From martin at v.loewis.de Wed Dec 1 00:06:47 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 01 Dec 2010 00:06:47 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87r5e236hj.fsf@benfinney.id.au> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> Message-ID: <4CF58387.1000609@v.loewis.de> Am 30.11.2010 21:24, schrieb Ben Finney: > haiyang kang writes: > >> I think it is a little ugly to have code like this: num = >> float("?.?"), expected result is: num = 1.1 > > That's a straw man, though. The string need not be a literal in the > program; it can be input to the program. > > num = float(input_from_the_external_world) > > Does that change your assessment of whether non-ASCII digits are used? I think the OP (haiyang kang) already indicated that he finds it quite unlikely that anybody would possibly want to enter that. You would need a number of key strokes to enter each individual ideograph, plus you have to press the keys for keyboard layout switching to enter the Latin decimal separator (which you normally wouldn't use along with the Han numerals). Regards, Martin From martin at v.loewis.de Wed Dec 1 00:09:18 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 01 Dec 2010 00:09:18 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4CF5841E.6020401@v.loewis.de> Am 30.11.2010 23:43, schrieb Terry Reedy: > On 11/30/2010 3:23 AM, Stephen J. Turnbull wrote: > >> I see no reason not to make a similar promise for numeric literals. I >> see no good reason to allow compatibility full-width Japanese "ASCII" >> numerals or Arabic cursive numerals in "for i in range(...)" for >> example. > > I do not think that anyone, at least not me, has argued for anything > other than 0-9 digits (or 0-f for hex) in literals in program code. The > only issue is whether non-programmer *users* should be able to use their > native digits in applications in response to input prompts. And here, my observation stands: if they wanted to, they currently couldn't - at least not for real numbers (and also not for integers if they want to use grouping). So the presumed application of this feature doesn't actually work, despite the presence of the feature it was supposedly meant to enable. Regards, Martin From tjreedy at udel.edu Wed Dec 1 00:19:30 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 30 Nov 2010 18:19:30 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2F067.5020705@pearwood.info> <4CF354C6.9020302@v.loewis.de> <20101129193302.115dbcd5@pitrou.net> <4CF41785.5020807@v.loewis.de> Message-ID: On 11/30/2010 10:05 AM, Alexander Belopolsky wrote: My general answers to the questions you have raised are as follows: 1. Each new feature release should use the latest version of the UCD as of the first beta release (or perhaps a week or so before). New chars are new features and the beta period can be used to (hopefully) iron out any bugs introduced by a new UCD version. 2. The language specification should not be UCD version specific. Martin pointed out that the definition of identifiers was intentionally written to not be, bu referring to 'current version' or some such. On the other hand, the UCD version used should be programatically discoverable, perhaps as an attribute of sys or str. 3.. The UCD should not change in bugfix releases. New chars are new features. Adding them in bugfix releases will introduce gratuitous imcompatibilities between releases. People who want the latest Unicode should either upgrade to the latest Python version or patch an older version (but not expect core support for any problems that creates). > Given that 2.7 will be maintained for 5 years and arguably Unicode > Consortium takes backward compatibility very seriously, wouldn't it > make sense to consider a backport at some point? > > I am sure we will soon see a bug report that the following does not > work in 2.7: :-) >>>> ord('\N{CAT FACE WITH WRY SMILE}') > 128572 3 (cont). 2.7 is no different in that regard. It is feature frozen just like all other x.y releases. And that is the answer to any such report. If that code became valid in 2.7.2, for instance, it would still not work in 2.7 and 2.7.1. Not working is not a bug; working is a new feature introduced after 2.7 was released. >>> - How specific should library reference manual be in defining methods >>> affected by UCD such as str.upper()? >> >> It should specify what this actually does in Unicode terminology >> (probably in addition to a layman's rephrase of that) >> > > I opened an issue for this: > > http://bugs.python.org/issue10587 1,2 (cont). Good idea in general. > I was more concerned about wide an narrow unicode CPython builds. Is > it a bug that '\UXXXXXXXX'.isalpha() may disagree even when the two > implementations are based on the same version of UCD? 4. While the difference between narrow/wide builds of (CPython) x.y (which should have once constant UCD) cannot be completely masked, I appreciate and generally agree with your efforts to minimize them. In some cases, there will be a conflict/tradeoff between eliminating this difference versus that. -- Terry Jan Reedy From ben+python at benfinney.id.au Wed Dec 1 01:13:23 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Wed, 01 Dec 2010 11:13:23 +1100 Subject: [Python-Dev] Python and the Unicode Character Database References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> Message-ID: <87k4ju2vvg.fsf@benfinney.id.au> "Martin v. L?wis" writes: > Am 30.11.2010 21:24, schrieb Ben Finney: > > The string need not be a literal in the program; it can be input to > > the program. > > > > num = float(input_from_the_external_world) > > > > Does that change your assessment of whether non-ASCII digits are > > used? > > I think the OP (haiyang kang) already indicated that he finds it quite > unlikely that anybody would possibly want to enter that. Who's talking about *entering* it into the program at a keyboard directly, though? Input to a program can come from all kinds of crazy sources. Just because it wasn't typed by the person at the keyboard using this program doesn't stop it being input to the program. A concrete example, but certainly not the only possible case: non-ASCII digit characters representing integers, stored as text in a file. Note that I'm not saying this is common. Nor am I saying it's a desirable situation. I'm saying it is a feasible use case, to be dismissed only if there is strong evidence that it's not used by existing Python code. -- \ ?When a well-packaged web of lies has been sold to the masses | `\ over generations, the truth will seem utterly preposterous and | _o__) its speaker a raving lunatic.? ?Dresden James | Ben Finney From stutzbach at google.com Wed Dec 1 02:11:30 2010 From: stutzbach at google.com (Daniel Stutzbach) Date: Tue, 30 Nov 2010 17:11:30 -0800 Subject: [Python-Dev] I/O ABCs Message-ID: The documentation for the collections Abstract Base Classes (ABCs) [1] contains a table listing all of the collections ABCs, their parent classes, their abstract methods, and the methods they provide. This table makes it very easy to figure out which methods I must override when I derive from one of the ABCs, as well as which methods will be provided for me. I'm working on a similar table for the I/O ABCs ( http://bugs.python.org/issue10589). The existing documentation [2] describes the methods of each class but doesn't describe which methods provide a meaningful implementation and which methods a user should override. If I want to inherit from one of the I/O ABCs, I have to go poking into Lib/_pyio.py to figure out which methods I need to override. While starting to examine the I/O ABCs, I discovered that there are some inconsistencies. For example, RawIOBase provides .read() if the subclass provides .readinto(). BufferedIOBase does the opposite; it provides .readinto() if the subclass provides .read() [3]. I would like to fix some of these inconsistencies. However, this will be a backwards-incompatible change. A Google Code Search suggests that the ABCs are currently only used within the standard library [4]. Just to be clear, the changes would NOT impact code that merely uses I/O objects; they would only impact code that implements I/O by subclassing one of the I/O ABCs and depending on features that are currently undocumented. Does anyone have any categorical objections? [1]: http://docs.python.org/py3k/library/collections.html#abcs-abstract-base-classes [2]: http://docs.python.org/py3k/library/io.html#class-hierarchy [3]: Possibly hurting performance by forcing .readinto() to perform the extra allocations, copies, and deallocations required by .read(). [4]: http://www.google.com/codesearch?hl=en&sa=N&q=BufferedIOBase++lang:python&ct=rr&cs_r=lang:python -- Daniel Stutzbach -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Dec 1 02:19:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Dec 2010 11:19:52 +1000 Subject: [Python-Dev] python3k : imp.find_module raises SyntaxError In-Reply-To: References: <201011251530.23947.emile.anclin@logilab> <4CEE9B72.1070002@ronadam.com> <20101129115311.GD18888@lupus.logilab.fr> Message-ID: On Wed, Dec 1, 2010 at 8:48 AM, Ron Adam wrote: > * It almost seems like the concept of a sub-module (in a package) is flawed. > ?I'm not sure I can explain what causes me to feel that way at the moment > though. It isn't flawed, it is just a *lot* more complicated than most people realise (cf. PEP 302). In this case, the signature of find_module (returning an already open file) is unfortunate, but probably necessary given the way the import internals currently work. As Brett says, returning a loader would be preferable, but the builtin import machinery doesn't have proper loaders defined (and won't until we manage to get to the point where importlib *is* the import machinery). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From songofacandy at gmail.com Wed Dec 1 03:30:43 2010 From: songofacandy at gmail.com (INADA Naoki) Date: Wed, 1 Dec 2010 11:30:43 +0900 Subject: [Python-Dev] What is PyBuffer_SizeFromFormat? Message-ID: PyBuffer_SizeFromFormat is documented and defined in abstract.h. But I can't find an implementation of the function. Do I overlook anything? -- INADA Naoki? From ncoghlan at gmail.com Wed Dec 1 04:51:11 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Dec 2010 13:51:11 +1000 Subject: [Python-Dev] What is PyBuffer_SizeFromFormat? In-Reply-To: References: Message-ID: On Wed, Dec 1, 2010 at 12:30 PM, INADA Naoki wrote: > PyBuffer_SizeFromFormat is documented and defined in abstract.h. > But I can't find an implementation of the function. > Do I overlook anything? PEP 3118 describes what it is *meant* to do. Looks like it might be yet another thing that was missed in the implementation of that PEP though :P Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From rrr at ronadam.com Wed Dec 1 06:59:18 2010 From: rrr at ronadam.com (Ron Adam) Date: Tue, 30 Nov 2010 23:59:18 -0600 Subject: [Python-Dev] python3k : imp.find_module raises SyntaxError In-Reply-To: References: <201011251530.23947.emile.anclin@logilab> <4CEE9B72.1070002@ronadam.com> <20101129115311.GD18888@lupus.logilab.fr> Message-ID: <4CF5E436.3070503@ronadam.com> On 11/30/2010 07:19 PM, Nick Coghlan wrote: > On Wed, Dec 1, 2010 at 8:48 AM, Ron Adam wrote: >> * It almost seems like the concept of a sub-module (in a package) is flawed. >> I'm not sure I can explain what causes me to feel that way at the moment >> though. > > It isn't flawed, it is just a *lot* more complicated than most people > realise (cf. PEP 302). Yes, it's realising that it is a *lot* more *complicated*, that gets me. Flawed isn't the right word, it's rather a feeling things could have been simpler if perhaps some things were done differently. Here is the gist of ideas I got from these feelings. (Food for thought and YMMV and all that.) Python doesn't have a nice way to define a collection of modules that isn't also a package. So we have packages used to organise modules, and packages inside other packages. A collection of modules wouldn't require importing a package before importing a module in it. Another idea is, to have a way to split a large module into files, and have it still *be* a module, and not a package. And also be able to tell what is what, by looking at the directory structure. The train of thought these things came from is, how can we get back to having the directory tree have enough info in it so it's clear what is what? And how can we avoid some of the *interdependent* nesting? > In this case, the signature of find_module (returning an already open > file) is unfortunate, but probably necessary given the way the import > internals currently work. As Brett says, returning a loader would be > preferable, but the builtin import machinery doesn't have proper > loaders defined (and won't until we manage to get to the point where > importlib *is* the import machinery). I'll be looking forward to the new loaders. :-) Cheers, Ron From rrr at ronadam.com Wed Dec 1 06:59:18 2010 From: rrr at ronadam.com (Ron Adam) Date: Tue, 30 Nov 2010 23:59:18 -0600 Subject: [Python-Dev] python3k : imp.find_module raises SyntaxError In-Reply-To: References: <201011251530.23947.emile.anclin@logilab> <4CEE9B72.1070002@ronadam.com> <20101129115311.GD18888@lupus.logilab.fr> Message-ID: <4CF5E436.3070503@ronadam.com> On 11/30/2010 07:19 PM, Nick Coghlan wrote: > On Wed, Dec 1, 2010 at 8:48 AM, Ron Adam wrote: >> * It almost seems like the concept of a sub-module (in a package) is flawed. >> I'm not sure I can explain what causes me to feel that way at the moment >> though. > > It isn't flawed, it is just a *lot* more complicated than most people > realise (cf. PEP 302). Yes, it's realising that it is a *lot* more *complicated*, that gets me. Flawed isn't the right word, it's rather a feeling things could have been simpler if perhaps some things were done differently. Here is the gist of ideas I got from these feelings. (Food for thought and YMMV and all that.) Python doesn't have a nice way to define a collection of modules that isn't also a package. So we have packages used to organise modules, and packages inside other packages. A collection of modules wouldn't require importing a package before importing a module in it. Another idea is, to have a way to split a large module into files, and have it still *be* a module, and not a package. And also be able to tell what is what, by looking at the directory structure. The train of thought these things came from is, how can we get back to having the directory tree have enough info in it so it's clear what is what? And how can we avoid some of the *interdependent* nesting? > In this case, the signature of find_module (returning an already open > file) is unfortunate, but probably necessary given the way the import > internals currently work. As Brett says, returning a loader would be > preferable, but the builtin import machinery doesn't have proper > loaders defined (and won't until we manage to get to the point where > importlib *is* the import machinery). I'll be looking forward to the new loaders. :-) Cheers, Ron From ncoghlan at gmail.com Wed Dec 1 08:41:07 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Dec 2010 17:41:07 +1000 Subject: [Python-Dev] python3k : imp.find_module raises SyntaxError In-Reply-To: <4CF5E436.3070503@ronadam.com> References: <201011251530.23947.emile.anclin@logilab> <4CEE9B72.1070002@ronadam.com> <20101129115311.GD18888@lupus.logilab.fr> <4CF5E436.3070503@ronadam.com> Message-ID: On Wed, Dec 1, 2010 at 3:59 PM, Ron Adam wrote: > Yes, it's realising that it is a *lot* more *complicated*, that gets me. > Flawed isn't the right word, it's rather a feeling things could have been > simpler if perhaps some things were done differently. *That* feeling I can understand. The import system has steadily acquired features over time, with each addition constrained by backwards compatibility concerns with all the past additions, including the exotic hacks people were using to fake features that were added more cleanly later. For the directory-as-module-not-package idea, you could probably implement a PEP 302 importer/loader that did that (independent of the stdlib). It would have the advantage of avoiding a lot of the pickle compatibility problems that a "flat package" like the new unittest layout can cause. However, you would need to be very careful with it, since all the files would be sharing a common globals() namespace. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From mal at egenix.com Wed Dec 1 10:15:57 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 01 Dec 2010 10:15:57 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2F067.5020705@pearwood.info> <4CF354C6.9020302@v.loewis.de> <20101129193302.115dbcd5@pitrou.net> <4CF41785.5020807@v.loewis.de> Message-ID: <4CF6124D.6000904@egenix.com> Terry Reedy wrote: > On 11/30/2010 10:05 AM, Alexander Belopolsky wrote: > > My general answers to the questions you have raised are as follows: > > 1. Each new feature release should use the latest version of the UCD as > of the first beta release (or perhaps a week or so before). New chars > are new features and the beta period can be used to (hopefully) iron out > any bugs introduced by a new UCD version. The UCD is versioned just like Python is, so if the Unicode Consortium decides to ship a 5.2.1 version of the UCD, we can add that to Python 2.7.x, since Python 2.7 started out with 5.2.0. > 2. The language specification should not be UCD version specific. Martin > pointed out that the definition of identifiers was intentionally written > to not be, bu referring to 'current version' or some such. On the other > hand, the UCD version used should be programatically discoverable, > perhaps as an attribute of sys or str. It already is and has been for while, e.g. Python 2.5: >>> import unicodedata >>> unicodedata.unidata_version '4.1.0' > 3.. The UCD should not change in bugfix releases. New chars are new > features. Adding them in bugfix releases will introduce gratuitous > imcompatibilities between releases. People who want the latest Unicode > should either upgrade to the latest Python version or patch an older > version (but not expect core support for any problems that creates). See above. Patch level revisions of the UCD are fine for patch level releases of Python, since those patch level revisions of the UCD fix bugs just like we do in Python. Note that each new UCD major.minor version is a new standard on its own, so it's perfectly ok to stick with one such standard version per Python version. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 01 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Dec 1 10:28:10 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 01 Dec 2010 10:28:10 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF58387.1000609@v.loewis.de> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> Message-ID: <4CF6152A.3050400@egenix.com> "Martin v. L?wis" wrote: > Am 30.11.2010 21:24, schrieb Ben Finney: >> haiyang kang writes: >> >>> I think it is a little ugly to have code like this: num = >>> float("?.?"), expected result is: num = 1.1 >> >> That's a straw man, though. The string need not be a literal in the >> program; it can be input to the program. >> >> num = float(input_from_the_external_world) >> >> Does that change your assessment of whether non-ASCII digits are used? > > I think the OP (haiyang kang) already indicated that he finds it quite > unlikely that anybody would possibly want to enter that. You would need > a number of key strokes to enter each individual ideograph, plus you > have to press the keys for keyboard layout switching to enter the Latin > decimal separator (which you normally wouldn't use along with the Han > numerals). That's a somewhat limited view, IMHO. Numbers are not always entered using a computer keyboard, you have tool like cash registries, special numeric keypads, scanners, OCR, etc. for external entry, and you also have other programs producing such output, e.g. MS Office if configured that way. The argument with the decimal point doesn't work well either, since it's obvious that float() and int() do not support localized input. E.g. in Germany we write 3,141 instead of 3.141: >>> float('3,141') Traceback (most recent call last): File "", line 1, in ValueError: invalid literal for float(): 3,141 No surprise there. The localization of the input data, e.g. removal of thousands separators and conversion of decimal marks to the dot, have to be done by the application, just like you have to now for German floating point number literals. The locale module already has locale.atof() and locale.atoi() for just this purpose. FYI, here's a list of decimal digits supported by Python 2.7: http://www.unicode.org/Public/5.2.0/ucd/extracted/DerivedNumericType.txt: """ 0030..0039 ; Decimal # Nd [10] DIGIT ZERO..DIGIT NINE 0660..0669 ; Decimal # Nd [10] ARABIC-INDIC DIGIT ZERO..ARABIC-INDIC DIGIT NINE 06F0..06F9 ; Decimal # Nd [10] EXTENDED ARABIC-INDIC DIGIT ZERO..EXTENDED ARABIC-INDIC DIGIT NINE 07C0..07C9 ; Decimal # Nd [10] NKO DIGIT ZERO..NKO DIGIT NINE 0966..096F ; Decimal # Nd [10] DEVANAGARI DIGIT ZERO..DEVANAGARI DIGIT NINE 09E6..09EF ; Decimal # Nd [10] BENGALI DIGIT ZERO..BENGALI DIGIT NINE 0A66..0A6F ; Decimal # Nd [10] GURMUKHI DIGIT ZERO..GURMUKHI DIGIT NINE 0AE6..0AEF ; Decimal # Nd [10] GUJARATI DIGIT ZERO..GUJARATI DIGIT NINE 0B66..0B6F ; Decimal # Nd [10] ORIYA DIGIT ZERO..ORIYA DIGIT NINE 0BE6..0BEF ; Decimal # Nd [10] TAMIL DIGIT ZERO..TAMIL DIGIT NINE 0C66..0C6F ; Decimal # Nd [10] TELUGU DIGIT ZERO..TELUGU DIGIT NINE 0CE6..0CEF ; Decimal # Nd [10] KANNADA DIGIT ZERO..KANNADA DIGIT NINE 0D66..0D6F ; Decimal # Nd [10] MALAYALAM DIGIT ZERO..MALAYALAM DIGIT NINE 0E50..0E59 ; Decimal # Nd [10] THAI DIGIT ZERO..THAI DIGIT NINE 0ED0..0ED9 ; Decimal # Nd [10] LAO DIGIT ZERO..LAO DIGIT NINE 0F20..0F29 ; Decimal # Nd [10] TIBETAN DIGIT ZERO..TIBETAN DIGIT NINE 1040..1049 ; Decimal # Nd [10] MYANMAR DIGIT ZERO..MYANMAR DIGIT NINE 1090..1099 ; Decimal # Nd [10] MYANMAR SHAN DIGIT ZERO..MYANMAR SHAN DIGIT NINE 17E0..17E9 ; Decimal # Nd [10] KHMER DIGIT ZERO..KHMER DIGIT NINE 1810..1819 ; Decimal # Nd [10] MONGOLIAN DIGIT ZERO..MONGOLIAN DIGIT NINE 1946..194F ; Decimal # Nd [10] LIMBU DIGIT ZERO..LIMBU DIGIT NINE 19D0..19DA ; Decimal # Nd [11] NEW TAI LUE DIGIT ZERO..NEW TAI LUE THAM DIGIT ONE 1A80..1A89 ; Decimal # Nd [10] TAI THAM HORA DIGIT ZERO..TAI THAM HORA DIGIT NINE 1A90..1A99 ; Decimal # Nd [10] TAI THAM THAM DIGIT ZERO..TAI THAM THAM DIGIT NINE 1B50..1B59 ; Decimal # Nd [10] BALINESE DIGIT ZERO..BALINESE DIGIT NINE 1BB0..1BB9 ; Decimal # Nd [10] SUNDANESE DIGIT ZERO..SUNDANESE DIGIT NINE 1C40..1C49 ; Decimal # Nd [10] LEPCHA DIGIT ZERO..LEPCHA DIGIT NINE 1C50..1C59 ; Decimal # Nd [10] OL CHIKI DIGIT ZERO..OL CHIKI DIGIT NINE A620..A629 ; Decimal # Nd [10] VAI DIGIT ZERO..VAI DIGIT NINE A8D0..A8D9 ; Decimal # Nd [10] SAURASHTRA DIGIT ZERO..SAURASHTRA DIGIT NINE A900..A909 ; Decimal # Nd [10] KAYAH LI DIGIT ZERO..KAYAH LI DIGIT NINE A9D0..A9D9 ; Decimal # Nd [10] JAVANESE DIGIT ZERO..JAVANESE DIGIT NINE AA50..AA59 ; Decimal # Nd [10] CHAM DIGIT ZERO..CHAM DIGIT NINE ABF0..ABF9 ; Decimal # Nd [10] MEETEI MAYEK DIGIT ZERO..MEETEI MAYEK DIGIT NINE FF10..FF19 ; Decimal # Nd [10] FULLWIDTH DIGIT ZERO..FULLWIDTH DIGIT NINE 104A0..104A9 ; Decimal # Nd [10] OSMANYA DIGIT ZERO..OSMANYA DIGIT NINE 1D7CE..1D7FF ; Decimal # Nd [50] MATHEMATICAL BOLD DIGIT ZERO..MATHEMATICAL MONOSPACE DIGIT NINE """ The Chinese and Japanese ideographs are not supported because of the way they are defined in the Unihan database. I'm currently investigating how we could support them as well. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 01 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Wed Dec 1 10:29:26 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 01 Dec 2010 10:29:26 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4CF61576.8030607@egenix.com> Terry Reedy wrote: > On 11/30/2010 3:23 AM, Stephen J. Turnbull wrote: > >> I see no reason not to make a similar promise for numeric literals. I >> see no good reason to allow compatibility full-width Japanese "ASCII" >> numerals or Arabic cursive numerals in "for i in range(...)" for >> example. > > I do not think that anyone, at least not me, has argued for anything > other than 0-9 digits (or 0-f for hex) in literals in program code. The > only issue is whether non-programmer *users* should be able to use their > native digits in applications in response to input prompts. Me neither. This is solely about Python being able to parse numeric input in the float(), int() and complex() constructors. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 01 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From steve at pearwood.info Wed Dec 1 10:47:45 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 01 Dec 2010 20:47:45 +1100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF5841E.6020401@v.loewis.de> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> Message-ID: <4CF619C1.8030403@pearwood.info> Martin v. L?wis wrote: > Am 30.11.2010 23:43, schrieb Terry Reedy: >> On 11/30/2010 3:23 AM, Stephen J. Turnbull wrote: >> >>> I see no reason not to make a similar promise for numeric literals. I >>> see no good reason to allow compatibility full-width Japanese "ASCII" >>> numerals or Arabic cursive numerals in "for i in range(...)" for >>> example. >> I do not think that anyone, at least not me, has argued for anything >> other than 0-9 digits (or 0-f for hex) in literals in program code. The >> only issue is whether non-programmer *users* should be able to use their >> native digits in applications in response to input prompts. > > And here, my observation stands: if they wanted to, they currently > couldn't - at least not for real numbers (and also not for integers > if they want to use grouping). So the presumed application of this > feature doesn't actually work, despite the presence of the feature it > was supposedly meant to enable. By that argument, English speakers wanting to enter integers using Arabic numerals can't either! I'd like to use grouping for large literals, if only I could think of a half-decent syntax, and if only Python supported it. This fails on both counts: x = 123_456_789_012_345 The lack of grouping and the lack of a native decimal point doesn't mean that the feature "doesn't work" -- it merely means the feature requires some compromise before it can be used. In the same way, if I wanted to enter a number using non-Arabic digits, it works provided I compromise by using the Anglo-American decimal point instead of the European comma or the native decimal point I might prefer. The lack of support for non-dot decimal points is arguably a bug that should be fixed, not a reason to remove functionality. -- Steven From greg.ewing at canterbury.ac.nz Wed Dec 1 11:22:42 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 01 Dec 2010 23:22:42 +1300 Subject: [Python-Dev] python3k : imp.find_module raises SyntaxError In-Reply-To: References: <201011251530.23947.emile.anclin@logilab> <4CEE9B72.1070002@ronadam.com> <20101129115311.GD18888@lupus.logilab.fr> <4CF5E436.3070503@ronadam.com> Message-ID: <4CF621F2.3060203@canterbury.ac.nz> Nick Coghlan wrote: > For the directory-as-module-not-package idea ... > you would need to be very careful with it, > since all the files would be sharing a common globals() namespace. One of the things I like about Python's module system is that once I know which module a name was imported from, I also know which file to look in for its definition. If a module can be spread over several files, that feature would be lost. -- Greg From ncoghlan at gmail.com Wed Dec 1 11:39:47 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 Dec 2010 20:39:47 +1000 Subject: [Python-Dev] python3k : imp.find_module raises SyntaxError In-Reply-To: <4CF621F2.3060203@canterbury.ac.nz> References: <201011251530.23947.emile.anclin@logilab> <4CEE9B72.1070002@ronadam.com> <20101129115311.GD18888@lupus.logilab.fr> <4CF5E436.3070503@ronadam.com> <4CF621F2.3060203@canterbury.ac.nz> Message-ID: On Wed, Dec 1, 2010 at 8:22 PM, Greg Ewing wrote: > Nick Coghlan wrote: > >> For the directory-as-module-not-package idea ... >> you would need to be very careful with it, >> since all the files would be sharing a common globals() namespace. > > One of the things I like about Python's module system > is that once I know which module a name was imported > from, I also know which file to look in for its > definition. If a module can be spread over several > files, that feature would be lost. There are many potential problems with the idea, I just chose to mention one of the ones that could easily make the affected code *break* :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From regebro at gmail.com Wed Dec 1 13:58:01 2010 From: regebro at gmail.com (Lennart Regebro) Date: Wed, 1 Dec 2010 13:58:01 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: On Tue, Nov 30, 2010 at 09:23, Stephen J. Turnbull wrote: > Sure you can. ?In Python program text, all keywords will be ASCII Yes, yes, sure, but not the contents of variables, > I see no reason not to make a similar promise for numeric literals. Wait what, literas? The example was >>> float('????.??') Which doesn't have any numeric literals in them at all. Do that work? Nope, it's a syntax error. Too badm that would have been cool, but whatever. Why would this be a problem: >>> T1234 = float('????.??') >>> T1234 1234.56 But this OK? >>> T???? = float('1234.56') >>> T???? 1234.56 I don't see that. Should we bother to implement ????.?? as a literal equivalent to 1234.56? Well, not unless somebody askes for it, or it turns out to be easy. :-) But that's another question. From rrr at ronadam.com Wed Dec 1 16:20:12 2010 From: rrr at ronadam.com (Ron Adam) Date: Wed, 01 Dec 2010 09:20:12 -0600 Subject: [Python-Dev] python3k : imp.find_module raises SyntaxError In-Reply-To: References: <201011251530.23947.emile.anclin@logilab> <4CEE9B72.1070002@ronadam.com> <20101129115311.GD18888@lupus.logilab.fr> <4CF5E436.3070503@ronadam.com> <4CF621F2.3060203@canterbury.ac.nz> Message-ID: On 12/01/2010 04:39 AM, Nick Coghlan wrote: > On Wed, Dec 1, 2010 at 8:22 PM, Greg Ewing wrote: >> Nick Coghlan wrote: >> >>> For the directory-as-module-not-package idea ... >>> you would need to be very careful with it, >>> since all the files would be sharing a common globals() namespace. >> >> One of the things I like about Python's module system >> is that once I know which module a name was imported >> from, I also know which file to look in for its >> definition. If a module can be spread over several >> files, that feature would be lost. > > There are many potential problems with the idea, I just chose to > mention one of the ones that could easily make the affected code > *break* :) Right. It would require additional pieces as well. Ron :-) From alexander.belopolsky at gmail.com Wed Dec 1 18:55:28 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 1 Dec 2010 12:55:28 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF2DC5B.4020702@egenix.com> References: <4CF2DC5B.4020702@egenix.com> Message-ID: On Sun, Nov 28, 2010 at 5:48 PM, M.-A. Lemburg wrote: .. >> With Python 3.1: >> >>>>> exec('\u0CF1 = 1') >> Traceback (most recent call last): >> ?File "", line 1, in >> ?File "", line 1 >> ? ?? = 1 >> ? ? ?^ >> SyntaxError: invalid character in identifier >> >> but with Python 3.2a4: >> >>>>> exec('\u0CF1 = 1') >>>>> eval('\u0CF1') >> 1 > > Such changes are not new, but I agree that they should probably > be highlighted in the "What's new in Python x.x". > As of today, "What?s New In Python 3.2" [1] does not even mention the unicodedata upgrade to 6.0.0. Here are the features form the unicode.org summary [2] that I think should be reflected in Python's "What's New" document: """ * adds 2,088 characters, including over 1,000 additional symbols?chief among them the additional emoji symbols, which are especially important for mobile phones; * corrects character properties for existing characters including - a general category change to two Kannada characters (U+0CF1, U+0CF2), which has the effect of making them newly eligible for inclusion in identifiers; - a general category change to one New Tai Lue numeric character (U+19DA), which would have the effect of disqualifying it from inclusion in identifiers unless grandfathering measures are in place for the defining identifier syntax. """ The above may be too verbose for inclusion to "What?s New In Python 3.2", but I think we should add a possibly shorter summary with a link to unicode.org for details. PS: Yes, I think everyone should know about the Python 3.2 killer feature: ('\N{CAT FACE WITH WRY SMILE}'! [1] http://docs.python.org/dev/whatsnew/3.2.html [2] http://www.unicode.org/versions/Unicode6.0.0/ From tjreedy at udel.edu Wed Dec 1 19:29:06 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 01 Dec 2010 13:29:06 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <4CF2DC5B.4020702@egenix.com> Message-ID: On 12/1/2010 12:55 PM, Alexander Belopolsky wrote: > On Sun, Nov 28, 2010 at 5:48 PM, M.-A. Lemburg wrote: > .. >>> With Python 3.1: >>> >>>>>> exec('\u0CF1 = 1') >>> Traceback (most recent call last): >>> File "", line 1, in >>> File "", line 1 >>> ? = 1 >>> ^ >>> SyntaxError: invalid character in identifier >>> >>> but with Python 3.2a4: >>> >>>>>> exec('\u0CF1 = 1') >>>>>> eval('\u0CF1') >>> 1 >> >> Such changes are not new, but I agree that they should probably >> be highlighted in the "What's new in Python x.x". >> > > As of today, "What?s New In Python 3.2" [1] does not even mention the > unicodedata upgrade to 6.0.0. Here are the features form the > unicode.org summary [2] that I think should be reflected in Python's > "What's New" document: > > """ > * adds 2,088 characters, including over 1,000 additional symbols?chief > among them the additional emoji symbols, which are especially > important for mobile phones; > > * corrects character properties for existing characters including > - a general category change to two Kannada characters (U+0CF1, > U+0CF2), which has the effect of making them newly eligible for > inclusion in identifiers; > > - a general category change to one New Tai Lue numeric character > (U+19DA), which would have the effect of disqualifying it from > inclusion in identifiers unless grandfathering measures are in place > for the defining identifier syntax. > """ > The above may be too verbose for inclusion to "What?s New In Python > 3.2", I think those 11 lines are pretty good. Put them in ('\N{CAT FACE WITH WRY SMILE}'! Plus give a link to Unicode site (Issue numbers are implicit links). -- Terry Jan Reedy From contactprashantat at gmail.com Wed Dec 1 19:51:03 2010 From: contactprashantat at gmail.com (Prashant Kumar) Date: Thu, 2 Dec 2010 00:21:03 +0530 Subject: [Python-Dev] Porting Ideas Message-ID: Hello everyone. My name is Prashant. I and my friend Zubin recently ported 'Configobj'. It would be great if somebody can suggest about any utilities or scripts that are being widely used and need to be ported. From brian.curtin at gmail.com Wed Dec 1 20:02:00 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 1 Dec 2010 13:02:00 -0600 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: Message-ID: On Wed, Dec 1, 2010 at 12:51, Prashant Kumar wrote: > Hello everyone. My name is Prashant. I and my friend Zubin recently > ported 'Configobj'. It would be great if somebody can suggest about > any utilities or scripts that are being widely used and need to be > ported. http://onpython3yet.com/ might be helpful to you. It orders the projects on PyPI with the most dependencies which are not yet ported to 3.x. Note that there are a number of false positives, e.g., the first result -- NumPy, since people don't seem to keep their classifiers up-to-date. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chef at ghum.de Wed Dec 1 20:17:08 2010 From: chef at ghum.de (Massa, Harald Armin) Date: Wed, 1 Dec 2010 20:17:08 +0100 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: Message-ID: > > http://onpython3yet.com/ might be helpful to you. It orders the projects > on PyPI with the most dependencies which are not yet ported to 3.x. > > Note that there are a number of false positives, e.g., the first result -- > NumPy, since people don't seem to keep their classifiers up-to-date. > > That could be a nice list. But quite disturbing content, as "Python, the programming language" is stated as "not being ported to 3.0". Does not really provoke trust. Harald -- GHUM GmbH Harald Armin Massa Spielberger Stra?e 49 70435 Stuttgart 0173/9409607 Amtsgericht Stuttgart, HRB 734971 - persuadere. et programmare -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Dec 1 20:17:16 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 1 Dec 2010 20:17:16 +0100 Subject: [Python-Dev] Porting Ideas References: Message-ID: <20101201201716.4357a316@pitrou.net> On Wed, 1 Dec 2010 13:02:00 -0600 Brian Curtin wrote: > On Wed, Dec 1, 2010 at 12:51, Prashant Kumar wrote: > > > Hello everyone. My name is Prashant. I and my friend Zubin recently > > ported 'Configobj'. It would be great if somebody can suggest about > > any utilities or scripts that are being widely used and need to be > > ported. > > http://onpython3yet.com/ might be helpful to you. It orders the projects on > PyPI with the most dependencies which are not yet ported to 3.x. I don't know who did that page but it seems like there's some FUD there. simplejson, ctypes, pysqlite and others are available in the 3.x stdlib. Mercurial is a command-line tool and doesn't need to be ported to be used for Python 3 projects. setuptools is supplanted by distribute which should Python 3 compatible. And I'm not sure what this package called "Python" is (?a high-level object-oriented programming language?? like Java?), but I'm pretty sure I've heard there's a Python 3 compatible version. (granted, it's probably less FUD than stupid automation) Regards Antoine. From brian.curtin at gmail.com Wed Dec 1 20:39:43 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Wed, 1 Dec 2010 13:39:43 -0600 Subject: [Python-Dev] Porting Ideas In-Reply-To: <20101201201716.4357a316@pitrou.net> References: <20101201201716.4357a316@pitrou.net> Message-ID: On Wed, Dec 1, 2010 at 13:17, Antoine Pitrou wrote: > On Wed, 1 Dec 2010 13:02:00 -0600 > Brian Curtin wrote: > > On Wed, Dec 1, 2010 at 12:51, Prashant Kumar < > contactprashantat at gmail.com>wrote: > > > > > Hello everyone. My name is Prashant. I and my friend Zubin recently > > > ported 'Configobj'. It would be great if somebody can suggest about > > > any utilities or scripts that are being widely used and need to be > > > ported. > > > > http://onpython3yet.com/ might be helpful to you. It orders the projects > on > > PyPI with the most dependencies which are not yet ported to 3.x. > > I don't know who did that page but it seems like there's some FUD there. > > simplejson, ctypes, pysqlite and others are available in the 3.x > stdlib. It grabs the info from their PyPI pages, which are probably not kept up-to-date. This was brought up at a local user group meeting and I think it can be a useful tool, but as you can see it requires good input data which isn't always the case for some packages. Package authors: if you spent time making your project work on 3.x -- let the world know, update your classifiers. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Dec 1 21:23:03 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 01 Dec 2010 15:23:03 -0500 Subject: [Python-Dev] Deprecating undocumented, unused functions in difflib. Message-ID: Difflib.SequenceMatcher object currently get two feature attributes: self.isbjunk = junk.__contains__ self.isbpopular = popular.__contains__ Tim Peters agrees that the junk and popular sets should be directly exposed and documented as part of the api, thereby making the functions redundant. The two functions are not currently documented (and should not be now). A google codesearch of 'isbjunk' and 'isbpopular' only returns hits in difflib.py itself (and its predecessor, ndiff.py). It would be easiest to just remove the two lines above. Or should I define functions _xxx names that issue a deprecation warning and attach them as attributes to each object? (Defining instance methods would not be the same). There is only one internal use of one of the two functions which is easily edited. -- Terry Jan Reedy From ddasilva at umd.edu Wed Dec 1 22:15:04 2010 From: ddasilva at umd.edu (Daniel da Silva) Date: Wed, 1 Dec 2010 16:15:04 -0500 Subject: [Python-Dev] [Preview] Comments and change proposals on documentation In-Reply-To: References: <4CF18220.7000202@pearwood.info> <4CF19B3C.2000308@pearwood.info> Message-ID: I think it looks great.! If you are looking for some suggestions to make it a little more elegant: 1. If I delete a comment that has no children, it should remove it completely (currently, it just replaces it with [deleted]). If there are children, I think it is doing the right thing. 2. When I post a comment, it should automatically vote that comment up. I wouldn't have posted it if I didn't like it. 3. As far as text formatting, I personally think there should be some hilighting support for code spans/blocks (IMO that should match the idle colors). Also, I seemed to manage to trigger a visible system warning in my badly formatted comment on math.fabs(x), :) -Daniel -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Wed Dec 1 23:32:25 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 01 Dec 2010 23:32:25 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF619C1.8030403@pearwood.info> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> Message-ID: <4CF6CCF9.6000100@v.loewis.de> >> And here, my observation stands: if they wanted to, they currently >> couldn't - at least not for real numbers (and also not for integers >> if they want to use grouping). So the presumed application of this >> feature doesn't actually work, despite the presence of the feature it >> was supposedly meant to enable. > > By that argument, English speakers wanting to enter integers using > Arabic numerals can't either! That's correct, and the key point here for the argument. It's just not *meant* to support localized number forms, but deliberately constrains them to a formal grammar which users using it must be aware of in order to use it. > I'd like to use grouping for large > literals, if only I could think of a half-decent syntax, and if only > Python supported it. This fails on both counts: > > x = 123_456_789_012_345 Here you are confusing issues, though: this fragment uses the syntax of the Python programming language. Whether or not the syntax of the float() constructor arguments matches that syntax is also a subject of the debate. I take it that you speak in favor of the float syntax also being used for the float() constructor. > The lack of grouping and the lack of a native decimal point doesn't mean > that the feature "doesn't work" -- it merely means the feature requires > some compromise before it can be used. No, it means that the Python programming language syntax for floating point numbers just doesn't take local notation into account *at all*. This is not a flaw - it just means that this feature is non-existent. Now, for the float() constructor, some people in this thread have claimed that it *is* aimed at people who want to enter numbers in their local spellings. I claim that this feature either doesn't work, or is absent also. > In the same way, if I wanted to enter a number using non-Arabic digits, > it works provided I compromise by using the Anglo-American decimal point > instead of the European comma or the native decimal point I might prefer. Why would you want that, if, what you really wanted, could not be done. There certainly *is* a way to convert strings into floats, and there would be a way if that restricted itself to the digits 0..9. So it can't be the mere desire to convert strings to float that make you ask for non-ASCII digits. > The lack of support for non-dot decimal points is arguably a bug that > should be fixed, not a reason to remove functionality. I keep repeating my two concerns: a) if that was a feature, it is not specified at all in the documentation. In fact, the documentation was recently clarified to deny existence of that feature. b) fixing it will be much more difficult than you apparently think. Regards, Martin From martin at v.loewis.de Wed Dec 1 23:36:13 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 01 Dec 2010 23:36:13 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87k4ju2vvg.fsf@benfinney.id.au> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> Message-ID: <4CF6CDDD.4060109@v.loewis.de> >> I think the OP (haiyang kang) already indicated that he finds it quite >> unlikely that anybody would possibly want to enter that. > > Who's talking about *entering* it into the program at a keyboard > directly, though? Input to a program can come from all kinds of crazy > sources. Just because it wasn't typed by the person at the keyboard > using this program doesn't stop it being input to the program. I think haiyang kang claimed exactly that - it won't ever be input to a program. I trust him on that - and so should you, unless you have sufficient experience with the Chinese language and writing system. > Note that I'm not saying this is common. Nor am I saying it's a > desirable situation. I'm saying it is a feasible use case, to be > dismissed only if there is strong evidence that it's not used by > existing Python code. And indeed, for the Chinese numerals, we have such strong evidence. Regards, Martin From martin at v.loewis.de Wed Dec 1 23:39:17 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 01 Dec 2010 23:39:17 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <4CF2DC5B.4020702@egenix.com> Message-ID: <4CF6CE95.1060707@v.loewis.de> > As of today, "What?s New In Python 3.2" [1] does not even mention the > unicodedata upgrade to 6.0.0. One reason was that I was instructed not to change "What's New" a few years ago. Regards, Martin From martin at v.loewis.de Wed Dec 1 23:48:29 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 01 Dec 2010 23:48:29 +0100 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: Message-ID: <4CF6D0BD.6010204@v.loewis.de> Am 01.12.2010 20:02, schrieb Brian Curtin: > On Wed, Dec 1, 2010 at 12:51, Prashant Kumar > > wrote: > > Hello everyone. My name is Prashant. I and my friend Zubin recently > ported 'Configobj'. It would be great if somebody can suggest about > any utilities or scripts that are being widely used and need to be > ported. > > > http://onpython3yet.com/ might be helpful to you. Another such list is at http://www.python.org/3kpoll Regards, Martin From steve at pearwood.info Thu Dec 2 00:18:10 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 02 Dec 2010 10:18:10 +1100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF6CDDD.4060109@v.loewis.de> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> Message-ID: <4CF6D7B2.6030506@pearwood.info> Martin v. L?wis wrote: >>> I think the OP (haiyang kang) already indicated that he finds it quite >>> unlikely that anybody would possibly want to enter that. >> Who's talking about *entering* it into the program at a keyboard >> directly, though? Input to a program can come from all kinds of crazy >> sources. Just because it wasn't typed by the person at the keyboard >> using this program doesn't stop it being input to the program. > > I think haiyang kang claimed exactly that - it won't ever be input to a > program. I trust him on that - and so should you, unless you have > sufficient experience with the Chinese language and writing system. > >> Note that I'm not saying this is common. Nor am I saying it's a >> desirable situation. I'm saying it is a feasible use case, to be >> dismissed only if there is strong evidence that it's not used by >> existing Python code. > > And indeed, for the Chinese numerals, we have such strong evidence. With full respect to haiyang kang, hear-say from one person can hardly be described as "strong" evidence -- particularly, as Alexander Belopolsky pointed out, the use-case described isn't currently supported by Python. Given that what haiyang kang describes *can't* be done, the fact that people don't do it is hardly surprising -- nor is it a good reason for taking away functionality that does exist. -- Steven From alexander.belopolsky at gmail.com Thu Dec 2 00:32:32 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 1 Dec 2010 18:32:32 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF6CDDD.4060109@v.loewis.de> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> Message-ID: On Wed, Dec 1, 2010 at 5:36 PM, "Martin v. L?wis" wrote: .. >> Note that I'm not saying this is common. Nor am I saying it's a >> desirable situation. I'm saying it is a feasible use case, to be >> dismissed only if there is strong evidence that it's not used by >> existing Python code. > > And indeed, for the Chinese numerals, we have such strong evidence. > Indeed: it over 10 years that Python's int() accepted Arabic-Indic numerals, nobody has complained that it *did not* accept Chinese. From steve at pearwood.info Thu Dec 2 01:17:51 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 02 Dec 2010 11:17:51 +1100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF6CCF9.6000100@v.loewis.de> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> Message-ID: <4CF6E5AF.7080308@pearwood.info> Martin v. L?wis wrote: >>> And here, my observation stands: if they wanted to, they currently >>> couldn't - at least not for real numbers (and also not for integers >>> if they want to use grouping). So the presumed application of this >>> feature doesn't actually work, despite the presence of the feature it >>> was supposedly meant to enable. >> By that argument, English speakers wanting to enter integers using >> Arabic numerals can't either! > > That's correct, and the key point here for the argument. It's just not > *meant* to support localized number forms, but deliberately constrains > them to a formal grammar which users using it must be aware of in order > to use it. You're *agreeing* that English speakers can't enter integers using Arabic numerals? What do you think I'm doing when I do this? >>> int("1234") 1234 Ah wait... did you think I meant Arabic numerals in the sense of digits used by Arabs in Arabia? I meant Arabic numerals as opposed to Roman numerals. Sorry for the confusion. Your argument was that even though Python's int() supports many non-ASCII digits, the lack of grouping means that it "doesn't actually work". If that argument were correct, then it applies equally to ASCII digits as well. It's clearly nonsense to say that int("1234") "doesn't work" just because of the lack of grouping. It's equally nonsense to say that int("????") "doesn't work" because of the lack of grouping. [...] > I take it that you speak in favor of the float syntax also being used > for the float() constructor. I'm sorry, I don't understand what you mean here. I've repeatedly said that the syntax for numeric literals should remain constrained to the ASCII digits, as it currently is. n = ???? gives a SyntaxError, and I don't want to see that change. But I've also argued that the float constructor currently accepts non-ASCII strings: n = int("????") we should continue to support the existing behaviour. None of the arguments against it seem convincing to me, particularly since the opponents of the current behaviour admit that there is a use-case for it, but they just want it to move elsewhere, such as the locale module. We've even heard from one person -- I forget who, sorry -- who claimed that C++ has the same behaviour, and if you want ASCII-only digits, you have to explicitly ask for it. For what it's worth, Microsoft warns developers not to assume users will enter numeric data using ASCII digits: "Number representation can also use non-ASCII native digits, so your application may encounter characters other than 0-9 as inputs. Avoid filtering on U+0030 through U+0039 to prevent frustration for users who are trying to enter data using non-ASCII digits." http://msdn.microsoft.com/en-us/magazine/cc163506.aspx There was a similar discussion going on in Perl-land recently: http://www.nntp.perl.org/group/perl.perl5.porters/2010/07/msg162400.html although, being Perl, the discussion was dominated by concerns about regexes and implicit conversions, rather than an explicit call to float() or int() as we are discussing here. [...] >> In the same way, if I wanted to enter a number using non-Arabic digits, >> it works provided I compromise by using the Anglo-American decimal point >> instead of the European comma or the native decimal point I might prefer. > > Why would you want that, if, what you really wanted, could not be > done. There certainly *is* a way to convert strings into floats, > and there would be a way if that restricted itself to the digits 0..9. > So it can't be the mere desire to convert strings to float that make > you ask for non-ASCII digits. Why do Europeans use programming languages that force them to use a dot instead of a comma for the decimal place? Why do I misspell string.centre as string.center? Because if you want to get something done, you use the tools you have and not the tools you'd like to have. -- Steven From stephen at xemacs.org Thu Dec 2 01:31:54 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 02 Dec 2010 09:31:54 +0900 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87tyixxbet.fsf@uwakimon.sk.tsukuba.ac.jp> Lennart Regebro writes: > On Tue, Nov 30, 2010 at 09:23, Stephen J. Turnbull wrote: > > Sure you can. ?In Python program text, all keywords will be ASCII > > Yes, yes, sure, but not the contents of variables, Irrelevant, you're not converting these to a string representation. If you're generating numerals for internal use, I don't see why you would want to do arithmetic on them; conversion is a YAGNI. This is only interesting to allow naive users to input in a comfortable way. As yet there is no evidence that there are *any* such naive users, 1.3 billion of "possibles" are shut out, and at least two cultures which use non-ASCII numerals every day, representing 1.3 billion naive users (the coincidence of numbers is no coincidence), have reported that nobody in their right mind would would *input* the numbers that way, and at least for Japanese, the use cases are not really numeric anyway. > > I see no reason not to make a similar promise for numeric literals. > > Wait what, literas? Sorry, my bad. -------------- next part -------------- > Why would this be a problem: > > >>> T1234 = float('~~~~.~~') > >>> T1234 > 1234.56 > > But this OK? > > >>> T~~~~ = float('1234.56') > >>> T~~~~ > 1234.56 (Sorry, the Arabic is going to get munged, my mailer is beta and somebody screwed up.) Because the characters in the identifier are uninterpreted and have no syntactic content other than their identity. They're arbitrary. That's not true of numerics. Because that works, but print(T1234) doesn't (it prints ASCII). You can't round-trip, but users will want/expect that. Because that works but this doesn't: -------------- next part -------------- T1000 = float('?.???') Violates TOOWTDI. If you're proposing to fix the numeric parsers, I still don't like it but I could go to -0 on it. However as Alexander points out and MAL admits, it's apparently not so easy to do that. From alexander.belopolsky at gmail.com Thu Dec 2 01:44:28 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 1 Dec 2010 19:44:28 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF6E5AF.7080308@pearwood.info> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> Message-ID: On Wed, Dec 1, 2010 at 7:17 PM, Steven D'Aprano wrote: .. > we should continue to support the existing behaviour. None of the arguments > against it seem convincing to me, particularly since the opponents of the > current behaviour admit that there is a use-case for it, but they just want > it to move elsewhere, such as the locale module. > I don't remember who made this argument, but I think you misunderstood it. The argument was that if there was a use case for parsing Eastern Arabic numerals, it would be better served by a module written by someone who speaks one of the Arabic languages and knows the details of how Eastern Arabic numerals are written. So far nobody has even claimed to know conclusively that Arabic-Indic digits are always written left-to-right. >>> unicodedata.bidirectional('?') 'AN' is not very helpful because it means "any Arabic-Indic digit" according to unicode.org. (To me, a special category hints that it may be written in either direction and the proper interpretation may depend on context.) I have not seen a real use case reported in this thread and for theoretical use cases, the current implementation is either outright wrong or does not solve the problem completely. Given that a function that replaces all Unicode digits in a string with 0-9 can be written in 3 lines of Python code, it is very unlikely that anyone would prefer to rely on undocumented behavior of Python builtins instead of having explicit control over parsing of their data. From stephen at xemacs.org Thu Dec 2 01:45:55 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 02 Dec 2010 09:45:55 +0900 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF6D7B2.6030506@pearwood.info> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> Message-ID: <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > With full respect to haiyang kang, hear-say from one person can hardly > be described as "strong" evidence That's *disrespectful* nonsense. What Haiyang reported was not hearsay, it's direct observation of what he sees around him and personal experience, plus extrapolation. Look up "hearsay," please. Furthermore, he provided good *objective* reason (excessive cost, to which I can also testify, in several different input methods for Japanese) why numbers simply would not be input that way. What's left is copy/paste via the mouse. I assure you, every day I see dozens of Japanese copy/pasting *only* ASCII numerals, and the sales figures for Microsoft Excel (not to mention the download numbers for Open Office) strongly suggest that 30 million Japanese salarymen are similarly dedicated to ASCII. (That's not "hearsay" either, that's direct observation and extrapolation, which is more than the "we need float to translate Arabic" supporters can offer.) I have seen only *one* use case: it's a toy for sophisticated programmers who want to think of themselves as broadminded. We've seen several examples of that in this thread, so I can't deny that is a real use case. Please, give us just *one* more real use case that isn't "somebody might". From fuzzyman at voidspace.org.uk Thu Dec 2 02:17:43 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 02 Dec 2010 01:17:43 +0000 Subject: [Python-Dev] Porting Ideas In-Reply-To: <20101201201716.4357a316@pitrou.net> References: <20101201201716.4357a316@pitrou.net> Message-ID: <4CF6F3B7.1040805@voidspace.org.uk> On 01/12/2010 19:17, Antoine Pitrou wrote: > On Wed, 1 Dec 2010 13:02:00 -0600 > Brian Curtin wrote: >> On Wed, Dec 1, 2010 at 12:51, Prashant Kumarwrote: >> >>> Hello everyone. My name is Prashant. I and my friend Zubin recently >>> ported 'Configobj'. It would be great if somebody can suggest about >>> any utilities or scripts that are being widely used and need to be >>> ported. >> http://onpython3yet.com/ might be helpful to you. It orders the projects on >> PyPI with the most dependencies which are not yet ported to 3.x. > I don't know who did that page but it seems like there's some FUD there. > > simplejson, ctypes, pysqlite and others are available in the 3.x > stdlib. Mercurial is a command-line tool and doesn't need to be ported > to be used for Python 3 projects. setuptools is supplanted by > distribute which should Python 3 compatible. > > And I'm not sure what this package called "Python" is (?a high-level > object-oriented programming language?? like Java?), but I'm pretty sure > I've heard there's a Python 3 compatible version. > > (granted, it's probably less FUD than stupid automation) From what I can tell it simply looks at dependencies and availability of those dependencies with a "Python 3" trove classification. Some manual filtering may well be useful. It is well *possible* that there are packages with a runtime dependency on libraries in mercurial however. Those would need mercurial porting to Python 3 if they are to run on Python 3. If they simply shell out to mercurial that wouldn't be the case. Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ncoghlan at gmail.com Thu Dec 2 02:22:15 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Dec 2010 11:22:15 +1000 Subject: [Python-Dev] Deprecating undocumented, unused functions in difflib. In-Reply-To: References: Message-ID: On Thu, Dec 2, 2010 at 6:23 AM, Terry Reedy wrote: > It would be easiest to just remove the two lines above. > Or should I define functions _xxx names that issue a deprecation warning and > attach them as attributes to each object? (Defining instance methods would > not be the same). Given that functions are converted to bound methods only on retrieval from an instance, why wouldn't it be the same? But yes, if you want to get rid of them, then deprecation for 3.2 and removal in 3.3 is the way to go. Alternatively, not deprecating them at all and just leaving them undocumented with a comment in the source to say they have been deliberately omitted from the docs would also be fine. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fuzzyman at voidspace.org.uk Thu Dec 2 02:22:18 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 02 Dec 2010 01:22:18 +0000 Subject: [Python-Dev] Deprecating undocumented, unused functions in difflib. In-Reply-To: References: Message-ID: <4CF6F4CA.4030005@voidspace.org.uk> On 01/12/2010 20:23, Terry Reedy wrote: > Difflib.SequenceMatcher object currently get two feature attributes: > self.isbjunk = junk.__contains__ > self.isbpopular = popular.__contains__ > > Tim Peters agrees that the junk and popular sets should be directly > exposed and documented as part of the api, thereby making the > functions redundant. The two functions are not currently documented > (and should not be now). A google codesearch of 'isbjunk' and > 'isbpopular' only returns hits in difflib.py itself (and its > predecessor, ndiff.py). > > It would be easiest to just remove the two lines above. > Or should I define functions _xxx names that issue a deprecation > warning and attach them as attributes to each object? (Defining > instance methods would not be the same). > > There is only one internal use of one of the two functions which is > easily edited. > I would still be tempted to go through a single release of deprecation. You can add a test that the names are gone if the version of Python is 3.3. When the tests start failing the code and the tests can be ripped out. All the best, Michael -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ben+python at benfinney.id.au Thu Dec 2 03:01:29 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 02 Dec 2010 13:01:29 +1100 Subject: [Python-Dev] Python and the Unicode Character Database References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87bp552arq.fsf@benfinney.id.au> "Stephen J. Turnbull" writes: > Furthermore, he provided good *objective* reason (excessive cost, to > which I can also testify, in several different input methods for > Japanese) why numbers simply would not be input that way. > > What's left is copy/paste via the mouse. For direct entry by an interactive user, yes. Why are some people in this discussion thinking only of direct entry by an interactive user? Input to a program comes from various sources other than direct entry by the interactive user, as has been pointed out many times. > Please, give us just *one* more real use case that isn't "somebody > might". Input from an existing text file, as I said earlier. Or any other way of text data making its way into a Python program. Direct entry at the console is a red herring. -- \ ?First things first, but not necessarily in that order.? ?The | `\ Doctor, _Doctor Who_ | _o__) | Ben Finney From tjreedy at udel.edu Thu Dec 2 03:42:03 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 01 Dec 2010 21:42:03 -0500 Subject: [Python-Dev] Deprecating undocumented, unused functions in difflib. In-Reply-To: <4CF6F4CA.4030005@voidspace.org.uk> References: <4CF6F4CA.4030005@voidspace.org.uk> Message-ID: On 12/1/2010 8:22 PM, Michael Foord wrote: > I would still be tempted to go through a single release of deprecation. > You can add a test that the names are gone if the version of Python is > 3.3. When the tests start failing the code and the tests can be ripped out. I was wondering how people remember... It would be nice is there were instead a central place to 'deposit' simple future patches that just consist of removals -- Terry Jan Reedy From tjreedy at udel.edu Thu Dec 2 03:53:47 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 01 Dec 2010 21:53:47 -0500 Subject: [Python-Dev] Porting Ideas In-Reply-To: <4CF6F3B7.1040805@voidspace.org.uk> References: <20101201201716.4357a316@pitrou.net> <4CF6F3B7.1040805@voidspace.org.uk> Message-ID: On 12/1/2010 8:17 PM, Michael Foord wrote: > It is well *possible* that there are packages with a runtime dependency > on libraries in mercurial however. Those would need mercurial porting to > Python 3 if they are to run on Python 3. If they simply shell out to > mercurial that wouldn't be the case. It would be nice is all the Python-coded tools needed to work on Python3 ran on Python3, so one did not have to install 2.x just for that purpose ;-). Does Sphinx run on PY3 yet? -- Terry Jan Reedy From alexander.belopolsky at gmail.com Thu Dec 2 04:06:24 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 1 Dec 2010 22:06:24 -0500 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: <20101201201716.4357a316@pitrou.net> <4CF6F3B7.1040805@voidspace.org.uk> Message-ID: On Wed, Dec 1, 2010 at 9:53 PM, Terry Reedy wrote: .. > Does Sphinx run on PY3 yet? It does, but see issue10224 for details. http://bugs.python.org/issue10224 From tjreedy at udel.edu Thu Dec 2 04:09:14 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 01 Dec 2010 22:09:14 -0500 Subject: [Python-Dev] Deprecating undocumented, unused functions in difflib. In-Reply-To: References: Message-ID: On 12/1/2010 8:22 PM, Nick Coghlan wrote: > On Thu, Dec 2, 2010 at 6:23 AM, Terry Reedy wrote: >> It would be easiest to just remove the two lines above. >> Or should I define functions _xxx names that issue a deprecation warning and >> attach them as attributes to each object? (Defining instance methods would >> not be the same). > > Given that functions are converted to bound methods only on retrieval > from an instance, why wouldn't it be the same? The two SequenceMatcher instance attributes are bound functions of the two sets, not of the instance. But you are right in the sense that the usage would be the same. Since, as of a week ago, the sets were implemented as dicts, any code depending on the class of the underlying instance is already broken. So I will go with S-M methods and add a doc string like "Undocumented, deprecated method that will disappear in 3.3. Do not use!" to show up in a help() listing. > But yes, if you want to get rid of them, then deprecation for 3.2 and > removal in 3.3 is the way to go. OK. > Alternatively, not deprecating them at all and just leaving them > undocumented with a comment in the source to say they have been > deliberately omitted from the docs would also be fine. Too messy and too useless ;-). -- Terry Jan Reedy From ncoghlan at gmail.com Thu Dec 2 04:14:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Dec 2010 13:14:12 +1000 Subject: [Python-Dev] [Python-checkins] r86924 - python/branches/py3k/Doc/library/random.rst In-Reply-To: <20101202024133.CCAF1EE985@mail.python.org> References: <20101202024133.CCAF1EE985@mail.python.org> Message-ID: On Thu, Dec 2, 2010 at 12:41 PM, raymond.hettinger wrote: > +A more general approach is to arrange the weights in a cumulative probability > +distribution with :func:`itertools.accumulate`, and then locate the random value > +with :func:`bisect.bisect`:: > + > + ? ?>>> choices, weights = zip(*weighted_choices) > + ? ?>>> cumdist = list(itertools.accumulate(weights)) > + ? ?>>> x = random.random() * cumdist[-1] > + ? ?>>> choices[bisect.bisect(cumdist, x)] > + ? ?'Blue' Neat example, although it would be easier to follow if you broke that last line into two pieces: .>>> random_index = bisect.bisect(cumdist, x) .>>> choices[random_index] 'Blue' It took me a moment to remember how bisect.bisect worked, but it would have been instant if the return value was assigned to an appropriately named variable. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Thu Dec 2 04:11:21 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 01 Dec 2010 22:11:21 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> Message-ID: On 12/1/2010 7:44 PM, Alexander Belopolsky wrote: > it. The argument was that if there was a use case for parsing Eastern > Arabic numerals, it would be better served by a module written by > someone who speaks one of the Arabic languages and knows the details > of how Eastern Arabic numerals are written. So far nobody has even > claimed to know conclusively that Arabic-Indic digits are always > written left-to-right. Both my personal observations when travelling from Turkey to India and Wikipedia say yes. "When representing a number in Arabic, the lowest-valued position is placed on the right, so the order of positions is the same as in left-to-right scripts." https://secure.wikimedia.org/wikipedia/en/wiki/Arabic_language#Numerals -- Terry Jan Reedy From alexander.belopolsky at gmail.com Thu Dec 2 04:28:49 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 1 Dec 2010 22:28:49 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> Message-ID: On Wed, Dec 1, 2010 at 10:11 PM, Terry Reedy wrote: > On 12/1/2010 7:44 PM, Alexander Belopolsky wrote: > >> it. ?The argument was that if there was a use case for parsing Eastern >> Arabic numerals, it would be better served by a module written by >> someone who speaks one of the Arabic languages and knows the details >> of how ?Eastern Arabic numerals are written. ?So far nobody has even >> claimed to know conclusively that Arabic-Indic digits are always >> written left-to-right. > > Both my personal observations when travelling from Turkey to India and > Wikipedia say yes. "When representing a number in Arabic, the lowest-valued > position is placed on the right, so the order of positions is the same as in > left-to-right scripts." > https://secure.wikimedia.org/wikipedia/en/wiki/Arabic_language#Numerals This matches my limited research on this topic as well. However, I am not sure that when these codes are embedded in Arabic text, their logical order always matches their display order. It seems to me that it can go either way depending on the surrounding text and/or presence of explicit formatting codes. Also, I don't understand why Eastern Arabic-Indic digits have the same Bidi-Class as European digits, but Arabic-Indic digits, Arabic decimal and thousands separators have Bidi-Class "AN". http://www.unicode.org/reports/tr9/tr9-23.html#Bidirectional_Character_Types From alexander.belopolsky at gmail.com Thu Dec 2 05:45:26 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 1 Dec 2010 23:45:26 -0500 Subject: [Python-Dev] ICU In-Reply-To: <1291147987.8628.13.camel@localhost.localdomain> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2F067.5020705@pearwood.info> <4CF354C6.9020302@v.loewis.de> <20101129193302.115dbcd5@pitrou.net> <4CF53724.8090000@voidspace.org.uk> <4CF54DA1.5080900@v.loewis.de> <1291144993.8628.1.camel@localhost.localdomain> <4CF55346.1040108@v.loewis.de> <1291146254.8628.4.camel@localhost.localdomain> <4CF556C8.9010704@v.loewis.de> <1291147987.8628.13.camel@localhost.localdomain> Message-ID: On Tue, Nov 30, 2010 at 3:13 PM, Antoine Pitrou wrote: > > Oh, about ICU: > >> > Actually, I remember you saying that locale should ideally be replaced >> > with a wrapper around the ICU library. >> >> By that, I stand - however, I have given up the hope that this will >> happen anytime soon. > > Perhaps this could be made a GSOC topic. > Incidentally, this may also address another Python's Achilles' heel: the timezone support. http://icu-project.org/download/icutzu.html From a.badger at gmail.com Thu Dec 2 06:11:27 2010 From: a.badger at gmail.com (Toshio Kuratomi) Date: Wed, 1 Dec 2010 21:11:27 -0800 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: <20101201201716.4357a316@pitrou.net> <4CF6F3B7.1040805@voidspace.org.uk> Message-ID: <20101202051127.GG6965@unaka.lan> On Wed, Dec 01, 2010 at 10:06:24PM -0500, Alexander Belopolsky wrote: > On Wed, Dec 1, 2010 at 9:53 PM, Terry Reedy wrote: > .. > > Does Sphinx run on PY3 yet? > > It does, but see issue10224 for details. > > http://bugs.python.org/issue10224 > Also, docutils has an unported module. /me needs to write a bug report for that as he really doesn't have the time he thought he did to perform the port. -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From stephen at xemacs.org Thu Dec 2 08:49:24 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 02 Dec 2010 16:49:24 +0900 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87bp552arq.fsf@benfinney.id.au> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> Message-ID: <87oc94y5q3.fsf@uwakimon.sk.tsukuba.ac.jp> Ben Finney writes: > Input from an existing text file, as I said earlier. Or any other way of > text data making its way into a Python program. > Direct entry at the console is a red herring. I don't think it is. Not at all. Here's why: '''print "%d" % some_integer''' doesn't now, and never will (unless Kristan gets his Python 2.8), produce Arabic or Han numerals. Not in any language I know of, not in Microsoft Excel, and definitely not in Python 2. *Somebody* typed that text at some point. If it's Han, that somebody had *way* too much time on his hands, not a working accountant nor a graduate assistant in a research lab for sure. How about old archived texts, copied and recopied? At least for Japanese, old archival (text) data will *all* be in ASCII, because the earliest implementations of Japanese language text used JIS X 0201 (or its predecessor), which doesn't have Han digits (and kana digits don't exist even if you write with a brush and ink AFAIK). Ditto Arabic, I would imagine; ISO 8859/6 (aka Latin/Arabic) does not contain the Arabic digits that have been presented here earlier AFAICT. Note that there's plenty of space for them in that code table (eg, 0xB0-0xB9 is empty). Apparently nobody *ever* thought it was useful to have them! So, which culture, using which script and in which application, inputs numeric data in other than ASCII digits? Or would want to, if only somebody would tell them they can do it in Python? Hearsay will do, for starters. From ncoghlan at gmail.com Thu Dec 2 10:32:19 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 2 Dec 2010 19:32:19 +1000 Subject: [Python-Dev] [Python-checkins] r86930 - in python/branches/py3k: Doc/library/os.rst Lib/os.py Lib/test/test_os.py Misc/ACKS Misc/NEWS In-Reply-To: <20101202070557.1D971EE98C@mail.python.org> References: <20101202070557.1D971EE98C@mail.python.org> Message-ID: On Thu, Dec 2, 2010 at 5:05 PM, terry.reedy wrote: > + If > + ? the target directory with the same mode as we specified already exists, > + ? raises an :exc:`OSError` exception if *exist_ok* is False, otherwise no > + ? exception is raised. ?If the directory cannot be created in other cases, > + ? raises an :exc:`OSError` exception. I would suggest being explicit here that "directory exists, but has a mode other than the one requested" always triggers an exception. Perhaps something like the following: "Raises an :exc:`OSError` exception if the target directory already exists, unless *exist_ok* is True and the existing directory has the same mode as is specified in the current call. Also raises an :exc:`OSError` exception if the directory cannot be created for any other reason." Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From sable at users.sourceforge.net Thu Dec 2 11:13:44 2010 From: sable at users.sourceforge.net (=?ISO-8859-1?Q?S=E9bastien_Sabl=E9?=) Date: Thu, 02 Dec 2010 11:13:44 +0100 Subject: [Python-Dev] AIX 5.3 - Enabling Shared Library Support Vs Extensions In-Reply-To: References: Message-ID: <4CF77158.3010005@users.sourceforge.net> Hi Anurag, Le 25/11/2010 10:24, Anurag Chourasia a ?crit : > All, > > When I configure python to enable shared libraries, none of the extensions are getting built during the make step due to this error. > you may want to take a look at the following issue: http://bugs.python.org/issue941346 Python compiled with shared libraries was broken on AIX until recently. There are some patches there to get it to work, or you may want to test the latest 2.7 or 3.x releases. regards -- S?bastien Sabl? From merwok at netwok.org Thu Dec 2 13:18:53 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 02 Dec 2010 13:18:53 +0100 Subject: [Python-Dev] [Python-checkins] r86924 - python/branches/py3k/Doc/library/random.rst In-Reply-To: References: <20101202024133.CCAF1EE985@mail.python.org> Message-ID: <4CF78EAD.1060408@netwok.org> > On Thu, Dec 2, 2010 at 12:41 PM, raymond.hettinger > wrote: >> +A more general approach is to arrange the weights in a cumulative probability >> +distribution with :func:`itertools.accumulate`, and then locate the random value >> +with :func:`bisect.bisect`:: >> + >> + >>> choices, weights = zip(*weighted_choices) >> + >>> cumdist = list(itertools.accumulate(weights)) >> + >>> x = random.random() * cumdist[-1] >> + >>> choices[bisect.bisect(cumdist, x)] >> + 'Blue' ?pydoc bisect.bisect? is empty (?Alias for bisect_right()?); in the code, bisect.bisect is noted as compatibility alias. Wouldn?t it be more helpful to use the newer name? Regards From nyamatongwe at gmail.com Thu Dec 2 13:40:45 2010 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Thu, 2 Dec 2010 23:40:45 +1100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87oc94y5q3.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <87oc94y5q3.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Stephen J. Turnbull: > Here's why: '''print "%d" % > some_integer''' doesn't now, and never will (unless Kristan gets his > Python 2.8), produce Arabic or Han numerals. ?Not in any > language I know of, not in Microsoft Excel, and definitely not in > Python 2. While I don't have Excel to test with, OpenOffice.org Calc will display in Arabic or Han numerals using the NatNum format codes. http://www.scintilla.org/ArabicNumbers.png > Ditto Arabic, I > would imagine; ISO 8859/6 (aka Latin/Arabic) does not contain the > Arabic digits that have been presented here earlier AFAICT. ?Note that > there's plenty of space for them in that code table (eg, 0xB0-0xB9 is > empty). ?Apparently nobody *ever* thought it was useful to have them! DOS code page 864 does use 0xB0-0xB9 for ? .. ?. http://www.ascii.ca/cp864.htm Neil From g.brandl at gmx.net Thu Dec 2 13:57:41 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 02 Dec 2010 13:57:41 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF6CE95.1060707@v.loewis.de> References: <4CF2DC5B.4020702@egenix.com> <4CF6CE95.1060707@v.loewis.de> Message-ID: Am 01.12.2010 23:39, schrieb "Martin v. L?wis": >> As of today, "What?s New In Python 3.2" [1] does not even mention the >> unicodedata upgrade to 6.0.0. > > One reason was that I was instructed not to change "What's New" a few > years ago. Maybe all past, present and future whatsnew maintainers can agree on these rules, which I copied directly from whatsnew/3.2.rst? Rules for maintenance: * Anyone can add text to this document. Do not spend very much time on the wording of your changes, because your text will probably get rewritten to some degree. * The maintainer will go through Misc/NEWS periodically and add changes; it's therefore more important to add your changes to Misc/NEWS than to this file. * This is not a complete list of every single change; completeness is the purpose of Misc/NEWS. Some changes I consider too small or esoteric to include. If such a change is added to the text, I'll just remove it. (This is another reason you shouldn't spend too much time on writing your addition.) * If you want to draw your new text to the attention of the maintainer, add 'XXX' to the beginning of the paragraph or section. * It's OK to just add a fragmentary note about a change. For example: "XXX Describe the transmogrify() function added to the socket module." The maintainer will research the change and write the necessary text. * You can comment out your additions if you like, but it's not necessary (especially when a final release is some months away). * Credit the author of a patch or bugfix. Just the name is sufficient; the e-mail address isn't necessary. It's helpful to add the issue number: XXX Describe the transmogrify() function added to the socket module. (Contributed by P.Y. Developer; :issue:`12345`.) This saves the maintainer the effort of going through the SVN log when researching a change. Georg From ziade.tarek at gmail.com Thu Dec 2 14:08:37 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Dec 2010 14:08:37 +0100 Subject: [Python-Dev] Change to the Distutils / Distutils2 workflow Message-ID: Hey We discussed with Eric about the debugging workflow and we agreed that our life would be easier if every bug fix would land first in Distutils2 when it makes sense, then get backported to Distutils1. For other core-devs that would mean that your patches should be done against hg.python.org/distutils2, which uses unittest2. Then Eric and I would take care of the backporting. I am planning to set up a wiki page with the workflow as soon as I get a chance. Thanks Tarek -- Tarek Ziad? | http://ziade.org From regebro at gmail.com Thu Dec 2 14:28:33 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 2 Dec 2010 14:28:33 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87tyixxbet.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87tyixxbet.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: 2010/12/2 Stephen J. Turnbull : > Because that works, but > > print(T1234) > > doesn't (it prints ASCII). ?You can't round-trip, but users will > want/expect that. You should be able to round-trip, absolutely. I don't think you should expect print() to do that. str(56) possibly. :) That's an argument for it to be in a module, as you then would need to send in a parameter on which decimal characters you want. > T1000 = float('?.???') That was already discussed here, and it's clear that unicode does not consider these characters to be something you can use in a decimal number, and hence it's not broken. From solipsis at pitrou.net Thu Dec 2 14:36:33 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 2 Dec 2010 14:36:33 +0100 Subject: [Python-Dev] Python and the Unicode Character Database References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> Message-ID: <20101202143633.6eb17b8d@pitrou.net> On Wed, 1 Dec 2010 22:28:49 -0500 Alexander Belopolsky wrote: > > > > Both my personal observations when travelling from Turkey to India and > > Wikipedia say yes. "When representing a number in Arabic, the lowest-valued > > position is placed on the right, so the order of positions is the same as in > > left-to-right scripts." > > https://secure.wikimedia.org/wikipedia/en/wiki/Arabic_language#Numerals > > This matches my limited research on this topic as well. However, I am > not sure that when these codes are embedded in Arabic text, their > logical order always matches their display order. That shouldn't matter, since unicode text follows logical order. The display order is up to the graphical representation library. Regards Antoine. From regebro at gmail.com Thu Dec 2 14:36:33 2010 From: regebro at gmail.com (Lennart Regebro) Date: Thu, 2 Dec 2010 14:36:33 +0100 Subject: [Python-Dev] Porting Ideas In-Reply-To: <20101201201716.4357a316@pitrou.net> References: <20101201201716.4357a316@pitrou.net> Message-ID: On Wed, Dec 1, 2010 at 20:17, Antoine Pitrou wrote: > And I'm not sure what this package called "Python" is (?a high-level > object-oriented programming language?? like Java?), but I'm pretty sure > I've heard there's a Python 3 compatible version. Uhm... http://pypi.python.org/pypi/Python Anybody wanna remove that, or update it or something? :-) From anurag.chourasia at gmail.com Thu Dec 2 15:11:07 2010 From: anurag.chourasia at gmail.com (Anurag Chourasia) Date: Thu, 2 Dec 2010 19:41:07 +0530 Subject: [Python-Dev] AIX 5.3 - Enabling Shared Library Support Vs Extensions In-Reply-To: <4CF77158.3010005@users.sourceforge.net> References: <4CF77158.3010005@users.sourceforge.net> Message-ID: Hi Sebastian, Thanks for your response. I looked at http://bugs.python.org/issue941346 earlier. I was referred to this link by Stefan Krah through another bug that i created at http://bugs.python.org/issue10555 for this issue. I confirm that my problem is solved with the Python 2.7.1 release which contains the changes done by you. Great work done by you and other folks for enabling the Shared Library build on AIX. Hats Off !!! Regards, Anurag 2010/12/2 S?bastien Sabl? > > Hi Anurag, > > Le 25/11/2010 10:24, Anurag Chourasia a ?crit : > > > All, > > > > When I configure python to enable shared libraries, none of the > extensions are getting built during the make step due to this error. > > > > you may want to take a look at the following issue: > > http://bugs.python.org/issue941346 > > Python compiled with shared libraries was broken on AIX until recently. > There are some patches there to get it to work, or you may want to test the > latest 2.7 or 3.x releases. > > regards > > -- > S?bastien Sabl? > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Dec 2 16:47:08 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 2 Dec 2010 07:47:08 -0800 Subject: [Python-Dev] ICU In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2F067.5020705@pearwood.info> <4CF354C6.9020302@v.loewis.de> <20101129193302.115dbcd5@pitrou.net> <4CF53724.8090000@voidspace.org.uk> <4CF54DA1.5080900@v.loewis.de> <1291144993.8628.1.camel@localhost.localdomain> <4CF55346.1040108@v.loewis.de> <1291146254.8628.4.camel@localhost.localdomain> <4CF556C8.9010704@v.loewis.de> <1291147987.8628.13.camel@localhost.localdomain> Message-ID: On Wed, Dec 1, 2010 at 8:45 PM, Alexander Belopolsky wrote: > On Tue, Nov 30, 2010 at 3:13 PM, Antoine Pitrou wrote: >> >> Oh, about ICU: >> >>> > Actually, I remember you saying that locale should ideally be replaced >>> > with a wrapper around the ICU library. >>> >>> By that, I stand - however, I have given up the hope that this will >>> happen anytime soon. >> >> Perhaps this could be made a GSOC topic. >> > > Incidentally, this may also address another Python's Achilles' heel: > the timezone support. > > http://icu-project.org/download/icutzu.html I work with people who speak highly of ICU, so I want to encourage work in this area. At the same time, I'm skeptical -- IIRC, ICU is a large amount of C++ code. I don't know how easy it will be to integrate this into our build processes for various platforms, nor how "Pythonic" the resulting APIs will look to the experienced Python user. Still, those are not roadblocks, the benefits are potentially great, so it's definitely worth investigating! -- --Guido van Rossum (python.org/~guido) From ocean-city at m2.ccsnet.ne.jp Thu Dec 2 16:50:51 2010 From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto) Date: Fri, 03 Dec 2010 00:50:51 +0900 Subject: [Python-Dev] [Python-checkins] r86817 - python/branches/py3k-stat-on-windows/Lib/test/test_shutil.py In-Reply-To: References: <20101126184428.E04A0EE984@mail.python.org> <4CF00E4E.6030507@m2.ccsnet.ne.jp> <4CF01632.8070504@m2.ccsnet.ne.jp> Message-ID: <4CF7C05B.7030909@m2.ccsnet.ne.jp> On 2010/11/27 5:31, Brian Curtin wrote: > On Fri, Nov 26, 2010 at 14:18, Hirokazu Yamamoto> wrote: > >> On 2010/11/27 5:02, Brian Curtin wrote: >> >>> We briefly chatted about this on the os.link >>> feature issue, but I never found a way around it. >>> >> >> How about implementing os.path.samefile in >> Modules/posixmodule.c like this? >> >> http://bugs.python.org/file19262/py3k_fix_kill_python_for_short_path.patch >> >> # I hope this works. >> > > That's almost identical to what the current os.path.sameopenfile is. > > Lib/ntpath.py opens both files, then compares them via _getfileinformation. > That function is implemented to take in a file descriptor, call > GetFileInformationByHandle with it, then returns a tuple > of dwVolumeSerialNumber, nFileIndexHigh, and nFileIndexLow. > Yes. Difference is, file object cannot represent directory, and probably FILE_FLAG_BACKUP_SEMANTICS makes it faster to open file. From foom at fuhm.net Thu Dec 2 17:15:54 2010 From: foom at fuhm.net (James Y Knight) Date: Thu, 2 Dec 2010 11:15:54 -0500 Subject: [Python-Dev] ICU In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2F067.5020705@pearwood.info> <4CF354C6.9020302@v.loewis.de> <20101129193302.115dbcd5@pitrou.net> <4CF53724.8090000@voidspace.org.uk> <4CF54DA1.5080900@v.loewis.de> <1291144993.8628.1.camel@localhost.localdomain> <4CF55346.1040108@v.loewis.de> <1291146254.8628.4.camel@localhost.localdomain> <4CF556C8.9010704@v.loewis.de> <1291147987.8628.13.camel@localhost.localdomain> Message-ID: <96F96B19-67B5-4526-9553-B517C1D0AE38@fuhm.net> On Dec 1, 2010, at 11:45 PM, Alexander Belopolsky wrote: > On Tue, Nov 30, 2010 at 3:13 PM, Antoine Pitrou wrote: >> >> Oh, about ICU: >> >>>> Actually, I remember you saying that locale should ideally be replaced >>>> with a wrapper around the ICU library. >>> >>> By that, I stand - however, I have given up the hope that this will >>> happen anytime soon. >> >> Perhaps this could be made a GSOC topic. >> > > Incidentally, this may also address another Python's Achilles' heel: > the timezone support. > > http://icu-project.org/download/icutzu.html Does ICU do anything regarding timezones that datetime + pytz doesn't already do? Wouldn't it make more sense to integrate the already-existing-and-pythonic pytz into Python than to make a new wrapper based on ICU? James From alexander.belopolsky at gmail.com Thu Dec 2 17:41:11 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 2 Dec 2010 11:41:11 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <20101202143633.6eb17b8d@pitrou.net> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> Message-ID: On Thu, Dec 2, 2010 at 8:36 AM, Antoine Pitrou wrote: > On Wed, 1 Dec 2010 22:28:49 -0500 > Alexander Belopolsky wrote: .. >> This matches my limited research on this topic as well. ?However, I am >> not sure that when these codes are embedded in Arabic text, their >> logical order always matches their display order. > > That shouldn't matter, since unicode text follows logical order. The > display order is up to the graphical representation library. > I am not so sure. On my Mac, U+200F (RIGHT-TO-LEFT MARK) affects 0-9 and Arabic-Indic decimals differently: >>> print('\u200F123') ?123 >>> print('\u200F\u0661\u0662\u0663') 231 I replaced Arabic-Indic decimals with 0-9 in the output to demonstrate the point. Cut-n-paste does not work well in the presence of RTL directives. and U+202E (RIGHT-TO-LEFT OVERRIDE) reverts the display order for both: >>> print('\u202E123') 321 >>> print('\u202E\u0661\u0662\u0663') 321 (again, the output display is simulated not copied.) I don't know if explicit RTL directives are ever used in Arabic texts, but it is quite possible that texts converted from older formats would use them for efficiency. Note that my point is not to find the correct answer here, but to demonstrate that we as a group don't have the expertise to get parsing of Arabic text right. If we've got it right for Arabic, it is by chance and not by design. This still leaves us with 41 other types of digits for at least 30 different languages. Nobody will ever assume that python builtins are suitable for use with all these variants. This "feature" is only good for nefarious purposes such as hiding extra digits in innocent-looking files or smuggling binary data through naive interfaces. PS: BTW, shouldn't int('\u0661\u0662\u06DD') be valid? or is it int('\u06DD\u0661\u0662')? From solipsis at pitrou.net Thu Dec 2 17:56:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 02 Dec 2010 17:56:00 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> Message-ID: <1291308960.3576.28.camel@localhost.localdomain> Le jeudi 02 d?cembre 2010 ? 11:41 -0500, Alexander Belopolsky a ?crit : > > Note that my point is not to find the correct answer here, but to > demonstrate that we as a group don't have the expertise to get parsing > of Arabic text right. I don't understand why you think Arabic or Hebrew text is any different from Western text. Surely right-to-left isn't more conceptually complicated than left-to-right, is it? The fact that mixed rtl + ltr can render bizarrely or is awkward to cut and paste is quite off-topic for our discussion. > If we've got it right for Arabic, it is by > chance and not by design. This still leaves us with 41 other types of > digits for at least 30 different languages. So why do you trust the Unicode standard on other things and not on this one? Regards Antoine. From benjamin at python.org Thu Dec 2 18:03:57 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 2 Dec 2010 11:03:57 -0600 Subject: [Python-Dev] ICU In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2F067.5020705@pearwood.info> <4CF354C6.9020302@v.loewis.de> <20101129193302.115dbcd5@pitrou.net> <4CF53724.8090000@voidspace.org.uk> <4CF54DA1.5080900@v.loewis.de> <1291144993.8628.1.camel@localhost.localdomain> <4CF55346.1040108@v.loewis.de> <1291146254.8628.4.camel@localhost.localdomain> <4CF556C8.9010704@v.loewis.de> <1291147987.8628.13.camel@localhost.localdomain> Message-ID: 2010/12/2 Guido van Rossum : > On Wed, Dec 1, 2010 at 8:45 PM, Alexander Belopolsky > wrote: >> On Tue, Nov 30, 2010 at 3:13 PM, Antoine Pitrou wrote: >>> >>> Oh, about ICU: >>> >>>> > Actually, I remember you saying that locale should ideally be replaced >>>> > with a wrapper around the ICU library. >>>> >>>> By that, I stand - however, I have given up the hope that this will >>>> happen anytime soon. >>> >>> Perhaps this could be made a GSOC topic. >>> >> >> Incidentally, this may also address another Python's Achilles' heel: >> the timezone support. >> >> http://icu-project.org/download/icutzu.html > > I work with people who speak highly of ICU, so I want to encourage > work in this area. > > At the same time, I'm skeptical -- IIRC, ICU is a large amount of C++ > code. I don't know how easy it will be to integrate this into our > build processes for various platforms, nor how "Pythonic" the > resulting APIs will look to the experienced Python user. There's a nice C-API. -- Regards, Benjamin From tjreedy at udel.edu Thu Dec 2 18:34:27 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 02 Dec 2010 12:34:27 -0500 Subject: [Python-Dev] [Python-checkins] r86930 - in python/branches/py3k: Doc/library/os.rst Lib/os.py Lib/test/test_os.py Misc/ACKS Misc/NEWS In-Reply-To: References: <20101202070557.1D971EE98C@mail.python.org> Message-ID: <4CF7D8A3.3050800@udel.edu> On 12/2/2010 4:32 AM, Nick Coghlan wrote: > On Thu, Dec 2, 2010 at 5:05 PM, terry.reedy wrote: (except I did not write most of the patch) >> + If >> + the target directory with the same mode as we specified already exists, >> + raises an :exc:`OSError` exception if *exist_ok* is False, otherwise no >> + exception is raised. If the directory cannot be created in other cases, >> + raises an :exc:`OSError` exception. > > I would suggest being explicit here that "directory exists, but has a > mode other than the one requested" always triggers an exception. > Perhaps something like the following: > > "Raises an :exc:`OSError` exception if the target directory already > exists, unless *exist_ok* is True and the existing directory has the > same mode as is specified in the current call. Also raises an > :exc:`OSError` exception if the directory cannot be created for any > other reason." Georg has already patched that paragraph. I will let him decide if any further change is needed. Terry From pje at telecommunity.com Thu Dec 2 18:16:33 2010 From: pje at telecommunity.com (P.J. Eby) Date: Thu, 02 Dec 2010 12:16:33 -0500 Subject: [Python-Dev] ICU In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2F067.5020705@pearwood.info> <4CF354C6.9020302@v.loewis.de> <20101129193302.115dbcd5@pitrou.net> <4CF53724.8090000@voidspace.org.uk> <4CF54DA1.5080900@v.loewis.de> <1291144993.8628.1.camel@localhost.localdomain> <4CF55346.1040108@v.loewis.de> <1291146254.8628.4.camel@localhost.localdomain> <4CF556C8.9010704@v.loewis.de> <1291147987.8628.13.camel@localhost.localdomain> Message-ID: <20101202171652.024943A4119@sparrow.telecommunity.com> At 07:47 AM 12/2/2010 -0800, Guido van Rossum wrote: >On Wed, Dec 1, 2010 at 8:45 PM, Alexander Belopolsky > wrote: > > On Tue, Nov 30, 2010 at 3:13 PM, Antoine Pitrou > wrote: > >> > >> Oh, about ICU: > >> > >>> > Actually, I remember you saying that locale should ideally be replaced > >>> > with a wrapper around the ICU library. > >>> > >>> By that, I stand - however, I have given up the hope that this will > >>> happen anytime soon. > >> > >> Perhaps this could be made a GSOC topic. > >> > > > > Incidentally, this may also address another Python's Achilles' heel: > > the timezone support. > > > > http://icu-project.org/download/icutzu.html > >I work with people who speak highly of ICU, so I want to encourage >work in this area. > >At the same time, I'm skeptical -- IIRC, ICU is a large amount of C++ >code. I don't know how easy it will be to integrate this into our >build processes for various platforms, nor how "Pythonic" the >resulting APIs will look to the experienced Python user. > >Still, those are not roadblocks, the benefits are potentially great, >so it's definitely worth investigating! FWIW, OSAF did a wrapping for Chandler, though I personally haven't used it: http://pyicu.osafoundation.org/ The README explains the mapping from the ICU APIs to Python ones, including iteration, string conversion, and timezone mapping for use with the datetime type. >-- >--Guido van Rossum (python.org/~guido) >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/pje%40telecommunity.com From tjreedy at udel.edu Thu Dec 2 18:59:00 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 02 Dec 2010 12:59:00 -0500 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: <20101201201716.4357a316@pitrou.net> Message-ID: On 12/2/2010 8:36 AM, Lennart Regebro wrote: > On Wed, Dec 1, 2010 at 20:17, Antoine Pitrou wrote: >> And I'm not sure what this package called "Python" is (?a high-level >> object-oriented programming language?? like Java?), but I'm pretty sure >> I've heard there's a Python 3 compatible version. > > Uhm... http://pypi.python.org/pypi/Python > > Anybody wanna remove that, or update it or something? :-) Entry is for Python 2.5. # Package Index Owner: guido, anthony, barry -- Terry Jan Reedy From alexander.belopolsky at gmail.com Thu Dec 2 19:14:29 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 2 Dec 2010 13:14:29 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <1291308960.3576.28.camel@localhost.localdomain> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> <1291308960.3576.28.camel@localhost.localdomain> Message-ID: On Thu, Dec 2, 2010 at 11:56 AM, Antoine Pitrou wrote: > Le jeudi 02 d?cembre 2010 ? 11:41 -0500, Alexander Belopolsky a ?crit : >> >> Note that my point is not to find the correct answer here, but to >> demonstrate that we as a group don't have the expertise to get parsing >> of Arabic text right. > > I don't understand why you think Arabic or Hebrew text is any different > from Western text. Surely right-to-left isn't more conceptually > complicated than left-to-right, is it? > No, but a mix of LTR and RTL is certainly more difficult that either of the two. I invite you to digest Unicode Standard Annex #9 before we continue this discussion. See . > The fact that mixed rtl + ltr can render bizarrely or is awkward to cut > and paste is quite off-topic for our discussion. > No, it is not. One of the invented use cases in this thread was naive users' desire to enter numbers using their preferred local decimals. Same users may want to be able to cut and paste their decimals as well. More importantly, however, legacy formats may not have support for mixed-direction text and may require that "John is 41" be stored as "41 si nhoJ" and Unicode converter would turn it into "[RTL]John is 14" that will still display as "41 si nhoJ", but int(s[-2:]) will return 14, not 41. >> If we've got it right for Arabic, it is by >> chance and not by design. ?This still leaves us with 41 other types of >> digits for at least 30 different languages. > > So why do you trust the Unicode standard on other things and not on this > one? What other things? As far as I understand the only str method that was designed to comply with Unicode recomendations was str.isidentifier(). And we have some really bizarre results: >>> '\u2164'.isidentifier() True >>> '\u2164'.isalpha() False and can you describe the difference between str.isdigit() and str.isdecimal()? According to the reference manual, """ str.isdecimal() Return true if all characters in the string are decimal characters and there is at least one character, false otherwise. Decimal characters include digit characters, and all characters that that can be used to form decimal-radix numbers, e.g. U+0660, ARABIC-INDIC DIGIT ZERO. str.isdigit() Return true if all characters in the string are digits and there is at least one character, false otherwise. """ http://docs.python.org/dev/library/stdtypes.html#str.isdecimal Since U+0660 is mentioned in the first definition and not in the second, I may conclude that it is not a digit, but >>> '\u0660'.isdigit() True If you know the correct answer, please contribute it here: . From solipsis at pitrou.net Thu Dec 2 19:55:31 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 02 Dec 2010 19:55:31 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> <1291308960.3576.28.camel@localhost.localdomain> Message-ID: <1291316131.3576.43.camel@localhost.localdomain> Le jeudi 02 d?cembre 2010 ? 13:14 -0500, Alexander Belopolsky a ?crit : > > I don't understand why you think Arabic or Hebrew text is any different > > from Western text. Surely right-to-left isn't more conceptually > > complicated than left-to-right, is it? > > > > No, but a mix of LTR and RTL is certainly more difficult that either > of the two. I invite you to digest Unicode Standard Annex #9 before > we continue this discussion. > > See . ?This annex describes specifications for the *positioning* of characters flowing from right to left? (emphasis mine) Looks like something for implementors of rendering engines, which python-dev is not AFAICT. > Same users may want to be able to cut and paste their decimals as > well. More importantly, however, legacy formats may not have support > for mixed-direction text and may require that "John is 41" be stored > as "41 si nhoJ" and Unicode converter would turn it into "[RTL]John is > 14" that will still display as "41 si nhoJ", but int(s[-2:]) will > return 14, not 41. The legacy format argument looks like a red herring to me. When converting from a format to another it is the programmer's job to his/her job right. > >> If we've got it right for Arabic, it is by > >> chance and not by design. This still leaves us with 41 other types of > >> digits for at least 30 different languages. > > > > So why do you trust the Unicode standard on other things and not on this > > one? > > What other things? Everything which the Unicode database stores and that we already rely on. > As far as I understand the only str method that was > designed to comply with Unicode recomendations was str.isidentifier(). I don't think so. str.split() and str.splitlines() are also defined in conformance to the SPEC, AFAIK. They certainly try to. And, outside of str itself, the re module tries to follow Unicode categories as well (for example, "\d" should match non-ASCII digits). Regards Antoine. From barry at python.org Thu Dec 2 20:06:50 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 2 Dec 2010 14:06:50 -0500 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: <20101201201716.4357a316@pitrou.net> Message-ID: <20101202140650.11d96108@mission> On Dec 02, 2010, at 12:59 PM, Terry Reedy wrote: >On 12/2/2010 8:36 AM, Lennart Regebro wrote: >> On Wed, Dec 1, 2010 at 20:17, Antoine Pitrou wrote: >>> And I'm not sure what this package called "Python" is (?a high-level >>> object-oriented programming language?? like Java?), but I'm pretty sure >>> I've heard there's a Python 3 compatible version. >> >> Uhm... http://pypi.python.org/pypi/Python >> >> Anybody wanna remove that, or update it or something? :-) > >Entry is for Python 2.5. > ># Package Index Owner: guido, anthony, barry Well, I definitely can't remember ever seeing that before. Of course, that doesn't mean I haven't. ;) -Barry Aside: how does one log into the Cheeseshop with your Launchpad OpenID? When I try to do it I end up on a "Manual user registration" page. I fill out the username with what I think my PyPI user name is, and add my python.org email address, but then it tells me 'barry' is already taken. Do I need some kind of back door linking of my lp openid and my pypi user id? -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From sridharr at activestate.com Thu Dec 2 20:24:36 2010 From: sridharr at activestate.com (Sridhar Ratnakumar) Date: Thu, 2 Dec 2010 11:24:36 -0800 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: Message-ID: On 2010-12-01, at 11:02 AM, Brian Curtin wrote: > http://onpython3yet.com/ might be helpful to you. It orders the projects on PyPI with the most dependencies which are not yet ported to 3.x. > > Note that there are a number of false positives, e.g., the first result -- NumPy, since people don't seem to keep their classifiers up-to-date. Also note that the dependency information is incomplete. For instance, onpython3yet.com shows just 14 packages depending on "Twisted", http://onpython3yet.com/packages/show/Twisted while, in reality, there are 68 of them, http://code.activestate.com/pypm/twisted/#requiredby (see the right sidebar) -srid -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Thu Dec 2 20:37:29 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 02 Dec 2010 20:37:29 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87bp552arq.fsf@benfinney.id.au> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> Message-ID: <4CF7F579.5050603@v.loewis.de> Am 02.12.2010 03:01, schrieb Ben Finney: > "Stephen J. Turnbull" writes: > >> Furthermore, he provided good *objective* reason (excessive cost, to >> which I can also testify, in several different input methods for >> Japanese) why numbers simply would not be input that way. >> >> What's left is copy/paste via the mouse. > > For direct entry by an interactive user, yes. Why are some people in > this discussion thinking only of direct entry by an interactive user? Ultimately, somebody will have entered the data. > Input from an existing text file, as I said earlier. Which *specific* existing text file? Have you actually *seen* such a text file? > Direct entry at the console is a red herring. And we don't need powerhouses because power comes out of the socket. Regards, Martin From martin at v.loewis.de Thu Dec 2 20:40:52 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 02 Dec 2010 20:40:52 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <4CF2DC5B.4020702@egenix.com> <4CF6CE95.1060707@v.loewis.de> Message-ID: <4CF7F644.8020207@v.loewis.de> > Maybe all past, present and future whatsnew maintainers can agree on these > rules, which I copied directly from whatsnew/3.2.rst? I don't think all past maintainers can (I'm pretty certain that AMK would disagree), but if that's the current policy, I can certainly try following it (I didn't know it exists because I never look at the file). Regards, Martin From martin at v.loewis.de Thu Dec 2 20:44:42 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Dec 2010 20:44:42 +0100 Subject: [Python-Dev] Porting Ideas In-Reply-To: <20101202140650.11d96108@mission> References: <20101201201716.4357a316@pitrou.net> <20101202140650.11d96108@mission> Message-ID: <4CF7F72A.6090406@v.loewis.de> > Aside: how does one log into the Cheeseshop with your Launchpad OpenID? When > I try to do it I end up on a "Manual user registration" page. I fill out the > username with what I think my PyPI user name is, and add my python.org email > address, but then it tells me 'barry' is already taken. Do I need some kind > of back door linking of my lp openid and my pypi user id? Since the "barry" account already exists, you first need to log into that (likely using a password). You can then claim the LP OpenID as being associated with that account, and use LP in the future. Regards, Martin From barry at python.org Thu Dec 2 20:55:51 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 2 Dec 2010 14:55:51 -0500 Subject: [Python-Dev] Porting Ideas In-Reply-To: <4CF7F72A.6090406@v.loewis.de> References: <20101201201716.4357a316@pitrou.net> <20101202140650.11d96108@mission> <4CF7F72A.6090406@v.loewis.de> Message-ID: <20101202145551.06643709@mission> On Dec 02, 2010, at 08:44 PM, Martin v. L?wis wrote: >Since the "barry" account already exists, you first need to log into >that (likely using a password). You can then claim the LP OpenID as >being associated with that account, and use LP in the future. Thanks Martin. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From benjamin at python.org Thu Dec 2 20:59:42 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 2 Dec 2010 13:59:42 -0600 Subject: [Python-Dev] PEP 384 accepted Message-ID: Hi, Since discussion has trailed off without any blocking objections, I'm accepting PEP 384. Martin, you may mark the PEP accepted and proceed with merging the implementation for the beta on Saturday. -- Regards, Benjamin From mal at egenix.com Thu Dec 2 21:05:21 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 02 Dec 2010 21:05:21 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF2DFD1.10901@v.loewis.de> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> Message-ID: <4CF7FC01.80307@egenix.com> "Martin v. L?wis" wrote: >>> Now, one may wonder what precisely a "possibly signed floating point >>> number" is, but most likely, this refers to >>> >>> floatnumber ::= pointfloat | exponentfloat >>> pointfloat ::= [intpart] fraction | intpart "." >>> exponentfloat ::= (intpart | pointfloat) exponent >>> intpart ::= digit+ >>> fraction ::= "." digit+ >>> exponent ::= ("e" | "E") ["+" | "-"] digit+ >>> digit ::= "0"..."9" >> >> I don't see why the language spec should limit the wealth of number >> formats supported by float(). > > If it doesn't, there should be some other specification of what > is correct and what is not. It must not be unspecified. True. >> It is not uncommon for Asians and other non-Latin script users to >> use their own native script symbols for numbers. Just because these >> digits may look strange to someone doesn't mean that they are >> meaningless or should be discarded. > > Then these users should speak up and indicate their need, or somebody > should speak up and confirm that there are users who actually want > '????.??' to denote 1234.56. To my knowledge, there is no writing > system in which '????.??e4' means 12345600.0. I'm not sure what you're after here. >> Please also remember that Python3 now allows Unicode names for >> identifiers for much the same reasons. > > No no no. Addition of Unicode identifiers has a well-designed, > deliberate specification, with a PEP and all. The support for > non-ASCII digits in float appears to be ad-hoc, and not founded > on actual needs of actual users. Please note that we didn't have PEPs and the PEP process at the time. The Unicode proposal predates and in some respects inspired the PEP process. The decision to add this support was deliberate based on the desire to support as much of the nice features of Unicode in Python as we could. At least that was what was driving me at the time. Regarding actual needs of actual users: I don't buy that as an argument when it comes to supporting a standard that is meant to attract users with non-ASCII origins. Some references you may want to read up on: http://en.wikipedia.org/wiki/Numbers_in_Chinese_culture http://en.wikipedia.org/wiki/Vietnamese_numerals http://en.wikipedia.org/wiki/Korean_numerals http://en.wikipedia.org/wiki/Japanese_numerals Even MS Office supports them: http://languages.siuc.edu/Chinese/Language_Settings.html >> Note that the support in float() (and the other numeric constructors) >> to work with Unicode code points was explicitly added when Unicode >> support was added to Python and has been available since Python 1.6. > > That doesn't necessarily make it useful. Alexander's complaint is that > it makes Python unstable (i.e. changing as the UCD changes). If that were true, then all Unicode database (UCD) changes would make Python unstable. However, most changes to existing code points in the UCS are bug fixes, so they actually have a stabilizing quality more than a destabilizing one. >> It is not a bug by any definition of "bug" > > Most certainly it is: the documentation is either underspecified, > or deviates from the implementation (when taking the most plausible > interpretation). This is the very definition of "bug". The implementation is not a bug and neither was this a bug in the 2.x series of the Python documentation. The Python 3.x docs apparently introduced a reference to the language spec which is clearly not capturing the wealth of possible inputs. So, yes, we're talking about a documentation bug, but not an implementation bug. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Nov 29 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From g.brandl at gmx.net Thu Dec 2 21:05:05 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 02 Dec 2010 21:05:05 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF7F644.8020207@v.loewis.de> References: <4CF2DC5B.4020702@egenix.com> <4CF6CE95.1060707@v.loewis.de> <4CF7F644.8020207@v.loewis.de> Message-ID: Am 02.12.2010 20:40, schrieb "Martin v. L?wis": >> Maybe all past, present and future whatsnew maintainers can agree on these >> rules, which I copied directly from whatsnew/3.2.rst? > > I don't think all past maintainers can Yes, and the same goes for the future ones, since they may not even know yet that they will be whatsnew maintainers. Or maybe they aren't born yet (let's hope for a long life of Python 3...). > (I'm pretty certain that AMK > would disagree), but if that's the current policy, I can certainly try > following it (I didn't know it exists because I never look at the file). The large chunk of rules appeared in 2.6, where AMK still was maintainer. But even in the whatsnew for 2.4, there is this: .. Don't write extensive text for new sections; I'll do that. .. Feel free to add commented-out reminders of things that need .. to be covered. --amk But in any case, they are certainly valid for the current whatsnew -- even if Raymond likes to grumble about too expansive commits :) Georg From g.brandl at gmx.net Thu Dec 2 21:09:02 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 02 Dec 2010 21:09:02 +0100 Subject: [Python-Dev] Porting Ideas In-Reply-To: <20101202140650.11d96108@mission> References: <20101201201716.4357a316@pitrou.net> <20101202140650.11d96108@mission> Message-ID: Am 02.12.2010 20:06, schrieb Barry Warsaw: > On Dec 02, 2010, at 12:59 PM, Terry Reedy wrote: > >>On 12/2/2010 8:36 AM, Lennart Regebro wrote: >>> On Wed, Dec 1, 2010 at 20:17, Antoine Pitrou wrote: >>>> And I'm not sure what this package called "Python" is (?a high-level >>>> object-oriented programming language?? like Java?), but I'm pretty sure >>>> I've heard there's a Python 3 compatible version. >>> >>> Uhm... http://pypi.python.org/pypi/Python >>> >>> Anybody wanna remove that, or update it or something? :-) >> >>Entry is for Python 2.5. >> >># Package Index Owner: guido, anthony, barry > > Well, I definitely can't remember ever seeing that before. Of course, that > doesn't mean I haven't. ;) No idea what that entry is about. * Development Status :: 3 - Alpha * Development Status :: 6 - Mature Aha. Let's just delete it. > Aside: how does one log into the Cheeseshop with your Launchpad OpenID? When > I try to do it I end up on a "Manual user registration" page. I fill out the > username with what I think my PyPI user name is, and add my python.org email > address, but then it tells me 'barry' is already taken. Do I need some kind > of back door linking of my lp openid and my pypi user id? In addition to what Martin said, the "Claim OpenID" form is on the "Your Details" page. Georg From martin at v.loewis.de Thu Dec 2 21:23:41 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 02 Dec 2010 21:23:41 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF7FC01.80307@egenix.com> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> Message-ID: <4CF8004D.1080408@v.loewis.de> >> Then these users should speak up and indicate their need, or somebody >> should speak up and confirm that there are users who actually want >> '????.??' to denote 1234.56. To my knowledge, there is no writing >> system in which '????.??e4' means 12345600.0. > > I'm not sure what you're after here. That the current float() constructor accepts tons of bogus character strings and accepts them as numbers, and that it should stop doing so. > The decision to add this support was deliberate based on the desire > to support as much of the nice features of Unicode in Python as > we could. At least that was what was driving me at the time. At the time, this may have been the right thing to do. With the experience gained, we should now conclude to revert this particular aspect. > Some references you may want to read up on: > > http://en.wikipedia.org/wiki/Numbers_in_Chinese_culture > http://en.wikipedia.org/wiki/Vietnamese_numerals > http://en.wikipedia.org/wiki/Korean_numerals > http://en.wikipedia.org/wiki/Japanese_numerals I don't question that people use non-ASCII characters to denote numbers. I claim that the specific support in Python for that has no connection to reality. I further claim that the use of non-ASCII numbers is a local convention, and that if you provide a library to parse numbers, users (of that library) will somehow have to specify which notational convention(s) is reasonable for the input they have. > Even MS Office supports them: > > http://languages.siuc.edu/Chinese/Language_Settings.html That's printing, though, not parsing. Notice that Python does *not* currently support printing numbers in other scripts - even though this may actually be more useful than parsing. >>> Note that the support in float() (and the other numeric constructors) >>> to work with Unicode code points was explicitly added when Unicode >>> support was added to Python and has been available since Python 1.6. >> >> That doesn't necessarily make it useful. Alexander's complaint is that >> it makes Python unstable (i.e. changing as the UCD changes). > > If that were true, then all Unicode database (UCD) changes would make > Python unstable. That's indeed the case - they do (see the recent bug report on white space processing). However, any change makes Python unstable (in the sense that it can potentially break existing applications), and, in many cases, the risk of breaking something is well worth it. In the case of number parsing, I think Python would be better if float() rejected non-ASCII strings, and any support for such parsing should be redone correctly in a different place (preferably along with printing of numbers). >> Most certainly it is: the documentation is either underspecified, >> or deviates from the implementation (when taking the most plausible >> interpretation). This is the very definition of "bug". > > The implementation is not a bug and neither was this a bug in the > 2.x series of the Python documentation. Of course the 2.x documentation is wrong, in that it is severely underspecified, and the most straight-forward interpretation of the specific wording gives an incorrect impression of the implementation. > The Python 3.x docs apparently > introduced a reference to the language spec which is clearly not > capturing the wealth of possible inputs. Right - but only because the 2.x documentation *already* suggested that the supported syntax matches the literal syntax - as that's the most natural thing to assume. Regards, Martin From martin at v.loewis.de Thu Dec 2 21:24:43 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Dec 2010 21:24:43 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: Message-ID: <4CF8008B.60402@v.loewis.de> > Since discussion has trailed off without any blocking objections, I'm > accepting PEP 384. Martin, you may mark the PEP accepted and proceed > with merging the implementation for the beta on Saturday. Thanks! will do (I'll also take into consideration the proposed changes). Regards, Martin From dirkjan at ochtman.nl Thu Dec 2 21:29:52 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Thu, 2 Dec 2010 21:29:52 +0100 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: Message-ID: On Thu, Dec 2, 2010 at 20:24, Sridhar Ratnakumar wrote: > Also note that the dependency information is incomplete. Also, a python3 version of chardet is available (from the website only, looks like). Cheers, Dirkjan From ziade.tarek at gmail.com Thu Dec 2 21:48:00 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Dec 2010 21:48:00 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF8008B.60402@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> Message-ID: On Thu, Dec 2, 2010 at 9:24 PM, "Martin v. L?wis" wrote: >> Since discussion has trailed off without any blocking objections, I'm >> accepting PEP 384. Martin, you may mark the PEP accepted and proceed >> with merging the implementation for the beta on Saturday. > > Thanks! will do (I'll also take into consideration the proposed changes). I did not get an answer to my last mail about distutils / distutils2 > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com > -- Tarek Ziad? | http://ziade.org From martin at v.loewis.de Thu Dec 2 22:02:28 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Dec 2010 22:02:28 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> Message-ID: <4CF80964.4070300@v.loewis.de> Am 02.12.2010 21:48, schrieb Tarek Ziad?: > On Thu, Dec 2, 2010 at 9:24 PM, "Martin v. L?wis" wrote: >>> Since discussion has trailed off without any blocking objections, I'm >>> accepting PEP 384. Martin, you may mark the PEP accepted and proceed >>> with merging the implementation for the beta on Saturday. >> >> Thanks! will do (I'll also take into consideration the proposed changes). > > I did not get an answer to my last mail about distutils / distutils2 What was the question again, and whom did you want an answer from? Regards, Martin From ziade.tarek at gmail.com Thu Dec 2 22:11:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Dec 2010 22:11:20 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF80964.4070300@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> Message-ID: 2010/12/2 "Martin v. L?wis" : > Am 02.12.2010 21:48, schrieb Tarek Ziad?: >> On Thu, Dec 2, 2010 at 9:24 PM, "Martin v. L?wis" wrote: >>>> Since discussion has trailed off without any blocking objections, I'm >>>> accepting PEP 384. Martin, you may mark the PEP accepted and proceed >>>> with merging the implementation for the beta on Saturday. >>> >>> Thanks! will do (I'll also take into consideration the proposed changes). >> >> I did not get an answer to my last mail about distutils / distutils2 > > What was the question again, and whom did you want an answer from? You can read it in the archives here: http://mail.python.org/pipermail/python-dev/2010-November/106138.html tldr: The question was "Why not implementing this in Distutils2 ?" Your answer was "No, PEP 3149 was accepted, I will do this in Distutils1" My answer was "Having an accepted PEP does not imply your code lands in the sdtlib (like PEP 376 and 345)" So the question still stands: "Why not implementing this in Distutils2 ?" Regards Tarek > > Regards, > Martin > -- Tarek Ziad? | http://ziade.org From mal at egenix.com Thu Dec 2 22:14:34 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 02 Dec 2010 22:14:34 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF7F579.5050603@v.loewis.de> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> Message-ID: <4CF80C3A.9000705@egenix.com> "Martin v. L?wis" wrote: >> [...] >> For direct entry by an interactive user, yes. Why are some people in >> this discussion thinking only of direct entry by an interactive user? > > Ultimately, somebody will have entered the data. I don't think you really believe that all data processed by a computer was eventually manually entered by a someone :-) I already gave you a couple of examples of how such data can end up being input for Python number constructors. If you are still curious, please see the Wikipedia pages I linked to, or have a look at these keyboards: http://en.wikipedia.org/wiki/File:KB_Arabic_MAC.svg http://en.wikipedia.org/wiki/File:Keyboard_Layout_Sanskrit.png http://en.wikipedia.org/wiki/File:800px-KB_Thai_Kedmanee.png http://en.wikipedia.org/wiki/File:Tibetan_Keyboard.png http://en.wikipedia.org/wiki/File:KBD-DZ-noshift-2009.png (all referenced on http://en.wikipedia.org/wiki/Keyboard_layout) and then compare these to: http://www.unicode.org/Public/5.2.0/ucd/extracted/DerivedNumericType.txt Arabic numerals are being used a lot nowadays in Asian countries, but that doesn't mean that the native script versions are not being used anymore. Furthermore, data can well originate from texts that were written hundreds or even thousands of years ago, so there is plenty of material available for processing. Even if not entered directly, there are plenty of ways to convert Arabic numerals (or other numeral systems) to the above forms, e.g. in MS Office for Thai: http://office.microsoft.com/en-us/excel-help/convert-arabic-numbers-to-thai-text-format-HP003074364.aspx Anyway, as mentioned before: all this is really besides the point: If we want to support Unicode in Python, we have to also support conversion of numerals declared in Unicode into a form that can be processed by Python. Regardless of where such data originates. If we were not to follow this approach, we could just as well decide not support support reading Egyptian Hieroglyphs based on the argument that there's no keyboard to enter them... http://www.unicode.org/charts/PDF/U13000.pdf :-) (from http://www.unicode.org/charts/) >> Input from an existing text file, as I said earlier. > > Which *specific* existing text file? Have you actually *seen* such a > text file? Have you tried Google ? http://www.google.com/search?q=??? http://www.google.com/search?q=?+site%3Agov.lb Some examples: http://www.bdl.gov.lb/circ/intpdf/int123.pdf http://www.cdr.gov.lb/study/sdatl/Arabic/Chapter3.PDF http://www.batroun.gov.lb/PDF/Waredat2006.pdf (these all use http://en.wikipedia.org/wiki/Eastern_Arabic_numerals) >> Direct entry at the console is a red herring. > > And we don't need powerhouses because power comes out of the socket. Martin, the argument simply doesn't fit well with the discussion about Python and Unicode. We introduced Unicode in Python not because there was a need for each and every code point in Unicode, but because we wanted to adopt a standard which doesn't prefer any one way of writing things over another. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 02 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Thu Dec 2 22:19:40 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Dec 2010 22:19:40 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> Message-ID: <4CF80D6C.6050703@v.loewis.de> > So the question still stands: "Why not implementing this in Distutils2 ?" Because it then wouldn't be available in Python 3.2, which is the target release of the PEP. If that really causes too much pain, I'll refrain from making any changes to distutils; PEP 384 doesn't specify any changes, anyway. Regards, Martin From martin at v.loewis.de Thu Dec 2 22:27:39 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 02 Dec 2010 22:27:39 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF80C3A.9000705@egenix.com> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> <4CF80C3A.9000705@egenix.com> Message-ID: <4CF80F4B.1020905@v.loewis.de> > Arabic numerals are being used a lot nowadays in Asian countries, > but that doesn't mean that the native script versions are not > being used anymore. I never claimed that people are not using their local scripts to enter numbers. However, none of your examples is about Chinese numerals using an ASCII full stop as a decimal point. The only thing I claimed about usage (actually only repeating haiyang kang's earlier claim) is that nobody would enter Chinese numerals with a keyboard and then use full stop as the decimal separator. So all your counter-examples just don't apply - I don't deny them. Regards, Martin From steve at pearwood.info Thu Dec 2 22:30:38 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 03 Dec 2010 08:30:38 +1100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF8004D.1080408@v.loewis.de> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> Message-ID: <4CF80FFE.8020505@pearwood.info> Martin v. L?wis wrote: >>> Then these users should speak up and indicate their need, or somebody >>> should speak up and confirm that there are users who actually want >>> '????.??' to denote 1234.56. To my knowledge, there is no writing >>> system in which '????.??e4' means 12345600.0. >> I'm not sure what you're after here. > > That the current float() constructor accepts tons of bogus character > strings and accepts them as numbers, and that it should stop doing so. What bogus characters do the float() and int() constructors accept? As far as I can see, they only accepts numerals. [...] > Notice that Python does *not* currently support printing numbers in > other scripts - even though this may actually be more useful than > parsing. Lack of one function, even if more useful, does not imply that an existing function should be removed. [...] > In the case of number parsing, I think Python would be better if > float() rejected non-ASCII strings, and any support for such parsing > should be redone correctly in a different place (preferably along with > printing of numbers). So your problems with the current behaviour are: (1) in some unspecified way, it's not done correctly; (2) it belongs somewhere other than float() and int(). That second is awfully close to bike-shedding. Since you accept that Python *should* have the current behaviour, and Python *already* has the current behaviour, it seems strange that you are kicking up such a fuss merely to *move* the implementation of that behaviour out of the numeric constructors into some unspecified "different place". I think it would be constructive to explain: - how the current behaviour is incorrect; - your suggestions for correcting it; and - a concrete suggestion for where you would like to see the behaviour moved to, and why that would be better than where it currently is. -- Steven From ziade.tarek at gmail.com Thu Dec 2 22:33:24 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Dec 2010 22:33:24 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF80D6C.6050703@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> Message-ID: 2010/12/2 "Martin v. L?wis" : > >> So the question still stands: "Why not implementing this in Distutils2 ?" > > Because it then wouldn't be available in Python 3.2, which is the target > release of the PEP. The exact feature I am mentioning is the ability to compile extensions with new options, so I am not sure which PEP is involved since distutils changes refer to PEP 384 in the other PEP. I was told not to touch to Distutils code to avoid any regression since it's patched to the bones in third party products. So we decided to freeze distutils and add all new features in Distutils2, which is at alpha stage now. So this move seems contradictory to me. Grouping all new features in the new version and keep Distutils1 in maintenance mode seems to make more sense to me, if we want to make Distutils die and push forward Distutils2 for its new features etc. Or we might get back into backward hell again :) So, I am +1 on a patch on distutils2 and -1 on de-freezing Distutils for any new feature. > If that really causes too much pain, I'll refrain from making any > changes to distutils; PEP 384 doesn't specify any changes, anyway. That would be awesome, and we can work on a patch for distutils2 to provide that abi option. > > Regards, > Martin > -- Tarek Ziad? | http://ziade.org From alexander.belopolsky at gmail.com Thu Dec 2 22:34:34 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 2 Dec 2010 16:34:34 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <1291316131.3576.43.camel@localhost.localdomain> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> <1291308960.3576.28.camel@localhost.localdomain> <1291316131.3576.43.camel@localhost.localdomain> Message-ID: On Thu, Dec 2, 2010 at 1:55 PM, Antoine Pitrou wrote: .. > I don't think so. ?str.split() and str.splitlines() are also defined in > conformance to the SPEC, AFAIK. ?They certainly try to. You are joking, right? Where exactly does Unicode specify something like this: >>> ''.join('???'.split('\udf00\ud800')) '??' ? OK, splitting on a given separator has very little to do with Unicode or UCD, but str.splitlines() makes absolutely no attempt to conform to Unicode Standard Annex #14 ("Unicode line breaking algorithm"). Wait, UAX #14 is actually relevant to textwrap module which saw very little change since 2.x days. So, what exactly does str.splitlines() do? And which part of the Unicode standard defines how it is different from str.split(.., '\n')? Reference manual does not help me here either: """ str.splitlines([keepends]) Return a list of the lines in the string, breaking at line boundaries. Line breaks are not included in the resulting list unless keepends is given and true. """ http://docs.python.org/dev/library/stdtypes.html#str.splitlines From solipsis at pitrou.net Thu Dec 2 22:36:58 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 02 Dec 2010 22:36:58 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> <1291308960.3576.28.camel@localhost.localdomain> <1291316131.3576.43.camel@localhost.localdomain> Message-ID: <1291325818.3576.46.camel@localhost.localdomain> Le jeudi 02 d?cembre 2010 ? 16:34 -0500, Alexander Belopolsky a ?crit : > On Thu, Dec 2, 2010 at 1:55 PM, Antoine Pitrou wrote: > .. > > I don't think so. str.split() and str.splitlines() are also defined in > > conformance to the SPEC, AFAIK. They certainly try to. > > You are joking, right? Perhaps you could look at the implementation. From martin at v.loewis.de Thu Dec 2 22:39:20 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Dec 2010 22:39:20 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> Message-ID: <4CF81208.8090705@v.loewis.de> > I was told not to touch to Distutils code to avoid any regression > since it's patched to the bones in third party products. So we decided > to freeze distutils and add all new features in Distutils2, which is > at alpha stage now. So this move seems contradictory to me. I think it was a bad decision to freeze distutils, and "we" certainly didn't make that (not any we that includes me, that is). This freeze made the situation worse. IIRC, it was really the incompatible changes that made people ask you to stop changing distutils. Regards, Martin From martin at v.loewis.de Thu Dec 2 22:48:33 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 02 Dec 2010 22:48:33 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF80FFE.8020505@pearwood.info> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CF80FFE.8020505@pearwood.info> Message-ID: <4CF81431.2040900@v.loewis.de> Am 02.12.2010 22:30, schrieb Steven D'Aprano: > Martin v. L?wis wrote: >>>> Then these users should speak up and indicate their need, or somebody >>>> should speak up and confirm that there are users who actually want >>>> '????.??' to denote 1234.56. To my knowledge, there is no writing >>>> system in which '????.??e4' means 12345600.0. >>> I'm not sure what you're after here. >> >> That the current float() constructor accepts tons of bogus character >> strings and accepts them as numbers, and that it should stop doing so. > > What bogus characters do the float() and int() constructors accept? As > far as I can see, they only accepts numerals. Not bogus characters, but bogus character strings. E.g. strings that mix digits from different scripts, and mix them with the Python decimal separator. >> Notice that Python does *not* currently support printing numbers in >> other scripts - even though this may actually be more useful than >> parsing. > > Lack of one function, even if more useful, does not imply that an > existing function should be removed. No. But if the specific function(ality) is not useful and underspecified, it should be removed. > So your problems with the current behaviour are: > > (1) in some unspecified way, it's not done correctly; No. My main concern is that it is not properly specified. If it was specified, I could then tell you what precisely is wrong about it. Right now, I can only give examples for input that it should not accept, and examples of input that it should, but does not accept. > (2) it belongs somewhere other than float() and int(). That's only because it also needs a parameter to specify what syntax to follow, somehow. That parameter could be explicit or implicit, and it could be to float or to some other function. But it must be available, and is not. > That second is awfully close to bike-shedding. Since you accept that > Python *should* have the current behaviour No, I don't. I think it behaves incorrectly, accepting garbage input and guessing some meaning out of it. > - how the current behaviour is incorrect; See above: it accepts strings that do not denote real numbers in any writing system, and, despite the claim that the feature is there to support other writing systems, actually does not truly support other writing systems. > - your suggestions for correcting it; and Make the current implementation exactly match the current documentation. I think the documentation is correct; the implementation is wrong. > - a concrete suggestion for where you would like to see the behaviour > moved to, and why that would be better than where it currently is. The current behavior should go nowhere; it is not useful. Something very similar to the current behavior (but done correctly) should go into the locale module. Regards, Martin From alexander.belopolsky at gmail.com Thu Dec 2 22:50:47 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 2 Dec 2010 16:50:47 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF80C3A.9000705@egenix.com> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> <4CF80C3A.9000705@egenix.com> Message-ID: On Thu, Dec 2, 2010 at 4:14 PM, M.-A. Lemburg wrote: .. > Have you tried Google ? > I tried google at I could not find any plain text or HTML file that would use Arabic-Indic numerals. What was interesting, though that a search for "quran unicode" (without quotes). Brought me to http://www.sacred-texts.com which says that they've been using unicode since 2002 in their archives. Interestingly enough, their version of Qur'an uses ordinary digits for ayah numbers. See, for example . I will change my mind on this issue when you present a machine-readable file with Arabic-Indic numerals and a program capable of reading it and show that this program uses the same number parsing algorithm as Python's int() or float(). From ziade.tarek at gmail.com Thu Dec 2 22:54:50 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Dec 2010 22:54:50 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF81208.8090705@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> Message-ID: 2010/12/2 "Martin v. L?wis" : >> I was told not to touch to Distutils code to avoid any regression >> since it's patched to the bones in third party products. So we decided >> to freeze distutils and add all new features in Distutils2, which is >> at alpha stage now. ?So this move seems contradictory to me. > > I think it was a bad decision to freeze distutils, and "we" certainly > didn't make that (not any we that includes me, that is). "We" is the people at the last language summit. Sorry if I used such a vague word. > This freeze made the situation worse. Can you extend on this and explains why it makes it worse ? If we (as you included) don't agree it's the best solution, I would not want to be pushed back to square one at the next summit.. I happily reverted all my changes last year when asked, and started to work on Distutils2. But I'll get out of steam if the direction changes again, with you stating that it makes the situation worse. > > IIRC, it was really the incompatible changes that made people ask you to > stop changing distutils. Who is "people" ? Are you suggesting that we could have added all the new features in Distutils in the stdlib ? The decision was because we had a mix of: - incompatible changes in private parts -- and some packages where patching distutils internals - changes on public APIs behavior, whith a behavior that was not clearly documented and suggest to interpretation - some mistakes I made as well But that's what you would expect for a project that needs to evolve a lot. Thus the freezing. So how would you make the situation better, if not by doing the work in distutils2 ? > Regards, > Martin > -- Tarek Ziad? | http://ziade.org From fuzzyman at voidspace.org.uk Thu Dec 2 22:54:54 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 02 Dec 2010 21:54:54 +0000 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF81208.8090705@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> Message-ID: <4CF815AE.1040404@voidspace.org.uk> On 02/12/2010 21:39, "Martin v. L?wis" wrote: >> I was told not to touch to Distutils code to avoid any regression >> since it's patched to the bones in third party products. So we decided >> to freeze distutils and add all new features in Distutils2, which is >> at alpha stage now. So this move seems contradictory to me. > I think it was a bad decision to freeze distutils, and "we" certainly > didn't make that (not any we that includes me, that is). This freeze > made the situation worse. What situation worse? "We" certainly did ask Tarek to become bdfl of distutils and fix/improve it (at a language summit 2 years ago). "We" then asked him to revert distutils and do the work in a new package instead of inside distutils (at the language summit this year). I would perhaps argue for a case by case exception on PEPs that *required* distutils support that are being accepted and implemented prior to distutils2 moving into the standard library. It doesn't sound like your changes are *required* by the PEP though. As I recall Tarek thought it was a bad idea to freeze distutils as well, but "we" insisted. :-) > IIRC, it was really the incompatible changes that made people ask you to > stop changing distutils. > Which included virtually any change to even private APIs. Given the issues freezing the distutils APIs except for essential bugfixes is a reasonable response. I don't know of any situation it has made worse. Things are getting very much better, but happening in distutils2 not distutils. All the best, Michael Foord > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From dickinsm at gmail.com Thu Dec 2 22:57:45 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Thu, 2 Dec 2010 21:57:45 +0000 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF8004D.1080408@v.loewis.de> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> Message-ID: On Thu, Dec 2, 2010 at 8:23 PM, "Martin v. L?wis" wrote: > In the case of number parsing, I think Python would be better if > float() rejected non-ASCII strings, and any support for such parsing > should be redone correctly in a different place (preferably along with > printing of numbers). +1. The set of strings currently accepted by the float constructor just seems too ad hoc to be at all useful. Apart from the decimal separator issue, and the question of exactly which decimal digits are accepted and which aren't, there are issues like this one: >>> x = '\uff11\uff25\uff0b\uff11\uff10' >>> x '?????' >>> float(x) Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'decimal' codec can't encode character '\uff25' in position 1: invalid decimal Unicode string >>> y = '\uff11E+\uff11\uff10' >>> y '?E+??' >>> float(y) 10000000000.0 That is, fullwidth *digits* are allowed, but none of the other characters can be fullwidth variants. Unfortunately, a float string doesn't consist solely of digits, and it seems to me to make little sense to allow variation in the digits without allowing corresponding variations in the other characters that might appear ('.', 'e', 'E', '+', '-'). A couple of slightly trickier decisions: (1) the float constructor currently does accept leading and trailing whitespace; should it allow any Unicode whitespace characters here? I'd say yes. (2) For int() rather than float(), there's a bit more value in allowing the variant digits, since it provides an easy way to interpret those digits. The decimal module currently makes use of this, for example (the decimal spec requires that non-European digits be accepted). I'd be happier if this functionality were moved elsewhere, though. The int constructor is, if anything, currently worse off than float, thanks to its attempts to support non-decimal bases. There's value in having an easy-to-specify, easy-to-maintain API for these basic builtin functions. For one thing, it helps non-CPython implementations. [MAL] >> The Python 3.x docs apparently >> introduced a reference to the language spec which is clearly not >> capturing the wealth of possible inputs. That documentation update was my fault; I was motivated to make the update by issues unrelated to this one (mostly to do with Python 3's more consistent handling of inf and nan, as a result of all the new float<->string conversion code). If I'd been thinking harder, I would have remembered that float accepted the non-European digits and added a note to that effect. This (unintentional) omission does underline the point that it's difficult right now to document and understand exactly what the float constructor does or doesn't accept. Mark From eric at trueblade.com Thu Dec 2 22:57:45 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 02 Dec 2010 16:57:45 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF81431.2040900@v.loewis.de> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CF80FFE.8020505@pearwood.info> <4CF81431.2040900@v.loewis.de> Message-ID: <4CF81659.4030302@trueblade.com> On 12/2/2010 4:48 PM, "Martin v. L?wis" wrote: > Am 02.12.2010 22:30, schrieb Steven D'Aprano: >> Martin v. L?wis wrote: >>>>> Then these users should speak up and indicate their need, or somebody >>>>> should speak up and confirm that there are users who actually want >>>>> '????.??' to denote 1234.56. To my knowledge, there is no writing >>>>> system in which '????.??e4' means 12345600.0. >>>> I'm not sure what you're after here. >>> >>> That the current float() constructor accepts tons of bogus character >>> strings and accepts them as numbers, and that it should stop doing so. >> >> What bogus characters do the float() and int() constructors accept? As >> far as I can see, they only accepts numerals. > > Not bogus characters, but bogus character strings. E.g. strings that mix > digits from different scripts, and mix them with the Python decimal > separator. > >>> Notice that Python does *not* currently support printing numbers in >>> other scripts - even though this may actually be more useful than >>> parsing. >> >> Lack of one function, even if more useful, does not imply that an >> existing function should be removed. > > No. But if the specific function(ality) is not useful and > underspecified, it should be removed. > >> So your problems with the current behaviour are: >> >> (1) in some unspecified way, it's not done correctly; > > No. My main concern is that it is not properly specified. If it was > specified, I could then tell you what precisely is wrong about it. > Right now, I can only give examples for input that it should not accept, > and examples of input that it should, but does not accept. > >> (2) it belongs somewhere other than float() and int(). > > That's only because it also needs a parameter to specify what syntax to > follow, somehow. That parameter could be explicit or implicit, and it > could be to float or to some other function. But it must be available, > and is not. > >> That second is awfully close to bike-shedding. Since you accept that >> Python *should* have the current behaviour > > No, I don't. I think it behaves incorrectly, accepting garbage input and > guessing some meaning out of it. > >> - how the current behaviour is incorrect; > > See above: it accepts strings that do not denote real numbers in any > writing system, and, despite the claim that the feature is there to > support other writing systems, actually does not truly support other > writing systems. > >> - your suggestions for correcting it; and > > Make the current implementation exactly match the current documentation. > I think the documentation is correct; the implementation is wrong. > >> - a concrete suggestion for where you would like to see the behaviour >> moved to, and why that would be better than where it currently is. > > The current behavior should go nowhere; it is not useful. Something very > similar to the current behavior (but done correctly) should go into the > locale module. I agree with everything Martin says here. I think the basic premise is: you won't find strings "in the wild" that use non-ASCII digits but do use the ASCII dot as a decimal point. And that's what float() is looking for. (And that doesn't even begin to address what it expects for an exponent 'e'.) Eric. From martin at v.loewis.de Thu Dec 2 23:17:05 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Dec 2010 23:17:05 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> Message-ID: <4CF81AE1.5050900@v.loewis.de> >> This freeze made the situation worse. > > Can you extend on this and explains why it makes it worse ? Before the freeze, distutils was unmaintained (i.e. before you started maintaining it), but people who want to improve it gradually atleast could. Now gradual improvements are also banned, so it's not only unmaintained, but I can't even provide support for the PEP in Python that was just accepted. >> IIRC, it was really the incompatible changes that made people ask you to >> stop changing distutils. > > Who is "people" ? Are you suggesting that we could have added all the > new features in Distutils in the stdlib ? No, only the ones that didn't cause backwards incompatibilities, and broke existing packages. > But that's what you would expect for a project that needs to evolve a > lot. Thus the freezing. Instead of evolving a lot, and instead of freezing, I would have preferred "evolve a little". > So how would you make the situation better, if not by doing the work > in distutils2 ? Lift the freeze. I'm all for replacing distutils with distutils2, but I'm not sure whether you will declare distutils2 ready tomorrow, next year, or ten years from now. Regards, Martin From martin at v.loewis.de Thu Dec 2 23:21:25 2010 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Dec 2010 23:21:25 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF815AE.1040404@voidspace.org.uk> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> Message-ID: <4CF81BE5.1060003@v.loewis.de> Am 02.12.2010 22:54, schrieb Michael Foord: > On 02/12/2010 21:39, "Martin v. L?wis" wrote: >>> I was told not to touch to Distutils code to avoid any regression >>> since it's patched to the bones in third party products. So we decided >>> to freeze distutils and add all new features in Distutils2, which is >>> at alpha stage now. So this move seems contradictory to me. >> I think it was a bad decision to freeze distutils, and "we" certainly >> didn't make that (not any we that includes me, that is). This freeze >> made the situation worse. > > What situation worse? The "distutils is unmaintained" situation. It's not only unmaintained now, but proposed improvements are rejected without consideration, on the grounds that they are changes. > I would perhaps argue for a case by case exception on PEPs that > *required* distutils support that are being accepted and implemented > prior to distutils2 moving into the standard library. It doesn't sound > like your changes are *required* by the PEP though. Well, the PEP 384 text in PEP 3149 specifies a change. It's not clear whether this change was accepted when PEP 3149 was accepted, or whether it was accepted when PEP 384 was accepted, or whether it was not accepted at all, or whether it was just proposed. In any case, without the change, you won't naturally get extension modules that use the abi3 tag proposed in 3149. Regards, Martin From ziade.tarek at gmail.com Thu Dec 2 23:23:21 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Dec 2010 23:23:21 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF81BE5.1060003@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> Message-ID: 2010/12/2 "Martin v. L?wis" : > Am 02.12.2010 22:54, schrieb Michael Foord: >> On 02/12/2010 21:39, "Martin v. L?wis" wrote: >>>> I was told not to touch to Distutils code to avoid any regression >>>> since it's patched to the bones in third party products. So we decided >>>> to freeze distutils and add all new features in Distutils2, which is >>>> at alpha stage now. ?So this move seems contradictory to me. >>> I think it was a bad decision to freeze distutils, and "we" certainly >>> didn't make that (not any we that includes me, that is). This freeze >>> made the situation worse. >> >> What situation worse? > > The "distutils is unmaintained" situation. It's not only unmaintained > now, but proposed improvements are rejected without consideration, on > the grounds that they are changes. I welcome those changes in Distutils2. That's the whole point. From martin at v.loewis.de Thu Dec 2 23:29:10 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Dec 2010 23:29:10 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> Message-ID: <4CF81DB6.7040209@v.loewis.de> >> The "distutils is unmaintained" situation. It's not only unmaintained >> now, but proposed improvements are rejected without consideration, on >> the grounds that they are changes. > > I welcome those changes in Distutils2. That's the whole point. That would be useful if there was a clear vision of when distutils2 will be released. Please understand that I'm not blaming you for not releasing it (it *is* too much for a single person), but please understand that it's also not helpful to submit changes to a codebase that is not going to be released in a foreseeable future. Regards, Martin From mal at egenix.com Thu Dec 2 23:43:12 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 02 Dec 2010 23:43:12 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF81659.4030302@trueblade.com> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CF80FFE.8020505@pearwood.info> <4CF81431.2040900@v.loewis.de> <4CF81659.4030302@trueblade.com> Message-ID: <4CF82100.5080308@egenix.com> Eric Smith wrote: >> The current behavior should go nowhere; it is not useful. Something very >> similar to the current behavior (but done correctly) should go into the >> locale module. > > I agree with everything Martin says here. I think the basic premise is: > you won't find strings "in the wild" that use non-ASCII digits but do > use the ASCII dot as a decimal point. And that's what float() is looking > for. (And that doesn't even begin to address what it expects for an > exponent 'e'.) http://en.wikipedia.org/wiki/Decimal_mark "In China, comma and space are used to mark digit groups because dot is used as decimal mark." Note that float() can also parse integers, it just returns them as floats :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 02 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ziade.tarek at gmail.com Thu Dec 2 23:44:57 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Dec 2010 23:44:57 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF81AE1.5050900@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> Message-ID: 2010/12/2 "Martin v. L?wis" : >>> This freeze made the situation worse. >> >> Can you extend on this and explains why it makes it worse ? > > Before the freeze, distutils was unmaintained (i.e. before you started > maintaining it), but people who want to improve it gradually atleast > could. Now gradual improvements are also banned, so it's not only > unmaintained, but I can't even provide support for the PEP in Python > that was just accepted. > >>> IIRC, it was really the incompatible changes that made people ask you to >>> stop changing distutils. >> >> Who is "people" ? Are you suggesting that we could have added all the >> new features in Distutils in the stdlib ? > > No, only the ones that didn't cause backwards incompatibilities, > and broke existing packages. This is impossible. I can point you to some third party project that can break if you touch some distutils internals, like setuptools. Setuptools also uses some privates global variables in some other modules in the stdlib FYI. The right answer was maybe back then: make setuptools and other projects evolve with distutils. But it did not happen. So we left the status quo and moved forward in distutils2. Because we knew distutils needed deeper changes anyways, and we knew setuptools was used everywhere and unfortunately not evolving at the same pace. (note: I am not blaming PJE or anyone when I say this -- the way distutils worked and was poorly maintained was the main reason) > >> But that's what you would expect for a project that needs to evolve a >> lot. Thus the freezing. > > Instead of evolving a lot, and instead of freezing, I would have > preferred "evolve a little". > >> So how would you make the situation better, if not by doing the work >> in distutils2 ? > > Lift the freeze. I'm all for replacing distutils with distutils2, but > I'm not sure whether you will declare distutils2 ready tomorrow, next > year, or ten years from now. Depends on what "ready" means. If by ready you mean it can be used to replace Distutils1 in a project, I declare Distutils2 ready for usage NOW. It's in alpha stage. I want a solid beta before Pycon. I would even remove Distutils from 3.x altogether at some point since setuptools is not Python 3 compatible, and just put distutils2. 3.3 sounds like a good target. Regards Tarek -- Tarek Ziad? | http://ziade.org From solipsis at pitrou.net Thu Dec 2 23:51:14 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 2 Dec 2010 23:51:14 +0100 Subject: [Python-Dev] PEP 384 accepted References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> Message-ID: <20101202235114.1dc77cdc@pitrou.net> On Thu, 02 Dec 2010 23:21:25 +0100 "Martin v. L?wis" wrote: > Am 02.12.2010 22:54, schrieb Michael Foord: > > On 02/12/2010 21:39, "Martin v. L?wis" wrote: > >>> I was told not to touch to Distutils code to avoid any regression > >>> since it's patched to the bones in third party products. So we decided > >>> to freeze distutils and add all new features in Distutils2, which is > >>> at alpha stage now. So this move seems contradictory to me. > >> I think it was a bad decision to freeze distutils, and "we" certainly > >> didn't make that (not any we that includes me, that is). This freeze > >> made the situation worse. > > > > What situation worse? > > The "distutils is unmaintained" situation. It's not only unmaintained > now, but proposed improvements are rejected without consideration, on > the grounds that they are changes. I think distutils is simply a bugfix branch for distutils2. Similarly as how we don't commit improvements in e.g. 2.7 or 3.1, neither do we commit improvements to distutils. (and I think that's how Guido wanted it anyway) Regards Antoine. From ziade.tarek at gmail.com Thu Dec 2 23:53:08 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Thu, 2 Dec 2010 23:53:08 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF81DB6.7040209@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> <4CF81DB6.7040209@v.loewis.de> Message-ID: 2010/12/2 "Martin v. L?wis" : >>> The "distutils is unmaintained" situation. It's not only unmaintained >>> now, but proposed improvements are rejected without consideration, on >>> the grounds that they are changes. >> >> I welcome those changes in Distutils2. That's the whole point. > > That would be useful if there was a clear vision of when distutils2 > will be released. Please understand that I'm not blaming you for not > releasing it (it *is* too much for a single person), but please > understand that it's also not helpful to submit changes to a codebase > that is not going to be released in a foreseeable future. I know you're not blaming me. Distutils 2 alpha3 is currently released and available at PyPI. I use it in some of my professional projects FWIW. alpha4 was postponed but should be out this month. It contains major features, people from the GSOC worked on. The initial roadmap was to have a final by the time 3.2 final is out, but that'll be too short. So the target is to have a beta release for Pycon, and to sync the final release with 3.3, with lots of feedback in the meantime hopefully, and people using it from 2.4 onward. > > Regards, > Martin > -- Tarek Ziad? | http://ziade.org From barry at python.org Thu Dec 2 23:54:01 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 2 Dec 2010 17:54:01 -0500 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF81BE5.1060003@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> Message-ID: <20101202175401.1a94a5f4@mission> On Dec 02, 2010, at 11:21 PM, Martin v. L?wis wrote: >Well, the PEP 384 text in PEP 3149 specifies a change. It's not clear >whether this change was accepted when PEP 3149 was accepted, or whether >it was accepted when PEP 384 was accepted, or whether it was not >accepted at all, or whether it was just proposed. From my point of view, the PEP 3149 text is just a proposal. It leaves the final decision to PEP 384, but tries to address some of the issues raised during the PEP 3149 discussion. I think it is within PEP 384's scope to make the final decisions about it. >In any case, without the change, you won't naturally get extension >modules that use the abi3 tag proposed in 3149. I would favor changing distutils, if it can be done in a way that reasonably preserves backward compatibility. I suppose it's impossible to know all the ways 3rd party code has reached into distutils, but I think you can make fairly good judgements about whether a change is backward compatible or not. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From martin at v.loewis.de Thu Dec 2 23:54:56 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 02 Dec 2010 23:54:56 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> Message-ID: <4CF823C0.7070002@v.loewis.de> >> No, only the ones that didn't cause backwards incompatibilities, >> and broke existing packages. > > This is impossible. I can point you to some third party project that > can break if you touch some distutils internals, like setuptools. > Setuptools also uses some privates global variables in some other > modules in the stdlib FYI. So what would break if Extension accepted an abi= keyword parameter? >> Lift the freeze. I'm all for replacing distutils with distutils2, but >> I'm not sure whether you will declare distutils2 ready tomorrow, next >> year, or ten years from now. > > Depends on what "ready" means. Included in Python, so that changes become possible again. > If by ready you mean it can be used to replace Distutils1 in a > project, I declare Distutils2 ready for usage NOW. It's in alpha > stage. I want a solid beta before Pycon. > > I would even remove Distutils from 3.x altogether at some point since > setuptools is not Python 3 compatible, and just put distutils2. > > 3.3 sounds like a good target. So will distuils2 be released before that? If so, when? Regards, Martin From mal at egenix.com Thu Dec 2 23:58:07 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 02 Dec 2010 23:58:07 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> <4CF80C3A.9000705@egenix.com> Message-ID: <4CF8247F.4000603@egenix.com> Alexander Belopolsky wrote: > On Thu, Dec 2, 2010 at 4:14 PM, M.-A. Lemburg wrote: > .. >> Have you tried Google ? >> > > I tried google at I could not find any plain text or HTML file that > would use Arabic-Indic numerals. What was interesting, though that a > search for "quran unicode" (without quotes). Brought me to > http://www.sacred-texts.com which says that they've been using unicode > since 2002 in their archives. Interestingly enough, their version of > Qur'an uses ordinary digits for ayah numbers. See, for example > . > > I will change my mind on this issue when you present a > machine-readable file with Arabic-Indic numerals and a program capable > of reading it and show that this program uses the same number parsing > algorithm as Python's int() or float(). Have you had a look at the examples I posted ? They include texts and tables with numbers written using east asian arabic numerals. Here's an example of a a famous Chinese text using Chinese numerals: http://ctext.org/nine-chapters Unfortunately, the Chinese numerals are not listed in the Category "Nd", so Python won't be able to parse them. This has various reasons, it seems, one of them being that the numeral code points were not defined as range of code points. I'm sure you can find other books on mathematics in sanscrit or arabic scripts as well. But this whole branch of the discussion is not going to go anywhere. The point is that we support all of Unicode in Python, not just a fragment, and therefore the numeric constructors support all of Unicode. Using them, it's very easy to support numbers in all kinds of variants, whether bound to a locale or not. Adding more locale aware numeric parsers and formatters to the locale module, based on these APIs is certainly a good idea, but orthogonal to the ongoing discussion, IMO. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 02 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From mal at egenix.com Fri Dec 3 00:01:24 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 03 Dec 2010 00:01:24 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <87pqto6bnv.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF38861.5090309@egenix.com> <4CF3C477.1020007@egenix.com> Message-ID: <4CF82544.2080506@egenix.com> Terry Reedy wrote: > On 11/29/2010 10:19 AM, M.-A. Lemburg wrote: >> Nick Coghlan wrote: >>> On Mon, Nov 29, 2010 at 9:02 PM, M.-A. Lemburg wrote: >>>> If we would go down that road, we would also have to disable other >>>> Unicode features based on locale, e.g. whether to apply non-ASCII >>>> case mappings, what to consider whitespace, etc. >>>> >>>> We don't do that for a good reason: Unicode is supposed to be >>>> universal and not limited to a single locale. >>> >>> Because parsing numbers is about more than just the characters used >>> for the individual digits. There are additional semantics associated >>> with digit ordering (for any number) and decimal separators and >>> exponential notation (for floating point numbers) and those vary by >>> locale. We deliberately chose to make the builtin numeric parsers >>> unaware of all of those things, and assuming that we can simply parse >>> other digits as if they were their ASCII equivalents and otherwise >>> assume a C locale seems questionable. >> >> Sure, and those additional semantics are locale dependent, even >> between ASCII-only locales. However, that does not apply to the >> basic building blocks, the decimal digits themselves. >> >>> If the existing semantics can be adequately defined, documented and >>> defended, then retaining them would be fine. However, the language >>> reference needs to define the behaviour properly so that other >>> implementations know what they need to support and what can be chalked >>> up as being just an implementation accident of CPython. (As a point in >>> the plus column, both decimal.Decimal and fractions.Fraction were able >>> to handle the '????.??' example in a manner consistent with the int >>> and float handling) >> >> The support is built into the C API, so there's not really much >> surprise there. >> >> Regarding documentation, we'd just have to add that numbers may >> be made up of an Unicode code point in the category "Nd". >> >> See http://www.unicode.org/versions/Unicode5.2.0/ch04.pdf, section >> 4.6 for details.... >> >> """ >> Decimal digits form a large subcategory of numbers consisting of those >> digits that can be >> used to form decimal-radix numbers. They include script-specific >> digits, but exclude char- >> acters such as Roman numerals and Greek acrophonic numerals. (Note >> that<1, 5> = 15 = >> fifteen, but = IV = four.) Decimal digits also exclude the >> compatibility subscript or >> superscript digits to prevent simplistic parsers from misinterpreting >> their values in context. >> """ >> >> int(), float() and long() (in Python2) are such simplistic >> parsers. > > Since you are the knowledgable advocate of the current behavior, perhaps > you could open an issue and propose a doc patch, even if not .rst > formatted. Good suggestion. I tried to collect as much context as possible: http://bugs.python.org/issue10610 I'll leave the rst-magic to someone else, but will certainly help if you have more questions about the details. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 02 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Fri Dec 3 00:01:52 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 03 Dec 2010 00:01:52 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <20101202235114.1dc77cdc@pitrou.net> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> <20101202235114.1dc77cdc@pitrou.net> Message-ID: <4CF82560.7020001@v.loewis.de> > I think distutils is simply a bugfix branch for distutils2. Similarly > as how we don't commit improvements in e.g. 2.7 or 3.1, neither do we > commit improvements to distutils. It's different, though, in the sense that Python has a release schedule and multiple committers working on it, and that it normally gets released even if some changes don't get included in a specific release yet. All this seems not to be true for distutils2. So my motivation to contribute changes to it is *much* lower than my desire to contribute to distutils, and it is also provably lower than my motivation to contribute to distribute (say). I'm just getting tired having to talk to five projects just to make a single change to the build infrastructure available to the Python community. Regards, Martin From alexander.belopolsky at gmail.com Fri Dec 3 00:11:24 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 2 Dec 2010 18:11:24 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF8247F.4000603@egenix.com> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> <4CF80C3A.9000705@egenix.com> <4CF8247F.4000603@egenix.com> Message-ID: On Thu, Dec 2, 2010 at 5:58 PM, M.-A. Lemburg wrote: .. >> I will change my mind on this issue when you present a >> machine-readable file with Arabic-Indic numerals and a program capable >> of reading it and show that this program uses the same number parsing >> algorithm as Python's int() or float(). > > Have you had a look at the examples I posted ? They include texts > and tables with numbers written using east asian arabic numerals. Yes, but this was all about output. I am pretty sure TeX was able to typeset Qur'an in all its glory long before Unicode was invented. Yet, in machine readable form it would be something like {\quran 1} (invented directive). I have asked for a file that is intended for machine processing, not for human enjoyment in print or on a display. I claim that if such file exists, the program that reads it does not use the same rules as Python and converting non-ascii digits would be a tiny portion of what that program does. From martin at v.loewis.de Fri Dec 3 00:19:20 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Dec 2010 00:19:20 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF82100.5080308@egenix.com> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CF80FFE.8020505@pearwood.info> <4CF81431.2040900@v.loewis.de> <4CF81659.4030302@trueblade.com> <4CF82100.5080308@egenix.com> Message-ID: <4CF82978.1060406@v.loewis.de> Am 02.12.2010 23:43, schrieb M.-A. Lemburg: > Eric Smith wrote: >>> The current behavior should go nowhere; it is not useful. Something very >>> similar to the current behavior (but done correctly) should go into the >>> locale module. >> >> I agree with everything Martin says here. I think the basic premise is: >> you won't find strings "in the wild" that use non-ASCII digits but do >> use the ASCII dot as a decimal point. And that's what float() is looking >> for. (And that doesn't even begin to address what it expects for an >> exponent 'e'.) > > http://en.wikipedia.org/wiki/Decimal_mark > > "In China, comma and space are used to mark digit groups because dot is used as decimal mark." I may be misinterpreting that, but I think that refers to the case of writing numbers using Arabic digits. "Chinese" digits are, e.g., used in the Suzhou numerals http://en.wikipedia.org/wiki/Suzhou_numerals This doesn't have a decimal point at all. Instead, the second line (below or left to the actual digits) describes the power of ten and the unit of measurement (i.e. similar to scientific notation, but with ideographs for the powers of ten). In another writing system, they use ? (U+70B9) as the decimal separator, see http://en.wikipedia.org/wiki/Chinese_numerals#Fractional_values In the same system, the integral part uses multipliers, i.e. 12345 is [1][10000][2][1000][3][100][4][10][5]; the fractional part uses regular digits. Regards, Martin From eric at trueblade.com Thu Dec 2 23:45:19 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 02 Dec 2010 17:45:19 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF82100.5080308@egenix.com> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CF80FFE.8020505@pearwood.info> <4CF81431.2040900@v.loewis.de> <4CF81659.4030302@trueblade.com> <4CF82100.5080308@egenix.com> Message-ID: <4CF8217F.2010005@trueblade.com> On 12/2/2010 5:43 PM, M.-A. Lemburg wrote: > Eric Smith wrote: >>> The current behavior should go nowhere; it is not useful. Something very >>> similar to the current behavior (but done correctly) should go into the >>> locale module. >> >> I agree with everything Martin says here. I think the basic premise is: >> you won't find strings "in the wild" that use non-ASCII digits but do >> use the ASCII dot as a decimal point. And that's what float() is looking >> for. (And that doesn't even begin to address what it expects for an >> exponent 'e'.) > > http://en.wikipedia.org/wiki/Decimal_mark > > "In China, comma and space are used to mark digit groups because dot is used as decimal mark." Is that an ASCII dot? That page doesn't say. > Note that float() can also parse integers, it just returns them as > floats :-) :) From fuzzyman at voidspace.org.uk Fri Dec 3 00:23:46 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 02 Dec 2010 23:23:46 +0000 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF82560.7020001@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> <20101202235114.1dc77cdc@pitrou.net> <4CF82560.7020001@v.loewis.de> Message-ID: <4CF82A82.2020804@voidspace.org.uk> On 02/12/2010 23:01, "Martin v. L?wis" wrote: >> [snip...] > I'm just getting tired having to talk to > five projects just to make a single change to the build infrastructure > available to the Python community. > The very best hope of resolving that particular problem is distutils2. :-) distutils2 is *already* available to the Python community, and whether or not there is a fixed release date it will have betas and then a 1.0 release in the foreseeable future. The team working on it has made an enormous amount of progress. We're much better off as a development community putting our support and energy into distutils2 rather than pining for evolution of distutils. Michael > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From ziade.tarek at gmail.com Fri Dec 3 00:25:48 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 3 Dec 2010 00:25:48 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF823C0.7070002@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF823C0.7070002@v.loewis.de> Message-ID: 2010/12/2 "Martin v. L?wis" : >>> No, only the ones that didn't cause backwards incompatibilities, >>> and broke existing packages. >> >> This is impossible. I can point you to some third party project that >> can break if you touch some distutils internals, like setuptools. >> Setuptools also uses some privates global variables in some other >> modules in the stdlib FYI. > > So what would break if Extension accepted an abi= keyword parameter? I suppose you have code behind this, that will be in build_ext and in the compilers. So you will need to try out ALL projects out there that customize build_ext, like numpy or setuptools, etc, But you won't be able to try out all projects because they are not listed somewhere. For starters, the Extension class is replaced by another one in setuptools, that patches the constructor if Pyrex is installed, which is unlikely I guess, so no big deal. But you will also get a replaced version of the Distribution class that uses a private method from distutils, and another version of build_ext with custom compiling flags. Now depending on how you do your thing it could work if you are careful at doing things on the top of setuptools. And then, if numpy.distutils is installed, it relies on distutils build_ext and tries to rely on setuptools one's too, so it gets in the mix of the patched classes, and you get an horrible mix and possible bad interactions. So I am not saying it's impossible to add the feature, but it is impossible to be sure nothing gets broken in third party. So the freeze seems wise indeed >>> Lift the freeze. I'm all for replacing distutils with distutils2, but >>> I'm not sure whether you will declare distutils2 ready tomorrow, next >>> year, or ten years from now. >> >> Depends on what "ready" means. > > Included in Python, so that changes become possible again. > >> If by ready you mean it can be used to replace Distutils1 in a >> project, I declare Distutils2 ready for usage NOW. ?It's in alpha >> stage. I want a solid beta before Pycon. >> >> I would even remove Distutils from 3.x altogether at some point since >> setuptools is not Python 3 compatible, and just put distutils2. >> >> 3.3 sounds like a good target. > > So will distuils2 be released before that? If so, when? An alpha is already released. A beta will be released for Pycon (I need it for my talk :) ) Then hopefully the final before 3.2 > Regards, > Martin > -- Tarek Ziad? | http://ziade.org From martin at v.loewis.de Fri Dec 3 00:27:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 03 Dec 2010 00:27:16 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF8247F.4000603@egenix.com> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> <4CF80C3A.9000705@egenix.com> <4CF8247F.4000603@egenix.com> Message-ID: <4CF82B54.8090000@v.loewis.de> > The point is that we support all of Unicode in Python, not just a fragment, > and therefore the numeric constructors support all of Unicode. That conclusion is as false today as it was in Python 1.6, but only now people start caring about that. a) we don't support all of Unicode in numeric constructors. There are lots of things that you can write down that readers would recognize as a real/rational/integral number that float() won't parse. b) if float() would restrict itself to the scientific notation of real numbers (as it should), Python could well continue to claim all of Unicode. > Adding more locale aware numeric parsers and formatters to the > locale module, based on these APIs is certainly a good idea, > but orthogonal to the ongoing discussion, IMO. Not at all. The concept of "Unicode numbers" is flawed: Unicode does *not* prescribe any specific way to denote numbers. Unicode is about characters, and Python supports the Unicode characters for digits as well as it supports all the other Unicode characters. Instead, support for non-scientific notation of real numbers should be based on user needs, which probably can be approximated by looking at actual scripts. This, in turn, is inherently locale-dependent. Regards, Martin From mal at egenix.com Fri Dec 3 00:37:13 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 03 Dec 2010 00:37:13 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF8217F.2010005@trueblade.com> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CF80FFE.8020505@pearwood.info> <4CF81431.2040900@v.loewis.de> <4CF81659.4030302@trueblade.com> <4CF82100.5080308@egenix.com> <4CF8217F.2010005@trueblade.com> Message-ID: <4CF82DA9.4070100@egenix.com> Eric Smith wrote: > On 12/2/2010 5:43 PM, M.-A. Lemburg wrote: >> Eric Smith wrote: >>>> The current behavior should go nowhere; it is not useful. Something >>>> very >>>> similar to the current behavior (but done correctly) should go into the >>>> locale module. >>> >>> I agree with everything Martin says here. I think the basic premise is: >>> you won't find strings "in the wild" that use non-ASCII digits but do >>> use the ASCII dot as a decimal point. And that's what float() is looking >>> for. (And that doesn't even begin to address what it expects for an >>> exponent 'e'.) >> >> http://en.wikipedia.org/wiki/Decimal_mark >> >> "In China, comma and space are used to mark digit groups because dot >> is used as decimal mark." > > Is that an ASCII dot? That page doesn't say. Yes, but to be fair: I think that the page actually refers to the use of the Arabic numeral format in China, rather than with their own script symbols. >> Note that float() can also parse integers, it just returns them as >> floats :-) > > :) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 02 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From martin at v.loewis.de Fri Dec 3 00:38:36 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 03 Dec 2010 00:38:36 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <20101202175401.1a94a5f4@mission> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> <20101202175401.1a94a5f4@mission> Message-ID: <4CF82DFC.8090301@v.loewis.de> > From my point of view, the PEP 3149 text is just a proposal. It leaves the > final decision to PEP 384, but tries to address some of the issues raised > during the PEP 3149 discussion. I think it is within PEP 384's scope to make > the final decisions about it. Ok, then it looks like there just won't be any support for module tagging of ABI-conforming modules. It might be possible to support something like this in the import code, but I would consider this pointless without accompanying distutils support. Then, by default, the modules just use the ABI tag that distutils assigns to them by default. It's interesting to note that #9807 got into distutils despite it being frozen (but this is not about ABI tags, right - so does distutils in 3.2 actually assign any ABI tag at all?) > I would favor changing distutils, if it can be done in a way that reasonably > preserves backward compatibility. It seems this is right out for policy reasons. Regards, Martin From martin at v.loewis.de Fri Dec 3 00:42:10 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 03 Dec 2010 00:42:10 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF823C0.7070002@v.loewis.de> Message-ID: <4CF82ED2.5020304@v.loewis.de> > An alpha is already released. A beta will be released for Pycon (I > need it for my talk :) ) Then hopefully the final before 3.2 Ok, that's promising. Regards, Martin From ziade.tarek at gmail.com Fri Dec 3 00:50:19 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Fri, 3 Dec 2010 00:50:19 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF82560.7020001@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> <20101202235114.1dc77cdc@pitrou.net> <4CF82560.7020001@v.loewis.de> Message-ID: On Fri, Dec 3, 2010 at 12:01 AM, "Martin v. L?wis" wrote: >> I think distutils is simply a bugfix branch for distutils2. Similarly >> as how we don't commit improvements in e.g. 2.7 or 3.1, neither do we >> commit improvements to distutils. > > It's different, though, in the sense that Python has a release schedule > and multiple committers working on it, and that it normally gets > released even if some changes don't get included in a specific release > yet. > > All this seems not to be true for distutils2. We have 3 or 4 regular contributors. That's not a lot for sure. > So my motivation to > contribute changes to it is *much* lower than my desire to contribute > to distutils, and it is also provably lower than my motivation to > contribute to distribute (say). I'm just getting tired having to talk to > five projects just to make a single change to the build infrastructure > available to the Python community. I am not trying to motivate you to contribute to Distutils2. I am trying to make sure we are all on the same page for what's good for Python. So if we work in Distutils2 and you work in Distutils saying publicly that you don't want to contribute to Distutils2, that's a total nonsense. We took some decisions, and you want to go against them. So I want to have a consensus here for the packaging eco-system and make sure we are still on track. I am sorry if you get tired of it, but I don't want to be told at the next summit: sorry Tarek, now we need to do changes little by little in distutils1 From amauryfa at gmail.com Fri Dec 3 00:51:37 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Fri, 3 Dec 2010 00:51:37 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF82A82.2020804@voidspace.org.uk> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> <20101202235114.1dc77cdc@pitrou.net> <4CF82560.7020001@v.loewis.de> <4CF82A82.2020804@voidspace.org.uk> Message-ID: Hi, 2010/12/3 Michael Foord > On 02/12/2010 23:01, "Martin v. L?wis" wrote: > >> [snip...] >>> >> I'm just getting tired having to talk to >> five projects just to make a single change to the build infrastructure >> available to the Python community. >> >> > The very best hope of resolving that particular problem is distutils2. :-) > > distutils2 is *already* available to the Python community, and whether or > not there is a fixed release date it will have betas and then a 1.0 release > in the foreseeable future. The team working on it has made an enormous > amount of progress. We're much better off as a development community putting > our support and energy into distutils2 rather than pining for evolution of > distutils. > Sure. But today (before 3.2b1) we want to merge PEP3149 and PEP384; they change the paths and filenames used by python. Either we modify distutils to comply with the new names, or defer these PEPs until distutils2 is ready. -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Dec 3 00:54:10 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 2 Dec 2010 18:54:10 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF80C3A.9000705@egenix.com> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> <4CF80C3A.9000705@egenix.com> Message-ID: On Thu, Dec 2, 2010 at 4:14 PM, M.-A. Lemburg wrote: .. > Some examples: > > http://www.bdl.gov.lb/circ/intpdf/int123.pdf I looked at this one more closely. While I cannot understand what it says, It appears that Arabic numerals are used in dates. It looks like Python want be able to deal with those: >>> datetime.strptime('????/??/??', '%Y/%m/%d') .. ValueError: time data '????/??/??' does not match format '%Y/%m/%d' Interestingly, >>> datetime.strptime('????', '%Y') datetime.datetime(1999, 1, 1, 0, 0) which further suggests that support of such numerals is accidental. As I think more about it, though I am becoming less avert to accepting these numerals for base 10 integers. Integers can be easily extracted from text using simple regex and '\d' accepts all category Nd characters. I would require though that all digits be from the same block, which is not hard because Unicode now promises to only have them in contiguous blocks of 10. This rule seems to address some of security issues because it is unlikely that a system that can display some of the local digits would not be able to display all of them properly. I still don't think it makes any sense to accept them in float(). From steve at pearwood.info Fri Dec 3 01:12:30 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 03 Dec 2010 11:12:30 +1100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4CF835EE.1060604@pearwood.info> Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > With full respect to haiyang kang, hear-say from one person can hardly > > be described as "strong" evidence > > That's *disrespectful* nonsense. What Haiyang reported was not > hearsay, it's direct observation of what he sees around him and > personal experience, plus extrapolation. Look up "hearsay," please. Fair enough. I choose my words poorly and apologise. A better description would be anecdotal evidence. -- Steven From fuzzyman at voidspace.org.uk Fri Dec 3 01:35:51 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 03 Dec 2010 00:35:51 +0000 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> <20101202235114.1dc77cdc@pitrou.net> <4CF82560.7020001@v.loewis.de> <4CF82A82.2020804@voidspace.org.uk> Message-ID: <4CF83B67.10809@voidspace.org.uk> On 02/12/2010 23:51, Amaury Forgeot d'Arc wrote: > Hi, > > 2010/12/3 Michael Foord > > > On 02/12/2010 23:01, "Martin v. L?wis" wrote: > > [snip...] > > I'm just getting tired having to talk to > five projects just to make a single change to the build > infrastructure > available to the Python community. > > > The very best hope of resolving that particular problem is > distutils2. :-) > > distutils2 is *already* available to the Python community, and > whether or not there is a fixed release date it will have betas > and then a 1.0 release in the foreseeable future. The team working > on it has made an enormous amount of progress. We're much better > off as a development community putting our support and energy into > distutils2 rather than pining for evolution of distutils. > > > Sure. But today (before 3.2b1) we want to merge PEP3149 and PEP384; > they change the paths and filenames used by python. > Either we modify distutils to comply with the new names, > or defer these PEPs until distutils2 is ready. Or put support for them into distutils2 now? Michael > > -- > Amaury Forgeot d'Arc > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies ("BOGUS AGREEMENTS") that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri Dec 3 02:07:34 2010 From: barry at python.org (Barry Warsaw) Date: Thu, 2 Dec 2010 20:07:34 -0500 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF815AE.1040404@voidspace.org.uk> <4CF81BE5.1060003@v.loewis.de> <20101202235114.1dc77cdc@pitrou.net> <4CF82560.7020001@v.loewis.de> <4CF82A82.2020804@voidspace.org.uk> Message-ID: <20101202200734.60649f05@mission> On Dec 03, 2010, at 12:51 AM, Amaury Forgeot d'Arc wrote: >Sure. But today (before 3.2b1) we want to merge PEP3149 and PEP384; >they change the paths and filenames used by python. >Either we modify distutils to comply with the new names, >or defer these PEPs until distutils2 is ready. I do not think it would be a good idea to revert PEP 3149. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From doko at ubuntu.com Fri Dec 3 02:48:25 2010 From: doko at ubuntu.com (Matthias Klose) Date: Fri, 03 Dec 2010 02:48:25 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF823C0.7070002@v.loewis.de> Message-ID: <4CF84C69.9020105@ubuntu.com> On 03.12.2010 00:25, Tarek Ziad? wrote: > 2010/12/2 "Martin v. L?wis": >>>> No, only the ones that didn't cause backwards incompatibilities, >>>> and broke existing packages. >>> >>> This is impossible. I can point you to some third party project that >>> can break if you touch some distutils internals, like setuptools. >>> Setuptools also uses some privates global variables in some other >>> modules in the stdlib FYI. >> >> So what would break if Extension accepted an abi= keyword parameter? > > I suppose you have code behind this, that will be in build_ext and in > the compilers. So you will need to try out ALL projects out there that > customize build_ext, like numpy or setuptools, etc, But you won't be > able to try out all projects because they are not listed somewhere. is this necessary? are all these projects known to work with 3.2, without having changes compared to 3.1 *without* this pep? hardly ... how many extensions will use this restricted api at all? Is it a legitimate solution to back up building an extension in the "default" mode? even without having any changes in distutils it would make sense to know if an extension can be built with the restricted ABI, so maybe it is better to defer any changes to the extension soname, and provide a check for an extension if it conforms to the restricted ABI, even if the extension still uses the python version specific soname. I did not mean to block this pep by choosing any installation names. Matthias From tjreedy at udel.edu Fri Dec 3 02:52:23 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 02 Dec 2010 20:52:23 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> <4CF80C3A.9000705@egenix.com> Message-ID: On 12/2/2010 6:54 PM, Alexander Belopolsky wrote: > On Thu, Dec 2, 2010 at 4:14 PM, M.-A. Lemburg wrote: > .. >> Some examples: >> >> http://www.bdl.gov.lb/circ/intpdf/int123.pdf > > I looked at this one more closely. While I cannot understand what it > says, It appears that Arabic numerals are used in dates. It looks > like Python want be able to deal with those: When I travelled in S. Asia around 25 years ago, arabic and indic numerals were in obvious use in stores, road signs, and banks (as with money exchange receipts). I learned the digits partly for self-protestions ;-). I have no real idea of what is done *now* in computerized business, but I assume the native digits are used. It may well be that there is no Python software yet that operates with native digits. The lack of direct output capability would hinder that. Of course, someone could run both input and output through language-specific str.translate digit translators. >>>> datetime.strptime('????/??/??', '%Y/%m/%d') Googling ???? gets about 83,000 hits. > .. > ValueError: time data '????/??/??' does not match format '%Y/%m/%d' > > Interestingly, > >>>> datetime.strptime('????', '%Y') > datetime.datetime(1999, 1, 1, 0, 0) > > which further suggests that support of such numerals is accidental. > > As I think more about it, though I am becoming less avert to accepting > these numerals for base 10 integers. Both input and output are needed for educational programming, though translation tables might be enough. > Integers can be easily extracted > from text using simple regex and '\d' accepts all category Nd > characters. I would require though that all digits be from the same > block, which is not hard because Unicode now promises to only have > them in contiguous blocks of 10. That seems sensible. > This rule seems to address some of > security issues because it is unlikely that a system that can display > some of the local digits would not be able to display all of them > properly. > > I still don't think it makes any sense to accept them in float(). For the present, I would pretty well agree with that, at least until we know more. You have raised an important issue. It is a bit of a chicken and egg problem though. We will not really know what is needed until Python is used more in non-english/non-euro contexts, while such usage may await better support. -- Terry Jan Reedy From merwok at netwok.org Fri Dec 3 03:14:21 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 03 Dec 2010 03:14:21 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF84C69.9020105@ubuntu.com> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF823C0.7070002@v.loewis.de> <4CF84C69.9020105@ubuntu.com> Message-ID: <4CF8527D.9030503@netwok.org> > even without having any changes in distutils it would make sense to know if an > extension can be built with the restricted ABI, so maybe it is better to defer > any changes to the extension soname, and provide a check for an extension if it > conforms to the restricted ABI, even if the extension still uses the python > version specific soname. Python?s setup.py has an example in Martin?s branch: ext = Extension('xxlimited', ['xxlimited.c'], define_macros=[('Py_LIMITED_API', 1)]) This is possible with today?s distutils. I don?t know if it?s enough to build stable-ABI-conformant extension modules. Regards From stephen at xemacs.org Fri Dec 3 03:20:19 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 03 Dec 2010 11:20:19 +0900 Subject: [Python-Dev] Porting Ideas In-Reply-To: <4CF7F72A.6090406@v.loewis.de> References: <20101201201716.4357a316@pitrou.net> <20101202140650.11d96108@mission> <4CF7F72A.6090406@v.loewis.de> Message-ID: <87lj47y4v0.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > > Aside: how does one log into the Cheeseshop with your Launchpad OpenID? When > > I try to do it I end up on a "Manual user registration" page. I fill out the > > username with what I think my PyPI user name is, and add my python.org email > > address, but then it tells me 'barry' is already taken. Do I need some kind > > of back door linking of my lp openid and my pypi user id? > > Since the "barry" account already exists, you first need to log into > that (likely using a password). You can then claim the LP OpenID as > being associated with that account, and use LP in the future. It would be nice if the UI told users that, and offered an opportunity to log in. Better yet would be a option for an OpenID to claim a user name by giving the password for it (ie, automatically on a successful login from that page). From stephen at xemacs.org Fri Dec 3 03:27:45 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 03 Dec 2010 11:27:45 +0900 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87tyixxbet.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87k4jry4im.fsf@uwakimon.sk.tsukuba.ac.jp> Lennart Regebro writes: > 2010/12/2 Stephen J. Turnbull : > > T1000 = float('?.???') > > That was already discussed here, and it's clear that unicode does not > consider these characters to be something you can use in a decimal > number, and hence it's not broken. Huh? IOW, use Unicode features just because they're there, what the users want and use doesn't matter? The only evidence I've seen so far that this feature is anything but a a toy for a small faction of developers is Neil Hodgson's information that OOo will generate these kinds of digits (note that it *will* do Han! so the evidence is as good for users demanding Han numerals as for any other kind, Unicode.org definitions notwithstanding), and that DOS CP 864 contains the Indo/Arabic versions. Of course, it's quite possible that those were toys for the developers of those software packages too. From merwok at netwok.org Fri Dec 3 03:39:12 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 03 Dec 2010 03:39:12 +0100 Subject: [Python-Dev] Porting Ideas In-Reply-To: References: Message-ID: <4CF85850.2070305@netwok.org> Hi Prashant, Python 3 support in distutils2 is not entirely finished, it?s an interesting and challenging task. Another idea: convert the python.org internal scripts to use Python 3, for example starting with patches for http://code.python.org/hg/peps/ . This would not have any impact on the community, but it?s easy work that?d help the Python developers to eat their own dogfood. Regards From cornsea at gmail.com Fri Dec 3 04:18:43 2010 From: cornsea at gmail.com (haiyang kang) Date: Fri, 3 Dec 2010 11:18:43 +0800 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CF80C3A.9000705@egenix.com> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> <4CF80C3A.9000705@egenix.com> Message-ID: > Furthermore, data can well originate from texts that were written > hundreds or even thousands of years ago, so there is plenty of > material available for processing. humm..., for this, i think we need a special tuned language processing system to handle this, and one subsystem for one language :)... (sometimes a single word is not enough, we also need context) Take pi for example, in modern math, it is wrote as: 3.1415...; in old China, it is sometimes wrote as: ????? or ?????? or ??????; And if these texts are extracted through scanner (OCR or other image processing tech), in my POV, it is the job of this image processing subsystem (or some other subsystem between the image processing and database) to do the mapping between number and raw text data, example table in DB: text | raw data |raw image data -----------|---------------------------------|----------------------- 3.1415 | ????? | image... br, khy From stephen at xemacs.org Fri Dec 3 04:54:05 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 03 Dec 2010 12:54:05 +0900 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <87oc94y5q3.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <87hbevy0iq.fsf@uwakimon.sk.tsukuba.ac.jp> Neil Hodgson writes: > While I don't have Excel to test with, OpenOffice.org Calc will > display in Arabic or Han numerals using the NatNum format codes. Display is different from input, but at least this is concrete evidence. Will it accept Arabic on input? (Han might be too much to ask for since Unicode considers Han digits to be "impure".) > > Ditto Arabic, I would imagine; ISO 8859/6 (aka Latin/Arabic) does > > not contain the Arabic digits that have been presented here > > earlier AFAICT. > > DOS code page 864 does use 0xB0-0xB9 OK, Microsoft thought it would be useful. I'd still like to know whether people actually use them for input (or output, for that matter -- anybody have a corpus of Arabic Form 10-Ks to grep through?), but that's more concrete evidence than we've seen before. Thank you! From stephen at xemacs.org Fri Dec 3 05:58:55 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 03 Dec 2010 13:58:55 +0900 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <1291316131.3576.43.camel@localhost.localdomain> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> <1291308960.3576.28.camel@localhost.localdomain> <1291316131.3576.43.camel@localhost.localdomain> Message-ID: <87fwufxxio.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > The legacy format argument looks like a red herring to me. When > converting from a format to another it is the programmer's job to > his/her job right. Uhmmmmmm, the argument *for* this "feature" proposed by several people is that Python's numeric constructors do it (right) so that the programmer doesn't have to. If Python *doesn't* do it right, why should Python do it at all? From alexander.belopolsky at gmail.com Fri Dec 3 06:10:29 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 3 Dec 2010 00:10:29 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> Message-ID: On Thu, Dec 2, 2010 at 4:57 PM, Mark Dickinson wrote: .. > (the decimal spec requires that non-European digits be accepted). Mark, I think *requires* is too strong of a word to describe what the spec says. The decimal module documentation refers to two authorities: 1. IBM?s General Decimal Arithmetic Specification 2. IEEE standard 854-1987 The IEEE standards predates Unicode and unsurprisingly does not have anything related to the issue. the IBM's spec says the following in the Conversions section: """ It is recommended that implementations also provide additional number formatting routines (including some which are locale-dependent), and if available should accept non-European decimal digits in strings. """ http://speleotrove.com/decimal/daconvs.html This cannot possibly be interpreted as normative text. The emphasis is clearly on "formatting routines" with "non-European decimal digits" added as an afterthought. This recommendation can reasonably be interpreted as a requirement that conversion routines should accept what formatting routines can produce. In Python there are no formatting routines to produce non-European numerals, so there is no requirement to accept them in conversions. I don't think decimal module should support non-European decimal digits. The only place where it can make some sense is in int() because here we have a fighting chance of producing a reasonable definition. The motivating use case is conversion of numerical data extracted from text using simple '\d+' regex matches. Here is how I would do it: 1. String x of non-European decimal digits is only accepted in int(x), but not by int(x, 0) or int(x, 10). 2. If x contains one or more non-European digits, then (a) all digits must be from the same block: def basepoint(c): return ord(c) - unicodedata.digit(c) all(basepoint(c) == basepoint(x[0]) for c in x) -> True (b) and '+' or '-' sign is not alowed. 3. A character c is a digit if it matches '\d' regex. I think this means unicodedata.category(c) -> 'Nd'. Condition 2(b) is important because there is no clear way to define what is acceptable as '+' or '-' using Unicode character properties and not all number systems even support local form of negation. (It is also YAGNI.) From merwok at netwok.org Fri Dec 3 06:32:39 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 03 Dec 2010 06:32:39 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF81AE1.5050900@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> Message-ID: <4CF880F7.9000800@netwok.org> Le 02/12/2010 23:17, Martin v. L?wis a ?crit : > Before the freeze, distutils was unmaintained (i.e. before you started > maintaining it), but people who want to improve it gradually atleast > could. Now gradual improvements are also banned, so it's not only > unmaintained, but I can't even provide support for the PEP in Python > that was just accepted. I wonder what your definition of ?unmaintained? is. Tarek has been fixing bugs for two years, and recently I have been made a committer to assist him. It?s true that I?ve not been as active as I would have liked*, but I did fix some bugs, as I think you know, given that you?ve helped me in some reports. Sure, distutils is not as well-maintained as other modules, but a dozen bugs have been fixed by five or six of us since the revert. I do feel responsible for all 116 remaining bugs, and intend to address all of them. * This is partly normal, since I had warned before I was accepted as a committer that my time would be scarce for a year, partly due to the fact that I also do bug triage, doc work and patch reviews, and partly due to some personal problems with focusing. On the matter of freeze exceptions, there have been two: - reading the makefile with surogateescape error handler so that python can build with an ASCII locale in a non-ASCII path (haypo, #6011) - handle soabiflags (barry, #9807). I took part in the discussion before those changes and did not object to them: they are very small changes that enable a new feature of Python 3.2. Maybe I should have requested Tarek?s approval for those changes; he knows better than me how third parties may break because of changes that don?t seem to break anything. Regards From merwok at netwok.org Fri Dec 3 07:34:21 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 03 Dec 2010 07:34:21 +0100 Subject: [Python-Dev] Change to the Distutils / Distutils2 workflow In-Reply-To: References: Message-ID: <4CF88F6D.7000804@netwok.org> Hi everyone, I have sketched a workflow guide on http://wiki.python.org/moin/Distutils/FixingBugs Cheers From martin at v.loewis.de Fri Dec 3 08:16:35 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Dec 2010 08:16:35 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF8527D.9030503@netwok.org> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF823C0.7070002@v.loewis.de> <4CF84C69.9020105@ubuntu.com> <4CF8527D.9030503@netwok.org> Message-ID: <4CF89953.3070906@v.loewis.de> > Python?s setup.py has an example in Martin?s branch: > > ext = Extension('xxlimited', ['xxlimited.c'], > define_macros=[('Py_LIMITED_API', 1)]) > > > > This is possible with today?s distutils. I don?t know if it?s enough to > build stable-ABI-conformant extension modules. It is. However, there is also the proposal that they use an ABI tag in the SO name; having that generated automatically would require a distutils change. Regards, Martin From martin at v.loewis.de Fri Dec 3 08:31:27 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Dec 2010 08:31:27 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF880F7.9000800@netwok.org> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> Message-ID: <4CF89CCF.20307@v.loewis.de> > I wonder what your definition of ?unmaintained? is. In this specific case: doesn't get feature requests acted upon. I'm well aware that you are fixing bugs, and that is appreciated. > Sure, distutils is not as well-maintained as other modules, but a dozen > bugs have been fixed by five or six of us since the revert. I do feel > responsible for all 116 remaining bugs, and intend to address all of them. But if the resolution of the bug would require a new feature, your answer will be "this is going to be fixed in distutils2 (if at all), it's out of scope for distutils". Before, if the submitter contributed a patch, the patch was just unreviewed for a long time, unless one of the committers picked it up. Now, the patch will be rejected, which I consider worse - because the patch is not being rejected on its own merits, but just because of a policy decision to not improve distutils anymore. For example, I keep running into the issue that distutils doesn't currently support parallel builds. I have been pondering supporting "-j" for building extensions, using both unbounded "-j" and the GNU make style -jN build server. However, I know that the patch will be rejected, so I don't even start working on it. > On the matter of freeze exceptions, there have been two: > - reading the makefile with surogateescape error handler so that python > can build with an ASCII locale in a non-ASCII path (haypo, #6011) > - handle soabiflags (barry, #9807). > I took part in the discussion before those changes and did not object to > them: they are very small changes that enable a new feature of Python > 3.2. Maybe I should have requested Tarek?s approval for those changes; > he knows better than me how third parties may break because of changes > that don?t seem to break anything. I see. Now, I'd claim that the reasoning as to why an abi= parameter on Extension may break things also applies to the soabiflags: to support soabiflags, the INSTALL_SCHEMES syntax was modified. If the install command is subclassed, that could lead to funny interactions, e.g. where the subclass fails to put abiflags into config_vars. IIUC, subst_vars will then eventually raise a ValueError. I'm not saying that this is a likely scenario - only that the reasoning "if a change can possibly affect existing code, it should not be made" applies to essentially any change. So if you want to avoid breaking things with certainty, not even bug fixes would be acceptable. Regards, Martin From martin at v.loewis.de Fri Dec 3 08:33:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 03 Dec 2010 08:33:23 +0100 Subject: [Python-Dev] Porting Ideas In-Reply-To: <87lj47y4v0.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20101201201716.4357a316@pitrou.net> <20101202140650.11d96108@mission> <4CF7F72A.6090406@v.loewis.de> <87lj47y4v0.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4CF89D43.5030805@v.loewis.de> > It would be nice if the UI told users that, and offered an opportunity > to log in. > > Better yet would be a option for an OpenID to claim a user name by > giving the password for it (ie, automatically on a successful login > from that page). So many projects, so little time. Contributions are welcome. IOW, it's easier for me to educate users. Regards, Martin From victor.stinner at haypocalc.com Fri Dec 3 10:16:04 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 3 Dec 2010 10:16:04 +0100 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <20101202180651.F2AB0EE9CD@mail.python.org> References: <20101202180651.F2AB0EE9CD@mail.python.org> Message-ID: <201012031016.05063.victor.stinner@haypocalc.com> On Thursday 02 December 2010 19:06:51 georg.brandl wrote: > Author: georg.brandl > Date: Thu Dec 2 19:06:51 2010 > New Revision: 86934 > > Log: > #7475: add (un)transform method to bytes/bytearray and str, add back codecs > that can be used with them from Python 2. Oh no, someone did it. Was it really needed to reintroduce rot13 and friends? I'm not strongly opposed to .transform()/.untranform() if it can be complelty separated to text encodings (ascii, latin9, utf-8 & cie). But str.encode() and bytes.decode() do accept transform codec names and raise strange error messages. Quote of Martin von L?wis (#7475): "If the codecs are restored, one half of them becomes available to .encode/.decode methods, since the codec registry cannot tell which ones implement real character encodings, and which ones are other conversion methods. So adding them would be really confusing." >>> 'abc'.transform('hex') TypeError: 'str' does not support the buffer interface >>> b'abc'.transform('rot13') TypeError: expected an object with the buffer interface >>> b'abcd'.decode('hex') TypeError: decoder did not return a str object (type=bytes) >>> 'abc'.encode('rot13') TypeError: encoder did not return a bytes object (type=str) I don't like transform() and untransform() because I think that we should not add too much operations to the base types (bytes and str), and they do implicit module import. I prefer explicit module import (eg. import binascii; binascii.hexlify(b'to hex')). It remembers me PHP and it's ugly namespace with +5000 functions. I prefer Python because it uses smaller and more namespaces which are more specific and well defined. If we add email and compression functions to bytes, why not adding a web browser to the str? Victor From nyamatongwe at gmail.com Fri Dec 3 11:07:32 2010 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Fri, 3 Dec 2010 21:07:32 +1100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87hbevy0iq.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <87oc94y5q3.fsf@uwakimon.sk.tsukuba.ac.jp> <87hbevy0iq.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: Stephen J. Turnbull: > Will it accept Arabic on input? ?(Han might be too much to ask for > since Unicode considers Han digits to be "impure".) I couldn't find a direct way to input Arabic digits into OO Calc, the normal use of Alt+number didn't work in Calc although it did in WordPad where Alt+1632 is ? and so on. OO Calc does have settings in the Complex Text Layout section for choosing different numerals but I don't understand the interaction of choices here. Neil From mal at egenix.com Fri Dec 3 11:15:51 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 03 Dec 2010 11:15:51 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <87r5e236hj.fsf@benfinney.id.au> <4CF58387.1000609@v.loewis.de> <87k4ju2vvg.fsf@benfinney.id.au> <4CF6CDDD.4060109@v.loewis.de> <4CF6D7B2.6030506@pearwood.info> <87sjyhxarg.fsf@uwakimon.sk.tsukuba.ac.jp> <87bp552arq.fsf@benfinney.id.au> <4CF7F579.5050603@v.loewis.de> <4CF80C3A.9000705@egenix.com> <4CF8247F.4000603@egenix.com> Message-ID: <4CF8C357.4020608@egenix.com> Alexander Belopolsky wrote: > On Thu, Dec 2, 2010 at 5:58 PM, M.-A. Lemburg wrote: > .. >>> I will change my mind on this issue when you present a >>> machine-readable file with Arabic-Indic numerals and a program capable >>> of reading it and show that this program uses the same number parsing >>> algorithm as Python's int() or float(). >> >> Have you had a look at the examples I posted ? They include texts >> and tables with numbers written using east asian arabic numerals. > > Yes, but this was all about output. I am pretty sure TeX was able to > typeset Qur'an in all its glory long before Unicode was invented. > Yet, in machine readable form it would be something like {\quran 1} > (invented directive). I have asked for a file that is intended for > machine processing, not for human enjoyment in print or on a display. > I claim that if such file exists, the program that reads it does not > use the same rules as Python and converting non-ascii digits would be > a tiny portion of what that program does. Well, programs that take input from the keyboards I posted in this thread will have to deal with the digits. Since Python's input() accepts keyboard input, you have your use case :-) Seriously, I find the distinction between input and output forms of numerals somewhat misguided. Any output can also serve as input. For books and other printed material, images, etc. you have scanners and OCR. For screen output you have screen readers. For spreadsheets and data, you have CSV, TSV, XML, etc. etc. etc. Just for the fun of it, I created a CSV file with Thai and Dzongkha numerals (in addition to Arabic ones) using OpenOffice. Here's the cut and paste version: """ Numbers in various scripts Arabic Thai Dzongkha 1 ? ? 2 ? ? 3 ? ? 4 ? ? 5 ? ? 6 ? ? 7 ? ? 8 ? ? 9 ? ? 10 ?? ?? 11 ?? ?? 12 ?? ?? 13 ?? ?? 14 ?? ?? 15 ?? ?? 16 ?? ?? 17 ?? ?? 18 ?? ?? 19 ?? ?? 20 ?? ?? """ And here's the script that goes with it: import csv c = csv.reader(open('Numbers-in-various-scripts.csv')) headers = [c.next() for i in range(3)] while c: print [int(unicode(x, 'utf-8')) for x in c.next()] and the output using Python 2.7: [1, 1, 1] [2, 2, 2] [3, 3, 3] [4, 4, 4] [5, 5, 5] [6, 6, 6] [7, 7, 7] [8, 8, 8] [9, 9, 9] [10, 10, 10] [11, 11, 11] [12, 12, 12] [13, 13, 13] [14, 14, 14] [15, 15, 15] [16, 16, 16] [17, 17, 17] [18, 18, 18] [19, 19, 19] [20, 20, 20] If you need more such files, I can generate as many as you like ;-) I can send the OOo file as well, if you like to play around with it. I'd say: case closed :-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 03 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: Numbers-in-various-scripts.csv URL: From ncoghlan at gmail.com Fri Dec 3 11:53:04 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 3 Dec 2010 20:53:04 +1000 Subject: [Python-Dev] [Python-checkins] r86965 - python/branches/py3k/Lib/test/__main__.py In-Reply-To: <20101203104203.E5461EE982@mail.python.org> References: <20101203104203.E5461EE982@mail.python.org> Message-ID: On Fri, Dec 3, 2010 at 8:42 PM, michael.foord wrote: > +# When tests are run from the Python build directory, it is best practice > +# to keep the test files in a subfolder. ?It eases the cleanup of leftover > +# files using command "make distclean". > +if sysconfig.is_python_build(): > + ? ?TEMPDIR = os.path.join(sysconfig.get_config_var('srcdir'), 'build') > + ? ?TEMPDIR = os.path.abspath(TEMPDIR) > + ? ?if not os.path.exists(TEMPDIR): > + ? ? ? ?os.mkdir(TEMPDIR) > + ? ?regrtest.TEMPDIR = TEMPDIR If syconfig.is_python_build() returns False... > +# Define a writable temp dir that will be used as cwd while running > +# the tests. The name of the dir includes the pid to allow parallel > +# testing (see the -j option). > +TESTCWD = 'test_python_{}'.format(os.getpid()) > + > +TESTCWD = os.path.join(TEMPDIR, TESTCWD) ... this line is going to blow up with a NameError. I would suggest putting this common setup code into a _make_test_dir() helper function in regrtest, then have both regrtest and test.__main__ call it. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fuzzyman at voidspace.org.uk Fri Dec 3 11:56:21 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 03 Dec 2010 10:56:21 +0000 Subject: [Python-Dev] [Python-checkins] r86965 - python/branches/py3k/Lib/test/__main__.py In-Reply-To: References: <20101203104203.E5461EE982@mail.python.org> Message-ID: <4CF8CCD5.6000905@voidspace.org.uk> On 03/12/2010 10:53, Nick Coghlan wrote: > On Fri, Dec 3, 2010 at 8:42 PM, michael.foord > wrote: >> +# When tests are run from the Python build directory, it is best practice >> +# to keep the test files in a subfolder. It eases the cleanup of leftover >> +# files using command "make distclean". >> +if sysconfig.is_python_build(): >> + TEMPDIR = os.path.join(sysconfig.get_config_var('srcdir'), 'build') >> + TEMPDIR = os.path.abspath(TEMPDIR) >> + if not os.path.exists(TEMPDIR): >> + os.mkdir(TEMPDIR) >> + regrtest.TEMPDIR = TEMPDIR > If syconfig.is_python_build() returns False... > >> +# Define a writable temp dir that will be used as cwd while running >> +# the tests. The name of the dir includes the pid to allow parallel >> +# testing (see the -j option). >> +TESTCWD = 'test_python_{}'.format(os.getpid()) >> + >> +TESTCWD = os.path.join(TEMPDIR, TESTCWD) > ... this line is going to blow up with a NameError. > > I would suggest putting this common setup code into a _make_test_dir() > helper function in regrtest, then have both regrtest and test.__main__ > call it. > Ok, good suggestion. Thanks Michael > Cheers, > Nick. > -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From merwok at netwok.org Fri Dec 3 13:27:23 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 03 Dec 2010 13:27:23 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF89CCF.20307@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> Message-ID: <4CF8E22B.7050005@netwok.org> Le 03/12/2010 08:31, Martin v. L?wis a ?crit : >> I wonder what your definition of ?unmaintained? is. > In this specific case: doesn't get feature requests acted upon. Thanks for clarifying. I think that?s a stretch, but I see your meaning now. >> Sure, distutils is not as well-maintained as other modules, but a dozen >> bugs have been fixed by five or six of us since the revert. I do feel >> responsible for all 116 remaining bugs, and intend to address all of them. > > But if the resolution of the bug would require a new feature, your > answer will be "this is going to be fixed in distutils2 (if at all), > it's out of scope for distutils". Before, if the submitter contributed > a patch, the patch was just unreviewed for a long time, unless one > of the committers picked it up. Now, the patch will be rejected, which > I consider worse - because the patch is not being rejected on its own > merits, but just because of a policy decision to not improve distutils > anymore. The patch would not be rejected, but assigned to a different version. It?s not different than replying to an old bug with a patch for Python 2.5 and requesting that it be updated for py3k. It?s also not uncommon to have another contributor or a core dev updating the patch if the original poster does not reply. > For example, I keep running into the issue that distutils doesn't > currently support parallel builds. I have been pondering supporting > "-j" for building extensions, using both unbounded "-j" and the GNU make > style -jN build server. However, I know that the patch will be rejected, > so I don't even start working on it. This would be a very useful feature for distutils2. >> On the matter of freeze exceptions, there have been two: [snip] > I see. Now, I'd claim that the reasoning as to why an abi= parameter > on Extension may break things also applies to the soabiflags: > to support soabiflags, the INSTALL_SCHEMES syntax was modified. > If the install command is subclassed, that could lead to funny > interactions, e.g. where the subclass fails to put abiflags into > config_vars. IIUC, subst_vars will then eventually raise a ValueError. This is a concern. > I'm not saying that this is a likely scenario - only that the > reasoning "if a change can possibly affect existing code, it > should not be made" applies to essentially any change. So if you > want to avoid breaking things with certainty, not even bug > fixes would be acceptable. If we wanted 100% certainty, yes. Regards From solipsis at pitrou.net Fri Dec 3 13:51:06 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 03 Dec 2010 13:51:06 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <87fwufxxio.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> <1291308960.3576.28.camel@localhost.localdomain> <1291316131.3576.43.camel@localhost.localdomain> <87fwufxxio.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1291380666.3624.0.camel@localhost.localdomain> Le vendredi 03 d?cembre 2010 ? 13:58 +0900, Stephen J. Turnbull a ?crit : > Antoine Pitrou writes: > > > The legacy format argument looks like a red herring to me. When > > converting from a format to another it is the programmer's job to > > his/her job right. > > Uhmmmmmm, the argument *for* this "feature" proposed by several people > is that Python's numeric constructors do it (right) so that the > programmer doesn't have to. As far as I understand, Alexander was talking about a legacy pre-unicode text format. We don't have to support this. Regards Antoine. From solipsis at pitrou.net Fri Dec 3 15:00:15 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Dec 2010 15:00:15 +0100 Subject: [Python-Dev] r86962 - in python/branches/py3k: Doc/library/pydoc.rst Doc/whatsnew/3.2.rst Lib/pydoc.py Lib/test/test_pydoc.py Misc/ACKS Misc/NEWS References: <20101203092912.3D2A0EE981@mail.python.org> Message-ID: <20101203150015.62d89632@pitrou.net> On Fri, 3 Dec 2010 10:29:12 +0100 (CET) nick.coghlan wrote: > Author: nick.coghlan > Date: Fri Dec 3 10:29:11 2010 > New Revision: 86962 > > Log: > Improve Pydoc interactive browsing (#2001). Patch by Ron Adam. Tests seem to fail under Windows: ====================================================================== FAIL: test_url_requests (test.test_pydoc.PyDocUrlHandlerTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_pydoc.py", line 456, in test_url_requests self.assertEqual(result, title) AssertionError: 'Python: Read Error' != 'Python: getfile /D:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows\\build\\l [truncated]... - Python: Read Error + Python: getfile /D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\string.py (http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/3708/steps/test/logs/stdio) From rdmurray at bitdance.com Fri Dec 3 16:11:29 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 03 Dec 2010 10:11:29 -0500 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <201012031016.05063.victor.stinner@haypocalc.com> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> Message-ID: <20101203151129.74C5222A88F@kimball.webabinitio.net> On Fri, 03 Dec 2010 10:16:04 +0100, Victor Stinner wrote: > On Thursday 02 December 2010 19:06:51 georg.brandl wrote: > > Author: georg.brandl > > Date: Thu Dec 2 19:06:51 2010 > > New Revision: 86934 > > > > Log: > > #7475: add (un)transform method to bytes/bytearray and str, add back codecs > > that can be used with them from Python 2. > > Oh no, someone did it. Was it really needed to reintroduce rot13 and friends? > > I'm not strongly opposed to .transform()/.untranform() if it can be complelty > separated to text encodings (ascii, latin9, utf-8 & cie). But str.encode() and > bytes.decode() do accept transform codec names and raise strange error > messages. Quote of Martin von L??wis (#7475): > > "If the codecs are restored, one half of them becomes available to > .encode/.decode methods, since the codec registry cannot tell which > ones implement real character encodings, and which ones are other > conversion methods. So adding them would be really confusing." > > >>> 'abc'.transform('hex') > TypeError: 'str' does not support the buffer interface > >>> b'abc'.transform('rot13') > TypeError: expected an object with the buffer interface I find these 'buffer interface' error messages to be the most confusing error message I get out of Python3 no matter what context they show up in. I have no idea what they are telling me. That issue is more general than transform/untransform, but perhaps it could be fixed for transform/untransform in particular. > >>> b'abcd'.decode('hex') > TypeError: decoder did not return a str object (type=bytes) > >>> 'abc'.encode('rot13') > TypeError: encoder did not return a bytes object (type=str) These error messages make perfect sense to me. I think it is called "duck typing" :) > I don't like transform() and untransform() because I think that we should not > add too much operations to the base types (bytes and str), and they do > implicit module import. I prefer explicit module import (eg. import binascii; > binascii.hexlify(b'to hex')). It remembers me PHP and it's ugly namespace with > +5000 functions. I prefer Python because it uses smaller and more namespaces > which are more specific and well defined. If we add email and compression > functions to bytes, why not adding a web browser to the str? As MAL says, the codec machinery is a general purpose tool. I think it, and the transform methods, are a useful level of abstraction over a general class of problems. Please also recall that transform/untransform was discussed before the release of Python 3.0 and was approved at the time, but it just did not get implemented before the 3.0 release. -- R. David Murray www.bitdance.com From alexander.belopolsky at gmail.com Fri Dec 3 17:14:56 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 3 Dec 2010 11:14:56 -0500 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <20101203151129.74C5222A88F@kimball.webabinitio.net> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> Message-ID: On Fri, Dec 3, 2010 at 10:11 AM, R. David Murray wrote: .. > Please also recall that transform/untransform was discussed before > the release of Python 3.0 and was approved at the time, but it just > did not get implemented before the 3.0 release. > Can you provide a link? My search for transform on python-dev came out with http://mail.python.org/pipermail/python-dev/2010-June/100564.html where you seem to oppose these methods. Also, new methods to builtins fall under the language moratorium (but can be approved on a case-by-case basis): http://www.python.org/dev/peps/pep-3003/#case-by-case-exemptions Is there an effort to document these exceptions? I expected such approvals to be added to PEP 3003, but apparently this was not the case. From solipsis at pitrou.net Fri Dec 3 17:28:14 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Dec 2010 17:28:14 +0100 Subject: [Python-Dev] "buffer interface" messages References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> Message-ID: <20101203172814.6aa8ba0e@pitrou.net> On Fri, 03 Dec 2010 10:11:29 -0500 "R. David Murray" wrote: > > > > >>> 'abc'.transform('hex') > > TypeError: 'str' does not support the buffer interface > > >>> b'abc'.transform('rot13') > > TypeError: expected an object with the buffer interface > > I find these 'buffer interface' error messages to be the most confusing > error message I get out of Python3 no matter what context they show up > in. I have no idea what they are telling me. That issue is more > general than transform/untransform, but perhaps it could be fixed > for transform/untransform in particular. I agree. "buffer interface" is a technicality that the Python user doesn't do about (unless (s)he also writes C extensions). How about "expected a bytes-compatible object"? Regards Antoine. From ncoghlan at gmail.com Fri Dec 3 17:28:41 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Dec 2010 02:28:41 +1000 Subject: [Python-Dev] r86962 - in python/branches/py3k: Doc/library/pydoc.rst Doc/whatsnew/3.2.rst Lib/pydoc.py Lib/test/test_pydoc.py Misc/ACKS Misc/NEWS In-Reply-To: <20101203150015.62d89632@pitrou.net> References: <20101203092912.3D2A0EE981@mail.python.org> <20101203150015.62d89632@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 12:00 AM, Antoine Pitrou wrote: > On Fri, ?3 Dec 2010 10:29:12 +0100 (CET) > nick.coghlan wrote: >> Author: nick.coghlan >> Date: Fri Dec ?3 10:29:11 2010 >> New Revision: 86962 >> >> Log: >> Improve Pydoc interactive browsing (#2001). ?Patch by Ron Adam. > > Tests seem to fail under Windows: > > ====================================================================== > FAIL: test_url_requests (test.test_pydoc.PyDocUrlHandlerTest) > ---------------------------------------------------------------------- > Traceback (most recent call last): > ?File "D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\test\test_pydoc.py", line 456, in test_url_requests > ? ?self.assertEqual(result, title) > AssertionError: 'Python: Read Error' != 'Python: getfile /D:\\cygwin\\home\\db3l\\buildarea\\3.x.bolen-windows\\build\\l [truncated]... > - Python: Read Error > + Python: getfile /D:\cygwin\home\db3l\buildarea\3.x.bolen-windows\build\lib\string.py > > > (http://www.python.org/dev/buildbot/all/builders/x86%20XP-4%203.x/builds/3708/steps/test/logs/stdio) Yeah, Georg pointed that one out. My latest checkin *should* have fixed it (but we'll see what the buildbots have to say). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From victor.stinner at haypocalc.com Fri Dec 3 17:44:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 3 Dec 2010 17:44:17 +0100 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <20101203151129.74C5222A88F@kimball.webabinitio.net> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> Message-ID: <201012031744.17152.victor.stinner@haypocalc.com> On Friday 03 December 2010 16:11:29 R. David Murray wrote: > > >>> 'abc'.transform('hex') > > TypeError: 'str' does not support the buffer interface > > > > >>> b'abc'.transform('rot13') > > TypeError: expected an object with the buffer interface > > I find these 'buffer interface' error messages to be the most confusing > error message I get out of Python3 no matter what context they show up > in. I have no idea what they are telling me. That issue is more > general than transform/untransform, but perhaps it could be fixed > for transform/untransform in particular. If it's more general, let's open an issue for that: http://bugs.python.org/issue10616 > > >>> b'abcd'.decode('hex') > > TypeError: decoder did not return a str object (type=bytes) > > > > >>> 'abc'.encode('rot13') > > TypeError: encoder did not return a bytes object (type=str) > > These error messages make perfect sense to me. I think it > is called "duck typing" :) > > (...) > > Please also recall that transform/untransform was discussed before > the release of Python 3.0 and was approved at the time, but it just > did not get implemented before the 3.0 release. Ok. Victor From ncoghlan at gmail.com Fri Dec 3 17:45:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Dec 2010 02:45:42 +1000 Subject: [Python-Dev] "buffer interface" messages In-Reply-To: <20101203172814.6aa8ba0e@pitrou.net> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203172814.6aa8ba0e@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 2:28 AM, Antoine Pitrou wrote: > On Fri, 03 Dec 2010 10:11:29 -0500 > "R. David Murray" wrote: >> > >> > >>> 'abc'.transform('hex') >> > TypeError: 'str' does not support the buffer interface >> > >>> b'abc'.transform('rot13') >> > TypeError: expected an object with the buffer interface >> >> I find these 'buffer interface' error messages to be the most confusing >> error message I get out of Python3 no matter what context they show up >> in. ?I have no idea what they are telling me. ?That issue is more >> general than transform/untransform, but perhaps it could be fixed >> for transform/untransform in particular. > > I agree. "buffer interface" is a technicality that the Python user > doesn't do about (unless (s)he also writes C extensions). How about > "expected a bytes-compatible object"? Why not "binary data interface"? That's what they're actually looking for. It seems odd for 'rot13' to be throwing that error though. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Fri Dec 3 17:57:22 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Dec 2010 17:57:22 +0100 Subject: [Python-Dev] "buffer interface" messages In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203172814.6aa8ba0e@pitrou.net> Message-ID: <20101203175722.6124d0a6@pitrou.net> On Sat, 4 Dec 2010 02:45:42 +1000 Nick Coghlan wrote: > On Sat, Dec 4, 2010 at 2:28 AM, Antoine Pitrou wrote: > > On Fri, 03 Dec 2010 10:11:29 -0500 > > "R. David Murray" wrote: > >> > > >> > >>> 'abc'.transform('hex') > >> > TypeError: 'str' does not support the buffer interface > >> > >>> b'abc'.transform('rot13') > >> > TypeError: expected an object with the buffer interface > >> > >> I find these 'buffer interface' error messages to be the most confusing > >> error message I get out of Python3 no matter what context they show up > >> in. ?I have no idea what they are telling me. ?That issue is more > >> general than transform/untransform, but perhaps it could be fixed > >> for transform/untransform in particular. > > > > I agree. "buffer interface" is a technicality that the Python user > > doesn't do about (unless (s)he also writes C extensions). How about > > "expected a bytes-compatible object"? > > Why not "binary data interface"? That's what they're actually looking for. I don't think it's more understandable, since it's not a well-known Python concept. (and in this specific case, the transform() method is on str and bytes, not on arbitrary bytes-like objects) > It seems odd for 'rot13' to be throwing that error though. Agreed. Regards Antoine. From ncoghlan at gmail.com Fri Dec 3 18:05:32 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 4 Dec 2010 03:05:32 +1000 Subject: [Python-Dev] "buffer interface" messages In-Reply-To: <20101203175722.6124d0a6@pitrou.net> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203172814.6aa8ba0e@pitrou.net> <20101203175722.6124d0a6@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 2:57 AM, Antoine Pitrou wrote: > On Sat, 4 Dec 2010 02:45:42 +1000 > Nick Coghlan wrote: >> On Sat, Dec 4, 2010 at 2:28 AM, Antoine Pitrou wrote: >> > On Fri, 03 Dec 2010 10:11:29 -0500 >> > "R. David Murray" wrote: >> >> > >> >> > >>> 'abc'.transform('hex') >> >> > TypeError: 'str' does not support the buffer interface >> >> > >>> b'abc'.transform('rot13') >> >> > TypeError: expected an object with the buffer interface >> >> >> >> I find these 'buffer interface' error messages to be the most confusing >> >> error message I get out of Python3 no matter what context they show up >> >> in. ?I have no idea what they are telling me. ?That issue is more >> >> general than transform/untransform, but perhaps it could be fixed >> >> for transform/untransform in particular. >> > >> > I agree. "buffer interface" is a technicality that the Python user >> > doesn't do about (unless (s)he also writes C extensions). How about >> > "expected a bytes-compatible object"? >> >> Why not "binary data interface"? That's what they're actually looking for. > > I don't think it's more understandable, since it's not a well-known > Python concept. > (and in this specific case, the transform() method is on str and bytes, > not on arbitrary bytes-like objects) Indeed, "bytes-compatible" is likely to be the most meaningful phrase (and then we can have the bytes docs explain further as to what "bytes-compatible" means, likely by reference to memoryview). Although, as Victor's mod to the complex constructor error message shows, the default error message is still going to be overly specific in many cases that also accept non-binary data. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From status at bugs.python.org Fri Dec 3 18:07:01 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 3 Dec 2010 18:07:01 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20101203170701.A2821104031@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-11-26 - 2010-12-03) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2537 ( +4) closed 19861 (+69) total 22398 (+73) Open issues with patches: 1080 Issues opened (47) ================== #2690: Precompute range length and enhance range subscript support http://bugs.python.org/issue2690 reopened by ncoghlan #9333: Expose a way to enable os.symlink on Windows http://bugs.python.org/issue9333 reopened by brian.curtin #10544: yield expression inside generator expression does nothing http://bugs.python.org/issue10544 opened by Inyeol.Lee #10545: remove or rewrite "Using Backslash to Continue Statements" ant http://bugs.python.org/issue10545 opened by rurpy2 #10546: UTF-16-LE and UTF-16-BE support non-BMP characters http://bugs.python.org/issue10546 opened by haypo #10548: Error in setUp not reported as expectedFailure (unittest) http://bugs.python.org/issue10548 opened by michael.foord #10550: Windows: leak in test_concurrent_futures http://bugs.python.org/issue10550 opened by skrah #10551: mimetypes read from the registry should not overwrite standard http://bugs.python.org/issue10551 opened by kovid #10552: Tools/unicode/gencodec.py error http://bugs.python.org/issue10552 opened by belopolsky #10553: Add optimize argument to builtin compile() and byte-compilatio http://bugs.python.org/issue10553 opened by georg.brandl #10556: test_zipimport_support mucks up with modules http://bugs.python.org/issue10556 opened by pitrou #10557: Malformed error message from float() http://bugs.python.org/issue10557 opened by belopolsky #10558: non-standard processing of several configure options ignores " http://bugs.python.org/issue10558 opened by ned.deily #10559: NameError in tutorial/interpreter http://bugs.python.org/issue10559 opened by eric.araujo #10560: Fixes for Windows sources http://bugs.python.org/issue10560 opened by Carlo_Bramini #10563: Spurious newline in time.ctime http://bugs.python.org/issue10563 opened by Gerrit.Holl #10566: gdb debugging support additions (Tools/gdb/libpython.py) http://bugs.python.org/issue10566 opened by eggy #10570: curses.tigetstr() returns bytes, but curses.tparm() expects a http://bugs.python.org/issue10570 opened by jwilk #10571: "setup.py upload --sign" broken: TypeError: 'str' does not sup http://bugs.python.org/issue10571 opened by jwilk #10572: Move unittest test package to Lib/test http://bugs.python.org/issue10572 opened by michael.foord #10573: Consistency in unittest assert methods: order of actual, expec http://bugs.python.org/issue10573 opened by michael.foord #10574: email.header.decode_header fails if the string contains multip http://bugs.python.org/issue10574 opened by starsareblueandfaraway #10576: Add a progress callback to gcmodule http://bugs.python.org/issue10576 opened by krisvale #10577: (Fancy) URL opener stuck when trying to open redirected url http://bugs.python.org/issue10577 opened by xhresko #10580: Installer sentence in bold http://bugs.python.org/issue10580 opened by Retro #10581: Review and document string format accepted in numeric data typ http://bugs.python.org/issue10581 opened by belopolsky #10582: PyErr_PrintEx exits silently when passed SystemExit exception http://bugs.python.org/issue10582 opened by Marc.Horowitz #10583: Encoding issue with chm help in 2.7.1 http://bugs.python.org/issue10583 opened by flashk #10587: Document the meaning of str methods http://bugs.python.org/issue10587 opened by belopolsky #10588: imp.find_module raises unexpected SyntaxError http://bugs.python.org/issue10588 opened by emile.anclin #10589: I/O ABC docs should specify which methods have implementations http://bugs.python.org/issue10589 opened by stutzbach #10590: Parameter type error for xml.sax.parseString(string, ...) http://bugs.python.org/issue10590 opened by Thomas.Ryan #10592: pprint module doesn't work well with OrderedDicts http://bugs.python.org/issue10592 opened by mikez302 #10595: Adding a syslog.conf reader in syslog http://bugs.python.org/issue10595 opened by tarek #10596: modulo operator bug http://bugs.python.org/issue10596 opened by Sergio.??lut??in #10598: curses fails to import on Solaris http://bugs.python.org/issue10598 opened by rtarnell #10599: sgmllib.parse_endtag() is not respecting quoted text http://bugs.python.org/issue10599 opened by Michael.Brooks #10601: sys.displayhook: use backslashreplace error handler if repr(va http://bugs.python.org/issue10601 opened by haypo #10605: ElementTree documentation http://bugs.python.org/issue10605 opened by adrian_nye #10608: Add a section to Windows FAQ explaining os.symlink http://bugs.python.org/issue10608 opened by brian.curtin #10609: dbm documentation example doesn't work (iteritems()) http://bugs.python.org/issue10609 opened by sandro.tosi #10610: Correct the float(), int() and complex() documentation http://bugs.python.org/issue10610 opened by lemburg #10611: sys.exit() in a test causes a test run to die http://bugs.python.org/issue10611 opened by michael.foord #10612: StopTestRun exception to halt test run http://bugs.python.org/issue10612 opened by michael.foord #10614: ZipFile and CP932 encoding http://bugs.python.org/issue10614 opened by ocean-city #10615: Trivial mingw compile fixes http://bugs.python.org/issue10615 opened by jonny #10616: Change PyObject_AsCharBuffer() error message http://bugs.python.org/issue10616 opened by haypo Most recent 15 issues with no replies (15) ========================================== #10615: Trivial mingw compile fixes http://bugs.python.org/issue10615 #10612: StopTestRun exception to halt test run http://bugs.python.org/issue10612 #10611: sys.exit() in a test causes a test run to die http://bugs.python.org/issue10611 #10608: Add a section to Windows FAQ explaining os.symlink http://bugs.python.org/issue10608 #10605: ElementTree documentation http://bugs.python.org/issue10605 #10595: Adding a syslog.conf reader in syslog http://bugs.python.org/issue10595 #10590: Parameter type error for xml.sax.parseString(string, ...) http://bugs.python.org/issue10590 #10588: imp.find_module raises unexpected SyntaxError http://bugs.python.org/issue10588 #10583: Encoding issue with chm help in 2.7.1 http://bugs.python.org/issue10583 #10582: PyErr_PrintEx exits silently when passed SystemExit exception http://bugs.python.org/issue10582 #10573: Consistency in unittest assert methods: order of actual, expec http://bugs.python.org/issue10573 #10556: test_zipimport_support mucks up with modules http://bugs.python.org/issue10556 #10546: UTF-16-LE and UTF-16-BE support non-BMP characters http://bugs.python.org/issue10546 #10545: remove or rewrite "Using Backslash to Continue Statements" ant http://bugs.python.org/issue10545 #10538: PyArg_ParseTuple("s*") does not always incref object http://bugs.python.org/issue10538 Most recent 15 issues waiting for review (15) ============================================= #10615: Trivial mingw compile fixes http://bugs.python.org/issue10615 #10601: sys.displayhook: use backslashreplace error handler if repr(va http://bugs.python.org/issue10601 #10598: curses fails to import on Solaris http://bugs.python.org/issue10598 #10589: I/O ABC docs should specify which methods have implementations http://bugs.python.org/issue10589 #10576: Add a progress callback to gcmodule http://bugs.python.org/issue10576 #10571: "setup.py upload --sign" broken: TypeError: 'str' does not sup http://bugs.python.org/issue10571 #10566: gdb debugging support additions (Tools/gdb/libpython.py) http://bugs.python.org/issue10566 #10563: Spurious newline in time.ctime http://bugs.python.org/issue10563 #10559: NameError in tutorial/interpreter http://bugs.python.org/issue10559 #10558: non-standard processing of several configure options ignores " http://bugs.python.org/issue10558 #10557: Malformed error message from float() http://bugs.python.org/issue10557 #10553: Add optimize argument to builtin compile() and byte-compilatio http://bugs.python.org/issue10553 #10552: Tools/unicode/gencodec.py error http://bugs.python.org/issue10552 #10550: Windows: leak in test_concurrent_futures http://bugs.python.org/issue10550 #10546: UTF-16-LE and UTF-16-BE support non-BMP characters http://bugs.python.org/issue10546 Top 10 most discussed issues (10) ================================= #3243: Support iterable bodies in httplib http://bugs.python.org/issue3243 42 msgs #10557: Malformed error message from float() http://bugs.python.org/issue10557 34 msgs #10542: Py_UNICODE_NEXT and other macros for surrogates http://bugs.python.org/issue10542 23 msgs #10552: Tools/unicode/gencodec.py error http://bugs.python.org/issue10552 11 msgs #10576: Add a progress callback to gcmodule http://bugs.python.org/issue10576 11 msgs #9264: trace.py documentation is incomplete http://bugs.python.org/issue9264 10 msgs #10262: Add --soabi option to `configure` http://bugs.python.org/issue10262 10 msgs #10572: Move unittest test package to Lib/test http://bugs.python.org/issue10572 10 msgs #10544: yield expression inside generator expression does nothing http://bugs.python.org/issue10544 8 msgs #10551: mimetypes read from the registry should not overwrite standard http://bugs.python.org/issue10551 8 msgs Issues closed (64) ================== #2001: Pydoc interactive browsing enhancement http://bugs.python.org/issue2001 closed by ncoghlan #4113: Add custom __repr__ to functools.partial http://bugs.python.org/issue4113 closed by belopolsky #4335: inspect.getsourcelines ignores last line in module http://bugs.python.org/issue4335 closed by belopolsky #6594: json C serializer performance tied to structure depth on some http://bugs.python.org/issue6594 closed by pitrou #6628: IDLE freezes after encountering a syntax error http://bugs.python.org/issue6628 closed by ned.deily #6886: cgi.py runs python, not python3 http://bugs.python.org/issue6886 closed by eric.araujo #6887: executables in lib use /usr/bin/env python, not python3 http://bugs.python.org/issue6887 closed by eric.araujo #7830: Flatten nested functools.partial http://bugs.python.org/issue7830 closed by belopolsky #7911: unittest.TestCase.longMessage should default to True in Python http://bugs.python.org/issue7911 closed by michael.foord #8685: set(range(100000)).difference(set()) is slow http://bugs.python.org/issue8685 closed by pitrou #8710: test_xpickle: compat tests: workaround for missing test_suppor http://bugs.python.org/issue8710 closed by skrah #8805: urllib should support SSL contexts http://bugs.python.org/issue8805 closed by pitrou #8989: email.utils.make_msgid: specify domain http://bugs.python.org/issue8989 closed by r.david.murray #9198: Should repr() print unicode characters outside the BMP? http://bugs.python.org/issue9198 closed by haypo #9299: os.makedirs(): Add a keyword argument to suppress "File exists http://bugs.python.org/issue9299 closed by eric.araujo #9528: Add pure Python implementation of time module to CPython http://bugs.python.org/issue9528 closed by belopolsky #9573: importing a module that executes fork() raises RuntimeError http://bugs.python.org/issue9573 closed by ncoghlan #9598: untabify.py fails on files that contain non-ascii characters http://bugs.python.org/issue9598 closed by belopolsky #9873: urllib.parse: Allow bytes in some APIs that use string literal http://bugs.python.org/issue9873 closed by ncoghlan #9915: speeding up sorting with a key http://bugs.python.org/issue9915 closed by stutzbach #10047: python-2.6.6 coredump running newspipe http://bugs.python.org/issue10047 closed by wiz #10273: Clean-up Unittest API http://bugs.python.org/issue10273 closed by ezio.melotti #10323: Final state of underlying sequence in islice http://bugs.python.org/issue10323 closed by rhettinger #10360: _weakrefset.WeakSet.__contains__ should not propagate TypeErro http://bugs.python.org/issue10360 closed by georg.brandl #10383: test_os leaks under Windows http://bugs.python.org/issue10383 closed by skrah #10464: netrc module not parsing passwords containing #s. http://bugs.python.org/issue10464 closed by r.david.murray #10499: Modular interpolation in configparser http://bugs.python.org/issue10499 closed by lukasz.langa #10500: Palevo.DZ worm msix86 installer 3.x installer http://bugs.python.org/issue10500 closed by loewis #10518: Bring back callable() http://bugs.python.org/issue10518 closed by pitrou #10532: A bug related to matching the empty string http://bugs.python.org/issue10532 closed by ned.deily #10539: Regular expression not checking 'range' element on 1st char in http://bugs.python.org/issue10539 closed by georg.brandl #10547: FreeBSD: wrong value for LDSHARED in sysconfig http://bugs.python.org/issue10547 closed by skrah #10549: help(cls1) breaks when cls1 has staticmethod(cls2) attribute http://bugs.python.org/issue10549 closed by georg.brandl #10554: Context management support for subprocess.Popen http://bugs.python.org/issue10554 closed by eric.araujo #10555: AIX 5.3 - GCC - Python 2.7 Shared Library Support - Fatal Pyth http://bugs.python.org/issue10555 closed by Anurag.Chourasia #10561: The pdb command 'clear bpnumber' may delete more than one brea http://bugs.python.org/issue10561 closed by orsenthil #10562: Change 'j' for imaginary unit into an 'i' http://bugs.python.org/issue10562 closed by mark.dickinson #10564: maths float error http://bugs.python.org/issue10564 closed by mark.dickinson #10565: isinstance(x, collections.Iterator) can return True, when x is http://bugs.python.org/issue10565 closed by belopolsky #10567: Unicode space character \u200b unrecognised a space http://bugs.python.org/issue10567 closed by belopolsky #10568: Python and the Unicode Character Database http://bugs.python.org/issue10568 closed by belopolsky #10569: abc: `issubclass([], my_abstract_type)` raises exception http://bugs.python.org/issue10569 closed by amaury.forgeotdarc #10575: makeunicodedata.py does not support Unihan digit data http://bugs.python.org/issue10575 closed by loewis #10578: Add mock PyPI server to test distutils http://bugs.python.org/issue10578 closed by eric.araujo #10579: Is ``o[key]`` equivalent to PyMapping_HasKeyString? http://bugs.python.org/issue10579 closed by naoki #10584: Bad links in doc of zipfile.open http://bugs.python.org/issue10584 closed by georg.brandl #10585: 2.6 maintenance branch erroneously uses unittest.TestCase.asse http://bugs.python.org/issue10585 closed by barry #10586: Enhanced cache access API for functools.lru_cache http://bugs.python.org/issue10586 closed by ncoghlan #10591: test_os failure in refleak runs http://bugs.python.org/issue10591 closed by brian.curtin #10593: LRU Cache with maxsize=None http://bugs.python.org/issue10593 closed by rhettinger #10594: Typo in PyList_New doc. http://bugs.python.org/issue10594 closed by georg.brandl #10597: Py_SetPythonHome document shows same url twice. http://bugs.python.org/issue10597 closed by georg.brandl #10600: surrogateescape'd paths not readable on Windows XP. http://bugs.python.org/issue10600 closed by loewis #10602: Wrong assert* method in test_csv.test_register_kwargs masks er http://bugs.python.org/issue10602 closed by eric.araujo #10603: __file__ not usable, with certain paths on windows XP. http://bugs.python.org/issue10603 closed by r.david.murray #10604: Allocating too much memory for pathes http://bugs.python.org/issue10604 closed by georg.brandl #10606: Misspelling of Jamie Zawinski's surname in urllib.parse docstr http://bugs.python.org/issue10606 closed by dmalcolm #10607: Document of PyOS_(v)snprintf is wrong. http://bugs.python.org/issue10607 closed by naoki #10613: gzip._PaddedFile calls getattr with arguments in reversed orde http://bugs.python.org/issue10613 closed by eric.araujo #940286: pydoc.Helper.help() ignores input/output init parameters http://bugs.python.org/issue940286 closed by georg.brandl #1745035: DoS smtpd vulnerability http://bugs.python.org/issue1745035 closed by georg.brandl #1486713: HTMLParser : A auto-tolerant parsing mode http://bugs.python.org/issue1486713 closed by r.david.murray #975556: HTMLParser lukewarm on bogus bare attribute chars http://bugs.python.org/issue975556 closed by r.david.murray #1046092: HTMLParser fix to accept malformed tag attributes http://bugs.python.org/issue1046092 closed by r.david.murray From g.brandl at gmx.net Fri Dec 3 18:09:39 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 03 Dec 2010 18:09:39 +0100 Subject: [Python-Dev] "buffer interface" messages In-Reply-To: <20101203175722.6124d0a6@pitrou.net> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203172814.6aa8ba0e@pitrou.net> <20101203175722.6124d0a6@pitrou.net> Message-ID: Am 03.12.2010 17:57, schrieb Antoine Pitrou: > On Sat, 4 Dec 2010 02:45:42 +1000 > Nick Coghlan wrote: >> On Sat, Dec 4, 2010 at 2:28 AM, Antoine Pitrou wrote: >> > On Fri, 03 Dec 2010 10:11:29 -0500 >> > "R. David Murray" wrote: >> >> > >> >> > >>> 'abc'.transform('hex') >> >> > TypeError: 'str' does not support the buffer interface >> >> > >>> b'abc'.transform('rot13') >> >> > TypeError: expected an object with the buffer interface >> >> >> >> I find these 'buffer interface' error messages to be the most confusing >> >> error message I get out of Python3 no matter what context they show up >> >> in. I have no idea what they are telling me. That issue is more >> >> general than transform/untransform, but perhaps it could be fixed >> >> for transform/untransform in particular. >> > >> > I agree. "buffer interface" is a technicality that the Python user >> > doesn't do about (unless (s)he also writes C extensions). How about >> > "expected a bytes-compatible object"? >> >> Why not "binary data interface"? That's what they're actually looking for. > > I don't think it's more understandable, since it's not a well-known > Python concept. > (and in this specific case, the transform() method is on str and bytes, > not on arbitrary bytes-like objects) "object that can be handled as bytes"? "object that provides bytes"? >> It seems odd for 'rot13' to be throwing that error though. > > Agreed. rot13 is a str<->str codec. Georg From solipsis at pitrou.net Fri Dec 3 18:29:23 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 Dec 2010 18:29:23 +0100 Subject: [Python-Dev] "buffer interface" messages References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203172814.6aa8ba0e@pitrou.net> <20101203175722.6124d0a6@pitrou.net> Message-ID: <20101203182923.500be9da@pitrou.net> On Fri, 03 Dec 2010 18:09:39 +0100 Georg Brandl wrote: > Am 03.12.2010 17:57, schrieb Antoine Pitrou: > > On Sat, 4 Dec 2010 02:45:42 +1000 > > Nick Coghlan wrote: > >> On Sat, Dec 4, 2010 at 2:28 AM, Antoine Pitrou wrote: > >> > On Fri, 03 Dec 2010 10:11:29 -0500 > >> > "R. David Murray" wrote: > >> >> > > >> >> > >>> 'abc'.transform('hex') > >> >> > TypeError: 'str' does not support the buffer interface > >> >> > >>> b'abc'.transform('rot13') > >> >> > TypeError: expected an object with the buffer interface > >> >> > >> >> I find these 'buffer interface' error messages to be the most confusing > >> >> error message I get out of Python3 no matter what context they show up > >> >> in. I have no idea what they are telling me. That issue is more > >> >> general than transform/untransform, but perhaps it could be fixed > >> >> for transform/untransform in particular. > >> > > >> > I agree. "buffer interface" is a technicality that the Python user > >> > doesn't do about (unless (s)he also writes C extensions). How about > >> > "expected a bytes-compatible object"? > >> > >> Why not "binary data interface"? That's what they're actually looking for. > > > > I don't think it's more understandable, since it's not a well-known > > Python concept. > > (and in this specific case, the transform() method is on str and bytes, > > not on arbitrary bytes-like objects) > > "object that can be handled as bytes"? > "object that provides bytes"? > > >> It seems odd for 'rot13' to be throwing that error though. > > > > Agreed. > > rot13 is a str<->str codec. Then why does it claim to expect an object with the buffer interface? bytes has the buffer interface, while str doesn't ;) I'll also mention that getting a TypeError when you call a method with the correct type of argument is a bit surprising. cheers Antoine. From g.brandl at gmx.net Fri Dec 3 18:45:08 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 03 Dec 2010 18:45:08 +0100 Subject: [Python-Dev] "buffer interface" messages In-Reply-To: <20101203182923.500be9da@pitrou.net> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203172814.6aa8ba0e@pitrou.net> <20101203175722.6124d0a6@pitrou.net> <20101203182923.500be9da@pitrou.net> Message-ID: Am 03.12.2010 18:29, schrieb Antoine Pitrou: > On Fri, 03 Dec 2010 18:09:39 +0100 > Georg Brandl wrote: >> Am 03.12.2010 17:57, schrieb Antoine Pitrou: >> > On Sat, 4 Dec 2010 02:45:42 +1000 >> > Nick Coghlan wrote: >> >> On Sat, Dec 4, 2010 at 2:28 AM, Antoine Pitrou wrote: >> >> > On Fri, 03 Dec 2010 10:11:29 -0500 >> >> > "R. David Murray" wrote: >> >> >> > >> >> >> > >>> 'abc'.transform('hex') >> >> >> > TypeError: 'str' does not support the buffer interface >> >> >> > >>> b'abc'.transform('rot13') >> >> >> > TypeError: expected an object with the buffer interface >> >> >> >> >> >> I find these 'buffer interface' error messages to be the most confusing >> >> >> error message I get out of Python3 no matter what context they show up >> >> >> in. I have no idea what they are telling me. That issue is more >> >> >> general than transform/untransform, but perhaps it could be fixed >> >> >> for transform/untransform in particular. >> >> > >> >> > I agree. "buffer interface" is a technicality that the Python user >> >> > doesn't do about (unless (s)he also writes C extensions). How about >> >> > "expected a bytes-compatible object"? >> >> >> >> Why not "binary data interface"? That's what they're actually looking for. >> > >> > I don't think it's more understandable, since it's not a well-known >> > Python concept. >> > (and in this specific case, the transform() method is on str and bytes, >> > not on arbitrary bytes-like objects) >> >> "object that can be handled as bytes"? >> "object that provides bytes"? >> >> >> It seems odd for 'rot13' to be throwing that error though. >> > >> > Agreed. >> >> rot13 is a str<->str codec. > > Then why does it claim to expect an object with the buffer interface? > bytes has the buffer interface, while str doesn't ;) I'll leave it to you to figure it out. (Hint: look at the traceback.) > I'll also mention that getting a TypeError when you call a method with > the correct type of argument is a bit surprising. Why? The codec has the wrong type :) But seriously, MAL already proposed a bit more information on codecs, so that they can let it be known which types they convert between. Georg From rdmurray at bitdance.com Fri Dec 3 18:58:32 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 03 Dec 2010 12:58:32 -0500 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> Message-ID: <20101203175832.88F00236D8B@kimball.webabinitio.net> On Fri, 03 Dec 2010 11:14:56 -0500, Alexander Belopolsky wrote: > On Fri, Dec 3, 2010 at 10:11 AM, R. David Murray wrote: > .. > > Please also recall that transform/untransform was discussed before > > the release of Python 3.0 and was approved at the time, but it just > > did not get implemented before the 3.0 release. > > > > Can you provide a link? My search for transform on python-dev came out with It was linked from the issue, if I recall correctly. I do remember reading the thread from the python-3000 list, linked by someone somewhere :) > http://mail.python.org/pipermail/python-dev/2010-June/100564.html > > where you seem to oppose these methods. Also, new methods to builtins It looks to me like I was agreeing that transform/untrasnform should do only bytes->bytes or str->str regardless of what codec name you passed them. > fall under the language moratorium (but can be approved on a > case-by-case basis): > > http://www.python.org/dev/peps/pep-3003/#case-by-case-exemptions > > Is there an effort to document these exceptions? I expected such > approvals to be added to PEP 3003, but apparently this was not the > case. I believe MAL's thought was that the addition of these methods had been approved pre-moratorium, but I don't know if that is a sufficient argument or not. -- R. David Murray www.bitdance.com From guido at python.org Fri Dec 3 19:05:32 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 3 Dec 2010 10:05:32 -0800 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <20101203175832.88F00236D8B@kimball.webabinitio.net> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> Message-ID: On Fri, Dec 3, 2010 at 9:58 AM, R. David Murray wrote: > On Fri, 03 Dec 2010 11:14:56 -0500, Alexander Belopolsky wrote: >> On Fri, Dec 3, 2010 at 10:11 AM, R. David Murray wrote: >> .. >> > Please also recall that transform/untransform was discussed before >> > the release of Python 3.0 and was approved at the time, but it just >> > did not get implemented before the 3.0 release. >> > >> >> Can you provide a link? ?My search for transform on python-dev came out with > > It was linked from the issue, if I recall correctly. ?I do remember > reading the thread from the python-3000 list, linked by someone > somewhere :) > >> http://mail.python.org/pipermail/python-dev/2010-June/100564.html >> >> where you seem to oppose these methods. ?Also, new methods to builtins > > It looks to me like I was agreeing that transform/untrasnform should > do only bytes->bytes or str->str regardless of what codec name you > passed them. > >> fall under the language moratorium (but can be approved on a >> case-by-case basis): >> >> http://www.python.org/dev/peps/pep-3003/#case-by-case-exemptions >> >> Is there an effort to document these exceptions? ?I expected such >> approvals to be added to PEP 3003, but apparently this was not the >> case. > > I believe MAL's thought was that the addition of these methods had > been approved pre-moratorium, but I don't know if that is a > sufficient argument or not. It is not. The moratorium is intended to freeze the state of the language as implemented, not whatever was discussed and approved but didn't get implemented (that'd be a hole big enough to drive a truck through, as the saying goes :-). Regardless of what I or others may have said before, I am not currently a fan of adding transform() to either str or bytes. -- --Guido van Rossum (python.org/~guido) From contactprashantat at gmail.com Fri Dec 3 19:25:10 2010 From: contactprashantat at gmail.com (Prashant Kumar) Date: Fri, 3 Dec 2010 23:55:10 +0530 Subject: [Python-Dev] Porting Ideas In-Reply-To: <4CF85850.2070305@netwok.org> References: <4CF85850.2070305@netwok.org> Message-ID: On 12/3/10, ?ric Araujo wrote: > Hi Prashant, > > Python 3 support in distutils2 is not entirely finished, it?s an > interesting and challenging task. > > Another idea: convert the python.org internal scripts to use Python 3, > for example starting with patches for http://code.python.org/hg/peps/ . > This would not have any impact on the community, but it?s easy work > that?d help the Python developers to eat their own dogfood. > > Regards > > Thanks for the suggestion. From alexander.belopolsky at gmail.com Fri Dec 3 19:28:08 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 3 Dec 2010 13:28:08 -0500 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <201012031016.05063.victor.stinner@haypocalc.com> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> Message-ID: On Fri, Dec 3, 2010 at 4:16 AM, Victor Stinner wrote: .. > I don't like transform() and untransform() because I think that we should not > add too much operations to the base types (bytes and str), and they do > implicit module import. I prefer explicit module import (eg. import binascii; > binascii.hexlify(b'to hex')). +1 Implicit imports are currently subtly broken with no solution in sight. See http://bugs.python.org/issue8098 http://bugs.python.org/issue7980 In fact, once the language moratorium is over, I will argue that str.encode() and byte.decode() should deprecate encoding argument and just do UTF-8 encoding/decoding. Hopefully by that time most people will forget that other encodings exist. (I can dream, right?) Python can still provide the codec machinery but not tie it to str/bytes builtins. After all, neither encode/decode nor transform/unstransform fully utilize the power of codec design. From brett at python.org Fri Dec 3 20:22:18 2010 From: brett at python.org (Brett Cannon) Date: Fri, 3 Dec 2010 11:22:18 -0800 Subject: [Python-Dev] [Python-checkins] r86945 - python/branches/py3k/Lib/test/__main__.py In-Reply-To: <20101203013401.A454BEE981@mail.python.org> References: <20101203013401.A454BEE981@mail.python.org> Message-ID: I just want to say thanks for doing this, Michael. __main__.py is IMO woefully underused and it's great to see Python dogfooding the feature along with making it easier to explain how to run our unit tests. On Thu, Dec 2, 2010 at 17:34, michael.foord wrote: > Author: michael.foord > Date: Fri Dec ?3 02:34:01 2010 > New Revision: 86945 > > Log: > Initial implementation of Lib/test/__main__.py so we can run tests with 'python -m test' > > Added: > ? python/branches/py3k/Lib/test/__main__.py ? (contents, props changed) > > Added: python/branches/py3k/Lib/test/__main__.py > ============================================================================== > --- (empty file) > +++ python/branches/py3k/Lib/test/__main__.py ? Fri Dec ?3 02:34:01 2010 > @@ -0,0 +1,38 @@ > +import os > +import sys > +import sysconfig > + > +from test import support > +from test.regrtest import main > + > +# findtestdir() gets the dirname out of __file__, so we have to make it > +# absolute before changing the working directory. > +# For example __file__ may be relative when running trace or profile. > +# See issue #9323. > +__file__ = os.path.abspath(__file__) > + > +# sanity check > +assert __file__ == os.path.abspath(sys.argv[0]) > + > +# When tests are run from the Python build directory, it is best practice > +# to keep the test files in a subfolder. ?It eases the cleanup of leftover > +# files using command "make distclean". > +if sysconfig.is_python_build(): > + ? ?TEMPDIR = os.path.join(sysconfig.get_config_var('srcdir'), 'build') > + ? ?TEMPDIR = os.path.abspath(TEMPDIR) > + ? ?if not os.path.exists(TEMPDIR): > + ? ? ? ?os.mkdir(TEMPDIR) > + > +# Define a writable temp dir that will be used as cwd while running > +# the tests. The name of the dir includes the pid to allow parallel > +# testing (see the -j option). > +TESTCWD = 'test_python_{}'.format(os.getpid()) > + > +TESTCWD = os.path.join(TEMPDIR, TESTCWD) > + > +# Run the tests in a context manager that temporary changes the CWD to a > +# temporary and writable directory. If it's not possible to create or > +# change the CWD, the original CWD will be used. The original CWD is > +# available from support.SAVEDCWD. > +with support.temp_cwd(TESTCWD, quiet=True): > + ? ?main() > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From martin at v.loewis.de Fri Dec 3 23:39:19 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Dec 2010 23:39:19 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF8E22B.7050005@netwok.org> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> Message-ID: <4CF97197.2050105@v.loewis.de> >> For example, I keep running into the issue that distutils doesn't >> currently support parallel builds. I have been pondering supporting >> "-j" for building extensions, using both unbounded "-j" and the GNU make >> style -jN build server. However, I know that the patch will be rejected, >> so I don't even start working on it. > This would be a very useful feature for distutils2. But I'm not interested at all in having it in distutils2. I want the Python build itself to use it, and alas, I can't because of the freeze. Regards, Martin From merwok at netwok.org Fri Dec 3 23:48:48 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 03 Dec 2010 23:48:48 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF97197.2050105@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> Message-ID: <4CF973D0.4060800@netwok.org> > But I'm not interested at all in having it in distutils2. I want the > Python build itself to use it, and alas, I can't because of the freeze. You can?t in 3.2, true. Neither can you in 3.1, or any previous version. If you implement it in distutils2, you have very good chances to get it for 3.3. Isn?t that a win? (BTW: congratulations on having PEP 384 accepted and merged.) Regards From martin at v.loewis.de Fri Dec 3 23:52:41 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 03 Dec 2010 23:52:41 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF973D0.4060800@netwok.org> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> Message-ID: <4CF974B9.6040509@v.loewis.de> Am 03.12.2010 23:48, schrieb ?ric Araujo: >> But I'm not interested at all in having it in distutils2. I want the >> Python build itself to use it, and alas, I can't because of the freeze. > You can?t in 3.2, true. Neither can you in 3.1, or any previous > version. If you implement it in distutils2, you have very good chances > to get it for 3.3. Isn?t that a win? It is, unfortunately, a very weak promise. Until distutils2 is integrated in Python, I probably won't spend any time on it. Regards, Martin From dimaqq at gmail.com Fri Dec 3 23:55:41 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 3 Dec 2010 15:55:41 -0700 Subject: [Python-Dev] gc ideas -- sparse memory Message-ID: How hard or reasonable would it be to free memory pages on OS level? [pcmiiw] Gabage collection within a generation involves moving live objects to compact the generation storage. This separates the memory region into 2 parts "live" and "cleared", the pointer to the beginning of the "cleared" part is where next allocation is going to happen. When this is done, does Python gc move the objects preserving order or does it try to populate garbaged slot with some live object disregarding order? Earlier case is more applicable, but latter case is a target for below too. If we were to look at memory regions from the OS point of view, they are allocated as pages (or possibly as hugetlb pages). So if we are to compact something like this [LL__][_L__][____][L_LL][LFFF] where [] is a page, L is live object and _ is garbage and F is free memory, would it not make more sense to tell OS that [____] is not needed anymore, and not move some of the consequtive [L_LL][LFFF] at all, or at least not move those objects as far down the memory region? This would of course have a certain overhead of tracking which pages are given back to OS and mapping them back when needed, but at the same time, be beneficial because fewer objets are moved and also possibly improve cpu cache performance because objects won't be moved so far out. p.s. if someone has an athoritative link to modern python gc design, please let me know. From tjreedy at udel.edu Sat Dec 4 00:04:58 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 03 Dec 2010 18:04:58 -0500 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: Message-ID: On 12/3/2010 5:55 PM, Dima Tisnek wrote: > How hard or reasonable would it be to free memory pages on OS level? > > [pcmiiw] Gabage collection within a generation involves moving live > objects to compact the generation storage. This separates the memory > region into 2 parts "live" and "cleared", the pointer to the beginning > of the "cleared" part is where next allocation is going to happen. > > When this is done, does Python gc move the objects preserving order or > does it try to populate garbaged slot with some live object > disregarding order? Earlier case is more applicable, but latter case > is a target for below too. > > If we were to look at memory regions from the OS point of view, they > are allocated as pages (or possibly as hugetlb pages). So if we are to > compact something like this [LL__][_L__][____][L_LL][LFFF] where [] > is a page, L is live object and _ is garbage and F is free memory, > would it not make more sense to tell OS that [____] is not needed > anymore, and not move some of the consequtive [L_LL][LFFF] at all, or > at least not move those objects as far down the memory region? > > This would of course have a certain overhead of tracking which pages > are given back to OS and mapping them back when needed, but at the > same time, be beneficial because fewer objets are moved and also > possibly improve cpu cache performance because objects won't be moved > so far out. > > p.s. if someone has an athoritative link to modern python gc design, > please let me know. gc is implementation specific. CPython uses ref counting + cycle gc. A constraint on all implementations is that objects have a fixed, unique id during their lifetime. CPython uses the address as the id, so it cannot move objects. Other implementations do differently. Compacting gc requires an id to current address table or something. -- Terry Jan Reedy From lukasz at langa.pl Fri Dec 3 23:45:33 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Fri, 3 Dec 2010 23:45:33 +0100 Subject: [Python-Dev] [Python-checkins] r86976 - in python/branches/py3k: Doc/library/configparser.rst Doc/library/fileformats.rst Lib/configparser.py Lib/test/test_cfgparser.py Misc/NEWS In-Reply-To: <4CF9386E.4000205@netwok.org> References: <20101203162800.9A740EE9B9@mail.python.org> <4CF9386E.4000205@netwok.org> Message-ID: <5CE836CF-4A20-4F50-97D5-91E35E194004@langa.pl> Wiadomo?? napisana przez ?ric Araujo w dniu 2010-12-03, o godz. 19:35: > Hello, > >> Author: lukasz.langa >> New Revision: 86976 >> Log: Issue 10499: Modular interpolation in configparser >> >> Modified: python/branches/py3k/Doc/library/configparser.rst > Is the module still backward compatible with the 3.1 version, modulo > fixed bugs? I haven?t been able to follow closely all the great > improvements you?ve been making, and there has been a lot of movement in > the code, so I?m not sure. There have been minor incompatibilities, all have been thoroughly discussed with Fred and other developers. No changes should cause silent breakage, though. > Thanks for taking up configparser. Maybe it will become so useful and > extensible that you?ll have to release it on PyPI for older Pythons :) Sure. Why not? :) >> Modified: python/branches/py3k/Doc/library/fileformats.rst > This looks like unrelated changes that slipped in the commit. As we discussed this on IRC, this is an unfortunate slip caused by me trying to wrap up documentation updates in one go. Will remember to do that separately now. Thanks! >> Modified: python/branches/py3k/Lib/configparser.py >> >> Raise DuplicateSectionError if a section by the specified name >> already exists. Raise ValueError if name is DEFAULT. >> """ >> - if section == self._default_section: >> + if section == self.default_section: >> raise ValueError('Invalid section name: %s' % section) > I think it?s the only error message using %s instead of %r. The quotes > added by %r typically help spot names with spaces (embedded or trailing) > and the like. Corrected in rev 86999, thanks. >> + options = list(d.keys()) >> + if raw: >> + return [(option, d[option]) >> + for option in options] >> + else: >> + return [(option, self._interpolation.before_get(self, section, >> + option, d[option], >> + d)) >> + for option in options] > The list call seems unneeded. Minor style point: I avoid such dangling > arguments that don?t read great, I prefer to go to a newline after the > opening paren: > return [(option, self._interpolation.before_get( > self, section, option, d[option], d)) for > option in options] You're right that my formatting looks ugly but yours is not that much better either ;) How about: value_getter = lambda option: self._interpolation.before_get(self, section, option, d[option], d) if raw: value_getter = lambda option: d[option] return [(option, value_getter(option)) for option in d.keys()] -- Best regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From foom at fuhm.net Sat Dec 4 00:11:57 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 3 Dec 2010 18:11:57 -0500 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF974B9.6040509@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> Message-ID: <294C7411-A987-45CE-ACE1-6DEFB17DA0C6@fuhm.net> On Dec 3, 2010, at 5:52 PM, Martin v. L?wis wrote: > Am 03.12.2010 23:48, schrieb ?ric Araujo: >>> But I'm not interested at all in having it in distutils2. I want the >>> Python build itself to use it, and alas, I can't because of the freeze. >> You can?t in 3.2, true. Neither can you in 3.1, or any previous >> version. If you implement it in distutils2, you have very good chances >> to get it for 3.3. Isn?t that a win? > > It is, unfortunately, a very weak promise. Until distutils2 is > integrated in Python, I probably won't spend any time on it. It seems like it'd be a good idea to start integrating distutils2 into python trunk immediately after the 3.2 branch is cut, then. James From dimaqq at gmail.com Sat Dec 4 00:13:34 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 3 Dec 2010 16:13:34 -0700 Subject: [Python-Dev] gc ideas -- dynamic profiling Message-ID: Python organizes objects into 3 generations, ephemeral, short- and long-lived. When object is created it is place in ephemeral, if it lives long enough, it is move to short-lived and so on. q1 are generations placed in separate memory regions, or are all generations in one memory regions and there is a pointer that signifies the boundary between generations? I propose to track hot spots in python, that is contexts where most of allocations occur, and instrument these with counters that essentially tell how often an object generated here ends up killed in ephemeral, short-, long-lived garbage collector run or is in fac tstill alive. If a particular allocation context creates objects that are likely to be long-lived, allocator could skip frst 2 generations altogether (generations are separate regions) or preload the object with high survival count (if q1 is single region). On the other hand, if we know where most allocations occur, we can presume that most of these allocations are ephemeral, otherwise we run out of memory anyway, if this is indeed so, it makes my point moot. Implications are extra code to define context, extra pointer back to context from every allocation (or alterntively a weakref from allocation point to every object it generated) and real-time accounting as to what happens to these objects. It should be possible to approach this problem statistically, that is instrument only every 100s object or so. Context could be simple, e.g. bytecode operation 3 on line 45 in module "junk", or more complex, e.g. call stack str<-generator at 238382<-function "foo"<-function "boo"<-...; or even patterns like ''' str<-any depth<-function "big" '''; clearly figuring out what hotspots are is already non-trivial, the more coplex the definition of context the more it borders downright impossible. p.s. can anyone share modern cpython profiling results to shed some light on how important gc optimization really is? From foom at fuhm.net Sat Dec 4 00:15:17 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 3 Dec 2010 18:15:17 -0500 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: Message-ID: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> On Dec 3, 2010, at 6:04 PM, Terry Reedy wrote: > gc is implementation specific. CPython uses ref counting + cycle gc. A constraint on all implementations is that objects have a fixed, unique id during their lifetime. CPython uses the address as the id, so it cannot move objects. Other implementations do differently. Compacting gc requires an id to current address table or something. It's somewhat unfortuante that python has this constraint, instead of the looser: "objects have a fixed id during their lifetime", which is much easier to implement, and practically as useful. James From martin at v.loewis.de Sat Dec 4 00:15:48 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Dec 2010 00:15:48 +0100 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: Message-ID: <4CF97A24.5020106@v.loewis.de> Am 03.12.2010 23:55, schrieb Dima Tisnek: > How hard or reasonable would it be to free memory pages on OS level? Very easy. Python already does that. > [pcmiiw] Gabage collection within a generation involves moving live > objects to compact the generation storage. This separates the memory > region into 2 parts "live" and "cleared", the pointer to the beginning > of the "cleared" part is where next allocation is going to happen. I think you are talking about copying collectors here. This is not how Python's garbage collection works. > When this is done, does Python gc move the objects preserving order or > does it try to populate garbaged slot with some live object > disregarding order? Earlier case is more applicable, but latter case > is a target for below too. (C)Python's garbage collector is not moving objects at all. > If we were to look at memory regions from the OS point of view, they > are allocated as pages (or possibly as hugetlb pages). So if we are to > compact something like this [LL__][_L__][____][L_LL][LFFF] where [] > is a page, L is live object and _ is garbage and F is free memory, > would it not make more sense to tell OS that [____] is not needed > anymore, and not move some of the consequtive [L_LL][LFFF] at all, or > at least not move those objects as far down the memory region? See above. Python does no moving of objects whatsoever. Regards, Martin From dimaqq at gmail.com Sat Dec 4 00:19:44 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 3 Dec 2010 16:19:44 -0700 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: Message-ID: Oh my bad, I must've confused python with some research paper. Unique id is not so hard to make without an address. While on this topic, what is the real need for unique ids? Also I reckon not all objects need a unique id like this, e.g. interned strings, simple data types and hashable and comparable objects could perhaps survive without unique id? On 3 December 2010 16:04, Terry Reedy wrote: > On 12/3/2010 5:55 PM, Dima Tisnek wrote: >> >> How hard or reasonable would it be to free memory pages on OS level? >> >> [pcmiiw] Gabage collection within a generation involves moving live >> objects to compact the generation storage. This separates the memory >> region into 2 parts "live" and "cleared", the pointer to the beginning >> of the "cleared" part is where next allocation is going to happen. >> >> When this is done, does Python gc move the objects preserving order or >> does it try to populate garbaged slot with some live object >> disregarding order? Earlier case is more applicable, but latter case >> is a target for below too. >> >> If we were to look at memory regions from the OS point of view, they >> are allocated as pages (or possibly as hugetlb pages). So if we are to >> compact something like this [LL__][_L__][____][L_LL][LFFF] ?where [] >> is a page, L is live object and _ is garbage and F is free memory, >> would it not make more sense to tell OS that [____] is not needed >> anymore, and not move some of the consequtive [L_LL][LFFF] at all, or >> at least not move those objects as far down the memory region? >> >> This would of course have a certain overhead of tracking which pages >> are given back to OS and mapping them back when needed, but at the >> same time, be beneficial because fewer objets are moved and also >> possibly improve cpu cache performance because objects won't be moved >> so far out. >> >> p.s. if someone has an athoritative link to modern python gc design, >> please let me know. > > gc is implementation specific. CPython uses ref counting + cycle gc. A > constraint on all implementations is that objects have a fixed, unique id > during their lifetime. CPython uses the address as the id, so it cannot move > objects. Other implementations do differently. Compacting gc requires an id > to current address table or something. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/dimaqq%40gmail.com > From martin at v.loewis.de Sat Dec 4 00:21:52 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Dec 2010 00:21:52 +0100 Subject: [Python-Dev] gc ideas -- dynamic profiling In-Reply-To: References: Message-ID: <4CF97B90.8090802@v.loewis.de> > q1 are generations placed in separate memory regions, or are all > generations in one memory regions and there is a pointer that > signifies the boundary between generations? You should really start reading the source code. See Modules/gcmodule.c. To answer your question: neither, nor. All objects are in one region, and there is no pointer separating the generations. Instead, all objects belonging to one generation form a double-linked list. > I propose to track hot spots in python, that is contexts where most of > allocations occur, and instrument these with counters that essentially > tell how often an object generated here ends up killed in ephemeral, > short-, long-lived garbage collector run or is in fac tstill alive. If > a particular allocation context creates objects that are likely to be > long-lived, allocator could skip frst 2 generations altogether > (generations are separate regions) or preload the object with high > survival count (if q1 is single region). We would consider such a proposal only if you had *actual evidence* that it improves things, rather than just having a reasoning that it might. Regards, Martin From tjreedy at udel.edu Sat Dec 4 00:35:04 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 03 Dec 2010 18:35:04 -0500 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF974B9.6040509@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> Message-ID: On 12/3/2010 5:52 PM, "Martin v. L?wis" wrote: > Am 03.12.2010 23:48, schrieb ?ric Araujo: >>> But I'm not interested at all in having it in distutils2. I want the >>> Python build itself to use it, and alas, I can't because of the freeze. >> You can?t in 3.2, true. Neither can you in 3.1, or any previous >> version. If you implement it in distutils2, you have very good chances >> to get it for 3.3. Isn?t that a win? > > It is, unfortunately, a very weak promise. Until distutils2 is > integrated in Python, I probably won't spend any time on it. ?ric, I have the impression from Tarek and you together that D2 is still in alpha only because it is not feature frozen and that it is as capable and stable as D1. I do not know what Martin means by 'integrate' (other than that he be able to use it to build Python), but if my first sentence is true, I cannot help but wonder whether a snapshot of D2 could be included with 3.2, perhaps as '_distribute2' (note leading underscore) at least for Python's use. The doc, if any, could just say 'Development snapshot of D2.a4 (or whatever) for building Python. Other uses should get the latest public release from PyPI." -- Terry Jan Reedy From martin at v.loewis.de Sat Dec 4 00:45:27 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 04 Dec 2010 00:45:27 +0100 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: Message-ID: <4CF98117.5060301@v.loewis.de> > Oh my bad, I must've confused python with some research paper. > Unique id is not so hard to make without an address. > > While on this topic, what is the real need for unique ids? They are absolutely needed for mutable objects. For immutable ones, it would be ok to claim that they are identical if they are equal (assuming they support equality - which is tricky for things like NaN). Of course, the C API has lots of assumptions that identity and address are really the same thing. Regards, Martin From martin at v.loewis.de Sat Dec 4 00:46:49 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 04 Dec 2010 00:46:49 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> Message-ID: <4CF98169.6020106@v.loewis.de> Am 04.12.2010 00:35, schrieb Terry Reedy: > On 12/3/2010 5:52 PM, "Martin v. L?wis" wrote: >> Am 03.12.2010 23:48, schrieb ?ric Araujo: >>>> But I'm not interested at all in having it in distutils2. I want the >>>> Python build itself to use it, and alas, I can't because of the freeze. >>> You can?t in 3.2, true. Neither can you in 3.1, or any previous >>> version. If you implement it in distutils2, you have very good chances >>> to get it for 3.3. Isn?t that a win? >> >> It is, unfortunately, a very weak promise. Until distutils2 is >> integrated in Python, I probably won't spend any time on it. > > ?ric, I have the impression from Tarek and you together that D2 is still > in alpha only because it is not feature frozen and that it is as capable > and stable as D1. I do not know what Martin means by 'integrate' (other > than that he be able to use it to build Python) That the master copy of the source code is in the Python source repository. Regards, Martin From tjreedy at udel.edu Sat Dec 4 01:00:15 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 03 Dec 2010 19:00:15 -0500 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF98169.6020106@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> <4CF98169.6020106@v.loewis.de> Message-ID: On 12/3/2010 6:46 PM, "Martin v. L?wis" wrote: >> and stable as D1. I do not know what Martin means by 'integrate' (other >> than that he be able to use it to build Python) > > That the master copy of the source code is in the Python source repository. Is a separate branch acceptible, as long as you can commit changes? Does the meaning of 'repository' change any with the hg switch? -- Terry Jan Reedy From tjreedy at udel.edu Sat Dec 4 01:05:19 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 03 Dec 2010 19:05:19 -0500 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> Message-ID: On 12/3/2010 6:15 PM, James Y Knight wrote: > On Dec 3, 2010, at 6:04 PM, Terry Reedy wrote: >> gc is implementation specific. CPython uses ref counting + cycle >> gc. A constraint on all implementations is that objects have a >> fixed, unique id during their lifetime. CPython uses the address as >> the id, so it cannot move objects. Other implementations do >> differently. Compacting gc requires an id to current address table >> or something. I left out that the id must be an int. > It's somewhat unfortuante that python has this constraint, instead of > the looser: "objects have a fixed id during their lifetime", which is > much easier to implement, and practically as useful. Given that the only different between 'fixed and unique' and 'fixed' is the uniqueness part, I do not understand 'practically as useful'. Duplicate ids (in the extreme, that same for all objects) hardly seem useful at all. -- Terry Jan Reedy From martin at v.loewis.de Sat Dec 4 01:41:32 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 04 Dec 2010 01:41:32 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> <4CF98169.6020106@v.loewis.de> Message-ID: <4CF98E3C.3050803@v.loewis.de> Am 04.12.2010 01:00, schrieb Terry Reedy: > On 12/3/2010 6:46 PM, "Martin v. L?wis" wrote: > >>> and stable as D1. I do not know what Martin means by 'integrate' (other >>> than that he be able to use it to build Python) >> >> That the master copy of the source code is in the Python source >> repository. > > Is a separate branch acceptible, as long as you can commit changes? No. I want the buildbots be able to use the code, in the Python trunk. > Does the meaning of 'repository' change any with the hg switch? No. The code would need to be pushed to the master repository, so that the buildbots can fetch it from there. In short, I'm not very interested in contributing to a tool that has no users (and is the fifths of its kind). Regards, Martin From foom at fuhm.net Sat Dec 4 01:46:46 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 3 Dec 2010 19:46:46 -0500 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> Message-ID: On Dec 3, 2010, at 7:05 PM, Terry Reedy wrote: > I left out that the id must be an int. > >> It's somewhat unfortuante that python has this constraint, instead of >> the looser: "objects have a fixed id during their lifetime", which is >> much easier to implement, and practically as useful. > > Given that the only different between 'fixed and unique' and 'fixed' is the uniqueness part, I do not understand 'practically as useful'. Duplicate ids (in the extreme, that same for all objects) hardly seem useful at all. Sure they are. This is what Java provides you, for example. If you have fixed, but potentially non-unique ids (in Java you get this using "identityHashCode()"), you can still make an identity hashtable. You simply need to *also* check using "is" that the two objects really are the same one after finding the hash bin using id. It'd be a quality of implementation issue whether an implementation gives you the same value for every object. It should not, of course, if it wants programs to have reasonable performance. Same sort of thing as how __hash__ should not return 0 for everything. Besides identity-hashtables, the main other thing id gets used for is to have some identifier string in a printer (e.g. ""). There, it's generally "good enough" to use an id which is not guaranteed to be, but often is, unique. It works for Java... James From tjreedy at udel.edu Sat Dec 4 04:50:10 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 03 Dec 2010 22:50:10 -0500 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> Message-ID: On 12/3/2010 7:46 PM, James Y Knight wrote: > Sure they are. This is what Java provides you, for example. If you > have fixed, but potentially non-unique ids (in Java you get this > using "identityHashCode()"), you can still make an identity I do not see the point of calling a (non-unique) hash value the identity > hashtable. You simply need to *also* check using "is" that the two In Python, that unique isness is the identify. (a is b) == (id(a) == id(b)) by definition. > objects really are the same one after finding the hash bin using id. by using the hash value, which is how Python dict operate. -- Terry Jan Reedy From cs at zip.com.au Sat Dec 4 05:06:45 2010 From: cs at zip.com.au (Cameron Simpson) Date: Sat, 4 Dec 2010 15:06:45 +1100 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> Message-ID: <20101204040645.GA32257@cskk.homeip.net> On 03Dec2010 18:15, James Y Knight wrote: | On Dec 3, 2010, at 6:04 PM, Terry Reedy wrote: | > gc is implementation specific. CPython uses ref counting + cycle | > gc. A constraint on all implementations is that objects have a fixed, | > unique id during their lifetime. CPython uses the address as the id, so | > it cannot move objects. Other implementations do differently. Compacting | > gc requires an id to current address table or something. | | It's somewhat unfortuante that python has this constraint, instead of | the looser: "objects have a fixed id during their lifetime", which is | much easier to implement, and practically as useful. Python doesn't have the constraint you imagine; it _does_ have "objects have a fixed id during their lifetime". _CPython_ has this constraint because it uses the address as the id, which is free and needs no papping or extra storage. Of course, it removes certain freedoms from the GC etc as a side effect. -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ The number of cylinders for this disk is set to 364737. There is nothing wrong with that, but this is larger than 1024, [...] - fdisk on our new RAID 07oct2007 :-) From foom at fuhm.net Sat Dec 4 05:34:43 2010 From: foom at fuhm.net (James Y Knight) Date: Fri, 3 Dec 2010 23:34:43 -0500 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> Message-ID: On Dec 3, 2010, at 10:50 PM, Terry Reedy wrote: > On 12/3/2010 7:46 PM, James Y Knight wrote: > >> Sure they are. This is what Java provides you, for example. If you >> have fixed, but potentially non-unique ids (in Java you get this >> using "identityHashCode()"), you can still make an identity > > I do not see the point of calling a (non-unique) hash value the identity My point was simply that a) it's an unfortunate constraint on potential GC implementations that objects need to have a fixed and unique id in Python, and b) that it's not actually necessary to have such a constraint (in the abstract sense of required; obviously it's a requirement upon Python *today*, due to existing code which depends upon that promise). Would you be happier if I had said "it's unfortunate that Python has an "id" function instead of an "identityHashValue" function? I suppose that's what I really meant. Python the language would not have been harmed had it had from the start an identityHashValue() function instead of an id() function. In the CPython implementation, it may even have had the exact same behavior, but would've allowed other implementations more flexibility. James From tjreedy at udel.edu Sat Dec 4 06:39:23 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 04 Dec 2010 00:39:23 -0500 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <20101204040645.GA32257@cskk.homeip.net> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <20101204040645.GA32257@cskk.homeip.net> Message-ID: On 12/3/2010 11:06 PM, Cameron Simpson wrote: > On 03Dec2010 18:15, James Y Knight wrote: > | On Dec 3, 2010, at 6:04 PM, Terry Reedy wrote: > |> gc is implementation specific. CPython uses ref counting + cycle > |> gc. A constraint on all implementations is that objects have a fixed, > |> unique id during their lifetime. CPython uses the address as the id, so > |> it cannot move objects. Other implementations do differently. Compacting > |> gc requires an id to current address table or something. > | > | It's somewhat unfortuante that python has this constraint, instead of > | the looser: "objects have a fixed id during their lifetime", which is > | much easier to implement, and practically as useful. > > Python doesn't have the constraint you imagine; it _does_ have "objects > have a fixed id during their lifetime". "id(object) Return the ?identity? of an object. This is an integer which is guaranteed to be unique and constant for this object during its lifetime." Of course, other implementations are free to change builtins, but code that depends on CPython's definitions will not run. -- Terry Jan Reedy From dimaqq at gmail.com Sat Dec 4 06:47:53 2010 From: dimaqq at gmail.com (Dima Tisnek) Date: Fri, 3 Dec 2010 22:47:53 -0700 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <4CF98117.5060301@v.loewis.de> References: <4CF98117.5060301@v.loewis.de> Message-ID: On 3 December 2010 16:45, "Martin v. L?wis" wrote: >> Oh my bad, I must've confused python with some research paper. >> Unique id is not so hard to make without an address. >> >> While on this topic, what is the real need for unique ids? > > They are absolutely needed for mutable objects. For immutable ones, > it would be ok to claim that they are identical if they are equal > (assuming they support equality - which is tricky for things like NaN). Indeed, but do ids really need to be unique and fixed at the same time? a is b # (if atomic) needs unique ids, but doesn't really need fixed ids a[b] # needs fixed hash, but not strictly a globally unique id I can imagine an implementaion of pickle for example that uses unique and fixed as a given to detect cycles, etc; but that would be implementation detail. It seems to me unique and fixed id implies that it is stored somewhere (with incref beforehands and decref afterwards), however a proper reference to an object could be used just as well. Am I still missing something? > > Of course, the C API has lots of assumptions that identity and address > are really the same thing. > > Regards, > Martin > From fijall at gmail.com Sat Dec 4 07:21:30 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 4 Dec 2010 08:21:30 +0200 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> Message-ID: On Sat, Dec 4, 2010 at 6:34 AM, James Y Knight wrote: > On Dec 3, 2010, at 10:50 PM, Terry Reedy wrote: >> On 12/3/2010 7:46 PM, James Y Knight wrote: >> >>> Sure they are. This is what Java provides you, for example. If you >>> have fixed, but potentially non-unique ids (in Java you get this >>> using "identityHashCode()"), you can still make an identity >> >> I do not see the point of calling a (non-unique) hash value the identity > > My point was simply that a) it's an unfortunate constraint on potential GC implementations that objects need to have a fixed and unique id in Python, and b) that it's not actually necessary to have such a constraint (in the abstract sense of required; obviously it's a requirement upon Python *today*, due to existing code which depends upon that promise). > > Would you be happier if I had said "it's unfortunate that Python has an "id" function instead of an "identityHashValue" function? I suppose that's what I really meant. Python the language would not have been harmed had it had from the start an identityHashValue() function instead of an id() function. In the CPython implementation, it may even have had the exact same behavior, but would've allowed other implementations more flexibility. > > James > I don't see how this related to moving vs non-moving GC. PyPy (and I believe IronPython and Java) all have fixed unique ids that are not necesarilly their addresses. The only problem is that id() computed that way is more costly performance-wise, but works. From steve at pearwood.info Sat Dec 4 07:48:13 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 04 Dec 2010 17:48:13 +1100 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> Message-ID: <4CF9E42D.3010302@pearwood.info> James Y Knight wrote: > On Dec 3, 2010, at 10:50 PM, Terry Reedy wrote: >> On 12/3/2010 7:46 PM, James Y Knight wrote: >> >>> Sure they are. This is what Java provides you, for example. If you >>> have fixed, but potentially non-unique ids (in Java you get this >>> using "identityHashCode()"), you can still make an identity >> I do not see the point of calling a (non-unique) hash value the identity > > My point was simply that a) it's an unfortunate constraint on potential GC implementations that objects need to have a fixed and unique id in Python, and b) that it's not actually necessary to have such a constraint (in the abstract sense of required; obviously it's a requirement upon Python *today*, due to existing code which depends upon that promise). I'm afraid I don't follow you. Unless you're suggesting some sort of esoteric object system whereby objects *don't* have identity (e.g. where objects are emergent properties of some sort of distributed, non-localised "information"), any object naturally has an identity -- itself. It seems to me that an identify function must obey one constraint: * Two objects which exist simultaneously have the same identity if and only if they are the same object i.e. id(x) == id(y) if and only if x is y. Other than that, an implementation is free to make id() behave any way they like. CPython uses the memory location of the object, but Jython and IronPython use an incrementing counter which is never re-used for the lifetime of the Python process. CPython's implementation implies that objects may not be moved in memory, but that's not a language constraint, that's an implementation issue. It seems counter-productive to me to bother with an identity function which doesn't meet that constraint. If id(x) == id(y) implies nothing about x and y (they may, or may not, be the same object) then what's the point? Why would you bother using that function when you could just use x == y instead? > Would you be happier if I had said "it's unfortunate that Python has an "id" function instead of an "identityHashValue" function? I suppose that's what I really meant. Python the language would not have been harmed had it had from the start an identityHashValue() function instead of an id() function. In the CPython implementation, it may even have had the exact same behavior, but would've allowed other implementations more flexibility. No, because I don't see what the point of identityHashValue or why you would ever bother to use it. -- Steven From stephen at xemacs.org Sat Dec 4 09:13:45 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Dec 2010 17:13:45 +0900 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <1291380666.3624.0.camel@localhost.localdomain> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> <1291308960.3576.28.camel@localhost.localdomain> <1291316131.3576.43.camel@localhost.localdomain> <87fwufxxio.fsf@uwakimon.sk.tsukuba.ac.jp> <1291380666.3624.0.camel@localhost.localdomain> Message-ID: <877hfqx8ee.fsf@uwakimon.sk.tsukuba.ac.jp> Antoine Pitrou writes: > Le vendredi 03 d?cembre 2010 ? 13:58 +0900, Stephen J. Turnbull a > ?crit : > > Antoine Pitrou writes: > > > > > The legacy format argument looks like a red herring to me. When > > > converting from a format to another it is the programmer's job to > > > his/her job right. > > > > Uhmmmmmm, the argument *for* this "feature" proposed by several people > > is that Python's numeric constructors do it (right) so that the > > programmer doesn't have to. > > As far as I understand, Alexander was talking about a legacy pre-unicode > text format. We don't have to support this. *I* didn't say we *should* support it. I'm saying that *others'* argument for not restricting the formats accepting by string to number converters to something well-defined and AFAIK universally understood by users (developers of Python programs *and* end-users) is that we *already* support this. Alexander, Martin, and I are basically just pointing out that no, the "support" we have via the built-in numeric constructors is incomplete and nonconforming. We feel that is a bug to be fixed by (1) implementing the definition as currently found in the documents, and (2) moving the non-ASCII support to another module (or, as a compromise, supporting non-ASCII digits via an argument to the built-ins -- that was my proposal, I don't know if Alexander or Martin would find it acceptable). Given that some committers (MAL, you?) don't even consider that accepting and converting a string containing digits from multiple scripts as a single number is a bug, I'd really rather that this bug/feature not be embedded in the interpreter. I suppose that as a built-in rather than syntax, technically it doesn't fall under the moratorium, but it makes me nervous.... From stephen at xemacs.org Sat Dec 4 09:31:04 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Dec 2010 17:31:04 +0900 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> Message-ID: <8762vax7lj.fsf@uwakimon.sk.tsukuba.ac.jp> Alexander Belopolsky writes: > In fact, once the language moratorium is over, I will argue that > str.encode() and byte.decode() should deprecate encoding argument and > just do UTF-8 encoding/decoding. Hopefully by that time most people > will forget that other encodings exist. (I can dream, right?) It's just a dream. There's a pile of archival material, often on R/O media, out there that won't be transcoded any more quickly than the inscriptions on Tutankhamun's tomb. Remember, Python is a language used to implement such translations. It's not an application. I think it would be reasonable to make UTF-8 the *default* encoding on all platforms, except for internal OS functions, where Windows will presumably continue to use UTF-16 and *nix distros will probably continue to agree to disagree about whether on-disk format is NFD or NFC (and the Python language as yet doesn't know about NFC v. NFD, although the library does). In the discussion of PEP 263, I proposed that the external encoding of Python scripts themselves be fixed as UTF-8, and other encodings would have to be pretranslated by an appropriate codec. That was shouted down by the European contingent, who wanted to continue using Latin-1 and Latin-2 without codecs or a wrapper to call them transparently. However, this time around you might get a more sympathetic hearing. From stefan_ml at behnel.de Sat Dec 4 09:35:11 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 04 Dec 2010 09:35:11 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF8008B.60402@v.loewis.de> References: <4CF8008B.60402@v.loewis.de> Message-ID: "Martin v. L?wis", 02.12.2010 21:24: >> Since discussion has trailed off without any blocking objections, I'm >> accepting PEP 384. Martin, you may mark the PEP accepted and proceed >> with merging the implementation for the beta on Saturday. > > Thanks! will do (I'll also take into consideration the proposed changes). Note that the PEP states under "Signature Changes": """ A number of functions currently expect a specific struct, even though callers typically have PyObject* available. These have been changed to expect PyObject* as the parameter; this will cause warnings in applications that currently explicitly cast to the parameter type. """ That's not true for C++ compilation where I now get an error - 3 failing tests in Cython's test suite tonight due to the new PyEval_EvalCode() signature. I'm not really complaining (the API fixes are long overdue), just leaving a comment that what a compiler considers a warning or error pretty much depends on compiler, platform and configuration. So the above statement can be considered euphemistic. Stefan From martin at v.loewis.de Sat Dec 4 09:43:55 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 04 Dec 2010 09:43:55 +0100 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: References: <4CF98117.5060301@v.loewis.de> Message-ID: <4CF9FF4B.1030004@v.loewis.de> > Am I still missing something? Apparently. The hole C API would break if objects would move in memory. Since they have to stay at fixed addresses, it's easy enough to use the address as ID. There is no problem to be solved here. Regards, Martin From eliben at gmail.com Sat Dec 4 09:48:38 2010 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 4 Dec 2010 10:48:38 +0200 Subject: [Python-Dev] python 2 for building python 3 Message-ID: To build the trunk Python 3k, it seems that Python 2 has to be installed. This is because Objects/typeslots.py is called in the Makefile, and this file uses Python 2 "print statement" syntax, which fails if $(PYTHON) is actually Python 3. Can anyone reproduce this? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sat Dec 4 09:49:12 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 04 Dec 2010 09:49:12 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> Message-ID: <4CFA0088.8020407@v.loewis.de> > I'm not really complaining (the API fixes are long overdue), just > leaving a comment that what a compiler considers a warning or error > pretty much depends on compiler, platform and configuration. No no no. It does *not* depend on compiler, platform, or configuration. It *only* depends on the programming language. So for C++, it's an API change; for C, it's not. Still, I can point that out. Regards, Martin From martin at v.loewis.de Sat Dec 4 09:59:08 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Dec 2010 09:59:08 +0100 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <4CF9E42D.3010302@pearwood.info> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <4CF9E42D.3010302@pearwood.info> Message-ID: <4CFA02DC.3060807@v.loewis.de> > I'm afraid I don't follow you. Unless you're suggesting some sort of > esoteric object system whereby objects *don't* have identity (e.g. where > objects are emergent properties of some sort of distributed, > non-localised "information"), any object naturally has an identity -- > itself. Not in Java or C#. It is in these languages possible to determine whether to references refer to the same object. However, objects don't naturally have a distinct identification (be it an integer or something else). If you really want to associate unique numbers with objects in these languages, the common approach is to put them into an identity dictionary as keys. > It seems counter-productive to me to bother with an identity function > which doesn't meet that constraint. If id(x) == id(y) implies nothing > about x and y (they may, or may not, be the same object) then what's the > point? See James' explanation: it would be possible to use this as the foundation of an identity hash table. > Why would you bother using that function when you could just use > x == y instead? Because in a hash table, you also need a hash value. Regards, Martn From martin at v.loewis.de Sat Dec 4 10:13:11 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Dec 2010 10:13:11 +0100 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: References: Message-ID: <4CFA0627.7010202@v.loewis.de> Am 04.12.2010 09:48, schrieb Eli Bendersky: > To build the trunk Python 3k, it seems that Python 2 has to be installed. > This is because Objects/typeslots.py is called in the Makefile, and this > file uses Python 2 "print statement" syntax, which fails if $(PYTHON) is > actually Python 3. This shouldn't be necessary, as typeslots.inc is also checked into subversion, and should have a newer time stamp than typeslots.inc (perhaps not currently, but that is the plan, anyway). In any case, I now made the script 2-vs-3-agnostic. Regards, Martin From eliben at gmail.com Sat Dec 4 10:20:57 2010 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 4 Dec 2010 11:20:57 +0200 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: <4CFA0627.7010202@v.loewis.de> References: <4CFA0627.7010202@v.loewis.de> Message-ID: On Dec 4, 2010, at 11:13, "Martin v. L?wis" wrote: > Am 04.12.2010 09:48, schrieb Eli Bendersky: >> To build the trunk Python 3k, it seems that Python 2 has to be installed. >> This is because Objects/typeslots.py is called in the Makefile, and this >> file uses Python 2 "print statement" syntax, which fails if $(PYTHON) is >> actually Python 3. > > This shouldn't be necessary, as typeslots.inc is also checked into > subversion, and should have a newer time stamp than typeslots.inc > (perhaps not currently, but that is the plan, anyway). > > In any case, I now made the script 2-vs-3-agnostic. > > Regards, > Martin I had the error pop out in a real build, so the time stamp is probably not a 100% guarantee. Therefore I think it's good that you made it version agnostic. This should be probably be a requirement for all scripts that are part of the build. Thanks, Eli From steve at pearwood.info Sat Dec 4 11:54:31 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 04 Dec 2010 21:54:31 +1100 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <4CFA02DC.3060807@v.loewis.de> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <4CF9E42D.3010302@pearwood.info> <4CFA02DC.3060807@v.loewis.de> Message-ID: <4CFA1DE7.3010409@pearwood.info> Martin v. L?wis wrote: >> I'm afraid I don't follow you. Unless you're suggesting some sort of >> esoteric object system whereby objects *don't* have identity (e.g. where >> objects are emergent properties of some sort of distributed, >> non-localised "information"), any object naturally has an identity -- >> itself. > > Not in Java or C#. It is in these languages possible to determine > whether to references refer to the same object. However, objects don't > naturally have a distinct identification (be it an integer or something > else). Surely even in Java or C#, objects have an *identity* even if the language doesn't provide a way to query their distinct *identification*. An object stored at one memory location is distinct from any other object stored at different memory locations at that same moment in time, regardless of whether or not the language gives you a convenient label for that identity. Even if that memory location can change during the lifetime of the object, at any one moment, object X is a different object from every other object. The fact that we can even talk about "this object" versus "that object" implies that objects have identity. To put it in Python terms, if the id() function were removed, it would no longer be possible to get the unique identification associated with an object, but you could still compare the identity of two objects using `is`. Of course, I'm only talking about objects. In Java there are values which aren't objects, such as ints and floats. That's irrelevant for our discussion, because Python has no such values. > If you really want to associate unique numbers with objects in these > languages, the common approach is to put them into an identity > dictionary as keys. > >> It seems counter-productive to me to bother with an identity function >> which doesn't meet that constraint. If id(x) == id(y) implies nothing >> about x and y (they may, or may not, be the same object) then what's the >> point? > > See James' explanation: it would be possible to use this as the > foundation of an identity hash table. I'm afraid James' explanation didn't shed any light on the question to me. It seems to me that Java's IdentityHashValue[sic -- I think the correct function name is actually IdentityHashCode] is equivalent to Python's hash(), not to Python's id(), and claiming it is related to identity is misleading and confusing. I don't think I'm alone here -- it seems to me that even among Java programmers, the same criticisms have been raised: http://bugs.sun.com/bugdatabase/view_bug.do?bug%5Fid=6321873 http://deepakjha.wordpress.com/2008/07/31/interesting-fact-about-identityhashcode-method-in-javalangsystem-class/ Like hash(), IdentityHashCode doesn't make any promises about identity at all. Two distinct objects could have the same hash code, and a further test is needed to distinguish them. >> Why would you bother using that function when you could just use >> x == y instead? > > Because in a hash table, you also need a hash value. Well, sure, in a hash table you need a hash value. But I was talking about an id() function. So is that it? Is IdentityHashValue (or *Code, as the case may be) just a longer name for hash()? -- Steven From solipsis at pitrou.net Sat Dec 4 12:17:18 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 04 Dec 2010 12:17:18 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <877hfqx8ee.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrnv43v5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CF5841E.6020401@v.loewis.de> <4CF619C1.8030403@pearwood.info> <4CF6CCF9.6000100@v.loewis.de> <4CF6E5AF.7080308@pearwood.info> <20101202143633.6eb17b8d@pitrou.net> <1291308960.3576.28.camel@localhost.localdomain> <1291316131.3576.43.camel@localhost.localdomain> <87fwufxxio.fsf@uwakimon.sk.tsukuba.ac.jp> <1291380666.3624.0.camel@localhost.localdomain> <877hfqx8ee.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <1291461438.3534.2.camel@localhost.localdomain> Le samedi 04 d?cembre 2010 ? 17:13 +0900, Stephen J. Turnbull a ?crit : > Antoine Pitrou writes: > > Le vendredi 03 d?cembre 2010 ? 13:58 +0900, Stephen J. Turnbull a > > ?crit : > > > Antoine Pitrou writes: > > > > > > > The legacy format argument looks like a red herring to me. When > > > > converting from a format to another it is the programmer's job to > > > > his/her job right. > > > > > > Uhmmmmmm, the argument *for* this "feature" proposed by several people > > > is that Python's numeric constructors do it (right) so that the > > > programmer doesn't have to. > > > > As far as I understand, Alexander was talking about a legacy pre-unicode > > text format. We don't have to support this. > > *I* didn't say we *should* support it. I'm saying that *others'* > argument for not restricting the formats accepting by string to number > converters to something well-defined and AFAIK universally understood > by users (developers of Python programs *and* end-users) is that we > *already* support this. As far as I can parse your sentence, I think you are mistaken. Regards Antoine. From solipsis at pitrou.net Sat Dec 4 12:27:10 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Dec 2010 12:27:10 +0100 Subject: [Python-Dev] PEP 384 accepted References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> <294C7411-A987-45CE-ACE1-6DEFB17DA0C6@fuhm.net> Message-ID: <20101204122710.794d5fe8@pitrou.net> On Fri, 3 Dec 2010 18:11:57 -0500 James Y Knight wrote: > On Dec 3, 2010, at 5:52 PM, Martin v. L?wis wrote: > > > Am 03.12.2010 23:48, schrieb ?ric Araujo: > >>> But I'm not interested at all in having it in distutils2. I want the > >>> Python build itself to use it, and alas, I can't because of the freeze. > >> You can?t in 3.2, true. Neither can you in 3.1, or any previous > >> version. If you implement it in distutils2, you have very good chances > >> to get it for 3.3. Isn?t that a win? > > > > It is, unfortunately, a very weak promise. Until distutils2 is > > integrated in Python, I probably won't spend any time on it. > > It seems like it'd be a good idea to start integrating distutils2 into python trunk immediately after the 3.2 branch is cut, then. +1 from me. Antoine. From solipsis at pitrou.net Sat Dec 4 12:37:35 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Dec 2010 12:37:35 +0100 Subject: [Python-Dev] gc ideas -- sparse memory References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <20101204040645.GA32257@cskk.homeip.net> Message-ID: <20101204123735.56609926@pitrou.net> On Sat, 4 Dec 2010 15:06:45 +1100 Cameron Simpson wrote: > On 03Dec2010 18:15, James Y Knight wrote: > | On Dec 3, 2010, at 6:04 PM, Terry Reedy wrote: > | > gc is implementation specific. CPython uses ref counting + cycle > | > gc. A constraint on all implementations is that objects have a fixed, > | > unique id during their lifetime. CPython uses the address as the id, so > | > it cannot move objects. Other implementations do differently. Compacting > | > gc requires an id to current address table or something. > | > | It's somewhat unfortuante that python has this constraint, instead of > | the looser: "objects have a fixed id during their lifetime", which is > | much easier to implement, and practically as useful. > > Python doesn't have the constraint you imagine; it _does_ have "objects > have a fixed id during their lifetime". > > _CPython_ has this constraint because it uses the address as the id, > which is free and needs no papping or extra storage. Well, the main reason CPython has this constraint is that the C API mandates constant object pointers. That we can then reuse the pointer for id() is only a consequence of this. Regards Antoine. From solipsis at pitrou.net Sat Dec 4 12:41:09 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Dec 2010 12:41:09 +0100 Subject: [Python-Dev] python 2 for building python 3 References: <4CFA0627.7010202@v.loewis.de> Message-ID: <20101204124109.07745d71@pitrou.net> On Sat, 4 Dec 2010 11:20:57 +0200 Eli Bendersky wrote: > > I had the error pop out in a real build, so the time stamp is probably not a 100% guarantee. Therefore I think it's good that you made it version agnostic. This should be probably be a requirement for all scripts that are part of the build. Er, normally you don't need *any* Python installed to build Python (be it 3.x or 2.x). It would be better if this didn't change. Regards Antoine. From solipsis at pitrou.net Sat Dec 4 12:45:02 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 4 Dec 2010 12:45:02 +0100 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py References: <20101204091044.C6CBEEE985@mail.python.org> Message-ID: <20101204124502.37ec1594@pitrou.net> On Sat, 4 Dec 2010 10:10:44 +0100 (CET) gregory.p.smith wrote: > Author: gregory.p.smith > Date: Sat Dec 4 10:10:44 2010 > New Revision: 87010 > > Log: > issue7213 + issue2320: Cause a DeprecationWarning if the close_fds argument is > not passed to subprocess.Popen as the default value will be changing in a > future Python to the safer and more often desired value of True. That doesn't seem to be a good idea under Windows, is it? (?Note that on Windows, you cannot set *close_fds* to true and also redirect the standard handles by setting *stdin*, *stdout* or *stderr*.?) From martin at v.loewis.de Sat Dec 4 13:22:07 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Dec 2010 13:22:07 +0100 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <4CFA1DE7.3010409@pearwood.info> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <4CF9E42D.3010302@pearwood.info> <4CFA02DC.3060807@v.loewis.de> <4CFA1DE7.3010409@pearwood.info> Message-ID: <4CFA326F.7030202@v.loewis.de> > Surely even in Java or C#, objects have an *identity* even if the > language doesn't provide a way to query their distinct *identification*. When people said "the id of an object should this or that", they always meant identification, not identity (perhaps except for you). Nobody would propose that the identity (in your wording) of two distinct objects might be the same, or might change over time. > I'm afraid James' explanation didn't shed any light on the question to > me. It seems to me that Java's IdentityHashValue[sic -- I think the > correct function name is actually IdentityHashCode] is equivalent to > Python's hash() No, it's not. java.lang.Object.hashCode() is equivalent to Python's hash(). In particular, for both of these, the following requirement holds: object that *compare* equal must hash equal. This is not the case for the identity hash code, or Python's id function: objects that compare equal may still have different ids, or different java identity hash codes. > not to Python's id(), and claiming it is related to > identity is misleading and confusing. No, it's right on spot. This is *not* about the regular value hash, but about the identity hash. > Like hash(), IdentityHashCode doesn't make any promises about identity > at all. It sure does: calling identityHashCode on the very same object twice will always return the same value - even after a garbage collection. So if two references refer to the same object, calling identityHashCode on both of them gives the same value. This is similar to the relationship between equals and hashCode. > Two distinct objects could have the same hash code, and a > further test is needed to distinguish them. Correct. However, references with different identity hash codes will never refer to the same object. >>> Why would you bother using that function when you could just use >>> x == y instead? >> >> Because in a hash table, you also need a hash value. > > Well, sure, in a hash table you need a hash value. But I was talking > about an id() function. > > So is that it? Is IdentityHashValue (or *Code, as the case may be) just > a longer name for hash()? Of course not. Regards, Martin From stephen at xemacs.org Sat Dec 4 13:33:06 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Dec 2010 21:33:06 +0900 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <4CFA1DE7.3010409@pearwood.info> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <4CF9E42D.3010302@pearwood.info> <4CFA02DC.3060807@v.loewis.de> <4CFA1DE7.3010409@pearwood.info> Message-ID: <87y685wwe5.fsf@uwakimon.sk.tsukuba.ac.jp> Steven D'Aprano writes: > Martin v. L?wis wrote: > >> It seems counter-productive to me to bother with an identity function > >> which doesn't meet that constraint. If id(x) == id(y) implies nothing > >> about x and y (they may, or may not, be the same object) then what's the > >> point? > > > > See James' explanation: it would be possible to use this as the > > foundation of an identity hash table. > > I'm afraid James' explanation didn't shed any light on the question to > me. It seems to me that Java's IdentityHashValue[sic -- I think the > correct function name is actually IdentityHashCode] is equivalent to > Python's hash(), not to Python's id(), and claiming it is related to > identity is misleading and confusing. Not quite equivalent. Python's hash() obeys a certain additional constraint (that numeric values that compare equal have the same hash), and IdentityHashValue presumably won't work at all on numbers in Java (since they aren't objects). Bikeshedding aside, I have to agree with you: there's no point in calling such a function id() or IdentityAnything(). Everything you need to know (including both the fact that it could be an efficient way to search for a unique object, and the possible non-uniqueness of the object assigned that code) is contained in the word "hash". From fijall at gmail.com Sat Dec 4 14:01:47 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Sat, 4 Dec 2010 15:01:47 +0200 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <20101204123735.56609926@pitrou.net> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <20101204040645.GA32257@cskk.homeip.net> <20101204123735.56609926@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 1:37 PM, Antoine Pitrou wrote: > On Sat, 4 Dec 2010 15:06:45 +1100 > Cameron Simpson wrote: >> On 03Dec2010 18:15, James Y Knight wrote: >> | On Dec 3, 2010, at 6:04 PM, Terry Reedy wrote: >> | > gc is implementation specific. CPython uses ref counting + cycle >> | > gc. A constraint on all implementations is that objects have a fixed, >> | > unique id during their lifetime. CPython uses the address as the id, so >> | > it cannot move objects. Other implementations do differently. Compacting >> | > gc requires an id to current address table or something. >> | >> | It's somewhat unfortuante that python has this constraint, instead of >> | the looser: "objects have a fixed id during their lifetime", which is >> | much easier to implement, and practically as useful. >> >> Python doesn't have the constraint you imagine; it _does_ have "objects >> have a fixed id during their lifetime". >> >> _CPython_ has this constraint because it uses the address as the id, >> which is free and needs no papping or extra storage. > > Well, the main reason CPython has this constraint is that the C API > mandates constant object pointers. That we can then reuse the pointer > for id() is only a consequence of this. > > Regards > > Antoine. > Yes. I think further discussion of Java-vs-Python makes no sense (and it's proven that C-API-less python can exist and be compatible) From dickinsm at gmail.com Sat Dec 4 14:23:19 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 4 Dec 2010 13:23:19 +0000 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: <20101204124109.07745d71@pitrou.net> References: <4CFA0627.7010202@v.loewis.de> <20101204124109.07745d71@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 11:41 AM, Antoine Pitrou wrote: > Er, normally you don't need *any* Python installed to build Python (be > it 3.x or 2.x). Are you sure about this? I remember needing an existing Python to building Python 2.7 on a new python-less install of FreeBSD a couple of months ago. But maybe that was just an issue with timestamps on files. I'll see if I can reproduce. Mark From solipsis at pitrou.net Sat Dec 4 14:36:06 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 04 Dec 2010 14:36:06 +0100 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: References: <4CFA0627.7010202@v.loewis.de> <20101204124109.07745d71@pitrou.net> Message-ID: <1291469766.3534.6.camel@localhost.localdomain> Le samedi 04 d?cembre 2010 ? 13:23 +0000, Mark Dickinson a ?crit : > On Sat, Dec 4, 2010 at 11:41 AM, Antoine Pitrou wrote: > > Er, normally you don't need *any* Python installed to build Python (be > > it 3.x or 2.x). > > Are you sure about this? I remember needing an existing Python to > building Python 2.7 on a new python-less install of FreeBSD a couple > of months ago. I remember compiling a Python 2.6 (2.5?) on a machine with no Python installed. But that was long ago. (of course, this is difficult to check on a standard development machine :-)) Regards Antoine. From dickinsm at gmail.com Sat Dec 4 14:39:20 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 4 Dec 2010 13:39:20 +0000 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: References: <4CFA0627.7010202@v.loewis.de> <20101204124109.07745d71@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 1:23 PM, Mark Dickinson wrote: > On Sat, Dec 4, 2010 at 11:41 AM, Antoine Pitrou wrote: >> Er, normally you don't need *any* Python installed to build Python (be >> it 3.x or 2.x). > > Are you sure about this? ?I remember needing an existing Python to > building Python 2.7 on a new python-less install of FreeBSD a couple > of months ago. ?But maybe that was just an issue with timestamps on > files. ?I'll see if I can reproduce. With a fresh checkout of the release27-maint branch on an Ubuntu 64-bit VM, with /usr/bin/python renamed to /usr/bin/python_not_here, a './configure && make' fails with: [...] checking for socklen_t... yes checking for build directories... done configure: creating ./config.status config.status: creating Makefile.pre config.status: creating Modules/Setup.config config.status: creating Misc/python.pc config.status: creating Modules/ld_so_aix config.status: creating pyconfig.h creating Modules/Setup creating Modules/Setup.local creating Makefile ./Parser/asdl_c.py -h ./Include ./Parser/Python.asdl /usr/bin/env: python: No such file or directory make: *** [Include/Python-ast.h] Error 127 I think this is the same problem that I saw on the FreeBSD VM. Mark From benjamin at python.org Sat Dec 4 14:40:41 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sat, 4 Dec 2010 07:40:41 -0600 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: References: <4CFA0627.7010202@v.loewis.de> <20101204124109.07745d71@pitrou.net> Message-ID: 2010/12/4 Mark Dickinson : > On Sat, Dec 4, 2010 at 1:23 PM, Mark Dickinson wrote: >> On Sat, Dec 4, 2010 at 11:41 AM, Antoine Pitrou wrote: >>> Er, normally you don't need *any* Python installed to build Python (be >>> it 3.x or 2.x). >> >> Are you sure about this? ?I remember needing an existing Python to >> building Python 2.7 on a new python-less install of FreeBSD a couple >> of months ago. ?But maybe that was just an issue with timestamps on >> files. ?I'll see if I can reproduce. > > With a fresh checkout of the release27-maint branch on an Ubuntu > 64-bit VM, with /usr/bin/python renamed to /usr/bin/python_not_here, a > './configure && make' fails with: > > [...] > checking for socklen_t... yes > checking for build directories... done > configure: creating ./config.status > config.status: creating Makefile.pre > config.status: creating Modules/Setup.config > config.status: creating Misc/python.pc > config.status: creating Modules/ld_so_aix > config.status: creating pyconfig.h > creating Modules/Setup > creating Modules/Setup.local > creating Makefile > ./Parser/asdl_c.py -h ./Include ./Parser/Python.asdl > /usr/bin/env: python: No such file or directory > make: *** [Include/Python-ast.h] Error 127 > > I think this is the same problem that I saw on the FreeBSD VM. You have to touch Include/Python-ast.h and Python/Python-ast.c. We do this for release tarballs. -- Regards, Benjamin From solipsis at pitrou.net Sat Dec 4 14:41:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 04 Dec 2010 14:41:55 +0100 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: References: <4CFA0627.7010202@v.loewis.de> <20101204124109.07745d71@pitrou.net> Message-ID: <1291470115.3534.8.camel@localhost.localdomain> Le samedi 04 d?cembre 2010 ? 13:39 +0000, Mark Dickinson a ?crit : > On Sat, Dec 4, 2010 at 1:23 PM, Mark Dickinson wrote: > > On Sat, Dec 4, 2010 at 11:41 AM, Antoine Pitrou wrote: > >> Er, normally you don't need *any* Python installed to build Python (be > >> it 3.x or 2.x). > > > > Are you sure about this? I remember needing an existing Python to > > building Python 2.7 on a new python-less install of FreeBSD a couple > > of months ago. But maybe that was just an issue with timestamps on > > files. I'll see if I can reproduce. > > With a fresh checkout of the release27-maint branch on an Ubuntu > 64-bit VM, with /usr/bin/python renamed to /usr/bin/python_not_here, a > './configure && make' fails with: How about with the release tarball? Perhaps SVN doesn't get timestamps right. From dickinsm at gmail.com Sat Dec 4 14:58:07 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Sat, 4 Dec 2010 13:58:07 +0000 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: References: <4CFA0627.7010202@v.loewis.de> <20101204124109.07745d71@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 1:40 PM, Benjamin Peterson wrote: > You have to touch Include/Python-ast.h and Python/Python-ast.c. We do > this for release tarballs. Ah, that makes sense. Thanks. Mark From eliben at gmail.com Sat Dec 4 15:03:42 2010 From: eliben at gmail.com (Eli Bendersky) Date: Sat, 4 Dec 2010 16:03:42 +0200 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: <1291470115.3534.8.camel@localhost.localdomain> References: <4CFA0627.7010202@v.loewis.de> <20101204124109.07745d71@pitrou.net> <1291470115.3534.8.camel@localhost.localdomain> Message-ID: On Sat, Dec 4, 2010 at 15:41, Antoine Pitrou wrote: > Le samedi 04 d?cembre 2010 ? 13:39 +0000, Mark Dickinson a ?crit : > > On Sat, Dec 4, 2010 at 1:23 PM, Mark Dickinson > wrote: > > > On Sat, Dec 4, 2010 at 11:41 AM, Antoine Pitrou > wrote: > > >> Er, normally you don't need *any* Python installed to build Python (be > > >> it 3.x or 2.x). > > > > > > Are you sure about this? I remember needing an existing Python to > > > building Python 2.7 on a new python-less install of FreeBSD a couple > > > of months ago. But maybe that was just an issue with timestamps on > > > files. I'll see if I can reproduce. > > > > With a fresh checkout of the release27-maint branch on an Ubuntu > > 64-bit VM, with /usr/bin/python renamed to /usr/bin/python_not_here, a > > './configure && make' fails with: > > How about with the release tarball? Perhaps SVN doesn't get timestamps > right. > > My original problem was that I was re-running 'make' on a svn py3k branch checkout, which already had a compiled ./python exe in its root (Python 3.2, of course). Since the script Objects/typeslots.py (which Martin checked in just yesterday) required Python 2, this failed, although "python" on my machine actually refers to 2.6.5. The failure then happened since in the root of the Python build, "python" referred to the local Python 3 executable. Relying on timestamps sounds a bit too brittle. I think it would be nice if: 1. Parts of the Makefile that use Python checked if Python is installed and generate a useful error if not. 2. All Python scripts that are part of the build should be 2-vs-3 version agnostic for the time being (= until Python 2 is distant history, which won't happen soon) Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Sat Dec 4 15:22:54 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 04 Dec 2010 23:22:54 +0900 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <4CFA326F.7030202@v.loewis.de> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <4CF9E42D.3010302@pearwood.info> <4CFA02DC.3060807@v.loewis.de> <4CFA1DE7.3010409@pearwood.info> <4CFA326F.7030202@v.loewis.de> Message-ID: <87wrnpwrb5.fsf@uwakimon.sk.tsukuba.ac.jp> Quotes are out of order. "Martin v. L?wis" writes: > No, it's not. java.lang.Object.hashCode() is equivalent to Python's > hash(). In particular, for both of these, the following requirement > holds: object that *compare* equal must hash equal. Aha. I see, now. That is indeed a different concept. Mutable objects retain their identities across mutations. > > Two distinct objects could have the same hash code, and a > > further test is needed to distinguish them. > > Correct. However, references with different identity hash codes will > never refer to the same object. Why is useful to expose an identity hash? AFAICS it is *only* useful in building an identity hash table. If so, why not just provide id() or the is operator or both and be done with it? This is different from exposing the value hash AFAICS, where it may be possible to optimize value hashing of composite objects by computing their hashes from the hashes of subobjects, perhaps omitting some subobjects that don't affect equality (eg, Emacs strings may carry display properties that are ignored when they are compared as strings). With identity, I don't see much point in leaving that up to the programmer. In implementations with stable object addresses, the address is a fine implementation of identity_hash() -- perfect, in fact. Otherwise, you either have to omit mutable members from the computation or store the hashcode in the object. In the former situation, you will likely get very coarse hashing, and the set of immutable members (if any ;-) can be computed by the compiler. In the latter, you may as well store an id directly and avoid the whole hash concept for that type (except as a space optimization, but that again implies coarse hashing). > [F]or the identity hash code, or Python's id function: objects that > compare equal may still have different ids, or different java > identity hash codes. ISTM that's no problem, you have to do an extra comparison for identity when the codes are equal either way. Of course the identity hash code is a more precise optimization, but that doesn't make hash() unusable for this purpose (cf Sun's built-in implementation of IdentityHashCode that always returns 2). The problem presumably lies in the fact that there are "id()-ifiable" objects that, because they are mutable, either are unhashable (Python hash) or would have an unstable value hash (Lisp sxhash). Ie, AFAICS def identity_hash (thing): "Wrapper for hash() suitable for building an identity hash table." try: return hash(thing) except TypeError: return 0 would be a fine implementation for Python. > > Like hash(), IdentityHashCode doesn't make any promises about identity > > at all. > > It sure does: calling identityHashCode on the very same object twice > will always return the same value - even after a garbage > collection. Well, if you want to put it that way, so does hash(). I read Steven as placing more emphasis on "like hash()" than on "at all". From a.badger at gmail.com Sat Dec 4 17:42:32 2010 From: a.badger at gmail.com (Toshio Kuratomi) Date: Sat, 4 Dec 2010 08:42:32 -0800 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CF974B9.6040509@v.loewis.de> References: <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> Message-ID: <20101204164232.GH3015@unaka.lan> On Fri, Dec 03, 2010 at 11:52:41PM +0100, "Martin v. L?wis" wrote: > Am 03.12.2010 23:48, schrieb ?ric Araujo: > >> But I'm not interested at all in having it in distutils2. I want the > >> Python build itself to use it, and alas, I can't because of the freeze. > > You can?t in 3.2, true. Neither can you in 3.1, or any previous > > version. If you implement it in distutils2, you have very good chances > > to get it for 3.3. Isn?t that a win? > > It is, unfortunately, a very weak promise. Until distutils2 is > integrated in Python, I probably won't spend any time on it. > At the language summit it was proposed and seemed generally accepted (maybe I took silence as consent... it's been almost a year now) that bold new modules (and bold rewrites of existing modules since it fell out of the distutils/2 discussion) should get implemented in a module on pypi before being merged into the python stdlib. If you wouldn't want to work on any of those modules until they were actually integrated into Python, it sounds like you disagree with that as a general practice? -Toshio -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 198 bytes Desc: not available URL: From ziade.tarek at gmail.com Sat Dec 4 17:55:13 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 4 Dec 2010 17:55:13 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <20101204122710.794d5fe8@pitrou.net> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> <294C7411-A987-45CE-ACE1-6DEFB17DA0C6@fuhm.net> <20101204122710.794d5fe8@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 12:27 PM, Antoine Pitrou wrote: > On Fri, 3 Dec 2010 18:11:57 -0500 > James Y Knight wrote: > >> On Dec 3, 2010, at 5:52 PM, Martin v. L?wis wrote: >> >> > Am 03.12.2010 23:48, schrieb ?ric Araujo: >> >>> But I'm not interested at all in having it in distutils2. I want the >> >>> Python build itself to use it, and alas, I can't because of the freeze. >> >> You can?t in 3.2, true. ?Neither can you in 3.1, or any previous >> >> version. ?If you implement it in distutils2, you have very good chances >> >> to get it for 3.3. ?Isn?t that a win? >> > >> > It is, unfortunately, a very weak promise. Until distutils2 is >> > integrated in Python, I probably won't spend any time on it. >> >> It seems like it'd be a good idea to start integrating distutils2 into python trunk immediately after the 3.2 branch is cut, then. > > +1 from me. +1 too. Just to clarify my position in a few sentences: - I was told not to progressively change distutils - the PEPs for my changes were accepted under the condition that the changes would be made in distutils2 - PEP 384 is yet another accepted PEP, why the rule would change - because it's a small feature ? - I have tons of small features I had to revert and push back in distutils2 If Martin changes make it into Distutils1, I am going to ask that a collection of small features that I have should go there too, because they "should not break things" Regards Tarek -- Tarek Ziad? | http://ziade.org From merwok at netwok.org Sat Dec 4 18:49:04 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sat, 04 Dec 2010 18:49:04 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> <294C7411-A987-45CE-ACE1-6DEFB17DA0C6@fuhm.net> <20101204122710.794d5fe8@pitrou.net> Message-ID: <4CFA7F10.3010308@netwok.org> Le 04/12/2010 17:55, Tarek Ziad? a ?crit : > On Sat, Dec 4, 2010 at 12:27 PM, Antoine Pitrou wrote: >>> It seems like it'd be a good idea to start integrating distutils2 into >>> python trunk immediately after the 3.2 branch is cut, then. >> >> +1 from me. > > +1 too. +1, and I take responsibility to propose a merge plan to python-dev after 3.2 is released and carry it out. Regards From greg at krypto.org Sat Dec 4 19:14:28 2010 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 4 Dec 2010 10:14:28 -0800 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: <20101204124502.37ec1594@pitrou.net> References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 3:45 AM, Antoine Pitrou wrote: > On Sat, 4 Dec 2010 10:10:44 +0100 (CET) > gregory.p.smith wrote: > > Author: gregory.p.smith > > Date: Sat Dec 4 10:10:44 2010 > > New Revision: 87010 > > > > Log: > > issue7213 + issue2320: Cause a DeprecationWarning if the close_fds > argument is > > not passed to subprocess.Popen as the default value will be changing in a > > future Python to the safer and more often desired value of True. > > That doesn't seem to be a good idea under Windows, is it? > > (?Note that on Windows, you cannot set *close_fds* to true and > also redirect the standard handles by setting *stdin*, *stdout* or > *stderr*.?) > Regardless of what platform you are on I think the warning makes sense, it was just worded too strongly. It is asking people to be explicit with close_fds. Explicitly setting close_fds=False when that is desired is good. I modified the docs and warning message to just say that the default close_fds behavior will change in the future without specifying exactly what it will be. It could make sense for the default to be a soft-True on windows that changes behavior if any of the std handles are set if that matches what users expect and find easiest. We may want to reconsider its entire future in the face of the new pass_fds parameter, etc. Those are 3.3 decisions. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sat Dec 4 20:28:34 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 4 Dec 2010 19:28:34 +0000 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: On 4 December 2010 18:14, Gregory P. Smith wrote: > > > On Sat, Dec 4, 2010 at 3:45 AM, Antoine Pitrou wrote: >> >> On Sat, ?4 Dec 2010 10:10:44 +0100 (CET) >> gregory.p.smith wrote: >> > Author: gregory.p.smith >> > Date: Sat Dec ?4 10:10:44 2010 >> > New Revision: 87010 >> > >> > Log: >> > issue7213 + issue2320: Cause a DeprecationWarning if the close_fds >> > argument is >> > not passed to subprocess.Popen as the default value will be changing in >> > a >> > future Python to the safer and more often desired value of True. >> >> That doesn't seem to be a good idea under Windows, is it? >> >> (?Note that on Windows, you cannot set *close_fds* to true and >> also redirect the standard handles by setting *stdin*, *stdout* or >> *stderr*.?) > > Regardless of what platform you are on I think the warning makes sense, it > was just worded too strongly. ?It is asking people to be explicit with > close_fds. ?Explicitly setting close_fds=False when that is desired is good. > I modified the docs and warning message to just say that the default > close_fds behavior will change in the future without specifying exactly what > it will be. ?It could make sense for the default to be a soft-True on > windows that changes behavior if any of the std handles are set if that > matches what users expect and find easiest. ?We may want to reconsider its > entire future in the face of the new pass_fds parameter, etc. ?Those are 3.3 > decisions. This sounds like omitting the close_fds parameter is now considered ill-advised, if not outright wrong. That's silly - I certainly never specify close_fds, expecting the module to do the correct thing if I don't know/care enough to say. I use Popen as a convenience function, so ignoring low-level details is the whole point in my opinion (and close_fds *is* a low-level detail for what I do, on Windows). At the very least, the whole of the section "Replacing Older Functions with the subprocess Module" (with a couple of small exceptions) is in need of updating, as it is recommending bad practice (not specifying close_fds) at the moment... OK, I'm exaggerating a touch here. But can someone point me at the discussion of this change? Or if there hasn't been one, can we have one (i.e., to start the ball rolling, can someone justify the change, please). Paul. From martin at v.loewis.de Sat Dec 4 20:37:14 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Dec 2010 20:37:14 +0100 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <87wrnpwrb5.fsf@uwakimon.sk.tsukuba.ac.jp> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <4CF9E42D.3010302@pearwood.info> <4CFA02DC.3060807@v.loewis.de> <4CFA1DE7.3010409@pearwood.info> <4CFA326F.7030202@v.loewis.de> <87wrnpwrb5.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4CFA986A.8050200@v.loewis.de> > Why is useful to expose an identity hash? AFAICS it is *only* useful > in building an identity hash table. If so, why not just provide id() > or the is operator or both and be done with it? That's precisely James' point: Java provides the identity hash *instead* of the id() function (i.e. it does not have an equivalent of id()). Doing so gives greater liberties in implementing Java. For example, an implementation that would only use the type and not the instance for identity hash would still be conforming (as would one that always returns 0). > With identity, I don't see much point in leaving that up to the > programmer. In implementations with stable object addresses, the > address is a fine implementation of identity_hash() -- perfect, in > fact. Of course. James' complaint is that Python-the-language mandates support for an id() function, though - a requirement that even implementations that don't have stable object addresses now must support. If Python mandated only identity hash properties of the id() function, alternative implementations could be simplified. > ISTM that's no problem, you have to do an extra comparison for > identity when the codes are equal either way. Of course the identity > hash code is a more precise optimization, but that doesn't make hash() > unusable for this purpose (cf Sun's built-in implementation of > IdentityHashCode that always returns 2). That's only the case if the hash() result is guaranteed not to change. In some applications, it may be desirable to have that as an absolute guarantee (rather than just being a convention). No, you can't substitute identity hash with hash: the value hash of an object may change over time, whereas the identity hash must not. Regards, Martin From martin at v.loewis.de Sat Dec 4 20:43:27 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Dec 2010 20:43:27 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <20101204164232.GH3015@unaka.lan> References: <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> <20101204164232.GH3015@unaka.lan> Message-ID: <4CFA99DF.9040907@v.loewis.de> > At the language summit it was proposed and seemed generally accepted (maybe > I took silence as consent... it's been almost a year now) that bold new > modules (and bold rewrites of existing modules since it fell out of the > distutils/2 discussion) should get implemented in a module on pypi before > being merged into the python stdlib. If you wouldn't want to work on any of > those modules until they were actually integrated into Python, it sounds > like you disagree with that as a general practice? No - I just don't feel responsible for projects outside of CPython (unless I have an interest in the project as a user, i.e. to scratch my own itches). I have no use for a distutils2 package whatsoever - to install Python packages, I would either use the builtin distutils (preferred), or setuptools, or distribute (if the package author requires setuptools). I agree that new functionality should stand on its own merits first. And if it fails that test (i.e. it doesn't get much attraction from users when stand-alone), it shouldn't be included into Python at all. Now, distutils2 is different: it's *not* new functionality. So perhaps yes: I disagree with the principle that bold rewrites should be developed independently. Such work would be much better carried out in a branch - it will never stand on its own. Regards, Martin From martin at v.loewis.de Sat Dec 4 20:59:01 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 04 Dec 2010 20:59:01 +0100 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: References: <4CFA0627.7010202@v.loewis.de> <20101204124109.07745d71@pitrou.net> <1291470115.3534.8.camel@localhost.localdomain> Message-ID: <4CFA9D85.3030707@v.loewis.de> > 1. Parts of the Makefile that use Python checked if Python is installed > and generate a useful error if not. > 2. All Python scripts that are part of the build should be 2-vs-3 > version agnostic for the time being (= until Python 2 is distant > history, which won't happen soon) I think this is asked too much, unless you want to also contribute code for that (which I really consider unnecessary). Users who build from the source repository just need to get their build environment right. It's unfortunate that subversion doesn't provide relative ordering of files after an update, perhaps Mercurial is better with that. For a release, the time stamps in the tar file will do fine. Regards, Martin From greg at krypto.org Sat Dec 4 21:13:15 2010 From: greg at krypto.org (Gregory P. Smith) Date: Sat, 4 Dec 2010 12:13:15 -0800 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: On Sat, Dec 4, 2010 at 11:28 AM, Paul Moore wrote: > On 4 December 2010 18:14, Gregory P. Smith wrote: > > > > > > On Sat, Dec 4, 2010 at 3:45 AM, Antoine Pitrou > wrote: > >> > >> On Sat, 4 Dec 2010 10:10:44 +0100 (CET) > >> gregory.p.smith wrote: > >> > Author: gregory.p.smith > >> > Date: Sat Dec 4 10:10:44 2010 > >> > New Revision: 87010 > >> > > >> > Log: > >> > issue7213 + issue2320: Cause a DeprecationWarning if the close_fds > >> > argument is > >> > not passed to subprocess.Popen as the default value will be changing > in > >> > a > >> > future Python to the safer and more often desired value of True. > >> > >> That doesn't seem to be a good idea under Windows, is it? > >> > >> (?Note that on Windows, you cannot set *close_fds* to true and > >> also redirect the standard handles by setting *stdin*, *stdout* or > >> *stderr*.?) > > > > Regardless of what platform you are on I think the warning makes sense, > it > > was just worded too strongly. It is asking people to be explicit with > > close_fds. Explicitly setting close_fds=False when that is desired is > good. > > I modified the docs and warning message to just say that the default > > close_fds behavior will change in the future without specifying exactly > what > > it will be. It could make sense for the default to be a soft-True on > > windows that changes behavior if any of the std handles are set if that > > matches what users expect and find easiest. We may want to reconsider > its > > entire future in the face of the new pass_fds parameter, etc. Those are > 3.3 > > decisions. > > This sounds like omitting the close_fds parameter is now considered > ill-advised, if not outright wrong. That's silly - I certainly never > specify close_fds, expecting the module to do the correct thing if I > don't know/care enough to say. I use Popen as a convenience function, > so ignoring low-level details is the whole point in my opinion (and > close_fds *is* a low-level detail for what I do, on Windows). > > At the very least, the whole of the section "Replacing Older Functions > with the subprocess Module" (with a couple of small exceptions) is in > need of updating, as it is recommending bad practice (not specifying > close_fds) at the moment... > > OK, I'm exaggerating a touch here. But can someone point me at the > discussion of this change? Or if there hasn't been one, can we have > one (i.e., to start the ball rolling, can someone justify the change, > please). > > Paul. > Making the change was intended to force the discussion. I'm glad that worked. :) I don't like the thought of requiring people to specify close_fds either but the default is a bad one (see http://bugs.python.org/issue7213 and http://bugs.python.org/issue2320) so we should change it. The real question is how should we go about doing the change? This warning asking people to always specify close_fds=True is heavy handed. Other places in the stdlib and docs no doubt still need to be updated to reflect it, etc. Options that seem worthy of debate: A) The DeprecationWarning that exists in py3k as of today. B) Remove the DeprecationWarning, simply document that people should be explicit about it if they care at all and that the default will change in 3.3. C) Never change close_fds behavior. we're stuck with it for life. D) Deprecate close_fds so that it becomes a legacy no-op. the new pass_fds flag could evolve into this. this will break existing code that depends on close_fds one way or another. I'm in 100% agreement that forcing people to pass close_fds in makes the API less friendly (granted, people already find it unfriendly but why make it worse?). Option B seems the most friendly to me. Option D is always on the table but I haven't planned out what a future without it should look like. I prefer requiring people who need open file descriptors to pass them in; a semaphore for "all fds" could be created and pass_fds=ALL becomes the new "close_fds=False" (I think one of the bugs suggests this). -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Sat Dec 4 21:29:41 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 4 Dec 2010 15:29:41 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> Message-ID: On Fri, Dec 3, 2010 at 12:10 AM, Alexander Belopolsky wrote: .. > I don't think decimal module should support non-European decimal > digits. ?The only place where it can make some sense is in int() > because here we have a fighting chance of producing a reasonable > definition. ? The motivating use case is conversion of numerical data > extracted from text using simple '\d+' ?regex matches. > It turns out, this use case does not quite work in Python either: >>> re.compile(r'\s+(\d+)\s+').match(' \u2081\u2082\u2083 ').group(1) '???' >>> int(_) Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'decimal' codec can't encode character '\u2081' in position 0: invalid decimal Unicode string This may actually be a bug in Python regex implementation because Unicode standard seems to recommend that '\d' be interpreted as gc = Decimal_Number (Nd): http://unicode.org/reports/tr18/#Compatibility_Properties I actually wonder if Python's re module can claim to provide even Basic Unicode Support. http://unicode.org/reports/tr18/#Basic_Unicode_Support > Here is how I would do it: > > 1. ?String x of non-European decimal digits is only accepted in > int(x), but not by int(x, 0) or int(x, 10). > 2. ?If x contains one or more non-European digits, then > > ? ?(a) ?all digits must be from the same block: > > ? ? ?def basepoint(c): > ? ? ? ? ? ?return ord(c) - unicodedata.digit(c) > ? ? ?all(basepoint(c) == basepoint(x[0]) for c in x) -> True > > ? ? (b) and '+' or '-' sign is not alowed. > > 3. A character c is a digit if it matches '\d' regex. ?I think this > means unicodedata.category(c) -> 'Nd'. > > Condition 2(b) is important because there is no clear way to define > what is acceptable as '+' or '-' using Unicode character properties > and not all number systems even support local form of negation. ?(It > is also YAGNI.) > From tjreedy at udel.edu Sat Dec 4 23:51:44 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 04 Dec 2010 17:51:44 -0500 Subject: [Python-Dev] [Python-checkins] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: <20101204091044.C6CBEEE985@mail.python.org> References: <20101204091044.C6CBEEE985@mail.python.org> Message-ID: <4CFAC600.5040905@udel.edu> > DeprecationWarnings that show up in a lot of existing code are controversial > and have caused pain in the past. I'd like to leave this on for 3.2 beta1 and > see how things go. We can remove the warning if it is deemed too noisy during > any betas. (case study: the md5 and sha module DeprecationWarnings are loathed > around the world as those modules were never going to be removed in 2.x and > 2to3 has a fixer for code that uses them) My understanding is that DeprecationWarnings are now off by default in part because of issues like the above. So they should only be seen by programmers who turn them of for the purpose of seeing them so they can future-proof their code before it fails. As a result, there have been several added recently in efforts to clean up the APis. The real issue is changing the default. If allowed, should it be switched in one release or should there be an intermediate release with no default. Which are you planning? My feeling is that we should be able to correct bad-enough mistakes. Code can easily be made portable across all 3.x versions by given an explicit value and not depending on defaults. I think there should be an intermediate version to cause overt failure of code that is not upgraded, because the programmer neglected either to turn DeprecationWarnings on for test runs or to read the News releases. Terry J. Reedy From martin at v.loewis.de Sat Dec 4 23:58:56 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 04 Dec 2010 23:58:56 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> Message-ID: <4CFAC7B0.3080307@v.loewis.de> > I actually wonder if Python's re module can claim to provide even > Basic Unicode Support. Do you really wonder? Most definitely it does not. Regards, Martin From p.f.moore at gmail.com Sun Dec 5 00:07:04 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 4 Dec 2010 23:07:04 +0000 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: On 4 December 2010 20:13, Gregory P. Smith wrote: >> This sounds like omitting the close_fds parameter is now considered >> ill-advised, if not outright wrong. [...] > Making the change was intended to force the discussion. ?I'm glad that > worked. :) :-) > I don't like the thought of requiring people to specify close_fds either but > the default is a bad one (see http://bugs.python.org/issue7213 > and?http://bugs.python.org/issue2320) so we should change it. Both of these issues seem to be for Unix (based on the fact that they use grep and cat as executables - I haven't dug very deeply into this). I work on Windows only, so I'm happy to take the experts' opinions on that OS. Is there an issue on Windows? If not, and given how different FD inheritance is on Windows, I'd argue that in the absence of bug reports, there's no need to change behaviour on Windows. > The real question is how should we go about doing the change? ?This warning > asking people to always specify close_fds=True is heavy handed. ?Other > places in the stdlib and docs no doubt still need to be updated to reflect > it, etc. > > Options that seem worthy of debate: > A) The DeprecationWarning that exists in py3k as of today. Given that the option appears not to be getting deprecated (just the default changed), this seems wrong. I know DeprecationWarnings are off by default. If someone switches them on, they'd expect to have to address them. But what should they do in this case? Unless they have thought hard about FD inheritance, and then explicitly decided to use the default because it matches what they want (as opposed to specifying it explicitly, and documenting their intent clearly), nothing. Or they use the default expecting it to be good enough, and suddenly have to worry if they are relying on something that's changing (I have no feel for how likely it is that they'll be affected by the change, other than to say that personally, none of my code would be). That seems to me like a bad use of a DeprecationWarning, as a non-trivial amount of the time there's nothing to do. > B) Remove the DeprecationWarning, simply document that people should be > explicit about it if they care at all and that the default will change in > 3.3. That seems reasonable to me. But I can see that the backward compatibility rules might apply here. On the other hand, given the bug reports, surely this is a bug fix? > C) Never change close_fds behavior. ?we're stuck with it for life. Sounds harsh. But look at it another way - if compatibility rules mean that we can't change the default, the reported bugs need to be fixed another way. Or behaviour needs to be documented more explicitly so that the bugs can be downgraded to "user error" caused by not reading the (new, improved) documentation. Compatibility rules are to protect working programs. What proportion of programs using the default are "working" and what proportion are "wrong"? It seems to me that most are "working", so compatibility applies. > D) Deprecate close_fds so that it becomes a legacy no-op. ?the new pass_fds > flag could evolve into this. ?this will break existing code that depends on > close_fds one way or another. This works for me, as I never specify close_fds. > I'm in 100% agreement that forcing people to pass close_fds in makes the API > less friendly (granted, people already find it unfriendly but why make it > worse?). I'd say making the parameter mandatory is highly unfriendly - to the point of making the API feel like an "advanced" one, not suitable for simple cases like replacing os.system. > Option B seems the most friendly to me. > Option D is always on the table but I haven't planned out what a future > without it should look like. ?I prefer requiring people who need open file > descriptors to pass them in; a semaphore for "all fds" could be created and > pass_fds=ALL becomes the new "close_fds=False" (I think one of the bugs > suggests this). In my view: 1. Don't change Windows behaviour unless there's a demonstrable issue. The Popen code is split into 2 distinct paths internally anyway, so that shouldn't be a problem. If documenting the behaviour becomes awkward should Unix behaviour change and Windows doesn't, that indicates to me that there's a problem with the behaviour ("if it's hard to explain..."). 2. Users who don't specify close_fds because they don't care (as opposed to because they explicitly want the current default, but choose not to say so - I appreciate that this is a fine distinction, difficult to police in practice) should see no change in behaviour, apart from identified bugs being fixed. If you're not hitting a bug, you should see no change at all. Keep the interface straightforward for people who don't know (or don't want to worry about) the subtleties of FD inheritance. The original goal was for subprocess to replace os.system, os.popen, os.spawn, etc. That's never quite happened because subprocess is just a little bit too conceptually complex for those basic tasks. Let's not make it worse! Paul. From fuzzyman at voidspace.org.uk Sun Dec 5 00:10:01 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 04 Dec 2010 23:10:01 +0000 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <20101204122710.794d5fe8@pitrou.net> References: <4CF8008B.60402@v.loewis.de> <4CF80964.4070300@v.loewis.de> <4CF80D6C.6050703@v.loewis.de> <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> <294C7411-A987-45CE-ACE1-6DEFB17DA0C6@fuhm.net> <20101204122710.794d5fe8@pitrou.net> Message-ID: <4CFACA49.5060102@voidspace.org.uk> On 04/12/2010 11:27, Antoine Pitrou wrote: > On Fri, 3 Dec 2010 18:11:57 -0500 > James Y Knight wrote: > >> On Dec 3, 2010, at 5:52 PM, Martin v. L?wis wrote: >> >>> Am 03.12.2010 23:48, schrieb ?ric Araujo: >>>>> But I'm not interested at all in having it in distutils2. I want the >>>>> Python build itself to use it, and alas, I can't because of the freeze. >>>> You can?t in 3.2, true. Neither can you in 3.1, or any previous >>>> version. If you implement it in distutils2, you have very good chances >>>> to get it for 3.3. Isn?t that a win? >>> It is, unfortunately, a very weak promise. Until distutils2 is >>> integrated in Python, I probably won't spend any time on it. >> It seems like it'd be a good idea to start integrating distutils2 into python trunk immediately after the 3.2 branch is cut, then. > +1 from me. > And from me. Michael > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From p.f.moore at gmail.com Sun Dec 5 00:15:19 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 4 Dec 2010 23:15:19 +0000 Subject: [Python-Dev] [Python-checkins] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: <4CFAC600.5040905@udel.edu> References: <20101204091044.C6CBEEE985@mail.python.org> <4CFAC600.5040905@udel.edu> Message-ID: On 4 December 2010 22:51, Terry Reedy wrote: > My feeling is that we should be able to correct bad-enough mistakes. That's fair. Is this a "bad-enough mistake"? From a brief reading of the 2 bug reports, combined with my own trouble-free experience with Popen (always leaving close_fds as default), it doesn't seem so, to me. If someone provides evidence that this is a serious bug in the API, then that's a different matter. Paul. From p.f.moore at gmail.com Sun Dec 5 00:17:49 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 4 Dec 2010 23:17:49 +0000 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: On 4 December 2010 23:07, Paul Moore wrote: > Is there an issue on Windows? If not, and given how different FD > inheritance is on Windows, I'd argue that in the absence of bug > reports, there's no need to change behaviour on Windows. Actually, from the error message I just got: ValueError: close_fds is not supported on Windows platforms if you redirect stdin/stdout/stderr So on Windows, for the issues mentioned (both of which involve redirected stdin/out/err), leaving the current default appears to be not only advisable, but required. I suspect an os-dependent default would be ugly to document... Paul. From v+python at g.nevcal.com Sun Dec 5 01:26:47 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 04 Dec 2010 16:26:47 -0800 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: <4CFADC47.7020702@g.nevcal.com> On 12/4/2010 3:07 PM, Paul Moore wrote: > The original goal was for subprocess to replace os.system, os.popen, > os.spawn, etc. That's never quite happened because subprocess is just > a little bit too conceptually complex for those basic tasks. Is that way? I didn't find it particularly hard to learn, given the "cheat sheet" of techniques for doing the replacements. However, it is a bit deficient in providing non-blocking handling primitives for actually communicating with interactive spawned programs. subprocess.communicate provides one technique, which works for an extremely limited set of circumstances: I've offered some alternatives in http://bugs.python.org/issue10482 that greatly expand the types of communications that can be achieved without deadlock. Of course, it is still a small fraction of the possible circumstances, and doesn't really handle the hard case of feeding a program and concurrently analyzing its output to determine its future feedings: that still requires a complex, explicitly threaded and synchronized program. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian.curtin at gmail.com Sun Dec 5 03:08:35 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Sat, 4 Dec 2010 20:08:35 -0600 Subject: [Python-Dev] [Python-checkins] r87070 - python/branches/py3k/Lib/test/test_shutil.py In-Reply-To: <20101205020417.16069EE9C9@mail.python.org> References: <20101205020417.16069EE9C9@mail.python.org> Message-ID: On Sat, Dec 4, 2010 at 20:04, hirokazu.yamamoto wrote: > Author: hirokazu.yamamoto > Date: Sun Dec 5 03:04:16 2010 > New Revision: 87070 > > Log: > Now can reproduce the error on AMD64 Windows Server 2008 > even where os.symlink is not supported. > > > Modified: > python/branches/py3k/Lib/test/test_shutil.py > > Modified: python/branches/py3k/Lib/test/test_shutil.py > > ============================================================================== > --- python/branches/py3k/Lib/test/test_shutil.py (original) > +++ python/branches/py3k/Lib/test/test_shutil.py Sun Dec 5 03:04:16 > 2010 > @@ -271,25 +271,33 @@ > shutil.rmtree(src_dir) > shutil.rmtree(os.path.dirname(dst_dir)) > > - @unittest.skipUnless(hasattr(os, "symlink"), > - "Missing symlink implementation") > + @unittest.skipUnless(hasattr(os, 'link'), 'requires os.link') > def test_dont_copy_file_onto_link_to_itself(self): > # bug 851123. > os.mkdir(TESTFN) > src = os.path.join(TESTFN, 'cheese') > dst = os.path.join(TESTFN, 'shop') > try: > - f = open(src, 'w') > - f.write('cheddar') > - f.close() > - > - if hasattr(os, "link"): > - os.link(src, dst) > - self.assertRaises(shutil.Error, shutil.copyfile, src, dst) > - with open(src, 'r') as f: > - self.assertEqual(f.read(), 'cheddar') > - os.remove(dst) > + with open(src, 'w') as f: > + f.write('cheddar') > + os.link(src, dst) > + self.assertRaises(shutil.Error, shutil.copyfile, src, dst) > + with open(src, 'r') as f: > + self.assertEqual(f.read(), 'cheddar') > + os.remove(dst) > + finally: > + shutil.rmtree(TESTFN, ignore_errors=True) > > + @unittest.skipUnless(hasattr(os, "symlink"), > + "Missing symlink implementation") > + def test_dont_copy_file_onto_symlink_to_itself(self): > + # bug 851123. > + os.mkdir(TESTFN) > + src = os.path.join(TESTFN, 'cheese') > + dst = os.path.join(TESTFN, 'shop') > + try: > + with open(src, 'w') as f: > + f.write('cheddar') > # Using `src` here would mean we end up with a symlink pointing > # to TESTFN/TESTFN/cheese, while it should point at > # TESTFN/cheese. > @@ -299,10 +307,7 @@ > self.assertEqual(f.read(), 'cheddar') > os.remove(dst) > finally: > - try: > - shutil.rmtree(TESTFN) > - except OSError: > - pass > + shutil.rmtree(TESTFN, ignore_errors=True) > > @unittest.skipUnless(hasattr(os, "symlink"), > "Missing symlink implementation") > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > I created #10540 for this issue, but the patch I have on there is just a bad hack. I need to fix os.path.samefile for hard links, which might be easier if we keep st_ino data in stat structures on Windows. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ocean-city at m2.ccsnet.ne.jp Sun Dec 5 04:15:33 2010 From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto) Date: Sun, 05 Dec 2010 12:15:33 +0900 Subject: [Python-Dev] [Python-checkins] r87070 - python/branches/py3k/Lib/test/test_shutil.py In-Reply-To: References: <20101205020417.16069EE9C9@mail.python.org> Message-ID: <4CFB03D5.2060204@m2.ccsnet.ne.jp> On 2010/12/05 11:08, Brian Curtin wrote: > I created #10540 for this issue, but the patch I have on there is just a bad > hack. I need to fix os.path.samefile for hard links, which might be easier > if we keep st_ino data in stat structures on Windows. MSDN says, http://msdn.microsoft.com/en-us/library/aa363788%28v=VS.85%29.aspx > In the NTFS file system, a file keeps the same file ID until it is deleted. Hard link can be live in NTFS only, so maybe we can say st_ino data is valid until hard links are established. From ocean-city at m2.ccsnet.ne.jp Sun Dec 5 05:09:37 2010 From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto) Date: Sun, 05 Dec 2010 13:09:37 +0900 Subject: [Python-Dev] [Python-checkins] r87070 - python/branches/py3k/Lib/test/test_shutil.py In-Reply-To: References: <20101205020417.16069EE9C9@mail.python.org> Message-ID: <4CFB1081.8030805@m2.ccsnet.ne.jp> I missed it, st_dev is not set yet. When I set st_dev with dwVolumeSerialNumber, it was sometimes negative. Is it OK? From stephen at xemacs.org Sun Dec 5 09:17:37 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sun, 05 Dec 2010 17:17:37 +0900 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <4CFA986A.8050200@v.loewis.de> References: <203DD6FC-D9F7-4E7D-B49C-CABC1DF7352B@fuhm.net> <4CF9E42D.3010302@pearwood.info> <4CFA02DC.3060807@v.loewis.de> <4CFA1DE7.3010409@pearwood.info> <4CFA326F.7030202@v.loewis.de> <87wrnpwrb5.fsf@uwakimon.sk.tsukuba.ac.jp> <4CFA986A.8050200@v.loewis.de> Message-ID: <87sjycws4e.fsf@uwakimon.sk.tsukuba.ac.jp> "Martin v. L?wis" writes: > > Why is useful to expose an identity hash? AFAICS it is *only* useful > > in building an identity hash table. If so, why not just provide id() > > or the is operator or both and be done with it? > > That's precisely James' point: Java provides the identity hash > *instead* of the id() function (i.e. it does not have an equivalent > of id()). Doing so gives greater liberties in implementing Java. Yes, we understand that it makes the implementer's job easier. *Why bother having an identity hash at all?* Having taken away id() and provided maximum leisure to the implementer via def identity_hash(object): return 42 is there *any* benefit left for the user/developer? All I see is costs: costs in implementation, costs in debugging. And AFAICS this is a problem that can be solved once and reused by everybody who needs id(); why does every developer need to write his own id() function? From ncoghlan at gmail.com Sun Dec 5 10:39:47 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 5 Dec 2010 19:39:47 +1000 Subject: [Python-Dev] [Python-checkins] r87066 - python/branches/py3k/Doc/whatsnew/3.2.rst In-Reply-To: <20101204225625.CDFD8EEA29@mail.python.org> References: <20101204225625.CDFD8EEA29@mail.python.org> Message-ID: On Sun, Dec 5, 2010 at 8:56 AM, raymond.hettinger wrote: > +.. seealso:: > + > + ? :pep:`384` - PYC Repository Directories > + ? ? ?PEP written by Martin von Loewis. > + The PEP title here should be "Defining a Stable ABI" (I noticed a later checkin fixing the PEP 3148 see-also title, but I didn't see anything for this one). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Sun Dec 5 12:36:05 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 5 Dec 2010 11:36:05 +0000 (UTC) Subject: [Python-Dev] Repo frozen for 3.2 Message-ID: I've just been notified via being added to the nosy list of http://bugs.python.org/issue10627 about the deprecation of ConfigParser for 3.2. I presume I was added to this list because logging.config uses ConfigParser, but logging.config doesn't use any interpolation features so I could easily change all usages to SafeConfigParser; should I do it now (before the 3.2 beta is cut) or hold off for now? Regards, Vinay Sajip From tseaver at palladion.com Sun Dec 5 13:45:02 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 05 Dec 2010 07:45:02 -0500 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/04/2010 03:13 PM, Gregory P. Smith wrote: > On Sat, Dec 4, 2010 at 11:28 AM, Paul Moore wrote: > >> On 4 December 2010 18:14, Gregory P. Smith wrote: >>> >>> >>> On Sat, Dec 4, 2010 at 3:45 AM, Antoine Pitrou >> wrote: >>>> >>>> On Sat, 4 Dec 2010 10:10:44 +0100 (CET) >>>> gregory.p.smith wrote: >>>>> Author: gregory.p.smith >>>>> Date: Sat Dec 4 10:10:44 2010 >>>>> New Revision: 87010 >>>>> >>>>> Log: >>>>> issue7213 + issue2320: Cause a DeprecationWarning if the close_fds >>>>> argument is >>>>> not passed to subprocess.Popen as the default value will be changing >> in >>>>> a >>>>> future Python to the safer and more often desired value of True. >>>> >>>> That doesn't seem to be a good idea under Windows, is it? >>>> >>>> (?Note that on Windows, you cannot set *close_fds* to true and >>>> also redirect the standard handles by setting *stdin*, *stdout* or >>>> *stderr*.?) >>> >>> Regardless of what platform you are on I think the warning makes sense, >> it >>> was just worded too strongly. It is asking people to be explicit with >>> close_fds. Explicitly setting close_fds=False when that is desired is >> good. >>> I modified the docs and warning message to just say that the default >>> close_fds behavior will change in the future without specifying exactly >> what >>> it will be. It could make sense for the default to be a soft-True on >>> windows that changes behavior if any of the std handles are set if that >>> matches what users expect and find easiest. We may want to reconsider >> its >>> entire future in the face of the new pass_fds parameter, etc. Those are >> 3.3 >>> decisions. >> >> This sounds like omitting the close_fds parameter is now considered >> ill-advised, if not outright wrong. That's silly - I certainly never >> specify close_fds, expecting the module to do the correct thing if I >> don't know/care enough to say. I use Popen as a convenience function, >> so ignoring low-level details is the whole point in my opinion (and >> close_fds *is* a low-level detail for what I do, on Windows). >> >> At the very least, the whole of the section "Replacing Older Functions >> with the subprocess Module" (with a couple of small exceptions) is in >> need of updating, as it is recommending bad practice (not specifying >> close_fds) at the moment... >> >> OK, I'm exaggerating a touch here. But can someone point me at the >> discussion of this change? Or if there hasn't been one, can we have >> one (i.e., to start the ball rolling, can someone justify the change, >> please). >> >> Paul. >> > > Making the change was intended to force the discussion. I'm glad that > worked. :) > > I don't like the thought of requiring people to specify close_fds either but > the default is a bad one (see http://bugs.python.org/issue7213 and > http://bugs.python.org/issue2320) so we should change it. > > The real question is how should we go about doing the change? This warning > asking people to always specify close_fds=True is heavy handed. Other > places in the stdlib and docs no doubt still need to be updated to reflect > it, etc. > > > Options that seem worthy of debate: > > A) The DeprecationWarning that exists in py3k as of today. > > B) Remove the DeprecationWarning, simply document that people should be > explicit about it if they care at all and that the default will change in > 3.3. > > C) Never change close_fds behavior. we're stuck with it for life. > > D) Deprecate close_fds so that it becomes a legacy no-op. the new pass_fds > flag could evolve into this. this will break existing code that depends on > close_fds one way or another. > > > I'm in 100% agreement that forcing people to pass close_fds in makes the API > less friendly (granted, people already find it unfriendly but why make it > worse?). > > Option B seems the most friendly to me. +1 from me. People who don't care about the 'close_fds' behavior one way or the other shouldn't get a warning which only helps the tiny (I assert) minority who a) *do* care but b) don't pass the flag explicitly. Tres.. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkz7iU4ACgkQ+gerLs4ltQ4SJgCfePUImv5OSHzzZ4QJvzUz1oYJ LhAAoKRut3AfGkS23hghQx9pd3D0WF3p =y8hn -----END PGP SIGNATURE----- From lukasz at langa.pl Sun Dec 5 14:29:16 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 5 Dec 2010 14:29:16 +0100 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: References: Message-ID: Reposted for reference what originally only went to Vinay. Wiadomo?? napisana przez Vinay Sajip w dniu 2010-12-05, o godz. 12:36: > I've just been notified via being added to the nosy list of > > http://bugs.python.org/issue10627 > > about the deprecation of ConfigParser for 3.2. I presume I was added to this > list because logging.config uses ConfigParser, but logging.config doesn't use > any interpolation features so I could easily change all usages to > SafeConfigParser; should I do it now (before the 3.2 beta is cut) or hold off > for now? > Hold off, just like distutils2 etc. :) I finished the deprecation patch too late and so #10627 was created literally hours (if not minutes) before the freeze. Sorry for that. On a related note, if you're sure logging users don't use any interpolation, you can also use SafeConfigParser(interpolation=None) so then all values become raw by default (e.g. people can use Python string formatting directives, % signs etc.). We can discuss this later on when the time comes for that. -- Best regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From merwok at netwok.org Sun Dec 5 15:19:16 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sun, 05 Dec 2010 15:19:16 +0100 Subject: [Python-Dev] Unanswered reactions to python-checkins In-Reply-To: <4C8D59F3.9020101@netwok.org> References: <20100912163158.7902EF11D@mail.python.org> <4C8D59F3.9020101@netwok.org> Message-ID: <4CFB9F64.90504@netwok.org> Hello, Three messages sent in reaction to python-checkins email have not got any reply so far, so I?m resending them. Regards Nick, in reaction to the reprlib.recursive_repr commit: >> > + # Can't use functools.wraps() here because of bootstrap issues >> > + wrapper.__module__ = getattr(user_function, '__module__') >> > + wrapper.__doc__ = getattr(user_function, '__doc__') >> > + wrapper.__name__ = getattr(user_function, '__name__') >> > + return wrapper > Perhaps add __wrapped__ as well? > > (I assume that, similar to _collections before it was made a builtin, > the bootstrap issue is that _functools is an extension module rather > than builtin, but reprlib is needed when building the extension > modules?) Me, about a zlib-related change in Misc/NEWS: > +zlib > +---- > + > +The :mod:`zlib` extension is built using an included copy of the zlib > +sources unless the zlib version found on the system is too old to be > +used for the build:: > > Unless or if? Building with an included copy *if* the system one is too > old makes sense to me, not the contrary. Am I not seeing something? Me, about a change to winsound.PlaySound: > Extension Modules > ----------------- > > +- Issue #6317: Now winsound.PlaySound only accepts unicode. > + > - Issue #6317: Now winsound.PlaySound can accept non ascii filename. I think the new entry should have replaced the older: ?only accepts? trumps ?can accept?. From solipsis at pitrou.net Sun Dec 5 15:20:41 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 5 Dec 2010 15:20:41 +0100 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: <20101205152041.06be7433@pitrou.net> On Sat, 4 Dec 2010 23:17:49 +0000 Paul Moore wrote: > On 4 December 2010 23:07, Paul Moore wrote: > > Is there an issue on Windows? If not, and given how different FD > > inheritance is on Windows, I'd argue that in the absence of bug > > reports, there's no need to change behaviour on Windows. > > Actually, from the error message I just got: > > ValueError: close_fds is not supported on Windows platforms if you > redirect stdin/stdout/stderr > > So on Windows, for the issues mentioned (both of which involve > redirected stdin/out/err), leaving the current default appears to be > not only advisable, but required. > > I suspect an os-dependent default would be ugly to document... How about a best-effort behaviour? Setting close_fds to True would only close fds where possible (i.e., not under Windows when piping either of stdin, stdout, stderr). Regards Antoine. From g.brandl at gmx.net Sun Dec 5 15:23:42 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 05 Dec 2010 15:23:42 +0100 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: <20101205152041.06be7433@pitrou.net> References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> <20101205152041.06be7433@pitrou.net> Message-ID: Am 05.12.2010 15:20, schrieb Antoine Pitrou: > On Sat, 4 Dec 2010 23:17:49 +0000 > Paul Moore wrote: >> On 4 December 2010 23:07, Paul Moore wrote: >> > Is there an issue on Windows? If not, and given how different FD >> > inheritance is on Windows, I'd argue that in the absence of bug >> > reports, there's no need to change behaviour on Windows. >> >> Actually, from the error message I just got: >> >> ValueError: close_fds is not supported on Windows platforms if you >> redirect stdin/stdout/stderr >> >> So on Windows, for the issues mentioned (both of which involve >> redirected stdin/out/err), leaving the current default appears to be >> not only advisable, but required. >> >> I suspect an os-dependent default would be ugly to document... > > How about a best-effort behaviour? Setting close_fds to True would only > close fds where possible (i.e., not under Windows when piping either of > stdin, stdout, stderr). That sounds reasonable. Georg From ocean-city at m2.ccsnet.ne.jp Sun Dec 5 17:12:15 2010 From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto) Date: Mon, 06 Dec 2010 01:12:15 +0900 Subject: [Python-Dev] Unanswered reactions to python-checkins In-Reply-To: <4CFB9F64.90504@netwok.org> References: <20100912163158.7902EF11D@mail.python.org> <4C8D59F3.9020101@netwok.org> <4CFB9F64.90504@netwok.org> Message-ID: <4CFBB9DF.1010607@m2.ccsnet.ne.jp> On 2010/12/05 23:19, ?ric Araujo wrote: > Me, about a change to winsound.PlaySound: >> Extension Modules >> ----------------- >> >> +- Issue #6317: Now winsound.PlaySound only accepts unicode. >> + >> - Issue #6317: Now winsound.PlaySound can accept non ascii filename. > I think the new entry should have replaced the older: ?only accepts? > trumps ?can accept?. This is two difference thing. Ordinary winsound.PlaySound accepted both bytes and str, but cannot accept non ascii filename. Now it accepts only str, but can accept non ascii filename. I can read "can accept" as "can accept str, but also bytes", but this is not true. From fdrake at acm.org Sun Dec 5 18:10:25 2010 From: fdrake at acm.org (Fred Drake) Date: Sun, 5 Dec 2010 12:10:25 -0500 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: References: Message-ID: 2010/12/5 ?ukasz Langa : > On a related note, if you're sure logging users don't use any interpolation, > you can also use SafeConfigParser(interpolation=None) so then all values > become raw by default (e.g. people can use Python string formatting > directives, % signs etc.). We can discuss this later on ?when the time comes > for that. This is the hard part, though. So long as the users decide whether to use the interpolation features, it has to be treated as an API compatibility issue. The interpolation syntax is a feature of the language being parsed more than a code-level feature. It's actually a good thing logging is using the ancient ConfigParser, since the interpolation handling is so broken there's unlikely to be any affected uses of '%' in working configurations. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From p.f.moore at gmail.com Sun Dec 5 19:02:07 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 5 Dec 2010 18:02:07 +0000 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: <20101205152041.06be7433@pitrou.net> References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> <20101205152041.06be7433@pitrou.net> Message-ID: On 5 December 2010 14:20, Antoine Pitrou wrote: > How about a best-effort behaviour? Setting close_fds to True would only > close fds where possible (i.e., not under Windows when piping either of > stdin, stdout, stderr). Is that plausible? I thought that it's possible to close fds, but doesn't necessarily do the right thing. If it's possible to do this, I'd be OK with it, but if it could cause problems just as subtle as the ones we already have, I don't see the benefit. Paul. From skip at pobox.com Sun Dec 5 19:03:33 2010 From: skip at pobox.com (skip at pobox.com) Date: Sun, 5 Dec 2010 12:03:33 -0600 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: <4CFADC47.7020702@g.nevcal.com> References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> <4CFADC47.7020702@g.nevcal.com> Message-ID: <19707.54261.55334.594655@montanaro.dyndns.org> Glenn> On 12/4/2010 3:07 PM, Paul Moore wrote: >> The original goal was for subprocess to replace os.system, os.popen, >> os.spawn, etc. That's never quite happened because subprocess is just >> a little bit too conceptually complex for those basic tasks. Glenn> Is that way? I didn't find it particularly hard to learn, given Glenn> the "cheat sheet" of techniques for doing the replacements. For 99% of my usage (I suspect for most other peoples' as well, at least on Unix-y systems), this is all I need: for line in os.popen("some pipeline"): do_stuff(line) No cheat sheet necessary. I don't see how subprocess could have made that common idiom any simpler. Maybe it's better at doing esoteric stuff, however that falls into the 1% where a simple os.popen isn't adequate. Skip From skip at pobox.com Sun Dec 5 19:06:32 2010 From: skip at pobox.com (skip at pobox.com) Date: Sun, 5 Dec 2010 12:06:32 -0600 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: References: Message-ID: <19707.54440.612130.268676@montanaro.dyndns.org> Vinay> ... the deprecation of ConfigParser for 3.2.... What's the rush? It's been deprecated, not removed. Skip From greg at krypto.org Sun Dec 5 20:08:43 2010 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 5 Dec 2010 11:08:43 -0800 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: On Sun, Dec 5, 2010 at 4:45 AM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 12/04/2010 03:13 PM, Gregory P. Smith wrote: > > > > Making the change was intended to force the discussion. I'm glad that > > worked. :) > > > > I don't like the thought of requiring people to specify close_fds either > but > > the default is a bad one (see http://bugs.python.org/issue7213 and > > http://bugs.python.org/issue2320) so we should change it. > > > > The real question is how should we go about doing the change? This > warning > > asking people to always specify close_fds=True is heavy handed. Other > > places in the stdlib and docs no doubt still need to be updated to > reflect > > it, etc. > > > > > > Options that seem worthy of debate: > > > > A) The DeprecationWarning that exists in py3k as of today. > > > > B) Remove the DeprecationWarning, simply document that people should be > > explicit about it if they care at all and that the default will change in > > 3.3. > > > > C) Never change close_fds behavior. we're stuck with it for life. > > > > D) Deprecate close_fds so that it becomes a legacy no-op. the new > pass_fds > > flag could evolve into this. this will break existing code that depends > on > > close_fds one way or another. > > > > > > I'm in 100% agreement that forcing people to pass close_fds in makes the > API > > less friendly (granted, people already find it unfriendly but why make it > > worse?). > > > > Option B seems the most friendly to me. > > +1 from me. People who don't care about the 'close_fds' behavior one > way or the other shouldn't get a warning which only helps the tiny (I > assert) minority who a) *do* care but b) don't pass the flag explicitly. > > Sleeping on the issue some more and pondering it... Is there any _good_ reason not to just make the close_fds default change in 3.2 today? No warning (since they're never seen by most people anyways). Document it in Misc/NEWS and whatsnew.rst. The most common use case is likely the simpler one where close_fds is not specified because the caller doesn't care, not because they are relying on the default being False. Another reason why I think changing it today is okay: This is Python 3.x. It has not yet seen wide adoption. It isn't likely to see wide adoption until 3.2. People migrating code to 3.2 are more likely to be migrating from 2.x than from 3.1. 2to3 can have a fixer added to specify close_fds=False on subprocess calls. Consider this option B-prime. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From tseaver at palladion.com Sun Dec 5 20:11:18 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sun, 05 Dec 2010 14:11:18 -0500 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: <19707.54440.612130.268676@montanaro.dyndns.org> References: <19707.54440.612130.268676@montanaro.dyndns.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/05/2010 01:06 PM, skip at pobox.com wrote: > > Vinay> ... the deprecation of ConfigParser for 3.2.... > > What's the rush? It's been deprecated, not removed. Having modules in the stdlib ship trigger (not raise) DeprecationWarnings is bad housekeeping / "broken window" stuff: it leads application developers to ignore them. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkz749YACgkQ+gerLs4ltQ6WyACcDrHkqRVYl11ctTJ5AswTK51Q evMAni7X2BdCbkyXzX7vLQCNMzoMj1yg =Ui6Y -----END PGP SIGNATURE----- From solipsis at pitrou.net Sun Dec 5 20:12:51 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 5 Dec 2010 20:12:51 +0100 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> Message-ID: <20101205201251.2339b854@pitrou.net> On Sun, 5 Dec 2010 11:08:43 -0800 "Gregory P. Smith" wrote: > Sleeping on the issue some more and pondering it... > > Is there any _good_ reason not to just make the close_fds default change in > 3.2 today? No warning (since they're never seen by most people anyways). > Document it in Misc/NEWS and whatsnew.rst. Yes, it will break many scripts under Windows. That's the core of the issue, really. I agree under Unix it makes sense to change the default. Regards Antoine. From greg at krypto.org Sun Dec 5 20:21:32 2010 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 5 Dec 2010 11:21:32 -0800 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: <20101205201251.2339b854@pitrou.net> References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> <20101205201251.2339b854@pitrou.net> Message-ID: On Sun, Dec 5, 2010 at 11:12 AM, Antoine Pitrou wrote: > On Sun, 5 Dec 2010 11:08:43 -0800 > "Gregory P. Smith" wrote: > > Sleeping on the issue some more and pondering it... > > > > Is there any _good_ reason not to just make the close_fds default change > in > > 3.2 today? No warning (since they're never seen by most people anyways). > > Document it in Misc/NEWS and whatsnew.rst. > > Yes, it will break many scripts under Windows. > That's the core of the issue, really. I agree under Unix it makes > sense to change the default. > Sorry, yes, I was ignoring windows in the above statement. I only want the change in default on unix, windows has different needs. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sun Dec 5 22:48:49 2010 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 05 Dec 2010 22:48:49 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems Message-ID: <4CFC08C1.3080609@v.loewis.de> I'd like to tighten PEP 11, and declare a policy that systems older than ten years at the point of a feature release are not supported anymore by default. Older systems where support is still maintained need to be explicitly listed in the PEP, along with the name of the responsible maintainer (I think this would currently only cover OS2/EMX which is maintained by Andrew MacIntyre). Support for such old platforms can then be removed from the codebase immediately, no need to go through a PEP 11 deprecation cycle. As a consequence, I would then like to remove support for Solaris versions older than Solaris 8 (released in January 2000, last updated by Sun in 2004) from the configure script for 3.2b2. A number of other tests in configure.in could probably also be removed, although I personally won't touch them before 3.2. The other major system affected by this would be Windows 2000, for which we already decided to not support it anymore. Opinions? Regards, Martin From solipsis at pitrou.net Sun Dec 5 23:14:30 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 5 Dec 2010 23:14:30 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems References: <4CFC08C1.3080609@v.loewis.de> Message-ID: <20101205231430.1e12ad59@pitrou.net> On Sun, 05 Dec 2010 22:48:49 +0100 "Martin v. L?wis" wrote: > I'd like to tighten PEP 11, and declare a policy that systems > older than ten years at the point of a feature release are not > supported anymore by default. Older systems where support is still > maintained need to be explicitly listed in the PEP, along with > the name of the responsible maintainer (I think this would currently > only cover OS2/EMX which is maintained by Andrew MacIntyre). > > Support for such old platforms can then be removed from the codebase > immediately, no need to go through a PEP 11 deprecation cycle. Seconded! > The other major system affected by this would be Windows 2000, for which > we already decided to not support it anymore. Is there any 2000-specific code (as opposed to XP-compatible)? Regards Antoine. From victor.stinner at haypocalc.com Sun Dec 5 23:25:27 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 5 Dec 2010 23:25:27 +0100 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <8762vax7lj.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20101202180651.F2AB0EE9CD@mail.python.org> <8762vax7lj.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <201012052325.27903.victor.stinner@haypocalc.com> On Saturday 04 December 2010 09:31:04 you wrote: > Alexander Belopolsky writes: > > In fact, once the language moratorium is over, I will argue that > > str.encode() and byte.decode() should deprecate encoding argument and > > just do UTF-8 encoding/decoding. Hopefully by that time most people > > will forget that other encodings exist. (I can dream, right?) > > It's just a dream. There's a pile of archival material, often on R/O > media, out there that won't be transcoded any more quickly than the > inscriptions on Tutankhamun's tomb. Not only, many libraries expect use bytes arguments encoded to a specific encoding (eg. locale encoding). Said differenlty, only few libraries written in C accept wchar* strings. The Linux kernel (or many, or all, UNIX/BSD kernels) only manipulate byte strings. The libc only accept wide characters for a few operations. I don't know how to open a file with an unicode path with the Linux libc: you have to encode it... Alexander: you should first patch all UNIX/BSD kernels to use unicode everywhere, then patch all libc implementations, and then all libraries (written in C). After that, you can have a break. Victor From greg.ewing at canterbury.ac.nz Sun Dec 5 23:29:27 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 06 Dec 2010 11:29:27 +1300 Subject: [Python-Dev] gc ideas -- sparse memory In-Reply-To: <4CF9FF4B.1030004@v.loewis.de> References: <4CF98117.5060301@v.loewis.de> <4CF9FF4B.1030004@v.loewis.de> Message-ID: <4CFC1247.4070703@canterbury.ac.nz> Martin v. L?wis wrote: > The hole C API would break if objects would move in memory. > Since they have to stay at fixed addresses, it's easy enough to use the > address as ID. Yes. Some of the discussion here seems to be assuming that the reason Python doesn't move objects is so that it can use the address as a unique ID. But it's the other way around. -- Greg From brett at python.org Mon Dec 6 00:26:05 2010 From: brett at python.org (Brett Cannon) Date: Sun, 5 Dec 2010 15:26:05 -0800 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <20101205231430.1e12ad59@pitrou.net> References: <4CFC08C1.3080609@v.loewis.de> <20101205231430.1e12ad59@pitrou.net> Message-ID: On Sun, Dec 5, 2010 at 14:14, Antoine Pitrou wrote: > On Sun, 05 Dec 2010 22:48:49 +0100 > "Martin v. L?wis" wrote: >> I'd like to tighten PEP 11, and declare a policy that systems >> older than ten years at the point of a feature release are not >> supported anymore by default. Older systems where support is still >> maintained need to be explicitly listed in the PEP, along with >> the name of the responsible maintainer (I think this would currently >> only cover OS2/EMX which is maintained by Andrew MacIntyre). >> >> Support for such old platforms can then be removed from the codebase >> immediately, no need to go through a PEP 11 deprecation cycle. > > Seconded! Thirded! -Brett > >> The other major system affected by this would be Windows 2000, for which >> we already decided to not support it anymore. > > Is there any 2000-specific code (as opposed to XP-compatible)? > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From martin at v.loewis.de Mon Dec 6 00:35:47 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 06 Dec 2010 00:35:47 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <20101205231430.1e12ad59@pitrou.net> References: <4CFC08C1.3080609@v.loewis.de> <20101205231430.1e12ad59@pitrou.net> Message-ID: <4CFC21D3.4060607@v.loewis.de> >> The other major system affected by this would be Windows 2000, for which >> we already decided to not support it anymore. > > Is there any 2000-specific code (as opposed to XP-compatible)? Yes: a number of APIs didn't exist in W2k, so we currently use LoadLibrary/GetProcAddress to call them. These can be replaced with regular calls. _WINNT (or whatever the macro is called) then needs to be bumped accordingly. Regards, Martin From tjreedy at udel.edu Mon Dec 6 00:45:02 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 05 Dec 2010 18:45:02 -0500 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFC08C1.3080609@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> Message-ID: On 12/5/2010 4:48 PM, "Martin v. L?wis" wrote: > I'd like to tighten PEP 11, and declare a policy that systems > older than ten years at the point of a feature release are not > supported anymore by default. Older systems where support is still > maintained need to be explicitly listed in the PEP, along with > the name of the responsible maintainer (I think this would currently > only cover OS2/EMX which is maintained by Andrew MacIntyre). > > Support for such old platforms can then be removed from the codebase > immediately, no need to go through a PEP 11 deprecation cycle. > > As a consequence, I would then like to remove support for Solaris > versions older than Solaris 8 (released in January 2000, last updated > by Sun in 2004) from the configure script for 3.2b2. A number of other > tests in configure.in could probably also be removed, although I > personally won't touch them before 3.2. > > The other major system affected by this would be Windows 2000, for which > we already decided to not support it anymore. WinXP (released August 2001) should be supported a lot longer than another year ;-) . It is still supported and installed on new systems. -- Terry Jan Reedy From v+python at g.nevcal.com Mon Dec 6 00:55:43 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 05 Dec 2010 15:55:43 -0800 Subject: [Python-Dev] r87010 - in python/branches/py3k: Doc/library/subprocess.rst Lib/subprocess.py Lib/test/test_subprocess.py In-Reply-To: <19707.54261.55334.594655@montanaro.dyndns.org> References: <20101204091044.C6CBEEE985@mail.python.org> <20101204124502.37ec1594@pitrou.net> <4CFADC47.7020702@g.nevcal.com> <19707.54261.55334.594655@montanaro.dyndns.org> Message-ID: <4CFC267F.7000105@g.nevcal.com> On 12/5/2010 10:03 AM, skip at pobox.com wrote: > Glenn> On 12/4/2010 3:07 PM, Paul Moore wrote: > >> The original goal was for subprocess to replace os.system, os.popen, > >> os.spawn, etc. That's never quite happened because subprocess is just > >> a little bit too conceptually complex for those basic tasks. > > Glenn> Is that way? I didn't find it particularly hard to learn, given > Glenn> the "cheat sheet" of techniques for doing the replacements. > > For 99% of my usage (I suspect for most other peoples' as well, at least on > Unix-y systems), this is all I need: > > for line in os.popen("some pipeline"): > do_stuff(line) > > No cheat sheet necessary. I don't see how subprocess could have made that > common idiom any simpler. Maybe it's better at doing esoteric stuff, > however that falls into the 1% where a simple os.popen isn't adequate. > > Skip So it couldn't make it any simpler. For your 99% usage, the question is, does it make it harder? And the answer is, at least a little bit... you have to type more... import subprocess for line in subprocess.Popen("cmd", shell=True, stdout=PIPE).stdout: do_stuff(line) Sad. It does seem like some wrapper functions could have been provided to make the easy cases at least as easy as they were before... My 99% case involves web servers and CGI-type interfaces. And subprocess doesn't provide quite enough to handle the job. It is a bit more complex than your pipeline case, and subprocess does allow the job to be done, and allows it to be done better than popen or popen2 could ever do it. But the helper functions I suggest in the issue would make it lots easier. And probably, it would be nice to include starting the threads within the helper functions, too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eliben at gmail.com Mon Dec 6 05:18:43 2010 From: eliben at gmail.com (Eli Bendersky) Date: Mon, 6 Dec 2010 06:18:43 +0200 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: References: <4CFC08C1.3080609@v.loewis.de> Message-ID: > The other major system affected by this would be Windows 2000, for which >> we already decided to not support it anymore. >> > > WinXP (released August 2001) should be supported a lot longer than another > year ;-) . It is still supported and installed on new systems. > Good catch. Windows XP, according to reports online has somewhere between 50-75% marketshare of all personal computers in the world. The Vista fiasco has created a large void which is only now starting to get filled with Windows 7, but it will yet take a long time to dethrone XP. Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Dec 6 05:25:30 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 6 Dec 2010 14:25:30 +1000 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <201012052325.27903.victor.stinner@haypocalc.com> References: <20101202180651.F2AB0EE9CD@mail.python.org> <8762vax7lj.fsf@uwakimon.sk.tsukuba.ac.jp> <201012052325.27903.victor.stinner@haypocalc.com> Message-ID: On Mon, Dec 6, 2010 at 8:25 AM, Victor Stinner wrote: > Not only, many libraries expect use bytes arguments encoded to a specific > encoding (eg. locale encoding). Said differenlty, only few libraries written in > C accept wchar* strings. > > The Linux kernel (or many, or all, UNIX/BSD kernels) only manipulate byte > strings. The libc only accept wide characters for a few operations. I don't > know how to open a file with an unicode path with the Linux libc: you have to > encode it... > > Alexander: you should first patch all UNIX/BSD kernels to use unicode > everywhere, then patch all libc implementations, and then all libraries > (written in C). After that, you can have a break. Slightly less ambitious is to get them all to agree to standardise on UTF-8 as the encoding mechanism (which is actually in the process of happening, it just has a long way to go). However, as a glue language, it is part of Python's role to help manage the transition from legacy encodings to UTF-8, so it will be a very long time before the idea of removing support for the encoding argument becomes even remotely feasible. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Dec 6 05:36:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 6 Dec 2010 14:36:13 +1000 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFC08C1.3080609@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> Message-ID: On Mon, Dec 6, 2010 at 7:48 AM, "Martin v. L?wis" wrote: > I'd like to tighten PEP 11, and declare a policy that systems > older than ten years at the point of a feature release are not > supported anymore by default. Older systems where support is still > maintained need to be explicitly listed in the PEP, along with > the name of the responsible maintainer (I think this would currently > only cover OS2/EMX which is maintained by Andrew MacIntyre). > > Support for such old platforms can then be removed from the codebase > immediately, no need to go through a PEP 11 deprecation cycle. > > As a consequence, I would then like to remove support for Solaris > versions older than Solaris 8 (released in January 2000, last updated > by Sun in 2004) from the configure script for 3.2b2. A number of other > tests in configure.in could probably also be removed, although I > personally won't touch them before 3.2. > > The other major system affected by this would be Windows 2000, for which > we already decided to not support it anymore. > > Opinions? I would prefer to be guided by vendor EOL dates rather than our own arbitrary 10 year limit. The EOL guide I would suggest is "Is the vendor still fixing bugs in that release?". For Solaris 8, the answer to that question is no (Phase 1 EOL support ended in October: http://www.sun.com/service/eosl/eosl_solaris.html) For Windows 2000, the answer is no (Extended Support ended in July: http://support.microsoft.com/lifecycle/?p1=7274) For Windows XP, the answer is yes (Extended Support doesn't end until April 2014: http://windows.microsoft.com/en-us/windows/products/lifecycle#section_2) Since the "Is the vendor still patching it?" guideline gives the "right" answer for the 3 systems mentioned in this thread, it will likely do a better job of covering anomalies like XP than a flat year limit would. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ocean-city at m2.ccsnet.ne.jp Mon Dec 6 06:04:46 2010 From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto) Date: Mon, 06 Dec 2010 14:04:46 +0900 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFC08C1.3080609@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> Message-ID: <4CFC6EEE.2080505@m2.ccsnet.ne.jp> On 2010/12/06 6:48, "Martin v. L?wis" wrote: > The other major system affected by this would be Windows 2000, for which > we already decided to not support it anymore. > > Opinions? I'm +1/2 for supporting Windows 2000... From martin at v.loewis.de Mon Dec 6 08:28:27 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 06 Dec 2010 08:28:27 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: References: <4CFC08C1.3080609@v.loewis.de> Message-ID: <4CFC909B.10208@v.loewis.de> Am 06.12.2010 05:36, schrieb Nick Coghlan: > On Mon, Dec 6, 2010 at 7:48 AM, "Martin v. L?wis" wrote: >> I'd like to tighten PEP 11, and declare a policy that systems >> older than ten years at the point of a feature release are not >> supported anymore by default. Older systems where support is still >> maintained need to be explicitly listed in the PEP, along with >> the name of the responsible maintainer (I think this would currently >> only cover OS2/EMX which is maintained by Andrew MacIntyre). >> >> Support for such old platforms can then be removed from the codebase >> immediately, no need to go through a PEP 11 deprecation cycle. >> >> As a consequence, I would then like to remove support for Solaris >> versions older than Solaris 8 (released in January 2000, last updated >> by Sun in 2004) from the configure script for 3.2b2. A number of other >> tests in configure.in could probably also be removed, although I >> personally won't touch them before 3.2. >> >> The other major system affected by this would be Windows 2000, for which >> we already decided to not support it anymore. >> >> Opinions? > > I would prefer to be guided by vendor EOL dates rather than our own > arbitrary 10 year limit. The EOL guide I would suggest is "Is the > vendor still fixing bugs in that release?". If available, perhaps. Not all system vendors have such a policy, or it may not be easy to find. If that is part of the policy, I'd like to add the specific systems where we use this approach along with the URLs that list the policies in the PEP. As a counter-example, I think the only way to phase out support for old OpenBSD releases is that we set a date. If this policy is applied to Debian also, we may get unhappy users, since Debian releases get out of love fairly quickly. OTOH, supporting old Linux releases really isn't that much of a problem. Regards, Martin From sungsuha at gmail.com Mon Dec 6 08:26:54 2010 From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha) Date: Mon, 6 Dec 2010 07:26:54 +0000 (UTC) Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems References: <4CFC08C1.3080609@v.loewis.de> Message-ID: Nick Coghlan gmail.com> writes: > On Mon, Dec 6, 2010 at 7:48 AM, "Martin v. L?wis" v.loewis.de> wrote: > > I'd like to tighten PEP 11 > > Opinions? > > I would prefer to be guided by vendor EOL dates rather than our own > arbitrary 10 year limit. The EOL guide I would suggest is "Is the > vendor still fixing bugs in that release?". > > Since the "Is the vendor still patching it?" guideline gives the > "right" answer for the 3 systems mentioned in this thread, it will > likely do a better job of covering anomalies like XP than a flat year > limit would. +1 I think Nick has a point. If the vendor is willing to patch/support said version, then I think it would be worthwhile for the Python community to provide support EOL dates of prominent Linux distribution : RHEL: https://access.redhat.com/support/policy/updates/errata/ Ubuntu: http://en.wikipedia.org/wiki/List_of_Ubuntu_releases#Version_timeline (http://www.ubuntu.com/products/ubuntu/release-cycle seems to be down) SuSe http://support.novell.com/lifecycle/ I do foresee a possible side-effect regarding Fedora as "maintained for approximately 13 months", because "Release X is supported until one month after the release of Release X+2." http://fedoraproject.org/wiki/LifeCycle Considering the nature of the Fedora project, dropping unsupported fedora distributions may or may not be helpful for Pyhton and it's users. Also, it is not clear what to do about distributions/OSs without any official EOL or life cycles. From ncoghlan at gmail.com Mon Dec 6 08:51:30 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 6 Dec 2010 17:51:30 +1000 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFC909B.10208@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFC909B.10208@v.loewis.de> Message-ID: On Mon, Dec 6, 2010 at 5:28 PM, "Martin v. L?wis" wrote: > Am 06.12.2010 05:36, schrieb Nick Coghlan: >> On Mon, Dec 6, 2010 at 7:48 AM, "Martin v. L?wis" wrote: >>> I'd like to tighten PEP 11, and declare a policy that systems >>> older than ten years at the point of a feature release are not >>> supported anymore by default. Older systems where support is still >>> maintained need to be explicitly listed in the PEP, along with >>> the name of the responsible maintainer (I think this would currently >>> only cover OS2/EMX which is maintained by Andrew MacIntyre). >>> >>> Support for such old platforms can then be removed from the codebase >>> immediately, no need to go through a PEP 11 deprecation cycle. >>> >>> As a consequence, I would then like to remove support for Solaris >>> versions older than Solaris 8 (released in January 2000, last updated >>> by Sun in 2004) from the configure script for 3.2b2. A number of other >>> tests in configure.in could probably also be removed, although I >>> personally won't touch them before 3.2. >>> >>> The other major system affected by this would be Windows 2000, for which >>> we already decided to not support it anymore. >>> >>> Opinions? >> >> I would prefer to be guided by vendor EOL dates rather than our own >> arbitrary 10 year limit. The EOL guide I would suggest is "Is the >> vendor still fixing bugs in that release?". > > If available, perhaps. Not all system vendors have such a policy, > or it may not be easy to find. If that is part of the policy, I'd > like to add the specific systems where we use this approach along > with the URLs that list the policies in the PEP. > > As a counter-example, I think the only way to phase out support > for old OpenBSD releases is that we set a date. I would be fine with an EOL based policy for single-vendor platforms (specifically Solaris and Windows) and a date-based policy for everything else. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From asmodai at in-nomine.org Mon Dec 6 09:36:09 2010 From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven) Date: Mon, 6 Dec 2010 09:36:09 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFC909B.10208@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFC909B.10208@v.loewis.de> Message-ID: <20101206083609.GE29719@nexus.in-nomine.org> -On [20101206 08:30], "Martin v. L?wis" (martin at v.loewis.de) wrote: >As a counter-example, I think the only way to phase out support >for old OpenBSD releases is that we set a date. If you want, I can provide you with specifics on the BSD platforms of what is currently in use, EOL and the future. Also, with regard to Windows 2000, XP and its APIs. Didn't XP already switch a bunch of APIs around that 2000 doesn't use? (Need to find that migration document.) -- Jeroen Ruigrok van der Werven / asmodai ????? ?????? ??? ?? ?????? http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B Nothing is more honorable than enlightenment, nothing is more beautiful than virtue... From martin at v.loewis.de Mon Dec 6 10:08:05 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 06 Dec 2010 10:08:05 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <20101206083609.GE29719@nexus.in-nomine.org> References: <4CFC08C1.3080609@v.loewis.de> <4CFC909B.10208@v.loewis.de> <20101206083609.GE29719@nexus.in-nomine.org> Message-ID: <4CFCA7F5.8080709@v.loewis.de> Am 06.12.2010 09:36, schrieb Jeroen Ruigrok van der Werven: > -On [20101206 08:30], "Martin v. L?wis" (martin at v.loewis.de) wrote: >> As a counter-example, I think the only way to phase out support >> for old OpenBSD releases is that we set a date. > > If you want, I can provide you with specifics on the BSD platforms of what > is currently in use, EOL and the future. If that's publicly documented from a single starting page, having the URL of that page would be appreciated. Also, do users accept that policy? (i.e. would they feel sad if Python releases don't support BSD releases anymore that don't get patches?) For Windows and Solaris, it seems that some users continue to use the system after the vendor stops producing patches, and dislike the prospect of not having Python releases for it anymore. However, they are in clear minority, so by our current policy for minority platforms (no need to support them), that's fine. This really triggered the "ten years" proposal: for quite some time now (20 years) people stop losing interest in operating systems after ten years. > Also, with regard to Windows 2000, XP and its APIs. Didn't XP already switch > a bunch of APIs around that 2000 doesn't use? (Need to find that migration > document.) I don't understand the question: what does it meant to switch an API around? XP has added new API that wasn't available in 2000, if that's what you are asking. Regards, Martin From martin at v.loewis.de Mon Dec 6 10:18:30 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 06 Dec 2010 10:18:30 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: References: <4CFC08C1.3080609@v.loewis.de> Message-ID: <4CFCAA66.6060901@v.loewis.de> > EOL dates of prominent Linux distribution : I think I would need more information than that. Nick's proposal was more specific: when does the vendor stop producing patches? This is a clear criterion, and one that I support. > RHEL: > https://access.redhat.com/support/policy/updates/errata/ My interpretation: Python support until end of production phase 3 (7 years). > Ubuntu: > http://en.wikipedia.org/wiki/List_of_Ubuntu_releases#Version_timeline > (http://www.ubuntu.com/products/ubuntu/release-cycle seems to be down) I'd prefer something more official than Wikipedia, though. > SuSe > http://support.novell.com/lifecycle/ My interpretation: Python support until end of Extended support phase (7 years). So by this policy, RHEL and SuSE users would be off worse than with my original proposal (10 years). > Considering the nature of the Fedora project, dropping unsupported fedora > distributions may or may not be helpful for Pyhton and it's users. Again, for Linux, I think the issue is somewhat less critical: in terms of portability and ABI stability, it seems like they manage best (i.e. we have least version-dependent code for Linux in Python, probably because a "Linux version" doesn't exist in the first place, so distributions must provide source and binary compatibility even across vendors, making such support across versions more easy). > Also, it is not clear what to do about distributions/OSs > without any official EOL or life cycles. Here my proposal stands: 10 years, by default. Regards, Martin From ziade.tarek at gmail.com Mon Dec 6 10:42:50 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Mon, 6 Dec 2010 10:42:50 +0100 Subject: [Python-Dev] PEP 384 accepted In-Reply-To: <4CFA99DF.9040907@v.loewis.de> References: <4CF81208.8090705@v.loewis.de> <4CF81AE1.5050900@v.loewis.de> <4CF880F7.9000800@netwok.org> <4CF89CCF.20307@v.loewis.de> <4CF8E22B.7050005@netwok.org> <4CF97197.2050105@v.loewis.de> <4CF973D0.4060800@netwok.org> <4CF974B9.6040509@v.loewis.de> <20101204164232.GH3015@unaka.lan> <4CFA99DF.9040907@v.loewis.de> Message-ID: On Sat, Dec 4, 2010 at 8:43 PM, "Martin v. L?wis" wrote: ... > Now, distutils2 is different: it's *not* new functionality. So perhaps > yes: I disagree with the principle that bold rewrites should be > developed independently. Such work would be much better carried out in a > branch - it will never stand on its own. So we had several +1 on James' idea to bring back distutils2 to the trunk after 3.2 final, and it seems that you are not against this idea. So we will do this unless you object Regards Tarek -- Tarek Ziad? | http://ziade.org From sungsuha at gmail.com Mon Dec 6 12:04:19 2010 From: sungsuha at gmail.com (Seung =?utf-8?b?U29vLA==?= Ha) Date: Mon, 6 Dec 2010 11:04:19 +0000 (UTC) Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems References: <4CFC08C1.3080609@v.loewis.de> <4CFC909B.10208@v.loewis.de> Message-ID: Nick Coghlan gmail.com> writes: > I would be fine with an EOL based policy for single-vendor platforms > (specifically Solaris and Windows) and a date-based policy for > everything else. +1 I also think this would be for the best. From andrew at bemusement.org Mon Dec 6 12:10:12 2010 From: andrew at bemusement.org (Andrew Bennetts) Date: Mon, 6 Dec 2010 22:10:12 +1100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFCAA66.6060901@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFCAA66.6060901@v.loewis.de> Message-ID: <20101206111012.GA22072@aihal.home.puzzling.org> "Martin v. L?wis" wrote: [...] > > Ubuntu: > > http://en.wikipedia.org/wiki/List_of_Ubuntu_releases#Version_timeline > > (http://www.ubuntu.com/products/ubuntu/release-cycle seems to be down) > > I'd prefer something more official than Wikipedia, though. https://wiki.ubuntu.com/Releases -Andrew. From mal at egenix.com Mon Dec 6 12:39:43 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 06 Dec 2010 12:39:43 +0100 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> Message-ID: <4CFCCB7F.9080605@egenix.com> Guido van Rossum wrote: > On Fri, Dec 3, 2010 at 9:58 AM, R. David Murray wrote: >> On Fri, 03 Dec 2010 11:14:56 -0500, Alexander Belopolsky wrote: >>> On Fri, Dec 3, 2010 at 10:11 AM, R. David Murray wrote: >>> .. >>>> Please also recall that transform/untransform was discussed before >>>> the release of Python 3.0 and was approved at the time, but it just >>>> did not get implemented before the 3.0 release. >>>> >>> >>> Can you provide a link? My search for transform on python-dev came out with >> >> It was linked from the issue, if I recall correctly. I do remember >> reading the thread from the python-3000 list, linked by someone >> somewhere :) >> >>> http://mail.python.org/pipermail/python-dev/2010-June/100564.html >>> >>> where you seem to oppose these methods. Also, new methods to builtins >> >> It looks to me like I was agreeing that transform/untrasnform should >> do only bytes->bytes or str->str regardless of what codec name you >> passed them. >> >>> fall under the language moratorium (but can be approved on a >>> case-by-case basis): >>> >>> http://www.python.org/dev/peps/pep-3003/#case-by-case-exemptions >>> >>> Is there an effort to document these exceptions? I expected such >>> approvals to be added to PEP 3003, but apparently this was not the >>> case. >> >> I believe MAL's thought was that the addition of these methods had >> been approved pre-moratorium, Indeed. >> but I don't know if that is a >> sufficient argument or not. > > It is not. > > The moratorium is intended to freeze the state of the language as > implemented, not whatever was discussed and approved but didn't get > implemented (that'd be a hole big enough to drive a truck through, as > the saying goes :-). Sure, but those two particular methods only provide interfaces to the codecs sub-system without actually requiring any major implementation changes. Furthermore, they "help ease adoption of Python 3.x" (quoted from PEP 3003), since the functionality they add back was removed from Python 3.0 in a way that makes it difficult to port Python2 applications to Python3. > Regardless of what I or others may have said before, I am not > currently a fan of adding transform() to either str or bytes. How should I read this ? Do want the methods to be removed again and added back in 3.3 ? Frankly, I'm a bit tired of constantly having to argue against cutting down the Unicode and codec support in Python3. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 06 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From ulrich.eckhardt at dominolaser.com Mon Dec 6 13:18:55 2010 From: ulrich.eckhardt at dominolaser.com (Ulrich Eckhardt) Date: Mon, 6 Dec 2010 13:18:55 +0100 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: <1291470115.3534.8.camel@localhost.localdomain> References: <1291470115.3534.8.camel@localhost.localdomain> Message-ID: <201012061318.55922.ulrich.eckhardt@dominolaser.com> On Saturday 04 December 2010, Antoine Pitrou wrote: > Perhaps SVN doesn't get timestamps right. SVN doesn't touch timestamps explicitly, it just writes the files as they come and the system gives them the timestamps. This also makes sense, because the build system depends on them - you don't want the build to skip compiling files after rewinding your working copy to an older version. That said, you can tell SVN to do something with timestamps, I'd have to search a bit in order to find out what exactly that is. Uli ************************************************************************************** Domino Laser GmbH, Fangdieckstra?e 75a, 22547 Hamburg, Deutschland Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932 ************************************************************************************** Visit our website at ************************************************************************************** Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden. E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Domino Laser GmbH ist f?r diese Folgen nicht verantwortlich. ************************************************************************************** From tseaver at palladion.com Mon Dec 6 13:47:30 2010 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 06 Dec 2010 07:47:30 -0500 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: <201012061318.55922.ulrich.eckhardt@dominolaser.com> References: <1291470115.3534.8.camel@localhost.localdomain> <201012061318.55922.ulrich.eckhardt@dominolaser.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/06/2010 07:18 AM, Ulrich Eckhardt wrote: > On Saturday 04 December 2010, Antoine Pitrou wrote: >> Perhaps SVN doesn't get timestamps right. > > SVN doesn't touch timestamps explicitly, it just writes the files as they come > and the system gives them the timestamps. This also makes sense, because the > build system depends on them - you don't want the build to skip compiling > files after rewinding your working copy to an older version. > > That said, you can tell SVN to do something with timestamps, I'd have to > search a bit in order to find out what exactly that is. If you set the 'use-commit-times' config flag, subversion will keep timestamps in sync with the repository commit times during 'checkout', 'update', 'switch', etc. operations (it does this by default during 'export', which is likekt why the tarball process works): http://svnbook.red-bean.com/en/1.1/ch07.html Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkz822EACgkQ+gerLs4ltQ5tdwCfV/iScJcoxymb/REIBm6VFcbK TUIAoNl5LLaFgifdV8BjiuOIw5YmnaqR =H7+7 -----END PGP SIGNATURE----- From flub at devork.be Mon Dec 6 14:40:08 2010 From: flub at devork.be (Floris Bruynooghe) Date: Mon, 6 Dec 2010 13:40:08 +0000 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFCAA66.6060901@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFCAA66.6060901@v.loewis.de> Message-ID: On 6 December 2010 09:18, "Martin v. L?wis" wrote: >> Also, it is not clear what to do about distributions/OSs >> without any official EOL or life cycles. > > Here my proposal stands: 10 years, by default. How about max(EOL, 10years)? That sounds like it could be a useful guideline. (Personally I'd be sad to see Solaris 8 go in the next few years) Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From martin at v.loewis.de Mon Dec 6 20:03:26 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 06 Dec 2010 20:03:26 +0100 Subject: [Python-Dev] python 2 for building python 3 In-Reply-To: <201012061318.55922.ulrich.eckhardt@dominolaser.com> References: <1291470115.3534.8.camel@localhost.localdomain> <201012061318.55922.ulrich.eckhardt@dominolaser.com> Message-ID: <4CFD337E.9050300@v.loewis.de> Am 06.12.2010 13:18, schrieb Ulrich Eckhardt: > On Saturday 04 December 2010, Antoine Pitrou wrote: >> Perhaps SVN doesn't get timestamps right. > > SVN doesn't touch timestamps explicitly, it just writes the files as they come > and the system gives them the timestamps. This also makes sense, because the > build system depends on them - you don't want the build to skip compiling > files after rewinding your working copy to an older version. > > That said, you can tell SVN to do something with timestamps, I'd have to > search a bit in order to find out what exactly that is. What you can ask it to is to use commit times. As you say, this is risky because it may cause build steps to be skipped. What I'm asking for is that the version control gives the files new time stamps, but still imposes a timestamp order, at least on modern file systems (and, as an option, also on old ones). Suppose you do an update spanning 20 revisions. The update should: 1. get the current time T at the beginning of the update 2. fetch all files, and update the contents 3. Touch all files, in the order of the revisions, using microsecond steps (i.e. the oldest file gets T, the next revision T+1?s, the next one T+2?s). As the update probably takes longer than 20?s, all files will have timestamps in the past when the update is done. If the file system does not support subsecond timestamps, you can do the same in second resolution, but that might give you files dated in the future (which make complains about - but you'ld get over with that after 20s). ms might offer some middle ground (but I think all filesystems supporting ms resolution will also support ?s). Regards, Martin From martin at v.loewis.de Mon Dec 6 19:55:15 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 06 Dec 2010 19:55:15 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: References: <4CFC08C1.3080609@v.loewis.de> <4CFCAA66.6060901@v.loewis.de> Message-ID: <4CFD3193.2030906@v.loewis.de> Am 06.12.2010 14:40, schrieb Floris Bruynooghe: > On 6 December 2010 09:18, "Martin v. L?wis" wrote: >>> Also, it is not clear what to do about distributions/OSs >>> without any official EOL or life cycles. >> >> Here my proposal stands: 10 years, by default. > > How about max(EOL, 10years)? That sounds like it could be a useful guideline. > > (Personally I'd be sad to see Solaris 8 go in the next few years) I guess we'll be sorry, then: under that policy, max(EOL, 10years) comes out as "not supported" (not sure whether you were aware of that). Of course, with these old systems, I really wonder: why do they need current Python releases? 2.7 will remain available and maintained for some time, and 3.1 will at least see security fixes for some more time - something that the base system itself doesn't receive anymore. So if you needed a Python release for Solaris 8, you could just use Python 2.3, no? We are not going to take the sources of old releases offline. Regards, Martin From raymond.hettinger at gmail.com Mon Dec 6 20:06:56 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 6 Dec 2010 11:06:56 -0800 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> Message-ID: <76C191DD-614F-4E45-A294-E6527A9935DF@gmail.com> On Dec 3, 2010, at 10:05 AM, Guido van Rossum wrote: > Regardless of what I or others may have said before, I am not > currently a fan of adding transform() to either str or bytes. Just to clarify, is your preference to go back to the Python 2.x way and use encode()/decode() for both unicode encodings and other other transformations (uu, zlib, bz2, quopri, etc)? FWIW, transform()/untransform() are already in the 3.2 beta. Do you want them taken out? If so, do you want the codecs made accessible with encode/decode or have them continue to be inaccessible in 3.x? Raymond From raymond.hettinger at gmail.com Mon Dec 6 20:21:28 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 6 Dec 2010 11:21:28 -0800 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: References: Message-ID: <5E183509-BBED-426B-BC70-DDA5640AD302@gmail.com> On Dec 5, 2010, at 3:36 AM, Vinay Sajip wrote: > I've just been notified via being added to the nosy list of > > http://bugs.python.org/issue10627 > > about the deprecation of ConfigParser for 3.2. I presume I was added to this > list because logging.config uses ConfigParser, but logging.config doesn't use > any interpolation features so I could easily change all usages to > SafeConfigParser; should I do it now (before the 3.2 beta is cut) or hold off > for now? +1 for doing the update now. There's no advantage in continuing to use the old ConfigParser. With respect to beta feature freeze, that should really just mean not adding prominent new major features. We're still free to implement bug fixes, make the language more consistent, propagate the existing changes (such as SafeConfigParser), tweak any of the new APIs, and reconsider or rework some of the new features (such as transform/untransform). Raymond P.S. We really ought to stop with the SafeFoo naming convention. It is only descriptive to the person who wrote the class or function, not to the user who will immediately wonder, "safe from what?" Safe can mean "sanitized", "won't raise an error", "has a default value", "doesn't use eval/exec", "is secure", "has a known encoding", etc. From tjreedy at udel.edu Mon Dec 6 20:25:56 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 06 Dec 2010 14:25:56 -0500 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFCA7F5.8080709@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFC909B.10208@v.loewis.de> <20101206083609.GE29719@nexus.in-nomine.org> <4CFCA7F5.8080709@v.loewis.de> Message-ID: On 12/6/2010 4:08 AM, "Martin v. L?wis" wrote: > For Windows and Solaris, it seems that some users continue to use the > system after the vendor stops producing patches, and dislike the > prospect of not having Python releases for it anymore. However, they > are in clear minority, so by our current policy for minority platforms I quite suspect that XP will be in major use (more than say, current BSD) for some years after MS stops official support. Why rush to drop it? Is there much XP specific stuff in the current xp/vista/7 code? It seems to me that the rule should be something like "around 10 years or end of support, as modified by popularity, the burden of continued support, the availability of test machines, and the availability of people". -- Terry Jan Reedy From martin at v.loewis.de Mon Dec 6 20:26:12 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 06 Dec 2010 20:26:12 +0100 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <76C191DD-614F-4E45-A294-E6527A9935DF@gmail.com> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <76C191DD-614F-4E45-A294-E6527A9935DF@gmail.com> Message-ID: <4CFD38D4.10507@v.loewis.de> > FWIW, transform()/untransform() are already in the 3.2 beta. > Do you want them taken out? If so, do you want the codecs > made accessible with encode/decode or have them continue > to be inaccessible in 3.x? Notice that they wouldn't be inaccessible even if transform is taken out again: codecs.lookup would still find them. Regards, Martin From me+python at ixokai.io Mon Dec 6 20:34:41 2010 From: me+python at ixokai.io (Stephen Hansen) Date: Mon, 06 Dec 2010 11:34:41 -0800 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFD3193.2030906@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFCAA66.6060901@v.loewis.de> <4CFD3193.2030906@v.loewis.de> Message-ID: <4CFD3AD1.70801@ixokai.io> On 12/6/10 10:55 AM, "Martin v. L?wis" wrote: > Of course, with these old systems, I really wonder: why do they need > current Python releases? 2.7 will remain available and maintained for > some time, and 3.1 will at least see security fixes for some more time - > something that the base system itself doesn't receive anymore. So > if you needed a Python release for Solaris 8, you could just use Python > 2.3, no? We are not going to take the sources of old releases offline. For things like Solaris 8, I have no thoughts one way or the other-- but considering Windows XP is hitting 10 years next year-- Personally? My entirely theoretical timetable for when I think I'll be able to finally upgrade to Python 3 (where I'll skip to whatever the latest Python 3 is), is actually shorter by at least a few years then my timetable of when I think I'll be able to drop support of Windows XP. Unfortunately. WinXP is old but *pervasive* still in my experience with small businesses / customers. Many aren't even considering making a plan for Win7 yet. So if two years rolls around and Python 3.x (where 'x' is 'whatever is current') isn't supported on Windows XP, I'll be very sad, and will have to be stuck on Python 3.x-1 for .. awhile, where "awhile" is out of my control and up to the Masses who are unable or can't be bothered with fixing what works for them w/ WinXP. -- Stephen Hansen ... Also: Ixokai ... Mail: me+python (AT) ixokai (DOT) io ... Blog: http://meh.ixokai.io/ -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 487 bytes Desc: OpenPGP digital signature URL: From fdrake at acm.org Mon Dec 6 20:40:25 2010 From: fdrake at acm.org (Fred Drake) Date: Mon, 6 Dec 2010 14:40:25 -0500 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: <5E183509-BBED-426B-BC70-DDA5640AD302@gmail.com> References: <5E183509-BBED-426B-BC70-DDA5640AD302@gmail.com> Message-ID: On Mon, Dec 6, 2010 at 2:21 PM, Raymond Hettinger wrote: > We really ought to stop with the SafeFoo naming convention. > It is only descriptive to the person who wrote the class or function, > not to the user who will immediately wonder, "safe from what?" Safe from bad vampire movies, of course! I'd not recognize the current Safe* class names as a pattern; there are currently two in the py3k trunk: configparser.SafeConfigParser -- very much a poor name xmlrpc.client.SafeTransport -- perhaps should have been SSLTransport or HTTPSTransport I agree the "Safe" prefix isn't meaningful. SafeConfigParser might even be my fault. Michael Foord has lobbied to end up with the "preferred" configparser class to be named (eventually) ConfigParser, with good reason. It's not clear to me that we want to do that for backward compatibility reasons (as I've argued elsewhere). Were it not for that issue, I'd be in favor of using different/better names. (And there's still space for better names, if we can create new names that avoid the b/w compatibility issues.) ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From martin at v.loewis.de Mon Dec 6 21:46:15 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 06 Dec 2010 21:46:15 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: References: <4CFC08C1.3080609@v.loewis.de> <4CFC909B.10208@v.loewis.de> <20101206083609.GE29719@nexus.in-nomine.org> <4CFCA7F5.8080709@v.loewis.de> Message-ID: <4CFD4B97.2090409@v.loewis.de> Am 06.12.2010 20:25, schrieb Terry Reedy: > On 12/6/2010 4:08 AM, "Martin v. L?wis" wrote: > >> For Windows and Solaris, it seems that some users continue to use the >> system after the vendor stops producing patches, and dislike the >> prospect of not having Python releases for it anymore. However, they >> are in clear minority, so by our current policy for minority platforms > > I quite suspect that XP will be in major use (more than say, current > BSD) for some years after MS stops official support. Why rush to drop > it? What rush to drop it, specifically? It will be supported as long as Microsoft provides patches, as per Nick's amendment. For Windows XP, the extended lifecycle support (end of security patches) will be on April 8, 2014. I wouldn't call that "rushing", at this point in time. By our current release pace, Python 3.5 might be the first release to not support XP anymore. > Is there much XP specific stuff in the current xp/vista/7 code? I don't know; I haven't investigated unsupporting XP yet. I'm concerned about Windows 2000, at the moment. > It seems to me that the rule should be something like "around 10 years > or end of support, as modified by popularity, the burden of continued > support, the availability of test machines, and the availability of > people". In my original posting, I proposed a clause where support could be extended as long as an individual steps forward to provide that support. So if XP remains popular by the time Microsoft stops providing patches for it, some volunteer would have to step forward. Regards, Martin From raymond.hettinger at gmail.com Mon Dec 6 22:02:41 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 6 Dec 2010 13:02:41 -0800 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: References: <5E183509-BBED-426B-BC70-DDA5640AD302@gmail.com> Message-ID: <19D1A664-2C13-4D9F-9A45-B4DEBB4E77E1@gmail.com> On Dec 6, 2010, at 11:40 AM, Fred Drake wrote: > On Mon, Dec 6, 2010 at 2:21 PM, Raymond Hettinger > wrote: >> We really ought to stop with the SafeFoo naming convention. >> It is only descriptive to the person who wrote the class or function, >> not to the user who will immediately wonder, "safe from what?" > > Safe from bad vampire movies, of course! > > I'd not recognize the current Safe* class names as a pattern; there > are currently two in the py3k trunk: > > configparser.SafeConfigParser > -- very much a poor name > > xmlrpc.client.SafeTransport > -- perhaps should have been SSLTransport or HTTPSTransport > > I agree the "Safe" prefix isn't meaningful. IIRC, pprint has a safe_repr() and string.Template has safe_substitute() and pydoc has a safe import. Never new there was so much danger in the standard library :-) Raymond From amauryfa at gmail.com Mon Dec 6 22:02:48 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Mon, 6 Dec 2010 22:02:48 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFD4B97.2090409@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFC909B.10208@v.loewis.de> <20101206083609.GE29719@nexus.in-nomine.org> <4CFCA7F5.8080709@v.loewis.de> <4CFD4B97.2090409@v.loewis.de> Message-ID: Hi, 2010/12/6 "Martin v. L?wis" > In my original posting, I proposed a clause where support could be > extended as long as an individual steps forward to provide that support. > So if XP remains popular by the time Microsoft stops providing patches > for it, some volunteer would have to step forward. > +1. Such a clause is already used to keep supporting the old VC6.0 compiler (a.k.a VC98!) -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From fdrake at acm.org Mon Dec 6 22:09:17 2010 From: fdrake at acm.org (Fred Drake) Date: Mon, 6 Dec 2010 16:09:17 -0500 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: <19D1A664-2C13-4D9F-9A45-B4DEBB4E77E1@gmail.com> References: <5E183509-BBED-426B-BC70-DDA5640AD302@gmail.com> <19D1A664-2C13-4D9F-9A45-B4DEBB4E77E1@gmail.com> Message-ID: On Mon, Dec 6, 2010 at 4:02 PM, Raymond Hettinger wrote: > IIRC, pprint has a safe_repr() and string.Template has safe_substitute() > and pydoc has a safe import. pprint.saferepr Ok, this one's my fault as well. Probably should just be named repr. string.Template.safe_substitute Agree on this as well. pydoc.safeimport Not documented, so I don't really care what it's called. > Never new there was so much danger in the standard library :-) You should have known this, Raymond! It's software. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From tjreedy at udel.edu Mon Dec 6 23:15:19 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 06 Dec 2010 17:15:19 -0500 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFD4B97.2090409@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFC909B.10208@v.loewis.de> <20101206083609.GE29719@nexus.in-nomine.org> <4CFCA7F5.8080709@v.loewis.de> <4CFD4B97.2090409@v.loewis.de> Message-ID: On 12/6/2010 3:46 PM, "Martin v. L?wis" wrote: > Am 06.12.2010 20:25, schrieb Terry Reedy: >> I quite suspect that XP will be in major use (more than say, current >> BSD) for some years after MS stops official support. Why rush to drop >> it? > > What rush to drop it, On the day MS stops support. But it is a moot support until them. -- Terry Jan Reedy From georg at python.org Mon Dec 6 22:46:48 2010 From: georg at python.org (Georg Brandl) Date: Mon, 06 Dec 2010 22:46:48 +0100 Subject: [Python-Dev] [RELEASED] Python 3.2 beta 1 Message-ID: <4CFD59C8.4070006@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm happy to announce the first of two beta preview releases of Python 3.2. Python 3.2 is a continuation of the efforts to improve and stabilize the Python 3.x line. Since the final release of Python 2.7, the 2.x line will only receive bugfixes, and new features are developed for 3.x only. Since PEP 3003, the Moratorium on Language Changes, is in effect, there are no changes in Python's syntax and built-in types in Python 3.2. Development efforts concentrated on the standard library and support for porting code to Python 3. Highlights are: * numerous improvements to the unittest module * PEP 3147, support for .pyc repository directories * PEP 3149, support for version tagged dynamic libraries * PEP 3148, a new futures library for concurrent programming * PEP 384, a stable ABI for extension modules * PEP 391, dictionary-based logging configuration * an overhauled GIL implementation that reduces contention * an extended email package that handles bytes messages * countless fixes regarding bytes/string issues; among them full support for a bytes environment (filenames, environment variables) * many consistency and behavior fixes for numeric operations * a sysconfig module to access configuration information * a pure-Python implementation of the datetime module * additions to the shutil module, among them archive file support * improvements to pdb, the Python debugger For a more extensive list of changes in 3.2, see http://docs.python.org/3.2/whatsnew/3.2.html To download Python 3.2 visit: http://www.python.org/download/releases/3.2/ Please consider trying Python 3.2 with your code and reporting any bugs you may notice to: http://bugs.python.org/ Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.2's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.16 (GNU/Linux) iEYEARECAAYFAkz9WcgACgkQN9GcIYhpnLBRYwCeMmH1GMmKOx9fVk8a/F0/TOzj Vp0AoIHYBNcxV/U0AXIwMGWFHi1bAB+a =KBam -----END PGP SIGNATURE----- From solipsis at pitrou.net Mon Dec 6 23:24:57 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 06 Dec 2010 23:24:57 +0100 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.2 beta 1 In-Reply-To: <4CFD59C8.4070006@python.org> References: <4CFD59C8.4070006@python.org> Message-ID: <1291674297.3579.5.camel@localhost.localdomain> > Development efforts concentrated on the standard library and support for > porting code to Python 3. Highlights are: > > [snip] * ...a much improved ssl module Regards Antoine. From dmalcolm at redhat.com Mon Dec 6 23:55:03 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Mon, 06 Dec 2010 17:55:03 -0500 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFCAA66.6060901@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFCAA66.6060901@v.loewis.de> Message-ID: <1291676103.15083.28.camel@radiator.bos.redhat.com> On Mon, 2010-12-06 at 10:18 +0100, "Martin v. L?wis" wrote: > > EOL dates of prominent Linux distribution : > > I think I would need more information than that. Nick's proposal was > more specific: when does the vendor stop producing patches? This is > a clear criterion, and one that I support. > > > RHEL: > > https://access.redhat.com/support/policy/updates/errata/ > > My interpretation: Python support until end of production phase 3 (7 years). (...) > So by this policy, RHEL and SuSE users would be off worse than with > my original proposal (10 years). Red Hat continues to provide patches for RHEL within the "Extended Life Cycle" (years 8, 9 and 10), but it's an optional add-on. So another interpretation of the above with Nick's proposal could be 10 years on RHEL. (though obviously I'm biased in favor of RHEL) Approaching this from another angle: please do add me to the "nosy" on any compatibility bugs with running latest python code on RHEL. I'm also looking into getting RHEL buildbot machines, FWIW. > > Considering the nature of the Fedora project, dropping unsupported fedora > > distributions may or may not be helpful for Pyhton and it's users. > > Again, for Linux, I think the issue is somewhat less critical: in terms > of portability and ABI stability, it seems like they manage best (i.e. > we have least version-dependent code for Linux in Python, probably > because a "Linux version" doesn't exist in the first place, so > distributions must provide source and binary compatibility even > across vendors, making such support across versions more easy). The other compat issues are in the toolchain: e.g. very recent versions of gcc . In downstream Fedora, we tend to be amongst the first to run into new compilation warnings (and, occasionally, "exciting" code-generation bugs...) But this tends to be the opposite kind of problem: beginning of life, rather than end-of-life, and these sorts of things will need fixing for every Linux build eventually. FWIW, I'm trying to keep Fedora's "system" python 2 and python 3 builds as up-to-date as reasonable, so Fedora users will (I hope) be running fairly recent code python as is. We have 2.7 as /usr/bin/python as of F14, for instance. Hope this is helpful Dave From martin at v.loewis.de Tue Dec 7 00:05:15 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 07 Dec 2010 00:05:15 +0100 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <1291676103.15083.28.camel@radiator.bos.redhat.com> References: <4CFC08C1.3080609@v.loewis.de> <4CFCAA66.6060901@v.loewis.de> <1291676103.15083.28.camel@radiator.bos.redhat.com> Message-ID: <4CFD6C2B.8030708@v.loewis.de> >> So by this policy, RHEL and SuSE users would be off worse than with >> my original proposal (10 years). > > Red Hat continues to provide patches for RHEL within the "Extended Life > Cycle" (years 8, 9 and 10), but it's an optional add-on. My understanding is that you keep the patches available - but you don't produce any new ones, right? > So another interpretation of the above with Nick's proposal could be 10 > years on RHEL. (though obviously I'm biased in favor of RHEL) I wouldn't count mere availability of old patches on the server as "support". > Approaching this from another angle: please do add me to the "nosy" on > any compatibility bugs with running latest python code on RHEL. I'll try to remember, but really can't promise. But then, as I said before: I think Linux support in Python is particularly easy. For example, there isn't a single distribution-specific test in configure.in. > The other compat issues are in the toolchain: e.g. very recent versions > of gcc . In downstream Fedora, we tend to be amongst the first to run > into new compilation warnings (and, occasionally, "exciting" > code-generation bugs...) Dropping support for old gcc versions (or other old compiler versions) is probably an issue on its own. It will be difficult to figure out what work-arounds are in place for what particular compiler glitch. Regards, Martin From fuzzyman at voidspace.org.uk Tue Dec 7 00:45:33 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 06 Dec 2010 23:45:33 +0000 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: <19D1A664-2C13-4D9F-9A45-B4DEBB4E77E1@gmail.com> References: <5E183509-BBED-426B-BC70-DDA5640AD302@gmail.com> <19D1A664-2C13-4D9F-9A45-B4DEBB4E77E1@gmail.com> Message-ID: <4CFD759D.1090700@voidspace.org.uk> On 06/12/2010 21:02, Raymond Hettinger wrote: > On Dec 6, 2010, at 11:40 AM, Fred Drake wrote: > >> On Mon, Dec 6, 2010 at 2:21 PM, Raymond Hettinger >> wrote: >>> We really ought to stop with the SafeFoo naming convention. >>> It is only descriptive to the person who wrote the class or function, >>> not to the user who will immediately wonder, "safe from what?" >> Safe from bad vampire movies, of course! >> >> I'd not recognize the current Safe* class names as a pattern; there >> are currently two in the py3k trunk: >> >> configparser.SafeConfigParser >> -- very much a poor name >> >> xmlrpc.client.SafeTransport >> -- perhaps should have been SSLTransport or HTTPSTransport >> >> I agree the "Safe" prefix isn't meaningful. > IIRC, pprint has a safe_repr() and string.Template has safe_substitute() > and pydoc has a safe import. > > Never new there was so much danger in the standard library :-) What would you name those functions instead? (SafeConfigParser is a naff name and only needed because ConfigParser is broken.) Michael > > Raymond > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From db3l.net at gmail.com Tue Dec 7 01:24:39 2010 From: db3l.net at gmail.com (David Bolen) Date: Mon, 06 Dec 2010 19:24:39 -0500 Subject: [Python-Dev] kill_python on windows buildbots (was Re: Stable buildbots) References: <20101113133712.60e9be27@pitrou.net> <4CEB7E12.1070201@snakebite.org> Message-ID: I previously wrote: > I suspect the problem may be on the "identify which process to kill" > rather than the "kill it" part, but it's definitely going to take time > to figure that out for sure. While the approach kill_python takes is > much more appropriate, since we don't currently have multiple builds > running simultaneously (and for me the machines are dedicated as build > slaves, so I won't be having my own python_d), a more blanket kill > operation is safe enough. For anyone interested, I caught (well, Georg Brandl caught it first) a case on Saturday with some hung processes on the Win7 buildbot that I was able to verify kill_python failed to kill. This was after having a few instances where it did work fine. I've created issue 10641 to track. I also noticed another recent issue (10136) that is also related to kill_python missing processes, but given that it works in my case some of the time, and is always called via the same path, not sure if that can be my problem. I also realized that some fraction of the other cases I have seen might not have truly been an issue, since from what I can see kill_python is only run at the start of a build process, so hung processes (even killable ones) from a prior build hang around until the next build takes place. They can however, interfere with the svn checkout so things never get to the point of using kill_python. So maybe kill_python's use should be moved to the clean stage? -- David From rdmurray at bitdance.com Tue Dec 7 03:30:08 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 06 Dec 2010 21:30:08 -0500 Subject: [Python-Dev] Repo frozen for 3.2 In-Reply-To: <19D1A664-2C13-4D9F-9A45-B4DEBB4E77E1@gmail.com> References: <5E183509-BBED-426B-BC70-DDA5640AD302@gmail.com> <19D1A664-2C13-4D9F-9A45-B4DEBB4E77E1@gmail.com> Message-ID: <20101207023008.12D21207603@kimball.webabinitio.net> On Mon, 06 Dec 2010 13:02:41 -0800, Raymond Hettinger wrote: > On Dec 6, 2010, at 11:40 AM, Fred Drake wrote: > > On Mon, Dec 6, 2010 at 2:21 PM, Raymond Hettinger > > wrote: > >> We really ought to stop with the SafeFoo naming convention. > >> It is only descriptive to the person who wrote the class or function, > >> not to the user who will immediately wonder, "safe from what?" > > > > Safe from bad vampire movies, of course! > > > > I'd not recognize the current Safe* class names as a pattern; there > > are currently two in the py3k trunk: > > > > configparser.SafeConfigParser > > -- very much a poor name > > > > xmlrpc.client.SafeTransport > > -- perhaps should have been SSLTransport or HTTPSTransport > > > > I agree the "Safe" prefix isn't meaningful. > > IIRC, pprint has a safe_repr() and string.Template has safe_substitute() > and pydoc has a safe import. > > Never new there was so much danger in the standard library :-) unittest has an (internal) safe_repr, too. I understand all four of these to be "versions of xxx that won't raise". I think that's a reasonable use of the word 'safe', but perhaps there is something better. SafeConfigParser doesn't follow that pattern, of course, though it is perhaps true that it would raise errors a bit less often... :) -- R. David Murray www.bitdance.com From alexander.belopolsky at gmail.com Tue Dec 7 04:03:16 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 6 Dec 2010 22:03:16 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: <4CFAC7B0.3080307@v.loewis.de> References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CFAC7B0.3080307@v.loewis.de> Message-ID: On Sat, Dec 4, 2010 at 5:58 PM, "Martin v. L?wis" wrote: >> I actually wonder if Python's re module can claim to provide even >> Basic Unicode Support. > > Do you really wonder? Most definitely it does not. > Were you more optimistic four years ago? http://bugs.python.org/issue1528154#msg54864 I was hoping that regex syntax would be useful in explaining/documenting Python text processing routines (including string to number conversions) without a heavy dose of Unicode terminology. From alexander.belopolsky at gmail.com Tue Dec 7 05:46:54 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 6 Dec 2010 23:46:54 -0500 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <201012052325.27903.victor.stinner@haypocalc.com> References: <20101202180651.F2AB0EE9CD@mail.python.org> <8762vax7lj.fsf@uwakimon.sk.tsukuba.ac.jp> <201012052325.27903.victor.stinner@haypocalc.com> Message-ID: On Sun, Dec 5, 2010 at 5:25 PM, Victor Stinner wrote: > On Saturday 04 December 2010 09:31:04 you wrote: >> Alexander Belopolsky writes: >> ?> In fact, once the language moratorium is over, I will argue that >> ?> str.encode() and byte.decode() should deprecate encoding argument and >> ?> just do UTF-8 encoding/decoding. ?Hopefully by that time most people >> ?> will forget that other encodings exist. ?(I can dream, right?) >> >> It's just a dream. ?There's a pile of archival material, often on R/O >> media, out there that won't be transcoded any more quickly than the >> inscriptions on Tutankhamun's tomb. > > Not only, many libraries expect use bytes arguments encoded to a specific > encoding (eg. locale encoding). Said differenlty, only few libraries written in > C accept wchar* strings. > My proposal has nothing to do with C-API. It only concerns Python API of the builtin str type. > The Linux kernel (or many, or all, UNIX/BSD kernels) only manipulate byte > strings. The libc only accept wide characters for a few operations. I don't > know how to open a file with an unicode path with the Linux libc: you have to > encode it... > Yes, but hopefully the encoding used by the filesystem will be UTF-8. For Python users, however, encoding details will hopefully be hidden by the open() call. Yes, I am aware of the many problems with divining the filesystem encoding, but instructing application developers to supply their own fsencoding in open(filepath.encode(fsencoding)) calls is not very helpful. > Alexander: you should first patch all UNIX/BSD kernels to use unicode > everywhere, then patch all libc implementations, and then all libraries > (written in C). After that, you can have a break. > As Martin explained later in this thread with respect to the transform() method, removing codec argument from str.encode() method does not imply removing the codecs themselves. If I need a method to encode strings to say koi8_r encoding, I can easily access it directly: >>> from encodings import koi8_r >>> to_koi8_r = koi8_r.Codec().encode >>> to_koi8_r('???') (b'\xcb\xcf\xc4', 3) More likely, however, I will only need en/decoding to read/write legacy files and rather than encoding the strings explicitly before writing into a file, I will just open that file with the correct encoding. Having all encodings accessible in a str method only promotes a programming style where bytes objects can contain differently encoded strings in different parts of the program. Instead, well-written programs should decode bytes on input, do all processing with str type and decode on output. When strings need to be passed to char* C APIs, they should be encoded in UTF-8. Many C APIs originally designed for ASCII actually produce meaningful results when given UTF-8 bytes. (Supporting such usage was one of the design goals of UTF-8.) From ncoghlan at gmail.com Tue Dec 7 06:06:13 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 7 Dec 2010 15:06:13 +1000 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <8762vax7lj.fsf@uwakimon.sk.tsukuba.ac.jp> <201012052325.27903.victor.stinner@haypocalc.com> Message-ID: On Tue, Dec 7, 2010 at 2:46 PM, Alexander Belopolsky wrote: > Having all encodings accessible in a str method only promotes a > programming style where bytes objects can contain differently encoded > strings in different parts of the program. ?Instead, well-written > programs should decode bytes on input, do all processing with str type > and decode on output. ?When strings need to be passed to char* C APIs, > they should be encoded in UTF-8. ?Many C APIs originally designed for > ASCII actually produce meaningful results when given ?UTF-8 bytes. > (Supporting such usage was one of the design goals of UTF-8.) This world sounds nice, but it isn't the one that exists right now. Practicality beats purity and all that :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From alexander.belopolsky at gmail.com Tue Dec 7 06:57:43 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 7 Dec 2010 00:57:43 -0500 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <8762vax7lj.fsf@uwakimon.sk.tsukuba.ac.jp> <201012052325.27903.victor.stinner@haypocalc.com> Message-ID: On Tue, Dec 7, 2010 at 12:06 AM, Nick Coghlan wrote: > On Tue, Dec 7, 2010 at 2:46 PM, Alexander Belopolsky > wrote: >> Having all encodings accessible in a str method only promotes a >> programming style where bytes objects can contain differently encoded >> strings in different parts of the program. ?Instead, well-written >> programs should decode bytes on input, do all processing with str type >> and decode on output. ?When strings need to be passed to char* C APIs, >> they should be encoded in UTF-8. ?Many C APIs originally designed for >> ASCII actually produce meaningful results when given ?UTF-8 bytes. >> (Supporting such usage was one of the design goals of UTF-8.) > > This world sounds nice, but it isn't the one that exists right now. > Practicality beats purity and all that :) .. and default encoding being fixed as UTF-8 already goes 99% of the way to that world. As long as I can use encode/decode without an argument, it does not bother me much that they can take one. These methods are also much easier to ignore than the transform/untransform pair simply because it is only one method per class. transform/untransform have much larger mental footprint not only because there are two of them in both str and bytes, but also because both str and bytes have a synonymously named translate method. With 43 non-special methods, str interface is already huge. The transform() method with a suitable set of codecs could possibly replace things like expandtabs() or swapcase(), but that would be like writing x.transform('exp') and x.unstransform('exp') instead of math.exp(x) and math.log(x). From lukasz at langa.pl Tue Dec 7 09:24:37 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Tue, 7 Dec 2010 09:24:37 +0100 Subject: [Python-Dev] [RELEASED] Python 3.2 beta 1 In-Reply-To: <4CFD59C8.4070006@python.org> References: <4CFD59C8.4070006@python.org> Message-ID: <433FCEE2-CD6C-48BD-90DF-2990B27E4EED@langa.pl> Wiadomo?? napisana przez Georg Brandl w dniu 2010-12-06, o godz. 22:46: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On behalf of the Python development team, I'm happy to announce the > first of two beta preview releases of Python 3.2. > > Python 3.2 is a continuation of the efforts to improve and stabilize the > Python 3.x line. Since the final release of Python 2.7, the 2.x line > will only receive bugfixes, and new features are developed for 3.x only. > > Since PEP 3003, the Moratorium on Language Changes, is in effect, there > are no changes in Python's syntax and built-in types in Python 3.2. > Development efforts concentrated on the standard library and support for > porting code to Python 3. Highlights are: > > [snip] * configparser 1.1: new API using the mapping protocol access, support for pluggable interpolation handlers, additional interpolation handler (ExtendedInterpolation) which supports the zc.buildout syntax, support for alternative option/value delimiters, support for customization of accepted INI file structure (e.g. comment prefixes, name of the DEFAULT section, indentation, empty lines in multiline values, etc.), support for specifying encoding for read operations, ConfigParser class deprecated in favor of SafeConfigParser, lots of other small changes. -- Best regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From flub at devork.be Tue Dec 7 13:22:29 2010 From: flub at devork.be (Floris Bruynooghe) Date: Tue, 7 Dec 2010 12:22:29 +0000 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFD3193.2030906@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFCAA66.6060901@v.loewis.de> <4CFD3193.2030906@v.loewis.de> Message-ID: On 6 December 2010 18:55, "Martin v. L?wis" wrote: > Am 06.12.2010 14:40, schrieb Floris Bruynooghe: >> On 6 December 2010 09:18, "Martin v. L?wis" wrote: >>>> Also, it is not clear what to do about distributions/OSs >>>> without any official EOL or life cycles. >>> >>> Here my proposal stands: 10 years, by default. >> >> How about max(EOL, 10years)? ?That sounds like it could be a useful guideline. >> >> (Personally I'd be sad to see Solaris 8 go in the next few years) > > I guess we'll be sorry, then: under that policy, max(EOL, 10years) comes > out as "not supported" (not sure whether you were aware of that). I was :-) Which is why I said "sad" and not "panic" or something. I do realise I'm probably in the minority and that it doesn't justify the burden on everyone in the Python community. > We are not going to take the sources of old releases offline. exactly Regards Floris -- Debian GNU/Linux -- The Power of Freedom www.debian.org | www.gnu.org | www.kernel.org From vlastimil.brom at gmail.com Tue Dec 7 14:02:47 2010 From: vlastimil.brom at gmail.com (Vlastimil Brom) Date: Tue, 7 Dec 2010 14:02:47 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CFAC7B0.3080307@v.loewis.de> Message-ID: 2010/12/7 Alexander Belopolsky : > On Sat, Dec 4, 2010 at 5:58 PM, "Martin v. L?wis" wrote: >>> I actually wonder if Python's re module can claim to provide even >>> Basic Unicode Support. >> >> Do you really wonder? Most definitely it does not. >> > > Were you more optimistic four years ago? > > http://bugs.python.org/issue1528154#msg54864 > > I was hoping that regex syntax would be useful in > explaining/documenting Python text processing routines (including > string to number conversions) without a heavy dose of Unicode > terminology. > The new regex version http://bugs.python.org/issue2636 supports much more features, including unicode properties, and the mentioned possix classes etc. but definitely not all of the requirements of that rather "generous" list. http://www.unicode.org/reports/tr18/ It seems, e.g. in Perl, there are some omissions too http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression-Support-Level Do you know of any re engine fully complying to to tr18, even at the first level: "Basic Unicode Support"? vbr From tseaver at palladion.com Tue Dec 7 15:38:30 2010 From: tseaver at palladion.com (Tres Seaver) Date: Tue, 07 Dec 2010 09:38:30 -0500 Subject: [Python-Dev] kill_python on windows buildbots (was Re: Stable buildbots) In-Reply-To: References: <20101113133712.60e9be27@pitrou.net> <4CEB7E12.1070201@snakebite.org> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/06/2010 07:24 PM, David Bolen wrote: > I previously wrote: > >> I suspect the problem may be on the "identify which process to kill" >> rather than the "kill it" part, but it's definitely going to take time >> to figure that out for sure. While the approach kill_python takes is >> much more appropriate, since we don't currently have multiple builds >> running simultaneously (and for me the machines are dedicated as build >> slaves, so I won't be having my own python_d), a more blanket kill >> operation is safe enough. > > For anyone interested, I caught (well, Georg Brandl caught it first) a > case on Saturday with some hung processes on the Win7 buildbot that I > was able to verify kill_python failed to kill. This was after having > a few instances where it did work fine. > > I've created issue 10641 to track. I also noticed another recent > issue (10136) that is also related to kill_python missing processes, > but given that it works in my case some of the time, and is always > called via the same path, not sure if that can be my problem. > > I also realized that some fraction of the other cases I have seen > might not have truly been an issue, since from what I can see > kill_python is only run at the start of a build process, so hung > processes (even killable ones) from a prior build hang around until > the next build takes place. They can however, interfere with the svn > checkout so things never get to the point of using kill_python. So > maybe kill_python's use should be moved to the clean stage? Maybe belt-and-suspenders it in both places. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkz+RuYACgkQ+gerLs4ltQ58OgCgiMs1JAdFjkOGxrJ3X3nB1k18 iKcAoMCP+MYumDe0r/XkZr29e7loACxP =wrWe -----END PGP SIGNATURE----- From alexander.belopolsky at gmail.com Tue Dec 7 15:41:59 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 7 Dec 2010 09:41:59 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CFAC7B0.3080307@v.loewis.de> Message-ID: On Tue, Dec 7, 2010 at 8:02 AM, Vlastimil Brom wrote: .. > It seems, e.g. in Perl, there are some omissions too > http://perldoc.perl.org/perlunicode.html#Unicode-Regular-Expression-Support-Level > > Do you know of any re engine fully complying to to tr18, even at the > first level: "Basic Unicode Support"? > I would say Perl comes very close. At least it explicitly documents the missing features and offers workarounds in its reference manual. I am actually not as concerned about missing features as I am about non-conformance in the widely used features such as digits' matching with '\d'. From db3l.net at gmail.com Tue Dec 7 16:11:58 2010 From: db3l.net at gmail.com (David Bolen) Date: Tue, 07 Dec 2010 10:11:58 -0500 Subject: [Python-Dev] kill_python on windows buildbots References: <20101113133712.60e9be27@pitrou.net> <4CEB7E12.1070201@snakebite.org> Message-ID: Tres Seaver writes: > Maybe belt-and-suspenders it in both places. The clean batch file is also called from the build step, so relocating it there should maintain the existing behavior as well. Hirokazu (ocean-city) pointed out in my new issue an earlier issue he created (#9973) that included a patch for such a change. In thinking about it some more, I suppose there's still a small window for a loss of communication during a test which results in clean not being run (which could then block the next svn checkout without an opportunity to kill the processes), so maybe the right place is actually at the end of the test batch file which is the step during which such hung processes might get generated? I don't know the history, if any, of it's current location in the flow. Relocating the use of kill_python seems reasonable to me, after which we could sort of wait and see if we run into any other hangs that interfere with the builds. I can't make such a change myself though. -- David From dmalcolm at redhat.com Tue Dec 7 16:56:47 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Tue, 07 Dec 2010 10:56:47 -0500 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFD6C2B.8030708@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFCAA66.6060901@v.loewis.de> <1291676103.15083.28.camel@radiator.bos.redhat.com> <4CFD6C2B.8030708@v.loewis.de> Message-ID: <1291737407.20128.17.camel@radiator.bos.redhat.com> On Tue, 2010-12-07 at 00:05 +0100, "Martin v. L?wis" wrote: > >> So by this policy, RHEL and SuSE users would be off worse than with > >> my original proposal (10 years). > > > > Red Hat continues to provide patches for RHEL within the "Extended Life > > Cycle" (years 8, 9 and 10), but it's an optional add-on. > > My understanding is that you keep the patches available - but you > don't produce any new ones, right? It typically involves backporting (and testing) security fixes to the older versions of the various OS packages. Whether or not the results of that work count as "new patches" is debatable. I don't know if CentOS and the other people who rebuild the RHEL sources track those final 3 years. Dave From dmalcolm at redhat.com Tue Dec 7 16:56:47 2010 From: dmalcolm at redhat.com (David Malcolm) Date: Tue, 07 Dec 2010 10:56:47 -0500 Subject: [Python-Dev] PEP 11: Dropping support for ten year old systems In-Reply-To: <4CFD6C2B.8030708@v.loewis.de> References: <4CFC08C1.3080609@v.loewis.de> <4CFCAA66.6060901@v.loewis.de> <1291676103.15083.28.camel@radiator.bos.redhat.com> <4CFD6C2B.8030708@v.loewis.de> Message-ID: <1291737407.20128.17.camel@radiator.bos.redhat.com> On Tue, 2010-12-07 at 00:05 +0100, "Martin v. L?wis" wrote: > >> So by this policy, RHEL and SuSE users would be off worse than with > >> my original proposal (10 years). > > > > Red Hat continues to provide patches for RHEL within the "Extended Life > > Cycle" (years 8, 9 and 10), but it's an optional add-on. > > My understanding is that you keep the patches available - but you > don't produce any new ones, right? It typically involves backporting (and testing) security fixes to the older versions of the various OS packages. Whether or not the results of that work count as "new patches" is debatable. I don't know if CentOS and the other people who rebuild the RHEL sources track those final 3 years. Dave From merwok at netwok.org Tue Dec 7 18:24:33 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Tue, 07 Dec 2010 18:24:33 +0100 Subject: [Python-Dev] [Python-checkins] r87118 - in python/branches/py3k: Mac/BuildScript/build-installer.py Makefile.pre.in In-Reply-To: <20101207144105.C0E88EEB67@mail.python.org> References: <20101207144105.C0E88EEB67@mail.python.org> Message-ID: <4CFE6DD1.9030801@netwok.org> Hello, > Author: ronald.oussoren > New Revision: 87118 > Log: Two small changes to adjust framework builds to the new stable ABI > > Modified: python/branches/py3k/Mac/BuildScript/build-installer.py > ============================================================================== > + LDVERSION=None > + VERSION=None > + ABIFLAGS=None > + > + with open(os.path.join(buildDir, 'Makefile')) as fp: > + for ln in fp: > + if ln.startswith('VERSION='): > + VERSION=ln.split()[1] > + if ln.startswith('ABIFLAGS='): > + ABIFLAGS=ln.split()[1] > + > + if ln.startswith('LDVERSION='): > + LDVERSION=ln.split()[1] > + > + LDVERSION = LDVERSION.replace('$(VERSION)', VERSION) > + LDVERSION = LDVERSION.replace('$(ABIFLAGS)', ABIFLAGS) Isn?t this a perfect use case for the new sysconfig module? Regards From martin at v.loewis.de Tue Dec 7 21:20:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 07 Dec 2010 21:20:21 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CFAC7B0.3080307@v.loewis.de> Message-ID: <4CFE9705.4020800@v.loewis.de> Am 07.12.2010 04:03, schrieb Alexander Belopolsky: > On Sat, Dec 4, 2010 at 5:58 PM, "Martin v. L?wis" wrote: >>> I actually wonder if Python's re module can claim to provide even >>> Basic Unicode Support. >> >> Do you really wonder? Most definitely it does not. >> > > Were you more optimistic four years ago? > > http://bugs.python.org/issue1528154#msg54864 Not at all. I thought back then, and think now, that Python should, but doesn't, support TR#18. I don't view that lack as a severe problem, though, and apparently none of the other contributors did so, either. Regards, Martin From vinay_sajip at yahoo.co.uk Tue Dec 7 21:26:06 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 7 Dec 2010 20:26:06 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests Message-ID: This issue was brought to my notice today: http://bugs.python.org/issue10626 and reference was made in the comments to possible obstacles facing stdlib maintainers who might wish to use logging in the stdlib and in its unit tests. >From my perspective and as mentioned in the logging documentation, library code which uses logging should add a NullHandler instance to any top-level logger, which will avoid any "No handlers could be found for logger XXX" message if no logging handlers have been set up. This applies to stdlib code, too, though it would be good if a logger namespace could be agreed for stdlib usage. (The logging package itself uses the logger "py.warnings" to redirect warnings to logging when configured to do so. Perhaps we could standardize on "py.XXX" for stdlib usage?) I would suggest that when unit testing, rather than adding StreamHandlers to log to stderr, that something like TestHandler and Matcher from this post: http://plumberjack.blogspot.com/2010/09/unit-testing-and-logging.html This will allow assertion checking of logged messages without resorting to StringIO, getvalue() etc. If people think it's a good idea, I can add the TestHandler/Matcher classes to the unit test infrastructure (they wouldn't become part of the public API, at least until 3.3, but I presume they could be used in the stdlib unit tests). On the question of using logging in the stdlib and its unit tests, I would like to throw this open to python-dev: is there anyone here who wants to use logging but feels that they can't because of some shortcoming in logging? AFAIK there should be no obstacles. The preferred approach in stdlib code is as I have outlined above: add a NullHandler to top-level loggers to avoid misconfiguration, document the logger names you use, and avoid adding any other handlers by default. For unit testing, add a TestHandler (or equivalent) to your top-level logger at the start of the test, make any assertions you need to regarding what has been logged, remove the handler and close it at the end of the test. I don't believe there is any inherent conflict or incompatibility between logger usage in the stdlib, use of logging assertions in unit tests and use of handlers by application code, but I am happy to have any mistake on my part pointed out. >From what I've seen, concurrent.futures adds a StreamHandler which is removed in the unit test and replaced by StreamHandler pointing to a different stream. This, I believe, should be changed in line with what I've said above. Comments? Regards, Vinay Sajip From nad at acm.org Tue Dec 7 21:48:18 2010 From: nad at acm.org (Ned Deily) Date: Tue, 07 Dec 2010 12:48:18 -0800 Subject: [Python-Dev] [Python-checkins] r87118 - in python/branches/py3k: Mac/BuildScript/build-installer.py Makefile.pre.in References: <20101207144105.C0E88EEB67@mail.python.org> <4CFE6DD1.9030801@netwok.org> Message-ID: In article <4CFE6DD1.9030801 at netwok.org>, ?ric Araujo wrote: > > Author: ronald.oussoren > > New Revision: 87118 > > Log: Two small changes to adjust framework builds to the new stable ABI > > > > Modified: python/branches/py3k/Mac/BuildScript/build-installer.py > > ============================================================================ > > == > > + LDVERSION=None > > + VERSION=None > > + ABIFLAGS=None > > + > > + with open(os.path.join(buildDir, 'Makefile')) as fp: > > + for ln in fp: > > + if ln.startswith('VERSION='): > > + VERSION=ln.split()[1] > > + if ln.startswith('ABIFLAGS='): > > + ABIFLAGS=ln.split()[1] > > + > > + if ln.startswith('LDVERSION='): > > + LDVERSION=ln.split()[1] > > + > > + LDVERSION = LDVERSION.replace('$(VERSION)', VERSION) > > + LDVERSION = LDVERSION.replace('$(ABIFLAGS)', ABIFLAGS) > > Isn?t this a perfect use case for the new sysconfig module? There is no guarantee which version of Python the build-installer script is being run with and, hence, no guarantee which version of sysconfig is available. Typically it is an older system-supplied Python which is fine as it is merely used to run build-installer (which includes running Sphinx to build the documentation) and otherwise has no impact on the Python being built. -- Ned Deily, nad at acm.org From solipsis at pitrou.net Tue Dec 7 21:50:53 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Dec 2010 21:50:53 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: Message-ID: <20101207215053.025ecf52@pitrou.net> On Tue, 7 Dec 2010 20:26:06 +0000 (UTC) Vinay Sajip wrote: > > >From my perspective and as mentioned in the logging documentation, library code > which uses logging should add a NullHandler instance to any top-level logger, > which will avoid any "No handlers could be found for logger XXX" message if no > logging handlers have been set up. This applies to stdlib code, too, though it > would be good if a logger namespace could be agreed for stdlib usage. (The > logging package itself uses the logger "py.warnings" to redirect warnings to > logging when configured to do so. Perhaps we could standardize on "py.XXX" for > stdlib usage?) I thought "error" and "critical" messages were logged to stderr by default? Isn't it the case? If any library defining a logger must also add a NullHandler just in case, ISTM that complicates a lot the use of logging (and could explain its impopularity). Both for library writers and application writers, apparently (since the former will have to add the NullHandler, and the latter will have to override it so that error messages get printed out rather than lost). Regards Antoine. From lukasz at langa.pl Tue Dec 7 22:04:36 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Tue, 7 Dec 2010 22:04:36 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <20101207215053.025ecf52@pitrou.net> References: <20101207215053.025ecf52@pitrou.net> Message-ID: <8E88AE41-04D5-4B5A-9ECB-1CC6A137B3C0@langa.pl> Wiadomo?? napisana przez Antoine Pitrou w dniu 2010-12-07, o godz. 21:50: > If any library defining a logger must also add a NullHandler just in > case, ISTM that complicates a lot the use of logging (and could explain > its impopularity). Both for library writers and application writers, > apparently (since the former will have to add the NullHandler, and the > latter will have to override it so that error messages get printed out > rather than lost). ISTM that the stdlib is a special case here. If you're writing an application then the "No handlers could be found" message is actually useful because there's hardly any reason no to include one. If you're writing a library, well, that depends. Part of libraries might want to have their specific default handlers, part of them might want to leave that to the library user (e.g. "No handlers can be found" by default) and part of them might want to silence logs by default. One way or the other, we should really default to the convenience of application developers. This is currently the case. -- Best regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Dec 7 22:19:34 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 Dec 2010 22:19:34 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <8E88AE41-04D5-4B5A-9ECB-1CC6A137B3C0@langa.pl> References: <20101207215053.025ecf52@pitrou.net> <8E88AE41-04D5-4B5A-9ECB-1CC6A137B3C0@langa.pl> Message-ID: <20101207221934.725be79d@pitrou.net> On Tue, 7 Dec 2010 22:04:36 +0100 ?ukasz Langa wrote: > Wiadomo?? napisana przez Antoine Pitrou w dniu 2010-12-07, o godz. 21:50: > > > If any library defining a logger must also add a NullHandler just in > > case, ISTM that complicates a lot the use of logging (and could explain > > its impopularity). Both for library writers and application writers, > > apparently (since the former will have to add the NullHandler, and the > > latter will have to override it so that error messages get printed out > > rather than lost). > > ISTM that the stdlib is a special case here. If you're writing an application then the "No handlers could be found" message is actually useful because there's hardly any reason no to include one. Why do you say that? Not having to add a handler is certainly useful when you are doing some quick prototyping or simply writing a script (situations in which you still want to get error messages displayed properly by the libraries). > One way or the other, we should really default to the convenience of application developers. This is currently the case. Why wouldn't there be a default convenience of printing out errors? It's already the case for the root handler, so why would other handler be treated differently? >>> import logging >>> logging.debug("foo") >>> logging.error("bar") ERROR:root:bar Thanks Antoine. From alexander.belopolsky at gmail.com Tue Dec 7 22:21:46 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 7 Dec 2010 16:21:46 -0500 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CFAC7B0.3080307@v.loewis.de> Message-ID: On Tue, Dec 7, 2010 at 8:02 AM, Vlastimil Brom wrote: .. > Do you know of any re engine fully complying to to tr18, even at the > first level: "Basic Unicode Support"? > """ ICU Regular Expressions conform to Unicode Technical Standard #18 , Unicode Regular Expressions, level 1, and in addition include Default Word boundaries and Name Properties from level 2. """ http://userguide.icu-project.org/strings/regexp From eric at trueblade.com Tue Dec 7 22:57:20 2010 From: eric at trueblade.com (Eric Smith) Date: Tue, 07 Dec 2010 16:57:20 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: Message-ID: <4CFEADC0.50609@trueblade.com> On 12/07/2010 03:26 PM, Vinay Sajip wrote: > I would suggest that when unit testing, rather than adding StreamHandlers to log > to stderr, that something like TestHandler and Matcher from this post: > > http://plumberjack.blogspot.com/2010/09/unit-testing-and-logging.html > > This will allow assertion checking of logged messages without resorting to > StringIO, getvalue() etc. If people think it's a good idea, I can add the > TestHandler/Matcher classes to the unit test infrastructure (they wouldn't > become part of the public API, at least until 3.3, but I presume they could be > used in the stdlib unit tests). Wouldn't it make more sense to add these to test.support? I don't think we make any guarantees about its API being stable, although I have a faint recollection of that being debated in the past. Eric. From vlastimil.brom at gmail.com Tue Dec 7 23:19:44 2010 From: vlastimil.brom at gmail.com (Vlastimil Brom) Date: Tue, 7 Dec 2010 23:19:44 +0100 Subject: [Python-Dev] Python and the Unicode Character Database In-Reply-To: References: <20101128214311.092abd35@pitrou.net> <4CF2D4E9.3060607@v.loewis.de> <4CF2DAD7.2000408@egenix.com> <4CF2DFD1.10901@v.loewis.de> <4CF7FC01.80307@egenix.com> <4CF8004D.1080408@v.loewis.de> <4CFAC7B0.3080307@v.loewis.de> Message-ID: 2010/12/7 Alexander Belopolsky : > On Tue, Dec 7, 2010 at 8:02 AM, Vlastimil Brom wrote: > .. >> Do you know of any re engine fully complying to to tr18, even at the >> first level: "Basic Unicode Support"? >> > """ > ICU Regular Expressions conform to Unicode Technical Standard #18 , > Unicode Regular Expressions, level 1, and in addition include Default > Word boundaries and Name Properties from level 2. > """ http://userguide.icu-project.org/strings/regexp > Thanks for the pointer, I wasn't aware of that project. Anyway I am quite happy with the mentioned regex library and can live with trading this full compliance for some non-unicode goodies (like unbounded lookbehinds etc.), but I see, it's beyond the point here. Not that my opinion matters, but I can't think of, say, "union, intersection and set-difference of Unicode sets" as a basic feature or consider it a part of "a minimal level for useful Unicode support." vbr From lukasz at langa.pl Tue Dec 7 23:33:24 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Tue, 7 Dec 2010 23:33:24 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <20101207221934.725be79d@pitrou.net> References: <20101207215053.025ecf52@pitrou.net> <8E88AE41-04D5-4B5A-9ECB-1CC6A137B3C0@langa.pl> <20101207221934.725be79d@pitrou.net> Message-ID: <9CF9C003-B62D-444E-B988-6ECEB6229F93@langa.pl> Wiadomo?? napisana przez Antoine Pitrou w dniu 2010-12-07, o godz. 22:19: >> If you're writing an application then the "No handlers could be found" message is actually useful because there's hardly any reason no to include one. > > Why do you say that? Not having to add a handler is certainly useful > when you are doing some quick prototyping or simply writing a script > (situations in which you still want to get error messages displayed > properly by the libraries). > >> One way or the other, we should really default to the convenience of application developers. This is currently the case. > > Why wouldn't there be a default convenience of printing out errors? > It's already the case for the root handler, so why would other handler > be treated differently? > >>>> import logging >>>> logging.debug("foo") >>>> logging.error("bar") > ERROR:root:bar If you're arguing that instead of writing "No handler", our logging library could just as easily default to a simplistic stderr handler for errors, then I agree. Both the convenience and consistency arguments you provided are convincing. See, that was 3 times a con* in one sentence! Then again, warning an application developer that some handler is not configured that probably should be, is still valuable IMHO. Unless explicitly silenced. -- Best regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Tue Dec 7 23:59:36 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 07 Dec 2010 16:59:36 -0600 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: Message-ID: On 12/7/10 2:26 PM, Vinay Sajip wrote: > This issue was brought to my notice today: > > http://bugs.python.org/issue10626 > > and reference was made in the comments to possible obstacles facing stdlib > maintainers who might wish to use logging in the stdlib and in its unit tests. > >> From my perspective and as mentioned in the logging documentation, library code > which uses logging should add a NullHandler instance to any top-level logger, > which will avoid any "No handlers could be found for logger XXX" message if no > logging handlers have been set up. I've done that before in my own library code, then quickly realized that it was a bad idea. Adding a NullHandler silently prevents logging.basicConfig() from working. Embedding that kind of restriction into library code (usually by a mere import!) made users unhappy and added an additional burden on the library developers who had to make sure to do that everywhere they used logging. If I had my druthers, I would simply remove the "No handlers could be found for logger XXX" message. If we always wrote entire applications from the ground up, it makes a lot of sense. The person that writes the code that issues logs is the same person that writes the code to configure logging for reading. If you add logging in one place, you almost certainly want to use it somewhere else and not setting up logging is probably an error. But when you want to write reusable libraries, those roles become separated. As a library author, I would dearly love to just add logging liberally without placing any additional burden to the users of my library. If my users wants to read those logs, he will configure logging. If he doesn't, he won't. With the current behavior, I can't do that. If I add logging, he has to add code just to silence a message that is meaningless to him (after I get the support emails asking if things are broken and explain how to silence it). If I add a NullHandler, I remove the ability for him to use logging.basicConfig(), the easiest and most straightforward way for him to add logging to his application. This is only my personal anecdotal experience, but the current behavior has always wasted my time and never saved any of my time. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From brett at python.org Wed Dec 8 00:03:47 2010 From: brett at python.org (Brett Cannon) Date: Tue, 7 Dec 2010 15:03:47 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <4CFEADC0.50609@trueblade.com> References: <4CFEADC0.50609@trueblade.com> Message-ID: On Tue, Dec 7, 2010 at 13:57, Eric Smith wrote: > On 12/07/2010 03:26 PM, Vinay Sajip wrote: >> >> I would suggest that when unit testing, rather than adding StreamHandlers >> to log >> to stderr, that something like TestHandler and Matcher from this post: >> >> http://plumberjack.blogspot.com/2010/09/unit-testing-and-logging.html >> >> This will allow assertion checking of logged messages without resorting to >> StringIO, getvalue() etc. If people think it's a good idea, I can add the >> TestHandler/Matcher classes to the unit test infrastructure (they wouldn't >> become part of the public API, at least until 3.3, but I presume they >> could be >> used in the stdlib unit tests). > > Wouldn't it make more sense to add these to test.support? I don't think we > make any guarantees about its API being stable, although I have a faint > recollection of that being debated in the past. We make no guarantees. The debate was whether we accidentally painted ourselves into a corner by documenting the module a tad for core developer's benefit. The warning has been made much more explicit in the docs that test.support is not meant for outside usage. From robert.kern at gmail.com Wed Dec 8 00:21:44 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 07 Dec 2010 17:21:44 -0600 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: Message-ID: On 12/7/10 4:59 PM, Robert Kern wrote: > On 12/7/10 2:26 PM, Vinay Sajip wrote: >> This issue was brought to my notice today: >> >> http://bugs.python.org/issue10626 >> >> and reference was made in the comments to possible obstacles facing stdlib >> maintainers who might wish to use logging in the stdlib and in its unit tests. >> >>> From my perspective and as mentioned in the logging documentation, library code >> which uses logging should add a NullHandler instance to any top-level logger, >> which will avoid any "No handlers could be found for logger XXX" message if no >> logging handlers have been set up. > > I've done that before in my own library code, then quickly realized that it was > a bad idea. Adding a NullHandler silently prevents logging.basicConfig() from > working. Only on the root handler. Forget this bit of my argument (and the statements that directly follow from it). The rest of my argument should stand on its own, though. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From barry at python.org Wed Dec 8 00:38:55 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Dec 2010 18:38:55 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: Message-ID: <20101207183855.52ad8926@mission> On Dec 07, 2010, at 04:59 PM, Robert Kern wrote: >As a library author, I would dearly love to just add logging liberally >without placing any additional burden to the users of my library. If my users >wants to read those logs, he will configure logging. If he doesn't, he >won't. With the current behavior, I can't do that. If I add logging, he has >to add code just to silence a message that is meaningless to him (after I get >the support emails asking if things are broken and explain how to silence >it). If I add a NullHandler, I remove the ability for him to use >logging.basicConfig(), the easiest and most straightforward way for him to >add logging to his application. +1 -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Wed Dec 8 00:45:39 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 7 Dec 2010 23:45:39 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > > I thought "error" and "critical" messages were logged to stderr by > default? Isn't it the case? > Only if you call basicConfig() or use the logging.debug(), logging.info(), etc. module-level convenience functions (which call basicConfig under the hood). > If any library defining a logger must also add a NullHandler just in > case, ISTM that complicates a lot the use of logging (and could explain > its impopularity). Both for library writers and application writers, > apparently (since the former will have to add the NullHandler, and the > latter will have to override it so that error messages get printed out > rather than lost). > No - the application developer doesn't have to do anything special. Adding the NullHandler is only needed for the top-level logger of a library package, so it's hardly more onerous than doing an import. It doesn't complicate anything, as far as I can see. It's to be done once per library, and trivial in comparison with all the work needed to develop, test, and package a library. For the stdlib, for example, if we were to standardize and say that all stdlib loggers were to be located below the "py" namespace, then I could put the relevant NullHandler-adding code in the logging package and no other stdlib maintainer would have to do anything other than name their stdlib loggers beginning with "py." (e.g. "py.concurrent.futures") for things to work as expected. If you don't like "py", then pick "std", "stdlib" or whatever. When is the NullHandler needed? Only for cases where an application developer uses a library which does logging under the covers (for those users who might be interested in logging its operations), but where that application developer doesn't use logging themselves for that application. In such a case, a library call which generated a logging event would cause logging to look for handlers, and if one is not found, print that one-off "no handlers could be found for logger XXX" message. This message is there because errors should not pass silently; but in the case of an application developer oblivious of logging who uses a library which uses logging under the hood, you can't argue that that developer has made any error. So the library developer, by adding a NullHandler, can improve things for their users by not forcing them to configure logging handlers (in order to avoid the message). #mylib.py import logging logger = logging.getLogger(__name__) logger.addHandler(logging.NullHandler()) def func(): logger.warning('Something happened!') #myapp.py import mylib mylib.func() If you run myapp.py, no message is printed. If you comment out the "addHandler" line and re-run, then you see No handlers could be found for logger "mylib" which is the misconfiguration message. With the line commented back in, you as an application developer get expected behaviour (no spurious messages about missing handlers). When you decide to use logging in myapp.py, simply insert import logging logging.basicConfig() before calling mylib.func(), and you get WARNING:mylib:Something happened! which is what you would expect. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Dec 8 00:58:54 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 7 Dec 2010 23:58:54 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: Message-ID: Robert Kern gmail.com> writes: > If I had my druthers, I would simply remove the "No handlers could be > found for logger XXX" message. If we always wrote entire applications > from the ground up, it makes a lot of sense. The person that writes the > code that issues logs is the same person that writes the code to configure > logging for reading. If you add logging in one place, you almost certainly > want to use it somewhere else and not setting up logging is probably an > error. But when you want to write reusable libraries, those roles become > separated. Exactly - we do use third-party libraries. When logging was added there was some debate about whether this one-off message should be printed, and in balance it was thought better to print this, not least because people would be unfamiliar with logging (as it was then new) and so be not unlikely to misconfigure it. No indication of this at all would be double-plus-ungood. > As a library author, I would dearly love to just add logging liberally > without placing any additional burden to the users of my library. > If my users wants to read those logs, he will configure logging. If he > doesn't, he won't. With the current behavior, I can't do that. If I add > logging, he has to add code just to silence a message that is meaningless > to him (after I get the support emails asking if things are broken and > explain how to silence it). If I add a NullHandler, I remove the ability > for him to use logging.basicConfig(), the easiest and most straightforward > way for him to add logging to his application. You don't remove the ability for him to use basicConfig() - that's another mistake in your post (you already corrected the other mistake in your self-response). See my example with mylib.py/myapp.py in another post on this thread, in response to Antoine. > This is only my personal anecdotal experience, but the current behavior has > always wasted my time and never saved any of my time. > That's because I think you're doing it wrong, or else I've misunderstood your use case. NullHandler is the way to go, and doesn't preclude the use of basicConfig() UNLESS you choose to add the NullHandler to the root logger (not what you're supposed to do - you're supposed to add it to the top-level logger *of your library*, not the top-level logger of the logging hierarchy.) See the mylib/myapp example and tell me which it is - your mistake or mine? Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Dec 8 01:01:25 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 00:01:25 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207183855.52ad8926@mission> Message-ID: Barry Warsaw python.org> writes: > > On Dec 07, 2010, at 04:59 PM, Robert Kern wrote: > > >As a library author, I would dearly love to just add logging liberally > >without placing any additional burden to the users of my library. If my users > >wants to read those logs, he will configure logging. If he doesn't, he > >won't. With the current behavior, I can't do that. If I add logging, he has > >to add code just to silence a message that is meaningless to him (after I get > >the support emails asking if things are broken and explain how to silence > >it). If I add a NullHandler, I remove the ability for him to use > >logging.basicConfig(), the easiest and most straightforward way for him to > >add logging to his application. > > +1 > Barry, if you mean +1 as in "I agree this is how it should work", then we're all agreed. But I think Robert is wrong that NullHandler precludes use of basicConfig - when NullHandler is added to a library logger (rather than the root logger), basicConfig() works as expected. See the mylib/myapp example in my other post on this thread. Regards, Vinay Sajip From python at mrabarnett.plus.com Wed Dec 8 01:07:56 2010 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 08 Dec 2010 00:07:56 +0000 Subject: [Python-Dev] Is PyPI down? Message-ID: <4CFECC5C.8040001@mrabarnett.plus.com> I can't get http://pypi.python.org and I've double-checked using http://downforeveryoneorjustme.com/. From solipsis at pitrou.net Wed Dec 8 01:09:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Dec 2010 01:09:13 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> Message-ID: <20101208010913.363c6105@pitrou.net> On Tue, 7 Dec 2010 23:45:39 +0000 (UTC) Vinay Sajip wrote: > Antoine Pitrou pitrou.net> writes: > > > > > I thought "error" and "critical" messages were logged to stderr by > > default? Isn't it the case? > > > > Only if you call basicConfig() or use the logging.debug(), logging.info(), etc. > module-level convenience functions (which call basicConfig under the hood). Why wouldn't it be the default for all logging calls ? Such special cases don't really make things easy to remember. > When is the NullHandler needed? Only for cases where an application developer > uses a library which does logging under the covers (for those users who might be > interested in logging its operations), but where that application developer > doesn't use logging themselves for that application. You seem pretty tied up to the "application developer" situation. There are cases (scripts, prototyping, etc.) where you certainly want to see error messages (errors should not pass silently) but don't want to configure logging for each of the libraries you use. Having convenient and understandable defaults would go a long way towards making logging more usable, IMO. Regards Antoine. From vinay_sajip at yahoo.co.uk Wed Dec 8 01:09:28 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 00:09:28 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <4CFEADC0.50609@trueblade.com> Message-ID: Eric Smith trueblade.com> writes: > Wouldn't it make more sense to add these to test.support? I don't think > we make any guarantees about its API being stable, although I have a > faint recollection of that being debated in the past. > That's what I mean when I said "unit test infrastructure" - adding to test.support. Sorry, I should have been more explicit. Regards, Vinay Sajip From richard at python.org Wed Dec 8 01:50:17 2010 From: richard at python.org (Richard Jones) Date: Wed, 8 Dec 2010 11:50:17 +1100 Subject: [Python-Dev] Is PyPI down? In-Reply-To: <4CFECC5C.8040001@mrabarnett.plus.com> References: <4CFECC5C.8040001@mrabarnett.plus.com> Message-ID: The monitor said it went away at 23:50 UTC and came back 22 minutes later at 00:12 UTC. Richard On Wed, Dec 8, 2010 at 11:07 AM, MRAB wrote: > I can't get http://pypi.python.org and I've double-checked using > http://downforeveryoneorjustme.com/. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/richard%40python.org > From eric at trueblade.com Wed Dec 8 01:57:58 2010 From: eric at trueblade.com (Eric Smith) Date: Tue, 07 Dec 2010 19:57:58 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <4CFEADC0.50609@trueblade.com> Message-ID: <4CFED816.1000204@trueblade.com> On 12/07/2010 07:09 PM, Vinay Sajip wrote: > Eric Smith trueblade.com> writes: > > >> Wouldn't it make more sense to add these to test.support? I don't think >> we make any guarantees about its API being stable, although I have a >> faint recollection of that being debated in the past. >> > > That's what I mean when I said "unit test infrastructure" - adding to > test.support. Sorry, I should have been more explicit. My bad. I read it as "unittest infrastructure". Eric. From ncoghlan at gmail.com Wed Dec 8 02:00:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Dec 2010 11:00:50 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <20101208010913.363c6105@pitrou.net> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: On Wed, Dec 8, 2010 at 10:09 AM, Antoine Pitrou wrote: > On Tue, 7 Dec 2010 23:45:39 +0000 (UTC) > Vinay Sajip wrote: >> Antoine Pitrou pitrou.net> writes: >> >> > >> > I thought "error" and "critical" messages were logged to stderr by >> > default? Isn't it the case? >> > >> >> Only if you call basicConfig() or use the logging.debug(), logging.info(), etc. >> module-level convenience functions (which call basicConfig under the hood). > > Why wouldn't it be the default for all logging calls ? Such special > cases don't really make things easy to remember. Indeed - I was very surprised to find just now that calling "logging.warn('Whatever')" is not the same as doing "log = logging.getLogger(); log.warn('Whatever')". Adding a NullHandler isn't the right thing to do - the behaviour I would want for any standard library logging that hasn't been explicitly configured otherwise is to do what the root logger does under basicConfig(): debug() and info() get suppressed, warn(), error(), critical() and exception() go to stderr. It seems to me like that would be more useful default behaviour in general than emitting a warning about not finding a handler. So my suggestion would be: 1. In the absence of a "config" call, make the "no handler" path in loggers emit messages *as if* basicConfig() has been called (without actually calling it) 2. Remove the implicit calls to basicConfig() from the module level convenience function How *feasible* that idea is to implement, I don't know. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Wed Dec 8 02:19:45 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 01:19:45 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > > Why wouldn't it be the default for all logging calls ? Such special > cases don't really make things easy to remember. > One size doesn't fit all. Everything's documented reasonably well. If you use it often, you remember it. If you use it seldom, you might need to look things up. > You seem pretty tied up to the "application developer" situation. There > are cases (scripts, prototyping, etc.) where you certainly want to see > error messages (errors should not pass silently) but don't want to > configure logging for each of the libraries you use. You don't have to configure each of the libraries you use. The "application developer" = "The developer who is not writing a library" = "The person writing a script, prototyping, etc." I don't mean Application with a capital A. I'm just using it as a convenient label to mean a library user who is not writing a library themselves. > Having convenient and understandable defaults would go a long way > towards making logging more usable, IMO. > The defaults are I believe reasonably convenient, or at least not inconvenient: 1. If writing a library, add a NullHandler to the top-level logger of your library, don't add any other handlers by default (i.e. on import), document the names of the loggers you use. Otherwise, see the following. 2. If writing a simple script, prototyping etc. i.e. for casual use, use logging.debug(), logging.info() etc. That's easy enough and convenient enough for casual users to remember, I hope. All WARNING, ERROR and CRITICAL events in your code and in that of any libraries you use will be printed to sys.stderr. 3. If slightly less than very casual use (e.g. you want to log to file, or want a specific message format), call basicConfig(). Just the one extra line to remember to do, hopefully not too inconvenient. All WARNING, ERROR and CRITICAL events in your code and in that of any libraries you use will be printed to sys.stderr, unless you specify a file to log to and/or a different logging level, in which case what you specified will be used as destination/threshold for logging messages. 4. For more advanced use, use programmatic configuration, fileConfig or dictConfig to specify a configuration. Convenience is inversely proportional to how involved the configuration is. If you can specify what you think is inconvenient in the above, that would be helpful. Regards, Vinay Sajip From barry at python.org Wed Dec 8 02:20:21 2010 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Dec 2010 20:20:21 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207183855.52ad8926@mission> Message-ID: <20101207202021.794d25bd@snowdog> On Dec 08, 2010, at 12:01 AM, Vinay Sajip wrote: >Barry, if you mean +1 as in "I agree this is how it should work", then >we're all agreed. Yep, that's what I meant! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From vinay_sajip at yahoo.co.uk Wed Dec 8 02:51:44 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 01:51:44 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: Nick Coghlan gmail.com> writes: > Indeed - I was very surprised to find just now that calling > "logging.warn('Whatever')" is not the same as doing "log = > logging.getLogger(); log.warn('Whatever')". Don't know why you'd be surprised: it's been that way since logging was added to Python, and the logging.debug() etc. are documented as convenience methods for casual use in throwaway scripts, interactive sessions etc. The convenience is in that you don't need to specify a logger (the root logger is used) and that basicConfig() is called for you. > Adding a NullHandler isn't the right thing to do - the behaviour I > would want for any standard library logging that hasn't been > explicitly configured otherwise is to do what the root logger does > under basicConfig(): debug() and info() get suppressed, warn(), > error(), critical() and exception() go to stderr. It seems to me like > that would be more useful default behaviour in general than emitting a > warning about not finding a handler. > > So my suggestion would be: > 1. In the absence of a "config" call, make the "no handler" path in > loggers emit messages *as if* basicConfig() has been called (without > actually calling it) > 2. Remove the implicit calls to basicConfig() from the module level > convenience function > > How *feasible* that idea is to implement, I don't know. > What about "Special cases aren't special enough to break the rules."? Why should logging from stdlib modules be treated differently from the case of logging from library modules in general? It could be argued that the behaviour you're proposing is confusing/inconsistent to some people. Regards, Vinay Sajip From robert.kern at gmail.com Wed Dec 8 03:33:33 2010 From: robert.kern at gmail.com (Robert Kern) Date: Tue, 07 Dec 2010 20:33:33 -0600 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: Message-ID: On 2010-12-07 17:58 , Vinay Sajip wrote: > Robert Kern gmail.com> writes: > > >> If I had my druthers, I would simply remove the "No handlers could be >> found for logger XXX" message. If we always wrote entire applications >> from the ground up, it makes a lot of sense. The person that writes the >> code that issues logs is the same person that writes the code to configure >> logging for reading. If you add logging in one place, you almost certainly >> want to use it somewhere else and not setting up logging is probably an >> error. But when you want to write reusable libraries, those roles become >> separated. > > Exactly - we do use third-party libraries. When logging was added there was some > debate about whether this one-off message should be printed, and in balance it > was thought better to print this, not least because people would be unfamiliar > with logging (as it was then new) and so be not unlikely to misconfigure it. No > indication of this at all would be double-plus-ungood. I really don't understand how this view can be consistent with the practice of adding NullHandler to loggers. If this message is so important to prevent misconfiguration, then why should a library author decide to silence it for his users? I think that the chances of a misconfiguration that the warning would catch. are small in practice. I don't have any solid survey data on how people configure their loggers, but I strongly suspect that almost all configurations include a catch-all root logger and that most of those *only* consist of that root logger. >> As a library author, I would dearly love to just add logging liberally >> without placing any additional burden to the users of my library. >> If my users wants to read those logs, he will configure logging. If he >> doesn't, he won't. With the current behavior, I can't do that. If I add >> logging, he has to add code just to silence a message that is meaningless >> to him (after I get the support emails asking if things are broken and >> explain how to silence it). If I add a NullHandler, I remove the ability >> for him to use logging.basicConfig(), the easiest and most straightforward >> way for him to add logging to his application. > > You don't remove the ability for him to use basicConfig() - that's another > mistake in your post (you already corrected the other mistake in your > self-response). See my example with mylib.py/myapp.py in another post on this > thread, in response to Antoine. Same mistake. I intended the correction to apply to all such statements in my post. >> This is only my personal anecdotal experience, but the current behavior has >> always wasted my time and never saved any of my time. > > That's because I think you're doing it wrong, or else I've misunderstood your > use case. I will try to state it more plainly: the message has always been a false positive in my experience. It has never helped me fix a real problem and has otherwise wasted my time. > NullHandler is the way to go, and doesn't preclude the use of > basicConfig() UNLESS you choose to add the NullHandler to the root logger (not > what you're supposed to do - you're supposed to add it to the top-level logger > *of your library*, not the top-level logger of the logging hierarchy.) > > See the mylib/myapp example and tell me which it is - your mistake or mine? I think that boilerplate should be minimized. If using getLogger() should almost always be followed by adding a NullHandler, then it should be the default behavior. The easiest way to achieve this effect is to simply not issue the warning message. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ocean-city at m2.ccsnet.ne.jp Wed Dec 8 07:12:32 2010 From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto) Date: Wed, 08 Dec 2010 15:12:32 +0900 Subject: [Python-Dev] kill_python on windows buildbots In-Reply-To: References: <20101113133712.60e9be27@pitrou.net> <4CEB7E12.1070201@snakebite.org> Message-ID: <4CFF21D0.7050705@m2.ccsnet.ne.jp> On 2010/12/08 0:11, David Bolen wrote: > In thinking about it some more, I suppose there's still a small window > for a loss of communication during a test which results in clean not > being run (which could then block the next svn checkout without an > opportunity to kill the processes), so maybe the right place is > actually at the end of the test batch file which is the step during > which such hung processes might get generated? I don't know the > history, if any, of it's current location in the flow. Yes, but test can freeze. In that case, I'm worried that (snip) rt.bat .... # freeze here (will be halt by buildbot) vcbuild .... & kill_python_d # Will this be called? in test.bat. From ncoghlan at gmail.com Wed Dec 8 08:28:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Dec 2010 17:28:34 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: On Wed, Dec 8, 2010 at 11:51 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> Indeed - I was very surprised to find just now that calling >> "logging.warn('Whatever')" is not the same as doing "log = >> logging.getLogger(); log.warn('Whatever')". > > Don't know why you'd be surprised: it's been that way since logging was added to > Python, and the logging.debug() etc. are documented as convenience methods for > casual use in throwaway scripts, interactive sessions etc. The convenience is in > that you don't need to specify a logger (the root logger is used) and that > basicConfig() is called for you. I've previously only used logging as an application author, hence I hadn't noticed some of the subtleties that come up when using logging in library code where you don't control which handlers are going to be installed at runtime. The surprise came from not realising there was a basicConfig() call hidden inside the convenience APIs, a fact which is *not* mentioned in the docstrings. It may be mentioned in the main documentation, but I didn't look at that at the time - there was nothing to suggest the docstrings were lacking pertinent information: >>> print(logging.warn.__doc__) Log a message with severity 'WARNING' on the root logger. >>> print(logging.getLogger().warn.__doc__) Log 'msg % args' with severity 'WARNING'. To pass exception information, use the keyword argument exc_info with a true value, e.g. logger.warning("Houston, we have a %s", "bit of a problem", exc_info=1) >> So my suggestion would be: >> 1. In the absence of a "config" call, make the "no handler" path in >> loggers emit messages *as if* basicConfig() has been called (without >> actually calling it) >> 2. Remove the implicit calls to basicConfig() from the module level >> convenience function >> >> How *feasible* that idea is to implement, I don't know. >> > > What about "Special cases aren't special enough to break the rules."? Why should > logging from stdlib modules be treated differently from the case of logging from > library modules in general? It could be argued that the behaviour you're > proposing is confusing/inconsistent to some people. I'm not proposing that the standard library be special-cased, I'm proposing that the default behaviour of an unconfigured logging module in general be changed to something more useful (i.e. warnings and errors printed to stderr, everything else suppressed), rather than unhelpfully suppressing *all* output except for an "Oh, I'm throwing output away" message the first time it happens. The specific use case that triggered my interest in this default behaviour is one where concurrent.futures *can't* raise an exception because it knows nothing is going to be in a position to handle that exception (due to the way callbacks are handled, c.f knows it provides the outermost exception handler in the affected call stack). Instead it has to catch the error and display it *somewhere* (i.e. it's very similar to the use cases for PyErr_WriteUnraisable at the C level). The options for handling this are: 1. Write the error detail directly to stderr. (Unless the default behaviour of logging changes, that is what I am going to suggest Brian do, as it exactly mimics the behaviour of the PyErr_WriteUnraisable API). 2. Write it to the root logger with the convenience APIs (Possible option, but I don't like the global state impact of implicitly calling basicConfig() from library code) 3. Add a StdErr handler for the logger (this is what is currently implemented, and again, I don't like it because of the global state impact on something that should be completely under an application's control) Basically, the current behaviour of logging is such that libraries *cannot* use it for unraisable warnings and error messages, as the messages will be suppressed unless the application takes steps to see them. That is OK for debug and info messages, but unacceptable for warnings and errors. A throwaway script using concurrent.futures needs to know if callbacks are failing, and that needs to happen without any logging related boilerplate in the script itself. If, however, an application completely lacking even a call to logging.basicConfig() would still see warnings and errors, then libraries could safely use the module without needing to worry about applications needing an particular boilerplate in order to see the unraisable errors and warnings that are emitted. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From g.brandl at gmx.net Wed Dec 8 09:09:45 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 08 Dec 2010 09:09:45 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <20101208010913.363c6105@pitrou.net> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: Am 08.12.2010 01:09, schrieb Antoine Pitrou: > On Tue, 7 Dec 2010 23:45:39 +0000 (UTC) > Vinay Sajip wrote: >> Antoine Pitrou pitrou.net> writes: >> >> > >> > I thought "error" and "critical" messages were logged to stderr by >> > default? Isn't it the case? >> > >> >> Only if you call basicConfig() or use the logging.debug(), logging.info(), etc. >> module-level convenience functions (which call basicConfig under the hood). > > Why wouldn't it be the default for all logging calls ? Such special > cases don't really make things easy to remember. > >> When is the NullHandler needed? Only for cases where an application developer >> uses a library which does logging under the covers (for those users who might be >> interested in logging its operations), but where that application developer >> doesn't use logging themselves for that application. > > You seem pretty tied up to the "application developer" situation. There > are cases (scripts, prototyping, etc.) where you certainly want to see > error messages (errors should not pass silently) but don't want to > configure logging for each of the libraries you use. But errors don't pass silently, do they? The usual way to present errors is still by raising exceptions. Georg From g.brandl at gmx.net Wed Dec 8 09:13:49 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 08 Dec 2010 09:13:49 +0100 Subject: [Python-Dev] [RELEASED] Python 3.2 beta 1 In-Reply-To: <433FCEE2-CD6C-48BD-90DF-2990B27E4EED@langa.pl> References: <4CFD59C8.4070006@python.org> <433FCEE2-CD6C-48BD-90DF-2990B27E4EED@langa.pl> Message-ID: Am 07.12.2010 09:24, schrieb ?ukasz Langa: > Wiadomo?? napisana przez Georg Brandl w dniu 2010-12-06, o godz. 22:46: > >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On behalf of the Python development team, I'm happy to announce the >> first of two beta preview releases of Python 3.2. >> >> Python 3.2 is a continuation of the efforts to improve and stabilize the >> Python 3.x line. Since the final release of Python 2.7, the 2.x line >> will only receive bugfixes, and new features are developed for 3.x only. >> >> Since PEP 3003, the Moratorium on Language Changes, is in effect, there >> are no changes in Python's syntax and built-in types in Python 3.2. >> Development efforts concentrated on the standard library and support for >> porting code to Python 3. Highlights are: >> >> [snip] > > * configparser 1.1: new API using the mapping protocol access, support for > pluggable interpolation handlers, additional interpolation handler > (ExtendedInterpolation) which supports the zc.buildout syntax, support for > alternative option/value delimiters, support for customization of accepted INI > file structure (e.g. comment prefixes, name of the DEFAULT section, indentation, > empty lines in multiline values, etc.), support for specifying encoding for read > operations, ConfigParser class deprecated in favor of SafeConfigParser, lots of > other small changes. Thanks, I've added this and the ssl module for the next announcement. Georg From vinay_sajip at yahoo.co.uk Wed Dec 8 09:32:58 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 08:32:58 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: Nick Coghlan gmail.com> writes: > The surprise came from not realising there was a basicConfig() call > hidden inside the convenience APIs, a fact which is *not* mentioned in > the docstrings. It may be mentioned in the main documentation, but I > didn't look at that at the time - there was nothing to suggest the > docstrings were lacking pertinent information: > You're right that this is missing from the docstrings; I'll rectify that. > I'm not proposing that the standard library be special-cased, I'm > proposing that the default behaviour of an unconfigured logging module > in general be changed to something more useful (i.e. warnings and > errors printed to stderr, everything else suppressed), rather than > unhelpfully suppressing *all* output except for an "Oh, I'm throwing > output away" message the first time it happens. I don't have a philosophical problem with this, but there might be differing opinions about this not just on python-dev but also in the wider community. I'd definitely want logging to do the right thing and minimize inconvenience as much as possible. There's also possibly a backwards compatibility dimension to this, but I'm not sure to what extent a change like this would really affect people in practice (as nothing will change if logging is configured). > The specific use case that triggered my interest in this default > behaviour is one where concurrent.futures *can't* raise an exception > because it knows nothing is going to be in a position to handle that > exception (due to the way callbacks are handled, c.f knows it provides > the outermost exception handler in the affected call stack). Instead > it has to catch the error and display it *somewhere* (i.e. it's very > similar to the use cases for PyErr_WriteUnraisable at the C level). > > The options for handling this are: > > 1. Write the error detail directly to stderr. (Unless the default > behaviour of logging changes, that is what I am going to suggest Brian > do, as it exactly mimics the behaviour of the PyErr_WriteUnraisable > API). > 2. Write it to the root logger with the convenience APIs (Possible > option, but I don't like the global state impact of implicitly calling > basicConfig() from library code) > 3. Add a StdErr handler for the logger (this is what is currently > implemented, and again, I don't like it because of the global state > impact on something that should be completely under an application's > control) > > Basically, the current behaviour of logging is such that libraries > *cannot* use it for unraisable warnings and error messages, as the > messages will be suppressed unless the application takes steps to see > them. That is OK for debug and info messages, but unacceptable for > warnings and errors. A throwaway script using concurrent.futures needs > to know if callbacks are failing, and that needs to happen without any > logging related boilerplate in the script itself. > > If, however, an application completely lacking even a call to > logging.basicConfig() would still see warnings and errors, then > libraries could safely use the module without needing to worry about > applications needing an particular boilerplate in order to see the > unraisable errors and warnings that are emitted. > Thanks for the detailed explanation. I agree that unraisable warnings and errors need to be handled somehow. There is a way in which this can be done without affecting a logging configuration, viz. logging can define a "handler of last resort" (not attached to any logger) which is invoked when there are no user-specified handlers. This would by default be a StreamHandler writing to sys.stderr with a threshold of WARNING (or perhaps ERROR). Thus sounds like a better option than a direct write to sys.stderr, since you can't change the latter behaviour easily if you want to do something else instead. This is of course a backwards-incompatible change to logging semantics: instead of saying that logging will be silent unless explicitly asked to produce output, we're saying that logging will always produce output for warnings and errors (or perhaps just errors), unless explicitly silenced. This is of course in line with the Zen of Python; the present behaviour, which is not so aligned, is based on the idea that logging should not affect program behaviour if it's not wanted by the program developer (as opposed to library developer). It would also mean changing the documentation about NullHandler to say: "If you have messages which must get out when you can't raise an exception, then don't add a NullHandler to your top-level loggers." Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Dec 8 09:51:44 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 08:51:44 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: Message-ID: Robert Kern gmail.com> writes: > > I really don't understand how this view can be consistent with the > practice of adding NullHandler to loggers. If this message is so important > to prevent misconfiguration, then why should a library author decide to > silence it for his users? Because the application developer knows more about the end-user audience for their application, they are better placed to know how logging should work for their application. It's not an error for a particular application developer to decide that nothing should be produced by logging for a particular application; they (particularly when casual users) would be confused by the misconfiguration message due to logging by a library they're using. The library author's users are the application developers who use the library, not the end users who use their applications. Sometimes they're the same people, I know, but I just think of them as wearing different hats :-) > [...] I strongly suspect that almost all configurations include a > catch-all root logger and that most of those *only* consist of that > root logger. That doesn't seem right: your comment might be conflating loggers with handlers. The common pattern would be (or should be) to name loggers according to __name__ in the modules which use logging, but only configure *handlers* for the root logger. That way, logging messages indicate their origin (because of the __name__ convention) but you only need to add handlers at the root logger to capture all the logging information. > Same mistake. I intended the correction to apply to all such statements in > my post. Oh, right. I thought this was a different case. > I think that boilerplate should be minimized. If using getLogger() should > almost always be followed by adding a NullHandler, then it should be the > default behavior. The easiest way to achieve this effect is to simply not > issue the warning message. getLogger() should NOT "almost always be followed by adding a NullHandler". For example, in Django, only the logger named "django" would have that handler added; no other Django logger (e.g. "django.db.models") would need to have that handler added. Regards, Vinay Sajip From solipsis at pitrou.net Wed Dec 8 10:41:45 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Dec 2010 10:41:45 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: <20101208104145.41ed925b@pitrou.net> On Wed, 8 Dec 2010 01:51:44 +0000 (UTC) Vinay Sajip wrote: > > > Adding a NullHandler isn't the right thing to do - the behaviour I > > would want for any standard library logging that hasn't been > > explicitly configured otherwise is to do what the root logger does > > under basicConfig(): debug() and info() get suppressed, warn(), > > error(), critical() and exception() go to stderr. It seems to me like > > that would be more useful default behaviour in general than emitting a > > warning about not finding a handler. > > > > So my suggestion would be: > > 1. In the absence of a "config" call, make the "no handler" path in > > loggers emit messages *as if* basicConfig() has been called (without > > actually calling it) > > 2. Remove the implicit calls to basicConfig() from the module level > > convenience function > > > > How *feasible* that idea is to implement, I don't know. > > > > What about "Special cases aren't special enough to break the rules."? Well, it seems we are asking to remove the special cases, since we are asking for the special case to become the norm. Antoine. From solipsis at pitrou.net Wed Dec 8 10:42:59 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Dec 2010 10:42:59 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: <20101208104259.70f5699a@pitrou.net> On Wed, 08 Dec 2010 09:09:45 +0100 Georg Brandl wrote: > Am 08.12.2010 01:09, schrieb Antoine Pitrou: > > On Tue, 7 Dec 2010 23:45:39 +0000 (UTC) > > Vinay Sajip wrote: > >> Antoine Pitrou pitrou.net> writes: > >> > >> > > >> > I thought "error" and "critical" messages were logged to stderr by > >> > default? Isn't it the case? > >> > > >> > >> Only if you call basicConfig() or use the logging.debug(), logging.info(), etc. > >> module-level convenience functions (which call basicConfig under the hood). > > > > Why wouldn't it be the default for all logging calls ? Such special > > cases don't really make things easy to remember. > > > >> When is the NullHandler needed? Only for cases where an application developer > >> uses a library which does logging under the covers (for those users who might be > >> interested in logging its operations), but where that application developer > >> doesn't use logging themselves for that application. > > > > You seem pretty tied up to the "application developer" situation. There > > are cases (scripts, prototyping, etc.) where you certainly want to see > > error messages (errors should not pass silently) but don't want to > > configure logging for each of the libraries you use. > > But errors don't pass silently, do they? The usual way to present errors > is still by raising exceptions. Or logging them. http://docs.python.org/dev/library/logging.html#logging.Logger.exception From solipsis at pitrou.net Wed Dec 8 11:03:09 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Dec 2010 11:03:09 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: <20101208110309.56cf2a22@pitrou.net> On Wed, 8 Dec 2010 01:19:45 +0000 (UTC) Vinay Sajip wrote: > Antoine Pitrou pitrou.net> writes: > > > > > Why wouldn't it be the default for all logging calls ? Such special > > cases don't really make things easy to remember. > > > > One size doesn't fit all. Everything's documented reasonably well. If you use it > often, you remember it. If you use it seldom, you might need to look things up. My point is that the default behaviour should be helpful. I would point out that logging is really the kind of thing people don't want to spend time with. I know it's not very gratifying for the module author (you!), but nobody expects (*) to have to first configure a logging library when they want to use a third-party library. They expect it to work *and* give useful error report by default. (*) (not in the Python land anyway; Java might be different) > The defaults are I believe reasonably convenient, or at least not inconvenient: > > 1. If writing a library, add a NullHandler to the top-level logger of your > library, don't add any other handlers by default (i.e. on import), document the > names of the loggers you use. Otherwise, see the following. First, you cannot call it a "default", since the library writer has to make it explicit. Second, I don't find that convenient at all. When I use a third-party lib I want the errors to be displayed, not silenced. I'm willing to bet that most people have the same expectation. *Especially* when prototyping, since you're not sure you've got everything right. So, that's two problems: - the library author has to explicit activate what you argue is a "default" behaviour (otherwise the library user gets an unhelpful message that handlers are not configured) - the library user still has to override that "default" since it's not satisfying for most uses There's a third problem with it: - all this is not easily discoverable. You cannot expect each user of a third-party lib to *also* read the logging docs in addition to the third-party lib docs. On my Web browser, http://docs.python.org/dev/library/logging.html seems to be at least 50 pages long. For a comparison, the warnings module logs all warnings by default on stderr without requiring any a priori configuration. And that's what stderr is for (reporting errors). > 2. If writing a simple script, prototyping etc. i.e. for casual use, use > logging.debug(), logging.info() etc. That's easy enough and convenient enough > for casual users to remember, I hope. All WARNING, ERROR and CRITICAL events in > your code and in that of any libraries you use will be printed to sys.stderr. Indeed, this is convenient. And that's what I'm arguing should be the default for *all* loggers, not just the top-level one. There's no reason that what is convenient for the root logger isn't for other loggers. > 3. If slightly less than very casual use (e.g. you want to log to file, or want > a specific message format), call basicConfig(). Just the one extra line to > remember to do, hopefully not too inconvenient. I know that one line sounds "not too inconvenient" since you are the library author and you are so used to it (and certainly like using it and spend a lot of time improving its features, which is good). I think most people, however, have different expectations (see beginning of this message). > 4. For more advanced use, use programmatic configuration, fileConfig or > dictConfig to specify a configuration. Convenience is inversely proportional to > how involved the configuration is. That I have already noticed :) Regards Antoine. From ncoghlan at gmail.com Wed Dec 8 11:29:57 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 8 Dec 2010 20:29:57 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: On Wed, Dec 8, 2010 at 6:32 PM, Vinay Sajip wrote: > Thanks for the detailed explanation. I agree that unraisable warnings and errors > need to be handled somehow. There is a way in which this can be done without > affecting a logging configuration, viz. logging can define a "handler of last > resort" (not attached to any logger) which is invoked when there are no > user-specified handlers. This would by default be a StreamHandler writing to > sys.stderr with a threshold of WARNING (or perhaps ERROR). A "handler of last resort" is exactly the kind of concept I had in mind. Just not expressed quite so succinctly :) > Thus sounds like a > better option than a direct write to sys.stderr, since you can't change the > latter behaviour easily if you want to do something else instead. Yeah, this is why I can understand Brian's desire to use the logging module rather than a direct write to stderr in the futures implementation. > This is of course a backwards-incompatible change to logging semantics: instead > of saying that logging will be silent unless explicitly asked to produce output, > we're saying that logging will always produce output for warnings and errors (or > perhaps just errors), unless explicitly silenced. This is of course in line with > the Zen of Python; the present behaviour, which is not so aligned, is based on > the idea that logging should not affect program behaviour if it's not wanted by > the program developer (as opposed to library developer). Libraries that write directly to stderr affect program behaviour as well - at least with logging, the application developer will have an easier time of silencing (or redirecting) them. Also, in situations where this default handler will be used, logging currently produces output at least once (the warning that it couldn't find a handler to use). So while it is a behavioural change, it is one that should be acceptable from a compatibility point of view. > It would also mean changing the documentation about NullHandler to say: "If you > have messages which must get out when you can't raise an exception, then don't > add a NullHandler to your top-level loggers." Agreed. We could also make it easy to replace the handler of last resort so people can easily recover the old behaviour (by inserting NullHandler in place of the default last resort handler). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From g.brandl at gmx.net Wed Dec 8 11:48:16 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 08 Dec 2010 11:48:16 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <20101208104259.70f5699a@pitrou.net> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208104259.70f5699a@pitrou.net> Message-ID: Am 08.12.2010 10:42, schrieb Antoine Pitrou: >> But errors don't pass silently, do they? The usual way to present errors >> is still by raising exceptions. > > Or logging them. > http://docs.python.org/dev/library/logging.html#logging.Logger.exception Yes, thank you I'm aware of the exception() method. But hopefully standard library modules don't use it to report exceptions to code that uses them? Georg From solipsis at pitrou.net Wed Dec 8 12:12:20 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Dec 2010 12:12:20 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208104259.70f5699a@pitrou.net> Message-ID: <20101208121220.1ac1ed97@pitrou.net> On Wed, 08 Dec 2010 11:48:16 +0100 Georg Brandl wrote: > Am 08.12.2010 10:42, schrieb Antoine Pitrou: > > >> But errors don't pass silently, do they? The usual way to present errors > >> is still by raising exceptions. > > > > Or logging them. > > http://docs.python.org/dev/library/logging.html#logging.Logger.exception > > Yes, thank you I'm aware of the exception() method. But hopefully standard > library modules don't use it to report exceptions to code that uses them? I'm not aware of that, but there are certainly third-party libs using it (think an HTTP server that wants to log an error in one of its request handlers without the error taking the whole server down). Antoine. From vinay_sajip at yahoo.co.uk Wed Dec 8 13:15:53 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 12:15:53 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > > My point is that the default behaviour should be helpful. > I can't disagree with that. Now if only we could agree what we mean by "default" and "helpful" ;-) > I would point out that logging is really the kind of thing people don't > want to spend time with. I know it's not very gratifying for the module > author (you!), but nobody expects (*) to have to first configure a Logging's just a tool - I completely understand that it's just an adjunct to the interesting stuff. Doesn't bother me a bit :-) > logging library when they want to use a third-party library. They > expect it to work *and* give useful error report by default. > > (*) (not in the Python land anyway; Java might be different) The Java thing is a red herring, I believe. It's more akin to the Unix idea of minimum verbosity as a default. In general, you don't need to "configure a library" except when you want specific behaviour from that library's loggers. > First, you cannot call it a "default", since the library writer has to > make it explicit. > Second, I don't find that convenient at all. When I use a third-party > lib I want the errors to be displayed, not silenced. I'm willing to bet > that most people have the same expectation. *Especially* when > prototyping, since you're not sure you've got everything right. > > So, that's two problems: > - the library author has to explicit activate what you argue is a > "default" behaviour (otherwise the library user gets an unhelpful > message that handlers are not configured) > - the library user still has to override that "default" since it's not > satisfying for most uses > See my comments to Nick Coghlan's post about getting messages out when you can't raise an exception. I think the case is not as common as you suggest (because in many instances, you would raise an exception to signal an error). I'm not opposed to changing the way things work, but I think we need more of a consensus because some backwards incompatibility will result, albeit not likely to affect too many users of logging. > There's a third problem with it: > - all this is not easily discoverable. You cannot expect each user of a > third-party lib to *also* read the logging docs in addition to the > third-party lib docs. On my Web browser, > http://docs.python.org/dev/library/logging.html seems to be at least > 50 pages long. You're complaining about too much documentation?! Don't measure it by weight! Seriously, I don't see what's wrong with a library developer reading the documentation for any module they're using, logging included. "Configuring logging for a library" is easily visible in the sidebar on that page. So I wouldn't agree that "it's not easily discoverable". It's not exactly buried. And it's the library developer who has to read it, not the user of the library. The user of the library only needs to know what logger names are used by that library, and that too only if they want to configure those loggers. > For a comparison, the warnings module logs all warnings by default on > stderr without requiring any a priori configuration. And that's what > stderr is for (reporting errors). > But that behaviour is *not* always wanted. That's why I was asked to add warnings integration to logging - to redirect warning messages to logging. Things are seldom as black and white as you're making them out to be here. > Indeed, this is convenient. And that's what I'm arguing should be the > default for *all* loggers, not just the top-level one. There's no > reason that what is convenient for the root logger isn't for other > loggers. Loggers work in a hierarchy, and the package is specifically designed so that you can attach handlers just to the root logger, and get messages from all code in your application + its libraries to destinations that you configure there. This is not special case behaviour for root logger versus other loggers - it's just that the root logger is a convenient, logical place to attach handlers. > I know that one line sounds "not too inconvenient" since you are the > library author and you are so used to it (and certainly like using it > and spend a lot of time improving its features, which is good). > I think most people, however, have different expectations (see > beginning of this message). Hey, that's not why I think "not too inconvenient"! It's because (a) It's no less inconvenient than "import foo", and it's hard to argue that one line is really inconvenient for this case - remember, for the simplest uses you can omit the basicConfig() call, since you can use logging.info() etc. (b) For this case, since there's no telepathy in this world, you need some way for the programmer to communicate their intent about the logging level, or format string, or log file that they want to use. I stand by the statement that one line (or more precisely, one function call) is not too much to ask here, so yes - "not too inconvenient". That's emphatically not because I'm the module's author and "used to it". Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Dec 8 13:22:07 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 12:22:07 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208104259.70f5699a@pitrou.net> <20101208121220.1ac1ed97@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > > On Wed, 08 Dec 2010 11:48:16 +0100 > Georg Brandl gmx.net> wrote: But hopefully standard > > library modules don't use it to report exceptions to code that uses them? > > I'm not aware of that, but there are certainly third-party libs using > it (think an HTTP server that wants to log an error in one of its > request handlers without the error taking the whole server down). > That's not the same thing as Georg is talking about, IIUC. The exception() method is used in exception *handler* code to record that the exception occurred, but the correct thing for any code to do when an error condition is detected (in most situations at least) is to raise an exception. It would be quite wrong for code to e.g. call logger.error(...) instead of raise ... Regards, Vinay Sajip From p.f.moore at gmail.com Wed Dec 8 14:02:00 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Dec 2010 13:02:00 +0000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: On 8 December 2010 08:32, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: >> I'm not proposing that the standard library be special-cased, I'm >> proposing that the default behaviour of an unconfigured logging module >> in general be changed to something more useful (i.e. warnings and >> errors printed to stderr, everything else suppressed), rather than >> unhelpfully suppressing *all* output except for an "Oh, I'm throwing >> output away" message the first time it happens. > > I don't have a philosophical problem with this, but there might be differing > opinions about this not just on python-dev but also in the wider community. I'd > definitely want logging to do the right thing and minimize inconvenience as much > as possible. There's also possibly a backwards compatibility dimension to this, > but I'm not sure to what extent a change like this would really affect people in > practice (as nothing will change if logging is configured). To provide an application developer point of view, I'm +1 on this change. I want library developers to be able to report errors/warnings and have them appear by default, without me doing anything (or even needing to be aware that the logging module is involved). The benefit of the library developer using the logging module to me is that if I deem it appropriate, I can direct the warnings/errors elsewhere simply by configuring logging. So, I'd like the library developer to feel that they can use logging in precisely the same ways as they would write to sys.stderr, and with the same expectations (output will be seen by default), but with less reluctance because they know that I can easily adapt their output according to my needs. >> The options for handling this are: >> >> 1. Write the error detail directly to stderr. (Unless the default >> behaviour of logging changes, that is what I am going to suggest Brian >> do, as it exactly mimics the behaviour of the PyErr_WriteUnraisable >> API). I find library code that does this annoying. It's the right effect, but leaves me with little or no control should I want to do something like log to a file. >> 2. Write it to the root logger with the convenience APIs (Possible >> option, but I don't like the global state impact of implicitly calling >> basicConfig() from library code) Library code shouldn't use the root logger, and shouldn't call basicConfig - I agree with Nick again. >> 3. Add a StdErr handler for the logger (this is what is currently >> implemented, and again, I don't like it because of the global state >> impact on something that should be completely under an application's >> control) That seems like a best of a bad job answer. As an application user, I'd expect to find it awkward to modify this default, purely because I have to undo the configuration that the library had to do to work around the logging library's defaults. > Thanks for the detailed explanation. I agree that unraisable warnings and errors > need to be handled somehow. There is a way in which this can be done without > affecting a logging configuration, viz. logging can define a "handler of last > resort" (not attached to any logger) which is invoked when there are no > user-specified handlers. This would by default be a StreamHandler writing to > sys.stderr with a threshold of WARNING (or perhaps ERROR). Thus sounds like a > better option than a direct write to sys.stderr, since you can't change the > latter behaviour easily if you want to do something else instead. >From my POV as an application writer, I don't care how you implement it (but I agree that your suggestion sounds more flexible). As long as errors and warnings are printed by default, and I can set my own configuration without having to undo the existing configuration, then I'm happy. > This is of course a backwards-incompatible change to logging semantics: instead > of saying that logging will be silent unless explicitly asked to produce output, > we're saying that logging will always produce output for warnings and errors (or > perhaps just errors), unless explicitly silenced. This is of course in line with > the Zen of Python; the present behaviour, which is not so aligned, is based on > the idea that logging should not affect program behaviour if it's not wanted by > the program developer (as opposed to library developer). I can see that it's an incompatible change. But I'd still be in favour of it for 3.2. In my view, it needs to be done, and it'll only get harder to do as time goes on. Paul. From p.f.moore at gmail.com Wed Dec 8 14:19:26 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Dec 2010 13:19:26 +0000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> Message-ID: On 8 December 2010 12:15, Vinay Sajip wrote: > The Java thing is a red herring, I believe. It's more akin to the Unix idea of > minimum verbosity as a default. And yet Unix invented the concept of stderr, precisely to ensure that there's a route for things the program wants to inform the user about to get out. The NullHandler approach could be seen as the equivalent of adding 2>/dev/null to every command by default. > In general, you don't need to "configure a > library" except when you want specific behaviour from that library's loggers. The "specific behaviour" I want is for errors and warnings to go to stderr. Every time. (Unless I'm writing a service that wants to redirect all output to a log file, or something like that - and that's where I'm really glad to have the logging module! And the fact that the defaults tend to encourage library authors not to use it really frustrate me as an applicatioo developer.) > I'm not opposed to changing the way things work, but I think we need more of a > consensus because some backwards incompatibility will result, albeit not likely > to affect too many users of logging. Hmm. As far as I can tell from reading the thread, everyone but you is in favour of errors and warnings going to stderr by default. I may have missed some dissenting views, but it certainly feels like a consensus to me! If you feel that the proposal needs to be presented to a wider group than just python-dev then that's fair. But personally, I think that python-dev will be a reasonably good indicator in this case. >> For a comparison, the warnings module logs all warnings by default on >> stderr without requiring any a priori configuration. And that's what >> stderr is for (reporting errors). >> > > But that behaviour is *not* always wanted. That's why I was asked to add > warnings integration to logging - to redirect warning messages to logging. > Things are seldom as black and white as you're making them out to be here. Exactly!!!! Logging should be a BETTER option than direct writes to stderr. At the moment, though, it's a CHOICE - writes to stderr, which are visible by default but (very) inflexible, or logging which is flexible but suppressed by default (OK, "if you follow the recommended practice to avoid a user-hostile warning"). The problem is that some people are voting with their feet that visibility matters more than flexibility. >> Indeed, this is convenient. And that's what I'm arguing should be the >> default for *all* loggers, not just the top-level one. There's no >> reason that what is convenient for the root logger isn't for other >> loggers. > > Loggers work in a hierarchy, and the package is specifically designed so that > you can attach handlers just to the root logger, and get messages from all code > in your application + its libraries to destinations that you configure there. > This is not special case behaviour for root logger versus other loggers - it's > just that the root logger is a convenient, logical place to attach handlers. But you don't because the library developer added a NullHandler which you have to switch off!!! > Hey, that's not why I think "not too inconvenient"! It's because > > (a) It's no less inconvenient than "import foo", and it's hard to argue that one > line is really inconvenient for this case - remember, for the simplest uses you > can omit the basicConfig() call, since you can use logging.info() etc. > > (b) For this case, since there's no telepathy in this world, you need some way > for the programmer to communicate their intent about the logging level, or > format string, or log file that they want to use. I stand by the statement that > one line (or more precisely, one function call) is not too much to ask here, so > yes - "not too inconvenient". That's emphatically not because I'm the module's > author and "used to it". FWIW, basicConfig has *never* been what I want. Except in the sense that I have to use it to switch on logging to stderr (if only it worked, but see the comment above about NullHandler) because that's not the default. In all other cases, I've wanted a little bit more (which may not be hard to achieve, but that's not the point here). Saying that's convenient is like saying that adding sys.stderr=open("/dev/tty", "w") is a convenient way of getting errors displayed when you don't like the default destination of /dev/null. Paul. From g.brandl at gmx.net Wed Dec 8 14:33:04 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 08 Dec 2010 14:33:04 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208104259.70f5699a@pitrou.net> <20101208121220.1ac1ed97@pitrou.net> Message-ID: Am 08.12.2010 13:22, schrieb Vinay Sajip: > Antoine Pitrou pitrou.net> writes: > >> >> On Wed, 08 Dec 2010 11:48:16 +0100 >> Georg Brandl gmx.net> wrote: > But hopefully standard >> > library modules don't use it to report exceptions to code that uses them? >> >> I'm not aware of that, but there are certainly third-party libs using >> it (think an HTTP server that wants to log an error in one of its >> request handlers without the error taking the whole server down). >> > > That's not the same thing as Georg is talking about, IIUC. The exception() > method is used in exception *handler* code to record that the exception > occurred, but the correct thing for any code to do when an error condition is > detected (in most situations at least) is to raise an exception. It would be > quite wrong for code to e.g. call logger.error(...) instead of raise ... Exactly. The HTTP server is of course a good example of an application, and one of the most obvious candidates in the stdilb to use the logging module instead of sys.stderr.write(). Georg From vinay_sajip at yahoo.co.uk Wed Dec 8 14:57:40 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 13:57:40 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> Message-ID: Paul Moore gmail.com> writes: > And yet Unix invented the concept of stderr, precisely to ensure that > there's a route for things the program wants to inform the user about > to get out. > > The NullHandler approach could be seen as the equivalent of adding > 2>/dev/null to every command by default. > Not quite true. The program can (as of right now) ensure that all warnings and messages get directed to stderr, simply by calling basicConfig(). That'll work even if every library has added a NullHandler. Although you can argue that the call to basicConfig() shouldn't be necessary, you tone comes across as a bit of an overreaction. > The "specific behaviour" I want is for errors and warnings to go to > stderr. Every time. (Unless I'm writing a service that wants to > redirect all output to a log file, or something like that - and that's > where I'm really glad to have the logging module! And the fact that > the defaults tend to encourage library authors not to use it really > frustrate me as an applicatioo developer.) > I don't see where your assertion that "the defaults tend to encourage library developers not to use it" comes from. If it's Antoine's post in this thread, then I believe he was referring to the stdlib rather than libraries in general, and I would also say that I don't believe it's a major factor; I would say that inertia was a more pertinent factor. Of course I understand I could be wrong about this, but I don't recall when a stdlib maintainer has said to me, "I want to start using logging in stdlib module XXX, but can't justify it because ..." I doubt whether making this change will result in a flurry of stdlib modules which suddenly adopt logging, but stdlib maintainers can certainly prove me wrong easily enough by stating otherwise in response to this. It's not as if I'm not approachable - you will see that many improvements to the logging package exist because of user feedback, both from python-dev and elsewhere. > Hmm. As far as I can tell from reading the thread, everyone but you is > in favour of errors and warnings going to stderr by default. I may > have missed some dissenting views, but it certainly feels like a > consensus to me! "Everyone but you" - I think you mean "everyone on this thread but you", and "everyone else on this thread" is not remotely close to "everyone else, period". > If you feel that the proposal needs to be presented to a wider group > than just python-dev then that's fair. But personally, I think that > python-dev will be a reasonably good indicator in this case. I don't dispute that, but there's not really been enough time for more than a few people on python-dev to respond. > > But that behaviour is *not* always wanted. That's why I was asked to add > > warnings integration to logging - to redirect warning messages to logging. > > Things are seldom as black and white as you're making them out to be here. > > Exactly!!!! Logging should be a BETTER option than direct writes to > stderr. At the moment, though, it's a CHOICE - writes to stderr, which > are visible by default but (very) inflexible, or logging which is > flexible but suppressed by default (OK, "if you follow the recommended > practice to avoid a user-hostile warning"). The problem is that some > people are voting with their feet that visibility matters more than > flexibility. > I don't think it's unreasonable to ask which people are voting with their feet, that you know of. This discussion was initiated by me after I got added to the nosy list of an issue. It's not as if there has been a host of voters who have asked for change and are walking away because of some intransigence on my part. As you can see in my response to Nick's post, I am making concrete suggestions on how the situation could be improved. It's not helping to talk about people voting with their feet, especially as no serious attempt to engage with me on this issue has been made in the past. Because of backward-compatibility considerations, I would consider it premature to make this kind of change just based on a request by one or two python-dev members. I have the highest level of respect for people who have responded on this thread, but how can I be sure that they're representing everyone else on this list? It's a great community, with a lot of cohesion, but people do have divergent opinions. (I'm not talking about me here. If there's a consensus about this, I've no problem implementing a "handler of last resort".) > But you don't because the library developer added a NullHandler which > you have to switch off!!! > Where are you getting this "switch off" from? There's no switching off that happens. The NullHandler remains in place. What you are doing is "switching on" output to some specific place you want - console, file etc. - with a level of verbosity and format that you choose. > FWIW, basicConfig has *never* been what I want. Except in the sense > that I have to use it to switch on logging to stderr (if only it > worked, but see the comment above about NullHandler) because that's > not the default. In all other cases, I've wanted a little bit more > (which may not be hard to achieve, but that's not the point here). Well, "may not be hard to achieve" is in line with "not too inconvenient", if you ask me. And as for "if only it worked", that makes it sound like there's some effect you can't achieve at very modest cost in terms of any code you have to write. > Saying that's convenient is like saying that adding > sys.stderr=open("/dev/tty", "w") is a convenient way of getting errors > displayed when you don't like the default destination of /dev/null. I don't think I'm saying that at all. I see where you're coming from, but apart from stating your point of view, IMO you're not adding to the discussion in the same way that Nick, say, has. Regards, Vinay Sajip From fdrake at acm.org Wed Dec 8 15:09:59 2010 From: fdrake at acm.org (Fred Drake) Date: Wed, 8 Dec 2010 09:09:59 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> Message-ID: On Wed, Dec 8, 2010 at 8:19 AM, Paul Moore wrote: > But you don't because the library developer added a NullHandler which > you have to switch off!!! I'm suspecting there's misunderstanding on this point. If I have a logger "myns.mypackage" for my library, and register a NullHandler for that, that doesn't need to be turned off for applications. What it accomplishes is that for all messages sent to the "myns.mypackage" logger (and descendants) are handled by at least one handler. Log records are still propagated toward the root for outer loggers to handle. An application can also still choose to install additional handlers for "myns.mypackage" if desired, and do whatever output handling is appropriate. The library-installed NullHandler on a library-scoped logger simply ensures that the library doesn't trigger the message about un-handled log records. We seem to have consensus on not wanting that message triggered for library logging is desirable, and this NullHandler trick provides a way to handle that independent from other possible changes. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From solipsis at pitrou.net Wed Dec 8 15:27:59 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Dec 2010 15:27:59 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> Message-ID: <20101208152759.32b53cb2@pitrou.net> On Wed, 8 Dec 2010 12:15:53 +0000 (UTC) Vinay Sajip wrote: > > First, you cannot call it a "default", since the library writer has to > > make it explicit. > > Second, I don't find that convenient at all. When I use a third-party > > lib I want the errors to be displayed, not silenced. I'm willing to bet > > that most people have the same expectation. *Especially* when > > prototyping, since you're not sure you've got everything right. > > > > So, that's two problems: > > - the library author has to explicit activate what you argue is a > > "default" behaviour (otherwise the library user gets an unhelpful > > message that handlers are not configured) > > - the library user still has to override that "default" since it's not > > satisfying for most uses > > > > See my comments to Nick Coghlan's post about getting messages out when you can't > raise an exception. I think the case is not as common as you suggest (because in > many instances, you would raise an exception to signal an error). I'm not talking specifically about exceptions, but about errors in general. If the case wasn't common, I'm not sure why the error() and critical() methods would exist at all. (of course I'm assuming error() is meant to output error messages. If that's a wrong interpretation then I'm a bit puzzled :-)) > > There's a third problem with it: > > - all this is not easily discoverable. You cannot expect each user of a > > third-party lib to *also* read the logging docs in addition to the > > third-party lib docs. On my Web browser, > > http://docs.python.org/dev/library/logging.html seems to be at least > > 50 pages long. > > You're complaining about too much documentation?! Don't measure it by weight! > Seriously, I don't see what's wrong with a library developer reading the > documentation for any module they're using, logging included. I was talking about the user of a library, not its developer (see the snippet quoted above). > The > user of the library only needs to know what logger names are used by that > library, and that too only if they want to configure those loggers. The thing is, they don't *want* to configure them, but you force them to do some configuration if they don't want error messages to be silenced. > > For a comparison, the warnings module logs all warnings by default on > > stderr without requiring any a priori configuration. And that's what > > stderr is for (reporting errors). > > > > But that behaviour is *not* always wanted. Sure. What I'm arguing is that the warnings module's default behaviour is much more useful than the logging module's default behaviour. > That's why I was asked to add > warnings integration to logging - to redirect warning messages to logging. That's a separate topic :) > (b) For this case, since there's no telepathy in this world, you need some way > for the programmer to communicate their intent about the logging level, or > format string, or log file that they want to use. Well, the programmer doesn't have to communicate their intent about what and how warnings get displayed by default, and I'm not sure the warnings module has a telepathy routine inside (although you never know, since IIRC Brett did some philosophy studies). Regards Antoine. From fdrake at acm.org Wed Dec 8 15:43:09 2010 From: fdrake at acm.org (Fred Drake) Date: Wed, 8 Dec 2010 09:43:09 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <20101208152759.32b53cb2@pitrou.net> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: On Wed, Dec 8, 2010 at 9:27 AM, Antoine Pitrou wrote: > The thing is, they don't *want* to configure them, but you force them > to do some configuration if they don't want error messages to be > silenced. As I tried to explain earlier, a NullHandler doesn't silence anything except the message about logging not being configured. Propagation is not controlled by the handlers, but by the loggers. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From vinay_sajip at yahoo.co.uk Wed Dec 8 15:54:09 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 14:54:09 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > I'm not talking specifically about exceptions, but about errors in > general. If the case wasn't common, I'm not sure why the error() and > critical() methods would exist at all. > > (of course I'm assuming error() is meant to output error messages. If > that's a wrong interpretation then I'm a bit puzzled ) > Then again, it's not always helpful when *library* code prints out messages to stderr. A user of the library might prefer that an error code be returned or an exception raised, so they can deal with the condition most appropriately. > I was talking about the user of a library, not its developer (see the > snippet quoted above). Yes, but see my response to that. A user of a library doesn't need to know the specifics of what loggers that library uses, they just need to make a positive statement of where they want warning and error messages go, and how they want them formatted. BTW we haven't discussed this yet - but these enabled-by-default messages, what format should they have? (e.g. just message, logger name + message, logger name + severity + message etc.) It might sound like a trivial point, but it also seems like whatever you pick, some bikeshed discussions would ensue. At present, the developer is encouraged to choose (as they have to call basicConfig anyway), avoiding that bikeshedding altogether. > The thing is, they don't *want* to configure them, but you force them > to do some configuration if they don't want error messages to be > silenced. > True, but it need be no more than a call to basicConfig(), so "not too inconvenient" ;-) > Sure. What I'm arguing is that the warnings module's default behaviour > is much more useful than the logging module's default behaviour. > Sure, that's what we're discussing, all right. >> That's why I was asked to add >> warnings integration to logging - to redirect warning messages to logging. > > That's a separate topic :) > If you say so ;-) >> (b) For this case, since there's no telepathy in this world, you need some >> way for the programmer to communicate their intent about the logging level, >> or format string, or log file that they want to use. > > Well, the programmer doesn't have to communicate their intent about > what and how warnings get displayed by default, and I'm not sure the > warnings module has a telepathy routine inside (although you never > know, since IIRC Brett did some philosophy studies). See my earlier comment about formatting and the potential for bikeshedding ;-) Regards, Vinay Sajip From ncoghlan at gmail.com Wed Dec 8 15:56:14 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Dec 2010 00:56:14 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: On Thu, Dec 9, 2010 at 12:43 AM, Fred Drake wrote: > On Wed, Dec 8, 2010 at 9:27 AM, Antoine Pitrou wrote: >> The thing is, they don't *want* to configure them, but you force them >> to do some configuration if they don't want error messages to be >> silenced. > > As I tried to explain earlier, a NullHandler doesn't silence anything > except the message about logging not being configured. Propagation is > not controlled by the handlers, but by the loggers. If the only handler is NullHandler though, (or if there are no handlers configured at all) then output does get silenced by default, even errors and warning messages. The application has to (directly or indirectly) configure logging with additional handlers in order for the output to appear somewhere. That's the behaviour that is inappropriate for use cases like the concurrent.futures handling of unraisable errors that started this discussion over on the bug tracker. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Dec 8 15:52:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Dec 2010 00:52:52 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <20101208152759.32b53cb2@pitrou.net> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: On Thu, Dec 9, 2010 at 12:27 AM, Antoine Pitrou wrote: > On Wed, 8 Dec 2010 12:15:53 +0000 (UTC) > Vinay Sajip wrote: >> See my comments to Nick Coghlan's post about getting messages out when you can't >> raise an exception. I think the case is not as common as you suggest (because in >> many instances, you would raise an exception to signal an error). > > I'm not talking specifically about exceptions, but about errors in > general. If the case wasn't common, I'm not sure why the error() and > critical() methods would exist at all. > > (of course I'm assuming error() is meant to output error messages. If > that's a wrong interpretation then I'm a bit puzzled :-)) As I see it, there aren't many cases at the *library* level where logging errors is more appropriate than raising exceptions: - operations where there is no outer exception handler, and the library knows it (e.g. callback processing) - methods that are guaranteed not to raise exceptions (e.g. top level event handlers) Some batch processing operations may also work that way, but those are a little more questionable (it may be better to report a result summary and leave it up to the application to decide what to do with any errors encountered along the way) However, it *is* the case that logging is currently ill-suited to tasks that are otherwise handled by writing directly to sys.stderr. I suspect that has influenced some people's views of its suitability for use in published libraries, even though it doesn't actually affect most libraries (I suspect it is a significant chunk of what underlay my reservations about using logging in library code). Since the standard library *does* include modules that currently write to stderr, then adjusting logging to make it better suited to this task will help us as well as anyone else with similar problems. That said, while I think Vinay's suggested "handler of last resort" solution is a good one and something we should be doing for 3.2, I'm also happy to let the idea bake for at least a few weeks. Cheers, Nick. P.S. On a completely unrelated note, has anyone thought about creating a write-only TextIO stream that outputs received writes via the logging module? -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From alexander.belopolsky at gmail.com Wed Dec 8 17:28:30 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Wed, 8 Dec 2010 11:28:30 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: On Wed, Dec 8, 2010 at 9:52 AM, Nick Coghlan wrote: .. > P.S. On a completely unrelated note, has anyone thought about creating > a write-only TextIO stream that outputs received writes via the > logging module? I've done something similar for C++ iostreams many moons ago. The idea was to prepend all lines written to std::log with timestamp and other information. It worked reasonably well, but involved a lot of C++ hackery at the time. I think this is a great idea and would allow existing libraries that use sys.stderr for messages start using say the root logger instead when user redefines sys.stderr. From robert.kern at gmail.com Wed Dec 8 17:35:59 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 08 Dec 2010 10:35:59 -0600 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: Message-ID: On 12/8/10 2:51 AM, Vinay Sajip wrote: > Robert Kern gmail.com> writes: > >> >> I really don't understand how this view can be consistent with the >> practice of adding NullHandler to loggers. If this message is so important >> to prevent misconfiguration, then why should a library author decide to >> silence it for his users? > > Because the application developer knows more about the end-user audience for > their application, they are better placed to know how logging should work for > their application. It's not an error for a particular application developer to > decide that nothing should be produced by logging for a particular application; > they (particularly when casual users) would be confused by the misconfiguration > message due to logging by a library they're using. > > The library author's users are the application developers who use the library, > not the end users who use their applications. Sometimes they're the same people, > I know, but I just think of them as wearing different hats :-) I'm sorry, but it's not at all clear that you have understood my point. There is no way for me to parse your words as a sensible reply to what I said. Let's say I write a library called Foo. I want to add logging to my functions. You want to write an application called Bar that uses Foo and you want to configure logging for your application (at the very least to provide a default if not production). The warning is supposed to help you not make mistakes when configuring logging in your application. If I, library author, attach NullHandlers to all of Foo's loggers, then you will not get that warning if you forget to add handlers the Foo loggers. My adding the NullHandler to Foo prevented that warning that you consider to be so important. I don't think the warning helps much, if at all. >> [...] I strongly suspect that almost all configurations include a >> catch-all root logger and that most of those *only* consist of that >> root logger. > > That doesn't seem right: your comment might be conflating loggers with handlers. > The common pattern would be (or should be) to name loggers according to __name__ > in the modules which use logging, but only configure *handlers* for the root > logger. That way, logging messages indicate their origin (because of the > __name__ convention) but you only need to add handlers at the root logger to > capture all the logging information. Yes. That's what I meant. >> I think that boilerplate should be minimized. If using getLogger() should >> almost always be followed by adding a NullHandler, then it should be the >> default behavior. The easiest way to achieve this effect is to simply not >> issue the warning message. > > getLogger() should NOT "almost always be followed by adding a NullHandler". For > example, in Django, only the logger named "django" would have that handler > added; no other Django logger (e.g. "django.db.models") would need to have that > handler added. In a large package (particularly a namespace package), I can't guarantee that any particular module will get imported. I will want to be able to import just foo.bar.baz without needing to worry about whether foo.setup_logging got imported and ran the logging configuration as a side-effect. I want to be able to loosen the coupling between modules across my package, not add more coupling. But in any case, while adding a NullHandler to just the package's root logger helps you to avoid needing a NullHandler on every logger, the effect is the same. Almost all loggers effectively terminate in a NullHandler either directly or through a chain of parent loggers. So why not just make it the default? Personally, I would back the proposals being made elsewhere in this thread, that in the absence of configuration, warnings and errors should be printed to stderr no matter where they come from. This gives useful behavior out-of-the-box without configuration but remains completely configurable. Library errors don't pass silently, but logging allows people to silence them explicitly. It separates the concerns of library authors (who should never touch logging configuration and shouldn't be required to think about it) from those of the application authors and application users. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From vinay_sajip at yahoo.co.uk Wed Dec 8 17:46:59 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 16:46:59 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: Nick Coghlan gmail.com> writes: > That said, while I think Vinay's suggested "handler of last resort" > solution is a good one and something we should be doing for 3.2, I'm > also happy to let the idea bake for at least a few weeks. I agree on the baking part, since it will allow time for any drawbacks to be spotted or better solutions found. There are also the questions of what level and format to use for the handler of last resort, before it can actually be implemented. > > P.S. On a completely unrelated note, has anyone thought about creating > a write-only TextIO stream that outputs received writes via the > logging module? > Is this for use at the C level? At the Python level, there's a post I wrote a while back which shows how to use a logger like an output stream: http://plumberjack.blogspot.com/2009/09/how-to-treat-logger-like-output-stream.html Regards, Vinay Sajip From ncoghlan at gmail.com Wed Dec 8 18:28:36 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Dec 2010 03:28:36 +1000 Subject: [Python-Dev] Redirecting stdout/stderr to the logging module (was Re: Using logging in the stdlib and its unit tests) Message-ID: On Thu, Dec 9, 2010 at 2:46 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: >> P.S. On a completely unrelated note, has anyone thought about creating >> a write-only TextIO stream that outputs received writes via the >> logging module? >> > > Is this for use at the C level? At the Python level, there's a post I wrote a > while back which shows how to use a logger like an output stream: > > http://plumberjack.blogspot.com/2009/09/how-to-treat-logger-like-output-stream.html Similar in concept, but more full-featured (i.e. supporting the relevant IO ABCs), allowing it to be used as a replacement for stdout/stderr. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Dec 8 18:32:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Dec 2010 03:32:27 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: On Thu, Dec 9, 2010 at 2:46 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> That said, while I think Vinay's suggested "handler of last resort" >> solution is a good one and something we should be doing for 3.2, I'm >> also happy to let the idea bake for at least a few weeks. > > I agree on the baking part, since it will allow time for any drawbacks to be > spotted or better solutions found. There are also the questions of what level > and format to use for the handler of last resort, before it can actually be > implemented. As a starting point, I'd say warnings and above, no formatting (i.e. just the message). To minimise bikeshedding, I'd like to be guided by the idea that this is a more configurable alternative to printing directly to stderr, but in the absence of application level configuration, you wouldn't be able to tell which approach the library was using just by looking at the program output. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Wed Dec 8 18:43:06 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 17:43:06 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: Message-ID: Robert Kern gmail.com> writes: > I'm sorry, but it's not at all clear that you have understood my point. > There is no way for me to parse your words as a sensible reply to what I > said. > > Let's say I write a library called Foo. I want to add logging to my > functions. You want to write an application called Bar that uses Foo and > you want to configure logging for your application (at the very least to > provide a default if not production). The warning is supposed to help you > not make mistakes when configuring logging in your application. If I, > library author, attach NullHandlers to all of Foo's loggers, then you will > not get that warning if you forget to add handlers the Foo loggers. > My adding the NullHandler to Foo prevented that warning that you consider > to be so important. I understand, and what you say above is correct. > I don't think the warning helps much, if at all. Perhaps not. I'm not so hung up on the importance of the message now, but it certainly *was* important when logging was first introduced, and users couldn't expect to get the same level of help on comp.lang.python that they now can. Today there are quite a few people who can help people with finger-trouble logging issues. Consider the scenarios when not having the current behaviour would bite you: 1. You're new to Python/new to logging. You write some code, perhaps across several modules, which code makes logging calls, but you omit to configure any handlers, whether through forgetting or not understanding what needs to be done. Without the message, you've no idea why no logging messages appear, no matter how much you fiddle with levels. 2. You write some code and decide you don't want to use logging, but use some third party libraries which do. You don't care about that, so it's annoying to have "no handlers could be found for logger XXX" messages printed to console. You berate the library developer for their lack of consideration. Perhaps you don't find yourself in these situations, but surely you sympathize with people who do? How would you propose to address both those scenarios? > In a large package (particularly a namespace package), I can't guarantee > that any particular module will get imported. I will want to be able to > import just foo.bar.baz without needing to worry about whether > foo.setup_logging got imported and ran the logging configuration as a > side-effect. I want to be able to loosen the coupling between modules > across my package, not add more coupling. > I'm not sure what coupling you're talking about - perhaps you can illustrate with an example. If I develop a package "foo.bar" which is part of namespace package "foo", and use loggers named __name__ in my code, and add a NullHandler to logger "foo.bar", that's all I have to do. Likewise, if another person develops "foo.baz" and they add a NullHandler to "foo.baz", then where's the coupling between the two packages? They needn't even know about each other. > But in any case, while adding a NullHandler to just the package's root logger > helps you to avoid needing a NullHandler on every logger, the effect is the > same. Almost all loggers effectively terminate in a NullHandler either > directly or through a chain of parent loggers. So why not just make it the > default? There's no "termination" when a NullHandler is encountered. Django has dozens of modules, many of which could use logging, but only one NullHandler needs to be added for the whole of Django. The effect you say is the same is not: when adding new modules to Django, for example, no additional NullHandler adding needs to be done. I don't want to appear rude, but this seems to be another mistake (or perhaps a variant of the same mistake as before) you are making about how logging works. You obviously feel strongly about it, and if you have a specific use case which is causing/has caused you pain, please spell it out for me (on comp.lang.python, say) and I will try to help sort out the problem. As am off-topic example, Armin Ronacher kept on saying in various posts and presentations that you couldn't use stdlib logging for web applications, that there were fundamental problems with it. But when he actually sent me his specific problem statement, I gave him a solution without spending too much time on it (see http://plumberjack.blogspot.com/2010/09/configuring-logging-for-web.html if you care). I'm not trying to be obstructive, honestly. > Personally, I would back the proposals being made elsewhere in this thread, > that in the absence of configuration, warnings and errors should be printed > to stderr no matter where they come from. This gives useful behavior > out-of-the-box As you might have seen in my response to Nick's post, I've made specific proposals about this myself. I agree with Nick's viee that some more time should be given for discussion, suggestions, others to express views, and just for the ideas to bake for a bit. > without configuration but remains completely configurable. Library errors > don't pass silently, but logging allows people to silence them explicitly. > It separates the concerns of library authors (who should never touch logging > configuration and shouldn't be required to think about it) from those of the > application authors and application users. You surprise me with "library authors (who should never touch logging configuration and shouldn't be required to think about it)". If you use logging, you should think about it at least a little, just as with any other software you use. For example, about default levels, propagation, etc. etc. The Django team, for example, have expended a reasonable amount of thought on how best to add logging support to their library; I don't believe their time has been wasted. Regards, Vinay Sajip From v+python at g.nevcal.com Wed Dec 8 18:51:44 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 08 Dec 2010 09:51:44 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> Message-ID: <4CFFC5B0.3040006@g.nevcal.com> On 12/8/2010 4:15 AM, Vinay Sajip wrote: > You're complaining about too much documentation?! Don't measure it by weight! On 12/8/2010 5:57 AM, Vinay Sajip wrote: > Of course I understand I could be wrong > about this, but I don't recall when a stdlib maintainer has said to me, "I want > to start using logging in stdlib module XXX, but can't justify it because ..." So I'm a fairly new Python user, but 30 years programming experience. When I first looked at the Python logging facility (and again today, when I looked again), I found a simple example of debugging logging. Then a bunch of stuff about how to configure rotating logs. Then something about logging levels. And then a dissertation on the theory of loggers "The logging library takes a modular approach and offers the several categories of components: loggers, handlers, filters, and formatters." And I hadn't gotten through 10% of the documentation yet, per the scrollbar. My reaction the first time was "Sounds sophisticated and complex. I think I'll do something simpler for now, and maybe someday, when I have a spare week, I'll read the documentation and see if the benefits are worth the effort." OK, so now you've discovered that too much documentation can be a turn off... at least, if it is presented from the top to describe the sophistication of the facility, rather than how easy it is to use (if it is, I still haven't gotten to 10%, and I still don't know that). From this thread, it sounds like it might be that logging could be easy to use. Although part of my turn off from too much documentation and sophistication is a concern about the overhead of reading in and compiling a module that is that sophisticated for simple programs. Now if I was convinced that (1) it always gets read in anyway as part of Python startup (8% of the documentation didn't say that), so that no matter how simple my alternate facility, I pay the cost of loading the logger anyway, and (2) that it would take less than a week to comprehend the basics of what I need to learn to use it, (in other words, if the 8% of the documentation I've read, actually gave sufficient simple examples to use applications and libraries, and left the sohistication and complexity and theory to later sections) then I might have used the logger, instead of writing, in my personal library of "useful functions": _logfh = sys.stderr def log( text=None, textpre=None, filename=None, mode=None ): # version 2010/11/11 global _logfh if filename is not None: if mode is None: mode = "a" _logfh = open( filename, mode, **_wenc ) if text is not None: if textpre is None: _logfh.write( str( text ) + "\n") else: _logfh.write( str( textpre ) + ": " + str( text ) + "\n") _logfh.flush() return _logfh You see, I can create that faster than I read 8% of the documentation for logger, which didn't convince me that using the logger was better than the above, and it has been working fine... and I can tweak it when and if I need more sophistication. I'm well aware that my function provides much less than what logger provides. But in the first 8% of the documentation, I hadn't learned how to be able to use logging from a multiple module program in an easy and consistent manner... The example shows a 4 line cost... it is not clear from the first 8% of the documentation if it requires 4 lines in every module; it is not clear if I can or should or if by default, the file I configure for one module is available to other modules, such that only two lines are necessary in those modules (import, logging.debug (or other levels of message)), etc. And I'd have to read more than 8% of the documentation to find that out. Now I'm not saying that I only read 8% of the documentation for modules that I want to use, in general, before I give up in disgust. But I start reading, and estimate the cost of learning vs the cost of rewriting as I go. logger documentation is large, so reading it all before starting to use it is unlikely; since the first part didn't get me started soon enough, I "put it on a shelf", where it still is. If the first part had shown an example of how to use logger, in simple mode, in a trivial but multi-module application (because most trivial application do involve multiple modules, even if only one is user-written), then as I went along, I'd have likely learned more about the features as needed. There are lots of interesting sounding features in the Table of Contents. "Someday" I might decide to spend the week to read about them. Hope this perspective helps, somehow. Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Dec 8 19:15:37 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Dec 2010 19:15:37 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: <20101208191537.44fc3ad8@pitrou.net> On Wed, 8 Dec 2010 14:54:09 +0000 (UTC) Vinay Sajip wrote: > Antoine Pitrou pitrou.net> writes: > > > I'm not talking specifically about exceptions, but about errors in > > general. If the case wasn't common, I'm not sure why the error() and > > critical() methods would exist at all. > > > > (of course I'm assuming error() is meant to output error messages. If > > that's a wrong interpretation then I'm a bit puzzled ) > > > > Then again, it's not always helpful when *library* code prints out messages to > stderr. A user of the library might prefer that an error code be returned or an > exception raised, so they can deal with the condition most appropriately. Sorry, what are you arguing about and what kind of answer are you expecting? Obviously error() is preferrable in some cases and other means of communicating the error are preferrable in other cases. That's kind of stating the obvious to me. But since you are the one you wrote the library and added error() in the first place, why are you trying to convince me that error() is not useful? Perhaps you should explain how error() is supposed to be used for if it's not supposed to log errors. > > I was talking about the user of a library, not its developer (see the > > snippet quoted above). > > Yes, but see my response to that. A user of a library doesn't need to know the > specifics of what loggers that library uses, they just need to make a positive > statement of where they want warning and error messages go, and how they want > them formatted. And my point, again, is that they want it to work by default *without* making such a positive statement. I'm not sure if I'm supposed to explain this using different words since you're not appearing to get it. > BTW we haven't discussed this yet - but these enabled-by-default messages, what > format should they have? (e.g. just message, logger name + message, logger name > + severity + message etc.) It might sound like a trivial point, but it also > seems like whatever you pick, some bikeshed discussions would ensue. I'm sorry but your way of argumenting is really puzzling to me. You are trying to find counter-arguments against something which you *already* implemented yourself in logging.error(). If you're interested in having a theoretical argument about what a beautiful design should be (and if you want to argue about your own design decisions), I'll leave the discussion here. I'm only interested in the *practical* matter of logging having an useful behaviour by default, which apparently you don't really care about. Fair enough, but this whole thread is quite telling, IMO, about the general impedance mismatch between the logging design and the expectations of casual users (which is 99% of them, certainly). Regards Antoine. From solipsis at pitrou.net Wed Dec 8 19:18:07 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Dec 2010 19:18:07 +0100 Subject: [Python-Dev] Redirecting stdout/stderr to the logging module (was Re: Using logging in the stdlib and its unit tests) References: Message-ID: <20101208191807.5979d53d@pitrou.net> On Thu, 9 Dec 2010 03:28:36 +1000 Nick Coghlan wrote: > On Thu, Dec 9, 2010 at 2:46 AM, Vinay Sajip wrote: > > Nick Coghlan gmail.com> writes: > >> P.S. On a completely unrelated note, has anyone thought about creating > >> a write-only TextIO stream that outputs received writes via the > >> logging module? > >> > > > > Is this for use at the C level? At the Python level, there's a post I wrote a > > while back which shows how to use a logger like an output stream: > > > > http://plumberjack.blogspot.com/2009/09/how-to-treat-logger-like-output-stream.html > > Similar in concept, but more full-featured (i.e. supporting the > relevant IO ABCs), allowing it to be used as a replacement for > stdout/stderr. That would be an interesting idea. It would fit in the logging module rather than the io module IMO. However, there are some limitations due to the fact that logging is line-based while people may output arbitrary text on stderr (perhaps ASCII-formatted tables, who knows). Regards Antoine. From vinay_sajip at yahoo.co.uk Wed Dec 8 19:44:46 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 18:44:46 +0000 (UTC) Subject: [Python-Dev] Redirecting stdout/stderr to the logging module (was Re: Using logging in the stdlib and its unit tests) References: <20101208191807.5979d53d@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > However, there are some limitations due to the fact that logging is > line-based while people may output arbitrary text on stderr (perhaps > ASCII-formatted tables, who knows). True, though the wrapper could easily buffer partial output internally to support line-based output formats. Actually the line-based nature is a function of the handler and/or formatter and just reflects what people tend to want most of the time. Regards, Vinay Sajip From p.f.moore at gmail.com Wed Dec 8 19:49:52 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Dec 2010 18:49:52 +0000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: On 8 December 2010 14:52, Nick Coghlan wrote: > As I see it, there aren't many cases at the *library* level where > logging errors is more appropriate than raising exceptions: On a slightly tangential note, what do you think of the idea of library code including info or debug level logging? In effect, tracing and diagnostic code built in and available simply by changing the logging level? Paul. From vinay_sajip at yahoo.co.uk Wed Dec 8 20:00:31 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 19:00:31 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > But since you are the one you wrote the library and added error() in > the first place, why are you trying to convince me that error() is > not useful? Perhaps you should explain how error() is supposed to be > used for if it's not supposed to log errors. I didn't say error() [or actually exception(), which you mentioned] was useless, I merely pointed out what the normal pattern of usage was. Georg, at least, seemed to be agreeing with me. > And my point, again, is that they want it to work by default *without* > making such a positive statement. I'm not sure if I'm supposed to > explain this using different words since you're not appearing to get it. No, I'm getting it, I think. See my responses to Nick's posts. > I'm sorry but your way of argumenting is really puzzling to me. You are > trying to find counter-arguments against something which you *already* > implemented yourself in logging.error(). I don't think I am, so one of us is misunderstanding the other, or perhaps both of us are misunderstanding each other. > If you're interested in having a theoretical argument about what a > beautiful design should be (and if you want to argue about your own > design decisions), I'll leave the discussion here. I'm only interested No, I'm not interested in that. Like you I'm interested in practical matters, but unlike you I have to make some implementation choices, going forward - so it shouldn't be surprising that I want to talk around the issue to make sure that no wrong decision gets made. Sorry if that offends you because it sometimes seems to be stating the (to you) obvious. > in the *practical* matter of logging having an useful behaviour by > default, which apparently you don't really care about. Actually, I don't think my response to Nick's post (about concurrent.futures) could be characterized as "I don't care", as I even made a specific proposal about how a change could be implemented. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Dec 8 20:04:36 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 19:04:36 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: Paul Moore gmail.com> writes: > > On 8 December 2010 14:52, Nick Coghlan gmail.com> wrote: > > As I see it, there aren't many cases at the *library* level where > > logging errors is more appropriate than raising exceptions: > > On a slightly tangential note, what do you think of the idea of > library code including info or debug level logging? In effect, tracing > and diagnostic code built in and available simply by changing the > logging level? That's how it works right now. You get info() and debug() messages sent via calls in library code, just by changing the level of (say) the root logger. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Dec 8 20:11:48 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 19:11:48 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: Nick Coghlan gmail.com> writes: > As a starting point, I'd say warnings and above, no formatting (i.e. > just the message). To minimise bikeshedding, I'd like to be guided by > the idea that this is a more configurable alternative to printing > directly to stderr, but in the absence of application level > configuration, you wouldn't be able to tell which approach the library > was using just by looking at the program output. Makes sense. I know it's only a small change at the implementation level but the impact may be larger (due to it being a backwards-incompatible behaviour change), and the little details need to be agreed, so does it make sense to create a PEP about this? What do people think - is this bureaucratic overkill? Regards, Vinay Sajip From skip at pobox.com Wed Dec 8 20:14:25 2010 From: skip at pobox.com (skip at pobox.com) Date: Wed, 8 Dec 2010 13:14:25 -0600 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: <19711.55569.982114.312501@montanaro.dyndns.org> >> On a slightly tangential note, what do you think of the idea of >> library code including info or debug level logging? In effect, >> tracing and diagnostic code built in and available simply by changing >> the logging level? Vinay> That's how it works right now. You get info() and debug() Vinay> messages sent via calls in library code, just by changing the Vinay> level of (say) the root logger. There can be performance implications if you log heavily. I don't know how the code is organized, but functionally these two calls are equivalent: >>> logging.error("error 1 2 3 %s" % "yup") ERROR:root:error 1 2 3 yup >>> logging.error("error 1 2 3 %s", "yup") ERROR:root:error 1 2 3 yup The second form should be preferred in library code as long as the format string expansion is deferred until after the test is made to emit the message. Skip From solipsis at pitrou.net Wed Dec 8 20:16:42 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 8 Dec 2010 20:16:42 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> Message-ID: <20101208201642.429851ff@pitrou.net> On Wed, 8 Dec 2010 19:00:31 +0000 (UTC) Vinay Sajip wrote: > > > If you're interested in having a theoretical argument about what a > > beautiful design should be (and if you want to argue about your own > > design decisions), I'll leave the discussion here. I'm only interested > > No, I'm not interested in that. Like you I'm interested in practical matters, > but unlike you I have to make some implementation choices, going forward - so it > shouldn't be surprising that I want to talk around the issue to make sure that > no wrong decision gets made. Sorry if that offends you because it sometimes > seems to be stating the (to you) obvious. Ok, I'm sorry for the harsh words. I really hope this discussions leads to somewhere. Regards Antoine. From vinay_sajip at yahoo.co.uk Wed Dec 8 20:40:06 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 19:40:06 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> <20101208201642.429851ff@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > Ok, I'm sorry for the harsh words. > I really hope this discussions leads to somewhere. No offence taken, and do do I :-) Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Wed Dec 8 20:42:00 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 19:42:00 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <19711.55569.982114.312501@montanaro.dyndns.org> Message-ID: pobox.com> writes: > >>> logging.error("error 1 2 3 %s" % "yup") > ERROR:root:error 1 2 3 yup > >>> logging.error("error 1 2 3 %s", "yup") > ERROR:root:error 1 2 3 yup > > The second form should be preferred in library code as long as the format > string expansion is deferred until after the test is made to emit the > message. Yes, and the string expansion normally is deferred to happen as late as possible. Regards, Vinay From vinay_sajip at yahoo.co.uk Wed Dec 8 20:44:52 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 19:44:52 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> <20101208201642.429851ff@pitrou.net> Message-ID: Vinay Sajip yahoo.co.uk> writes: > No offence taken, and do do I s/do do/so do/ Perhaps it was a Freudian slip admitting that I *am* a dodo! Regards, Vinay From kirklin.mcdonald at gmail.com Wed Dec 8 21:09:22 2010 From: kirklin.mcdonald at gmail.com (Kirk McDonald) Date: Wed, 8 Dec 2010 12:09:22 -0800 Subject: [Python-Dev] Status of PEP 3143? Message-ID: What is the status of PEP 3143? It appears to be aimed at 3.2, but there does not appear to be a resolution on it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed Dec 8 21:12:31 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 8 Dec 2010 14:12:31 -0600 Subject: [Python-Dev] Status of PEP 3143? In-Reply-To: References: Message-ID: 2010/12/8 Kirk McDonald : > What is the status of PEP 3143? It appears to be aimed at 3.2, but there > does not appear to be a resolution on it. It's after beta freeze now, so definitely not 3.2. AFAIK, though, it's never been discussed here. -- Regards, Benjamin From robert.kern at gmail.com Wed Dec 8 21:20:45 2010 From: robert.kern at gmail.com (Robert Kern) Date: Wed, 08 Dec 2010 14:20:45 -0600 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: Message-ID: If we're all converging on adding a "handler of last resort" and dropping the warning message, we can just let this branch of the thread drop. But if you want to continue, I already had most of the following already written. I hope it clears some things up more than it muddies them further. :-) On 12/8/10 11:43 AM, Vinay Sajip wrote: > Robert Kern gmail.com> writes: >> I don't think the warning helps much, if at all. > > Perhaps not. I'm not so hung up on the importance of the message now, but it > certainly *was* important when logging was first introduced, and users couldn't > expect to get the same level of help on comp.lang.python that they now can. > Today there are quite a few people who can help people with finger-trouble > logging issues. I am not commenting on the reasonableness of the behavior when it was introduced, just what I think is the best behavior now. > Consider the scenarios when not having the current behaviour would bite you: > > 1. You're new to Python/new to logging. You write some code, perhaps across > several modules, which code makes logging calls, but you omit to configure any > handlers, whether through forgetting or not understanding what needs to be done. > Without the message, you've no idea why no logging messages appear, no matter > how much you fiddle with levels. > 2. You write some code and decide you don't want to use logging, but use some > third party libraries which do. You don't care about that, so it's annoying to > have "no handlers could be found for logger XXX" messages printed to console. > You berate the library developer for their lack of consideration. > > Perhaps you don't find yourself in these situations, but surely you sympathize > with people who do? How would you propose to address both those scenarios? I am quite familiar with the latter in the third-party library author role. However, the messages I get aren't about lack of consideration but "What is wrong with my code? Are things broken?" They think the warning comes from my code. As for the former, I'm not sure how that would cause much confusion. If I'm interested in getting the logged information, wouldn't I first find out how to configure logging? It's just about the very first thing you see reading the docs. It's one of the fundamental operations using the logging package. I can't imagine that this is a common failure mode. I think that the documentation ameliorates it better than the warning message. >> In a large package (particularly a namespace package), I can't guarantee >> that any particular module will get imported. I will want to be able to >> import just foo.bar.baz without needing to worry about whether >> foo.setup_logging got imported and ran the logging configuration as a >> side-effect. I want to be able to loosen the coupling between modules >> across my package, not add more coupling. > > I'm not sure what coupling you're talking about - perhaps you can illustrate > with an example. If I develop a package "foo.bar" which is part of namespace > package "foo", and use loggers named __name__ in my code, and add a NullHandler > to logger "foo.bar", that's all I have to do. Likewise, if another person > develops "foo.baz" and they add a NullHandler to "foo.baz", then where's the > coupling between the two packages? They needn't even know about each other. You were saying (in the part of the conversation that you snipped) that one would just add a NullHandler to the "django" logger to take care of messages sent to "django.db.models" and all of the other child loggers. *That* introduces a tighter coupling than I am comfortable with. All modules that use logging must make sure that the code that adds the NullHandler to the "django" logger gets executed. It's certainly reasonable to make that effort sometimes, but there are good reasons to avoid it, too. Just using logging should not force my hand. If I do keep things decoupled, then I am adding a NullHandler to nearly every logger anyways, and I return to my original point that this is repetitive boilerplate and should be eliminated by appropriate defaults. >> But in any case, while adding a NullHandler to just the package's root logger >> helps you to avoid needing a NullHandler on every logger, the effect is the >> same. Almost all loggers effectively terminate in a NullHandler either >> directly or through a chain of parent loggers. So why not just make it the >> default? > > There's no "termination" when a NullHandler is encountered. All I meant is that it eventually reaches a NullHandler so no warning message is issued. Whether it actually stops there or not is irrelevant to my point. My point is that it is rarely the case that you deliberately want to have a logger which will cause the ?No handlers could be found" message. Whether you add the NullHandler to each logger or to one parent logger, you almost always want that behavior. That's why I think it should be the default behavior. > Django has dozens of > modules, many of which could use logging, but only one NullHandler needs to be > added for the whole of Django. The effect you say is the same is not: when > adding new modules to Django, for example, no additional NullHandler adding > needs to be done. The "effect" that I am talking about is that the log messages eventually get handled by a NullHandler and no warning message about missing handlers is sent. I'm not talking about the effect on developer effort. The only thing adding a NullHandler accomplishes is to silence the warning. I think that accomplishing that effect in this manner is bad design. It's not obvious, particularly pre-2.7 where you had to roll your own NullHandler. It introduces a special case ("library authors should never add Handlers except they *should* add a NullHandler"). > I don't want to appear rude, but this seems to be another mistake (or perhaps a > variant of the same mistake as before) you are making about how logging works. It seems that I am being particularly unclear in stating my points. You also seem to be addressing small errors or misunderstandings in my statements at length and ignoring the substantive points. I understand that it is not intentional, but it is frustrating. > You obviously feel strongly about it, and if you have a specific use case which > is causing/has caused you pain, please spell it out for me (on comp.lang.python, > say) and I will try to help sort out the problem. I want to write library code that liberally uses logging. I want to avoid doing any configuration of logging in my library code. I do not want users of my library code to need to configure logging until they actually want to change the (sensible) defaults. Is that clear enough? >> without configuration but remains completely configurable. Library errors >> don't pass silently, but logging allows people to silence them explicitly. >> It separates the concerns of library authors (who should never touch logging >> configuration and shouldn't be required to think about it) from those of the >> application authors and application users. > > You surprise me with "library authors (who should never touch logging > configuration and shouldn't be required to think about it)". If you use logging, > you should think about it at least a little, just as with any other software you > use. For example, about default levels, propagation, etc. etc. The Django team, > for example, have expended a reasonable amount of thought on how best to add > logging support to their library; I don't believe their time has been wasted. Fair enough. I don't disagree. Please drop "and shouldn't be required to think about it" from my statements. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From g.brandl at gmx.net Wed Dec 8 21:45:03 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 08 Dec 2010 21:45:03 +0100 Subject: [Python-Dev] Status of PEP 3143? In-Reply-To: References: Message-ID: Am 08.12.2010 21:12, schrieb Benjamin Peterson: > 2010/12/8 Kirk McDonald : >> What is the status of PEP 3143? It appears to be aimed at 3.2, but there >> does not appear to be a resolution on it. > > It's after beta freeze now, so definitely not 3.2. AFAIK, though, it's > never been discussed here. I might add that now would be a good time to put any candidate library into PyPI, in order to experience enough real-world usage in time for inclusion in Python 3.3. Georg From tjreedy at udel.edu Wed Dec 8 22:09:45 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 08 Dec 2010 16:09:45 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> Message-ID: On 12/8/2010 2:00 PM, Vinay Sajip wrote: > Actually, I don't think my response to Nick's post (about concurrent.futures) > could be characterized as "I don't care", as I even made a specific proposal > about how a change could be implemented. Your proposal struck me as probably the best way forward. Can you code it up and put a patch on the tracker that people can test before the next beta? A couple of other notes: 1. The number of participants in this thread is probably larger than average. So I think the responses so far can be taken as representative of those here who care enough to say something. 2. To me, Django is more an app framework than a mere 'library', and one with extra problems of error reporting in that it needs to send messages to both client and server and possibly elsewhere. So I as a user would expect it to embody extra thought about message disposition relative to a typical library. -- Terry Jan Reedy From ben+python at benfinney.id.au Wed Dec 8 22:16:54 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Thu, 09 Dec 2010 08:16:54 +1100 Subject: [Python-Dev] Status of PEP 3143? References: Message-ID: <87oc8wyng9.fsf@benfinney.id.au> Benjamin Peterson writes: > 2010/12/8 Kirk McDonald : > > What is the status of PEP 3143? It appears to be aimed at 3.2, but > > there does not appear to be a resolution on it. I am the PEP 3143 author and champion. It hasn't been presented for a resolution yet. Development of the reference implementation is complete but not yet suitable for standard library inclusion; progress in that direction has stalled due to intermittent communication between the developers. Further questions specifically about the PEP content or its implementation to me directly, if you like. Or in ?comp.lang.python?, if you prefer. > It's after beta freeze now, so definitely not 3.2. So it's not abandoned, but I don't know which version should be the current target. What change should I make to the PEP in such a case? -- \ ?I used to be an airline pilot. I got fired because I kept | `\ locking the keys in the plane. They caught me on an 80 foot | _o__) stepladder with a coathanger.? ?Steven Wright | Ben Finney From benjamin at python.org Wed Dec 8 22:34:14 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 8 Dec 2010 15:34:14 -0600 Subject: [Python-Dev] Status of PEP 3143? In-Reply-To: <87oc8wyng9.fsf@benfinney.id.au> References: <87oc8wyng9.fsf@benfinney.id.au> Message-ID: 2010/12/8 Ben Finney : > Benjamin Peterson writes: > >> 2010/12/8 Kirk McDonald : >> > What is the status of PEP 3143? It appears to be aimed at 3.2, but >> > there does not appear to be a resolution on it. > > I am the PEP 3143 author and champion. It hasn't been presented for a > resolution yet. > > Development of the reference implementation is complete but not yet > suitable for standard library inclusion; progress in that direction has > stalled due to intermittent communication between the developers. > > Further questions specifically about the PEP content or its > implementation to me directly, if you like. Or in ?comp.lang.python?, if > you prefer. > >> It's after beta freeze now, so definitely not 3.2. > > So it's not abandoned, but I don't know which version should be the > current target. What change should I make to the PEP in such a case? Put 3.3 or 3.x if you're thinking really long term. :) -- Regards, Benjamin From p.f.moore at gmail.com Thu Dec 9 00:06:46 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 8 Dec 2010 23:06:46 +0000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: On 8 December 2010 19:04, Vinay Sajip wrote: > Paul Moore gmail.com> writes: > >> >> On 8 December 2010 14:52, Nick Coghlan gmail.com> wrote: >> > As I see it, there aren't many cases at the *library* level where >> > logging errors is more appropriate than raising exceptions: >> >> On a slightly tangential note, what do you think of the idea of >> library code including info or debug level logging? In effect, tracing >> and diagnostic code built in and available simply by changing the >> logging level? > > That's how it works right now. You get info() and debug() messages sent via > calls in library code, just by changing the level of (say) the root logger. You misunderstand me. I know that's how those levels work. What I'm not sure about (and I think would be interesting and potentially useful information) is whether the individuals participating in this thread feel that liberal use of info and debug level logging in library code is useful. All the discussion thus far has been about warning and error levels (mainly because people seem less comfortable with the "suppress by default" approach for those levels, whereas it's natural for debug and info levels). I guess I'm wondering what best practices for logging might be. (Actually, that might make the subject of an interesting blog posting, if you're looking for suggestions :-)). Paul. From vinay_sajip at yahoo.co.uk Thu Dec 9 00:35:34 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Wed, 8 Dec 2010 23:35:34 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: Paul Moore gmail.com> writes: > You misunderstand me. I know that's how those levels work. What I'm > not sure about (and I think would be interesting and potentially > useful information) is whether the individuals participating in this > thread feel that liberal use of info and debug level logging in > library code is useful. All the discussion thus far has been about > warning and error levels (mainly because people seem less comfortable > with the "suppress by default" approach for those levels, whereas it's > natural for debug and info levels). OK I see, sorry - I just didn't get that from your original choice of words. I'm probably not firing on all cylinders right now :-) > I guess I'm wondering what best practices for logging might be. > (Actually, that might make the subject of an interesting blog posting, > if you're looking for suggestions ). Thanks for the suggestion, I'll think about posting on this topic after ruminating on it for a bit. Regards, Vinay Sajip From v+python at g.nevcal.com Thu Dec 9 01:50:48 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 08 Dec 2010 16:50:48 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: Message-ID: <4D0027E8.5020808@g.nevcal.com> On 12/8/2010 9:43 AM, Vinay Sajip wrote: > > > As am off-topic example, Armin Ronacher kept on saying in various posts and > presentations that you couldn't use stdlib logging for web applications, that > there were fundamental problems with it. But when he actually sent me his > specific problem statement, I gave him a solution without spending too much time > on it (see > http://plumberjack.blogspot.com/2010/09/configuring-logging-for-web.html if you > care). I'm not trying to be obstructive, honestly. > > Since my application is also web server related, I was curious enough to look at this blog posting to see if it would be a good starter example that is missing from the manual. I don't think it is, as it is somewhat complex, although it is multi-module, and even multi-thread, so it might make a good example for other parts of the documentation (but maybe they already have good examples; I've only looked at the first 8%). In trying to understand it, I couldn't figure out why the WebApp class needs to keep track of the threads that are running the that web app. Why isn't it good enough for the thread to know the name of the app? The filter copies the appName from the thread to the record; the InjectingFilter.filter not replace record.appName = tlocal.appName tname = threading.currentThread().getName() return tname in self.app.threads by record.appName = tlocal.appName return appName == self.app.name and get the same effect, without needing to mainting the self.threads "database"? Or what am I missing? Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Thu Dec 9 03:56:40 2010 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 09 Dec 2010 02:56:40 +0000 Subject: [Python-Dev] Can't compile regex module with Python 3.2 Message-ID: <4D004568.4000905@mrabarnett.plus.com> The regex module calls _PyUnicode_IsWhitespace, which is mapped by unicodeobject.h to either _PyUnicodeUCS2_IsWhitespace or _PyUnicodeUCS4_IsWhitespace. From Python 2.5 to Python 3.1 the library pythonXX.lib contains either _PyUnicodeUCS2_IsWhitespace or _PyUnicodeUCS4_IsWhitespace. However, in Python 3.2b1 the library python32.lib contains only _PyUnicode_IsWhitespace, therefore breaking the build. Is this change intentional? If so, why does unicodeobject.h still do the mapping? Windows XP Pro, 32-bit. From merwok at netwok.org Thu Dec 9 04:45:37 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 09 Dec 2010 04:45:37 +0100 Subject: [Python-Dev] [Python-checkins] r86855 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py In-Reply-To: <20101129013825.EF8DAEEB1E@mail.python.org> References: <20101129013825.EF8DAEEB1E@mail.python.org> Message-ID: <4D0050E1.3000707@netwok.org> Hello, > Author: raymond.hettinger > Date: Mon Nov 29 02:38:25 2010 > New Revision: 86855 > Log: Do not add an obsolete unittest name to Py3.2. > Modified: python/branches/py3k/Lib/unittest/case.py > - # Old name for assertCountEqual() > - assertItemsEqual = assertCountEqual When we merge distutils2 back into the stdlib, our tests will have to work with stdlib unittest in 3.3 or unittest2 for older Pythons (we?ll still make standalone releases for them). unittest2 doesn?t have assertCountEqual, unittest in 3.2+ doesn?t have assertItemsEqual, what?s the plan for compatibility? Regards From martin at v.loewis.de Thu Dec 9 06:47:52 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 09 Dec 2010 06:47:52 +0100 Subject: [Python-Dev] Can't compile regex module with Python 3.2 In-Reply-To: <4D004568.4000905@mrabarnett.plus.com> References: <4D004568.4000905@mrabarnett.plus.com> Message-ID: <4D006D88.10300@v.loewis.de> Am 09.12.2010 03:56, schrieb MRAB: > The regex module calls _PyUnicode_IsWhitespace, which is mapped by > unicodeobject.h to either _PyUnicodeUCS2_IsWhitespace or > _PyUnicodeUCS4_IsWhitespace. > > From Python 2.5 to Python 3.1 the library pythonXX.lib contains either > _PyUnicodeUCS2_IsWhitespace or _PyUnicodeUCS4_IsWhitespace. > > However, in Python 3.2b1 the library python32.lib contains only > _PyUnicode_IsWhitespace, therefore breaking the build. > > Is this change intentional? If so, why does unicodeobject.h still do > the mapping? Are you sure about this? It's not intentional (except in the limited ABI). Regards, Martin From alexander.belopolsky at gmail.com Thu Dec 9 06:57:04 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 9 Dec 2010 00:57:04 -0500 Subject: [Python-Dev] Can't compile regex module with Python 3.2 In-Reply-To: <4D006D88.10300@v.loewis.de> References: <4D004568.4000905@mrabarnett.plus.com> <4D006D88.10300@v.loewis.de> Message-ID: On Thu, Dec 9, 2010 at 12:47 AM, "Martin v. L?wis" wrote: .. >> However, in Python 3.2b1 the library python32.lib contains only >> _PyUnicode_IsWhitespace, therefore breaking the build. >> >> Is this change intentional? If so, why does unicodeobject.h still do >> the mapping? > > Are you sure about this? It's not intentional (except in the limited ABI). this does seem to be intentional: $ svn log -r 84177 ------------------------------------------------------------------------ r84177 | amaury.forgeotdarc | 2010-08-18 16:44:58 -0400 (Wed, 18 Aug 2010) | 9 lines #5127: Even on narrow unicode builds, the C functions that access the Unicode Database (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others) now accept and return characters from the full Unicode range (Py_UCS4). The differences from Python code are few: - unicodedata.numeric(), unicodedata.decimal() and unicodedata.digit() now return the correct value for large code points - repr() may consider more characters as printable. ------------------------------------------------------------------------ http://svn.python.org/view?view=rev&revision=84177 From ncoghlan at gmail.com Thu Dec 9 06:57:16 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 9 Dec 2010 15:57:16 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> Message-ID: On Thu, Dec 9, 2010 at 4:49 AM, Paul Moore wrote: > On 8 December 2010 14:52, Nick Coghlan wrote: >> As I see it, there aren't many cases at the *library* level where >> logging errors is more appropriate than raising exceptions: > > On a slightly tangential note, what do you think of the idea of > library code including info or debug level logging? In effect, tracing > and diagnostic code built in and available simply by changing the > logging level? It's a trade-off between performance and the ability to diagnose faults and what you want to do will vary by application and problem domain. My background is in working on distributed projects where we need enough diagnostic information to debug one-off failures of the live system, so I'm in favour of the idea in general, but doing it *can* carry a fairly hefty speed cost (you can ameliorate that to some degree by handing off the actual I/O to a separate thread, but the system would still be faster if the log messages weren't there). As Skip says, you definitely want to take advantage of the lazy string formatting when doing this with the Python logging module. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Thu Dec 9 09:26:35 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 9 Dec 2010 08:26:35 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <4D0027E8.5020808@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > Or what am I missing? That threads are not necessarily dedicated to apps, in a real world setting. Depending on the server implementation, a single thread could be asked to handle requests for different apps over its lifetime. So the only way of knowing which threads are currently servicing a particular app is to maintain a set of them. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Thu Dec 9 09:39:13 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 9 Dec 2010 08:39:13 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <4D0027E8.5020808@g.nevcal.com> Message-ID: Vinay Sajip yahoo.co.uk> writes: > > Glenn Linderman g.nevcal.com> writes: > > > Or what am I missing? > And one more thing: the filters for *both* apps are called for a given request. One will return True, the other will return False. Bear in mind that the intention of the post is to be didactic, so it's possible there are some improvements that could be made when implementing for production. Regards, Vinay Sajip From v+python at g.nevcal.com Thu Dec 9 10:29:51 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Thu, 09 Dec 2010 01:29:51 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <4D0027E8.5020808@g.nevcal.com> Message-ID: <4D00A18F.60208@g.nevcal.com> On 12/9/2010 12:26 AM, Vinay Sajip wrote: > Glenn Linderman g.nevcal.com> writes: >> > Or what am I missing? > That threads are not necessarily dedicated to apps, in a real world setting. > Depending on the server implementation, a single thread could be asked to handle > requests for different apps over its lifetime. So the only way of knowing which > threads are currently servicing a particular app is to maintain a set of them. Agreed, they are not necessarily dedicated to apps. But while they are running the app, they have the appname in their thread local storage, no? So if a thread has the appname in its thread local storage, is it not servicing that app, at that moment? And if it is, why is that insufficient? That is my question, and you've sidestepped it. > And one more thing: the filters for*both* apps are called for a given request. > One will return True, the other will return False. > > Bear in mind that the intention of the post is to be didactic, so it's possible > there are some improvements that could be made when implementing for production. OK, I just learned what "didactic" meant, so you've taught me something. I understood that both filters would be called. I understood, that in production, it would probably not be necessary to log messages to both the app log and the global log, that the global log would be there just for things that are not app-specific. But I still don't see a need to maintain a set of threads running the app in the app object, if the app keeps track of what app it is running, in a spot that is accessible to the filter (which it seems to be). I don't see how it is beneficial to keep track of two separate data structures that could serve the same purpose. I realize you designed this in 10 minutes, and probably took twice that long to write it up, so didn't necessarily analyze it for efficiency. But I'm asking for that analysis now, if there is any real need for the app to track the set of threads, _for the purpose of the problem being solved_? I understand there might be other reasons for which that might be useful, but for the logging it simply seems to be inefficient redundancy... and if it isn't, then I don't understand the example, yet, and I'm trying to. So since you hand-waved, instead of giving a straight answer, and then maybe your second message was an attempt to back-pedal a bit, not sure, I went ahead and downloaded your code, made changes to remove the set of threads as outlined (see modified functions below), and it seems to run just as correctly. I thought it was an obvious question, while trying to understand the code, and maybe learn about the logger, and I guess I have, a little. And maybe some other things too. Please further explain if I am still missing something. Under what conditions of the problem you were solving does the code below fail? class InjectingFilter(logging.Filter): def __init__(self, app): self.app = app def filter(self, record): record.method = tlocal.request.method record.ip = tlocal.request.ip record.appName = tlocal.appName return record.appName == self.app.name # tname = threading.currentThread().getName() # return tname in self.app.threads class WebApp: def __init__(self, name): self.name = name self.threads = set() handler = logging.FileHandler(name + '.log', 'w') f = InjectingFilter(self) handler.setFormatter(formatter) handler.addFilter(f) root.addHandler(handler) self.num_requests = 0 def process_request(self, request): tlocal.request = request tlocal.appName = self.name # tname = threading.currentThread().getName() # self.threads.add(tname) self.num_requests += 1 try: logger.debug('Request processing started') webapplib.useful() logger.debug('Request processing finished') finally: pass # self.threads.remove(tname) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Thu Dec 9 10:52:13 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Thu, 9 Dec 2010 09:52:13 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <4D0027E8.5020808@g.nevcal.com> <4D00A18F.60208@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > Agreed, they are not necessarily dedicated to apps.? But while they > are running the app, they have the appname in their thread local > storage, no???? So if a thread has the appname in its thread local > storage, is it not servicing that app, at that moment?? And if it > is, why is that insufficient?? That is my question, and you've > sidestepped it. Sorry, I didn't mean to side-step: it's just that when reading your earlier post (on the Web using Gmane), all the lines of Python in your post ran together, so I didn't quite see what you were getting at. Having looked at it more carefully, you're right: the threads set is not needed. You can just "return tlocal.appName == self.app.name". I missed that, so thanks for teaching me something, too :-) Regards, Vinay Sajip From fuzzyman at voidspace.org.uk Thu Dec 9 11:57:27 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 09 Dec 2010 10:57:27 +0000 Subject: [Python-Dev] [Python-checkins] r86855 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py In-Reply-To: <4D0050E1.3000707@netwok.org> References: <20101129013825.EF8DAEEB1E@mail.python.org> <4D0050E1.3000707@netwok.org> Message-ID: <4D00B617.5090602@voidspace.org.uk> On 09/12/2010 03:45, ?ric Araujo wrote: > Hello, > >> Author: raymond.hettinger >> Date: Mon Nov 29 02:38:25 2010 >> New Revision: 86855 >> Log: Do not add an obsolete unittest name to Py3.2. >> Modified: python/branches/py3k/Lib/unittest/case.py >> - # Old name for assertCountEqual() >> - assertItemsEqual = assertCountEqual > When we merge distutils2 back into the stdlib, our tests will have to > work with stdlib unittest in 3.3 or unittest2 for older Pythons (we?ll > still make standalone releases for them). unittest2 doesn?t have > assertCountEqual, unittest in 3.2+ doesn?t have assertItemsEqual, what?s > the plan for compatibility? unittest2 will continue to track changes in unittest. A 0.6.0 release is planned, with feature parity with the version of unittest in Python 3.2. Of course if you want to help that would be much appreciated. :-) All the best, Michael Foord > Regards -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From tnagyemail-mail at yahoo.fr Thu Dec 9 13:26:02 2010 From: tnagyemail-mail at yahoo.fr (Thomas Nagy) Date: Thu, 9 Dec 2010 12:26:02 +0000 (GMT) Subject: [Python-Dev] futures API Message-ID: <330887.27553.qm@web27506.mail.ukl.yahoo.com> Hello, I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86) by the futures module which was announced in python 3.2 beta. I am a bit stuck with it, so I have a few questions about the futures: 1. Is the futures API frozen? 2. How hard would it be to return the tasks processed in an output queue to process/consume the results while they are returned? The code does not seem to be very open for monkey patching. 3. How hard would it be to add new tasks dynamically (after a task is executed) and have the futures object never complete? 4. Is there a performance evaluation of the futures code (execution overhead) ? Thanks, Thomas From ocean-city at m2.ccsnet.ne.jp Thu Dec 9 13:49:49 2010 From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto) Date: Thu, 09 Dec 2010 21:49:49 +0900 Subject: [Python-Dev] OpenSSL Vulnerability (openssl-1.0.0a) In-Reply-To: <20101124162312.2109.1025683352.divmod.xquotient.215@localhost.localdomain> References: <4CEB3F72.7000006@m2.ccsnet.ne.jp> <20101123150219.29e20374@pitrou.net> <720EFE43-119F-4F2F-BCB1-939275B5FA6E@twistedmatrix.com> <1290585733.3642.2.camel@localhost.localdomain> <20101124150106.2109.660794265.divmod.xquotient.197@localhost.localdomain> <20101124161120.5ddd106c@pitrou.net> <20101124162312.2109.1025683352.divmod.xquotient.215@localhost.localdomain> Message-ID: <4D00D06D.5060201@m2.ccsnet.ne.jp> On 2010/11/25 1:23, exarkun at twistedmatrix.com wrote: > Ah. Okay, then Python 3.2 would be vulnerable. Good thing it isn't > released yet. ;) It seems OpenSSL 1.0.0c out. http://openssl.org/news/secadv_20101202.txt > 02-Dec-2010: Security Advisory: ciphersuite downgrade fix > 02-Dec-2010: OpenSSL 1.0.0c is now available, including important > bug and security fixes > 02-Dec-2010: OpenSSL 0.9.8q is now available, including important > bug and security fixes From alexander.belopolsky at gmail.com Thu Dec 9 15:51:36 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 9 Dec 2010 09:51:36 -0500 Subject: [Python-Dev] The fate of transform() and untransform() methods Message-ID: On Fri, Dec 3, 2010 at 1:05 PM, Guido van Rossum wrote: > On Fri, Dec 3, 2010 at 9:58 AM, R. David Murray wrote: .. >> I believe MAL's thought was that the addition of these methods had >> been approved pre-moratorium, but I don't know if that is a >> sufficient argument or not. > > It is not. > > The moratorium is intended to freeze the state of the language as > implemented, not whatever was discussed and approved but didn't get > implemented (that'd be a hole big enough to drive a truck through, as > the saying goes :-). > > Regardless of what I or others may have said before, I am not > currently a fan of adding transform() to either str or bytes. > I would like to restart the discussion under a separate subject because the original thread [1] went off the specific topic of the six new methods (2 methods x 3 types) added to builtins shortly before 3.2 beta was released. [2] The ticket that introduced the change is currently closed [3] even though the last message suggests that at least part of the change needs to be reverted. Note that reverting part of the patch is not entirely trivial because new codecs' documentation refers to bytes.[un]transform() both in the docstrings and the library reference. I think it will be the best to revert r86934 and resume the discussion of adding this functionality to 3.3 when we won't be constrained by the language moratorium. I will write a separate message with my thoughts about adding bytes codecs in 3.3. Let's keep this thread focused on what has to be done for 3.2. [1] http://mail.python.org/pipermail/python-dev/2010-December/106353.html [2] http://svn.python.org/view?view=rev&revision=86934 [3] http://bugs.python.org/issue7475 From mal at egenix.com Thu Dec 9 16:03:20 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 09 Dec 2010 16:03:20 +0100 Subject: [Python-Dev] The fate of transform() and untransform() methods In-Reply-To: References: Message-ID: <4D00EFB8.1090204@egenix.com> Alexander Belopolsky wrote: > On Fri, Dec 3, 2010 at 1:05 PM, Guido van Rossum wrote: >> On Fri, Dec 3, 2010 at 9:58 AM, R. David Murray wrote: > .. >>> I believe MAL's thought was that the addition of these methods had >>> been approved pre-moratorium, but I don't know if that is a >>> sufficient argument or not. >> >> It is not. >> >> The moratorium is intended to freeze the state of the language as >> implemented, not whatever was discussed and approved but didn't get >> implemented (that'd be a hole big enough to drive a truck through, as >> the saying goes :-). >> >> Regardless of what I or others may have said before, I am not >> currently a fan of adding transform() to either str or bytes. >> > > I would like to restart the discussion under a separate subject > because the original thread [1] went off the specific topic of the six > new methods (2 methods x 3 types) added to builtins shortly before 3.2 > beta was released. [2] The ticket that introduced the change is > currently closed [3] even though the last message suggests that at > least part of the change needs to be reverted. That's for Guido to decide. The moratorium, if at all, would only cover the new methods, not the other changes (readding the codecs and fixing the codecs.py module to work with bytes as well as Unicode), all of which were already discussed at length in several previous discussion, on tickets and on python-dev. I don't see much point in going through the same discussions over and over again. Fortunately, I'm on vacation next week, so don't have to go through all this again ;-) Cheers, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 09 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From fuzzyman at voidspace.org.uk Thu Dec 9 16:13:36 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 09 Dec 2010 15:13:36 +0000 Subject: [Python-Dev] The fate of transform() and untransform() methods In-Reply-To: <4D00EFB8.1090204@egenix.com> References: <4D00EFB8.1090204@egenix.com> Message-ID: <4D00F220.4070305@voidspace.org.uk> On 09/12/2010 15:03, M.-A. Lemburg wrote: > Alexander Belopolsky wrote: >> On Fri, Dec 3, 2010 at 1:05 PM, Guido van Rossum wrote: >>> On Fri, Dec 3, 2010 at 9:58 AM, R. David Murray wrote: >> .. >>>> I believe MAL's thought was that the addition of these methods had >>>> been approved pre-moratorium, but I don't know if that is a >>>> sufficient argument or not. >>> It is not. >>> >>> The moratorium is intended to freeze the state of the language as >>> implemented, not whatever was discussed and approved but didn't get >>> implemented (that'd be a hole big enough to drive a truck through, as >>> the saying goes :-). >>> >>> Regardless of what I or others may have said before, I am not >>> currently a fan of adding transform() to either str or bytes. >>> >> I would like to restart the discussion under a separate subject >> because the original thread [1] went off the specific topic of the six >> new methods (2 methods x 3 types) added to builtins shortly before 3.2 >> beta was released. [2] The ticket that introduced the change is >> currently closed [3] even though the last message suggests that at >> least part of the change needs to be reverted. > That's for Guido to decide. > Well, Guido *already* said no to transform / untransform in the previous thread. Michael > The moratorium, if at all, would only cover the new methods, > not the other changes (readding the codecs and fixing the > codecs.py module to work with bytes as well as Unicode), all > of which were already discussed at length in several previous > discussion, on tickets and on python-dev. > > I don't see much point in going through the same discussions > over and over again. Fortunately, I'm on vacation next week, > so don't have to go through all this again ;-) > > Cheers, -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From brett at python.org Thu Dec 9 17:36:56 2010 From: brett at python.org (Brett Cannon) Date: Thu, 9 Dec 2010 08:36:56 -0800 Subject: [Python-Dev] futures API In-Reply-To: <330887.27553.qm@web27506.mail.ukl.yahoo.com> References: <330887.27553.qm@web27506.mail.ukl.yahoo.com> Message-ID: On Thu, Dec 9, 2010 at 04:26, Thomas Nagy wrote: > Hello, > > I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86) by the futures module which was announced in python 3.2 beta. I am a bit stuck with it, so I have a few questions about the futures: > > 1. Is the futures API frozen? It will be once Python 3.2 final is released. -Brett > 2. How hard would it be to return the tasks processed in an output queue to process/consume the results while they are returned? The code does not seem to be very open for monkey patching. > 3. How hard would it be to add new tasks dynamically (after a task is executed) and have the futures object never complete? > 4. Is there a performance evaluation of the futures code (execution overhead) ? > > Thanks, > Thomas > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From fuzzyman at voidspace.org.uk Thu Dec 9 17:48:25 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 09 Dec 2010 16:48:25 +0000 Subject: [Python-Dev] futures API In-Reply-To: References: <330887.27553.qm@web27506.mail.ukl.yahoo.com> Message-ID: <4D010859.1020807@voidspace.org.uk> On 09/12/2010 16:36, Brett Cannon wrote: > On Thu, Dec 9, 2010 at 04:26, Thomas Nagy wrote: >> Hello, >> >> I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86) by the futures module which was announced in python 3.2 beta. I am a bit stuck with it, so I have a few questions about the futures: >> >> 1. Is the futures API frozen? > It will be once Python 3.2 final is released. Now that 3.2 beta 1 is out the api is effectively frozen though, the only changes that should be made are bugfixes unless you can convince the release manager that an api change is really *essential*. Michael > -Brett > >> 2. How hard would it be to return the tasks processed in an output queue to process/consume the results while they are returned? The code does not seem to be very open for monkey patching. >> 3. How hard would it be to add new tasks dynamically (after a task is executed) and have the futures object never complete? >> 4. Is there a performance evaluation of the futures code (execution overhead) ? >> >> Thanks, >> Thomas >> >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From mal at egenix.com Thu Dec 9 17:56:07 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 09 Dec 2010 17:56:07 +0100 Subject: [Python-Dev] The fate of transform() and untransform() methods In-Reply-To: <4D00F220.4070305@voidspace.org.uk> References: <4D00EFB8.1090204@egenix.com> <4D00F220.4070305@voidspace.org.uk> Message-ID: <4D010A27.1070902@egenix.com> Michael Foord wrote: > On 09/12/2010 15:03, M.-A. Lemburg wrote: >> Alexander Belopolsky wrote: >>> On Fri, Dec 3, 2010 at 1:05 PM, Guido van Rossum >>> wrote: >>>> On Fri, Dec 3, 2010 at 9:58 AM, R. David >>>> Murray wrote: >>> .. >>>>> I believe MAL's thought was that the addition of these methods had >>>>> been approved pre-moratorium, but I don't know if that is a >>>>> sufficient argument or not. >>>> It is not. >>>> >>>> The moratorium is intended to freeze the state of the language as >>>> implemented, not whatever was discussed and approved but didn't get >>>> implemented (that'd be a hole big enough to drive a truck through, as >>>> the saying goes :-). >>>> >>>> Regardless of what I or others may have said before, I am not >>>> currently a fan of adding transform() to either str or bytes. >>>> >>> I would like to restart the discussion under a separate subject >>> because the original thread [1] went off the specific topic of the six >>> new methods (2 methods x 3 types) added to builtins shortly before 3.2 >>> beta was released. [2] The ticket that introduced the change is >>> currently closed [3] even though the last message suggests that at >>> least part of the change needs to be reverted. >> That's for Guido to decide. >> > > Well, Guido *already* said no to transform / untransform in the previous > thread. I'm not sure he did and asked for clarification (see attached email). -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 09 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ -------------- next part -------------- An embedded message was scrubbed... From: "M.-A. Lemburg" Subject: Re: [Python-Dev] transform() and untransform() methods, and the codec registry Date: Mon, 06 Dec 2010 12:39:43 +0100 Size: 7468 URL: From alexander.belopolsky at gmail.com Thu Dec 9 18:06:46 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 9 Dec 2010 12:06:46 -0500 Subject: [Python-Dev] The fate of transform() and untransform() methods In-Reply-To: <4D00EFB8.1090204@egenix.com> References: <4D00EFB8.1090204@egenix.com> Message-ID: On Thu, Dec 9, 2010 at 10:03 AM, M.-A. Lemburg wrote: > Alexander Belopolsky wrote: .. >>?The ticket that introduced the change is >> currently closed [3] even though the last message suggests that at >> least part of the change needs to be reverted. > > That's for Guido to decide. > The decision will probably rest with the release manager, but Guido has clearly voiced his opinion. Note that I deliberately removed the codecs part form the subject, so that we can focus on the fate of {str,bytes,bytearray}.{transform,untransform} methods. I don't see any dispute about the fact that adding these methods breaks the moratorium or any suggestion that a case-by-case exception has been approved for this particular case. I don't think there is an option to keep these methods. The two available options are: 1. Revert r86934. 2. Revert C code changes, but keep the codecs and remove references to transform/untransform from documentation. From brian at sweetapp.com Thu Dec 9 18:02:41 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Thu, 9 Dec 2010 09:02:41 -0800 Subject: [Python-Dev] futures API In-Reply-To: <330887.27553.qm@web27506.mail.ukl.yahoo.com> References: <330887.27553.qm@web27506.mail.ukl.yahoo.com> Message-ID: <85AA5A0F-9EC4-4BBF-9A30-1547F4DA59E8@sweetapp.com> On Dec 9, 2010, at 4:26 AM, Thomas Nagy wrote: > Hello, > > I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86 > ) by the futures module which was announced in python 3.2 beta. I am > a bit stuck with it, so I have a few questions about the futures: > > 1. Is the futures API frozen? Yes. > 2. How hard would it be to return the tasks processed in an output > queue to process/consume the results while they are returned? The > code does not seem to be very open for monkey patching. You can associate a callback with a submitted future. That callback could add the future to your queue. > 3. How hard would it be to add new tasks dynamically (after a task > is executed) and have the futures object never complete? I'm not sure that I understand your question. You can submit new work to an Executor at until time until it is shutdown and a work item can take as long to complete as you want. If you are contemplating tasks that don't complete then maybe you could be better just scheduling a thread. > 4. Is there a performance evaluation of the futures code (execution > overhead) ? No. Scott Dial did make some performance improvements so he might have a handle on its overhead. Cheers, Brian > Thanks, > Thomas > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From mal at egenix.com Thu Dec 9 18:32:29 2010 From: mal at egenix.com (M.-A. Lemburg) Date: Thu, 09 Dec 2010 18:32:29 +0100 Subject: [Python-Dev] The fate of transform() and untransform() methods In-Reply-To: References: <4D00EFB8.1090204@egenix.com> Message-ID: <4D0112AD.6030103@egenix.com> Alexander Belopolsky wrote: > On Thu, Dec 9, 2010 at 10:03 AM, M.-A. Lemburg wrote: >> Alexander Belopolsky wrote: > .. >>> The ticket that introduced the change is >>> currently closed [3] even though the last message suggests that at >>> least part of the change needs to be reverted. >> >> That's for Guido to decide. >> > The decision will probably rest with the release manager, but Guido > has clearly voiced his opinion. FYI: Georg explicitly asked me whether I would have the patch ready for 3.2 and since I didn't have time to work on it, he volunteered to implement it, which I'd like to thank him for ! -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Dec 09 2010) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try our new mxODBC.Connect Python Database Interface for free ! :::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ From guido at python.org Thu Dec 9 19:42:27 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 9 Dec 2010 10:42:27 -0800 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <4CFCCB7F.9080605@egenix.com> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> Message-ID: On Mon, Dec 6, 2010 at 3:39 AM, M.-A. Lemburg wrote: > Guido van Rossum wrote: >> The moratorium is intended to freeze the state of the language as >> implemented, not whatever was discussed and approved but didn't get >> implemented (that'd be a hole big enough to drive a truck through, as >> the saying goes :-). > > Sure, but those two particular methods only provide interfaces > to the codecs sub-system without actually requiring any major > implementation changes. > > Furthermore, they "help ease adoption of Python 3.x" (quoted from > PEP 3003), since the functionality they add back was removed from > Python 3.0 in a way that makes it difficult to port Python2 > applications to Python3. > >> Regardless of what I or others may have said before, I am not >> currently a fan of adding transform() to either str or bytes. > > How should I read this ? Do want the methods to be removed again > and added back in 3.3 ? Given that it's in 3.2b1 I'm okay with keeping it. That's at best a +0. I'd be -0 if it wasn't already in. But anyway this should suffice to keep it in unless there are others strongly opposed. > Frankly, I'm a bit tired of constantly having to argue against > cutting down the Unicode and codec support in Python3. But transform() isn't really about Unicode or codec support -- it is about string-to-string and bytes-to-bytes transformations. At least the transform() API is clear about the distinction between codecs (which translate between bytes and string) and transforms (which keep the type unchanged) -- though I still don't like that the registries for transforms and codecs use the same namespace. Also bytes-bytes and string-string transforms use the same namespace even though the typical transform only supports one or the other. E.g. IMO all of the following should raise LookupError: >>> b'abc'.transform('rot13') Traceback (most recent call last): File "", line 1, in File "/Users/guido/p3/Lib/encodings/rot_13.py", line 16, in encode return (input.translate(rot13_map), len(input)) TypeError: expected an object with the buffer interface >>> b'abc'.decode('rot13') Traceback (most recent call last): File "", line 1, in File "/Users/guido/p3/Lib/encodings/rot_13.py", line 19, in decode return (input.translate(rot13_map), len(input)) AttributeError: 'memoryview' object has no attribute 'translate' >>> 'abc'.encode('rot13') Traceback (most recent call last): File "", line 1, in TypeError: encoder did not return a bytes object (type=str) >>> b''.decode('rot13') '' The latter may be a separate bug; b''.decode('anything') seems to not even attempt to look up the codec. -- --Guido van Rossum (python.org/~guido) From alexander.belopolsky at gmail.com Thu Dec 9 19:55:08 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 9 Dec 2010 13:55:08 -0500 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> Message-ID: On Thu, Dec 9, 2010 at 1:42 PM, Guido van Rossum wrote: .. > string-string transforms use the same namespace even though the > typical transform only supports one or the other. E.g. IMO all of the > following should raise LookupError: > >>>> b'abc'.transform('rot13') > Traceback (most recent call last): > .. > ? ?return (input.translate(rot13_map), len(input)) > TypeError: expected an object with the buffer interface This is actually *very* misleading because >>> 'abc'.transform('rot13') 'nop' works even though 'abc' is not "an object with the buffer interface". >>> memoryview('abc') Traceback (most recent call last): .. TypeError: cannot make memory view because object does not have the buffer interface From solipsis at pitrou.net Thu Dec 9 20:17:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 9 Dec 2010 20:17:49 +0100 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> Message-ID: <20101209201749.79edd5ce@pitrou.net> On Thu, 9 Dec 2010 13:55:08 -0500 Alexander Belopolsky wrote: > On Thu, Dec 9, 2010 at 1:42 PM, Guido van Rossum wrote: > .. > > string-string transforms use the same namespace even though the > > typical transform only supports one or the other. E.g. IMO all of the > > following should raise LookupError: > > > >>>> b'abc'.transform('rot13') > > Traceback (most recent call last): > > .. > > ? ?return (input.translate(rot13_map), len(input)) > > TypeError: expected an object with the buffer interface > > This is actually *very* misleading because > > >>> 'abc'.transform('rot13') > 'nop' > > works even though 'abc' is not "an object with the buffer interface". Agreed. It was already pointed out in the parent thread. I would say my opinion on keeping transform()/untransform() is +0 if these error messages (and preferably the exception type as well) get improved. Otherwise we'd better revert them and add a more polished version in 3.3. Regards Antoine. From merwok at netwok.org Thu Dec 9 20:19:06 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 09 Dec 2010 20:19:06 +0100 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> Message-ID: <4D012BAA.80401@netwok.org> Le 09/12/2010 19:42, Guido van Rossum a ?crit : > Given that it's in 3.2b1 I'm okay with keeping it. That's at best a > +0. [...] > though I still don't like that the registries for transforms and > codecs use the same namespace. Also bytes-bytes and > string-string transforms use the same namespace even though the > typical transform only supports one or the other. E.g. IMO all of the > following should raise LookupError: [...] Although I?d regret not having transform/untransform in 3.2, I think there needs to be a discussion about this namespace issue before the new methods ship. Regards From python at mrabarnett.plus.com Thu Dec 9 21:21:32 2010 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 09 Dec 2010 20:21:32 +0000 Subject: [Python-Dev] Can't compile regex module with Python 3.2 In-Reply-To: References: <4D004568.4000905@mrabarnett.plus.com> <4D006D88.10300@v.loewis.de> Message-ID: <4D013A4C.4090703@mrabarnett.plus.com> On 09/12/2010 05:57, Alexander Belopolsky wrote: > On Thu, Dec 9, 2010 at 12:47 AM, "Martin v. L?wis" wrote: > .. >>> However, in Python 3.2b1 the library python32.lib contains only >>> _PyUnicode_IsWhitespace, therefore breaking the build. >>> >>> Is this change intentional? If so, why does unicodeobject.h still do >>> the mapping? >> >> Are you sure about this? It's not intentional (except in the limited ABI). > > this does seem to be intentional: > > > $ svn log -r 84177 > ------------------------------------------------------------------------ > r84177 | amaury.forgeotdarc | 2010-08-18 16:44:58 -0400 (Wed, 18 Aug > 2010) | 9 lines > > #5127: Even on narrow unicode builds, the C functions that access the Unicode > Database (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others) now accept > and return characters from the full Unicode range (Py_UCS4). > > The differences from Python code are few: > - unicodedata.numeric(), unicodedata.decimal() and unicodedata.digit() > now return the correct value for large code points > - repr() may consider more characters as printable. > > ------------------------------------------------------------------------ > http://svn.python.org/view?view=rev&revision=84177 > I have it compiling now. From alexander.belopolsky at gmail.com Thu Dec 9 21:29:35 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 9 Dec 2010 15:29:35 -0500 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <20101209201749.79edd5ce@pitrou.net> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> <20101209201749.79edd5ce@pitrou.net> Message-ID: On Thu, Dec 9, 2010 at 2:17 PM, Antoine Pitrou wrote: > On Thu, 9 Dec 2010 13:55:08 -0500 > Alexander Belopolsky wrote: .. >> This is actually *very* misleading because >> >> >>> 'abc'.transform('rot13') >> 'nop' >> >> works even though 'abc' is not "an object with the buffer interface". > > Agreed. It was already pointed out in the parent thread. > I would say my opinion on keeping transform()/untransform() is +0 if > these error messages (and preferably the exception type as well) get > improved. Otherwise we'd better revert them and add a more polished > version in 3.3. Error messages is only one of the problems. User confusion over which codec supports which types is another. Why, for example rot13 works on str and not on bytes? It only affects ASCII range, so isn't it natural to expect b'abc'.transform('rot13') to work? Well, presumably this is so because Caesar did not know about bytes and his "cypher" was about character shuffling. In this case, should't it also shuffle other code points assigned to Latin letters? Given how "useful" rot13 is in practice, I feel that it was only added to justify adding str.transform(). There are other problems raised on the issue and not addressed in the tracker discussion. For example, both Victor and I expressed concern about having builitn methods that do import behind the scenes. Granted, this issue already exists with encode/decode methods, but these are usable without providing an explicit encoding and in this form do not have side-effects. Another problem is that with str.transform(), users are encouraged to write programs in which data stored in strings is not always interpreted as Unicode. For example, when I see a 'n' in a string variable, it may actually mean 'a' because it has been scrambled by rot13. Again, rot13 is not a realistic example, but as users are encouraged to create their own string to string codecs, we may soon find ourselves in the same mess as we have with 2.x programs trying to support multiple locales. As far as realistic examples go, Unicode transformations such as case folding, normalization or decimal to ASCII translation have not been considered in str.transform() design. The str.transform/str.untransform pair may or may not be a good solution for these cases. One obvious issue being that these transformations are often not invertible. I admit I have more questions than answers at this point, but a design that adds the same two methods to three builtin types with very different usage patterns (str, bytes and bytearray) does not seem to be well thought out. The str type already has 40+ methods many of which are not well-suited to handle the complexities inherent in Unicode. Rather than rushing in two more methods that will prove to be about as useful as str.swapcase(), lets consider actual use cases and come up with a design that will properly address them. From db3l.net at gmail.com Thu Dec 9 22:34:25 2010 From: db3l.net at gmail.com (David Bolen) Date: Thu, 09 Dec 2010 16:34:25 -0500 Subject: [Python-Dev] kill_python on windows buildbots References: <20101113133712.60e9be27@pitrou.net> <4CEB7E12.1070201@snakebite.org> <4CFF21D0.7050705@m2.ccsnet.ne.jp> Message-ID: Hirokazu Yamamoto writes: > Yes, but test can freeze. In that case, I'm worried that > (snip) > rt.bat .... # freeze here (will be halt by buildbot) > vcbuild .... & kill_python_d # Will this be called? > in test.bat. Yeah, you're right. It may be impossible to completely eliminate the risk of a process getting stuck without doing something external to the build process, so long as the first thing a new build tries to do is an svn checkout that will fail if the process is still running. Having the kill operation in clean.bat covers the vast majority of the common cases with a minimum change, so seems the simplest. -- David From ncoghlan at gmail.com Thu Dec 9 23:18:00 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Dec 2010 08:18:00 +1000 Subject: [Python-Dev] [Python-checkins] r87145 - python/branches/py3k/Doc/whatsnew/3.2.rst In-Reply-To: <20101209164154.5A45DEEAA3@mail.python.org> References: <20101209164154.5A45DEEAA3@mail.python.org> Message-ID: On Fri, Dec 10, 2010 at 2:41 AM, raymond.hettinger wrote: > @@ -588,7 +593,12 @@ > ? pointing to the original callable function. ?This allows wrapped functions to > ? be introspected. ?It also copies :attr:`__annotations__` if defined. ?And now > ? it also gracefully skips over missing attributes such as :attr:`__doc__` which > - ?might not be defined for the wrapped callable. > + ?might not be defined for the wrapped callable: > + > + ?>>> callable(max) > + ?True > + ?>>> callable(20) > + ?False Placeholder example? Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From raymond.hettinger at gmail.com Thu Dec 9 23:39:38 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 9 Dec 2010 14:39:38 -0800 Subject: [Python-Dev] futures API In-Reply-To: <85AA5A0F-9EC4-4BBF-9A30-1547F4DA59E8@sweetapp.com> References: <330887.27553.qm@web27506.mail.ukl.yahoo.com> <85AA5A0F-9EC4-4BBF-9A30-1547F4DA59E8@sweetapp.com> Message-ID: <1A7546B0-2999-481B-904A-D1711DB65FEB@gmail.com> On Dec 9, 2010, at 9:02 AM, Brian Quinlan wrote: > > On Dec 9, 2010, at 4:26 AM, Thomas Nagy wrote: > >> Hello, >> >> I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86) by the futures module which was announced in python 3.2 beta. I am a bit stuck with it, so I have a few questions about the futures: >> >> 1. Is the futures API frozen? > > Yes. Yes, unless the current API is defective in some way. A beta1 release is a chance for everyone to exercise the new API and discover whether it is problematic in any real world applications. > >> 2. How hard would it be to return the tasks processed in an output queue to process/consume the results while they are returned? The code does not seem to be very open for monkey patching. > > You can associate a callback with a submitted future. That callback could add the future to your queue. The would be a good example for the docs. > >> 3. How hard would it be to add new tasks dynamically (after a task is executed) and have the futures object never complete? > > I'm not sure that I understand your question. You can submit new work to an Executor at until time until it is shutdown and a work item can take as long to complete as you want. If you are contemplating tasks that don't complete then maybe you could be better just scheduling a thread. > >> 4. Is there a performance evaluation of the futures code (execution overhead) ? > > No. Scott Dial did make some performance improvements so he might have a handle on its overhead. FWIW, the source code is short and readable. From my quick read, it looks to be a relatively thin wrapper/adapter around existing tools. Most of the work still gets done by the threads or processes themselves. Think of this as a cleaner, more centralized API around the current toolset -- there is no deep new technology under the hood. Raymond From raymond.hettinger at gmail.com Thu Dec 9 23:45:57 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 9 Dec 2010 14:45:57 -0800 Subject: [Python-Dev] [Python-checkins] r87145 - python/branches/py3k/Doc/whatsnew/3.2.rst In-Reply-To: References: <20101209164154.5A45DEEAA3@mail.python.org> Message-ID: <09990EAC-E8CA-4ADD-807B-93E6650215D7@gmail.com> On Dec 9, 2010, at 2:18 PM, Nick Coghlan wrote: > On Fri, Dec 10, 2010 at 2:41 AM, raymond.hettinger > wrote: >> @@ -588,7 +593,12 @@ >> pointing to the original callable function. This allows wrapped functions to >> be introspected. It also copies :attr:`__annotations__` if defined. And now >> it also gracefully skips over missing attributes such as :attr:`__doc__` which >> - might not be defined for the wrapped callable. >> + might not be defined for the wrapped callable: >> + >> + >>> callable(max) >> + True >> + >>> callable(20) >> + False > > Placeholder example? > Trivial function begets a trivial example :-) If you think it warrants more, I'm open to suggestions. Raymond From raymond.hettinger at gmail.com Thu Dec 9 23:54:24 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 9 Dec 2010 14:54:24 -0800 Subject: [Python-Dev] sWAPcASE Was: transform() and untransform() methods, and the codec registry In-Reply-To: References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> <20101209201749.79edd5ce@pitrou.net> Message-ID: <22FDCC84-2D68-41DE-A421-80C823CB3020@gmail.com> On Dec 9, 2010, at 12:29 PM, Alexander Belopolsky wrote: > > The str type already has 40+ methods many of which are not well-suited > to handle the complexities inherent in Unicode. Rather than rushing > in two more methods that will prove to be about as useful as > str.swapcase(), lets consider actual use cases and come up with a > design that will properly address them. It would make me happy if we could agree to kill or at least mortally wound str.swapcase(). I did some research on what it is go for and found that it is a vestige of an old word processor command to handle the case where a user accidentally left the caps lock key turned-on. AFAICT using Google's code search, it has nearly zero value for Python scripts. It does have a cost however, the code search turned-up many cases where people were writing string like objects and included swapcase() just so they could match the built-in API. It's time for swapcase() to go the way of the dinosaurs. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Thu Dec 9 23:54:12 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 09 Dec 2010 17:54:12 -0500 Subject: [Python-Dev] [Python-checkins] r87145 - python/branches/py3k/Doc/whatsnew/3.2.rst In-Reply-To: <09990EAC-E8CA-4ADD-807B-93E6650215D7@gmail.com> References: <20101209164154.5A45DEEAA3@mail.python.org> <09990EAC-E8CA-4ADD-807B-93E6650215D7@gmail.com> Message-ID: <4D015E14.3040807@trueblade.com> On 12/9/2010 5:45 PM, Raymond Hettinger wrote: > > On Dec 9, 2010, at 2:18 PM, Nick Coghlan wrote: > >> On Fri, Dec 10, 2010 at 2:41 AM, raymond.hettinger >> wrote: >>> @@ -588,7 +593,12 @@ >>> pointing to the original callable function. This allows wrapped functions to >>> be introspected. It also copies :attr:`__annotations__` if defined. And now >>> it also gracefully skips over missing attributes such as :attr:`__doc__` which >>> - might not be defined for the wrapped callable. >>> + might not be defined for the wrapped callable: >>> + >>> +>>> callable(max) >>> + True >>> +>>> callable(20) >>> + False >> >> Placeholder example? >> > > Trivial function begets a trivial example :-) > > If you think it warrants more, I'm open to suggestions. I think the issue is that the section is talking about functools.wraps, the example is for callable(). From merwok at netwok.org Fri Dec 10 00:03:39 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 10 Dec 2010 00:03:39 +0100 Subject: [Python-Dev] [Python-checkins] r86855 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py In-Reply-To: <4D00B617.5090602@voidspace.org.uk> References: <20101129013825.EF8DAEEB1E@mail.python.org> <4D0050E1.3000707@netwok.org> <4D00B617.5090602@voidspace.org.uk> Message-ID: <4D01604B.4070400@netwok.org> Le 09/12/2010 11:57, Michael Foord a ?crit : > unittest2 will continue to track changes in unittest. A 0.6.0 release is > planned, with feature parity with the version of unittest in Python 3.2. All right. We?ll just run a sed over our tests and bump our required unittest2 version. Thanks for the answer. > Of course if you want to help that would be much appreciated. :-) I would if I could, but I have my own synchronization to take care of :) Best regards From martin at v.loewis.de Fri Dec 10 00:17:28 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 10 Dec 2010 00:17:28 +0100 Subject: [Python-Dev] OpenSSL Vulnerability (openssl-1.0.0a) In-Reply-To: <4D00D06D.5060201@m2.ccsnet.ne.jp> References: <4CEB3F72.7000006@m2.ccsnet.ne.jp> <20101123150219.29e20374@pitrou.net> <720EFE43-119F-4F2F-BCB1-939275B5FA6E@twistedmatrix.com> <1290585733.3642.2.camel@localhost.localdomain> <20101124150106.2109.660794265.divmod.xquotient.197@localhost.localdomain> <20101124161120.5ddd106c@pitrou.net> <20101124162312.2109.1025683352.divmod.xquotient.215@localhost.localdomain> <4D00D06D.5060201@m2.ccsnet.ne.jp> Message-ID: <4D016388.1060906@v.loewis.de> Am 09.12.2010 13:49, schrieb Hirokazu Yamamoto: > On 2010/11/25 1:23, exarkun at twistedmatrix.com wrote: >> Ah. Okay, then Python 3.2 would be vulnerable. Good thing it isn't >> released yet. ;) > > It seems OpenSSL 1.0.0c out. > > http://openssl.org/news/secadv_20101202.txt > >> 02-Dec-2010: Security Advisory: ciphersuite downgrade fix >> 02-Dec-2010: OpenSSL 1.0.0c is now available, including > important > bug and security fixes >> 02-Dec-2010: OpenSSL 0.9.8q is now available, including > important > bug and security fixes I don't plan upgrading the Windows build before 3.2; I have already patched the OpenSSL copy that we use. Regards, Martin From eric at trueblade.com Fri Dec 10 00:10:38 2010 From: eric at trueblade.com (Eric Smith) Date: Thu, 09 Dec 2010 18:10:38 -0500 Subject: [Python-Dev] sWAPcASE Was: transform() and untransform() methods, and the codec registry In-Reply-To: <22FDCC84-2D68-41DE-A421-80C823CB3020@gmail.com> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> <20101209201749.79edd5ce@pitrou.net> <22FDCC84-2D68-41DE-A421-80C823CB3020@gmail.com> Message-ID: <4D0161EE.2080506@trueblade.com> On 12/9/2010 5:54 PM, Raymond Hettinger wrote: > It would make me happy if we could agree to kill or at least mortally wound > str.swapcase(). I did some research on what it is go for and found > that it is a vestige of an old word processor command to handle > the case where a user accidentally left the caps lock key turned-on. > AFAICT using Google's code search, it has nearly zero value for > Python scripts. It does have a cost however, the code search turned-up > many cases where people were writing string like objects and included > swapcase() just so they could match the built-in API. > > It's time for swapcase() to go the way of the dinosaurs. +1, assuming the normal deprecation process. If we're looking to reduce the number of methods on str, I wouldn't mind seeing center() and zfill() also go away, since they're trivially replaced by format(). >>> '3'.zfill(10) '0000000003' >>> format('3', '>010') '0000000003' >>> '3'.center(10) ' 3 ' >>> format('3', '^10') ' 3 ' >>> '3'.center(10, '-') '----3-----' >>> format('3', '-^10') '----3-----' Although I'll grant this the case for the demise of center() and zfill() isn't as strong as for swapcase(). It's truly useless. Eric. From martin at v.loewis.de Fri Dec 10 00:23:32 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 10 Dec 2010 00:23:32 +0100 Subject: [Python-Dev] Can't compile regex module with Python 3.2 In-Reply-To: References: <4D004568.4000905@mrabarnett.plus.com> <4D006D88.10300@v.loewis.de> Message-ID: <4D0164F4.9040500@v.loewis.de> Am 09.12.2010 06:57, schrieb Alexander Belopolsky: > On Thu, Dec 9, 2010 at 12:47 AM, "Martin v. L?wis" wrote: > .. >>> However, in Python 3.2b1 the library python32.lib contains only >>> _PyUnicode_IsWhitespace, therefore breaking the build. >>> >>> Is this change intentional? If so, why does unicodeobject.h still do >>> the mapping? >> >> Are you sure about this? It's not intentional (except in the limited ABI). > > this does seem to be intentional: Oops, right; I was confusing this issue 8654. Then going back to the OP: Why exactly does that break the build? There should be no problem - you shouldn't see _PyUnicodeUCS2_IsWhitespace anywhere in 3.2. Regards, Martin From ncoghlan at gmail.com Fri Dec 10 00:28:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Dec 2010 09:28:55 +1000 Subject: [Python-Dev] [Python-checkins] r87145 - python/branches/py3k/Doc/whatsnew/3.2.rst In-Reply-To: <4D015E14.3040807@trueblade.com> References: <20101209164154.5A45DEEAA3@mail.python.org> <09990EAC-E8CA-4ADD-807B-93E6650215D7@gmail.com> <4D015E14.3040807@trueblade.com> Message-ID: On Fri, Dec 10, 2010 at 8:54 AM, Eric Smith wrote: > On 12/9/2010 5:45 PM, Raymond Hettinger wrote: >> Trivial function begets a trivial example :-) >> >> If you think it warrants more, I'm open to suggestions. > > I think the issue is that the section is talking about functools.wraps, the > example is for callable(). Yeah, sorry about that - I should have been clearer as to what I was asking about. It looks like the callable example had been copied to get the formatting started for the updated functools.wraps example, but the actual example for the latter didn't get filled in. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From solipsis at pitrou.net Fri Dec 10 00:29:30 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 10 Dec 2010 00:29:30 +0100 Subject: [Python-Dev] sWAPcASE Was: transform() and untransform() methods, and the codec registry References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> <20101209201749.79edd5ce@pitrou.net> <22FDCC84-2D68-41DE-A421-80C823CB3020@gmail.com> <4D0161EE.2080506@trueblade.com> Message-ID: <20101210002930.451efe65@pitrou.net> On Thu, 09 Dec 2010 18:10:38 -0500 Eric Smith wrote: > On 12/9/2010 5:54 PM, Raymond Hettinger wrote: > > It would make me happy if we could agree to kill or at least mortally wound > > str.swapcase(). I did some research on what it is go for and found > > that it is a vestige of an old word processor command to handle > > the case where a user accidentally left the caps lock key turned-on. > > AFAICT using Google's code search, it has nearly zero value for > > Python scripts. It does have a cost however, the code search turned-up > > many cases where people were writing string like objects and included > > swapcase() just so they could match the built-in API. > > > > It's time for swapcase() to go the way of the dinosaurs. > > +1, assuming the normal deprecation process. > > If we're looking to reduce the number of methods on str, I wouldn't mind > seeing center() and zfill() also go away, since they're trivially > replaced by format(). Well, it's only trivial when you know format()'s wicked mini-language by heart :/ center() is easy and obvious. zfill() is arguably a bit too specialized. Regards Antoine. From stutzbach at google.com Fri Dec 10 00:35:18 2010 From: stutzbach at google.com (Daniel Stutzbach) Date: Thu, 9 Dec 2010 15:35:18 -0800 Subject: [Python-Dev] Can't compile regex module with Python 3.2 In-Reply-To: <4D004568.4000905@mrabarnett.plus.com> References: <4D004568.4000905@mrabarnett.plus.com> Message-ID: On Wed, Dec 8, 2010 at 6:56 PM, MRAB wrote: > Is this change intentional? If so, why does unicodeobject.h still do > the mapping? > In 3.2b1, unicodeobject.h doesn't map _PyUnicode_IsWhitespace: http://svn.python.org/view/python/tags/r32b1/Include/unicodeobject.h?view=markup Do you have an old unicodeobject.h somehow? -- Daniel Stutzbach -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Dec 10 00:59:50 2010 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 09 Dec 2010 23:59:50 +0000 Subject: [Python-Dev] Can't compile regex module with Python 3.2 In-Reply-To: References: <4D004568.4000905@mrabarnett.plus.com> Message-ID: <4D016D76.2060806@mrabarnett.plus.com> On 09/12/2010 23:35, Daniel Stutzbach wrote: > On Wed, Dec 8, 2010 at 6:56 PM, MRAB > wrote: > > Is this change intentional? If so, why does unicodeobject.h still do > the mapping? > > > In 3.2b1, unicodeobject.h doesn't map _PyUnicode_IsWhitespace: > > http://svn.python.org/view/python/tags/r32b1/Include/unicodeobject.h?view=markup > > Do you have an old unicodeobject.h somehow? > Basically, yes. I'm already having to deal with 2.5, 2.6, 2.7 and 3.1, and there was a mismatch between older 3.2 files from the trunk downloaded a while back and the 3.2b1 .lib file. It all seems to be working now. From raymond.hettinger at gmail.com Fri Dec 10 01:18:14 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 9 Dec 2010 16:18:14 -0800 Subject: [Python-Dev] API for the new sysconfig module Message-ID: Does anyone know why this needed a separate module and so many accessor functions? ISTM it mostly could have been reduced to single call returning a nested dictionary. Raymond from sysconfig import * import json def sysconf(): return dict(paths = get_paths(), config_vars = get_config_vars(), platform = get_platform(), version = get_python_version(), scheme_names = get_scheme_names(), ) print(json.dumps(sysconf(), indent=2)) { "config_vars": { "EXE": ".exe", "VERSION": "32", "py_version_nodot": "32", "exec_prefix": "C:\\Python32", "platbase": "C:\\Python32", "userbase": "C:\\Documents and Settings\\Raymond\\Application Data\\Python", "LIBDEST": "C:\\Python32\\Lib", "py_version_short": "3.2", "prefix": "C:\\Python32", "base": "C:\\Python32", "SO": ".pyd", "projectbase": "C:\\Python32", "BINLIBDEST": "C:\\Python32\\Lib", "srcdir": "C:\\Python32", "py_version": "3.2b1", "abiflags": "", "INCLUDEPY": "C:\\Python32\\Include", "BINDIR": "C:\\Python32" }, "platform": "win32", "version": "3.2", "scheme_names": [ "nt", "nt_user", "os2", "os2_home", "osx_framework_user", "posix_home", "posix_prefix", "posix_user" ], "paths": { "platinclude": "C:\\Python32\\Include", "platstdlib": "C:\\Python32\\Lib", "platlib": "C:\\Python32\\Lib\\site-packages", "purelib": "C:\\Python32\\Lib\\site-packages", "stdlib": "C:\\Python32\\Lib", "scripts": "C:\\Python32\\Scripts", "include": "C:\\Python32\\Include", "data": "C:\\Python32" } } From brian at sweetapp.com Fri Dec 10 01:22:03 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Thu, 9 Dec 2010 16:22:03 -0800 Subject: [Python-Dev] futures API In-Reply-To: <1A7546B0-2999-481B-904A-D1711DB65FEB@gmail.com> References: <330887.27553.qm@web27506.mail.ukl.yahoo.com> <85AA5A0F-9EC4-4BBF-9A30-1547F4DA59E8@sweetapp.com> <1A7546B0-2999-481B-904A-D1711DB65FEB@gmail.com> Message-ID: On Dec 9, 2010, at 2:39 PM, Raymond Hettinger wrote: > > On Dec 9, 2010, at 9:02 AM, Brian Quinlan wrote: > >> >> On Dec 9, 2010, at 4:26 AM, Thomas Nagy wrote: >> >>> Hello, >>> >>> I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86 >>> ) by the futures module which was announced in python 3.2 beta. I >>> am a bit stuck with it, so I have a few questions about the futures: >>> >>> 1. Is the futures API frozen? >> >> Yes. > > Yes, unless the current API is defective in some way. A beta1 > release is a chance for everyone to exercise the new API and > discover whether it is problematic in any real world applications. > >> >>> 2. How hard would it be to return the tasks processed in an output >>> queue to process/consume the results while they are returned? The >>> code does not seem to be very open for monkey patching. >> >> You can associate a callback with a submitted future. That callback >> could add the future to your queue. > > The would be a good example for the docs. I don't know what Thomas' use case is but I expect that taking the results of a future and asynchronously sticking it in another queue is not typical. Cheers, Brian >> >>> 3. How hard would it be to add new tasks dynamically (after a task >>> is executed) and have the futures object never complete? >> >> I'm not sure that I understand your question. You can submit new >> work to an Executor at until time until it is shutdown and a work >> item can take as long to complete as you want. If you are >> contemplating tasks that don't complete then maybe you could be >> better just scheduling a thread. >> >>> 4. Is there a performance evaluation of the futures code >>> (execution overhead) ? >> >> No. Scott Dial did make some performance improvements so he might >> have a handle on its overhead. > > FWIW, the source code is short and readable. From my quick read, it > looks to be a relatively thin wrapper/adapter around existing > tools. Most of the work still gets done by the threads or processes > themselves. Think of this as a cleaner, more centralized API around > the current toolset -- there is no deep new technology under the hood. > > Raymond > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From ncoghlan at gmail.com Fri Dec 10 01:26:09 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Dec 2010 10:26:09 +1000 Subject: [Python-Dev] Format factories (was Re: sWAPcASE Was: transform() and untransform() methods, and the codec registry) Message-ID: On Fri, Dec 10, 2010 at 9:29 AM, Antoine Pitrou wrote: > On Thu, 09 Dec 2010 18:10:38 -0500 > Eric Smith wrote: >> If we're looking to reduce the number of methods on str, I wouldn't mind >> seeing center() and zfill() also go away, since they're trivially >> replaced by format(). > > Well, it's only trivial when you know format()'s wicked mini-language by > heart :/ ?center() is easy and obvious. zfill() is arguably a bit too > specialized. I've occasionally wondered if string formatting [1] and the struct module [2] would benefit from format building functions that made them easier to use without necessarily learning the cryptic mini-languages off by heart. For example, a "string.make_format_spec" function might have a signature like: def make_format_spec(fill=None, align=None, sign=None, width=None, precision=None, display_type='s', alternate=False, commas=False, numeric_case=None) "align" would accept not only the actual format symbols ('<', '>', '=', '^'), but also the corresponding names ('left', 'right', 'numeric', 'center'). "numeric_case" would accept None, 'upper' or 'lower' (affecting the formatting of hex and floating point values). If the stated numeric case differs from that specified by the display type, raise a ValueError. For an unspecified numeric case, the affected display types would default to 'lower'. Similarly, "display_type" would accept long names in addition to the character codes: 's': 'str' 'b': 'binary' 'c': 'chr' 'd': 'int' 'o': 'octal' 'x', 'X': 'hex' (numeric case controls display of digits A-F) 'n': 'locale' 'e', 'E': 'exponent' (numeric case controls display of exponent as well as infinite and NaN values) 'f', 'F': 'float' (numeric case controls display of exponent as well as infinite and NaN values) 'g', 'G': 'general' (numeric case controls display of exponent as well as infinite and NaN values) '%': 'percent' (numeric case controls display of exponent as well as infinite and NaN values) There could also be a corresponding parse_format_spec that produced a named tuple with the appropriate details. Cheers, Nick. [1] http://docs.python.org/dev/library/string#format-specification-mini-language [2] http://docs.python.org/dev/library/struct#format-strings -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Fri Dec 10 01:31:43 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Dec 2010 10:31:43 +1000 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: References: Message-ID: On Fri, Dec 10, 2010 at 10:18 AM, Raymond Hettinger wrote: > Does anyone know why this needed a separate module and so many accessor functions? > ISTM it mostly could have been reduced to single call returning a nested dictionary. Tarek will likely answer for himself, but I believe it is a matter of providing like-for-like replacements of existing distutils APIs, specifically: http://docs.python.org/dev/distutils/apiref.html#module-distutils.sysconfig The old way required source code available ability at runtime, the new sysconfig module captures everything it needs at build time. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Fri Dec 10 01:59:11 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 9 Dec 2010 16:59:11 -0800 Subject: [Python-Dev] sWAPcASE Was: transform() and untransform() methods, and the codec registry In-Reply-To: <22FDCC84-2D68-41DE-A421-80C823CB3020@gmail.com> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> <20101209201749.79edd5ce@pitrou.net> <22FDCC84-2D68-41DE-A421-80C823CB3020@gmail.com> Message-ID: On Thu, Dec 9, 2010 at 2:54 PM, Raymond Hettinger wrote: > > On Dec 9, 2010, at 12:29 PM, Alexander Belopolsky wrote: > > The str type already has 40+ methods many of which are not well-suited > to handle the complexities inherent in Unicode. ??Rather than rushing > in two more methods that will prove to be about as useful as > str.swapcase(), lets consider actual use cases and come up with a > design that will properly address them. > > It would make me happy if we could agree to kill or at least mortally wound > str.swapcase(). ? ?I did some research on what it is go for and found > that it is a vestige of an old word processor command to handle > the case where a user accidentally left the caps lock key turned-on. > AFAICT using Google's code search, it has nearly zero value for > Python scripts. ? It does have a cost however, the code search turned-up > many cases where people were writing string like objects and included > swapcase() just so they could match the built-in API. > It's time for swapcase() to go the way of the dinosaurs. I fully admit the mistakes I made by introducing swapcase() and a few other weird string methods. It happened many eons ago; before they were string methods they were functions in the "string" module. At the same time I cannot bring up any enthusiasm for trying to kill these off. The cost of the deprecation process is probably more than the cost of keeping them around, if you integrate over the entire community for t -> infinity. I don't see them as harmful. We had our chance to kill them in Py3k, and we didn't. -- --Guido van Rossum (python.org/~guido) From greg at krypto.org Fri Dec 10 05:24:33 2010 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 9 Dec 2010 20:24:33 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: On Tue, Dec 7, 2010 at 5:51 PM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > > > Indeed - I was very surprised to find just now that calling > > "logging.warn('Whatever')" is not the same as doing "log = > > logging.getLogger(); log.warn('Whatever')". > > Don't know why you'd be surprised: it's been that way since logging was > added to > Python, and the logging.debug() etc. are documented as convenience methods > for > casual use in throwaway scripts, interactive sessions etc. The convenience > is in > that you don't need to specify a logger (the root logger is used) and that > basicConfig() is called for you. > Hahaha. :) Well, I won't be suggesting to anyone at work that we throw away our entire bazillion line codebase just because all of it happily relies on logging.{debug,info,warn,error,exception} functions and all log messages go through a single root logger. I'd argue that anyone using a multi-logger hierarchy has already implemented overkill and that the default for anyone wanting to log something should be to simply call the above functions directly from the logging module. This simplistic easy usage somewhat echo's Glenn's comment on this thread about logging seeming way to daunting as presented today. It needn't be. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri Dec 10 05:29:19 2010 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 9 Dec 2010 20:29:19 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <4CFFC5B0.3040006@g.nevcal.com> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <4CFFC5B0.3040006@g.nevcal.com> Message-ID: On Wed, Dec 8, 2010 at 9:51 AM, Glenn Linderman > wrote: > On 12/8/2010 4:15 AM, Vinay Sajip wrote: > > You're complaining about too much documentation?! Don't measure it by weight! > > On 12/8/2010 5:57 AM, Vinay Sajip wrote: > > Of course I understand I could be wrong > about this, but I don't recall when a stdlib maintainer has said to me, "I want > to start using logging in stdlib module XXX, but can't justify it because ..." > > > So I'm a fairly new Python user, but 30 years programming experience. > > When I first looked at the Python logging facility (and again today, when I > looked again), I found a simple example of debugging logging. Then a bunch > of stuff about how to configure rotating logs. Then something about logging > levels. And then a dissertation on the theory of loggers "The logging > library takes a modular approach and offers the several categories of > components: loggers, handlers, filters, and formatters." And I hadn't > gotten through 10% of the documentation yet, per the scrollbar. > > My reaction the first time was "Sounds sophisticated and complex. I think > I'll do something simpler for now, and maybe someday, when I have a spare > week, I'll read the documentation and see if the benefits are worth the > effort." > > OK, so now you've discovered that too much documentation can be a turn > off... at least, if it is presented from the top to describe the > sophistication of the facility, rather than how easy it is to use (if it is, > I still haven't gotten to 10%, and I still don't know that). > Exactly. All I ever recommend people do is: import logging ... logging.warn('doing something a bit odd.') ... for x in thing: logging.debug('working on %r', x) ... And be done with it. If they are controlling their __main__ they'll probably want to call a common function to setup the log message formatting and where it gets written to from there. Otherwise ignore that and leave that up to the person using your library (directly, indirectly or otherwise) to do that in their __main__ code. For logging to be useful it needs to be dirt simple. The above is. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Fri Dec 10 09:06:49 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 10 Dec 2010 08:06:49 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: Gregory P. Smith krypto.org> writes: > Hahaha. :) ?Well, I won't be suggesting to anyone at work that we throw away > our entire bazillion line codebase just because all of it happily relies on > logging.{debug,info,warn,error,exception} functions and all log messages go > through a single root logger. That's a perfectly valid way of using logging: if the convenience functions suit your needs, there's no need to use anything else. > I'd argue that anyone using a multi-logger hierarchy has already implemented > overkill and that the default for anyone wanting to log something should be to > simply call the above functions directly from the logging module. Some people need the logger hierarchy and it's there for them, but no-one is forced to use it. > > This simplistic easy usage somewhat echo's Glenn's comment on this thread about logging seeming way to daunting as presented today. ?It needn't be. > Indeed, and the very first code sample in the logging documentation shows exactly the simplistic easy usage you're talking about. I can't see why anyone would be scared off by that example. Regards, Vinay Sajip From v+python at g.nevcal.com Fri Dec 10 09:31:23 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 10 Dec 2010 00:31:23 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: <4D01E55B.2050507@g.nevcal.com> On 12/10/2010 12:06 AM, Vinay Sajip wrote: >> > This simplistic easy usage somewhat echo's Glenn's comment on this thread > about logging seeming way to daunting as presented today. It needn't be. >> > > Indeed, and the very first code sample in the logging documentation shows > exactly the simplistic easy usage you're talking about. I can't see why anyone > would be scared off by that example. It didn't scare me off. But it wasn't _quite_ comprehensive enough to keep me from being put off by the next bit of manual, (about 8%, as mentioned before), which had lots of discussion, but no more examples, so what is lacking up front is a few more examples of using the convenience methods as Greg apparently does, without configuration, but showing use in multiple modules. After it is shown to be simple to get started in a multi-module code base, then all the refinements can be explained and added, but by then people are already using the logger, and learning becomes incremental. Otherwise, people are scared off or put off by the need to learn lots of new terms (logger, filter, handler, formatter, and configuration) before they get any value out of it. The next thing after the simple example is a rotating log configuration. A handy feature, of course, but hardly the second thing that one needs to learn about logging. Maybe the 4th or 5th or later. -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Fri Dec 10 09:36:57 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 10 Dec 2010 00:36:57 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <4CFFC5B0.3040006@g.nevcal.com> Message-ID: <4D01E6A9.80209@g.nevcal.com> On 12/9/2010 8:29 PM, Gregory P. Smith wrote: > Exactly. All I ever recommend people do is: > > import logging > ... > logging.warn('doing something a bit odd.') > ... > for x in thing: > logging.debug('working on %r', x) > ... > > > And be done with it. If they are controlling their __main__ they'll > probably want to call a common function to setup the log message > formatting and where it gets written to from there. Greg -- can you flesh out that last line, to remove the need for my parenthetical uncertainties below)? So with your warn and debug calls, and the "common function to setup ..." (whatever this is, and however often and from where it needs to be called in a multi-module environment) at the top of the logging manual, it might be more approachable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at xemacs.org Fri Dec 10 10:47:50 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 10 Dec 2010 18:47:50 +0900 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> Message-ID: <87tyimrmbd.fsf@uwakimon.sk.tsukuba.ac.jp> Vinay Sajip writes: > Indeed, and the very first code sample in the logging documentation > shows exactly the simplistic easy usage you're talking about. I > can't see why anyone would be scared off by that example. They're not scared by that example. What you need is a paragraph below it that says """ Do you think the above is all you should need? If so, you're right. You can stop reading now. If you think you need more, we've got that, too. Read on (you may need more coffee). """ No? From hrvoje.niksic at avl.com Fri Dec 10 11:37:19 2010 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Fri, 10 Dec 2010 11:37:19 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <87tyimrmbd.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <87tyimrmbd.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4D0202DF.3040707@avl.com> On 12/10/2010 10:47 AM, Stephen J. Turnbull wrote: > Vinay Sajip writes: > > > Indeed, and the very first code sample in the logging documentation > > shows exactly the simplistic easy usage you're talking about. I > > can't see why anyone would be scared off by that example. > > They're not scared by that example. What you need is a paragraph > below it that says > > """ > Do you think the above is all you should need? If so, you're > right. You can stop reading now. If you think you need more, > we've got that, too. Read on (you may need more coffee). > """ +1 From phd at phd.pp.ru Fri Dec 10 12:33:27 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Fri, 10 Dec 2010 14:33:27 +0300 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <87tyimrmbd.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <87tyimrmbd.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20101210113327.GA16785@phd.pp.ru> On Fri, Dec 10, 2010 at 06:47:50PM +0900, Stephen J. Turnbull wrote: > Vinay Sajip writes: > > Indeed, and the very first code sample in the logging documentation > > shows exactly the simplistic easy usage you're talking about. I > > can't see why anyone would be scared off by that example. > > They're not scared by that example. What you need is a paragraph > below it that says > > """ > Do you think the above is all you should need? If so, you're > right. You can stop reading now. If you think you need more, > we've got that, too. Read on (you may need more coffee). > """ Better yet (IMHO) would be to split the huge page into "Logging: Simple start" and "Logging: Advanced usage (for the brave of of heart)". Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From vinay_sajip at yahoo.co.uk Fri Dec 10 12:46:18 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 10 Dec 2010 11:46:18 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> Message-ID: Terry Reedy udel.edu> writes: > Your proposal struck me as probably the best way forward. Can you code > it up and put a patch on the tracker that people can test before the > next beta? Coded up (including unit test) and checked into py3k branch, r87157. Regressions pass OK. Old behaviour can be obtained by setting logging.lastResort = None. Regards, Vinay Sajip From solipsis at pitrou.net Fri Dec 10 12:58:50 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 10 Dec 2010 12:58:50 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> Message-ID: <20101210125850.76b6e65b@pitrou.net> Hey Vinay, On Fri, 10 Dec 2010 11:46:18 +0000 (UTC) Vinay Sajip wrote: > Terry Reedy udel.edu> writes: > > > Your proposal struck me as probably the best way forward. Can you code > > it up and put a patch on the tracker that people can test before the > > next beta? > > Coded up (including unit test) and checked into py3k branch, r87157. Regressions > pass OK. Old behaviour can be obtained by setting logging.lastResort = None. When you make significant changes outside of logging, it would be nice if you first opened an issue and posted the patch for review. There's nothing obviously wrong with what you committed, but I think it's principally better to ask first for advice. (then there's nothing wrong with committing if you didn't get an opinion past a week or two) Regards Antoine. From ncoghlan at gmail.com Fri Dec 10 13:22:10 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 10 Dec 2010 22:22:10 +1000 Subject: [Python-Dev] [Python-checkins] r87157 - in python/branches/py3k: Lib/logging/__init__.py Lib/test/test_logging.py Misc/NEWS In-Reply-To: <20101210114257.EB346ED47@mail.python.org> References: <20101210114257.EB346ED47@mail.python.org> Message-ID: On Fri, Dec 10, 2010 at 9:42 PM, vinay.sajip wrote: > Author: vinay.sajip > Date: Fri Dec 10 12:42:57 2010 > New Revision: 87157 > > Log: > logging: added handler of last resort. Nice, that ended up fitting in quite neatly. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Fri Dec 10 13:34:48 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 10 Dec 2010 12:34:48 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> <20101210125850.76b6e65b@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: Hi Antoine, > When you make significant changes outside of logging, it would be nice > if you first opened an issue and posted the patch for review. There's > nothing obviously wrong with what you committed, but I think it's > principally better to ask first for advice. Sorry, which significant changes outside of logging do you mean? If you're referring to my addition of TestHandler and Matcher to test.support, I did mention this earlier in the thread, with a link to the code (though it wasn't on a patch added to the tracker, I agree). I can revert this change, if you want. The changes in r87157 are to Lib/logging/__init__.py, Lib/test/test_logging.py and Misc/NEWS - so IMO not outside of logging. Perhaps wrongly, I expected that these straightforward-seeming changes wouldn't adversely affect anyone else's workflow, except perhaps making it easier for Brian Quinlan to make changes to test_concurrent_futures to resolve #10626. I did run regressions and everything passed OK. > (then there's nothing wrong with committing if you didn't get an > opinion past a week or two) Sorry if my eagerness has stepped on any toes - it's not intentional. The diff of the changes is here: https://gist.github.com/736120 Comments are of course welcome. Regards, Vinay Sajip From solipsis at pitrou.net Fri Dec 10 14:11:07 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 10 Dec 2010 14:11:07 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> <20101210125850.76b6e65b@pitrou.net> Message-ID: <20101210141107.2c21f9f4@pitrou.net> On Fri, 10 Dec 2010 12:34:48 +0000 (UTC) Vinay Sajip wrote: > > When you make significant changes outside of logging, it would be nice > > if you first opened an issue and posted the patch for review. There's > > nothing obviously wrong with what you committed, but I think it's > > principally better to ask first for advice. > > Sorry, which significant changes outside of logging do you mean? If you're > referring to my addition of TestHandler and Matcher to test.support, I did > mention this earlier in the thread, with a link to the code (though it wasn't on > a patch added to the tracker, I agree). I can revert this change, if you want. Yes, I am talking about this one. I am not asking that you revert it; I am just saying that it is generally appreciated if substantial patches get proposed on the tracker before being committed. In this case since it's in test.support we can probably improve it on the fly, of course. Regards Antoine. From tnagyemail-mail at yahoo.fr Fri Dec 10 14:36:17 2010 From: tnagyemail-mail at yahoo.fr (Thomas Nagy) Date: Fri, 10 Dec 2010 13:36:17 +0000 (GMT) Subject: [Python-Dev] futures API Message-ID: <774742.79683.qm@web27506.mail.ukl.yahoo.com> --- El jue, 9/12/10, Brian Quinlan escribi?: > On Dec 9, 2010, at 4:26 AM, Thomas Nagy wrote: > > > I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86) > by the futures module which was announced in python 3.2 > beta. I am a bit stuck with it, so I have a few questions > about the futures: > > > > 1. Is the futures API frozen? > > Yes. > > > 2. How hard would it be to return the tasks processed > in an output queue to process/consume the results while they > are returned? The code does not seem to be very open for > monkey patching. > > You can associate a callback with a submitted future. That > callback could add the future to your queue. Ok, it works. I was thinking the object was cleaned up immediately after it was used. > > 3. How hard would it be to add new tasks dynamically > (after a task is executed) and have the futures object never > complete? > > I'm not sure that I understand your question. You can > submit new work to an Executor at until time until it is > shutdown and a work item can take as long to complete as you > want. If you are contemplating tasks that don't complete > then maybe you could be better just scheduling a thread. > > > 4. Is there a performance evaluation of the futures > code (execution overhead) ? > > No. Scott Dial did make some performance improvements so he > might have a handle on its overhead. Ok. I have a process running for a long time, and which may use futures of different max_workers count. I think it is not too far-fetched to create a new futures object each time. Yet, the execution becomes slower after each call, for example with http://freehackers.org/~tnagy/futures_test.py: """ import concurrent.futures from queue import Queue import datetime class counter(object): ? ? def __init__(self, fut): ? ? ? ? self.fut = fut ? ? def run(self): ? ? ? ? def look_busy(num, obj): ? ? ? ? ? ? tot = 0 ? ? ? ? ? ? for x in range(num): ? ? ? ? ? ? ? ? tot += x ? ? ? ? ? ? obj.out_q.put(tot) ? ? ? ? start = datetime.datetime.utcnow() ? ? ? ? self.count = 0 ? ? ? ? self.out_q = Queue(0) ? ? ? ? for x in range(1000): ? ? ? ? ? ? self.count += 1 ? ? ? ? ? ? self.fut.submit(look_busy, self.count, self) ? ? ? ? while self.count: ? ? ? ? ? ? self.count -= 1 ? ? ? ? ? ? self.out_q.get() ? ? ? ? delta = datetime.datetime.utcnow() - start ? ? ? ? print(delta.total_seconds()) fut = concurrent.futures.ThreadPoolExecutor(max_workers=20) for x in range(100): ? ? # comment the following line ? ? fut = concurrent.futures.ThreadPoolExecutor(max_workers=20) ? ? c = counter(fut) ? ? c.run() """ The runtime grows after each step: 0.216451 0.225186 0.223725 0.222274 0.230964 0.240531 0.24137 0.252393 0.249948 0.257153 ... Is there a mistake in this piece of code? Thanks, Thomas From vinay_sajip at yahoo.co.uk Fri Dec 10 14:37:22 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 10 Dec 2010 13:37:22 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <20101208110309.56cf2a22@pitrou.net> <20101208152759.32b53cb2@pitrou.net> <20101208191537.44fc3ad8@pitrou.net> <20101210125850.76b6e65b@pitrou.net> <20101210141107.2c21f9f4@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > Yes, I am talking about this one. I am not asking that you revert it; I > am just saying that it is generally appreciated if substantial patches > get proposed on the tracker before being committed. > OK, sorry - point taken. > In this case since it's in test.support we can probably improve it on > the fly, of course. For sure. Regards, Vinay From merwok at netwok.org Fri Dec 10 15:20:17 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Fri, 10 Dec 2010 15:20:17 +0100 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: References: Message-ID: <4D023721.1070904@netwok.org> Hi, Original discussion for the sysconfig module was short: http://mail.python.org/pipermail/python-dev/2009-May/089520.html Tarek will reply better, but I think the issue to solve was to move sysconfig out of distutils, improving its API a bit in the process but not overhauling it completely. A single function returning a nested dictionary could be added if deemed useful, but to me the existing API separates concerns (paths and variables for example) without being confusing nor having too many names. Exceptions: get_python_version could be removed; get_scheme_names and get_path_names could be replaced by constants. Maybe Tarek chose accessors to keep freedom for later, or just for consistency. sysconfig being new in 2.7 and 3.2, I don?t know how compatibility concerns apply. I think it would be bad to have highlighted the new module in 2.7 and then remove it in 3.2, since 2.6 and 2.7 have the goal of helping people moving to 3.x. Also, I don?t see where those 800 lines of code would live if the sysconfig module is removed; maybe in platform, which is already long? (it could be made smaller with some code modernization, removal of the popen function, but not much smaller). Final note: with 3.2 being in beta, I don?t know how much can be changed now. > The old way required source code available ability at runtime, the new > sysconfig module captures everything it needs at build time. Not yet :) See http://bugs.python.org/issue9878 Regards From stephen at xemacs.org Fri Dec 10 17:01:10 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 11 Dec 2010 01:01:10 +0900 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <20101210113327.GA16785@phd.pp.ru> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <87tyimrmbd.fsf@uwakimon.sk.tsukuba.ac.jp> <20101210113327.GA16785@phd.pp.ru> Message-ID: <87oc8tsjll.fsf@uwakimon.sk.tsukuba.ac.jp> Oleg Broytman writes: > Better yet (IMHO) would be to split the huge page into "Logging: Simple > start" and "Logging: Advanced usage (for the brave of of heart)". Splitting is OK, but I disagree about the gloss "for the brave of heart". In my experience, if it is a YAGNI, the complexity is nearly impenetrable. If you *do* need it, it's not at all difficult to understand what the complexity is for, and it doesn't even look all that complex because it matches up with the problem you need to solve. If the documentation is still a deterrent, that's a problem with the documentation and it should be improved. AFAICT, making it clear that exporting all the internal flexibility is for those who need it, while most users will rarely need it, should be enough. But I'm not a good test case, since I already am familiar with XEmacs's similar system. From status at bugs.python.org Fri Dec 10 18:07:04 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 10 Dec 2010 18:07:04 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20101210170704.5A4541DBA6@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-12-03 - 2010-12-10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2557 (+20) closed 19899 (+38) total 22456 (+58) Open issues with patches: 1083 Issues opened (40) ================== #9101: reference json format in file formats chapter http://bugs.python.org/issue9101 reopened by eric.araujo #10553: Add optimize argument to builtin compile() and byte-compilatio http://bugs.python.org/issue10553 reopened by eric.araujo #10617: Collections ABCs can???t be linked to http://bugs.python.org/issue10617 opened by eric.araujo #10618: regression in subprocess.call() command quoting http://bugs.python.org/issue10618 opened by djc #10619: Failed module loading in test discovery loses traceback http://bugs.python.org/issue10619 opened by michael.foord #10620: `python -m uniittest` should work with file paths as well as t http://bugs.python.org/issue10620 opened by michael.foord #10622: WebKit browsers show superfluous scrollbars in html docs http://bugs.python.org/issue10622 opened by davide.rizzo #10626: Bad interaction between test_logging and test_concurrent_futur http://bugs.python.org/issue10626 opened by pitrou #10627: Remove usage of deprecated configparser.ConfigParser class in http://bugs.python.org/issue10627 opened by lukasz.langa #10631: ZipFile and current directory change http://bugs.python.org/issue10631 opened by ocean-city #10632: multiprocessing generates a fatal error http://bugs.python.org/issue10632 opened by bquinlan #10634: Windows timezone changes not reflected by time.localtime http://bugs.python.org/issue10634 opened by eric.pruitt #10635: Calling subprocess.Popen with preexec_fn=signal.pause blocks f http://bugs.python.org/issue10635 opened by joseph.h.garvin #10636: subprocess module has race condition with SIGCHLD handlers http://bugs.python.org/issue10636 opened by joseph.h.garvin #10639: reindent.py converts newlines to platform default http://bugs.python.org/issue10639 opened by jaraco #10640: SystemError on pickling bytes >= 4GB http://bugs.python.org/issue10640 opened by hagen #10641: kill_python sometimes fails to kill processes on buildbots http://bugs.python.org/issue10641 opened by db3l #10642: site.py crashes on python startup due to defective .pth file http://bugs.python.org/issue10642 opened by dwr2 #10644: socket loses data, calling send()/sendall() on invalid socket http://bugs.python.org/issue10644 opened by diekmann #10645: egg-info file in lib-dynload http://bugs.python.org/issue10645 opened by ronaldoussoren #10646: ntpath.samefile doesn't work for hard links http://bugs.python.org/issue10646 opened by brian.curtin #10647: scrollbar crash in non-US locale format settings http://bugs.python.org/issue10647 opened by hfischer #10648: Extend peepholer to reverse loads or stores instead of build/u http://bugs.python.org/issue10648 opened by serprex #10650: decimal.py: quantize(): excess digits with watchexp=0 http://bugs.python.org/issue10650 opened by skrah #10652: test___all_ + test_tcl fails (Windows installed binary) http://bugs.python.org/issue10652 opened by ocean-city #10653: test_time test_strptime fails on windows http://bugs.python.org/issue10653 opened by ocean-city #10654: test_datetime fails on Python3.2 windows binary http://bugs.python.org/issue10654 opened by ocean-city #10655: Wrong powerpc define in Python/ceval.c http://bugs.python.org/issue10655 opened by adrian #10656: "Out of tree" build fails on AIX 5.3 http://bugs.python.org/issue10656 opened by pedz #10657: os.lstat/os.stat/os.fstat don't set st_dev (st_rdev) on Window http://bugs.python.org/issue10657 opened by ocean-city #10663: configure shouldn't set a default OPT http://bugs.python.org/issue10663 opened by pitrou #10664: xml.sax.expatreader should support namespace prefixes http://bugs.python.org/issue10664 opened by fdrake #10665: Update and expand unicodedata module documentation http://bugs.python.org/issue10665 opened by belopolsky #10666: OS X installer variants have confusing readline differences http://bugs.python.org/issue10666 opened by ned.deily #10667: collections.Counter object in C http://bugs.python.org/issue10667 opened by jpeel #10669: Remove Deprecation Warnings http://bugs.python.org/issue10669 opened by RusiMody #10670: Provide search scope limits http://bugs.python.org/issue10670 opened by techtonik #10671: urllib2 redirect to another host doesn't work http://bugs.python.org/issue10671 opened by kirax #10673: multiprocess.Process join method - timeout indistinguishable f http://bugs.python.org/issue10673 opened by Brian.Cain #10674: Unused dictmaker symbol in 2.* grammar http://bugs.python.org/issue10674 opened by salty-horse Most recent 15 issues with no replies (15) ========================================== #10674: Unused dictmaker symbol in 2.* grammar http://bugs.python.org/issue10674 #10670: Provide search scope limits http://bugs.python.org/issue10670 #10666: OS X installer variants have confusing readline differences http://bugs.python.org/issue10666 #10664: xml.sax.expatreader should support namespace prefixes http://bugs.python.org/issue10664 #10663: configure shouldn't set a default OPT http://bugs.python.org/issue10663 #10657: os.lstat/os.stat/os.fstat don't set st_dev (st_rdev) on Window http://bugs.python.org/issue10657 #10656: "Out of tree" build fails on AIX 5.3 http://bugs.python.org/issue10656 #10646: ntpath.samefile doesn't work for hard links http://bugs.python.org/issue10646 #10632: multiprocessing generates a fatal error http://bugs.python.org/issue10632 #10619: Failed module loading in test discovery loses traceback http://bugs.python.org/issue10619 #10617: Collections ABCs can???t be linked to http://bugs.python.org/issue10617 #10612: StopTestRun exception to halt test run http://bugs.python.org/issue10612 #10608: Add a section to Windows FAQ explaining os.symlink http://bugs.python.org/issue10608 #10605: ElementTree documentation http://bugs.python.org/issue10605 #10595: Adding a syslog.conf reader in syslog http://bugs.python.org/issue10595 Most recent 15 issues waiting for review (15) ============================================= #10673: multiprocess.Process join method - timeout indistinguishable f http://bugs.python.org/issue10673 #10671: urllib2 redirect to another host doesn't work http://bugs.python.org/issue10671 #10667: collections.Counter object in C http://bugs.python.org/issue10667 #10665: Update and expand unicodedata module documentation http://bugs.python.org/issue10665 #10657: os.lstat/os.stat/os.fstat don't set st_dev (st_rdev) on Window http://bugs.python.org/issue10657 #10653: test_time test_strptime fails on windows http://bugs.python.org/issue10653 #10652: test___all_ + test_tcl fails (Windows installed binary) http://bugs.python.org/issue10652 #10648: Extend peepholer to reverse loads or stores instead of build/u http://bugs.python.org/issue10648 #10639: reindent.py converts newlines to platform default http://bugs.python.org/issue10639 #10627: Remove usage of deprecated configparser.ConfigParser class in http://bugs.python.org/issue10627 #10626: Bad interaction between test_logging and test_concurrent_futur http://bugs.python.org/issue10626 #10622: WebKit browsers show superfluous scrollbars in html docs http://bugs.python.org/issue10622 #10615: Trivial mingw compile fixes http://bugs.python.org/issue10615 #10611: sys.exit() in a test causes a test run to die http://bugs.python.org/issue10611 #10598: curses fails to import on Solaris http://bugs.python.org/issue10598 Top 10 most discussed issues (10) ================================= #10516: Add list.clear() and list.copy() http://bugs.python.org/issue10516 22 msgs #10615: Trivial mingw compile fixes http://bugs.python.org/issue10615 19 msgs #10626: Bad interaction between test_logging and test_concurrent_futur http://bugs.python.org/issue10626 15 msgs #6697: Check that _PyUnicode_AsString() result is not NULL http://bugs.python.org/issue6697 13 msgs #10367: "python setup.py sdist upload --show-response" can fail with " http://bugs.python.org/issue10367 10 msgs #8194: Incompatible API change in xmlrpclib.Transport.parse_response( http://bugs.python.org/issue8194 9 msgs #10453: Add -h/--help option to compileall http://bugs.python.org/issue10453 9 msgs #10653: test_time test_strptime fails on windows http://bugs.python.org/issue10653 9 msgs #10654: test_datetime fails on Python3.2 windows binary http://bugs.python.org/issue10654 9 msgs #9517: Make test.script_helper more comprehensive, and use it in the http://bugs.python.org/issue9517 7 msgs Issues closed (37) ================== #7245: better Ctrl-C support in pdb (program can be resumed) (issue21 http://bugs.python.org/issue7245 closed by georg.brandl #7905: Shelf 'keyencoding' keyword argument is undocumented and does http://bugs.python.org/issue7905 closed by georg.brandl #9227: can't import Tkinter / use IDLE after installing Python 2.7 on http://bugs.python.org/issue9227 closed by ronaldoussoren #9518: PyModuleDef_HEAD_INIT does not explicitly initialize all field http://bugs.python.org/issue9518 closed by dmalcolm #10107: Quitting IDLE on Mac doesn't save unsaved code http://bugs.python.org/issue10107 closed by ronaldoussoren #10272: SSL timeouts should raise socket.timeout, not a generic SSLErr http://bugs.python.org/issue10272 closed by pitrou #10292: tarinfo should use relative symlinks http://bugs.python.org/issue10292 closed by lars.gustaebel #10406: IDLE 2.7 on OS X does not enable Rstrip extension by default http://bugs.python.org/issue10406 closed by ronaldoussoren #10478: Ctrl-C locks up the interpreter http://bugs.python.org/issue10478 closed by pitrou #10545: remove or rewrite "Using Backslash to Continue Statements" ant http://bugs.python.org/issue10545 closed by terry.reedy #10546: UTF-16-LE and UTF-16-BE support non-BMP characters http://bugs.python.org/issue10546 closed by haypo #10596: modulo operator bug http://bugs.python.org/issue10596 closed by mark.dickinson #10601: sys.displayhook: use backslashreplace error handler if repr(va http://bugs.python.org/issue10601 closed by haypo #10621: >>> 1 + 2j --> (1 + 2j) and not (1+2j) http://bugs.python.org/issue10621 closed by mark.dickinson #10623: What???s New In Python 3.2 document refers to PEP 382: Definin http://bugs.python.org/issue10623 closed by loewis #10624: Move requires_IEEE_754 decorator from test_complex into test.s http://bugs.python.org/issue10624 closed by eric.smith #10625: There is no test for repr(complex(-0.,1.)) special handling http://bugs.python.org/issue10625 closed by eric.smith #10628: Typos in 3.2 what???s new http://bugs.python.org/issue10628 closed by georg.brandl #10629: Arbitrary precision http://bugs.python.org/issue10629 closed by georg.brandl #10630: dict_proxy.keys() / values() / items() are lists http://bugs.python.org/issue10630 closed by benjamin.peterson #10633: string.format() Unexpected output with numeric '#' prefix and http://bugs.python.org/issue10633 closed by r.david.murray #10637: Calling CloseHandle twice (Modules/posixmodule.c) http://bugs.python.org/issue10637 closed by ocean-city #10638: PyArg_ParseTuple: refcount in nested tuples http://bugs.python.org/issue10638 closed by arigo #10643: OSX proxy exceptions parse bug in urllib (crash) http://bugs.python.org/issue10643 closed by ned.deily #10649: Attempting to override special methods of a function object do http://bugs.python.org/issue10649 closed by r.david.murray #10651: chr(13) is added before chr(10) when issuing ".write" under wi http://bugs.python.org/issue10651 closed by amaury.forgeotdarc #10658: Link to source code is broken http://bugs.python.org/issue10658 closed by georg.brandl #10659: Hook scripts for immediate doc build system http://bugs.python.org/issue10659 closed by georg.brandl #10660: format() to lower and uppercase http://bugs.python.org/issue10660 closed by r.david.murray #10661: ElementTree QName has a very uninformative repr() http://bugs.python.org/issue10661 closed by georg.brandl #10662: Typo in concurrent.futures, "seperate" http://bugs.python.org/issue10662 closed by georg.brandl #10668: Array tests have nonsense in them http://bugs.python.org/issue10668 closed by georg.brandl #10672: ["with"] new in version 2.6 instead of 2.5 http://bugs.python.org/issue10672 closed by georg.brandl #1772833: -q (quiet) option for python interpreter http://bugs.python.org/issue1772833 closed by georg.brandl #1513299: Clean up usage of map() in the stdlib http://bugs.python.org/issue1513299 closed by georg.brandl #1569291: Speed-up in array_repeat() http://bugs.python.org/issue1569291 closed by georg.brandl #1571170: Some numeric characters are still not recognized http://bugs.python.org/issue1571170 closed by andersch From ncoghlan at gmail.com Fri Dec 10 18:31:08 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 03:31:08 +1000 Subject: [Python-Dev] futures API In-Reply-To: <774742.79683.qm@web27506.mail.ukl.yahoo.com> References: <774742.79683.qm@web27506.mail.ukl.yahoo.com> Message-ID: On Fri, Dec 10, 2010 at 11:36 PM, Thomas Nagy wrote: > fut = concurrent.futures.ThreadPoolExecutor(max_workers=20) > for x in range(100): > ? ? # comment the following line > ? ? fut = concurrent.futures.ThreadPoolExecutor(max_workers=20) > ? ? c = counter(fut) > ? ? c.run() > """ > The runtime grows after each step: > Is there a mistake in this piece of code? This isn't an "overhead" question, it's a "how prompt is the resource release" question. Given that you've created circular references between the futures and your counter class, the answer is probably "not prompt at all". Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From brian at sweetapp.com Fri Dec 10 18:41:52 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 10 Dec 2010 09:41:52 -0800 Subject: [Python-Dev] futures API In-Reply-To: <774742.79683.qm@web27506.mail.ukl.yahoo.com> References: <774742.79683.qm@web27506.mail.ukl.yahoo.com> Message-ID: <961F4C62-DD20-4C11-88B0-A45427B27616@sweetapp.com> Oops. I accidentally replied off-list: On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: > --- El jue, 9/12/10, Brian Quinlan escribi?: >> On Dec 9, 2010, at 4:26 AM, Thomas Nagy wrote: >> >>> I am looking forward to replacing a piece of code (http://code.google.com/p/waf/source/browse/trunk/waflib/Runner.py#86 >>> ) >> by the futures module which was announced in python 3.2 >> beta. I am a bit stuck with it, so I have a few questions >> about the futures: >>> >>> 1. Is the futures API frozen? >> >> Yes. >> >>> 2. How hard would it be to return the tasks processed >> in an output queue to process/consume the results while they >> are returned? The code does not seem to be very open for >> monkey patching. >> >> You can associate a callback with a submitted future. That >> callback could add the future to your queue. > > Ok, it works. I was thinking the object was cleaned up immediately > after it was used. > >>> 3. How hard would it be to add new tasks dynamically >> (after a task is executed) and have the futures object never >> complete? >> >> I'm not sure that I understand your question. You can >> submit new work to an Executor at until time until it is >> shutdown and a work item can take as long to complete as you >> want. If you are contemplating tasks that don't complete >> then maybe you could be better just scheduling a thread. >> >>> 4. Is there a performance evaluation of the futures >> code (execution overhead) ? >> >> No. Scott Dial did make some performance improvements so he >> might have a handle on its overhead. > > Ok. > > I have a process running for a long time, and which may use futures > of different max_workers count. I think it is not too far-fetched to > create a new futures object each time. Yet, the execution becomes > slower after each call, for example with http://freehackers.org/~tnagy/futures_test.py > : > > """ > import concurrent.futures > from queue import Queue > import datetime > > class counter(object): > def __init__(self, fut): > self.fut = fut > > def run(self): > def look_busy(num, obj): > tot = 0 > for x in range(num): > tot += x > obj.out_q.put(tot) > > start = datetime.datetime.utcnow() > self.count = 0 > self.out_q = Queue(0) > for x in range(1000): > self.count += 1 > self.fut.submit(look_busy, self.count, self) > > while self.count: > self.count -= 1 > self.out_q.get() > > delta = datetime.datetime.utcnow() - start > print(delta.total_seconds()) > > fut = concurrent.futures.ThreadPoolExecutor(max_workers=20) > for x in range(100): > # comment the following line > fut = concurrent.futures.ThreadPoolExecutor(max_workers=20) > c = counter(fut) > c.run() > """ > > The runtime grows after each step: > 0.216451 > 0.225186 > 0.223725 > 0.222274 > 0.230964 > 0.240531 > 0.24137 > 0.252393 > 0.249948 > 0.257153 > ... > > Is there a mistake in this piece of code? There is no mistake that I can see but I suspect that the circular references that you are building are causing the ThreadPoolExecutor to take a long time to be collected. Try adding: c = counter(fut) c.run() + fut.shutdown() Even if that fixes your problem, I still don't fully understand these numbers because I would expect the runtime to fall after a while as ThreadPoolExecutors are collected. Cheers, Brian > Thanks, > Thomas > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Dec 10 18:54:56 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 10 Dec 2010 09:54:56 -0800 Subject: [Python-Dev] transform() and untransform() methods, and the codec registry In-Reply-To: <20101209201749.79edd5ce@pitrou.net> References: <20101202180651.F2AB0EE9CD@mail.python.org> <201012031016.05063.victor.stinner@haypocalc.com> <20101203151129.74C5222A88F@kimball.webabinitio.net> <20101203175832.88F00236D8B@kimball.webabinitio.net> <4CFCCB7F.9080605@egenix.com> <20101209201749.79edd5ce@pitrou.net> Message-ID: On Thu, Dec 9, 2010 at 11:17 AM, Antoine Pitrou wrote: > On Thu, 9 Dec 2010 13:55:08 -0500 > Alexander Belopolsky wrote: > >> On Thu, Dec 9, 2010 at 1:42 PM, Guido van Rossum wrote: >> .. >> > string-string transforms use the same namespace even though the >> > typical transform only supports one or the other. E.g. IMO all of the >> > following should raise LookupError: >> > >> >>>> b'abc'.transform('rot13') >> > Traceback (most recent call last): >> > .. >> > ? ?return (input.translate(rot13_map), len(input)) >> > TypeError: expected an object with the buffer interface >> >> This is actually *very* misleading because >> >> >>> 'abc'.transform('rot13') >> 'nop' >> >> works even though 'abc' is not "an object with the buffer interface". > > Agreed. It was already pointed out in the parent thread. > I would say my opinion on keeping transform()/untransform() is +0 if > these error messages (and preferably the exception type as well) get > improved. Otherwise we'd better revert them and add a more polished > version in 3.3. Let me change my opinion then and say that I'm -1 on keeping the confusing situation. It can be fixed either by postponing transform() until 3.3 or by fixing the error checking and namespace issue. Sorry about the reversal of pronouncements, but my original approval was very weak indeed and mostly driven by "stick to status quo unless there's a serious problem". The followup made it clear that the errors and namespace issue are more seriously than I estimated at first. -- --Guido van Rossum (python.org/~guido) From janssen at parc.com Fri Dec 10 19:16:18 2010 From: janssen at parc.com (Bill Janssen) Date: Fri, 10 Dec 2010 10:16:18 PST Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <4D01E55B.2050507@g.nevcal.com> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> Message-ID: <3231.1292004978@parc.com> Glenn Linderman wrote: > On 12/10/2010 12:06 AM, Vinay Sajip wrote: > >> > This simplistic easy usage somewhat echo's Glenn's comment on this thread > > about logging seeming way to daunting as presented today. It needn't be. > >> > > > Indeed, and the very first code sample in the logging documentation shows > > exactly the simplistic easy usage you're talking about. I can't see why anyone > > would be scared off by that example. > > It didn't scare me off. But it wasn't _quite_ comprehensive enough to > keep me from being put off by the next bit of manual, (about 8%, as > mentioned before), which had lots of discussion, but no more examples, > so what is lacking up front is a few more examples of using the > convenience methods as Greg apparently does, without configuration, > but showing use in multiple modules. After it is shown to be simple > to get started in a multi-module code base, then all the refinements > can be explained and added, but by then people are already using the > logger, and learning becomes incremental. Just a note: after spacing out over this thread for a bit, I thought I'd try out logging, which I'd never tried before, in some code I was writing. I added a couple of lines like logging.info("foo is %s", foo) where I'd usually write sys.stderr.write("foo is %s\n" % foo) Liked it just fine -- easier to write. But nothing came out! Because it's got this odd idea (from Java?) about "severities" of messages, and by default passes nothing less "threatening" than warnings, because logger.root is initialized by default to WARNING. So, still a bit disconcerting for newbies. Bill From tnagyemail-mail at yahoo.fr Fri Dec 10 19:51:11 2010 From: tnagyemail-mail at yahoo.fr (Thomas Nagy) Date: Fri, 10 Dec 2010 18:51:11 +0000 (GMT) Subject: [Python-Dev] futures API Message-ID: <441626.55995.qm@web27505.mail.ukl.yahoo.com> --- El vie, 10/12/10, Brian Quinlan escribi?: > On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: > > I have a process running for a long time, and which > may use futures of different max_workers count. I think it > is not too far-fetched to create a new futures object each > time. Yet, the execution becomes slower after each call, for > example with http://freehackers.org/~tnagy/futures_test.py: > > > > """ > > import concurrent.futures > > from queue import Queue > > import datetime > > > > class counter(object): > >? ???def __init__(self, fut): > >? ? ? ???self.fut = fut > > > >? ???def run(self): > >? ? ? ???def > look_busy(num, obj): > >? ? ? ? ? > ???tot = 0 > >? ? ? ? ? > ???for x in range(num): > >? ? ? ? ? ? ? > ???tot += x > >? ? ? ? ? > ???obj.out_q.put(tot) > > > >? ? ? ???start = > datetime.datetime.utcnow() > >? ? ? ???self.count = 0 > >? ? ? ???self.out_q = > Queue(0) > >? ? ? ???for x in > range(1000): > >? ? ? ? ? > ???self.count += 1 > >? ? ? ? ? > ???self.fut.submit(look_busy, self.count, > self) > > > >? ? ? ???while > self.count: > >? ? ? ? ? > ???self.count -= 1 > >? ? ? ? ? > ???self.out_q.get() > > > >? ? ? ???delta = > datetime.datetime.utcnow() - start > >? ? ? > ???print(delta.total_seconds()) > > > > fut = > concurrent.futures.ThreadPoolExecutor(max_workers=20) > > for x in range(100): > >? ???# comment the following line > >? ???fut = > concurrent.futures.ThreadPoolExecutor(max_workers=20) > >? ???c = counter(fut) > >? ???c.run() > > """ > > > > The runtime grows after each step: > > 0.216451 > > 0.225186 > > 0.223725 > > 0.222274 > > 0.230964 > > 0.240531 > > 0.24137 > > 0.252393 > > 0.249948 > > 0.257153 > > ... > > > > Is there a mistake in this piece of code? > > There is no mistake that I can see but I suspect that the > circular references that you are building are causing the > ThreadPoolExecutor to take a long time to be collected. Try > adding: > > ??? c = counter(fut) > ??? c.run() > +??? fut.shutdown() > > Even if that fixes your problem, I still don't fully > understand this because I would expect the runtime to fall > after a while as ThreadPoolExecutors are collected. The shutdown call is indeed a good fix :-) Here is the time response of the calls to counter() when shutdown is not called: http://www.freehackers.org/~tnagy/runtime_futures.png After trying to stop the program by using CTRL+C, the following error may appear, after which the process cannot be interrupted: """ 19:18:12 /tmp/build> python3.2 futures_test.py 0.389657 0.417173 0.416513 0.421424 0.449666 0.482273 ^CTraceback (most recent call last): ? File "futures_test.py", line 36, in ? ? c.run() ? File "futures_test.py", line 22, in run ? ? self.fut.submit(look_busy, self.count, self) ? File "/usr/local/lib/python3.2/concurrent/futures/thread.py", line 114, in submit ? ? self._work_queue.put(w) ? File "/usr/local/lib/python3.2/queue.py", line 135, in put ? ? self.not_full.acquire() KeyboardInterrupt """ It is not expected, is it? Thomas From brian at sweetapp.com Fri Dec 10 20:24:42 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 10 Dec 2010 11:24:42 -0800 Subject: [Python-Dev] futures API In-Reply-To: <441626.55995.qm@web27505.mail.ukl.yahoo.com> References: <441626.55995.qm@web27505.mail.ukl.yahoo.com> Message-ID: On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote: > --- El vie, 10/12/10, Brian Quinlan escribi?: >> On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: >>> I have a process running for a long time, and which >> may use futures of different max_workers count. I think it >> is not too far-fetched to create a new futures object each >> time. Yet, the execution becomes slower after each call, for >> example with http://freehackers.org/~tnagy/futures_test.py: >>> >>> """ >>> import concurrent.futures >>> from queue import Queue >>> import datetime >>> >>> class counter(object): >>> def __init__(self, fut): >>> self.fut = fut >>> >>> def run(self): >>> def >> look_busy(num, obj): >>> >> tot = 0 >>> >> for x in range(num): >>> >> tot += x >>> >> obj.out_q.put(tot) >>> >>> start = >> datetime.datetime.utcnow() >>> self.count = 0 >>> self.out_q = >> Queue(0) >>> for x in >> range(1000): >>> >> self.count += 1 >>> >> self.fut.submit(look_busy, self.count, >> self) >>> >>> while >> self.count: >>> >> self.count -= 1 >>> >> self.out_q.get() >>> >>> delta = >> datetime.datetime.utcnow() - start >>> >> print(delta.total_seconds()) >>> >>> fut = >> concurrent.futures.ThreadPoolExecutor(max_workers=20) >>> for x in range(100): >>> # comment the following line >>> fut = >> concurrent.futures.ThreadPoolExecutor(max_workers=20) >>> c = counter(fut) >>> c.run() >>> """ >>> >>> The runtime grows after each step: >>> 0.216451 >>> 0.225186 >>> 0.223725 >>> 0.222274 >>> 0.230964 >>> 0.240531 >>> 0.24137 >>> 0.252393 >>> 0.249948 >>> 0.257153 >>> ... >>> >>> Is there a mistake in this piece of code? >> >> There is no mistake that I can see but I suspect that the >> circular references that you are building are causing the >> ThreadPoolExecutor to take a long time to be collected. Try >> adding: >> >> c = counter(fut) >> c.run() >> + fut.shutdown() >> >> Even if that fixes your problem, I still don't fully >> understand this because I would expect the runtime to fall >> after a while as ThreadPoolExecutors are collected. > > The shutdown call is indeed a good fix :-) Here is the time response > of the calls to counter() when shutdown is not called: > http://www.freehackers.org/~tnagy/runtime_futures.png FWIW, I think that you are confusion the term "future" with "executor". A future represents a single work item. An executor creates futures and schedules their underlying work. Hmmm....that is very suspicious - it looks like the ThreadPoolExecutors are not being collected. If you are feeling bored you could figure out why not :-) > After trying to stop the program by using CTRL+C, the following > error may appear, after which the process cannot be interrupted: > > """ > 19:18:12 /tmp/build> python3.2 futures_test.py > 0.389657 > 0.417173 > 0.416513 > 0.421424 > 0.449666 > 0.482273 > ^CTraceback (most recent call last): > File "futures_test.py", line 36, in > c.run() > File "futures_test.py", line 22, in run > self.fut.submit(look_busy, self.count, self) > File "/usr/local/lib/python3.2/concurrent/futures/thread.py", line > 114, in submit > self._work_queue.put(w) > File "/usr/local/lib/python3.2/queue.py", line 135, in put > self.not_full.acquire() > KeyboardInterrupt > """ > > It is not expected, is it? It isn't surprising. Python lock acquisitions are not interruptible and anytime you interrupt a program that manipulates locks you may kill the code that was going to cause the lock to be released. Cheers, Brian > Thomas > > > > From vinay_sajip at yahoo.co.uk Fri Dec 10 20:39:08 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 10 Dec 2010 19:39:08 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> Message-ID: Bill Janssen parc.com> writes: > Liked it just fine -- easier to write. But nothing came out! Because > it's got this odd idea (from Java?) about "severities" of messages, and > by default passes nothing less "threatening" than warnings, because > logger.root is initialized by default to WARNING. The idea about severity of messages is based on the premise that there might be different audiences for logged messages, and that these can change over time. A fairly common developer perspective on logging is that a logger equates to a destination (e.g. console or file) and the only audience for logging is the developer themselves. So, "handlers" and "loggers" seem complicated, perhaps over-complicated. In more complex systems, there are different stakeholders: end users, sys admins, support teams, development teams. You have to consider "what happened" (logging event), "where it happened" (logger name), "how important it is" (severity or level) and "who wants to know" (handler), because not everyone wants to know about everything all the time. That's pretty independent of which language you're developing in; it's more a function of the complexity of the system you're developing. Since Python is used in some fairly large system scenarios - NASA, Google, etc. - it doesn't make sense not to have the whole enchilada of logging functionality, at the risk of making it seem complicated to users with less complex needs. And importance maps reasonably naturally onto an idea of integer levels. The default setting of WARNING is not from Java either - it's the Unix idea that verbosity should be kept to a minimum unless specifically requested. So, warnings and errors should be shown, but info and debug not. Imagine if the default level was DEBUG - lots of detailed stuff would be displayed, which would be off-putting for most users. Regards, Vinay Sajip From tnagyemail-mail at yahoo.fr Fri Dec 10 20:39:18 2010 From: tnagyemail-mail at yahoo.fr (Thomas Nagy) Date: Fri, 10 Dec 2010 19:39:18 +0000 (GMT) Subject: [Python-Dev] futures API In-Reply-To: <441626.55995.qm@web27505.mail.ukl.yahoo.com> Message-ID: <953842.41772.qm@web27502.mail.ukl.yahoo.com> --- El vie, 10/12/10, Thomas Nagy escribi?: > --- El vie, 10/12/10, Brian Quinlan > escribi?: > > On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: > > > I have a process running for a long time, and > which > > may use futures of different max_workers count. I > think it > > is not too far-fetched to create a new futures object > each > > time. Yet, the execution becomes slower after each > call, for > > example with http://freehackers.org/~tnagy/futures_test.py: > > > > > > """ > > > import concurrent.futures > > > from queue import Queue > > > import datetime > > > > > > class counter(object): > > >? ???def __init__(self, fut): > > >? ? ? ???self.fut = fut > > > > > >? ???def run(self): > > >? ? ? ???def > > look_busy(num, obj): > > >? ? ? ? ? > > ???tot = 0 > > >? ? ? ? ? > > ???for x in range(num): > > >? ? ? ? ? ? ? > > ???tot += x > > >? ? ? ? ? > > ???obj.out_q.put(tot) > > > > > >? ? ? ???start = > > datetime.datetime.utcnow() > > >? ? ? ???self.count = 0 > > >? ? ? ???self.out_q = > > Queue(0) > > >? ? ? ???for x in > > range(1000): > > >? ? ? ? ? > > ???self.count += 1 > > >? ? ? ? ? > > ???self.fut.submit(look_busy, self.count, > > self) > > > > > >? ? ? ???while > > self.count: > > >? ? ? ? ? > > ???self.count -= 1 > > >? ? ? ? ? > > ???self.out_q.get() > > > > > >? ? ? ???delta = > > datetime.datetime.utcnow() - start > > >? ? ? > > ???print(delta.total_seconds()) > > > > > > fut = > > concurrent.futures.ThreadPoolExecutor(max_workers=20) > > > for x in range(100): > > >? ???# comment the following line > > >? ???fut = > > concurrent.futures.ThreadPoolExecutor(max_workers=20) > > >? ???c = counter(fut) > > >? ???c.run() > > > """ > > > > > > The runtime grows after each step: > > > 0.216451 > > > 0.225186 > > > 0.223725 > > > 0.222274 > > > 0.230964 > > > 0.240531 > > > 0.24137 > > > 0.252393 > > > 0.249948 > > > 0.257153 > > > ... > > > > > > Is there a mistake in this piece of code? > > > > There is no mistake that I can see but I suspect that > the > > circular references that you are building are causing > the > > ThreadPoolExecutor to take a long time to be > collected. Try > > adding: > > > > ??? c = counter(fut) > > ??? c.run() > > +??? fut.shutdown() > > > > Even if that fixes your problem, I still don't fully > > understand this because I would expect the runtime to > fall > > after a while as ThreadPoolExecutors are collected. > > The shutdown call is indeed a good fix :-) Here is the time > response of the calls to counter() when shutdown is not > called: > http://www.freehackers.org/~tnagy/runtime_futures.png > > After trying to stop the program by using CTRL+C, the > following error may appear, after which the process cannot > be interrupted: > > """ > 19:18:12 /tmp/build> python3.2 futures_test.py > 0.389657 > 0.417173 > 0.416513 > 0.421424 > 0.449666 > 0.482273 > ^CTraceback (most recent call last): > ? File "futures_test.py", line 36, in > ? ? c.run() > ? File "futures_test.py", line 22, in run > ? ? self.fut.submit(look_busy, self.count, self) > ? File > "/usr/local/lib/python3.2/concurrent/futures/thread.py", > line 114, in submit > ? ? self._work_queue.put(w) > ? File "/usr/local/lib/python3.2/queue.py", line 135, in > put > ? ? self.not_full.acquire() > KeyboardInterrupt > """ > > It is not expected, is it? The problem also occurs when using a callback: http://www.freehackers.org/~tnagy/futures_test2.py If it is necessary to catch KeyboardInterrupt exceptions to cancel the futures execution, then how about adding this detail to the docs? Thomas From brian at sweetapp.com Fri Dec 10 21:07:48 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Fri, 10 Dec 2010 12:07:48 -0800 Subject: [Python-Dev] futures API In-Reply-To: <953842.41772.qm@web27502.mail.ukl.yahoo.com> References: <953842.41772.qm@web27502.mail.ukl.yahoo.com> Message-ID: <8674D52A-07EF-4D4A-898A-983DE0AC6BC3@sweetapp.com> On Dec 10, 2010, at 11:39 AM, Thomas Nagy wrote: > --- El vie, 10/12/10, Thomas Nagy escribi?: >> --- El vie, 10/12/10, Brian Quinlan >> escribi?: >>> On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: >>>> I have a process running for a long time, and >> which >>> may use futures of different max_workers count. I >> think it >>> is not too far-fetched to create a new futures object >> each >>> time. Yet, the execution becomes slower after each >> call, for >>> example with http://freehackers.org/~tnagy/futures_test.py: >>>> >>>> """ >>>> import concurrent.futures >>>> from queue import Queue >>>> import datetime >>>> >>>> class counter(object): >>>> def __init__(self, fut): >>>> self.fut = fut >>>> >>>> def run(self): >>>> def >>> look_busy(num, obj): >>>> >>> tot = 0 >>>> >>> for x in range(num): >>>> >>> tot += x >>>> >>> obj.out_q.put(tot) >>>> >>>> start = >>> datetime.datetime.utcnow() >>>> self.count = 0 >>>> self.out_q = >>> Queue(0) >>>> for x in >>> range(1000): >>>> >>> self.count += 1 >>>> >>> self.fut.submit(look_busy, self.count, >>> self) >>>> >>>> while >>> self.count: >>>> >>> self.count -= 1 >>>> >>> self.out_q.get() >>>> >>>> delta = >>> datetime.datetime.utcnow() - start >>>> >>> print(delta.total_seconds()) >>>> >>>> fut = >>> concurrent.futures.ThreadPoolExecutor(max_workers=20) >>>> for x in range(100): >>>> # comment the following line >>>> fut = >>> concurrent.futures.ThreadPoolExecutor(max_workers=20) >>>> c = counter(fut) >>>> c.run() >>>> """ >>>> >>>> The runtime grows after each step: >>>> 0.216451 >>>> 0.225186 >>>> 0.223725 >>>> 0.222274 >>>> 0.230964 >>>> 0.240531 >>>> 0.24137 >>>> 0.252393 >>>> 0.249948 >>>> 0.257153 >>>> ... >>>> >>>> Is there a mistake in this piece of code? >>> >>> There is no mistake that I can see but I suspect that >> the >>> circular references that you are building are causing >> the >>> ThreadPoolExecutor to take a long time to be >> collected. Try >>> adding: >>> >>> c = counter(fut) >>> c.run() >>> + fut.shutdown() >>> >>> Even if that fixes your problem, I still don't fully >>> understand this because I would expect the runtime to >> fall >>> after a while as ThreadPoolExecutors are collected. >> >> The shutdown call is indeed a good fix :-) Here is the time >> response of the calls to counter() when shutdown is not >> called: >> http://www.freehackers.org/~tnagy/runtime_futures.png >> >> After trying to stop the program by using CTRL+C, the >> following error may appear, after which the process cannot >> be interrupted: >> >> """ >> 19:18:12 /tmp/build> python3.2 futures_test.py >> 0.389657 >> 0.417173 >> 0.416513 >> 0.421424 >> 0.449666 >> 0.482273 >> ^CTraceback (most recent call last): >> File "futures_test.py", line 36, in >> c.run() >> File "futures_test.py", line 22, in run >> self.fut.submit(look_busy, self.count, self) >> File >> "/usr/local/lib/python3.2/concurrent/futures/thread.py", >> line 114, in submit >> self._work_queue.put(w) >> File "/usr/local/lib/python3.2/queue.py", line 135, in >> put >> self.not_full.acquire() >> KeyboardInterrupt >> """ >> >> It is not expected, is it? > > The problem also occurs when using a callback: > http://www.freehackers.org/~tnagy/futures_test2.py > > If it is necessary to catch KeyboardInterrupt exceptions to cancel > the futures execution, then how about adding this detail to the docs? AFAIK, catching KeyboardInterrupt exceptions is not sufficient. Cheers, Brian > Thomas > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From raymond.hettinger at gmail.com Fri Dec 10 21:27:26 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 10 Dec 2010 12:27:26 -0800 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <4D023721.1070904@netwok.org> References: <4D023721.1070904@netwok.org> Message-ID: On Dec 10, 2010, at 6:20 AM, ?ric Araujo wrote: > Final note: with 3.2 being in beta, I don?t know how much can be changed > now. Part of the purpose of a beta, and in our case, two betas is to give people a chance to exercise new APIs and fix them before they become set in stone two months later. IMO, sysconfig did not warrant a whole module. The pile of awkward accessor functions is harder to learn / remember than the simple data structure shown in the last email. Rather than using two levels of dictionary, it's also possible to use a named tuple if you think that is more clean looking: >>> c = sys.sysconfig() >>> c.config_vars.get('SO) '.pyd' >>> c.platform 'win32' >>> c.paths.get('stdlib') 'C:\\Python32\\Lib' >>> # the named tuple fields: >>> c.fields ['config_vars', 'platform', 'version', 'scheme_names', 'paths'] This would be a straight-forward API that uses existing, well-known tools (attribute access and dict.get) instead of an unnecessary nest of ad-hoc accessor functions living in a new module. Tastes may vary on writing getter functions for every category of interest but we did not need a whole new module for this. The referenced email didn't indicate much if any thought about the API design, so I think should get that thought now. If those functions were already public in distutils, it is trivial to write an adapter for them there. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Dec 10 21:31:19 2010 From: guido at python.org (Guido van Rossum) Date: Fri, 10 Dec 2010 12:31:19 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> Message-ID: On Fri, Dec 10, 2010 at 11:39 AM, Vinay Sajip wrote: > Bill Janssen parc.com> writes: > >> Liked it just fine -- easier to write. ?But nothing came out! ?Because >> it's got this odd idea (from Java?) about "severities" of messages, and >> by default passes nothing less "threatening" than warnings, because >> logger.root is initialized by default to WARNING. > > The idea about severity of messages is based on the premise that there might be > different audiences for logged messages, and that these can change over time. A > fairly common developer perspective on logging is that a logger equates to a > destination (e.g. console or file) and the only audience for logging is the > developer themselves. So, "handlers" and "loggers" seem complicated, perhaps > over-complicated. > > In more complex systems, there are different stakeholders: end users, sys > admins, support teams, development teams. You have to consider "what happened" > (logging event), "where it happened" (logger name), "how important it is" > (severity or level) and "who wants to know" (handler), because not everyone > wants to know about everything all the time. That's pretty independent of which > language you're developing in; it's more a function of the complexity of the > system you're developing. Since Python is used in some fairly large system > scenarios - NASA, Google, etc. - it doesn't make sense not to have the whole > enchilada of logging functionality, at the risk of making it seem complicated to > users with less complex needs. And importance maps reasonably naturally onto an > idea of integer levels. > > The default setting of WARNING is not from Java either - it's the Unix idea that > verbosity should be kept to a minimum unless specifically requested. So, > warnings and errors should be shown, but info and debug not. Imagine if the > default level was DEBUG - lots of detailed stuff would be displayed, which would > be off-putting for most users. And yet, I have helped many people who were baffled by exactly what Bill observed: logging.info() didn't do anything. Maybe the default should be INFO? -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Fri Dec 10 21:49:19 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 10 Dec 2010 21:49:19 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> Message-ID: <20101210214919.4ac63d27@pitrou.net> On Fri, 10 Dec 2010 12:31:19 -0800 Guido van Rossum wrote: > > The default setting of WARNING is not from Java either - it's the Unix idea that > > verbosity should be kept to a minimum unless specifically requested. So, > > warnings and errors should be shown, but info and debug not. Imagine if the > > default level was DEBUG - lots of detailed stuff would be displayed, which would > > be off-putting for most users. > > And yet, I have helped many people who were baffled by exactly what > Bill observed: logging.info() didn't do anything. Maybe the default > should be INFO? Funny, because displaying only errors and silencing other messages is exactly what I expected logging to do by default. Regards Antoine. From solipsis at pitrou.net Fri Dec 10 21:56:00 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 10 Dec 2010 21:56:00 +0100 Subject: [Python-Dev] API for the new sysconfig module References: <4D023721.1070904@netwok.org> Message-ID: <20101210215600.56f4eaf2@pitrou.net> On Fri, 10 Dec 2010 12:27:26 -0800 Raymond Hettinger wrote: > > IMO, sysconfig did not warrant a whole module. Where would you put it? > Rather than using two levels of dictionary, it's also possible > to use a named tuple if you think that is more clean looking: > > >>> c = sys.sysconfig() > >>> c.config_vars.get('SO) > '.pyd' Some of these things are computed at runtime by parsing makefiles and other stuff. You don't want to do it as soon as the module is imported. There may also be some bootstrap issues when importing some of the necessary modules from the top level. The current API is fine IMO. Perhaps it could be refined a bit, but there's nothing horrendous. It's not for everyone to use; creating dict-like classes over these APIs would be a terrible waste of time, and add a useless maintenance burden. > If those functions were already public in distutils, it is trivial > to write an adapter for them there. The whole point was to separate them from distutils. Regards Antoine. From v+python at g.nevcal.com Fri Dec 10 21:57:29 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 10 Dec 2010 12:57:29 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <20101210214919.4ac63d27@pitrou.net> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: <4D029439.7040708@g.nevcal.com> On 12/10/2010 12:49 PM, Antoine Pitrou wrote: >> And yet, I have helped many people who were baffled by exactly what >> > Bill observed: logging.info() didn't do anything. Maybe the default >> > should be INFO? > Funny, because displaying only errors and silencing other messages is > exactly what I expected logging to do by default. So we are slowly learning the things that should be on the first couple pages of the logging docs... 1) simple example for one file programs, include an example of specifying output severity threshold. I'm with Antoine here on my expectations. 2) example for multi-module, showing how a single logging destination causes logging to happen in all modules, at the same level (if that is the case, which I hope it is). 3) Maybe a small discussion of log formatting should be next? So the user realizes he shouldn't do the message formatting himself? 4) Then rotating logs for long-running programs. And then the rest of the more feature-filled stuff. I agree logging needs to be full-featured, but many of the features are only necessary in complex environments, and simple stuff should be simple, if it can be, and those cases explained first. Hopefully a useful subset for simple cases can be described in 1-3 pages, and then the complex features after that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From janssen at parc.com Fri Dec 10 22:26:02 2010 From: janssen at parc.com (Bill Janssen) Date: Fri, 10 Dec 2010 13:26:02 PST Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> Message-ID: <9235.1292016362@parc.com> Guido van Rossum wrote: > And yet, I have helped many people who were baffled by exactly what > Bill observed: logging.info() didn't do anything. Maybe the default > should be INFO? Yeah, I was curious enough to read the code and find out why. But many won't. By the way, I tried reading the 2.7 docs first, and didn't figure it out from them. So I looked at the code. Bill From janssen at parc.com Fri Dec 10 22:30:11 2010 From: janssen at parc.com (Bill Janssen) Date: Fri, 10 Dec 2010 13:30:11 PST Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <4D029439.7040708@g.nevcal.com> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D029439.7040708@g.nevcal.com> Message-ID: <9308.1292016611@parc.com> Glenn Linderman wrote: > 1) simple example for one file programs, include an example of > specifying output severity threshold. I'm with Antoine here on my > expectations. Yes, once I put logging.basicConfig(stream=sys.stdout, level=logging.DEBUG) in my main(), I got what I thought I'd get in the first place. All logging messages. But I can see Antoine's point: if we use this in the stdlib, you don't want that default. I think that logging events (context) have to come into this; you can't do it with just severity alone. I'd expect to have different settings, by default, for __main__ and for http.client, for instance. Bill From raymond.hettinger at gmail.com Fri Dec 10 22:35:03 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 10 Dec 2010 13:35:03 -0800 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <20101210215600.56f4eaf2@pitrou.net> References: <4D023721.1070904@netwok.org> <20101210215600.56f4eaf2@pitrou.net> Message-ID: <948611D4-66C3-4924-8737-DAC2F8A5DE35@gmail.com> On Dec 10, 2010, at 12:56 PM, Antoine Pitrou wrote: > On Fri, 10 Dec 2010 12:27:26 -0800 > Raymond Hettinger wrote: >> >> IMO, sysconfig did not warrant a whole module. > > Where would you put it? A single function in the sys module. > >> Rather than using two levels of dictionary, it's also possible >> to use a named tuple if you think that is more clean looking: >> >>>>> c = sys.sysconfig() >>>>> c.config_vars.get('SO) >> '.pyd' > > Some of these things are computed at runtime by parsing makefiles and > other stuff. You don't want to do it as soon as the module is imported. The proposal is for a function that does the computation when invoked, not when imported. The function returns a named tuple so that we can use Python's builtin accessors like attributes and dict.get(). Raymond From rdmurray at bitdance.com Fri Dec 10 22:59:02 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 10 Dec 2010 16:59:02 -0500 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: References: Message-ID: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> On Thu, 09 Dec 2010 16:18:14 -0800, Raymond Hettinger wrote: > Does anyone know why this needed a separate module and so many accessor functions? Originally sysconfig was moved *out* of distutils, and distutils was changed to use it. But that proved to be as fragile as many other distutils changes. When distutils was reverted, sysconfig was kept because of the long term desire to move it out of distutils. At that point I think it would have been very easy to propose API changes, but I suppose it didn't occur to anyone to think about that possibility. > ISTM it mostly could have been reduced to single call returning a nested dictionary. If what was returned was, as you suggested on IRC, a set of named tuples, it seems to me that would be a very convenient interface. And since all values (as I understand it) are intended to be static (even if some aren't at the moment), it seems to make sense from an implementation standpoint as well. Like ??ric, I'm not sure what the implications of the existing module having been released in 2.7 and 3.2 beta are in terms of making such an API change. -- R. David Murray www.bitdance.com From vinay_sajip at yahoo.co.uk Fri Dec 10 23:16:39 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 10 Dec 2010 22:16:39 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: Antoine Pitrou pitrou.net> writes: > Guido van Rossum python.org> wrote: > > And yet, I have helped many people who were baffled by exactly what > > Bill observed: logging.info() didn't do anything. Maybe the default > > should be INFO? > > Funny, because displaying only errors and silencing other messages is > exactly what I expected logging to do by default. The one thing I'm sure of is that it's hard to find more than a couple of people with the same view about what the default should be. Anyone for bikesheds? :-) IMO as long as it's just a small amount of work to get the specific effect that you want, it doesn't really matter too much what the default is - though of course I'd like it to be "right", whatever that means ;-) And whatever it is, it's going to upset *some* group of people because they have to add one line (or so) to get it to do what they want, and they feel really strongly that they shouldn't have to add that one line. In a way, I suppose I'm not unhappy that this is all we're arguing about and not anything more substantive. Some useful feedback about the documentation has emerged, too. I would (naturally) like to see broader adoption of logging in the standard library, and the view from Python core developers who have expressed opinions on this thread seems to be that the present behaviour is an obstacle to that adoption. I have my views but I'm not hidebound by them, hence the proposed change and the posting about it on the logging blog and c.l.py, just to let people know what's coming. It'll be interesting to see, once this change is implemented, whether the suggested obstacle really has been an obstacle, rather than (as I suspect) the low level of logging adoption in the stdlib being largely being a matter of plain inertia. But even if it helps make possible a better resolution of #10626 (as an archetype of unraisable-exception scenarios), that would be a good outcome. Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Fri Dec 10 23:21:11 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Fri, 10 Dec 2010 22:21:11 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D029439.7040708@g.nevcal.com> <9308.1292016611@parc.com> Message-ID: Bill Janssen parc.com> writes: > stdlib, you don't want that default. I think that logging events > (context) have to come into this; you can't do it with just severity > alone. I'd expect to have different settings, by default, for __main__ > and for http.client, for instance. > Although e.g. http.client may set a default severity, the developer integrating it into an application is free to override that severity for the purposes of their application. Typically, there are different severities set for different parts of an application at different times, depending on where the focus of diagnostic scrutiny is at those times. Severity is not the only filtering mechanism, merely the simplest and most common one, that people can readily grasp. Regards, Vinay Sajip From ncoghlan at gmail.com Sat Dec 11 03:07:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 12:07:18 +1000 Subject: [Python-Dev] futures API In-Reply-To: <8674D52A-07EF-4D4A-898A-983DE0AC6BC3@sweetapp.com> References: <953842.41772.qm@web27502.mail.ukl.yahoo.com> <8674D52A-07EF-4D4A-898A-983DE0AC6BC3@sweetapp.com> Message-ID: On Sat, Dec 11, 2010 at 6:07 AM, Brian Quinlan wrote: >> The problem also occurs when using a callback: >> http://www.freehackers.org/~tnagy/futures_test2.py >> >> If it is necessary to catch KeyboardInterrupt exceptions to cancel the >> futures execution, then how about adding this detail to the docs? > > AFAIK, catching KeyboardInterrupt exceptions is not sufficient. finally blocks and with statements can get you a fairly long way, though. futures does everything right on this front, as far as I can see. In this case, the problem is in the design of the test program. It *must* get exactly as many items in the queue as were submitted to the executor. If one is ever missing (e.g. due to a poorly timed KeyboardInterrupt in the callback, as clearly happened in the presented trace), it will hang in the final call to self.out_q.get(). There's a reason Queue.get offers both nonblocking and block-with-timeout functionality. (Alternately, the management of out_q and count could be made smarter, such that an exception in adding a result to out_q triggered an appropriate adjustment in the count of expected values) I also agree with Brian that the variable naming for the sample code and comments like "may use futures of different max_workers count. I think it is not too far-fetched to create a new futures object each time" suggests a fundamental confusion of the terms "future" and "executor". Executors are not futures - the return value from an executor's submit method is a future. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sat Dec 11 03:13:37 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 12:13:37 +1000 Subject: [Python-Dev] (Non-) use of weakref.WeakSet in concurrent.futures (was Re: futures API) Message-ID: On Sat, Dec 11, 2010 at 6:07 AM, Brian Quinlan wrote: > AFAIK, catching KeyboardInterrupt exceptions is not sufficient. Getting away from the flaws in the test code in this thread, is there any particular reason futures is rolling its own variant of weakref.WeakSet for the _thread_references cleanup sets? It seems to me that using WeakSet would mean you could get rid of _remove_dead_thread_references altogether (the implicit callbacks would handle that part), and then a set() call in _python_exit would give you concrete references to work with for cleanup purposes. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From stephen at xemacs.org Sat Dec 11 03:35:50 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 11 Dec 2010 11:35:50 +0900 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <4D029439.7040708@g.nevcal.com> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D029439.7040708@g.nevcal.com> Message-ID: <87k4jhrq7t.fsf@uwakimon.sk.tsukuba.ac.jp> Glenn Linderman writes: > 1) simple example for one file programs, include an example of > specifying output severity threshold. I'm with Antoine here on my > expectations. > > 2) example for multi-module, showing how a single logging destination > causes logging to happen in all modules, at the same level (if that is > the case, which I hope it is). > > 3) Maybe a small discussion of log formatting should be next? So the > user realizes he shouldn't do the message formatting himself? As advice, this shouldn't need to be more than a couple of lines, with a two line example, right? But it does deserve a prominent header. > 4) Then rotating logs for long-running programs. > > And then the rest of the more feature-filled stuff. +1 for this agenda. From tjreedy at udel.edu Sat Dec 11 03:42:19 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 10 Dec 2010 21:42:19 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On 12/10/2010 5:16 PM, Vinay Sajip wrote: > IMO as long as it's just a small amount of work to get the specific effect > that you want, it doesn't really matter too much what the default is - though > of course I'd like it to be "right", whatever that means ;-) I think the default should accomodate both internal uses (stdlib and elsewhere) and beginner uses. I suspect the latter would mean outputting most everything, to the convenience functions work (as a beginner might expect). -- Terry Jan Reedy From tjreedy at udel.edu Sat Dec 11 03:44:12 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 10 Dec 2010 21:44:12 -0500 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> Message-ID: On 12/10/2010 4:59 PM, R. David Murray wrote: > Like ?ric, I'm not sure what the implications of the existing module > having been released in 2.7 and 3.2 beta are in terms of making such an > API change. I am with Raymond on this: the purpose of betas is so we can test *and* make changes. No one should base production software on a beta release. -- Terry Jan Reedy From ncoghlan at gmail.com Sat Dec 11 03:51:17 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 12:51:17 +1000 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> Message-ID: On Sat, Dec 11, 2010 at 7:59 AM, R. David Murray wrote: > On Thu, 09 Dec 2010 16:18:14 -0800, Raymond Hettinger wrote: >> ISTM it mostly could have been reduced to single call returning a nested dictionary. > > If what was returned was, as you suggested on IRC, a set of named tuples, > it seems to me that would be a very convenient interface. ?And since > all values (as I understand it) are intended to be static (even if some > aren't at the moment), it seems to make sense from an implementation > standpoint as well. > > Like ?ric, I'm not sure what the implications of the existing module > having been released in 2.7 and 3.2 beta are in terms of making such an > API change. We've reverted from beta to pseudo-alpha after discovering sufficiently egregious mistakes in the past. (we didn't *actually* revert the build naming because release IDs that don't increase monotonically can confuse matters). However, I don't think this is sufficiently egregious to qualify. My own reasons for advocating retention of the separate module: 1. Something equivalent to the current sysconfig module is going to be needed to actually work out the contents of the data structure. Rewriting all that in C would be silly, so the most that could happen is to rename "sysconfig" to "_sysconfig". 2. Invoking Python code from sys module functions is possible but untidy (generally involving implicit module imports). Since we need a module anyway for implementation purposes, why not make it public? 3. The sysconfig module docs provide a nicer space to talk about the purpose of the provided information rather than squirreling it away in a corner of the sys module docs 4. "python -m sysconfig" provides a useful dump of the system status 5. The API design did not get much *new* thought, as the idea was to allow users to do anything that the old distutils.sysconfig could do, but without requiring them to depend on distutils to get that functionality. The easiest way to ensure that is to just copy the existing API rather than trying to get creative. Don't think of it as adding a "whole new module", thinking of it as decoupling a relatively small existing module (i.e. distutils.sysconfig) from a larger package that isn't installed by default on all systems (i.e. distutils). A concrete problem with Raymond's idea in particular is that there is actually a full set of paths defined for *each scheme*. get_path() and get_paths() let you ignore this most of the time, since there is an appropriate default scheme defined for each platform, so they just use that if you don't specify one explicitly. A named tuple could obviously expose the paths for all of the schemes, with the paths for the default scheme duplicated at the top level, but the data structure starts looking very messy at that point. The named tuple idea also conflates meta-data about the sysconfig data (the list of scheme names and path names) with the actual data for the current platform (platform, version, paths, config_vars). So I think reusing the tried and tested API is easily justified in this case. What may be desirable in 3.3 (particularly with the distutils2 reintegration taking place), is to revisit the sysconfig API to see if there are better ways to expose some of the data (such as a function returning a named tuple containing the raw data underlying the output of "python -m sysconfig"). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sat Dec 11 03:55:25 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 12:55:25 +1000 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> Message-ID: On Sat, Dec 11, 2010 at 12:44 PM, Terry Reedy wrote: > On 12/10/2010 4:59 PM, R. David Murray wrote: > >> Like ?ric, I'm not sure what the implications of the existing module >> having been released in 2.7 and 3.2 beta are in terms of making such an >> API change. > > I am with Raymond on this: the purpose of betas is so we can test *and* make > changes. No one should base production software on a beta release. It's also the key difference between betas and release candidates. Completely new features are still out after the first beta, but tweaking the APIs of new-in-this-release features is still possible with sufficient justification. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ethan at stoneleaf.us Sat Dec 11 05:19:23 2010 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 10 Dec 2010 20:19:23 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <87oc8tsjll.fsf@uwakimon.sk.tsukuba.ac.jp> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <87tyimrmbd.fsf@uwakimon.sk.tsukuba.ac.jp> <20101210113327.GA16785@phd.pp.ru> <87oc8tsjll.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4D02FBCB.50600@stoneleaf.us> Stephen J. Turnbull wrote: > They're not scared by that example. What you need is a paragraph > below it that says > > """ > Do you think the above is all you should need? If so, you're > right. You can stop reading now. If you think you need more, > we've got that, too. Read on (you may need more coffee). > """ +1 Oleg Broytman writes: > > > Better yet (IMHO) would be to split the huge page into "Logging: Simple > > start" and "Logging: Advanced usage (for the brave of of heart)". > > Splitting is OK, but I disagree about the gloss "for the brave of > heart". > > In my experience, if it is a YAGNI, the complexity is nearly > impenetrable. If you *do* need it, it's not at all difficult to > understand what the complexity is for, and it doesn't even look all > that complex because it matches up with the problem you need to solve. > > If the documentation is still a deterrent, that's a problem with the > documentation and it should be improved. AFAICT, making it clear that > exporting all the internal flexibility is for those who need it, while > most users will rarely need it, should be enough. But I'm not a good > test case, since I already am familiar with XEmacs's similar system. I think I'm a pretty good test case -- knew nothing about logging, still don't know much, found documentation comprehensive but unweildy, and would *still* benefit from a more extensive (though still short) beginner's section, with the prominent paragraph stating I now know enough for simple cases. :) Oh, and awesome module, by the way. Thank you. ~Ethan~ From ncoghlan at gmail.com Sat Dec 11 06:06:20 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 15:06:20 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On Sat, Dec 11, 2010 at 12:42 PM, Terry Reedy wrote: > On 12/10/2010 5:16 PM, Vinay Sajip wrote: > >> IMO as long as it's just a small amount of work to get the specific effect >> that you want, it doesn't really matter too much what the default is - >> though >> of course I'd like it to be "right", whatever that means ;-) > > I think the default should accomodate both internal uses (stdlib and > elsewhere) and beginner uses. I suspect the latter would mean outputting > most everything, to the convenience functions work (as a beginner might > expect). I think the main point of confusion is going to be what happens to .info() messages by default. With Vinay's latest changes, the 3.2 default is going to be: debug() -> /dev/null info() -> /dev/null warning() -> stderr error() -> stderr critical() -> stderr exception() -> stderr This will apply both to the convenience functions (where this is already the case due to the implicit basicConfig() call and the default level setting on the root logger) and to logger methods (where the new "handler of last resort" will apply instead of emitting the comparatively unhelpful "no handler found" error message). This is the correct behaviour for library code and even many applications - silent when all goes well, but if anything goes wrong that can't be flagged with an exception, the errors are displayed by default. While Vinay is right that inertia plays a large part in not using the logging module in the standard library, the new behaviour for direct calls to loggers *is* a direct result of a problem encountered in the concurrent.futures tests due to the fact that Brian previously couldn't rely on an unconfigured logging module doing the right thing. However, the confusion that this setup will engender is that encountered by Bill: by default, info() messages are silenced rather than displayed on stdout. Notice that even the recommended "basicConfig" approach to resolving this is subtly flawed, since your choices are to either display info messages on stderr (by specifying just a level argument) or to redirect warning() et al to stdout instead of stderr (by also specifying a stream argument). The "correct" answer (info and below to stdout, warning and above to stderr) is actually quite tricky to set up, since handlers have no innate concept of a "maximum level", so you have to provide a filter function (more on that below [1]). Basically, as of 3.2, the correct "default" use is likely to be: print() for information you want to appear on stdout by default (especially in scripts) logging.debug() and .info() for additional debugging information that should be silenced by default warnings.warn() for actual programmatic warnings that should obey warning filter rules (logging can hook the warnings display mechanism) logging.warn() for warnings that should never raise an exception and should be emitted every time they are encountered raise an exception for normal error reporting logging.error(), .critical() and .exception() for error reporting that cannot raise an exception for some reason (e.g. long running processes that are simply recording that an exception occurred) Libraries can now safely use their own getLogger(__name__) loggers for the latter two uses without worrying about the output being lost by the default logging configuration, so they become a much better option than printing directly to sys.stderr. Most importantly, someone debugging the app will be able to insert additional context information (e.g. stack traces, local variables values) without needing to alter the code emitting the error message. Regards, Nick. [1] As an exercise, I decided to actually configure recommended console logging output style via the logging module. It proved to be quite informative as to where some of the flaws in the current documentation lie. First and foremost, it was genuinely hard for me to figure out that one of my key problems was actually that the root logger is created with a level of WARNING by default so my handlers weren't even being invoked. The only place I found this very important detail was in the documentation of Logger.setLevel()! There really should be a section early on in the logging documentation describing the root logger, how to use it for basic logging, and its default settings (with and without a call to "basicConfig"). Such a section would then include a link to the latter section on the logging heirarchy. In addition to that, the documentation of the LogRecord class should really include a table of attribute names that are available for use in formatting and filtering. Initially I thought the attribute names corresponded to the parameter names, and subsequently resorted to a dir() call in my filter to figure out what the actual names were after "lvl" caused an AttributeError. Armed with the fact that the attribute I wanted was called "levelno", I was able to discover the list of LogRecord attributes was actually in the *Formatter* documentation. If a proper list of attributes was added to the LogRecord documentation, then the current table in the formatting section could be removed and replaced with a reference to the table of attributes and some notes on formatting (when the documentation for the new styles feature goes in would probably be an appropriate time to fix this). Similarly, the section on filters could point directly to that table when describing possible filtering criteria. Anyway, the shortest way I could find of setting this up (debug silenced, info written to stdout, warning and above written to stderr): import sys, logging root = logging.getLogger() # Turns out the level of the root logger is set to WARNING by default # That should be much easier to find out from the logging docs! root.setLevel(logging.DEBUG) # Send WARNING and above to stderr # basicConfig() is no help, since it sets the specified level on the root logger # but NOT on the default handler it installs. To emulate console output # we want to skip the formatting anyway. err = logging.StreamHandler() err.setLevel(logging.WARNING) root.addHandler(err) # Send INFO to stdout def is_info(record): return record.levelno == logging.INFO out = logging.StreamHandler(stream=sys.stdout) out.addFilter(is_info) root.addHandler(out) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sat Dec 11 06:24:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 15:24:39 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On Sat, Dec 11, 2010 at 3:06 PM, Nick Coghlan wrote: > Basically, as of 3.2, the correct "default" use is likely to be: > > print() for information you want to appear on stdout by default > (especially in scripts) > logging.debug() and .info() for additional debugging information that > should be silenced by default > > warnings.warn() for actual programmatic warnings that should obey > warning filter rules (logging can hook the warnings display mechanism) > logging.warn() for warnings that should never raise an exception and > should be emitted every time they are encountered > > raise an exception for normal error reporting > logging.error(), .critical() and .exception() for error reporting that > cannot raise an exception for some reason (e.g. long running processes > that are simply recording that an exception occurred) This could actually make a reasonably good basic for a "task oriented" subsection of the logging documentation. Something like: =============== Task: Display console output for ordinary usage of a command line script or program Tool: Use print() Task: Report events that occur during normal operation of a program (e.g. for status monitoring or fault investigation) Tool: Use logging.info() (or logging.debug() for especially voluminous or expensive to calculate output) Task: Issue a warning regarding a particular runtime event Tool: Use warnings.warn() in library code if the issue is avoidable and the client application should be modified to eliminate the warning. Use logging.warn() if there is nothing the client application can do about the situation, but the event should still be noted Task: Report an error regarding a particular runtime event Tool: Raise an exception Task: Report suppression of an error without raising an exception (e.g. error handler in a long-running server process) Tool: Use logging.error(), logging.exception(), or logging.critical() as appropriate to the specific error and application domain Note: all references to the root level logging functions in the above recommendations also cover direct invocation of the corresponding methods on specific logger instances. =============== Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sat Dec 11 06:38:52 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 15:38:52 +1000 Subject: [Python-Dev] [Python-checkins] r87162 - python/branches/release27-maint/Doc/library/stdtypes.rst In-Reply-To: <20101211004102.E773DEE98A@mail.python.org> References: <20101211004102.E773DEE98A@mail.python.org> Message-ID: On Sat, Dec 11, 2010 at 10:41 AM, raymond.hettinger wrote: > Author: raymond.hettinger > Date: Sat Dec 11 01:41:02 2010 > New Revision: 87162 > > Log: > Issue 2690: ?Doc fixup. ?xrange() objects are slicable. > > > Modified: > ? python/branches/release27-maint/Doc/library/stdtypes.rst Are you sure about that one? Python 2.7.0+ (trunk:86033, Nov 1 2010, 00:42:57) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> xrange(10)[1:2] Traceback (most recent call last): File "", line 1, in TypeError: sequence index must be integer, not 'slice' Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Sat Dec 11 07:18:57 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 11 Dec 2010 06:18:57 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: Nick, Thanks for the detailed exposition. > Notice that even the recommended "basicConfig" approach to resolving > this is subtly flawed, since your choices are to either display info Right. basicConfig() is pretty basic. > Basically, as of 3.2, the correct "default" use is likely to be: The suggestions make sense. > [1] As an exercise, I decided to actually configure recommended > console logging output style via the logging module. It proved to be > quite informative as to where some of the flaws in the current > documentation lie. [snip] > documentation of Logger.setLevel()! There really should be a section > early on in the logging documentation describing the root logger, how Agreed. > In addition to that, the documentation of the LogRecord class should > really include a table of attribute names that are available for use > in formatting and filtering. Initially I thought the attribute names Yes. > formatting (when the documentation for the new styles feature goes in > would probably be an appropriate time to fix this). Similarly, the Sorry, what do you mean by "new styles feature"? > Anyway, the shortest way I could find of setting this up (debug Yes, except that you could have set the root logger level to INFO rather than DEBUG. This raises some more questions. Obviously, we don't want users to have to go through these steps themselves. Once these steps have been finalised, the last resort handler could be modified to do all this. Should this handler be part of the public API, so that it can be inherited from etc? I realise we're in beta and hence feature freeze ... just sayin'. We're all consenting adults here, after all :-) Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Sat Dec 11 07:21:14 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 11 Dec 2010 06:21:14 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: Nick Coghlan gmail.com> writes: > This could actually make a reasonably good basic for a "task oriented" > subsection of the logging documentation. Something like: > Good suggestion, I'll see what I can do. Thanks, Vinay Sajip From v+python at g.nevcal.com Sat Dec 11 07:25:54 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 10 Dec 2010 22:25:54 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: <4D031972.5090105@g.nevcal.com> On 12/10/2010 9:06 PM, Nick Coghlan wrote: > Anyway, the shortest way I could find of setting this up (debug > silenced, info written to stdout, warning and above written to > stderr): > > import sys, logging > root = logging.getLogger() > # Turns out the level of the root logger is set to WARNING by default > # That should be much easier to find out from the logging docs! > root.setLevel(logging.DEBUG) > # Send WARNING and above to stderr > # basicConfig() is no help, since it sets the specified level on the > root logger > # but NOT on the default handler it installs. To emulate console output > # we want to skip the formatting anyway. > err = logging.StreamHandler() > err.setLevel(logging.WARNING) > root.addHandler(err) > # Send INFO to stdout > def is_info(record): > return record.levelno == logging.INFO > out = logging.StreamHandler(stream=sys.stdout) > out.addFilter(is_info) > root.addHandler(out) Wow, that's way longer than I expected... almost as long as my "cheap logger". Greg didn't flesh out the "setup" stuff, but you have, so this is educational. Thanks. Not sure I want my INFO to go to stdout, by the way. That corresponds to "debugging print statement" behavior, but even my "cheap logger" actually puts stuff in a file instead, and gets it out of the way of the normal output. I'd suggest that this be "simplified" to put all the messages in the same (disk) file, but then I'm sure that there are other preferences than mine... you clearly thought about what you wanted/thought would be useful, in coming up with the above. On 12/10/2010 9:24 PM, Nick Coghlan wrote: > This could actually make a reasonably good basic for a "task oriented" > subsection of the logging documentation. Something like: Yep, agree. But sadly, for each point, there may be multiple options (your StreamHandler, but I'd want a FileHandler; your separation of messages by level, my wanting them combined; etc.) Your comment about basicConfig setting the level on the root logger, but not on the default handler making it useless is opaque to me, but is there perhaps room for another basic setup API that could get the setup code down to a line or two in simple cases? Maybe 3 parameters: 1. minimum level to be generated, which would be passed through to the root logger and anything else defined by this basic setup API 2. whether debug and info should go to the same or different stream/file as warn+ messages. Maybe this is a stream or filename or None, the last implying to use the warn+ output. 3. Where to send the warn+ output. maybe a 4. Maximum (size, count) of the round-robin log files, assuming that either or both of 2 & 3 specify files. Would that be a useful set of functionality to bundle? And could it be extended, when the user wants more power, or would it have to be replaced, because it gets in the way of the user that wants more power? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 11 09:00:39 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 18:00:39 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <4D031972.5090105@g.nevcal.com> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D031972.5090105@g.nevcal.com> Message-ID: On Sat, Dec 11, 2010 at 4:25 PM, Glenn Linderman wrote: > On 12/10/2010 9:24 PM, Nick Coghlan wrote: > This could actually make a reasonably good basic for a "task oriented" > subsection of the logging documentation. Something like: > > Yep, agree.? But sadly, for each point, there may be multiple options (your > StreamHandler, but I'd want a FileHandler; your separation of messages by > level, my wanting them combined; etc.) No, no, no, that's the *whole point* of using logging. The library writer doesn't really *care* about where the messages end up - that is entirely in the hands of the application developer when they choose which handlers to install. The only situation that the library writer cares about is the one that tripped up concurrent.futures and that has already been changed for 3.2b2: that warnings and errors written to a logger were silenced by default if the application never even called basicConfig(). That has now been fixed so they will be emitted on sys.stderr instead (including obeying any redirections of stderr over the lifetime of the program). Notice that my task list is *entirely* from the point of view of the person emitting the messages. How those messages are later displayed is then up to the application writer (or the logging module default settings, if the application writer doesn't do anything specific). > Your comment about basicConfig setting the level on the root logger, but not > on the default handler making it useless is opaque to me, An early version of my code used basicConfig to create the stderr StreamHandler, but that turned out to be pointless since I wanted different levels on the logger and the handler. > but is there > perhaps room for another basic setup API that could get the setup code down > to a line or two in simple cases? For 3.2? No. For 3.3? Maybe. > Would that be a useful set of functionality to bundle?? And could it be > extended, when the user wants more power, or would it have to be replaced, > because it gets in the way of the user that wants more power? Logging already has powerful configuration mechanisms, especially following the addition of dictConfig (http://docs.python.org/dev/library/logging#logging-config-dictschema), so it really doesn't need anything else along those lines. At the simpler end, basicConfig already covers sending all messages to a single stream with a single format - it's only split-stream handling (such as stderr/stdout potentially being different endpoints) that it can't cope with. What may make more sense than yet another global config mechanism, is a module level "addHandler" helper function along the following lines: from logging import Formatter, FileHandler, StreamHandler, getLogger def addHandler(*, handler=None, stream=None, filename=None, filemode='a', format=None, datefmt=None, style='{', level=None, max_level=None, filters=(), logger=None): """stream, filename, level, format, datefmt, style: as per logging.basicConfig handler: use a precreated handler instead of creating a new one logger: logger to add the handler to (uses root logger if none specified) filters: an iterable of filters to add to the handler max_level: can optionally limit the handler to messages of a certain level and below """ # Create the handler if one hasn't been passed in if handler is None: if filename is not None: handler = FileHandler(filename, filemode) else: handler = StreamHandler(stream) # Set up the formatting of the log messages # New API, so it can default to str.format instead of %-formatting formatter = Formatter(format, datefmt, style) handler.setFormatter(formatter) # Set up filtering of which messages to handle if level is not None: handler.setLevel(level) if max_level is not None: def level_ok(record): return record.levelno <= max_level handler.addFilter(level_ok) for filter in filters: handler.addFilter(filter) # Add the fully configured handler to the specified logger if logger is None: logger = getLogger() logger.addHandler(handler) return handler # Previous set up example is now only three lines long import sys, logging # Let root logger handlers see all messages logging.getLogger().setLevel(logging.NOTSET) # Send WARNING and above to stderr addHandler(stream=sys.stderr, level=logging.WARNING) # Send INFO to stdout addHandler(stream=sys.stdout, level=logging.INFO, max_level=logging.INFO) logging.info("Hello world!") logging.warn("Hello world!") Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sat Dec 11 09:14:27 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 18:14:27 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D031972.5090105@g.nevcal.com> Message-ID: On Sat, Dec 11, 2010 at 6:00 PM, Nick Coghlan wrote: > What may make more sense than yet another global config mechanism, is > a module level "addHandler" helper function along the following lines: For a more readable version of that example, take a look at the copy I put up over on the Python Cookbook site: http://code.activestate.com/recipes/577496-simple-creation-configuration-and-installation-of-/ Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ziade.tarek at gmail.com Sat Dec 11 09:26:35 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sat, 11 Dec 2010 09:26:35 +0100 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> Message-ID: Sorry to get late in the discussion I am travelling. Nick resumes well the motivations behind sysconfig. I'll emphase that this module could hold more functions in the future that could be useful to other python implementations to abstract what is a python installation. E.g. more than paths and variables. On Dec 11, 2010 3:53 AM, "Nick Coghlan" wrote: On Sat, Dec 11, 2010 at 7:59 AM, R. David Murray wrote: > On Thu, 09 Dec 201... >> ISTM it mostly could have been reduced to single call returning a nested dictionary. > > If what ... We've reverted from beta to pseudo-alpha after discovering sufficiently egregious mistakes in the past. (we didn't *actually* revert the build naming because release IDs that don't increase monotonically can confuse matters). However, I don't think this is sufficiently egregious to qualify. My own reasons for advocating retention of the separate module: 1. Something equivalent to the current sysconfig module is going to be needed to actually work out the contents of the data structure. Rewriting all that in C would be silly, so the most that could happen is to rename "sysconfig" to "_sysconfig". 2. Invoking Python code from sys module functions is possible but untidy (generally involving implicit module imports). Since we need a module anyway for implementation purposes, why not make it public? 3. The sysconfig module docs provide a nicer space to talk about the purpose of the provided information rather than squirreling it away in a corner of the sys module docs 4. "python -m sysconfig" provides a useful dump of the system status 5. The API design did not get much *new* thought, as the idea was to allow users to do anything that the old distutils.sysconfig could do, but without requiring them to depend on distutils to get that functionality. The easiest way to ensure that is to just copy the existing API rather than trying to get creative. Don't think of it as adding a "whole new module", thinking of it as decoupling a relatively small existing module (i.e. distutils.sysconfig) from a larger package that isn't installed by default on all systems (i.e. distutils). A concrete problem with Raymond's idea in particular is that there is actually a full set of paths defined for *each scheme*. get_path() and get_paths() let you ignore this most of the time, since there is an appropriate default scheme defined for each platform, so they just use that if you don't specify one explicitly. A named tuple could obviously expose the paths for all of the schemes, with the paths for the default scheme duplicated at the top level, but the data structure starts looking very messy at that point. The named tuple idea also conflates meta-data about the sysconfig data (the list of scheme names and path names) with the actual data for the current platform (platform, version, paths, config_vars). So I think reusing the tried and tested API is easily justified in this case. What may be desirable in 3.3 (particularly with the distutils2 reintegration taking place), is to revisit the sysconfig API to see if there are better ways to expose some of the data (such as a function returning a named tuple containing the raw data underlying the output of "python -m sysconfig"). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia ________________... Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Uns... -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 11 09:35:29 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 18:35:29 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On Sat, Dec 11, 2010 at 4:18 PM, Vinay Sajip wrote: >> formatting (when the documentation for the new styles feature goes in >> would probably be an appropriate time to fix this). Similarly, the > > Sorry, what do you mean by "new styles feature"? The switching between percent, str.format and string.Template formatting. It turns out that has been documented, but the table in question is still written from a percent-style formatting point of view. >> Anyway, the shortest way I could find of setting this up (debug > > Yes, except that you could have set the root logger level to INFO rather than > DEBUG. > > This raises some more questions. Obviously, we don't want users to have to go > through these steps themselves. Once these steps have been finalised, the last > resort handler could be modified to do all this. Should this handler be part of > the public API, so that it can be inherited from etc? The "lazy" stream handler might be useful to make public in some way. For example, rather than hardcoding sys.stderr, it could take a callable that it uses to retrieve the stream. That kind of change can wait until 3.3 though - what is there now should be fine to let us get concurrent.futures and the logging tests to play nicely together. As far as whether anything else in the logging defaults needs to change, I don't think so. I'm now happy that if an application needs to emit text on stdout as part of its normal operation, then that isn't what the logging module is for - logging messages are intended to record "this happened, then that happened, then this other thing happened". They are *not* properly part of an application's console UI (which is reflected in the whole "silent by default" and "console output defaults to sys.stderr" philosophy). The change to the default behaviour was just to make logging.warning() etc a suitable replacement for writing directly to sys.stderr when exceptions can't be raised, which is in keeping with the existing spirit of the logging module. > I realise we're in beta and hence feature freeze ... just sayin'. We're all > consenting adults here, after all :-) I don't think we should be adding more *code* at this stage. But documentation fixes and adjustments may be a good idea. I've learned quite a bit myself about the logging module in the course of this discussion, so it would be nice if some of that could be captured and used to improve the documentation. That said, something that might be interesting is to see what my addHandler() recipe [1] would do for the examples in the logging documentation. If it makes a big enough difference, then it may be worth talking to Georg about adding it, even for 3.2. Alternatively, if you like the idea, but we don't want to break the feature freeze for it, then it may make a nice "see-also" link from the docs. I find it useful from a "here are all the things I can configure on a handler in one easy bundle" point of view, even aside from any benefits in reduced typing. Cheers, Nick. [1] http://code.activestate.com/recipes/577496-simple-creation-configuration-and-installation-of-/ -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Sat Dec 11 09:52:45 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 11 Dec 2010 08:52:45 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D031972.5090105@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > Yep, agree.? But sadly, for each point, there may be multiple > options (your StreamHandler, but I'd want a FileHandler; your > separation of messages by level, my wanting them combined; etc.) That's partly why logging hasn't made much effort in the "default" department: everyone wants something different. > Your comment about basicConfig setting the level on the root logger, > but not on the default handler making it useless is opaque to me, I think he just means that he can't use basicConfig because it just sets the root logger's level to whatever's passed in, but what he actually wants to set are two different levels: INFO on the logger, WARNING on the stderr handler. On top of that, he wants to have a stdout handler which only outputs INFO events, and to get that effect he has to add the filter to the stdout handler (otherwise it would output INFO + anything at a higher level, too). > but is there perhaps room for another basic setup API that could get > the setup code down to a line or two in simple cases?? Maybe 3 > parameters: [snip] > Would that be a useful set of functionality to bundle?? And could it > be extended, when the user wants more power, or would it have to be > replaced, because it gets in the way of the user that wants more > power? > It's possible that perhaps there could be some 'canned configs', as I've mentioned in another post on this thread. Some more thinking needs to be done around this area. Regards, Vinay Sajip From ncoghlan at gmail.com Sat Dec 11 10:07:55 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 19:07:55 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On Sat, Dec 11, 2010 at 3:06 PM, Nick Coghlan wrote: > However, the confusion that this setup will engender is that > encountered by Bill: by default, info() messages are silenced rather > than displayed on stdout. > > Notice that even the recommended "basicConfig" approach to resolving > this is subtly flawed, since your choices are to either display info > messages on stderr (by specifying just a level argument) or to > redirect warning() et al to stdout instead of stderr (by also > specifying a stream argument). The "correct" answer (info and below to > stdout, warning and above to stderr) is actually quite tricky to set > up, since handlers have no innate concept of a "maximum level", so you > have to provide a filter function (more on that below [1]). I need to correct this comment, since it's wrong and basicConfig(level=logging.INFO) actually does the right thing. As Glenn mentioned later in the thread, the output of logging.info and logging.debug messages is *distinct* from an application's normal operational output that is emitted on stdout. So making it easy to emit such messages on stderr is the right thing to do - it's OK that it requires a bit of additional set up to get them onto stdout instead (since they really don't belong there most of the time). I know my own rule of thumb is going to be updated along the lines of "Am I writing to sys.stderr, or guarding output with a verbosity flag? Then perhaps I should be using the logging module rather than print statements". Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Sat Dec 11 10:28:14 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 11 Dec 2010 09:28:14 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: Nick Coghlan gmail.com> writes: > The switching between percent, str.format and string.Template > formatting. It turns out that has been documented, but the table in > question is still written from a percent-style formatting point of > view. Oh, right - yes. I presume you mean the table in the formatter documentation which shows %(message)s etc. > The "lazy" stream handler might be useful to make public in some way. > For example, rather than hardcoding sys.stderr, it could take a > callable that it uses to retrieve the stream. That kind of change can > wait until 3.3 though - what is there now should be fine to let us get > concurrent.futures and the logging tests to play nicely together. Actually if we're to change things to print INFO to stdout and WARNING+ to stderr, I'd change the last resort handler class to be more than just defining a lazy stream property: perhaps something like [untested] class _DefaultHandler(StreamHandler): def __init__(self, level=NOTSET): """ Initialize the handler. """ Handler.__init__(self, level) def emit(self, record): if record.levelno == INFO: self.stream = sys.stdout else: self.stream = sys.stderr super(_DefaultHandler, self).emit(record) > As far as whether anything else in the logging defaults needs to > change, I don't think so. I'm now happy that if an application needs I agree it can be refined later, but if we agreed that INFO goes to stdout, then something like the above could be put in place for 3.2 (perhaps 3.2b2). > output defaults to sys.stderr" philosophy). The change to the default > behaviour was just to make logging.warning() etc a suitable > replacement for writing directly to sys.stderr when exceptions can't > be raised, which is in keeping with the existing spirit of the logging > module. Agreed. > I don't think we should be adding more *code* at this stage. But > documentation fixes and adjustments may be a good idea. I've learned > quite a bit myself about the logging module in the course of this > discussion, so it would be nice if some of that could be captured and > used to improve the documentation. Yes, you (and Glenn, and others) have given me quite a bit to work on, on that score! > That said, something that might be interesting is to see what my > addHandler() recipe [1] would do for the examples in the logging > documentation. If it makes a big enough difference, then it may be > worth talking to Georg about adding it, even for 3.2. Alternatively, > if you like the idea, but we don't want to break the feature freeze > for it, then it may make a nice "see-also" link from the docs. I find > it useful from a "here are all the things I can configure on a handler > in one easy bundle" point of view, even aside from any benefits in > reduced typing. Actually I've been thinking about a handlers= optional argument for basicConfig for some time (which, if provided, would override file and stream arguments). But it seemed a little unwieldy and came across plainly as the afterthought it admittedly was :-) Regards, Vinay Sajip From v+python at g.nevcal.com Sat Dec 11 10:30:58 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 11 Dec 2010 01:30:58 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D031972.5090105@g.nevcal.com> Message-ID: <4D0344D2.3070006@g.nevcal.com> On 12/11/2010 12:00 AM, Nick Coghlan wrote: > On Sat, Dec 11, 2010 at 4:25 PM, Glenn Linderman wrote: >> On 12/10/2010 9:24 PM, Nick Coghlan wrote: >> This could actually make a reasonably good basic for a "task oriented" >> subsection of the logging documentation. Something like: >> >> Yep, agree. But sadly, for each point, there may be multiple options (your >> StreamHandler, but I'd want a FileHandler; your separation of messages by >> level, my wanting them combined; etc.) > No, no, no, that's the *whole point* of using logging. The library > writer doesn't really *care* about where the messages end up - that is > entirely in the hands of the application developer when they choose > which handlers to install. The only situation that the library writer > cares about is the one that tripped up concurrent.futures and that has > already been changed for 3.2b2: that warnings and errors written to a > logger were silenced by default if the application never even called > basicConfig(). That has now been fixed so they will be emitted on > sys.stderr instead (including obeying any redirections of stderr over > the lifetime of the program). > > Notice that my task list is *entirely* from the point of view of the > person emitting the messages. How those messages are later displayed > is then up to the application writer (or the logging module default > settings, if the application writer doesn't do anything specific). Yes, yes, yes... that's what your list was! I see that now! But it is interesting that in missing that point, and reading your logging setup code, I saw that the same list of tasks is also exactly the sorts of things that you would want to control, for simple uses of the logger. Does basicConfig give that control? I don't know, the first 8% of the logger documentation don't tell me that, they only give me a canned example (actually 3) without explanation of its full function. Someday I'll read more. Until then, basicConfig is opaque: I know only three incantations for using it, and not even any surety that they can be mixed, or that it can be called more than once in the same program. >> Your comment about basicConfig setting the level on the root logger, but not >> on the default handler making it useless is opaque to me, > An early version of my code used basicConfig to create the stderr > StreamHandler, but that turned out to be pointless since I wanted > different levels on the logger and the handler. > >> but is there >> perhaps room for another basic setup API that could get the setup code down >> to a line or two in simple cases? > For 3.2? No. > > For 3.3? Maybe. > >> Would that be a useful set of functionality to bundle? And could it be >> extended, when the user wants more power, or would it have to be replaced, >> because it gets in the way of the user that wants more power? > Logging already has powerful configuration mechanisms, especially > following the addition of dictConfig > (http://docs.python.org/dev/library/logging#logging-config-dictschema), > so it really doesn't need anything else along those lines. Sure; that is the replacement technique. And maybe that is OK, if there are no useful intermediates between basicConfig (whatever it might be capable of) and dictConfig (or the other various config techniques that preceded it and still exist, if I understand correctly from has been said here on python-dev). > At the simpler end, basicConfig already covers sending all messages to > a single stream with a single format - it's only split-stream handling > (such as stderr/stdout potentially being different endpoints) that it > can't cope with. OK, so I'd probably be fine with basicConfig, even though you weren't, once I understand it a little better. > What may make more sense than yet another global config mechanism, is > a module level "addHandler" helper function along the following lines: And once I learn enough to understand the addHandler code, I will have read a lot of the logger documentation than I have to date, or it will have been restructured enough that re-reading the first 8%, or maybe 12% if the easy-to-understand part grows! But I like what you said here: > # Previous set up example is now only three lines long > import sys, logging > # Let root logger handlers see all messages > logging.getLogger().setLevel(logging.NOTSET) > # Send WARNING and above to stderr > addHandler(stream=sys.stderr, level=logging.WARNING) > # Send INFO to stdout > addHandler(stream=sys.stdout, level=logging.INFO, max_level=logging.INFO) > > logging.info("Hello world!") > logging.warn("Hello world!") And this subset usage of addHandler is reasonably understandable for beginners, although clearly the complexity of all the parameters must add power that is not needed in the simple case, that you feel could be useful in the complex case. And I would change the third line from level=logging.INFO to level=logging.DEBUG to see lots more on stdout, correct? Looking back at the logger's simple examples section, I see that getLogger is used before being defined, is not explained, nor is its parameter explained. So it is not clear how to go from the first simple example to the second simple example. I wonder if that is where I started getting a bit of uncertainty about the ability to quickly learn what was going on... the rotating log example uses several APIs that haven't been used before, without explaining them at all. Can that example actually be added to the previous example? What is the magic 'MyLogger'? Where has basicConfig() gone in the second example? I'm not expecting answers to the above questions here, but I'd love to see the documentation improved to the point that I don't have them any more :) Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Sat Dec 11 10:40:29 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 11 Dec 2010 01:40:29 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: <4D03470D.4090407@g.nevcal.com> On 12/11/2010 1:28 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: >> The "lazy" stream handler might be useful to make public in some way. >> For example, rather than hardcoding sys.stderr, it could take a >> callable that it uses to retrieve the stream. That kind of change can >> wait until 3.3 though - what is there now should be fine to let us get >> concurrent.futures and the logging tests to play nicely together. > Actually if we're to change things to print INFO to stdout and WARNING+ to > stderr, I'd change the last resort handler class to be more than just defining > a lazy stream property: perhaps something like [untested] > > class _DefaultHandler(StreamHandler): > def __init__(self, level=NOTSET): > """ > Initialize the handler. > """ > Handler.__init__(self, level) > > def emit(self, record): > if record.levelno == INFO: > self.stream = sys.stdout > else: > self.stream = sys.stderr > super(_DefaultHandler, self).emit(record) So I think Nick's configuration stuff is something that is supposed to be done by the application, not by the library, or am I missing that too? And so if it is done by the application, then Nick's, or any other application writer's, preferences for where to send which messages can likely be accommodated. But I think that by default anything that is not part of the application's defined output, should go to stderr... even INFO messages. I have no problem with Nick wanting them on stdout, but I would have a problem with libraries (or the logger of last resort) sending them there by default. >> As far as whether anything else in the logging defaults needs to >> change, I don't think so. I'm now happy that if an application needs > I agree it can be refined later, but if we agreed that INFO goes to stdout, > then something like the above could be put in place for 3.2 (perhaps 3.2b2). I'd find that hard to agree to, but I probably don't count. I like Nick's statement. Libraries spewing messages to stderr is fine, spewing them to stdout is not, by default. -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Sat Dec 11 10:44:33 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 11 Dec 2010 01:44:33 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: <4D034801.5020308@g.nevcal.com> On 12/11/2010 1:07 AM, Nick Coghlan wrote: > As Glenn mentioned later in the thread, the output of logging.info and > logging.debug messages is*distinct* from an application's normal > operational output that is emitted on stdout. So making it easy to > emit such messages on stderr is the right thing to do - it's OK that > it requires a bit of additional set up to get them onto stdout instead > (since they really don't belong there most of the time). > > I know my own rule of thumb is going to be updated along the lines of > "Am I writing to sys.stderr, or guarding output with a verbosity flag? > Then perhaps I should be using the logging module rather than print > statements". I'm in agreement with all this. Thanks for the correction. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Dec 11 11:51:53 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 11 Dec 2010 20:51:53 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On Sat, Dec 11, 2010 at 7:28 PM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > Actually if we're to change things to print INFO to stdout and WARNING+ to > stderr, I suspect our messages crossed paths in mid-stream, but I actually realised I was wrong on that front - the behaviour already provided by logging.basicConfig is the better way to go. The missing piece in the story of "how do I create X kind of output" is plain old vanilla print statements and similar direct writes to sys.stdout. The logging module is a tool to track what is happening in a program, not a tool for providing a console based UI. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From contactprashantat at gmail.com Sat Dec 11 11:36:46 2010 From: contactprashantat at gmail.com (Prashant Kumar) Date: Sat, 11 Dec 2010 16:06:46 +0530 Subject: [Python-Dev] porting python.org Message-ID: I was wondering if we could contribute in porting of python.org website over to python3k. If we can, what are the steps to be taken and other necessary requirements? regards, prashant -------------- next part -------------- An HTML attachment was scrubbed... URL: From vinay_sajip at yahoo.co.uk Sat Dec 11 12:52:27 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 11 Dec 2010 11:52:27 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D031972.5090105@g.nevcal.com> <4D0344D2.3070006@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > the logger.? Does basicConfig give that control?? I don't know, the > first 8% of the logger documentation don't tell me that, they only > give me a canned example (actually 3) without explanation of its > full function.? Someday I'll read more.? Until then, basicConfig is > opaque: I know only three incantations for using it, and not even > any surety that they can be mixed, or that it can be called more > than once in the same program. You've made some helpful observations about improving the logging documentation. I'm grateful to you for those and would like to incorporate your feedback. But I feel you're making somewhat heavy weather about basicConfig() and the "8%". It's just 20 lines of code and not especially complicated code at that. It would probably take you less time to look at that code than to post again about how little information there is in the top 8% of the logging documentation, I think you've got your point across. Here are a couple of hopefully useful links about basicConfig(): >From the 2.7.1 docs - documentation gleaned from the docstring: http://docs.python.org/library/logging.html#logging.basicConfig >From the source code for 2.7.1, docstring followed by the code: http://svn.python.org/view/python/branches/release27-maint/Lib/logging/__init__.py?revision=87157&view=markup#basicConfig I will try to incorporate more basic examples at the top of the documentation, but surely you don't want me to add more verbiage about basicConfig when your overall feeling is that there's too much documentation? Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Sat Dec 11 13:06:40 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sat, 11 Dec 2010 12:06:40 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: Nick Coghlan gmail.com> writes: > the logging module is a tool to track what is happening in a program, not a > tool for providing a console based UI. That was certainly the thinking behind how it worked before my recent changes, but I completely accept that it wasn't helpful in the concurrent.futures scenario, and agree with the comments you made around this in your earlier posts. However, what Antoine has said on this thread (and others have concurred) leads me to believe that he wants to use it not purely as a tracking tool (as I've termed it, "an adjunct to doing real work") but also as a tool to provide program output (e.g. doing some of the program's real work, say by calling error() to print an error message). Regards, Vinay Sajip From solipsis at pitrou.net Sat Dec 11 15:17:03 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 11 Dec 2010 15:17:03 +0100 Subject: [Python-Dev] API for the new sysconfig module References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> Message-ID: <20101211151703.09c8675e@pitrou.net> On Sat, 11 Dec 2010 12:55:25 +1000 Nick Coghlan wrote: > On Sat, Dec 11, 2010 at 12:44 PM, Terry Reedy wrote: > > On 12/10/2010 4:59 PM, R. David Murray wrote: > > > >> Like ?ric, I'm not sure what the implications of the existing module > >> having been released in 2.7 and 3.2 beta are in terms of making such an > >> API change. > > > > I am with Raymond on this: the purpose of betas is so we can test *and* make > > changes. No one should base production software on a beta release. > > It's also the key difference between betas and release candidates. Seems that both you and Terry have missed the part where the sysconfig API is already part of 2.7, though. Regards Antoine. From tnagyemail-mail at yahoo.fr Sat Dec 11 15:44:35 2010 From: tnagyemail-mail at yahoo.fr (Thomas Nagy) Date: Sat, 11 Dec 2010 14:44:35 +0000 (GMT) Subject: [Python-Dev] futures API In-Reply-To: Message-ID: <608275.24121.qm@web27507.mail.ukl.yahoo.com> --- El vie, 10/12/10, Brian Quinlan escribi?: > On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote: > > --- El vie, 10/12/10, Brian Quinlan escribi?: > >> On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: > >>> I have a process running for a long time, and > which > >> may use futures of different max_workers count. I > think it > >> is not too far-fetched to create a new futures > object each > >> time. Yet, the execution becomes slower after each > call, for > >> example with http://freehackers.org/~tnagy/futures_test.py: > >>> > >>> """ > >>> import concurrent.futures > >>> from queue import Queue > >>> import datetime > >>> > >>> class counter(object): > >>>? ? ? def __init__(self, fut): > >>>? ? ? ? ? self.fut = > fut > >>> > >>>? ? ? def run(self): > >>>? ? ? ? ? def > >> look_busy(num, obj): > >>> > >>? ? tot = 0 > >>> > >>? ? for x in range(num): > >>> > >>? ? tot += x > >>> > >>? ? obj.out_q.put(tot) > >>> > >>>? ? ? ? ? start = > >> datetime.datetime.utcnow() > >>>? ? ? ? ? self.count = > 0 > >>>? ? ? ? ? self.out_q > = > >> Queue(0) > >>>? ? ? ? ? for x in > >> range(1000): > >>> > >>? ? self.count += 1 > >>> > >>? ? self.fut.submit(look_busy, > self.count, > >> self) > >>> > >>>? ? ? ? ? while > >> self.count: > >>> > >>? ? self.count -= 1 > >>> > >>? ? self.out_q.get() > >>> > >>>? ? ? ? ? delta = > >> datetime.datetime.utcnow() - start > >>> > >>? ? print(delta.total_seconds()) > >>> > >>> fut = > >> > concurrent.futures.ThreadPoolExecutor(max_workers=20) > >>> for x in range(100): > >>>? ? ? # comment the following > line > >>>? ? ? fut = > >> > concurrent.futures.ThreadPoolExecutor(max_workers=20) > >>>? ? ? c = counter(fut) > >>>? ? ? c.run() > >>> """ > >>> > >>> The runtime grows after each step: > >>> 0.216451 > >>> 0.225186 > >>> 0.223725 > >>> 0.222274 > >>> 0.230964 > >>> 0.240531 > >>> 0.24137 > >>> 0.252393 > >>> 0.249948 > >>> 0.257153 > >>> ... > >>> > >>> Is there a mistake in this piece of code? > >> > >> There is no mistake that I can see but I suspect > that the > >> circular references that you are building are > causing the > >> ThreadPoolExecutor to take a long time to be > collected. Try > >> adding: > >> > >>? ???c = counter(fut) > >>? ???c.run() > >> +? ? fut.shutdown() > >> > >> Even if that fixes your problem, I still don't > fully > >> understand this because I would expect the runtime > to fall > >> after a while as ThreadPoolExecutors are > collected. > > > > The shutdown call is indeed a good fix :-) Here is the > time response? > > of the calls to counter() when shutdown is not > called: > > http://www.freehackers.org/~tnagy/runtime_futures.png > > FWIW, I think that you are confusion the term "future" > with? > "executor". A future represents a single work item. An > executor? > creates futures and schedules their underlying work. Ah yes, sorry. I have also realized that the executor is not the killer feature I was expecting, it can only replace a little part of the code I have: controlling the exceptions and the workflow is the most complicated part. I have also observed a minor performance degradation with the executor replacement (3 seconds for 5000 work items). The amount of work items processed by unit of time does not seem to be a straight line: http://www.freehackers.org/~tnagy/runtime_futures_2.png . Out of curiosity, what is the "_thread_references" for? The source file for the example is in: http://www.freehackers.org/~tnagy/futures_test3.py The diagram was created by: http://www.freehackers.org/~tnagy/futures_test3.plot Thomas From scott+python-dev at scottdial.com Sat Dec 11 17:17:30 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sat, 11 Dec 2010 11:17:30 -0500 Subject: [Python-Dev] futures API In-Reply-To: <608275.24121.qm@web27507.mail.ukl.yahoo.com> References: <608275.24121.qm@web27507.mail.ukl.yahoo.com> Message-ID: <4D03A41A.5090302@scottdial.com> On 12/11/2010 9:44 AM, Thomas Nagy wrote: > The amount of work items processed by unit of time does not seem to be a straight line: http://www.freehackers.org/~tnagy/runtime_futures_2.png . Out of curiosity, what is the "_thread_references" for? > > The source file for the example is in: > http://www.freehackers.org/~tnagy/futures_test3.py > > The diagram was created by: > http://www.freehackers.org/~tnagy/futures_test3.plot You're test code does 50,000,000 of list appends. I suspect your benchmark is telling you more about the behavior of large lists than the overhead of the futures module. You should retry that experiment with the list pre-allocated. Beyond that, the curve in that line is not exactly a large amount of variance from a straight line. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From ncoghlan at gmail.com Sat Dec 11 18:18:43 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Dec 2010 03:18:43 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On Sat, Dec 11, 2010 at 10:06 PM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> the logging module is a tool to track what is happening in a program, not a >> tool for providing a console based UI. > > That was certainly the thinking behind how it worked before my recent changes, > but I completely accept that it wasn't helpful in the concurrent.futures > scenario, and agree with the comments you made around this in your earlier > posts. However, what Antoine has said on this thread (and others have concurred) > leads me to believe that he wants to use it not purely as a tracking tool (as > I've termed it, "an adjunct to doing real work") but also as a tool to provide > program output (e.g. doing some of the program's real work, say by calling > error() to print an error message). I think that's actually just a case where the lines get a little blurry. From logging's point of view, it is still being used to say "Hey, this particular event happened during program execution". We're just adjusting our display policy to say that there are certain kinds of events that should be made visible to the user by default, but piping them into the logging machinery is preferable to writing them directly to sys,stderr. By contrast, most information that is written to sys.stdout isn't really reporting events that happened in the program - it is actual meaningful data that is related to the task the program accomplishes. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From g.brandl at gmx.net Sat Dec 11 18:19:59 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 11 Dec 2010 18:19:59 +0100 Subject: [Python-Dev] porting python.org In-Reply-To: References: Message-ID: Am 11.12.2010 11:36, schrieb Prashant Kumar: > I was wondering if we could contribute in porting of python.org > website over to python3k. If we can, what are the steps to > be taken and other necessary requirements? The SVN repository for python.org is currently not publicly available, but you can get access asking on the pydotorg-www at python.org mailing list. An excerpt from the README: External packages required: Mako (http://www.makotemplates.org/) [Note that Mako requires setuptools to install out-of-the-box. You can also just copy the lib/mako/ tree into your Python's site-packages directory if you prefer not to install setuptools.] PyYAML (http://pyyaml.org/wiki/PyYAML) Docutils (http://docutils.sf.net) So the first step would be to determine whether Mako and PyYAML are ported yet, and if not, help porting them. Docutils is ported already. cheers, Georg From ncoghlan at gmail.com Sat Dec 11 18:21:43 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Dec 2010 03:21:43 +1000 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <20101211151703.09c8675e@pitrou.net> References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> <20101211151703.09c8675e@pitrou.net> Message-ID: On Sun, Dec 12, 2010 at 12:17 AM, Antoine Pitrou wrote: > On Sat, 11 Dec 2010 12:55:25 +1000 > Nick Coghlan wrote: > >> On Sat, Dec 11, 2010 at 12:44 PM, Terry Reedy wrote: >> > On 12/10/2010 4:59 PM, R. David Murray wrote: >> > >> >> Like ?ric, I'm not sure what the implications of the existing module >> >> having been released in 2.7 and 3.2 beta are in terms of making such an >> >> API change. >> > >> > I am with Raymond on this: the purpose of betas is so we can test *and* make >> > changes. No one should base production software on a beta release. >> >> It's also the key difference between betas and release candidates. > > Seems that both you and Terry have missed the part where the sysconfig > API is already part of 2.7, though. I had missed that, yes*. I suspect Raymond may have missed it as well. Cheers, Nick. *(I sometimes lose track of which changes were made in both branches pre-2.7, which ones were mode post-2.7 release, and which ones went in pre-2.7, but were 3.x only regardless) -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tseaver at palladion.com Sat Dec 11 18:24:22 2010 From: tseaver at palladion.com (Tres Seaver) Date: Sat, 11 Dec 2010 12:24:22 -0500 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/11/2010 04:28 AM, Vinay Sajip wrote: > Actually if we're to change things to print INFO to stdout and WARNING+ to > stderr, ACK! That would be an awful default -- stdout belongs to the application, not to "meta" stuff like logging, which should only go to stderr. Much better to document the "how-to" for setting up something like that than to make it a default. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0Ds8YACgkQ+gerLs4ltQ6BBACfTWWcfl6000DX9SWCYvd5Tf7n qIcAniTD6EHRhrXA+eVpgyrlqvv12apq =oR03 -----END PGP SIGNATURE----- From sandro.tosi at gmail.com Sat Dec 11 18:37:14 2010 From: sandro.tosi at gmail.com (Sandro Tosi) Date: Sat, 11 Dec 2010 18:37:14 +0100 Subject: [Python-Dev] porting python.org In-Reply-To: References: Message-ID: Hi, On Sat, Dec 11, 2010 at 18:19, Georg Brandl wrote: > ?Mako (http://www.makotemplates.org/) > ?[Note that Mako requires setuptools to install out-of-the-box. ?You can > ? also just copy the lib/mako/ tree into your Python's site-packages > ? directory if you prefer not to install setuptools.] Mako is compatible with py3k (via 2to3) since version 0.3 (but later versions has a better support). [1] explains how to have a py3k installation of mako. [1] http://www.makotemplates.org/trac/browser/README.py3k > ?PyYAML (http://pyyaml.org/wiki/PyYAML) "PyYAML also supports Python 3" from the homepage, the version introducing py3k support is 3.08 . Just my 2c, -- Sandro Tosi (aka morph, morpheus, matrixhasu) My website: http://matrixhasu.altervista.org/ Me at Debian: http://wiki.debian.org/SandroTosi From ncoghlan at gmail.com Sat Dec 11 18:51:06 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Dec 2010 03:51:06 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On Sun, Dec 12, 2010 at 3:24 AM, Tres Seaver wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On 12/11/2010 04:28 AM, Vinay Sajip wrote: > >> Actually if we're to change things to print INFO to stdout and WARNING+ to >> stderr, > > ACK! ?That would be an awful default -- stdout belongs to the > application, not to "meta" stuff like logging, which should only go to > stderr. ?Much better to document the "how-to" for setting up something > like that than to make it a default. Don't worry, that isn't going to happen. Vinay was just responding to something he thought I was suggesting. I wasn't really, but I can understand someone getting that impression (as I was using that behaviour as my running example of "so just what *is* involved if an applications wants to configure logging like that?"). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From v+python at g.nevcal.com Sat Dec 11 20:17:05 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 11 Dec 2010 11:17:05 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D031972.5090105@g.nevcal.com> <4D0344D2.3070006@g.nevcal.com> Message-ID: <4D03CE31.3040700@g.nevcal.com> On 12/11/2010 3:52 AM, Vinay Sajip wrote: > I will try to incorporate more basic examples at the top of the documentation, > but surely you don't want me to add more verbiage about basicConfig when your > overall feeling is that there's too much documentation? I try not to post unless I feel there is a new detail; yes the 8% is probably repetitive, but at present, going further, at least sequentially, does feel like a large mountain to climb... suddenly becoming significantly steeper there. I really can't say much about the 92% of the documentation, because of the mountain. Whether the existing documentation can be reordered to avoid the steep part, but rather provide a gentle upward path, I cannot say. If it is properly organized reference material, then probably adding more about something at the beginning is appropriate, to give the foundation, justification, and assistance for climbing the slope. My overall feeling is not necessarily that there is too much documentation (there is a lot, but maybe that is a good thing, I should read it some week) but that it suffers in not providing useful information for the beginner at the front. I've listed what I've gleaned here and tried to point out the things that I still don't know, and tried to point out where things became less than obvious along the way. I do think it was primarily because the intro material is too brief and too disconnected, that I quit reading at the 8% mark (or so). If the rest of the large document were to be similarly disconnected, it wouldn't be worth my time to read it... and if I had to read it all, just to understand enough to get started with simple logging, then it would be faster to write my own, which is the path I took. I suffer with my own being primitive, but I can tweak it when I need to. I still don't fully understand the nuances between handlers and filters, and likely won't until I read more of the documentation. Jumping directly to the documentation (at the 28% mark, I'd have had to read a long way to find it) for basicConfig per your links helps, but sadly, because the first paragraph is full of the terms "logger", "handler", and "formatter", which I have only rudimentary ideas about what they do, being a beginner, I almost gave up again. But the description of the valid keywords for **kwargs was actually understandable, at the end of that documentation. I also learned that calling basicConfig more than once in a program is probably useless, and it should be nearly the first logger API called, if it is used. The first paragraph should be split, perhaps: "Does basic configuration for the logging system." is almost redundant with the name, but is OK if first. The rest of that paragraph, and the next three, should probably be moved after the list of kwargs keys and descriptions. Yes, the word handler is used a bit, but it is associated there with concepts like file names and stream handles, and that eases the uncertainty; together with the simple example at the front, and the verbage there, I can see that I specify a file name, and it makes the logger log to that file :) And the level parameter is fine. I'm less sure about format and datefmt and why I would want to specify them or how, and am surprised to find that format is to be used by a "handler" not a "formatter", but the simple example shows me I can leave those out for starters. Personally, I suffer in writing documentation; when I attempt to do so, and people read it, they usually tell me that after about the third read, they finally understand it, but sadly it takes them three reads. It is interesting to be looking at the logger from the other side... maybe I'll be able to improve the documentation I write, after analyzing why the logger documentation was hard for me to approach. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Dec 11 20:21:41 2010 From: brett at python.org (Brett Cannon) Date: Sat, 11 Dec 2010 11:21:41 -0800 Subject: [Python-Dev] Format factories (was Re: sWAPcASE Was: transform() and untransform() methods, and the codec registry) In-Reply-To: References: Message-ID: On Thu, Dec 9, 2010 at 16:26, Nick Coghlan wrote: > On Fri, Dec 10, 2010 at 9:29 AM, Antoine Pitrou wrote: >> On Thu, 09 Dec 2010 18:10:38 -0500 >> Eric Smith wrote: >>> If we're looking to reduce the number of methods on str, I wouldn't mind >>> seeing center() and zfill() also go away, since they're trivially >>> replaced by format(). >> >> Well, it's only trivial when you know format()'s wicked mini-language by >> heart :/ ?center() is easy and obvious. zfill() is arguably a bit too >> specialized. > > I've occasionally wondered if string formatting [1] and the struct > module [2] would benefit from format building functions that made them > easier to use without necessarily learning the cryptic mini-languages > off by heart. > > For example, a "string.make_format_spec" function might have a signature like: > > def make_format_spec(fill=None, align=None, sign=None, width=None, > precision=None, display_type='s', alternate=False, commas=False, > numeric_case=None) > > "align" would accept not only the actual format symbols ('<', '>', > '=', '^'), but also the corresponding names ('left', 'right', > 'numeric', 'center'). > > "numeric_case" would accept None, 'upper' or 'lower' (affecting the > formatting of hex and floating point values). If the stated numeric > case differs from that specified by the display type, raise a > ValueError. For an unspecified numeric case, the affected display > types would default to 'lower'. > > Similarly, "display_type" would accept long names in addition to the > character codes: > > 's': 'str' > 'b': 'binary' > 'c': 'chr' > 'd': 'int' > 'o': 'octal' > 'x', 'X': 'hex' (numeric case controls display of digits A-F) > 'n': 'locale' > 'e', 'E': 'exponent' (numeric case controls display of exponent as > well as infinite and NaN values) > 'f', 'F': 'float' (numeric case controls display of exponent as well > as infinite and NaN values) > 'g', 'G': 'general' (numeric case controls display of exponent as well > as infinite and NaN values) > '%': 'percent' (numeric case controls display of exponent as well as > infinite and NaN values) > > There could also be a corresponding parse_format_spec that produced a > named tuple with the appropriate details. But is this worth it since once you write it you won't be changing it again, suggesting that taking the time to look up the formatting rules isn't that much harder and wouldn't burden us w/ writing such functions and trying to come up with a good API. I suspect what would be more helpful is either have a rather detailed docstring for str.format or to at least put the URL to the docs which explains the mini-language to make it easier to find. -Brett > > Cheers, > Nick. > > [1] http://docs.python.org/dev/library/string#format-specification-mini-language > [2] http://docs.python.org/dev/library/struct#format-strings > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From brett at python.org Sat Dec 11 20:32:55 2010 From: brett at python.org (Brett Cannon) Date: Sat, 11 Dec 2010 11:32:55 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On Fri, Dec 10, 2010 at 22:21, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > > >> This could actually make a reasonably good basic for a "task oriented" >> subsection of the logging documentation. Something like: >> > > Good suggestion, I'll see what I can do. Just wanted to +1 on some task-oriented (or at least simple intro) docs going into the logging module. The length of the docs is rather daunting I always have a hard time getting started (the first example is for file logging and the second one does log rotation when all I want is printing to stderr!). All I want is an opener saying, "here are the functions to call to send stuff to the log; the defaults for each go here and here (e.g., stderr, /dev/null), and if you want to change those settings, here is how to e.g., get info() to go to stderr. Read on for more if you need something fancier (e.g., writing to a file)". That and making it not an error when I call logging.error() w/o setting an explicit logger would go a long way to making the logging module much more accessible IMO. From jyasskin at gmail.com Sat Dec 11 21:40:16 2010 From: jyasskin at gmail.com (Jeffrey Yasskin) Date: Sat, 11 Dec 2010 12:40:16 -0800 Subject: [Python-Dev] Format factories (was Re: sWAPcASE Was: transform() and untransform() methods, and the codec registry) In-Reply-To: References: Message-ID: On Sat, Dec 11, 2010 at 11:21 AM, Brett Cannon wrote: > On Thu, Dec 9, 2010 at 16:26, Nick Coghlan wrote: >> On Fri, Dec 10, 2010 at 9:29 AM, Antoine Pitrou wrote: >>> On Thu, 09 Dec 2010 18:10:38 -0500 >>> Eric Smith wrote: >>>> If we're looking to reduce the number of methods on str, I wouldn't mind >>>> seeing center() and zfill() also go away, since they're trivially >>>> replaced by format(). >>> >>> Well, it's only trivial when you know format()'s wicked mini-language by >>> heart :/ ?center() is easy and obvious. zfill() is arguably a bit too >>> specialized. >> >> I've occasionally wondered if string formatting [1] and the struct >> module [2] would benefit from format building functions that made them >> easier to use without necessarily learning the cryptic mini-languages >> off by heart. >> >> For example, a "string.make_format_spec" function might have a signature like: >> >> def make_format_spec(fill=None, align=None, sign=None, width=None, >> precision=None, display_type='s', alternate=False, commas=False, >> numeric_case=None) >> >> "align" would accept not only the actual format symbols ('<', '>', >> '=', '^'), but also the corresponding names ('left', 'right', >> 'numeric', 'center'). >> >> "numeric_case" would accept None, 'upper' or 'lower' (affecting the >> formatting of hex and floating point values). If the stated numeric >> case differs from that specified by the display type, raise a >> ValueError. For an unspecified numeric case, the affected display >> types would default to 'lower'. >> >> Similarly, "display_type" would accept long names in addition to the >> character codes: >> >> 's': 'str' >> 'b': 'binary' >> 'c': 'chr' >> 'd': 'int' >> 'o': 'octal' >> 'x', 'X': 'hex' (numeric case controls display of digits A-F) >> 'n': 'locale' >> 'e', 'E': 'exponent' (numeric case controls display of exponent as >> well as infinite and NaN values) >> 'f', 'F': 'float' (numeric case controls display of exponent as well >> as infinite and NaN values) >> 'g', 'G': 'general' (numeric case controls display of exponent as well >> as infinite and NaN values) >> '%': 'percent' (numeric case controls display of exponent as well as >> infinite and NaN values) >> >> There could also be a corresponding parse_format_spec that produced a >> named tuple with the appropriate details. > > But is this worth it since once you write it you won't be changing it > again, Having a make_format_spec() would also help people trying to read the code. > suggesting that taking the time to look up the formatting rules > isn't that much harder and wouldn't burden us w/ writing such > functions and trying to come up with a good API. I suspect what would > be more helpful is either have a rather detailed docstring for > str.format or to at least put the URL to the docs which explains the > mini-language to make it easier to find. >> >> Cheers, >> Nick. >> >> [1] http://docs.python.org/dev/library/string#format-specification-mini-language >> [2] http://docs.python.org/dev/library/struct#format-strings >> >> -- >> Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jyasskin%40gmail.com > From brian at sweetapp.com Sat Dec 11 21:53:15 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sat, 11 Dec 2010 12:53:15 -0800 Subject: [Python-Dev] futures API In-Reply-To: <608275.24121.qm@web27507.mail.ukl.yahoo.com> References: <608275.24121.qm@web27507.mail.ukl.yahoo.com> Message-ID: On Dec 11, 2010, at 6:44 AM, Thomas Nagy wrote: > --- El vie, 10/12/10, Brian Quinlan escribi?: >> On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote: >>> --- El vie, 10/12/10, Brian Quinlan escribi?: >>>> On Dec 10, 2010, at 5:36 AM, Thomas Nagy wrote: >>>>> I have a process running for a long time, and >> which >>>> may use futures of different max_workers count. I >> think it >>>> is not too far-fetched to create a new futures >> object each >>>> time. Yet, the execution becomes slower after each >> call, for >>>> example with http://freehackers.org/~tnagy/futures_test.py: >>>>> >>>>> """ >>>>> import concurrent.futures >>>>> from queue import Queue >>>>> import datetime >>>>> >>>>> class counter(object): >>>>> def __init__(self, fut): >>>>> self.fut = >> fut >>>>> >>>>> def run(self): >>>>> def >>>> look_busy(num, obj): >>>>> >>>> tot = 0 >>>>> >>>> for x in range(num): >>>>> >>>> tot += x >>>>> >>>> obj.out_q.put(tot) >>>>> >>>>> start = >>>> datetime.datetime.utcnow() >>>>> self.count = >> 0 >>>>> self.out_q >> = >>>> Queue(0) >>>>> for x in >>>> range(1000): >>>>> >>>> self.count += 1 >>>>> >>>> self.fut.submit(look_busy, >> self.count, >>>> self) >>>>> >>>>> while >>>> self.count: >>>>> >>>> self.count -= 1 >>>>> >>>> self.out_q.get() >>>>> >>>>> delta = >>>> datetime.datetime.utcnow() - start >>>>> >>>> print(delta.total_seconds()) >>>>> >>>>> fut = >>>> >> concurrent.futures.ThreadPoolExecutor(max_workers=20) >>>>> for x in range(100): >>>>> # comment the following >> line >>>>> fut = >>>> >> concurrent.futures.ThreadPoolExecutor(max_workers=20) >>>>> c = counter(fut) >>>>> c.run() >>>>> """ >>>>> >>>>> The runtime grows after each step: >>>>> 0.216451 >>>>> 0.225186 >>>>> 0.223725 >>>>> 0.222274 >>>>> 0.230964 >>>>> 0.240531 >>>>> 0.24137 >>>>> 0.252393 >>>>> 0.249948 >>>>> 0.257153 >>>>> ... >>>>> >>>>> Is there a mistake in this piece of code? >>>> >>>> There is no mistake that I can see but I suspect >> that the >>>> circular references that you are building are >> causing the >>>> ThreadPoolExecutor to take a long time to be >> collected. Try >>>> adding: >>>> >>>> c = counter(fut) >>>> c.run() >>>> + fut.shutdown() >>>> >>>> Even if that fixes your problem, I still don't >> fully >>>> understand this because I would expect the runtime >> to fall >>>> after a while as ThreadPoolExecutors are >> collected. >>> >>> The shutdown call is indeed a good fix :-) Here is the >> time response >>> of the calls to counter() when shutdown is not >> called: >>> http://www.freehackers.org/~tnagy/runtime_futures.png >> >> FWIW, I think that you are confusion the term "future" >> with >> "executor". A future represents a single work item. An >> executor >> creates futures and schedules their underlying work. > > Ah yes, sorry. I have also realized that the executor is not the > killer feature I was expecting, it can only replace a little part of > the code I have: controlling the exceptions and the workflow is the > most complicated part. > > I have also observed a minor performance degradation with the > executor replacement (3 seconds for 5000 work items). The amount of > work items processed by unit of time does not seem to be a straight > line: http://www.freehackers.org/~tnagy/runtime_futures_2.png . That looks pretty linear to me. > Out of curiosity, what is the "_thread_references" for? There is a big comment above it in the code: # Workers are created as daemon threads. This is done to allow the interpreter # to exit when there are still idle threads in a ThreadPoolExecutor's thread # pool (i.e. shutdown() was not called). However, allowing workers to die with # the interpreter has two undesirable properties: # - The workers would still be running during interpretor shutdown, # meaning that they would fail in unpredictable ways. # - The workers could be killed while evaluating a work item, which could # be bad if the callable being evaluated has external side-effects e.g. # writing to a file. # # To work around this problem, an exit handler is installed which tells the # workers to exit when their work queues are empty and then waits until the # threads finish. _thread_references = set() _shutdown = False def _python_exit(): global _shutdown _shutdown = True for thread_reference in _thread_references: thread = thread_reference() if thread is not None: thread.join() Is it still unclear why it is there? Maybe you could propose some additional documentation. Cheers, Brian > The source file for the example is in: > http://www.freehackers.org/~tnagy/futures_test3.py > > The diagram was created by: > http://www.freehackers.org/~tnagy/futures_test3.plot > > Thomas > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From ben+python at benfinney.id.au Sat Dec 11 22:15:11 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 12 Dec 2010 08:15:11 +1100 Subject: [Python-Dev] Status of PEP 3143? References: <87oc8wyng9.fsf@benfinney.id.au> Message-ID: <877hfgypsw.fsf@benfinney.id.au> Benjamin Peterson writes: > 2010/12/8 Ben Finney : > > So it's not abandoned, but I don't know which version should be the > > current target. What change should I make to the PEP in such a case? > > Put 3.3 or 3.x if you're thinking really long term. :) Done now (, revision 87170). -- \ ?We must respect the other fellow's religion, but only in the | `\ sense and to the extent that we respect his theory that his | _o__) wife is beautiful and his children smart.? ?Henry L. Mencken | Ben Finney From raymond.hettinger at gmail.com Sat Dec 11 22:18:49 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 11 Dec 2010 13:18:49 -0800 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> <20101211151703.09c8675e@pitrou.net> Message-ID: <037003C3-70F1-4430-8CA4-55C122F40C86@gmail.com> On Dec 11, 2010, at 9:21 AM, Nick Coghlan wrote: > On Sun, Dec 12, 2010 at 12:17 AM, Antoine Pitrou wrote: >> On Sat, 11 Dec 2010 12:55:25 +1000 >> Nick Coghlan wrote: >> >>> On Sat, Dec 11, 2010 at 12:44 PM, Terry Reedy wrote: >>>> On 12/10/2010 4:59 PM, R. David Murray wrote: >>>> >>>>> Like ?ric, I'm not sure what the implications of the existing module >>>>> having been released in 2.7 and 3.2 beta are in terms of making such an >>>>> API change. >>>> >>>> I am with Raymond on this: the purpose of betas is so we can test *and* make >>>> changes. No one should base production software on a beta release. >>> >>> It's also the key difference between betas and release candidates. >> >> Seems that both you and Terry have missed the part where the sysconfig >> API is already part of 2.7, though. > > I had missed that, yes*. I suspect Raymond may have missed it as well. > > Cheers, > Nick. > > *(I sometimes lose track of which changes were made in both branches > pre-2.7, which ones were mode post-2.7 release, and which ones went in > pre-2.7, but were 3.x only regardless) Right. I missed that it was already in 2.7. So, now we're stuck with it, forever. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Dec 11 22:28:25 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 11 Dec 2010 16:28:25 -0500 Subject: [Python-Dev] porting python.org In-Reply-To: References: Message-ID: On 12/11/2010 12:19 PM, Georg Brandl wrote: > Am 11.12.2010 11:36, schrieb Prashant Kumar: >> I was wondering if we could contribute in porting of python.org >> website over to python3k. I think this is an excellent idea. It will test Python3 and the modules and external packages used and once successful, advertise that all such are production-ready Py3 software. If and when there is a test version I can access (py3.python.org?), I will happily help test by browsing around and downloading stuff. -- Terry Jan Reedy From ben+python at benfinney.id.au Sat Dec 11 22:47:51 2010 From: ben+python at benfinney.id.au (Ben Finney) Date: Sun, 12 Dec 2010 08:47:51 +1100 Subject: [Python-Dev] Status of PEP 3143? References: <87oc8wyng9.fsf@benfinney.id.au> <877hfgypsw.fsf@benfinney.id.au> Message-ID: <871v5oyoag.fsf@benfinney.id.au> Ben Finney writes: > Done now (, revision > 87170). Thanks to Georg Brandl for applying the change on my behalf. -- \ ?As the most participatory form of mass speech yet developed, | `\ the Internet deserves the highest protection from governmental | _o__) intrusion.? ?U.S. District Court Judge Dalzell | Ben Finney From g.brandl at gmx.net Sat Dec 11 23:15:58 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 11 Dec 2010 23:15:58 +0100 Subject: [Python-Dev] porting python.org In-Reply-To: References: Message-ID: Am 11.12.2010 22:28, schrieb Terry Reedy: > On 12/11/2010 12:19 PM, Georg Brandl wrote: >> Am 11.12.2010 11:36, schrieb Prashant Kumar: >>> I was wondering if we could contribute in porting of python.org >>> website over to python3k. > > I think this is an excellent idea. It will test Python3 and the modules > and external packages used and once successful, advertise that all such > are production-ready Py3 software. If and when there is a test version I > can access (py3.python.org?), I will happily help test by browsing > around and downloading stuff. I don't think a separate test site is useful. Either the build process works with Python 3 or it doesn't -- in the latter case you'll just get an exception while running "make". Georg From steve at pearwood.info Sun Dec 12 02:22:43 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 12 Dec 2010 12:22:43 +1100 Subject: [Python-Dev] futures API In-Reply-To: References: <608275.24121.qm@web27507.mail.ukl.yahoo.com> Message-ID: <4D0423E3.4020805@pearwood.info> Brian Quinlan wrote: > > On Dec 11, 2010, at 6:44 AM, Thomas Nagy wrote: >> I have also observed a minor performance degradation with the executor >> replacement (3 seconds for 5000 work items). The amount of work items >> processed by unit of time does not seem to be a straight line: >> http://www.freehackers.org/~tnagy/runtime_futures_2.png . > > That looks pretty linear to me. Close to, but not quite. The graph seems to be slightly curved, with the amount of work done per second decreasing for large amounts of work. Assuming that this performance degradation is real, and not an artifact of the measurement technique, it seems to be quite small. I'd be happy to describe it as "linear" in the same way we describe dictionary lookups as constant-time, even though technically that's not strictly true. (They're linear in the number of items with a matching hash, and there are probably other complications as well.) As drawn, the curve seems to fall away like a log graph, which might suggest to the casual viewer that this is a good thing. It may be better to reverse the axes, that is to have the independent variable, number of tasks, on the horizontal axis, and the dependent variable, cost (time taken), vertically. This will make it clear that the incremental cost of doing one extra task increases (slightly) as the number of tasks goes up. -- Steven From ncoghlan at gmail.com Sun Dec 12 03:30:37 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Dec 2010 12:30:37 +1000 Subject: [Python-Dev] Format factories (was Re: sWAPcASE Was: transform() and untransform() methods, and the codec registry) In-Reply-To: References: Message-ID: On Sun, Dec 12, 2010 at 5:21 AM, Brett Cannon wrote: > But is this worth it since once you write it you won't be changing it > again, suggesting that taking the time to look up the formatting rules > isn't that much harder and wouldn't burden us w/ writing such > functions and trying to come up with a good API. I suspect what would > be more helpful is either have a rather detailed docstring for > str.format or to at least put the URL to the docs which explains the > mini-language to make it easier to find. Yes, it may be worth it, since it greatly simplifies things like generating a format spec programmatically rather than hardcoding it. You can also run it once at the interactive prompt to figure out the format string you want to hard code, then include the API call as a comment for concise documentation of what your format string does. I helped *define* the new format spec and I still need to reference the docs to create trickier ones. A factory function would be far more convenient than burdening str.format's docstring with a full definition of the format spec syntax. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sun Dec 12 03:33:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Dec 2010 12:33:34 +1000 Subject: [Python-Dev] futures API In-Reply-To: References: <608275.24121.qm@web27507.mail.ukl.yahoo.com> Message-ID: On Sun, Dec 12, 2010 at 6:53 AM, Brian Quinlan wrote: > Is it still unclear why it is there? Maybe you could propose some additional > documentation. Did you get my question the other day as to whether a weakref.WeakKeySet might be a better choice? I believe you would be able to get rid of the periodic sweep for dead references if you did that, and I didn't spot any obvious downsides. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tnagyemail-mail at yahoo.fr Sun Dec 12 03:32:18 2010 From: tnagyemail-mail at yahoo.fr (Thomas Nagy) Date: Sun, 12 Dec 2010 02:32:18 +0000 (GMT) Subject: [Python-Dev] futures API Message-ID: <229168.97019.qm@web27508.mail.ukl.yahoo.com> --- El s?b, 11/12/10, Brian Quinlan escribi?: > > On Dec 11, 2010, at 6:44 AM, Thomas Nagy wrote: > > > --- El vie, 10/12/10, Brian Quinlan escribi?: > >> On Dec 10, 2010, at 10:51 AM, Thomas Nagy wrote: > >>> --- El vie, 10/12/10, Brian Quinlan > escribi?: > >>>> On Dec 10, 2010, at 5:36 AM, Thomas Nagy > wrote: > >>>>> I have a process running for a long > time, and > >> which > >>>> may use futures of different max_workers > count. I > >> think it > >>>> is not too far-fetched to create a new > futures > >> object each > >>>> time. Yet, the execution becomes slower > after each > >> call, for > >>>> example with http://freehackers.org/~tnagy/futures_test.py: > >>>>> > >>>>> """ > >>>>> import concurrent.futures > >>>>> from queue import Queue > >>>>> import datetime > >>>>> > >>>>> class counter(object): > >>>>>? ? ???def > __init__(self, fut): > >>>>>? ? ? ? > ???self.fut = > >> fut > >>>>> > >>>>>? ? ???def > run(self): > >>>>>? ? ? ? > ???def > >>>> look_busy(num, obj): > >>>>> > >>>>? ???tot = 0 > >>>>> > >>>>? ???for x in > range(num): > >>>>> > >>>>? ???tot += x > >>>>> > >>>>? > ???obj.out_q.put(tot) > >>>>> > >>>>>? ? ? ? > ???start = > >>>> datetime.datetime.utcnow() > >>>>>? ? ? ? > ???self.count = > >> 0 > >>>>>? ? ? ? > ???self.out_q > >> = > >>>> Queue(0) > >>>>>? ? ? ? > ???for x in > >>>> range(1000): > >>>>> > >>>>? ???self.count += 1 > >>>>> > >>>>? > ???self.fut.submit(look_busy, > >> self.count, > >>>> self) > >>>>> > >>>>>? ? ? ? > ???while > >>>> self.count: > >>>>> > >>>>? ???self.count -= 1 > >>>>> > >>>>? ???self.out_q.get() > >>>>> > >>>>>? ? ? ? > ???delta = > >>>> datetime.datetime.utcnow() - start > >>>>> > >>>>? > ???print(delta.total_seconds()) > >>>>> > >>>>> fut = > >>>> > >> > concurrent.futures.ThreadPoolExecutor(max_workers=20) > >>>>> for x in range(100): > >>>>>? ? ???# > comment the following > >> line > >>>>>? ? ???fut = > >>>> > >> > concurrent.futures.ThreadPoolExecutor(max_workers=20) > >>>>>? ? ???c = > counter(fut) > >>>>>? ? > ???c.run() > >>>>> """ > >>>>> > >>>>> The runtime grows after each step: > >>>>> 0.216451 > >>>>> 0.225186 > >>>>> 0.223725 > >>>>> 0.222274 > >>>>> 0.230964 > >>>>> 0.240531 > >>>>> 0.24137 > >>>>> 0.252393 > >>>>> 0.249948 > >>>>> 0.257153 > >>>>> ... > >>>>> > >>>>> Is there a mistake in this piece of > code? > >>>> > >>>> There is no mistake that I can see but I > suspect > >> that the > >>>> circular references that you are building > are > >> causing the > >>>> ThreadPoolExecutor to take a long time to > be > >> collected. Try > >>>> adding: > >>>> > >>>>? ? ? c = counter(fut) > >>>>? ? ? c.run() > >>>> +? ? fut.shutdown() > >>>> > >>>> Even if that fixes your problem, I still > don't > >> fully > >>>> understand this because I would expect the > runtime > >> to fall > >>>> after a while as ThreadPoolExecutors are > >> collected. > >>> > >>> The shutdown call is indeed a good fix :-) > Here is the > >> time response > >>> of the calls to counter() when shutdown is > not > >> called: > >>> http://www.freehackers.org/~tnagy/runtime_futures.png > >> > >> FWIW, I think that you are confusion the term > "future" > >> with > >> "executor". A future represents a single work > item. An > >> executor > >> creates futures and schedules their underlying > work. > > > > Ah yes, sorry. I have also realized that the executor > is not the killer feature I was expecting, it can only > replace a little part of the code I have: controlling the > exceptions and the workflow is the most complicated part. > > > > I have also observed a minor performance degradation > with the executor replacement (3 seconds for 5000 work > items). The amount of work items processed by unit of time > does not seem to be a straight line: http://www.freehackers.org/~tnagy/runtime_futures_2.png > . > > That looks pretty linear to me. Ok. > > Out of curiosity, what is the "_thread_references" > for? > > There is a big comment above it in the code: > > # Workers are created as daemon threads. This is done to > allow the interpreter > # to exit when there are still idle threads in a > ThreadPoolExecutor's thread > # pool (i.e. shutdown() was not called). However, allowing > workers to die with > # the interpreter has two undesirable properties: > #???- The workers would still be running > during interpretor shutdown, > #? ???meaning that they would fail in > unpredictable ways. > #???- The workers could be killed while > evaluating a work item, which could > #? ???be bad if the callable being > evaluated has external side-effects e.g. > #? ???writing to a file. > # > # To work around this problem, an exit handler is installed > which tells the > # workers to exit when their work queues are empty and then > waits until the > # threads finish. > > _thread_references = set() > _shutdown = False > > def _python_exit(): > ? ? global _shutdown > ? ? _shutdown = True > ? ? for thread_reference in _thread_references: > ? ? ? ? thread = thread_reference() > ? ? ? ? if thread is not None: > ? ? ? ? ? ? thread.join() > > Is it still unclear why it is there? Maybe you could > propose some additional documentation. I was thinking that if exceptions have to be caught - and it is likely to be the case in general - then this scheme is redundant. Now I see that the threads are getting their work items from a queue, so it is clear now. Thanks for all the information, Thomas From brian at sweetapp.com Sun Dec 12 03:36:47 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Sat, 11 Dec 2010 18:36:47 -0800 Subject: [Python-Dev] futures API In-Reply-To: References: <608275.24121.qm@web27507.mail.ukl.yahoo.com> Message-ID: <2FA356B1-88E6-43C7-8118-70CAD2BBB315@sweetapp.com> On Dec 11, 2010, at 6:33 PM, Nick Coghlan wrote: > On Sun, Dec 12, 2010 at 6:53 AM, Brian Quinlan > wrote: >> Is it still unclear why it is there? Maybe you could propose some >> additional >> documentation. > > Did you get my question the other day as to whether a > weakref.WeakKeySet might be a better choice? I believe you would be > able to get rid of the periodic sweep for dead references if you did > that, and I didn't spot any obvious downsides. No I didn't, sorry! Could you resent it if it has more details then the paragraph above? Cheers, Brian > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Dec 12 07:05:38 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 12 Dec 2010 16:05:38 +1000 Subject: [Python-Dev] futures API In-Reply-To: <2FA356B1-88E6-43C7-8118-70CAD2BBB315@sweetapp.com> References: <608275.24121.qm@web27507.mail.ukl.yahoo.com> <2FA356B1-88E6-43C7-8118-70CAD2BBB315@sweetapp.com> Message-ID: On Sun, Dec 12, 2010 at 12:36 PM, Brian Quinlan wrote: > > On Dec 11, 2010, at 6:33 PM, Nick Coghlan wrote: > >> On Sun, Dec 12, 2010 at 6:53 AM, Brian Quinlan wrote: >>> >>> Is it still unclear why it is there? Maybe you could propose some >>> additional >>> documentation. >> >> Did you get my question the other day as to whether a >> weakref.WeakKeySet might be a better choice? I believe you would be >> able to get rid of the periodic sweep for dead references if you did >> that, and I didn't spot any obvious downsides. > > No I didn't, sorry! Could you resent it if it has more details then the > paragraph above? There wasn't much more detail, as there actually isn't a great deal to the idea. This was the main comment in the previous email: "It seems to me that using WeakSet would mean you could get rid of _remove_dead_thread_references altogether (the implicit callbacks would handle that part), and then a set() call in _python_exit would give you concrete references to work with for cleanup purposes." Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From lukasz at langa.pl Sun Dec 12 13:01:42 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Sun, 12 Dec 2010 13:01:42 +0100 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <037003C3-70F1-4430-8CA4-55C122F40C86@gmail.com> References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> <20101211151703.09c8675e@pitrou.net> <037003C3-70F1-4430-8CA4-55C122F40C86@gmail.com> Message-ID: <27F9BCAE-74C1-4863-85A4-1E411E1AC06A@langa.pl> Wiadomo?? napisana przez Raymond Hettinger w dniu 2010-12-11, o godz. 22:18: >> *(I sometimes lose track of which changes were made in both branches >> pre-2.7, which ones were mode post-2.7 release, and which ones went in >> pre-2.7, but were 3.x only regardless) > > Right. I missed that it was already in 2.7. > So, now we're stuck with it, forever. Why not deprecate it for 3.2 (easy since it's probably not yet used anywhere anyway, even in 2.7) and introduce sys.sysconfig. I really like that much better than Java-like accessor functions. -- Best regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun Dec 12 14:42:31 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 12 Dec 2010 14:42:31 +0100 Subject: [Python-Dev] API for the new sysconfig module References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> <20101211151703.09c8675e@pitrou.net> <037003C3-70F1-4430-8CA4-55C122F40C86@gmail.com> <27F9BCAE-74C1-4863-85A4-1E411E1AC06A@langa.pl> Message-ID: <20101212144231.772efa58@pitrou.net> On Sun, 12 Dec 2010 13:01:42 +0100 ?ukasz Langa wrote: > Wiadomo?? napisana przez Raymond Hettinger w dniu 2010-12-11, o godz. 22:18: > > >> *(I sometimes lose track of which changes were made in both branches > >> pre-2.7, which ones were mode post-2.7 release, and which ones went in > >> pre-2.7, but were 3.x only regardless) > > > > Right. I missed that it was already in 2.7. > > So, now we're stuck with it, forever. > > Why not deprecate it for 3.2 (easy since it's probably not yet used anywhere anyway, even in 2.7) and introduce sys.sysconfig. We already had a lot of churn around these APIs (distutils & friends). I don't think it's a good idea to introduce even more churn. Oh and by the way it's too late to add or remove features from 3.2. > I really like that much better than Java-like accessor functions. Do you actually use sysconfig yourself? It's quite a specialized module, and I don't think API elegance arguments have a great weight here. Regards Antoine. From barry at python.org Sun Dec 12 15:05:06 2010 From: barry at python.org (Barry Warsaw) Date: Sun, 12 Dec 2010 09:05:06 -0500 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <20101212144231.772efa58@pitrou.net> References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> <20101211151703.09c8675e@pitrou.net> <037003C3-70F1-4430-8CA4-55C122F40C86@gmail.com> <27F9BCAE-74C1-4863-85A4-1E411E1AC06A@langa.pl> <20101212144231.772efa58@pitrou.net> Message-ID: <20101212090506.7c22638e@mission> On Dec 12, 2010, at 02:42 PM, Antoine Pitrou wrote: >On Sun, 12 Dec 2010 13:01:42 +0100 >?ukasz Langa wrote: > >> Wiadomo?? napisana przez Raymond Hettinger w dniu 2010-12-11, o godz. 22:18: >> >> >> *(I sometimes lose track of which changes were made in both branches >> >> pre-2.7, which ones were mode post-2.7 release, and which ones went in >> >> pre-2.7, but were 3.x only regardless) >> > >> > Right. I missed that it was already in 2.7. >> > So, now we're stuck with it, forever. >> >> Why not deprecate it for 3.2 (easy since it's probably not yet used anywhere anyway, even in 2.7) and introduce sys.sysconfig. > >We already had a lot of churn around these APIs (distutils & friends). I >don't think it's a good idea to introduce even more churn. >Oh and by the way it's too late to add or remove features from 3.2. > >> I really like that much better than Java-like accessor functions. > >Do you actually use sysconfig yourself? It's quite a specialized >module, and I don't think API elegance arguments have a great weight >here. I have used them and I do think they're fairly ugly and unwieldy. However, I agree that we should not rush into a different design. Since sysconfig was essentially refactored out of distutils (and now we have two ways to do it!), and we're getting distutils2 for 3.3, I think it would be better to work out a more natural design for sysconfig for 3.3. Then the sysconfig module can go through the natural deprecation cycle. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From ncoghlan at gmail.com Sun Dec 12 15:05:44 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 00:05:44 +1000 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <20101212144231.772efa58@pitrou.net> References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> <20101211151703.09c8675e@pitrou.net> <037003C3-70F1-4430-8CA4-55C122F40C86@gmail.com> <27F9BCAE-74C1-4863-85A4-1E411E1AC06A@langa.pl> <20101212144231.772efa58@pitrou.net> Message-ID: On Sun, Dec 12, 2010 at 11:42 PM, Antoine Pitrou wrote: >> I really like that much better than Java-like accessor functions. > > Do you actually use sysconfig yourself? It's quite a specialized > module, and I don't think API elegance arguments have a great weight > here. I would also like those advocating replacement of a tried and tested API with "oh, you can just use a single function for it" to rewrite distutils2 and the other tools that currently used distutils.sysconfig based on their "single function" approach before *anything* gets touched. This is not an API that was invented on a whim. It is well-established, with existing use cases that are essential to the wider Python ecosystem, and are more easily managed on some Linux distributions if they don't involve a dependency on distutils. Proposing to throw it away in favour of an ill-defined single function that indiscriminately mixes system data with metadata about that data because some people that don't even use the module find it aesthetically displeasing seems... unwise. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Sun Dec 12 15:44:22 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 00:44:22 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: On Sun, Dec 12, 2010 at 5:32 AM, Brett Cannon wrote: > On Fri, Dec 10, 2010 at 22:21, Vinay Sajip wrote: >> Nick Coghlan gmail.com> writes: >> >> >>> This could actually make a reasonably good basic for a "task oriented" >>> subsection of the logging documentation. Something like: >>> >> >> Good suggestion, I'll see what I can do. > > Just wanted to +1 on some task-oriented (or at least simple intro) > docs going into the logging module. I think Vinay has made some great improvements to the logging module docs in the last day or two. The latest version out of SVN is available on the site at the usual location: http://docs.python.org/dev/library/logging I am putting some minor notes here for Vinay's benefit (I can put them on the tracker instead, if he would prefer): General It may be worth talking to Georg about how best to split the logging docs up into multiple files. The sidebar menu is getting kinda overwhelmed. 14.7.1.1 Parenthetical comment in first row of second table should start with "(e.g. for" not "(e.g. or" 14.7.1.8 Probably best to say "that's it for the basic tutorial" and then point people towards the advanced tutorial in 14.7.2 before setting them loose on the rest of the docs. The advanced tutorial defines the terminology and gives the necessary structure to help keep the detailed docs in perspective without being overwhelmed by the detail. 14.7.2.1 Something appears to have gone wrong with the first bulleted list. It is missing the "These are the configuration methods:" intro text, as well as a bullet for add/removeHandler The "does not address filters" part should cross-reference the detailed section on filter objects This section should state explicitly whether or not the level setting on a child logger affects which messages it passes to its parent logger 14.7.2.5 This section is out of date, and needs to be caveated to make it clear that it applies only to version prior to Python 3.2 (for 3.2, it can describe the new handler of last resort behaviour) And that's the end of the two tutorials... very nice update :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From stefan-usenet at bytereef.org Sun Dec 12 15:37:03 2010 From: stefan-usenet at bytereef.org (Stefan Krah) Date: Sun, 12 Dec 2010 15:37:03 +0100 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <27F9BCAE-74C1-4863-85A4-1E411E1AC06A@langa.pl> References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> <20101211151703.09c8675e@pitrou.net> <037003C3-70F1-4430-8CA4-55C122F40C86@gmail.com> <27F9BCAE-74C1-4863-85A4-1E411E1AC06A@langa.pl> Message-ID: <20101212143703.GA15235@yoda.bytereef.org> Lukasz Langa wrote: > Wiadomo?? napisana przez Raymond Hettinger w dniu 2010-12-11, o godz. 22:18: > > Right. I missed that it was already in 2.7. > So, now we're stuck with it, forever. > > Why not deprecate it for 3.2 (easy since it's probably not yet used anywhere > anyway, even in 2.7) and introduce sys.sysconfig. I really like that much > better than Java-like accessor functions. When I use sysconfig, I just want to get things done as quickly and painlessly as possible. The API suits me just fine (in fact, I find it one of the APIs that are easy to remember). Given that sysconfig will always contain a certain amount of hackery and will always change to accommodate new systems, I'd prefer that it remains a standalone module. Stefan Krah From ziade.tarek at gmail.com Sun Dec 12 16:53:20 2010 From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=) Date: Sun, 12 Dec 2010 16:53:20 +0100 Subject: [Python-Dev] API for the new sysconfig module In-Reply-To: <20101212090506.7c22638e@mission> References: <20101210215902.CDB5B1AB60E@kimball.webabinitio.net> <20101211151703.09c8675e@pitrou.net> <037003C3-70F1-4430-8CA4-55C122F40C86@gmail.com> <27F9BCAE-74C1-4863-85A4-1E411E1AC06A@langa.pl> <20101212144231.772efa58@pitrou.net> <20101212090506.7c22638e@mission> Message-ID: On Sun, Dec 12, 2010 at 3:05 PM, Barry Warsaw wrote: > On Dec 12, 2010, at 02:42 PM, Antoine Pitrou wrote: > >>On Sun, 12 Dec 2010 13:01:42 +0100 >>?ukasz Langa wrote: >> >>> Wiadomo?? napisana przez Raymond Hettinger w dniu 2010-12-11, o godz. 22:18: >>> >>> >> *(I sometimes lose track of which changes were made in both branches >>> >> pre-2.7, which ones were mode post-2.7 release, and which ones went in >>> >> pre-2.7, but were 3.x only regardless) >>> > >>> > Right. ?I missed that it was already in 2.7. >>> > So, now we're stuck with it, forever. >>> >>> Why not deprecate it for 3.2 (easy since it's probably not yet used anywhere anyway, even in 2.7) and introduce sys.sysconfig. >> >>We already had a lot of churn around these APIs (distutils & friends). I >>don't think it's a good idea to introduce even more churn. >>Oh and by the way it's too late to add or remove features from 3.2. >> >>> I really like that much better than Java-like accessor functions. >> >>Do you actually use sysconfig yourself? It's quite a specialized >>module, and I don't think API elegance arguments have a great weight >>here. > > I have used them and I do think they're fairly ugly and unwieldy. ?However, I > agree that we should not rush into a different design. ?Since sysconfig was > essentially refactored out of distutils (and now we have two ways to do it!), > and we're getting distutils2 for 3.3, I think it would be better to work out a > more natural design for sysconfig for 3.3. ?Then the sysconfig module can go > through the natural deprecation cycle. I don't think any API refactoring worth the pain here. I don't see the proposed changes make the caller's code that much better. The existing one is good enough in my opinion. Tarek -- Tarek Ziad? | http://ziade.org From vinay_sajip at yahoo.co.uk Sun Dec 12 18:41:46 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Sun, 12 Dec 2010 17:41:46 +0000 (UTC) Subject: [Python-Dev] Using logging in the stdlib and its unit tests References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: Nick Coghlan gmail.com> writes: Gosh, Nick, that was fast! I'm still making changes, but thanks for spotting and highlighting the typos and omissions. I've just checked in a further update; hopefully it'll get built soon so we can all see the latest changes. Regards, Vinay Sajip From zeljko.grk at gmail.com Sun Dec 12 20:04:49 2010 From: zeljko.grk at gmail.com (Zeljko) Date: Sun, 12 Dec 2010 19:04:49 +0000 (UTC) Subject: [Python-Dev] use case for bytes.format Message-ID: I'm considering to write some pure python pdf lib from from scratch, but found there is no built-in formating for bytes. It's very frustrating to use some inefficient surogate funtion, which is gong to be called thousands times. From p.f.moore at gmail.com Sun Dec 12 23:26:10 2010 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 12 Dec 2010 22:26:10 +0000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <4D029439.7040708@g.nevcal.com> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D029439.7040708@g.nevcal.com> Message-ID: On 10 December 2010 20:57, Glenn Linderman wrote: > On 12/10/2010 12:49 PM, Antoine Pitrou wrote: > > And yet, I have helped many people who were baffled by exactly what >> Bill observed: logging.info() didn't do anything. Maybe the default >> should be INFO? > > Funny, because displaying only errors and silencing other messages is > exactly what I expected logging to do by default. > > So we are slowly learning the things that should be on the first couple > pages of the logging docs... > > 1) simple example for one file programs, include an example of specifying > output severity threshold.? I'm with Antoine here on my expectations. > > 2) example for multi-module, showing how a single logging destination causes > logging to happen in all modules, at the same level (if that is the case, > which I hope it is). > > 3) Maybe a small discussion of log formatting should be next?? So the user > realizes he shouldn't do the message formatting himself? > > 4) Then rotating logs for long-running programs. The thing *I* hit very early was wanting to add a command lime option to my script to set the logging level. I'd have liked to be able to add --log=INFO/DEBUG/... but to do that I seem to need to write my own mapping between level names and numbers. A simple example of how to tie command line options to logging config would be a great addition to the documentation. Paul. From tjreedy at udel.edu Sun Dec 12 23:33:19 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 12 Dec 2010 17:33:19 -0500 Subject: [Python-Dev] use case for bytes.format In-Reply-To: References: Message-ID: On 12/12/2010 2:04 PM, Zeljko wrote: This post should have gone to python-list, mirrored as gmane.comp.python.general. Please limit any further response to either of those (or c.l.p) and delete pydev. > I'm considering to write some pure python pdf lib from from scratch, but found > there is no built-in formating for bytes. Bytes do, but you should use text str for general text manipulation. > It's very frustrating to use some inefficient surogate funtion, which is gong to > be called thousands times. This sentence has 3 spelling mistakes, 2 of which would be caught with a mail client like Thunderbird (free) that has spelling correction. Please consider switching. -- Terry Jan Reedy From merwok at netwok.org Sun Dec 12 23:33:22 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sun, 12 Dec 2010 23:33:22 +0100 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: <4D054DB2.8020204@netwok.org> Hi, I suggest to replace ?error? with ?event? in the module doc synopsis. Regards From v+python at g.nevcal.com Mon Dec 13 01:42:49 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 12 Dec 2010 16:42:49 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D029439.7040708@g.nevcal.com> Message-ID: <4D056C09.3090104@g.nevcal.com> On 12/12/2010 2:26 PM, Paul Moore wrote: > The thing*I* hit very early was wanting to add a command lime option > to my script to set the logging level. I'd have liked to be able to > add --log=INFO/DEBUG/... but to do that I seem to need to write my own > mapping between level names and numbers. A simple example of how to > tie command line options to logging config would be a great addition > to the documentation. Oh? import * from logger # change some details to avoid this basicConfig( level= eval( opt.loglevel ) -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Mon Dec 13 01:53:05 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 12 Dec 2010 16:53:05 -0800 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> Message-ID: <4D056E71.3020704@g.nevcal.com> On 12/12/2010 9:41 AM, Vinay Sajip wrote: > Gosh, Nick, that was fast! I'm still making changes, but thanks for spotting and > highlighting the typos and omissions. I've just checked in a further update; > hopefully it'll get built soon so we can all see the latest changes. I'm not as fast as Nick, but let me add that these changes to the documentation are surely helpful to me. I've read 12% now, of a bigger base, but it was very approachable, and I've come away with being ready to scrap my little logger, I know what I need to do to make my multi-module logging work with the logging module instead, to greater benefit than my little logger, and the only "advanced technique" that I think I need to learn at the moment is formatters, so next chance I get I'll read about those. The mountain doesn't look as steep, now! Thanks for the fast reaction time. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.kern at gmail.com Mon Dec 13 02:22:47 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 12 Dec 2010 19:22:47 -0600 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: <4D056C09.3090104@g.nevcal.com> References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D029439.7040708@g.nevcal.com> <4D056C09.3090104@g.nevcal.com> Message-ID: On 2010-12-12 18:42 , Glenn Linderman wrote: > On 12/12/2010 2:26 PM, Paul Moore wrote: >> The thing*I* hit very early was wanting to add a command lime option >> to my script to set the logging level. I'd have liked to be able to >> add --log=INFO/DEBUG/... but to do that I seem to need to write my own >> mapping between level names and numbers. A simple example of how to >> tie command line options to logging config would be a great addition >> to the documentation. > > Oh? > > import * from logger # change some details to avoid this > basicConfig( level= eval( opt.loglevel ) level = getattr(logging, opt.logLevel) or level = logging._levelNames[opt.logLevel] -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ncoghlan at gmail.com Mon Dec 13 04:30:24 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 13:30:24 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D029439.7040708@g.nevcal.com> <4D056C09.3090104@g.nevcal.com> Message-ID: On Mon, Dec 13, 2010 at 11:22 AM, Robert Kern wrote: > level = getattr(logging, opt.logLevel) While this is the approach I would recommend, it does have a couple of downsides: 1. An upper() call is also needed to allow strings like "info" instead of "INFO": 2. If an integer is available, it would be nice to return it unmodified (or, if we ever get named values in the standard library, convert it to that) 3. The asymmetry with "logging.getLevelName" grates a bit So it would be far more obvious if there was a "logging.getLevel" counterpart to "logging.getLevelName" that looked something like: def getLevel(level): try: return operator.index(level) # Integers and equivalents except TypeError: pass try: key = level.upper() except Exception as ex: raise TypeError("Log level must be an integer or string") from ex return globals()[key] > level = logging._levelNames[opt.logLevel] That doesn't work (_levelNames maps from integers to strings, we want the mapping from strings to integers and it is only the module globals that provides that). Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Dec 13 04:45:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 13:45:50 +1000 Subject: [Python-Dev] [Python-checkins] r87202 - python/branches/py3k/Doc/library/logging.rst In-Reply-To: <20101212224535.C828DEE995@mail.python.org> References: <20101212224535.C828DEE995@mail.python.org> Message-ID: On Mon, Dec 13, 2010 at 8:45 AM, vinay.sajip wrote: > +to get the value which you'll pass to :func:`basicConfig` via the *level* > +argument. You may want to error check any user input value, perhaps as in the > +following example:: > + > + ? # assuming loglevel is bound to the string value obtained from the > + ? # command line argument. Convert to upper case to allow the user to > + ? # specify --log=DEBUG or --log=debug > + ? numeric_level = getattr(logging, loglevel.upper(), None) > + ? assert numeric_level is not None, 'Invalid log level: %s' % loglevel > + ? logging.basicConfig(level=numeric_level, ...) Minor nit - using asserts to check user input is generally a bad idea. A more explicit check might be a better example: if not isinstance(numeric_level, int): raise ValueError('Invalid log level: %s' % loglevel) This also covers the case where someone does something weird like pass in "basic_format" as a logging level. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From robert.kern at gmail.com Mon Dec 13 05:13:12 2010 From: robert.kern at gmail.com (Robert Kern) Date: Sun, 12 Dec 2010 22:13:12 -0600 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D029439.7040708@g.nevcal.com> <4D056C09.3090104@g.nevcal.com> Message-ID: On 2010-12-12 21:30 , Nick Coghlan wrote: > On Mon, Dec 13, 2010 at 11:22 AM, Robert Kern wrote: >> level = getattr(logging, opt.logLevel) > > While this is the approach I would recommend, it does have a couple of > downsides: > > 1. An upper() call is also needed to allow strings like "info" instead > of "INFO": > 2. If an integer is available, it would be nice to return it > unmodified (or, if we ever get named values in the standard library, > convert it to that) > 3. The asymmetry with "logging.getLevelName" grates a bit > > So it would be far more obvious if there was a "logging.getLevel" > counterpart to "logging.getLevelName" that looked something like: > > def getLevel(level): > try: > return operator.index(level) # Integers and equivalents > except TypeError: > pass > try: > key = level.upper() > except Exception as ex: > raise TypeError("Log level must be an integer or string") from ex > return globals()[key] I don't think that the implementation should use globals(). I wouldn't want logging.getLevel('basic_format') to work. Instead, it should look it up in the known set of levels. >> level = logging._levelNames[opt.logLevel] > > That doesn't work (_levelNames maps from integers to strings, we want > the mapping from strings to integers and it is only the module globals > that provides that). At least in Python 2.6, it maps both ways. But then again, it is an _implementation _detail that should not be relied upon in your programs. I would suggest that there should be two dictionaries as part of the documented API, one mapping numbers to names and one mapping names to numbers. Or functions/methods returning said dictionaries. Having the entire mappings at hand is more useful than having functions that do the translation. They would allow you to auto-generate UIs (e.g. the help text for a --log-level argument or a radio box widget in a GUI). Having separate mappings makes them easier to work with than the 2.6-style _levelNames mapping. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco From ncoghlan at gmail.com Mon Dec 13 05:27:21 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 14:27:21 +1000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: <20101207215053.025ecf52@pitrou.net> <20101208010913.363c6105@pitrou.net> <4D01E55B.2050507@g.nevcal.com> <3231.1292004978@parc.com> <20101210214919.4ac63d27@pitrou.net> <4D029439.7040708@g.nevcal.com> <4D056C09.3090104@g.nevcal.com> Message-ID: On Mon, Dec 13, 2010 at 2:13 PM, Robert Kern wrote: >>> level = logging._levelNames[opt.logLevel] >> >> That doesn't work (_levelNames maps from integers to strings, we want >> the mapping from strings to integers and it is only the module globals >> that provides that). > > At least in Python 2.6, it maps both ways. But then again, it is an > _implementation _detail that should not be relied upon in your programs. Ah, you're quite right - I didn't notice that when looking at the contents (the first entries happened to map levels to names) > I > would suggest that there should be two dictionaries as part of the > documented API, one mapping numbers to names and one mapping names to > numbers. Or functions/methods returning said dictionaries. Having the entire > mappings at hand is more useful than having functions that do the > translation. They would allow you to auto-generate UIs (e.g. the help text > for a --log-level argument or a radio box widget in a GUI). Having separate > mappings makes them easier to work with than the 2.6-style _levelNames > mapping. Definitely something worth considering for 3.3. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From rich at noir.com Fri Dec 10 21:20:13 2010 From: rich at noir.com (K. Richard Pixley) Date: Fri, 10 Dec 2010 12:20:13 -0800 Subject: [Python-Dev] logging doc broken Message-ID: <4D028B7D.9030308@noir.com> I'm not sure where to report this but the online doc appears to be mismatched to python-2.6.5 for the logging module. Specifically, for a dir of an instance of a LogRecord, I'm seeing: ['__doc__', '__init__', '__module__', '__str__', 'args', 'created', 'exc_info', 'exc_text', 'filename', 'funcName', 'getMessage', 'levelname', 'levelno', 'lineno', 'module', 'msecs', 'msg', 'name', 'pathname', 'process', 'processName', 'relativeCreated', 'thread', 'threadName'] while the documentation lists a different set of attributes including "lvl". --rich From ncoghlan at gmail.com Mon Dec 13 11:21:18 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 20:21:18 +1000 Subject: [Python-Dev] logging doc broken In-Reply-To: <4D028B7D.9030308@noir.com> References: <4D028B7D.9030308@noir.com> Message-ID: On Sat, Dec 11, 2010 at 6:20 AM, K. Richard Pixley wrote: > I'm not sure where to report this but the online doc appears to be > mismatched to python-2.6.5 for the logging module. > > Specifically, for a dir of an instance of a LogRecord, I'm seeing: > > ['__doc__', '__init__', '__module__', '__str__', 'args', 'created', > 'exc_info', 'exc_text', 'filename', 'funcName', 'getMessage', 'levelname', > 'levelno', 'lineno', 'module', 'msecs', 'msg', 'name', 'pathname', > 'process', 'processName', 'relativeCreated', 'thread', 'threadName'] > > while the documentation lists a different set of attributes including "lvl". As discussed on this list recently, the logging documentation can unfortunately give the impression that the attributes of the log record are the same as the arguments to the log record constructor. This is not the case - the actual list of attributes can be found in the table showing the useful LogRecord attributes that are available to formatters (and filters) when processing records: http://docs.python.org/library/logging#formatter-objects Vinay is currently working on updates to the logging documentation, and this is one of the things that is likely to change (with the table of attributes moved to the LogRecord section and referenced from the sections on Formatter and Filter objects). Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Mon Dec 13 11:31:35 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Dec 2010 10:31:35 +0000 (UTC) Subject: [Python-Dev] logging doc broken References: <4D028B7D.9030308@noir.com> Message-ID: K. Richard Pixley noir.com> writes: > > I'm not sure where to report this but the online doc appears to be > mismatched to python-2.6.5 for the logging module. > > Specifically, for a dir of an instance of a LogRecord, I'm seeing: > > ['__doc__', '__init__', '__module__', '__str__', 'args', 'created', > 'exc_info', 'exc_text', 'filename', 'funcName', 'getMessage', > 'levelname', 'levelno', 'lineno', 'module', 'msecs', 'msg', 'name', > 'pathname', 'process', 'processName', 'relativeCreated', 'thread', > 'threadName'] > > while the documentation lists a different set of attributes including "lvl". > Please report this on bugs.python.org. Check that you were referring to the 2.6 documentation - I didn't see any references to attributes there (following a quick scan). Ideally, link to the wrong doc section on docs.python.org in your bug report. Thanks, Vinay Sajip From mail at timgolden.me.uk Mon Dec 13 11:35:08 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 13 Dec 2010 10:35:08 +0000 Subject: [Python-Dev] logging doc broken In-Reply-To: References: <4D028B7D.9030308@noir.com> Message-ID: <4D05F6DC.5040900@timgolden.me.uk> On 13/12/2010 10:31, Vinay Sajip wrote: [...] > Ideally, link to the wrong doc section on docs.python.org in your bug report. Now that's not a piece of advice you see very often :) TJG From ncoghlan at gmail.com Mon Dec 13 11:38:35 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 20:38:35 +1000 Subject: [Python-Dev] logging doc broken In-Reply-To: References: <4D028B7D.9030308@noir.com> Message-ID: On Mon, Dec 13, 2010 at 8:31 PM, Vinay Sajip wrote: > Please report this on bugs.python.org. Check that you were referring to the 2.6 > documentation - I didn't see any references to attributes there (following a > quick scan). Ideally, link to the wrong doc section on docs.python.org in your > bug report. The reference to "lvl" makes me think this is the same mistake I made the other day (i.e. thinking the constructor parameters are also the LogRecord attributes) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Mon Dec 13 11:58:32 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Dec 2010 10:58:32 +0000 (UTC) Subject: [Python-Dev] logging doc broken References: <4D028B7D.9030308@noir.com> Message-ID: Nick Coghlan gmail.com> writes: > Vinay is currently working on updates to the logging documentation, > and this is one of the things that is likely to change (with the table > of attributes moved to the LogRecord section and referenced from the > sections on Formatter and Filter objects). Yes, and to compound matters, the LogRecord constructor documentation lists the attribute values in the place you'd expect the parameters to appear, using the doc markup for attributes. This I think is what causes the confusion, although the markup is different to that used for parameters it's not clear at first glance. However, these changes are after 2.7 and 3.1: In 2.6 documentation the parameters are named as in the source, but there's no reference in the LogRecord reference documentation about any attributes of the LogRecord. Perhaps it would be best to do the following: - Add constructor params using normal parameter markup (:param:) - Create a separate section "LogRecord Attributes" with a table containing attr name, description, how to use with % formatting and how to use with {}-formatting. There is some overlap here with the parameter documentation for the LogRecord constructor, but I can put in a rider which explains that the param names and attr names are slightly different. (I could kick myself now for not being more scrupulous originally, but I'm not sure I can go back and change the param names in the source code now because in theory, someone might be constructing a LogRecord using LogRecord(**kwargs)). - Link to the latter section from the Formatter, Filter sections. Anyone see problems with this, or have a better suggestion? Regards, Vinay Sajip From vinay_sajip at yahoo.co.uk Mon Dec 13 12:04:26 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Dec 2010 11:04:26 +0000 (UTC) Subject: [Python-Dev] logging doc broken References: <4D028B7D.9030308@noir.com> <4D05F6DC.5040900@timgolden.me.uk> Message-ID: Tim Golden timgolden.me.uk> writes: > > On 13/12/2010 10:31, Vinay Sajip wrote: > [...] > > Ideally, link to the wrong doc section on docs.python.org in your bug report. > > Now that's not a piece of advice you see very often :) > True, but this area changed after 2.6 was released (after even 2.7, IIRC), so I want to be sure that if I'm going to change the doc sources on release26-maint, I'm doing the right thing. The 2.6 docs seem consistent with the 2.6 code, and while confusing (because the distinction between constructor param names and attr names is not spelled out), I don't see 2.6 docs as broken (or at least, I don't see where the breakage is). Regards, Vinay Sajip From ncoghlan at gmail.com Mon Dec 13 12:17:26 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 21:17:26 +1000 Subject: [Python-Dev] logging doc broken In-Reply-To: References: <4D028B7D.9030308@noir.com> Message-ID: On Mon, Dec 13, 2010 at 8:58 PM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > >> Vinay is currently working on updates to the logging documentation, >> and this is one of the things that is likely to change (with the table >> of attributes moved to the LogRecord section and referenced from the >> sections on Formatter and Filter objects). > > Yes, and to compound matters, the LogRecord constructor documentation lists the > attribute values in the place you'd expect the parameters to appear, using the > doc markup for attributes. This I think is what causes the confusion, although > the markup is different to that used for parameters it's not clear at first > glance. > > However, these changes are after 2.7 and 3.1: In 2.6 documentation the > parameters are named as in the source, but there's no reference in the > LogRecord reference documentation about any attributes of the LogRecord. That's the problem though - if you don't read the intro paragraph closely (which I didn't do), it's easy to jump to the conclusion that those are also the attributes (since there is nothing else in the section). > Perhaps it would be best to do the following: > > - Add constructor params using normal parameter markup (:param:) > - Create a separate section "LogRecord Attributes" with a table containing attr > name, description, how to use with % formatting and how to use with > {}-formatting. There is some overlap here with the parameter documentation for > the LogRecord constructor, but I can put in a rider which explains that the > param names and attr names are slightly different. (I could kick myself now for > not being more scrupulous originally, but I'm not sure I can go back and change > the param names in the source code now because in theory, someone might be > constructing a LogRecord using LogRecord(**kwargs)). > > - Link to the latter section from the Formatter, Filter sections. > > Anyone see problems with this, or have a better suggestion? Yep, that's what I had assumed you were going to do (this did come up in the big thread about the logging docs, but you may have missed it). You're also right about the backwards compatibility problem with changing the parameter names. It's the one downside of keyword arguments - it makes the parameter names part of a function's public API, so they are much harder to fix when you make a mistake. That said, you can't get a one-to-one correspondence in this case anyway, since some of the parameters relate to multiple attributes (e.g. "lvl" becomes both "levelNo" and "levelName"). It is rare that anyone will be constructing a LogRecord manually though, so I don't think the parameter names matter all that much in practice. It is far more common that people will be creating them implicitly through the Logger event notification methods or by deserialising them. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Dec 13 12:21:36 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 21:21:36 +1000 Subject: [Python-Dev] logging doc broken In-Reply-To: References: <4D028B7D.9030308@noir.com> <4D05F6DC.5040900@timgolden.me.uk> Message-ID: On Mon, Dec 13, 2010 at 9:04 PM, Vinay Sajip wrote: > Tim Golden timgolden.me.uk> writes: > >> >> On 13/12/2010 10:31, Vinay Sajip wrote: >> [...] >> > Ideally, link to the wrong doc section on docs.python.org in your bug report. >> >> Now that's not a piece of advice you see very often :) >> > > True, but this area changed after 2.6 was released (after even 2.7, IIRC), so I > want to be sure that if I'm going to change the doc sources on release26-maint, > I'm doing the right thing. It was more a comment on the fact that, at first glance, that sentence looks like an instuction to provide an incorrect link, rather than to provide a link to the section of the docs that is incorrect. Just a quirk of English grammar :) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From zuo at chopin.edu.pl Mon Dec 13 12:44:49 2010 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Mon, 13 Dec 2010 12:44:49 +0100 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation Message-ID: <20101213114449.GA2348@chopin.edu.pl> Dear Python Developers, It is s my first post to python-dev, so let me introduce myself briefly: Jan Kaliszewski, programmer and composer, sometimes also NGO activist. Coming to the matter... The discussion started with remark by Mark Dickinson about such a syntax oddity: > def f(a, b,): ... is fine, but > def f(*, a, b,): ... is a SyntaxError Then some other similar oddities were pointed at (*args/**kwargs-related ones as well as calls like f(*, a=3,) causing SyntaxError too). References: * http://mail.python.org/pipermail/python-dev/2010-July/101636.html * http://bugs.python.org/issue9232 * http://bugs.python.org/issue10682 But yesterday both mentioned issues has been closed as rejected -- with suggestion that it would probably require a PEP to modify Python in this aspect (as there is no clear consensus). So I'd opt for re-opening the discussion -- I suppose that more people could be interested in solving the issue (at least after the end of PEP 3003 moratorium period). I think that seeing that: def f(a, b): ... def f(a, b,): ... def f(a, *, b): ... def f(a, *args, b): ... x(1, 2, 3, 4, z=5) x(1, 2, 3, 4, z=5,) x(1, *(2,3,4), z=5) ...are ok, then -- def f(a, *, b,): ... def f(a, *args, b,): ... x(1, *(2,3,4), z=5,): ... ...should be ok as well, and consequently -- def f(a, *args,): ... def f(a, **kwargs,): ... x(1, *(2,3,4),) x(1, **dict(z=6),) ...should also be ok. Please also note that Py3k's function annotations make one-def-argument- -per-line formattig style the most suitable in some cases, e.g.: def my_func( spam:"Very tasty and nutritious piece of food", ham:"For experts only", *more_spam:"Not less tasty and not less nutritious!", spammish_inquisition:"Nobody expects this!", ) -> "Spam, spam, spam, spam, spam, spam, spam, spam, spam, spam": ... Regards, Jan Kaliszewski From ncoghlan at gmail.com Mon Dec 13 14:25:58 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 23:25:58 +1000 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: <20101213114449.GA2348@chopin.edu.pl> References: <20101213114449.GA2348@chopin.edu.pl> Message-ID: On Mon, Dec 13, 2010 at 9:44 PM, Jan Kaliszewski wrote: > I think that seeing that: > > ? ?def f(a, b): ... > ? ?def f(a, *, b): ... > ? ?def f(a, *args, b): ... > ? ?x(1, 2, 3, 4, z=5) > ? ?x(1, *(2,3,4), z=5) As per the closure of the affected tickets, the likely outcome of such a discussion would be the deprecation and subsequent removal of support for the following two options: def f(a, b,): ... x(1, 2, 3, 4, z=5,): ... Function arguments are not lists. Even when separated onto multiple lines, the closing "):" should remain on the final line with other content. That would be a lot of hassle to get rid of something that people probably aren't doing in the first place, though. Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From mail at timgolden.me.uk Mon Dec 13 14:29:12 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 13 Dec 2010 13:29:12 +0000 Subject: [Python-Dev] logging doc broken In-Reply-To: References: <4D028B7D.9030308@noir.com> <4D05F6DC.5040900@timgolden.me.uk> Message-ID: <4D061FA8.2000100@timgolden.me.uk> On 13/12/2010 11:21, Nick Coghlan wrote: > On Mon, Dec 13, 2010 at 9:04 PM, Vinay Sajip wrote: >> Tim Golden timgolden.me.uk> writes: >> >>> >>> On 13/12/2010 10:31, Vinay Sajip wrote: >>> [...] >>>> Ideally, link to the wrong doc section on docs.python.org in your bug report. >>> >>> Now that's not a piece of advice you see very often :) >>> >> >> True, but this area changed after 2.6 was released (after even 2.7, IIRC), so I >> want to be sure that if I'm going to change the doc sources on release26-maint, >> I'm doing the right thing. > > It was more a comment on the fact that, at first glance, that sentence > looks like an instuction to provide an incorrect link, rather than to > provide a link to the section of the docs that is incorrect. Just a > quirk of English grammar :) Thanks, Nick. That is what I meant. I wanted to indicate that it was tongue-in-cheek, but I never can remember the sequence of characters which means that... TJG From fuzzyman at voidspace.org.uk Mon Dec 13 14:39:44 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 13 Dec 2010 13:39:44 +0000 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> Message-ID: <4D062220.8040501@voidspace.org.uk> On 13/12/2010 13:25, Nick Coghlan wrote: > On Mon, Dec 13, 2010 at 9:44 PM, Jan Kaliszewski wrote: >> I think that seeing that: >> >> def f(a, b): ... >> def f(a, *, b): ... >> def f(a, *args, b): ... >> x(1, 2, 3, 4, z=5) >> x(1, *(2,3,4), z=5) > As per the closure of the affected tickets, the likely outcome of such > a discussion would be the deprecation and subsequent removal of > support for the following two options: > > def f(a, b,): ... > x(1, 2, 3, 4, z=5,): ... > > Function arguments are not lists. Even when separated onto multiple > lines, the closing "):" should remain on the final line with other > content. Why? For very long signatures I still mildly prefer this: def f(self, first, second, third, fourth, foo=None, bar=None, baz=None, spam=None, eggs=None, ham=None ): Over putting the closing paren: on the last line of the def. Of course not having such long signatures is even more preferable, but *sometimes* they are needed. All the best, Michael Foord > That would be a lot of hassle to get rid of something that people > probably aren't doing in the first place, though. > > Regards, > Nick. > -- http://www.voidspace.org.uk/ READ CAREFULLY. By accepting and reading this email you agree, on behalf of your employer, to release me from all obligations and waivers arising from any and all NON-NEGOTIATED agreements, licenses, terms-of-service, shrinkwrap, clickwrap, browsewrap, confidentiality, non-disclosure, non-compete and acceptable use policies (?BOGUS AGREEMENTS?) that I have entered into with your employer, its partners, licensors, agents and assigns, in perpetuity, without prejudice to my ongoing rights and privileges. You further represent that you have the authority to release me from any BOGUS AGREEMENTS on behalf of your employer. From tseaver at palladion.com Mon Dec 13 14:42:34 2010 From: tseaver at palladion.com (Tres Seaver) Date: Mon, 13 Dec 2010 08:42:34 -0500 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 12/13/2010 08:25 AM, Nick Coghlan wrote: > On Mon, Dec 13, 2010 at 9:44 PM, Jan Kaliszewski wrote: >> I think that seeing that: >> >> def f(a, b): ... >> def f(a, *, b): ... >> def f(a, *args, b): ... >> x(1, 2, 3, 4, z=5) >> x(1, *(2,3,4), z=5) > > As per the closure of the affected tickets, the likely outcome of such > a discussion would be the deprecation and subsequent removal of > support for the following two options: > > def f(a, b,): ... > x(1, 2, 3, 4, z=5,): ... > > Function arguments are not lists. Even when separated onto multiple > lines, the closing "):" should remain on the final line with other > content. > > That would be a lot of hassle to get rid of something that people > probably aren't doing in the first place, though. I actually make use of the feature when dealing with APIs which both a) take lots of arguments (more than fit comfortably on two lines at whatever indentation they are called), and b) have optional trailing arguments: I always leave the trailing comma in place in such cases, with the closing paren on the following line, so that adding or removing an argument at the end of the list stays consistent (the diffs are better, too, when I use this pattern). Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0GIsoACgkQ+gerLs4ltQ5HLwCfYFSyVrFtt04h6a39hyK6BD2c t8oAoJdXNS7wIsjF34ZiOQCwQGq9Qs2v =ZWqW -----END PGP SIGNATURE----- From ncoghlan at gmail.com Mon Dec 13 14:54:28 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 13 Dec 2010 23:54:28 +1000 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> Message-ID: On Mon, Dec 13, 2010 at 11:42 PM, Tres Seaver wrote: > I actually make use of the feature when dealing with APIs which both a) > take lots of arguments (more than fit comfortably on two lines at > whatever indentation they are called), and b) have optional trailing > arguments: ?I always leave the trailing comma in place in such cases, > with the closing paren on the following line, so that adding or removing > an argument at the end of the list stays consistent (the diffs are > better, too, when I use this pattern). My personal preferences aren't strong either way, but the issues were closed because committers voiced opinions against making this consistent in the other direction (i.e. always allowing the trailing comma). I don't know that a full PEP is really needed, but the status quo is that some committers said no and others don't really care about the issue all that much, so the current behaviour is going to remain in place unless those in the first group change their mind (or Guido weighs in and says "change it"). Creating a PEP is one way to carry out such persuasion (probably overkill though). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From zuo at chopin.edu.pl Mon Dec 13 15:18:34 2010 From: zuo at chopin.edu.pl (Jan Kaliszewski) Date: Mon, 13 Dec 2010 15:18:34 +0100 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> Message-ID: <20101213141834.GA5056@chopin.edu.pl> Nick Coghlan dixit (2010-12-13, 23:25): > Function arguments are not lists. Even when separated onto multiple > lines, the closing "):" should remain on the final line with other > content. Not necessarily, IMHO. 1. What about my example with '-> xxx' return-value annotation? (especially when that annotation is a long expression) 2. There are two argument-list-formatting idioms I apply -- depending on which is more suitable in a particular case: a) when argument specs/expressions are not very long and rather if their number is not very big: def function(argument_spec1, argument_spec2, argument_spec3, argument_spec4, argument_spec5, argument_spec6): function_call(expression1, expression2, expression3, expression4, expression5, expression6) b) for long argument lists and/or argument specs/expressions (e.g. when default values or argument annotations are defined as long expressions): def function( long_argument_spec1, long_argument_spec2, long_argument_spec3, long_argument_spec4, long_argument_spec5, long_argument_spec6, ): function_call( long_expression1, long_expression2, long_expression3, long_expression4, long_expression5, long_expression6, ) Note that option 'b' is more convenient for refactorization, diffs etc. Regards, *j From vinay_sajip at yahoo.co.uk Mon Dec 13 16:01:09 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Dec 2010 15:01:09 +0000 (UTC) Subject: [Python-Dev] logging doc broken References: <4D028B7D.9030308@noir.com> Message-ID: Nick Coghlan gmail.com> writes: > Yep, that's what I had assumed you were going to do (this did come up > in the big thread about the logging docs, but you may have missed it). I hadn't missed it - I'm just spelling out in more detail what I'm going to do. > That said, you can't get a one-to-one correspondence in this case > anyway, since some of the parameters relate to multiple attributes > (e.g. "lvl" becomes both "levelNo" and "levelName"). > > It is rare that anyone will be constructing a LogRecord manually > though, so I don't think the parameter names matter all that much in > practice. It is far more common that people will be creating them > implicitly through the Logger event notification methods or by > deserialising them. I agree, it's a pretty unlikely scenario. Of course, I do make use of the parameter names in the dictConfig approach, so I can't really complain. I should hopefully remember this when I'm writing out a method or function signature for a public API :-) Regards, Vinay From vinay_sajip at yahoo.co.uk Mon Dec 13 16:04:15 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Dec 2010 15:04:15 +0000 (UTC) Subject: [Python-Dev] logging doc broken References: <4D028B7D.9030308@noir.com> <4D05F6DC.5040900@timgolden.me.uk> <4D061FA8.2000100@timgolden.me.uk> Message-ID: Tim Golden timgolden.me.uk> writes: > > It was more a comment on the fact that, at first glance, that sentence > > looks like an instuction to provide an incorrect link, rather than to > > provide a link to the section of the docs that is incorrect. Just a > > quirk of English grammar :) > > Thanks, Nick. That is what I meant. I wanted to indicate that > it was tongue-in-cheek, but I never can remember the sequence > of characters which means that... > Aaargh! You're right of course, I was too hasty in typing "link to the wrong doc section" rather than the only slightly longer "link to the doc section which is wrong". Tim - when you find those emoticons, be sure to let me know the one for "facepalm" ;-) Regards, Vinay From rdmurray at bitdance.com Mon Dec 13 16:51:19 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 13 Dec 2010 10:51:19 -0500 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> Message-ID: <20101213155119.A52A621CD91@kimball.webabinitio.net> On Mon, 13 Dec 2010 23:25:58 +1000, Nick Coghlan wrote: > On Mon, Dec 13, 2010 at 9:44 PM, Jan Kaliszewski wrote: > > I think that seeing that: > > > > =A0 =A0def f(a, b): ... > > =A0 =A0def f(a, *, b): ... > > =A0 =A0def f(a, *args, b): ... > > =A0 =A0x(1, 2, 3, 4, z=3D5) > > =A0 =A0x(1, *(2,3,4), z=3D5) > > As per the closure of the affected tickets, the likely outcome of such > a discussion would be the deprecation and subsequent removal of > support for the following two options: > > def f(a, b,): ... > x(1, 2, 3, 4, z=3D5,): ... > > Function arguments are not lists. Even when separated onto multiple > lines, the closing "):" should remain on the final line with other > content. > > That would be a lot of hassle to get rid of something that people > probably aren't doing in the first place, though. Counter examples from google code search: http://www.google.com/codesearch/p?hl=en#copo3dCwf5E/django/utils/simplejson/encoder.py&q=^\s*\):&sa=N&cd=5&ct=rc (also appears in json in the stdlib) http://www.google.com/codesearch/p?hl=en#algXCqBNNP0/vendor/python-clientform/ClientForm.py&q=^\ *\):&sa=N&cd=3&ct=rc (class def) http://www.google.com/codesearch/p?hl=en#KT-ZlRkUunU/trunk/code/output/ExprParser.py&q=def\(.*,\s\):&sa=N&cd=2&ct=rc http://www.google.com/codesearch/p?hl=en#XnG7n8Mjf2s/GoogleSearch.py&q=def\(.*,\s\):&sa=N&cd=3&ct=rc http://www.google.com/codesearch/p?hl=en#MokQ50OeeyU/src/python/ndogen/parser/matlab/parser.py&q=def\(.*,\s\):&sa=N&cd=5&ct=rc Not many, granted, but not zero, either, and I'm sure there are lots more out there[*]. I do especially like the fact that there is one in the stdlib :) It seems like the status quo is fine. I wouldn't object to it being made more consistent. I would object to removing the existing cases. -- R. David Murray www.bitdance.com [*] code search's response to various regexes was somewhat surprising; expressions I thought should have been supersets resulted in fewer hits. Nor could I think of a way to search for function invocations ending with a comma. Then again, I usually make lots of mistakes with regexes. From guido at python.org Mon Dec 13 17:54:16 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 13 Dec 2010 08:54:16 -0800 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> Message-ID: On Mon, Dec 13, 2010 at 5:42 AM, Tres Seaver wrote: > I actually make use of the feature when dealing with APIs which both a) > take lots of arguments (more than fit comfortably on two lines at > whatever indentation they are called), and b) have optional trailing > arguments: ?I always leave the trailing comma in place in such cases, > with the closing paren on the following line, so that adding or removing > an argument at the end of the list stays consistent (the diffs are > better, too, when I use this pattern). Same here, and it's a soft style rule at Google that trailing commas are good -- they can help produce shorter diffs. I'm at least +0 on allowing trailing commas in the situation the OP mentioned. -- --Guido van Rossum (python.org/~guido) From vinay_sajip at yahoo.co.uk Mon Dec 13 19:55:12 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 13 Dec 2010 18:55:12 +0000 (UTC) Subject: [Python-Dev] logging doc broken References: <4D028B7D.9030308@noir.com> Message-ID: Nick Coghlan gmail.com> writes: > Yep, that's what I had assumed you were going to do (this did come up > in the big thread about the logging docs, but you may have missed it). > Ok, I've now checked in this change, and would be grateful for any feedback. Time for a break :-) Thanks a lot for all your patience and valuable feedback, Nick. It's been very helpful, and I hope it will make the logging docs more accessible to those who've been put off in the past. I've emailed Georg about how best to reorganise into: intro + basic tutorial (at current location) advanced tutorial reference cookbook and I'm waiting for his suggestions on how best to implement (in terms of breaking up into different files etc). Thanks and regards, Vinay From alexander.belopolsky at gmail.com Mon Dec 13 20:09:02 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 13 Dec 2010 14:09:02 -0500 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> Message-ID: On Mon, Dec 13, 2010 at 11:54 AM, Guido van Rossum wrote: > I'm at least +0 on > allowing trailing commas in the situation the OP mentioned. > FWIW, I am also about +0.5 on allowing trailing comma. Note that in a similar situation, the C standardization committee has erred on the side of consistency: """ A new feature of C99: a common extension in many implementations allows a trailing comma after the list of enumeration constants. The Committee decided to adopt this feature as an innocuous extension that mirrors the trailing commas allowed in initializers. """ http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf Similarly, I find allowing trailing comma in keyword only arguments lists to be an innocuous extension that mirrors the trailing commas allowed in the positional arguments lists. A possible benefit that I have not seen mentioned is that if developer decides to convert some of the trailing arguments in her function to keyword only, she does not have to worry about removing the trailing comma. From solipsis at pitrou.net Mon Dec 13 20:17:59 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 13 Dec 2010 20:17:59 +0100 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation References: <20101213114449.GA2348@chopin.edu.pl> Message-ID: <20101213201759.422b15de@pitrou.net> On Mon, 13 Dec 2010 14:09:02 -0500 Alexander Belopolsky wrote: > On Mon, Dec 13, 2010 at 11:54 AM, Guido van Rossum wrote: > > I'm at least +0 on > > allowing trailing commas in the situation the OP mentioned. > > > > FWIW, I am also about +0.5 on allowing trailing comma. Note that in a > similar situation, the C standardization committee has erred on the > side of consistency: > > """ > A new feature of C99: a common extension in many implementations > allows a trailing comma after the list of enumeration constants. The > Committee decided to adopt this feature as an innocuous extension that > mirrors the trailing commas allowed in initializers. > """ http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf > > Similarly, I find allowing trailing comma in keyword only arguments > lists to be an innocuous extension that mirrors the trailing commas > allowed in the positional arguments lists. +1 from me as well. Special cases are hard to remember. Regards Antoine. From v+python at g.nevcal.com Mon Dec 13 20:26:44 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 13 Dec 2010 11:26:44 -0800 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: <20101213201759.422b15de@pitrou.net> References: <20101213114449.GA2348@chopin.edu.pl> <20101213201759.422b15de@pitrou.net> Message-ID: <4D067374.8050200@g.nevcal.com> On 12/13/2010 11:17 AM, Antoine Pitrou wrote: > On Mon, 13 Dec 2010 14:09:02 -0500 > Alexander Belopolsky wrote: > >> On Mon, Dec 13, 2010 at 11:54 AM, Guido van Rossum wrote: >>> I'm at least +0 on >>> allowing trailing commas in the situation the OP mentioned. >>> >> FWIW, I am also about +0.5 on allowing trailing comma. Note that in a >> similar situation, the C standardization committee has erred on the >> side of consistency: >> >> """ >> A new feature of C99: a common extension in many implementations >> allows a trailing comma after the list of enumeration constants. The >> Committee decided to adopt this feature as an innocuous extension that >> mirrors the trailing commas allowed in initializers. >> """ http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf >> >> Similarly, I find allowing trailing comma in keyword only arguments >> lists to be an innocuous extension that mirrors the trailing commas >> allowed in the positional arguments lists. > +1 from me as well. Special cases are hard to remember. +1. I tend to put them in, and then take out the ones that Python won't accept. Then, when I add another item to the list, Python tells me to go back and put the one I took out back in again, and take out the one I put in at the new end of the list. Annoying. (for vertically arranged lists, one per line, primarily, with ) on the last line by itself. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Mon Dec 13 20:34:14 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 13 Dec 2010 14:34:14 -0500 Subject: [Python-Dev] logging doc broken In-Reply-To: References: <4D028B7D.9030308@noir.com> <4D05F6DC.5040900@timgolden.me.uk> Message-ID: On 12/13/2010 6:04 AM, Vinay Sajip wrote: > True, but this area changed after 2.6 was released (after even 2.7, IIRC), so I > want to be sure that if I'm going to change the doc sources on release26-maint, 2.6 is in security-fix only mode. I am not sure if that precludes doc fixes, on the chance that there will ever be a security fix, as it does non-security bug fixes, but is it hardly worth the bother compared to improving active releases. -- Terry Jan Reedy From dickinsm at gmail.com Mon Dec 13 20:39:32 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 13 Dec 2010 19:39:32 +0000 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: <20101213155119.A52A621CD91@kimball.webabinitio.net> References: <20101213114449.GA2348@chopin.edu.pl> <20101213155119.A52A621CD91@kimball.webabinitio.net> Message-ID: On Mon, Dec 13, 2010 at 3:51 PM, R. David Murray wrote: > It seems like the status quo is fine. ?I wouldn't object to it being > made more consistent. ?I would object to removing the existing cases. Same here, on all three counts. In one of the projects I'm currently working on, we've settled on a style that does quite a lot of: my_thing = Thing( foo = Foo(arg1, arg2, ...), bar = Bar(arg3, arg4, ...), ... ) and I've found the trailing comma very convenient during refactoring and API experimentation. (There's still good fun to be had arguing about the indentation of that closing parenthesis, though.) Mar From v+python at g.nevcal.com Mon Dec 13 21:08:31 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 13 Dec 2010 12:08:31 -0800 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> <20101213155119.A52A621CD91@kimball.webabinitio.net> Message-ID: <4D067D3F.8030206@g.nevcal.com> On 12/13/2010 11:39 AM, Mark Dickinson wrote: > my_thing = Thing( > foo = Foo(arg1, arg2, ...), > bar = Bar(arg3, arg4, ...), > ... > ) > > and I've found the trailing comma very convenient during refactoring > and API experimentation. (There's still good fun to be had arguing > about the indentation of that closing parenthesis, though.) Clearly it needs to be indented one level, because it is a continuation of the prior line, just like the foo and bar and ... lines are continuations and therefore indented. I'd have argued differently for languages that use {} to delimit blocks. Enjoy! From cs at zip.com.au Mon Dec 13 22:06:50 2010 From: cs at zip.com.au (Cameron Simpson) Date: Tue, 14 Dec 2010 08:06:50 +1100 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: <20101213201759.422b15de@pitrou.net> References: <20101213201759.422b15de@pitrou.net> Message-ID: <20101213210650.GA31778@cskk.homeip.net> On 13Dec2010 20:17, Antoine Pitrou wrote: | On Mon, 13 Dec 2010 14:09:02 -0500 | Alexander Belopolsky wrote: | | > On Mon, Dec 13, 2010 at 11:54 AM, Guido van Rossum wrote: | > > I'm at least +0 on | > > allowing trailing commas in the situation the OP mentioned. | > > | > | > FWIW, I am also about +0.5 on allowing trailing comma. Note that in a | > similar situation, the C standardization committee has erred on the | > side of consistency: | > | > """ | > A new feature of C99: a common extension in many implementations | > allows a trailing comma after the list of enumeration constants. The | > Committee decided to adopt this feature as an innocuous extension that | > mirrors the trailing commas allowed in initializers. | > """ http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf | > | > Similarly, I find allowing trailing comma in keyword only arguments | > lists to be an innocuous extension that mirrors the trailing commas | > allowed in the positional arguments lists. | | +1 from me as well. Special cases are hard to remember. +1 again. Both the special cases and probably an example of diff friendliness: x = f(a, b=3, ##c=4, hacking at dev time d=5) Cheers, -- Cameron Simpson DoD#743 http://www.cskk.ezoshosting.com/cs/ A Master is someone who started before you did. - Gary Zukav From tjreedy at udel.edu Mon Dec 13 22:21:20 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 13 Dec 2010 16:21:20 -0500 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: <20101213201759.422b15de@pitrou.net> References: <20101213114449.GA2348@chopin.edu.pl> <20101213201759.422b15de@pitrou.net> Message-ID: On 12/13/2010 2:17 PM, Antoine Pitrou wrote: > On Mon, 13 Dec 2010 14:09:02 -0500 > Alexander Belopolsky wrote: > >> On Mon, Dec 13, 2010 at 11:54 AM, Guido van Rossum wrote: >>> I'm at least +0 on >>> allowing trailing commas in the situation the OP mentioned. >>> >> >> FWIW, I am also about +0.5 on allowing trailing comma. Note that in a >> similar situation, the C standardization committee has erred on the >> side of consistency: >> >> """ >> A new feature of C99: a common extension in many implementations >> allows a trailing comma after the list of enumeration constants. The >> Committee decided to adopt this feature as an innocuous extension that >> mirrors the trailing commas allowed in initializers. >> """ http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf >> >> Similarly, I find allowing trailing comma in keyword only arguments >> lists to be an innocuous extension that mirrors the trailing commas >> allowed in the positional arguments lists. > > +1 from me as well. Special cases are hard to remember. Same here. A strong +1 for a consistent rule (always or never allowed) with a +1 for always given others use case of one param/arg per line. So I think the issues should be reopened. -- Terry Jan Reedy From raymond.hettinger at gmail.com Mon Dec 13 22:55:17 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 13 Dec 2010 13:55:17 -0800 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> <20101213201759.422b15de@pitrou.net> Message-ID: <0DE0FA27-5792-4017-A0B0-4B1AF109FA76@gmail.com> On Dec 13, 2010, at 1:21 PM, Terry Reedy wrote: > Same here. A strong +1 for a consistent rule (always or never allowed) with a +1 for always given others use case of one param/arg per line. It seems to me that a trailing comma in an argument list is more likely to be a user error than a deliberate comma-for-the-future. Raymond From guido at python.org Mon Dec 13 23:16:58 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 13 Dec 2010 14:16:58 -0800 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: <0DE0FA27-5792-4017-A0B0-4B1AF109FA76@gmail.com> References: <20101213114449.GA2348@chopin.edu.pl> <20101213201759.422b15de@pitrou.net> <0DE0FA27-5792-4017-A0B0-4B1AF109FA76@gmail.com> Message-ID: On Mon, Dec 13, 2010 at 1:55 PM, Raymond Hettinger wrote: > > On Dec 13, 2010, at 1:21 PM, Terry Reedy wrote: > >> Same here. A strong +1 for a consistent rule (always or never allowed) with a +1 for always given others use case of one param/arg per line. > > > > It seems to me that a trailing comma in an argument list is more likely to be a user error than a deliberate comma-for-the-future. Really? Have you observed this? Even if it was inserted by mistake, it is harmless. Python has a long tradition of allowing redundant trailing commas in comma-separated lists, and it is habit-forming. That's the issue the OP had: he expected it to work in the one context where it doesn't. -- --Guido van Rossum (python.org/~guido) From lukasz at langa.pl Mon Dec 13 23:22:29 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Mon, 13 Dec 2010 23:22:29 +0100 Subject: [Python-Dev] configparser 1.1 - one last bug fix needed Message-ID: <4A6BBCF7-6491-48C3-A02A-84EEA299B1DB@langa.pl> Hi there. There's one last thing that needs to be done with configparser for 3.2. Raymond, Fred, Michael and Georg already expressed their approval on that so unless anybody finds a flaw in the idea expressed below, I'm going to make the change for 3.2b2: - the ConfigParser class will be removed - the SafeConfigParser class will be renamed to ConfigParser - 2to3 will rename SafeConfigParser classes to ConfigParser - 2to3 will warn on the subtle behaviour change when ConfigParser classes are found What's the difference? ---------------------- Both ConfigParser and SafeConfigParser implement interpolation, e.g. option values can contain special tokens similar to those implemented by Python's string formatting: %(example)s. These tokens are replaced during get() operations by values from respective keys (either from the current section or from the DEFAULT section). SafeConfigParser was originally introduced to fix a couple of ConfigParser problems: - when a token didn't match the %(name)s syntax, it was simply treated as a raw value. This caused configuration errors like %var or %(no_closing_s) to be missed. - if someone actually wanted to store arbitrary strings in values, including Python formatting strings, there was no way to escape %(name)s in the configuration. The programmer had to know in advance that some value may hold %(name)s and only get() values from that option using get('section', 'option', raw=True) Then however, that option could not use interpolation anymore. - set() originally allowed to store non-string values in the parser. This was not meant to be a feature and caused trouble when the user tried to save the configuration to a file or get the stored values back using typed get() methods. SafeConfigParser solves these problems by validating interpolation syntax (only %(name)s or %% are allowed, the latter being an escaped percent sign) and raising exceptions on syntax errors, and validating type on set() operations so that no non-string values can be passed to the parser using the API. Why change that? ---------------- When ConfigParser was left alone, it remained the default choice for most end users, including our own distutils and logging libs. This was a very weak choice, and most current ConfigParser users are not aware of the interpolation quirks. I had to close a couple of issues related to people trying to store non-string values internally in the parser. The current situation needlessly complicates the documentation. Explaining all the above quirks to each new user who only wants to parse an INI file is weak at best. Moreover, users trust Python to do the right thing by default and according to their intuition. In this case, going for the default configparser.ConfigParser class without consulting the documentation is clearly a suboptimal choice. One last argument is that SafeConfigParser is an awkward name. It implicates the other parsers are somehow unsafe, or that this specific parser protects users from something. This is generally considered a naming antipattern. When? ----- You might ask whether this can be done for 3.2 (e.g. is that a feature or a bugfix). In Raymond's words, the beta process should be used to flesh out the APIs, test whether they work as expected and fix suboptimal decisions before we hit the release candidate stage. He consideres this essentially a bugfix and I agree. You might ask why do that now and not for 3.3. We believe that 3.2 is the last possible moment of introducing a change like that. The adoption rate is currently still low and application authors porting projects from 2.x expect incompatibilities. When they are non-issues, handled by 2to3, there's nothing to be afraid of. But isn't that... INCOMPATIBLE?! -------------------------------- Yes, it is. Thanks to the low py3k adoption rate now's the only moment where there's marginal risk of introducing silent incompatibility (there are hardly any py3k projects out there). Projects ported from Python 2.x will be informed by 2to3 of the change. We believe that this will fix more bugs than it introduces. Support for bare % signs would be the single case where ConfigParser might have appeared a more natural solution. In those cases we expect that users will rather choose to turn off interpolation whatsoever. Summary ------- If you have any strong, justified arguments against this bugfix, speak up. Otherwise the change will be made on Thursday. -- Interpolating regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ From chris at simplistix.co.uk Tue Dec 14 00:41:28 2010 From: chris at simplistix.co.uk (Chris Withers) Date: Mon, 13 Dec 2010 23:41:28 +0000 Subject: [Python-Dev] Using logging in the stdlib and its unit tests In-Reply-To: References: Message-ID: <4D06AF28.3040601@simplistix.co.uk> On 07/12/2010 20:26, Vinay Sajip wrote: > I would suggest that when unit testing, rather than adding StreamHandlers to log > to stderr, that something like TestHandler and Matcher from this post: > > http://plumberjack.blogspot.com/2010/09/unit-testing-and-logging.html For Python 2, my testfixtures package has had some helpful bits in this area for a while now: http://packages.python.org/testfixtures/logging.html I find it important to be able to check my code is logging what I think it should be logging! cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk From v+python at g.nevcal.com Tue Dec 14 00:46:46 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 13 Dec 2010 15:46:46 -0800 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: <0DE0FA27-5792-4017-A0B0-4B1AF109FA76@gmail.com> References: <20101213114449.GA2348@chopin.edu.pl> <20101213201759.422b15de@pitrou.net> <0DE0FA27-5792-4017-A0B0-4B1AF109FA76@gmail.com> Message-ID: <4D06B066.5090603@g.nevcal.com> On 12/13/2010 1:55 PM, Raymond Hettinger wrote: > It seems to me that a trailing comma in an argument list is more likely to be a user error than a deliberate comma-for-the-future. It seems to me that a trailing comma, especially in the case of one parameter per line, is a deliberate comma-for-the-future. It's a good reminder that you are dealing with a list of some sort, rather than a statement, when you look at just one parameter on the line. Especially if the ) is on the next line, which I prefer. Yes, a parameter list is not a python list, nor a python tuple, but it is still a generic, comma-separated list, and all such are more conveniently dealt with, in the multi-line case, if trailing commas are permitted. And, of course, the one-entry tuple needs the comma to differentiate it from some other expression, forcing the trailing comma into the syntax... so there can be no consistent rule for all commas that doesn't permit trailing commas. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 14 01:08:32 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Dec 2010 10:08:32 +1000 Subject: [Python-Dev] logging doc broken In-Reply-To: References: <4D028B7D.9030308@noir.com> <4D05F6DC.5040900@timgolden.me.uk> Message-ID: On Tue, Dec 14, 2010 at 5:34 AM, Terry Reedy wrote: > On 12/13/2010 6:04 AM, Vinay Sajip wrote: > > True, but this area changed after 2.6 was released (after even 2.7, IIRC), >> so I >> want to be sure that if I'm going to change the doc sources on >> release26-maint, >> > > 2.6 is in security-fix only mode. I am not sure if that precludes doc > fixes, on the chance that there will ever be a security fix, as it does > non-security bug fixes, but is it hardly worth the bother compared to > improving active releases. > Fixing it in the 2.7 maintenance branch will be sufficient to modify docs.python.org, which is likely the most important place to update. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 14 01:16:07 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Dec 2010 10:16:07 +1000 Subject: [Python-Dev] logging doc broken In-Reply-To: References: <4D028B7D.9030308@noir.com> Message-ID: On Tue, Dec 14, 2010 at 4:55 AM, Vinay Sajip wrote: > Nick Coghlan gmail.com> writes: > > > Yep, that's what I had assumed you were going to do (this did come up > > in the big thread about the logging docs, but you may have missed it). > > > > Ok, I've now checked in this change, and would be grateful for any > feedback. > Time for a break :-) > Hmm, that may not have built correctly, since it isn't showing up in the web version of the dev docs yet. I skimmed the diff on python-checkins though, and it looked good to me. I think the revised docs should make it *much* easier for people to get a handle on how the logging system works (I know my own understanding of it is significantly better than it was a couple of weeks ago). Cheers, Nick. P.S. I'm off visiting family for a couple of weeks, so my email access is going to be sketchy for a while. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 14 01:20:04 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Dec 2010 10:20:04 +1000 Subject: [Python-Dev] configparser 1.1 - one last bug fix needed In-Reply-To: <4A6BBCF7-6491-48C3-A02A-84EEA299B1DB@langa.pl> References: <4A6BBCF7-6491-48C3-A02A-84EEA299B1DB@langa.pl> Message-ID: +1 from me. If anyone complains too much, perhaps we can offer to retain the old ConfigParser as "_BuggyConfigParser"? (that idea is only partially tongue-in-cheek...) Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Tue Dec 14 01:26:44 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Tue, 14 Dec 2010 01:26:44 +0100 Subject: [Python-Dev] configparser 1.1 - one last bug fix needed In-Reply-To: References: <4A6BBCF7-6491-48C3-A02A-84EEA299B1DB@langa.pl> Message-ID: <16704819-23F0-4C2D-8A42-3C3D06FE99EF@langa.pl> Wiadomo?? napisana przez Nick Coghlan w dniu 2010-12-14, o godz. 01:20: > +1 from me. > > If anyone complains too much, perhaps we can offer to retain the old ConfigParser as "_BuggyConfigParser"? (that idea is only partially tongue-in-cheek...) > We may leave the LegacyInterpolation class if someone *really* needs to support that. That way RawConfigParser(interpolation=LegacyInterpolation()) is effectively the old ConfigParser. -- Nondeprecatorily yours, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 14 01:31:01 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 14 Dec 2010 10:31:01 +1000 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> <20101213155119.A52A621CD91@kimball.webabinitio.net> Message-ID: On Tue, Dec 14, 2010 at 5:39 AM, Mark Dickinson wrote: > On Mon, Dec 13, 2010 at 3:51 PM, R. David Murray > wrote: > > It seems like the status quo is fine. I wouldn't object to it being > > made more consistent. I would object to removing the existing cases. > > Same here, on all three counts. In one of the projects I'm currently > working on, we've settled on a style that does quite a lot of: > > my_thing = Thing( > foo = Foo(arg1, arg2, ...), > bar = Bar(arg3, arg4, ...), > ... > ) > > and I've found the trailing comma very convenient during refactoring > and API experimentation. (There's still good fun to be had arguing > about the indentation of that closing parenthesis, though.) > Another valid use case that occurred to me is building up a string-keyed dictionary: mapping = dict( x=1, y=2, z=3, ) So, on reflection, removing the existing cases where it is supported is certainly unreasonable, which makes the consistency argument that much stronger. For the record, I reopened issue #9232 (noting the lack of consensus), and (as someone suggested on the tracker) changed the resolution on the other one to be as a duplicate of #9232. Cheers, Nick. P.S. As I noted in the logging discussion, my email access is going to be a bit sketchy for the next couple of weeks. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Dec 14 01:34:40 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 14 Dec 2010 01:34:40 +0100 Subject: [Python-Dev] configparser 1.1 - one last bug fix needed References: <4A6BBCF7-6491-48C3-A02A-84EEA299B1DB@langa.pl> Message-ID: <20101214013440.290f2a37@pitrou.net> On Tue, 14 Dec 2010 10:20:04 +1000 Nick Coghlan wrote: > +1 from me. > > If anyone complains too much, perhaps we can offer to retain the old > ConfigParser as "_BuggyConfigParser"? (that idea is only partially > tongue-in-cheek...) Or we can put it in the "buggy" module which receives all buggy code. Regards Antoine. From vinay_sajip at yahoo.co.uk Tue Dec 14 02:22:43 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 14 Dec 2010 01:22:43 +0000 (UTC) Subject: [Python-Dev] logging doc broken References: <4D028B7D.9030308@noir.com> Message-ID: Nick Coghlan gmail.com> writes: > Hmm, that may not have built correctly, since it isn't showing up in the web > version of the dev docs yet. I skimmed the diff on python-checkins though, and > it looked good to me. I'll keep an eye on the build (built OK on my machine but I'm not sure what the build trigger is for docs.python.org). > I think the revised docs should make it *much* easier for people to get a > handle on how the logging system works (I know my own understanding of it is > significantly better than it was a couple of weeks ago). Glad to hear that. > P.S. I'm off visiting family for a couple of weeks, so my email access is > going to be sketchy for a while. Have a nice time, and thanks for all the help. Regards, Vinay From raymond.hettinger at gmail.com Tue Dec 14 07:21:33 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 13 Dec 2010 22:21:33 -0800 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> <20101213201759.422b15de@pitrou.net> <0DE0FA27-5792-4017-A0B0-4B1AF109FA76@gmail.com> Message-ID: <6406C2A8-3042-4D88-992D-7389C4C413FC@gmail.com> On Dec 13, 2010, at 2:16 PM, Guido van Rossum wrote: > On Mon, Dec 13, 2010 at 1:55 PM, Raymond Hettinger > wrote: >> >> It seems to me that a trailing comma in an argument list is more likely to be a user error than a deliberate comma-for-the-future. > > Really? Have you observed this? Even if it was inserted by mistake, it > is harmless. I only have one data point, my own mistakes. The SyntaxError has occasionally been helpful to me when working out a function signature or to detect a copy and paste error. In both cases, it meant that there was supposed to be another argument and it had been either forgotten or mispasted. Also, if I were reviewing someone else's code and saw a trailing comma in a function definition, it would seem weird and I would wonder if the author intended a different signature. FWIW, this isn't important to me at all. Was just noting my own experience. Don't put assign much weight to it, I don't have much of a preference either way. > Python has a long tradition of allowing redundant > trailing commas in comma-separated lists, and it is habit-forming. Right. I see that in the wild quite often and use it myself. Raymond From steve at pearwood.info Tue Dec 14 12:24:07 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 14 Dec 2010 22:24:07 +1100 Subject: [Python-Dev] configparser 1.1 - one last bug fix needed In-Reply-To: <20101214013440.290f2a37@pitrou.net> References: <4A6BBCF7-6491-48C3-A02A-84EEA299B1DB@langa.pl> <20101214013440.290f2a37@pitrou.net> Message-ID: <4D0753D7.9060102@pearwood.info> Antoine Pitrou wrote: > On Tue, 14 Dec 2010 10:20:04 +1000 > Nick Coghlan wrote: > >> +1 from me. >> >> If anyone complains too much, perhaps we can offer to retain the old >> ConfigParser as "_BuggyConfigParser"? (that idea is only partially >> tongue-in-cheek...) > > Or we can put it in the "buggy" module which receives all buggy code. The good thing about that idea is that maintenance of buggy.py will be so simple! -- Steven From fdrake at acm.org Tue Dec 14 12:38:51 2010 From: fdrake at acm.org (Fred Drake) Date: Tue, 14 Dec 2010 06:38:51 -0500 Subject: [Python-Dev] configparser 1.1 - one last bug fix needed In-Reply-To: <4D0753D7.9060102@pearwood.info> References: <4A6BBCF7-6491-48C3-A02A-84EEA299B1DB@langa.pl> <20101214013440.290f2a37@pitrou.net> <4D0753D7.9060102@pearwood.info> Message-ID: On Tue, Dec 14, 2010 at 6:24 AM, Steven D'Aprano wrote: > The good thing about that idea is that maintenance of buggy.py will be so > simple! It's self-describing, and needs no further documentation. :-) ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From raymond.hettinger at gmail.com Tue Dec 14 18:01:28 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 14 Dec 2010 09:01:28 -0800 Subject: [Python-Dev] configparser 1.1 - one last bug fix needed In-Reply-To: References: <4A6BBCF7-6491-48C3-A02A-84EEA299B1DB@langa.pl> <20101214013440.290f2a37@pitrou.net> <4D0753D7.9060102@pearwood.info> Message-ID: <7549C51E-0E6F-456F-9469-BBBECB5E8867@gmail.com> On Dec 14, 2010, at 3:38 AM, Fred Drake wrote: > On Tue, Dec 14, 2010 at 6:24 AM, Steven D'Aprano wrote: >> The good thing about that idea is that maintenance of buggy.py will be so >> simple! > > It's self-describing, and needs no further documentation. :-) > And psychologically more effective than deprecation. Anyone writing "import buggy" probably feels like they have got some explaining to do :-) Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From mail at timgolden.me.uk Tue Dec 14 18:06:20 2010 From: mail at timgolden.me.uk (Tim Golden) Date: Tue, 14 Dec 2010 17:06:20 +0000 Subject: [Python-Dev] configparser 1.1 - one last bug fix needed In-Reply-To: <7549C51E-0E6F-456F-9469-BBBECB5E8867@gmail.com> References: <4A6BBCF7-6491-48C3-A02A-84EEA299B1DB@langa.pl> <20101214013440.290f2a37@pitrou.net> <4D0753D7.9060102@pearwood.info> <7549C51E-0E6F-456F-9469-BBBECB5E8867@gmail.com> Message-ID: <4D07A40C.90507@timgolden.me.uk> On 14/12/2010 17:01, Raymond Hettinger wrote: > > On Dec 14, 2010, at 3:38 AM, Fred Drake wrote: > >> On Tue, Dec 14, 2010 at 6:24 AM, Steven D'Aprano wrote: >>> The good thing about that idea is that maintenance of buggy.py will be so >>> simple! >> >> It's self-describing, and needs no further documentation. :-) >> > > And psychologically more effective than deprecation. > Anyone writing "import buggy" probably feels like they > have got some explaining to do :-) They might be intro retro-robotics: http://www.anf.nildram.co.uk/beebcontrol/buggies/bbc_buggy/index.html (For those not in the UK, this was a very popular schools project kit on the back of the famous BBC micro initiative) TJG From ethan at stoneleaf.us Tue Dec 14 18:30:51 2010 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 14 Dec 2010 09:30:51 -0800 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> <20101213201759.422b15de@pitrou.net> Message-ID: <4D07A9CB.4060409@stoneleaf.us> Terry Reedy wrote: > On 12/13/2010 2:17 PM, Antoine Pitrou wrote: >> On Mon, 13 Dec 2010 14:09:02 -0500 >> Alexander Belopolsky wrote: >> >>> On Mon, Dec 13, 2010 at 11:54 AM, Guido van Rossum >>> wrote: >>>> I'm at least +0 on >>>> allowing trailing commas in the situation the OP mentioned. >>>> >>> >>> FWIW, I am also about +0.5 on allowing trailing comma. Note that in a >>> similar situation, the C standardization committee has erred on the >>> side of consistency: >>> >>> """ >>> A new feature of C99: a common extension in many implementations >>> allows a trailing comma after the list of enumeration constants. The >>> Committee decided to adopt this feature as an innocuous extension that >>> mirrors the trailing commas allowed in initializers. >>> """ http://www.open-std.org/jtc1/sc22/wg14/www/C99RationaleV5.10.pdf >>> >>> Similarly, I find allowing trailing comma in keyword only arguments >>> lists to be an innocuous extension that mirrors the trailing commas >>> allowed in the positional arguments lists. >> >> +1 from me as well. Special cases are hard to remember. > > Same here. A strong +1 for a consistent rule (always or never allowed) > with a +1 for always given others use case of one param/arg per line. +1 on consistency. +1 on allowing the trailing comma. ~Ethan~ From orsenthil at gmail.com Wed Dec 15 09:51:32 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Wed, 15 Dec 2010 16:51:32 +0800 Subject: [Python-Dev] Two urllib issues Message-ID: I know it is late to add features in beta release, but still I thought I would ask for a little leeway for these issues, especially as they don't change any API signatures. http://bugs.python.org/issue3243 Has patch, tests and docs http://bugs.python.org/issue1508475 I have patch ready and shall add the tests and docs too. Nothing is dependent on those changes, just that that it would be good to have. Any suggestions on the above? Georg, is it okay if I push this in before 17th? -- Senthil -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Dec 15 13:00:24 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 15 Dec 2010 13:00:24 +0100 Subject: [Python-Dev] r87253 - tracker/instances/python-dev/detectors/autonosy.py References: <20101215005337.D91D4EE9EF@mail.python.org> Message-ID: <20101215130024.31612572@pitrou.net> On Wed, 15 Dec 2010 01:53:37 +0100 (CET) ezio.melotti wrote: > Author: ezio.melotti > Date: Wed Dec 15 01:53:37 2010 > New Revision: 87253 > > Log: > #363: add automatically release managers to the nosy list for release blockers. Initial patch by Georg Brandl. You should probably add deferred blockers too. Antoine. From solipsis at pitrou.net Wed Dec 15 19:39:40 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 15 Dec 2010 19:39:40 +0100 Subject: [Python-Dev] Remove HTTP 0.9 support Message-ID: <20101215193940.31262534@pitrou.net> Hello, I would like to remove HTTP 0.9 support from http.client and http.server. I've opened an issue at http://bugs.python.org/issue10711 for that. Would anyone think it's a bad idea? (HTTP 1.0 was devised in 1996) Regards Antoine. From fdrake at acm.org Wed Dec 15 21:02:24 2010 From: fdrake at acm.org (Fred Drake) Date: Wed, 15 Dec 2010 15:02:24 -0500 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101215193940.31262534@pitrou.net> References: <20101215193940.31262534@pitrou.net> Message-ID: On Wed, Dec 15, 2010 at 1:39 PM, Antoine Pitrou wrote: > I would like to remove HTTP 0.9 support from http.client and > http.server. +1 ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From v+python at g.nevcal.com Wed Dec 15 21:58:51 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 15 Dec 2010 12:58:51 -0800 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101215193940.31262534@pitrou.net> References: <20101215193940.31262534@pitrou.net> Message-ID: <4D092C0B.8090300@g.nevcal.com> On 12/15/2010 10:39 AM, Antoine Pitrou wrote: > Hello, > > I would like to remove HTTP 0.9 support from http.client and > http.server. I've opened an issue at http://bugs.python.org/issue10711 > for that. Would anyone think it's a bad idea? > > (HTTP 1.0 was devised in 1996) Please address the following comment from the server.py source: # The default request version. This only affects responses up until # the point where the request line is parsed, so it mainly decides what # the client gets back when sending a malformed request line. # Most web servers default to HTTP 0.9, i.e. don't send a status line. default_request_version = "HTTP/0.9" I realize this is a somewhat obscure point, and in general, if your interest in http.client and http.server implies that some of the many outstanding bug reports for that code will get resolved, I have no concern for dropping support for HTTP 0.9 protocol, other than the above. Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Dec 15 22:25:10 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 15 Dec 2010 22:25:10 +0100 Subject: [Python-Dev] Remove HTTP 0.9 support References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> Message-ID: <20101215222510.0b8aa7e7@pitrou.net> On Wed, 15 Dec 2010 12:58:51 -0800 Glenn Linderman wrote: > On 12/15/2010 10:39 AM, Antoine Pitrou wrote: > > Hello, > > > > I would like to remove HTTP 0.9 support from http.client and > > http.server. I've opened an issue at http://bugs.python.org/issue10711 > > for that. Would anyone think it's a bad idea? > > > > (HTTP 1.0 was devised in 1996) > > Please address the following comment from the server.py source: > > # The default request version. This only affects responses up until > # the point where the request line is parsed, so it mainly decides what > # the client gets back when sending a malformed request line. > # Most web servers default to HTTP 0.9, i.e. don't send a status line. > default_request_version = "HTTP/0.9" What do you mean by "address"? The patch changes this to 1.0. And, as the comment says, this only affects what happens when the client sends a syntactically invalid request line, so whether the server does a 0.9-style or 1.0-style response is unimportant. Regards Antoine. From v+python at g.nevcal.com Wed Dec 15 23:20:54 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 15 Dec 2010 14:20:54 -0800 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101215222510.0b8aa7e7@pitrou.net> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> Message-ID: <4D093F46.6050106@g.nevcal.com> On 12/15/2010 1:25 PM, Antoine Pitrou wrote: > On Wed, 15 Dec 2010 12:58:51 -0800 > Glenn Linderman wrote: >> On 12/15/2010 10:39 AM, Antoine Pitrou wrote: >>> Hello, >>> >>> I would like to remove HTTP 0.9 support from http.client and >>> http.server. I've opened an issue at http://bugs.python.org/issue10711 >>> for that. Would anyone think it's a bad idea? >>> >>> (HTTP 1.0 was devised in 1996) >> Please address the following comment from the server.py source: >> >> # The default request version. This only affects responses up until >> # the point where the request line is parsed, so it mainly decides what >> # the client gets back when sending a malformed request line. >> # Most web servers default to HTTP 0.9, i.e. don't send a status line. >> default_request_version = "HTTP/0.9" > What do you mean by "address"? The patch changes this to 1.0. > And, as the comment says, this only affects what happens when the > client sends a syntactically invalid request line, so whether the server > does a 0.9-style or 1.0-style response is unimportant. Just what you did... justify the unimportance of not changing it :) Since now it is different than "most web servers". -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Dec 15 23:29:27 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 15 Dec 2010 23:29:27 +0100 Subject: [Python-Dev] Remove HTTP 0.9 support References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> Message-ID: <20101215232927.475f925c@pitrou.net> On Wed, 15 Dec 2010 14:20:54 -0800 Glenn Linderman wrote: > On 12/15/2010 1:25 PM, Antoine Pitrou wrote: > > On Wed, 15 Dec 2010 12:58:51 -0800 > > Glenn Linderman wrote: > >> On 12/15/2010 10:39 AM, Antoine Pitrou wrote: > >>> Hello, > >>> > >>> I would like to remove HTTP 0.9 support from http.client and > >>> http.server. I've opened an issue at http://bugs.python.org/issue10711 > >>> for that. Would anyone think it's a bad idea? > >>> > >>> (HTTP 1.0 was devised in 1996) > >> Please address the following comment from the server.py source: > >> > >> # The default request version. This only affects responses up until > >> # the point where the request line is parsed, so it mainly decides what > >> # the client gets back when sending a malformed request line. > >> # Most web servers default to HTTP 0.9, i.e. don't send a status line. > >> default_request_version = "HTTP/0.9" > > What do you mean by "address"? The patch changes this to 1.0. > > And, as the comment says, this only affects what happens when the > > client sends a syntactically invalid request line, so whether the server > > does a 0.9-style or 1.0-style response is unimportant. > > Just what you did... justify the unimportance of not changing it :) > Since now it is different than "most web servers". Well, I think the "most web servers" comment itself is outdated. Try e.g. www.mozilla.org or www.google.com or www.msn.com. (but www.python.org or www.apache.org still have the legacy behaviour) Regards Antoine. From fuzzyman at voidspace.org.uk Thu Dec 16 01:35:59 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 16 Dec 2010 00:35:59 +0000 Subject: [Python-Dev] [Python-checkins] r87296 - python/branches/py3k/Doc/library/test.rst In-Reply-To: <20101216002330.E0DBAEE993@mail.python.org> References: <20101216002330.E0DBAEE993@mail.python.org> Message-ID: On 16 December 2010 00:23, eric.araujo wrote: > Author: eric.araujo > Date: Thu Dec 16 01:23:30 2010 > New Revision: 87296 > > Log: > Advertise ?python -m? instead of direct filename. > > > Modified: > python/branches/py3k/Doc/library/test.rst > > Modified: python/branches/py3k/Doc/library/test.rst > > ============================================================================== > --- python/branches/py3k/Doc/library/test.rst (original) > +++ python/branches/py3k/Doc/library/test.rst Thu Dec 16 01:23:30 2010 > @@ -168,14 +168,14 @@ > > Running :mod:`test.regrtest` directly allows what resources are available > for > tests to use to be set. You do this by using the ``-u`` command-line > -option. Run :program:`python regrtest.py -uall` to turn on all > +option. Run :program:`python -m regrtest -uall` to turn on all > Shouldn't this be `python -m test.regrtest`, or even just `python -m test`? Michael > resources; specifying ``all`` as an option for ``-u`` enables all > possible resources. If all but one resource is desired (a more common > case), a > comma-separated list of resources that are not desired may be listed after > -``all``. The command :program:`python regrtest.py -uall,-audio,-largefile` > +``all``. The command :program:`python -m regrtest -uall,-audio,-largefile` > will run :mod:`test.regrtest` with all resources except the ``audio`` and > ``largefile`` resources. For a list of all resources and more command-line > -options, run :program:`python regrtest.py -h`. > +options, run :program:`python -m regrtest -h`. > > Some other ways to execute the regression tests depend on what platform > the > tests are being executed on. On Unix, you can run :program:`make test` at > the > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > -- http://www.voidspace.org.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: From merwok at netwok.org Thu Dec 16 02:07:43 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 16 Dec 2010 02:07:43 +0100 Subject: [Python-Dev] [Python-checkins] r87296 - python/branches/py3k/Doc/library/test.rst In-Reply-To: References: <20101216002330.E0DBAEE993@mail.python.org> Message-ID: <4D09665F.603@netwok.org> >> -option. Run :program:`python regrtest.py -uall` to turn on all >> +option. Run :program:`python -m regrtest -uall` to turn on all > > Shouldn't this be `python -m test.regrtest`, or even just `python -m test`? It should :) I was about to fix just that but then noticed I could remove more references to regrtest (after all just an implementation detail), so I rewrote the section title and first few lines of the section. My stupid email client forces rewrapping, so I?m attaching the diff as a separate file. I can repost the patch to the tracker if it?s inconvenient to review in email, but I?d like to commit soon since I have a big doc merge to do after this :) Best regards -------------- next part -------------- A non-text attachment was scrubbed... Name: fix-test.rst.diff Type: text/x-diff Size: 2611 bytes Desc: not available URL: From fuzzyman at voidspace.org.uk Thu Dec 16 02:13:42 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 16 Dec 2010 01:13:42 +0000 Subject: [Python-Dev] [Python-checkins] r87296 - python/branches/py3k/Doc/library/test.rst In-Reply-To: <4D09665F.603@netwok.org> References: <20101216002330.E0DBAEE993@mail.python.org> <4D09665F.603@netwok.org> Message-ID: <4D0967C6.4040408@voidspace.org.uk> On 16/12/2010 01:07, ?ric Araujo wrote: >>> -option. Run :program:`python regrtest.py -uall` to turn on all >>> +option. Run :program:`python -m regrtest -uall` to turn on all >> Shouldn't this be `python -m test.regrtest`, or even just `python -m test`? > It should :) I was about to fix just that but then noticed I could > remove more references to regrtest (after all just an implementation > detail), so I rewrote the section title and first few lines of the section. > > My stupid email client forces rewrapping, so I?m attaching the diff as a > separate file. I can repost the patch to the tracker if it?s > inconvenient to review in email, but I?d like to commit soon since I > have a big doc merge to do after this :) The diff looks good to me. All the best, Michael > Best regards -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From merwok at netwok.org Thu Dec 16 02:36:12 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 16 Dec 2010 02:36:12 +0100 Subject: [Python-Dev] [Python-checkins] r87299 - in python/branches/py3k: Doc/library/configparser.rst Lib/configparser.py Lib/test/test_cfgparser.py In-Reply-To: <20101216011622.871CDEE9D2@mail.python.org> References: <20101216011622.871CDEE9D2@mail.python.org> Message-ID: <4D096D0C.8020506@netwok.org> > Author: lukasz.langa > New Revision: 87299 > > Log: > Broken ConfigParser removed, SafeConfigParser renamed to ConfigParser. > Life is beatiful once again. IIIUC, this change makes bugs requesting use of SafeConfigParser in distutils and logging obsolete. > @@ -1139,6 +1122,6 @@ > > if __name__ == "__main__": > if "-c" in sys.argv: > - test_coverage('/tmp/cmd.cover') > + test_coverage('/tmp/configparser.cover') > else: > test_main() Consider using the tempfile module. You need to print the filename on stderr, I think. Alternatively, remove this custom functionality entirely and move it to regrtest or unittest. Cheers From merwok at netwok.org Thu Dec 16 02:41:26 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 16 Dec 2010 02:41:26 +0100 Subject: [Python-Dev] [Python-checkins] r87296 - python/branches/py3k/Doc/library/test.rst In-Reply-To: <4D0967C6.4040408@voidspace.org.uk> References: <20101216002330.E0DBAEE993@mail.python.org> <4D09665F.603@netwok.org> <4D0967C6.4040408@voidspace.org.uk> Message-ID: <4D096E46.20801@netwok.org> > The diff looks good to me. Committed as r87300, thank you sir! From merwok at netwok.org Thu Dec 16 03:41:48 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 16 Dec 2010 03:41:48 +0100 Subject: [Python-Dev] PEPs and versionadded Message-ID: <4D097C6C.1010109@netwok.org> Hi, I noticed that changes related to PEP 3147 and PEP 3149 in Doc haven?t been accompanied by versionadded/versionchanged directives. Is that on purpose, meaning that everyone should be aware of these PEPs when reading 3.2 docs, or just an oversight? Regards From barry at python.org Thu Dec 16 02:20:56 2010 From: barry at python.org (Barry Warsaw) Date: Wed, 15 Dec 2010 20:20:56 -0500 Subject: [Python-Dev] PEPs and versionadded In-Reply-To: <4D097C6C.1010109@netwok.org> References: <4D097C6C.1010109@netwok.org> Message-ID: <20101215202056.3665b462@mission> On Dec 16, 2010, at 03:41 AM, ?ric Araujo wrote: >I noticed that changes related to PEP 3147 and PEP 3149 in Doc haven?t >been accompanied by versionadded/versionchanged directives. > >Is that on purpose, meaning that everyone should be aware of these PEPs >when reading 3.2 docs, or just an oversight? It's an oversight. The changes should be accompanied by version* directives. I see you already committed a change for compileall.rst - thanks for that! Please feel free to add anything you find missing or submit a bug report and assign it to me. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: not available URL: From merwok at netwok.org Thu Dec 16 07:34:37 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 16 Dec 2010 07:34:37 +0100 Subject: [Python-Dev] PEPs and versionadded In-Reply-To: <20101215202056.3665b462@mission> References: <4D097C6C.1010109@netwok.org> <20101215202056.3665b462@mission> Message-ID: <4D09B2FD.7080806@netwok.org> > It's an oversight. The changes should be accompanied by version* directives. > I see you already committed a change for compileall.rst - thanks for that! > Please feel free to add anything you find missing or submit a bug report and > assign it to me. The PEP 3149 commits did not touch Doc, apart from sys.rst which got its versionadded added (duh) a short time after the code changes. I don?t know if the introduction of ABI flags ought to be documented more prominently; Doc/using seems still accurate, and Doc/distutils is rather vague about extensions, so I guess we?re good. PEP 3147 modified a handful of files in Doc; I?ve added missing directives in r87312. __cached__ is only described in library/runpy.rst and c-api/import.rst. If there is one canonical place to define what a module is (I couldn?t tell from http://docs.python.org/dev/genindex-M), I think it should be mentioned there. Copy-pasting the paragraph about __cached__ from the PEP would suffice. Both PEPs are well covered in whatsnew. Three cheers for Raymond! Regards From orsenthil at gmail.com Thu Dec 16 09:14:04 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Thu, 16 Dec 2010 16:14:04 +0800 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <4D093F46.6050106@g.nevcal.com> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> Message-ID: <20101216081404.GA4603@rubuntu> On Wed, Dec 15, 2010 at 02:20:54PM -0800, Glenn Linderman wrote: > On 12/15/2010 10:39 AM, Antoine Pitrou wrote: > I would like to remove HTTP 0.9 support from http.client and > http.server. I've opened an issue at http://bugs.python.org/issue10711 > for that. Would anyone think it's a bad idea? > > (HTTP 1.0 was devised in 1996) > > Please address the following comment from the server.py source: > > # The default request version. This only affects responses up until > # the point where the request line is parsed, so it mainly decides what > # the client gets back when sending a malformed request line. > # Most web servers default to HTTP 0.9, i.e. don't send a status line. > default_request_version = "HTTP/0.9" > > What do you mean by "address"? The patch changes this to 1.0. > And, as the comment says, this only affects what happens when the > client sends a syntactically invalid request line, so whether the server > does a 0.9-style or 1.0-style response is unimportant. > > > Just what you did... justify the unimportance of not changing it :) Since now > it is different than "most web servers". +1 to removing HTTP 0.9 related code in http.client and http.server. Until today, I hadn't cared to read any details of HTTP 0.9 except that I knew that some code was present in http library to support it. Reading a bit about the HTTP 0.9 'Internet Draft' written 1991 is enough to convince that, if we have to fall back on any old behavior falling back to HTTP 1.0 is perfectly fine. Regarding this comment # Most web servers default to HTTP 0.9, i.e. don't send a status line. Even HTTP 0.9 says that response SHOULD start with status line, but gives a suggestion that clients can "tolerate" bad server server behaviors when they don't send the status line and in that the case response is the body. http://www.w3.org/Protocols/HTTP/Response.html So, It cannot be associated with the behavior "most webservers", back then and even more so now. -- Senthil From orsenthil at gmail.com Thu Dec 16 09:14:56 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Thu, 16 Dec 2010 16:14:56 +0800 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101215232927.475f925c@pitrou.net> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101215232927.475f925c@pitrou.net> Message-ID: <20101216081456.GB4603@rubuntu> On Wed, Dec 15, 2010 at 11:29:27PM +0100, Antoine Pitrou wrote: > Well, I think the "most web servers" comment itself is outdated. > Try e.g. www.mozilla.org or www.google.com or www.msn.com. > (but www.python.org or www.apache.org still have the legacy behaviour) What legacy behavior did you observe in these? -- Senthil From nd at perlig.de Thu Dec 16 07:42:08 2010 From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=) Date: Thu, 16 Dec 2010 07:42:08 +0100 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101215193940.31262534@pitrou.net> References: <20101215193940.31262534@pitrou.net> Message-ID: <201012160742.08474@news.perlig.de> * Antoine Pitrou wrote: > Hello, > > I would like to remove HTTP 0.9 support from http.client and > http.server. I've opened an issue at http://bugs.python.org/issue10711 > for that. Would anyone think it's a bad idea? > > (HTTP 1.0 was devised in 1996) HTTP/0.9 support is still recommended (RFC 2616 is from 1999, but still current). I'm wondering, why you would consider touching that at all. Is it broken? Does it stand in the way of anything? If not, why throw away a feature? nd -- Already I've seen people (really!) write web URLs in the form: http:\\some.site.somewhere [...] How soon until greengrocers start writing "apples $1\pound" or something? -- Joona I Palaste in clc From dpritsos at extremepro.gr Thu Dec 16 12:09:41 2010 From: dpritsos at extremepro.gr (Dimitrios Pritsos) Date: Thu, 16 Dec 2010 13:09:41 +0200 Subject: [Python-Dev] Multiprocessing module - Synergeticprocessing (my custom module) - Pickling unPickling issues Message-ID: <4D09F375.40208@extremepro.gr> Hello Core Developers, My name is Dimitrios and I am newbie in python. I am working on a Project (part of my PhD) that is called Synergeticprocessing module. Initially is imitating the multiprocessing built in module but the processes are distributed on a LAN and not Locally. The main issue I have is with Pickle module. And I think I found some kind of BUG in the built in multiprocessing module. (Synergeticprocessing module is located at GitHub: https://github.com/dpritsos/synergeticprocessing) Starting with the "BUG". In case someone uses the multiprocessing.Pool of processes he/she has to face the problem of types.MehtodType Impossible pickling. That is you cannot dispatch an class instance method to the to the Process Pool. However, digging in to the Source Code of the module there are few lines that resolve this issue however are not activated or they are faultily activated so they do not work. This is the 'BUG' _@ ....../multiprocessing/forking.py_ . . . # # Try making some callable types picklable # from pickle import Pickler class ForkingPickler(Pickler): dispatch = Pickler.dispatch.copy() @classmethod def register(cls, type, reduce): def dispatcher(self, obj): rv = reduce(obj) self.save_reduce(obj=obj, *rv) cls.dispatch[type] = dispatcher def _reduce_method(m): if m.im_self is None: return getattr, (m.im_class, m.im_func.func_name) else: return getattr, (m.im_self, m.im_func.func_name) ForkingPickler.register(type(ForkingPickler.save), _reduce_method) def _reduce_method_descriptor(m): return getattr, (m.__objclass__, m.__name__) ForkingPickler.register(type(list.append), _reduce_method_descriptor) ForkingPickler.register(type(int.__add__), _reduce_method_descriptor) #def _reduce_builtin_function_or_method(m): # return getattr, (m.__self__, m.__name__) #ForkingPickler.register(type(list().append), _reduce_builtin_function_or_method) #ForkingPickler.register(type(int().__add__), _reduce_builtin_function_or_method) . . . The RED lines are not doing the job, for some reason they are not managing to register the GREEN function as a global reduce/pickling function even if you call the registration function into you __main__ script. The solution I found is just to do this * import copy_reg import types* def _reduce_method(m): if m.im_self is None: return getattr, (m.im_class, m.im_func.func_name) else: return getattr, (m.im_self, m.im_func.func_name) *copy_reg.pickle(types.MethodType, _reduce_method)* . . . Doing that everything works FINE. But ONLY for local methods i.e. the ones that their class is defined on the __main__ script or other import-ed. In case you want to send something remotely (in an other machine) or to an other __main__ script running separately then you get a message like this: 'module' object has no attribute '' The only way to resolve this is firstly to import a script that has defined there and everything works fine. SO the problems it seems to be that the *m.im_class* (look code above) has some attribute __module__ defined as __module__ = '__main__' or something like that. And this is the reason why remote script cannot execute the function. I mean that the _reduce_method() above DOES is pickling the whole CLASS object so there is no reason not to be executed at the remote script. Besides it does as mentioned above in you just import this the user defined class form an other script. I have already spent about 12 weeks working on building my synergeticPool and resolve the issue of Pickling and only 2 days needed for the code of the Pool and the rest of the time was spent for the Pickling issues, and study all the Class related mechanics of python. That was the reason I ve started digging the multipocessessing module and found this say 'BUG', and finally sent this email. Best Regards, Dimitrios -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Thu Dec 16 13:29:59 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 16 Dec 2010 12:29:59 +0000 Subject: [Python-Dev] Multiprocessing module - Synergeticprocessing (my custom module) - Pickling unPickling issues In-Reply-To: <4D09F375.40208@extremepro.gr> References: <4D09F375.40208@extremepro.gr> Message-ID: <4D0A0647.2050900@voidspace.org.uk> On 16/12/2010 11:09, Dimitrios Pritsos wrote: > > Hello Core Developers, > > My name is Dimitrios and I am newbie in python. I am working on a > Project (part of my PhD) that is called Synergeticprocessing module. > Initially is imitating the multiprocessing built in module but the > processes are distributed on a LAN and not Locally. The main issue I > have is with Pickle module. And I think I found some kind of BUG in > the built in multiprocessing module. > Hello Dimitrios, Please post your bug report to the Python bug tracker. As you think you have a fix for the issue it is much more likely to be applied quickly if you can package it in the form of a test that demonstrates the issue and a patch that fixes it. http://bugs.python.org/ All the best, Michael Foord > (Synergeticprocessing module is located at GitHub: > https://github.com/dpritsos/synergeticprocessing) > > Starting with the "BUG". In case someone uses the multiprocessing.Pool > of processes he/she has to face the problem of types.MehtodType > Impossible pickling. That is you cannot dispatch an class instance > method to the to the Process Pool. However, digging in to the Source > Code of the module there are few lines that resolve this issue however > are not activated or they are faultily activated so they do not work. > This is the 'BUG' > > _@ ....../multiprocessing/forking.py_ > > > . > . > . > > # > # Try making some callable types picklable > # > > from pickle import Pickler > class ForkingPickler(Pickler): > dispatch = Pickler.dispatch.copy() > > @classmethod > def register(cls, type, reduce): > def dispatcher(self, obj): > rv = reduce(obj) > self.save_reduce(obj=obj, *rv) > cls.dispatch[type] = dispatcher > > def _reduce_method(m): > if m.im_self is None: > return getattr, (m.im_class, m.im_func.func_name) > else: > return getattr, (m.im_self, m.im_func.func_name) > ForkingPickler.register(type(ForkingPickler.save), _reduce_method) > > def _reduce_method_descriptor(m): > return getattr, (m.__objclass__, m.__name__) > ForkingPickler.register(type(list.append), _reduce_method_descriptor) > ForkingPickler.register(type(int.__add__), _reduce_method_descriptor) > > #def _reduce_builtin_function_or_method(m): > # return getattr, (m.__self__, m.__name__) > #ForkingPickler.register(type(list().append), > _reduce_builtin_function_or_method) > #ForkingPickler.register(type(int().__add__), > _reduce_builtin_function_or_method) > . > . > . > > The RED lines are not doing the job, for some reason they are not > managing to register the GREEN function as a global reduce/pickling > function even if you call the registration function into you __main__ > script. > > The solution I found is just to do this > * > import copy_reg > import types* > > def _reduce_method(m): > if m.im_self is None: > return getattr, (m.im_class, m.im_func.func_name) > else: > return getattr, (m.im_self, m.im_func.func_name) > > *copy_reg.pickle(types.MethodType, _reduce_method)* > . > . > . > > Doing that everything works FINE. But ONLY for local methods i.e. the > ones that their class is defined on the __main__ script or other > import-ed. > > In case you want to send something remotely (in an other machine) or > to an other __main__ script running separately then you get a message > like this: > > 'module' object has no attribute '' > > The only way to resolve this is firstly to import a script that has > defined there and everything works fine. > > SO the problems it seems to be that the *m.im_class* (look code > above) has some attribute __module__ defined as __module__ = > '__main__' or something like that. And this is the reason why remote > script cannot execute the function. I mean that the _reduce_method() > above DOES is pickling the whole CLASS object so there is no reason > not to be executed at the remote script. Besides it does as mentioned > above in you just import this the user defined class form an other > script. > > > I have already spent about 12 weeks working on building my > synergeticPool and resolve the issue of Pickling and only 2 days > needed for the code of the Pool and the rest of the time was spent for > the Pickling issues, and study all the Class related mechanics of > python. That was the reason I ve started digging the multipocessessing > module and found this say 'BUG', and finally sent this email. > > Best Regards, > > > Dimitrios > > > > > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Dec 16 14:20:37 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Dec 2010 14:20:37 +0100 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101216081456.GB4603@rubuntu> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101215232927.475f925c@pitrou.net> <20101216081456.GB4603@rubuntu> Message-ID: <1292505637.3689.0.camel@localhost.localdomain> Le jeudi 16 d?cembre 2010 ? 16:14 +0800, Senthil Kumaran a ?crit : > On Wed, Dec 15, 2010 at 11:29:27PM +0100, Antoine Pitrou wrote: > > Well, I think the "most web servers" comment itself is outdated. > > Try e.g. www.mozilla.org or www.google.com or www.msn.com. > > (but www.python.org or www.apache.org still have the legacy behaviour) > > What legacy behavior did you observe in these? -> Request: xyzzy -> Response: 405 Method Not Allowed

Method Not Allowed

The requested method xyzzy is not allowed for the URL /.

Apache/2.3.8 (Unix) mod_ssl/2.3.8 OpenSSL/1.0.0a Server at www.apache.org Port 80
(notice how the response has no headers) From dpritsos at extremepro.gr Thu Dec 16 14:52:19 2010 From: dpritsos at extremepro.gr (Dimitrios Pritsos) Date: Thu, 16 Dec 2010 15:52:19 +0200 Subject: [Python-Dev] Multiprocessing module - Synergeticprocessing (my custom module) - Pickling unPickling issues In-Reply-To: <4D0A0647.2050900@voidspace.org.uk> References: <4D09F375.40208@extremepro.gr> <4D0A0647.2050900@voidspace.org.uk> Message-ID: <4D0A1993.2030309@extremepro.gr> On 12/16/2010 02:29 PM, Michael Foord wrote: > On 16/12/2010 11:09, Dimitrios Pritsos wrote: >> >> Hello Core Developers, >> >> My name is Dimitrios and I am newbie in python. I am working on a >> Project (part of my PhD) that is called Synergeticprocessing module. >> Initially is imitating the multiprocessing built in module but the >> processes are distributed on a LAN and not Locally. The main issue I >> have is with Pickle module. And I think I found some kind of BUG in >> the built in multiprocessing module. >> > > Hello Dimitrios, > > Please post your bug report to the Python bug tracker. As you think > you have a fix for the issue it is much more likely to be applied > quickly if you can package it in the form of a test that demonstrates > the issue and a patch that fixes it. > > http://bugs.python.org/ > > All the best, > > Michael Foord > Hello Michael, OK I will do sent it to the bug tracker. But what about the last issue i.e. that even if the class is transfered-and-pickled-unpickled it raises an exception if the class is defined into the __main__ script and not imported from an other one. (see below) could you please guide me where should I post this for someone to help me fix it! Best Regards, Dimitrios > > >> (Synergeticprocessing module is located at GitHub: >> https://github.com/dpritsos/synergeticprocessing) >> >> Starting with the "BUG". In case someone uses the >> multiprocessing.Pool of processes he/she has to face the problem of >> types.MehtodType Impossible pickling. That is you cannot dispatch an >> class instance method to the to the Process Pool. However, digging in >> to the Source Code of the module there are few lines that resolve >> this issue however are not activated or they are faultily activated >> so they do not work. This is the 'BUG' >> >> _@ ....../multiprocessing/forking.py_ >> >> >> . >> . >> . >> >> # >> # Try making some callable types picklable >> # >> >> from pickle import Pickler >> class ForkingPickler(Pickler): >> dispatch = Pickler.dispatch.copy() >> >> @classmethod >> def register(cls, type, reduce): >> def dispatcher(self, obj): >> rv = reduce(obj) >> self.save_reduce(obj=obj, *rv) >> cls.dispatch[type] = dispatcher >> >> def _reduce_method(m): >> if m.im_self is None: >> return getattr, (m.im_class, m.im_func.func_name) >> else: >> return getattr, (m.im_self, m.im_func.func_name) >> ForkingPickler.register(type(ForkingPickler.save), _reduce_method) >> >> def _reduce_method_descriptor(m): >> return getattr, (m.__objclass__, m.__name__) >> ForkingPickler.register(type(list.append), _reduce_method_descriptor) >> ForkingPickler.register(type(int.__add__), _reduce_method_descriptor) >> >> #def _reduce_builtin_function_or_method(m): >> # return getattr, (m.__self__, m.__name__) >> #ForkingPickler.register(type(list().append), >> _reduce_builtin_function_or_method) >> #ForkingPickler.register(type(int().__add__), >> _reduce_builtin_function_or_method) >> . >> . >> . >> >> The RED lines are not doing the job, for some reason they are not >> managing to register the GREEN function as a global reduce/pickling >> function even if you call the registration function into you __main__ >> script. >> >> The solution I found is just to do this >> * >> import copy_reg >> import types* >> >> def _reduce_method(m): >> if m.im_self is None: >> return getattr, (m.im_class, m.im_func.func_name) >> else: >> return getattr, (m.im_self, m.im_func.func_name) >> >> *copy_reg.pickle(types.MethodType, _reduce_method)* >> . >> . >> . >> >> Doing that everything works FINE. But ONLY for local methods i.e. the >> ones that their class is defined on the __main__ script or other >> import-ed. >> >> In case you want to send something remotely (in an other machine) or >> to an other __main__ script running separately then you get a message >> like this: >> >> 'module' object has no attribute '' >> >> The only way to resolve this is firstly to import a script that has >> defined there and everything works fine. >> >> SO the problems it seems to be that the *m.im_class* (look code >> above) has some attribute __module__ defined as __module__ = >> '__main__' or something like that. And this is the reason why remote >> script cannot execute the function. I mean that the _reduce_method() >> above DOES is pickling the whole CLASS object so there is no reason >> not to be executed at the remote script. Besides it does as mentioned >> above in you just import this the user defined class form an other >> script. >> >> >> I have already spent about 12 weeks working on building my >> synergeticPool and resolve the issue of Pickling and only 2 days >> needed for the code of the Pool and the rest of the time was spent >> for the Pickling issues, and study all the Class related mechanics of >> python. That was the reason I ve started digging the >> multipocessessing module and found this say 'BUG', and finally sent >> this email. >> >> Best Regards, >> >> >> Dimitrios >> >> >> >> >> >> >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > -- > > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > May you share freely, never taking more than you give. > -- the sqlite blessinghttp://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Dec 16 15:00:34 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Dec 2010 15:00:34 +0100 Subject: [Python-Dev] Multiprocessing module - Synergeticprocessing (my custom module) - Pickling unPickling issues References: <4D09F375.40208@extremepro.gr> <4D0A0647.2050900@voidspace.org.uk> <4D0A1993.2030309@extremepro.gr> Message-ID: <20101216150034.04711183@pitrou.net> On Thu, 16 Dec 2010 15:52:19 +0200 Dimitrios Pritsos wrote: > > Hello Michael, > > OK I will do sent it to the bug tracker. But what about the last issue > i.e. that even if the class is transfered-and-pickled-unpickled it > raises an exception if the class is defined into the __main__ script and > not imported from an other one. (see below) could you please guide me > where should I post this for someone to help me fix it! The likely explanation is that the __main__ module in the parent process isn't the same as in the child process, so fetching the class from that module doesn't work. If that's really important for you you could open a separate issue. Regards Antoine. From solipsis at pitrou.net Thu Dec 16 15:23:05 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Dec 2010 15:23:05 +0100 Subject: [Python-Dev] Remove HTTP 0.9 support References: <20101215193940.31262534@pitrou.net> <201012160742.08474@news.perlig.de> Message-ID: <20101216152305.13b1fd3a@pitrou.net> On Thu, 16 Dec 2010 07:42:08 +0100 Andr? Malo wrote: > * Antoine Pitrou wrote: > > > Hello, > > > > I would like to remove HTTP 0.9 support from http.client and > > http.server. I've opened an issue at http://bugs.python.org/issue10711 > > for that. Would anyone think it's a bad idea? > > > > (HTTP 1.0 was devised in 1996) > > HTTP/0.9 support is still recommended (RFC 2616 is from 1999, but still > current). > > I'm wondering, why you would consider touching that at all. Is it broken? > Does it stand in the way of anything? If not, why throw away a feature? Well, it complicates maintenance and makes fixing issues such as http://bugs.python.org/issue6791 less likely. Note that the patch still accepts servers and clients which advertise themselves as 0.9 (using "HTTP/0.9" as a version string). It just removes support for the style of "simple response" without headers that 0.9 allowed. Regards Antoine. From exarkun at twistedmatrix.com Thu Dec 16 16:25:56 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 16 Dec 2010 15:25:56 -0000 Subject: [Python-Dev] Multiprocessing module - Synergeticprocessing (my custom module) - Pickling unPickling issues In-Reply-To: <20101216150034.04711183@pitrou.net> References: <4D09F375.40208@extremepro.gr> <4D0A0647.2050900@voidspace.org.uk> <4D0A1993.2030309@extremepro.gr> <20101216150034.04711183@pitrou.net> Message-ID: <20101216152556.2058.1297404892.divmod.xquotient.262@localhost.localdomain> On 02:00 pm, solipsis at pitrou.net wrote: >On Thu, 16 Dec 2010 15:52:19 +0200 >Dimitrios Pritsos wrote: >> >>Hello Michael, >> >>OK I will do sent it to the bug tracker. But what about the last issue >>i.e. that even if the class is transfered-and-pickled-unpickled it >>raises an exception if the class is defined into the __main__ script >>and >>not imported from an other one. (see below) could you please guide me >>where should I post this for someone to help me fix it! > >The likely explanation is that the __main__ module in the parent >process isn't the same as in the child process, so fetching the class >from that module doesn't work. >If that's really important for you you could open a separate issue. And another option is to use the simple, pleasant fix of not defining any functions or classes in __main__. Define them in a real module and import them for use in __main__. Jean-Paul From nd at perlig.de Thu Dec 16 16:52:43 2010 From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=) Date: Thu, 16 Dec 2010 16:52:43 +0100 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101216152305.13b1fd3a@pitrou.net> References: <20101215193940.31262534@pitrou.net> <201012160742.08474@news.perlig.de> <20101216152305.13b1fd3a@pitrou.net> Message-ID: <201012161652.43949.nd@perlig.de> On Thursday 16 December 2010 15:23:05 Antoine Pitrou wrote: > On Thu, 16 Dec 2010 07:42:08 +0100 > > Andr? Malo wrote: > > * Antoine Pitrou wrote: > > > Hello, > > > > > > I would like to remove HTTP 0.9 support from http.client and > > > http.server. I've opened an issue at http://bugs.python.org/issue10711 > > > for that. Would anyone think it's a bad idea? > > > > > > (HTTP 1.0 was devised in 1996) > > > > HTTP/0.9 support is still recommended (RFC 2616 is from 1999, but still > > current). > > > > I'm wondering, why you would consider touching that at all. Is it broken? > > Does it stand in the way of anything? If not, why throw away a feature? > > Well, it complicates maintenance and makes fixing issues such as > http://bugs.python.org/issue6791 less likely. I'd vote for removing it from the client code and keeping it in the server. > Note that the patch still accepts servers and clients which advertise > themselves as 0.9 (using "HTTP/0.9" as a version string). HTTP/0.9 doesn't *have* a version string. GET /foo is a HTTP/0.9 request. GET /foo HTTP/0.9 isn't actually (it's a paradoxon, alright ;). It simply isn't a valid HTTP request, which would demand a 505 response. nd From rdmurray at bitdance.com Thu Dec 16 17:04:46 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 16 Dec 2010 11:04:46 -0500 Subject: [Python-Dev] [Python-checkins] r87310 - in python/branches/release27-maint: Doc/library/compileall.rst Lib/compileall.py In-Reply-To: <20101216061502.43686EE992@mail.python.org> References: <20101216061502.43686EE992@mail.python.org> Message-ID: <20101216160446.40E9D1FFBDE@kimball.webabinitio.net> On Thu, 16 Dec 2010 07:15:02 +0100, eric.araujo wrote: > Modified: python/branches/release27-maint/Doc/library/compileall.rst > ============================================================================== > --- python/branches/release27-maint/Doc/library/compileall.rst (original) > +++ python/branches/release27-maint/Doc/library/compileall.rst Thu Dec 16 07:15:02 2010 > @@ -1,4 +1,3 @@ > - > :mod:`compileall` --- Byte-compile Python libraries > =================================================== > > @@ -50,14 +49,14 @@ > > Expand list with its content (file and directory names). I realize you didn't write this line, but note that '-' is accepted as an argument and means "read the list from stdin". > -.. versionadded:: 2.7 > - The ``-i`` option. > +.. versionchanged:: 2.7 > + Added the ``-i`` option. > > > Public functions > ---------------- > > -.. function:: compile_dir(dir[, maxlevels[, ddir[, force[, rx[, quiet]]]]]) > +.. function:: compile_dir(dir[, maxlevels[, ddir[, force[, rx[, quiet]]]]]) > > Recursively descend the directory tree named by *dir*, compiling all :file:`.py` > files along the way. The *maxlevels* parameter is used to limit the depth of > @@ -72,6 +71,23 @@ > If *quiet* is true, nothing is printed to the standard output in normal > operation. > > + > +.. function:: compile_file(fullname[, ddir[, force[, rx[, quiet]]]]) > + > + Compile the file with path *fullname*. If *ddir* is given, it is used as the > + base path from which the filename used in error messages will be generated. > + If *force* is true, modules are re-compiled even if the timestamp is up to > + date. Although this is copied from the other descriptions of *ddir*, it and the other instances (and the description of the -d option) should all really be fixed. As I discovered when writing the tests for the -d option, what ddir is is the path that is "baked in" to the .pyc file. In very old versions of Python that meant it was the path that would show up in tracebacks as the path to the source file. In modern Pythons the ddir path shows up if and only if the .py file does not exist and the .pyc file is being run directly. In 3.2, this means it will never show up normally, since you can't even run the .pyc file without moving it out of __pycache__. Which means 'ddir' is henceforth useful only to those people who want to package sourceless distributions of the python code. (If you want to see this in action check out the -d tests in test_compileall.) So, 'in error messages' really means 'in tracebacks if the .py file doesn't exist'. -- R. David Murray www.bitdance.com From fdrake at acm.org Thu Dec 16 17:20:04 2010 From: fdrake at acm.org (Fred Drake) Date: Thu, 16 Dec 2010 11:20:04 -0500 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <201012161652.43949.nd@perlig.de> References: <20101215193940.31262534@pitrou.net> <201012160742.08474@news.perlig.de> <20101216152305.13b1fd3a@pitrou.net> <201012161652.43949.nd@perlig.de> Message-ID: On Thu, Dec 16, 2010 at 10:52 AM, Andr? Malo wrote: > I'd vote for removing it from the client code and keeping it in the server. If it must be maintained anywhere, it should be in the client, according to the basic principle of "accept what you can, generate carefully." Python.org's HTTP/0.9 responses appear to be in response to HTTP/0.9 requests only. A request claiming to be HTTP 1.0, but without a Host: header, gets a redirect to the same page. I'm still in favor of removing HTTP 0.9 support entirely. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From merwok at netwok.org Thu Dec 16 17:41:22 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 16 Dec 2010 17:41:22 +0100 Subject: [Python-Dev] [Python-checkins] r87310 - in python/branches/release27-maint: Doc/library/compileall.rst Lib/compileall.py In-Reply-To: <20101216160446.40E9D1FFBDE@kimball.webabinitio.net> References: <20101216061502.43686EE992@mail.python.org> <20101216160446.40E9D1FFBDE@kimball.webabinitio.net> Message-ID: <4D0A4132.5020407@netwok.org> Hi, Thanks for double-checking. When I first looked into compileall, I opened http://bugs.python.org/issue10454 where I state that I find the description of those options unclear or even not understandable, so your diagnosis that I just copied text is right. A rewrite to fully cover the module functionality in clear English is needed. Your email was very helpful; can you turn it into a patch and post it to the bug? Thanks in advance. Regards From orsenthil at gmail.com Thu Dec 16 17:52:14 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Fri, 17 Dec 2010 00:52:14 +0800 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <1292505637.3689.0.camel@localhost.localdomain> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101215232927.475f925c@pitrou.net> <20101216081456.GB4603@rubuntu> <1292505637.3689.0.camel@localhost.localdomain> Message-ID: <20101216165214.GA7964@rubuntu> On Thu, Dec 16, 2010 at 02:20:37PM +0100, Antoine Pitrou wrote: > > > Try e.g. www.mozilla.org or www.google.com or www.msn.com. > > > (but www.python.org or www.apache.org still have the legacy behaviour) > > > > What legacy behavior did you observe in these? > > -> Request: > xyzzy > > -> Response: > > (notice how the response has no headers) > Well, Error response without headers was observed in www.mozilla.org and www.google.com for Invalid requests. But, I observed something surprising at www.apache.org If you do GET / HTTP/1.0 You do get the valid Response with headers. But if you do GET / You get a valid response Without headers. I was afraid if this behavior was to support HTTP 0.9 style where the the reponse is sent without the headers. Actually, it is turning out to be true: http://ftp.ics.uci.edu/pub/ietf/http/rfc1945.html#Response According to HTTP 1.0, When a request is Simple-Request, it means a VERB URL (without a version) and it generally corresponds to HTTP 0.9 And when a server receives such a Simple-Request, it sends a Simple-Response where it does not send the headers back. I think, the same is exhibited by other Servers as well www.google.com, www.mozilla.org where for Invalid Request without version, you are sending a Simple-Request (HTTP 0.9) style and getting the corresponding response. Given these, any assumption that servers no longer support HTTP/0.9 becomes false. So nuking it will require some thought. -- Senthil -- From foom at fuhm.net Thu Dec 16 17:21:37 2010 From: foom at fuhm.net (James Y Knight) Date: Thu, 16 Dec 2010 11:21:37 -0500 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101216081404.GA4603@rubuntu> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101216081404.GA4603@rubuntu> Message-ID: <892B9BB4-C2F4-43FF-9CA4-98A2A8A51EDA@fuhm.net> On Dec 16, 2010, at 3:14 AM, Senthil Kumaran wrote: > Even HTTP 0.9 says that response SHOULD start with status line, but > gives a suggestion that clients can "tolerate" bad server server > behaviors when they don't send the status line and in that the case > response is the body. > > http://www.w3.org/Protocols/HTTP/Response.html > > So, It cannot be associated with the behavior "most webservers", back > then and even more so now. Actually no. That document is describing almost-HTTP 1.0. Here is the actual document you were looking for: http://www.w3.org/Protocols/HTTP/AsImplemented.html HTTP 0.9 had no headers, no status line, nothing but "GET $url " and a stream of data in response. James From orsenthil at gmail.com Thu Dec 16 17:56:13 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Fri, 17 Dec 2010 00:56:13 +0800 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <892B9BB4-C2F4-43FF-9CA4-98A2A8A51EDA@fuhm.net> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101216081404.GA4603@rubuntu> <892B9BB4-C2F4-43FF-9CA4-98A2A8A51EDA@fuhm.net> Message-ID: <20101216165613.GB7964@rubuntu> On Thu, Dec 16, 2010 at 11:21:37AM -0500, James Y Knight wrote: > > Even HTTP 0.9 says that response SHOULD start with status line, but > > gives a suggestion that clients can "tolerate" bad server server > > behaviors when they don't send the status line and in that the case > > response is the body. > > > > http://www.w3.org/Protocols/HTTP/Response.html > > > > So, It cannot be associated with the behavior "most webservers", back > > then and even more so now. > > Actually no. That document is describing almost-HTTP 1.0. Yeah. I know it was almost-HTTP 1.0, but the same docs say that if protocol version was not specified, it is assumed to be 0.9. So, I thought it was a good reference point to understand the behavior. > Here is the actual document you were looking for: > http://www.w3.org/Protocols/HTTP/AsImplemented.html > > HTTP 0.9 had no headers, no status line, nothing but "GET $url " and a stream of data in response. Actually, you are right. I seems be be actual defined behavior of HTTP 0.9. As explained in that above doc and also in RFC 1945 Request section. -- Senthil From orsenthil at gmail.com Thu Dec 16 17:58:00 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Fri, 17 Dec 2010 00:58:00 +0800 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <201012161652.43949.nd@perlig.de> References: <20101215193940.31262534@pitrou.net> <201012160742.08474@news.perlig.de> <20101216152305.13b1fd3a@pitrou.net> <201012161652.43949.nd@perlig.de> Message-ID: <20101216165800.GC7964@rubuntu> On Thu, Dec 16, 2010 at 04:52:43PM +0100, Andr? Malo wrote: > HTTP/0.9 doesn't *have* a version string. > > GET /foo > > is a HTTP/0.9 request. > > GET /foo HTTP/0.9 > > isn't actually (it's a paradoxon, alright ;). It simply isn't a valid HTTP > request, which would demand a 505 response. Yes, this is an important point. Many webservers seem to exhibit the HTTP 0.9 response behaviors when you don't specify the version. -- Senthil From solipsis at pitrou.net Thu Dec 16 18:02:45 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 Dec 2010 18:02:45 +0100 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101216165214.GA7964@rubuntu> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101215232927.475f925c@pitrou.net> <20101216081456.GB4603@rubuntu> <1292505637.3689.0.camel@localhost.localdomain> <20101216165214.GA7964@rubuntu> Message-ID: <20101216180245.19251acf@pitrou.net> On Fri, 17 Dec 2010 00:52:14 +0800 Senthil Kumaran wrote: > Actually, it is turning out to be true: > > http://ftp.ics.uci.edu/pub/ietf/http/rfc1945.html#Response > > According to HTTP 1.0, When a request is Simple-Request, it means a > VERB URL (without a version) and it generally corresponds to HTTP 0.9 > And when a server receives such a Simple-Request, it sends a > Simple-Response where it does not send the headers back. > > I think, the same is exhibited by other Servers as well > www.google.com, www.mozilla.org where for Invalid Request without > version, you are sending a Simple-Request (HTTP 0.9) style and getting > the corresponding response. Yes, but only error or redirect responses: $ nc www.google.fr 80 GET / HTTP/1.0 302 Found Location: http://www.google.fr/ [etc.] $ nc www.mozilla.org 80 GET / 403 Forbidden

Forbidden

You don't have permission to access /error/noindex.html on this server.

That's quite understandable, since most HTTP servers will expect a "host" header to know which site is actually desired. So a HTTP 0.9 client sending Simple-Requests has very little chance of being useful these days. Regards Antoine. From guido at python.org Thu Dec 16 18:34:09 2010 From: guido at python.org (Guido van Rossum) Date: Thu, 16 Dec 2010 09:34:09 -0800 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101216180245.19251acf@pitrou.net> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101215232927.475f925c@pitrou.net> <20101216081456.GB4603@rubuntu> <1292505637.3689.0.camel@localhost.localdomain> <20101216165214.GA7964@rubuntu> <20101216180245.19251acf@pitrou.net> Message-ID: All this talk of modern servers that also still support HTTP/0.9 is irrelevant. Unless anybody knows of a server that *only* supports HTTP 0.9 (and that's relevant to users of httplib) let's please kill support in the client. -- --Guido van Rossum (python.org/~guido) From exarkun at twistedmatrix.com Thu Dec 16 19:30:18 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Thu, 16 Dec 2010 18:30:18 -0000 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101216180245.19251acf@pitrou.net> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101215232927.475f925c@pitrou.net> <20101216081456.GB4603@rubuntu> <1292505637.3689.0.camel@localhost.localdomain> <20101216165214.GA7964@rubuntu> <20101216180245.19251acf@pitrou.net> Message-ID: <20101216183018.2058.1108731637.divmod.xquotient.281@localhost.localdomain> On 05:02 pm, solipsis at pitrou.net wrote: >On Fri, 17 Dec 2010 00:52:14 +0800 >Senthil Kumaran wrote: >>Actually, it is turning out to be true: >> >>http://ftp.ics.uci.edu/pub/ietf/http/rfc1945.html#Response >> >>According to HTTP 1.0, When a request is Simple-Request, it means a >>VERB URL (without a version) and it generally corresponds to HTTP 0.9 >>And when a server receives such a Simple-Request, it sends a >>Simple-Response where it does not send the headers back. >> >>I think, the same is exhibited by other Servers as well >>www.google.com, www.mozilla.org where for Invalid Request without >>version, you are sending a Simple-Request (HTTP 0.9) style and getting >>the corresponding response. > >Yes, but only error or redirect responses: > >$ nc www.google.fr 80 >GET / >HTTP/1.0 302 Found >Location: http://www.google.fr/ >[etc.] Note that by using `nc` to test this, you're already generating an illegal request (unless you're doing something very special with your keyboard ;). When you hit return, nc will send \n. However, lines are delimited by \r\n in HTTP. I doubt this makes a difference to the point being discussed, but it _could_. I suggest performing your tests with telnet, instead. Jean-Paul From nd at perlig.de Thu Dec 16 19:48:11 2010 From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=) Date: Thu, 16 Dec 2010 19:48:11 +0100 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: References: <20101215193940.31262534@pitrou.net> <201012161652.43949.nd@perlig.de> Message-ID: <201012161948.11644@news.perlig.de> * Fred Drake wrote: > On Thu, Dec 16, 2010 at 10:52 AM, Andr? Malo wrote: > > I'd vote for removing it from the client code and keeping it in the > > server. > > If it must be maintained anywhere, it should be in the client, > according to the basic principle of "accept what you can, generate > carefully." *scratching head* exactly why I would keep support in the server. nd -- package Hacker::Perl::Another::Just;print qq~@{[reverse split/::/ =>__PACKAGE__]}~; # Andr? Malo # http://www.perlig.de # From fdrake at acm.org Thu Dec 16 19:47:49 2010 From: fdrake at acm.org (Fred Drake) Date: Thu, 16 Dec 2010 13:47:49 -0500 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101216183018.2058.1108731637.divmod.xquotient.281@localhost.localdomain> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101215232927.475f925c@pitrou.net> <20101216081456.GB4603@rubuntu> <1292505637.3689.0.camel@localhost.localdomain> <20101216165214.GA7964@rubuntu> <20101216180245.19251acf@pitrou.net> <20101216183018.2058.1108731637.divmod.xquotient.281@localhost.localdomain> Message-ID: On Thu, Dec 16, 2010 at 1:30 PM, wrote: > I doubt this makes a difference to the point being discussed, but it > _could_. ?I suggest performing your tests with telnet, instead. I received similar results using telnet earlier today. ? -Fred -- Fred L. Drake, Jr.? ? "A storm broke loose in my mind."? --Albert Einstein From greg.ewing at canterbury.ac.nz Thu Dec 16 23:15:24 2010 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 17 Dec 2010 11:15:24 +1300 Subject: [Python-Dev] Remove HTTP 0.9 support In-Reply-To: <20101216165214.GA7964@rubuntu> References: <20101215193940.31262534@pitrou.net> <4D092C0B.8090300@g.nevcal.com> <20101215222510.0b8aa7e7@pitrou.net> <4D093F46.6050106@g.nevcal.com> <20101215232927.475f925c@pitrou.net> <20101216081456.GB4603@rubuntu> <1292505637.3689.0.camel@localhost.localdomain> <20101216165214.GA7964@rubuntu> Message-ID: <4D0A8F7C.4080007@canterbury.ac.nz> Senthil Kumaran wrote: > Given these, any assumption that servers no longer support HTTP/0.9 > becomes false. But as long as httplib only sends requests with a version number >= 1.0, it should be able to expect headers in the response, shouldn't it? -- Greg From stefan_ml at behnel.de Fri Dec 17 08:19:07 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 17 Dec 2010 08:19:07 +0100 Subject: [Python-Dev] nonlocal x = value Message-ID: Hi, it seems that Py3 doesn't support setting a "nonlocal" value as part of the "nonlocal" command Python 3.2a4+ (py3k:86480, Nov 16 2010, 16:43:22) [GCC 4.4.3] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> def x(): ... y = 5 ... def f(): ... nonlocal y = 6 File "", line 4 nonlocal y = 6 ^ SyntaxError: invalid syntax even though the PEP clearly describes this feature. http://www.python.org/dev/peps/pep-3104/#proposed-solution Either the PEP or the implementation should be updated. Personally, I think the PEP should be changed as I think that the syntax complicates the grammar more than it's worth. Also, the moratorium applies here, given that Py3.1 does not implement this. Comments? Stefan From benjamin at python.org Fri Dec 17 17:08:06 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 17 Dec 2010 10:08:06 -0600 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: 2010/12/17 Stefan Behnel : > Hi, > > it seems that Py3 doesn't support setting a "nonlocal" value as part of the > "nonlocal" command > > ? ?Python 3.2a4+ (py3k:86480, Nov 16 2010, 16:43:22) > ? ?[GCC 4.4.3] on linux2 > ? ?Type "help", "copyright", "credits" or "license" for more information. > ? ?>>> def x(): > ? ?... ? y = 5 > ? ?... ? def f(): > ? ?... ? ? nonlocal y = 6 > ? ? ?File "", line 4 > ? ? ? ?nonlocal y = 6 > ? ? ? ? ? ? ? ? ? ^ > ? ?SyntaxError: invalid syntax > > even though the PEP clearly describes this feature. > > http://www.python.org/dev/peps/pep-3104/#proposed-solution > > Either the PEP or the implementation should be updated. Personally, I think > the PEP should be changed as I think that the syntax complicates the grammar > more than it's worth. Also, the moratorium applies here, given that Py3.1 > does not implement this. > > Comments? There's a issue and a patch for this somewhere. I personally don't care; it's not too painful to write two lines. -- Regards, Benjamin From lvh at laurensvh.be Fri Dec 17 17:52:17 2010 From: lvh at laurensvh.be (Laurens Van Houtven) Date: Fri, 17 Dec 2010 17:52:17 +0100 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: +1 for throwing it out of the PEP. Assignment is a thing, nonlocal/global is a thing, don't mix them up :) (That in addition to the grammar cleanliness argument Stephan already made) cheers lvh. From status at bugs.python.org Fri Dec 17 18:07:04 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 17 Dec 2010 18:07:04 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20101217170704.4F0151CCAD@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-12-10 - 2010-12-17) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2537 (-20) closed 19971 (+72) total 22508 (+52) Open issues with patches: 1070 Issues opened (25) ================== #9232: Allow trailing comma in any function argument list. http://bugs.python.org/issue9232 reopened by ncoghlan #10642: Improve the error message of addpackage() (site.py) for defect http://bugs.python.org/issue10642 reopened by r.david.murray #10679: "make altinstall" may clobber OS provided scripts http://bugs.python.org/issue10679 opened by ncoghlan #10680: argparse: titles and add_mutually_exclusive_group don't mix (e http://bugs.python.org/issue10680 opened by Mads.Michelsen #10684: Shutil.move deletes file/folder in windows while renaming http://bugs.python.org/issue10684 opened by harish #10685: trace does nto ignore --ignore-module http://bugs.python.org/issue10685 opened by RusiMody #10686: email.Generator should use unknown-8bit encoded words for head http://bugs.python.org/issue10686 opened by r.david.murray #10690: IDLE Crash when running/saving Module http://bugs.python.org/issue10690 opened by David_Anon #10694: zipfile.py end of central directory detection not robust http://bugs.python.org/issue10694 opened by KevinH #10697: host and port attributes not documented well in function urlli http://bugs.python.org/issue10697 opened by JTMoon79 #10701: Error pickling a dict http://bugs.python.org/issue10701 opened by belopolsky #10702: bytes and bytearray methods are not documented http://bugs.python.org/issue10702 opened by belopolsky #10708: Misc/porting should be folded in to the development FAQ http://bugs.python.org/issue10708 opened by pitrou #10709: Misc/AIX-NOTES needs updating http://bugs.python.org/issue10709 opened by pitrou #10711: Rip out HTTP 0.9 client support http://bugs.python.org/issue10711 opened by pitrou #10712: 2to3 fixer for deprecated unittest method names http://bugs.python.org/issue10712 opened by ezio.melotti #10713: re module doesn't describe string boundaries for \b http://bugs.python.org/issue10713 opened by ralph.corderoy #10715: uninformative error message http://bugs.python.org/issue10715 opened by Phillip.M.Feldman #10716: Modernize pydoc to use CSS http://bugs.python.org/issue10716 opened by rhettinger #10717: Multiprocessing module Pickling unPickling issues http://bugs.python.org/issue10717 opened by dimitrios #10720: test_threadsignals hang on FreeBSD 6.4 http://bugs.python.org/issue10720 opened by pitrou #10721: Remove HTTP 0.9 server support http://bugs.python.org/issue10721 opened by pitrou #10722: IDLE's subprocess didnit make connection ..... Python 2.7 http://bugs.python.org/issue10722 opened by plovet #10723: Undocumented built-in exceptions http://bugs.python.org/issue10723 opened by flashk #10725: Better cache instrumentation http://bugs.python.org/issue10725 opened by rhettinger Most recent 15 issues with no replies (15) ========================================== #10720: test_threadsignals hang on FreeBSD 6.4 http://bugs.python.org/issue10720 #10717: Multiprocessing module Pickling unPickling issues http://bugs.python.org/issue10717 #10713: re module doesn't describe string boundaries for \b http://bugs.python.org/issue10713 #10709: Misc/AIX-NOTES needs updating http://bugs.python.org/issue10709 #10708: Misc/porting should be folded in to the development FAQ http://bugs.python.org/issue10708 #10701: Error pickling a dict http://bugs.python.org/issue10701 #10686: email.Generator should use unknown-8bit encoded words for head http://bugs.python.org/issue10686 #10685: trace does nto ignore --ignore-module http://bugs.python.org/issue10685 #10684: Shutil.move deletes file/folder in windows while renaming http://bugs.python.org/issue10684 #10666: OS X installer variants have confusing readline differences http://bugs.python.org/issue10666 #10664: xml.sax.expatreader should support namespace prefixes http://bugs.python.org/issue10664 #10663: configure shouldn't set a default OPT http://bugs.python.org/issue10663 #10657: os.lstat/os.stat/os.fstat don't set st_dev (st_rdev) on Window http://bugs.python.org/issue10657 #10656: "Out of tree" build fails on AIX 5.3 http://bugs.python.org/issue10656 #10646: ntpath.samefile doesn't work for hard links http://bugs.python.org/issue10646 Most recent 15 issues waiting for review (15) ============================================= #10725: Better cache instrumentation http://bugs.python.org/issue10725 #10723: Undocumented built-in exceptions http://bugs.python.org/issue10723 #10721: Remove HTTP 0.9 server support http://bugs.python.org/issue10721 #10712: 2to3 fixer for deprecated unittest method names http://bugs.python.org/issue10712 #10711: Rip out HTTP 0.9 client support http://bugs.python.org/issue10711 #10694: zipfile.py end of central directory detection not robust http://bugs.python.org/issue10694 #10673: multiprocess.Process join method - timeout indistinguishable f http://bugs.python.org/issue10673 #10671: urllib2 redirect to another host doesn't work http://bugs.python.org/issue10671 #10665: Expand unicodedata module documentation http://bugs.python.org/issue10665 #10657: os.lstat/os.stat/os.fstat don't set st_dev (st_rdev) on Window http://bugs.python.org/issue10657 #10653: test_time test_strptime fails on windows http://bugs.python.org/issue10653 #10652: test___all_ + test_tcl fails (Windows installed binary) http://bugs.python.org/issue10652 #10648: Extend peepholer to reverse loads or stores instead of build/u http://bugs.python.org/issue10648 #10639: reindent.py converts newlines to platform default http://bugs.python.org/issue10639 #10626: Bad interaction between test_logging and test_concurrent_futur http://bugs.python.org/issue10626 Top 10 most discussed issues (10) ================================= #10716: Modernize pydoc to use CSS http://bugs.python.org/issue10716 12 msgs #6791: httplib read status memory usage http://bugs.python.org/issue6791 11 msgs #9234: argparse: aliases for positional arguments (subparsers) http://bugs.python.org/issue9234 11 msgs #10254: unicodedata.normalize('NFC', s) regression http://bugs.python.org/issue10254 11 msgs #10711: Rip out HTTP 0.9 client support http://bugs.python.org/issue10711 11 msgs #9232: Allow trailing comma in any function argument list. http://bugs.python.org/issue9232 8 msgs #9264: trace.py documentation is incomplete http://bugs.python.org/issue9264 8 msgs #9286: email.utils.parseaddr returns garbage for invalid input http://bugs.python.org/issue9286 5 msgs #10515: csv sniffer does not recognize quotes at the end of line http://bugs.python.org/issue10515 5 msgs #10642: Improve the error message of addpackage() (site.py) for defect http://bugs.python.org/issue10642 5 msgs Issues closed (60) ================== #2576: httplib read() very slow due to lack of socket buffer http://bugs.python.org/issue2576 closed by pitrou #4236: Crash when importing builtin module during interpreter shutdow http://bugs.python.org/issue4236 closed by r.david.murray #4402: os.getenv('PATH') return different result between 2.5 and 3. http://bugs.python.org/issue4402 closed by r.david.murray #4661: email.parser: impossible to read messages encoded in a differe http://bugs.python.org/issue4661 closed by r.david.murray #4766: email documentation needs to be precise about strings/bytes http://bugs.python.org/issue4766 closed by r.david.murray #6454: Add "example" keyword argument to optparse constructor http://bugs.python.org/issue6454 closed by rhettinger #6559: add pass_fds paramter to subprocess.Popen() http://bugs.python.org/issue6559 closed by gregory.p.smith #7183: did 2.6.3 regress for some uses of the __doc__ property? http://bugs.python.org/issue7183 closed by r.david.murray #7213: subprocess leaks open file descriptors between Popen instances http://bugs.python.org/issue7213 closed by gregory.p.smith #8127: Add link to PortingPythonToPy3k to What's New documentation http://bugs.python.org/issue8127 closed by r.david.murray #8753: Py_ReprEnter and Py_ReprLeave are undocumented http://bugs.python.org/issue8753 closed by stutzbach #8844: Condition.wait() doesn't raise KeyboardInterrupt http://bugs.python.org/issue8844 closed by pitrou #9048: no OS X buildbots in the stable list http://bugs.python.org/issue9048 closed by r.david.murray #9162: License for multiprocessing files http://bugs.python.org/issue9162 closed by r.david.murray #9721: urlparse.urljoin() cuts off last base character with semicolon http://bugs.python.org/issue9721 closed by orsenthil #10188: tempfile.TemporaryDirectory may throw errors at shutdown http://bugs.python.org/issue10188 closed by ncoghlan #10454: Clarify compileall command-line options http://bugs.python.org/issue10454 closed by r.david.murray #10509: PyTokenizer_FindEncoding can lead to a segfault if bad charact http://bugs.python.org/issue10509 closed by Trundle #10534: difflib.SequenceMatcher: expose junk sets, deprecate undocumen http://bugs.python.org/issue10534 closed by terry.reedy #10559: NameError in tutorial/interpreter http://bugs.python.org/issue10559 closed by r.david.murray #10618: regression in subprocess.call() command quoting http://bugs.python.org/issue10618 closed by r.david.murray #10627: Remove usage of deprecated configparser.ConfigParser class in http://bugs.python.org/issue10627 closed by lukasz.langa #10634: Windows timezone changes not reflected by time.localtime http://bugs.python.org/issue10634 closed by eric.pruitt #10667: collections.Counter object in C http://bugs.python.org/issue10667 closed by rhettinger #10670: Provide search scope limits http://bugs.python.org/issue10670 closed by r.david.murray #10674: Unused dictmaker symbol in 2.* grammar http://bugs.python.org/issue10674 closed by benjamin.peterson #10675: unittest should have an assertChanges context manager http://bugs.python.org/issue10675 closed by r.david.murray #10676: Confusing note in Numeric Types http://bugs.python.org/issue10676 closed by georg.brandl #10677: "make altinstall" includes ABI codes in versioned executable n http://bugs.python.org/issue10677 closed by barry #10678: email.utils.mktime_tz Giving wrong result , by ignoring Timezo http://bugs.python.org/issue10678 closed by r.david.murray #10681: PySlice_GetIndices() signature changed http://bugs.python.org/issue10681 closed by loewis #10682: With '*args' or even bare '*' in def/call argument list, trail http://bugs.python.org/issue10682 closed by loewis #10683: PreLinkEvent error under VC2010 http://bugs.python.org/issue10683 closed by krisvale #10687: Python fails to install with empty ABI flags http://bugs.python.org/issue10687 closed by barry #10688: pydoc removes lib directory http://bugs.python.org/issue10688 closed by r.david.murray #10689: _scproxy extension is NOT build http://bugs.python.org/issue10689 closed by ned.deily #10691: make testall no longer working on Darwin http://bugs.python.org/issue10691 closed by r.david.murray #10692: imap lib server compabilities http://bugs.python.org/issue10692 closed by silversky #10693: error in documentation of distutils.archive_util.make_zipfile http://bugs.python.org/issue10693 closed by eric.araujo #10695: telnetlib.Telnet port number int/str inconsistency http://bugs.python.org/issue10695 closed by r.david.murray #10696: port not split in function urllib.parse.urlsplit http://bugs.python.org/issue10696 closed by r.david.murray #10698: doctest load_tests() typo http://bugs.python.org/issue10698 closed by r.david.murray #10699: fix incorrect help doc with time.tzset http://bugs.python.org/issue10699 closed by r.david.murray #10700: python pickle.dumps AssertionError http://bugs.python.org/issue10700 closed by belopolsky #10703: Regex 0.1.20101210 http://bugs.python.org/issue10703 closed by belopolsky #10704: Regex 0.1.20101210 Python 3.1 install problem Mac OS X 10.6.5 http://bugs.python.org/issue10704 closed by belopolsky #10705: HTTPConnection.set_debuglevel has no information about level r http://bugs.python.org/issue10705 closed by r.david.murray #10706: kill runtests.sh http://bugs.python.org/issue10706 closed by pitrou #10707: compileall is broken http://bugs.python.org/issue10707 closed by r.david.murray #10710: Is Misc/setuid-prog.c still needed? http://bugs.python.org/issue10710 closed by pitrou #10714: httpserver request length http://bugs.python.org/issue10714 closed by pitrou #10718: Idle 2.7.1 from 64-bit installer crashes on OS X when source i http://bugs.python.org/issue10718 closed by ned.deily #10719: compileall no longer warns when cli arguments name non-existen http://bugs.python.org/issue10719 closed by r.david.murray #10724: socket.close close telnet with RST http://bugs.python.org/issue10724 closed by r.david.murray #10726: pydoc: don???t display raw reST in keyword help http://bugs.python.org/issue10726 closed by georg.brandl #1459867: Message.as_string should use "mangle_from_=unixfrom"? http://bugs.python.org/issue1459867 closed by r.david.murray #1078919: email.Header (via add_header) encodes non-ASCII content incorr http://bugs.python.org/issue1078919 closed by r.david.murray #1628205: socket.readline() interface doesn't handle EINTR properly http://bugs.python.org/issue1628205 closed by pitrou #1243654: Faster output if message already has a boundary http://bugs.python.org/issue1243654 closed by r.david.murray #775964: fix test_grp failing when NIS entries present http://bugs.python.org/issue775964 closed by r.david.murray From alexander.belopolsky at gmail.com Fri Dec 17 18:13:47 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 17 Dec 2010 12:13:47 -0500 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 11:52 AM, Laurens Van Houtven wrote: > +1 for throwing it out of the PEP. Assignment is a thing, > nonlocal/global is a thing, don't mix them up :) (That in addition to > the grammar cleanliness argument Stephan already made) Another +1 for the same reasons. Also, since global does not allow assignment, neither should nonlocal. From benjamin at python.org Fri Dec 17 18:33:50 2010 From: benjamin at python.org (Benjamin Peterson) Date: Fri, 17 Dec 2010 11:33:50 -0600 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: 2010/12/17 Alexander Belopolsky : > On Fri, Dec 17, 2010 at 11:52 AM, Laurens Van Houtven wrote: >> +1 for throwing it out of the PEP. Assignment is a thing, >> nonlocal/global is a thing, don't mix them up :) (That in addition to >> the grammar cleanliness argument Stephan already made) > > Another +1 for the same reasons. ?Also, since global does not allow > assignment, neither should nonlocal. Note that the PEP stated that global would also be extended. -- Regards, Benjamin From alexander.belopolsky at gmail.com Fri Dec 17 19:00:53 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 17 Dec 2010 13:00:53 -0500 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: On Fri, Dec 17, 2010 at 12:33 PM, Benjamin Peterson wrote: .. >> Another +1 for the same reasons. ?Also, since global does not allow >> assignment, neither should nonlocal. > > Note that the PEP stated that global would also be extended. I missed that, so for future reference, the PEP says: """ A shorthand form is also permitted, in which nonlocal is prepended to an assignment or augmented assignment: nonlocal x = 3 The above has exactly the same meaning as nonlocal x; x = 3. (Guido supports a similar form of the global statement.) """ and refers to Guido's post at http://mail.python.org/pipermail/python-3000/2006-November/004166.html In any case, the relevant issue is http://bugs.python.org/issue4199 and it should probably be marked as "after moratorium". Meanwhile an implementation status note can be added to the PEP to avoid this issue being brought up again. From python at mrabarnett.plus.com Sat Dec 18 01:08:47 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 18 Dec 2010 00:08:47 +0000 Subject: [Python-Dev] Locale-specific formatting Message-ID: <4D0BFB8F.2040301@mrabarnett.plus.com> I had a thought about locale-specific formatting. Currently, when we want to do locale-specific formatting we use the locale module like this: >>> locale.format("%d", 12345, grouping=False) '12345' >>> locale.format("%d", 12345, grouping=True) '12,345' This makes it harder to use more than one locale at a time, or one which is different from the default. My thought was that we could specify a locale in the format specification mini-language and the parameter list of str.format, something like this: >>> loc = locale.getlocale() >>> "{0:@1}".format(12345, loc) '12345' >>> "{0:, at 1}".format(12345, loc) '12,345' ... >>> "UK says {value:,.1f at uk} and France says {value:,.1f at france}".format(value=12345, uk=uk_loc, france=france_loc) 'UK says 1,234.5 and France says 1 234,5' Comments? From victor.stinner at haypocalc.com Sat Dec 18 01:55:36 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 18 Dec 2010 01:55:36 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable Message-ID: <4D0C0688.1080108@haypocalc.com> Hi, In 2008, I proposed a patch to raise a Python exception on SIGSEVG signal. In some cases, it's possible to catch the exception, log it, and continue the execution. But because in some cases, the Python internal state is corrupted, the idea was rejected (see the issue #3999). Someone asked me to display the Python backtrace on SIGSEGV, instead of raising an exception. I implemented this idea in issue #8863. After 9 versions, I think that the patch is ready for inclusion. It catchs SIGSEGV, SIGFPE, SIGBUS and SIGILL signals, and also display the Python backtrace on fatal errors. Because some operating systems have their own fault handler (eg. Ubuntu with apport), Dave Malcolm asked me to add an option disable the Python handler. That's why I added the PYTHONNOFAULTHANDLER environment variable (we all love long variable names!). For an OS vendor, it looks like an environment variable is more practical than a command line option. Eg. Mandriva sets PYTHONDONTWRITEBYTECODE for the whole system. Raymond Hettinger asked me on IRC to write an email about the new environment variable, so here is the question: do you agree to add the new variable? Victor From rdmurray at bitdance.com Sat Dec 18 05:00:28 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Fri, 17 Dec 2010 23:00:28 -0500 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <4D0BFB8F.2040301@mrabarnett.plus.com> References: <4D0BFB8F.2040301@mrabarnett.plus.com> Message-ID: <20101218040028.08CBD237968@kimball.webabinitio.net> On Sat, 18 Dec 2010 00:08:47 +0000, MRAB wrote: > I had a thought about locale-specific formatting. > > Currently, when we want to do locale-specific formatting we use the > locale module like this: > > >>> locale.format("%d", 12345, grouping=False) > '12345' > >>> locale.format("%d", 12345, grouping=True) > '12,345' > > This makes it harder to use more than one locale at a time, or one > which is different from the default. > > My thought was that we could specify a locale in the format > specification mini-language and the parameter list of str.format, > something like this: > > >>> loc = locale.getlocale() > >>> "{0:@1}".format(12345, loc) > '12345' > >>> "{0:, at 1}".format(12345, loc) > '12,345' > ... > >>> "UK says {value:,.1f at uk} and France says > {value:,.1f at france}".format(value=12345, uk=uk_loc, france=france_loc) > 'UK says 1,234.5 and France says 1 234,5' > > Comments? There was at least one long thread on this on python-ideas. Probably worth finding and reading before proceeding with a new discussion... :) I think it was part of the discussion that ultimately led to PEP 378. -- R. David Murray www.bitdance.com From martin at v.loewis.de Sat Dec 18 10:26:34 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Dec 2010 10:26:34 +0100 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <4D0BFB8F.2040301@mrabarnett.plus.com> References: <4D0BFB8F.2040301@mrabarnett.plus.com> Message-ID: <4D0C7E4A.3010302@v.loewis.de> > Comments? How do you implement that? In particular, how do you retrieve information for different locales in a single program? Regards, Martin From g.brandl at gmx.net Sat Dec 18 12:37:09 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 18 Dec 2010 12:37:09 +0100 Subject: [Python-Dev] r87188 - python/branches/py3k/Doc/library/test.rst In-Reply-To: <20101212182525.C5C6DEE9F3@mail.python.org> References: <20101212182525.C5C6DEE9F3@mail.python.org> Message-ID: I'd rather keep this a note. We don't want to use warnings except for cases where there is a possibility of security implications or crashes. Georg Am 12.12.2010 19:25, schrieb antoine.pitrou: > Author: antoine.pitrou > Date: Sun Dec 12 19:25:25 2010 > New Revision: 87188 > > Log: > Make this a warning and fix indentation > > > > Modified: > python/branches/py3k/Doc/library/test.rst > > Modified: python/branches/py3k/Doc/library/test.rst > ============================================================================== > --- python/branches/py3k/Doc/library/test.rst (original) > +++ python/branches/py3k/Doc/library/test.rst Sun Dec 12 19:25:25 2010 > @@ -5,12 +5,12 @@ > :synopsis: Regression tests package containing the testing suite for Python. > .. sectionauthor:: Brett Cannon > > -.. note:: > - The :mod:`test` package is meant for internal use by Python only. It is > - documented for the benefit of the core developers of Python. Any use of > - this package outside of Python's standard library is discouraged as code > - mentioned here can change or be removed without notice between releases of > - Python. > +.. warning:: > + The :mod:`test` package is meant for internal use by Python only. It is > + documented for the benefit of the core developers of Python. Any use of > + this package outside of Python's standard library is discouraged as code > + mentioned here can change or be removed without notice between releases of > + Python. > > > The :mod:`test` package contains all regression tests for Python as well as the From solipsis at pitrou.net Sat Dec 18 13:07:34 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 18 Dec 2010 13:07:34 +0100 Subject: [Python-Dev] r87188 - python/branches/py3k/Doc/library/test.rst References: <20101212182525.C5C6DEE9F3@mail.python.org> Message-ID: <20101218130734.1f2f471a@pitrou.net> On Sat, 18 Dec 2010 12:37:09 +0100 Georg Brandl wrote: > I'd rather keep this a note. We don't want to use warnings except for cases > where there is a possibility of security implications or crashes. Well, there'll be a crash as soon as someone relies on an API we decide to change or remove :) From g.brandl at gmx.net Sat Dec 18 13:11:09 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 18 Dec 2010 13:11:09 +0100 Subject: [Python-Dev] r87188 - python/branches/py3k/Doc/library/test.rst In-Reply-To: <20101218130734.1f2f471a@pitrou.net> References: <20101212182525.C5C6DEE9F3@mail.python.org> <20101218130734.1f2f471a@pitrou.net> Message-ID: Am 18.12.2010 13:07, schrieb Antoine Pitrou: > On Sat, 18 Dec 2010 12:37:09 +0100 > Georg Brandl wrote: >> I'd rather keep this a note. We don't want to use warnings except for cases >> where there is a possibility of security implications or crashes. > > Well, there'll be a crash as soon as someone relies on an API we decide > to change or remove :) Ah, but I meant a real crash, not an exception. Think marshal or ctypes. Georg From g.brandl at gmx.net Sat Dec 18 13:12:33 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 18 Dec 2010 13:12:33 +0100 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: Am 17.12.2010 17:52, schrieb Laurens Van Houtven: > +1 for throwing it out of the PEP. Assignment is a thing, > nonlocal/global is a thing, don't mix them up :) (That in addition to > the grammar cleanliness argument Stephan already made) The trouble is what to make of nonlocal x = 3, y Is it two nonlocal declarations or one with a tuple assignment? Georg From g.brandl at gmx.net Sat Dec 18 13:21:45 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 18 Dec 2010 13:21:45 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <4D0C0688.1080108@haypocalc.com> References: <4D0C0688.1080108@haypocalc.com> Message-ID: Am 18.12.2010 01:55, schrieb Victor Stinner: > Hi, > > In 2008, I proposed a patch to raise a Python exception on SIGSEVG > signal. In some cases, it's possible to catch the exception, log it, and > continue the execution. But because in some cases, the Python internal > state is corrupted, the idea was rejected (see the issue #3999). > > Someone asked me to display the Python backtrace on SIGSEGV, instead of > raising an exception. I implemented this idea in issue #8863. After 9 > versions, I think that the patch is ready for inclusion. It catchs > SIGSEGV, SIGFPE, SIGBUS and SIGILL signals, and also display the Python > backtrace on fatal errors. Because some operating systems have their own > fault handler (eg. Ubuntu with apport), Dave Malcolm asked me to add an > option disable the Python handler. That's why I added the > PYTHONNOFAULTHANDLER environment variable (we all love long variable > names!). For an OS vendor, it looks like an environment variable is more > practical than a command line option. Eg. Mandriva sets > PYTHONDONTWRITEBYTECODE for the whole system. > > Raymond Hettinger asked me on IRC to write an email about the new > environment variable, so here is the question: do you agree to add the > new variable? I very much like having a traceback on (some) segmentation faults, but it's clear there needs to be a way to turn it off. An environment variable seems to be the obvious choice (for the reasons you stated for PYTHONDONTWRITEBYTECODE). Georg From solipsis at pitrou.net Sat Dec 18 13:25:37 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 18 Dec 2010 13:25:37 +0100 Subject: [Python-Dev] r87188 - python/branches/py3k/Doc/library/test.rst References: <20101212182525.C5C6DEE9F3@mail.python.org> <20101218130734.1f2f471a@pitrou.net> Message-ID: <20101218132537.649de53b@pitrou.net> On Sat, 18 Dec 2010 13:11:09 +0100 Georg Brandl wrote: > Am 18.12.2010 13:07, schrieb Antoine Pitrou: > > On Sat, 18 Dec 2010 12:37:09 +0100 > > Georg Brandl wrote: > >> I'd rather keep this a note. We don't want to use warnings except for cases > >> where there is a possibility of security implications or crashes. > > > > Well, there'll be a crash as soon as someone relies on an API we decide > > to change or remove :) > > Ah, but I meant a real crash, not an exception. Think marshal or ctypes. We can add one if necessary ;) Antoine. From foom at fuhm.net Sat Dec 18 14:50:49 2010 From: foom at fuhm.net (James Y Knight) Date: Sat, 18 Dec 2010 08:50:49 -0500 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <4D0C0688.1080108@haypocalc.com> References: <4D0C0688.1080108@haypocalc.com> Message-ID: On Dec 17, 2010, at 7:55 PM, Victor Stinner wrote: > Hi, > > In 2008, I proposed a patch to raise a Python exception on SIGSEVG signal. In some cases, it's possible to catch the exception, log it, and continue the execution. But because in some cases, the Python internal state is corrupted, the idea was rejected (see the issue #3999). > > Someone asked me to display the Python backtrace on SIGSEGV, instead of raising an exception. I implemented this idea in issue #8863. After 9 versions, I think that the patch is ready for inclusion. It catchs SIGSEGV, SIGFPE, SIGBUS and SIGILL signals, and also display the Python backtrace on fatal errors. Because some operating systems have their own fault handler (eg. Ubuntu with apport), Dave Malcolm asked me to add an option disable the Python handler. I think instead of calling abort() to kill the process, you should: - install the signal handler with SA_NODEFER|SA_RESETHAND (or if sigaction is not present, explicitly reset the action to SIG_DFL and unblock first thing upon entering the handler), and then, - at the end of the handler, kill(getpid(), orig_signal) in order to abort the process. This has two advantages: 1) the process's exit code will actually show the correct signal, 2) it might let the OS fault handlers work properly as well -- I'm not sure. If it does, you may want to experiment with whether having or omitting SA_NODEFER gives a better backtrace (from the OS mechanism) in that case. If #2 actually works, you may not even need the env var, which would be nice. James From victor.stinner at haypocalc.com Sat Dec 18 14:57:12 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 18 Dec 2010 14:57:12 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> Message-ID: <4D0CBDB8.6010608@haypocalc.com> Le 18/12/2010 13:21, Georg Brandl a ?crit : > I very much like having a traceback on (some) segmentation faults, Why do you say "some" segmentation faults? > but it's clear there needs to be a way to turn it off. An environment variable > seems to be the obvious choice (for the reasons you stated for > PYTHONDONTWRITEBYTECODE). Amaury proposed a new function sys.setsegfaultenabled(). Such function can be used in a customized site module. I think that I will implement it in my next patch, before commiting the new feature. I suppose that we also need a function to get the status of the fault handler. I just don't like the name. I would prefer sys.setfaulthandlerenabled(), I maybe just sys.setfaulthandler(). As this feature is specific to CPython, should it be protected function? So: * sys._setfaulthandler(bool): enable/disable the fault handler * sys._getfaulthandler()->bool: get the status (enabled/disabled) of the fault handler Victor From phd at phd.pp.ru Sat Dec 18 16:33:31 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Sat, 18 Dec 2010 18:33:31 +0300 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <4D0BFB8F.2040301@mrabarnett.plus.com> References: <4D0BFB8F.2040301@mrabarnett.plus.com> Message-ID: <20101218153331.GA22017@phd.pp.ru> On Sat, Dec 18, 2010 at 12:08:47AM +0000, MRAB wrote: > This makes it harder to use more than one locale at a time This is quite a known problem, not specific to Python. Locale settings are global for a process, and this is one of the thousands reasons why locale is considered so horrible. ICU is perhaps the only way around the problem. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From g.brandl at gmx.net Sat Dec 18 17:23:12 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 18 Dec 2010 17:23:12 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <4D0CBDB8.6010608@haypocalc.com> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> Message-ID: Am 18.12.2010 14:57, schrieb Victor Stinner: > Le 18/12/2010 13:21, Georg Brandl a ?crit : >> I very much like having a traceback on (some) segmentation faults, > Why do you say "some" segmentation faults? Well, without a closer I assume that for some crashes it's just not possible anymore for the Python interpreter to even print out the traceback? Georg From g.brandl at gmx.net Sat Dec 18 17:50:38 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 18 Dec 2010 17:50:38 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <4D0CBDB8.6010608@haypocalc.com> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> Message-ID: Am 18.12.2010 14:57, schrieb Victor Stinner: > Le 18/12/2010 13:21, Georg Brandl a ?crit : >> I very much like having a traceback on (some) segmentation faults, > Why do you say "some" segmentation faults? >> but it's clear there needs to be a way to turn it off. An environment variable >> seems to be the obvious choice (for the reasons you stated for >> PYTHONDONTWRITEBYTECODE). > Amaury proposed a new function sys.setsegfaultenabled(). Such function > can be used in a customized site module. I think that I will implement > it in my next patch, before commiting the new feature. I suppose that we > also need a function to get the status of the fault handler. > > I just don't like the name. I would prefer sys.setfaulthandlerenabled(), > I maybe just sys.setfaulthandler(). As this feature is specific to > CPython, should it be protected function? So: > * sys._setfaulthandler(bool): enable/disable the fault handler > * sys._getfaulthandler()->bool: get the status (enabled/disabled) of > the fault handler In any case, this is coming pretty late; beta 2 is scheduled for this weekend, and even if this is something that only kicks in when all hope is lost anyway, it is a new feature. I should like to hear approval from a few more devs before I will let this go into 3.2. Georg From solipsis at pitrou.net Sat Dec 18 18:50:03 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 18 Dec 2010 18:50:03 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> Message-ID: <20101218185003.3298ba57@pitrou.net> On Sat, 18 Dec 2010 17:50:38 +0100 Georg Brandl wrote: > Am 18.12.2010 14:57, schrieb Victor Stinner: > > Le 18/12/2010 13:21, Georg Brandl a ?crit : > >> I very much like having a traceback on (some) segmentation faults, > > Why do you say "some" segmentation faults? > >> but it's clear there needs to be a way to turn it off. An environment variable > >> seems to be the obvious choice (for the reasons you stated for > >> PYTHONDONTWRITEBYTECODE). > > Amaury proposed a new function sys.setsegfaultenabled(). Such function > > can be used in a customized site module. I think that I will implement > > it in my next patch, before commiting the new feature. I suppose that we > > also need a function to get the status of the fault handler. > > > > I just don't like the name. I would prefer sys.setfaulthandlerenabled(), > > I maybe just sys.setfaulthandler(). As this feature is specific to > > CPython, should it be protected function? So: > > * sys._setfaulthandler(bool): enable/disable the fault handler > > * sys._getfaulthandler()->bool: get the status (enabled/disabled) of > > the fault handler > > In any case, this is coming pretty late; beta 2 is scheduled for this > weekend, and even if this is something that only kicks in when all hope > is lost anyway, it is a new feature. I should like to hear approval > from a few more devs before I will let this go into 3.2. I'm +1 on Victor's patch, although I don't think the sys function is useful at all. The env var is enough IMO. (I would even be for backporting it to the bugfix branches if there's no downside) Regards Antoine. From python at mrabarnett.plus.com Sat Dec 18 19:26:43 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 18 Dec 2010 18:26:43 +0000 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <4D0C7E4A.3010302@v.loewis.de> References: <4D0BFB8F.2040301@mrabarnett.plus.com> <4D0C7E4A.3010302@v.loewis.de> Message-ID: <4D0CFCE3.2050906@mrabarnett.plus.com> On 18/12/2010 09:26, "Martin v. L?wis" wrote: >> Comments? > > How do you implement that? In particular, how do you retrieve > information for different locales in a single program? > The locale module would be able to return a named locale dict: >>> loc = locale.getnamedlocale('en_UK') or: >>> loc = locale.namedlocales['en_UK'] The dict would be like what locale.localeconv() returns. The str.format method would then be able to use that dict. From alexander.belopolsky at gmail.com Sat Dec 18 19:55:00 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sat, 18 Dec 2010 13:55:00 -0500 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> Message-ID: On Sat, Dec 18, 2010 at 11:50 AM, Georg Brandl wrote: .. > In any case, this is coming pretty late; beta 2 is scheduled for this > weekend, and even if this is something that only kicks in when all hope > is lost anyway, it is a new feature. ?I should like to hear approval > from a few more devs before I will let this go into 3.2. > I am -1 on the feature as written. I would be -0 if it did not install signal handlers by default and even better was implemented in a separate module, not in core. This feature is similar to the one that is implemented by R and with R, it was always a problem when R was embedded as a library. It has always been a welcome feature of Python that its core did not mess with the application global state. Before this is approved, I would like to see a discussion of the consequences for embedded Python and particularly for the case when there are multiple interpreters in the same process. I think it would be best to start with this feature as an library module or even as a 3rd party add-on and see how useful it will be found in the wild. From martin at v.loewis.de Sat Dec 18 20:36:50 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 18 Dec 2010 20:36:50 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> Message-ID: <4D0D0D52.5020204@v.loewis.de> Am 18.12.2010 19:55, schrieb Alexander Belopolsky: > On Sat, Dec 18, 2010 at 11:50 AM, Georg Brandl wrote: > .. >> In any case, this is coming pretty late; beta 2 is scheduled for this >> weekend, and even if this is something that only kicks in when all hope >> is lost anyway, it is a new feature. I should like to hear approval >> from a few more devs before I will let this go into 3.2. >> > > I am -1 on the feature as written. I would be -0 if it did not > install signal handlers by default and even better was implemented in > a separate module, not in core. This is also what I think. Installing a signal handler is a fairly drastic action, and I don't think the code has been sufficiently reviewed yet. So I'd rather see this deferred until after 3.2, to allow for additional reviews (e.g. if some of the Twisted guys would approve it - that would be convincing). I also share Alexander's concern that Python just shouldn't mess with signal handlers at all, ideally. So some trade-off has to be found to address that concern (e.g. by making the signal handlers only active for the executable interpreter, but not in the embedded case). Regards, Martin From solipsis at pitrou.net Sat Dec 18 21:18:36 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 18 Dec 2010 21:18:36 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> Message-ID: <20101218211836.0c2b0792@pitrou.net> On Sat, 18 Dec 2010 20:36:50 +0100 "Martin v. L?wis" wrote: > Am 18.12.2010 19:55, schrieb Alexander Belopolsky: > > On Sat, Dec 18, 2010 at 11:50 AM, Georg Brandl wrote: > > .. > >> In any case, this is coming pretty late; beta 2 is scheduled for this > >> weekend, and even if this is something that only kicks in when all hope > >> is lost anyway, it is a new feature. I should like to hear approval > >> from a few more devs before I will let this go into 3.2. > >> > > > > I am -1 on the feature as written. I would be -0 if it did not > > install signal handlers by default and even better was implemented in > > a separate module, not in core. > > This is also what I think. Installing a signal handler is a fairly > drastic action, and I don't think the code has been sufficiently > reviewed yet. How much more review should it receive? > I also share Alexander's concern that Python just shouldn't mess with > signal handlers at all, ideally. So some trade-off has to be found to > address that concern (e.g. by making the signal handlers only active > for the executable interpreter, but not in the embedded case). Well, Python already does (and also apparently messes with other things such as FPU state), so that's a separate issue altogether. Regards Antoine. From solipsis at pitrou.net Sat Dec 18 21:48:30 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 18 Dec 2010 21:48:30 +0100 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS References: <20101218200004.99110EE985@mail.python.org> Message-ID: <20101218214830.1aa8be32@pitrou.net> On Sat, 18 Dec 2010 21:00:04 +0100 (CET) ezio.melotti wrote: > Author: ezio.melotti > Date: Sat Dec 18 21:00:04 2010 > New Revision: 87389 > > Log: > #10573: use actual/expected consistently in unittest methods. IMHO, this should be reverted. The API currently doesn't treat these arguments differently, so they should really be labeled "first" and "second". Otherwise, the user will wrongly assume that the signature is asymmetric and that they should be careful about which order they pass the arguments in. Regards Antoine. From g.brandl at gmx.net Sat Dec 18 22:04:02 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 18 Dec 2010 22:04:02 +0100 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: <4D067D3F.8030206@g.nevcal.com> References: <20101213114449.GA2348@chopin.edu.pl> <20101213155119.A52A621CD91@kimball.webabinitio.net> <4D067D3F.8030206@g.nevcal.com> Message-ID: Am 13.12.2010 21:08, schrieb Glenn Linderman: > On 12/13/2010 11:39 AM, Mark Dickinson wrote: >> my_thing = Thing( >> foo = Foo(arg1, arg2, ...), >> bar = Bar(arg3, arg4, ...), >> ... >> ) >> >> and I've found the trailing comma very convenient during refactoring >> and API experimentation. (There's still good fun to be had arguing >> about the indentation of that closing parenthesis, though.) > > > Clearly it needs to be indented one level, because it is a continuation > of the prior line, just like the foo and bar and ... lines are > continuations and therefore indented. Clearly Emacs is superior to Vim because (insert some random fact here). Clearly the only thing that is clear about coding style details (even if we all more or less agree on PEP 8) is that it is a matter of personal taste. Georg From v+python at g.nevcal.com Sat Dec 18 23:41:09 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 18 Dec 2010 14:41:09 -0800 Subject: [Python-Dev] A grammatical oddity: trailing commas in argument lists -- continuation In-Reply-To: References: <20101213114449.GA2348@chopin.edu.pl> <20101213155119.A52A621CD91@kimball.webabinitio.net> <4D067D3F.8030206@g.nevcal.com> Message-ID: <4D0D3885.80903@g.nevcal.com> On 12/18/2010 1:04 PM, Georg Brandl wrote: > Am 13.12.2010 21:08, schrieb Glenn Linderman: >> On 12/13/2010 11:39 AM, Mark Dickinson wrote: >>> my_thing = Thing( >>> foo = Foo(arg1, arg2, ...), >>> bar = Bar(arg3, arg4, ...), >>> ... >>> ) >>> >>> and I've found the trailing comma very convenient during refactoring >>> and API experimentation. (There's still good fun to be had arguing >>> about the indentation of that closing parenthesis, though.) >> >> Clearly it needs to be indented one level, because it is a continuation >> of the prior line, just like the foo and bar and ... lines are >> continuations and therefore indented. > Clearly Emacs is superior to Vim because (insert some random fact here). Sure you have that right. > Clearly the only thing that is clear about coding style details (even if > we all more or less agree on PEP 8) is that it is a matter of personal > taste. And this too. But apparently you missed the fact that Mark wanted some good fun arguing about the indentation of the closing parenthesis... and didn't quote my "Enjoy!" that implied that that was all I was giving him. But then, you are release manager, which would make it very difficult, but hopefully you can still have a Merry Christmas! (or whatever end-of-year holiday suits your fancy) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Dec 18 23:46:23 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 18 Dec 2010 17:46:23 -0500 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <20101218214830.1aa8be32@pitrou.net> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> Message-ID: On 12/18/2010 3:48 PM, Antoine Pitrou wrote: > On Sat, 18 Dec 2010 21:00:04 +0100 (CET) > ezio.melotti wrote: >> Author: ezio.melotti >> Date: Sat Dec 18 21:00:04 2010 >> New Revision: 87389 >> >> Log: >> #10573: use actual/expected consistently in unittest methods. Change was requested by M. Foord and R. Hettinger (and G.Brandl for b2). > IMHO, this should be reverted. The API currently doesn't treat these > arguments differently, so they should really be labeled "first" and > "second". Otherwise, the user will wrongly assume that the signature is > asymmetric and that they should be careful about which order they pass > the arguments in. The error report on assert failure *is* often asymmetrical ;=). From Michael's post: "This is particularly relevant for the methods that produce 'diffed' output on failure - as the order determines whether mismatched items are missing from the expected or additional to the expected." This change struck me as a nice bit of polishing. -- Terry Jan Reedy From tjreedy at udel.edu Sun Dec 19 00:21:24 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 18 Dec 2010 18:21:24 -0500 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <20101218153331.GA22017@phd.pp.ru> References: <4D0BFB8F.2040301@mrabarnett.plus.com> <20101218153331.GA22017@phd.pp.ru> Message-ID: On 12/18/2010 10:33 AM, Oleg Broytman wrote: > This is quite a known problem, not specific to Python. Locale > settings are global for a process, and this is one of the thousands > reasons why locale is considered so horrible. > ICU is perhaps the only way around the problem. This is about the third mention of 'ICU' in the last few weeks, so I looked it up: International Components for Unicode http://site.icu-project.org/ Several libraries (C/C++,Java), including prebuilt binaries for Windows (and some others). There is already a Python .cpp wrapper (but no Windows binaries, which limits usefulness) http://pyicu.osafoundation.org/ http://pypi.python.org/pypi/PyICU/1.0.1 -- Terry Jan Reedy From grygoriy.fuchedzhy at gmail.com Sat Dec 18 22:37:40 2010 From: grygoriy.fuchedzhy at gmail.com (Grygoriy Fuchedzhy) Date: Sat, 18 Dec 2010 23:37:40 +0200 Subject: [Python-Dev] [feature request] add .svgz to mimetypes.suffix_map Message-ID: Hi. I've created bug on bugtracker, and then I was told there that I should post this on this mailing list for discussion. Here is link to bug: http://bugs.python.org/issue10730 Please add '.svgz': '.svg.gz' map to mimetypes.suffix_map -- Grygoriy Fuchedzhy -------------- next part -------------- An HTML attachment was scrubbed... URL: From martin at v.loewis.de Sun Dec 19 01:31:37 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 19 Dec 2010 01:31:37 +0100 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <4D0CFCE3.2050906@mrabarnett.plus.com> References: <4D0BFB8F.2040301@mrabarnett.plus.com> <4D0C7E4A.3010302@v.loewis.de> <4D0CFCE3.2050906@mrabarnett.plus.com> Message-ID: <4D0D5269.1060304@v.loewis.de> Am 18.12.2010 19:26, schrieb MRAB: > On 18/12/2010 09:26, "Martin v. L?wis" wrote: >>> Comments? >> >> How do you implement that? In particular, how do you retrieve >> information for different locales in a single program? >> > The locale module would be able to return a named locale dict: > >>>> loc = locale.getnamedlocale('en_UK') Ok, so, in turn: how do you implement *that* (what C API specifically do you use to implement that getnamedlocale function)? Regards, Martin From python at mrabarnett.plus.com Sun Dec 19 02:11:36 2010 From: python at mrabarnett.plus.com (MRAB) Date: Sun, 19 Dec 2010 01:11:36 +0000 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <4D0D5269.1060304@v.loewis.de> References: <4D0BFB8F.2040301@mrabarnett.plus.com> <4D0C7E4A.3010302@v.loewis.de> <4D0CFCE3.2050906@mrabarnett.plus.com> <4D0D5269.1060304@v.loewis.de> Message-ID: <4D0D5BC8.7080804@mrabarnett.plus.com> On 19/12/2010 00:31, "Martin v. L?wis" wrote: > Am 18.12.2010 19:26, schrieb MRAB: >> On 18/12/2010 09:26, "Martin v. L?wis" wrote: >>>> Comments? >>> >>> How do you implement that? In particular, how do you retrieve >>> information for different locales in a single program? >>> >> The locale module would be able to return a named locale dict: >> >>>>> loc = locale.getnamedlocale('en_UK') > > Ok, so, in turn: how do you implement *that* (what C API > specifically do you use to implement that getnamedlocale function)? > I suppose there could be some sort of locale database. A downloadable, up-to-date copy of the database could be maintained on the Python website. From tjreedy at udel.edu Sun Dec 19 03:24:32 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 18 Dec 2010 21:24:32 -0500 Subject: [Python-Dev] [feature request] add .svgz to mimetypes.suffix_map In-Reply-To: References: Message-ID: On 12/18/2010 4:37 PM, Grygoriy Fuchedzhy wrote: > Hi. > I've created bug on bugtracker, and then I was told there that I should > post this on this mailing list for discussion. > Here is link to bug: http://bugs.python.org/issue10730 > > Please add'.svgz':'.svg.gz' map to mimetypes.suffix_map This issue is actually a request to add .svg to the types map and the the abbreviation .svgz for .svg.gx to the suffix_map. I believe Scalable Vector Graphics are well on the way to becoming *the* standard vector graphics format for the web, especially with upcoming IE9 support. https://secure.wikimedia.org/wikipedia/en/wiki/Svg It (they) are already supported by the other major browsers. In addition, "Google announced on 31 August 2010 that it had begun to index SVG content on the web, whether it is in standalone files or embedded in HTML." So it might be sensible to add these now rather than two years from now. But if it is too late for 3.2, then I expect addition for 3.3 will be obvious. -- Terry Jan Reedy From guido at python.org Sun Dec 19 05:23:49 2010 From: guido at python.org (Guido van Rossum) Date: Sat, 18 Dec 2010 20:23:49 -0800 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> Message-ID: I may be unique, but I fear there is no great answer. On the one hand I almost always code it as e.g. assertEqual(actual, expected), which matches my preference for e.g. "if x == 5:" rather than "if 5 == x:". On the other hand in those assert* functions that show a nice diff of two lists, when reading such a diff my expectation is that "old, new" corresponds to "expected, actual". Which then freaks me out until I realize that I coded it as "actual, expected"... And yet "expected, actual" still looks weird to me. :-( On Sat, Dec 18, 2010 at 2:46 PM, Terry Reedy wrote: > On 12/18/2010 3:48 PM, Antoine Pitrou wrote: >> >> On Sat, 18 Dec 2010 21:00:04 +0100 (CET) >> ezio.melotti ?wrote: >>> >>> Author: ezio.melotti >>> Date: Sat Dec 18 21:00:04 2010 >>> New Revision: 87389 >>> >>> Log: >>> #10573: use actual/expected consistently in unittest methods. > > Change was requested by M. Foord and R. Hettinger (and G.Brandl for b2). > >> IMHO, this should be reverted. The API currently doesn't treat these >> arguments differently, so they should really be labeled "first" and >> "second". Otherwise, the user will wrongly assume that the signature is >> asymmetric and that they should be careful about which order they pass >> the arguments in. > > The error report on assert failure *is* often asymmetrical ;=). > From Michael's post: > "This is particularly relevant for the methods that produce 'diffed' output > on failure - as the order determines whether mismatched items are missing > from the expected or additional to the expected." > > This change struck me as a nice bit of polishing. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From rdmurray at bitdance.com Sun Dec 19 07:52:37 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Sun, 19 Dec 2010 01:52:37 -0500 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> Message-ID: <20101219065237.9FF242379ED@kimball.webabinitio.net> On Sat, 18 Dec 2010 20:23:49 -0800, Guido van Rossum wrote: > I may be unique, but I fear there is no great answer. On the one hand > I almost always code it as e.g. assertEqual(actual, expected), which > matches my preference for e.g. "if x =3D=3D 5:" rather than "if 5 =3D=3D x:= > ". > On the other hand in those assert* functions that show a nice diff of > two lists, when reading such a diff my expectation is that "old, new" > corresponds to "expected, actual". Which then freaks me out until I > realize that I coded it as "actual, expected"... And yet "expected, > actual" still looks weird to me. :-( You aren't unique, I feel the same way. But it seems to me that the most important thing is to be consistent, so that I don't freak out for long. -- R. David Murray www.bitdance.com From phd at phd.pp.ru Sun Dec 19 11:44:00 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Sun, 19 Dec 2010 13:44:00 +0300 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: References: <4D0BFB8F.2040301@mrabarnett.plus.com> <20101218153331.GA22017@phd.pp.ru> Message-ID: <20101219104400.GA21740@phd.pp.ru> On Sat, Dec 18, 2010 at 06:21:24PM -0500, Terry Reedy wrote: > On 12/18/2010 10:33 AM, Oleg Broytman wrote: > >> This is quite a known problem, not specific to Python. Locale >> settings are global for a process, and this is one of the thousands >> reasons why locale is considered so horrible. >> ICU is perhaps the only way around the problem. > > This is about the third mention of 'ICU' in the last few weeks, so I > looked it up: International Components for Unicode > http://site.icu-project.org/ > Several libraries (C/C++,Java), including prebuilt binaries for Windows > (and some others). > There is already a Python .cpp wrapper (but no Windows binaries, which > limits usefulness) > http://pyicu.osafoundation.org/ > http://pypi.python.org/pypi/PyICU/1.0.1 A month ago there was a long thread that mentioned ICU many times: http://mail.python.org/pipermail/python-dev/2010-November/106068.html Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Sun Dec 19 14:13:55 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Dec 2010 14:13:55 +0100 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> Message-ID: <20101219141355.5ac7cd2f@pitrou.net> On Sat, 18 Dec 2010 20:23:49 -0800 Guido van Rossum wrote: > I may be unique, but I fear there is no great answer. On the one hand > I almost always code it as e.g. assertEqual(actual, expected), which > matches my preference for e.g. "if x == 5:" rather than "if 5 == x:". > On the other hand in those assert* functions that show a nice diff of > two lists, when reading such a diff my expectation is that "old, new" > corresponds to "expected, actual". Which then freaks me out until I > realize that I coded it as "actual, expected"... And yet "expected, > actual" still looks weird to me. :-( This could be nicely resolved by renaming the arguments "a" and "b", and having the diff display "a, b". It's quite natural (both the diff ordering and the arguments ordering), and they are consistent with each other. Regards Antoine. From solipsis at pitrou.net Sun Dec 19 14:23:02 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Dec 2010 14:23:02 +0100 Subject: [Python-Dev] r87399 - in python/branches/py3k: Doc/library/http.client.rst Doc/library/urllib.request.rst Lib/http/client.py Lib/test/test_httplib.py Lib/test/test_urllib2.py Lib/urllib/request.py Misc/NEWS References: <20101219104952.EABC8EE9A9@mail.python.org> Message-ID: <20101219142302.399ef3b4@pitrou.net> Hello Senthil, > + raise TypeError("data should be byte-like object\ Should be "bytes-like". > + request.add_unredirected_header( > + 'Content-length', '%d' % len(mv) * mv.itemsize) There is an operator precedence problem here: >>> "%d" % 4 * 5 '44444' > + """ > + file_obj = io.StringIO() > + file_obj.write("Something\nSomething\nSomething\n") > + Why is this whole thing commented out? If it wasn't, you would have seen the test failing. Thanks Antoine. From guido at python.org Sun Dec 19 19:41:34 2010 From: guido at python.org (Guido van Rossum) Date: Sun, 19 Dec 2010 10:41:34 -0800 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <20101219141355.5ac7cd2f@pitrou.net> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> Message-ID: On Sun, Dec 19, 2010 at 5:13 AM, Antoine Pitrou wrote: > On Sat, 18 Dec 2010 20:23:49 -0800 > Guido van Rossum wrote: >> I may be unique, but I fear there is no great answer. On the one hand >> I almost always code it as e.g. assertEqual(actual, expected), which >> matches my preference for e.g. "if x == 5:" rather than "if 5 == x:". >> On the other hand in those assert* functions that show a nice diff of >> two lists, when reading such a diff my expectation is that "old, new" >> corresponds to "expected, actual". Which then freaks me out until I >> realize that I coded it as "actual, expected"... And yet "expected, >> actual" still looks weird to me. :-( > > This could be nicely resolved by renaming the arguments "a" and "b", > and having the diff display "a, b". It's quite natural (both the diff > ordering and the arguments ordering), and they are consistent with each > other. So 'a' stands for 'after' and 'b' for 'before', right? :-) -- --Guido van Rossum (python.org/~guido) From solipsis at pitrou.net Sun Dec 19 19:49:49 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Dec 2010 19:49:49 +0100 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> Message-ID: <1292784589.3679.1.camel@localhost.localdomain> Le dimanche 19 d?cembre 2010 ? 10:41 -0800, Guido van Rossum a ?crit : > On Sun, Dec 19, 2010 at 5:13 AM, Antoine Pitrou wrote: > > On Sat, 18 Dec 2010 20:23:49 -0800 > > Guido van Rossum wrote: > >> I may be unique, but I fear there is no great answer. On the one hand > >> I almost always code it as e.g. assertEqual(actual, expected), which > >> matches my preference for e.g. "if x == 5:" rather than "if 5 == x:". > >> On the other hand in those assert* functions that show a nice diff of > >> two lists, when reading such a diff my expectation is that "old, new" > >> corresponds to "expected, actual". Which then freaks me out until I > >> realize that I coded it as "actual, expected"... And yet "expected, > >> actual" still looks weird to me. :-( > > > > This could be nicely resolved by renaming the arguments "a" and "b", > > and having the diff display "a, b". It's quite natural (both the diff > > ordering and the arguments ordering), and they are consistent with each > > other. > > So 'a' stands for 'after' and 'b' for 'before', right? :-) Ouch. I guess I don't natively think in English. From martin at v.loewis.de Sun Dec 19 19:49:30 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 19 Dec 2010 19:49:30 +0100 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <4D0D5BC8.7080804@mrabarnett.plus.com> References: <4D0BFB8F.2040301@mrabarnett.plus.com> <4D0C7E4A.3010302@v.loewis.de> <4D0CFCE3.2050906@mrabarnett.plus.com> <4D0D5269.1060304@v.loewis.de> <4D0D5BC8.7080804@mrabarnett.plus.com> Message-ID: <4D0E53BA.40101@v.loewis.de> > I suppose there could be some sort of locale database. A downloadable, > up-to-date copy of the database could be maintained on the Python > website. I think you are quite underestimating the implementation effort. So -0 on your original proposal until such a thing actually exists. Regards, Martin From martin at v.loewis.de Sun Dec 19 19:53:43 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 19 Dec 2010 19:53:43 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <20101218211836.0c2b0792@pitrou.net> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> <20101218211836.0c2b0792@pitrou.net> Message-ID: <4D0E54B7.2070604@v.loewis.de> >> This is also what I think. Installing a signal handler is a fairly >> drastic action, and I don't think the code has been sufficiently >> reviewed yet. > > How much more review should it receive? There should be at least one reviewer with an established track record of being interested/knowledgable in the POSIX APIs and cross-platform aspects of it. As I said, any of the Twisted guys would qualify. > Well, Python already does (and also apparently messes with other things > such as FPU state), so that's a separate issue altogether. Not at all. Some signal handlers are more likely to interfere with the the rest of the application than others. Regards, Martin From solipsis at pitrou.net Sun Dec 19 20:20:22 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Dec 2010 20:20:22 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <4D0E54B7.2070604@v.loewis.de> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> <20101218211836.0c2b0792@pitrou.net> <4D0E54B7.2070604@v.loewis.de> Message-ID: <1292786422.3679.24.camel@localhost.localdomain> Le dimanche 19 d?cembre 2010 ? 19:53 +0100, "Martin v. L?wis" a ?crit : > >> This is also what I think. Installing a signal handler is a fairly > >> drastic action, and I don't think the code has been sufficiently > >> reviewed yet. > > > > How much more review should it receive? > > There should be at least one reviewer with an established track record > of being interested/knowledgable in the POSIX APIs and cross-platform > aspects of it. > As I said, any of the Twisted guys would qualify. James Knight already posted in this thread. Is this sufficient to you? I don't think it's reasonable to expect "the Twisted guys" (whoever they are...) to come and review our code, while they struggle with their own bug queue. It would certainly be beneficial for the stdlib to get as much review attention as you are asking for this patch, but it is also totally impractical to expect such a level of attention from a community of volunteers. If we aren't knowledgeable enough to maintain signal handling code, then we should simply remove the signal module from the stdlib (with an appropriate deprecation period, that is) and let interested people maintain it as a third-party module. That would also close some lingering issues in the tracker, and free up some of our time. > > Well, Python already does (and also apparently messes with other things > > such as FPU state), so that's a separate issue altogether. > > Not at all. Some signal handlers are more likely to interfere with the > the rest of the application than others. Can you clarify why you think those signal handlers fall into that description? After all, SIGINT, SIGILL and friends only get triggered in catastrophic conditions. Thanks Antoine. From raymond.hettinger at gmail.com Sun Dec 19 20:55:55 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 19 Dec 2010 11:55:55 -0800 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> Message-ID: <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> On Dec 19, 2010, at 10:41 AM, Guido van Rossum wrote: > On Sun, Dec 19, 2010 at 5:13 AM, Antoine Pitrou wrote: >> On Sat, 18 Dec 2010 20:23:49 -0800 >> Guido van Rossum wrote: >>> I may be unique, but I fear there is no great answer. On the one hand >>> I almost always code it as e.g. assertEqual(actual, expected), which >>> matches my preference for e.g. "if x == 5:" rather than "if 5 == x:". >>> On the other hand in those assert* functions that show a nice diff of >>> two lists, when reading such a diff my expectation is that "old, new" >>> corresponds to "expected, actual". Which then freaks me out until I >>> realize that I coded it as "actual, expected"... And yet "expected, >>> actual" still looks weird to me. :-( >> >> This could be nicely resolved by renaming the arguments "a" and "b", >> and having the diff display "a, b". It's quite natural (both the diff >> ordering and the arguments ordering), and they are consistent with each >> other. > > So 'a' stands for 'after' and 'b' for 'before', right? :-) If you go down the a / b path instead of actual/expected, the diffs are straight-forward but some of the other output styles needed to be changed also (replace the messages for "unexpected" and "missing" elements to "things in a but not in b" and "things in b but not in a". Raymond From alexander.belopolsky at gmail.com Sun Dec 19 21:38:41 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Sun, 19 Dec 2010 15:38:41 -0500 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <1292786422.3679.24.camel@localhost.localdomain> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> <20101218211836.0c2b0792@pitrou.net> <4D0E54B7.2070604@v.loewis.de> <1292786422.3679.24.camel@localhost.localdomain> Message-ID: On Sun, Dec 19, 2010 at 2:20 PM, Antoine Pitrou wrote: .. >> There should be at least one reviewer with an established track record >> of being interested/knowledgable in the POSIX APIs and cross-platform >> aspects of it. >> As I said, any of the Twisted guys would qualify. > > James Knight already posted in this thread. Is this sufficient to you? > I cannot speak for Martin of course, but I think we should first see if James Knight's recommendations get incorporated into the patch and whether that will be sufficient for James Knight to speak in favor of applying the patch. From victor.stinner at haypocalc.com Sun Dec 19 22:59:01 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 19 Dec 2010 22:59:01 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <4D0E54B7.2070604@v.loewis.de> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> <20101218211836.0c2b0792@pitrou.net> <4D0E54B7.2070604@v.loewis.de> Message-ID: <1292795941.5914.23.camel@marge> Le dimanche 19 d?cembre 2010 ? 19:53 +0100, "Martin v. L?wis" a ?crit : > There should be at least one reviewer with an established track record > of being interested/knowledgable in the POSIX APIs and cross-platform > aspects of it. Functions used by the fault handler: - write() - PyUnicode_Check() - PyFrame_GetLineNumber() - DebugBreak() (Windows, in debug mode, only) - abort() - (macro) PyUnicode_GET_SIZE() and PyUnicode_AS_UNICODE() - PyUnicode_Check(), PyFrame_Check() - PyFrame_GetLineNumber() - _Py_atomic_load_relaxed() - _PyThreadState_GetFrame() I suppose that all *Py*() functions are POSIX compliants. DebugBreak() is specific to Windows. Only write() and abort() have to be POSIX compliant. According to Linux manual pages: - abort() conforms to SVr4, POSIX.1-2001, 4.3BSD, C89, C99. - write() conforms to SVr4, 4.3BSD, POSIX.1-2001 -- Functions used by the other parts of the patch: - signal() or sigaction()+sigemptyset() - (optional) sigaltstack() - PyMem_Malloc(), PyMem_Free() - Py_GETENV() - (optional) getpid(), kill() (used by the tests) signal() conforms to C89, C99, POSIX.1-2001. So I don't see any function incompatible with POSIX. (Don't read old versions of my patch, read the latest version) -- About the cross-platform question: it looks like SIGFPE fault is not catchable on Windows. I only tried Linux and Windows, but there are a unit test for each signal and we have 60+ buildbots with various platforms. Can't we wait for the buildbot results? > > Well, Python already does (and also apparently messes with other things > > such as FPU state), so that's a separate issue altogether. > > Not at all. Some signal handlers are more likely to interfere with the > the rest of the application than others. Python installs its own SIGINT handler. I think that a SIGINT handler is more intrusive than a SIGSEGV or SIGILL handler. Victor From victor.stinner at haypocalc.com Sun Dec 19 23:10:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 19 Dec 2010 23:10:43 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> Message-ID: <1292796643.5914.34.camel@marge> Le samedi 18 d?cembre 2010 ? 17:23 +0100, Georg Brandl a ?crit : > Well, without a closer I assume that for some crashes it's just not > possible anymore for the Python interpreter to even print out the > traceback? The worst case occurs if the frame objects are corrupted, eg. if the filename of a frame is an invalid object (eg. pointer to NULL or to a corrupted unicode object). If the file descriptor 2 is not stderr (eg. if there is no file descriptor 2), the backtrace cannot be printed. Except of these two issues, I think that the fault handler is always able to display the backtrace, even on stack overflow or invalid memory write. -- Older versions of my patch were less reliable: - allocate memory on the heap => only use the stack (only few bytes) - call "complex" Python functions (like UTF-8 codec with backslashreplace error handler) => use a very simple ASCII +backslashreplace implementation and avoid complex functions - use not signal-safe functions like printf => use write() and simple functions to format numbers - may go into an unlimited loop if there is a loop in the frame linked list => limit the loop to MAX_DEPTH (100) iterations Victor From victor.stinner at haypocalc.com Sun Dec 19 23:32:03 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 19 Dec 2010 23:32:03 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> Message-ID: <1292797923.5914.55.camel@marge> Le samedi 18 d?cembre 2010 ? 13:55 -0500, Alexander Belopolsky a ?crit : > I am -1 on the feature as written. I would be -0 if it did not > install signal handlers by default and even better was implemented in > a separate module, not in core. I think that the feature is useless if it is disabled by default. How do you know that you application will crash? The idea is to give informations to the user when an application crashs: the user can use the backtrace or send it to the developer. Segmentation faults are usually not (easilly) reproductible :-( So even if you enable the fault handler, you may not be able to replay the crash. Or even worse, the fault may not occurs at all when you enable the fault handler... (Heisenbugs!) > This feature is similar to the one that is implemented by R and with > R, it was always a problem when R was embedded as a library. Which kind of problem? A conflict with an existing fault handler? > It has always been a welcome feature of Python that its core did > not mess with the application global state. What do mess the application global state? > Before this is approved, I would like to see a discussion of the > consequences for embedded Python I suppose that if a fault occurs outside Python, the fault handler displays an irrevelent or empty Python backtrace. The fault handler is installed by Py_InitializeEx(), and uninstalled by Py_Finalize(). If the program has its own fault handler or doesn't want Python fault handler, it can set PYTHONNOFAULTHANDLER environment variable. Should we add a C function to the Python API to configure (disable) the fault handler? I only know *two* applications catching faults: Mplayer and aMule. I don't think that they embed Python. If they do, I think that these *two* applications can do something special. Since there are only two applications having their own fault handler, I don't think that we should disable the fault handler by default. (System wide fault handler, like Apport on Ubuntu, can be configured system wide by setting the PYTHONNOFAULTHANDLER environment variable) > and particularly for the case when there are multiple interpreters > in the same process. What? Is it a myth or does Python really support multiple interpreters in the same process? How is it possible? Who uses this? _Py_DumpBacktrace() function (of my patch) uses _Py_atomic_load_relaxed(&_PyThreadState_Current) to get the current thread state. I picked up this instruction from PyThreadState_Get(). If it doesn't work, PyThreadState_Get() should be fixed too. Victor From scott+python-dev at scottdial.com Sun Dec 19 23:08:51 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sun, 19 Dec 2010 17:08:51 -0500 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> Message-ID: <4D0E8273.1040606@scottdial.com> On 12/18/2010 8:50 AM, James Y Knight wrote: > I think instead of calling abort() to kill the process, you should: > - install the signal handler with SA_NODEFER|SA_RESETHAND (or if > sigaction is not present, explicitly reset the action to SIG_DFL and > unblock first thing upon entering the handler), and then, > - at the end of the handler, kill(getpid(), orig_signal) in order to > abort the process. I concur with this being the correct way to right such a handler. > This has two advantages: 1) the process's exit code will actually > show the correct signal, It's more than an advantage: it's the only correct way to handle a termination signal. > 2) it might let the OS fault handlers work properly > as well -- I'm not sure. If it does, you may want to experiment with > whether having or omitting SA_NODEFER gives a better backtrace (from the > OS mechanism) in that case. Even if that doesn't work, things like the grsecurity patches to linux use these signals to detect exploits and log them and do throttling. Calling abort() effectively converts all of these faults into SIGABRT terminations that are considered (more?) innocent terminations. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From victor.stinner at haypocalc.com Sun Dec 19 23:37:58 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 19 Dec 2010 23:37:58 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <1292786422.3679.24.camel@localhost.localdomain> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> <20101218211836.0c2b0792@pitrou.net> <4D0E54B7.2070604@v.loewis.de> <1292786422.3679.24.camel@localhost.localdomain> Message-ID: <1292798278.5914.56.camel@marge> Le dimanche 19 d?cembre 2010 ? 20:20 +0100, Antoine Pitrou a ?crit : > Can you clarify why you think those signal handlers fall into that > description? After all, SIGINT, SIGILL and friends only get triggered in > catastrophic conditions. SIGSEGV, not SIGINT Victor From solipsis at pitrou.net Sun Dec 19 23:51:03 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 19 Dec 2010 23:51:03 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> <20101218211836.0c2b0792@pitrou.net> <4D0E54B7.2070604@v.loewis.de> <1292786422.3679.24.camel@localhost.localdomain> <1292798278.5914.56.camel@marge> Message-ID: <20101219235103.227aaad8@pitrou.net> On Sun, 19 Dec 2010 23:37:58 +0100 Victor Stinner wrote: > Le dimanche 19 d?cembre 2010 ? 20:20 +0100, Antoine Pitrou a ?crit : > > Can you clarify why you think those signal handlers fall into that > > description? After all, SIGINT, SIGILL and friends only get triggered in > > catastrophic conditions. > > SIGSEGV, not SIGINT Oops, sorry for the mistake. I hope this didn't stir any confusion beyond my own ;) cheers Antoine. From victor.stinner at haypocalc.com Mon Dec 20 00:02:25 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 20 Dec 2010 00:02:25 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> Message-ID: <1292799745.5914.76.camel@marge> Le samedi 18 d?cembre 2010 ? 08:50 -0500, James Y Knight a ?crit : > I think instead of calling abort() to kill the process, you should: > - install the signal handler with SA_NODEFER|SA_RESETHAND (or if sigaction is not present, explicitly reset the action to SIG_DFL and unblock first thing upon entering the handler), and then, > - at the end of the handler, kill(getpid(), orig_signal) in order to abort the process. I don't think that kill() is available on Windows. We may call directly the previous handler in the new handler, but it is more difficult. It looks like the prototype of a signal handler depends on its options. Or is it possible to always pass (signum, siginfo, ucontext) arguments? I don't know if siginfo and ucontect are portable or not... Another possible approach is to restore the previous handler in the fault handler, and don't call abort(). The fault should occur again, and so the previous signal handler will be called. I choosed to not call the previous handler to keep the patch simple. If you consider that we must call the previous handler, I can work on a new patch doing that. > This has two advantages: 1) the process's exit code will actually show the correct signal Oh, I didn't noticed that the exit code is different. > 2) it might let the OS fault handlers work properly as well -- I'm not sure. If it does, you may want to experiment with whether having or omitting SA_NODEFER gives a better backtrace (from the OS mechanism) in that case. For the OS fault handler: we should try, I am not sure that calling another signal handler (my fault handler) keeps the process state unchanged (especially if sigaltstk() is not available). I don't understand the SA_NODEFER option. What does it change for this fault handler? > If #2 actually works, you may not even need the env var, which would be nice. I prefer to keep it optional because we don't know yet how it interacts exactly with existing debug tools, OS tools, embedded Python interpreters, etc. Victor From tjreedy at udel.edu Mon Dec 20 00:54:55 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 19 Dec 2010 18:54:55 -0500 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> Message-ID: On 12/19/2010 1:41 PM, Guido van Rossum wrote: > On Sun, Dec 19, 2010 at 5:13 AM, Antoine Pitrou wrote: >> This could be nicely resolved by renaming the arguments "a" and "b", >> and having the diff display "a, b". It's quite natural (both the diff >> ordering and the arguments ordering), and they are consistent with each >> other. > > So 'a' stands for 'after' and 'b' for 'before', right? :-) difflib uses 'a' and 'b' for before and after (orig,new in svn terms, with edits/diffs from a to b) respectively. Not really great. The docs then have to explain what 'a' and 'b' are and the implications for interpreting the output. -- Terry Jan Reedy From merwok at netwok.org Mon Dec 20 01:09:19 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Mon, 20 Dec 2010 01:09:19 +0100 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <20101218153331.GA22017@phd.pp.ru> References: <4D0BFB8F.2040301@mrabarnett.plus.com> <20101218153331.GA22017@phd.pp.ru> Message-ID: <4D0E9EAF.6010000@netwok.org> Le 18/12/2010 16:33, Oleg Broytman a ?crit : > This is quite a known problem, not specific to Python. Locale > settings are global for a process, and this is one of the thousands > reasons why locale is considered so horrible. > ICU is perhaps the only way around the problem. Babel rocks: http://babel.edgewall.org/ Cheers From victor.stinner at haypocalc.com Mon Dec 20 01:19:35 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 20 Dec 2010 01:19:35 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <1292799745.5914.76.camel@marge> References: <4D0C0688.1080108@haypocalc.com> <1292799745.5914.76.camel@marge> Message-ID: <1292804375.5914.84.camel@marge> Le lundi 20 d?cembre 2010 ? 00:02 +0100, Victor Stinner a ?crit : > Another possible approach is to restore the previous handler in the > fault handler, and don't call abort(). The fault should occur again, and > so the previous signal handler will be called. I implemented this simple approach in the version 10 of my patch: the previous signal handler is called. http://bugs.python.org/issue8863#msg124373 On my Debian Sid, the libc fatal error message is written as expected (eg. "Segmentation fault") and the exit code is different for each fault (eg. 132 for SIGILL). So we keep the existing behaviour, the new Python fault handler just write more information if a fault occurs. I suppose that it is compatible with the grsecurity thing and Apport, but I didn't tried. Can someone test it? I prefer to restore the previous signal handler than restoring the *default* signal handler. > I choosed to not call the previous handler to keep the patch simple. It's cool: restoring the signal approach keeps the code simple, and it looks like it works :-) Victor From solipsis at pitrou.net Mon Dec 20 01:48:40 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 20 Dec 2010 01:48:40 +0100 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> Message-ID: <20101220014840.6b734274@pitrou.net> On Sun, 19 Dec 2010 18:54:55 -0500 Terry Reedy wrote: > On 12/19/2010 1:41 PM, Guido van Rossum wrote: > > On Sun, Dec 19, 2010 at 5:13 AM, Antoine Pitrou wrote: > > >> This could be nicely resolved by renaming the arguments "a" and "b", > >> and having the diff display "a, b". It's quite natural (both the diff > >> ordering and the arguments ordering), and they are consistent with each > >> other. > > > > So 'a' stands for 'after' and 'b' for 'before', right? :-) > > difflib uses 'a' and 'b' for before and after (orig,new in svn terms, > with edits/diffs from a to b) respectively. Not really great. For a non-native English speaker, 'a' and 'b' don't evoke 'after' and 'before' but simply the first two letters of the latin alphabet, and their ordering is therefore obvious with respect to function arguments. By the way, hg uses a/b as well, and so does git apparently, so Python's difflib is not exotic in that regard: $ hg diff diff -r 56867877575b README --- a/README Fri Dec 17 21:43:27 2010 +0100 +++ b/README Mon Dec 20 01:42:57 2010 +0100 @@ -1,3 +1,4 @@ +some change This is Python version 3.2 beta 1 ================================= Regards Antoine. From martin at v.loewis.de Mon Dec 20 02:05:30 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 20 Dec 2010 02:05:30 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <1292795941.5914.23.camel@marge> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> <20101218211836.0c2b0792@pitrou.net> <4D0E54B7.2070604@v.loewis.de> <1292795941.5914.23.camel@marge> Message-ID: <4D0EABDA.3060702@v.loewis.de> > Functions used by the fault handler: > - write() > - PyUnicode_Check() > - PyFrame_GetLineNumber() > - DebugBreak() (Windows, in debug mode, only) > - abort() > - (macro) PyUnicode_GET_SIZE() and PyUnicode_AS_UNICODE() > - PyUnicode_Check(), PyFrame_Check() > - PyFrame_GetLineNumber() > - _Py_atomic_load_relaxed() > - _PyThreadState_GetFrame() > > I suppose that all *Py*() functions are POSIX compliants. The problem is that merely being POSIX compliant would not be enough to allow calling a function in a signal handler. Instead, the function *also* needs to be async-signal safe. However, > - abort() conforms to SVr4, POSIX.1-2001, 4.3BSD, C89, C99. > - write() conforms to SVr4, 4.3BSD, POSIX.1-2001 These two functions are indeed async-signal safe. So assuming that none of the functions above indirectly calls a non-async-signal-safe function, this part is fine. Looking at your function list, my other concern is that you are calling Python API without holding the GIL, IIUC. In particular, you are accessing _PyThreadState_Current, which may not point to the current thread if the current thread has released the GIL. Regards, Martin From martin at v.loewis.de Mon Dec 20 02:11:34 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 20 Dec 2010 02:11:34 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <1292797923.5914.55.camel@marge> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <1292797923.5914.55.camel@marge> Message-ID: <4D0EAD46.209@v.loewis.de> > What? Is it a myth or does Python really support multiple interpreters > in the same process? How is it possible? Who uses this? [Not sure if you are joking] There is certainly some support for multiple interpreters in Python; the most prominent user of this feature is mod_python. Regards, Martin From fuzzyman at voidspace.org.uk Mon Dec 20 01:57:04 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 20 Dec 2010 00:57:04 +0000 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <1292799745.5914.76.camel@marge> References: <4D0C0688.1080108@haypocalc.com> <1292799745.5914.76.camel@marge> Message-ID: <4D0EA9E0.4080707@voidspace.org.uk> On 19/12/2010 23:02, Victor Stinner wrote: > Le samedi 18 d?cembre 2010 ? 08:50 -0500, James Y Knight a ?crit : >> I think instead of calling abort() to kill the process, you should: >> - install the signal handler with SA_NODEFER|SA_RESETHAND (or if sigaction is not present, explicitly reset the action to SIG_DFL and unblock first thing upon entering the handler), and then, >> - at the end of the handler, kill(getpid(), orig_signal) in order to abort the process. > I don't think that kill() is available on Windows. Thanks to Brian Curtin there is now an os.kill() on Windows, but it doesn't handle the full range of signals. Michael -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From victor.stinner at haypocalc.com Mon Dec 20 05:53:49 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 20 Dec 2010 05:53:49 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <4D0EABDA.3060702@v.loewis.de> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> <20101218211836.0c2b0792@pitrou.net> <4D0E54B7.2070604@v.loewis.de> <1292795941.5914.23.camel@marge> <4D0EABDA.3060702@v.loewis.de> Message-ID: <1292820829.4763.16.camel@marge> Le lundi 20 d?cembre 2010 ? 02:05 +0100, "Martin v. L?wis" a ?crit : > The problem is that merely being POSIX compliant would not be enough > to allow calling a function in a signal handler. Instead, the function > *also* needs to be async-signal safe. Yes, this issue was fixed in an older version of my patch. Starting at version 9, the signal handler only call signal-safe functions. > Looking at your function list, my other concern is that you are calling > Python API without holding the GIL, IIUC. In particular, you are > accessing _PyThreadState_Current, which may not point to the current > thread if the current thread has released the GIL. Ah? Where does _PyThreadState_Current point to if the GIL is not hold when the fault handler is called? My patch changes also Py_FatalError() to display the Python backtrace. It looks that _PyThreadState_Current can be NULL if the GIL is released. In this case, _Py_DumpBacktrace() just do nothing. There is also a gil_last_holder variable: can it be used to get the thread state (especially if the thread state was deleted)? I don't think that it will possible the acquire the GIL in Py_FatalError() or in the fault handler. Victor From martin at v.loewis.de Mon Dec 20 08:22:40 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 20 Dec 2010 08:22:40 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <1292820829.4763.16.camel@marge> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <4D0D0D52.5020204@v.loewis.de> <20101218211836.0c2b0792@pitrou.net> <4D0E54B7.2070604@v.loewis.de> <1292795941.5914.23.camel@marge> <4D0EABDA.3060702@v.loewis.de> <1292820829.4763.16.camel@marge> Message-ID: <4D0F0440.5070806@v.loewis.de> >> Looking at your function list, my other concern is that you are calling >> Python API without holding the GIL, IIUC. In particular, you are >> accessing _PyThreadState_Current, which may not point to the current >> thread if the current thread has released the GIL. > > Ah? Where does _PyThreadState_Current point to if the GIL is not hold > when the fault handler is called? The GIL is likely held by a different thread, then. _PyThreadState_Current will point to the state of this other thread. > It looks that _PyThreadState_Current can be NULL if the GIL is released. > In this case, _Py_DumpBacktrace() just do nothing. There is also a > gil_last_holder variable: can it be used to get the thread state > (especially if the thread state was deleted)? Of this thread? I don't think so. gil_last_holder might also refer to a different thread. > I don't think that it will possible the acquire the GIL in > Py_FatalError() or in the fault handler. I agree. Regards, Martin From phd at phd.pp.ru Mon Dec 20 10:23:02 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Mon, 20 Dec 2010 12:23:02 +0300 Subject: [Python-Dev] Locale-specific formatting In-Reply-To: <4D0E9EAF.6010000@netwok.org> References: <4D0BFB8F.2040301@mrabarnett.plus.com> <20101218153331.GA22017@phd.pp.ru> <4D0E9EAF.6010000@netwok.org> Message-ID: <20101220092302.GA5973@phd.pp.ru> On Mon, Dec 20, 2010 at 01:09:19AM +0100, ??ric Araujo wrote: > Le 18/12/2010 16:33, Oleg Broytman a ??crit : > > This is quite a known problem, not specific to Python. Locale > > settings are global for a process, and this is one of the thousands > > reasons why locale is considered so horrible. > > ICU is perhaps the only way around the problem. > > Babel rocks: http://babel.edgewall.org/ Unicode CLDR! Never heard of it before, thank you for pointing this out. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From steve at pearwood.info Mon Dec 20 10:55:10 2010 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 20 Dec 2010 20:55:10 +1100 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <20101220014840.6b734274@pitrou.net> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <20101220014840.6b734274@pitrou.net> Message-ID: <4D0F27FE.4020706@pearwood.info> Antoine Pitrou wrote: > For a non-native English speaker, 'a' and 'b' don't evoke 'after' and > 'before' but simply the first two letters of the latin alphabet, and > their ordering is therefore obvious with respect to function arguments. It's not just non-native English speakers either. I too think of a, b as being first, second rather than after, before. -- Steven From stefan at bytereef.org Mon Dec 20 12:08:35 2010 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 20 Dec 2010 12:08:35 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> Message-ID: <20101220110835.GA13825@yoda.bytereef.org> Alexander Belopolsky wrote: > On Sat, Dec 18, 2010 at 11:50 AM, Georg Brandl wrote: > .. > > In any case, this is coming pretty late; beta 2 is scheduled for this > > weekend, and even if this is something that only kicks in when all hope > > is lost anyway, it is a new feature. ?I should like to hear approval > > from a few more devs before I will let this go into 3.2. > > > > I am -1 on the feature as written. I would be -0 if it did not > install signal handlers by default and even better was implemented in > a separate module, not in core. I would not want this to be the default either. I think the output is not particularly informative: $ ./python Lib/test/crashers/gc_inspection.py (, , , , , , , , , ) Fatal Python error: Segmentation fault Traceback (most recent call first): File "Lib/test/crashers/gc_inspection.py", line 29 in g File "Lib/test/crashers/gc_inspection.py", line 32 in Segmentation fault Segmentation fault Another test is hanging indefinitely (Ubuntu 64-bit): $ ./python Lib/test/crashers/nasty_eq_vs_dict.py [hanging with no output] It's hard to get signal handlers right across multiple platforms; even SIGINT catching does not work properly on OpenBSD: http://bugs.python.org/issue8714 In short, I agree that having more signal handlers by default is not a good idea. Stefan Krah From stefan at bytereef.org Mon Dec 20 13:11:52 2010 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 20 Dec 2010 13:11:52 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <20101220110835.GA13825@yoda.bytereef.org> References: <4D0C0688.1080108@haypocalc.com> <4D0CBDB8.6010608@haypocalc.com> <20101220110835.GA13825@yoda.bytereef.org> Message-ID: <20101220121152.GA15386@yoda.bytereef.org> Stefan Krah wrote: > Another test is hanging indefinitely (Ubuntu 64-bit): > > $ ./python Lib/test/crashers/nasty_eq_vs_dict.py > [hanging with no output] And this test does not report anything at all: $ ./python Lib/test/crashers/compiler_recursion.py [no output at all] $ echo $? 0 Without the patch: $ ./python Lib/test/crashers/compiler_recursion.py Segmentation fault $ echo $? 139 Stefan Krah From fuzzyman at voidspace.org.uk Mon Dec 20 14:00:26 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 20 Dec 2010 13:00:26 +0000 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> Message-ID: <4D0F536A.8050808@voidspace.org.uk> On 19/12/2010 19:55, Raymond Hettinger wrote: > On Dec 19, 2010, at 10:41 AM, Guido van Rossum wrote: > >> On Sun, Dec 19, 2010 at 5:13 AM, Antoine Pitrou wrote: >>> On Sat, 18 Dec 2010 20:23:49 -0800 >>> Guido van Rossum wrote: >>>> I may be unique, but I fear there is no great answer. On the one hand >>>> I almost always code it as e.g. assertEqual(actual, expected), which >>>> matches my preference for e.g. "if x == 5:" rather than "if 5 == x:". >>>> On the other hand in those assert* functions that show a nice diff of >>>> two lists, when reading such a diff my expectation is that "old, new" >>>> corresponds to "expected, actual". Which then freaks me out until I >>>> realize that I coded it as "actual, expected"... And yet "expected, >>>> actual" still looks weird to me. :-( >>> This could be nicely resolved by renaming the arguments "a" and "b", >>> and having the diff display "a, b". It's quite natural (both the diff >>> ordering and the arguments ordering), and they are consistent with each >>> other. >> So 'a' stands for 'after' and 'b' for 'before', right? :-) > If you go down the a / b path instead of actual/expected, > the diffs are straight-forward but some of the other > output styles needed to be changed also (replace the > messages for "unexpected" and "missing" elements to > "things in a but not in b" and "things in b but not in a". > > Ah man, we've *nearly* finished bikeshedding about the names of unittest assert methods so its time to move onto the names and order of the arguments. Really? I wouldn't use a/b but first/second [1] as they have a more obvious meaning. However, I'm reluctant to move away from the actual/expected terminology. It's standard terminology for testing (used by all the other unit testing frameworks I looked at phpunit, JUnit and NUnit), but more importantly it is a useful way to think about testing - and one used by most devs I've worked with. You fetch an 'actual' result by calling your code and compare it against a pre-computed 'expected' result. Hopefully the two are the same. Talking about your actual value and your expected value is a natural way to talk in testing, so it's a useful concept. Once you use the 'actual' and 'expected' terminology you have a natural order for displaying failure message results: if an element is present in your actual but not in your expected then it is extra. If an element is in your expected but not in your actual then it is missing. Straightforward. (Of course it maybe that your actual is correct and it is your expected result needs correcting, that doesn't affect how failure messages should be presented though.) The only thing left to decide is then the order - (actual, expected) or (expected, actual). Raymond, myself, Guido and Ezio have all expressed a preference for (actual, expected). I like this comment from Raymond on the relevant issue [2]: I also tend to use actual/expected and find the reverse to be a form Yoda-speak, "assert 5 == x", "perhaps misread the prophecy was", etc. As the current ordering used within unittest is (actual, expected), to reverse it would be dumb (why should everyone using the current ordering reformat all their tests for the new order?). So, -1 on dropping actual and expected. They're standard and useful terminology / concepts for testing. If we do move to a more "agnostic" wording in the failure messages (whilst keeping actual / expected as argument names and in the documentation perhaps?) then I prefer first / second to a / b. All the best, Michael Foord [1] Interestingly unittest did use (first, second) for assert argument names back in 2.1 when it was added: http://svn.python.org/view/python/branches/release21-maint/Lib/unittest.py?revision=24536&view=markup [2] http://bugs.python.org/issue10573 > Raymond > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessinghttp://www.sqlite.org/different.html From solipsis at pitrou.net Mon Dec 20 14:47:21 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 20 Dec 2010 14:47:21 +0100 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <4D0F536A.8050808@voidspace.org.uk> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> Message-ID: <1292852841.3737.25.camel@localhost.localdomain> Le lundi 20 d?cembre 2010 ? 13:00 +0000, Michael Foord a ?crit : > Ah man, we've *nearly* finished bikeshedding about the names of unittest > assert methods so its time to move onto the names and order of the > arguments. Really? Apparently someone decided this bikeshedding was important enough to make an SVN commit out of it. If you think it isn't worth discussing then perhaps it wasn't worth changing in the first place :) > The only thing left to decide is then the order - (actual, expected) or > (expected, actual). Raymond, myself, Guido and Ezio have all expressed a > preference for (actual, expected). I like this comment from Raymond on > the relevant issue [2]: > > I also tend to use actual/expected and find the reverse to be a > form Yoda-speak, "assert 5 == x", "perhaps misread the prophecy was", etc. Isn't it some kind of ethnocentric comment? "Natural" order is not the same in all languages, and I don't see why "actual" should come before "expected". And the problem here is that (actual, expected) is in reverse order of the diff displayed on error. Now if you look at various TypeErrors raised in Python, the error message is most often "expected , got ". So there expected always comes before actual, and apparently it was natural to the authors of that code. Perhaps they are all Yoda-speakers. And moreover, some methods such as assertRaises() already have their args in the (expected, actual) order. > If we do move to a more "agnostic" wording in the failure messages > (whilst keeping actual / expected as argument names and in the > documentation perhaps?) then I prefer first / second to a / b. Well, no. The whole point of displaying results as a/b (or first/second) is that they map intuitively to the first and second args. If, however, you name those args "actual" and "expected", it becomes confusing. Also, it wouldn't fix that your diff would still be in a weird order compared to the method args (unless you think it's fine to display an "expected - actual" diff). Regards Antoine. From fuzzyman at voidspace.org.uk Mon Dec 20 15:03:26 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 20 Dec 2010 14:03:26 +0000 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <1292852841.3737.25.camel@localhost.localdomain> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> Message-ID: <4D0F622E.7050902@voidspace.org.uk> On 20/12/2010 13:47, Antoine Pitrou wrote: > Le lundi 20 d?cembre 2010 ? 13:00 +0000, Michael Foord a ?crit : >> Ah man, we've *nearly* finished bikeshedding about the names of unittest >> assert methods so its time to move onto the names and order of the >> arguments. Really? > Apparently someone decided this bikeshedding was important enough to > make an SVN commit out of it. If you think it isn't worth discussing > then perhaps it wasn't worth changing in the first place :) > The only change was to use them consistently and the only code change was to re-order the arguments in a method that is new and not in any previous version of Python. You're arguing for a much bigger change. >> The only thing left to decide is then the order - (actual, expected) or >> (expected, actual). Raymond, myself, Guido and Ezio have all expressed a >> preference for (actual, expected). I like this comment from Raymond on >> the relevant issue [2]: >> >> I also tend to use actual/expected and find the reverse to be a >> form Yoda-speak, "assert 5 == x", "perhaps misread the prophecy was", etc. > Isn't it some kind of ethnocentric comment? "Natural" order is not the > same in all languages, and I don't see why "actual" should come before > "expected". Agreement that actual, expected was preferred came from an American, a Dutchman, an Englishman and an Italian. :-) As it is what unittest currently uses anyway you'll need more than "I don't see why" to reverse it. > And the problem here is that (actual, expected) is in reverse order of > the diff displayed on error. > Diffing is completely an implementation detail of how the failure messages are generated. The important thing is that failure messages make sense with respect to actual result and expected result. > Now if you look at various TypeErrors raised in Python, the error > message is most often "expected, got". So there > expected always comes before actual, and apparently it was natural to > the authors of that code. Perhaps they are all Yoda-speakers. > > And moreover, some methods such as assertRaises() already have their > args in the (expected, actual) order. > >> If we do move to a more "agnostic" wording in the failure messages >> (whilst keeping actual / expected as argument names and in the >> documentation perhaps?) then I prefer first / second to a / b. > Well, no. The whole point of displaying results as a/b (or first/second) > is that they map intuitively to the first and second args. If, however, > you name those args "actual" and "expected", it becomes confusing. > > Also, it wouldn't fix that your diff would still be in a weird order > compared to the method args (unless you think it's fine to display an > "expected - actual" diff). > I told you how I think the diff should be generated and gave the reasons for it. If we use first / second in failure messages (but not necessarily in documentation or argument names - your objections notwithstanding) then the messages for the container asserts don't need to imply a direction (as Raymond suggested, wording like "in first but not in second" and "in second but not in first"). The string diffs are more problematic as they actually use difflib to generate the failure output. I'm also against dropping the use of actual / expected concepts and terminology within unittest as I think they are useful. We don't necessarily need to use them in the failure outputs but it seems like you want them to be dropped altogether. All the best, Michael > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From vstinner at edenwall.com Mon Dec 20 14:39:29 2010 From: vstinner at edenwall.com (Victor Stinner) Date: Mon, 20 Dec 2010 14:39:29 +0100 Subject: [Python-Dev] =?iso-8859-1?q?Issue_=238863_adds_a_new_PYTHONNOFAUL?= =?iso-8859-1?q?THANDLER=09environment_variable?= In-Reply-To: <20101220110835.GA13825@yoda.bytereef.org> References: <4D0C0688.1080108@haypocalc.com> <20101220110835.GA13825@yoda.bytereef.org> Message-ID: <201012201439.29282.vstinner@edenwall.com> Le lundi 20 d?cembre 2010 12:08:35, Stefan Krah a ?crit : > I think the output is not particularly informative: > > $ ./python Lib/test/crashers/gc_inspection.py > (, , , , , , > , , , ) Fatal Python error: Segmentation fault > > Traceback (most recent call first): > File "Lib/test/crashers/gc_inspection.py", line 29 in g > File "Lib/test/crashers/gc_inspection.py", line 32 in > Segmentation fault > Segmentation fault The backtrace is valid. Don't you think that this backtrace is more useful than just "Segmentation fault"? > Another test is hanging indefinitely (Ubuntu 64-bit): > > $ ./python Lib/test/crashers/nasty_eq_vs_dict.py > [hanging with no output] I didn't tried with patch version 9, but with the patch version 10 I get: ----------------- $ ./python Lib/test/crashers/nasty_eq_vs_dict.py Fatal Python error: Segmentation fault Traceback (most recent call first): File "Lib/test/crashers/nasty_eq_vs_dict.py", line 16 in __hash__ File "Lib/test/crashers/nasty_eq_vs_dict.py", line 40 in __fill_dict File "Lib/test/crashers/nasty_eq_vs_dict.py", line 30 in __eq__ File "Lib/test/crashers/nasty_eq_vs_dict.py", line 47 in Segmentation fault Erreur de segmentation $ echo $? 139 ----------------- > And this test does not report anything at all: > > $ ./python Lib/test/crashers/compiler_recursion.py > [no output at all] > $ echo $? > 0 Patch version 9 doesn't call the previous handler. Please retry with patch version 10 (which call the previous handler). With the patch version 10 I get: ----------------- $ ./python Lib/test/crashers/compiler_recursion.py Fatal Python error: Segmentation fault Traceback (most recent call first): File "Lib/test/crashers/compiler_recursion.py", line 5 in Segmentation fault Erreur de segmentation $ echo $? 139 ----------------- Victor From victor.stinner at haypocalc.com Mon Dec 20 15:24:22 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Mon, 20 Dec 2010 15:24:22 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <4D0F0440.5070806@v.loewis.de> References: <4D0C0688.1080108@haypocalc.com> <1292820829.4763.16.camel@marge> <4D0F0440.5070806@v.loewis.de> Message-ID: <201012201524.23023.victor.stinner@haypocalc.com> Le lundi 20 d?cembre 2010 08:22:40, vous avez ?crit : > >> Looking at your function list, my other concern is that you are calling > >> Python API without holding the GIL, IIUC. In particular, you are > >> accessing _PyThreadState_Current, which may not point to the current > >> thread if the current thread has released the GIL. > > > > Ah? Where does _PyThreadState_Current point to if the GIL is not hold > > when the fault handler is called? > > The GIL is likely held by a different thread, then. > _PyThreadState_Current will point to the state of this other thread. I tested when the GIL released: the fault handler is unable to retrieve "the" thread state and so the backtrace is not printed. Which thread state should be retrieve? I don't know yet: maybe the last active thread state? It doesn't look trivial to fix this issue because if there is no thread (only the main thread), release the GIL clears the thread state pointer (to it to NULL) without storing it somewhere else (whereas with multiple threads, the last active thread is stored in "gil_last_holder". I should ask Antoine (or someone one) how the new GIL works. > Of this thread? I don't think so. gil_last_holder might also refer to > a different thread. Another variable might be created to store the (pointer to the) last active thread state. > > I don't think that it will possible the acquire the GIL in > > Py_FatalError() or in the fault handler. > > I agree. We all love deadlocks :-) Victor From solipsis at pitrou.net Mon Dec 20 15:31:37 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 20 Dec 2010 15:31:37 +0100 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <4D0F622E.7050902@voidspace.org.uk> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> Message-ID: <1292855497.3737.55.camel@localhost.localdomain> Le lundi 20 d?cembre 2010 ? 14:03 +0000, Michael Foord a ?crit : > On 20/12/2010 13:47, Antoine Pitrou wrote: > > Le lundi 20 d?cembre 2010 ? 13:00 +0000, Michael Foord a ?crit : > >> Ah man, we've *nearly* finished bikeshedding about the names of unittest > >> assert methods so its time to move onto the names and order of the > >> arguments. Really? > > Apparently someone decided this bikeshedding was important enough to > > make an SVN commit out of it. If you think it isn't worth discussing > > then perhaps it wasn't worth changing in the first place :) > > The only change was to use them consistently and the only code change > was to re-order the arguments in a method that is new and not in any > previous version of Python. You're arguing for a much bigger change. No, I'm first of all arguing for a "first/second" or "a/b" argument naming. Which was exactly the case before the change that triggered this thread. > >> The only thing left to decide is then the order - (actual, expected) or > >> (expected, actual). Raymond, myself, Guido and Ezio have all expressed a > >> preference for (actual, expected). I like this comment from Raymond on > >> the relevant issue [2]: > >> > >> I also tend to use actual/expected and find the reverse to be a > >> form Yoda-speak, "assert 5 == x", "perhaps misread the prophecy was", etc. > > Isn't it some kind of ethnocentric comment? "Natural" order is not the > > same in all languages, and I don't see why "actual" should come before > > "expected". > > Agreement that actual, expected was preferred came from an American, a > Dutchman, an Englishman and an Italian. :-) I'm not sure what that's supposed to prove, unless you have problems with the idea that what is natural for a couple of people isn't natural for everyone. You also apparently missed that part: > > Now if you look at various TypeErrors raised in Python, the error > > message is most often "expected, got". So there > > expected always comes before actual, and apparently it was natural to > > the authors of that code. Perhaps they are all Yoda-speakers. > As it is what unittest currently uses anyway you'll need more than "I > don't see why" to reverse it. unittest doesn't "use it anyway", since it used first/second before that change. Actually, as I pointed out, (expected, actual) is used in assertRaises in friends. > > And the problem here is that (actual, expected) is in reverse order of > > the diff displayed on error. > > > > Diffing is completely an implementation detail of how the failure > messages are generated. The important thing is that failure messages > make sense with respect to actual result and expected result. Which, again, they don't. Let's see: self.assertEqual(actual, expected) AssertionError: 'a\nb\nc\ne\n' != 'a\nb\nc\nd\n' a b c - e + d The diff shows "expected - actual", but it would be logical (in your own logic) to display "actual - expected". The whole issue disappears if you drop this idea of naming the arguments "actual" and "expected". > I'm also against dropping the use of actual / expected concepts and > terminology within unittest as I think they are useful. We don't > necessarily need to use them in the failure outputs but it seems like > you want them to be dropped altogether. I'm saying that they cause confusion wrt. to the actual error display (as Guido also admitted). Feel free to come up with a solution that doesn't get rid of actual/expected, if that's what you want ;) Regards Antoine. From stefan at bytereef.org Mon Dec 20 15:55:57 2010 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 20 Dec 2010 15:55:57 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER?environment variable In-Reply-To: <201012201439.29282.vstinner@edenwall.com> References: <4D0C0688.1080108@haypocalc.com> <20101220110835.GA13825@yoda.bytereef.org> <201012201439.29282.vstinner@edenwall.com> Message-ID: <20101220145557.GA17397@yoda.bytereef.org> Victor Stinner wrote: > Le lundi 20 d?cembre 2010 12:08:35, Stefan Krah a ?crit : > > I think the output is not particularly informative: > > > > $ ./python Lib/test/crashers/gc_inspection.py > > (, , , , , , > > , , , ) Fatal Python error: Segmentation fault > > > > Traceback (most recent call first): > > File "Lib/test/crashers/gc_inspection.py", line 29 in g > > File "Lib/test/crashers/gc_inspection.py", line 32 in > > Segmentation fault > > Segmentation fault > > The backtrace is valid. Don't you think that this backtrace is more useful > than just "Segmentation fault"? Perhaps I misunderstood, but I thought the purpose of the patch was to let developers act more quickly on bug reports. I wonder if this output really speeds up the process. Do you have an example bug where this patch helps in finding the precise location of a segfault? > Patch version 9 doesn't call the previous handler. Please retry with patch > version 10 (which call the previous handler). I did use version 10. I've verified the same behavior with a fresh py3k checkout and this patch: $ md5sum segfault_handler-10.patch e1b5a9439d2b51d0a63f701dd694796d segfault_handler-10.patch My machine currently has a load average of 2. Perhaps you'll be able to reproduce it if you crank up the load average. Stefan Krah From vstinner at edenwall.com Mon Dec 20 16:33:12 2010 From: vstinner at edenwall.com (Victor Stinner) Date: Mon, 20 Dec 2010 16:33:12 +0100 Subject: [Python-Dev] =?iso-8859-1?q?Issue_=238863_adds_a_new=09PYTHONNOFA?= =?iso-8859-1?q?ULTHANDLER=3Fenvironment_variable?= In-Reply-To: <20101220145557.GA17397@yoda.bytereef.org> References: <4D0C0688.1080108@haypocalc.com> <201012201439.29282.vstinner@edenwall.com> <20101220145557.GA17397@yoda.bytereef.org> Message-ID: <201012201633.12988.vstinner@edenwall.com> Le lundi 20 d?cembre 2010 15:55:57, Stefan Krah a ?crit : > > The backtrace is valid. Don't you think that this backtrace is more > > useful than just "Segmentation fault"? > > Perhaps I misunderstood, but I thought the purpose of the patch was to > let developers act more quickly on bug reports. I wonder if this output > really speeds up the process. The fault handler helps developers because they don't have to have a Python compiled in debug mode and to run the application in a debugger (like gdb). If the developer is unable to reproduce an error, because it's an Heisenbug, or because the developer doesn't have the same OS, libraries and/or applications, the fault handler helps the developer to isolate the problem. Many bugs only occur once and you cannot report a bug report because you don't have any useful information. But the fault handler doesn't help graphical applications not started from a terminal... > Do you have an example bug where this patch helps in finding the precise > location of a segfault? Any bug report which only contain "Segmentation fault" information. But I cannot give you a revelant example because users are too shy to report an issue if they don't have more information than just "the program crashed". With a fault handler displaying the Python backtrace, I hope that more *users* will report bugs. Example of an issue with only the "Segmentation fault" information: http://bugs.python.org/issue6011#msg121551 I am unable to reproduce this crash, and I'm very frustrated because I want to fix it! :-) Another example which might benefit from the fault handler: http://bugs.python.org/issue7424 --- Compiling /usr/pkg/lib/python2.6/test/test_builtin.py ... [1] Segmentation fault (core dumped) PYTHONPATH=/usr/... *** Error code 139 --- If the reporter already wrote a minimal script, it doesn't help. The fault handler helps to write the minimal script (to isolate the bug). > I did use version 10. I've verified the same behavior with a fresh py3k > checkout and this patch: (...) > > My machine currently has a load average of 2. Perhaps you'll be able to > reproduce it if you crank up the load average. What is your OS (name, version)? I only tested crashers on Linux (Debian Sid). Victor From rrr at ronadam.com Mon Dec 20 17:47:33 2010 From: rrr at ronadam.com (Ron Adam) Date: Mon, 20 Dec 2010 10:47:33 -0600 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> Message-ID: <4D0F88A5.9070306@ronadam.com> On 12/18/2010 04:46 PM, Terry Reedy wrote: > On 12/18/2010 3:48 PM, Antoine Pitrou wrote: >> On Sat, 18 Dec 2010 21:00:04 +0100 (CET) >> ezio.melotti wrote: >>> Author: ezio.melotti >>> Date: Sat Dec 18 21:00:04 2010 >>> New Revision: 87389 >>> >>> Log: >>> #10573: use actual/expected consistently in unittest methods. > > Change was requested by M. Foord and R. Hettinger (and G.Brandl for b2). > >> IMHO, this should be reverted. The API currently doesn't treat these >> arguments differently, so they should really be labeled "first" and >> "second". Otherwise, the user will wrongly assume that the signature is >> asymmetric and that they should be careful about which order they pass >> the arguments in. I've always presumed it would make a difference in error displayed anyway. > The error report on assert failure *is* often asymmetrical ;=). > From Michael's post: > "This is particularly relevant for the methods that produce 'diffed' output > on failure - as the order determines whether mismatched items are missing > from the expected or additional to the expected." > > This change struck me as a nice bit of polishing. I like ("actual", "expected") in the asserts. It matches my expected ordering of "input"/"output" and how I use comparisons in 'if' statements. I feel it is more important that the diffs are consistent with other diffs in python. So (for me), changing the asymmetrical output to be symmetrical would be in the category of foolish consistency because changing that, introduces other inconsistencies I'd rather not have. It doesn't bother me that the functions arguments aren't the same order of the diffs as long as the labels and wording are obvious enough in the messages. So maybe the diff output can be improved a bit instead of changing the terms and ordering. Ron From scott.urban at isilon.com Mon Dec 20 17:48:25 2010 From: scott.urban at isilon.com (Scott Urban) Date: Mon, 20 Dec 2010 08:48:25 -0800 Subject: [Python-Dev] sqlite, DDL, and transactions Message-ID: <20101220164825.GZ21083@surban.west.isilon.com> Hi The python sqlite module automatically commits open transactions when it encounters a DDL statement. This is unnecessary; DDL is transaction in my testing (see attached). Attached patch addresses the issue. Patch is against 2.6.1, but looking at Trunk in svn, it seems like the patch is needed and would apply. One issue I could foresee is that this behavior might depend on the sqlite version in use (I'm on 3.6.10). Thanks Scott -------------- next part -------------- A non-text attachment was scrubbed... Name: pysql-transactions.2.diff Type: text/x-diff Size: 2545 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: test_sqlite_ddl.py Type: text/x-python Size: 3433 bytes Desc: not available URL: From stefan at bytereef.org Mon Dec 20 17:56:32 2010 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 20 Dec 2010 17:56:32 +0100 Subject: [Python-Dev] Issue #8863 adds a new?PYTHONNOFAULTHANDLER?environment variable In-Reply-To: <201012201633.12988.vstinner@edenwall.com> References: <4D0C0688.1080108@haypocalc.com> <201012201439.29282.vstinner@edenwall.com> <20101220145557.GA17397@yoda.bytereef.org> <201012201633.12988.vstinner@edenwall.com> Message-ID: <20101220165632.GA19009@yoda.bytereef.org> Victor Stinner wrote: > Le lundi 20 d?cembre 2010 15:55:57, Stefan Krah a ?crit : > > > The backtrace is valid. Don't you think that this backtrace is more > > > useful than just "Segmentation fault"? > > > > Perhaps I misunderstood, but I thought the purpose of the patch was to > > let developers act more quickly on bug reports. I wonder if this output > > really speeds up the process. > > The fault handler helps developers because they don't have to have a Python > compiled in debug mode and to run the application in a debugger (like gdb). > > If the developer is unable to reproduce an error, because it's an Heisenbug, > or because the developer doesn't have the same OS, libraries and/or > applications, the fault handler helps the developer to isolate the problem. This is what I'm sceptical about. I think the tracebacks are don't carry enough information for that. > Any bug report which only contain "Segmentation fault" information. But I > cannot give you a revelant example because users are too shy to report an > issue if they don't have more information than just "the program crashed". > With a fault handler displaying the Python backtrace, I hope that more *users* > will report bugs. I wonder. It would be fine to make the tracebacks optional and tell the user to rerun the program with the proper environment variable set. > > I did use version 10. I've verified the same behavior with a fresh py3k > > checkout and this patch: (...) > > > > My machine currently has a load average of 2. Perhaps you'll be able to > > reproduce it if you crank up the load average. > > What is your OS (name, version)? I only tested crashers on Linux (Debian Sid). Ubuntu 8.10, 64-bit, real hardware. I also tested on FreeBSD-8.0-qemu: $ ./python Lib/test/crashers/compiler_recursion.py Illegal instruction: 4 (core dumped) $ ./python Lib/test/crashers/nasty_eq_vs_dict.py [hanging with no output] Stefan Krah From brian.curtin at gmail.com Mon Dec 20 18:03:23 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Mon, 20 Dec 2010 11:03:23 -0600 Subject: [Python-Dev] sqlite, DDL, and transactions In-Reply-To: <20101220164825.GZ21083@surban.west.isilon.com> References: <20101220164825.GZ21083@surban.west.isilon.com> Message-ID: On Mon, Dec 20, 2010 at 10:48, Scott Urban wrote: > Hi > > The python sqlite module automatically commits open transactions > when it encounters a DDL statement. This is unnecessary; DDL is > transaction in my testing (see attached). > > Attached patch addresses the issue. Patch is against 2.6.1, but > looking at Trunk in svn, it seems like the patch is needed and > would apply. One issue I could foresee is that this behavior might > depend on the sqlite version in use (I'm on 3.6.10). > > Thanks > Scott Can you submit this to bugs.python.org? Bug reports and their patches on the mailing list tend to get lost. Additionally, 2.7 would be the 2.x version to fix this against (branches/release27-maint when looking in svn). -------------- next part -------------- An HTML attachment was scrubbed... URL: From fuzzyman at voidspace.org.uk Mon Dec 20 18:03:51 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 20 Dec 2010 17:03:51 +0000 Subject: [Python-Dev] sqlite, DDL, and transactions In-Reply-To: <20101220164825.GZ21083@surban.west.isilon.com> References: <20101220164825.GZ21083@surban.west.isilon.com> Message-ID: <4D0F8C77.6020704@voidspace.org.uk> On 20/12/2010 16:48, Scott Urban wrote: > Hi > > The python sqlite module automatically commits open transactions > when it encounters a DDL statement. This is unnecessary; DDL is > transaction in my testing (see attached). > Hello Scott, Please post your patch to the Python bug tracker. http://bugs.python.org/ All the best, Michael Foord > Attached patch addresses the issue. Patch is against 2.6.1, but > looking at Trunk in svn, it seems like the patch is needed and > would apply. One issue I could foresee is that this behavior might > depend on the sqlite version in use (I'm on 3.6.10). > > Thanks > Scott > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe:http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessinghttp://www.sqlite.org/different.html -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Mon Dec 20 18:22:48 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 20 Dec 2010 12:22:48 -0500 Subject: [Python-Dev] Issue #8863 adds a new?PYTHONNOFAULTHANDLER?environment variable In-Reply-To: <20101220165632.GA19009@yoda.bytereef.org> References: <4D0C0688.1080108@haypocalc.com> <201012201439.29282.vstinner@edenwall.com> <20101220145557.GA17397@yoda.bytereef.org> <201012201633.12988.vstinner@edenwall.com> <20101220165632.GA19009@yoda.bytereef.org> Message-ID: On Mon, Dec 20, 2010 at 11:56 AM, Stefan Krah wrote: > Victor Stinner wrote: .. >> The fault handler helps developers because they don't have to have a Python >> compiled in debug mode and to run the application in a debugger (like gdb). >> >> If the developer is unable to reproduce an error, because it's an Heisenbug, >> or because the developer doesn't have the same OS, libraries and/or >> applications, the fault handler helps the developer to isolate the problem. > > This is what I'm sceptical about. I think the tracebacks are don't carry > enough information for that. > My concern is that the traceback generated by a signal handler may actually contain *less* information that what would otherwise be available from the core dump or other system diagnostic facilities. Whenever you continue to execute code after a fault occurs (even in an alternative stack), you change register values and run the risk that the program will not be able to produce a core dump at all or will produce a core dump with misleading information in it. I think Stefan's approach is the correct one: before we agree on this feature, the proponents should present actual crashers for which reports generated by the patch are actually helpful. From rrr at ronadam.com Mon Dec 20 17:47:33 2010 From: rrr at ronadam.com (Ron Adam) Date: Mon, 20 Dec 2010 10:47:33 -0600 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> Message-ID: <4D0F88A5.9070306@ronadam.com> On 12/18/2010 04:46 PM, Terry Reedy wrote: > On 12/18/2010 3:48 PM, Antoine Pitrou wrote: >> On Sat, 18 Dec 2010 21:00:04 +0100 (CET) >> ezio.melotti wrote: >>> Author: ezio.melotti >>> Date: Sat Dec 18 21:00:04 2010 >>> New Revision: 87389 >>> >>> Log: >>> #10573: use actual/expected consistently in unittest methods. > > Change was requested by M. Foord and R. Hettinger (and G.Brandl for b2). > >> IMHO, this should be reverted. The API currently doesn't treat these >> arguments differently, so they should really be labeled "first" and >> "second". Otherwise, the user will wrongly assume that the signature is >> asymmetric and that they should be careful about which order they pass >> the arguments in. I've always presumed it would make a difference in error displayed anyway. > The error report on assert failure *is* often asymmetrical ;=). > From Michael's post: > "This is particularly relevant for the methods that produce 'diffed' output > on failure - as the order determines whether mismatched items are missing > from the expected or additional to the expected." > > This change struck me as a nice bit of polishing. I like ("actual", "expected") in the asserts. It matches my expected ordering of "input"/"output" and how I use comparisons in 'if' statements. I feel it is more important that the diffs are consistent with other diffs in python. So (for me), changing the asymmetrical output to be symmetrical would be in the category of foolish consistency because changing that, introduces other inconsistencies I'd rather not have. It doesn't bother me that the functions arguments aren't the same order of the diffs as long as the labels and wording are obvious enough in the messages. So maybe the diff output can be improved a bit instead of changing the terms and ordering. Ron From gzlist at googlemail.com Mon Dec 20 18:31:24 2010 From: gzlist at googlemail.com (Martin (gzlist)) Date: Mon, 20 Dec 2010 17:31:24 +0000 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <4D0EA9E0.4080707@voidspace.org.uk> References: <4D0C0688.1080108@haypocalc.com> <1292799745.5914.76.camel@marge> <4D0EA9E0.4080707@voidspace.org.uk> Message-ID: On 20/12/2010, Michael Foord wrote: > On 19/12/2010 23:02, Victor Stinner wrote: >> I don't think that kill() is available on Windows. Can just use raise instead. But, is any of this change valid on Windows? "The SIGILL, SIGSEGV, and SIGTERM signals are not generated under Windows NT. They are included for ANSI compatibility." And the last signal this concerns is explicitly skipped in the tests on the latest patch under windows with the note "SIGFPE cannot be catched on Windows". Signals are a posix compatibility hack on win32, they're unlikely to do what you want. > Thanks to Brian Curtin there is now an os.kill() on Windows, but it > doesn't handle the full range of signals. Which uses GenerateConsoleCtrlEvent and is completely the wrong thing. Martin From vstinner at edenwall.com Mon Dec 20 18:34:19 2010 From: vstinner at edenwall.com (Victor Stinner) Date: Mon, 20 Dec 2010 18:34:19 +0100 Subject: [Python-Dev] =?iso-8859-1?q?Issue_=238863_adds_a_new_PYTHONNOFAUL?= =?iso-8859-1?q?THANDLER=09environment_variable?= In-Reply-To: <20101220110835.GA13825@yoda.bytereef.org> References: <4D0C0688.1080108@haypocalc.com> <20101220110835.GA13825@yoda.bytereef.org> Message-ID: <201012201834.19264.vstinner@edenwall.com> Le lundi 20 d?cembre 2010 12:08:35, Stefan Krah a ?crit : > Another test is hanging indefinitely (Ubuntu 64-bit): > > $ ./python Lib/test/crashers/nasty_eq_vs_dict.py > [hanging with no output] Oh, I understood. I always test with Python compiled using --with-pydebug. With pydebug, a segfault occurs. Without pydebug, there is an unlimited loop. The patch is not related to the lack of segfault :-) (Retry without the patch) Victor From vstinner at edenwall.com Mon Dec 20 18:51:09 2010 From: vstinner at edenwall.com (Victor Stinner) Date: Mon, 20 Dec 2010 18:51:09 +0100 Subject: [Python-Dev] Issue #8863 adds a new?PYTHONNOFAULTHANDLER?environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> <20101220165632.GA19009@yoda.bytereef.org> Message-ID: <201012201851.09810.vstinner@edenwall.com> Le lundi 20 d?cembre 2010 18:22:48, Alexander Belopolsky a ?crit : > On Mon, Dec 20, 2010 at 11:56 AM, Stefan Krah wrote: > > Victor Stinner wrote: > .. > > >> The fault handler helps developers because they don't have to have a > >> Python compiled in debug mode and to run the application in a debugger > >> (like gdb). > >> > >> If the developer is unable to reproduce an error, because it's an > >> Heisenbug, or because the developer doesn't have the same OS, libraries > >> and/or applications, the fault handler helps the developer to isolate > >> the problem. > > > > This is what I'm sceptical about. I think the tracebacks are don't carry > > enough information for that. Yes, in some cases the backtrace doesn't help. But why don't you want to display the backtrace if it can help in some other cases? > My concern is that the traceback generated by a signal handler may > actually contain *less* information that what would otherwise be > available from the core dump or other system diagnostic facilities. I changed my patch to call the previous signal handler starting at version 10. So the core dump function or OS fault handler will be called two. Said differently, it only gives *more* information, not less. If you have a better OS fault handler than Python fault handler, you should disable Python fault handler by setting the environment variable. > Whenever you continue to execute code after a fault occurs (even in > an alternative stack), you change register values and run the risk > that the program will not be able to produce a core dump at all or > will produce a core dump with misleading information in it. Are you sure that a signal handler changes the registers? At least Linux saves integer and float registers before calling the signal handler, and then restores them. The fault handler only changes some registers and write some bytes on the stack. The stack pointer is restored at exit, so I don't think that it causes troubles to use a core dump. Dump a core file doesn't rely on process registers or mappings. Why do you think that it will break the core dump function? Victor From scott.urban at isilon.com Mon Dec 20 19:05:17 2010 From: scott.urban at isilon.com (Scott Urban) Date: Mon, 20 Dec 2010 10:05:17 -0800 Subject: [Python-Dev] sqlite, DDL, and transactions In-Reply-To: <4D0F8C77.6020704@voidspace.org.uk> References: <20101220164825.GZ21083@surban.west.isilon.com> <4D0F8C77.6020704@voidspace.org.uk> Message-ID: <20101220180517.GB21083@surban.west.isilon.com> On Mon, Dec 20, 2010 at 17:03:51, Michael Foord sent: > On 20/12/2010 16:48, Scott Urban wrote: > >Hi > > > >The python sqlite module automatically commits open transactions > >when it encounters a DDL statement. This is unnecessary; DDL is > >transaction in my testing (see attached). > > > > Hello Scott, > > Please post your patch to the Python bug tracker. > > http://bugs.python.org/ http://bugs.python.org/issue10740 Thanks Scott From alexander.belopolsky at gmail.com Mon Dec 20 19:09:32 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 20 Dec 2010 13:09:32 -0500 Subject: [Python-Dev] Issue #8863 adds a new?PYTHONNOFAULTHANDLER?environment variable In-Reply-To: <201012201851.09810.vstinner@edenwall.com> References: <4D0C0688.1080108@haypocalc.com> <20101220165632.GA19009@yoda.bytereef.org> <201012201851.09810.vstinner@edenwall.com> Message-ID: On Mon, Dec 20, 2010 at 12:51 PM, Victor Stinner wrote: .. > Are you sure that a signal handler changes the registers? At least Linux saves > integer and float registers before calling the signal handler, and then > restores them. > What if the program crashes (again) in the signal handler before having a chance to restore registers? Also, can you point to documentation that describes register restoring behavior? I am not sure all registers are (or even can be) restored. > The fault handler only changes some registers and write some bytes on the > stack. The stack pointer is restored at exit, so I don't think that it causes > troubles to use a core dump. > > Dump a core file doesn't rely on process registers or mappings. Why do you > think that it will break the core dump function? Because I had actual experience with that using R. Your code may be better, but R brings up a dialog on segfault asking whether to produce a core dump. Occasionally, that core dump would be completely unusable because it would point to an unrelated point in the code. I don't have a specific case handy because long ago I disabled segfault handlers in the local R installation. (I actually use RPy which loads R into Python as a shared library, so the two handlers will conflict anyways.) From rdmurray at bitdance.com Mon Dec 20 19:46:01 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 20 Dec 2010 13:46:01 -0500 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER?environment variable In-Reply-To: <20101220145557.GA17397@yoda.bytereef.org> References: <4D0C0688.1080108@haypocalc.com> <20101220110835.GA13825@yoda.bytereef.org> <201012201439.29282.vstinner@edenwall.com> <20101220145557.GA17397@yoda.bytereef.org> Message-ID: <20101220184601.955F62002A9@kimball.webabinitio.net> On Mon, 20 Dec 2010 15:55:57 +0100, Stefan Krah wrote: > Victor Stinner wrote: > > Stefan Krah wrote: > > > I think the output is not particularly informative: > > > > > > $ ./python Lib/test/crashers/gc_inspection.py > > > (, , , , , , > > > , , , ) Fatal Python error: Segmentation fault > > > > > > Traceback (most recent call first): > > > File "Lib/test/crashers/gc_inspection.py", line 29 in g > > > File "Lib/test/crashers/gc_inspection.py", line 32 in > > > Segmentation fault > > > Segmentation fault > > > The backtrace is valid. Don't you think that this backtrace is more useful > > than just "Segmentation fault"? > > Perhaps I misunderstood, but I thought the purpose of the patch was to > let developers act more quickly on bug reports. I wonder if this output > really speeds up the process. > > Do you have an example bug where this patch helps in finding the precise > location of a segfault? How is 'line 29 in g' not more precise than 'Segmentation fault'? Now imagine that this error is being produced from a 20,000 line application program bundle... -- R. David Murray www.bitdance.com From stefan at bytereef.org Mon Dec 20 20:01:33 2010 From: stefan at bytereef.org (Stefan Krah) Date: Mon, 20 Dec 2010 20:01:33 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER?environment variable In-Reply-To: <201012201834.19264.vstinner@edenwall.com> References: <4D0C0688.1080108@haypocalc.com> <20101220110835.GA13825@yoda.bytereef.org> <201012201834.19264.vstinner@edenwall.com> Message-ID: <20101220190133.GA20442@yoda.bytereef.org> Victor Stinner wrote: > Le lundi 20 d?cembre 2010 12:08:35, Stefan Krah a ?crit : > > Another test is hanging indefinitely (Ubuntu 64-bit): > > > > $ ./python Lib/test/crashers/nasty_eq_vs_dict.py > > [hanging with no output] > > Oh, I understood. I always test with Python compiled using --with-pydebug. > With pydebug, a segfault occurs. Without pydebug, there is an unlimited loop. I'll open a feature request for more reliable segfaults. ;) Still, on FreeBSD-8.0 the patch does not produce a traceback and also appears to change the signal: [without the patch] $ ./python Lib/test/crashers/compiler_recursion.py Segmentation fault: 11 (core dumped) $ echo $? 139 [with the patch] $ ./python Lib/test/crashers/compiler_recursion.py Illegal instruction: 4 (core dumped) $ echo $? 132 Stefan Krah From v+python at g.nevcal.com Mon Dec 20 21:41:04 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 20 Dec 2010 12:41:04 -0800 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <1292855497.3737.55.camel@localhost.localdomain> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> Message-ID: <4D0FBF60.1040300@g.nevcal.com> On 12/20/2010 6:31 AM, Antoine Pitrou wrote: >> > Diffing is completely an implementation detail of how the failure >> > messages are generated. The important thing is that failure messages >> > make sense with respect to actual result and expected result. > Which, again, they don't. Let's see: > > self.assertEqual(actual, expected) > AssertionError: 'a\nb\nc\ne\n' != 'a\nb\nc\nd\n' > a > b > c > - e > + d > > The diff shows "expected - actual", but it would be logical (in your own > logic) to display "actual - expected". The whole issue disappears if you > drop this idea of naming the arguments "actual" and "expected". I'm not a unittest user, although I probably will become one, in time, when I learn enough to contribute to Python, instead of just find bugs in it from use. I don't much care what the parameters names are, although the terms actual and expected seem good for testing scenarios if properly used, but the above does not match what I would expect the behavior to be from a testing scenario: run the test, and tell me what changed from the expected results. If the expected result (not parameter) is d and the actual result (not parameter) is e, the diff should show a b c - d + e Thinking-that-sometimes-a-novice's-expectations-are-relevant-to-such-discussions'ly yours, Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Dec 20 23:29:50 2010 From: guido at python.org (Guido van Rossum) Date: Mon, 20 Dec 2010 14:29:50 -0800 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <4D0F27FE.4020706@pearwood.info> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <20101220014840.6b734274@pitrou.net> <4D0F27FE.4020706@pearwood.info> Message-ID: On Mon, Dec 20, 2010 at 1:55 AM, Steven D'Aprano wrote: > Antoine Pitrou wrote: > >> For a non-native English speaker, 'a' and 'b' don't evoke 'after' and >> 'before' but simply the first two letters of the latin alphabet, and >> their ordering is therefore obvious with respect to function arguments. > > It's not just non-native English speakers either. I too think of a, b as > being first, second rather than after, before. I was mostly being facetious (my main point being there's no perfect solution here), though I *have* seen serious code using the b=before/a=after convention. -- --Guido van Rossum (python.org/~guido) From martin at v.loewis.de Tue Dec 21 00:00:39 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 21 Dec 2010 00:00:39 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER?environment variable In-Reply-To: <20101220184601.955F62002A9@kimball.webabinitio.net> References: <4D0C0688.1080108@haypocalc.com> <20101220110835.GA13825@yoda.bytereef.org> <201012201439.29282.vstinner@edenwall.com> <20101220145557.GA17397@yoda.bytereef.org> <20101220184601.955F62002A9@kimball.webabinitio.net> Message-ID: <4D0FE017.50509@v.loewis.de> >> Do you have an example bug where this patch helps in finding the precise >> location of a segfault? > > How is 'line 29 in g' not more precise than 'Segmentation fault'? I think Stefan is treating "precise location" as an absolute property, not a relative one. The precise location would be the very instruction causing the fault; this is something that a core dump provides, unless a signal handler interferes with that. In many crashes I fixed, the Python stack trace was irrelevant to understand the nature of the crash. Of course, there were also counter- examples where you'ld need the Python stack (or, rather, the entire interpreter trace) to understand how the crashing case can occur in the first place. Regards, Martin From martin at v.loewis.de Tue Dec 21 00:05:31 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 21 Dec 2010 00:05:31 +0100 Subject: [Python-Dev] Issue #8863 adds a new PYTHONNOFAULTHANDLER environment variable In-Reply-To: <201012201524.23023.victor.stinner@haypocalc.com> References: <4D0C0688.1080108@haypocalc.com> <1292820829.4763.16.camel@marge> <4D0F0440.5070806@v.loewis.de> <201012201524.23023.victor.stinner@haypocalc.com> Message-ID: <4D0FE13B.2020003@v.loewis.de> >> The GIL is likely held by a different thread, then. >> _PyThreadState_Current will point to the state of this other thread. > > I tested when the GIL released: the fault handler is unable to retrieve "the" > thread state and so the backtrace is not printed. Which thread state should be > retrieve? I don't know yet: maybe the last active thread state? > > It doesn't look trivial to fix this issue because if there is no thread (only > the main thread), release the GIL clears the thread state pointer (to it to > NULL) without storing it somewhere else (whereas with multiple threads, the > last active thread is stored in "gil_last_holder". I should ask Antoine (or > someone one) how the new GIL works. I don't think this is possible to fix. Just put a sleep() call before the crash, and have a parallel thread perform a while-loop to see what I mean: it should give you a stack trace of the wrong thread, possibly touching transient objects which go away while you inspect them. > Another variable might be created to store the (pointer to the) last active > thread state. Please understand: the last active thread may not be the one that got the signal. Instead, it may be another thread that keeps running while your thread crashes. Regards, Martin From ncoghlan at gmail.com Tue Dec 21 02:38:21 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Dec 2010 12:38:21 +1100 Subject: [Python-Dev] Search-friendly shortcuts for Windows? Message-ID: Something I noticed after installing the 3.2 beta on my Windows 7 laptop is that the start menu shortcuts aren't particularly search friendly. Searching loses the heirarchical information, so attempting to directly locate "pyt" provides two separate "Python Command Line" shortcuts, with no indicator as to which one will start which version. Given the changing dynamics of the desktop launch menus to better support direct access as an alternative to hierarchical navigation, would it be reasonable to consider including the major version number in the start menu shortcut names? (Question is mainly for Martin, obviously, but I'm also curious if anyone else has encountered the same thing) Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Dec 21 02:57:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Dec 2010 12:57:50 +1100 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <1292855497.3737.55.camel@localhost.localdomain> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> Message-ID: On Tue, Dec 21, 2010 at 1:31 AM, Antoine Pitrou wrote: >> Diffing is completely an implementation detail of how the failure >> messages are generated. The important thing is that failure messages >> make sense with respect to actual result and expected result. > > Which, again, they don't. Let's see: > > ? ?self.assertEqual(actual, expected) > AssertionError: 'a\nb\nc\ne\n' != 'a\nb\nc\nd\n' > ?a > ?b > ?c > - e > + d > > The diff shows "expected - actual", but it would be logical (in your own > logic) to display "actual - expected". The whole issue disappears if you > drop this idea of naming the arguments "actual" and "expected". To make this a bit clearer... >>> class Ex(ut.TestCase): ... def demo(self): ... self.assertEqual("actual", "expected") ... >>> Ex("demo").demo() Traceback (most recent call last): AssertionError: 'actual' != 'expected' - actual + expected For the actual/expected terminology the diff is the wrong way around (as of 3.2b1, anyway). My own +1 goes to keeping the actual/expected terminology (and ordering) and adjusting the diffs accordingly (with a header noting that the diff is old=expected, new=actual). assertRaises() *is* an exception to the general actual/expected pattern, but that asymmetry is forced by the ability to pass arbitrary positional arguments to the function being tested (which later proved useful for the context manager form as well). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Tue Dec 21 06:45:07 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 21 Dec 2010 00:45:07 -0500 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: References: Message-ID: On 12/20/2010 8:38 PM, Nick Coghlan wrote: > Given the changing dynamics of the desktop launch menus to better > support direct access as an alternative to hierarchical navigation, > would it be reasonable to consider including the major version number > in the start menu shortcut names? I would very much like that. I once changed some the entries but bugfix or later alpha/beta/gamma/final installation reset everything. > (Question is mainly for Martin, obviously, but I'm also curious if > anyone else has encountered the same thing) Now that I have multiple versions of Python installed (for development testing), I have *three* identical 'IDLE (Python GUI)' entries in the left-hand frequently-used programs column of the Start panel. This is really useless and therefore a waste of space. -- Terry Jan Reedy From scott+python-dev at scottdial.com Tue Dec 21 06:47:09 2010 From: scott+python-dev at scottdial.com (Scott Dial) Date: Tue, 21 Dec 2010 00:47:09 -0500 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: References: Message-ID: <4D103F5D.7060300@scottdial.com> On 12/20/2010 8:38 PM, Nick Coghlan wrote: > would it be reasonable to consider including the major version number > in the start menu shortcut names? +1 First thing I did was add an x.y prefix to the Python shortcuts. There are a lot of application shortcuts on my Win7 system that have version numbers that have no naming conflicts, so even if a single version of Python was installed, it would not look out of place at all. -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From martin at v.loewis.de Tue Dec 21 08:36:43 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 21 Dec 2010 08:36:43 +0100 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: References: Message-ID: <4D10590B.9040208@v.loewis.de> > Given the changing dynamics of the desktop launch menus to better > support direct access as an alternative to hierarchical navigation, > would it be reasonable to consider including the major version number > in the start menu shortcut names? > > (Question is mainly for Martin, obviously, but I'm also curious if > anyone else has encountered the same thing) I can't see anything wrong with that, but I'm terrible with wording. So somebody would have to spell out the exact wording that each of the start menu items should have. In doing so, also take into consideration that some people install 32-bit and 64-bit versions simultaneously. Regards, Martin From ncoghlan at gmail.com Tue Dec 21 13:01:00 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 21 Dec 2010 23:01:00 +1100 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: <4D10590B.9040208@v.loewis.de> References: <4D10590B.9040208@v.loewis.de> Message-ID: On Tue, Dec 21, 2010 at 6:36 PM, "Martin v. L?wis" wrote: >> Given the changing dynamics of the desktop launch menus to better >> support direct access as an alternative to hierarchical navigation, >> would it be reasonable to consider including the major version number >> in the start menu shortcut names? >> >> (Question is mainly for Martin, obviously, but I'm also curious if >> anyone else has encountered the same thing) > > I can't see anything wrong with that, but I'm terrible with wording. > So somebody would have to spell out the exact wording that each of > the start menu items should have. In doing so, also take into > consideration that some people install 32-bit and 64-bit versions > simultaneously. I put my initial proposal (reproduced below) at http://bugs.python.org/issue10747. I'll let this thread run a bit and see if there are any substantial improvements suggested, or valid objections raised (I tried to pre-empt the length issue by pushing the typically least important information to the end). I also considered using the x86 and x64 abbreviations, but being explicit seemed clearer. Shortcuts currently installed: - Python (command line) - Python Manuals - Module Docs - IDLE (Python GUI) - Uninstall Python Initial proposal for 32 bit builds: - Python 3.2 (command line - 32 bit) - Python 3.2 Manuals - Python 3.2 Docs Server (pydoc - 32 bit) - IDLE (Python 3.2 GUI - 32 bit) - Uninstall Python 3.2 (32 bit) Initial proposal for 64 bit builds: - Python 3.2 (command line - 64 bit) - Python 3.2 Manuals - Python 3.2 Docs Server (pydoc - 64 bit) - IDLE (Python 3.2 GUI - 64 bit) - Uninstall Python 3.2 (64 bit) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From fuzzyman at voidspace.org.uk Tue Dec 21 13:17:46 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 21 Dec 2010 12:17:46 +0000 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> Message-ID: <4D109AEA.7010908@voidspace.org.uk> On 21/12/2010 01:57, Nick Coghlan wrote: > On Tue, Dec 21, 2010 at 1:31 AM, Antoine Pitrou wrote: >>> Diffing is completely an implementation detail of how the failure >>> messages are generated. The important thing is that failure messages >>> make sense with respect to actual result and expected result. >> Which, again, they don't. Let's see: >> >> self.assertEqual(actual, expected) >> AssertionError: 'a\nb\nc\ne\n' != 'a\nb\nc\nd\n' >> a >> b >> c >> - e >> + d >> >> The diff shows "expected - actual", but it would be logical (in your own >> logic) to display "actual - expected". The whole issue disappears if you >> drop this idea of naming the arguments "actual" and "expected". > To make this a bit clearer... > >>>> class Ex(ut.TestCase): > ... def demo(self): > ... self.assertEqual("actual", "expected") > ... >>>> Ex("demo").demo() > Traceback (most recent call last): > > AssertionError: 'actual' != 'expected' > - actual > + expected > > For the actual/expected terminology the diff is the wrong way around > (as of 3.2b1, anyway). > The recent commit that sparked the controversy was supposed to ensure that all the asserts were documented consistently *and* worked as per the documentation. The error above is from assertMultiLineEqual. assertListEqual has the same issue: >>> t.assertListEqual([1], [2]) Traceback (most recent call last): ... AssertionError: Lists differ: [1] != [2] First differing element 0: 1 2 - [1] + [2] Interestingly assertSetEqual already uses the first/second symmetric wording: >>> t.assertSetEqual({1}, {2}) ... AssertionError: Items in the first set but not the second: 1 Items in the second set but not the first: 2 > My own +1 goes to keeping the actual/expected terminology (and > ordering) and adjusting the diffs accordingly (with a header noting > that the diff is old=expected, new=actual). > Well we don't have consensus. Whatever we do we need to be consistent, and in the absence of an agreement about a change we should at least make all the behaviour and documentation consistent. From this discussion and the discussion on the issue tracker: Myself, Nick Coghlan and Ezio Melotti prefer (actual, expected) Raymond like (actual, expected) but would be happy with symmetrical diffs Guido prefers the (actual, expected) ordering but prefers diffs to show the other way round R David Murray agreed with Guido Terry Reedy liked the change Glenn Linderman wants (actual, expected) and diffing to follow that Ron Adam ditto Symmetrical diffs (element in first not in second, element in second not in first) solves the problem without imposing an order on the arguments. Actually unittest *has* used (first, second) to refer to the arguments to asserts pretty much since its inception. Losing the (actual, expected) terminology is a cost of this but unittest hasn't emphasised this terminology in the past (as I thought it had). This won't work for diffing strings (assertMultiLineEqual) which use difflib and needs a direction for the diff. As above it is currently the wrong way round for (actual, expected). The other alternative is to make them consistent and follow Nick's suggestion adding the header note to the diffs that old=expected, new=actual. > assertRaises() *is* an exception to the general actual/expected > pattern, but that asymmetry is forced by the ability to pass arbitrary > positional arguments to the function being tested (which later proved > useful for the context manager form as well). The (actual, expected) pattern matches the way almost everyone I've ever seen write if statements and asserts: if x == 5: rather than if 5 == x: assert x == 5 rather than assert 5 == x It also matches functions like isinstance and issubclass. On the other hand it doesn't match the way we report TypeErrors where we report "expected , got ". All the best, Michael Foord > Cheers, > Nick. > -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From georg at python.org Tue Dec 21 20:18:56 2010 From: georg at python.org (Georg Brandl) Date: Tue, 21 Dec 2010 20:18:56 +0100 Subject: [Python-Dev] [RELEASED] Python 3.2 beta 2 Message-ID: <4D10FDA0.7090808@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team, I'm happy to announce the second beta preview release of Python 3.2. Python 3.2 is a continuation of the efforts to improve and stabilize the Python 3.x line. Since the final release of Python 2.7, the 2.x line will only receive bugfixes, and new features are developed for 3.x only. Since PEP 3003, the Moratorium on Language Changes, is in effect, there are no changes in Python's syntax and built-in types in Python 3.2. Development efforts concentrated on the standard library and support for porting code to Python 3. Highlights are: * numerous improvements to the unittest module * PEP 3147, support for .pyc repository directories * PEP 3149, support for version tagged dynamic libraries * PEP 3148, a new futures library for concurrent programming * PEP 384, a stable ABI for extension modules * PEP 391, dictionary-based logging configuration * an overhauled GIL implementation that reduces contention * an extended email package that handles bytes messages * countless fixes regarding bytes/string issues; among them full support for a bytes environment (filenames, environment variables) * many consistency and behavior fixes for numeric operations * a sysconfig module to access configuration information * a pure-Python implementation of the datetime module * additions to the shutil module, among them archive file support * improvements to pdb, the Python debugger For a more extensive list of changes in 3.2, see http://docs.python.org/3.2/whatsnew/3.2.html To download Python 3.2 visit: http://www.python.org/download/releases/3.2/ Please consider trying Python 3.2 with your code and reporting any bugs you may notice to: http://bugs.python.org/ Enjoy! - -- Georg Brandl, Release Manager georg at python.org (on behalf of the entire python-dev team and 3.2's contributors) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk0Q/aAACgkQN9GcIYhpnLDf8gCgkLGAsE+T3R505jZc1RxXDYsa NSsAnRGaFjeTm9o2Z5O8FuIzTUG8t1PT =hHzz -----END PGP SIGNATURE----- From v+python at g.nevcal.com Tue Dec 21 21:27:50 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 21 Dec 2010 12:27:50 -0800 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <4D109AEA.7010908@voidspace.org.uk> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> Message-ID: <4D110DC6.6060905@g.nevcal.com> On 12/21/2010 4:17 AM, Michael Foord wrote: > Glenn Linderman wants (actual, expected) and diffing to follow that If you say that is what I said, fine. I might not have understood the example well enough to say the right thing. I liked Nick's explanation, using the actual and expected words in his example, but -expected +actual -old +new is what I would expect in the diff. I didn't say anything about the parameters, I don't care, except that the documentation leads me to use it correctly, so that I get the above diff results. Sadly, if the diff results are not as above, I would probably misuse the parameters to achieve it, unless doing something for which the standard is to do it "backwards". From guido at python.org Tue Dec 21 21:40:16 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Dec 2010 12:40:16 -0800 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <4D109AEA.7010908@voidspace.org.uk> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> Message-ID: On Tue, Dec 21, 2010 at 4:17 AM, Michael Foord wrote: > On 21/12/2010 01:57, Nick Coghlan wrote: >> >> On Tue, Dec 21, 2010 at 1:31 AM, Antoine Pitrou >> ?wrote: >>>> >>>> Diffing is completely an implementation detail of how the failure >>>> messages are generated. The important thing is that failure messages >>>> make sense with respect to actual result and expected result. >>> >>> Which, again, they don't. Let's see: >>> >>> ? ?self.assertEqual(actual, expected) >>> AssertionError: 'a\nb\nc\ne\n' != 'a\nb\nc\nd\n' >>> ?a >>> ?b >>> ?c >>> - e >>> + d >>> >>> The diff shows "expected - actual", but it would be logical (in your own >>> logic) to display "actual - expected". The whole issue disappears if you >>> drop this idea of naming the arguments "actual" and "expected". >> >> To make this a bit clearer... >> >>>>> class Ex(ut.TestCase): >> >> ... ? def demo(self): >> ... ? ? self.assertEqual("actual", "expected") >> ... >>>>> >>>>> Ex("demo").demo() >> >> Traceback (most recent call last): >> ? >> AssertionError: 'actual' != 'expected' >> - actual >> + expected >> >> For the actual/expected terminology the diff is the wrong way around >> (as of 3.2b1, anyway). >> > > The recent commit that sparked the controversy was supposed to ensure that > all the asserts were documented consistently *and* worked as per the > documentation. The error above is from assertMultiLineEqual. > > assertListEqual has the same issue: > >>>> t.assertListEqual([1], [2]) > Traceback (most recent call last): > ?... > AssertionError: Lists differ: [1] != [2] > > First differing element 0: > 1 > 2 > > - [1] > + [2] > > Interestingly assertSetEqual already uses the first/second symmetric > wording: > >>>> t.assertSetEqual({1}, {2}) > ?... > AssertionError: Items in the first set but not the second: > 1 > Items in the second set but not the first: > 2 > > >> My own +1 goes to keeping the actual/expected terminology (and >> ordering) and adjusting the diffs accordingly (with a header noting >> that the diff is old=expected, new=actual). >> > > Well we don't have consensus. Whatever we do we need to be consistent, and > in the absence of an agreement about a change we should at least make all > the behaviour and documentation consistent. > > From this discussion and the discussion on the issue tracker: > > Myself, Nick Coghlan and Ezio Melotti prefer (actual, expected) > Raymond like (actual, expected) but would be happy with symmetrical diffs > Guido prefers the (actual, expected) ordering but prefers diffs to show the > other way round Actually I said there was no right answer. I certainly do not want the diff output to treat the second arg as "old" and the first one as "new" -- that would be just as confusing. All in all I'd like to get rid of any vestiges of actual and expected; I think the first/second wording is the best we can come up with it. > R David Murray agreed with Guido That's hard to believe since I don't agree with myself. :-) > Terry Reedy liked the change > Glenn Linderman wants (actual, expected) and diffing to follow that > Ron Adam ditto > > Symmetrical diffs (element in first not in second, element in second not in > first) solves the problem without imposing an order on the arguments. > Actually unittest *has* used (first, second) to refer to the arguments to > asserts pretty much since its inception. Losing the (actual, expected) > terminology is a cost of this but unittest hasn't emphasised this > terminology in the past (as I thought it had). > > This won't work for diffing strings (assertMultiLineEqual) which use difflib > and needs a direction for the diff. As above it is currently the wrong way > round for (actual, expected). > > The other alternative is to make them consistent and follow Nick's > suggestion adding the header note to the diffs that old=expected, > new=actual. > >> assertRaises() *is* an exception to the general actual/expected >> pattern, but that asymmetry is forced by the ability to pass arbitrary >> positional arguments to the function being tested (which later proved >> useful for the context manager form as well). > > The (actual, expected) pattern matches the way almost everyone I've ever > seen write if statements and asserts: > > ? ?if x == 5: rather than if 5 == x: > ? ?assert x == 5 rather than assert 5 == x > > It also matches functions like isinstance and issubclass. > > On the other hand it doesn't match the way we report TypeErrors where we > report "expected , got ". > > All the best, > > Michael Foord >> >> Cheers, >> Nick. >> > > > -- > > http://www.voidspace.org.uk/ > > May you do good and not evil > May you find forgiveness for yourself and forgive others > May you share freely, never taking more than you give. > -- the sqlite blessing http://www.sqlite.org/different.html > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) From ncoghlan at gmail.com Wed Dec 22 02:12:29 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Dec 2010 12:12:29 +1100 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <4D109AEA.7010908@voidspace.org.uk> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> Message-ID: On Tue, Dec 21, 2010 at 11:17 PM, Michael Foord wrote: > On 21/12/2010 01:57, Nick Coghlan wrote: >> My own +1 goes to keeping the actual/expected terminology (and >> ordering) and adjusting the diffs accordingly (with a header noting >> that the diff is old=expected, new=actual). >> > > Well we don't have consensus. Whatever we do we need to be consistent, and > in the absence of an agreement about a change we should at least make all > the behaviour and documentation consistent. > > From this discussion and the discussion on the issue tracker: > > Myself, Nick Coghlan and Ezio Melotti prefer (actual, expected) > Raymond like (actual, expected) but would be happy with symmetrical diffs > Guido prefers the (actual, expected) ordering but prefers diffs to show the > other way round > R David Murray agreed with Guido > Terry Reedy liked the change > Glenn Linderman wants (actual, expected) and diffing to follow that > Ron Adam ditto > > Symmetrical diffs (element in first not in second, element in second not in > first) solves the problem without imposing an order on the arguments. > Actually unittest *has* used (first, second) to refer to the arguments to > asserts pretty much since its inception. Losing the (actual, expected) > terminology is a cost of this but unittest hasn't emphasised this > terminology in the past (as I thought it had). I actually agree with Guido that anything we do is going to be suboptimal in some way. Encouraging the actual/expected ordering and updating the diff output so "expected=old" strikes me as least bad, but using the neutral first/second terminology and doing the diffs as "first=old" wouldn't be terrible (although I'm personally -0 on it because it encourages putting the expected value first in order to get the diffs the right way around when an error occurs). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Wed Dec 22 02:15:34 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Dec 2010 12:15:34 +1100 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.2 beta 2 In-Reply-To: <4D10FDA0.7090808@python.org> References: <4D10FDA0.7090808@python.org> Message-ID: On Wed, Dec 22, 2010 at 6:18 AM, Georg Brandl wrote: > Since PEP 3003, the Moratorium on Language Changes, is in effect, there > are no changes in Python's syntax and built-in types in Python 3.2. Minor nit - we actually did tweak a few of the builtin types a bit (mostly the stuff to improve Sequence ABC conformance and to make range objects more list-like) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From guido at python.org Wed Dec 22 02:37:05 2010 From: guido at python.org (Guido van Rossum) Date: Tue, 21 Dec 2010 17:37:05 -0800 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> Message-ID: On Tue, Dec 21, 2010 at 5:12 PM, Nick Coghlan wrote: > I actually agree with Guido that anything we do is going to be > suboptimal in some way. Encouraging the actual/expected ordering and > updating the diff output so "expected=old" strikes me as least bad, > but using the neutral first/second terminology and doing the diffs as > "first=old" wouldn't be terrible (although I'm personally -0 on it > because it encourages putting the expected value first in order to get > the diffs the right way around when an error occurs). There are several problems with the actual/expected terminology. First of all, the arguments are primarily thought of as (and look) positional, not as having names (and there's nothing you can do about this in the docs -- people copy code without reading docs). Furthermore, Java's jUnit puts expected first (and makes this part of the culture/religion), so people coming from there will use that order and be freaked out if you were to swap them. And last, the order of diff arguments (old new) is also ingrained in the culture (which actually matches the expected/actual order in my mind). I think the least bad thing would be to drop any remnants of expected/actual terminology, keep the diffs in the first-second order, and let developers choose whether they put the expected value first or second. Then of course there will still be the examples in the doc (which some people *do* read and copy) -- I suppose we could alternate here to emphasize that we don't have a preferred order. -- --Guido van Rossum (python.org/~guido) From tjreedy at udel.edu Wed Dec 22 03:26:54 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 21 Dec 2010 21:26:54 -0500 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <4D109AEA.7010908@voidspace.org.uk> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> Message-ID: On 12/21/2010 7:17 AM, Michael Foord wrote: My first priority is that doc and code match. Close second is consistency (hence, ease of learning and use) between various AssertXs. > Symmetrical diffs (element in first not in second, element in second not > in first) solves the problem without imposing an order on the arguments. Where applicable, I prefer this as unambiguous output headings. -- Terry Jan Reedy From tjreedy at udel.edu Wed Dec 22 03:39:05 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 21 Dec 2010 21:39:05 -0500 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: References: <4D10590B.9040208@v.loewis.de> Message-ID: On 12/21/2010 7:01 AM, Nick Coghlan wrote: > On Tue, Dec 21, 2010 at 6:36 PM, "Martin v. L?wis" wrote: >>> Given the changing dynamics of the desktop launch menus to better >>> support direct access as an alternative to hierarchical navigation, >>> would it be reasonable to consider including the major version number >>> in the start menu shortcut names? >>> >>> (Question is mainly for Martin, obviously, but I'm also curious if >>> anyone else has encountered the same thing) >> >> I can't see anything wrong with that, but I'm terrible with wording. >> So somebody would have to spell out the exact wording that each of >> the start menu items should have. In doing so, also take into >> consideration that some people install 32-bit and 64-bit versions >> simultaneously. > > I put my initial proposal (reproduced below) at > http://bugs.python.org/issue10747. I'll let this thread run a bit and > see if there are any substantial improvements suggested, or valid > objections raised (I tried to pre-empt the length issue by pushing the > typically least important information to the end). I also considered > using the x86 and x64 abbreviations, but being explicit seemed > clearer. > > Shortcuts currently installed: > - Python (command line) > - Python Manuals > - Module Docs > - IDLE (Python GUI) > - Uninstall Python > > Initial proposal for 32 bit builds: > - Python 3.2 (command line - 32 bit) > - Python 3.2 Manuals > - Python 3.2 Docs Server (pydoc - 32 bit) Python 3.2 PyDoc Server (32 bit) is shorter. But since pydoc.py is has no 'bitness', PyDoc Server (Python 3.2 - 32 bit) is more in line with below > - IDLE (Python 3.2 GUI - 32 bit) I think of IDLE as an IDE, not a GUI, but IDLE IDE does not work too well. In any case, it also has no 'bitness'. IDLE GUI (Python 3.2 - 32 bit) -- Terry Jan Reedy From tjreedy at udel.edu Wed Dec 22 03:16:28 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 21 Dec 2010 21:16:28 -0500 Subject: [Python-Dev] [Python-checkins] r87433 - python/branches/py3k/Doc/reference/lexical_analysis.rst In-Reply-To: <20101222013736.8A9F7EE9A2@mail.python.org> References: <20101222013736.8A9F7EE9A2@mail.python.org> Message-ID: <4D115F7C.1020002@udel.edu> On 12/21/2010 8:37 PM, alexander.belopolsky wrote: > Author: alexander.belopolsky > Date: Wed Dec 22 02:37:36 2010 > New Revision: 87433 > > Log: > Both PEP 3131 and the current implementation use NFKC normalization > for identifiers. Fixed the documentation to agree. > > > Modified: > python/branches/py3k/Doc/reference/lexical_analysis.rst > > Modified: python/branches/py3k/Doc/reference/lexical_analysis.rst > ============================================================================== > --- python/branches/py3k/Doc/reference/lexical_analysis.rst (original) > +++ python/branches/py3k/Doc/reference/lexical_analysis.rst Wed Dec 22 02:37:36 2010 > @@ -309,8 +309,8 @@ > * *Nd* - decimal numbers > * *Pc* - connector punctuations > > -All identifiers are converted into the normal form NFC while parsing; comparison > -of identifiers is based on NFC. > +All identifiers are converted into the normal form NFKC while parsing; comparison > +of identifiers is based on NFKC. > > A non-normative HTML file listing all valid identifier characters for Unicode > 4.1 can be found at Has that file been updated for Unicode 6.0? From alexander.belopolsky at gmail.com Wed Dec 22 04:31:02 2010 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 21 Dec 2010 22:31:02 -0500 Subject: [Python-Dev] [Python-checkins] r87433 - python/branches/py3k/Doc/reference/lexical_analysis.rst In-Reply-To: <4D115F7C.1020002@udel.edu> References: <20101222013736.8A9F7EE9A2@mail.python.org> <4D115F7C.1020002@udel.edu> Message-ID: On Tue, Dec 21, 2010 at 9:16 PM, Terry Reedy wrote: .. >> ?A non-normative HTML file listing all valid identifier characters for >> Unicode >> ?4.1 can be found at http://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html > > Has that file been updated for Unicode 6.0? Apparently not. In Python 3.2: >>> '\N{KANNADA SIGN JIHVAMULIYA}'.isidentifier() True $ curl -s http://www.dcl.hpi.uni-potsdam.de/home/loewis/table-3131.html| grep JIHVAMULIYA $ From ncoghlan at gmail.com Wed Dec 22 07:21:05 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 Dec 2010 17:21:05 +1100 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: References: <4D10590B.9040208@v.loewis.de> Message-ID: On Wed, Dec 22, 2010 at 1:39 PM, Terry Reedy wrote: > ? ?Python 3.2 PyDoc Server (32 bit) > is shorter. But since pydoc.py is has no 'bitness', > > ? ?PyDoc Server (Python 3.2 - 32 bit) > is more in line with below > >> - IDLE (Python 3.2 GUI - 32 bit) > > I think of IDLE as an IDE, not a GUI, but IDLE IDE does not work too well. > In any case, it also has no 'bitness'. > > ? ?IDLE GUI (Python 3.2 - 32 bit) The IDLE and Pydoc links implicitly launch an interpreter though - that interpreter definitely has a defined pointer size. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From adal.chiriliuc at gmail.com Wed Dec 22 07:23:39 2010 From: adal.chiriliuc at gmail.com (Adal Chiriliuc) Date: Wed, 22 Dec 2010 08:23:39 +0200 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: References: <4D10590B.9040208@v.loewis.de> Message-ID: Microsoft recommendations: Avoid putting a version number in a program name unless that is how users normally refer to your program. Put only program shortcuts on the Start menu. Don't put shortcuts to the following items on the Start menu: - Program uninstallers. Users access uninstallers through the Programs control panel item. - Help files. Users access Help topics directly from your program. http://msdn.microsoft.com/en-us/library/aa511447.aspx I think it's fine for Python to use a version number and to put links to the manual in the Start Menu (since Python doesn't have a Help menu). But the uninstall shortcut should go. I also don't think that "command line" should be used in the title. And why is the .CHM file called "Manuals"? Why the plural? When seeing this title I thought that maybe it's a link to the Doc directory. Most of the .CHM files that I have in my Start Menu are called "abc Documentation", and a couple "abc Users Manual". The help file it's also titled "Python v2.7 documentation" when opened, so maybe it should be renamed to "Python 3.2 Documentation". > Initial proposal for 32 bit builds: > - Python 3.2 (command line - 32 bit) > - Python 3.2 Manuals > - Python 3.2 Docs Server (pydoc - 32 bit) > - IDLE (Python 3.2 GUI - 32 bit) > - Uninstall Python 3.2 (32 bit) > > Initial proposal for 64 bit builds: > - Python 3.2 (command line - 64 bit) > - Python 3.2 Manuals > - Python 3.2 Docs Server (pydoc - 64 bit) > - IDLE (Python 3.2 GUI - 64 bit) > - Uninstall Python 3.2 (64 bit) From eliben at gmail.com Wed Dec 22 07:40:36 2010 From: eliben at gmail.com (Eli Bendersky) Date: Wed, 22 Dec 2010 08:40:36 +0200 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: References: <4D10590B.9040208@v.loewis.de> Message-ID: > I put my initial proposal (reproduced below) at > http://bugs.python.org/issue10747. I'll let this thread run a bit and > see if there are any substantial improvements suggested, or valid > objections raised (I tried to pre-empt the length issue by pushing the > typically least important information to the end). I also considered > using the x86 and x64 abbreviations, but being explicit seemed > clearer. > > In general, definite +1 on the idea of having version numbers in there. A small comment: "command line" doesn't sound too helpful. Wouldn't "interactive shell" (as they do in ActiveState's Python) be better? Eli -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Dec 22 07:49:14 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 22 Dec 2010 01:49:14 -0500 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: References: <4D10590B.9040208@v.loewis.de> Message-ID: On 12/22/2010 1:23 AM, Adal Chiriliuc wrote: > Microsoft recommendations: Irrelevant. > > Avoid putting a version number in a program name unless that is how > users normally refer to your program. Version numbers are the point of this issue, because people *do* have multiple version installed. > Put only program shortcuts on the Start menu. Don't put shortcuts to > the following items on the Start menu: > - Program uninstallers. Users access uninstallers through the Programs > control panel item. Many programs put them there where they are EASY to access and invoke. Control panel and add/remove programs are absolute slugs when it comes to starting up. > - Help files. Users access Help topics directly from your program. Have you ever run the other items? IDLE has its own help. Control panel has no menu, and what help it does have is for its configuration dialog. \ > http://msdn.microsoft.com/en-us/library/aa511447.aspx > > I think it's fine for Python to use a version number and to put links > to the manual in the Start Menu (since Python doesn't have a Help > menu). But the uninstall shortcut should go. > > I also don't think that "command line" should be used in the title. > > And why is the .CHM file called "Manuals"? Why the plural? Because there are several. When seeing > this title I thought that maybe it's a link to the Doc directory. Most > of the .CHM files that I have in my Start Menu are called "abc > Documentation", and a couple "abc Users Manual". The help file it's > also titled "Python v2.7 documentation" when opened, so maybe it That is only true of the 2.7 doc. > should be renamed to "Python 3.2 Documentation". The 3.2 doc already is, at least on my system. >> Initial proposal for 32 bit builds: >> - Python 3.2 (command line - 32 bit) >> - Python 3.2 Manuals >> - Python 3.2 Docs Server (pydoc - 32 bit) >> - IDLE (Python 3.2 GUI - 32 bit) >> - Uninstall Python 3.2 (32 bit) >> >> Initial proposal for 64 bit builds: >> - Python 3.2 (command line - 64 bit) >> - Python 3.2 Manuals >> - Python 3.2 Docs Server (pydoc - 64 bit) >> - IDLE (Python 3.2 GUI - 64 bit) >> - Uninstall Python 3.2 (64 bit) -- Terry Jan Reedy From tjreedy at udel.edu Wed Dec 22 07:50:43 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 22 Dec 2010 01:50:43 -0500 Subject: [Python-Dev] Search-friendly shortcuts for Windows? In-Reply-To: References: <4D10590B.9040208@v.loewis.de> Message-ID: On 12/22/2010 1:21 AM, Nick Coghlan wrote: > On Wed, Dec 22, 2010 at 1:39 PM, Terry Reedy wrote: >> Python 3.2 PyDoc Server (32 bit) >> is shorter. But since pydoc.py is has no 'bitness', >> >> PyDoc Server (Python 3.2 - 32 bit) >> is more in line with below >> >>> - IDLE (Python 3.2 GUI - 32 bit) >> >> I think of IDLE as an IDE, not a GUI, but IDLE IDE does not work too well. >> In any case, it also has no 'bitness'. >> >> IDLE GUI (Python 3.2 - 32 bit) > > The IDLE and Pydoc links implicitly launch an interpreter though - > that interpreter definitely has a defined pointer size. That is exactly why I left the identity of the interpreter in parens. It is Python3.2 that is 32 or 64 bit, not IDLE or pydocs, so the modifier should be attached to Python 3.2 and not IDLE or pydocs. -- Terry Jan Reedy From phd at phd.pp.ru Wed Dec 22 09:07:41 2010 From: phd at phd.pp.ru (Oleg Broytman) Date: Wed, 22 Dec 2010 11:07:41 +0300 Subject: [Python-Dev] phd.pp.ru => phdru.name Message-ID: <20101222080741.GA4130@phd.pp.ru> Hello! I need to inform people I'm changing my online identity. Domain phd.pp.ru will die Dec 27 (I'll try to reregister it to extend its life for a few months). My new personal domain will be phdru.name, primary email will be "phd" (in case one makes a mistake and write "phdru" two times - the address is an alias for "phd"). The new domain is already delegated, email and site works. I'm working on changing all my email subscriptions. Oleg. -- Oleg Broytman http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From kristjan at ccpgames.com Wed Dec 22 14:58:17 2010 From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=) Date: Wed, 22 Dec 2010 21:58:17 +0800 Subject: [Python-Dev] issue 10296 Message-ID: <2E034B571A5CE44E949B9FCC3B6D24EE5E5AE047@exchcn.ccp.ad.local> I have this defect here http://bugs.python.org/issue10296 ctypes catches BreakPoint error on windows 32 that I posted a while back and had no response to. This is ctypes. Would anyone care to take a look? K -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Wed Dec 22 14:09:46 2010 From: eric at trueblade.com (Eric Smith) Date: Wed, 22 Dec 2010 08:09:46 -0500 Subject: [Python-Dev] [python-committers] [RELEASED] Python 3.2 beta 2 In-Reply-To: References: <4D10FDA0.7090808@python.org> Message-ID: <4D11F89A.3080103@trueblade.com> On 12/22/2010 8:46 AM, Georg Brandl wrote: > Am 22.12.2010 02:15, schrieb Nick Coghlan: >> On Wed, Dec 22, 2010 at 6:18 AM, Georg Brandl wrote: >>> Since PEP 3003, the Moratorium on Language Changes, is in effect, there >>> are no changes in Python's syntax and built-in types in Python 3.2. >> >> Minor nit - we actually did tweak a few of the builtin types a bit >> (mostly the stuff to improve Sequence ABC conformance and to make >> range objects more list-like) > > Indeed, I'll fix this wording for the next announcement. (And I will > mention SSL, sorry Antoine). If you're only going to mention some vague "some builtins had minor changes", then I'm fine with that. If you're going to enumerate all such changes, that will be a bigger job. There were 2 such changes I'm aware of: str.format_map (#6081) and the addition of alternate ("#") formatting to float, complex and decimal (#7094) __format__ methods. For this announcement I don't think it's necessary to list them all. From lvh at laurensvh.be Wed Dec 22 23:11:57 2010 From: lvh at laurensvh.be (Laurens Van Houtven) Date: Wed, 22 Dec 2010 23:11:57 +0100 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: On Sat, Dec 18, 2010 at 1:12 PM, Georg Brandl wrote: > Am 17.12.2010 17:52, schrieb Laurens Van Houtven: >> +1 for throwing it out of the PEP. Assignment is a thing, >> nonlocal/global is a thing, don't mix them up :) (That in addition to >> the grammar cleanliness argument Stephan already made) > > The trouble is what to make of > > nonlocal x = 3, y > > Is it two nonlocal declarations or one with a tuple assignment? > > Georg I'm not sure I understand. Isn't that another reason to throw it out? If you don't allow such assignments, there can't be any ambiguity, right? (Or am I missing something?) lvh From tjreedy at udel.edu Thu Dec 23 00:41:08 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 22 Dec 2010 18:41:08 -0500 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: On 12/22/2010 5:11 PM, Laurens Van Houtven wrote: > On Sat, Dec 18, 2010 at 1:12 PM, Georg Brandl wrote: >> Am 17.12.2010 17:52, schrieb Laurens Van Houtven: >>> +1 for throwing it out of the PEP. Assignment is a thing, >>> nonlocal/global is a thing, don't mix them up :) (That in addition to >>> the grammar cleanliness argument Stephan already made) >> >> The trouble is what to make of >> >> nonlocal x = 3, y >> >> Is it two nonlocal declarations or one with a tuple assignment? >> >> Georg > > I'm not sure I understand. Isn't that another reason to throw it out? I am sure he meant it to be, and I agree. We lived without global x = 3, y all these years. > If you don't allow such assignments, there can't be any ambiguity, > right? (Or am I missing something?) -- Terry Jan Reedy From victor.stinner at haypocalc.com Thu Dec 23 03:08:13 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 23 Dec 2010 03:08:13 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default Message-ID: <1293070093.12866.14.camel@marge> Hi, Thanks to all your useful remarks, I improved my patch twice (version 10 and 11). I think that the last version (11) addresses all reported issues. The most notable change is that the fault handler is now disabled by default. It fixes also the function getting the thread state, it now works in all cases (especially if the thread causing the fault doesn't hold the GIL). Summary of the patch (#8863): - Add an optional fault handler for SIGSEGV, SIGFPE, SIGILL and SIGBUS signals displaying the Python backtrace - Setting PYTHONFAULTHANDLER environment variable, or using "-X faulthandler" command line option, enables it - If it's enabled, display also the Python backtrace on a fatal error Use case: when a program crashs, the user reruns its application with the fault handler enabled and tries to reproduce the crash. He/She can send the Python backtrace to the developer, or use it directly (if he/she understands it). I think that this patch helps to isolate crashs in Python or libraries, especially if the user doesn't have all tools needed to get a Python backtrace (on a crash) and the developer is unable to reproduce the bug (because it doesn't have the same OS or libraries). For example, gdb 7 and python-gdb.py are rare on Windows :-) (even on Linux!) So, do you agree with the fault handler? Does someone want to give a last review because I commit it? http://bugs.python.org/issue8863 Victor From vstinner at edenwall.com Thu Dec 23 03:26:35 2010 From: vstinner at edenwall.com (Victor Stinner) Date: Thu, 23 Dec 2010 03:26:35 +0100 Subject: [Python-Dev] Issue #8863 adds a new?PYTHONNOFAULTHANDLER?environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> <20101220165632.GA19009@yoda.bytereef.org> <201012201851.09810.vstinner@edenwall.com> Message-ID: <1293071195.12866.32.camel@marge> Le lundi 20 d?cembre 2010 ? 13:09 -0500, Alexander Belopolsky a ?crit : > What if the program crashes (again) in the signal handler before > having a chance to restore registers? If the fault handler crashs, the registers are not restored. > Also, can you point to documentation that describes register restoring behavior? I found an article telling that the kernel saves integer and float registers when calling a signal handler, but I am unable to find it again, sorry. And I cannot find an article telling the opposite. Anyway, it would just be horrible if a signal handler was responsible to save/restore integer registers. If the kernel doesn't do that for us, I suppose that the compiler or something else does it for us. Eg. signal_default_int_handler(), the Python handler of SIGINT signal, doesn't save/restore registers, even if it erases many of them. > Because I had actual experience with that using R. Your code may be > better, but R brings up a dialog on segfault asking ... You cannot compare a simple handler using simple signal-safe functions like write(), and a complex handler calling graphical functions. It is not safe to call graphical functions in a signal handler. > (I actually use RPy which loads R into Python as a shared library, > so the two handlers will conflict anyways.) It depends if R handler calls the previous handler. My fault handler is "chained" with the previous handler (it calls indirectly the previous handler). If R fault handler does the same, all handlers will be executed correctly (I don't think that the execution order matter). Victor From martin at v.loewis.de Thu Dec 23 03:37:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Dec 2010 03:37:21 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <1293070093.12866.14.camel@marge> References: <1293070093.12866.14.camel@marge> Message-ID: <4D12B5E1.7060608@v.loewis.de> > So, do you agree with the fault handler? Does someone want to give a > last review because I commit it? It's a new feature, so regardless of whether it's correct or not (which I haven't reviewed yet), I don't think it should go in before 3.2 is released. Regards, Martin From simon at brunningonline.net Thu Dec 23 10:40:12 2010 From: simon at brunningonline.net (Simon Brunning) Date: Thu, 23 Dec 2010 09:40:12 +0000 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> Message-ID: On 22 December 2010 01:37, Guido van Rossum wrote: > Furthermore, Java's jUnit puts expected first (and makes this part of > the culture/religion), so people coming from there will use that order > and be freaked out if you were to swap them. And last, the order of > diff arguments (old new) is also ingrained in the culture (which > actually matches the expected/actual order in my mind). For what it's worth, none of the (numerous) Java projects that I've worked on over the last couple of years have used bare JUnit. Assertions are often done using the Hamcrest library - . The actual value comes first, and this combined with the use of matchers makes the failure messages very descriptive. I'd hate to go back to one of those projects where the tests tell you that two values don't match, but doesn?t tell me which is the expected value. There is a Python port - . I must get around to trying that. -- Cheers, Simon B. From orsenthil at gmail.com Thu Dec 23 11:04:43 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Thu, 23 Dec 2010 18:04:43 +0800 Subject: [Python-Dev] [Python-checkins] r87270 - python/branches/py3k/Doc/library/urllib.request.rst In-Reply-To: <20101215190726.2FFE2EE98A@mail.python.org> References: <20101215190726.2FFE2EE98A@mail.python.org> Message-ID: <20101223100443.GC1690@rubuntu> On Wed, Dec 15, 2010 at 08:07:26PM +0100, antoine.pitrou wrote: > Log: > Move the urllib-inherited API to a distinguished section Distinguished: Legacy. :) -- Senthil From victor.stinner at haypocalc.com Thu Dec 23 18:20:42 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Thu, 23 Dec 2010 18:20:42 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <4D12B5E1.7060608@v.loewis.de> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> Message-ID: <1293124842.26635.3.camel@marge> Le jeudi 23 d?cembre 2010 ? 03:37 +0100, "Martin v. L?wis" a ?crit : > > So, do you agree with the fault handler? Does someone want to give a > > last review because I commit it? > > It's a new feature, so regardless of whether it's correct or not > (which I haven't reviewed yet), I don't think it should go in before > 3.2 is released. I tested my patch on Windows (XP), Ubuntu (10.4) and FreeBSD (8) and it works correctly: all tests pass and the system fault handler (Windows popup, Apport and core dump) is also called. The fault handler is disabled by default and it is clearly separated (eg. it doesn't touch the API of a module). Can't you make an exception for this new feature? The first answer to my previous email thread was Georg, answer starting with "I very much like having a traceback on (some) segmentation faults" :-) Victor From martin at v.loewis.de Thu Dec 23 19:23:17 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 23 Dec 2010 19:23:17 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <1293124842.26635.3.camel@marge> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> Message-ID: <4D139395.3000709@v.loewis.de> > The fault handler is disabled by default and it is clearly separated > (eg. it doesn't touch the API of a module). Can't you make an exception > for this new feature? Ultimately, it's for the release manager to decide, so I don't need to make an exception. However, I think that special cases aren't special enough to break the rules. I still wish that the beta releases had been deferred until after the Mercurial switchover, but alas, the release manager has decided otherwise - explicitly pointing out that the rationale for releasing beta 1 was to stop accepting new features. The motivation for freezing features before feature releases is not so much that existing applications may be affected by API changes (3.2 will change APIs, anyway, and users fully expect to need some porting), but that the features that are about to be released need some time for testing. This specific feature has seen very little testing. Giving it a full release cycle (i.e. until 3.3) would somewhat reduce the need for a more careful code review. Regards, Martin From bobbyi at gmail.com Thu Dec 23 19:29:17 2010 From: bobbyi at gmail.com (Bobby Impollonia) Date: Thu, 23 Dec 2010 13:29:17 -0500 Subject: [Python-Dev] Issue #8863 adds a new?PYTHONNOFAULTHANDLER?environment variable In-Reply-To: <1293071195.12866.32.camel@marge> References: <4D0C0688.1080108@haypocalc.com> <20101220165632.GA19009@yoda.bytereef.org> <201012201851.09810.vstinner@edenwall.com> <1293071195.12866.32.camel@marge> Message-ID: On Wed, Dec 22, 2010 at 9:26 PM, Victor Stinner wrote: > If the kernel doesn't do that for us, I > suppose that the compiler or something else does it for us. GCC does this for you if you declare your function with __attribute__(signal). In general, the compiler doesn't know that a function will be used as a signal and no portable way to indicate this. > Anyway, it would just be horrible if a signal handler was responsible to > save/restore integer registers. Horrible or not, the existence of __attribute__(signal) seems to indicate that this is the case on some platform, or at least was historically. I don't know whether it's the case now on any platform that Python supports. From martin at v.loewis.de Thu Dec 23 19:57:21 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 23 Dec 2010 19:57:21 +0100 Subject: [Python-Dev] Issue #8863 adds a new?PYTHONNOFAULTHANDLER?environment variable In-Reply-To: References: <4D0C0688.1080108@haypocalc.com> <20101220165632.GA19009@yoda.bytereef.org> <201012201851.09810.vstinner@edenwall.com> <1293071195.12866.32.camel@marge> Message-ID: <4D139B91.3000501@v.loewis.de> > Horrible or not, the existence of __attribute__(signal) seems to > indicate that this is the case on some platform, or at least was > historically. The signal attribute has an effect only on ATMEL AVR processors, according to the documentation (and according to my source grep). On other processors, gcc typically implements __attribute__((interrupt)), meaning that the function is an interrupt handler - something that you cannot express in regular C. This may involve re-enabling interrupts on entry, but may also involve other code generation aspects (such as using iret instead of ret on x86). For AVR, there was apparently some compatibility issue, introducing the need for two different ways to declare an ISR: __attribute__((interrupt)) causes interrupts to be re-enabled on entry (using the "sei" instruction); __attribute__((signal)) keeps them disabled. So this has nothing to do with Unix signals. Regards, Martin From solipsis at pitrou.net Thu Dec 23 20:55:28 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Dec 2010 20:55:28 +0100 Subject: [Python-Dev] r87445 - python/branches/py3k/Lib/numbers.py References: <20101223184133.9DED1EE995@mail.python.org> Message-ID: <20101223205528.1c365abd@pitrou.net> On Thu, 23 Dec 2010 19:41:33 +0100 (CET) eric.araujo wrote: > > def __index__(self): > - """index(self)""" > + """someobject[self]""" This is misleading as to what the method actually does, as you can read in the implementation: > return int(self) From g.brandl at gmx.net Thu Dec 23 21:51:54 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Dec 2010 21:51:54 +0100 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: Am 22.12.2010 23:11, schrieb Laurens Van Houtven: > On Sat, Dec 18, 2010 at 1:12 PM, Georg Brandl wrote: >> Am 17.12.2010 17:52, schrieb Laurens Van Houtven: >>> +1 for throwing it out of the PEP. Assignment is a thing, >>> nonlocal/global is a thing, don't mix them up :) (That in addition to >>> the grammar cleanliness argument Stephan already made) >> >> The trouble is what to make of >> >> nonlocal x = 3, y >> >> Is it two nonlocal declarations or one with a tuple assignment? >> >> Georg > > I'm not sure I understand. Isn't that another reason to throw it out? > If you don't allow such assignments, there can't be any ambiguity, > right? (Or am I missing something?) Yes and no -- there may not be an ambiguity to the parser, but still to the human. Except if you disallow the syntax in any case, requiring people to write nonlocal x = (3, y) which is then again inconsistent with ordinary assignment statements. Georg From g.brandl at gmx.net Thu Dec 23 21:59:40 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Dec 2010 21:59:40 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <4D139395.3000709@v.loewis.de> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> Message-ID: Am 23.12.2010 19:23, schrieb "Martin v. L?wis": >> The fault handler is disabled by default and it is clearly separated >> (eg. it doesn't touch the API of a module). Can't you make an exception >> for this new feature? > > Ultimately, it's for the release manager to decide, so I don't need to > make an exception. However, I think that special cases aren't special > enough to break the rules. I still wish that the beta releases had been > deferred until after the Mercurial switchover, but alas, the release > manager has decided otherwise - explicitly pointing out that the > rationale for releasing beta 1 was to stop accepting new features. The main rationale was that the Mercurial switchover is not a fixed target and depends very much on volunteer effort, so in effect this would have been akin to deferring 3.2 for an indefinite period. > The motivation for freezing features before feature releases is not > so much that existing applications may be affected by API changes > (3.2 will change APIs, anyway, and users fully expect to need some > porting), but that the features that are about to be released need > some time for testing. This specific feature has seen very little > testing. Giving it a full release cycle (i.e. until 3.3) would > somewhat reduce the need for a more careful code review. Exactly. I did say I like the feature, but that was a) before beta 2 was released, now the next release is a release candidate, and b) this thread showed that it is not at all obvious how the feature should look like. The fact that it isn't enabled by default also makes it seem less attractive to me, but I understand the reasons why it shouldn't be on by default. Therefore, I'm not willing to make an exception here. Georg From lvh at laurensvh.be Thu Dec 23 22:03:49 2010 From: lvh at laurensvh.be (Laurens Van Houtven) Date: Thu, 23 Dec 2010 22:03:49 +0100 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: On Thu, Dec 23, 2010 at 9:51 PM, Georg Brandl wrote: > Yes and no -- there may not be an ambiguity to the parser, but still to > the human. ?Except if you disallow the syntax in any case, requiring > people to write > > nonlocal x = (3, y) > > which is then again inconsistent with ordinary assignment statements. > > Georg Right -- but (and hence the confusion) I was arguing for not mixing global/nonlocal with assignment at all, and instead having nonlocal and global only take one or more names. That would (obviously) remove any such ambiguity ;-) cheers lvh From merwok at netwok.org Thu Dec 23 22:09:15 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 23 Dec 2010 22:09:15 +0100 Subject: [Python-Dev] r87445 - python/branches/py3k/Lib/numbers.py In-Reply-To: <20101223205528.1c365abd@pitrou.net> References: <20101223184133.9DED1EE995@mail.python.org> <20101223205528.1c365abd@pitrou.net> Message-ID: <4D13BA7B.7010402@netwok.org> Le 23/12/2010 20:55, Antoine Pitrou a ?crit : >> def __index__(self): >> - """index(self)""" >> + """someobject[self]""" > > This is misleading as to what the method actually does, Really? Unless I misunderstood the docs, __index__ is used when the object is used as an index (or with bin or oct, but I didn?t want to complicate the docstring, just fix it). On IRC, R. David Murray said that I could just have deleted the docstring. I agree with that: I think magic methods never need a docstring, since they?re documented once and for all in the language reference. (__init__ is not an exception: its parameters can be documented in the class docstring.) > as you can read in the implementation: >> return int(self) The fact that __index__ is implemented thanks to int/__int__ here is a detail IMO. Regards From g.brandl at gmx.net Thu Dec 23 22:16:22 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 23 Dec 2010 22:16:22 +0100 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: Am 23.12.2010 22:03, schrieb Laurens Van Houtven: > On Thu, Dec 23, 2010 at 9:51 PM, Georg Brandl wrote: >> Yes and no -- there may not be an ambiguity to the parser, but still to >> the human. Except if you disallow the syntax in any case, requiring >> people to write >> >> nonlocal x = (3, y) >> >> which is then again inconsistent with ordinary assignment statements. >> >> Georg > > Right -- but (and hence the confusion) I was arguing for not mixing > global/nonlocal with assignment at all, and instead having nonlocal > and global only take one or more names. That would (obviously) remove > any such ambiguity ;-) Oh yes, I see -- not sure why I worded it like I did. I just wanted to reiterate the most obvious problematic point to people who hadn't followed the earlier discussions about it. Sorry. Georg From solipsis at pitrou.net Thu Dec 23 22:28:44 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 Dec 2010 22:28:44 +0100 Subject: [Python-Dev] r87445 - python/branches/py3k/Lib/numbers.py In-Reply-To: <4D13BA7B.7010402@netwok.org> References: <20101223184133.9DED1EE995@mail.python.org> <20101223205528.1c365abd@pitrou.net> <4D13BA7B.7010402@netwok.org> Message-ID: <20101223222844.654d9dfe@pitrou.net> On Thu, 23 Dec 2010 22:09:15 +0100 ?ric Araujo wrote: > Le 23/12/2010 20:55, Antoine Pitrou a ?crit : > >> def __index__(self): > >> - """index(self)""" > >> + """someobject[self]""" > > > > This is misleading as to what the method actually does, > Really? Unless I misunderstood the docs, __index__ is used when the > object is used as an index (or with bin or oct, but I didn?t want to > complicate the docstring, just fix it). Well, it's only used for converting to an integer (also, it's used in more cases than just indexing), so "someobject[self]" is not the actual operation. Regards Antoine. From martin at v.loewis.de Thu Dec 23 23:09:28 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 23 Dec 2010 23:09:28 +0100 Subject: [Python-Dev] r87445 - python/branches/py3k/Lib/numbers.py In-Reply-To: <4D13BA7B.7010402@netwok.org> References: <20101223184133.9DED1EE995@mail.python.org> <20101223205528.1c365abd@pitrou.net> <4D13BA7B.7010402@netwok.org> Message-ID: <4D13C898.6070509@v.loewis.de> Am 23.12.2010 22:09, schrieb ?ric Araujo: > Le 23/12/2010 20:55, Antoine Pitrou a ?crit : >>> def __index__(self): >>> - """index(self)""" >>> + """someobject[self]""" >> >> This is misleading as to what the method actually does, > Really? Unless I misunderstood the docs, __index__ is used when the > object is used as an index (or with bin or oct, but I didn?t want to > complicate the docstring, just fix it). In case Antoine's objection isn't clear yet: __index__ does *not* perform the actual indexing, as opposed to what the docstring suggests. Regards, Martin From gzlist at googlemail.com Thu Dec 23 23:58:23 2010 From: gzlist at googlemail.com (Martin (gzlist)) Date: Thu, 23 Dec 2010 22:58:23 +0000 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <1293124842.26635.3.camel@marge> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> Message-ID: On 23/12/2010, Victor Stinner wrote: > > I tested my patch on Windows (XP), Ubuntu (10.4) and FreeBSD (8) and it > works correctly: all tests pass and the system fault handler (Windows > popup, Apport and core dump) is also called. Doesn't build for me without #ifdef HAVE_UNISTD_H in Python/fault.c and you missed my comment about raise vs. kill in the other thread which lets the SIGILL test run as well. With those changes, I get: test_disabled (test.test_fault.FaultHandlerTests) ... ok test_fatal_error (test.test_fault.FaultHandlerTests) ... FAIL test_gil_released (test.test_fault.FaultHandlerTests) ... ok test_sigbus (test.test_fault.FaultHandlerTests) ... skipped 'need _testcapi.sigbus()' test_sigfpe (test.test_fault.FaultHandlerTests) ... skipped 'SIGFPE cannot be caught on Windows' test_sigill (test.test_fault.FaultHandlerTests) ... ok test_sigsegv (test.test_fault.FaultHandlerTests) ... ok test_xoption (test.test_fault.FaultHandlerTests) ... ok So, this does basically work as there is a SIGSEGV compatibility hack in VS8 and later (I'd neglected to compare the language across the different MSDN pages). The failure is due to the test not expecting an extra note the assert gives at the end. See attached interdiff for suggested changes. Martin -------------- next part -------------- A non-text attachment was scrubbed... Name: issue8863_incr.diff Type: application/octet-stream Size: 2346 bytes Desc: not available URL: From martin at v.loewis.de Fri Dec 24 00:39:02 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 24 Dec 2010 00:39:02 +0100 Subject: [Python-Dev] [Python-checkins] r87458 - python/branches/py3k/Lib/gettext.py In-Reply-To: <20101223224938.9A243EE997@mail.python.org> References: <20101223224938.9A243EE997@mail.python.org> Message-ID: <4D13DD96.2070400@v.loewis.de> > # "a?b:c" to "test(a,b,c)". > expr = re.compile(r'(.*?)\?(.*?):(.*)') > def repl(x): > - return "test(%s, %s, %s)" % (x.group(1), x.group(2), > - expr.sub(repl, x.group(3))) > + return "(%s if %s else %s)" % (x.group(2), x.group(1), > + expr.sub(repl, x.group(3))) Please update the comment as well. Regards, Martin From benjamin at python.org Fri Dec 24 00:45:41 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 23 Dec 2010 17:45:41 -0600 Subject: [Python-Dev] [Python-checkins] r87458 - python/branches/py3k/Lib/gettext.py In-Reply-To: <4D13DD96.2070400@v.loewis.de> References: <20101223224938.9A243EE997@mail.python.org> <4D13DD96.2070400@v.loewis.de> Message-ID: 2010/12/23 "Martin v. L?wis" : >> ? ? ?# "a?b:c" to "test(a,b,c)". >> ? ? ?expr = re.compile(r'(.*?)\?(.*?):(.*)') >> ? ? ?def repl(x): >> - ? ? ? ?return "test(%s, %s, %s)" % (x.group(1), x.group(2), >> - ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? expr.sub(repl, x.group(3))) >> + ? ? ? ?return "(%s if %s else %s)" % (x.group(2), x.group(1), >> + ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? expr.sub(repl, x.group(3))) > > Please update the comment as well. Done. -- Regards, Benjamin From ncoghlan at gmail.com Fri Dec 24 02:08:08 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Dec 2010 11:08:08 +1000 Subject: [Python-Dev] [Python-checkins] r87445 - python/branches/py3k/Lib/numbers.py In-Reply-To: <20101223184133.9DED1EE995@mail.python.org> References: <20101223184133.9DED1EE995@mail.python.org> Message-ID: On Fri, Dec 24, 2010 at 4:41 AM, eric.araujo wrote: > Author: eric.araujo > Date: Thu Dec 23 19:41:33 2010 > New Revision: 87445 > > Log: > Fix small inaccuracy: there is no index function Yes, there is, it just isn't a builtin - it lives in the operator module. > ? ? def __index__(self): > - ? ? ? ?"""index(self)""" > + ? ? ? ?"""someobject[self]""" > ? ? ? ? return int(self) Changing the docstring to say "operator.index(self)" would be the clearest solution here. (Choosing to accept arbitrary index objects as integer equivalents is up to the object being indexed, just like interpreting slices is - a dict, for example, will never invoke __index__ methods). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From victor.stinner at haypocalc.com Fri Dec 24 03:21:19 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 24 Dec 2010 03:21:19 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> Message-ID: <1293157279.16757.3.camel@marge> Le jeudi 23 d?cembre 2010 ? 21:59 +0100, Georg Brandl a ?crit : > this thread showed that it is not at all obvious how the feature should look like Ok, I understand. I closed #8863 (rejected) and I created a third party module on the Python cheese shop: http://pypi.python.org/pypi/faulthandler/ The module works on Linux, FreeBSD and Windows with Python 2.6 through 3.2. A third party module can evolve faster outside Python. Victor From victor.stinner at haypocalc.com Fri Dec 24 03:25:02 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Fri, 24 Dec 2010 03:25:02 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> Message-ID: <1293157502.16757.7.camel@marge> Le jeudi 23 d?cembre 2010 ? 22:58 +0000, Martin (gzlist) a ?crit : > On 23/12/2010, Victor Stinner wrote: > > > > I tested my patch on Windows (XP), Ubuntu (10.4) and FreeBSD (8) and it > > works correctly: all tests pass and the system fault handler (Windows > > popup, Apport and core dump) is also called. > > Doesn't build for me without #ifdef HAVE_UNISTD_H in Python/fault.c Yes, I wrote a comment about that in the issue. But then I realized that this header is not needed at all. > you missed my comment about raise vs. kill in the other thread > which lets the SIGILL test run as well Oh, I didn't noticed that this change enables sigill() on Windows. Nice :-) I created a third party module from my patch: http://pypi.python.org/pypi/faulthandler/ I commited your patch, thanks. I added you to the authors as a contributor. Victor From g.brandl at gmx.net Fri Dec 24 07:51:07 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 24 Dec 2010 07:51:07 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <1293157279.16757.3.camel@marge> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> <1293157279.16757.3.camel@marge> Message-ID: Am 24.12.2010 03:21, schrieb Victor Stinner: > Le jeudi 23 d?cembre 2010 ? 21:59 +0100, Georg Brandl a ?crit : >> this thread showed that it is not at all obvious how the feature should look like > > Ok, I understand. I closed #8863 (rejected) and I created a third party > module on the Python cheese shop: > http://pypi.python.org/pypi/faulthandler/ > > The module works on Linux, FreeBSD and Windows with Python 2.6 > through 3.2. > > A third party module can evolve faster outside Python. Very good. I take it you'll make a new issue then for inclusion in 3.3? Georg From hrvoje.niksic at avl.com Fri Dec 24 09:45:56 2010 From: hrvoje.niksic at avl.com (Hrvoje Niksic) Date: Fri, 24 Dec 2010 09:45:56 +0100 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: Message-ID: <4D145DC4.9000107@avl.com> On 12/23/2010 10:03 PM, Laurens Van Houtven wrote: > On Thu, Dec 23, 2010 at 9:51 PM, Georg Brandl wrote: >> Yes and no -- there may not be an ambiguity to the parser, but still to >> the human. Except if you disallow the syntax in any case, requiring >> people to write >> >> nonlocal x = (3, y) >> >> which is then again inconsistent with ordinary assignment statements. > > Right -- but (and hence the confusion) I was arguing for not mixing > global/nonlocal with assignment at all, and instead having nonlocal > and global only take one or more names. That would (obviously) remove > any such ambiguity ;-) I would like to offer the opposing viewpoint: nonlocal x = value is a useful shortcut because nonlocal is used in closure callbacks where brevity matters. The reason nonlocal is introduced is to change the variable, so it makes sense that the two can be done in the same line of code. As for global x = value being disallowed, I have been annoyed at times with that, so that sounds like a good argument to change both. Requiring the parentheses for tuple creation sounds like a good compromise for resolving the ambiguity, consistent with similar limitations of the generator expression syntax. From fuzzyman at voidspace.org.uk Fri Dec 24 13:59:41 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 24 Dec 2010 12:59:41 +0000 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <1293157279.16757.3.camel@marge> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> <1293157279.16757.3.camel@marge> Message-ID: <4D14993D.2040803@voidspace.org.uk> On 24/12/2010 02:21, Victor Stinner wrote: > Le jeudi 23 d?cembre 2010 ? 21:59 +0100, Georg Brandl a ?crit : >> this thread showed that it is not at all obvious how the feature should look like > Ok, I understand. I closed #8863 (rejected) and I created a third party > module on the Python cheese shop: > http://pypi.python.org/pypi/faulthandler/ > > The module works on Linux, FreeBSD and Windows with Python 2.6 > through 3.2. > > A third party module can evolve faster outside Python. I hope you will include it in 3.3 though; it is great functionality. I would really like to see it enabled by default as well. It seemed from the discussion that the biggest barrier to enabling it by default was possible difficulties when embedding Python (multiple interpreters, potential conflicts with application signal handling). A public C-API to disable the functionality per interpreter would be one option for this. Another possibility would be providing a C-API to enable it and have the Python interpreter application call this, so that the functionality remains off by default for embedded interpreters but on for normal uses. All the best, Michael > Victor > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From fuzzyman at voidspace.org.uk Fri Dec 24 17:09:42 2010 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 24 Dec 2010 16:09:42 +0000 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> Message-ID: <4D14C5C6.9070703@voidspace.org.uk> On 22/12/2010 02:26, Terry Reedy wrote: > On 12/21/2010 7:17 AM, Michael Foord wrote: > > My first priority is that doc and code match. > Close second is consistency (hence, ease of learning and use) between > various AssertXs. > >> Symmetrical diffs (element in first not in second, element in second not >> in first) solves the problem without imposing an order on the arguments. > > Where applicable, I prefer this as unambiguous output headings. > Could you explain what you mean? All the best, Michael Foord -- http://www.voidspace.org.uk/ May you do good and not evil May you find forgiveness for yourself and forgive others May you share freely, never taking more than you give. -- the sqlite blessing http://www.sqlite.org/different.html From status at bugs.python.org Fri Dec 24 18:07:05 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 24 Dec 2010 18:07:05 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20101224170705.3F7651CCBB@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-12-17 - 2010-12-24) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2542 ( +5) closed 20009 (+38) total 22551 (+43) Open issues with patches: 1069 Issues opened (35) ================== #3243: Support iterable bodies in httplib http://bugs.python.org/issue3243 reopened by georg.brandl #9090: Error code 10035 calling socket.recv() on a socket with a time http://bugs.python.org/issue9090 reopened by ehohenstein #10731: UnicodeDecodeError in OS X tkinter when binding to timestamps prior to 1970 on Windo http://bugs.python.org/issue2494 closed by r.david.murray #3500: unbound methods of different classes compare equal http://bugs.python.org/issue3500 closed by r.david.murray #4078: asyncore fixes are not backwards compatible http://bugs.python.org/issue4078 closed by r.david.murray #4322: function with modified __name__ uses original name when there' http://bugs.python.org/issue4322 closed by r.david.murray #4369: Error building to a nonstandard prefix (with patch) http://bugs.python.org/issue4369 closed by r.david.murray #4496: misleading comment in urllib2 http://bugs.python.org/issue4496 closed by r.david.murray #4871: zipfile can't decrypt http://bugs.python.org/issue4871 closed by r.david.murray #5587: vars() no longer has a useful __repr__ http://bugs.python.org/issue5587 closed by ezio.melotti #8073: Test fail for sha512 - http://bugs.python.org/issue8073 closed by gregory.p.smith #8863: Display Python backtrace on SIGSEGV, SIGFPE and fatal error http://bugs.python.org/issue8863 closed by haypo #9063: TZ examples in datetime.rst are incorrect http://bugs.python.org/issue9063 closed by belopolsky #9234: argparse: aliases for positional arguments (subparsers) http://bugs.python.org/issue9234 closed by bethard #9286: email.utils.parseaddr returns garbage for invalid input http://bugs.python.org/issue9286 closed by r.david.murray #9857: SkipTest in tearDown is reported an as an error http://bugs.python.org/issue9857 closed by michael.foord #9907: interactive mode TAB does not insert on OS X built with editli http://bugs.python.org/issue9907 closed by r.david.murray #10404: IDLE on OS X popup menus do not work: cannot set/clear breakpo http://bugs.python.org/issue10404 closed by r.david.murray #10587: Document the meaning of str methods http://bugs.python.org/issue10587 closed by belopolsky #10611: sys.exit() in a test causes a test run to die http://bugs.python.org/issue10611 closed by michael.foord #10622: WebKit browsers show superfluous scrollbars in html docs http://bugs.python.org/issue10622 closed by r.david.murray #10671: urllib2 redirect to another host doesn't work http://bugs.python.org/issue10671 closed by orsenthil #10711: Rip out HTTP 0.9 client support http://bugs.python.org/issue10711 closed by pitrou #10715: Command name missing from exception in subprocess.Popen http://bugs.python.org/issue10715 closed by eric.smith #10723: Undocumented built-in exceptions http://bugs.python.org/issue10723 closed by georg.brandl #10725: Better cache instrumentation http://bugs.python.org/issue10725 closed by rhettinger #10727: intermittent test_cmd_line failure http://bugs.python.org/issue10727 closed by ned.deily #10728: argparse.ArgumentParser.print_help uses sys.stdout http://bugs.python.org/issue10728 closed by r.david.murray #10729: fwbackups python lib error http://bugs.python.org/issue10729 closed by eric.smith #10730: add .svgz to mimetypes.suffix_map and .svg to types_map http://bugs.python.org/issue10730 closed by terry.reedy #10732: Wrong destructor function type in Python/getargs.c http://bugs.python.org/issue10732 closed by pitrou #10743: 3.2's sysconfig doesn't work with virtualenv http://bugs.python.org/issue10743 closed by srid #10749: lots of warnings when generating logging docs http://bugs.python.org/issue10749 closed by georg.brandl #10750: "raw" attribute of buffered IO objects is assignable http://bugs.python.org/issue10750 closed by pitrou #10754: os.path.isfile doesn't work with some greek characters http://bugs.python.org/issue10754 closed by r.david.murray #10748: zipfile does not write empty ZIP structure if close() called a http://bugs.python.org/issue10748 closed by georg.brandl #1155362: Bugs in parsedate_tz http://bugs.python.org/issue1155362 closed by r.david.murray From tjreedy at udel.edu Fri Dec 24 19:56:06 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 24 Dec 2010 13:56:06 -0500 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <4D14C5C6.9070703@voidspace.org.uk> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> <4D14C5C6.9070703@voidspace.org.uk> Message-ID: On 12/24/2010 11:09 AM, Michael Foord wrote: > On 22/12/2010 02:26, Terry Reedy wrote: >> On 12/21/2010 7:17 AM, Michael Foord wrote: >> >> My first priority is that doc and code match. >> Close second is consistency (hence, ease of learning and use) between >> various AssertXs. >> >>> Symmetrical diffs (element in first not in second, element in second not >>> in first) solves the problem without imposing an order on the arguments. >> >> Where applicable, I prefer this as unambiguous output headings. >> > Could you explain what you mean? I was referring back to an output example symmetric diff that was clipped somewhere along the way: In x not in y: ... In y not in x: ... rather than just using -,+ prefixes which are not necessarily self-explanatory. 'Not applicable' would refer to output from difflib which necessarily is ordered. -- Terry Jan Reedy From raymond.hettinger at gmail.com Fri Dec 24 21:03:38 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 24 Dec 2010 12:03:38 -0800 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> <4D14C5C6.9070703@voidspace.org.uk> Message-ID: <763DCF04-033D-455B-A7BF-9D5813BBDFD4@gmail.com> On Dec 24, 2010, at 10:56 AM, Terry Reedy wrote: > On 12/24/2010 11:09 AM, Michael Foord wrote: >> On 22/12/2010 02:26, Terry Reedy wrote: >>> On 12/21/2010 7:17 AM, Michael Foord wrote: >>> >>> My first priority is that doc and code match. >>> Close second is consistency (hence, ease of learning and use) between >>> various AssertXs. >>> >>>> Symmetrical diffs (element in first not in second, element in second not >>>> in first) solves the problem without imposing an order on the arguments. >>> >>> Where applicable, I prefer this as unambiguous output headings. >>> >> Could you explain what you mean? > > I was referring back to an output example symmetric diff that was clipped somewhere along the way: > > In x not in y: > ... > In y not in x: > ... > > rather than just using -,+ prefixes which are not necessarily self-explanatory. 'Not applicable' would refer to output from difflib which necessarily is ordered. FWIW, I think + and - prefixes are much better for diffs that some made-up verbiage. People are used to seeing diffs with + and -. Anything else will be so contrived that it's net effect will be to make the output confusing and hard to interpret. If you want, add two lines of explanation before the diff: + means "in x, not in y" - means "in y, not it x" The notion of "making symmetric" can easily get carried too far, which a corresponding loss of useability. You get 95% of the benefit from two small changes: * Change the parameter names from "actual" and "expected" to "first" and "second" * Change the words "unexpected" and "missing" to "in first, not in second" and "in second, not in first". We have a strong history in using +/- and shouldn't throw away its brevity and clarity. Raymond From stephen at xemacs.org Sat Dec 25 03:29:51 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Sat, 25 Dec 2010 11:29:51 +0900 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <4D14993D.2040803@voidspace.org.uk> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> <1293157279.16757.3.camel@marge> <4D14993D.2040803@voidspace.org.uk> Message-ID: <87wrmymvog.fsf@uwakimon.sk.tsukuba.ac.jp> Michael Foord writes: > It seemed from the discussion that the biggest barrier to enabling it by > default was possible difficulties when embedding Python (multiple > interpreters, potential conflicts with application signal handling). A > public C-API to disable the functionality per interpreter would be one > option for this. That's not really good enough. The point of installing a handler like this is to "catch them squirmers". All you have to do is override some incautious developer's own squirmer-trap handler once, and Python has made an Enemy-For-Life. (This happened to XEmacs with esound. I immediately removed esound and anything that depends on it from my workstation. ;-) YMMV and you may think that that is not so important; my point is that the proposal to provide a way to disable does not at all address the objection. > Another possibility would be providing a C-API to enable it and > have the Python interpreter application call this, so that the > functionality remains off by default for embedded interpreters but > on for normal uses. I think this is heading in the right direction. Note: My own experience with such handlers has been positive, but it does not involve embedding interpreters in either direction, so not really helpful in addressing this objection. Precisely *because* my own experience has been positive, I worry about interfering with some third party's handler. From jcea at jcea.es Sat Dec 25 05:40:19 2010 From: jcea at jcea.es (Jesus Cea) Date: Sat, 25 Dec 2010 05:40:19 +0100 Subject: [Python-Dev] Compiling CPython to javascript Message-ID: <4D1575B3.3050809@jcea.es> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Interesting development. http://syntensity.com/static/python.html http://syntensity.blogspot.com/ - -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/_/_/_/ . _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iQCVAwUBTRV1s5lgi5GaxT1NAQLO1gQAjwjfyVHmQOgNg9cBD/lDZFoHJhPSbfaY KGnXVOVbXoAo6+o7Ne7mXXWamD38VK9et7dWckyWbqubHk1IGLuP9eCncKKjuX4j UTFMG26i4p24oQ6AMyUs2wuC7ockZ4PK8r7vcPs3YxSRR5rx81ojaH6SFQ2qJJGc kRTtNe+iExc= =UPG8 -----END PGP SIGNATURE----- From stefan_ml at behnel.de Sat Dec 25 11:59:36 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 25 Dec 2010 11:59:36 +0100 Subject: [Python-Dev] nonlocal x = value In-Reply-To: <4D145DC4.9000107@avl.com> References: <4D145DC4.9000107@avl.com> Message-ID: Hrvoje Niksic, 24.12.2010 09:45: > On 12/23/2010 10:03 PM, Laurens Van Houtven wrote: >> On Thu, Dec 23, 2010 at 9:51 PM, Georg Brandl wrote: >>> Yes and no -- there may not be an ambiguity to the parser, but still to >>> the human. Except if you disallow the syntax in any case, requiring >>> people to write >>> >>> nonlocal x = (3, y) >>> >>> which is then again inconsistent with ordinary assignment statements. >> >> Right -- but (and hence the confusion) I was arguing for not mixing >> global/nonlocal with assignment at all, and instead having nonlocal >> and global only take one or more names. That would (obviously) remove >> any such ambiguity ;-) > > I would like to offer the opposing viewpoint: nonlocal x = value is a > useful shortcut because nonlocal is used in closure callbacks where > brevity matters. I doubt that it really matters so much that one line more kills readability. It's still a relatively rare use case after all. > The reason nonlocal is introduced is to change the > variable, so it makes sense that the two can be done in the same line of > code. But still, this is just a special case. If the variable is changed more than once, you'd end up with one assignment with nonlocal and one without. This just adds to the growing list of code inconsistencies presented in this thread. > As for global x = value being disallowed, I have been annoyed at times with > that, so that sounds like a good argument to change both. > > Requiring the parentheses for tuple creation sounds like a good compromise > for resolving the ambiguity, consistent with similar limitations of the > generator expression syntax. Why introduce such a pitfall for coders here? "Require" doesn't mean the parser can enforce it. If it's not there, it just means something else, so it's up to the coder to get it right. The exact same kind of situation that was fixed for the except syntax in Python 3. Stefan From g.brandl at gmx.net Sat Dec 25 20:35:58 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 25 Dec 2010 20:35:58 +0100 Subject: [Python-Dev] r87478 - python/branches/py3k/Misc/maintainers.rst In-Reply-To: <20101224215903.3E813EE9BA@mail.python.org> References: <20101224215903.3E813EE9BA@mail.python.org> Message-ID: Eh, no. That is a different user. Alan didn't have Developer tracker privileges yet though, so I fixed that. Please revert. Georg Am 24.12.2010 22:59, schrieb terry.reedy: > Author: terry.reedy > Date: Fri Dec 24 22:59:03 2010 > New Revision: 87478 > > Log: > Match current tracker name, though I do not know if still active. > > > Modified: > python/branches/py3k/Misc/maintainers.rst > > Modified: python/branches/py3k/Misc/maintainers.rst > ============================================================================== > --- python/branches/py3k/Misc/maintainers.rst (original) > +++ python/branches/py3k/Misc/maintainers.rst Fri Dec 24 22:59:03 2010 > @@ -247,7 +247,7 @@ > xml.sax.saxutils > xml.sax.xmlreader > xmlrpc loewis > -zipfile alanmcintyre > +zipfile aimacintyre > zipimport > zlib > ================== =========== From victor.stinner at haypocalc.com Sat Dec 25 22:58:00 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sat, 25 Dec 2010 22:58:00 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <4D14993D.2040803@voidspace.org.uk> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> <1293157279.16757.3.camel@marge> <4D14993D.2040803@voidspace.org.uk> Message-ID: <1293314280.21272.183.camel@marge> Le vendredi 24 d?cembre 2010 ? 12:59 +0000, Michael Foord a ?crit : > I hope you will include it in 3.3 though; it is great functionality. I don't know, Python 3.3 is not planned yet. Anyway, faulthandler is already available and you can use it on Python 2.5 through 3.2. > I would really like to see it enabled by default as well. For different reasons, it is not a good idea to enable it by default (see the recent discussion on this list). > It seemed from the discussion that the biggest barrier to enabling it by > default was possible difficulties when ... Yes, that's why it was disabled by default in the latest version of my patch. > (multiple interpreters, I think that my fault handler works with multiple interpreters. > potential conflicts with application signal handling) Very few programs have an handler for SIGSEGV, SIGFPE, SIGILL or SIGBUS signals (but yes, there are such programs/libraries like RPy or XEmacs). > A public C-API to disable the functionality per interpreter would be one > option for this. Another possibility would be providing a C-API to > enable it and have the Python interpreter application call this, so that > the functionality remains off by default for embedded interpreters but > on for normal uses. I think that it's better to disable it by default, and only enabled by the user or by the developer of a specific application. -- Anyway, you don't have to wait Python 3.3 (or Python 3.2): you can already use faulthandler ;-) I created Windows installer for Python 2.6, 2.7 and 3.1 (I will do it for 2.5 too in the next faulthandler version, faulthandler 1.0 tests don't work with Python 2.5). faulthandler is a module: enable the handler is simple as "import faulthandler". Said differently, it's also disabled by default, and only enabled by the application developer. It's harder to enable it for the user, the site module should maybe be patched to load the module if an environment variable exist or a command line argument. Victor From merwok at netwok.org Sat Dec 25 23:31:18 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sat, 25 Dec 2010 23:31:18 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <1293314280.21272.183.camel@marge> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> <1293157279.16757.3.camel@marge> <4D14993D.2040803@voidspace.org.uk> <1293314280.21272.183.camel@marge> Message-ID: <4D1670B6.6000801@netwok.org> > faulthandler is a module: enable the handler is simple as "import > faulthandler". That sounds like a source of unwanted behavior (aka problems) if the handler is enabled by ?pydoc faulthandler? or by a pkgutil walk. You may want to consider using a function to enable the functionality (and add one to disable it). Regards From exarkun at twistedmatrix.com Sun Dec 26 15:10:49 2010 From: exarkun at twistedmatrix.com (exarkun at twistedmatrix.com) Date: Sun, 26 Dec 2010 14:10:49 -0000 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <4D1670B6.6000801@netwok.org> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> <1293157279.16757.3.camel@marge> <4D14993D.2040803@voidspace.org.uk> <1293314280.21272.183.camel@marge> <4D1670B6.6000801@netwok.org> Message-ID: <20101226141049.5919.1054241426.divmod.xquotient.369@localhost.localdomain> On 25 Dec, 10:31 pm, merwok at netwok.org wrote: >>faulthandler is a module: enable the handler is simple as "import >>faulthandler". > >That sounds like a source of unwanted behavior (aka problems) if the >handler is enabled by 1Cpydoc faulthandler 1D or by a pkgutil walk. You >may want to consider using a function to enable the functionality (and >add one to disable it). Enormous +1. Jean-Paul From merwok at netwok.org Sun Dec 26 18:13:53 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Sun, 26 Dec 2010 18:13:53 +0100 Subject: [Python-Dev] [Python-checkins] r87445 - python/branches/py3k/Lib/numbers.py In-Reply-To: References: <20101223184133.9DED1EE995@mail.python.org> Message-ID: <4D1777D1.3060104@netwok.org> Le 24/12/2010 02:08, Nick Coghlan a ?crit : > On Fri, Dec 24, 2010 at 4:41 AM, eric.araujo wrote: >> Fix small inaccuracy: there is no index function > > Yes, there is, it just isn't a builtin - it lives in the operator module. Defining object.__index__ with operator.index seems pretty circular to me :) http://docs.python.org/dev/reference/datamodel#object.__index__ does it, but it should be fixed IMO. The difference between __int__ and __index__ is also not clear. >> def __index__(self): >> - """index(self)""" >> + """someobject[self]""" >> return int(self) > > Changing the docstring to say "operator.index(self)" would be the > clearest solution here. I disagree. __add__ is documented as implementing +, not operator.add. > (Choosing to accept arbitrary index objects as > integer equivalents is up to the object being indexed, just like > interpreting slices is - a dict, for example, will never invoke > __index__ methods). I honestly don?t know what the best fix is. We could copy the doc from datamodel (?called whenever Python needs an integer object (such as in slicing, or in the built-in bin(), hex() and oct() functions)?). I?ve been told on IRC to let Mark Dickison decide how to fix the docstrings in the numbers module (deleting them being of course an option: magic methods are documented in the language reference, they don?t need docstrings). Regards From rrr at ronadam.com Sun Dec 26 19:09:34 2010 From: rrr at ronadam.com (Ron Adam) Date: Sun, 26 Dec 2010 12:09:34 -0600 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <763DCF04-033D-455B-A7BF-9D5813BBDFD4@gmail.com> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> <4D14C5C6.9070703@voidspace.org.uk> <763DCF04-033D-455B-A7BF-9D5813BBDFD4@gmail.com> Message-ID: <4D1784DE.6020809@ronadam.com> On 12/24/2010 02:03 PM, Raymond Hettinger wrote: > > On Dec 24, 2010, at 10:56 AM, Terry Reedy wrote: > >> On 12/24/2010 11:09 AM, Michael Foord wrote: >>> On 22/12/2010 02:26, Terry Reedy wrote: >>>> On 12/21/2010 7:17 AM, Michael Foord wrote: >>>> >>>> My first priority is that doc and code match. >>>> Close second is consistency (hence, ease of learning and use) between >>>> various AssertXs. >>>> >>>>> Symmetrical diffs (element in first not in second, element in second not >>>>> in first) solves the problem without imposing an order on the arguments. >>>> >>>> Where applicable, I prefer this as unambiguous output headings. >>>> >>> Could you explain what you mean? >> >> I was referring back to an output example symmetric diff that was >> clipped somewhere along the way: >> >> In x not in y: ... In y not in x: ... >> >> rather than just using -,+ prefixes which are not necessarily >> self-explanatory. 'Not applicable' would refer to output from difflib >> which necessarily is ordered. > FWIW, I think + and - prefixes are much better for diffs that some > made-up verbiage. People are used to seeing diffs with + and -. > Anything else will be so contrived that it's net effect will be to make > the output confusing and hard to interpret. Agree. > If you want, add two lines of explanation before the diff: + means "in > x, not in y" - means "in y, not it x" > > The notion of "making symmetric" can easily get carried too far, which > a corresponding loss of useability. I agree with this also. I don't understand the effort to make the tests be symmetric when many of the tests are non-symmetric. (see list below) I think the terms expected and actual are fine and help more than they hurt. I think of these as "actual result" and "expected result". A clearer terminology might be "expr" and "expected_result". Where a tests can be used *as if* they are symmetric, but the diff context is reversed, I think that that is ok. It just needs a entry in the docs that says that will happen if you do it. That won't break tests already written. Also notice (in the list below) that the use of 'a' and 'b' do not indicate a test is symmetric, but instead are used where they are *not-symmetric*. First and second could be used for those, but I think 'a' and 'b' have less mental luggage when it comes to visually seeing the meaning of the method signature in those cases. Tests where the order is not important usually use numbered but like arguments, such as "expr1" and "expr2" or "list1" and "list2". This makes sense to me. "obj1" and "obj2" are just two objects. The terms "x in y" and "x not in y" look like what you should get from containment or regex asserts. I guess what I'm try to say is think of the whole picture when trying to make improvements like these, an idea that works for one or two things may not scale well. Cheers, Ron Non-symmetric assert methods. assertDictContainsSubset(self, expected, actual, msg=None) assertFalse(self, expr, msg=None) assertGreater(self, a, b, msg=None) assertGreaterEqual(self, a, b, msg=None) assertIn(self, member, container, msg=None) assertIsInstance(self, obj, cls, msg=None) assertIsNone(self, obj, msg=None) assertIsNotNone(self, obj, msg=None) assertLess(self, a, b, msg=None) assertLessEqual(self, a, b, msg=None) assertNotIn(self, member, container, msg=None) assertIsInstance(self, obj, cls, msg=None) assertIsNone(self, obj, msg=None) assertIsNotNone(self, obj, msg=None) assertNotIn(self, member, container, msg=None) assertNotIsInstance(self, obj, cls, msg=None) assertRegex(self, text, expected_regex, msg=None) assertNotRegexMatches(self, text, unexpected_regex, msg=None) assertRaises(self, excClass, callableObj=None, *args, **kwargs) assertRaisesRegex(self, expected_exception, expected_regex, callable_obj=None, *args, **kwargs) assertRegex(self, text, expected_regex, msg=None) assertTrue(self, expr, msg=None) assertWarns(self, expected_warning, callable_obj=None, *args, **kwargs) assertWarnsRegex(self, expected_warning, expected_regex, callable_obj=None, *args, **kwargs) From rrr at ronadam.com Sun Dec 26 19:09:34 2010 From: rrr at ronadam.com (Ron Adam) Date: Sun, 26 Dec 2010 12:09:34 -0600 Subject: [Python-Dev] r87389 - in python/branches/py3k: Doc/library/unittest.rst Lib/unittest/case.py Misc/NEWS In-Reply-To: <763DCF04-033D-455B-A7BF-9D5813BBDFD4@gmail.com> References: <20101218200004.99110EE985@mail.python.org> <20101218214830.1aa8be32@pitrou.net> <20101219141355.5ac7cd2f@pitrou.net> <31185718-F276-46CE-82E2-6C6B6EC920E8@gmail.com> <4D0F536A.8050808@voidspace.org.uk> <1292852841.3737.25.camel@localhost.localdomain> <4D0F622E.7050902@voidspace.org.uk> <1292855497.3737.55.camel@localhost.localdomain> <4D109AEA.7010908@voidspace.org.uk> <4D14C5C6.9070703@voidspace.org.uk> <763DCF04-033D-455B-A7BF-9D5813BBDFD4@gmail.com> Message-ID: <4D1784DE.6020809@ronadam.com> On 12/24/2010 02:03 PM, Raymond Hettinger wrote: > > On Dec 24, 2010, at 10:56 AM, Terry Reedy wrote: > >> On 12/24/2010 11:09 AM, Michael Foord wrote: >>> On 22/12/2010 02:26, Terry Reedy wrote: >>>> On 12/21/2010 7:17 AM, Michael Foord wrote: >>>> >>>> My first priority is that doc and code match. >>>> Close second is consistency (hence, ease of learning and use) between >>>> various AssertXs. >>>> >>>>> Symmetrical diffs (element in first not in second, element in second not >>>>> in first) solves the problem without imposing an order on the arguments. >>>> >>>> Where applicable, I prefer this as unambiguous output headings. >>>> >>> Could you explain what you mean? >> >> I was referring back to an output example symmetric diff that was >> clipped somewhere along the way: >> >> In x not in y: ... In y not in x: ... >> >> rather than just using -,+ prefixes which are not necessarily >> self-explanatory. 'Not applicable' would refer to output from difflib >> which necessarily is ordered. > FWIW, I think + and - prefixes are much better for diffs that some > made-up verbiage. People are used to seeing diffs with + and -. > Anything else will be so contrived that it's net effect will be to make > the output confusing and hard to interpret. Agree. > If you want, add two lines of explanation before the diff: + means "in > x, not in y" - means "in y, not it x" > > The notion of "making symmetric" can easily get carried too far, which > a corresponding loss of useability. I agree with this also. I don't understand the effort to make the tests be symmetric when many of the tests are non-symmetric. (see list below) I think the terms expected and actual are fine and help more than they hurt. I think of these as "actual result" and "expected result". A clearer terminology might be "expr" and "expected_result". Where a tests can be used *as if* they are symmetric, but the diff context is reversed, I think that that is ok. It just needs a entry in the docs that says that will happen if you do it. That won't break tests already written. Also notice (in the list below) that the use of 'a' and 'b' do not indicate a test is symmetric, but instead are used where they are *not-symmetric*. First and second could be used for those, but I think 'a' and 'b' have less mental luggage when it comes to visually seeing the meaning of the method signature in those cases. Tests where the order is not important usually use numbered but like arguments, such as "expr1" and "expr2" or "list1" and "list2". This makes sense to me. "obj1" and "obj2" are just two objects. The terms "x in y" and "x not in y" look like what you should get from containment or regex asserts. I guess what I'm try to say is think of the whole picture when trying to make improvements like these, an idea that works for one or two things may not scale well. Cheers, Ron Non-symmetric assert methods. assertDictContainsSubset(self, expected, actual, msg=None) assertFalse(self, expr, msg=None) assertGreater(self, a, b, msg=None) assertGreaterEqual(self, a, b, msg=None) assertIn(self, member, container, msg=None) assertIsInstance(self, obj, cls, msg=None) assertIsNone(self, obj, msg=None) assertIsNotNone(self, obj, msg=None) assertLess(self, a, b, msg=None) assertLessEqual(self, a, b, msg=None) assertNotIn(self, member, container, msg=None) assertIsInstance(self, obj, cls, msg=None) assertIsNone(self, obj, msg=None) assertIsNotNone(self, obj, msg=None) assertNotIn(self, member, container, msg=None) assertNotIsInstance(self, obj, cls, msg=None) assertRegex(self, text, expected_regex, msg=None) assertNotRegexMatches(self, text, unexpected_regex, msg=None) assertRaises(self, excClass, callableObj=None, *args, **kwargs) assertRaisesRegex(self, expected_exception, expected_regex, callable_obj=None, *args, **kwargs) assertRegex(self, text, expected_regex, msg=None) assertTrue(self, expr, msg=None) assertWarns(self, expected_warning, callable_obj=None, *args, **kwargs) assertWarnsRegex(self, expected_warning, expected_regex, callable_obj=None, *args, **kwargs) From svenbrauch at googlemail.com Sun Dec 26 22:41:41 2010 From: svenbrauch at googlemail.com (Sven Brauch) Date: Sun, 26 Dec 2010 22:41:41 +0100 Subject: [Python-Dev] Column offsets for attribute nodes Message-ID: Hi there, I recently filed a feature request in the tracker to change the behaviour of the parser in terms of setting ranges on attribute AST nodes, because I'm working on an application which needs more information than is currently provided. I suggested to change the behaviour from foo.bar.baz # <- foo is said to start at column 0, bar at 0 and baz at 0 (current) to foo.bar.baz # <- foo starts at 0, bar at 3 and baz at 7 (suggestion) In that discussion, there's been different opinions about which behaviour is better; main arguments were "consistency" for the current and "usefulness" for the suggested behaviour. It has been proposed to ask the question on this list, that's why I'm doing that now. :) The thread can be found here: http://bugs.python.org/issue10769 So, which version do you think to be better: the current one or the suggested one? Best regards, Sven From victor.stinner at haypocalc.com Sun Dec 26 23:57:10 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Sun, 26 Dec 2010 23:57:10 +0100 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <20101226141049.5919.1054241426.divmod.xquotient.369@localhost.localdomain> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> <1293157279.16757.3.camel@marge> <4D14993D.2040803@voidspace.org.uk> <1293314280.21272.183.camel@marge> <4D1670B6.6000801@netwok.org> <20101226141049.5919.1054241426.divmod.xquotient.369@localhost.localdomain> Message-ID: <1293404230.6036.0.camel@marge> Le dimanche 26 d?cembre 2010 ? 14:10 +0000, exarkun at twistedmatrix.com a ?crit : > On 25 Dec, 10:31 pm, merwok at netwok.org wrote: > >>faulthandler is a module: enable the handler is simple as "import > >>faulthandler". > > > >That sounds like a source of unwanted behavior (aka problems) if the > >handler is enabled by 1Cpydoc faulthandler 1D or by a pkgutil walk. You > >may want to consider using a function to enable the functionality (and > >add one to disable it). > > Enormous +1. I don't know pkgutil. How does it work? In which case would it load the faulthandler module? faulthandler is currently only written in C. Victor From benjamin at python.org Mon Dec 27 00:34:55 2010 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 26 Dec 2010 17:34:55 -0600 Subject: [Python-Dev] Column offsets for attribute nodes In-Reply-To: References: Message-ID: 2010/12/26 Sven Brauch : > Hi there, > > I recently filed a feature request in the tracker to change the > behaviour of the parser in terms of setting ranges on attribute AST > nodes, because I'm working on an application which needs more > information than is currently provided. I suggested to change the > behaviour from > foo.bar.baz # <- foo is said to start at column 0, bar at 0 and baz at > 0 (current) > to > foo.bar.baz # <- foo starts at 0, bar at 3 and baz at 7 (suggestion) > > In that discussion, there's been different opinions about which > behaviour is better; main arguments were "consistency" for the current > and "usefulness" for the suggested behaviour. It has been proposed to > ask the question on this list, that's why I'm doing that now. :) My argument against this change is that an attribute includes the expression from which the attribute is being taken. This is consistent with subscripts and calls, which both have the lineno and col_offset of their source expressions. -- Regards, Benjamin From svenbrauch at googlemail.com Mon Dec 27 00:52:10 2010 From: svenbrauch at googlemail.com (Sven Brauch) Date: Mon, 27 Dec 2010 00:52:10 +0100 Subject: [Python-Dev] Column offsets for attribute nodes In-Reply-To: References: Message-ID: It should maybe be noted that the proposed patch would change that, too, so it would be the same behaviour for all three types (subscripts, calls, and attributes) again. Just more intuitive. :) 2010/12/27 Benjamin Peterson : > 2010/12/26 Sven Brauch : >> Hi there, >> >> I recently filed a feature request in the tracker to change the >> behaviour of the parser in terms of setting ranges on attribute AST >> nodes, because I'm working on an application which needs more >> information than is currently provided. I suggested to change the >> behaviour from >> foo.bar.baz # <- foo is said to start at column 0, bar at 0 and baz at >> 0 (current) >> to >> foo.bar.baz # <- foo starts at 0, bar at 3 and baz at 7 (suggestion) >> >> In that discussion, there's been different opinions about which >> behaviour is better; main arguments were "consistency" for the current >> and "usefulness" for the suggested behaviour. It has been proposed to >> ask the question on this list, that's why I'm doing that now. :) > > My argument against this change is that an attribute includes the > expression from which the attribute is being taken. This is consistent > with subscripts and calls, which both have the lineno and col_offset > of their source expressions. > > > > -- > Regards, > Benjamin > From ncoghlan at gmail.com Mon Dec 27 01:01:21 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Dec 2010 10:01:21 +1000 Subject: [Python-Dev] [Python-checkins] r87445 - python/branches/py3k/Lib/numbers.py In-Reply-To: <4D1777D1.3060104@netwok.org> References: <20101223184133.9DED1EE995@mail.python.org> <4D1777D1.3060104@netwok.org> Message-ID: On Mon, Dec 27, 2010 at 3:13 AM, ?ric Araujo wrote: > Le 24/12/2010 02:08, Nick Coghlan a ?crit : >> On Fri, Dec 24, 2010 at 4:41 AM, eric.araujo wrote: >>> Fix small inaccuracy: there is no index function >> >> Yes, there is, it just isn't a builtin - it lives in the operator module. > Defining object.__index__ with operator.index seems pretty circular to > me :) ?http://docs.python.org/dev/reference/datamodel#object.__index__ > does it, but it should be fixed IMO. ?The difference between __int__ and > __index__ is also not clear. Yes, the definition in the language reference could definitely be improved to mention the semantics first, and then reference operator.index second. Possible wording "Indicates to the Python interpreter that the object is semantically equivalent to the returned integer, rather than merely supporting a possibly lossy coercion to an integer (i.e. as the __int__ method allows for types like float and decimal.Decimal). This allows non-builtin objects to be used as sequence indices, elements of a slice definition, multiplies in sequence repetition, etc. Can be invoked explicitly from Python code via operator.index()" Removing the circularity from the definitions of __index__ and operator.index doesn't have a great deal to do with the docstrings in numbers.py, though. >>> ? ? def __index__(self): >>> - ? ? ? ?"""index(self)""" >>> + ? ? ? ?"""someobject[self]""" >>> ? ? ? ? return int(self) >> >> Changing the docstring to say "operator.index(self)" would be the >> clearest solution here. > I disagree. ?__add__ is documented as implementing +, not operator.add. That's because "+" is the idiomatic spelling. operator.index *is* the Python level spelling of obj.__index__() - there is no other way to spell it (other than calling the method explicitly, which is subtly different). >> (Choosing to accept arbitrary index objects as >> integer equivalents is up to the object being indexed, just like >> interpreting slices is - a dict, for example, will never invoke >> __index__ methods). > I honestly don?t know what the best fix is. ?We could copy the doc from > datamodel (?called whenever Python needs an integer object (such as in > slicing, or in the built-in bin(), hex() and oct() functions)?). ?I?ve > been told on IRC to let Mark Dickison decide how to fix the docstrings > in the numbers module (deleting them being of course an option: magic > methods are documented in the language reference, they don?t need > docstrings). Indeed. However, as a reference module for the numeric tower, it makes a certain amount of sense to keep the docstrings in this particular case. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From v+python at g.nevcal.com Mon Dec 27 01:03:06 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sun, 26 Dec 2010 16:03:06 -0800 Subject: [Python-Dev] Column offsets for attribute nodes In-Reply-To: References: Message-ID: <4D17D7BA.1020207@g.nevcal.com> On 12/26/2010 1:41 PM, Sven Brauch wrote: > Hi there, > > I recently filed a feature request in the tracker to change the > behaviour of the parser in terms of setting ranges on attribute AST > nodes, because I'm working on an application which needs more > information than is currently provided. I suggested to change the > behaviour from > foo.bar.baz #<- foo is said to start at column 0, bar at 0 and baz at > 0 (current) > to > foo.bar.baz #<- foo starts at 0, bar at 3 and baz at 7 (suggestion) > > In that discussion, there's been different opinions about which > behaviour is better; main arguments were "consistency" for the current > and "usefulness" for the suggested behaviour. It has been proposed to > ask the question on this list, that's why I'm doing that now. :) > The thread can be found here: http://bugs.python.org/issue10769 > > So, which version do you think to be better: the current one or the > suggested one? The current one is better, but maybe bar at 4 and baz at 8 would be even better. In other words, I don't think pointing at the "." is useful? Was that your intention? -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Dec 27 01:07:50 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Dec 2010 10:07:50 +1000 Subject: [Python-Dev] Fault handler updated, now disabled by default In-Reply-To: <1293404230.6036.0.camel@marge> References: <1293070093.12866.14.camel@marge> <4D12B5E1.7060608@v.loewis.de> <1293124842.26635.3.camel@marge> <4D139395.3000709@v.loewis.de> <1293157279.16757.3.camel@marge> <4D14993D.2040803@voidspace.org.uk> <1293314280.21272.183.camel@marge> <4D1670B6.6000801@netwok.org> <20101226141049.5919.1054241426.divmod.xquotient.369@localhost.localdomain> <1293404230.6036.0.camel@marge> Message-ID: On Mon, Dec 27, 2010 at 8:57 AM, Victor Stinner wrote: > Le dimanche 26 d?cembre 2010 ? 14:10 +0000, exarkun at twistedmatrix.com a > ?crit : >> On 25 Dec, 10:31 pm, merwok at netwok.org wrote: >> >>faulthandler is a module: enable the handler is simple as "import >> >>faulthandler". >> > >> >That sounds like a source of unwanted behavior (aka problems) if the >> >handler is enabled by ?1Cpydoc faulthandler 1D or by a pkgutil walk. ?You >> >may want to consider using a function to enable the functionality (and >> >add one to disable it). >> >> Enormous +1. > > I don't know pkgutil. How does it work? In which case would it load the > faulthandler module? > > faulthandler is currently only written in C. pkgutil includes a function that lets you walk the entire module heirarchy, implicitly importing everything, including all the builtin modules. It's one of the reasons doing things as side-effects of import is considered highly undesirable. The pydoc tests do this when they bring the (docstring-based) documentation server up to check its handling of HTTP requests. (we recently picked up an implicit addition of a logging handler by concurrent.futures due to this effect). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From svenbrauch at googlemail.com Mon Dec 27 01:14:53 2010 From: svenbrauch at googlemail.com (Sven Brauch) Date: Mon, 27 Dec 2010 01:14:53 +0100 Subject: [Python-Dev] Column offsets for attribute nodes In-Reply-To: <4D17D7BA.1020207@g.nevcal.com> References: <4D17D7BA.1020207@g.nevcal.com> Message-ID: Hey, yeah, you're right, that would be even better. Currently it would also give a 3 for bar in "foo. bar.baz". I'd be fine with both solutions, but yours seems more logical. Bye, Sven 2010/12/27 Glenn Linderman : > On 12/26/2010 1:41 PM, Sven Brauch wrote: > > Hi there, > > I recently filed a feature request in the tracker to change the > behaviour of the parser in terms of setting ranges on attribute AST > nodes, because I'm working on an application which needs more > information than is currently provided. I suggested to change the > behaviour from > foo.bar.baz # <- foo is said to start at column 0, bar at 0 and baz at > 0 (current) > to > foo.bar.baz # <- foo starts at 0, bar at 3 and baz at 7 (suggestion) > > In that discussion, there's been different opinions about which > behaviour is better; main arguments were "consistency" for the current > and "usefulness" for the suggested behaviour. It has been proposed to > ask the question on this list, that's why I'm doing that now. :) > The thread can be found here: http://bugs.python.org/issue10769 > > So, which version do you think to be better: the current one or the > suggested one? > > The current one is better, but maybe bar at 4 and baz at 8 would be even > better. > > In other words, I don't think pointing at the "." is useful?? Was that your > intention? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/svenbrauch%40googlemail.com > > From ncoghlan at gmail.com Mon Dec 27 01:15:45 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Dec 2010 10:15:45 +1000 Subject: [Python-Dev] Range __contains__ and objects with __index__ methods Message-ID: Starting in Python 3.2, range() supports fast containment checking for integers (i.e. based on an O(1) arithmetic calculation rather than an O(N) iteration through the entire sequence). Currently, this fast path ignores objects that implement __index__ - they are relegated to the slow iterative search. This seems wrong to me - the semantics of __index__ are such that it is meant to be used for objects that are alternative representations of the corresponding Python integers (e.g. numpy scalars, or integers that use a particular number of bits in memory). Under that interpretation, if an object provides __index__, we should use the fast path instead of calling __eq__ multiple times. Thoughts? Regards, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Mon Dec 27 01:32:58 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Dec 2010 10:32:58 +1000 Subject: [Python-Dev] Column offsets for attribute nodes In-Reply-To: References: Message-ID: On Mon, Dec 27, 2010 at 9:34 AM, Benjamin Peterson wrote: > 2010/12/26 Sven Brauch : >> In that discussion, there's been different opinions about which >> behaviour is better; main arguments were "consistency" for the current >> and "usefulness" for the suggested behaviour. It has been proposed to >> ask the question on this list, that's why I'm doing that now. :) > > My argument against this change is that an attribute includes the > expression from which the attribute is being taken. This is consistent > with subscripts and calls, which both have the lineno and col_offset > of their source expressions. I'd like to see the impact on existing uses of this information (primarily SyntaxErrors) before forming a firm opinion, but my initial inclination is that retaining the column information for the subattribute names is a better option. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From tjreedy at udel.edu Mon Dec 27 02:52:23 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 26 Dec 2010 20:52:23 -0500 Subject: [Python-Dev] Range __contains__ and objects with __index__ methods In-Reply-To: References: Message-ID: On 12/26/2010 7:15 PM, Nick Coghlan wrote: > Starting in Python 3.2, range() supports fast containment checking for > integers (i.e. based on an O(1) arithmetic calculation rather than an > O(N) iteration through the entire sequence). > > Currently, this fast path ignores objects that implement __index__ - > they are relegated to the slow iterative search. > > This seems wrong to me - the semantics of __index__ are such that it > is meant to be used for objects that are alternative representations > of the corresponding Python integers (e.g. numpy scalars, or integers > that use a particular number of bits in memory). Under that > interpretation, if an object provides __index__, we should use the > fast path instead of calling __eq__ multiple times. If I understand, you are proposing 'replacing' o with o.__index__() (when possible) and proceeding on the fast path rather than iterating the range and comparing o for equality each value in the range (the slow path). I suppose this would change semantics if o != o.__index__(). Equality is not specified in the manual: "object.__index__(self) Called to implement operator.index(). Also called whenever Python needs an integer object (such as in slicing, or in the built-in bin(), hex() and oct() functions). Must return an integer." However "operator.__index__(a) Return a converted to an integer. Equivalent to a.__index__()." comes close to implying equality (if possible). What are the actual used of .__index__? -- Terry Jan Reedy From tjreedy at udel.edu Mon Dec 27 02:58:34 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 26 Dec 2010 20:58:34 -0500 Subject: [Python-Dev] [Python-checkins] r87445 - python/branches/py3k/Lib/numbers.py In-Reply-To: References: <20101223184133.9DED1EE995@mail.python.org> <4D1777D1.3060104@netwok.org> Message-ID: On 12/26/2010 7:01 PM, Nick Coghlan wrote: > Yes, the definition in the language reference could definitely be > improved to mention the semantics first, and then reference > operator.index second. > > Possible wording "Indicates to the Python interpreter that the object > is semantically equivalent to the returned integer, rather than merely > supporting a possibly lossy coercion to an integer If that is the intent of __index__, the doc should say so more clearly. That clarification would change my answer to your question about range. > (i.e. as the > __int__ method allows for types like float and decimal.Decimal). This > allows non-builtin objects to be used as sequence indices, elements of > a slice definition, multiplies in sequence repetition, etc. Can be > invoked explicitly from Python code via operator.index()" > > Removing the circularity from the definitions of __index__ and > operator.index doesn't have a great deal to do with the docstrings in > numbers.py, though. It is both related and needed though. IE, it is hard to answer questions about what to to with .index if the intended meaning of .index is not very clear ;-). -- Terry Jan Reedy From martin at v.loewis.de Mon Dec 27 03:16:11 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 27 Dec 2010 03:16:11 +0100 Subject: [Python-Dev] Range __contains__ and objects with __index__ methods In-Reply-To: References: Message-ID: <4D17F6EB.1080907@v.loewis.de> > What are the actual used of .__index__? Can you please rephrase this question? Regards, Martin From ncoghlan at gmail.com Mon Dec 27 07:13:16 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Dec 2010 16:13:16 +1000 Subject: [Python-Dev] Range __contains__ and objects with __index__ methods In-Reply-To: References: Message-ID: On Mon, Dec 27, 2010 at 11:52 AM, Terry Reedy wrote: > Return a converted to an integer. Equivalent to a.__index__()." > comes close to implying equality (if possible). > > What are the actual used of .__index__? PEP 357 gives the original rationale - it was to allow integer-like objects (such as numpy scalars) to be used as sequence indices, as well as slice parameters. I would have to grep the source to get a full list of uses, but I believe it is already used in at least those two cases, as well as for sequence repetition (via '*') and to identify permitted inputs to bin/oct/hex. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From raymond.hettinger at gmail.com Mon Dec 27 10:43:33 2010 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 27 Dec 2010 01:43:33 -0800 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: <4D145DC4.9000107@avl.com> Message-ID: <9D8DAD45-36DF-4694-89CA-65C42FA846C3@gmail.com> On Dec 25, 2010, at 2:59 AM, Stefan Behnel wrote: > Hrvoje Niksic, 24.12.2010 09:45: >> On 12/23/2010 10:03 PM, Laurens Van Houtven wrote: >>> On Thu, Dec 23, 2010 at 9:51 PM, Georg Brandl wrote: >>>> Yes and no -- there may not be an ambiguity to the parser, but still to >>>> the human. Except if you disallow the syntax in any case, requiring >>>> people to write >>>> >>>> nonlocal x = (3, y) >>>> >>>> which is then again inconsistent with ordinary assignment statements. >>> >>> Right -- but (and hence the confusion) I was arguing for not mixing >>> global/nonlocal with assignment at all, and instead having nonlocal >>> and global only take one or more names. That would (obviously) remove >>> any such ambiguity ;-) >> >> I would like to offer the opposing viewpoint: nonlocal x = value is a >> useful shortcut because nonlocal is used in closure callbacks where >> brevity matters. > > I doubt that it really matters so much that one line more kills readability. It's still a relatively rare use case after all. > > >> The reason nonlocal is introduced is to change the >> variable, so it makes sense that the two can be done in the same line of >> code. FWIW, I'm entirely opposed to doing an assignment in a nonlocal definition. * It is easily mis-parsed by human (as shown by Georg's examples). * It looks very much like an initialization of a local variable in many languages, but it is not -- the variable has already been initialized in another scope. * It is not clear how to extend it to multiple variables (which we already allow). * It is entirely unnecessary. Just add a real assignment on the following line: local x x = 3, y * We've had global declarations for a very long time and never needed (or wanted) an assignment for it: global x = 3, y * The purported use case is rare (at best). Special cases aren't worth breaking the rules. And the purported goal (saving one line) isn't much of a payoff. * The language moratorium is ending but the aversion to non-essential micro-syntax changes persists. * And, Georg doesn't like it :-) Raymond From dickinsm at gmail.com Mon Dec 27 11:19:20 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 27 Dec 2010 10:19:20 +0000 Subject: [Python-Dev] [Python-checkins] r87445 - python/branches/py3k/Lib/numbers.py In-Reply-To: References: <20101223184133.9DED1EE995@mail.python.org> Message-ID: On Fri, Dec 24, 2010 at 1:08 AM, Nick Coghlan wrote: >> ? ? def __index__(self): >> - ? ? ? ?"""index(self)""" >> + ? ? ? ?"""someobject[self]""" >> ? ? ? ? return int(self) > > Changing the docstring to say "operator.index(self)" would be the > clearest solution here. Agreed. Certainly "someobject[self]" isn't right. (There's also a question about whether __index__ should really be defaulting to int, but that's another issue...) Mark From dickinsm at gmail.com Mon Dec 27 11:31:54 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 27 Dec 2010 10:31:54 +0000 Subject: [Python-Dev] nonlocal x = value In-Reply-To: <9D8DAD45-36DF-4694-89CA-65C42FA846C3@gmail.com> References: <4D145DC4.9000107@avl.com> <9D8DAD45-36DF-4694-89CA-65C42FA846C3@gmail.com> Message-ID: On Mon, Dec 27, 2010 at 9:43 AM, Raymond Hettinger wrote: > FWIW, I'm entirely opposed to doing an assignment in a nonlocal definition. > [...] -1 for assignment in nonlocal and global statements from me, too. Mark From ncoghlan at gmail.com Mon Dec 27 13:25:03 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Dec 2010 22:25:03 +1000 Subject: [Python-Dev] nonlocal x = value In-Reply-To: References: <4D145DC4.9000107@avl.com> <9D8DAD45-36DF-4694-89CA-65C42FA846C3@gmail.com> Message-ID: On Mon, Dec 27, 2010 at 8:31 PM, Mark Dickinson wrote: > On Mon, Dec 27, 2010 at 9:43 AM, Raymond Hettinger > wrote: >> FWIW, I'm entirely opposed to doing an assignment in a nonlocal definition. >> [...] > > -1 for assignment in nonlocal and global statements from me, too. Indeed. The PEP should be updated to be clear that that part was never implemented (referencing Raymond's post for the reasons why). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From dickinsm at gmail.com Mon Dec 27 13:44:26 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 27 Dec 2010 12:44:26 +0000 Subject: [Python-Dev] Range __contains__ and objects with __index__ methods In-Reply-To: References: Message-ID: Bah. I meant to send this to the list. (I suspect that Nick also meant to send his reply to the list.) On Mon, Dec 27, 2010 at 12:43 PM, Mark Dickinson wrote: > On Mon, Dec 27, 2010 at 12:23 PM, Nick Coghlan wrote: >> The symmetry only breaks for a class that breaks the invariant: >> >> ? x == operator.index(x) >> >> Failing to guarantee that invariant seems to violate the whole spirit >> of operator.index(). > > Agreed. (Though one also needs transitivity, symmetry, etc., of > equality, since what we really need is that x == y is interchangeable > with operator.index(x) == y.) > >> Perhaps this is what we should be documenting as >> the true distinguishing feature between the intended semantics of >> __index__ and the semantics of __int__? > > Hmm. ?Perhaps. ?For me, this doesn't fully capture the intent of > __index__, though. > > So the semantics (ignoring performance issues for now) would become: > > def range_contains(self, x) > ? ?try: > ? ? ? ?i = operator.index(x) > ? ?except TypeError: > ? ? ? ?i = x > ? ?return any(i == y for y in self) > > ? > > This change sounds fine to me; ?it would be good to have a > documentation note somewhere indicating that the range implementation > assumes x == index(x), though. ?This might belong in the range > documentation, or perhaps the __index__ method documentation could > indicate that some builtins might have unpredictable behaviour if the > identity is violated. > > Mark > From dickinsm at gmail.com Mon Dec 27 13:46:21 2010 From: dickinsm at gmail.com (Mark Dickinson) Date: Mon, 27 Dec 2010 12:46:21 +0000 Subject: [Python-Dev] Fwd: Range __contains__ and objects with __index__ methods In-Reply-To: References: Message-ID: .. and here's my original reply to Nick, which was also intended to go to the list. Sorry, folks. Mark ---------- Forwarded message ---------- From: Mark Dickinson Date: Mon, Dec 27, 2010 at 10:27 AM Subject: Re: [Python-Dev] Range __contains__ and objects with __index__ methods To: Nick Coghlan On Mon, Dec 27, 2010 at 12:15 AM, Nick Coghlan wrote: > Starting in Python 3.2, range() supports fast containment checking for > integers (i.e. based on an O(1) arithmetic calculation rather than an > O(N) iteration through the entire sequence). > > Currently, this fast path ignores objects that implement __index__ - > they are relegated to the slow iterative search. > > This seems wrong to me [...] Is seems to me that there are two separate issues here. Namely, (1) Determine the semantics of 'x in range(...)' for an object x that implements __index__, and (2) implement the containment check efficiently. At the moment, it looks as though the __index__ method is ignored entirely for the containment check, so you're proposing a change in semantics. ?If/when that change in semantics is made, fixing up the code to do the containment check efficiently seems like it should be a straightforward task. With the proposed change in semantics, the simple definition (x in range(...) iff x == y for some element y of range(...)) no longer holds. ?I'm not convinced (but not unconvinced either) that it's worth breaking that simplicity. Mark From g.brandl at gmx.net Mon Dec 27 15:01:16 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 27 Dec 2010 15:01:16 +0100 Subject: [Python-Dev] nonlocal x = value In-Reply-To: <9D8DAD45-36DF-4694-89CA-65C42FA846C3@gmail.com> References: <4D145DC4.9000107@avl.com> <9D8DAD45-36DF-4694-89CA-65C42FA846C3@gmail.com> Message-ID: Am 27.12.2010 10:43, schrieb Raymond Hettinger: > FWIW, I'm entirely opposed to doing an assignment in a nonlocal definition. > > * It is easily mis-parsed by human (as shown by Georg's examples). > * It looks very much like an initialization of a local variable in many languages, > but it is not -- the variable has already been initialized in another scope. Hah, and we'd probably get requests for a mandatory declaration of locals, ala local a = 5 for reasons of "consistency" or "explicit is better than implicit". Georg From vinay_sajip at yahoo.co.uk Mon Dec 27 16:29:22 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Mon, 27 Dec 2010 15:29:22 +0000 (UTC) Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs Message-ID: The logging configuration calls fileConfig and dictConfig disable all existing loggers, and enable only loggers explicitly named in the configuration (and their children). Although there is a disable_existing_loggers option for each configuration API, which can be used to prevent disabling of existing loggers, the default value of this option is True (for backward compatibility) and so existing ("old") loggers are disabled by default. This can lead to unexpected behaviour when loggers are used by the standard library. A recent issue, http://bugs.python.org/issue10626 relates to test_concurrent_futures failing, but only when exercised twice with a test_logging call in between, as in regrtest.py test_concurrent_futures test_logging test_concurrent_futures I've fixed test_logging so that it remembers the disabled state of existing loggers and restores it, and now the above test works fine. However, since recent changes in logging, we are expecting to use it to e.g. display messages on stderr when exceptions can't be raised in stdlib code. Thus, I think I may need to change the way disabling of loggers works, so I wanted to get some feedback from python-dev both about possible approaches and when to apply them (With 3.2 so close to release I'm not proposing to do anything precipitate, just raising the issue for discussion). Essentially, some or all of the loggers in the stdlib itself should perhaps be immune to the "disable existing loggers" logic. Otherwise, a fileConfig() or dictConfig() call with default semantics will prevent a stdlib message (logged when exceptions can't be raised) from being displayed, unless a user explicitly names those stdlib loggers in their configuration. This is not practical because users would need to update their configurations whenever a new logger appeared in the stdlib. Some possible approaches are: 1. Change the default configuration behaviour so that by default, existing loggers are NOT disabled. This is of course strongly backwards-incompatible, but would it make sense to consider it for 3.3? Perhaps we should add a warning? 2. Change the disabling logic so that it never disables stdlib loggers. This raises the question of how to identify them, and there are several loggers in the stdlib in 3.2: "py.warnings" (used by the logging package when warnings are redirected to logging), "concurrent.futures", "http.cookiejar", "multiprocessing" However, it would be good to have some consistency of naming stdlib loggers - perhaps using __name__ as is recommended practice for library and application developers, but with a prefix such as "py." to indicate that it's a part of the stdlib. This would make the disabling logic change easier as it could just check the "py." prefix rather than have an internal list of logger names, which would need more maintenance and be more error-prone. There are quite possibly other approaches I haven't thought of. I'd be grateful for your suggestions, as I'd like to minimise the impact on users, while making it easy to add more logging to the stdlib. Regards, Vinay Sajip From ethan at stoneleaf.us Mon Dec 27 20:37:38 2010 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 27 Dec 2010 11:37:38 -0800 Subject: [Python-Dev] Issue #6210: Exception Chaining missing method for suppressing context Message-ID: <4D18EB02.9040409@stoneleaf.us> I see the last comment was added on the 4th. Where should continued discussion take place -- bug-tracker, python-dev, ... ? ~Ethan~ From brian.curtin at gmail.com Mon Dec 27 20:38:10 2010 From: brian.curtin at gmail.com (Brian Curtin) Date: Mon, 27 Dec 2010 13:38:10 -0600 Subject: [Python-Dev] Issue #6210: Exception Chaining missing method for suppressing context In-Reply-To: <4D18EB02.9040409@stoneleaf.us> References: <4D18EB02.9040409@stoneleaf.us> Message-ID: On Mon, Dec 27, 2010 at 13:37, Ethan Furman wrote: > I see the last comment was added on the 4th. > > Where should continued discussion take place -- bug-tracker, python-dev, > ... ? > > ~Ethan~ Preferably on the bug tracker as to keep any discussion attached to the issue. If the discussion becomes stalled for reasons other than time, a discussion here might be warranted. -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Mon Dec 27 20:41:39 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 27 Dec 2010 11:41:39 -0800 Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs In-Reply-To: References: Message-ID: <4D18EBF3.9050202@g.nevcal.com> On 12/27/2010 7:29 AM, Vinay Sajip wrote: > The logging configuration calls fileConfig and dictConfig disable all existing > loggers, and enable only loggers explicitly named in the configuration (and > their children). Although there is a disable_existing_loggers option for each > configuration API, which can be used to prevent disabling of existing loggers, > the default value of this option is True (for backward compatibility) and so > existing ("old") loggers are disabled by default. Good that you recognized the bigger issue than 10626, and are wanting to address it. It would be interesting to hear the use case for calling fileConfig and/or dictConfig and/or basicConfig more than once anyway (although I guess basicConfig doesn't disable anything). That might shed some light on the approach that should be taken. My "intuitive" thought about the matter is that loggers, once enabled, shouldn't be disabled except explicitly by name, not implicitly by a flag parameter. So I'd expect a parameter that takes a list of loggers to disable, rather than a flag whose default value disables all of them. Of course this doesn't address the how and why of backward compatibility needs. Part of the problem, perhaps, is that stdlib loggers might be enabled before the application gets to its logging configuration call, and those would then be disabled accidentally. As I prepare to add logging to my web application this week, I've only found the need to initialize logging in one place and at one time... I doubt that any of the loggers you mentioned as being in the 3.2 stdlib will have been enabled by then, so I'll not have a problem, but if they proliferate, I could eventually in future versions. Granted, my application will likely be one of the simpler applications of logging, at least initially. While I agree it would be nice to have a stdlib logger naming convention of some sort, I don't think the naming convention should be used to implicitly not disable particular loggers. The only difference between stdlib and 3rd party libraries is whether they've become stable enough and useful enough for the whole Python community; many 3rd party libraries are quite stable enough and useful enough for particular applications, and could have similar issues as the stdlib regarding logger configuration. Having an API to disable loggers using an explicit list would require a corresponding API to obtain from logging the list of currently enabled loggers, and perhaps currently disabled ones as well? That way, if an application truly wanted to disable all currently enabled loggers, they would have an easy way to know the list... and perhaps the list of loggers they disable during configuration should be the first thing that is logged as a result of the xxxConfig API? Then, at least, they would not be ignorant of the accidental disabling. I guess even if the parameter were left as a default flag to turn off all loggers, if the list were logged, that would be alerting. But some applications may not expect such a log entry.... but maybe that is the least instrusive backwards-compatible action that can be taken, because if an application does configure logging, they likely do expect to have log messages to view/process. -------------- next part -------------- An HTML attachment was scrubbed... URL: From svenbrauch at googlemail.com Mon Dec 27 20:42:59 2010 From: svenbrauch at googlemail.com (Sven Brauch) Date: Mon, 27 Dec 2010 20:42:59 +0100 Subject: [Python-Dev] Column offsets for attribute nodes In-Reply-To: References: Message-ID: Glenn wrote: > [...] but maybe bar at 4 and baz at 8 would be even better. > In other words, I don't think pointing at the "." is useful? [...] Hi, here's a patch which results in what Glenn Linderman suggested: --- python-orig/Python/ast.c 2010-10-19 03:22:07.000000000 +0200 +++ python-new/Python/ast.c 2010-12-27 20:33:55.000000000 +0100 @@ -1615,7 +1615,7 @@ if (!attr_id) return NULL; return Attribute(left_expr, attr_id, Load, - LINENO(n), n->n_col_offset, c->c_arena); + LINENO(CHILD(n, 1)), CHILD(n, 1)->n_col_offset, c->c_arena); } else { REQ(CHILD(n, 0), LSQB); @@ -1742,8 +1742,6 @@ tmp = ast_for_trailer(c, ch, e); if (!tmp) return NULL; - tmp->lineno = e->lineno; - tmp->col_offset = e->col_offset; e = tmp; } if (TYPE(CHILD(n, NCH(n) - 1)) == factor) { This gives 0, 4, 8 as column offsets for foo.bar.baz and 0, 6, 14 for foo. bar. baz, i.e. points to the beginning of the attribute identifier. What do you think? Bye, Sven 2010/12/27 Nick Coghlan : > On Mon, Dec 27, 2010 at 9:34 AM, Benjamin Peterson wrote: >> 2010/12/26 Sven Brauch : >>> In that discussion, there's been different opinions about which >>> behaviour is better; main arguments were "consistency" for the current >>> and "usefulness" for the suggested behaviour. It has been proposed to >>> ask the question on this list, that's why I'm doing that now. :) >> >> My argument against this change is that an attribute includes the >> expression from which the attribute is being taken. This is consistent >> with subscripts and calls, which both have the lineno and col_offset >> of their source expressions. > > I'd like to see the impact on existing uses of this information > (primarily SyntaxErrors) before forming a firm opinion, but my initial > inclination is that retaining the column information for the > subattribute names is a better option. > > Cheers, > Nick. > > -- > Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia > From g.brandl at gmx.net Mon Dec 27 22:22:05 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 27 Dec 2010 22:22:05 +0100 Subject: [Python-Dev] r87518 - python/branches/py3k/Parser/tokenizer.c In-Reply-To: <20101227201213.52E12EE989@mail.python.org> References: <20101227201213.52E12EE989@mail.python.org> Message-ID: Am 27.12.2010 21:12, schrieb victor.stinner: > Author: victor.stinner > Date: Mon Dec 27 21:12:13 2010 > New Revision: 87518 > > Log: > Issue #10778: decoding_fgets() decodes the filename from the filesystem > encoding instead of UTF-8. > > > Modified: > python/branches/py3k/Parser/tokenizer.c > > Modified: python/branches/py3k/Parser/tokenizer.c > ============================================================================== > --- python/branches/py3k/Parser/tokenizer.c (original) > +++ python/branches/py3k/Parser/tokenizer.c Mon Dec 27 21:12:13 2010 > @@ -545,6 +545,7 @@ > { > char *line = NULL; > int badchar = 0; > + PyObject *filename; > for (;;) { > if (tok->decoding_state == STATE_NORMAL) { > /* We already have a codec associated with > @@ -585,12 +586,16 @@ > if (badchar) { > /* Need to add 1 to the line number, since this line > has not been counted, yet. */ > - PyErr_Format(PyExc_SyntaxError, > - "Non-UTF-8 code starting with '\\x%.2x' " > - "in file %.200s on line %i, " > - "but no encoding declared; " > - "see http://python.org/dev/peps/pep-0263/ for details", > - badchar, tok->filename, tok->lineno + 1); > + filename = PyUnicode_DecodeFSDefault(tok->filename); > + if (filename != NULL) { > + PyErr_Format(PyExc_SyntaxError, > + "Non-UTF-8 code starting with '\\x%.2x' " > + "in file %.200U on line %i, " > + "but no encoding declared; " > + "see http://python.org/dev/peps/pep-0263/ for details", > + badchar, filename, tok->lineno + 1); > + Py_DECREF(filename); > + } Hmm, and in case decoding fails, we return a Unicode error (without context) instead of a syntax error? Doesn't seem like a good trade-off when the file name is just displayed in a message. Georg From victor.stinner at haypocalc.com Tue Dec 28 01:06:06 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 28 Dec 2010 01:06:06 +0100 Subject: [Python-Dev] r87518 - python/branches/py3k/Parser/tokenizer.c In-Reply-To: References: <20101227201213.52E12EE989@mail.python.org> Message-ID: <1293494766.31514.12.camel@marge> Le lundi 27 d?cembre 2010 ? 22:22 +0100, Georg Brandl a ?crit : > Am 27.12.2010 21:12, schrieb victor.stinner: > > Author: victor.stinner > > Date: Mon Dec 27 21:12:13 2010 > > New Revision: 87518 > > > > Log: > > Issue #10778: decoding_fgets() decodes the filename from the filesystem > > encoding instead of UTF-8. > > > Hmm, and in case decoding fails, we return a Unicode error (without context) > instead of a syntax error? Yes, but it is very unlikely. I don't see in which case the decoder can fail. But a memory error can occur. > Doesn't seem like a good trade-off when the file name is just displayed in a message. What do you suggest? -- From victor.stinner at haypocalc.com Tue Dec 28 01:07:44 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 28 Dec 2010 01:07:44 +0100 Subject: [Python-Dev] r87518 - python/branches/py3k/Parser/tokenizer.c In-Reply-To: References: <20101227201213.52E12EE989@mail.python.org> Message-ID: <1293494864.31514.14.camel@marge> (oops, I posted an incomplete message, stupid mailer) Le lundi 27 d?cembre 2010 ? 22:22 +0100, Georg Brandl a ?crit : > Am 27.12.2010 21:12, schrieb victor.stinner: > > Author: victor.stinner > > Date: Mon Dec 27 21:12:13 2010 > > New Revision: 87518 > > > > Log: > > Issue #10778: decoding_fgets() decodes the filename from the filesystem > > encoding instead of UTF-8. > > > Hmm, and in case decoding fails, we return a Unicode error (without context) > instead of a syntax error? Yes, but it is very unlikely. I don't see in which case the decoder can fail. But a memory error can occur. > Doesn't seem like a good trade-off when the file name is just displayed in a message. What do you suggest? -- Prepare the decoded filename in PyParser_ParseStringFlagsFilenameEx() and PyParser_ParseFileFlagsEx() avoids this issue. Victor From g.rodola at gmail.com Tue Dec 28 03:04:31 2010 From: g.rodola at gmail.com (=?ISO-8859-1?Q?Giampaolo_Rodol=E0?=) Date: Tue, 28 Dec 2010 03:04:31 +0100 Subject: [Python-Dev] os.getpriority() and os.setpriority() Message-ID: Hi all, after having implemented something very similar in psutil: http://code.google.com/p/psutil/issues/detail?id=142 ...I decided to contribute a patch for exposing getpriority() and setpriority() system calls in the os module: http://bugs.python.org/issue10784 This was also raised some time ago: http://mail.python.org/pipermail/python-ideas/2009-April/004028.html ...and it seems feedbacks were positive. I think it would be nice to have Windows equivalents (GetPriorityClass / SetPriorityClass) as well but I'm not sure how since their notation is different. Regards, --- Giampaolo http://code.google.com/p/pyftpdlib http://code.google.com/p/psutil From rdmurray at bitdance.com Tue Dec 28 05:07:40 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 27 Dec 2010 23:07:40 -0500 Subject: [Python-Dev] [Python-checkins] r87504 - in python/branches/py3k: Doc/c-api/exceptions.rst Include/pyerrors.h Include/warnings.h In-Reply-To: <20101227014927.51BF5EE9A9@mail.python.org> References: <20101227014927.51BF5EE9A9@mail.python.org> Message-ID: <20101228040740.E1AE0228FC8@kimball.webabinitio.net> On Mon, 27 Dec 2010 02:49:27 +0100, victor.stinner wrote: > Author: victor.stinner > Date: Mon Dec 27 02:49:26 2010 > New Revision: 87504 > > Log: > Issue #9738: Document encodings of error and warning functions > > Modified: > python/branches/py3k/Doc/c-api/exceptions.rst > python/branches/py3k/Include/pyerrors.h > python/branches/py3k/Include/warnings.h > > Modified: python/branches/py3k/Doc/c-api/exceptions.rst > ============================================================================== > --- python/branches/py3k/Doc/c-api/exceptions.rst (original) > +++ python/branches/py3k/Doc/c-api/exceptions.rst Mon Dec 27 02:49:26 2010 > @@ -148,7 +148,8 @@ > This function sets the error indicator and returns *NULL*. *exception* > should be a Python exception class. The *format* and subsequent > parameters help format the error message; they have the same meaning and > - values as in :c:func:`PyUnicode_FromFormat`. > + values as in :c:func:`PyUnicode_FromFormat`. *format* is an ASCII-encoding > + string. ASCII-encoded string, or ASCII encoding-string? > .. c:function:: int PyErr_WarnFormat(PyObject *category, Py_ssize_t stack_level, const char *format, ...) > > Function similar to :c:func:`PyErr_WarnEx`, but use > - :c:func:`PyUnicode_FromFormat` to format the warning message. > + :c:func:`PyUnicode_FromFormat` to format the warning message. *format* is > + an ASCII-encoded string. So, the former, I'd guess :) -- R. David Murray www.bitdance.com From rdmurray at bitdance.com Tue Dec 28 05:13:40 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Mon, 27 Dec 2010 23:13:40 -0500 Subject: [Python-Dev] [Python-checkins] r87505 - in python/branches/py3k: Doc/c-api/unicode.rst Include/unicodeobject.h In-Reply-To: <20101227014929.D926EEEA85@mail.python.org> References: <20101227014929.D926EEEA85@mail.python.org> Message-ID: <20101228041340.CDF93200C26@kimball.webabinitio.net> On Mon, 27 Dec 2010 02:49:29 +0100, victor.stinner wrote: > Author: victor.stinner > Date: Mon Dec 27 02:49:29 2010 > New Revision: 87505 > > Log: > Issue #9738: document encodings of unicode functions > > Modified: > python/branches/py3k/Doc/c-api/unicode.rst > python/branches/py3k/Include/unicodeobject.h > > Modified: python/branches/py3k/Doc/c-api/unicode.rst > ============================================================================== > --- python/branches/py3k/Doc/c-api/unicode.rst (original) > +++ python/branches/py3k/Doc/c-api/unicode.rst Mon Dec 27 02:49:29 2010 > @@ -1063,7 +1063,8 @@ > .. c:function:: int PyUnicode_CompareWithASCIIString(PyObject *uni, char *string) > > Compare a unicode object, *uni*, with *string* and return -1, 0, 1 for less > - than, equal, and greater than, respectively. > + than, equal, and greater than, respectively. *string* is an ASCII-encoded > + string (it is interpreted as ISO-8859-1). Does it mean anything to say that an ASCII string is interpreted as ISO-8859-1? If it is ASCII-encoded it shouldn't have any bytes with the 8th bit set, leaving no room for interpretation. So presumably you mean it is (treated as) an ISO-8859-1 encoded string, despite the function name? -- R. David Murray www.bitdance.com From ncoghlan at gmail.com Tue Dec 28 08:53:20 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Dec 2010 17:53:20 +1000 Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs In-Reply-To: References: Message-ID: On Tue, Dec 28, 2010 at 1:29 AM, Vinay Sajip wrote: > ? ? However, it would be good to have some consistency of naming stdlib loggers > - perhaps using __name__ as is recommended practice for library and application > developers, but with a prefix such as "py." to indicate that it's a part of the > stdlib. This would make the disabling logic change easier as it could just check > the "py." prefix rather than have an internal list of logger names, which would > need more maintenance and be more error-prone. Unfortunately, the "py" package already claimed that namespace, so it isn't really free for us to use in the standard library (even the current "py.warnings" for redirected warnings may be misleading, as it may give users the impression that package is involved somewhere along the line). It is probably best just to go with the "__name__" convention and not worry about being able to draw a clean distinction between "standard library" and "third party" (that distinction doesn't exist in the module heirarchy, so it isn't really reasonable to expect it to exist in the logging heirarchy). However, rather than a manually maintained list of low level loggers, it may be feasible to just have a flag we can set on loggers that makes them immune to the default implicit disabling. Then the config calls can support three levels of logger disabling: - leave all existing loggers enabled (existing option) - leave only flagged loggers enabled (new default behaviour) - disable all loggers not mentioned explicitly (new option) Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From vinay_sajip at yahoo.co.uk Tue Dec 28 09:19:45 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 28 Dec 2010 08:19:45 +0000 (UTC) Subject: [Python-Dev] =?utf-8?q?Loggers_in_the_stdlib_and_logging_configur?= =?utf-8?q?ation=09APIs?= References: Message-ID: Nick Coghlan gmail.com> writes: > Unfortunately, the "py" package already claimed that namespace, so it > isn't really free for us to use in the standard library (even the > current "py.warnings" for redirected warnings may be misleading, as it > may give users the impression that package is involved somewhere along Gosh, I had no idea the py package even existed. > the line). It is probably best just to go with the "__name__" > convention and not worry about being able to draw a clean distinction > between "standard library" and "third party" (that distinction doesn't > exist in the module heirarchy, so it isn't really reasonable to expect > it to exist in the logging heirarchy). True. > However, rather than a manually maintained list of low level loggers, > it may be feasible to just have a flag we can set on loggers that > makes them immune to the default implicit disabling. Then the config Yes, I thought about this option after I posted, using "leave_enabled" as the attribute name, as well as possibly an API to register loggers for being skipped by the disable logic. A logger flag is definitely easier all round. > calls can support three levels of logger disabling: > - leave all existing loggers enabled (existing option) I think you mean disabled - that's the current behaviour. > - leave only flagged loggers enabled (new default behaviour) > - disable all loggers not mentioned explicitly (new option) So far, this seems the best approach. Thanks, Vinay Sajip From v+python at g.nevcal.com Tue Dec 28 09:25:36 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 28 Dec 2010 00:25:36 -0800 Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs In-Reply-To: References: Message-ID: <4D199F00.4070503@g.nevcal.com> On 12/27/2010 11:53 PM, Nick Coghlan wrote: > However, rather than a manually maintained list of low level loggers, Yes, a manually maintained list would be bad. > it may be feasible to just have a flag we can set on loggers that > makes them immune to the default implicit disabling. Then the config > calls can support three levels of logger disabling: > - leave all existing loggers enabled (existing option) > - leave only flagged loggers enabled (new default behaviour) > - disable all loggers not mentioned explicitly (new option) I thought of the idea of a flag to make loggers immune, but it seemed it could lead to an arms race, where the knee-jerk reaction of discovering that a library's logger got disabled would be to set the flag, and, of course, the knee-jerk reaction of discovering that a library's logger produced an unwanted message would cause the application to use the new option, to "turn them all off again". Then a new library logger option, to become immune to the new disable them all option, would be requested. Etc. That's why I suggested a list, although perhaps your scheme would be "more" backward compatible, as it defines a meaning for the default behavior that sounds somewhat similar to the current default behavior.... but isn't, really. Interesting that you choose to disable all loggers not mentioned explicitly (a logger white list), whereas I suggested listing the ones to disable (a logger black list). I suppose you suggested the white list, figuring that the application could choose which messages it wishes to see. I chose a black list, figuring that new messages being introduced might be useful to the application, but if they never see them (because they are not in the white list), they wouldn't know about them. Of course, either scheme can be programmed around, given an obtainable list of defined loggers.... but, thinking about it more, maybe that list cannot be obtainable, at xxxConfig time, because not all the modules that have loggers have been loaded yet... so maybe the question is not just what to do with pre-existing loggers at xxxConfig time, but also what to do with loggers created in the future that are not configured explicitly? I can see where a white list would produce a more predictable, consistent set of known messages, and a black list would allow introduction of new messages (if the lists are both manually maintained, vs. programmatically adjusted to a specific white or black list), which could be disruptive to some applications... but perhaps shouldn't be. But a new version of a module on the logger white list could introduce new messages, so that is not much different than a new version of a module not on the logger black list introducing new messages. Applications that use logging, really should be prepared for new messages to be displayed. Testing won't discover them all, disabling them could result in useful messages being suppressed. Should a module's documentation be expected to include lists of log messages that it may produce at any level of logging? It really seems that applications that use logging should be prepared to discover messages in the log that they don't produce, and that an explicit configuration of the appropriate level should be done in the xxxConfig call for each logger.... and if they are not mentioned in the xxxConfig call, how much logging should they be allowed to do? Should their existence, and lack of configuration be logged? -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Tue Dec 28 09:28:18 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 28 Dec 2010 00:28:18 -0800 Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs In-Reply-To: References: Message-ID: <4D199FA2.2020504@g.nevcal.com> On 12/28/2010 12:19 AM, Vinay Sajip wrote: >> > calls can support three levels of logger disabling: >> > - leave all existing loggers enabled (existing option) > I think you mean disabled - that's the current behaviour. > >> > - leave only flagged loggers enabled (new default behaviour) >> > - disable all loggers not mentioned explicitly (new option) My words in his mouth, but I think he meant that the existing _default_ behavior should be changed, and the existing option (listed first), which is not default behaviour should be kept, and a third option to sort of achieve the current default behavior, but make it harder. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue Dec 28 09:53:12 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Dec 2010 18:53:12 +1000 Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs In-Reply-To: <4D199FA2.2020504@g.nevcal.com> References: <4D199FA2.2020504@g.nevcal.com> Message-ID: On Tue, Dec 28, 2010 at 6:28 PM, Glenn Linderman wrote: > On 12/28/2010 12:19 AM, Vinay Sajip wrote: > >> calls can support three levels of logger disabling: >> - leave all existing loggers enabled (existing option) > > I think you mean disabled - that's the current behaviour. > >> - leave only flagged loggers enabled (new default behaviour) >> - disable all loggers not mentioned explicitly (new option) > > My words in his mouth, but I think he meant that the existing _default_ > behavior should be changed, and the existing option (listed first), which is > not default behaviour should be kept, and a third option to sort of achieve > the current default behavior, but make it harder. It isn't really about making it harder - it is about changing the default behaviour in a backwards compatible way. As I see it, library level loggers (including those in the standard library) should be left enabled unless explicitly turned off, while application level loggers should be left disabled unless explicitly turned on. However, there is currently no way to indicate the default state for a given logger that isn't mentioned explicitly in a detailed config - they're all implicitly disabled (or enabled, if that option is passed in as part of the config). Changing to a 3-tier system lets libraries use logging for error messages without them being lost by the default configuration process. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From ncoghlan at gmail.com Tue Dec 28 09:55:42 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 28 Dec 2010 18:55:42 +1000 Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs In-Reply-To: References: <4D199FA2.2020504@g.nevcal.com> Message-ID: On Tue, Dec 28, 2010 at 6:53 PM, Nick Coghlan wrote: > On Tue, Dec 28, 2010 at 6:28 PM, Glenn Linderman wrote: >> On 12/28/2010 12:19 AM, Vinay Sajip wrote: >> >>> calls can support three levels of logger disabling: >>> - leave all existing loggers enabled (existing option) >> >> I think you mean disabled - that's the current behaviour. >> >>> - leave only flagged loggers enabled (new default behaviour) >>> - disable all loggers not mentioned explicitly (new option) >> >> My words in his mouth, but I think he meant that the existing _default_ >> behavior should be changed, and the existing option (listed first), which is >> not default behaviour should be kept, and a third option to sort of achieve >> the current default behavior, but make it harder. > > It isn't really about making it harder - it is about changing the > default behaviour in a backwards compatible way. > > As I see it, library level loggers (including those in the standard > library) should be left enabled unless explicitly turned off, while > application level loggers should be left disabled unless explicitly > turned on. To put a bit more nuance on that view: if the library uses the logger to report errors, it should *definitely* flag it to be implicitly enabled. If it is only used for info messages, then preferences will vary according to the library developer. The beauty of the logging system is that application authors can get at the logging information regardless of which default the library author chooses. Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From g.brandl at gmx.net Tue Dec 28 10:12:46 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 28 Dec 2010 10:12:46 +0100 Subject: [Python-Dev] r87518 - python/branches/py3k/Parser/tokenizer.c In-Reply-To: <1293494864.31514.14.camel@marge> References: <20101227201213.52E12EE989@mail.python.org> <1293494864.31514.14.camel@marge> Message-ID: Am 28.12.2010 01:07, schrieb Victor Stinner: > (oops, I posted an incomplete message, stupid mailer) > > Le lundi 27 d?cembre 2010 ? 22:22 +0100, Georg Brandl a ?crit : >> Am 27.12.2010 21:12, schrieb victor.stinner: >> > Author: victor.stinner >> > Date: Mon Dec 27 21:12:13 2010 >> > New Revision: 87518 >> > >> > Log: >> > Issue #10778: decoding_fgets() decodes the filename from the filesystem >> > encoding instead of UTF-8. >> > > >> Hmm, and in case decoding fails, we return a Unicode error (without context) >> instead of a syntax error? > > Yes, but it is very unlikely. I don't see in which case the decoder can > fail. But a memory error can occur. > >> Doesn't seem like a good trade-off when the file name is just displayed in a message. > > What do you suggest? If the probability is so low, it's probably not worth changing. I'm just somewhat sensitive to changes that enforce correctness by taking away useful information from the user. Georg From victor.stinner at haypocalc.com Tue Dec 28 10:28:51 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 28 Dec 2010 10:28:51 +0100 Subject: [Python-Dev] [Python-checkins] r87505 - in python/branches/py3k: Doc/c-api/unicode.rst Include/unicodeobject.h In-Reply-To: <20101228041340.CDF93200C26@kimball.webabinitio.net> References: <20101227014929.D926EEEA85@mail.python.org> <20101228041340.CDF93200C26@kimball.webabinitio.net> Message-ID: <1293528531.24502.10.camel@marge> Le lundi 27 d?cembre 2010 ? 23:13 -0500, R. David Murray a ?crit : > > Modified: python/branches/py3k/Doc/c-api/unicode.rst > > ============================================================================== > > --- python/branches/py3k/Doc/c-api/unicode.rst (original) > > +++ python/branches/py3k/Doc/c-api/unicode.rst Mon Dec 27 02:49:29 2010 > > @@ -1063,7 +1063,8 @@ > > .. c:function:: int PyUnicode_CompareWithASCIIString(PyObject *uni, char *string) > > > > Compare a unicode object, *uni*, with *string* and return -1, 0, 1 for less > > - than, equal, and greater than, respectively. > > + than, equal, and greater than, respectively. *string* is an ASCII-encoded > > + string (it is interpreted as ISO-8859-1). > > Does it mean anything to say that an ASCII string is interpreted as > ISO-8859-1? If it is ASCII-encoded it shouldn't have any bytes with > the 8th bit set, leaving no room for interpretation. So presumably > you mean it is (treated as) an ISO-8859-1 encoded string, despite the > function name? Oh. Someone noticed :-) I would like to say that it is better to pass only ASCII-encoded string, but the function supports ISO-8859-1. Would it be more clear to say that the function expects ISO-8859-1 encoded string? But I don't want to patch the function. Victor From victor.stinner at haypocalc.com Tue Dec 28 11:28:19 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 28 Dec 2010 11:28:19 +0100 Subject: [Python-Dev] [Python-checkins] r87504 - in python/branches/py3k: Doc/c-api/exceptions.rst Include/pyerrors.h Include/warnings.h In-Reply-To: <20101228040740.E1AE0228FC8@kimball.webabinitio.net> References: <20101227014927.51BF5EE9A9@mail.python.org> <20101228040740.E1AE0228FC8@kimball.webabinitio.net> Message-ID: <1293532099.24502.70.camel@marge> Le lundi 27 d?cembre 2010 ? 23:07 -0500, R. David Murray a ?crit : > On Mon, 27 Dec 2010 02:49:27 +0100, victor.stinner wrote: > > Author: victor.stinner > > Date: Mon Dec 27 02:49:26 2010 > > New Revision: 87504 > > > > Log: > > Issue #9738: Document encodings of error and warning functions > ASCII-encoded string, or ASCII encoding-string? Oh, I don't really care. Which one do you prefer? I used "ASCII-encoded string" and "UTF-8 encoded string", because "UTF-8-encoded" has too many "-". I should maybe be uniformized. Victor From martin at v.loewis.de Tue Dec 28 11:40:01 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 28 Dec 2010 11:40:01 +0100 Subject: [Python-Dev] [Python-checkins] r87504 - in python/branches/py3k: Doc/c-api/exceptions.rst Include/pyerrors.h Include/warnings.h In-Reply-To: <1293532099.24502.70.camel@marge> References: <20101227014927.51BF5EE9A9@mail.python.org> <20101228040740.E1AE0228FC8@kimball.webabinitio.net> <1293532099.24502.70.camel@marge> Message-ID: <4D19BE81.3070806@v.loewis.de> Am 28.12.2010 11:28, schrieb Victor Stinner: > Le lundi 27 d?cembre 2010 ? 23:07 -0500, R. David Murray a ?crit : >> On Mon, 27 Dec 2010 02:49:27 +0100, victor.stinner wrote: >>> Author: victor.stinner >>> Date: Mon Dec 27 02:49:26 2010 >>> New Revision: 87504 >>> >>> Log: >>> Issue #9738: Document encodings of error and warning functions > >> ASCII-encoded string, or ASCII encoding-string? > > Oh, I don't really care. Which one do you prefer? I used "ASCII-encoded > string" and "UTF-8 encoded string", because "UTF-8-encoded" has too many > "-". I should maybe be uniformized. David's other concern was whether it should be "encod*ed*" or "encod*ing*". It seems he would prefer "encoded". Not sure what he would prefer wrt. hyphens (but apparently, you would put the hyphen after if it's "encoding", and before if it's "encoded"; not sure why that is). Regards, Martin From victor.stinner at haypocalc.com Tue Dec 28 12:01:43 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 28 Dec 2010 12:01:43 +0100 Subject: [Python-Dev] r87518 - python/branches/py3k/Parser/tokenizer.c In-Reply-To: References: <20101227201213.52E12EE989@mail.python.org> <1293494864.31514.14.camel@marge> Message-ID: <1293534103.24502.88.camel@marge> Le mardi 28 d?cembre 2010 ? 10:12 +0100, Georg Brandl a ?crit : > >> > Author: victor.stinner > >> > Date: Mon Dec 27 21:12:13 2010 > >> > New Revision: 87518 > >> > > >> > Log: > >> > Issue #10778: decoding_fgets() decodes the filename from the filesystem > >> > encoding instead of UTF-8. > > > >> Hmm, and in case decoding fails, we return a Unicode error (without context) > >> instead of a syntax error? I created an issue for this problem, but also to prepare the full support of unicode in the import machinery. I patched the import machinery of Python 3 to support undecodable bytes, but Python 3 doesn't support unencodable characters on Windows (see issue #3080). http://bugs.python.org/issue10785 Victor From stefan_ml at behnel.de Tue Dec 28 12:20:36 2010 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 28 Dec 2010 12:20:36 +0100 Subject: [Python-Dev] [Python-checkins] r87504 - in python/branches/py3k: Doc/c-api/exceptions.rst Include/pyerrors.h Include/warnings.h In-Reply-To: <4D19BE81.3070806@v.loewis.de> References: <20101227014927.51BF5EE9A9@mail.python.org> <20101228040740.E1AE0228FC8@kimball.webabinitio.net> <1293532099.24502.70.camel@marge> <4D19BE81.3070806@v.loewis.de> Message-ID: "Martin v. L?wis", 28.12.2010 11:40: >> Le lundi 27 d?cembre 2010 ? 23:07 -0500, R. David Murray a ?crit : >>> ASCII-encoded string, or ASCII encoding-string? >> > [...] apparently, you would put > the hyphen after if it's "encoding", and before if it's "encoded"; > not sure why that is I think David just put it there to point to the semantic difference. It's either a string that is "ASCII encoded", or an "encoding string" that refers to ASCII in some way. Whether you put a hyphen or not is a secondary concern. Stefan From vinay_sajip at yahoo.co.uk Tue Dec 28 13:16:09 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 28 Dec 2010 12:16:09 +0000 (UTC) Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs References: <4D199F00.4070503@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > I thought of the idea of a flag to make loggers immune, but it > seemed it could lead to an arms race, where the knee-jerk reaction > of discovering that a library's logger got disabled would be to set > the flag, and, of course, the knee-jerk reaction of discovering that > a library's logger produced an unwanted message would cause the > application to use the new option, to "turn them all off again". > Then a new library logger option, to become immune to the new > disable them all option, would be requested. Etc. Although an arms race is theoretically possible, I doubt it would come to that. After all, people can just set a logger's disabled flag to False, and its messages would start flowing again. That would be as easy for users as making any new API call. > That's why I suggested a list, although perhaps your scheme would be > "more" backward compatible, as it defines a meaning for the default > behavior that sounds somewhat similar to the current default > behavior.... but isn't, really. > Interesting that you choose to disable all loggers not mentioned > explicitly (a logger white list), whereas I suggested listing the > ones to disable (a logger black list). That's the backward compatible behaviour. In general, when the configuration APIs are called, the design assumption was that you want to replace any existing configuration with a new one; that now needs to change because of places where logging will be used for notifications where exceptions cannot be usefully raised. > I suppose you suggested the white list, figuring that the > application could choose which messages it wishes to see. An application can always choose not to see or not see messages via setting verbosity levels on specific loggers and/or handlers to the desired values. > I chose a black list, figuring that new messages being introduced > might be useful to the application, but if they never see them > (because they are not in the white list), they wouldn't know about > them. Of course, either scheme can be programmed around, given an Don't forget that any loggers created after a fileConfig/dictConfig call wouldn't be disabled; and for many many uses, logging will be set up reasonably early in the application's execution, and the only loggers which would be disabled by a configuration call would be loggers created before the call, and which are not explicitly named in the configuration. There will be relatively few of those. > messages. Applications that use logging, really should be prepared > for new messages to be displayed. Testing won't discover them all, > disabling them could result in useful messages being suppressed. Yes, because third party libraries can introduce new messages either by themseleves or through adding a new dependency, but disabling via a config call is unlikely if the config call is made early. > Should a module's documentation be expected to include lists of log > messages that it may produce at any level of logging? That would never be kept up to date :-) The best we can hope for is for module authors to document that they use logging, mention any logger names used, the default level settings for at least the top-level logger, and any convenience APIs provided for configuring handlers. Suppose we just implement a "leave_enabled" flag on loggers and change the config code to honour this. Then, there will be no backward incompatible change until we modify stdlib loggers to set leave_enabled to True (I'm not sure if this could be done for concurrent.futures now as it's a new package in 3.2; it depends on whether this would be regarded as a new feature). I don't propose adding another optional argument to the configuration APIs unless and until there seems to be a need for it; in the absence of this argument, users would have to explicitly turn off loggers they wanted to silence in the normal way (i.e. by setting levels, and perhaps propagation flags). I would propose to document "leave_enabled" as part of the public API, but with a recommendation that due care and consideration be given before choosing to set it for a logger. How does that sound? Regards, Vinay Sajip From victor.stinner at haypocalc.com Tue Dec 28 13:58:24 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Tue, 28 Dec 2010 13:58:24 +0100 Subject: [Python-Dev] [Python-checkins] r87504 - in python/branches/py3k: Doc/c-api/exceptions.rst Include/pyerrors.h Include/warnings.h In-Reply-To: <4D19BE81.3070806@v.loewis.de> References: <20101227014927.51BF5EE9A9@mail.python.org> <20101228040740.E1AE0228FC8@kimball.webabinitio.net> <1293532099.24502.70.camel@marge> <4D19BE81.3070806@v.loewis.de> Message-ID: <1293541104.24502.89.camel@marge> Le mardi 28 d?cembre 2010 ? 11:40 +0100, "Martin v. L?wis" a ?crit : > Am 28.12.2010 11:28, schrieb Victor Stinner: > > Le lundi 27 d?cembre 2010 ? 23:07 -0500, R. David Murray a ?crit : > >> On Mon, 27 Dec 2010 02:49:27 +0100, victor.stinner wrote: > >>> Author: victor.stinner > >>> Date: Mon Dec 27 02:49:26 2010 > >>> New Revision: 87504 > >>> > >>> Log: > >>> Issue #9738: Document encodings of error and warning functions > > > >> ASCII-encoded string, or ASCII encoding-string? > > > > Oh, I don't really care. Which one do you prefer? I used "ASCII-encoded > > string" and "UTF-8 encoded string", because "UTF-8-encoded" has too many > > "-". I should maybe be uniformized. > > David's other concern was whether it should be "encod*ed*" or > "encod*ing*". Ooops, it's ASCII-encoded string (and not ASCII-encoding string): fixed by r87529. Victor From rdmurray at bitdance.com Tue Dec 28 18:10:12 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 28 Dec 2010 12:10:12 -0500 Subject: [Python-Dev] [Python-checkins] r87504 - in python/branches/py3k: Doc/c-api/exceptions.rst Include/pyerrors.h Include/warnings.h In-Reply-To: <1293541104.24502.89.camel@marge> References: <20101227014927.51BF5EE9A9@mail.python.org> <20101228040740.E1AE0228FC8@kimball.webabinitio.net> <1293532099.24502.70.camel@marge> <4D19BE81.3070806@v.loewis.de> <1293541104.24502.89.camel@marge> Message-ID: <20101228171012.4A2312361E2@kimball.webabinitio.net> On Tue, 28 Dec 2010 13:58:24 +0100, Victor Stinner wrote: > Le mardi 28 d??cembre 2010 ?? 11:40 +0100, "Martin v. L??wis" a ??crit : > > Am 28.12.2010 11:28, schrieb Victor Stinner: > > > Le lundi 27 d??cembre 2010 ?? 23:07 -0500, R. David Murray a ??crit : > > >> On Mon, 27 Dec 2010 02:49:27 +0100, victor.stinner wrote: > > >>> Author: victor.stinner > > >>> Date: Mon Dec 27 02:49:26 2010 > > >>> New Revision: 87504 > > >>> > > >>> Log: > > >>> Issue #9738: Document encodings of error and warning functions > > > > > >> ASCII-encoded string, or ASCII encoding-string? > > > > > > Oh, I don't really care. Which one do you prefer? I used "ASCII-encoded > > > string" and "UTF-8 encoded string", because "UTF-8-encoded" has too many > > > "-". I should maybe be uniformized. > > > > David's other concern was whether it should be "encod*ed*" or > > "encod*ing*". > > Ooops, it's ASCII-encoded string (and not ASCII-encoding string): fixed > by r87529. Yes, exactly. Sorry for being a bit too subtle in my error report. -- R. David Murray www.bitdance.com From rdmurray at bitdance.com Tue Dec 28 18:14:52 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 28 Dec 2010 12:14:52 -0500 Subject: [Python-Dev] [Python-checkins] r87505 - in python/branches/py3k: Doc/c-api/unicode.rst Include/unicodeobject.h In-Reply-To: <1293528531.24502.10.camel@marge> References: <20101227014929.D926EEEA85@mail.python.org> <20101228041340.CDF93200C26@kimball.webabinitio.net> <1293528531.24502.10.camel@marge> Message-ID: <20101228171452.C3FA72361E2@kimball.webabinitio.net> On Tue, 28 Dec 2010 10:28:51 +0100, Victor Stinner wrote: > Le lundi 27 d??cembre 2010 ?? 23:13 -0500, R. David Murray a ??crit : > > > Modified: python/branches/py3k/Doc/c-api/unicode.rst > > > ============================================================================== > > > --- python/branches/py3k/Doc/c-api/unicode.rst (original) > > > +++ python/branches/py3k/Doc/c-api/unicode.rst Mon Dec 27 02:49:29 2010 > > > @@ -1063,7 +1063,8 @@ > > > .. c:function:: int PyUnicode_CompareWithASCIIString(PyObject *uni, char *string) > > > > > > Compare a unicode object, *uni*, with *string* and return -1, 0, 1 for less > > > - than, equal, and greater than, respectively. > > > + than, equal, and greater than, respectively. *string* is an ASCII-encoded > > > + string (it is interpreted as ISO-8859-1). > > > > Does it mean anything to say that an ASCII string is interpreted as > > ISO-8859-1? If it is ASCII-encoded it shouldn't have any bytes with > > the 8th bit set, leaving no room for interpretation. So presumably > > you mean it is (treated as) an ISO-8859-1 encoded string, despite the > > function name? > > Oh. Someone noticed :-) I would like to say that it is better to pass > only ASCII-encoded string, but the function supports ISO-8859-1. > > Would it be more clear to say that the function expects ISO-8859-1 > encoded string? > > But I don't want to patch the function. I think your first paragraph is what you should put in the docs: "it is best to pass only ASCII-encoded strings, but the function interprets the input string as ISO-8859-1 if it contains non-ASCII characters". A bit harder to compress that into an in-line comment in the code... -- R. David Murray www.bitdance.com From v+python at g.nevcal.com Tue Dec 28 18:14:41 2010 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 28 Dec 2010 09:14:41 -0800 Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs In-Reply-To: References: <4D199F00.4070503@g.nevcal.com> Message-ID: <4D1A1B01.7080508@g.nevcal.com> On 12/28/2010 4:16 AM, Vinay Sajip wrote: > How does that sound? Sounds pretty rational, overall. If the leave_enabled flag can be turned on/off by the application, then I agree the arms race is unlikely. I didn't dig through the logging docs to discover if there is an API that returns a list of currently known loggers.... such that an application could easily discover the current set. It would be nice to have such a thing, in any case, but especially in the presence of the leave_enabled flag. I also wasn't sure whether or not the "default disable" applied to future loggers or only preexisting loggers, but you clarified that it only applies to preexisting ones, so such an API as above would suffice to discover any that might need special handling at xxxConfig time. I hope somewhere in the parts of the documentation I haven't read yet, that there is some verbiage that discusses the fact that loggers created after xxxConfig time and not configured by it, are enabled by default? And the concept that applications that use logging should be prepared to handle messages that they haven't seen before in some manner. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukas.lueg at googlemail.com Tue Dec 28 18:08:58 2010 From: lukas.lueg at googlemail.com (Lukas Lueg) Date: Tue, 28 Dec 2010 18:08:58 +0100 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? Message-ID: Consider the following code: def foobar(x): for i in range(5): x[i] = i The bytecode in python 2.7 is the following: 2 0 SETUP_LOOP 30 (to 33) 3 LOAD_GLOBAL 0 (range) 6 LOAD_CONST 1 (5) 9 CALL_FUNCTION 1 12 GET_ITER >> 13 FOR_ITER 16 (to 32) 16 STORE_FAST 1 (i) 3 19 LOAD_FAST 1 (i) 22 LOAD_FAST 0 (x) 25 LOAD_FAST 1 (i) 28 STORE_SUBSCR 29 JUMP_ABSOLUTE 13 >> 32 POP_BLOCK >> 33 LOAD_CONST 0 (None) 36 RETURN_VALUE Can't we optimize the LOAD_FAST in lines 19 and 25 to a single load and put the reference twice on the stack? There is no way that the reference of i might change in between the two lines. Also, the load_fast in lne 22 to reference x could be taken out of the loop as x will always point to the same object.... From benjamin at python.org Tue Dec 28 18:24:36 2010 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 28 Dec 2010 11:24:36 -0600 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: References: Message-ID: 2010/12/28 Lukas Lueg : > Consider the following code: > > def foobar(x): > ? ?for i in range(5): > ? ? ? ?x[i] = i > > The bytecode in python 2.7 is the following: > > ?2 ? ? ? ? ? 0 SETUP_LOOP ? ? ? ? ? ? ?30 (to 33) > ? ? ? ? ? ? ?3 LOAD_GLOBAL ? ? ? ? ? ? ?0 (range) > ? ? ? ? ? ? ?6 LOAD_CONST ? ? ? ? ? ? ? 1 (5) > ? ? ? ? ? ? ?9 CALL_FUNCTION ? ? ? ? ? ?1 > ? ? ? ? ? ? 12 GET_ITER > ? ? ? ?>> ? 13 FOR_ITER ? ? ? ? ? ? ? ?16 (to 32) > ? ? ? ? ? ? 16 STORE_FAST ? ? ? ? ? ? ? 1 (i) > > ?3 ? ? ? ? ?19 LOAD_FAST ? ? ? ? ? ? ? ?1 (i) > ? ? ? ? ? ? 22 LOAD_FAST ? ? ? ? ? ? ? ?0 (x) > ? ? ? ? ? ? 25 LOAD_FAST ? ? ? ? ? ? ? ?1 (i) > ? ? ? ? ? ? 28 STORE_SUBSCR > ? ? ? ? ? ? 29 JUMP_ABSOLUTE ? ? ? ? ? 13 > ? ? ? ?>> ? 32 POP_BLOCK > ? ? ? ?>> ? 33 LOAD_CONST ? ? ? ? ? ? ? 0 (None) > ? ? ? ? ? ? 36 RETURN_VALUE > > Can't we optimize the LOAD_FAST in lines 19 and 25 to a single load > and put the reference twice on the stack? Yes. Would it be useful? Unlikely. -- Regards, Benjamin From vinay_sajip at yahoo.co.uk Tue Dec 28 19:28:07 2010 From: vinay_sajip at yahoo.co.uk (Vinay Sajip) Date: Tue, 28 Dec 2010 18:28:07 +0000 (UTC) Subject: [Python-Dev] Loggers in the stdlib and logging configuration APIs References: <4D199F00.4070503@g.nevcal.com> <4D1A1B01.7080508@g.nevcal.com> Message-ID: Glenn Linderman g.nevcal.com> writes: > I didn't dig through the logging docs to discover if there is an API > that returns a list of currently known loggers.... such that an > application could easily discover the current set.? It would be nice > to have such a thing, in any case, but especially in the presence of > the leave_enabled flag. There isn't one, though I'll see about providing it in 3.3. > I hope somewhere in the parts of the documentation I haven't read > yet, that there is some verbiage that discusses the fact that > loggers created after xxxConfig time and not configured by it, are Not quite; the fact that existing loggers are disabled by the configuration calls is documented, as that would be the thing that might be unexpected. That newly-created loggers would be enabled isn't documented specifically, as it is what one would expect. > enabled by default?? And the concept that applications that use > logging should be prepared to handle messages that they haven't seen > before in some manner. That concept is also not mentioned explicitly, because the rationale given for having a logging module in the stdlib is that it can be used by modules from different sources (standard library, 3rd party libs, application code) and provide logs in an integrated manner. So the expectation you describe would follow naturally. Thanks for the feedback, Vinay Sajip From g.brandl at gmx.net Tue Dec 28 19:30:48 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 28 Dec 2010 19:30:48 +0100 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: References: Message-ID: Am 28.12.2010 18:24, schrieb Benjamin Peterson: > 2010/12/28 Lukas Lueg : >> Consider the following code: >> >> def foobar(x): >> for i in range(5): >> x[i] = i >> >> The bytecode in python 2.7 is the following: >> >> 2 0 SETUP_LOOP 30 (to 33) >> 3 LOAD_GLOBAL 0 (range) >> 6 LOAD_CONST 1 (5) >> 9 CALL_FUNCTION 1 >> 12 GET_ITER >> >> 13 FOR_ITER 16 (to 32) >> 16 STORE_FAST 1 (i) >> >> 3 19 LOAD_FAST 1 (i) >> 22 LOAD_FAST 0 (x) >> 25 LOAD_FAST 1 (i) >> 28 STORE_SUBSCR >> 29 JUMP_ABSOLUTE 13 >> >> 32 POP_BLOCK >> >> 33 LOAD_CONST 0 (None) >> 36 RETURN_VALUE >> >> Can't we optimize the LOAD_FAST in lines 19 and 25 to a single load >> and put the reference twice on the stack? > > Yes. Would it be useful? Unlikely. Is it tricky to get all the corner cases right? Very probably :) Georg From martin at v.loewis.de Tue Dec 28 21:11:56 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 28 Dec 2010 21:11:56 +0100 Subject: [Python-Dev] os.getpriority() and os.setpriority() In-Reply-To: References: Message-ID: <4D1A448C.40102@v.loewis.de> > I think it would be nice to have Windows equivalents (GetPriorityClass > / SetPriorityClass) as well but I'm not sure how since their notation > is different. I don't mind exposing it (somewhere), however, it should keep its original name. In addition, you probably need OpenProcess, as well as Get/SetThreadPriority along with it. I think Python should incorporate something like pywin32, to expose all APIs whole-sale, in a type-safe manner (unlike ctypes). Regards, Martin From tjreedy at udel.edu Tue Dec 28 21:22:07 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Dec 2010 15:22:07 -0500 Subject: [Python-Dev] [Python-checkins] r87537 - in python/branches/py3k: Doc/library/struct.rst Doc/whatsnew/3.2.rst Lib/test/test_struct.py Lib/wave.py Misc/NEWS Modules/_struct.c In-Reply-To: <20101228132642.83E95EE9B6@mail.python.org> References: <20101228132642.83E95EE9B6@mail.python.org> Message-ID: <4D1A46EF.5030509@udel.edu> On 12/28/2010 8:26 AM, victor.stinner wrote: > Author: victor.stinner > Date: Tue Dec 28 14:26:42 2010 > New Revision: 87537 > > Log: > Issue #10783: struct.pack() doesn't encode implicitly unicode to UTF-8 > Modified: python/branches/py3k/Doc/whatsnew/3.2.rst > ============================================================================== > --- python/branches/py3k/Doc/whatsnew/3.2.rst (original) > +++ python/branches/py3k/Doc/whatsnew/3.2.rst Tue Dec 28 14:26:42 2010 > @@ -1705,3 +1705,7 @@ > > (Contributed by Georg Brandl and Mattias Br?ndstr?m; > `appspot issue 53094`_.) > + > +* :func:`struct.pack` doesn't encode implicitly unicode to UTF-8 anymore: That phrasing is awkward at best. I believe you mean "struct.pack no longer implicitly encodes unicode to UTF-8", which will be clearer to most people. Terry From victor.stinner at haypocalc.com Wed Dec 29 00:38:13 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Dec 2010 00:38:13 +0100 Subject: [Python-Dev] [Python-checkins] r87537 - in python/branches/py3k: Doc/library/struct.rst Doc/whatsnew/3.2.rst Lib/test/test_struct.py Lib/wave.py Misc/NEWS Modules/_struct.c In-Reply-To: <4D1A46EF.5030509@udel.edu> References: <20101228132642.83E95EE9B6@mail.python.org> <4D1A46EF.5030509@udel.edu> Message-ID: <1293579493.29958.1.camel@marge> Le mardi 28 d?cembre 2010 ? 15:22 -0500, Terry Reedy a ?crit : > On 12/28/2010 8:26 AM, victor.stinner wrote: > > Author: victor.stinner > > Date: Tue Dec 28 14:26:42 2010 > > New Revision: 87537 > > > > Log: > > Issue #10783: struct.pack() doesn't encode implicitly unicode to UTF-8 > > > Modified: python/branches/py3k/Doc/whatsnew/3.2.rst > > ============================================================================== > > --- python/branches/py3k/Doc/whatsnew/3.2.rst (original) > > +++ python/branches/py3k/Doc/whatsnew/3.2.rst Tue Dec 28 14:26:42 2010 > > @@ -1705,3 +1705,7 @@ > > > > (Contributed by Georg Brandl and Mattias Br?ndstr?m; > > `appspot issue 53094`_.) > > + > > +* :func:`struct.pack` doesn't encode implicitly unicode to UTF-8 anymore: > > That phrasing is awkward at best. Sorry, I am still learning english :-) > I believe you mean > "struct.pack no longer implicitly encodes unicode to UTF-8", > which will be clearer to most people. Yes, done in r87559 Victor From victor.stinner at haypocalc.com Wed Dec 29 00:41:17 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Dec 2010 00:41:17 +0100 Subject: [Python-Dev] [Python-checkins] r87505 - in python/branches/py3k: Doc/c-api/unicode.rst Include/unicodeobject.h In-Reply-To: <20101228171452.C3FA72361E2@kimball.webabinitio.net> References: <20101227014929.D926EEEA85@mail.python.org> <20101228041340.CDF93200C26@kimball.webabinitio.net> <1293528531.24502.10.camel@marge> <20101228171452.C3FA72361E2@kimball.webabinitio.net> Message-ID: <1293579677.29958.2.camel@marge> Le mardi 28 d?cembre 2010 ? 12:14 -0500, R. David Murray a ?crit : > I think your first paragraph is what you should put in the docs: "it is > best to pass only ASCII-encoded strings, but the function interprets > the input string as ISO-8859-1 if it contains non-ASCII characters". Nice, done in r87560 Victor From amauryfa at gmail.com Wed Dec 29 01:13:13 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Dec 2010 01:13:13 +0100 Subject: [Python-Dev] Backport troubles with mercurial Message-ID: Hello, The PyPy project recently switched from svn to mercurial. Since this day I have some difficulties to perform simple tasks, and my questions did not receive satisfying answers. I was sure the Python project would have the same issues; so I started from the repositories in http://code.python.org/hg/ and tried simple merges between Python versions. I would like several people to try the same example, and tell how they handle it. I'm new to mercurial, and I may have missed something important! Let's say you have to do the simple change shown in the diff below, and want to "fix" the the 3 usual versions: py3k, release31-maint, release27-maint. The diff was made against py3k. How would you do it with Mercurial? Please try to do it for real! "hg merge" is not the correct command: it merges whole branches, a change comes with all its ancestors. What we want is to "cherry-pick" a single change. "hg transplant" fails to apply the change to release31-maint because of a tiny conflict (in the diff context!) that leaves you with an ugly "hunks FAILED" and a .rej file you have to parse and apply by hand. "hg transplant" seems to succeed in release27-maint, but the test fails to run because "support" should be changed to "test_support". The change is already committed - to fix it another changeset is needed. This does not look clean to me: both changesets will be pushed to the repository, and diff review (on the python-checkins mailing list) is almost impossible. Furthermore, "hg transplant" does not keep track of integrations. There is a "transplants" file, but it stays in my local clone. Faced with a similar case in pypy, we finally manually copied the files between directories... and the fastest with our example is probably to do the change manually in all three directories. There is surely a better way to work with maintenance branches! PEP374 suggests to first modify the oldest release, but this does not seems right to me (I have three reasons in mind) At the moment, I feel that mercurial just cannot work for core Python development. At the very least before the migration we need precise use cases like this and recipes for common cases. Thanks for your help, -- Amaury Forgeot d'Arc diff -r 2777ae4d10d9 Lib/test/test_contextlib.py --- a/Lib/test/test_contextlib.py Tue Dec 28 22:14:34 2010 +0100 +++ b/Lib/test/test_contextlib.py Wed Dec 29 00:08:18 2010 +0100 @@ -26,6 +26,7 @@ state.append(x) self.assertEqual(state, [1, 42, 999]) + @support.cpython_only def test_contextmanager_finally(self): state = [] @contextmanager @@ -36,10 +37,10 @@ finally: state.append(999) with self.assertRaises(ZeroDivisionError): - with woohoo() as x: + with woohoo() as y: self.assertEqual(state, [1]) - self.assertEqual(x, 42) - state.append(x) + self.assertEqual(y, 42) + state.append(y) raise ZeroDivisionError() self.assertEqual(state, [1, 42, 999]) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jimjjewett at gmail.com Wed Dec 29 03:35:09 2010 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 28 Dec 2010 21:35:09 -0500 Subject: [Python-Dev] [Python-checkins] r87523 - python/branches/py3k/Doc/tutorial/interpreter.rst In-Reply-To: <20101228091824.4FE70F4E4@mail.python.org> References: <20101228091824.4FE70F4E4@mail.python.org> Message-ID: It might still be worth saying something like: Note that this "python wrote: > Author: georg.brandl > Date: Tue Dec 28 10:18:24 2010 > New Revision: 87523 > > Log: > Remove confusing paragraph -- this is relevant only to advanced users anyway and does not belong into the tutorial. > > Modified: > ? python/branches/py3k/Doc/tutorial/interpreter.rst > > Modified: python/branches/py3k/Doc/tutorial/interpreter.rst > ============================================================================== > --- python/branches/py3k/Doc/tutorial/interpreter.rst ? (original) > +++ python/branches/py3k/Doc/tutorial/interpreter.rst ? Tue Dec 28 10:18:24 2010 > @@ -58,14 +58,6 @@ > ?``python -m module [arg] ...``, which executes the source file for *module* as > ?if you had spelled out its full name on the command line. > > -Note that there is a difference between ``python file`` and ``python > - -``sys.stdin.read()``, are satisfied from *file*. ?Since this file has already > -been read until the end by the parser before the program starts executing, the > -program will encounter end-of-file immediately. ?In the former case (which is > -usually what you want) they are satisfied from whatever file or device is > -connected to standard input of the Python interpreter. > - > ?When a script file is used, it is sometimes useful to be able to run the script > ?and enter interactive mode afterwards. ?This can be done by passing :option:`-i` > ?before the script. ?(This does not work if the script is read from standard > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > http://mail.python.org/mailman/listinfo/python-checkins > From rdmurray at bitdance.com Wed Dec 29 03:33:43 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Tue, 28 Dec 2010 21:33:43 -0500 Subject: [Python-Dev] [Python-checkins] r87537 - in python/branches/py3k: Doc/library/struct.rst Doc/whatsnew/3.2.rst Lib/test/test_struct.py Lib/wave.py Misc/NEWS Modules/_struct.c In-Reply-To: <1293579493.29958.1.camel@marge> References: <20101228132642.83E95EE9B6@mail.python.org> <4D1A46EF.5030509@udel.edu> <1293579493.29958.1.camel@marge> Message-ID: <20101229023343.C6B14238BCA@kimball.webabinitio.net> On Wed, 29 Dec 2010 00:38:13 +0100, Victor Stinner wrote: >Le mardi 28 d??cembre 2010 ?? 15:22 -0500, Terry Reedy a ??crit : >> On 12/28/2010 8:26 AM, victor.stinner wrote: >> > Author: victor.stinner >> > Date: Tue Dec 28 14:26:42 2010 >> > New Revision: 87537 >> > >> > Log: >> > Issue #10783: struct.pack() doesn't encode implicitly unicode to UTF-8 >> >> > Modified: python/branches/py3k/Doc/whatsnew/3.2.rst >> > >============================================================================== >> > --- python/branches/py3k/Doc/whatsnew/3.2.rst (original) >> > +++ python/branches/py3k/Doc/whatsnew/3.2.rst Tue Dec 28 14:26:42 2010 >> > @@ -1705,3 +1705,7 @@ >> > >> > (Contributed by Georg Brandl and Mattias Br??ndstr??m; >> > `appspot issue 53094`_.) >> > + >> > +* :func:`struct.pack` doesn't encode implicitly unicode to UTF-8 anymore: >> >> That phrasing is awkward at best. > >Sorry, I am still learning english :-) > >> I believe you mean >> "struct.pack no longer implicitly encodes unicode to UTF-8", >> which will be clearer to most people. > >Yes, done in r87559 You still have two words swapped. You changed it to no longer encodes implicitly unicode but it should be no longer implicitly encodes unicode -- R. David Murray www.bitdance.com From orsenthil at gmail.com Wed Dec 29 04:29:01 2010 From: orsenthil at gmail.com (Senthil Kumaran) Date: Wed, 29 Dec 2010 11:29:01 +0800 Subject: [Python-Dev] [Python-checkins] r87523 - python/branches/py3k/Doc/tutorial/interpreter.rst In-Reply-To: References: <20101228091824.4FE70F4E4@mail.python.org> Message-ID: On Wed, Dec 29, 2010 at 10:35 AM, Jim Jewett wrote: > It might still be worth saying something like: > > Note that this "python details are not included in this tutorial. No, it this in fact puzzling. I was fine with the previous paragraph and if someone found it confusing, removing it is okay. But the above suggested sentence would leave the reader with an incomplete information. -- Senthil From cournape at gmail.com Wed Dec 29 05:13:25 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Dec 2010 13:13:25 +0900 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: On Wed, Dec 29, 2010 at 9:13 AM, Amaury Forgeot d'Arc wrote: > Hello, > The PyPy project recently switched from svn to mercurial. Since this day I > have some > difficulties to perform simple tasks, and my questions did not receive > satisfying answers. > I was sure the Python project would have the same issues; > so I started from the repositories in?http://code.python.org/hg/?and > tried?simple > merges between Python versions. > I would like several people to try the same example, and tell how they > handle it. > I'm new to mercurial, and I may have missed something important! > > Let's say you have to do the simple change shown in the diff below, > and want to "fix" the the 3 usual versions: py3k, release31-maint, > release27-maint. > The diff was made against py3k. > How would you do it with Mercurial??Please try to do it for real! The easiest way I found to emulate git cherry-pick (which does exactly what you want) with hg is to use import/export commands: http://mercurial.selenic.com/wiki/CommunicatingChanges It is indeed quite a pain to use in my experience, because you cannot easily refer to the original commit the cherry pick is coming from (i.e. no equivalent to git cherry-pick -x), and the conflict resolution is quite dumb. One alternative is to be careful about where you apply your patch (http://stackoverflow.com/questions/3926906/what-is-the-most-efficient-way-to-handle-hg-import-rejects), but that's not very convenient either. cheers, David From stephen at xemacs.org Wed Dec 29 07:18:56 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 29 Dec 2010 15:18:56 +0900 Subject: [Python-Dev] [Python-checkins] r87537 - in python/branches/py3k: Doc/library/struct.rst Doc/whatsnew/3.2.rst Lib/test/test_struct.py Lib/wave.py Misc/NEWS Modules/_struct.c In-Reply-To: <1293579493.29958.1.camel@marge> References: <20101228132642.83E95EE9B6@mail.python.org> <4D1A46EF.5030509@udel.edu> <1293579493.29958.1.camel@marge> Message-ID: <87zkrpje3z.fsf@uwakimon.sk.tsukuba.ac.jp> Victor Stinner writes: > >Sorry, I am still learning english :-) Aren't we all! :-) From g.brandl at gmx.net Wed Dec 29 09:01:12 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 29 Dec 2010 09:01:12 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: Am 29.12.2010 01:13, schrieb Amaury Forgeot d'Arc: > Hello, > > The PyPy project recently switched from svn to mercurial. Since this day I have some > difficulties to perform simple tasks, and my questions did not receive > satisfying answers. > > I was sure the Python project would have the same issues; > so I started from the repositories in http://code.python.org/hg/ and tried simple > merges between Python versions. > I would like several people to try the same example, and tell how they handle it. > I'm new to mercurial, and I may have missed something important! > > > Let's say you have to do the simple change shown in the diff below, > and want to "fix" the the 3 usual versions: py3k, release31-maint, release27-maint. > The diff was made against py3k. > > How would you do it with Mercurial? Please try to do it for real! > > "hg merge" is not the correct command: it merges whole branches, a change > comes with all its ancestors. What we want is to "cherry-pick" a single change. > > "hg transplant" fails to apply the change to release31-maint because > of a tiny conflict (in the diff context!) that leaves you with an ugly > "hunks FAILED" and a .rej file you have to parse and apply by hand. Yes, this has been an irritation to me as well. It's probably not so hard for a Mercurial coredev to change transplant into generating inline conflict markers though. BTW, we would apply the diff in release31-maint and then "hg merge" it to py3k. transplant still applies to 2.7 though. > "hg transplant" seems to succeed in release27-maint, > but the test fails to run because "support" should be changed to "test_support". > The change is already committed - to fix it another changeset is needed. > This does not look clean to me: both changesets will be pushed to the repository, > and diff review (on the python-checkins mailing list) is almost impossible. Right. Seems to me transplant should get a command-line switch that always goes into break-and-continue mode, where the commit is only made after calling "hg transplant --continue". > Furthermore, "hg transplant" does not keep track of integrations. > There is a "transplants" file, but it stays in my local clone. You're wrong, it does keep track of integrations in the commit metadata. The only thing that's clone-local is the transplants cache file. Sadly, transplant only checks against this file, but it is very easy to write a hook that keeps it up to date; I've already done that work. > Faced with a similar case in pypy, we finally manually copied the files > between directories... and the fastest with our example is probably > to do the change manually in all three directories. > > There is surely a better way to work with maintenance branches! > PEP374 suggests to first modify the oldest release, but this does not > seems right to me (I have three reasons in mind) Would you care to explain those? > At the very least before the migration we need precise use cases like this > and recipes for common cases. Which is what we'll certainly get once we have the test repo and Brett is rewriting the dev guide. Georg From amauryfa at gmail.com Wed Dec 29 09:02:23 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Dec 2010 09:02:23 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: 2010/12/29 David Cournapeau > The easiest way I found to emulate git cherry-pick (which does exactly > what you want) with hg is to use import/export commands: > http://mercurial.selenic.com/wiki/CommunicatingChanges > > It is indeed quite a pain to use in my experience, because you cannot > easily refer to the original commit the cherry pick is coming from > (i.e. no equivalent to git cherry-pick -x), and the conflict > resolution is quite dumb. > This is precisely why I proposed a specific example. Which precise steps do you take? How much typing or manual copy/paste is required for this very simple case? Can you do the merge in less than 10 minutes? And finally the biased question: can you find one improvement over the current situation with svn? One alternative is to be careful about where > you apply your patch > ( > http://stackoverflow.com/questions/3926906/what-is-the-most-efficient-way-to-handle-hg-import-rejects > ), > but that's not very convenient either. > I've read all this, and this method does not work, for several reasons: - py3k / 2.7 / 3.1 cannot be ordered, there is no "earliest possible parent".. we usually consider py3k as a child of both 2.7 and 3.1, and there is no common parent. - even if there was one, there is the problem of changes specifically made for 2.7 that make no sense in py3k. You'd have to perform a "dummy merge" to py3k which has the disadvantage of cluttering the py3k change log. And think of the horror if someone forgets this dummy merge. - in any case, the actual repositories in http://code.python.org/hg/ are not clones of each other, so "hg merge" won't work and "hg pull" will clone the whole remote repository. (btw, now that I have "hg pull" another repo, how can I remove it? is my clone broken forever?) -- Amaury Forgeot d'Arc -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.brandl at gmx.net Wed Dec 29 09:20:18 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 29 Dec 2010 09:20:18 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc: > I've read all this, and this method does not work, for several reasons: > > - py3k / 2.7 / 3.1 cannot be ordered, there is no "earliest possible parent".. > we usually consider py3k as a child of both 2.7 and 3.1, and there is no common > parent. Yes. Think of 2.7 as the oddball one, where revisions must be cherry-picked to, whereas the 3.x line can use the "commit in parent branch" system (and much more easily, since there are no changes in syntax and much less changes in code overall). > - even if there was one, there is the problem of changes specifically made for 2.7 > that make no sense in py3k. You'd have to perform a "dummy merge" to py3k > which has the disadvantage of cluttering the py3k change log. And think of the > horror > if someone forgets this dummy merge. No horror at all: the next committer notices the extra changes in his merge and removes them. *Never* commit merges without reviewing the diff. (The situtation is similar, by the way, to someone typing the wrong revision number when using svnmerge, and not noticing it because he does not review the diff.) > - in any case, the actual repositories in http://code.python.org/hg/ are not clones > of each other, so "hg merge" won't work and "hg pull" will clone the whole > remote repository. Note that the repos on code.python.org are not the results of our conversion process. Those will be on hg.python.org. The former are repos contributed by Antoine (I think) that he uses with hgsubversion to work on Python using Mercurial right now; they will disappear after hg.python.org works. > (btw, now that I have "hg pull" another repo, how can I remove it? is my clone > broken forever?) No, you can "hg strip" away the root of the wrongly pulled revisions (which also strips all descendants), or use "hg clone -r XXX" to create a new clone with only a specified revision and all its ancestors. Georg From amauryfa at gmail.com Wed Dec 29 10:53:56 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Dec 2010 10:53:56 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: 2010/12/29 Georg Brandl > > Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc: > > > - even if there was one, there is the problem of changes specifically made for 2.7 > > that make no sense in py3k. You'd have to perform a "dummy merge" to py3k > > which has the disadvantage of cluttering the py3k change log. And think of the > > horror > > if someone forgets this dummy merge. > > No horror at all: the next committer notices the extra changes in his merge and > removes them. ?*Never* commit merges without reviewing the diff. ?(The > situtation is similar, by the way, to someone typing the wrong revision number > when using svnmerge, and not noticing it because he does not review the diff.) Except that it's easy to overlook a diff and not notice another small change mixed in your merge. Checking "hg merge -P" is probably better. But is it possible to "remove" the extra change? What worries me more is the requirement to find the correct branch before I can even edit the file.?How would you, Georg, deal with doc patches (typos, bad markup)? With subversion?it's easy to work on trunk, and then have svnmerge tell you which chunk is in conflict?and does not apply to the maintenance branch. > > > - in any case, the actual repositories in http://code.python.org/hg/ are not clones > > of each other, so "hg merge" won't work and "hg pull" will clone the whole > > remote repository. > > Note that the repos on code.python.org are not the results of our conversion > process. ?Those will be on hg.python.org. ?The former are repos contributed > by Antoine (I think) that he uses with hgsubversion to work on Python using > Mercurial right now; they will disappear after hg.python.org works. ok. I used them because other the repos I found were really old. And?http://hg.python.org/cpython/?probably needs an initial "dummy merge" on every branch. > > > (btw, now that I have "hg pull" another repo, how can I remove it? is my clone > > broken forever?) > > No, you can "hg strip" away the root of the wrongly pulled revisions (which > also strips all descendants), or use "hg clone -r XXX" to create a new clone > with only a specified revision and all its ancestors. "hg strip" worked for me. The root revision to strip was simply the first one (=oldest) in "hg outgoing". -- Amaury Forgeot d'Arc From dirkjan at ochtman.nl Wed Dec 29 10:59:07 2010 From: dirkjan at ochtman.nl (Dirkjan Ochtman) Date: Wed, 29 Dec 2010 10:59:07 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: On Wed, Dec 29, 2010 at 10:53, Amaury Forgeot d'Arc wrote: > And?http://hg.python.org/cpython/?probably needs an initial "dummy merge" > on every branch. Yes, the present delays are all about getting that right. Cheers, Dirkjan From g.brandl at gmx.net Wed Dec 29 11:00:59 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 29 Dec 2010 11:00:59 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: Am 29.12.2010 10:53, schrieb Amaury Forgeot d'Arc: > 2010/12/29 Georg Brandl >> >> Am 29.12.2010 09:02, schrieb Amaury Forgeot d'Arc: >> >> > - even if there was one, there is the problem of changes specifically made for 2.7 >> > that make no sense in py3k. You'd have to perform a "dummy merge" to py3k >> > which has the disadvantage of cluttering the py3k change log. And think of the >> > horror >> > if someone forgets this dummy merge. >> >> No horror at all: the next committer notices the extra changes in his merge and >> removes them. *Never* commit merges without reviewing the diff. (The >> situtation is similar, by the way, to someone typing the wrong revision number >> when using svnmerge, and not noticing it because he does not review the diff.) > > Except that it's easy to overlook a diff and not notice another small change > mixed in your merge. Checking "hg merge -P" is probably better. > But is it possible to "remove" the extra change? No; hg merging is DAG-based, so you will always merge all ancestors. The only way to "remove" it is a null-merge. > What worries me more is the requirement to find the correct branch before I can > even edit the file. How would you, Georg, deal with doc patches > (typos, bad markup)? Finding the correct branch is not really hard. Bugfixes go to maintenance, features to trunk. You need to think about which category your change is right now too, when deciding what to backport/svnmerge. Accordingly, I would apply doc patches in release31-maint and merge them to trunk, probably all at once with one merge commit. >> > - in any case, the actual repositories in http://code.python.org/hg/ are not clones >> > of each other, so "hg merge" won't work and "hg pull" will clone the whole >> > remote repository. >> >> Note that the repos on code.python.org are not the results of our conversion >> process. Those will be on hg.python.org. The former are repos contributed >> by Antoine (I think) that he uses with hgsubversion to work on Python using >> Mercurial right now; they will disappear after hg.python.org works. > > ok. I used them because other the repos I found were really old. > And http://hg.python.org/cpython/ probably needs an initial "dummy merge" > on every branch. Yes, that is a very old conversion result. We should get a new one in a few days. Georg From amauryfa at gmail.com Wed Dec 29 11:09:21 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Dec 2010 11:09:21 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: 2010/12/29 Georg Brandl : >> What worries me more is the requirement to find the correct branch before I can >> even edit the file. How would you, Georg, deal with doc patches >> (typos, bad markup)? > > Finding the correct branch is not really hard. ?Bugfixes go to maintenance, > features to trunk. This works well indeed, provided there is only one maintenance branch. > You need to think about which category your change is > right now too, when deciding what to backport/svnmerge. No, today this decision can take place after the development, some tickets got reopened because a backport was needed. -- Amaury Forgeot d'Arc From g.brandl at gmx.net Wed Dec 29 11:16:44 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 29 Dec 2010 11:16:44 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: Am 29.12.2010 11:09, schrieb Amaury Forgeot d'Arc: > 2010/12/29 Georg Brandl : >>> What worries me more is the requirement to find the correct branch before I can >>> even edit the file. How would you, Georg, deal with doc patches >>> (typos, bad markup)? >> >> Finding the correct branch is not really hard. Bugfixes go to maintenance, >> features to trunk. > > This works well indeed, provided there is only one maintenance branch. Which there is, except for security fixes (but they are rare and require release manager intervention anyway). >> You need to think about which category your change is >> right now too, when deciding what to backport/svnmerge. > > No, today this decision can take place after the development, > some tickets got reopened because a backport was needed. A change can of course also be transplanted after the fact. It requires another merge, but usually a trivial one. Of course you may have to think a bit more about bugfix *before* plunging into the code, but is that necessarily a bad thing? We had quite a few bugfixes not backported precisely because it is very easy not to care about them right now. Georg From victor.stinner at haypocalc.com Wed Dec 29 14:17:33 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Dec 2010 14:17:33 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD Message-ID: <1293628653.16756.13.camel@marge> Hi, FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of a concurrent.futures failure. The problem is that test_concurrent_futures uses many (multiprocessing) POSIX semaphores, whereas POSIX semaphores support in FreeBSD is recent and limited. We have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: * http://bugs.python.org/issue10348 * "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot ^-- thread in python-dev opened last month I would like to know if it should be considered as a release blocker. Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to write a patch, but I never used semaphores (POSIX or SysV). There is a third party module which looks complete and stable: http://semanchuk.com/philip/sysv_ipc/ It is released under the BSD license. It supports semaphores, but also shared memory and message queues. We don't need all of those, semaphores would be enough. I added its author (Philip Semanchuk) to this thread. I don't know how we should decide to use POSIX or SysV semaphores. It looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD based OSes), and POSIX is preferred on Linux. Victor From martin at v.loewis.de Wed Dec 29 14:20:18 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Dec 2010 14:20:18 +0100 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: References: Message-ID: <4D1B3592.3040403@v.loewis.de> Am 28.12.2010 18:08, schrieb Lukas Lueg: > Also, the > load_fast in lne 22 to reference x could be taken out of the loop as x > will always point to the same object.... That's not true; a debugger may change the value of x. Regards, Martin From cournape at gmail.com Wed Dec 29 14:40:50 2010 From: cournape at gmail.com (David Cournapeau) Date: Wed, 29 Dec 2010 22:40:50 +0900 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: On Wed, Dec 29, 2010 at 5:02 PM, Amaury Forgeot d'Arc wrote: > 2010/12/29 David Cournapeau >> >> The easiest way I found to emulate git cherry-pick (which does exactly >> what you want) with hg is to use import/export commands: >> http://mercurial.selenic.com/wiki/CommunicatingChanges >> >> It is indeed quite a pain to use in my experience, because you cannot >> easily refer to the original commit the cherry pick is coming from >> (i.e. no equivalent to git cherry-pick -x), and the conflict >> resolution is quite dumb. > > This is precisely why I proposed a specific example. > Which precise steps do you take? > How much typing or manual copy/paste is required for this very simple case? > Can you do the merge in less than 10 minutes? I don't know in this specific case. As I said, when I have to use hg, that's the technique I use, and you get the issue you mention. That's a hg limitation AFAICS. > And finally the biased question: > can you find one improvement over the current situation with svn? I am not involved in the hg conversion process nor its decision (I am not even a python committer). Cherry picking is actually easier to do with svn by "accident", because its merge method, up to 1.5 at least, was really dumb and never remembered the ancestors of a previous merge. As for a few data points which may or may not be relevant: in numpy we convereted from svn -> git recently, and it has worked pretty well, with numerous new contributions happening, and better, new contributors appearing. I have been the release manager for numpy for several years, and as such had to do the kind of tasks you mention numerous times with svn, and the only words that comes to mind when remember this period would not be appropriate on a public mailing list: I always found svn to be horrible. I started using git to make my life as release manager simpler, actually. I would be surprised if python's situation did not end up being similar to numpy's one. Other projects related to numpy made the changes to a DVCS (ipython, nipy, lean scikit) before and none of them ever regretted it AFAIK, and sometimes the people who become the most vocal proponents of the new tool were the most sceptic ones before. Certainly, very few people if any asked to revert the process. cheers, David From stephen at xemacs.org Wed Dec 29 15:12:28 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Wed, 29 Dec 2010 23:12:28 +0900 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> Amaury Forgeot d'Arc writes: > Which precise steps do you take [to cherrypick with export/import]? hg export -r $CHERRY .tmp.patch xemacs .tmp.patch ;; Move to end of log message, type "Cherry-picked from: ", then ;; C-u M-! hg id -i -r $CHERRY RET C-x C-s C-x C-c> hg import .tmp.patch worked for me the one time I tried it, IIRC. Now I use patch queues, and avoid cherry-picking as much as possible. (Avoiding cherry-pick is not going to be possible for 3.x <-> 2.x porting, of course, but in many cases there a simple patch application is not going to work at all, anyway. Should be relatively rare, no?) CHERRY can be a tag, revno, or other alias for the revision. You may want to add other identifying information with additional arguments to "hg id", but "hg id -i" give the canonical ID. You may want to fix up committer and date using the -u and -d arguments to hg import. > How much typing or manual copy/paste is required for this very simple case? For the simple case, the above can be scripted, and the script takes one argument, CHERRY. It would not be hard to automate adding the cherry-pick notice. > Can you do the merge in less than 10 minutes? If no conflicts, sure, no problem. Probably about 30 seconds, including adding the cherrypicked revid to the log message. With a script, maybe 1 second. If there are conflicts, it depends on the complexity of the conflicts. I can imagine this could get hairy with the export | import method and a megapatch, but I've never had to deal with that in practice. This is one reason why it's recommended that feature branches be constructed of multiple, clearly separated patches. (This may be difficult to do for complex changes, but it's usually a reasonable goal.) > And finally the biased question: > can you find one improvement over the current situation with svn? No, I can't find just one. How many are you willing to read? svn has the advantage (and yes, it's a big one) that developers are already used to it. Otherwise, everything goes in favor of any of the dVCSes. > - even if there was one, there is the problem of changes > specifically made for 2.7 that make no sense in py3k. This is a problem with the committer, not with the VCS. Such changes belong in a separate commit, with an appropriate log entry, in *any* version control system. If that is done, there's no problem, you just don't cherrypick that change. > You'd have to perform a "dummy merge" to py3k which has the > disadvantage of cluttering the py3k change log. I don't see why. Such dummy merges can help only if you are going to rely on hg merge. But you are not going to be merging 2.x and 3.x. True, some patches from the maintenance branch won't be applied to the trunk. This is expected, and it is a problem that needs to be solved, probably with a tool like svnmerge, which will do a dummy merge or a revert commit to exclude that revision from any future merge to trunk. Therefore, if we change our minds, it's perfectly safe to cherrypick. Any conflicts that occur are real conflicts, not merge artifacts. > And think of the horror if someone forgets this dummy merge. What horror? By cherry-picking you have assumed responsibility for managing future cherry-picks from that source to that target in any case; you can no longer expect reliable help from Mercurial. It's not at all clear to me that a dummy merge could accomplish anything useful, especially in the case of 2.x <-> 3.x cherry-picks. If on the other hand you're thinking of the merge-from-maint-branch case, as I say this is a problem that needs solving anyway, just as it did with svn. From amauryfa at gmail.com Wed Dec 29 15:17:24 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Dec 2010 15:17:24 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: 2010/12/29 Georg Brandl : >>> You need to think about which category your change is >>> right now too, when deciding what to backport/svnmerge. >> >> No, today this decision can take place after the development, >> some tickets got reopened because a backport was needed. > > A change can of course also be transplanted after the fact. ?It requires another > merge, but usually a trivial one. No, it's not trivial with hg: this is the very subject of the thread! -- Amaury Forgeot d'Arc From amauryfa at gmail.com Wed Dec 29 15:29:28 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Dec 2010 15:29:28 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: 2010/12/29 David Cournapeau : > On Wed, Dec 29, 2010 at 5:02 PM, Amaury Forgeot d'Arc > wrote: >> 2010/12/29 David Cournapeau >>> >>> The easiest way I found to emulate git cherry-pick (which does exactly >>> what you want) with hg is to use import/export commands: >>> http://mercurial.selenic.com/wiki/CommunicatingChanges >>> >>> It is indeed quite a pain to use in my experience, because you cannot >>> easily refer to the original commit the cherry pick is coming from >>> (i.e. no equivalent to git cherry-pick -x), and the conflict >>> resolution is quite dumb. >> >> This is precisely why I proposed a specific example. >> Which precise steps do you take? >> How much typing or manual copy/paste is required for this very simple case? >> Can you do the merge in less than 10 minutes? > > I don't know in this specific case. As I said, when I have to use hg, > that's the technique I use, and you get the issue you mention. That's > a hg limitation AFAICS. Yes, Georg identified three things that "hg transplant" should do better: - an option to not commit - a way to add conflict markers in the source instead of the .rej file (In this case, it may be just as easy to use the standard merge tools) - somehow share the "transplants" cache file between clones. > sometimes the people who become the most vocal proponents of the new > tool were the most sceptic ones before. I really was not sceptic before, and I certainly don't want to become one! But yesterday I was blocked the whole afternoon by something I still call an routine task with most other SCMs; and the only answer I get is "that's right, it's a pain" hg will certainly impose a change in the way we develop Python. I'm not sure everybody understands the consequences. -- Amaury Forgeot d'Arc From g.brandl at gmx.net Wed Dec 29 15:57:10 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 29 Dec 2010 15:57:10 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: Am 29.12.2010 15:17, schrieb Amaury Forgeot d'Arc: > 2010/12/29 Georg Brandl : >>>> You need to think about which category your change is >>>> right now too, when deciding what to backport/svnmerge. >>> >>> No, today this decision can take place after the development, >>> some tickets got reopened because a backport was needed. >> >> A change can of course also be transplanted after the fact. It requires another >> merge, but usually a trivial one. > > No, it's not trivial with hg: this is the very subject of the thread! I don't understand: if you make the same change in two branches, but in different changesets, and then merge these branches, Mercurial will usually notice that and not trouble you with conflicts. Georg From g.brandl at gmx.net Wed Dec 29 16:01:38 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 29 Dec 2010 16:01:38 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <1293628653.16756.13.camel@marge> References: <1293628653.16756.13.camel@marge> Message-ID: Am 29.12.2010 14:17, schrieb Victor Stinner: > Hi, > > FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of > a concurrent.futures failure. The problem is that > test_concurrent_futures uses many (multiprocessing) POSIX semaphores, > whereas POSIX semaphores support in FreeBSD is recent and limited. We > have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead > of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: > > * http://bugs.python.org/issue10348 > * "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot > ^-- thread in python-dev opened last month > > I would like to know if it should be considered as a release blocker. > Georg Brandl said yes on IRC. Under the condition that it is within reason to fix it before the release. Georg From amauryfa at gmail.com Wed Dec 29 16:05:40 2010 From: amauryfa at gmail.com (Amaury Forgeot d'Arc) Date: Wed, 29 Dec 2010 16:05:40 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: 2010/12/29 Georg Brandl : >>> A change can of course also be transplanted after the fact. ?It requires another >>> merge, but usually a trivial one. >> >> No, it's not trivial with hg: this is the very subject of the thread! > > I don't understand: if you make the same change in two branches, but > in different changesets, and then merge these branches, Mercurial will > usually notice that and not trouble you with conflicts. Actually I never passed the first step: make the same change to two branches. -- Amaury Forgeot d'Arc From philip at semanchuk.com Wed Dec 29 15:24:07 2010 From: philip at semanchuk.com (Philip Semanchuk) Date: Wed, 29 Dec 2010 09:24:07 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <1293628653.16756.13.camel@marge> References: <1293628653.16756.13.camel@marge> Message-ID: On Dec 29, 2010, at 8:17 AM, Victor Stinner wrote: > Hi, > > FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of > a concurrent.futures failure. The problem is that > test_concurrent_futures uses many (multiprocessing) POSIX semaphores, > whereas POSIX semaphores support in FreeBSD is recent and limited. We > have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead > of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: > > * http://bugs.python.org/issue10348 > * "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot > ^-- thread in python-dev opened last month > > I would like to know if it should be considered as a release blocker. > Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to > write a patch, but I never used semaphores (POSIX or SysV). > > There is a third party module which looks complete and stable: > http://semanchuk.com/philip/sysv_ipc/ > > It is released under the BSD license. It supports semaphores, but also > shared memory and message queues. We don't need all of those, semaphores > would be enough. I added its author (Philip Semanchuk) to this thread. Hi all, What Victor says above is correct, although I wasn't aware that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to 7.2 it was broken but 7.2 worked OK in my limited testing. In any case, the sysv_ipc module is mine and it's mature and you're welcome to pillage it in whole or in part. > I don't know how we should decide to use POSIX or SysV semaphores. It > looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD > based OSes), and POSIX is preferred on Linux. Hmmm, "preferred" is a tricky word here. I would phrase it slightly differently: POSIX IPC is preferred everywhere (by me, at least) because it's a more modern API. However, SysV IPC is fully supported everywhere while the same can't be said about POSIX IPC. Speaking of POSIX IPC, I also have a posix_ipc module that's quite similar to sysv_ipc and the platform notes in the documentation tell one what support to expect for POSIX IPC under various platforms: http://semanchuk.com/philip/posix_ipc/#platforms Cheers Philip From rdmurray at bitdance.com Wed Dec 29 16:25:10 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 29 Dec 2010 10:25:10 -0500 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20101229152510.1E766239DB5@kimball.webabinitio.net> On Wed, 29 Dec 2010 23:12:28 +0900, "Stephen J. Turnbull" wrote: > worked for me the one time I tried it, IIRC. Now I use patch queues, > and avoid cherry-picking as much as possible. (Avoiding cherry-pick > is not going to be possible for 3.x <-> 2.x porting, of course, but in > many cases there a simple patch application is not going to work at > all, anyway. Should be relatively rare, no?) No. We merge bug fixes to 2.7 on a routine basis. It is the rule rather than the exception, by a long shot. (Features of course only go into trunk...but those are irrelevant to this conversation, since there's no requirement to merge them anywhere...as are bug fixes we choose not to backport, since those don't go into the Python3 maint branch either.) Some such merges apply cleanly, many more have only a few conflicts requiring only trivial tweaks, which are made easy by the in-line merge conflict markers. A few require re-engineering of the patch, and of course it is a bit of a pain with svnmerge to deal with the ones where the target file names have changed (but again, there aren't that many of those). [*] So, since we are going to be maintaining 2.7 for a while, this is a workflow problem that we *must* solve to make the hg transition worthwhile. I have no doubt that we will, but I also have no doubt we need to *solve* it, not just wave our hands. Many thanks to Georg and Djirkan for working on the problem(s). -- R. David Murray www.bitdance.com [*] I'm speaking of stdlib changes, which is what I do, but I suspect the situation is similar for the C code, and certainly the fact that bug fixes backported to Python3 maint are routinely backported to 2.7 is true. From martin at v.loewis.de Wed Dec 29 16:28:20 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Dec 2010 16:28:20 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> Message-ID: <4D1B5394.10802@v.loewis.de> >> I would like to know if it should be considered as a release blocker. >> Georg Brandl said yes on IRC. > > Under the condition that it is within reason to fix it before the > release. What *should* be possible is to disable building SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, multiprocessing locks would stop working on FreeBSD, and concurrent futures; the tests would recognize this lack of features and get skipped. Regards, Martin From martin at v.loewis.de Wed Dec 29 16:43:11 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Dec 2010 16:43:11 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> Message-ID: <4D1B570F.1000502@v.loewis.de> > Hi all, What Victor says above is correct, although I wasn't aware > that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to > 7.2 it was broken but 7.2 worked OK in my limited testing. In any > case, the sysv_ipc module is mine and it's mature and you're welcome > to pillage it in whole or in part. The question really is whether you would be willing to port Modules/_multiprocessing/semaphore.c to SysV IPC. I realize that this would be practically new code, but it would be much appreciated. One question is whether it's actually feasible to implement that API with SysV semaphores. >> I don't know how we should decide to use POSIX or SysV semaphores. >> It looks like SysV is preferred on FreeBSD and Darwin (and maybe >> all BSD based OSes), and POSIX is preferred on Linux. > > Hmmm, "preferred" is a tricky word here. I would phrase it slightly > differently: POSIX IPC is preferred everywhere (by me, at least) > because it's a more modern API. However, SysV IPC is fully supported > everywhere while the same can't be said about POSIX IPC. If you can make the above change, the question then is what API multiprocessing semaphores should be built upon. It seems that you are saying that they should use SysV IPC, and only fall back to POSIX IPC if SysV IPC doesn't work/exist (are there any platforms where this would be the case?) Alternatively, we could have a whitelist of systems on which POSIX IPC is used (==['linux']), and use sysv ipc everywhere else. Regards, Martin From jnoller at gmail.com Wed Dec 29 18:54:57 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 29 Dec 2010 12:54:57 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1B5394.10802@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> Message-ID: On Wed, Dec 29, 2010 at 10:28 AM, "Martin v. L?wis" wrote: >>> I would like to know if it should be considered as a release blocker. >>> Georg Brandl said yes on IRC. >> >> Under the condition that it is within reason to fix it before the >> release. > > What *should* be possible is to disable building > SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, > multiprocessing locks would stop working on FreeBSD, and concurrent > futures; the tests would recognize this lack of features and get > skipped. > > Regards, > Martin The multiprocessing test suite already skips the tests which use the (broken) functionality on BSD correctly. This logic needs to be added to the concurrent.futures library. jesse From jnoller at gmail.com Wed Dec 29 18:58:55 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 29 Dec 2010 12:58:55 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <1293628653.16756.13.camel@marge> References: <1293628653.16756.13.camel@marge> Message-ID: On Wed, Dec 29, 2010 at 8:17 AM, Victor Stinner wrote: > Hi, > > FreeBSD 7.2 3.x buildbot is red since some weeks (or months?) because of > a concurrent.futures failure. The problem is that > test_concurrent_futures uses many (multiprocessing) POSIX semaphores, > whereas POSIX semaphores support in FreeBSD is recent and limited. We > have to use SysV semaphores (ftok, semget, semop, semctl, ...) instead > of POSIX semaphores (sem_open, sem_getvalue, sem_unlink, ...). See: > > ?* http://bugs.python.org/issue10348 > ?* "Too many open files" errors on "x86 FreeBSD 7.2 3.x" buildbot > ? ^-- thread in python-dev opened last month > > I would like to know if it should be considered as a release blocker. > Georg Brandl said yes on IRC. Does anyone know SysV API? I tried to > write a patch, but I never used semaphores (POSIX or SysV). > > There is a third party module which looks complete and stable: > http://semanchuk.com/philip/sysv_ipc/ > > It is released under the BSD license. It supports semaphores, but also > shared memory and message queues. We don't need all of those, semaphores > would be enough. I added its author (Philip Semanchuk) to this thread. > > I don't know how we should decide to use POSIX or SysV semaphores. It > looks like SysV is preferred on FreeBSD and Darwin (and maybe all BSD > based OSes), and POSIX is preferred on Linux. > > Victor The concurrent.futures tests should (like the multiprocessing test suite) detect the lack of support and skip the tests on the broken platforms. I'm sort of surprised FreeBSD support is still broken in this way though (echoed by Philip) although it could be an issue on that particular buildbot. Moving from POSIX IPC to SYSV should *not* be on the plate for a release blocker - that's a much larger task. jesse From martin at v.loewis.de Wed Dec 29 19:34:31 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Dec 2010 19:34:31 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> Message-ID: <4D1B7F37.3080201@v.loewis.de> Am 29.12.2010 18:54, schrieb Jesse Noller: > On Wed, Dec 29, 2010 at 10:28 AM, "Martin v. L?wis" wrote: >>>> I would like to know if it should be considered as a release blocker. >>>> Georg Brandl said yes on IRC. >>> >>> Under the condition that it is within reason to fix it before the >>> release. >> >> What *should* be possible is to disable building >> SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, >> multiprocessing locks would stop working on FreeBSD, and concurrent >> futures; the tests would recognize this lack of features and get >> skipped. >> >> Regards, >> Martin > > The multiprocessing test suite already skips the tests which use the > (broken) functionality on BSD correctly. This logic needs to be added > to the concurrent.futures library. I'm not so sure that skipping the test is the right approach. Doesn't that mean that the code will still fail at runtime with difficult-to-explain messages? I'd rather prefer if the functionality wasn't available in the first place. Also, what specific test are you referring to? Regards, Martin From rdmurray at bitdance.com Wed Dec 29 20:14:03 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Wed, 29 Dec 2010 14:14:03 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> Message-ID: <20101229191403.A166F239421@kimball.webabinitio.net> On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller wrote: > The concurrent.futures tests should (like the multiprocessing test > suite) detect the lack of support and skip the tests on the broken > platforms. I'm sort of surprised FreeBSD support is still broken in > this way though (echoed by Philip) although it could be an issue on > that particular buildbot. If I'm reading the issue correctly, it isn't that it doesn't work, it's that the concurrent.futures tests fail because they create more semaphores than the default FreeBSD install supports. In other words, a user of concurrent.futures really needs to tweak their FreeBSD install to make in fully functional. There should be a way (through sysctl, presumably) to query the maximum number of semaphores and skip the relevant tests on that basis. -- R. David Murray www.bitdance.com From tjreedy at udel.edu Wed Dec 29 20:31:20 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 29 Dec 2010 14:31:20 -0500 Subject: [Python-Dev] Compile() and Windows/Mac newlines Message-ID: In python-list thread "Does Python 3.1 accept \r\n in compile()?" jmfauth notes that compile('print(999)\r\n', '', 'exec') works in 2.7 but not 3.1 (and 3.2 not checked) because 3.1 sees '\r' as SyntaxError. I started to respond that this is part of Py3 cleanup with newlines converted on input and back-compatibility with ancient Python not needed. Then I saw in 3.2 manual "Changed in version 3.2: Allowed use of Windows and Mac newlines. Also input in 'exec' mode does not have to end in a newline anymore. Added the optimize parameter." I verified second statement ("print(999)" works) (and remember commit for third), but original above gives same error. Should "Allowed use of Windows and Mac newlines." be deleted? What else could it mean other than use of '\r' or '\r\n'? -- Terry Jan Reedy From philip at semanchuk.com Wed Dec 29 20:33:29 2010 From: philip at semanchuk.com (Philip Semanchuk) Date: Wed, 29 Dec 2010 14:33:29 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <20101229191403.A166F239421@kimball.webabinitio.net> References: <1293628653.16756.13.camel@marge> <20101229191403.A166F239421@kimball.webabinitio.net> Message-ID: <3C7FBFEC-ECB1-4D94-9233-261B135FE43C@semanchuk.com> On Dec 29, 2010, at 2:14 PM, R. David Murray wrote: > On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller wrote: >> The concurrent.futures tests should (like the multiprocessing test >> suite) detect the lack of support and skip the tests on the broken >> platforms. I'm sort of surprised FreeBSD support is still broken in >> this way though (echoed by Philip) although it could be an issue on >> that particular buildbot. > > If I'm reading the issue correctly, it isn't that it doesn't work, > it's that the concurrent.futures tests fail because they create more > semaphores than the default FreeBSD install supports. In other words, > a user of concurrent.futures really needs to tweak their FreeBSD install > to make in fully functional. I think that's correct. Furthermore, the default 7.2 install doesn't have the necessary kernel modules loaded in order to use POSIX semaphores. See the notes here for FreeBSD: http://semanchuk.com/philip/posix_ipc/#platforms kldstat on my barely-customized 7.2 installation gives a list of three modules: kernel, acpi.ko and linux.ko (I assume because I asked for Linux executable format compatibility when I installed). bye Philip From philip at semanchuk.com Wed Dec 29 20:37:29 2010 From: philip at semanchuk.com (Philip Semanchuk) Date: Wed, 29 Dec 2010 14:37:29 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1B570F.1000502@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B570F.1000502@v.loewis.de> Message-ID: <3C36C60B-717E-4550-AC2E-AB6E2B2E7ECB@semanchuk.com> On Dec 29, 2010, at 10:43 AM, Martin v. L?wis wrote: >> Hi all, What Victor says above is correct, although I wasn't aware >> that POSIX IPC under FreeBSD 7.2 was still having problems. Prior to >> 7.2 it was broken but 7.2 worked OK in my limited testing. In any >> case, the sysv_ipc module is mine and it's mature and you're welcome >> to pillage it in whole or in part. > > The question really is whether you would be willing to port > Modules/_multiprocessing/semaphore.c to SysV IPC. I realize that > this would be practically new code, but it would be much appreciated. > > One question is whether it's actually feasible to implement that API > with SysV semaphores. > >>> I don't know how we should decide to use POSIX or SysV semaphores. >>> It looks like SysV is preferred on FreeBSD and Darwin (and maybe >>> all BSD based OSes), and POSIX is preferred on Linux. >> >> Hmmm, "preferred" is a tricky word here. I would phrase it slightly >> differently: POSIX IPC is preferred everywhere (by me, at least) >> because it's a more modern API. However, SysV IPC is fully supported >> everywhere while the same can't be said about POSIX IPC. > > If you can make the above change, the question then is what API > multiprocessing semaphores should be built upon. It seems that you > are saying that they should use SysV IPC, and only fall back to > POSIX IPC if SysV IPC doesn't work/exist (are there any platforms > where this would be the case?) Actually, I'd probably recommend the opposite. IMO POSIX IPC is better designed and easier to work with so it would be my first choice. SysV would be a fallback for when POSIX IPC is unavailable, broken, incomplete, etc. bye Philip From benjamin at python.org Wed Dec 29 20:53:16 2010 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 29 Dec 2010 13:53:16 -0600 Subject: [Python-Dev] Compile() and Windows/Mac newlines In-Reply-To: References: Message-ID: 2010/12/29 Terry Reedy : > In python-list thread "Does Python 3.1 accept \r\n in compile()?" > jmfauth notes that > compile('print(999)\r\n', '', 'exec') > works in 2.7 but not 3.1 (and 3.2 not checked) because 3.1 sees '\r' as > SyntaxError. > > I started to respond that this is part of Py3 cleanup with newlines > converted on input and back-compatibility with ancient Python not needed. > Then I saw in 3.2 manual > > "Changed in version 3.2: Allowed use of Windows and Mac newlines. Also input > in 'exec' mode does not have to end in a newline anymore. Added the optimize > parameter." > > I verified second statement ("print(999)" works) (and remember commit for > third), but original above gives same error. Should "Allowed use of Windows > and Mac newlines." be deleted? What else could it mean other than use of > '\r' or '\r\n'? $ ./python Python 3.2b2 (py3k:87559, Dec 28 2010, 17:39:51) [GCC 4.4.4] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> compile("print(999)\r\n", "blah", "exec") at 0xb353e8, file "blah", line 1> -- Regards, Benjamin From tjreedy at udel.edu Wed Dec 29 21:02:34 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 29 Dec 2010 15:02:34 -0500 Subject: [Python-Dev] Compile() and Windows/Mac newlines In-Reply-To: References: Message-ID: On 12/29/2010 2:31 PM, Terry Reedy wrote: > In python-list thread "Does Python 3.1 accept \r\n in compile()?" > jmfauth notes that > compile('print(999)\r\n', '', 'exec') > works in 2.7 but not 3.1 (and 3.2 not checked) because 3.1 sees '\r' as > SyntaxError. > > I started to respond that this is part of Py3 cleanup with newlines > converted on input and back-compatibility with ancient Python not > needed. Then I saw in 3.2 manual > > "Changed in version 3.2: Allowed use of Windows and Mac newlines. Also > input in 'exec' mode does not have to end in a newline anymore. Added > the optimize parameter." > > I verified second statement ("print(999)" works) (and remember commit > for third), but original above gives same error. Should "Allowed use of > Windows and Mac newlines." be deleted? What else could it mean other > than use of '\r' or '\r\n'? After tracing the questioned comment to B.Peterson's r76232 merged from 2.7 r76230 "fix several compile() issues by translating newlines in the tokenizer", I decided to open http://bugs.python.org/issue10792 -- Terry Jan Reedy From solipsis at pitrou.net Wed Dec 29 21:04:11 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Dec 2010 21:04:11 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD References: <1293628653.16756.13.camel@marge> <20101229191403.A166F239421@kimball.webabinitio.net> Message-ID: <20101229210411.4b562b59@pitrou.net> On Wed, 29 Dec 2010 14:14:03 -0500 "R. David Murray" wrote: > On Wed, 29 Dec 2010 12:58:55 -0500, Jesse Noller wrote: > > The concurrent.futures tests should (like the multiprocessing test > > suite) detect the lack of support and skip the tests on the broken > > platforms. I'm sort of surprised FreeBSD support is still broken in > > this way though (echoed by Philip) although it could be an issue on > > that particular buildbot. > > If I'm reading the issue correctly, it isn't that it doesn't work, > it's that the concurrent.futures tests fail because they create more > semaphores than the default FreeBSD install supports. Doesn't it suggest a possible resource leak somewhere? Or do the concurrent tests really rely on creating many semaphores? (as opposed to say 5 or 10 of them) Regards Antoine. From tjreedy at udel.edu Wed Dec 29 21:09:07 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 29 Dec 2010 15:09:07 -0500 Subject: [Python-Dev] Compile() and Windows/Mac newlines In-Reply-To: References: Message-ID: On 12/29/2010 2:53 PM, Benjamin Peterson wrote: > Type "help", "copyright", "credits" or "license" for more information. >>>> compile("print(999)\r\n", "blah", "exec") > at 0xb353e8, file "blah", line 1> I made a mistake in testing. Issue closed. Sorry for the noise. -- Terry Jan Reedy From jnoller at gmail.com Wed Dec 29 21:17:32 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 29 Dec 2010 15:17:32 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1B7F37.3080201@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> Message-ID: On Wed, Dec 29, 2010 at 1:34 PM, "Martin v. L?wis" wrote: > Am 29.12.2010 18:54, schrieb Jesse Noller: >> On Wed, Dec 29, 2010 at 10:28 AM, "Martin v. L?wis" wrote: >>>>> I would like to know if it should be considered as a release blocker. >>>>> Georg Brandl said yes on IRC. >>>> >>>> Under the condition that it is within reason to fix it before the >>>> release. >>> >>> What *should* be possible is to disable building >>> SemLock/multiprocessing.synchronize on FreeBSD. As a consequence, >>> multiprocessing locks would stop working on FreeBSD, and concurrent >>> futures; the tests would recognize this lack of features and get >>> skipped. >>> >>> Regards, >>> Martin >> >> The multiprocessing test suite already skips the tests which use the >> (broken) functionality on BSD correctly. This logic needs to be added >> to the concurrent.futures library. > > I'm not so sure that skipping the test is the right approach. Doesn't > that mean that the code will still fail at runtime with > difficult-to-explain messages? I'd rather prefer if the functionality > wasn't available in the first place. > > Also, what specific test are you referring to? > > Regards, > Martin > If the functionality is not supported then users get an import error (within multiprocessing). However, RDM's understanding is correct, and the test is creating more than supported. From victor.stinner at haypocalc.com Wed Dec 29 21:28:26 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Dec 2010 21:28:26 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <20101229210411.4b562b59@pitrou.net> References: <1293628653.16756.13.camel@marge> <20101229191403.A166F239421@kimball.webabinitio.net> <20101229210411.4b562b59@pitrou.net> Message-ID: <1293654506.18690.15.camel@marge> Le mercredi 29 d?cembre 2010 ? 21:04 +0100, Antoine Pitrou a ?crit : > Doesn't it suggest a possible resource leak somewhere? I already checked that: all locks are destroyed correctly on each test. - test_all_completed_some_already_completed() uses 51 SemLock objects - test_first_completed() uses 41 SemLock objects - test_processes_terminate() uses 41 SemLock objects - etc. FreeBSD has an arbitrary limit of 30 semaphores. Victor From martin at v.loewis.de Wed Dec 29 21:42:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Dec 2010 21:42:16 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <3C36C60B-717E-4550-AC2E-AB6E2B2E7ECB@semanchuk.com> References: <1293628653.16756.13.camel@marge> <4D1B570F.1000502@v.loewis.de> <3C36C60B-717E-4550-AC2E-AB6E2B2E7ECB@semanchuk.com> Message-ID: <4D1B9D28.9080801@v.loewis.de> >> If you can make the above change, the question then is what API >> multiprocessing semaphores should be built upon. It seems that you >> are saying that they should use SysV IPC, and only fall back to >> POSIX IPC if SysV IPC doesn't work/exist (are there any platforms >> where this would be the case?) > > Actually, I'd probably recommend the opposite. IMO POSIX IPC is > better designed and easier to work with so it would be my first > choice. "better designed and easier to work with" don't count if you have to do both, anyway. The code will be ugly - so what, once it is written? It's not being exposed to the user. > SysV would be a fallback for when POSIX IPC is unavailable, > broken, incomplete, etc. The problem seems to be that it is not possible to reliably determine these cases. In particular, the ridiculous low limits seem to be difficult to detect, and it's not clear what a required minimum level should be. Regards, Martin From martin at v.loewis.de Wed Dec 29 21:43:38 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Dec 2010 21:43:38 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> Message-ID: <4D1B9D7A.9080003@v.loewis.de> >>> The multiprocessing test suite already skips the tests which use the >>> (broken) functionality on BSD correctly. This logic needs to be added >>> to the concurrent.futures library. >> Also, what specific test are you referring to? Can you kindly point me to the test that skips if broken functionality on BSD is detected? Regards, Martin From martin at v.loewis.de Wed Dec 29 21:49:38 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 29 Dec 2010 21:49:38 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> Message-ID: <4D1B9EE2.70806@v.loewis.de> > If the functionality is not supported then users get an import error > (within multiprocessing). However, RDM's understanding is correct, and > the test is creating more than supported. Hmm. The tests do the absolute minimum stuff that exercises the code; doing anything less, and they would be useless. Of course, one may wonder why test_first_completed manages to create 41 SemLock objects, when all it tries to do is two future calls. So if the minimal test case fails, I'd claim that the module doesn't work on FreeBSD, period. ISTM that Posix IPC is just not a feasible approach to do IPC synchronization on FreeBSD, so it's better to say that multiprocessing is not supported on FreeBSD (until SysV IPC is getting used, hoping that this will fare better). Regards, Martin From jnoller at gmail.com Wed Dec 29 22:34:45 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 29 Dec 2010 16:34:45 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1B9EE2.70806@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> Message-ID: <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> On Dec 29, 2010, at 3:49 PM, "Martin v. L?wis" wrote: >> If the functionality is not supported then users get an import error >> (within multiprocessing). However, RDM's understanding is correct, and >> the test is creating more than supported. > > Hmm. The tests do the absolute minimum stuff that exercises the code; > doing anything less, and they would be useless. Of course, one may > wonder why test_first_completed manages to create 41 SemLock objects, > when all it tries to do is two future calls. > > So if the minimal test case fails, I'd claim that the module doesn't > work on FreeBSD, period. ISTM that Posix IPC is just not a feasible > approach to do IPC synchronization on FreeBSD, so it's better to say > that multiprocessing is not supported on FreeBSD (until SysV IPC is > getting used, hoping that this will fare better). > Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes. From martin at v.loewis.de Wed Dec 29 22:54:41 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 29 Dec 2010 22:54:41 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> Message-ID: <4D1BAE21.5070001@v.loewis.de> Am 29.12.2010 22:34, schrieb Jesse Noller: > > > On Dec 29, 2010, at 3:49 PM, "Martin v. L?wis" wrote: > >>> If the functionality is not supported then users get an import error >>> (within multiprocessing). However, RDM's understanding is correct, and >>> the test is creating more than supported. >> >> Hmm. The tests do the absolute minimum stuff that exercises the code; >> doing anything less, and they would be useless. Of course, one may >> wonder why test_first_completed manages to create 41 SemLock objects, >> when all it tries to do is two future calls. >> >> So if the minimal test case fails, I'd claim that the module doesn't >> work on FreeBSD, period. ISTM that Posix IPC is just not a feasible >> approach to do IPC synchronization on FreeBSD, so it's better to say >> that multiprocessing is not supported on FreeBSD (until SysV IPC is >> getting used, hoping that this will fare better). >> > > Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes. So what do you propose to do? Regards, Martin From jnoller at gmail.com Wed Dec 29 23:13:43 2010 From: jnoller at gmail.com (Jesse Noller) Date: Wed, 29 Dec 2010 17:13:43 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1BAE21.5070001@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> <4D1BAE21.5070001@v.loewis.de> Message-ID: <681D6028-7898-4845-9941-1F8D48C02EEE@gmail.com> On Dec 29, 2010, at 4:54 PM, "Martin v. L?wis" wrote: > Am 29.12.2010 22:34, schrieb Jesse Noller: >> >> >> On Dec 29, 2010, at 3:49 PM, "Martin v. L?wis" wrote: >> >>>> If the functionality is not supported then users get an import error >>>> (within multiprocessing). However, RDM's understanding is correct, and >>>> the test is creating more than supported. >>> >>> Hmm. The tests do the absolute minimum stuff that exercises the code; >>> doing anything less, and they would be useless. Of course, one may >>> wonder why test_first_completed manages to create 41 SemLock objects, >>> when all it tries to do is two future calls. >>> >>> So if the minimal test case fails, I'd claim that the module doesn't >>> work on FreeBSD, period. ISTM that Posix IPC is just not a feasible >>> approach to do IPC synchronization on FreeBSD, so it's better to say >>> that multiprocessing is not supported on FreeBSD (until SysV IPC is >>> getting used, hoping that this will fare better). >>> >> >> Whatever you choose to say Martin. It does work, and is supported to the limitations that FreeBSD imposes. > > So what do you propose to do? > I don't have a good suggestion (or a computer with a keyboard anywhere near me) right now, but making a migration/fallback to SYSV style semaphores a release blocker seems like a mistake to me. I would document the limitation in the futures/multiprocessing documentation, and skip that particular test for now on FreeBSD. FreeBSDs limitations seem arbitrary, but I'm not a FBSD expert. This isn't a bug in either futures or multiprocessing though. From martin at v.loewis.de Wed Dec 29 23:24:32 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 29 Dec 2010 23:24:32 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <681D6028-7898-4845-9941-1F8D48C02EEE@gmail.com> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> <4D1BAE21.5070001@v.loewis.de> <681D6028-7898-4845-9941-1F8D48C02EEE@gmail.com> Message-ID: <4D1BB520.6030608@v.loewis.de> > I don't have a good suggestion (or a computer with a keyboard > anywhere near me) right now, but making a migration/fallback to SYSV > style semaphores a release blocker seems like a mistake to me. And indeed, I don't propose to make that a release blocker. Instead, I propose to disable support for the module (either multiprocessing or concurrent.futures only) on FreeBSD, and make such disabling a release blocker. > I would document the limitation in the futures/multiprocessing > documentation, and skip that particular test for now on FreeBSD. Just skipping the test case seems inappropriate, when it really tests the core functionality of the module in question. The failing test really means that the module does not work on the system. > FreeBSDs limitations seem arbitrary, but I'm not a FBSD expert. This > isn't a bug in either futures or multiprocessing though. I don't claim that it's a bug. I claim that FreeBSD is just not supported (saying that FreeBSD doesn't support the multiprocessing module might be more appropriate). Somebody will have to correctly fix it - either the FreeBSD people, or the multiprocessing authors, or the concurrent.future authors. Meanwhile, we must admit that it just doesn't work (or else the test case would not fail). Regards, Martin From philip at semanchuk.com Wed Dec 29 23:29:32 2010 From: philip at semanchuk.com (Philip Semanchuk) Date: Wed, 29 Dec 2010 17:29:32 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1BB520.6030608@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> <4D1BAE21.5070001@v.loewis.de> <681D6028-7898-4845-9941-1F8D48C02EEE@gmail.com> <4D1BB520.6030608@v.loewis.de> Message-ID: <092A9C27-CC75-4B5A-8731-7F8EF7E3DF68@semanchuk.com> On Dec 29, 2010, at 5:24 PM, Martin v. L?wis wrote: >> I don't have a good suggestion (or a computer with a keyboard >> anywhere near me) right now, but making a migration/fallback to SYSV >> style semaphores a release blocker seems like a mistake to me. > > And indeed, I don't propose to make that a release blocker. Instead, > I propose to disable support for the module (either multiprocessing > or concurrent.futures only) on FreeBSD, and make such disabling a > release blocker. Note that we only know that the test fails on FreeBSD 7.2. The current release is FreeBSD 8.1. The 8.x series might work just fine. I don't know which test is failing under 7.2. Has anyone tested it under 7.3 or 8.x? bye Philip From solipsis at pitrou.net Wed Dec 29 23:33:17 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 29 Dec 2010 23:33:17 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> <4D1BAE21.5070001@v.loewis.de> <681D6028-7898-4845-9941-1F8D48C02EEE@gmail.com> <4D1BB520.6030608@v.loewis.de> Message-ID: <20101229233317.11fc10e2@pitrou.net> On Wed, 29 Dec 2010 23:24:32 +0100 "Martin v. L?wis" wrote: > > I don't have a good suggestion (or a computer with a keyboard > > anywhere near me) right now, but making a migration/fallback to SYSV > > style semaphores a release blocker seems like a mistake to me. > > And indeed, I don't propose to make that a release blocker. Instead, > I propose to disable support for the module (either multiprocessing > or concurrent.futures only) on FreeBSD, and make such disabling a > release blocker. I don't really agree with this. There's no need to explicitly forbid use of multiprocessing from FreeBSD. First, it is not our task to validate that each and every OS conforms to the APIs it claims to implement. Second, such disabling would make life uselessly more complicated for users the day FreeBSD actually fixes their stuff. So, IMO, skipping tests is enough. At worse, multiprocessing can issue a warning when imported under these OSes. Regards Antoine. From db3l.net at gmail.com Wed Dec 29 23:44:51 2010 From: db3l.net at gmail.com (David Bolen) Date: Wed, 29 Dec 2010 17:44:51 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> <4D1BAE21.5070001@v.loewis.de> <681D6028-7898-4845-9941-1F8D48C02EEE@gmail.com> <4D1BB520.6030608@v.loewis.de> Message-ID: "Martin v. L?wis" writes: >> I don't have a good suggestion (or a computer with a keyboard >> anywhere near me) right now, but making a migration/fallback to SYSV >> style semaphores a release blocker seems like a mistake to me. > > And indeed, I don't propose to make that a release blocker. Instead, > I propose to disable support for the module (either multiprocessing > or concurrent.futures only) on FreeBSD, and make such disabling a > release blocker. Or, I'll make the same offer I think I made in the multiprocessing case, which is I can build a kernel on the buildbot with a higher limit, if that's needed just to ensure test completion. Yes, it would also mean that users would need to do the same (or in later FreeBSD releases it can be done dynamically via sysctl), but that didn't seem to be considered a big hurdle in the prior discussion for multiprocessing. That would essentially switch this to a documentation issue, to document that on older FreeBSD platforms a user needs to take some steps, either as a startup configuration, or a kernel rebuild depending on release. That assumes that normal use of the module will need as many semaphores as the tests, but if not, presumably the documentation comments could become more a caution than a requirement. I've also been considering retiring the 6.x buildbot in favor of an 8.x (I only have resources for 2, and even that is slow). Updating the 7.x buildbot would also include dynamic adjustment of the limit, which I think based on the prior discussion her was mentioned as being in 7.3. Though 7.4 releases late in January so could jump right to that. -- David PS: In regards to another comment in this thread, while 7.2 doesn't load the POSIX support by default, the buildbot does have the appropriate loader configuration to include it. PPS: I am about to be traveling on business through 1/10, so unlikely to make any buildbot changes before then. From ncoghlan at gmail.com Wed Dec 29 23:46:54 2010 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 Dec 2010 08:46:54 +1000 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <20101229233317.11fc10e2@pitrou.net> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> <4D1BAE21.5070001@v.loewis.de> <681D6028-7898-4845-9941-1F8D48C02EEE@gmail.com> <4D1BB520.6030608@v.loewis.de> <20101229233317.11fc10e2@pitrou.net> Message-ID: On Thu, Dec 30, 2010 at 8:33 AM, Antoine Pitrou wrote: > On Wed, 29 Dec 2010 23:24:32 +0100 > "Martin v. L?wis" wrote: >> > I don't have a good suggestion (or a computer with a keyboard >> > anywhere near me) right now, but making a migration/fallback to SYSV >> > style semaphores a release blocker seems like a mistake to me. >> >> And indeed, I don't propose to make that a release blocker. Instead, >> I propose to disable support for the module (either multiprocessing >> or concurrent.futures only) on FreeBSD, and make such disabling a >> release blocker. > > I don't really agree with this. There's no need to explicitly forbid > use of multiprocessing from FreeBSD. First, it is not our task to > validate that each and every OS conforms to the APIs it claims to > implement. Second, such disabling would make life uselessly more > complicated for users the day FreeBSD actually fixes their stuff. Also, I believe you can get it to work on FreeBSD 7.2 by fiddling with the arbitrary limits. So +1 for skipping the affected tests on FreeBSD so the buildbot state can tell us something useful (conditionally wrapping them with "expected failure" may be even better, as then we'll find out when the FreeBSD out-of-the-box configuration actually supports running them). Cheers, Nick. -- Nick Coghlan?? |?? ncoghlan at gmail.com?? |?? Brisbane, Australia From victor.stinner at haypocalc.com Wed Dec 29 23:55:48 2010 From: victor.stinner at haypocalc.com (Victor Stinner) Date: Wed, 29 Dec 2010 23:55:48 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1B9EE2.70806@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> Message-ID: <1293663348.18690.163.camel@marge> Le mercredi 29 d?cembre 2010 ? 21:49 +0100, "Martin v. L?wis" a ?crit : > Of course, one may wonder why test_first_completed manages > to create 41 SemLock objects, when all it tries to do is two future > calls. More numbers (on Linux): - Queue: 3 SemLock - Condition: 4 SemLock - Event: 5 SemLock - Call (test_concurrent_futures): 10 SemLock (2 Event) - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition) FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, you can only create *one* Call object, whereas some tests use 4 Call objects or more. Victor From tjreedy at udel.edu Thu Dec 30 00:23:44 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 29 Dec 2010 18:23:44 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> <4D1BAE21.5070001@v.loewis.de> <681D6028-7898-4845-9941-1F8D48C02EEE@gmail.com> <4D1BB520.6030608@v.loewis.de> Message-ID: On 12/29/2010 5:44 PM, David Bolen wrote: > Or, I'll make the same offer I think I made in the multiprocessing > case, which is I can build a kernel on the buildbot with a higher > limit, if that's needed just to ensure test completion. Yes, it would > also mean that users would need to do the same (or in later FreeBSD > releases it can be done dynamically via sysctl), but that didn't seem > to be considered a big hurdle in the prior discussion for > multiprocessing. > > That would essentially switch this to a documentation issue, to document > that on older FreeBSD platforms a user needs to take some steps, either > as a startup configuration, or a kernel rebuild depending on release. Testing on updated buildbots and documenting older BSD limits and fixes seems like the best solution. Martin is right that we really do want the tests to run to catch any real bugs (on our side). -- Terry Jan Reedy From philip at semanchuk.com Thu Dec 30 01:01:44 2010 From: philip at semanchuk.com (Philip Semanchuk) Date: Wed, 29 Dec 2010 19:01:44 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <22257F0D-AE3E-46F5-BE0D-1426D5F908CF@gmail.com> <4D1BAE21.5070001@v.loewis.de> <681D6028-7898-4845-9941-1F8D48C02EEE@gmail.com> <4D1BB520.6030608@v.loewis.de> <20101229233317.11fc10e2@pitrou.net> Message-ID: On Dec 29, 2010, at 5:46 PM, Nick Coghlan wrote: > On Thu, Dec 30, 2010 at 8:33 AM, Antoine Pitrou wrote: >> On Wed, 29 Dec 2010 23:24:32 +0100 >> "Martin v. L?wis" wrote: >>>> I don't have a good suggestion (or a computer with a keyboard >>>> anywhere near me) right now, but making a migration/fallback to SYSV >>>> style semaphores a release blocker seems like a mistake to me. >>> >>> And indeed, I don't propose to make that a release blocker. Instead, >>> I propose to disable support for the module (either multiprocessing >>> or concurrent.futures only) on FreeBSD, and make such disabling a >>> release blocker. >> >> I don't really agree with this. There's no need to explicitly forbid >> use of multiprocessing from FreeBSD. First, it is not our task to >> validate that each and every OS conforms to the APIs it claims to >> implement. Second, such disabling would make life uselessly more >> complicated for users the day FreeBSD actually fixes their stuff. > > Also, I believe you can get it to work on FreeBSD 7.2 by fiddling with > the arbitrary limits. So +1 for skipping the affected tests on FreeBSD > so the buildbot state can tell us something useful (conditionally > wrapping them with "expected failure" may be even better, as then > we'll find out when the FreeBSD out-of-the-box configuration actually > supports running them). A bit of info about FreeBSD 7.2 and 8.0 (the only versions I have installed). My installs are set up explicitly for testing and so they have had very little customization. - `sysctl -a|grep p1003` displays the config values specific to POSIX IPC. I think `sysctl -a|grep ipc` shows values for SysV IPC. It looks like the POSIX 1003_1b names in sysctl are similar to the names here (e.g. SEM_NSEMS_MAX, SEM_VALUE_MAX, etc.) -- http://pubs.opengroup.org/onlinepubs/009695399/basedefs/limits.h.html - Under my 7.2, sysctl.p1003_1b.sem_nsems_max == 0 and I can't create semaphores with my posix_ipc extension. After `kldload sem`, sysctl.p1003_1b.sem_nsems_max == 30 and my posix_ipc semaphore & shared mem demo works. - Under my 8.0, sysctl.p1003_1b.sem_nsems_max == 30 and my posix_ipc semaphore & shared mem demo works even without `kldload sem`. `kldstat` doesn't show sem.ko as loaded, so I assume that POSIX semaphores are now a default part of the kernel. - Under my 7.2 *and* 8.0, sysctl.p1003_1b.mq_open_max == 0 and I can't create a message queue with my posix_ipc extension. After `kldload mqueuefs`, sysctl.p1003_1b.mq_open_max remains 0 (huh?) but my posix_ipc message queue demo works. - Under my 8.0, sysctl.p1003_1b.sem_nsems_max is read only even if I change it in /etc/sysctl.conf. According to this post, sem_nsems_max doesn't become dynamic until 8.1: http://www.gossamer-threads.com/lists/python/dev/875319?do=post_view_threaded Hope this helps a bit Philip From brian at sweetapp.com Thu Dec 30 04:45:44 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 29 Dec 2010 19:45:44 -0800 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <1293663348.18690.163.camel@marge> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> Message-ID: <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> On Dec 29, 2010, at 2:55 PM, Victor Stinner wrote: > Le mercredi 29 d?cembre 2010 ? 21:49 +0100, "Martin v. L?wis" a > ?crit : >> Of course, one may wonder why test_first_completed manages >> to create 41 SemLock objects, when all it tries to do is two future >> calls. > > More numbers (on Linux): > > - Queue: 3 SemLock > - Condition: 4 SemLock > - Event: 5 SemLock > - Call (test_concurrent_futures): 10 SemLock (2 Event) > - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition) > > FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, > you can only create *one* Call object, whereas some tests use 4 Call > objects or more. Great detective work! This would suggest that ProcessPoolExecutors are useable on FreeBSD 7.2 so long as the user doesn't create more than two at once (which probably isn't a big deal for most apps). So skipping the test is probably the way to go. Cheers, Brian > > Victor > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From brian at sweetapp.com Thu Dec 30 04:52:39 2010 From: brian at sweetapp.com (Brian Quinlan) Date: Wed, 29 Dec 2010 19:52:39 -0800 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1B9EE2.70806@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> Message-ID: On Dec 29, 2010, at 12:49 PM, Martin v. L?wis wrote: >> If the functionality is not supported then users get an import error >> (within multiprocessing). However, RDM's understanding is correct, >> and >> the test is creating more than supported. > > Hmm. The tests do the absolute minimum stuff that exercises the code; > doing anything less, and they would be useless. Of course, one may > wonder why test_first_completed manages to create 41 SemLock objects, > when all it tries to do is two future calls. I actually think that my tests may be overdone - in order to probe for specific race conditions they use a lot of locks to force calls to complete in a specific order. I'm thinking about pairing the tests down to only demonstrate basic correctness. This should fix the tests on FreeBSD and Windows. Then, when Python 3.2 is released, I can gradually introduce more comprehensive tests while ensuring that I keep the buildbots green on all supported platforms. Thoughts? Cheers, Brian > > So if the minimal test case fails, I'd claim that the module doesn't > work on FreeBSD, period. ISTM that Posix IPC is just not a feasible > approach to do IPC synchronization on FreeBSD, so it's better to say > that multiprocessing is not supported on FreeBSD (until SysV IPC is > getting used, hoping that this will fare better). > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brian%40sweetapp.com From stephen at xemacs.org Thu Dec 30 06:42:48 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Thu, 30 Dec 2010 14:42:48 +0900 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <20101229152510.1E766239DB5@kimball.webabinitio.net> References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> Message-ID: <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> R. David Murray writes: > We merge bug fixes to 2.7 on a routine basis. It is the rule rather > than the exception, by a long shot. For bugfixes, of course it's routine. I understand that. My point was that the case Amaury fears, where the (new) VCS makes things harder than patching or porting by hand, is likely to be relatively uncommon, sandwiched between the "typo fixed in comment" conflicts (aka "trivial tweaks") and those that require reengineering. Also, while workflow helpers will make a big difference to the non-VCS-geeks (ie, almost all Python developers), properly speaking this isn't really an issue with Mercurial, because all of the methods for this purpose are basically "diff | patch", although the executables are called things like "svn" and "python". They all demand workflow helper scripts to regulate the flow of desired and undesired patches. The difference is that the tool for hg is a SMOP, while that for svn is a SMOEP[1]. > So, since we are going to be maintaining 2.7 for a while, this is > a workflow problem that we *must* solve to make the hg transition > worthwhile. I have no doubt that we will, but I also have no doubt we > need to *solve* it, not just wave our hands. Certainly. I think I already said that, no? My point is simply that while Amaury's expression of his requirements is welcome, and his experimenting with hg is extremely valuable, indeed a necessary part of the transition, everything he describes so far is a known problem that we basically know how to solve. He talks about changes to the workflow, but frankly, I don't see a *need* for that.[2] IMO, changes to the workflow will be driven by kaizen, not by some brave new world revolution (Guido inter alia insisted on that) nor by thumb-in-dike disaster recovery (PEP 374 did a pretty good job on that, if I do say so myself). I wish I had more time to do real work on this (not to mention email, thank *you*, David!), but it seems like every time I start programming, I fall asleep ... and six hours later, it's back to day job or family services. :-/ Footnotes: [1] Simpler Matter Of Existing Program. [2] Aside for a need for establishing which hg commands correspond to which parts of the existing workflow, and perhaps creating helper scripts. Ie, I think the chances are pretty good that most people who have already tried hg are at least a little VCS-geeky, and probably they adjust personal workflow to the new VCS. That will not be so transparent to the "the tool should work for me, not vice-versa" majority. From rdmurray at bitdance.com Thu Dec 30 08:50:46 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 30 Dec 2010 02:50:46 -0500 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20101230075046.B4DBD238F3D@kimball.webabinitio.net> On Thu, 30 Dec 2010 14:42:48 +0900, "Stephen J. Turnbull" wrote: > R. David Murray writes: > > > We merge bug fixes to 2.7 on a routine basis. It is the rule rather > > than the exception, by a long shot. > > For bugfixes, of course it's routine. I understand that. My point > was that the case Amaury fears, where the (new) VCS makes things > harder than patching or porting by hand, is likely to be relatively > uncommon, sandwiched between the "typo fixed in comment" conflicts > (aka "trivial tweaks") and those that require reengineering. I thought Amaury was saying it was harder than svnmerge, not harder than patching by hand (ie: that it *was* patching by hand, including .rej files and the lack of tracking). I also heard Georg say that this should be fixable, and that he's partly fixed the tracking issue, but not the patch/merge issue (and that doing so will require a change in the hg core). > Also, while workflow helpers will make a big difference to the > non-VCS-geeks (ie, almost all Python developers), properly speaking > this isn't really an issue with Mercurial, because all of the methods > for this purpose are basically "diff | patch", although the > executables are called things like "svn" and "python". They all > demand workflow helper scripts to regulate the flow of desired and > undesired patches. The difference is that the tool for hg is a SMOP, > while that for svn is a SMOEP[1]. Well, considering that the transition is "soon", the fact that it is a SMOP is a concern :) > > So, since we are going to be maintaining 2.7 for a while, this is > > a workflow problem that we *must* solve to make the hg transition > > worthwhile. I have no doubt that we will, but I also have no doubt we > > need to *solve* it, not just wave our hands. > > Certainly. I think I already said that, no? My point is simply that Ah, I guess I did miss that, sorry. > while Amaury's expression of his requirements is welcome, and his > experimenting with hg is extremely valuable, indeed a necessary part > of the transition, everything he describes so far is a known problem > that we basically know how to solve. He talks about changes to the > workflow, but frankly, I don't see a *need* for that.[2] Well, there will be *some* workflow change since we're talking about committing to 3.2 maint before the new trunk instead of vice versa. But you are right that that is, mostly, a detail. I'm not as convinced that changes in workflow will be that minimal, though. I don't have much experience with the dvcses yet, but what experience I have had leads me to think that understanding the DAG is a non-trivial and necessary mental leap and does affect the workflow. I don't feel as though I've made that leap yet. Hopefully Brett will be able to document this in the Python context so that the *required* leap will be much smaller. > IMO, changes to the workflow will be driven by kaizen, not by some > brave new world revolution (Guido inter alia insisted on that) nor by > thumb-in-dike disaster recovery (PEP 374 did a pretty good job on > that, if I do say so myself). Well, I hope you are right. I'm looking forward to the new version of the test repository and doing some real world experiments. > I wish I had more time to do real work on this (not to mention email, > thank *you*, David!), but it seems like every time I start You are welcome; thanks for the feedback. (I sometimes feel like I'm working in a bit of a vacuum, though Barry does chime in occasionally...but I do realize that people are busy; that's why I inherited this job in the first place, after all :) -- R. David Murray www.bitdance.com From martin at v.loewis.de Thu Dec 30 10:16:43 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Dec 2010 10:16:43 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> Message-ID: <4D1C4DFB.30302@v.loewis.de> Am 30.12.2010 04:45, schrieb Brian Quinlan: > On Dec 29, 2010, at 2:55 PM, Victor Stinner wrote: > >> Le mercredi 29 d?cembre 2010 ? 21:49 +0100, "Martin v. L?wis" a ?crit : >>> Of course, one may wonder why test_first_completed manages >>> to create 41 SemLock objects, when all it tries to do is two future >>> calls. >> >> More numbers (on Linux): >> >> - Queue: 3 SemLock >> - Condition: 4 SemLock >> - Event: 5 SemLock >> - Call (test_concurrent_futures): 10 SemLock (2 Event) >> - ProcessPoolExecutor: 11 SemLock (2 Queue, 1 Condition) >> >> FreeBSD 7.2 is limited to 30 semaphores, so with ProcessPoolExecutor, >> you can only create *one* Call object, whereas some tests use 4 Call >> objects or more. > > Great detective work! This would suggest that ProcessPoolExecutors are > useable on FreeBSD 7.2 so long as the user doesn't create more than two > at once (which probably isn't a big deal for most apps). IIUC, this is a system-wide limit. So if you run any non-trivial app a number of times will quickly cause failure. This is just not practical. > So skipping the test is probably the way to go. I'm still -1 on that proposal. Regards, Martin From lukasz at langa.pl Thu Dec 30 16:04:16 2010 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Thu, 30 Dec 2010 16:04:16 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1C4DFB.30302@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> Message-ID: Wiadomo?? napisana przez Martin v. L?wis w dniu 2010-12-30, o godz. 10:16: > Am 30.12.2010 04:45, schrieb Brian Quinlan: > >> So skipping the test is probably the way to go. > > I'm still -1 on that proposal. I agree with Martin, explanation follows. In general, I'm trying to think as the user: someone wrote a program and said it requires Python 3.2. I expect it to work on Python 3.2 available on my platform. Doesn't matter if it's bundled with the OS or custom built. When a module is not available, this should fail loudly and as fast as possible. This is currently the case with conditionally built modules like curses, sqlite3, tkinter and others. From the user's perspective: the import fails. From the administrator's perspective: the build reports skipped modules. The admin can then alter the environment to make the build behave as expected, and then run the tests to confirm it works. From an OS maintainer perspective it's even better: a module can be made available optionally with explicitly set dependencies and required configuration. In our specific case a bunch of things should be determined first: 1. Does it still fail on FreeBSD 7.3+? 2. Why is the semaphore limit so low in the first place? 3. What is a reasonable number for that limit? That should include situations where people run multiple applications. 4. What other configuration should be done to ensure the module won't break on runtime. Some answers Philip gave already. Knowing all these things would let us decide whether switching the module off on systems that don't meet the requirements is okay and can we get away with just documenting how to make it work. One problem is that the administrator can make the OS compatible/incompatible on runtime (with sysctl). ISTM concurrent.futures was included a bit too early in the distribution. The module has problems not only on BSD [1]. Fortunately, there is visible action from Brian, Philip and David to make the problems go away. I'm sure it will be fixed pretty soon. Ultimately I also agree with Martin that an implementation using SysV IPC would probably be better from the user's perspective. As he said, even if the underlying code is not as pretty as with the POSIX APIs, once it's written and tested nobody cares because it's not exposed. However, it's far too late to change that now for 3.2. [1] On RedHat/CentOS the test freezes badly (issue 10517). -- Best regards, ?ukasz Langa tel. +48 791 080 144 WWW http://lukasz.langa.pl/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Dec 30 16:40:14 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 30 Dec 2010 16:40:14 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> Message-ID: <20101230164014.218a1f27@pitrou.net> On Thu, 30 Dec 2010 16:04:16 +0100 ?ukasz Langa wrote: > > Some answers Philip gave already. Knowing all these things would let us decide whether switching the module off on systems that don't meet the requirements is okay and can we get away with just documenting how to make it work. I really don't think it is our job to maintain a list of OS/versions which work and don't work. At best, multiprocessing when imported could try to create a single semaphore, and raise an ImportError if such creating fails (which it does, IIUC, under some old FreeBSDs without a specific kernel tweak). > ISTM concurrent.futures was included a bit too early in the distribution. > The module has problems not only on BSD [1]. I'm not sure concurrent.futures is the culprit, rather than multiprocessing itself (or perhaps even some core Python functionality). Actually, the threading-based part of concurrent.futures probably works fine. Regards Antoine. From martin at v.loewis.de Thu Dec 30 17:05:05 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 30 Dec 2010 17:05:05 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <20101230164014.218a1f27@pitrou.net> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> <20101230164014.218a1f27@pitrou.net> Message-ID: <4D1CADB1.6060803@v.loewis.de> Am 30.12.2010 16:40, schrieb Antoine Pitrou: > On Thu, 30 Dec 2010 16:04:16 +0100 > ?ukasz Langa wrote: >> >> Some answers Philip gave already. Knowing all these things would let us decide whether switching the module off on systems that don't meet the requirements is okay and can we get away with just documenting how to make it work. > > I really don't think it is our job to maintain a list of OS/versions > which work and don't work. Of course not. I would propose a dynamic test: check how many POSIX semaphores the installation supports, and fail if it's less than 200 (say). >> ISTM concurrent.futures was included a bit too early in the distribution. >> The module has problems not only on BSD [1]. > > I'm not sure concurrent.futures is the culprit, rather than > multiprocessing itself (or perhaps even some core Python functionality). > Actually, the threading-based part of concurrent.futures probably works > fine. Well, "the culprit" really is FreeBSD. However, concurrent.futures apparently makes freehanded use of semaphores, in a way that the FreeBSD authors didn't expect them to be used (if they expected them to be used at all, that is). Regards, Martin From solipsis at pitrou.net Thu Dec 30 17:07:47 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 30 Dec 2010 17:07:47 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1CADB1.6060803@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> <20101230164014.218a1f27@pitrou.net> <4D1CADB1.6060803@v.loewis.de> Message-ID: <1293725267.3660.3.camel@localhost.localdomain> > > I'm not sure concurrent.futures is the culprit, rather than > > multiprocessing itself (or perhaps even some core Python functionality). > > Actually, the threading-based part of concurrent.futures probably works > > fine. > > Well, "the culprit" really is FreeBSD. However, concurrent.futures > apparently makes freehanded use of semaphores, in a way that the > FreeBSD authors didn't expect them to be used (if they expected them > to be used at all, that is). It seems to be multiprocessing as much as concurrent.futures itself. Apparently a basic multiprocessing queue already uses two (interprocess) locks and one semaphore. And, again, the threading-based API in concurrent.futures should work fine anyway. Do you suggest we selectively disable the process-based API? Regards Antoine. From martin at v.loewis.de Thu Dec 30 17:50:23 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Dec 2010 17:50:23 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <1293725267.3660.3.camel@localhost.localdomain> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> <20101230164014.218a1f27@pitrou.net> <4D1CADB1.6060803@v.loewis.de> <1293725267.3660.3.camel@localhost.localdomain> Message-ID: <4D1CB84F.403@v.loewis.de> > And, again, the threading-based API in concurrent.futures should work > fine anyway. Do you suggest we selectively disable the process-based > API? Yes. Importing concurrent.futures.process should fail. Unfortunately, it's imported from __init__.py, so either we change the API to move the executors to the submodules, or we let creation of process pools fail (rather than the import). Regards, Martin From stephen at xemacs.org Thu Dec 30 18:54:26 2010 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Fri, 31 Dec 2010 02:54:26 +0900 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <20101230075046.B4DBD238F3D@kimball.webabinitio.net> References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> <20101230075046.B4DBD238F3D@kimball.webabinitio.net> Message-ID: <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> R. David Murray writes: > I thought Amaury was saying it was harder than svnmerge, not harder > than patching by hand (ie: that it *was* patching by hand, including > .rej files and the lack of tracking). I also heard Georg say that this > should be fixable, and that he's partly fixed the tracking issue, I don't see why the tracking issue is any different than it would be for svn. If you're actually merging, either a dummy merge (what git calls an "ours merge") or reverting unwanted patches will "block" them, and hg will keep track of the ones that have been merged. If you're actually cherry-picking, then you have to keep a separate database of the patches that have been picked and those that are blocked, but this has been done successfully for years with svnmerge, right? > but not the patch/merge issue (and that doing so will require a change > in the hg core). I don't think so. For merges, there isn't a problem. For cherypicking, I haven't thought carefully about this, but ISTM that "hg export | hg import; merge post-patch /dev/null source" should give the traditional conflict markers. This will require a bit of care to get the files to merge right, since there will in general be multiple files that fail to patch, but the names can be fished out of the .rej file(s). Getting the source files will also be mildly tricky, since they live in a different branch, and aren't available in your local repository. It will also require a bit of care to get the commit log etc right. But I don't think it's that hard, conceptually. Of course it's preferable to get this feature in hg itself, but I don't think we need to wait for hg/maintain a fork. > Well, considering that the transition is "soon", the fact that it > is a SMOP is a concern :) Sure, but in this crowd, I wouldn't waste a good worry on this particular SMOP. > Well, there will be *some* workflow change since we're talking about > committing to 3.2 maint before the new trunk instead of vice versa. > But you are right that that is, mostly, a detail. AFAIK that's not hg-driven, though. > I'm not as convinced that changes in workflow will be that minimal, > though. I don't have much experience with the dvcses yet, but what > experience I have had leads me to think that understanding the DAG is > a non-trivial and necessary mental leap and does affect the workflow. Yes, no, and yes. That is, although understanding it is nontrivial, and once you do it will affect your workflow, it is unnecessary. The Emacs crew have proved that to my satisfaction; there are a bunch of folks there who understand DAGs quite well, but there are also a bunch who just don't want to know -- and they're doing fine. True, making the most of dVCS requires "understanding the DAG." Adapting an existing complex workflow to a new dVCS's commands also requires understanding the DAG. But using the adapted workflow simply requires learning new names for old operations. Annoying, but it will make a fair number of core devs quite happy. > Well, I hope you are right. I'm looking forward to the new version of > the test repository and doing some real world experiments. Yup. It's always risky to predict, especially the future, but I'm pretty confident that things will work out. We do need to (1) make sure we do everything for hg that we've always done for svn, (2) work out some features that hg doesn't have yet (Windows EOLs), and (3) listen carefully to those who are testing out the new repository -- there are always surprises in this kind of thing. But Python does that kind of thing very well. From martin at v.loewis.de Thu Dec 30 19:17:08 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 30 Dec 2010 19:17:08 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> Message-ID: <4D1CCCA4.5000200@v.loewis.de> > 1. Does it still fail on FreeBSD 7.3+? Yes, it still fails. The limits (30 semaphores) haven't changed. It also remains untunable. > 2. Why is the semaphore limit so low in the first place? I don't know - (Free)BSD is in the tradition of disliking SysV inventions, and POSIX inventions unless they originate from FreeBSD. This is polemics, of course :-) In any case, SysV IPC doesn't seem much better. In SysV, you allocate semaphores in sets. In FreeBSD 8.1, in the standard setting, you can have up to 10 semaphore identifiers, with up to 60 semaphores each. That may sound reasonable, except that there is also a limit on the total number of semaphores of 60. So it seems that switching to SysV IPC wouldn't actually improve things (except that these are tunable parameters already, so changing them requires only a reboot, not a recompile of the kernel). > 3. What is a reasonable number for that limit? That should include > situations where people run multiple applications. POSIX specifies that the minimum acceptable value for the SEM_NSEMS_MAX setting is _POSIX_SEM_NSEMS_MAX, which in turn must be at least 256. We could argue that we refuse to use POSIX semaphores if the system doesn't conform to POSIX, i.e. has semaphore limit of less than 256. Regards, Martin From martin at v.loewis.de Thu Dec 30 19:31:46 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Dec 2010 19:31:46 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> <20101230075046.B4DBD238F3D@kimball.webabinitio.net> <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <4D1CD012.3010407@v.loewis.de> Am 30.12.2010 18:54, schrieb Stephen J. Turnbull: > R. David Murray writes: > > > I thought Amaury was saying it was harder than svnmerge, not harder > > than patching by hand (ie: that it *was* patching by hand, including > > .rej files and the lack of tracking). I also heard Georg say that this > > should be fixable, and that he's partly fixed the tracking issue, > > I don't see why the tracking issue is any different than it would be > for svn. If you're actually merging, either a dummy merge (what git > calls an "ours merge") or reverting unwanted patches will "block" > them, and hg will keep track of the ones that have been merged. Are you still talking about merges from 3.x to 2.7? Then I don't think what you say is actually true. hg will *not* be able to track the ones that get merged, and will *not* be able to block by means of dummy merges. Perhaps we aren't "actually" merging, I suppose. > If > you're actually cherry-picking, then you have to keep a separate > database of the patches that have been picked and those that are > blocked, but this has been done successfully for years with svnmerge, > right? [Ok. So "cherry-picking" is not a special case of "merging", or "actually merging", I presume] Wrong. For subversion, merge tracking is built into the tools that we use for merging (i.e. svnmerge). For hg (IIUC), the standard procedure that people use for merging (or "cherry-picking") apparently does not track what has been merged (or "cherry-picked"). > > Well, considering that the transition is "soon", the fact that it > > is a SMOP is a concern :) > > Sure, but in this crowd, I wouldn't waste a good worry on this > particular SMOP. Given how little enthusiasm this crowd has in actually helping the migration, I doubt they show any enthusiasm in writing such tools. > > Well, there will be *some* workflow change since we're talking about > > committing to 3.2 maint before the new trunk instead of vice versa. > > But you are right that that is, mostly, a detail. > > AFAIK that's not hg-driven, though. It is. So far, we have *always* merged from the development branch to the maintenance branch (i.e. "backported"), although there has been opposition from a number of committers to this workflow in recent years. I personally consider this more appropriate: with backporting, you can defer decision to backport until the original change has been confirmed as correct. With constant forward-porting, you risk breaking your maintenance branch with incorrect changes. > > > I'm not as convinced that changes in workflow will be that minimal, > > though. I don't have much experience with the dvcses yet, but what > > experience I have had leads me to think that understanding the DAG is > > a non-trivial and necessary mental leap and does affect the workflow. > > But using the adapted workflow simply requires learning new names for > old operations. Annoying, but it will make a fair number of core devs > quite happy. I think the new workflow will simply result in (even) less care for the maintenance branches than currently. Some people just refuse to patch multiple branches, and will continue to do so. While it was previously possible to backport, it won't be that easy anymore in the future, so it just won't be done. This actually reduces workload, but also reduces quality. Regards, Martin From rdmurray at bitdance.com Thu Dec 30 20:16:14 2010 From: rdmurray at bitdance.com (R. David Murray) Date: Thu, 30 Dec 2010 14:16:14 -0500 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> <20101230075046.B4DBD238F3D@kimball.webabinitio.net> <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <20101230191614.9402E239FEA@kimball.webabinitio.net> On Fri, 31 Dec 2010 02:54:26 +0900, "Stephen J. Turnbull" wrote: > I don't see why the tracking issue is any different than it would be > for svn. If you're actually merging, either a dummy merge (what git Martin already said most of what I would have in response to your post. > For cherypicking, I haven't thought carefully about this, but ISTM > that "hg export | hg import; merge post-patch /dev/null source" should > give the traditional conflict markers. This will require a bit of > care to get the files to merge right, since there will in general be > multiple files that fail to patch, but the names can be fished out of > the .rej file(s). Getting the source files will also be mildly > tricky, since they live in a different branch, and aren't available in > your local repository. It will also require a bit of care to get the > commit log etc right. But I don't think it's that hard, conceptually. > > Of course it's preferable to get this feature in hg itself, but I > don't think we need to wait for hg/maintain a fork. The fact that I really haven't a clue what you are talking about here means that I for one am not likely to be helping develop that tool, at least not any time soon. So I hope there are people who understand this stuff who will make it work for the rest of us. > > Well, considering that the transition is "soon", the fact that it > > is a SMOP is a concern :) > > Sure, but in this crowd, I wouldn't waste a good worry on this > particular SMOP. Talent is one thing, available time, as you pointed out about yourself, is a different matter. I'm confident we can make this all work. The only question is when. -- R. David Murray www.bitdance.com From merwok at netwok.org Thu Dec 30 20:57:11 2010 From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=) Date: Thu, 30 Dec 2010 20:57:11 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: Message-ID: <4D1CE417.8070001@netwok.org> Hi, One thing confuses me in this thread: Do the problems come from merging across different versions (i.e. different syntax and module names), or are they regular problems that come up because people don?t want to use a merge tool? I for one immensely like Mercurial?s merge and utterly dislike Subversion?s (certainly because I?ve learned a good merge tool and don?t have a good editor to handle files with conflict markers). > There is surely a better way to work with maintenance branches! > PEP374 suggests to first modify the oldest release, but this does not > seems right to me (I have three reasons in mind) Can you give them? Regards From solipsis at pitrou.net Thu Dec 30 21:14:31 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 30 Dec 2010 21:14:31 +0100 Subject: [Python-Dev] Backport troubles with mercurial References: <4D1CE417.8070001@netwok.org> Message-ID: <20101230211431.67f5648e@pitrou.net> On Thu, 30 Dec 2010 20:57:11 +0100 ?ric Araujo wrote: > Hi, > > One thing confuses me in this thread: Do the problems come from merging > across different versions (i.e. different syntax and module names), or > are they regular problems that come up because people don?t want to use > a merge tool? I for one immensely like Mercurial?s merge and utterly > dislike Subversion?s (certainly because I?ve learned a good merge tool > and don?t have a good editor to handle files with conflict markers). IIUC this is not about merges, this is about transplants (or cherry-picking). It would be unannoying to lose svnmerge's relative convenience in that matter. Regards Antoine. From benjamin at python.org Thu Dec 30 21:24:31 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 30 Dec 2010 14:24:31 -0600 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <20101230211431.67f5648e@pitrou.net> References: <4D1CE417.8070001@netwok.org> <20101230211431.67f5648e@pitrou.net> Message-ID: 2010/12/30 Antoine Pitrou : > On Thu, 30 Dec 2010 20:57:11 +0100 > ?ric Araujo wrote: >> Hi, >> >> One thing confuses me in this thread: Do the problems come from merging >> across different versions (i.e. different syntax and module names), or >> are they regular problems that come up because people don?t want to use >> a merge tool? ?I for one immensely like Mercurial?s merge and utterly >> dislike Subversion?s (certainly because I?ve learned a good merge tool >> and don?t have a good editor to handle files with conflict markers). > > IIUC this is not about merges, this is about transplants (or > cherry-picking). It would be unannoying to lose svnmerge's relative > convenience in that matter. I assume you meant "annoying"? -- Regards, Benjamin From solipsis at pitrou.net Thu Dec 30 21:31:10 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 30 Dec 2010 21:31:10 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: <4D1CE417.8070001@netwok.org> <20101230211431.67f5648e@pitrou.net> Message-ID: <1293741070.14809.3.camel@localhost.localdomain> Le jeudi 30 d?cembre 2010 ? 14:24 -0600, Benjamin Peterson a ?crit : > 2010/12/30 Antoine Pitrou : > > On Thu, 30 Dec 2010 20:57:11 +0100 > > ?ric Araujo wrote: > >> Hi, > >> > >> One thing confuses me in this thread: Do the problems come from merging > >> across different versions (i.e. different syntax and module names), or > >> are they regular problems that come up because people don?t want to use > >> a merge tool? I for one immensely like Mercurial?s merge and utterly > >> dislike Subversion?s (certainly because I?ve learned a good merge tool > >> and don?t have a good editor to handle files with conflict markers). > > > > IIUC this is not about merges, this is about transplants (or > > cherry-picking). It would be unannoying to lose svnmerge's relative > > convenience in that matter. > > I assume you meant "annoying"? Well, yes. I should stop twisting my negatives (whatever it means). Antoine. From ethan at stoneleaf.us Thu Dec 30 21:36:47 2010 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 30 Dec 2010 12:36:47 -0800 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1CB84F.403@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> <20101230164014.218a1f27@pitrou.net> <4D1CADB1.6060803@v.loewis.de> <1293725267.3660.3.camel@localhost.localdomain> <4D1CB84F.403@v.loewis.de> Message-ID: <4D1CED5F.7090607@stoneleaf.us> Martin v. L?wis wrote: >> And, again, the threading-based API in concurrent.futures should work >> fine anyway. Do you suggest we selectively disable the process-based >> API? > > Yes. Importing concurrent.futures.process should fail. If I understand correctly, it is possible to adjust BSD so that this will work -- as a user I would much rather be informed of that, even as just a caution, than to have Python not have the functionality even after I had fixed the OS problems. Mind you, I have no idea how hard it would be to make this happen in the module in an intelligent way. ~Ethan~ From solipsis at pitrou.net Thu Dec 30 21:43:13 2010 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 30 Dec 2010 21:43:13 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> <20101230164014.218a1f27@pitrou.net> <4D1CADB1.6060803@v.loewis.de> <1293725267.3660.3.camel@localhost.localdomain> <4D1CB84F.403@v.loewis.de> <4D1CED5F.7090607@stoneleaf.us> Message-ID: <20101230214313.710e6b55@pitrou.net> On Thu, 30 Dec 2010 12:36:47 -0800 Ethan Furman wrote: > Martin v. L?wis wrote: > >> And, again, the threading-based API in concurrent.futures should work > >> fine anyway. Do you suggest we selectively disable the process-based > >> API? > > > > Yes. Importing concurrent.futures.process should fail. > > If I understand correctly, it is possible to adjust BSD so that this > will work -- as a user I would much rather be informed of that, even as > just a caution, than to have Python not have the functionality even > after I had fixed the OS problems. > > Mind you, I have no idea how hard it would be to make this happen in the > module in an intelligent way. As I said, we can just emit a Warning instead of raising an exception. Regards Antoine. From g.brandl at gmx.net Thu Dec 30 22:05:52 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 30 Dec 2010 22:05:52 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <4D1CD012.3010407@v.loewis.de> References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> <20101230075046.B4DBD238F3D@kimball.webabinitio.net> <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> <4D1CD012.3010407@v.loewis.de> Message-ID: Am 30.12.2010 19:31, schrieb "Martin v. L?wis": >> But using the adapted workflow simply requires learning new names for >> old operations. Annoying, but it will make a fair number of core devs >> quite happy. > > I think the new workflow will simply result in (even) less care for the > maintenance branches than currently. Some people just refuse to > patch multiple branches, and will continue to do so. While it was > previously possible to backport, it won't be that easy anymore in > the future, so it just won't be done. I refuse to believe that we cannot make our committers (in particular the active ones, which is quite a small percentage) follow such simple rules, especially when they have to learn a new VCS anyway... Also, it's not really necessary for everyone to merge every change from maintenance to trunk: a) they can do multiple commits on the same subject and merge once, and b) if the change is small and no problems can be expected from merging, merging can also be done by others, just like the mass merging we had once from trunk -> py3k (just more convenient for the one doing the merge). Georg From g.brandl at gmx.net Thu Dec 30 22:07:51 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 30 Dec 2010 22:07:51 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1CCCA4.5000200@v.loewis.de> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> <4D1CCCA4.5000200@v.loewis.de> Message-ID: Am 30.12.2010 19:17, schrieb "Martin v. L?wis": >> 1. Does it still fail on FreeBSD 7.3+? > > Yes, it still fails. The limits (30 semaphores) haven't > changed. It also remains untunable. > >> 2. Why is the semaphore limit so low in the first place? > > I don't know - (Free)BSD is in the tradition of disliking > SysV inventions, and POSIX inventions unless they originate from > FreeBSD. This is polemics, of course :-) BTW - can anyone contribute data points from other *BSDs? Georg From martin at v.loewis.de Thu Dec 30 22:38:29 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Dec 2010 22:38:29 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> <20101230075046.B4DBD238F3D@kimball.webabinitio.net> <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> <4D1CD012.3010407@v.loewis.de> Message-ID: <4D1CFBD5.1010109@v.loewis.de> > Also, it's not really necessary for everyone to merge every change from > maintenance to trunk: a) they can do multiple commits on the same > subject and merge once, and b) if the change is small and no problems can > be expected from merging, merging can also be done by others, just like > the mass merging we had once from trunk -> py3k (just more convenient for > the one doing the merge). But you can't use Mercurial's merge functionality for that, right? You have to use some kind of transplant/cherry-picking to merge from the 3.3 branch to the 3.2 branch, right? Regards, Martin From martin at v.loewis.de Thu Dec 30 22:40:28 2010 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 30 Dec 2010 22:40:28 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <4D1CE417.8070001@netwok.org> References: <4D1CE417.8070001@netwok.org> Message-ID: <4D1CFC4C.60606@v.loewis.de> Am 30.12.2010 20:57, schrieb ?ric Araujo: > Hi, > > One thing confuses me in this thread: Do the problems come from merging > across different versions (i.e. different syntax and module names), or > are they regular problems that come up because people don?t want to use > a merge tool? Neither nor. They come from "hg merge" being useless when it comes to having code that is meant to live both on 2.7, 3.2 (perhaps) and 3.3. Regards, Martin From martin at v.loewis.de Thu Dec 30 22:42:16 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Dec 2010 22:42:16 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: <4D1CED5F.7090607@stoneleaf.us> References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> <20101230164014.218a1f27@pitrou.net> <4D1CADB1.6060803@v.loewis.de> <1293725267.3660.3.camel@localhost.localdomain> <4D1CB84F.403@v.loewis.de> <4D1CED5F.7090607@stoneleaf.us> Message-ID: <4D1CFCB8.2020303@v.loewis.de> Am 30.12.2010 21:36, schrieb Ethan Furman: > Martin v. L?wis wrote: >>> And, again, the threading-based API in concurrent.futures should work >>> fine anyway. Do you suggest we selectively disable the process-based >>> API? >> >> Yes. Importing concurrent.futures.process should fail. > > If I understand correctly, it is possible to adjust BSD so that this > will work -- as a user I would much rather be informed of that, even as > just a caution, than to have Python not have the functionality even > after I had fixed the OS problems. That doesn't exclude each other. The module will fail on default (misconfigured) systems, and work on correctly-configured systems. See my patch for details. Regards, Martin From g.brandl at gmx.net Thu Dec 30 22:44:47 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 30 Dec 2010 22:44:47 +0100 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: <4D1CFBD5.1010109@v.loewis.de> References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> <20101230075046.B4DBD238F3D@kimball.webabinitio.net> <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> <4D1CD012.3010407@v.loewis.de> <4D1CFBD5.1010109@v.loewis.de> Message-ID: Am 30.12.2010 22:38, schrieb "Martin v. L?wis": >> Also, it's not really necessary for everyone to merge every change from >> maintenance to trunk: a) they can do multiple commits on the same >> subject and merge once, and b) if the change is small and no problems can >> be expected from merging, merging can also be done by others, just like >> the mass merging we had once from trunk -> py3k (just more convenient for >> the one doing the merge). > > But you can't use Mercurial's merge functionality for that, right? > You have to use some kind of transplant/cherry-picking to merge from the > 3.3 branch to the 3.2 branch, right? Oh, I wrote the above assuming 3.2->3.3 merging. For the other direction you need cherry-picking, yes. Georg From martin at v.loewis.de Thu Dec 30 23:19:00 2010 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 30 Dec 2010 23:19:00 +0100 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD In-Reply-To: References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> <4D1CCCA4.5000200@v.loewis.de> Message-ID: <4D1D0554.1090002@v.loewis.de> > BTW - can anyone contribute data points from other *BSDs? I don't have an installation of OpenBSD, but... In FreeBSD, POSIX semaphores are implemented in sys/kern/uipc_sem.c. In http://www.openbsd.org/cgi-bin/cvsweb/src/sys/kern/ that file doesn't exist. Also, in FreeBSD's limits.h, _POSIX_SEM_NSEMS_MAX is defined (surprisingly to 256); in http://www.openbsd.org/cgi-bin/cvsweb/~checkout~/src/include/limits.h?rev=1.15;content-type=text/plain this constant doesn't appear. So ISTM that OpenBSD doesn't implement POSIX semaphores. IIUC, this means that the multiprocessing module won't be fully functional, and its tests (and the concurrent.futures tests) will be skipped. NetBSD apparently supports sem_open since 2.0: http://netbsd.gw.com/cgi-bin/man-cgi?sem_open++NetBSD-current According to http://cvsweb.netbsd.org/bsdweb.cgi/src/sys/kern/uipc_sem.c?rev=1.22&content-type=text/x-cvsweb-markup&only_with_tag=MAIN SEM_MAX is 128 since 2007, and dynamically adjustable (no reboot). Regards, Martin From tjreedy at udel.edu Thu Dec 30 23:33:20 2010 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 30 Dec 2010 17:33:20 -0500 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> <20101230075046.B4DBD238F3D@kimball.webabinitio.net> <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> <4D1CD012.3010407@v.loewis.de> <4D1CFBD5.1010109@v.loewis.de> Message-ID: On 12/30/2010 4:44 PM, Georg Brandl wrote: >> But you can't use Mercurial's merge functionality for that, right? >> You have to use some kind of transplant/cherry-picking to merge from the >> 3.3 branch to the 3.2 branch, right? > > Oh, I wrote the above assuming 3.2->3.3 merging. For the other direction > you need cherry-picking, yes. I have the impression that Benjamin plans to continue 3.1 bug maintenance for months after 3.2 final rather than issue the last bug fix the traditional 1 week after. That would make the sequence 3.1->3.2->3.3 or 3.1<-3.2<-3.3 or ... . The transition would be a lot easier, I think, if 3.1.4 was released along with 3.2, so most of us would never have to bother with 3.1 and hg together. -- Terry Jan Reedy From g.brandl at gmx.net Thu Dec 30 23:42:19 2010 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 30 Dec 2010 23:42:19 +0100 Subject: [Python-Dev] Demo is gone Message-ID: According to the consensus (and loosely following the Google spreadsheet I created for that purpose), I have removed the Demo directory from py3k. Most demos have been deleted; some have been moved into Tools or into the docs as an example. If I removed something you think should have stayed, please speak up. I kept a few less-useful ones in a new directory called Tools/demo that aims to give a broad overview of what you can do with a few lines of Python. Please feel free to remove and/or add demos there, if you find better ones. Georg From benjamin at python.org Thu Dec 30 23:49:41 2010 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 30 Dec 2010 16:49:41 -0600 Subject: [Python-Dev] Backport troubles with mercurial In-Reply-To: References: <87wrmsk6r7.fsf@uwakimon.sk.tsukuba.ac.jp> <20101229152510.1E766239DB5@kimball.webabinitio.net> <87tyhvke93.fsf@uwakimon.sk.tsukuba.ac.jp> <20101230075046.B4DBD238F3D@kimball.webabinitio.net> <87lj37jgdp.fsf@uwakimon.sk.tsukuba.ac.jp> <4D1CD012.3010407@v.loewis.de> <4D1CFBD5.1010109@v.loewis.de> Message-ID: 2010/12/30 Terry Reedy : > On 12/30/2010 4:44 PM, Georg Brandl wrote: > >>> But you can't use Mercurial's merge functionality for that, right? >>> You have to use some kind of transplant/cherry-picking to merge from the >>> 3.3 branch to the 3.2 branch, right? >> >> Oh, I wrote the above assuming 3.2->3.3 merging. ?For the other direction >> you need cherry-picking, yes. > > I have the impression that Benjamin plans to continue 3.1 bug maintenance > for months after 3.2 final rather than issue the last bug fix the > traditional 1 week after. That would make the sequence 3.1->3.2->3.3 or > 3.1<-3.2<-3.3 or ... . The transition would be a lot easier, I think, if > 3.1.4 was released along with 3.2, so most of us would never have to bother > with 3.1 and hg together. I will try to release 3.1.4 soon after 3.2 is released, but February and the spring in general will be quite busy for me. -- Regards, Benjamin From db3l.net at gmail.com Fri Dec 31 01:36:44 2010 From: db3l.net at gmail.com (David Bolen) Date: Thu, 30 Dec 2010 19:36:44 -0500 Subject: [Python-Dev] Issue #10348: concurrent.futures doesn't work on BSD References: <1293628653.16756.13.camel@marge> <4D1B5394.10802@v.loewis.de> <4D1B7F37.3080201@v.loewis.de> <4D1B9EE2.70806@v.loewis.de> <1293663348.18690.163.camel@marge> <1FC42F4E-BFAE-47E0-984F-0040AACD6804@sweetapp.com> <4D1C4DFB.30302@v.loewis.de> <4D1CCCA4.5000200@v.loewis.de> Message-ID: "Martin v. L?wis" writes: >> 1. Does it still fail on FreeBSD 7.3+? > > Yes, it still fails. The limits (30 semaphores) haven't > changed. It also remains untunable. Yeah, my recollection about 7.3 appears to have been remembering when the kernel module was included by default as opposed to needing to be explicitly loaded. >From the older discussion (http://mail.python.org/pipermail/python-dev/2010-November/105380.html) it would appear that 7.x remains fixed sans a kernel build (not necessarily a big deal for FreeBSD users), but 8.1+ can be tuned with sysctl. -- David From cesare.di.mauro at gmail.com Fri Dec 31 08:02:20 2010 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Fri, 31 Dec 2010 08:02:20 +0100 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: References: Message-ID: 2010/12/28 Lukas Lueg > Consider the following code: > > def foobar(x): > for i in range(5): > x[i] = i > > The bytecode in python 2.7 is the following: > > 2 0 SETUP_LOOP 30 (to 33) > 3 LOAD_GLOBAL 0 (range) > 6 LOAD_CONST 1 (5) > 9 CALL_FUNCTION 1 > 12 GET_ITER > >> 13 FOR_ITER 16 (to 32) > 16 STORE_FAST 1 (i) > > 3 19 LOAD_FAST 1 (i) > 22 LOAD_FAST 0 (x) > 25 LOAD_FAST 1 (i) > 28 STORE_SUBSCR > 29 JUMP_ABSOLUTE 13 > >> 32 POP_BLOCK > >> 33 LOAD_CONST 0 (None) > 36 RETURN_VALUE > > Can't we optimize the LOAD_FAST in lines 19 and 25 to a single load > and put the reference twice on the stack? There is no way that the > reference of i might change in between the two lines. Also, the > load_fast in lne 22 to reference x could be taken out of the loop as x > will always point to the same object.... Yes, you can, but you need: - a better AST evaluator (to mark symbols/variables with proper attributes); - a better optimizer (usually located on compile.c) which has a "global vision" (not limited to single instructions and/or single expressions). It's not that simple, and the results aren't guaranteed to be good. Also, consider that Python, as a dynamic-and-not-statically-compiled language need to find a good trade-off between compilation time and execution. Just to be clear, a C program is usually compiled once, then executed, so you can spend even *hours* to better optimize the final binary code. With a dynamic language, usually the code is compiled and the executed as needed, in "realtime". So it isn't practical neither desirable having to wait too much time before execution begins (the "startup" problem). Python stays in a "gray area", because modules are usually compiled once (when they are first used), and executed many times, but it isn't the only case. You cannot assume that optimization techniques used on other (static) languages can be used/ported in Python. Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Fri Dec 31 08:17:21 2010 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Fri, 31 Dec 2010 08:17:21 +0100 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: <4D1B3592.3040403@v.loewis.de> References: <4D1B3592.3040403@v.loewis.de> Message-ID: 2010/12/29 "Martin v. L?wis" > Am 28.12.2010 18:08, schrieb Lukas Lueg: > > Also, the > > load_fast in lne 22 to reference x could be taken out of the loop as x > > will always point to the same object.... > > That's not true; a debugger may change the value of x. > > Regards, > Martin OK, but is it mandatory? For example, in the above code, I can unroll the loop because I found that range is the usual built-in, 5 is a low-enough constant, and the body is made by a simple statement. Another example. I can totally remove the variable i, just using the stack, so a debugger (or, in general, having the tracing enabled) cannot even find something to change about it. And so on with other optimization examples that can be possible. Are they "legal" with Python? I think that we need to make it clear what happens in such cases. My idea is that it should be made implementation-specific. What happens with local variables and the generated code must depend on the specific compiler & virtual machine, in order to have a greater flexibility. IMHO the most important thing should be that, under normal conditions, the executed code have the expected behavior. Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From fijall at gmail.com Fri Dec 31 11:00:21 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 31 Dec 2010 12:00:21 +0200 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: References: <4D1B3592.3040403@v.loewis.de> Message-ID: > OK, but is it mandatory? For example, in the above code, I can unroll the > loop because I found that range is the usual built-in, 5 is a low-enough > constant, How do you know xrange is xrange and not something else? Cheers, fijal From fijall at gmail.com Fri Dec 31 11:01:12 2010 From: fijall at gmail.com (Maciej Fijalkowski) Date: Fri, 31 Dec 2010 12:01:12 +0200 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: References: <4D1B3592.3040403@v.loewis.de> Message-ID: On Fri, Dec 31, 2010 at 12:00 PM, Maciej Fijalkowski wrote: >> OK, but is it mandatory? For example, in the above code, I can unroll the >> loop because I found that range is the usual built-in, 5 is a low-enough >> constant, > > How do you know xrange is xrange and not something else? > > Cheers, > fijal > Err, misread. How do you know that range is a builtin you're thinking about and not some other object? Cheers, fijal From cesare.di.mauro at gmail.com Fri Dec 31 16:02:43 2010 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Fri, 31 Dec 2010 16:02:43 +0100 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: References: <4D1B3592.3040403@v.loewis.de> Message-ID: 2010/12/31 Maciej Fijalkowski > On Fri, Dec 31, 2010 at 12:00 PM, Maciej Fijalkowski > wrote: > >> OK, but is it mandatory? For example, in the above code, I can unroll > the > >> loop because I found that range is the usual built-in, 5 is a low-enough > >> constant, > > > > How do you know xrange is xrange and not something else? > > > > Cheers, > > fijal > > > > Err, misread. How do you know that range is a builtin you're thinking > about and not some other object? > > Cheers, > fijal > By a special opcode which could do this work. ]:-) Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Dec 31 16:55:36 2010 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 31 Dec 2010 07:55:36 -0800 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: References: <4D1B3592.3040403@v.loewis.de> Message-ID: <4D1DFCF8.8080109@stoneleaf.us> Cesare Di Mauro wrote: > > 2010/12/29 "Martin v. L?wis" wrote: >> >> Am 28.12.2010 18:08, schrieb Lukas Lueg: >>> Also, the load_fast in lne 22 to reference x could be taken out of the >>> loop as x will always point to the same object.... >> >> That's not true; a debugger may change the value of x. > > Another example. I can totally remove the variable i, just using the > stack, so a debugger (or, in general, having the tracing enabled) cannot > even find something to change about it. -1 Debugging is challenging enough as it is -- why would you want to make it even more difficult? ~Ethan~ From skip at pobox.com Fri Dec 31 17:36:36 2010 From: skip at pobox.com (skip at pobox.com) Date: Fri, 31 Dec 2010 10:36:36 -0600 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: <4D1DFCF8.8080109@stoneleaf.us> References: <4D1B3592.3040403@v.loewis.de> <4D1DFCF8.8080109@stoneleaf.us> Message-ID: <19742.1684.846410.770977@montanaro.dyndns.org> >> Another example. I can totally remove the variable i, just using the >> stack, so a debugger (or, in general, having the tracing enabled) >> cannot even find something to change about it. Ethan> -1 Ethan> Debugging is challenging enough as it is -- why would you want to Ethan> make it even more difficult? I don't know. Maybe he wants his program to run faster. If you use print statements for the bulk of your debugging (many people do), unrolling loops doesn't affect your debugging ability. Skip From status at bugs.python.org Fri Dec 31 18:07:06 2010 From: status at bugs.python.org (Python tracker) Date: Fri, 31 Dec 2010 18:07:06 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20101231170706.209AB1CBEB@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2010-12-24 - 2010-12-31) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 2525 (-17) closed 20058 (+49) total 22583 (+32) Open issues with patches: 1063 Issues opened (19) ================== #10764: sysconfig and alternative implementations http://bugs.python.org/issue10764 reopened by michael.foord #10771: descriptor protocol documentation has two different definition http://bugs.python.org/issue10771 opened by Devin Jeanpierre #10772: Several actions for argparse arguments missing from docs http://bugs.python.org/issue10772 opened by ipatrol #10775: assertRaises as a context manager should accept a 'msg' keywor http://bugs.python.org/issue10775 opened by r.david.murray #10782: Not possible to cross-compile due to poor detection of %lld su http://bugs.python.org/issue10782 opened by bgamari #10784: os.getpriority() and os.setpriority() http://bugs.python.org/issue10784 opened by giampaolo.rodola #10785: parser: store the filename as an unicode object http://bugs.python.org/issue10785 opened by haypo #10786: unittest.TextTextRunner does not respect redirected stderr http://bugs.python.org/issue10786 opened by cooyeah #10787: [random.gammavariate] Add the expression of the distribution i http://bugs.python.org/issue10787 opened by David.Kremer #10788: test_logging failure http://bugs.python.org/issue10788 opened by pitrou #10789: Lock.acquire documentation is misleading http://bugs.python.org/issue10789 opened by Jyrki.Pulliainen #10790: Header.append's charset logic is bogus, 'shift_jis' and "euc_j http://bugs.python.org/issue10790 opened by r.david.murray #10791: Wrapping TextIOWrapper around gzip files http://bugs.python.org/issue10791 opened by dabeaz #10794: Infinite recursion while garbage collecting loops indefinitely http://bugs.python.org/issue10794 opened by Mihai.Rusu #10796: readline completion flaw http://bugs.python.org/issue10796 opened by rheise #10798: test_concurrent_futures fails on FreeBSD http://bugs.python.org/issue10798 opened by loewis #10799: Improve webbrowser.open doc (and, someday, behavior?) http://bugs.python.org/issue10799 opened by terry.reedy #10800: libffi build failure on HP-UX 11/PA http://bugs.python.org/issue10800 opened by bugs-python at vendor.thewrittenword.com #10801: zipfile.ZipFile().extractall() header mismatch for non-ASCII c http://bugs.python.org/issue10801 opened by M..Z. Most recent 15 issues with no replies (15) ========================================== #10800: libffi build failure on HP-UX 11/PA http://bugs.python.org/issue10800 #10799: Improve webbrowser.open doc (and, someday, behavior?) http://bugs.python.org/issue10799 #10798: test_concurrent_futures fails on FreeBSD http://bugs.python.org/issue10798 #10796: readline completion flaw http://bugs.python.org/issue10796 #10789: Lock.acquire documentation is misleading http://bugs.python.org/issue10789 #10787: [random.gammavariate] Add the expression of the distribution i http://bugs.python.org/issue10787 #10775: assertRaises as a context manager should accept a 'msg' keywor http://bugs.python.org/issue10775 #10772: Several actions for argparse arguments missing from docs http://bugs.python.org/issue10772 #10761: tarfile.extractall fails to overwrite symlinks http://bugs.python.org/issue10761 #10760: tarfile doesn't handle sysfs well http://bugs.python.org/issue10760 #10752: build_ssl.py is relying on unreliable behaviour of os.popen http://bugs.python.org/issue10752 #10751: WSGIREF - REMOTE_USER and REMOTE-USER collision http://bugs.python.org/issue10751 #10747: Include version info in Windows shortcuts http://bugs.python.org/issue10747 #10746: ctypes c_long & c_bool have incorrect PEP-3118 type codes http://bugs.python.org/issue10746 #10745: setup.py install --user option undocumented http://bugs.python.org/issue10745 Most recent 15 issues waiting for review (15) ============================================= #10801: zipfile.ZipFile().extractall() header mismatch for non-ASCII c http://bugs.python.org/issue10801 #10798: test_concurrent_futures fails on FreeBSD http://bugs.python.org/issue10798 #10790: Header.append's charset logic is bogus, 'shift_jis' and "euc_j http://bugs.python.org/issue10790 #10787: [random.gammavariate] Add the expression of the distribution i http://bugs.python.org/issue10787 #10786: unittest.TextTextRunner does not respect redirected stderr http://bugs.python.org/issue10786 #10785: parser: store the filename as an unicode object http://bugs.python.org/issue10785 #10784: os.getpriority() and os.setpriority() http://bugs.python.org/issue10784 #10766: optparse uses %s in gettext calls http://bugs.python.org/issue10766 #10765: Build regression from automation changes on windows http://bugs.python.org/issue10765 #10756: Error in atexit._run_exitfuncs [...] Exception expected for v http://bugs.python.org/issue10756 #10755: Add posix.fdlistdir http://bugs.python.org/issue10755 #10746: ctypes c_long & c_bool have incorrect PEP-3118 type codes http://bugs.python.org/issue10746 #10744: ctypes arrays have incorrect buffer information (PEP-3118) http://bugs.python.org/issue10744 #10740: sqlite3 module should allow DDL statements in transactions http://bugs.python.org/issue10740 #10735: platform.architecture() gives misleading results for OS X mult http://bugs.python.org/issue10735 Top 10 most discussed issues (10) ================================= #6210: Exception Chaining missing method for suppressing context http://bugs.python.org/issue6210 23 msgs #2636: Regexp 2.7 (modifications to current re 2.2.2) http://bugs.python.org/issue2636 22 msgs #10542: Py_UNICODE_NEXT and other macros for surrogates http://bugs.python.org/issue10542 19 msgs #10769: ast: provide more useful range information http://bugs.python.org/issue10769 17 msgs #10791: Wrapping TextIOWrapper around gzip files http://bugs.python.org/issue10791 13 msgs #6011: python doesn't build if prefix contains non-ascii characters http://bugs.python.org/issue6011 5 msgs #10348: multiprocessing: use SysV semaphores on FreeBSD http://bugs.python.org/issue10348 5 msgs #10764: sysconfig and alternative implementations http://bugs.python.org/issue10764 5 msgs #7995: On Mac / BSD sockets returned by accept inherit the parent's F http://bugs.python.org/issue7995 4 msgs #9893: Usefulness of the Misc/Vim/ files? http://bugs.python.org/issue9893 4 msgs Issues closed (45) ================== #2708: IDLE subprocess error http://bugs.python.org/issue2708 closed by terry.reedy #3194: Demo/loop.c passing "char *" instead of "wchar_t *" http://bugs.python.org/issue3194 closed by georg.brandl #4212: email.LazyImporter does not use absolute imports http://bugs.python.org/issue4212 closed by r.david.murray #4685: IDLE will not open (2.6.1 on WinXP pro) http://bugs.python.org/issue4685 closed by terry.reedy #5258: addpackage in site.py fails hard on badly formed .pth files http://bugs.python.org/issue5258 closed by r.david.murray #5423: Exception raised when attempting to call set_charset on an ema http://bugs.python.org/issue5423 closed by r.david.murray #6533: Make test_xmlrpc_net functional in the absence of time.xmlrpc. http://bugs.python.org/issue6533 closed by r.david.murray #7056: regrtest runtest_inner calls findtestdir unnecessarily http://bugs.python.org/issue7056 closed by r.david.murray #7962: Demo and Tools need to be tested and pruned http://bugs.python.org/issue7962 closed by georg.brandl #8009: email.parser.Parser is inefficient with large strings http://bugs.python.org/issue8009 closed by r.david.murray #8889: test_support.transient_internet fails on Freebsd because socke http://bugs.python.org/issue8889 closed by r.david.murray #9153: Run tools and demos as part of the test suite http://bugs.python.org/issue9153 closed by georg.brandl #9824: SimpleCookie should escape commas and semi-colons http://bugs.python.org/issue9824 closed by r.david.murray #9891: Minor doc typo at datamodel.rst: "list" -> "alist" http://bugs.python.org/issue9891 closed by eric.araujo #10254: unicodedata.normalize('NFC', s) regression http://bugs.python.org/issue10254 closed by belopolsky #10494: Demo/comparisons/regextest.py needs some usage information. http://bugs.python.org/issue10494 closed by georg.brandl #10495: Demo/comparisons/sortingtest.py needs some usage information. http://bugs.python.org/issue10495 closed by georg.brandl #10609: dbm documentation example doesn't work (iteritems()) http://bugs.python.org/issue10609 closed by georg.brandl #10626: Bad interaction between test_logging and test_concurrent_futur http://bugs.python.org/issue10626 closed by bquinlan #10679: "make altinstall" may clobber OS provided scripts http://bugs.python.org/issue10679 closed by georg.brandl #10738: webbrowser.py bug with Opera on Linux http://bugs.python.org/issue10738 closed by terry.reedy #10739: Subprocess behavior on Windows http://bugs.python.org/issue10739 closed by georg.brandl #10742: memoryview.readonly attribute is not documented http://bugs.python.org/issue10742 closed by georg.brandl #10753: request_uri method of wsgiref module does not support RFC1808 http://bugs.python.org/issue10753 closed by orsenthil #10759: HTMLParser.unescape() fails on HTML entities with incorrect sy http://bugs.python.org/issue10759 closed by orsenthil #10763: subprocess.communicate() doesn't close pipes on Windows http://bugs.python.org/issue10763 closed by haypo #10767: Lib/test/crashers/README is out of date http://bugs.python.org/issue10767 closed by georg.brandl #10768: Bug in scrolledtext http://bugs.python.org/issue10768 closed by georg.brandl #10770: zipinfo - fix of a typo in the doc http://bugs.python.org/issue10770 closed by eric.araujo #10773: "Building C and C++ Extensions on Windows" documentation shows http://bugs.python.org/issue10773 closed by georg.brandl #10774: test_logging leaves temp files http://bugs.python.org/issue10774 closed by vinay.sajip #10776: os.utime returns an error on NTFS-3G partition http://bugs.python.org/issue10776 closed by Aaron.Masover #10777: xml.etree.register_namespace dictionary changed size during it http://bugs.python.org/issue10777 closed by georg.brandl #10778: decoding_fgets() (tokenizer.c) decodes the filename from the w http://bugs.python.org/issue10778 closed by haypo #10779: Change filename encoding to FS encoding in PyErr_WarnExplicit( http://bugs.python.org/issue10779 closed by haypo #10780: Fix filename encoding in PyErr_SetFromWindowsErrWithFilename() http://bugs.python.org/issue10780 closed by haypo #10781: minidom Node.writexml method doesn't manage encoding parameter http://bugs.python.org/issue10781 closed by georg.brandl #10783: struct.pack() and Unicode strings http://bugs.python.org/issue10783 closed by r.david.murray #10792: Compile() and 'Windows/Mac newlines' http://bugs.python.org/issue10792 closed by terry.reedy #10793: hashlib.hash.digest() documentation incorrect re return type http://bugs.python.org/issue10793 closed by orsenthil #10795: standard library do not use ssl as recommended http://bugs.python.org/issue10795 closed by pitrou #10797: Wrong detection of lines in readlines() function http://bugs.python.org/issue10797 closed by eric.smith #1379416: email.Header encode() unicode P2.6 http://bugs.python.org/issue1379416 closed by r.david.murray #1515839: socket timeout inheritance on accept http://bugs.python.org/issue1515839 closed by ned.deily #1685453: email package should work better with unicode http://bugs.python.org/issue1685453 closed by r.david.murray From cesare.di.mauro at gmail.com Fri Dec 31 18:42:53 2010 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Fri, 31 Dec 2010 18:42:53 +0100 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: <4D1DFCF8.8080109@stoneleaf.us> References: <4D1B3592.3040403@v.loewis.de> <4D1DFCF8.8080109@stoneleaf.us> Message-ID: 2010/12/31 Ethan Furman > Cesare Di Mauro wrote: > > > > >> 2010/12/29 "Martin v. L?wis" wrote: >> > >> > >> Am 28.12.2010 18:08, schrieb Lukas Lueg: >>> >>>> Also, the load_fast in lne 22 to reference x could be taken out of the >>>> >>> >>> loop as x will always point to the same object.... > >> >>> That's not true; a debugger may change the value of x. >>> >> >> Another example. I can totally remove the variable i, just using the >> stack, so a debugger (or, in general, having the tracing enabled) cannot >> even find something to change about it. >> > > -1 > > Debugging is challenging enough as it is -- why would you want to make it > even more difficult? > > ~Ethan~ With a good test suite you can forget debuggers. In more than 6 years of Python programming, I have used it only two times (to debug an ANTLR generated parser). Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: From cesare.di.mauro at gmail.com Fri Dec 31 18:51:39 2010 From: cesare.di.mauro at gmail.com (Cesare Di Mauro) Date: Fri, 31 Dec 2010 18:51:39 +0100 Subject: [Python-Dev] Possible optimization for LOAD_FAST ? In-Reply-To: <19742.1684.846410.770977@montanaro.dyndns.org> References: <4D1B3592.3040403@v.loewis.de> <4D1DFCF8.8080109@stoneleaf.us> <19742.1684.846410.770977@montanaro.dyndns.org> Message-ID: 2010/12/31 > > >> Another example. I can totally remove the variable i, just using the > >> stack, so a debugger (or, in general, having the tracing enabled) > >> cannot even find something to change about it. > Ethan> -1 > > Ethan> Debugging is challenging enough as it is -- why would you want to > Ethan> make it even more difficult? > > > I don't know. Maybe he wants his program to run faster. > > :D "Aggressive" optimizations can be enabled with explicit options, in order to leave normal "debugger-prone" code. > If you use print statements for the bulk of your debugging (many people > do), > unrolling loops doesn't affect your debugging ability. > > Skip It's a common practice. Also IDEs helps a lot, and advanced interactive shells too (such as DreamPie). Cesare -------------- next part -------------- An HTML attachment was scrubbed... URL: