From jimjjewett at gmail.com Thu Feb 1 00:50:35 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 31 Jan 2007 18:50:35 -0500 Subject: [Python-Dev] Python's C interface for types Message-ID: Nick Maclaren wrote: > Martin v. L?wis wrote: >> It may be a bit problematic to implement, but I think a clean >> specification is possible. If a and b are numbers, and a==b, >> then hash(a)==hash(b). You don't even need that much complication. If a==b, then hash(a) == hash(b) If you have to break this, then at least one (preferably both) of (a,b) must be unhashable, so that it won't get used as a dict key. Hashing is only ever meaningful as a shortcut to quickly show that two objects are *not* equal. >> I'm not sure whether "approximately 5.0" >> equals 5 or not: if it does, it should hash the same as 5, >> if it doesn't, it may or may not hash the same (whatever is >> easier to implement). agreed. >> For 0: hash(+0.0)==hash(-0.0)==hash(0)=hash(0L)=0 > Unfortunately, that assumes that equality is transitive. No, but the (transitively closed set of equivalent objects) must have the same hash. If >>> myfloat(5.0) != 5.0 True then you could just return 47 as the hash. It might not be terribly efficient, but it would work. If >>> myfloat_exact(5.0) == 5.0 == myfloat_approx(5.0) != myfloat_exact(5.0) then at least one of (myfloat_exact, myfloat_approx) needs to be unhashable, so that it can't be used as a dictionary key. > let us say that I am implementing a special function and want to > distinguish -0.0 and +0.0. Why can't I use a dictionary? Because they are equal. They aren't identical, but they are equal. > >>> a = float("+0.0") > >>> b = float("-0.0") > >>> print a, b > 0.0 -0.0 With the standard windows distribution, I get just 0.0 0.0 > No, I don't have an answer. You are damned if you do, and damned > if you don't. It is an insoluble problem, and CURRENTLY doesn't > justify two hashing mechanisms (i.e. ANY difference and EQUALITY > difference). You want something in between "__eq__" and "is". (a.identical(b) ?) Hashing is weaker than either. >>> hash ("JimJ") == hash (2010274390) True -jJ From george.sakkis at gmail.com Thu Feb 1 07:10:43 2007 From: george.sakkis at gmail.com (George Sakkis) Date: Thu, 1 Feb 2007 01:10:43 -0500 Subject: [Python-Dev] dict(keys, values) Message-ID: <91ad5bf80701312210y22c8303obeac8504b6b5cbc2@mail.gmail.com> Perhaps this has been brought up in the past but I couldn't find it in the archives: far too often I use the idiom dict(zip(keys,values)), or the same with izip. How does letting dict take two positional arguments sound ? Pros: - Pretty obvious semantics, no mental overhead to learn and remember it. - More concise (especially if one imports itertools just to use izip). - At least as efficient as the current alternatives. - Backwards compatible. Cons: - ?? George From brian at sweetapp.com Thu Feb 1 10:36:53 2007 From: brian at sweetapp.com (Brian Quinlan) Date: Thu, 01 Feb 2007 10:36:53 +0100 Subject: [Python-Dev] dict(keys, values) In-Reply-To: <91ad5bf80701312210y22c8303obeac8504b6b5cbc2@mail.gmail.com> References: <91ad5bf80701312210y22c8303obeac8504b6b5cbc2@mail.gmail.com> Message-ID: <45C1B4B5.9080802@sweetapp.com> George Sakkis wrote: > Perhaps this has been brought up in the past but I couldn't find it in > the archives: far too often I use the idiom dict(zip(keys,values)), or > the same with izip. How does letting dict take two positional > arguments sound ? > > Pros: > - Pretty obvious semantics, no mental overhead to learn and remember it. > - More concise (especially if one imports itertools just to use izip). > - At least as efficient as the current alternatives. > - Backwards compatible. > > Cons: - Yet Another Way To Do It - Marginal benefit Also note that the keyword variant is longer than the zip variant e.g. dict(zip(keys, values)) dict(keys=keys, values=values) and the relationship between the keys and values seems far less obvious to me in the keyword variant. Cheers, Brian From nmm1 at cus.cam.ac.uk Thu Feb 1 10:41:59 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Thu, 01 Feb 2007 09:41:59 +0000 Subject: [Python-Dev] Python's C interface for types Message-ID: "Jim Jewett" wrote: > > >> For 0: hash(+0.0)==hash(-0.0)==hash(0)=hash(0L)=0 > > > Unfortunately, that assumes that equality is transitive. > > No, but the (transitively closed set of equivalent objects) must have > the same hash. ... Er, how do you have a transitive closure for a non-transitive operation? I really do mean that quite a lot of floating-point bells and whistles are non-transitive. The only one most people will have come across is IEEE NaNs, where 'a is b' does not imply 'a == b', but there are a lot of others (and have been since time immemorial). I don't THINK that IEEE 754R decimal introduces any, though I am not prepared to bet on it. > > let us say that I am implementing a special function and want to > > distinguish -0.0 and +0.0. Why can't I use a dictionary? > > Because they are equal. They aren't identical, but they are equal. You have missed my point, which is extended floating-points effectively downgrade the status of the purely numeric comparisons, and therefore introduce a reasonable requirement for using a tighter match. Note that I am merely commenting that this needs bearing in mind, and NOT that anything should be changed. > > >>> a = float("+0.0") > > >>> b = float("-0.0") > > >>> print a, b > > 0.0 -0.0 > > With the standard windows distribution, I get just > > 0.0 0.0 Watch that space :-) Expect it to change. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From george.sakkis at gmail.com Thu Feb 1 10:48:21 2007 From: george.sakkis at gmail.com (George Sakkis) Date: Thu, 1 Feb 2007 04:48:21 -0500 Subject: [Python-Dev] dict(keys, values) In-Reply-To: <45C1B4B5.9080802@sweetapp.com> References: <91ad5bf80701312210y22c8303obeac8504b6b5cbc2@mail.gmail.com> <45C1B4B5.9080802@sweetapp.com> Message-ID: <91ad5bf80702010148v2af9687as473af9948e8c72d9@mail.gmail.com> On 2/1/07, Brian Quinlan wrote: > George Sakkis wrote: > > Perhaps this has been brought up in the past but I couldn't find it in > > the archives: far too often I use the idiom dict(zip(keys,values)), or > > the same with izip. How does letting dict take two positional > > arguments sound ? > > > > Pros: > > - Pretty obvious semantics, no mental overhead to learn and remember it. > > - More concise (especially if one imports itertools just to use izip). > > - At least as efficient as the current alternatives. > > - Backwards compatible. > > > > Cons: > - Yet Another Way To Do It > - Marginal benefit > > Also note that the keyword variant is longer than the zip variant e.g. > > dict(zip(keys, values)) > dict(keys=keys, values=values) > > and the relationship between the keys and values seems far less obvious > to me in the keyword variant. > > Cheers, > Brian > Um, you do realize that dict(keys=keys, values=values) is already valid and quite different from dict(zip(keys, values)), don't you ? :) George From steve at holdenweb.com Thu Feb 1 11:53:59 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 01 Feb 2007 10:53:59 +0000 Subject: [Python-Dev] dict(keys, values) In-Reply-To: <45C1B4B5.9080802@sweetapp.com> References: <91ad5bf80701312210y22c8303obeac8504b6b5cbc2@mail.gmail.com> <45C1B4B5.9080802@sweetapp.com> Message-ID: Brian Quinlan wrote: > George Sakkis wrote: >> Perhaps this has been brought up in the past but I couldn't find it in >> the archives: far too often I use the idiom dict(zip(keys,values)), or >> the same with izip. How does letting dict take two positional >> arguments sound ? >> >> Pros: >> - Pretty obvious semantics, no mental overhead to learn and remember it. >> - More concise (especially if one imports itertools just to use izip). >> - At least as efficient as the current alternatives. >> - Backwards compatible. >> >> Cons: > - Yet Another Way To Do It > - Marginal benefit > > Also note that the keyword variant is longer than the zip variant e.g. > > dict(zip(keys, values)) > dict(keys=keys, values=values) > > and the relationship between the keys and values seems far less obvious > to me in the keyword variant. > Unfortunately dict(keys=keys, values=values) == {keys: values} regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From brian at sweetapp.com Thu Feb 1 13:14:14 2007 From: brian at sweetapp.com (Brian Quinlan) Date: Thu, 01 Feb 2007 13:14:14 +0100 Subject: [Python-Dev] dict(keys, values) In-Reply-To: <91ad5bf80702010148v2af9687as473af9948e8c72d9@mail.gmail.com> References: <91ad5bf80701312210y22c8303obeac8504b6b5cbc2@mail.gmail.com> <45C1B4B5.9080802@sweetapp.com> <91ad5bf80702010148v2af9687as473af9948e8c72d9@mail.gmail.com> Message-ID: <45C1D996.3070207@sweetapp.com> George Sakkis wrote: > Um, you do realize that dict(keys=keys, values=values) is already > valid and quite different from dict(zip(keys, values)), don't you ? :) Sorry, minor misreading on my part. Like that time in Sunday school when I missed the "not" in "Though shall not kill". That was a rough week for everyone involved. OK, the non-zip variant saves you 5 characters i.e. dict(zip(keys, values)) vs. dict(keys, values) I still don't like it :-) Cheers, Brian From ark at acm.org Thu Feb 1 17:59:18 2007 From: ark at acm.org (Andrew Koenig) Date: Thu, 1 Feb 2007 11:59:18 -0500 Subject: [Python-Dev] dict(keys, values) In-Reply-To: Message-ID: <003f01c74622$52f02250$6402a8c0@arkdesktop> > Unfortunately > > dict(keys=keys, values=values) == {keys: values} Ummm, no: dict(keys=keys, values=values) == {'keys': keys, 'values': values} From steve at holdenweb.com Thu Feb 1 18:27:11 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 01 Feb 2007 17:27:11 +0000 Subject: [Python-Dev] dict(keys, values) In-Reply-To: <003f01c74622$52f02250$6402a8c0@arkdesktop> References: <003f01c74622$52f02250$6402a8c0@arkdesktop> Message-ID: Andrew Koenig wrote: >> Unfortunately >> >> dict(keys=keys, values=values) == {keys: values} > > Ummm, no: > > dict(keys=keys, values=values) == {'keys': keys, 'values': values} > Of course I should really have written dict(keys=keys, values=values) != dict(zip(keys, values)) regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From martin at v.loewis.de Thu Feb 1 19:35:58 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 01 Feb 2007 19:35:58 +0100 Subject: [Python-Dev] Python's C interface for types In-Reply-To: References: Message-ID: <45C2330E.4040601@v.loewis.de> Nick Maclaren schrieb: >>>> For 0: hash(+0.0)==hash(-0.0)==hash(0)=hash(0L)=0 >>> Unfortunately, that assumes that equality is transitive. >> No, but the (transitively closed set of equivalent objects) must have >> the same hash. ... > > Er, how do you have a transitive closure for a non-transitive operation? > > I really do mean that quite a lot of floating-point bells and whistles > are non-transitive. If so, they just shouldn't use the equal operator (==). == ought to be transitive. It should be consistent with has(). > You have missed my point, which is extended floating-points effectively > downgrade the status of the purely numeric comparisons, and therefore > introduce a reasonable requirement for using a tighter match. Note > that I am merely commenting that this needs bearing in mind, and NOT > that anything should be changed. If introducing extended floating-points would cause trouble to existing operations, I think extended floating-points should not be introduced to Python. If all three of you really need them, come up with method names to express "almost equal" or "equal only after sunset". Regards, Martin From nmm1 at cus.cam.ac.uk Thu Feb 1 20:36:24 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Thu, 01 Feb 2007 19:36:24 +0000 Subject: [Python-Dev] Python's C interface for types Message-ID: =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > > > I really do mean that quite a lot of floating-point bells and whistles > > are non-transitive. > > If so, they just shouldn't use the equal operator (==). == ought to > be transitive. It should be consistent with has(). Fine. A very valid viewpoint. Would you like to explain that to the IEEE 754 people? Strictly, it is only the reflexive property that IEEE 754 and the Decimal module lack. Yes, A == A is False, if A is a NaN. But the definition of 'transitive' often requires 'reflexive'. >>> from decimal import * >>> x = Decimal("NaN") >>> x == x False I don't know any CURRENT systems where basic floating-point doesn't have the strict transitive relation, but I wouldn't bet that there aren't any. You don't need to extend floating-point to have trouble; even the basic forms often had it. I sincerely hope that one is dead, but people keep reinventing old mistakes :-( The most common form was where comparison was equivalent to subtraction, and there were numbers such that A-B == 0, B-C == 0 but A-C != 0. That could occur even for integers on some systems. I don't THINK that the Decimal specification has reintroduced this, but am not quite sure. > > You have missed my point, which is extended floating-points effectively > > downgrade the status of the purely numeric comparisons, and therefore > > introduce a reasonable requirement for using a tighter match. Note > > that I am merely commenting that this needs bearing in mind, and NOT > > that anything should be changed. > > If introducing extended floating-points would cause trouble to existing > operations, I think extended floating-points should not be introduced > to Python. If all three of you really need them, come up with method > names to express "almost equal" or "equal only after sunset". Fine. Again, a very valid viewpoint. Would you like to explain it to the IEEE 754, Decimal and C99 people, and the Python people who think that tracking C is a good idea? We already have the situation where A == B == 0, but where 'C op A' != 'C op B' != 'C op 0'. Both where op is a built-in operator and where 'C op' is a standard library function. This one is NOT going to go away, and is going to get more serious, especially if extended floating-point formats like Decimal take off. Note that it is not a fault in Decimal, but a feature of almost all extended floating-points. As I said, I have no answer to it. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From martin at v.loewis.de Thu Feb 1 21:02:48 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 01 Feb 2007 21:02:48 +0100 Subject: [Python-Dev] Python's C interface for types In-Reply-To: References: Message-ID: <45C24768.6030306@v.loewis.de> Nick Maclaren schrieb: >> If so, they just shouldn't use the equal operator (==). == ought to >> be transitive. It should be consistent with has(). > > Fine. A very valid viewpoint. Would you like to explain that to > the IEEE 754 people? Why should I? I don't talk about IEEE 754, I talk about Python. > Strictly, it is only the reflexive property that IEEE 754 and the > Decimal module lack. Yes, A == A is False, if A is a NaN. But > the definition of 'transitive' often requires 'reflexive'. I deliberately stated 'transitive', not 'reflexive'. The standard definition of 'transitive' is "if a==b and b==c then a==c". > The most common form was where comparison was equivalent to subtraction, > and there were numbers such that A-B == 0, B-C == 0 but A-C != 0. That > could occur even for integers on some systems. I don't THINK that the > Decimal specification has reintroduced this, but am not quite sure. I'm not talking about subtraction, either. I'm talking about == and hash. > Fine. Again, a very valid viewpoint. Would you like to explain it > to the IEEE 754, Decimal and C99 people, and the Python people who > think that tracking C is a good idea? I'm not explaining anything. I'm stating an opinion. > This one is NOT going to go away, and is going to get more serious, > especially if extended floating-point formats like Decimal take off. > Note that it is not a fault in Decimal, but a feature of almost all > extended floating-points. As I said, I have no answer to it. It doesn't look like you *need* to give an answer now. I thought you were proposing some change to Python (although I'm uncertain what that change could have been). If you are merely explaining things (to whom?), just keep going. Regards, Martin From jimjjewett at gmail.com Thu Feb 1 21:16:06 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 1 Feb 2007 15:16:06 -0500 Subject: [Python-Dev] dict(keys, values) Message-ID: George Sakkis wrote: > far too often I use the idiom dict(zip(keys,values)), or > the same with izip. How does letting dict take two positional > arguments sound ? I think the dict constructor is already a bit too complicated, and would prefer that it be a separate classmethod, such as dict.zip(keys, values=itertools.repeat(None)) The default argument on values should finish the replacement of dict.fromkeys functionality that sets and defaultdicts began. > At least as efficient as the current alternatives. I think it has to do better (at least by eliminating the temporary zip object) to be worthwhile; the do-it-yourself alternative is pretty short. >>> # from itertools import izip as zip >>> def dictz(keys, values): return dict(zip(keys,values)) -jJ From nmm1 at cus.cam.ac.uk Thu Feb 1 21:24:57 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Thu, 01 Feb 2007 20:24:57 +0000 Subject: [Python-Dev] Python's C interface for types Message-ID: =?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?= wrote: > > >> If so, they just shouldn't use the equal operator (==). == ought to > >> be transitive. It should be consistent with has(). > > > > Fine. A very valid viewpoint. Would you like to explain that to > > the IEEE 754 people? > > Why should I? I don't talk about IEEE 754, I talk about Python. The problem is that Python is increasingly assuming IEEE 754 by implication, and you were stating something as a requirement that isn't true in IEEE 754. > > Strictly, it is only the reflexive property that IEEE 754 and the > > Decimal module lack. Yes, A == A is False, if A is a NaN. But > > the definition of 'transitive' often requires 'reflexive'. > > I deliberately stated 'transitive', not 'reflexive'. The standard > definition of 'transitive' is "if a==b and b==c then a==c". When I was taught mathematics, the lecturer said that a transitive relation is a reflexive one that has that extra property. It was then (and may still be) a fairly common usage. I apologise for being confusing! > > The most common form was where comparison was equivalent to subtraction, > > and there were numbers such that A-B == 0, B-C == 0 but A-C != 0. That > > could occur even for integers on some systems. I don't THINK that the > > Decimal specification has reintroduced this, but am not quite sure. > > I'm not talking about subtraction, either. I'm talking about == and > hash. Grrk. Look again. So am I. But let this one pass, as I don't think that mistake will return - and I sincerely hope not! > > Fine. Again, a very valid viewpoint. Would you like to explain it > > to the IEEE 754, Decimal and C99 people, and the Python people who > > think that tracking C is a good idea? > > I'm not explaining anything. I'm stating an opinion. You are, however, stating an opinion that conflicts with the direction that Python is currently taking. > It doesn't look like you *need* to give an answer now. I thought > you were proposing some change to Python (although I'm uncertain > what that change could have been). If you are merely explaining > things (to whom?), just keep going. Thanks. I hope the above clarifies things a bit. My purpose in posting is to point out that some changes are already happening, by inclusion from other standards, that are introducing problems to Python. And to many other languages, incidentally, including Fortran and C. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From jimjjewett at gmail.com Thu Feb 1 22:59:37 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 1 Feb 2007 16:59:37 -0500 Subject: [Python-Dev] Python's C interface for types Message-ID: Nick Maclaren wrote: >>> I really do mean that quite a lot of floating-point bells and whistles >>> are non-transitive. Martin v. L?wis wrote: >> If so, they just shouldn't use the equal operator (==). == ought to >> be transitive. It should be consistent with has(). Nick Maclaren wrote: > Fine. A very valid viewpoint. Would you like to explain that to > the IEEE 754 people? When Decimal was being argued, Tim pointed out that the standard requires certain operations, but doesn't require specific spelling shortcuts. If you managed to do (and document) it right, people would be grateful for methods like a.exactly(b) a.close_enough(b) a.same_expected_value(b) but that doesn't mean any of them should be used when testing a==b (In Lisp, you typically can specify which equality predicate a hashtable should use on pairs of keys; in python, you only specify which it should use on objects of your class, and if the other object in the comparison disagrees, you're out of luck.) > Strictly, it is only the reflexive property that IEEE 754 and the > Decimal module lack. Yes, A == A is False, if A is a NaN. Therefore NaNs should never be used (in python) as dictionary keys. Therefore, they should be unhashable. Also note that PyObject_RichCompareBool (from Objects/object.c) assumes the reflexive property, and if you try to violate it, you will get occasional surprises. > We already have the situation where A == B == 0, but where > 'C op A' != 'C op B' != 'C op 0'. Both where op is a built-in > operator and where 'C op' is a standard library function. That's fine; it just means that numeric equality may not be the strongest possible equivalence. hash in particular just happens to be defined in terms of ==, however == is determined. -jJ From nmm1 at cus.cam.ac.uk Fri Feb 2 09:33:40 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Fri, 02 Feb 2007 08:33:40 +0000 Subject: [Python-Dev] Python's C interface for types Message-ID: "Jim Jewett" wrote: > > > Fine. A very valid viewpoint. Would you like to explain that to > > the IEEE 754 people? > > When Decimal was being argued, Tim pointed out that the standard > requires certain operations, but doesn't require specific spelling > shortcuts. If you managed to do (and document) it right, people would > be grateful for methods like > > a.exactly(b) > a.close_enough(b) > a.same_expected_value(b) > > but that doesn't mean any of them should be used when testing a==b Hmm. That is misleading, as you state it. IEEE 754R doesn't include specific spellings, but IEEE 754 assuredly does. For example, it states that the equality operator that delivers False for NaN = NaN is spelled .EQ. in Fortran. There was no C standard at the time, but the "ad hoc' spellings are clearly intended for C-like languages, and C99 is very clear that the above equality operator is spelled '=='. However, there is no requirement that Python uses those names. What IS important is (a) that the comparisons are consistent, (b) that IEEE 754 (and IEEE 754R) define no reflexivity-preserving equality operator and (c) that the current float type derives its comparisons from C. > (In Lisp, you typically can specify which equality predicate a > hashtable should use on pairs of keys; in python, you only specify > which it should use on objects of your class, and if the other object > in the comparison disagrees, you're out of luck.) Yup. > > Strictly, it is only the reflexive property that IEEE 754 and the > > Decimal module lack. Yes, A == A is False, if A is a NaN. > > Therefore NaNs should never be used (in python) as dictionary keys. > Therefore, they should be unhashable. Again, a very valid point. Are you suggesting a change? :-) Currently, on my Linux system, Decimal raises an exception when trying to hash a NaN value but float doesn't. Is that a bug? > Also note that PyObject_RichCompareBool (from Objects/object.c) > assumes the reflexive property, and if you try to violate it, you will > get occasional surprises. Oh, yes, indeed! > > We already have the situation where A == B == 0, but where > > 'C op A' != 'C op B' != 'C op 0'. Both where op is a built-in > > operator and where 'C op' is a standard library function. > > That's fine; it just means that numeric equality may not be the > strongest possible equivalence. hash in particular just happens to be > defined in terms of ==, however == is determined. NO!!! What it means is that the equality operator may not be the strongest numeric equivalence! A much stronger statement. As I said, I am not grinding an axe, and have no answers. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From mokeefe at okeefecreations.com Sat Feb 3 18:36:10 2007 From: mokeefe at okeefecreations.com (Michael O'Keefe) Date: Sat, 3 Feb 2007 17:36:10 +0000 (UTC) Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls Message-ID: I had an idea on some new functionality I'd like to see offered in the python object class. I did some searching on the list but I wouldn't doubt that someone has proposed something like this before as chaining method calls seems like something folks would want to do. Specifically, I'd like to see a built-in shorthand to allow me to chain method calls even when a method call does not explicity return a reference to the instance of the object (self). I have an example below. In this example, someFunc represents the current way of dealing with calling methods that modify an object's state with no return value (if you found yourself doing this a lot, you might just write a wrapper function). The function explicitReturn shows how things might go if python implicitly returned self when no other return value was specified (similar to Ruby). I'm not proposing this, but have included it for completeness. The final two functions give some options for what I'm proposing as a shorthand (ideally included in class object but used in a subclass of class list in this example). The first of the two functions, newFunc01, defines the method "_". I'm not crazy about this because it's not very descriptive but it is the shortest way I could think of. The second possible shorthand is in, newFunc02, which defines the method "self_". I didn't want the method to be too long but my intent was "apply the given method and return self". I used the trailing underscore so as not to confuse with the self instance variable. I'm sure there are other ways to do this as well but what do people think? Has something like this come up before? def someFunc(): a = list([8,9,7,1]) a.sort() a.reverse() a.pop(0) return a def explicitReturn(): a = ExplicitReturnList([8,9,7,1]).sort().reverse() a.pop(0) return a def newFunc01(): return NewList([8,9,7,1])._('sort')._('reverse')._('pop',0) def newFunc02(): return NewList([8,9,7,1]).self_('sort').self_('reverse').self_('pop',0) class NewList(list): def __init__(self,*args,**kwargs): list.__init__(self,*args,**kwargs) def _(self,methodName,*args,**kwargs): method = getattr(self,methodName) method(*args,**kwargs) return self def self_(self,methodName,*args,**kwargs): method = getattr(self,methodName) method(*args,**kwargs) return self class ExplicitReturnList(list): def __init__(self,*args,**kwargs): list.__init__(self,*args,**kwargs) def sort(self): super(ExplicitReturnList,self).sort() return self def reverse(self): super(ExplicitReturnList,self).reverse() return self print someFunc() # returns [8, 7, 1] print explicitReturn() # returns [8, 7, 1] print newFunc01() # returns [8, 7, 1] print newFunc02() # returns [8, 7, 1] From mokeefe at okeefecreations.com Sat Feb 3 20:01:47 2007 From: mokeefe at okeefecreations.com (Michael =?utf-8?b?T1wnS2VlZmU=?=) Date: Sat, 3 Feb 2007 19:01:47 +0000 (UTC) Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls References: Message-ID: Michael O'Keefe okeefecreations.com> writes: I kept playing with this since my last post went live and here is a little more thought on these ideas. After sitting with the "Zen of Python" document (>>> import this), I'm maybe waffling on my previous post already so I came up with a few more options here. Also, to play the devil's advocate, in terms of pure space, these ideas don't necessarily decrease the total amount of typing (see the length stackup at the end of the post) though they would allow a trade for horizontal typing versus vertical (down the page) typing making for compact looking code. I suppose what would be nice is an operator which would execute the method indicated at left but then return back the reference to the now modified object. This is given in the desired() function below. Since this object doesn't exist, it can't be demonstrated. The filtered01 and filtered02 functions demonstrate trying to put eval to work -- not crazy about that since it has the added baggage of the globals(), locals() plus the need to write the calls in string format. Anyhow, just curious for ideas and sparking discussion. def desired(): pass # IF we had a --> operator which would execute the method at # left but return a ref to object #return [8,9,7,1].sort()-->reverse()-->pop(0)--> # returns [8,7,1] # return [8,9,7,1].sort()-->reverse()-->pop(0) # returns 9 # return [8,9,7,1].sort()-->reverse()-->pop(0) # equivalent to above def filtered01(): popidx=0 return filter([8,9,7,1],'sort()','reverse()','pop(popidx)',locals=locals()) def filtered02(): return filter([8,9,7,1],'sort()','reverse()').pop(0) def filter(obj,*args,**kwargs): if 'globals' in kwargs: globals_=kwargs['globals'] else: globals_=globals() if 'locals' in kwargs: locals_ =kwargs['locals'] else: locals_ =locals() locals_.update(locals()) for methodCall in args: eval('obj.%s'%methodCall,globals_,locals_) return obj print filtered01() # returns [8, 7, 1] print filtered02() # returns 9 # LENGTH STACKUP (if all statements put on the same horizontal line) # someFunc(): # a=[8,9,7,1]a.sort()a.reverse()a.pop(0)return a # filtered01: # return filter([8,9,7,1],'sort()','reverse()','pop(0)',locals=locals()) # explicitReturn(): # a=[8,9,7,1].sort().reverse()a.pop(0)return a # newFunc01: # return [8,9,7,1]._('sort')._('reverse')._('pop',0) # newFunc02: # return [8,9,7,1].self_('sort').self_('reverse').self_('pop',0) # desired: # return [8,9,7,1]-->sort()-->reverse()-->pop(0)--> From dustin at v.igoro.us Sat Feb 3 20:39:52 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Sat, 3 Feb 2007 13:39:52 -0600 Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls In-Reply-To: References: Message-ID: <20070203193952.GG31813@v.igoro.us> On Sat, Feb 03, 2007 at 07:01:47PM +0000, Michael O\'Keefe wrote: > Anyhow, just curious for ideas and sparking discussion. ... I haven't been on the list long enough to know, but I would expect that this idea and its relatives have been batted around at least once before. I think a lot of people have been frustrated at the repetitive nature of operations on lists, for example, as you indicated in your first post. I think there's room for debate on whether specific list methods that currently return None should instead return the list, although I would definitely consult the archives before entering that fray. I expect that the idea of adding a new operator or any other syntactic change is, like the count of 5, "right out". For what it's worth, you can wrap an object so it behaves the way you like as follows, although of course this will discard the return value of any functions which produce one: class wrapcall(object): def __init__(self, inner): self.inner = inner def __getattr__(self, attr): rv = getattr(self.inner, attr) if callable(rv): def wrap(*args, **kwargs): rv(*args, **kwargs) return self return wrap else: return rv mylist = [1, 2, 3] wrapcall(mylist).reverse().append(2).reverse() assert mylist == [2, 1, 2, 3] Dustin From eopadoan at altavix.com Sat Feb 3 20:44:33 2007 From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan) Date: Sat, 3 Feb 2007 17:44:33 -0200 Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls In-Reply-To: References: Message-ID: Cool! A way to write big one-liners and break the law of Demeter. Also, this should be a Python-ideas, not a Python-dev Post. -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt Blog: http://edcrypt.blogspot.com Jabber: edcrypt at jabber dot org ICQ: 161480283 GTalk: eduardo dot padoan at gmail dot com MSN: eopadoan at altavix dot com From rasky at develer.com Sat Feb 3 20:47:17 2007 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 03 Feb 2007 20:47:17 +0100 Subject: [Python-Dev] __dir__ and __all__ Message-ID: Hello, I could not find a PEP for __dir__. I was thinking today that if __dir__ was added, I would find it useful if modules implemented it by returning __all__ (when __all__ is defined). Or even better, to return the same names that would be imported by a star-import (so, if __all__ is not defined, all names which do not start with '_'). This wouldn't prevent introspection tools to use mod.__dict__ to still access the module's global dictionary, of course. But it would allow module's authors to more clearly document the module's public API. -- Giovanni Bajo From tjreedy at udel.edu Sat Feb 3 20:55:54 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 3 Feb 2007 14:55:54 -0500 Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls References: <20070203193952.GG31813@v.igoro.us> Message-ID: wrote in message news:20070203193952.GG31813 at v.igoro.us... | On Sat, Feb 03, 2007 at 07:01:47PM +0000, Michael O\'Keefe wrote: | I haven't been on the list long enough to know, but I would expect that this | idea and its relatives have been batted around at least once before. Of course. | I think a | lot of people have been frustrated at the repetitive nature of operations on | lists, for example, as you indicated in your first post. I think there's room | for debate on whether specific list methods that currently return None should | instead return the list, although I would definitely consult the archives | before entering that fray. The current design of having mutation methods return None is Guido's explicit, intentional, conscious, and repeatedly affirmed design choice. He is convinced that the pain of repetition is worth the reduction of bugs that would happen when people forget that a list (self) -returning method also mutates. I think comp.lang.python would be a better place for debate. tjr From greg.ewing at canterbury.ac.nz Sun Feb 4 00:21:57 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 04 Feb 2007 12:21:57 +1300 Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls In-Reply-To: References: Message-ID: <45C51915.60308@canterbury.ac.nz> Michael O'Keefe wrote: > I'd like to see a built-in shorthand to allow me to > chain method calls even when a method call does not explicity > return a reference to the instance of the object (self). > def newFunc02(): > return NewList([8,9,7,1]).self_('sort').self_('reverse').self_('pop',0) My thought is that you're trying to write programs in some other language using Python. If you really want all that on one line, you can do it already: x = NewList([8,9,7,1]); x.sort(); x.reverse(); return x.pop(0) which is just as compact and doesn't suffer from the weirdness of passing method names in as quoted strings. -- Greg From greg.ewing at canterbury.ac.nz Sun Feb 4 00:34:27 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 04 Feb 2007 12:34:27 +1300 Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls In-Reply-To: <20070203193952.GG31813@v.igoro.us> References: <20070203193952.GG31813@v.igoro.us> Message-ID: <45C51C03.4030505@canterbury.ac.nz> dustin at v.igoro.us wrote: > I think there's room > for debate on whether specific list methods that currently return None should > instead return the list, although I would definitely consult the archives > before entering that fray. Indeed. It's been discussed many times before. It was a deliberate decision to have those methods return None, because it makes it clear that they operate in-place. I doubt that Guido's thinking on that has changed at all. -- Greg From fuzzyman at voidspace.org.uk Sun Feb 4 00:30:57 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Sat, 03 Feb 2007 23:30:57 +0000 Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls In-Reply-To: <45C51915.60308@canterbury.ac.nz> References: <45C51915.60308@canterbury.ac.nz> Message-ID: <45C51B31.1080101@voidspace.org.uk> Greg Ewing wrote: > Michael O'Keefe wrote: > >> I'd like to see a built-in shorthand to allow me to >> chain method calls even when a method call does not explicity >> return a reference to the instance of the object (self). >> > > >> def newFunc02(): >> return NewList([8,9,7,1]).self_('sort').self_('reverse').self_('pop',0) >> > > My thought is that you're trying to write programs > in some other language using Python. > > If you really want all that on one line, you can > do it already: > > x = NewList([8,9,7,1]); x.sort(); x.reverse(); return x.pop(0) > > which is just as compact and doesn't suffer from > the weirdness of passing method names in as quoted > strings. > You could write a metaclass which decorates all methods in a class. Any methods which return None could return self instead. Michael > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > From ncoghlan at gmail.com Sun Feb 4 01:51:09 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 04 Feb 2007 10:51:09 +1000 Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls In-Reply-To: References: Message-ID: <45C52DFD.70809@gmail.com> Michael O'Keefe wrote: > I'm sure there are other ways to do this as well but what do people > think? Has something like this come up before? Yes, this has come up before, and list.sort and list.reverse were identified as the methods people were most likely to want to chain. The sorted() and reversed() builtins were the result. Also, the spelling you propose creates a horrible confusion when calling a method which already has a real return value (like list.pop). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From jcarlson at uci.edu Sun Feb 4 20:34:31 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 04 Feb 2007 11:34:31 -0800 Subject: [Python-Dev] __dir__ and __all__ In-Reply-To: References: Message-ID: <20070204113319.5AC5.JCARLSON@uci.edu> See the two threads started in python-dev by Tomer Filiba on (according to my clock) July 6, 2006 and October 6, 2006, both have __dir__ in the subject. - Josiah Giovanni Bajo wrote: > > Hello, > > I could not find a PEP for __dir__. I was thinking today that if __dir__ was > added, I would find it useful if modules implemented it by returning __all__ > (when __all__ is defined). Or even better, to return the same names that would > be imported by a star-import (so, if __all__ is not defined, all names which > do not start with '_'). > > This wouldn't prevent introspection tools to use mod.__dict__ to still access > the module's global dictionary, of course. But it would allow module's authors > to more clearly document the module's public API. > -- > Giovanni Bajo > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jcarlson%40uci.edu From kbk at shore.net Mon Feb 5 04:32:32 2007 From: kbk at shore.net (Kurt B. Kaiser) Date: Sun, 4 Feb 2007 22:32:32 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200702050332.l153WW59011394@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 423 open ( +2) / 3553 closed ( +4) / 3976 total ( +6) Bugs : 963 open (+20) / 6479 closed ( +8) / 7442 total (+28) RFE : 260 open ( +0) / 250 closed ( +0) / 510 total ( +0) New / Reopened Patches ______________________ ConfigParser getboolean() consistency (2007-01-28) http://python.org/sf/1646432 opened by Tal Einat gzip.GzipFile has no name attribute (2007-01-29) http://python.org/sf/1647484 opened by Lars Gust?bel proxy_bypass in urllib handling of macro (2007-01-30) http://python.org/sf/1648102 opened by Anthony Tuininga pty.fork() python implementation leaks the slave fd (2007-01-31) CLOSED http://python.org/sf/1648435 opened by John Levon Adding support for _Bool to ctypes as c_bool (2007-01-31) http://python.org/sf/1649190 opened by David Remahl configHandler support for raw data (2007-02-01) http://python.org/sf/1650174 opened by Tal Einat Patches Closed ______________ file -> open in stdlib (2007-01-25) http://python.org/sf/1644218 closed by gbrandl compiler.pycodegen causes crashes when compiling 'with' (2007-01-18) http://python.org/sf/1638243 closed by gbrandl Add aliases for latin7/9/10 charsets (2007-01-13) http://python.org/sf/1634778 closed by gbrandl pty.fork() python implementation leaks the slave fd (2007-01-31) http://python.org/sf/1648435 closed by gbrandl New / Reopened Bugs ___________________ os.access now returns bool but docstring is not updated (2007-01-27) CLOSED http://python.org/sf/1645944 opened by Seo Sanghyeon Dict lookups fail if sizeof(Py_ssize_t) < sizeof(long) (2007-01-27) http://python.org/sf/1646068 opened by ked-tao ctypes.string_at(buf, 0) is seen as zero-terminated-string (2007-01-28) http://python.org/sf/1646630 opened by Johannes H?lzl datetime.fromtimestamp fails with negative fractional times (2007-01-29) http://python.org/sf/1646728 opened by James Henstridge os.path, %HOME% set: realpath contradicts expanduser on '~' (2007-01-29) http://python.org/sf/1646838 opened by wrstl prmpft cookielib.CookieJar does not handle cookies when port in url (2007-01-29) CLOSED http://python.org/sf/1647037 opened by STS zero-length match confuses re.finditer() (2007-01-29) http://python.org/sf/1647489 opened by Jacques Frechet SystemError with re.match(array) (2007-01-30) http://python.org/sf/1647541 opened by Armin Rigo No obvious and correct way to get the time zone offset (2007-01-30) http://python.org/sf/1647654 opened by James Henstridge set update problem with class derived from dict (2007-01-30) CLOSED http://python.org/sf/1648179 opened by duncan Grammatical error (2007-01-30) CLOSED http://python.org/sf/1648191 opened by Chris Beelby Parameter list mismatches (portation problem) (2007-01-30) http://python.org/sf/1648268 opened by ked-tao HP-UX: ld -Wl,+b... (2007-01-31) http://python.org/sf/1648890 opened by Johannes Abt HP-UX: -lcurses missing for readline.so (2007-01-31) http://python.org/sf/1648923 opened by Johannes Abt HP-UX: _ctypes/libffi/src/ia64/ffi/__attribute__/native cc (2007-01-31) http://python.org/sf/1648957 opened by Johannes Abt HP-UX11.23: module zlib missing (2007-01-31) http://python.org/sf/1648960 opened by Johannes Abt HP-UX: compiler warnings: alignment (2007-01-31) http://python.org/sf/1649011 opened by Johannes Abt non-standard: array[0] (2007-01-31) http://python.org/sf/1649098 opened by Johannes Abt Arithmetics behaving strange (2007-01-31) CLOSED http://python.org/sf/1649100 opened by Sascha Peilicke potential class with C++ in ceval.h (2007-01-31) http://python.org/sf/1649238 opened by thechao gettext.py incompatible with eggs (2007-01-31) http://python.org/sf/1649329 opened by Shannon -jj Behrens decimals compare badly to floats (2007-02-01) http://python.org/sf/1650053 opened by Brian Sutherland doctest doesn't find nested functions (2007-02-01) http://python.org/sf/1650090 opened by Daniel Brown sys.excepthook does not work with -m command line switch (2007-02-02) CLOSED http://python.org/sf/1650899 opened by Miki Tebeka PyFloat_FromString deprecated form (2007-02-02) http://python.org/sf/1650903 opened by Jim Jewett ctypes.Structure formal parameter dies given tuple (2007-02-03) http://python.org/sf/1651235 opened by Gary Bishop readline needs termcap on my FC6 (2007-02-03) http://python.org/sf/1651427 opened by guichaz sgmllib _convert_ref UnicodeDecodeError exception, new in 2. (2007-02-04) http://python.org/sf/1651995 opened by John Nagle Bugs Closed ___________ os.access now returns bool but docstring is not updated (2007-01-27) http://python.org/sf/1645944 closed by gbrandl cookielib.CookieJar does not handle cookies when port in url (2007-01-29) http://python.org/sf/1647037 closed by tools-sts set update problem with class derived from dict (2007-01-30) http://python.org/sf/1648179 closed by rhettinger Grammatical error (2007-01-30) http://python.org/sf/1648191 closed by gbrandl GUI scripts always return to an interpreter (2006-09-29) http://python.org/sf/1568075 closed by jejackson Arithmetics behaving strange and magic underscore (2007-01-31) http://python.org/sf/1649100 closed by gbrandl pty.fork() leaves slave fd's open on Solaris (2004-04-12) http://python.org/sf/933670 closed by gbrandl Idle 1.2 - Calltips Hotkey does not work (2006-09-27) http://python.org/sf/1566611 closed by sf-robot sys.excepthook does not work with -m command line switch (2007-02-02) http://python.org/sf/1650899 closed by tebeka From python at rcn.com Mon Feb 5 08:05:37 2007 From: python at rcn.com (Raymond Hettinger) Date: Sun, 4 Feb 2007 23:05:37 -0800 Subject: [Python-Dev] Shortcut Notation for Chaining Method Calls References: Message-ID: <003801c748f4$0dd37350$f101a8c0@RaymondLaptop1> [Michael O'Keefe] > def desired(): > pass > # IF we had a --> operator which would execute the method at > # left but return a ref to object > #return [8,9,7,1].sort()-->reverse()-->pop(0)--> # returns [8,7,1] > # return [8,9,7,1].sort()-->reverse()-->pop(0) # returns 9 > # return [8,9,7,1].sort()-->reverse()-->pop(0) # equivalent to above I would write these as: >>> sorted([8,9,7,1], reverse=True)[1:] [8, 7, 1] >>> sorted([8,9,7,1], reverse=True)[0] 9 FWIW, I think the proposed notation is somewhat weird in that the first example's --> chain ends with a pop but returns a list. For successive mutations, separate lines are much more clear: s = [8,9,7,1] s.sort() s.reverse() s.pop(0) Also, I don't think your example generalizes well. How many classes have multiple methods that can meaningfully be chained together in an arbitrary order. It is telling that the example chooses to call the reverse method when it could have simple chosen the reverse=True argument to sort(). Raymond From fabiofz at gmail.com Wed Feb 7 01:42:30 2007 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Tue, 6 Feb 2007 22:42:30 -0200 Subject: [Python-Dev] Changing a value in a frame (for a debugger) Message-ID: Hi All, I'm currently trying to change the value for a variable in the debugger using: frame = findFrame(thread_id, frame_id) exec '%s=%s' % (attr, expression) in frame.f_globals, frame.f_locals it works well when the frame for the change is the topmost frame, but fails otherwise... I found some references speaking about some issue with PyFrame_FastToLocals which made it appear like this was not possible to do... so, basically, is there some way to make this work from python (changing a variable having a given frame) or not? Thanks, Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070206/a61ef453/attachment.htm From greg.ewing at canterbury.ac.nz Wed Feb 7 07:13:39 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 07 Feb 2007 19:13:39 +1300 Subject: [Python-Dev] Changing a value in a frame (for a debugger) In-Reply-To: References: Message-ID: <45C96E13.5080602@canterbury.ac.nz> Fabio Zadrozny wrote: > frame = findFrame(thread_id, frame_id) > exec '%s=%s' % (attr, expression) in frame.f_globals, frame.f_locals The locals of a function are actually stored in an array. When you access them as a dict using locals(), all you get is a dict containing a copy of their current values. Modifying that dict doesn't affect the underlying array. It seems that reading the f_locals of a frame does the same thing. To modify the locals, you would need to poke values into the original array -- but it doesn't seem to be exposed to Python. So it looks like you're out of luck. -- Greg From fabiofz at gmail.com Wed Feb 7 11:11:36 2007 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Wed, 7 Feb 2007 08:11:36 -0200 Subject: [Python-Dev] Changing a value in a frame (for a debugger) In-Reply-To: <45C96E13.5080602@canterbury.ac.nz> References: <45C96E13.5080602@canterbury.ac.nz> Message-ID: On 2/7/07, Greg Ewing wrote: > > Fabio Zadrozny wrote: > > > frame = findFrame(thread_id, frame_id) > > exec '%s=%s' % (attr, expression) in frame.f_globals, frame.f_locals > > The locals of a function are actually stored in an array. > When you access them as a dict using locals(), all you > get is a dict containing a copy of their current values. > Modifying that dict doesn't affect the underlying array. > > It seems that reading the f_locals of a frame does the > same thing. To modify the locals, you would need to poke > values into the original array -- but it doesn't seem > to be exposed to Python. > > So it looks like you're out of luck. > Would it be ok to add a feature request for that? I initially thought it was completely read-only, but I find it strange that it affects the topmost frame correctly (so, it seems that even though I get a copy, when I alter that copy on the topmost frame it affects it correctly)... anyone has a clue why that happens? It seems to affect pdb too... Consider the code: if __name__ == '__main__': def call1(): v_on_frame1 = 10 print 'v_on_frame1', v_on_frame1 def call0(): import pdb;pdb.set_trace() v_on_frame0 = 10 call1() print 'v_on_frame0', v_on_frame0 call0() #when modifying in the current frame > x:\scbr15\source\python\tests_base\empty_test.py(9)call0() -> v_on_frame0 = 10 (Pdb) n > x:\scbr15\source\python\tests_base\empty_test.py(10)call0() -> call1() (Pdb) v_on_frame0 = 40 (Pdb) c v_on_frame1 10 v_on_frame0 40 #when modifying an upper frame it does not work > x:\scbr15\source\python\tests_base\empty_test.py(9)call0() -> v_on_frame0 = 10 (Pdb) n > x:\scbr15\source\python\tests_base\empty_test.py(10)call0() -> call1() (Pdb) s --Call-- > x:\scbr15\source\python\tests_base\empty_test.py(3)call1() -> def call1(): (Pdb) n > x:\scbr15\source\python\tests_base\empty_test.py(4)call1() -> v_on_frame1 = 10 (Pdb) u > x:\scbr15\source\python\tests_base\empty_test.py(10)call0() -> call1() (Pdb) v_on_frame0 = 40 (Pdb) d > x:\scbr15\source\python\tests_base\empty_test.py(4)call1() -> v_on_frame1 = 10 (Pdb) c v_on_frame1 10 v_on_frame0 10 -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070207/89e064ec/attachment.html From martin at v.loewis.de Wed Feb 7 11:23:58 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 07 Feb 2007 11:23:58 +0100 Subject: [Python-Dev] Changing a value in a frame (for a debugger) In-Reply-To: References: <45C96E13.5080602@canterbury.ac.nz> Message-ID: <45C9A8BE.4030206@v.loewis.de> Fabio Zadrozny schrieb: > Would it be ok to add a feature request for that? Nobody can stop you, anyway :-) Seriously, a feature request is likely to sit there forever. If you would come up with an actual patch, that would be a different thing. You'll likely answer your other question in the process of developing a patch, too. Regards, Martin From fabiofz at gmail.com Wed Feb 7 18:17:59 2007 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Wed, 7 Feb 2007 15:17:59 -0200 Subject: [Python-Dev] Changing a value in a frame (for a debugger) In-Reply-To: <45C9A8BE.4030206@v.loewis.de> References: <45C96E13.5080602@canterbury.ac.nz> <45C9A8BE.4030206@v.loewis.de> Message-ID: On 2/7/07, "Martin v. L?wis" wrote: > > > Seriously, a feature request is likely to sit there > forever. If you would come up with an actual patch, > that would be a different thing. You'll likely answer > your other question in the process of developing a > patch, too. > Ok... I've added it as a feature-request in https://sourceforge.net/tracker/index.php?func=detail&aid=1654367&group_id=5470&atid=355470(it explains the problem and proposes a patch), so, if someone could take some time to go through it, it would be nice ;-) Thanks, Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070207/31bf551d/attachment.html From pje at telecommunity.com Wed Feb 7 18:47:00 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 07 Feb 2007 12:47:00 -0500 Subject: [Python-Dev] Changing a value in a frame (for a debugger) In-Reply-To: References: <45C96E13.5080602@canterbury.ac.nz> <45C96E13.5080602@canterbury.ac.nz> Message-ID: <5.1.1.6.0.20070207123653.02a96020@sparrow.telecommunity.com> At 08:11 AM 2/7/2007 -0200, Fabio Zadrozny wrote: >Would it be ok to add a feature request for that? I initially thought it >was completely read-only, but I find it strange that it affects the >topmost frame correctly (so, it seems that even though I get a copy, when >I alter that copy on the topmost frame it affects it correctly)... anyone >has a clue why that happens? The code that calls the debugging hook calls PyFrame_FastToLocals() before invoking the debugger, and then PyFrame_LocalsToFast() afterwards. Thus, the interrupted frame can have its locals modified, but no others can. In order for this to work on other frames, you'd have to explicitly call PyFrame_FastToLocals(frame, 1) -- which you could probably do with ctypes or a C extension. From greg.ewing at canterbury.ac.nz Thu Feb 8 00:06:56 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 08 Feb 2007 12:06:56 +1300 Subject: [Python-Dev] Changing a value in a frame (for a debugger) In-Reply-To: References: <45C96E13.5080602@canterbury.ac.nz> Message-ID: <45CA5B90.6010908@canterbury.ac.nz> Fabio Zadrozny wrote: > Would it be ok to add a feature request for that? It seems a reasonable thing to suggest. Instead of a copy, locals() could return a mapping object that is a view of the underlying array. The only limitation then would be that you couldn't add new keys. > I initially thought it > was completely read-only, but I find it strange that it affects the > topmost frame correctly That's because the topmost frame has a module's dict as its locals, so in that case you are modifying them directly. It's only code compiled as the body of a function that uses an array for locals. -- Greg From pje at telecommunity.com Thu Feb 8 00:32:09 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 07 Feb 2007 18:32:09 -0500 Subject: [Python-Dev] Changing a value in a frame (for a debugger) In-Reply-To: <45CA5B90.6010908@canterbury.ac.nz> References: <45C96E13.5080602@canterbury.ac.nz> Message-ID: <5.1.1.6.0.20070207183000.037a3a10@sparrow.telecommunity.com> At 12:06 PM 2/8/2007 +1300, Greg Ewing wrote: >That's because the topmost frame has a module's dict >as its locals, so in that case you are modifying them >directly. It's only code compiled as the body of a >function that uses an array for locals. By "topmost", he means the frame that was interrupted by the debugger, not the code at the top level of a module. From Martin.vonLoewis at hpi.uni-potsdam.de Thu Feb 8 10:49:38 2007 From: Martin.vonLoewis at hpi.uni-potsdam.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 08 Feb 2007 10:49:38 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? Message-ID: <45CAF232.7000505@hpi.uni-potsdam.de> #1653736 reports that slot_nb_inplace_power has the wrong type: it should be a ternary function, but only is a binary. The proposed change is to make it ternary, and to invoke __ipow__ with three arguments. In researching this, I came to wonder why nb_inplace_power is ternary in the first place. It is the implementation of foo **= bar (and that's its only use), so it ought to be binary. Historically, the reason probably is that it was copied from nb_power, which is ternary because it also implements pow()). So here is my proposed change: 1. For 2.5.1, rewrite slot_nb_inplace_power to raise an exception if the third argument is not None, and then invoke __ipow__ with only one argument. 2. For 2.6, change the API to make nb_inplace_power binary. Optionally, in 2.5, the exception could be added into other places as well, e.g. PyNumber_InPlacePower and instance_ipow (rather than invoking __ipow__ with a third argument if its not None). Comments? Regards, Martin From python at rcn.com Thu Feb 8 19:58:26 2007 From: python at rcn.com (python at rcn.com) Date: Thu, 8 Feb 2007 13:58:26 -0500 (EST) Subject: [Python-Dev] Why is nb_inplace_power ternary? Message-ID: <20070208135826.AXL19441@ms09.lnh.mail.rcn.net> [MvL] >>> So here is my proposed change: >>> >>> 1. For 2.5.1, rewrite slot_nb_inplace_power to raise an exception >>> if the third argument is not None, and then invoke __ipow__ with only one argument. Why would you change Py2.5? There is no bug here. Raymond From Martin.vonLoewis at hpi.uni-potsdam.de Thu Feb 8 20:11:20 2007 From: Martin.vonLoewis at hpi.uni-potsdam.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 08 Feb 2007 20:11:20 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <20070208135826.AXL19441@ms09.lnh.mail.rcn.net> References: <20070208135826.AXL19441@ms09.lnh.mail.rcn.net> Message-ID: <45CB75D8.9060100@hpi.uni-potsdam.de> python at rcn.com schrieb: >>>> 1. For 2.5.1, rewrite slot_nb_inplace_power to raise an exception >>>> if the third argument is not None, and then invoke __ipow__ with only one argument. > > Why would you change Py2.5? There is no bug here. There is: slot_nb_inplace has the signature static PyObject * slot_nb_inplace_power(PyObject *self, PyObject * arg1) yet it is stored in as_number.nb_inplace_power, which is defined as typedef PyObject * (*ternaryfunc)(PyObject *, PyObject *, PyObject *); ternaryfunc nb_inplace_power; This has undefined behavior. Even if it had, slot_nb_inplace_power would silently discard its third argument. Regards, Martin From brett at python.org Thu Feb 8 20:48:27 2007 From: brett at python.org (Brett Cannon) Date: Thu, 8 Feb 2007 11:48:27 -0800 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <45CAF232.7000505@hpi.uni-potsdam.de> References: <45CAF232.7000505@hpi.uni-potsdam.de> Message-ID: On 2/8/07, "Martin v. L?wis" wrote: > #1653736 reports that slot_nb_inplace_power has the wrong > type: it should be a ternary function, but only is a binary. > The proposed change is to make it ternary, and to invoke __ipow__ > with three arguments. > > In researching this, I came to wonder why nb_inplace_power is > ternary in the first place. It is the implementation of > > foo **= bar > > (and that's its only use), so it ought to be binary. Historically, > the reason probably is that it was copied from nb_power, which is > ternary because it also implements pow()). > > So here is my proposed change: > > 1. For 2.5.1, rewrite slot_nb_inplace_power to raise an exception > if the third argument is not None, and then invoke __ipow__ > with only one argument. > > 2. For 2.6, change the API to make nb_inplace_power binary. > > Optionally, in 2.5, the exception could be added into other places > as well, e.g. PyNumber_InPlacePower and instance_ipow (rather than > invoking __ipow__ with a third argument if its not None). > > Comments? > Seems reasonable to me. Is the argument of None passed in automatically somewhere? I think the 2.6 change is definitely good, just don't know if the exception is really needed. I realize the signature is off, but it doesn't hurt anyone at this point if it stayed wrong since obviously the semantics would be the same as they were. -Brett From greg.ewing at canterbury.ac.nz Fri Feb 9 00:08:32 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 09 Feb 2007 12:08:32 +1300 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <45CAF232.7000505@hpi.uni-potsdam.de> References: <45CAF232.7000505@hpi.uni-potsdam.de> Message-ID: <45CBAD70.1040201@canterbury.ac.nz> Martin v. L?wis wrote: > It is the implementation of > > foo **= bar > > (and that's its only use), so it ought to be binary. Maybe it's so that a type can plug the same implementation into both nb_pow and nb_inplace_pow. Although the same effect could be achieved by just leaving nb_inplace_pow null, so I suppose that's not necessary. Might we want to add an in-place version of the 3-arg pow() function one day? If so, leaving the third argument there could be useful. -- Greg From guido at python.org Fri Feb 9 06:14:44 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 8 Feb 2007 21:14:44 -0800 Subject: [Python-Dev] Adding timeout option to httplib...connect() Message-ID: I recently needed to access an HTTP URL with a timeout. I ended up monkey-patching httplib.HTTPConnection so that the connect() method has an optional second paramer, timeout, defaulting to None; if not None, a call to settimeout() is added right after successful creation of the socket. Does anybody else think this is a good idea? (Personally I think this should've been done years ago. :-) Shall I check it into the head? Index: httplib.py =================================================================== --- httplib.py (revision 53456) +++ httplib.py (working copy) @@ -656,7 +656,7 @@ def set_debuglevel(self, level): self.debuglevel = level - def connect(self): + def connect(self, timeout=None): """Connect to the host and port specified in __init__.""" msg = "getaddrinfo returns an empty list" for res in socket.getaddrinfo(self.host, self.port, 0, @@ -664,6 +664,8 @@ af, socktype, proto, canonname, sa = res try: self.sock = socket.socket(af, socktype, proto) + if timeout is not None: + self.sock.settimeout(timeout) if self.debuglevel > 0: print "connect: (%s, %s)" % (self.host, self.port) self.sock.connect(sa) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Fri Feb 9 06:23:27 2007 From: python at rcn.com (Raymond Hettinger) Date: Thu, 8 Feb 2007 21:23:27 -0800 Subject: [Python-Dev] Adding timeout option to httplib...connect() References: Message-ID: <00ed01c74c0a$6e36fd70$f001a8c0@RaymondLaptop1> [GvR] >I recently needed to access an HTTP URL with a timeout. I ended up > monkey-patching httplib.HTTPConnection so that the connect() method > has an optional second paramer, timeout, defaulting to None; if not > None, a call to settimeout() is added right after successful creation > of the socket. > > Does anybody else think this is a good idea? (Personally I think this > should've been done years ago. :-) Shall I check it into the head? Yes. This is has been requested more than once, but no one ever got around to adding the timeout.. Raymond From jcarlson at uci.edu Fri Feb 9 06:31:55 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 08 Feb 2007 21:31:55 -0800 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: References: Message-ID: <20070208212829.AC84.JCARLSON@uci.edu> "Guido van Rossum" wrote: > > I recently needed to access an HTTP URL with a timeout. I ended up > monkey-patching httplib.HTTPConnection so that the connect() method > has an optional second paramer, timeout, defaulting to None; if not > None, a call to settimeout() is added right after successful creation > of the socket. > > Does anybody else think this is a good idea? (Personally I think this > should've been done years ago. :-) Shall I check it into the head? I've done similar things to urllib in the past (also handling timeouts and insufficient data transfer rates). Seems like a reasonable change to me. It also makes me think that I should switch to httplib (since all the urls I ever fetch are http). - Josiah From python at rcn.com Fri Feb 9 06:30:05 2007 From: python at rcn.com (Raymond Hettinger) Date: Thu, 8 Feb 2007 21:30:05 -0800 Subject: [Python-Dev] Why is nb_inplace_power ternary? References: <20070208135826.AXL19441@ms09.lnh.mail.rcn.net> <45CB75D8.9060100@hpi.uni-potsdam.de> Message-ID: <010101c74c0b$5c2617a0$f001a8c0@RaymondLaptop1> [MvL] >>>>> 1. For 2.5.1, rewrite slot_nb_inplace_power to raise an exception >>>>> if the third argument is not None, and then invoke __ipow__ with only one >>>>> argument. [Raymond] >> Why would you change Py2.5? There is no bug here. [MvL] > There is: slot_nb_inplace has the signature > > static PyObject * slot_nb_inplace_power(PyObject *self, PyObject * arg1) > > yet it is stored in as_number.nb_inplace_power, which is defined as > > typedef PyObject * (*ternaryfunc)(PyObject *, PyObject *, PyObject *); > ternaryfunc nb_inplace_power; > > This has undefined behavior. Even if it had, slot_nb_inplace_power > would silently discard its third argument. That made sense, but my question was whether there would be benefit to making the change in the middle of a major release. At worst, code that is currently working due to undefined behavior will stop working. I don't see any offsetting benefit. ISTM that Py2.5 should be left as-is and that the full signature change be made in Py2.6. Raymond From Martin.vonLoewis at hpi.uni-potsdam.de Fri Feb 9 09:38:56 2007 From: Martin.vonLoewis at hpi.uni-potsdam.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 09 Feb 2007 09:38:56 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: References: <45CAF232.7000505@hpi.uni-potsdam.de> Message-ID: <45CC3320.6080707@hpi.uni-potsdam.de> Brett Cannon schrieb: > Seems reasonable to me. Is the argument of None passed in > automatically somewhere? There are few callers of nb_inplace_power at all (AFAICT, only PyNumber_InPlacePower); in turn, PyNumber_InPlacePower is called with the implicit Py_None always: - ceval.c, for INPLACE_POWER (which is binary) - operator.ipow (which is also binary) - class.c, from bin_inplace_power, which in turn is called from instance_ipow if the instance's 3rd argument to nb_inplace_power is Py_None (if there is a non-None third argument, instance_ipow invokes __ipow__ with three arguments if __ipow__ is defined, else it invokes __pow__ with three arguments) The only case I could find where a third argument is non-None is when the builtin pow() is invoked, which then invokes nb_power (but not nb_inplace_power) with three arguments. Regards, Martin From Martin.vonLoewis at hpi.uni-potsdam.de Fri Feb 9 09:46:11 2007 From: Martin.vonLoewis at hpi.uni-potsdam.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Fri, 09 Feb 2007 09:46:11 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <010101c74c0b$5c2617a0$f001a8c0@RaymondLaptop1> References: <20070208135826.AXL19441@ms09.lnh.mail.rcn.net> <45CB75D8.9060100@hpi.uni-potsdam.de> <010101c74c0b$5c2617a0$f001a8c0@RaymondLaptop1> Message-ID: <45CC34D3.2070807@hpi.uni-potsdam.de> Raymond Hettinger schrieb: > That made sense, but my question was whether there would be benefit > to making the change in the middle of a major release. At worst, code > that is > currently working due to undefined behavior will stop working. I don't > see any offsetting benefit. ISTM that Py2.5 should be left as-is and that > the full signature change be made in Py2.6. I can't see "left as is" as an option, see #1648268 and #1653736. Apparently, there are platforms where the code traps when there is a signature mismatch. If raising an exception is unacceptable, I'd rather explicitly ignore the extra argument, instead of ignoring it implicitly. Regards, Martin From martin at v.loewis.de Fri Feb 9 09:51:04 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Feb 2007 09:51:04 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <45CBAD70.1040201@canterbury.ac.nz> References: <45CAF232.7000505@hpi.uni-potsdam.de> <45CBAD70.1040201@canterbury.ac.nz> Message-ID: <45CC35F8.9040308@v.loewis.de> Greg Ewing schrieb: > Might we want to add an in-place version of the 3-arg > pow() function one day? If so, leaving the third argument > there could be useful. What could the syntax for that be? Instead of writing x = pow(x, n, 10) would you write x pow n = 10 ? or perhaps x ** n = 10 or x * n *= 10 Also, it would break existing __ipow__ implementations that only receive two arguments (unless there would be another __ method introduced). Regards, Martin From martin at v.loewis.de Fri Feb 9 09:57:51 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 09 Feb 2007 09:57:51 +0100 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: References: Message-ID: <45CC378F.8090803@v.loewis.de> Guido van Rossum schrieb: > I recently needed to access an HTTP URL with a timeout. I ended up > monkey-patching httplib.HTTPConnection so that the connect() method > has an optional second paramer, timeout, defaulting to None; if not > None, a call to settimeout() is added right after successful creation > of the socket. > > Does anybody else think this is a good idea? (Personally I think this > should've been done years ago. :-) Shall I check it into the head? Please also consider #723312, which is more general, but also more outdated. Regards, Martin From skip at pobox.com Fri Feb 9 14:52:16 2007 From: skip at pobox.com (skip at pobox.com) Date: Fri, 9 Feb 2007 07:52:16 -0600 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: References: Message-ID: <17868.31888.60854.91462@montanaro.dyndns.org> Guido> I recently needed to access an HTTP URL with a timeout. I ended Guido> up monkey-patching httplib.HTTPConnection so that the connect() Guido> method has an optional second paramer, timeout, defaulting to Guido> None; if not None, a call to settimeout() is added right after Guido> successful creation of the socket. Guido> Does anybody else think this is a good idea? In principle it's probably a fine idea. We should consider if it's possible to develop a uniform approach to timeouts for all the libraries that use sockets though. Otherwise you run the risk of doing it in different ways for different libraries and having to make a (small, but annoying) semantic leap when going from, say, httplib to smtpllib or ftplib. Skip From jimjjewett at gmail.com Fri Feb 9 16:42:22 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 9 Feb 2007 10:42:22 -0500 Subject: [Python-Dev] Why is nb_inplace_power ternary? Message-ID: Greg Ewing schrieb: > Might we want to add an in-place version of the 3-arg > pow() function one day? If so, leaving the third argument > there could be useful. "Martin v. L?wis" martin at v.loewis.de replied: > What could the syntax for that be? > Instead of writing > x = pow(x, n, 10) Either x**= n % 10 # The **= changes the parse context, so that % is no longer # immediately evaluated or x**= (n, 10) # exponentiation to a tuple isn't currently defined, and it # does show that both arguments are part of the power # expression -- but I'm not sure it needs a 3-argument form # instead of just unpacking the tuple itself. -jJ From brett at python.org Fri Feb 9 19:27:51 2007 From: brett at python.org (Brett Cannon) Date: Fri, 9 Feb 2007 10:27:51 -0800 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <45CC3320.6080707@hpi.uni-potsdam.de> References: <45CAF232.7000505@hpi.uni-potsdam.de> <45CC3320.6080707@hpi.uni-potsdam.de> Message-ID: On 2/9/07, "Martin v. L?wis" wrote: > Brett Cannon schrieb: > > Seems reasonable to me. Is the argument of None passed in > > automatically somewhere? > > There are few callers of nb_inplace_power at all (AFAICT, only > PyNumber_InPlacePower); in turn, PyNumber_InPlacePower is called > with the implicit Py_None always: > - ceval.c, for INPLACE_POWER (which is binary) > - operator.ipow (which is also binary) > - class.c, from bin_inplace_power, which in turn is called from > instance_ipow if the instance's 3rd argument to > nb_inplace_power is Py_None (if there is a non-None third > argument, instance_ipow invokes __ipow__ with three arguments > if __ipow__ is defined, else it invokes __pow__ with three > arguments) > > The only case I could find where a third argument is non-None > is when the builtin pow() is invoked, which then invokes nb_power > (but not nb_inplace_power) with three arguments. > Well then explicitly ignoring the object makes sense to me. While I am personally fine with raising the exception, erring on the side of caution as Raymond is suggesting wouldn't hurt either. -Brett From fdrake at acm.org Fri Feb 9 19:53:24 2007 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 9 Feb 2007 13:53:24 -0500 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <17868.31888.60854.91462@montanaro.dyndns.org> References: <17868.31888.60854.91462@montanaro.dyndns.org> Message-ID: <200702091353.25069.fdrake@acm.org> On Friday 09 February 2007 08:52, skip at pobox.com wrote: > In principle it's probably a fine idea. We should consider if it's > possible to develop a uniform approach to timeouts for all the libraries > that use sockets though. Otherwise you run the risk of doing it in > different ways for different libraries and having to make a (small, but > annoying) semantic leap when going from, say, httplib to smtpllib or > ftplib. Agreed. In the meanwhile, there's socket.setdefaulttimeout(), which has proved quite useful. -Fred -- Fred L. Drake, Jr. From guido at python.org Fri Feb 9 21:08:26 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Feb 2007 12:08:26 -0800 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <200702091353.25069.fdrake@acm.org> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> Message-ID: On 2/9/07, Fred L. Drake, Jr. wrote: > On Friday 09 February 2007 08:52, skip at pobox.com wrote: > > In principle it's probably a fine idea. We should consider if it's > > possible to develop a uniform approach to timeouts for all the libraries > > that use sockets though. Otherwise you run the risk of doing it in > > different ways for different libraries and having to make a (small, but > > annoying) semantic leap when going from, say, httplib to smtpllib or > > ftplib. > > Agreed. In the meanwhile, there's socket.setdefaulttimeout(), which has > proved quite useful. Didn't work for me, since my ap is a multi-threaded webserver and I only want one specific type of request to time out. I'm not going to change ftplib.py and all the others. I do think it's relevant to decide whether the timeout should be passed to the constructor or to the connect() method. I think it may be better to pass the timeout to the constructor. > -Fred > > -- > Fred L. Drake, Jr. > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Fri Feb 9 22:06:06 2007 From: skip at pobox.com (skip at pobox.com) Date: Fri, 9 Feb 2007 15:06:06 -0600 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> Message-ID: <17868.57918.906768.490015@montanaro.dyndns.org> Guido> Didn't work for me, since my ap is a multi-threaded webserver and Guido> I only want one specific type of request to time out. Understood. Guido> I'm not going to change ftplib.py and all the others. Also understood. This has, as far as I know, been the response of everybody who has encountered this problem before. Most people are only interested in a single protocol (or module) and therefore aren't motivated to change/fix the other protocols which could benefit from timeouts. After all, who really wants to update gopherlib? So there is sits. Guido> I do think it's relevant to decide whether the timeout should be Guido> passed to the constructor or to the connect() method. I think it Guido> may be better to pass the timeout to the constructor. Another thing to consider is how this change might percolate up to urllib/urllib2. (I advocate leaving urllib alone, but urllib2 should probably grow a timeout feature.) Skip From guido at python.org Fri Feb 9 23:01:19 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Feb 2007 14:01:19 -0800 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <17868.57918.906768.490015@montanaro.dyndns.org> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> Message-ID: You're not helping, Skip. Can you spend a minute looking at urllib2 and seeing how the two variants of my proposal (pass to constructor vs. pass to connect()) would impact on it? --Guido On 2/9/07, skip at pobox.com wrote: > > Guido> Didn't work for me, since my ap is a multi-threaded webserver and > Guido> I only want one specific type of request to time out. > > Understood. > > Guido> I'm not going to change ftplib.py and all the others. > > Also understood. This has, as far as I know, been the response of everybody > who has encountered this problem before. Most people are only interested in > a single protocol (or module) and therefore aren't motivated to change/fix > the other protocols which could benefit from timeouts. After all, who > really wants to update gopherlib? So there is sits. > > Guido> I do think it's relevant to decide whether the timeout should be > Guido> passed to the constructor or to the connect() method. I think it > Guido> may be better to pass the timeout to the constructor. > > Another thing to consider is how this change might percolate up to > urllib/urllib2. (I advocate leaving urllib alone, but urllib2 should > probably grow a timeout feature.) > > Skip > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Sat Feb 10 01:09:50 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 10 Feb 2007 13:09:50 +1300 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <45CC35F8.9040308@v.loewis.de> References: <45CAF232.7000505@hpi.uni-potsdam.de> <45CBAD70.1040201@canterbury.ac.nz> <45CC35F8.9040308@v.loewis.de> Message-ID: <45CD0D4E.7050308@canterbury.ac.nz> Martin v. L?wis wrote: > Greg Ewing schrieb: > >> Might we want to add an in-place version of the 3-arg >> pow() function one day? > > What could the syntax for that be? It wouldn't be a syntax, just a function, e.g. ipow(x, n, 10) > Also, it would break existing __ipow__ implementations > that only receive two arguments You could consider them broken already, since the signature clearly implies that there could be a third argument. The fact that it's not currently used is no excuse. :-) -- Greg From martin at v.loewis.de Sat Feb 10 13:28:05 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 10 Feb 2007 13:28:05 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <45CD0D4E.7050308@canterbury.ac.nz> References: <45CAF232.7000505@hpi.uni-potsdam.de> <45CBAD70.1040201@canterbury.ac.nz> <45CC35F8.9040308@v.loewis.de> <45CD0D4E.7050308@canterbury.ac.nz> Message-ID: <45CDBA55.2000907@v.loewis.de> Greg Ewing schrieb: >> What could the syntax for that be? > > It wouldn't be a syntax, just a function, e.g. > > ipow(x, n, 10) In what way would that be inplace? A function cannot rebind the variables it gets as parameters. Regards, Martin From martin at v.loewis.de Sat Feb 10 13:42:37 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 10 Feb 2007 13:42:37 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: References: Message-ID: <45CDBDBD.6090502@v.loewis.de> Jim Jewett schrieb: > Either > > x**= n % 10 # The **= changes the parse context, so that % > is no longer > # immediately evaluated Are you seriously proposing such a change? I was asking for spellings that currently don't have a meaning (rather, I was suggesting that no such spelling exists, and the entire idea of supporting such notation in the language is ridiculous). > x**= (n, 10) # exponentiation to a tuple isn't currently > defined, and it Likewise: it currenly is well-defined: it invoked nb_inplace_power with the tuple (and __ipow__ if x's class is written in Python). Whether this gives a TypeError or not depends on the class of x. Regards, Martin From martin at v.loewis.de Sat Feb 10 13:46:06 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 10 Feb 2007 13:46:06 +0100 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <17868.57918.906768.490015@montanaro.dyndns.org> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> Message-ID: <45CDBE8E.2080805@v.loewis.de> skip at pobox.com schrieb: > Guido> I'm not going to change ftplib.py and all the others. > > Also understood. This has, as far as I know, been the response of everybody > who has encountered this problem before. You should read your SF bug list more frequently, then. You are currently assigned bug #723312 (and have been since 2003-04-17), which has a patch to change ftplib. Regards, Martin From skip at pobox.com Sat Feb 10 14:44:22 2007 From: skip at pobox.com (skip at pobox.com) Date: Sat, 10 Feb 2007 07:44:22 -0600 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <45CDBE8E.2080805@v.loewis.de> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> <45CDBE8E.2080805@v.loewis.de> Message-ID: <17869.52278.474979.867894@montanaro.dyndns.org> Guido> I'm not going to change ftplib.py and all the others. >> Also understood. This has, as far as I know, been the response of >> everybody who has encountered this problem before. Martin> You should read your SF bug list more frequently, then. You are Martin> currently assigned bug #723312 (and have been since 2003-04-17), Martin> which has a patch to change ftplib. Assigned by the author, because I apparently offered him some advice I no longer recall giving? After much chiding here I'll take a look. I don't know if feature requests for Roundup are still being accepted, but I hope one of its features is that it can remind people periodically of the tickets they own. My primary goal in life is not to close Python bugs and patches, so I hope people will understand if I don't think on a Thursday evening, "Gee, I think I'll drop by SourceForge and see what work I have to do tonight". Skip From martin at v.loewis.de Sat Feb 10 18:05:01 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 10 Feb 2007 18:05:01 +0100 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <17869.52278.474979.867894@montanaro.dyndns.org> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> <45CDBE8E.2080805@v.loewis.de> <17869.52278.474979.867894@montanaro.dyndns.org> Message-ID: <45CDFB3D.1010701@v.loewis.de> skip at pobox.com schrieb: > I don't know if feature requests for Roundup are still being accepted, but I > hope one of its features is that it can remind people periodically of the > tickets they own. My primary goal in life is not to close Python bugs and > patches, so I hope people will understand if I don't think on a Thursday > evening, "Gee, I think I'll drop by SourceForge and see what work I have to > do tonight". Maybe it would be better to remove you from the list of possible assignees, then? You have four more patches and four bugs assigned to you, please consider unassigning them if you don't plan to work on them. Regards, Martin From skip at pobox.com Sat Feb 10 19:52:05 2007 From: skip at pobox.com (skip at pobox.com) Date: Sat, 10 Feb 2007 12:52:05 -0600 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <45CDFB3D.1010701@v.loewis.de> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> <45CDBE8E.2080805@v.loewis.de> <17869.52278.474979.867894@montanaro.dyndns.org> <45CDFB3D.1010701@v.loewis.de> Message-ID: <17870.5205.327646.687317@montanaro.dyndns.org> >> I don't know if feature requests for Roundup are still being >> accepted, but I hope one of its features is that it can remind people >> periodically of the tickets they own. My primary goal in life is not >> to close Python bugs and patches, so I hope people will understand if >> I don't think on a Thursday evening, "Gee, I think I'll drop by >> SourceForge and see what work I have to do tonight". Martin> Maybe it would be better to remove you from the list of possible Martin> assignees, then? I'll take a look. Skip From skip at pobox.com Sat Feb 10 20:02:47 2007 From: skip at pobox.com (skip at pobox.com) Date: Sat, 10 Feb 2007 13:02:47 -0600 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <45CDBE8E.2080805@v.loewis.de> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> <45CDBE8E.2080805@v.loewis.de> Message-ID: <17870.5847.223422.932006@montanaro.dyndns.org> Guido> I'm not going to change ftplib.py and all the others. >> Also understood. This has, as far as I know, been the response of >> everybody who has encountered this problem before. Martin> You should read your SF bug list more frequently, then. You are Martin> currently assigned bug #723312 (and have been since 2003-04-17), Martin> which has a patch to change ftplib. I updated the patch to be compatible with current svn: http://python.org/sf/723312 All tests pass. Well, the only tests which fail for me seem to have nothing to do with networking. I looked at gopherlib for about five seconds then decided it wasn't worth it. (Shouldn't it be deprecated and moved out to PyPI for external maintenance?) There are still no documentation updates. I don't have the few minutes necessary at the present time to cobble something together (headed out the door). Maybe tomorrow. Guido, I looked at urllib2 and quickly gave up. I have no idea how that code works (where is a lower level library's connection object instantiated, for example?). I presume with timeouts in the lower level libraries someone who knows how urllib2 works will be able to graft timeouts onto it without too much work. Skip From guido at python.org Sat Feb 10 20:42:55 2007 From: guido at python.org (Guido van Rossum) Date: Sat, 10 Feb 2007 11:42:55 -0800 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <17870.5847.223422.932006@montanaro.dyndns.org> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> <45CDBE8E.2080805@v.loewis.de> <17870.5847.223422.932006@montanaro.dyndns.org> Message-ID: On 2/10/07, skip at pobox.com wrote: > Guido, I looked at urllib2 and quickly gave up. I have no idea how that > code works (where is a lower level library's connection object instantiated, > for example?). I presume with timeouts in the lower level libraries someone > who knows how urllib2 works will be able to graft timeouts onto it without > too much work. Thanks for looking anyway! I had a quick look myself; I think it relies on the implicit call to connect() in the HTTPConnection class's request() method. This answers my question: the timeout should be passed to the HTTPConnection constructor which stores it in an instance variable just like host and port, and then connect() will get it from there. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From dustin at v.igoro.us Sat Feb 10 22:27:02 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Sat, 10 Feb 2007 15:27:02 -0600 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib Message-ID: <20070210212702.GH4869@v.igoro.us> Mostly for my own curiosity, I'm working on a PEP-342-based microthreading library with a similar api to threads and threading (coalesced into a single module). It uses coroutines and a trampoline scheduler, and provides basic async wrappers for common IO operations. It's not a framework/environment like Twisted or Kamaelia -- it's just a microthreading library with some solid primitives. My thinking is that this would be the "next level" for apps which currently use asyncore. I won't go into a lot of detail on the module, because (a) it's not even nearly done and (b) my question is higher-level than that. Is there any interest in including a simple microthreading module in Python's standard library? If this sounds like a terrible idea, let fly the n00b-seeking missiles. If it sounds better than terrible, I'll keep working and post a reasonable prototype soon (a PEP would also be in order at some point, correct?). Dustin From brett at python.org Sat Feb 10 23:37:22 2007 From: brett at python.org (Brett Cannon) Date: Sat, 10 Feb 2007 14:37:22 -0800 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <17870.5847.223422.932006@montanaro.dyndns.org> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> <45CDBE8E.2080805@v.loewis.de> <17870.5847.223422.932006@montanaro.dyndns.org> Message-ID: On 2/10/07, skip at pobox.com wrote: > > Guido> I'm not going to change ftplib.py and all the others. > > >> Also understood. This has, as far as I know, been the response of > >> everybody who has encountered this problem before. > > Martin> You should read your SF bug list more frequently, then. You are > Martin> currently assigned bug #723312 (and have been since 2003-04-17), > Martin> which has a patch to change ftplib. > > I updated the patch to be compatible with current svn: > > http://python.org/sf/723312 > > All tests pass. Well, the only tests which fail for me seem to have nothing > to do with networking. I looked at gopherlib for about five seconds then > decided it wasn't worth it. (Shouldn't it be deprecated and moved out to > PyPI for external maintenance? PEP 3108 has gopherlib slated for removal. -Brett From brett at python.org Sat Feb 10 23:41:07 2007 From: brett at python.org (Brett Cannon) Date: Sat, 10 Feb 2007 14:41:07 -0800 Subject: [Python-Dev] Adding timeout option to httplib...connect() In-Reply-To: <17869.52278.474979.867894@montanaro.dyndns.org> References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> <45CDBE8E.2080805@v.loewis.de> <17869.52278.474979.867894@montanaro.dyndns.org> Message-ID: On 2/10/07, skip at pobox.com wrote: > > Guido> I'm not going to change ftplib.py and all the others. > > >> Also understood. This has, as far as I know, been the response of > >> everybody who has encountered this problem before. > > Martin> You should read your SF bug list more frequently, then. You are > Martin> currently assigned bug #723312 (and have been since 2003-04-17), > Martin> which has a patch to change ftplib. > > Assigned by the author, because I apparently offered him some advice I no > longer recall giving? After much chiding here I'll take a look. > > I don't know if feature requests for Roundup are still being accepted, but I > hope one of its features is that it can remind people periodically of the > tickets they own. Good idea. Kind of like a weekly "here is your Python todo list in terms of issues" email. I think it should be doable. As for taking new ideas, the goal right now is to get the transition done. The biggest thing holding it up now is documentation, especially howto docs. Once we are transitioned we are expecting people to want to change things or add features. But if you want to get the ball rolling on this for possible inclusion afterwards, email tracker-discuss about your idea. > My primary goal in life is not to close Python bugs and > patches, so I hope people will understand if I don't think on a Thursday > evening, "Gee, I think I'll drop by SourceForge and see what work I have to > do tonight". I understand. =) From brett at python.org Sun Feb 11 00:00:28 2007 From: brett at python.org (Brett Cannon) Date: Sat, 10 Feb 2007 15:00:28 -0800 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070210212702.GH4869@v.igoro.us> References: <20070210212702.GH4869@v.igoro.us> Message-ID: On 2/10/07, dustin at v.igoro.us wrote: > Mostly for my own curiosity, I'm working on a PEP-342-based > microthreading library with a similar api to threads and threading > (coalesced into a single module). It uses coroutines and a trampoline > scheduler, and provides basic async wrappers for common IO operations. > > It's not a framework/environment like Twisted or Kamaelia -- it's just a > microthreading library with some solid primitives. My thinking is that > this would be the "next level" for apps which currently use asyncore. > > I won't go into a lot of detail on the module, because (a) it's not even > nearly done and (b) my question is higher-level than that. > > Is there any interest in including a simple microthreading module in > Python's standard library? > I am sure there is. > If this sounds like a terrible idea, let fly the n00b-seeking missiles. > If it sounds better than terrible, I'll keep working and post a > reasonable prototype soon (a PEP would also be in order at some point, > correct?). I really need to get the informatiion PEP written for guidelines on what it usually takes to get something added to the stdlib. =) Basically, the list of things you need to do (typically, these are just guidelines) are: 1. Write it 2. Get the community to use it and like it 3. Make it follow PEP 7/8 style guidelines 4. Write docs 5. Write tests 6. Promise to help maintain the code. -Brett From dustin at v.igoro.us Sun Feb 11 00:13:38 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Sat, 10 Feb 2007 17:13:38 -0600 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: References: <20070210212702.GH4869@v.igoro.us> Message-ID: <20070210231338.GJ4869@v.igoro.us> On Sat, Feb 10, 2007 at 03:00:28PM -0800, Brett Cannon wrote: > 1. Write it > 2. Get the community to use it and like it > 3. Make it follow PEP 7/8 style guidelines > 4. Write docs > 5. Write tests > 6. Promise to help maintain the code. Thanks -- I hadn't really planned that far ahead yet. I expect #2 will be the hardest! Dustin From janssen at parc.com Sun Feb 11 01:28:07 2007 From: janssen at parc.com (Bill Janssen) Date: Sat, 10 Feb 2007 16:28:07 PST Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070210212702.GH4869@v.igoro.us> References: <20070210212702.GH4869@v.igoro.us> Message-ID: <07Feb10.162808pst."57996"@synergy1.parc.xerox.com> > If this sounds like a terrible idea, let fly the n00b-seeking missiles. Sounds like a good idea. We did this with ILU, and it helped manage the overhead of threads quite a bit. Brett's comments on "the next step" are right on target. Bill From aahz at pythoncraft.com Sun Feb 11 06:29:01 2007 From: aahz at pythoncraft.com (Aahz) Date: Sat, 10 Feb 2007 21:29:01 -0800 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: References: <20070210212702.GH4869@v.igoro.us> Message-ID: <20070211052901.GA17758@panix.com> On Sat, Feb 10, 2007, Brett Cannon wrote: > On 2/10/07, dustin at v.igoro.us wrote: >> >> Is there any interest in including a simple microthreading module in >> Python's standard library? > > Basically, the list of things you need to do (typically, these are > just guidelines) are: > > 1. Write it > 2. Get the community to use it and like it > 3. Make it follow PEP 7/8 style guidelines > 4. Write docs > 5. Write tests > 6. Promise to help maintain the code. Let me amplify step 2 a bit: once you get your code to the stage where it's usable by other people, upload it to PyPI, announce it on c.l.py.a and other appropriate places, and go through several rounds of improvements and updates based on feedback and bug reports. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From dustin at v.igoro.us Sun Feb 11 06:35:18 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Sat, 10 Feb 2007 23:35:18 -0600 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: References: <20070210212702.GH4869@v.igoro.us> Message-ID: <20070211053518.GL4869@v.igoro.us> On Sun, Feb 11, 2007 at 03:35:29AM +0200, Yotam Rubin wrote: > Why don't you use Stackless? It's very simple, stable, and solves > quite completely the problems in writing concurrect code. That's a great point -- I'm not necessarily producing this to solve a problem I'm having. Rather, I think that the new language features in PEP 342 cry out for a batteries-included library that makes asynchronous programming both natural and easy. Dustin From steve at holdenweb.com Sun Feb 11 07:46:06 2007 From: steve at holdenweb.com (Steve Holden) Date: Sun, 11 Feb 2007 06:46:06 +0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070211053518.GL4869@v.igoro.us> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> Message-ID: <45CEBBAE.70002@holdenweb.com> dustin at v.igoro.us wrote: > On Sun, Feb 11, 2007 at 03:35:29AM +0200, Yotam Rubin wrote: >> Why don't you use Stackless? It's very simple, stable, and solves >> quite completely the problems in writing concurrect code. > > That's a great point -- I'm not necessarily producing this to solve a > problem I'm having. Rather, I think that the new language features in > PEP 342 cry out for a batteries-included library that makes asynchronous > programming both natural and easy. > Of course Stackless isn't quite fully integrated with 2.5 (yet). When did someone last suggest that Stackless become part of the core CPython implementation, and why didn't that ever happen? regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From brett at python.org Sun Feb 11 18:42:59 2007 From: brett at python.org (Brett Cannon) Date: Sun, 11 Feb 2007 09:42:59 -0800 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45CEBBAE.70002@holdenweb.com> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> Message-ID: On 2/10/07, Steve Holden wrote: > dustin at v.igoro.us wrote: > > On Sun, Feb 11, 2007 at 03:35:29AM +0200, Yotam Rubin wrote: > >> Why don't you use Stackless? It's very simple, stable, and solves > >> quite completely the problems in writing concurrect code. > > > > That's a great point -- I'm not necessarily producing this to solve a > > problem I'm having. Rather, I think that the new language features in > > PEP 342 cry out for a batteries-included library that makes asynchronous > > programming both natural and easy. > > > Of course Stackless isn't quite fully integrated with 2.5 (yet). > > When did someone last suggest that Stackless become part of the core > CPython implementation, and why didn't that ever happen? > Don't remember the "when". The "why" has always been that Christian's hacks into the core were complicated and he didn't even think integration was worth it. -Brett From martin at v.loewis.de Sun Feb 11 19:09:29 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 11 Feb 2007 19:09:29 +0100 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> Message-ID: <45CF5BD9.4080109@v.loewis.de> Brett Cannon schrieb: >> Of course Stackless isn't quite fully integrated with 2.5 (yet). >> >> When did someone last suggest that Stackless become part of the core >> CPython implementation, and why didn't that ever happen? >> > > Don't remember the "when". The "why" has always been that Christian's > hacks into the core were complicated and he didn't even think > integration was worth it. With emphasis on the latter. Christian never proposed (to my knowledge) that Stackless should be integrated. Of course, he didn't propose it because he assumed that proposal would be turned down, anyway. Regards, Martin From skip at pobox.com Sun Feb 11 19:23:37 2007 From: skip at pobox.com (skip at pobox.com) Date: Sun, 11 Feb 2007 12:23:37 -0600 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? Message-ID: <17871.24361.24644.747186@montanaro.dyndns.org> Someone checked in Parser/Python.asdl. After rebuilding Subversion tells me that Python/Python-ast.c has been modified. I assume the two are related. Did whoever checked in the former need to check in the latter (and maybe add a note to Misc/NEWS)? Skip From richard.m.tew at gmail.com Sun Feb 11 20:05:35 2007 From: richard.m.tew at gmail.com (Richard Tew) Date: Sun, 11 Feb 2007 19:05:35 +0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45CF5BD9.4080109@v.loewis.de> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> Message-ID: <952d92df0702111105q1b81f106rd3516461408f3035@mail.gmail.com> On 2/11/07, "Martin v. L?wis" wrote: > Brett Cannon schrieb: > >> Of course Stackless isn't quite fully integrated with 2.5 (yet). > >> > >> When did someone last suggest that Stackless become part of the core > >> CPython implementation, and why didn't that ever happen? > >> > > > > Don't remember the "when". The "why" has always been that Christian's > > hacks into the core were complicated and he didn't even think > > integration was worth it. > > With emphasis on the latter. Christian never proposed (to my knowledge) > that Stackless should be integrated. Of course, he didn't propose it > because he assumed that proposal would be turned down, anyway. As I understand it, given that at that time continuations and actually trying to make Python stackless were a goal of Stackless Python, I would hope people do not assume that the changes it currently makes to the core are the same as the ones mentioned here as hacks. Both of these goals have been discarded and Christian also made it an aim to make very few changes to the core and to "keep it and Stackless' intersection a minimum". One improvement Stackless needs related to what is proposed for the generator coroutines, is wrapping calls out of the Python runtime to things like network and file IO, so that microthreads can naturally take advantage of them. The asyncore module does the job for socket IO and allows monkeypatching the socket module so that the current microthread blocks on the asynchronous IO so the scheduler can continue without the whole runtime being stalled. But there is no support for cross- platform (specifically on Windows) asyncronous file IO for instance. If these generator coroutine microthreads went ahead and part of it was improving the support for asynchronous calls in the runtime and standard library, this would also be something which also benefited Stackless Python. Even if they didn't go ahead I would be interested in hearing about whether these sort of changes would be of interest or not for whatever reason. Regarding the port of Stackless to 2.5, it passes both its own (albeit limited) set of tests and the core's set as well. And there are no known bugs which affect general usage. Hope this helps, Richard. From brett at python.org Sun Feb 11 20:45:43 2007 From: brett at python.org (Brett Cannon) Date: Sun, 11 Feb 2007 11:45:43 -0800 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: <17871.24361.24644.747186@montanaro.dyndns.org> References: <17871.24361.24644.747186@montanaro.dyndns.org> Message-ID: On 2/11/07, skip at pobox.com wrote: > Someone checked in Parser/Python.asdl. After rebuilding Subversion tells me > that Python/Python-ast.c has been modified. I assume the two are related. > Did whoever checked in the former need to check in the latter Yeah, sorry about that. It took a ``make distclean`` for it to get regenerated for me. I checked in the updated version. > (and maybe add > a note to Misc/NEWS)? > That change to Parser/Python.asdl was cosmetic so there is no need for a NEWS entry. -Brett From martin at v.loewis.de Sun Feb 11 22:32:34 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 11 Feb 2007 22:32:34 +0100 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <952d92df0702111105q1b81f106rd3516461408f3035@mail.gmail.com> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <952d92df0702111105q1b81f106rd3516461408f3035@mail.gmail.com> Message-ID: <45CF8B72.2000808@v.loewis.de> Richard Tew schrieb: > If these generator coroutine microthreads went ahead and part > of it was improving the support for asynchronous calls in the > runtime and standard library, this would also be something > which also benefited Stackless Python. Even if they didn't go > ahead I would be interested in hearing about whether these > sort of changes would be of interest or not for whatever reason. For me to respond to such a question, I first need to understand what changes precisely you propose. "wrapping calls out of the Python runtime to things like network and file IO, so that microthreads can naturally take advantage of them." is a bit vague: how precisely would you wrap them? I assume you would like to see operating system mechanism be used that currently aren't used (similar to the way asynchore uses select/poll). If so, what mechanism would you like to use, and on what systems? Regards, Martin From martin at v.loewis.de Sun Feb 11 22:52:41 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 11 Feb 2007 22:52:41 +0100 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: References: <17871.24361.24644.747186@montanaro.dyndns.org> Message-ID: <45CF9029.3060104@v.loewis.de> Brett Cannon schrieb: > On 2/11/07, skip at pobox.com wrote: >> Someone checked in Parser/Python.asdl. After rebuilding Subversion tells me >> that Python/Python-ast.c has been modified. I assume the two are related. >> Did whoever checked in the former need to check in the latter > > Yeah, sorry about that. It took a ``make distclean`` for it to get > regenerated for me. I checked in the updated version. Actually, the regenerating should happen immediately after commit, as this bumps the revision number of the asdl file. This means you have to make two commits per AST grammar change: one to change the grammar, and the other to update the regenerated file. Regards, Martin From arigo at tunes.org Sun Feb 11 22:28:08 2007 From: arigo at tunes.org (Armin Rigo) Date: Sun, 11 Feb 2007 22:28:08 +0100 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45CF5BD9.4080109@v.loewis.de> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> Message-ID: <20070211212808.GA8197@code0.codespeak.net> Hi Martin, On Sun, Feb 11, 2007 at 07:09:29PM +0100, "Martin v. L?wis" wrote: > > hacks into the core were complicated and he didn't even think > > integration was worth it. > > With emphasis on the latter. Christian never proposed (to my knowledge) > that Stackless should be integrated. Of course, he didn't propose it > because he assumed that proposal would be turned down, anyway. The history as I remember it is that Christian tried hard to integrate the first versions of Stackless with CPython, but was turned town by python-dev. On a side note, I also wrote "greenlets", which is a C extension module for the regular CPython that uses some code from Stackless to provide coroutines without changing the core. I have even built an experimental I/O library for it, which I'm using with some success, and I personally like the style; but I also know that it's a very subjective matter, so I'm happy to keep it outside the stdlib. As far as I gather from discussing with Christian, he has got a similar point of view. Armin From guido at python.org Mon Feb 12 00:02:24 2007 From: guido at python.org (Guido van Rossum) Date: Sun, 11 Feb 2007 15:02:24 -0800 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: <45CF9029.3060104@v.loewis.de> References: <17871.24361.24644.747186@montanaro.dyndns.org> <45CF9029.3060104@v.loewis.de> Message-ID: On 2/11/07, "Martin v. L?wis" wrote: > Actually, the regenerating should happen immediately after commit, > as this bumps the revision number of the asdl file. This means > you have to make two commits per AST grammar change: one to change > the grammar, and the other to update the regenerated file. Is this documented somewhere? It wouldn't hurt if there was a pointer to that documentation right next to the line in Python-ast.c that gets modified by the regeneration. (I've been wondering about this a few times myself.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ben at redfrontdoor.org Mon Feb 12 00:45:05 2007 From: ben at redfrontdoor.org (Ben North) Date: Sun, 11 Feb 2007 23:45:05 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access Message-ID: <45CFAA81.5040906@redfrontdoor.org> Hi, A few days ago I posted to python-ideas with a suggestion for some new Python syntax, which would allow easier access to object attributes where the attribute name is known only at run-time. For example: setattr(self, method_name, getattr(self.metadata, method_name)) from Lib/distutils/dist.py could be rewritten self.(method_name) = self.metadata.(method_name) The new syntax would also be usable in augmented assignments, as in obj.(attr_name) += 1 There was some discussion on python-ideas, which I've linked to in the draft PEP below. In particular, Guido van Rossum wrote: > I've thought of the same syntax. I think you should submit this to the > PEP editor and argue on Python-dev for its inclusion in Python 2.6 -- > there's no benefit that I see of waiting until 3.0. so here I am. Does anybody have any opinions/suggestions, particularly on the "open questions" referred to in the draft PEP? To summarise these open questions: * The draft currently allows a two-argument form, to supply a default value if the object has no attribute of that name. This mimics the behaviour of the three-argument form of getattr, but looks a bit wrong: s = obj.(attr_name, 'default string') I agree that it looks odd, but perhaps the extra expressive power gained might be worth the oddness. * The draft implementation (a sourceforge patch, linked to in the draft PEP) may have a general performance penalty of around 1%, although my initial measurements were quite noisy. Josiah Carlson thought this would not be too substantial, but he did suggest a couple of other implementation routes which could be explored. The general performance hit is offset by a speed gain of around 40% for attribute access using the new syntax over getattr etc. Is 1% "too much" for this feature? Ben. - - - - 8< - - - - PEP: 363 [PROVISIONAL NUMBER] Title: Syntax For Dynamic Attribute Access Version: $Revision$ Last-Modified: $Date$ Author: Ben North Status: Draft Type: Standards Track Content-Type: text/plain Created: 29-Jan-2007 Post-History: Abstract Dynamic attribute access is currently possible using the "getattr" and "setattr" builtins. The present PEP suggests a new syntax to make such access easier, allowing the coder for example to write x.('foo_%d' % n) += 1 z = y.('foo_%d' % n).('bar_%s' % s) instead of attr_name = 'foo_%d' % n setattr(x, attr_name, getattr(x, attr_name) + 1) z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s) Note (I wrote this patch mostly to advance my own understanding of and experiment with the python language, but I've written it up in the style of a PEP in case it might be a useful idea.) Rationale Dictionary access and indexing both have a friendly invocation syntax: instead of x.__getitem__(12) the coder can write x[12]. This also allows the use of subscripted elements in an augmented assignment, as in "x[12] += 1". The present proposal brings this ease-of-use to dynamic attribute access too. Attribute access is currently possible in two ways: * When the attribute name is known at code-writing time, the ".NAME" trailer can be used, as in x.foo = 42 y.bar += 100 * When the attribute name is computed dynamically at run-time, the "getattr" and "setattr" builtins must be used: x = getattr(y, 'foo_%d' % n) setattr(z, 'bar_%s' % s, 99) The "getattr" builtin also allows the coder to specify a default value to be returned in the event that the object does not have an attribute of the given name: x = getattr(y, 'foo_%d' % n, 0) This PEP describes a new syntax for dynamic attribute access --- "x.(expr)" --- with examples given in the Abstract above. The new syntax also allows the provision of a default value in the "get" case, as in: x = y.('foo_%d' % n, None) This 2-argument form of dynamic attribute access is not permitted as the target of an (augmented or normal) assignment. Also, this part of the new syntax was not as well received [6] in initial discussions on python-ideas, and I agree that it does not fit so cleanly. I'm happy to prepare a revised PEP/patch without the 2-argument form if the consensus is that this would be preferred. Finally, the new syntax can be used with the "del" statement, as in del x.(attr_name) Impact On Existing Code The proposed new syntax is not currently valid, so no existing well-formed programs have their meaning altered by this proposal. Across all "*.py" files in the 2.5 distribution, there are around 600 uses of "getattr", "setattr" or "delattr". They break down as follows (figures have some room for error because they were arrived at by partially-manual inspection): c.300 uses of plain "getattr(x, attr_name)", which could be replaced with the new syntax; c.150 uses of the 3-argument form, i.e., with the default value; these could be replaced with the 2-argument form of the new syntax (the cases break down into c.125 cases where the attribute name is a literal string, and c.25 where it's only known at run-time); c.5 uses of the 2-argument form with a literal string attribute name, which I think could be replaced with the standard "x.attribute" syntax; c.120 uses of setattr, of which 15 use getattr to find the new value; all could be replaced with the new syntax, the 15 where getattr is also involved would show a particular increase in clarity; c.5 uses which would have to stay as "getattr" because they are calls of a variable named "getattr" whose default value is the builtin "getattr"; c.5 uses of the 2-argument form, inside a try/except block which catches AttributeError and uses a default value instead; these could use 2-argument form of the new syntax; c.10 uses of "delattr", which could use the new syntax. As examples, the line setattr(self, attr, change_root(self.root, getattr(self, attr))) from Lib/distutils/command/install.py could be rewritten self.(attr) = change_root(self.root, self.(attr)) and the line setattr(self, method_name, getattr(self.metadata, method_name)) from Lib/distutils/dist.py could be rewritten self.(method_name) = self.metadata.(method_name) Performance Impact Initial pystone measurements are inconclusive, but suggest there may be a performance penalty of around 1% in the pystones score with the patched version. One suggestion is that this is because the longer main loop in ceval.c hurts the cache behaviour, but this has not been confirmed. (Maybe a tool like valgrind [2] could help here?) On the other hand, measurements suggest a speed-up of around 40--45% for dynamic attribute access. Discussion To Date Initial posting of this PEP in draft form was to python-ideas on 20070209 [4], and the response was generally positive: I've thought of the same syntax. I think you should submit this to the PEP editor and argue on Python-dev for its inclusion in Python 2.6 -- there's no benefit that I see of waiting until 3.0. --- Guido van Rossum [5] Wow! I have to say this is a compelling idea. The syntax is a bit foreign looking, but [...] I feel like I could learn to like it anyway. --- Greg Falcon [6] I look forward to seeing this in Python 2.6. --- Josiah Carlson, further down the thread [8] with Greg Falcon expressing the reservations about the 2-argument form already noted above, and Josiah Carlson raising a query about performance: My only concern with your propsed change is your draft implementation. [...] Specifically, your changes [...] may negatively affect general Python performance. --- Josiah Carlson [7] Some initial measurements (see above) suggest the performance penalty is small, and Josiah Carlson said of such cost that it "isn't really substantial". [8] Questions To Be Resolved * Whether to allow the 2-argument form for default arguments. * Whether the performance impact is real; whether it is acceptable; whether alternative implementations might improve this aspect. Alternative Syntax For The New Feature Other syntaxes could be used, for example braces are currently invalid in a "trailer", so could be used here, giving x{'foo_%d' % n} += 1 My personal preference is for the x.('foo_%d' % n) += 1 syntax though: the presence of the dot shows there is attribute access going on; the parentheses have an analogous meaning to the mathematical "work this out first" meaning. This is also the syntax used in the language Matlab [1] for dynamic "field" access (where "field" is the Matlab term analogous to Python's "attribute"). Discussions on python-ideas (see above) made no comment on the brace alternative, and the .() notation was well-enough received, so the brace alternative should be considered rejected, I think. Error Cases Only strings are permitted as attribute names, so for instance the following error is produced: >>> x.(99) = 8 Traceback (most recent call last): File "", line 1, in TypeError: attribute name must be string, not 'int' This is handled by the existing PyObject_GetAttr function. Draft Implementation A draft implementation adds a new alternative to the "trailer" clause in Grammar/Grammar; a new AST type, "DynamicAttribute" in Python.asdl, with accompanying changes to symtable.c, ast.c, and compile.c, and three new opcodes (load/store/del) with accompanying changes to opcode.h and ceval.c. The patch consists of c.180 additional lines in the core code, and c.100 additional lines of tests. It is available as sourceforge patch #1657573 [3]. References [1] Using Dynamic Field Names :: Data Types (MATLAB Programming) http://www.mathworks.com/access/helpdesk/help/techdoc/matlab_prog/f2-41859.html [2] Valgrind: "suite of tools for debugging and profiling Linux programs" http://www.valgrind.org/ [3] Sourceforge patch #1657573 http://sourceforge.net/tracker/index.php?func=detail&aid=1657573&group_id=5470&atid=305470 [4] http://mail.python.org/pipermail/python-ideas/2007-February/000210.html [5] http://mail.python.org/pipermail/python-ideas/2007-February/000211.html [6] http://mail.python.org/pipermail/python-ideas/2007-February/000212.html [7] http://mail.python.org/pipermail/python-ideas/2007-February/000213.html [8] http://mail.python.org/pipermail/python-ideas/2007-February/000227.html Copyright This document has been placed in the public domain. From tdelaney at avaya.com Mon Feb 12 02:19:02 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Mon, 12 Feb 2007 12:19:02 +1100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access Message-ID: <2773CAC687FD5F4689F526998C7E4E5F074462@au3010avexu1.global.avaya.com> Ben North wrote: > c.5 uses which would have to stay as "getattr" because they > are calls of a variable named "getattr" whose default > value is the builtin "getattr"; Have you checked if these are intended to bring the "getattr" name into local scope for fast lookup, or to force a binding to the builtin "gettattr" at compile time (two common (ab)uses of default arguments)? If they are, they would be better served by the new syntax. Overall, +1 on obj.(attr_name) -0 on obj.(attr_name, default_value) Tim Delaney From ntoronto at cs.byu.edu Mon Feb 12 03:17:22 2007 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Sun, 11 Feb 2007 19:17:22 -0700 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45CFAA81.5040906@redfrontdoor.org> References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: <45CFCE32.60405@cs.byu.edu> Ben North wrote: > Hi, > > A few days ago I posted to python-ideas with a suggestion for some new > Python syntax, which would allow easier access to object attributes > where the attribute name is known only at run-time. For example: > > setattr(self, method_name, getattr(self.metadata, method_name)) > > from Lib/distutils/dist.py could be rewritten > > self.(method_name) = self.metadata.(method_name) I like it. > The new syntax would also be usable in augmented assignments, as in > > obj.(attr_name) += 1 Even nicer; much, much better than the current spelling. > *snip* > > * The draft currently allows a two-argument form, to supply a default > value if the object has no attribute of that name. This mimics the > behaviour of the three-argument form of getattr, but looks a bit wrong: > > s = obj.(attr_name, 'default string') > > I agree that it looks odd, but perhaps the extra expressive power > gained might be worth the oddness. It's not just odd, but because you can't use the result of that syntax as an assignment target (according to the PEP), it smells special-casey. Neil From brett at python.org Mon Feb 12 03:57:54 2007 From: brett at python.org (Brett Cannon) Date: Sun, 11 Feb 2007 18:57:54 -0800 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: References: <17871.24361.24644.747186@montanaro.dyndns.org> <45CF9029.3060104@v.loewis.de> Message-ID: On 2/11/07, Guido van Rossum wrote: > On 2/11/07, "Martin v. L?wis" wrote: > > Actually, the regenerating should happen immediately after commit, > > as this bumps the revision number of the asdl file. This means > > you have to make two commits per AST grammar change: one to change > > the grammar, and the other to update the regenerated file. > > Is this documented somewhere? It wouldn't hurt if there was a pointer > to that documentation right next to the line in Python-ast.c that gets > modified by the regeneration. (I've been wondering about this a few > times myself.) > Don't think so. How about this for wording for the file's documentation? /* File automatically generated by %s. This module must be committed separately from each AST grammar change; the __version__ number is set to the revision number of the commit containing the grammar change. */ -Brett From tjreedy at udel.edu Mon Feb 12 04:19:07 2007 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 11 Feb 2007 22:19:07 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: "Ben North" wrote in message news:45CFAA81.5040906 at redfrontdoor.org... | so here I am. Does anybody have any opinions/suggestions, particularly | on the "open questions" referred to in the draft PEP? To summarise | these open questions: Need: Runtime attributes are a fairly frequent 'How?' question on c.l.p. Syntax: I initially did not like it and was surprised at its current invalidity (I expected the 'surperfluous' parens to be ignored) but could not think of anything better, and after 10 min. it looks better for a couple of reasons. First, I vaguely remember () being used for a similar purpose of indicating indirection in some assembly languages. Second, parens are already special cased as the call operator when following an object expression (without an intervening operator). So I can swallow a new special case when they follow '.'. The 'two-arg' form: this analogizes .() as being like a obj() call, but the sematics are different enough that I think it confuses. So I suggest drop it at least for now. Terry Jan Reedy From anthony at interlink.com.au Mon Feb 12 04:14:53 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon, 12 Feb 2007 14:14:53 +1100 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: References: <17871.24361.24644.747186@montanaro.dyndns.org> Message-ID: <200702121414.53972.anthony@interlink.com.au> On Monday 12 February 2007 13:57, Brett Cannon wrote: > On 2/11/07, Guido van Rossum wrote: > > On 2/11/07, "Martin v. L?wis" wrote: > > > Actually, the regenerating should happen immediately after > > > commit, as this bumps the revision number of the asdl file. > > > This means you have to make two commits per AST grammar > > > change: one to change the grammar, and the other to update > > > the regenerated file. > > > > Is this documented somewhere? It wouldn't hurt if there was a > > pointer to that documentation right next to the line in > > Python-ast.c that gets modified by the regeneration. (I've been > > wondering about this a few times myself.) > > Don't think so. How about this for wording for the file's > documentation? > > /* > File automatically generated by %s. > > This module must be committed separately from each AST grammar > change; the __version__ number is set to the revision number of > the commit containing the grammar change. > */ Note that the welease.py script that builds the releases does a "touch" on the relevant files to make sure that make gets the build right. We had bugs opened at one point because the timestamps meant you needed a python interpreter to build python. I'm not _too_ stressed if the svn isn't always perfect in this regard - the number of people who are checking out svn to build their very first python interpreter would be low, I'd think. Anthony From brett at python.org Mon Feb 12 04:53:22 2007 From: brett at python.org (Brett Cannon) Date: Sun, 11 Feb 2007 19:53:22 -0800 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: <45CF9029.3060104@v.loewis.de> References: <17871.24361.24644.747186@montanaro.dyndns.org> <45CF9029.3060104@v.loewis.de> Message-ID: On 2/11/07, "Martin v. L?wis" wrote: > Brett Cannon schrieb: > > On 2/11/07, skip at pobox.com wrote: > >> Someone checked in Parser/Python.asdl. After rebuilding Subversion tells me > >> that Python/Python-ast.c has been modified. I assume the two are related. > >> Did whoever checked in the former need to check in the latter > > > > Yeah, sorry about that. It took a ``make distclean`` for it to get > > regenerated for me. I checked in the updated version. > > Actually, the regenerating should happen immediately after commit, > as this bumps the revision number of the asdl file. This means > you have to make two commits per AST grammar change: one to change > the grammar, and the other to update the regenerated file. > Revision 53751 adds a comment to Python-ast.c about this now, along with putting __version__ in the header as well. -Brett From brett at python.org Mon Feb 12 04:59:14 2007 From: brett at python.org (Brett Cannon) Date: Sun, 11 Feb 2007 19:59:14 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45CFAA81.5040906@redfrontdoor.org> References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: On 2/11/07, Ben North wrote: [SNIP] > * The draft currently allows a two-argument form, to supply a default > value if the object has no attribute of that name. This mimics the > behaviour of the three-argument form of getattr, but looks a bit wrong: > > s = obj.(attr_name, 'default string') > > I agree that it looks odd, but perhaps the extra expressive power > gained might be worth the oddness. I don't think it is. getattr can just be kept around if need be in order to support this use case. It just makes it look like too much of a function call at that point to me, especially if someone ends up writing some expression that is so long that they put in a newline between the two values: s = obj.(attr_name, 'default_string') So -1 on the two-item version, +0 on the one-item version. -Brett From anthony at interlink.com.au Mon Feb 12 05:33:14 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon, 12 Feb 2007 15:33:14 +1100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: <200702121533.15305.anthony@interlink.com.au> I have to say that I'm not that impressed by either the 1-arg or 2-arg versions. Someone coming across this syntax for the first time will not have any hints as to what it means - and worse, it looks like a syntax error to me. -1 from me. From collinw at gmail.com Mon Feb 12 05:55:51 2007 From: collinw at gmail.com (Collin Winter) Date: Sun, 11 Feb 2007 22:55:51 -0600 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45CFAA81.5040906@redfrontdoor.org> References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: <43aa6ff70702112055v711410c2ue9e94db678540bf4@mail.gmail.com> I like the general idea, but the syntax looks like dirt on my monitor. The period is too easy to lose visually and without it, there's nothing to distinguish this from a function call. Also, like Anthony Baxter said, someone coming across this for the first time will think it's a syntax error, allusions to MATLAB and assembly indirection syntax not withstanding. Ignoring the syntax, I'm -1 on the 2-argument form, especially since it can only be used in an expression context; getattr() can be kept around for this. I'm +0 on the idea, -1 on the means. Collin Winter From martin at v.loewis.de Mon Feb 12 08:15:43 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Feb 2007 08:15:43 +0100 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070211212808.GA8197@code0.codespeak.net> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> Message-ID: <45D0141F.10904@v.loewis.de> Armin Rigo schrieb: > The history as I remember it is that Christian tried hard to integrate > the first versions of Stackless with CPython, but was turned town by > python-dev. Are there public records of that? As I remember it, Christian never actually submitted a patch for inclusion (at least not in public; he may have approached Guido in private). The oldest discussion I could find quickly is from August 2000 (Python 2.0 and Stackless); there, it was Ben Wolfson who asked whether it could be integrated. I also found a message where I said that it couldn't be integrated as-is, for a number of reasons I listed; I didn't mean to imply that it can never be integrated. Regards, Martin From ntoronto at cs.byu.edu Mon Feb 12 08:38:27 2007 From: ntoronto at cs.byu.edu (Neil Toronto) Date: Mon, 12 Feb 2007 00:38:27 -0700 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <200702121533.15305.anthony@interlink.com.au> References: <45CFAA81.5040906@redfrontdoor.org> <200702121533.15305.anthony@interlink.com.au> Message-ID: <45D01973.9000802@cs.byu.edu> Anthony Baxter wrote: > I have to say that I'm not that impressed by either the 1-arg or > 2-arg versions. Someone coming across this syntax for the first > time will not have any hints as to what it means - and worse, it > looks like a syntax error to me. -1 from me. > I'm not sure the "looks like a syntax error" argument holds much weight, because any new syntax is likely to be a syntax error until the syntax is changed. :) "No hints" is a decent argument against it, though. Parenthesis are already used for tuples, function calls, precedence, and generator comprehensions. The more something gets overloaded, the more ambiguous it looks. How about obj.*str_expression instead? "*" is pretty common in the C family of languages as a dereferencing operator. Neil From martin at v.loewis.de Mon Feb 12 08:39:56 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Feb 2007 08:39:56 +0100 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: References: <17871.24361.24644.747186@montanaro.dyndns.org> <45CF9029.3060104@v.loewis.de> Message-ID: <45D019CC.1000907@v.loewis.de> Guido van Rossum schrieb: > Is this documented somewhere? It wouldn't hurt if there was a pointer > to that documentation right next to the line in Python-ast.c that gets > modified by the regeneration. (I've been wondering about this a few > times myself.) Done! Martin From anthony at interlink.com.au Mon Feb 12 08:50:35 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon, 12 Feb 2007 18:50:35 +1100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D01973.9000802@cs.byu.edu> References: <45CFAA81.5040906@redfrontdoor.org> <200702121533.15305.anthony@interlink.com.au> <45D01973.9000802@cs.byu.edu> Message-ID: <200702121850.38279.anthony@interlink.com.au> On Monday 12 February 2007 18:38, Neil Toronto wrote: > Anthony Baxter wrote: > > I have to say that I'm not that impressed by either the 1-arg > > or 2-arg versions. Someone coming across this syntax for the > > first time will not have any hints as to what it means - and > > worse, it looks like a syntax error to me. -1 from me. > > I'm not sure the "looks like a syntax error" argument holds much > weight, because any new syntax is likely to be a syntax error > until the syntax is changed. :) Yes and no. My point is that it's extremely similar to existing syntax. (Worse yet, it looks the same but for what's possibly the smallest and hardest-to-see character in any character set) "foo(baz)" vs "foo.(baz)" is... not good. > "No hints" is a decent argument > against it, though. Parenthesis are already used for tuples, > function calls, precedence, and generator comprehensions. The > more something gets overloaded, the more ambiguous it looks. How > about > > obj.*str_expression > > instead? "*" is pretty common in the C family of languages as a > dereferencing operator. I don't know. This all still smacks of magic line noise to me :-) Although I should also add that after thinking about it a bit, I'm still not convinced this is something that needs shorthand syntax - particularly not one that's so line-noisy... Hey, we could always use backticks in Python 3.0! Anthony From martin at v.loewis.de Mon Feb 12 08:55:17 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 12 Feb 2007 08:55:17 +0100 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: References: <17871.24361.24644.747186@montanaro.dyndns.org> <45CF9029.3060104@v.loewis.de> Message-ID: <45D01D65.2000801@v.loewis.de> Brett Cannon schrieb: > /* > File automatically generated by %s. > > This module must be committed separately from each AST grammar change; > the __version__ number is set to the revision number of the commit > containing the grammar change. > */ It doesn't completely show up in "svn diff" (when only the number changes). I don't think that's a problem. Regards, Martin From martin at v.loewis.de Mon Feb 12 08:58:11 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 12 Feb 2007 08:58:11 +0100 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: <200702121414.53972.anthony@interlink.com.au> References: <17871.24361.24644.747186@montanaro.dyndns.org> <200702121414.53972.anthony@interlink.com.au> Message-ID: <45D01E13.2000705@v.loewis.de> Anthony Baxter schrieb: >> This module must be committed separately from each AST grammar >> change; the __version__ number is set to the revision number of >> the commit containing the grammar change. >> */ > > Note that the welease.py script that builds the releases does > a "touch" on the relevant files to make sure that make gets the > build right. We had bugs opened at one point because the timestamps > meant you needed a python interpreter to build python. > > I'm not _too_ stressed if the svn isn't always perfect in this > regard - the number of people who are checking out svn to build > their very first python interpreter would be low, I'd think. Still, an editor of Python.asdl should really commit Python-ast.c *after* committing Python.asdl (or perhaps committing it twice): not because of time stamps, but because the subsequent compile run will regenerate the file and modify it in the process. Then, all other developers will find modifications in their tree that they didn't make. Regards, Martin From g.brandl at gmx.net Mon Feb 12 08:58:17 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 12 Feb 2007 08:58:17 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <43aa6ff70702112055v711410c2ue9e94db678540bf4@mail.gmail.com> References: <45CFAA81.5040906@redfrontdoor.org> <43aa6ff70702112055v711410c2ue9e94db678540bf4@mail.gmail.com> Message-ID: Collin Winter schrieb: > I like the general idea, but the syntax looks like dirt on my monitor. > The period is too easy to lose visually and without it, there's > nothing to distinguish this from a function call. Also, like Anthony > Baxter said, someone coming across this for the first time will think > it's a syntax error, allusions to MATLAB and assembly indirection > syntax not withstanding. > > Ignoring the syntax, I'm -1 on the 2-argument form, especially since > it can only be used in an expression context; getattr() can be kept > around for this. > > I'm +0 on the idea, -1 on the means. -1 here too. I fear that this gets too indistinguishable from normal calling syntax, leading to confusion. (Of course, one could propose other syntax, such as obj->(attr) or whatnot, but I doubt an ideal and Pythonic syntax can be found here...) To me, dynamic attribute access is something "special", and it justifies a different way of doing it, namely getattr and setattr. For the speed argument -- there were quite a few proposals to take builtins as constants under certain conditions, in which case getattr() usage could be optimized just as well as new syntax. Georg From martin at v.loewis.de Mon Feb 12 09:00:58 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Feb 2007 09:00:58 +0100 Subject: [Python-Dev] Does Python/Python-ast.c need to be checked in? In-Reply-To: <45D019CC.1000907@v.loewis.de> References: <17871.24361.24644.747186@montanaro.dyndns.org> <45CF9029.3060104@v.loewis.de> <45D019CC.1000907@v.loewis.de> Message-ID: <45D01EBA.1000600@v.loewis.de> Martin v. L?wis schrieb: > Guido van Rossum schrieb: >> Is this documented somewhere? It wouldn't hurt if there was a pointer >> to that documentation right next to the line in Python-ast.c that gets >> modified by the regeneration. (I've been wondering about this a few >> times myself.) > > Done! I didn't notice that my commit failed because my code wasn't up-to-date. So it's done, but not my doing (but Brett's). Regards, Martin From Jack.Jansen at cwi.nl Mon Feb 12 09:55:52 2007 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Mon, 12 Feb 2007 09:55:52 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45CFAA81.5040906@redfrontdoor.org> References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: On 12-feb-2007, at 0:45, Ben North wrote: > > self.(method_name) = self.metadata.(method_name) I like the functionality, but I don't like the syntax, to me it looks too much like a method call. To me self.[method_name] = self.metadata.[method_name] looks better: what we're doing here is more like dictionary lookup than calling functions. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From greg.ewing at canterbury.ac.nz Mon Feb 12 05:44:31 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Feb 2007 17:44:31 +1300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: <45CFF0AF.6030108@canterbury.ac.nz> Terry Reedy wrote: > Need: Runtime attributes are a fairly frequent 'How?' question on c.l.p. That's true, but how many of those are due to an actual need for runtime attributes, as opposed to someone trying to transplant idioms from another language that would be better served by a dict? In my experience, the amount of code which truly needs to use getattr is extremely small. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiem! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From greg.ewing at canterbury.ac.nz Mon Feb 12 10:14:52 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 12 Feb 2007 22:14:52 +1300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <45CFAA81.5040906@redfrontdoor.org> <43aa6ff70702112055v711410c2ue9e94db678540bf4@mail.gmail.com> Message-ID: <45D0300C.4000401@canterbury.ac.nz> Georg Brandl wrote: > For the speed argument -- there were quite a few proposals to take builtins as > constants under certain conditions, in which case getattr() usage could be > optimized just as well as new syntax. Even more aggressively, the compiler could recognise it and make a direct call to the __getattr__ method, or maybe even have a new opcode for doing that. In other words, "special syntax" doesn't necessarily have to look like special syntax. :-) -- Greg From python at rcn.com Mon Feb 12 10:25:54 2007 From: python at rcn.com (Raymond Hettinger) Date: Mon, 12 Feb 2007 01:25:54 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: <000501c74e88$0da367c0$6a176b0a@RaymondLaptop1> [Jack Jansen] > I like the functionality, but I don't like the syntax, to me it looks > too much like a method call. > > To me self.[method_name] = self.metadata.[method_name] looks better: > what we're doing here is more like dictionary lookup than calling > functions. I also like the functionality. Rather than munge existing syntaxes, an altogether new one would be more clear: self->name = self.metadata->name I like the arrow syntax because is the lookup process can be more involved than a simple dictionary lookup (perhaps traveling up to base classes). IOW, getattr(a,n) is not always the same as a.__dict__[n]. The a.__getattribute__(n) process can be more complex than that and a bracketed dictionary-like syntax would misleadingly mask the lookup process. Raymond From gjcarneiro at gmail.com Mon Feb 12 12:20:41 2007 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Mon, 12 Feb 2007 11:20:41 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45CFAA81.5040906@redfrontdoor.org> References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: On 2/11/07, Ben North wrote: > > Hi, > > A few days ago I posted to python-ideas with a suggestion for some new > Python syntax, which would allow easier access to object attributes > where the attribute name is known only at run-time. For example: > > setattr(self, method_name, getattr(self.metadata, method_name)) > > from Lib/distutils/dist.py could be rewritten > > self.(method_name) = self.metadata.(method_name) > > The new syntax would also be usable in augmented assignments, as in > > obj.(attr_name) += 1 > -1 from me. It does not solve a common problem, therefore it does not deserve a special syntax. Moreover, the existing syntax may be longer to type but is far more readable. -- Gustavo J. A. M. Carneiro "The universe is always one step beyond logic." -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070212/680055fe/attachment.html From ben at redfrontdoor.org Mon Feb 12 14:29:43 2007 From: ben at redfrontdoor.org (Ben North) Date: Mon, 12 Feb 2007 13:29:43 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access Message-ID: <1171286983.45d06bc75b001@imp.hosting365.ie> Thanks for the comments so far on this. First, on the general idea: Neil Toronto: > I like it. > [...] > > obj.(attr_name) += 1 > Even nicer; much, much better than the current spelling. Brett Cannon: > +0 on the one-item version. Anthony Baxter: > -1 from me. Collin Winter: > I like the general idea [...] I'm +0 on the idea Jack Jansen: > I like the functionality, Raymond Hettinger: > I also like the functionality. Generally gently positive, with the exception of Anthony Baxter's "-1", which I understand to be motivated by concerns about newcomers to the syntax: > Someone coming across this syntax for the first time will not have any > hints as to what it means - and worse, it looks like a syntax error to > me. This was echoed by Collin Winter: > Also, like Anthony Baxter said, someone coming across this for the > first time will think it's a syntax error, allusions to MATLAB and > assembly indirection syntax not withstanding. [...] -1 on the means. I think the argument "someone who hasn't come across it before won't know what it is" could be applied to any new syntax, for instance the new "with" statement, or going further back in time, the introduction of Booleans, semantics of "__slots__", the "extended print" syntax, the "list comprehension" syntax, the "augmented assignment" syntax, etc. Some of these were perhaps more transparent than others, but for instance "with" needs a bit of study to fully understand what's going on. I would argue that the newness itself is not sufficient reason to reject this. Also, I think it's a good thing that it looks like a syntax error --- it *is* currently a syntax error. A newcomer will be alerted to the fact that something new is going on and will know to go and find out about it. I am happy to write a paragraph for the "what's new" release notes, and/or new section(s) for the documentation describing the new syntax. On the two-argument form, feeling was generally pretty negative, with a few "-1"s thrown in. I propose to cut this piece out of the PEP, leaving just the one-argument form. If a default value is required, the coder can still use the three-argument form of "getattr". To be clear (and I'll put this in the PEP), I'm *not* suggesting that python loses "getattr", "setattr", or "delattr". On the syntax: Brett Cannon: > It just makes it look like too much of a function call at that point > to me Collin Winter: > the syntax looks like dirt on my monitor. The period is too easy to > lose visually and without it, there's nothing to distinguish this from > a function call. Jack Jansen: > I don't like the syntax, to me it looks too much like a method call. (Some of the initial comments in python-ideas were along these lines too.) My personal opinion is that "obj.(foo)" is distinct enough from "obj(foo)" to not cause confusion, but perhaps in a proportional font it is less clear. Some people made concrete suggestions as to what syntax could be used instead: Jack Jansen: > To me self.[method_name] = self.metadata.[method_name] looks better: > what we're doing here is more like dictionary lookup than calling > functions. In the same way, though, would this be viewed as too similar to normal dictionary/list indexing? Raymond Hettinger: > Rather than munge existing syntaxes, an altogether new one would be > more clear: > > self->name = self.metadata->name One thing which comes to mind about this one is that, for C/C++ programmers, the difference between obj.member and obj->member is the interpretation of the thing on the *left* of the dot or arrow, whereas the PEP is discussing a new interpretation of the thing on the *right* of the dot. One idea mentioned in the PEP is to bring {}s into service here, but I think the dot should be kept to keep it looking like an attribute access. What would the reaction be to obj.{member} instead? Braces are already used to construct dictionaries, so this has some of the right connotations. It could not be confused with either a function call or a dictionary lookup/list indexing. (Nobody had any comments on the potential 1% performance hit --- is this because it's too early to worry about implementation details?) Thanks, Ben. From richard.m.tew at gmail.com Mon Feb 12 14:46:43 2007 From: richard.m.tew at gmail.com (Richard Tew) Date: Mon, 12 Feb 2007 13:46:43 +0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45CF8B72.2000808@v.loewis.de> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <952d92df0702111105q1b81f106rd3516461408f3035@mail.gmail.com> <45CF8B72.2000808@v.loewis.de> Message-ID: <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> On 2/11/07, "Martin v. L?wis" wrote: > Richard Tew schrieb: > > If these generator coroutine microthreads went ahead and part > > of it was improving the support for asynchronous calls in the > > runtime and standard library, this would also be something > > which also benefited Stackless Python. Even if they didn't go > > ahead I would be interested in hearing about whether these > > sort of changes would be of interest or not for whatever reason. > > For me to respond to such a question, I first need to understand > what changes precisely you propose. "wrapping calls out of the Python > runtime to things like network and file IO, so that microthreads > can naturally take advantage of them." is a bit vague: how precisely > would you wrap them? I was meaning wrapping on the Stackless side of things and it was not meant as a reference to how better support for asynchronous calls might be added. You can see how wrapping is done with Stackless channels in the examples I linked to at the end of this mail on the off chance you were curious about that instead and I am happy to give any further detail needed. > I assume you would like to see operating system mechanism be used > that currently aren't used (similar to the way asynchore uses > select/poll). Yes. > If so, what mechanism would you like to use, and on what systems? It depends what you mean by mechanism. I can't go into low level details because I do not have the necessary experience or knowledge to know which approach would be best. But I do have some general thoughts and would be willing to do any work involved, if there was a feasible solution and the changes required were thought worthwhile. Let me describe the situation I currently face in order to give context, please excuse the Windows specifics and the lack of consideration for other platforms. I currently use Windows. By using asyncore and monkeypatching in a replacement socket module based on it [1] I can give users of Stackless socket operations which invisibly block at the microthread level allowing the scheduler to continue to run. However there is no support for asynchronous file IO. Now I can monkeypatch in a replacement file object which uses IO completion ports [2], but at that stage I need to poll two different resources for events. In order to avoid this it makes sense to stop using asyncore for sockets and to write a new replacement socket object based on IO completion ports. Then there are other calls which come into the mix. Like in the subprocess module. As I understand it I would need to monkeypatch the Popen.wait, and os.waitpid calls so that they were polled internally to the monkeypatching and Stackless could handle the wake up of the caller there as well. And there might be other blocking calls in the runtime which it would be worth handling also, which might not have methods any way of operating asynchronously (like sockets, files and subprocesses). The ideal mechanism at the high level would be expanding asyncore into a "one-stop shop". Where all these things can be passed into it and it can do the work to notify of events on the objects in a standard way. However even if this were judged feasible, one immediate hurdle is that files passed to it would need to have been opened in overlapped mode for IO completion ports could be used with it. I presume generally opening all files in overlapped mode on Windows would address this thought. And I guess all the other "stuff" which does not relate to IO completion ports like subprocesses would be bundled into a WaitForMultipleObject call. Perhaps there is a better way. And I of course have no concept of how this might be done on other platforms. Thanks, Richard. [1] http://svn.python.org/view/stackless/sandbox/examples/stacklesssocket.py [2] http://svn.python.org/view/stackless/sandbox/libraries/slpmonkeypatch/resources/iocp.py From ben at redfrontdoor.org Mon Feb 12 15:40:53 2007 From: ben at redfrontdoor.org (Ben North) Date: Mon, 12 Feb 2007 14:40:53 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access Message-ID: <1171291253.45d07c75297b5@imp.hosting365.ie> Apologies: I overlooked a couple of replies in my summary earlier. Tim Delaney and Terry Reedy both responded in positive terms to the one-argument form and its syntax, and in negative terms to the two-argument form. Also, I missed the fact that Neil Toronto had made the same point as me when he said: > I'm not sure the "looks like a syntax error" argument holds much weight, > because any new syntax is likely to be a syntax error until the syntax > is changed. :) he also suggested: > obj.*str_expression but I'm a bit uneasy about this, although the similarity with C's dereferencing is in its favour. Also, '*' in python is already used for "and the rest" arguments in function calls. Anthony Baxter cooled off a bit on the idea too: > after thinking about it a bit, I'm still not convinced this is > something that needs shorthand syntax Maybe not, but for the cases where you do want to do dynamic attribute access, I think there is a win in readability from nested getattr and/or setattr calls. Georg Brandl: > -1 here too. I fear that this gets too indistinguishable from normal > calling syntax, I think, from the context, that this is "-1" on the syntax rather than the idea as a whole, but I could have misread Georg's message. Possibly a "-0" on the idea? Greg Ewing: > In my experience, the amount of code which truly needs > to use getattr is extremely small. Another "-0"? Gustavo Carneiro: > -1 from me. It does not solve a common problem, therefore it does not > deserve a special syntax. It's not *that* uncommon, judging by the c.600 uses in the existing library code. > Moreover, the existing syntax may be longer > to type but is far more readable. I disagree, although I can see that there might be a small time during which one is getting familiar with the new syntax. Others have voiced the opinion that it's a readability win. Tim Delaney asked in particular: > Have you checked if [the existing uses of getattr, where "getattr" in > that scope is a function argument with default value the built-in > "getattr"] are intended to bring the "getattr" name into local scope > for fast lookup, or to force a binding to the builtin "gettattr" at > compile time (two common (ab)uses of default arguments)? If they are, > they would be better served by the new syntax. They're all in Lib/codecs.py, and are of the form: class StreamRecoder: def __getattr__(self, name, getattr=getattr): """ Inherit all other methods from the underlying stream. """ return getattr(self.stream, name) Without digging deeper into that code I'm afraid I can't say precisely what is going on. Ben. From ncoghlan at gmail.com Mon Feb 12 16:02:18 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Feb 2007 01:02:18 +1000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <1171286983.45d06bc75b001@imp.hosting365.ie> References: <1171286983.45d06bc75b001@imp.hosting365.ie> Message-ID: <45D0817A.4020805@gmail.com> Ben North wrote: > Thanks for the comments so far on this. Count me as a +0 on the general idea, -1 on the specific proposed syntax (based on the 'syntax shall not look like grit on Tim's monitor' guideline, and the fact that nested parentheses make it hard to separate the dynamic attribute lookup from function calls in the same expression) I'd probably be +0 for a "x.{some_str}" brace based syntax, with the caveat that I think this would also prevent future use of the "x{whatever}" syntax. The 'probably' is based on the fact that I'd like to see some examples of standard library code converted to that syntax. And a -1 on the two argument version either way (we can keep getattr around for that). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From g.brandl at gmx.net Mon Feb 12 16:19:57 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 12 Feb 2007 16:19:57 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <1171291253.45d07c75297b5@imp.hosting365.ie> References: <1171291253.45d07c75297b5@imp.hosting365.ie> Message-ID: Ben North schrieb: > Apologies: I overlooked a couple of replies in my summary earlier. Tim > Delaney and Terry Reedy both responded in positive terms to the > one-argument form and its syntax, and in negative terms to the > two-argument form. > > Also, I missed the fact that Neil Toronto had made the same point as me > when he said: >> I'm not sure the "looks like a syntax error" argument holds much weight, >> because any new syntax is likely to be a syntax error until the syntax >> is changed. :) IMO there's a difference between e.g. "with" and this new syntax. Looking at code that uses the "with" statement, one cannot think other than "ah, a new keyword, let's look what it means". Here, when someone comes across "obj.(name)" the situation is different. You also have to take into account that not all programmers know the language to the fullest extent, so it won't only be newbies who'll be confused and probably say "hey, that doesn't look right to me". > he also suggested: >> obj.*str_expression > > but I'm a bit uneasy about this, although the similarity with C's > dereferencing is in its favour. Also, '*' in python is already used for > "and the rest" arguments in function calls. Additionally, if you want to use a more complicated expression, you'll still have to put parentheses there. > Anthony Baxter cooled off a bit on the idea too: >> after thinking about it a bit, I'm still not convinced this is >> something that needs shorthand syntax > > Maybe not, but for the cases where you do want to do dynamic attribute > access, I think there is a win in readability from nested getattr and/or > setattr calls. As said in the "grit" argument, not necessarily. It's more easily overlooked that "getattr". > Georg Brandl: >> -1 here too. I fear that this gets too indistinguishable from normal >> calling syntax, > > I think, from the context, that this is "-1" on the syntax rather than > the idea as a whole, but I could have misread Georg's message. Possibly > a "-0" on the idea? A +0 on the idea, but as I wrote in that post, I doubt an ideal and Pythonic syntax can be found here. > Tim Delaney asked in particular: >> Have you checked if [the existing uses of getattr, where "getattr" in >> that scope is a function argument with default value the built-in >> "getattr"] are intended to bring the "getattr" name into local scope >> for fast lookup, or to force a binding to the builtin "gettattr" at >> compile time (two common (ab)uses of default arguments)? If they are, >> they would be better served by the new syntax. > > They're all in Lib/codecs.py, and are of the form: > > class StreamRecoder: > def __getattr__(self, name, > getattr=getattr): > > """ Inherit all other methods from the underlying stream. > """ > return getattr(self.stream, name) > > Without digging deeper into that code I'm afraid I can't say precisely > what is going on. Since that is a special method and ought to have the signature __getattr__(self, name), I think it's safe to assume that that's meant as an optimization. Georg From larry at hastings.org Mon Feb 12 16:33:44 2007 From: larry at hastings.org (Larry Hastings) Date: Mon, 12 Feb 2007 07:33:44 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <1171286983.45d06bc75b001@imp.hosting365.ie> References: <1171286983.45d06bc75b001@imp.hosting365.ie> Message-ID: <45D088D8.8010600@hastings.org> Ben North wrote: > Jack Jansen: > >> To me self.[method_name] = self.metadata.[method_name] looks better: >> what we're doing here is more like dictionary lookup than calling >> functions. >> > In the same way, though, would this be viewed as too similar to normal > dictionary/list indexing? > I think that's its strength; it's pleasantly symmetric to using [] on a dict: dict attribute behavior -------------------------------- [] .[] pulls out named thing, throws exception on failure, no default, is an lvalue .get() .getattr() pulls out named thing, returns default value on error (specifiable as second argument), not an lvalue List comprehensions look like funny lists, generator expressions look like funny expressions, I see no reason why attribute dereferencing shouldn't look like funny dict dereferencing. +1. > What would the reaction be to > obj.{member} > instead? Braces are already used to construct dictionaries, so this has > some of the right connotations. -0.5. You're not constructing anything, you're looking up something. For the record, I'm also -1 on the two-argument form--this isn't a function call. /larry/ From mal at egenix.com Mon Feb 12 16:43:58 2007 From: mal at egenix.com (M.-A. Lemburg) Date: Mon, 12 Feb 2007 16:43:58 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171291253.45d07c75297b5@imp.hosting365.ie> Message-ID: <45D08B3E.5020105@egenix.com> On 2007-02-12 16:19, Georg Brandl wrote: >> Tim Delaney asked in particular: >>> Have you checked if [the existing uses of getattr, where "getattr" in >>> that scope is a function argument with default value the built-in >>> "getattr"] are intended to bring the "getattr" name into local scope >>> for fast lookup, or to force a binding to the builtin "gettattr" at >>> compile time (two common (ab)uses of default arguments)? If they are, >>> they would be better served by the new syntax. >> They're all in Lib/codecs.py, and are of the form: >> >> class StreamRecoder: >> def __getattr__(self, name, >> getattr=getattr): >> >> """ Inherit all other methods from the underlying stream. >> """ >> return getattr(self.stream, name) >> >> Without digging deeper into that code I'm afraid I can't say precisely >> what is going on. > > Since that is a special method and ought to have the signature > __getattr__(self, name), I think it's safe to assume that that's meant > as an optimization. I can confirm that: it's a case of fast-local-lookup optimization. You can add a -1 from me to the list as well: I don't think that dynamic lookups are common enough to warrant new syntax. Even if you do add a new syntax for this, using parenthesis is a poor choice IMHO as the resulting code looks too much like a function call (e.g. "callable.(variable)"). Other choices would be square brackets [], but these have the same problem as they are in use for indexing. The only brackets that are not yet overloaded in the context of applying them to an object are curly brackets, so "callable.{variable}" would cause enough raising eyebrows to not think of a typo. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Feb 12 2007) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: From guido at python.org Mon Feb 12 16:55:22 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Feb 2007 07:55:22 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <1171286983.45d06bc75b001@imp.hosting365.ie> References: <1171286983.45d06bc75b001@imp.hosting365.ie> Message-ID: My perspective: - There's a lot of support for the basic idea, and only a few naysayers, so let's keep looking for a syntax that works. - There's near-universal dislike for the two-arg form, so let's drop that part of the proposal. - I can't recall that x.[y] has been proposed yet, but thinking about it, that actually makes more sense than x.(y). (In fact, in JavaScript you can write x[y] to the same effect. I wouldn't discount the JS example; JS is probably closer to Python than almost any other language currently in existence except for Boo, and JS has successfully borrowed from Python.) - I'm not too concerned by the '.' being such a small character with this new proposal. x[y] is a lot less common than x(y), so you'll look twice when you think you see x[y] and it doesn't make sense, and then you'll notice it's really x.[y], which you either know or don't, and in the latter case you'll be looking it up or asking around. PS Thanks to Ben for excellent summaries of the discussion so far! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy at alum.mit.edu Mon Feb 12 16:55:40 2007 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon, 12 Feb 2007 10:55:40 -0500 Subject: [Python-Dev] safety of Py_CLEAR and self Message-ID: I was wondering today how I could convince myself that a sequence of Py_CLEAR() calls in a tp_clear function was safe. Take for example a really trivial sequence of code on frame_clear(): Py_CLEAR(f->f_exc_type); Py_CLEAR(f->f_exc_value); Py_CLEAR(f->f_exc_traceback); Py_CLEAR(f->f_trace); We use Py_CLEAR() so that multiple calls to frame_clear() are safe. The first time this runs it sets exc_type to NULL before calling DECREF. This guarantees that a subsequent frame_clear() or frame_dealloc() won't attempt to DECREF it a second time. I think I understand why that's desireable and why it works. The primary risk is that via DECREF we may execute an arbitrary amount of Python code via weakref callbacks, finalizers, and code in other threads that gets resumed while the callbacks and finalizers are running. My question, specifically, then: Why it is safe to assume that f doesn't point to trash after a particular call to Py_CLEAR()? Any particular call to Py_CLEAR() could break the cycle that the object is involved in an lead to a call to frame_dealloc(). The frame could get returned to an obmalloc pool, returned to the OS, or just re-used by another object before we get back to Py_CLEAR(). It seems like such behavior would be exceedingly unlikely, but I can't convince myself that it is impossible. Which is it: improbable or impossible? If it is only improbable, should we figure out how to write code that is safe against such an improbable event? Jeremy From mithrandi-python-dev at mithrandi.za.net Mon Feb 12 16:35:04 2007 From: mithrandi-python-dev at mithrandi.za.net (Tristan Seligmann) Date: Mon, 12 Feb 2007 17:35:04 +0200 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <952d92df0702111105q1b81f106rd3516461408f3035@mail.gmail.com> <45CF8B72.2000808@v.loewis.de> <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> Message-ID: <20070212153503.GF14464@mithrandi.za.net> * Richard Tew [2007-02-12 13:46:43 +0000]: > I currently use Windows. By using asyncore and monkeypatching in a > replacement socket module based on it [1] I can give users of Stackless > socket operations which invisibly block at the microthread level allowing > the scheduler to continue to run. However there is no support for > asynchronous file IO. Now I can monkeypatch in a replacement file > object which uses IO completion ports [2], but at that stage I need to > poll two different resources for events. In order to avoid this it makes > sense to stop using asyncore for sockets and to write a new > replacement socket object based on IO completion ports. ...sounds kinda like the Twisted reactor. Except the IOCP implementation thereof is still underway, I believe. > Perhaps there is a better way. And I of course have no concept of > how this might be done on other platforms. Building on an existing framework that does this seems better than reinventing the wheel, for something of this magnitude. -- mithrandi, i Ainil en-Balandor, a faer Ambar -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20070212/fd2311c5/attachment.pgp From richard.m.tew at gmail.com Mon Feb 12 18:11:39 2007 From: richard.m.tew at gmail.com (Richard Tew) Date: Mon, 12 Feb 2007 17:11:39 +0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070212153503.GF14464@mithrandi.za.net> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <952d92df0702111105q1b81f106rd3516461408f3035@mail.gmail.com> <45CF8B72.2000808@v.loewis.de> <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> <20070212153503.GF14464@mithrandi.za.net> Message-ID: <952d92df0702120911v7b315221nf8bf22304df8eb1c@mail.gmail.com> On 2/12/07, Tristan Seligmann wrote: > * Richard Tew [2007-02-12 13:46:43 +0000]: > > Perhaps there is a better way. And I of course have no concept of > > how this might be done on other platforms. > > Building on an existing framework that does this seems better than > reinventing the wheel, for something of this magnitude. This to me seems to be the low level support you would build something like Twisted on top of. Pushing Twisted so that others can get it seems a little over the top. In any case I'm not building an application. I am transparently making standard library calls yield to a scheduler while whatever they would do happens asynchronously. Generic support for this allows me to make Stackless usage like normal Python usage but with microthreads. I would hope if Dustin went ahead with a generator based microthread solution he find it useful as well. Lumbering Stackless at least with a framework like Twisted pushes complexity up to the user, something Stackless cannot afford. Cheers, Richard. From hrvoje.niksic at avl.com Mon Feb 12 18:21:48 2007 From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=) Date: Mon, 12 Feb 2007 18:21:48 +0100 Subject: [Python-Dev] Interning string subtype instances Message-ID: <1170859141.32307.32.camel@localhost> I propose modifying PyString_InternInPlace to better cope with string subtype instances. Current implementation of PyString_InternInPlace does nothing and returns if passed an instance of a subtype of PyString_Type. This is a problem for applications that need to support string subtypes, but also must intern the strings for faster equivalence testing. Such an application, upon receiving a string subtype, will silently fail to work. There is good reason for PyString_InternInPlace not accepting string subtypes: since a subtype can have modified behavior, interning it can cause problems for other users of the interned string. I agree with the reasoning, but propose a different solution: when interning an instance of a string subtype, PyString_InternInPlace could simply intern a copy. This should be a fully backward compatible change because: 1) code that passes PyString instances (99.99% cases) will work as before, and 2) code that passes something else silently failed to intern the string anyway. Speed should be exactly the same as before, with the added benefit that interning PyString subtype instances now does something, but without the problems that interning the actual instance can produce. The patch could look like this. If there is interest in this, I can produce a complete patch. @@ -5,10 +5,6 @@ PyObject *t; if (s == NULL || !PyString_Check(s)) Py_FatalError("PyString_InternInPlace: strings only please!"); - /* If it's a string subclass, we don't really know what putting - it in the interned dict might do. */ - if (!PyString_CheckExact(s)) - return; if (PyString_CHECK_INTERNED(s)) return; if (interned == NULL) { @@ -25,6 +21,18 @@ *p = t; return; } + /* Make sure we don't intern a string subclass, since we don't + really know what putting it in the interned dict might do. */ + if (!PyString_CheckExact(s)) { + PyObject *copy; + copy = PyString_FromStringAndSize(PyString_AS_STRING(*p), + PyString_GET_SIZE(*p)); + if (!copy) + return; + Py_DECREF(*p); + *p = copy; + s = (PyStringObject *) copy; + } if (PyDict_SetItem(interned, (PyObject *)s, (PyObject *)s) < 0) { PyErr_Clear(); From richard.m.tew at gmail.com Mon Feb 12 18:36:55 2007 From: richard.m.tew at gmail.com (Richard Tew) Date: Mon, 12 Feb 2007 17:36:55 +0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D0141F.10904@v.loewis.de> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> Message-ID: <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> On 2/12/07, "Martin v. L?wis" wrote: > Armin Rigo schrieb: > > The history as I remember it is that Christian tried hard to integrate > > the first versions of Stackless with CPython, but was turned town by > > python-dev. > > Are there public records of that? As I remember it, Christian never > actually submitted a patch for inclusion (at least not in public; > he may have approached Guido in private). The oldest discussion > I could find quickly is from August 2000 (Python 2.0 and Stackless); > there, it was Ben Wolfson who asked whether it could be integrated. > I also found a message where I said that it couldn't be integrated > as-is, for a number of reasons I listed; I didn't mean to imply > that it can never be integrated. I can't point you to the posts, but I can point you to something Christian has written on the subject which may indicate why it was never actually submitted for inclusion. See "A Bit Of History" http://svn.python.org/view/stackless/trunk/Stackless/readme.txt I have not talked to Christian about it but as the current maintainer of Stackless I was under the impression that there was a complete lack of interest in even considering it for integration. If addition of microthreading is being considered please consider Stackless. One reason why Stackless is not so popular is that as a fork people shy away from it because it is not the main version. It would benefit Stackless to be integrated and I would be willing to do any work involved to either integrate or maintain it afterwards. Cheers, Richard. From guido at python.org Mon Feb 12 18:42:16 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Feb 2007 09:42:16 -0800 Subject: [Python-Dev] safety of Py_CLEAR and self In-Reply-To: References: Message-ID: Looking for where tp_clear() is being called, the only caller is line 713 in gmodule.c, which explicitly surrounds the call with an INCREF/DECREF pair. Perhaps that's the clue you're looking for? --Guido On 2/12/07, Jeremy Hylton wrote: > I was wondering today how I could convince myself that a sequence of > Py_CLEAR() calls in a tp_clear function was safe. Take for example a > really trivial sequence of code on frame_clear(): > > Py_CLEAR(f->f_exc_type); > Py_CLEAR(f->f_exc_value); > Py_CLEAR(f->f_exc_traceback); > Py_CLEAR(f->f_trace); > > We use Py_CLEAR() so that multiple calls to frame_clear() are safe. > The first time this runs it sets exc_type to NULL before calling > DECREF. This guarantees that a subsequent frame_clear() or > frame_dealloc() won't attempt to DECREF it a second time. I think I > understand why that's desireable and why it works. The primary risk > is that via DECREF we may execute an arbitrary amount of Python code > via weakref callbacks, finalizers, and code in other threads that gets > resumed while the callbacks and finalizers are running. > > My question, specifically, then: Why it is safe to assume that f > doesn't point to trash after a particular call to Py_CLEAR()? Any > particular call to Py_CLEAR() could break the cycle that the object is > involved in an lead to a call to frame_dealloc(). The frame could get > returned to an obmalloc pool, returned to the OS, or just re-used by > another object before we get back to Py_CLEAR(). It seems like such > behavior would be exceedingly unlikely, but I can't convince myself > that it is impossible. Which is it: improbable or impossible? If it > is only improbable, should we figure out how to write code that is > safe against such an improbable event? > > Jeremy > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ton.vanvliet at skynet.be Mon Feb 12 18:52:43 2007 From: ton.vanvliet at skynet.be (Ton van Vliet) Date: Mon, 12 Feb 2007 18:52:43 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <200702121850.38279.anthony@interlink.com.au> References: <45CFAA81.5040906@redfrontdoor.org> <200702121533.15305.anthony@interlink.com.au> <45D01973.9000802@cs.byu.edu> <200702121850.38279.anthony@interlink.com.au> Message-ID: On Mon, 12 Feb 2007 18:50:35 +1100, you wrote: >Yes and no. My point is that it's extremely similar to existing >syntax. (Worse yet, it looks the same but for what's possibly the >smallest and hardest-to-see character in any character set) > >"foo(baz)" vs "foo.(baz)" is... not good. To me (as a newbee to the language) I only see possible confusion if one gives 'static' looking examples like: x.(foo) += 1 which does indeed look a bit like a function call. However when giving more 'dynamic' looking examples like: x.('foo_%d' % n) += 1 I don't get confused at all and intuitively recognize the intention immediately. In this example I consider the parenthesis 'grouping operators' (which would not be the case when square brackets would have been used) So, +1 for the idea from me, since the main intention is to use it in a 'dynamic' context, and there it would improve readability. All IMHO as a newbee of course :) Ton. From python at rcn.com Mon Feb 12 18:52:42 2007 From: python at rcn.com (Raymond Hettinger) Date: Mon, 12 Feb 2007 09:52:42 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access References: <1171286983.45d06bc75b001@imp.hosting365.ie> Message-ID: <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> [Raymond Hettinger] >> Rather than munge existing syntaxes, an altogether new one would be >> more clear: >> >> self->name = self.metadata->name [Ben North] > One thing which comes to mind about this one is that, for C/C++ > programmers, the difference between > > obj.member and obj->member > > is the interpretation of the thing on the *left* of the dot or arrow, > whereas the PEP is discussing a new interpretation of the thing on the > *right* of the dot. Try not to get hung-up on meanings from other languages. Any simple syntax will have associations in other languages. It is more important that we don't create a syntax which already has strong associations in Python (i.e. curly braces, dots, and square brackets). Those syntaxes would make the language harder to mentally parse. I would like to give the -> syntax a chance as is it simple and it is provides a nice visual distinction between closely related concepts: a.name -- getattr(a, 'name') a->name -- getattr(a, name) Raymond From jeremy at alum.mit.edu Mon Feb 12 19:04:21 2007 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon, 12 Feb 2007 13:04:21 -0500 Subject: [Python-Dev] safety of Py_CLEAR and self In-Reply-To: References: Message-ID: On 2/12/07, Guido van Rossum wrote: > Looking for where tp_clear() is being called, the only caller is line > 713 in gmodule.c, which explicitly surrounds the call with an > INCREF/DECREF pair. Perhaps that's the clue you're looking for? Yes, of course. The INCREF guarantees that the object can't be collected until the tp_clear() returns. Thanks. Jeremy > > --Guido > > On 2/12/07, Jeremy Hylton wrote: > > I was wondering today how I could convince myself that a sequence of > > Py_CLEAR() calls in a tp_clear function was safe. Take for example a > > really trivial sequence of code on frame_clear(): > > > > Py_CLEAR(f->f_exc_type); > > Py_CLEAR(f->f_exc_value); > > Py_CLEAR(f->f_exc_traceback); > > Py_CLEAR(f->f_trace); > > > > We use Py_CLEAR() so that multiple calls to frame_clear() are safe. > > The first time this runs it sets exc_type to NULL before calling > > DECREF. This guarantees that a subsequent frame_clear() or > > frame_dealloc() won't attempt to DECREF it a second time. I think I > > understand why that's desireable and why it works. The primary risk > > is that via DECREF we may execute an arbitrary amount of Python code > > via weakref callbacks, finalizers, and code in other threads that gets > > resumed while the callbacks and finalizers are running. > > > > My question, specifically, then: Why it is safe to assume that f > > doesn't point to trash after a particular call to Py_CLEAR()? Any > > particular call to Py_CLEAR() could break the cycle that the object is > > involved in an lead to a call to frame_dealloc(). The frame could get > > returned to an obmalloc pool, returned to the OS, or just re-used by > > another object before we get back to Py_CLEAR(). It seems like such > > behavior would be exceedingly unlikely, but I can't convince myself > > that it is impossible. Which is it: improbable or impossible? If it > > is only improbable, should we figure out how to write code that is > > safe against such an improbable event? > > > > Jeremy > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > From talin at acm.org Mon Feb 12 19:25:31 2007 From: talin at acm.org (Talin) Date: Mon, 12 Feb 2007 10:25:31 -0800 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> Message-ID: <45D0B11B.3080309@acm.org> Richard Tew wrote: > See "A Bit Of History" > http://svn.python.org/view/stackless/trunk/Stackless/readme.txt I admit that I haven't given Stackless more than a cursory look over, but it seems to me that the real source of its complexity is because its trying to add a fundamental architectural feature to Python without completely re-writing it. Writing a purely stateless, "stack-less" language interpreter is not that hard, as long as you are willing to drink the KoolAid from the very start - in other words, you don't allow any function call interfaces which are not continuable. This does make writing C extension functions a little bit harder than before, as you now have to explicitly manage all of your local variables instead of just pushing them on the hardware stack. (Actually, for "simple" C subroutines that don't call any other interpreted functions, you can write it like a normal C functions just as you always have.) The nightmare comes when you try to glue all this onto an existing language interpreter, which wasn't written from the start with these principles in mind - AND which doesn't want to suffer the impact of a large number of global changes. To be honest, I can't even imagine how you would do such a thing, and the fact that Stackless has done it is quite impressive to me. Now, I'm very interested in Stackless, but, as you say, like many people I've tended to shy away from using a fork. Now, the question I would like to ask the Stackless people is: Rather than thinking about wholesale integration of Stackless into Python mainline, what would make it easier to maintain the Stackless fork? In other words, what would you change about the current Python (it could be a global change) that would make your lives easier? What I'd like to see is for the Stackless people to come up with a list of design principles which they would like to see the implementation of Python conform to. Things like "All C functions should conform to such and such calling convention". What I am getting at is that rather that doing heroic efforts to add stackless-ness to the current Python code base without changing it, instead define a migration path which allows Python to eventually acquire the characteristics of a stackless implementation. The idea is to gradually shrink the actual Stackless patches to the point where they become small enough that a direct patch becomes uncontroversial. -- Talin From martin at v.loewis.de Mon Feb 12 19:44:41 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Feb 2007 19:44:41 +0100 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> Message-ID: <45D0B599.8040608@v.loewis.de> Richard Tew schrieb: > I can't point you to the posts, but I can point you to something > Christian has written on the subject which may indicate why > it was never actually submitted for inclusion. > > See "A Bit Of History" > http://svn.python.org/view/stackless/trunk/Stackless/readme.txt > > I have not talked to Christian about it but as the current maintainer > of Stackless I was under the impression that there was a complete > lack of interest in even considering it for integration. I think this comes from a major misunderstanding of how free software works. This is not commercial software, so there is no "company agenda" that drives it forward. There are various contributors, each having their own agenda; for most of us, it's "scratch your own itches". Surely, the voices of people have varying significance (in Python, Guido is the BDFL, for example), but "there was a complete lack of interest in even considering" just cannot be true: clearly, some contributors and users *were* interested. There were surely objections to the implementation strategy, and I also had objections to the specific execution of this implementation strategy, see http://tinyurl.com/27v23r Now, the question is whose "job" it would have been to integrate Stackless to Python. Apparently, Christian tried to make it happen (see http://tinyurl.com/3bqe5d), although I'm still uncertain whether he ever got to a point where he said: "this is a patch, please review it" (of that, I can find no archives, as I said). I find it understandable that nobody else took over to work on integrating it (although there is PEP 219, so apparently some people worked at some point on it). From my point of view, people suggesting to incorporate code as-is often misunderstand that it is not sufficient for the code to work in the applications were it is used. It must also be sound (i.e. work in some meaningful way also in cases for which it wasn't designed), and it must be maintainable. I can understand how discouraging it is when you see that your code "works" that then people object to incorporating it as-is. However, I believe the long-term quality of Python can only be guaranteed when careful review is applied to every change made, or else we will regret changes in the future (and indeed, Python 3000 would not have been necessary if no mistakes had been made). > If addition of microthreading is being considered please consider > Stackless. One reason why Stackless is not so popular is that > as a fork people shy away from it because it is not the main > version. It would benefit Stackless to be integrated and I would > be willing to do any work involved to either integrate or maintain > it afterwards. This is what I'm talking about: nothing "is" ever considered. Instead, individuals make contributions, individuals review them, and individuals support and object to changes. There is no abstract python-dev entity that likes or dislikes changes. Individuals have individual opinions. Now, where is the SF patch that makes Stackless part of Python ?-) (and if you are serious about it, try to break it down into multiple pieces, each individually correct and useful; start with one where the effort isn't too big in case it gets rejected by BDFL pronouncement) I haven't reviewed the Stackless code in a while; last I looked, I found it was possible to crash the interpreter if you use it "incorrectly". I *personally* object to changes where it is easy to crash the interpreter, unless there is a clear specification when this may happen, and there is a systematic way to avoid that (this is the ctypes clause). Regards, Martin From martin at v.loewis.de Mon Feb 12 19:57:51 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Feb 2007 19:57:51 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> Message-ID: <45D0B8AF.7070903@v.loewis.de> Guido van Rossum schrieb: > PS Thanks to Ben for excellent summaries of the discussion so far! I'd like to second this. This is how PEPs ought to work. Regards, Martin From mike.klaas at gmail.com Mon Feb 12 19:59:40 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Mon, 12 Feb 2007 10:59:40 -0800 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <1170859141.32307.32.camel@localhost> References: <1170859141.32307.32.camel@localhost> Message-ID: <3d2ce8cb0702121059n14fbec2bl951ac5d60d0bcb21@mail.gmail.com> On 2/12/07, Hrvoje Nik?i? wrote: > cause problems for other users of the interned string. I agree with the > reasoning, but propose a different solution: when interning an instance > of a string subtype, PyString_InternInPlace could simply intern a copy. Interning currently requires an external reference to prevent garbage collection (I believe). What will hold a reference to the string copy? -Mike From tim.peters at gmail.com Mon Feb 12 20:23:11 2007 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 12 Feb 2007 14:23:11 -0500 Subject: [Python-Dev] safety of Py_CLEAR and self In-Reply-To: References: Message-ID: <1f7befae0702121123i3f523085yb60aae42e5018a47@mail.gmail.com> [Jeremy Hylton] > I was wondering today how I could convince myself that a sequence of > Py_CLEAR() calls in a tp_clear function was safe. Take for example a > really trivial sequence of code on frame_clear(): > > Py_CLEAR(f->f_exc_type); > Py_CLEAR(f->f_exc_value); > Py_CLEAR(f->f_exc_traceback); > Py_CLEAR(f->f_trace); > > We use Py_CLEAR() so that multiple calls to frame_clear() are safe. > The first time this runs it sets exc_type to NULL before calling > DECREF. This guarantees that a subsequent frame_clear() or > frame_dealloc() won't attempt to DECREF it a second time. I think I > understand why that's desireable and why it works. The primary risk > is that via DECREF we may execute an arbitrary amount of Python code > via weakref callbacks, finalizers, and code in other threads that gets > resumed while the callbacks and finalizers are running. > > My question, specifically, then: Why it is safe to assume that f > doesn't point to trash after a particular call to Py_CLEAR()? Any > particular call to Py_CLEAR() could break the cycle that the object is > involved in an lead to a call to frame_dealloc(). The frame could get > returned to an obmalloc pool, returned to the OS, or just re-used by > another object before we get back to Py_CLEAR(). It seems like such > behavior would be exceedingly unlikely, but I can't convince myself > that it is impossible. Which is it: improbable or impossible? If it > is only improbable, should we figure out how to write code that is > safe against such an improbable event? As Guido pointed out, tp_clear is called by gc from only one place, which sticks an incref/decref pair around the call so that the refcount on `f` can't fall to 0 while frame_clear() is active. That doesn't mean frame_clear is always safe to call, it only means that gc's use of the tp_clear slot is safe. Nobody else "should be" calling frame_clear (and it so happens nothing else in the core does), but it's something to be dimly aware of. For example, IIRC, some of ZODB's C code internally invokes its XXX_clear() functions directly, as part of removing persistent object state (unrelated to object deallocation). Determining whether those kinds of uses are safe requires case-by-case analysis. From martin at v.loewis.de Mon Feb 12 20:25:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Feb 2007 20:25:12 +0100 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <952d92df0702111105q1b81f106rd3516461408f3035@mail.gmail.com> <45CF8B72.2000808@v.loewis.de> <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> Message-ID: <45D0BF18.7000404@v.loewis.de> Richard Tew schrieb: > It depends what you mean by mechanism. I can't go into low level details > because I do not have the necessary experience or knowledge to know > which approach would be best. Indeed, low-level details is what I'm interested in. > I currently use Windows. By using asyncore and monkeypatching in a > replacement socket module based on it [1] I can give users of Stackless > socket operations which invisibly block at the microthread level allowing > the scheduler to continue to run. However there is no support for > asynchronous file IO. That's actually a limitation of Windows: in a UNIX system, you can pass sockets and file descriptors alike to select(2) or poll(2) (in fact, sockets *are* file descriptors). > Now I can monkeypatch in a replacement file > object which uses IO completion ports [2] Ok, so you want to use IO completion ports on Windows. This is the mechanism I was asking for (overlapped IO would have been another mechanism available on Win32). Adding support for IO completion ports is certainly something that would be worthwhile. > but at that stage I need to > poll two different resources for events. In order to avoid this it makes > sense to stop using asyncore for sockets and to write a new > replacement socket object based on IO completion ports. I don't know whether/how this is possible. The underlying WaitForMultipleObjects routine does not accept WinSock objects, to my knowledge (perhaps it does in Vista, with the rewriting of WinSock?). However, if it would be possible, I would suggest to add support for this into the select module, and allow then to pass non-socket objects to select on Windows (just as it is possible on Unix). If this can't work (as you need to associate the file handle with a io completion port, after which you can't use ReadFile anymore), a new poll-like routine may be added, and then support for that may be added to asyncore. Notice that WaitForMultipleObjects is limited to MAXIMUM_WAIT_OBJECTS, which I believe is 64. So this is somewhat limiting. > Perhaps there is a better way. And I of course have no concept of > how this might be done on other platforms. For file descriptors, other platforms are ahead of Windows. For child process IDs, it's somewhat more tricky: you get a POSIX signal when some of them terminates, and any system call should terminate with EINTR. So the universal poll routine would abort when a child dies, and could then check a global variable set by the signal handler. In any case, one needs to specify such a "universal event loop" precisely, or else it will just fail. Look at Tcl's even mechanism for an example that has failed (IMO). Regards, Martin From ironfroggy at gmail.com Mon Feb 12 20:31:32 2007 From: ironfroggy at gmail.com (Calvin Spealman) Date: Mon, 12 Feb 2007 14:31:32 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> Message-ID: <76fd5acf0702121131r2ce9602cs5de810c4e587ef74@mail.gmail.com> I'll take this opportunity to pipe in with a response, since Guido summed up the many issues nicely and its a good bouncing point to mention my own thoughts. On 2/12/07, Guido van Rossum wrote: > My perspective: > > - There's a lot of support for the basic idea, and only a few > naysayers, so let's keep looking for a syntax that works. > > - There's near-universal dislike for the two-arg form, so let's drop > that part of the proposal. > > - I can't recall that x.[y] has been proposed yet, but thinking about > it, that actually makes more sense than x.(y). (In fact, in JavaScript > you can write x[y] to the same effect. I wouldn't discount the JS > example; JS is probably closer to Python than almost any other > language currently in existence except for Boo, and JS has > successfully borrowed from Python.) I'll second that. The JS case is valuable and a good representation of the mindset behind using foo.[bar] > - I'm not too concerned by the '.' being such a small character with > this new proposal. x[y] is a lot less common than x(y), so you'll look > twice when you think you see x[y] and it doesn't make sense, and then > you'll notice it's really x.[y], which you either know or don't, and > in the latter case you'll be looking it up or asking around. I definitely see the downside of using the nearly invisible . here. Looking like dirt on the screen, being misunderstood, deleted because someone thought it was a typo, etc. could all be problems. I am not convinced they will be problems, only that they could. Now, do we need a more readable attribute deference or is there some optimizational need to make a syntax specifically for it? Why can't we just wrap the object in question with some attribute lookup type? attr(o)[methodname]() vs o.[methodname]() I think if you read those two, at a quick glance the first is a lot more readable than the second. It also has the advantage of not modifying the language syntax, or introducing nearly as much new code. -- Read my blog! I depend on your acceptance of my opinion! I am interesting! http://ironfroggy-code.blogspot.com/ From martin at v.loewis.de Mon Feb 12 20:33:22 2007 From: martin at v.loewis.de (=?ISO-8859-2?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 12 Feb 2007 20:33:22 +0100 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <3d2ce8cb0702121059n14fbec2bl951ac5d60d0bcb21@mail.gmail.com> References: <1170859141.32307.32.camel@localhost> <3d2ce8cb0702121059n14fbec2bl951ac5d60d0bcb21@mail.gmail.com> Message-ID: <45D0C102.9050102@v.loewis.de> Mike Klaas schrieb: >> cause problems for other users of the interned string. I agree with the >> reasoning, but propose a different solution: when interning an instance >> of a string subtype, PyString_InternInPlace could simply intern a copy. > > Interning currently requires an external reference to prevent garbage > collection (I believe). What will hold a reference to the string > copy? For PyString_InternInPlace, the caller will receive a reference; the original object passed in is decref'ed. A typical caller of PyString_InternInPlace will just store the reference in some global/static variable. builtin_intern will return it to the caller, which then needs to store it. This was always the case with intern(); the usage pattern is foo = intern(foo) as intern may return a different object (e.g. if the string was already interned). Regards, Martin From martin at v.loewis.de Mon Feb 12 20:34:18 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Mon, 12 Feb 2007 20:34:18 +0100 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <1170859141.32307.32.camel@localhost> References: <1170859141.32307.32.camel@localhost> Message-ID: <45D0C13A.7030108@v.loewis.de> Hrvoje Nik?i? schrieb: > The patch could look like this. If there is interest in this, I can > produce a complete patch. I can't see a problem with that (although I do wonder why people create string subtypes in the first place). Regards, Martin From jcarlson at uci.edu Mon Feb 12 21:29:37 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 12 Feb 2007 12:29:37 -0800 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <1170859141.32307.32.camel@localhost> References: <1170859141.32307.32.camel@localhost> Message-ID: <20070212122602.ACCF.JCARLSON@uci.edu> Hrvoje Nik?ic wrote: > > I propose modifying PyString_InternInPlace to better cope with string > subtype instances. Any particular reason why the following won't work for you? _interned = {} def intern(st): #add optional type checking here if st not in _interned: _interned[st] = st return _interned[st] If I remember the implementation of intern correctly, that's more or less what happens under the covers. - Josiah From jcarlson at uci.edu Mon Feb 12 21:38:06 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 12 Feb 2007 12:38:06 -0800 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D0BF18.7000404@v.loewis.de> References: <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> <45D0BF18.7000404@v.loewis.de> Message-ID: <20070212123238.ACD2.JCARLSON@uci.edu> "Martin v. L?wis" wrote: > Richard Tew schrieb: > > but at that stage I need to > > poll two different resources for events. In order to avoid this it makes > > sense to stop using asyncore for sockets and to write a new > > replacement socket object based on IO completion ports. > > I don't know whether/how this is possible. The underlying > WaitForMultipleObjects routine does not accept WinSock objects, > to my knowledge (perhaps it does in Vista, with the rewriting > of WinSock?). It "works" in Windows 2000. I've got a variant of a prototype (and abandoned) Twisted Reactor that uses WaitForMultipleObjects for an alternate asyncore.poll() implementation on Windows. It sucks up 100% processor when sending trivial amounts of data from a single socket (where asyncore uses about 2%), but you do get the data. - Josiah From grig.gheorghiu at gmail.com Mon Feb 12 21:50:17 2007 From: grig.gheorghiu at gmail.com (Grig Gheorghiu) Date: Mon, 12 Feb 2007 12:50:17 -0800 Subject: [Python-Dev] Any value in setting up pybots for py3k? Message-ID: <3f09d5a00702121250j73fef0f8k3baf114db6276b9f@mail.gmail.com> Sidnei da Silva asked on the pybots mailing list if we should be testing packages with py3k. I think it's probably too early for that, but on the other hand I'm sure many package creators/maintainers would be curious to see how their packages fare with py3k. So is there any value or interest in setting up a svn notification hook for py3k commits that would go to the pybots buildbot master? Grig -- http://agiletesting.blogspot.com From brett at python.org Mon Feb 12 22:47:51 2007 From: brett at python.org (Brett Cannon) Date: Mon, 12 Feb 2007 13:47:51 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <000501c74e88$0da367c0$6a176b0a@RaymondLaptop1> References: <45CFAA81.5040906@redfrontdoor.org> <000501c74e88$0da367c0$6a176b0a@RaymondLaptop1> Message-ID: On 2/12/07, Raymond Hettinger wrote: > [Jack Jansen] > > I like the functionality, but I don't like the syntax, to me it looks > > too much like a method call. > > > > To me self.[method_name] = self.metadata.[method_name] looks better: > > what we're doing here is more like dictionary lookup than calling > > functions. > > I also like the functionality. > > Rather than munge existing syntaxes, an altogether new one would be more clear: > > self->name = self.metadata->name > > I like the arrow syntax because is the lookup process can be more involved > than a simple dictionary lookup (perhaps traveling up to base classes). > IOW, getattr(a,n) is not always the same as a.__dict__[n]. > The a.__getattribute__(n) process can be more complex than that > and a bracketed dictionary-like syntax would misleadingly mask the lookup > process. > I actually kind of like that. The connection to pointer indirection meshes well with the idea of indirectly figuring out what attribute to access at runtime. -Brett From brett at python.org Mon Feb 12 23:04:03 2007 From: brett at python.org (Brett Cannon) Date: Mon, 12 Feb 2007 14:04:03 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> Message-ID: On 2/12/07, Raymond Hettinger wrote: > [Raymond Hettinger] > >> Rather than munge existing syntaxes, an altogether new one would be > >> more clear: > >> > >> self->name = self.metadata->name > > [Ben North] > > One thing which comes to mind about this one is that, for C/C++ > > programmers, the difference between > > > > obj.member and obj->member > > > > is the interpretation of the thing on the *left* of the dot or arrow, > > whereas the PEP is discussing a new interpretation of the thing on the > > *right* of the dot. > > Try not to get hung-up on meanings from other languages. > Any simple syntax will have associations in other languages. An example of where Python has been willing in the past to hijack syntax and give it a tweaked meaning is decorators compared to Java annotations. > It is more important that we don't create a syntax which already > has strong associations in Python (i.e. curly braces, dots, and square > brackets). > Those syntaxes would make the language harder to mentally parse. > I think this is a nice argument against using square brackets as well. Currently square brackets are used for getting an item from a container (__getitem__) or generating one (list comprehensions). While using the .[] syntax does connect with the "container" idea if you want to treat an object as a container and its attributes as its items, I don't think the connection is that strong. Guido said he would be much more willing to look twice at a use of brackets for a dot, but that seems to require mental backtracking to me. If I am skimming some source and I see square brackets, right now I know that either something has defined __getitem__ and is probably a dict or list, or it is a listcomp if it contains 'for'. But with this I would now have to stop and look for that dot which could be lost in the noise. I think people are calling the syntax "dirt" because you have two pieces of syntax ('.' and '[') that are a single character. And on top of that they are extremely thin (which sucks for non-monospaced fonts). We don't have issues with dots these days since there is usually several characters of information specifying the meaning of what the dot is modified by the dot. But in this case there is very little visual queue for that. > I would like to give the -> syntax a chance as is it simple > and it is provides a nice visual distinction between closely > related concepts: > > a.name -- getattr(a, 'name') > a->name -- getattr(a, name) Yeah, I am definitely liking this one more. It's making the ``.[name]`` or ``.(name)`` look more like "dirt on Tim's monitor" to me. -Brett From collinw at gmail.com Mon Feb 12 23:42:07 2007 From: collinw at gmail.com (Collin Winter) Date: Mon, 12 Feb 2007 16:42:07 -0600 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <45CFAA81.5040906@redfrontdoor.org> <000501c74e88$0da367c0$6a176b0a@RaymondLaptop1> Message-ID: <43aa6ff70702121442v2d72d746u21ab920e3492fc0f@mail.gmail.com> On 2/12/07, Brett Cannon wrote: > On 2/12/07, Raymond Hettinger wrote: > > [Jack Jansen] > > > I like the functionality, but I don't like the syntax, to me it looks > > > too much like a method call. > > > > > > To me self.[method_name] = self.metadata.[method_name] looks better: > > > what we're doing here is more like dictionary lookup than calling > > > functions. > > > > I also like the functionality. > > > > Rather than munge existing syntaxes, an altogether new one would be more clear: > > > > self->name = self.metadata->name > > > > I like the arrow syntax because is the lookup process can be more involved > > than a simple dictionary lookup (perhaps traveling up to base classes). > > IOW, getattr(a,n) is not always the same as a.__dict__[n]. > > The a.__getattribute__(n) process can be more complex than that > > and a bracketed dictionary-like syntax would misleadingly mask the lookup > > process. > > > > I actually kind of like that. The connection to pointer indirection > meshes well with the idea of indirectly figuring out what attribute to > access at runtime. There's a connection, but I'd say it's the wrong one. In C, "x->y" dereferences x, while in Python, "x->y" would dereference y. That's just begging for trouble. Collin Winter From benji at benjiyork.com Mon Feb 12 23:48:59 2007 From: benji at benjiyork.com (Benji York) Date: Mon, 12 Feb 2007 17:48:59 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <43aa6ff70702121442v2d72d746u21ab920e3492fc0f@mail.gmail.com> References: <45CFAA81.5040906@redfrontdoor.org> <000501c74e88$0da367c0$6a176b0a@RaymondLaptop1> <43aa6ff70702121442v2d72d746u21ab920e3492fc0f@mail.gmail.com> Message-ID: <45D0EEDB.3080703@benjiyork.com> Collin Winter wrote: > On 2/12/07, Brett Cannon wrote: >> I actually kind of like that. The connection to pointer indirection >> meshes well with the idea of indirectly figuring out what attribute to >> access at runtime. > > There's a connection, but I'd say it's the wrong one. In C, "x->y" > dereferences x, while in Python, "x->y" would dereference y. That's > just begging for trouble. Then the syntax should obviously be "x<-y". [insert in-Soviet-Russia-variables-dereference-you joke here] -- Benji York From guido at python.org Mon Feb 12 23:52:45 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Feb 2007 14:52:45 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> Message-ID: FWIW, I'm strongly -1 on the "->" notation. As a C programmer it's got too many neurons committed to it. I recommend that you do some experiments with the readability of the .[...] notation, e.g. write a program that randomly generates x.[foo] and x[foo], and see how fast you can spot the difference. I bet that you won't have any trouble. --Guido On 2/12/07, Brett Cannon wrote: > On 2/12/07, Raymond Hettinger wrote: > > [Raymond Hettinger] > > >> Rather than munge existing syntaxes, an altogether new one would be > > >> more clear: > > >> > > >> self->name = self.metadata->name > > > > [Ben North] > > > One thing which comes to mind about this one is that, for C/C++ > > > programmers, the difference between > > > > > > obj.member and obj->member > > > > > > is the interpretation of the thing on the *left* of the dot or arrow, > > > whereas the PEP is discussing a new interpretation of the thing on the > > > *right* of the dot. > > > > Try not to get hung-up on meanings from other languages. > > Any simple syntax will have associations in other languages. > > An example of where Python has been willing in the past to hijack > syntax and give it a tweaked meaning is decorators compared to Java > annotations. > > > It is more important that we don't create a syntax which already > > has strong associations in Python (i.e. curly braces, dots, and square > > brackets). > > Those syntaxes would make the language harder to mentally parse. > > > > I think this is a nice argument against using square brackets as well. > Currently square brackets are used for getting an item from a > container (__getitem__) or generating one (list comprehensions). > While using the .[] syntax does connect with the "container" idea if > you want to treat an object as a container and its attributes as its > items, I don't think the connection is that strong. > > Guido said he would be much more willing to look twice at a use of > brackets for a dot, but that seems to require mental backtracking to > me. If I am skimming some source and I see square brackets, right > now I know that either something has defined __getitem__ and is > probably a dict or list, or it is a listcomp if it contains 'for'. > But with this I would now have to stop and look for that dot which > could be lost in the noise. > > I think people are calling the syntax "dirt" because you have two > pieces of syntax ('.' and '[') that are a single character. And on > top of that they are extremely thin (which sucks for non-monospaced > fonts). We don't have issues with dots these days since there is > usually several characters of information specifying the meaning of > what the dot is modified by the dot. But in this case there is very > little visual queue for that. > > > I would like to give the -> syntax a chance as is it simple > > and it is provides a nice visual distinction between closely > > related concepts: > > > > a.name -- getattr(a, 'name') > > a->name -- getattr(a, name) > > Yeah, I am definitely liking this one more. It's making the > ``.[name]`` or ``.(name)`` look more like "dirt on Tim's monitor" to > me. > > -Brett > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bjourne at gmail.com Tue Feb 13 00:02:43 2007 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Tue, 13 Feb 2007 00:02:43 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <1171286983.45d06bc75b001@imp.hosting365.ie> References: <1171286983.45d06bc75b001@imp.hosting365.ie> Message-ID: <740c3aec0702121502n7878df4nc2fe76b7f152244d@mail.gmail.com> Is even more syntactic sugar really what Python really needs? -- mvh Bj?rn From ben at redfrontdoor.org Tue Feb 13 00:10:34 2007 From: ben at redfrontdoor.org (Ben North) Date: Mon, 12 Feb 2007 23:10:34 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access Message-ID: <45D0F3EA.4090208@redfrontdoor.org> Guido van Rossum wrote: > - There's near-universal dislike for the two-arg form, so let's drop > that part of the proposal. This is a strong consensus, definitely, so we can conclude that this point has been decided. I will remove it from the PEP. Guido also wrote: > - There's a lot of support for the basic idea, and only a few > naysayers, so let's keep looking for a syntax that works. If we can take that as a decision, then the various proposals for the syntax break down as follows. By raw numbers, the most popular choice is self.(method_name) = self.metadata.(method_name) but many of the "I like that" messages were in the context of the idea as a whole, so can't really be counted as explicit votes for "obj.(foo)" as such. Next, and Guido's preferred choice, is self.[method_name] = self.metadata.[method_name] (I haven't been following long enough to know whether "Guido likes it" overrides everything else (the "D" of BDFL), or is more of a casting vote in the event of a tie.) With regard to the potential overlookability of the dot, Guido says: > I recommend that you do some experiments with the readability of the > .[...] notation, e.g. write a program that randomly generates x.[foo] > and x[foo], and see how fast you can spot the difference. I bet that > you won't have any trouble. I agree --- just as it's important to have a font that makes it easy to distinguish "I" from "l" and "1", "0" from "O", "(" from "{", etc., I would say it's important to program using a font which makes it easy to tell whether there's a "." in your code. I can imagine that a proportional font where "." might be a single pixel wouldn't help, though. (Gently wandering off-topic, but: do people use proportional fonts for coding? Doesn't it cause general awkwardness for indentation, especially relevant for python?) Also mentioned was self.{method_name} = self.metadata.{method_name} which could not be confused either with function call or indexing but perhaps would preclude braces from any potential future use. Exploring other ideas, the "arrow" notation self->method_name = self.metadata->method_name has support, and is clearly different, but I personally would be misled to think, from the meaning in C, that it ought to be the left-hand operand which is dereferenced somehow. Guido has the same opinion, and is "strongly -1" on this. The "star" form has the "dereference" meaning from C, and is certainly visually distinctive: self.*method_name = self.metadata.*method_name and explicit parentheses could be put in as a syntax requirement, or by individual coder preference, or for more complex attribute calculations: self.*(method_name) = self.metadata.*(method_name) self.*('foo_%d' % n) = self.metadata.*('foo_%d' % n) The downside is that it could be considered "visually distinctive" to the point of being line noise. Could we cut down the choice to self.[method_name] = self.metadata.[method_name] if the danger of overlooking the dot were deemed small enough, or self.*(method_name) = self.metadata.*(method_name) if the consensus was that something more noticeable was needed? (Or there's always the late arrival > > In C, "x->y" dereferences x, while in Python, "x->y" would dereference y. > > Then the syntax should obviously be "x<-y". > [insert in-Soviet-Russia-variables-dereference-you joke here] from Benji York.) Ben. From greg.ewing at canterbury.ac.nz Tue Feb 13 00:19:50 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Feb 2007 12:19:50 +1300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <1171286983.45d06bc75b001@imp.hosting365.ie> References: <1171286983.45d06bc75b001@imp.hosting365.ie> Message-ID: <45D0F616.8070005@canterbury.ac.nz> Ben North wrote: > Generally gently positive, with the exception of Anthony Baxter's > "-1", which I understand to be motivated by concerns about newcomers to > the syntax The more I think about it, the more I'm leaning towards -1 as well. Adding syntax is a very big step, and it needs a very solid reason to jusify it. I don't see this filling a use case that's anywhere near common enough to reach that threshold. -- Greg From flashk at gmail.com Tue Feb 13 00:22:37 2007 From: flashk at gmail.com (Farshid Lashkari) Date: Mon, 12 Feb 2007 15:22:37 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <740c3aec0702121502n7878df4nc2fe76b7f152244d@mail.gmail.com> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <740c3aec0702121502n7878df4nc2fe76b7f152244d@mail.gmail.com> Message-ID: <978d1eac0702121522q39c29950q5f28fced630555c4@mail.gmail.com> On 2/12/07, BJ?rn Lindqvist wrote: > Is even more syntactic sugar really what Python really needs? Yes, I need my fix!!! my 2 cents: I'm +1 on either the '.(name)' or '.[name]' syntax. I'm leaning more towards the parentheses though. I don't really buy into the argument that people will confuse it for a function call. That seems like more of an argument against the dot operator in general than the new syntax. I'm -1 on the '->' syntax. It just doesn't look very clean to me. -Farshid From imbaczek at gmail.com Tue Feb 13 00:24:31 2007 From: imbaczek at gmail.com (=?ISO-8859-2?Q?Marek_"Baczek"_Baczy=F1ski?=) Date: Tue, 13 Feb 2007 00:24:31 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D0EEDB.3080703@benjiyork.com> References: <45CFAA81.5040906@redfrontdoor.org> <000501c74e88$0da367c0$6a176b0a@RaymondLaptop1> <43aa6ff70702121442v2d72d746u21ab920e3492fc0f@mail.gmail.com> <45D0EEDB.3080703@benjiyork.com> Message-ID: <5f3d2c310702121524l27fa3763n12185251b2471f54@mail.gmail.com> 2007/2/12, Benji York : > Collin Winter wrote: > > There's a connection, but I'd say it's the wrong one. In C, "x->y" > > dereferences x, while in Python, "x->y" would dereference y. That's > > just begging for trouble. > > Then the syntax should obviously be "x<-y". Someone with OCaml background could confuse that with an assignment -- Marek Baczy?ski From greg.ewing at canterbury.ac.nz Tue Feb 13 00:33:46 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Feb 2007 12:33:46 +1300 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <952d92df0702111105q1b81f106rd3516461408f3035@mail.gmail.com> <45CF8B72.2000808@v.loewis.de> <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> Message-ID: <45D0F95A.4000507@canterbury.ac.nz> Richard Tew wrote: > The ideal mechanism at the high level would be expanding asyncore into > a "one-stop shop". Where all these things can be passed into it and > it can do the work to notify of events on the objects in a standard way. +1. This sounds like an excellent idea. It's downright silly having each thing that uses async I/O doing its own thing. There should be a standard mechanism in the stdlib that everything can use. -- Greg From brett at python.org Tue Feb 13 00:47:26 2007 From: brett at python.org (Brett Cannon) Date: Mon, 12 Feb 2007 15:47:26 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> Message-ID: On 2/12/07, Guido van Rossum wrote: > FWIW, I'm strongly -1 on the "->" notation. As a C programmer it's got > too many neurons committed to it. > > I recommend that you do some experiments with the readability of the > .[...] notation, e.g. write a program that randomly generates x.[foo] > and x[foo], and see how fast you can spot the difference. I bet that > you won't have any trouble. > OK, so real-world examples. First, ``foo.(name)``, from urllib.py:: name = 'open_' + urltype self.type = urltype name = name.replace('-', '_') if not hasattr(self, name): if proxy: return self.open_unknown_proxy(proxy, fullurl, data) else: return self.open_unknown(fullurl, data) try: if data is None: return self.(name)(url) else: return self.(name)(url, data) except socket.error, msg: raise IOError, ('socket error', msg), sys.exc_info()[2] and also:: name = 'http_error_%d' % errcode if hasattr(self, name): method = self.(name) if data is None: result = method(url, fp, errcode, errmsg, headers) else: result = method(url, fp, errcode, errmsg, headers, data) if result: return result return self.http_error_default(url, fp, errcode, errmsg, headers) And here is urllib2.py for ``.[]`` (used different files so you wouldn't just remember where the change was):: if attr[:12] == '_Request__r_': name = attr[12:] if hasattr(Request, 'get_' + name): self.['get_' + name]() return self.[attr] raise AttributeError, attr and:: handlers = chain.get(kind, ()) for handler in handlers: func = handler.[meth_name] result = func(*args) if result is not None: return result Neither version jumps out at me strongly, although between the two the ``.[]`` version shows up the best. But that might also be because of the lower noise when used in a call. -Brett From greg.ewing at canterbury.ac.nz Tue Feb 13 00:59:39 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Feb 2007 12:59:39 +1300 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D0B11B.3080309@acm.org> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> <45D0B11B.3080309@acm.org> Message-ID: <45D0FF6B.6040104@canterbury.ac.nz> Talin wrote: > What I am getting at is that rather that doing heroic efforts to add > stackless-ness to the current Python code base without changing it, > instead define a migration path which allows Python to eventually > acquire the characteristics of a stackless implementation. I think you've already answered your own question. This is not the sort of thing that can be done piecemeal -- it requires everything to change simultaneously to a very different underlying model. The other thing is that, even if some kind of migration path could be found, Guido et al wouldn't want to follow that path anyway -- because the end result would be too convoluted for ordinary people to understand and maintain. -- Greg From tdelaney at avaya.com Tue Feb 13 00:08:26 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Tue, 13 Feb 2007 10:08:26 +1100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access Message-ID: <2773CAC687FD5F4689F526998C7E4E5F074464@au3010avexu1.global.avaya.com> Benji York wrote: > Collin Winter wrote: >> On 2/12/07, Brett Cannon wrote: >>> I actually kind of like that. The connection to pointer indirection >>> meshes well with the idea of indirectly figuring out what attribute >>> to access at runtime. >> >> There's a connection, but I'd say it's the wrong one. In C, "x->y" >> dereferences x, while in Python, "x->y" would dereference y. That's >> just begging for trouble. > > Then the syntax should obviously be "x<-y". I'd actually gone through this process myself, and concluded that I wouldn't be happy with either. "x->y: is currently invalid syntax, but has *very* strong connotations from C. Whilst I'm happy to consider things that deviate from other languages' use, this one is just too strong. "x<-y" is currently valid syntax, and gives (to mee) the wrong impression that y is that thing that's going to be modified. Of all the proposals I've seen, I think I like "x.{y}" best. The use of {} has connotations of interpolation and they're the only braces that currently don't have meaning as x{} (and probably never will). I do worry that giving syntax to getattr/setattr/delattr will encourage the use of dynamic attributes, when they are a fairly advanced feature. OTOH, when you're using advanced features, you want the code to be as readable as possible, and I think this will improve readability over using getattr/setattr/delattr. So I think I've changed to +0.5. Tim Delaney From anthony at interlink.com.au Tue Feb 13 01:26:10 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue, 13 Feb 2007 11:26:10 +1100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D0F3EA.4090208@redfrontdoor.org> References: <45D0F3EA.4090208@redfrontdoor.org> Message-ID: <200702131126.11208.anthony@interlink.com.au> On Tuesday 13 February 2007 10:10, Ben North wrote: > (Gently wandering off-topic, > but: do people use proportional fonts for coding? Doesn't it > cause general awkwardness for indentation, especially relevant > for python?) The killer problem with backticks (to pick the syntax that currently causes this problem the most) is with webpages and with printed books with code. Sure, everyone can pick a font for coding that they can read, but that's not the only way you read code. This is my issue with the foo.(bar) syntax. The period is far far too small and easy to miss. -- Anthony Baxter It's never too late to have a happy childhood. From collinw at gmail.com Tue Feb 13 01:27:06 2007 From: collinw at gmail.com (Collin Winter) Date: Mon, 12 Feb 2007 18:27:06 -0600 Subject: [Python-Dev] Recent experience with the _ast module Message-ID: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> I've been working this past week at converting the stdlib's compiler package to use 2.5's new _ast module instead of the package's own AST. Overall, _ast was a joy to work with, save for the following nitpicks: 1) There are times when the _fields attribute on some AST nodes is None; if this was done to indicate that a given node has no AST-related attributes, it would be much easier if _fields was [] in this case. As it is, I had to special-case "node._fields is None" in the visitor so that I don't accidentally iterate over it. 2) {BinOp,AugAssign,BoolOp,etc}.op is an instance of, eg, Add, Sub, Mult, Mod, meaning you have to dispatch based on tests like "isinstance(node.op, x)" or "type(node.op) is x". I would much, much prefer to spell this "node.op is x", ie, use "node.op = Add" rather than the current "node.op = Add()" when constructing the nodes. 3) I'd like there to be an Else() node for If.orelse, While.orelse, etc. My motivation is that I need the lineno and col_offset values of the "else" statement for a code-coverage utility; as it is, I have to find the last lineno of the if/while/etc suite and the first lineno of the "else" suite and then search between those two for the "else" statement. Thoughts? Thanks, Collin Winter From scott+python-dev at scottdial.com Tue Feb 13 01:07:11 2007 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 12 Feb 2007 19:07:11 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> Message-ID: <45D1012F.8000805@scottdial.com> Brett Cannon wrote: > On 2/12/07, Guido van Rossum wrote: >> I recommend that you do some experiments with the readability of the >> .[...] notation, e.g. write a program that randomly generates x.[foo] >> and x[foo], and see how fast you can spot the difference. I bet that >> you won't have any trouble. >> > > OK, so real-world examples. First, ``foo.(name)``, from urllib.py:: > [snip] > > Neither version jumps out at me strongly, although between the two the > ``.[]`` version shows up the best. But that might also be because of > the lower noise when used in a call. > > -Brett After seeing a real-world example of the notation, I'm not a fan. Like Brett, of the two I find ".[]" to be the most readable. I do in fact immediately associate an identifier next to "()" with being a function call, regardless of the dot. The "[]" immediately associates me with a type of dereferencing action which is more appropriate for the context. The danger here is that "foo.bar" and "baz.(bar)" use "bar" in drastically different ways and it is not made obvious enough. "baz.[bar]" gets you closer because we are used to seeing "bar" as a variable containing a key (as opposed to "foo.bar" where 'bar' is in fact the key). At the risk of needing flame retardant, I would propose the use of something more visually jarring. The first thing that came to mind was something like "foo@[bar]". The only usage of the @ symbol I can recall is in Ruby where it is used instead of "self." for attribute access, which is in the right ballpark for what we mean here and the brackets differentiate it still. To borrow the urllib2.py example: if attr[:12] == '_Request__r_': name = attr[12:] if hasattr(Request, 'get_' + name): self@['get_' + name]() return self@[attr] raise AttributeError, attr -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From guido at python.org Tue Feb 13 01:32:07 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Feb 2007 16:32:07 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> Message-ID: Oh, now I am definitely in favor of .[]! I read it in gmail in FireFox which uses a small variable-pitch font whose dot is a single pixel. The .() example was hard to find; the .[] jumped out immediately. (When do you ever see self[anything]?) On 2/12/07, Brett Cannon wrote: > On 2/12/07, Guido van Rossum wrote: > > FWIW, I'm strongly -1 on the "->" notation. As a C programmer it's got > > too many neurons committed to it. > > > > I recommend that you do some experiments with the readability of the > > .[...] notation, e.g. write a program that randomly generates x.[foo] > > and x[foo], and see how fast you can spot the difference. I bet that > > you won't have any trouble. > > > > OK, so real-world examples. First, ``foo.(name)``, from urllib.py:: > > name = 'open_' + urltype > self.type = urltype > name = name.replace('-', '_') > if not hasattr(self, name): > if proxy: > return self.open_unknown_proxy(proxy, fullurl, data) > else: > return self.open_unknown(fullurl, data) > try: > if data is None: > return self.(name)(url) > else: > return self.(name)(url, data) > except socket.error, msg: > raise IOError, ('socket error', msg), sys.exc_info()[2] > > and also:: > > name = 'http_error_%d' % errcode > if hasattr(self, name): > method = self.(name) > if data is None: > result = method(url, fp, errcode, errmsg, headers) > else: > result = method(url, fp, errcode, errmsg, headers, data) > if result: return result > return self.http_error_default(url, fp, errcode, errmsg, headers) > > > And here is urllib2.py for ``.[]`` (used different files so you > wouldn't just remember where the change was):: > > if attr[:12] == '_Request__r_': > name = attr[12:] > if hasattr(Request, 'get_' + name): > self.['get_' + name]() > return self.[attr] > raise AttributeError, attr > > and:: > > handlers = chain.get(kind, ()) > for handler in handlers: > func = handler.[meth_name] > result = func(*args) > if result is not None: > return result > > > > Neither version jumps out at me strongly, although between the two the > ``.[]`` version shows up the best. But that might also be because of > the lower noise when used in a call. > > -Brett > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From scott+python-dev at scottdial.com Tue Feb 13 01:33:45 2007 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 12 Feb 2007 19:33:45 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <000501c74e88$0da367c0$6a176b0a@RaymondLaptop1> References: <45CFAA81.5040906@redfrontdoor.org> <000501c74e88$0da367c0$6a176b0a@RaymondLaptop1> Message-ID: <45D10769.7010002@scottdial.com> Raymond Hettinger wrote: > Rather than munge existing syntaxes, an altogether new one would be more clear: > > self->name = self.metadata->name > My problem with this is that it isn't a "name". It should grammatically be a test (i.e. it can take on the expression any function argument could take). How do you spell "getattr(self, 'req_' + state)" with that arrow? You need some kind of grouping delimiters to make this operator be a syntax benefit otherwise you didn't shorten anything. -Scott -- Scott Dial scott at scottdial.com scodial at cs.indiana.edu From barry at python.org Tue Feb 13 01:36:23 2007 From: barry at python.org (Barry Warsaw) Date: Mon, 12 Feb 2007 19:36:23 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 12, 2007, at 7:32 PM, Guido van Rossum wrote: > Oh, now I am definitely in favor of .[]! I read it in gmail in FireFox > which uses a small variable-pitch font whose dot is a single pixel. > The .() example was hard to find; the .[] jumped out immediately. > (When do you ever see self[anything]?) Raymond's -> suggestion was nice. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRdEIB3EjvBPtnXfVAQLENQP+KlLGE0rldGbOdpeMdsZ6dX7ga9Fb2mZR wkOIEYJW6H8n6OyzawPgkvQWpuJzUlC+5A42gxYUdkC4pTxWqzJQ6i6csoVOD+ya 1vztCUavDTG9HznH4tX6L2bRJI2fhBdfMxkNCnSpmRaW1SOpOgutLMQv5ZxmLzHU xjkaqp9ogPY= =gu74 -----END PGP SIGNATURE----- From dustin at v.igoro.us Tue Feb 13 01:37:10 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Mon, 12 Feb 2007 18:37:10 -0600 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D0F95A.4000507@canterbury.ac.nz> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <952d92df0702111105q1b81f106rd3516461408f3035@mail.gmail.com> <45CF8B72.2000808@v.loewis.de> <952d92df0702120546q7dab0fe2h7b1a42b9d5ef9deb@mail.gmail.com> <45D0F95A.4000507@canterbury.ac.nz> Message-ID: <20070213003710.GD28008@v.igoro.us> On Tue, Feb 13, 2007 at 12:33:46PM +1300, Greg Ewing wrote: > Richard Tew wrote: > > > The ideal mechanism at the high level would be expanding asyncore into > > a "one-stop shop". Where all these things can be passed into it and > > it can do the work to notify of events on the objects in a standard way. > > +1. This sounds like an excellent idea. It's downright > silly having each thing that uses async I/O doing its > own thing. There should be a standard mechanism in the > stdlib that everything can use. I'm workin' on it! ;) I guess #2 wasn't so hard, after all, Brett! Dustin From brett at python.org Tue Feb 13 01:47:19 2007 From: brett at python.org (Brett Cannon) Date: Mon, 12 Feb 2007 16:47:19 -0800 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> Message-ID: On 2/12/07, Collin Winter wrote: > I've been working this past week at converting the stdlib's compiler > package to use 2.5's new _ast module instead of the package's own AST. > Overall, _ast was a joy to work with, save for the following nitpicks: > > 1) There are times when the _fields attribute on some AST nodes is > None; if this was done to indicate that a given node has no > AST-related attributes, it would be much easier if _fields was [] in > this case. As it is, I had to special-case "node._fields is None" in > the visitor so that I don't accidentally iterate over it. > That makes total sense to me. > 2) {BinOp,AugAssign,BoolOp,etc}.op is an instance of, eg, Add, Sub, > Mult, Mod, meaning you have to dispatch based on tests like > "isinstance(node.op, x)" or "type(node.op) is x". I would much, much > prefer to spell this "node.op is x", ie, use "node.op = Add" rather > than the current "node.op = Add()" when constructing the nodes. > I can't think of a reason, off the top of my head, why they can't be singletons. It actually makes sense since they are basically an enumeration value in the AST grammar. > 3) I'd like there to be an Else() node for If.orelse, While.orelse, > etc. My motivation is that I need the lineno and col_offset values of > the "else" statement for a code-coverage utility; as it is, I have to > find the last lineno of the if/while/etc suite and the first lineno of > the "else" suite and then search between those two for the "else" > statement. > Ouch. I don't know how much work it would be to make this change, but it seems reasonable to want. > Thoughts? > They all sound reasonable. And it's nice to have a wanted feature be found from actual use. =) -Brett From sergio.correia at gmail.com Tue Feb 13 02:03:15 2007 From: sergio.correia at gmail.com (Sergio Correia) Date: Mon, 12 Feb 2007 20:03:15 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171291253.45d07c75297b5@imp.hosting365.ie> <45D08B3E.5020105@egenix.com> Message-ID: A few of you have expressed concern about how would that look to a newbie. Being one, this is what I think: (again, newbie alert) - The idea sounds good. Setattr and getattr seem kinda unpythonic and difficult to read. - please.(dont_torture) = me(with_dots,that_look,like.(function),calls). Ok, so the dot _is_ needed in order to indicate that we are talking about objects. But if I see something like spam.(eggs) , I would feel tempted to delete the dot thinking it's a typo. Besides, the parenthesis make the dot even harder to read. - x->y feels like assignment. I even recall that in Mathematica it IS some kind of assignment. Besides, it lacks the dot that tells me "this is an object". - Square brackets have a lot of overloading but are not so bad. - Braces feel good. I think they are the best choice of the ones proposed. Because spam{eggs} doesn't mean anything, then there would be no confusion with a typo in spam.{eggs} --Sergio On 2/12/07, M.-A. Lemburg wrote: > On 2007-02-12 16:19, Georg Brandl wrote: > >> Tim Delaney asked in particular: > >>> Have you checked if [the existing uses of getattr, where "getattr" in > >>> that scope is a function argument with default value the built-in > >>> "getattr"] are intended to bring the "getattr" name into local scope > >>> for fast lookup, or to force a binding to the builtin "gettattr" at > >>> compile time (two common (ab)uses of default arguments)? If they are, > >>> they would be better served by the new syntax. > >> They're all in Lib/codecs.py, and are of the form: > >> > >> class StreamRecoder: > >> def __getattr__(self, name, > >> getattr=getattr): > >> > >> """ Inherit all other methods from the underlying stream. > >> """ > >> return getattr(self.stream, name) > >> > >> Without digging deeper into that code I'm afraid I can't say precisely > >> what is going on. > > > > Since that is a special method and ought to have the signature > > __getattr__(self, name), I think it's safe to assume that that's meant > > as an optimization. > > I can confirm that: it's a case of fast-local-lookup optimization. > > You can add a -1 from me to the list as well: I don't think that > dynamic lookups are common enough to warrant new syntax. > > Even if you do add a new syntax for this, using parenthesis is > a poor choice IMHO as the resulting code looks too much like a > function call (e.g. "callable.(variable)"). > > Other choices would be square brackets [], but these have the > same problem as they are in use for indexing. > > The only brackets that are not yet overloaded in the context > of applying them to an object are curly brackets, so > "callable.{variable}" would cause enough raising eyebrows > to not think of a typo. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 12 2007) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/sergio.correia%2Bpydev%40gmail.com > From greg.ewing at canterbury.ac.nz Tue Feb 13 02:03:12 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 13 Feb 2007 14:03:12 +1300 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <20070212122602.ACCF.JCARLSON@uci.edu> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> Message-ID: <45D10E50.2080107@canterbury.ac.nz> Josiah Carlson wrote: > def intern(st): > ... > > If I remember the implementation of intern correctly, that's more or > less what happens under the covers. That doesn't quite give you everything that real interning does, though. The string comparison method knows when both strings are interned, so it can compare them quickly whether they are equal or not. Your version could detect equal strings quickly, but not unequal strings. -- Greg From rrr at ronadam.com Tue Feb 13 04:01:57 2007 From: rrr at ronadam.com (Ron Adam) Date: Mon, 12 Feb 2007 21:01:57 -0600 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> Message-ID: <45D12A25.802@ronadam.com> Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Feb 12, 2007, at 7:32 PM, Guido van Rossum wrote: > >> Oh, now I am definitely in favor of .[]! I read it in gmail in FireFox >> which uses a small variable-pitch font whose dot is a single pixel. >> The .() example was hard to find; the .[] jumped out immediately. >> (When do you ever see self[anything]?) > > Raymond's -> suggestion was nice. I think it's gets a bit awkward in some situations. if bar->'__%s__' % attr < -42: print 'Hello World' if bar.['__%s__' % attr] > -42: print 'Hello World' To me it's easier to parse the second one visually. Ron From maric at aristote.info Tue Feb 13 05:33:00 2007 From: maric at aristote.info (Maric Michaud) Date: Tue, 13 Feb 2007 05:33:00 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> Message-ID: <200702130533.02892.maric@aristote.info> Le mardi 13 f?vrier 2007 01:36, Barry Warsaw a ?crit?: > On Feb 12, 2007, at 7:32 PM, Guido van Rossum wrote: > > Oh, now I am definitely in favor of .[]! I read it in gmail in FireFox > > which uses a small variable-pitch font whose dot is a single pixel. > > The .() example was hard to find; the .[] jumped out immediately. > > (When do you ever see self[anything]?) > > Raymond's -> suggestion was nice. > I really dislikes the .[ or .( or .{ operators. Just on my mail editor the two expressions a.[b] and a,[b] are quite hard to differentiate while completely unrelated. Why did the brace syntax wasn't even discussed ? Seems clean to me. obj{expr} can be read as "given obj as a namespace, retrieve the name resulting by expr in obj and enclosing namespaces (supers)". We can even use slice-like syntax for further improvements : obj{expr:default} obj{expr:default:type_} something like getattr(super(type_, obj), expr, default), or even obj{expr::} for getattr(super(obj.__class__, obj), expr) and, why not, even obj{:} for super(obj.__class__, obj), regards, From nas at arctrix.com Tue Feb 13 06:05:14 2007 From: nas at arctrix.com (Neil Schemenauer) Date: Tue, 13 Feb 2007 05:05:14 +0000 (UTC) Subject: [Python-Dev] New syntax for 'dynamic' attribute access References: <1171291253.45d07c75297b5@imp.hosting365.ie> <45D08B3E.5020105@egenix.com> Message-ID: M.-A. Lemburg wrote: > You can add a -1 from me to the list as well: I don't think that > dynamic lookups are common enough to warrant new syntax. I agree. Also, I think the special syntax may make them too inviting to new programmers, who haven't figured out that usually there are better ways of doing things. > Even if you do add a new syntax for this, using parenthesis is > a poor choice IMHO as the resulting code looks too much like a > function call (e.g. "callable.(variable)"). Yes. The .[] notation looks much better, IMO. Neil From kbk at shore.net Tue Feb 13 06:05:31 2007 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue, 13 Feb 2007 00:05:31 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200702130505.l1D55V59013755@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 417 open ( -6) / 3565 closed (+12) / 3982 total ( +6) Bugs : 960 open ( -3) / 6498 closed (+19) / 7458 total (+16) RFE : 266 open ( +6) / 251 closed ( +1) / 517 total ( +7) New / Reopened Patches ______________________ stream writing support in wave.py (2007-02-05) http://python.org/sf/1652328 opened by tom tarfile append behavior (2007-02-05) CLOSED http://python.org/sf/1652681 opened by Witten operator.c slice functions need to parse ssize_t (2007-02-07) http://python.org/sf/1654417 opened by Andy Wingo sys.excepthook shows relevant bindings. (2007-02-08) http://python.org/sf/1654974 opened by Nefarious CodeMonkey, Jr. socketmodule fails to build because missing NETLINK_DNRTMSG (2007-02-11) http://python.org/sf/1657276 opened by Aleksandr Koltsoff Add syntax for dynamic attribute access (2007-02-11) http://python.org/sf/1657573 opened by Ben North documentation for element interface (2007-02-12) http://python.org/sf/1657613 opened by Tim Mitchell Patches Closed ______________ CodeContext visibility (2006-08-15) http://python.org/sf/1540869 closed by kbk Auto-completion list placement (2006-12-23) http://python.org/sf/1621265 closed by kbk Make Python Editor Useful Without Full IDLE (2005-01-26) http://python.org/sf/1110205 closed by kbk allow using normal indent width in shell in IDLE (2005-05-06) http://python.org/sf/1196946 closed by kbk configHandler support for raw data (2007-02-01) http://python.org/sf/1650174 closed by kbk tarfile append behavior (2007-02-05) http://python.org/sf/1652681 closed by gustaebel except too broad (2006-08-15) http://python.org/sf/1540849 closed by kbk Creating dicts for dict subclasses (2006-12-14) http://python.org/sf/1615701 closed by rhettinger Fix dict and set docs, re: immutability (2006-01-05) http://python.org/sf/1397711 closed by rhettinger Remove bad PREDICT in ceval.c (2006-03-04) http://python.org/sf/1443159 closed by rhettinger make trace.py --ignore-dir work (2006-10-05) http://python.org/sf/1571379 closed by montanaro Bugfix for #847665 (XMLGenerator dies in namespace mode) (2006-04-02) http://python.org/sf/1463026 closed by loewis New / Reopened Bugs ___________________ IDLE Hung up after open script by command line... (2006-09-20) CLOSED http://python.org/sf/1562193 reopened by faramir2 Nested Objects scope problem (2007-02-05) CLOSED http://python.org/sf/1652387 reopened by kkelchev Nested Objects scope problem (2007-02-05) CLOSED http://python.org/sf/1652387 reopened by kkelchev Nested Objects scope problem (2007-02-05) CLOSED http://python.org/sf/1652387 reopened by kkelchev Nested Objects scope problem (2007-02-05) CLOSED http://python.org/sf/1652387 opened by kkelchev logging formatter %(lineno)d does not work (2007-02-06) http://python.org/sf/1652788 opened by lx_jakal Double free/corruption? (2007-02-06) http://python.org/sf/1653121 opened by Jarek Zgoda print >> f, "Hello" produces no error: normal? (2007-02-06) http://python.org/sf/1653416 opened by E.-O. Le Bigot Python misbehaves when installed in / (patch attached) (2007-02-06) http://python.org/sf/1653457 opened by Chris Webb Problems in datetime.c and typeobject.c. (2007-02-07) CLOSED http://python.org/sf/1653736 opened by ked-tao crash / abort during install (2007-02-06) CLOSED http://python.org/sf/1653753 opened by SAndreason configure does not check/warn/stop for tk/tcl (2007-02-06) CLOSED http://python.org/sf/1653757 opened by SAndreason popen - wrong order on fileobjects in tuple returned (2007-02-07) CLOSED http://python.org/sf/1653940 opened by Emil Lind Installer should split tcl/tk and tkinter install options. (2007-02-07) http://python.org/sf/1654408 opened by Ron Adam thread join() with timeout hangs on Windows 2003 x64 (2007-02-07) http://python.org/sf/1654429 opened by bentoi thirdparty extensions, --enable-shared, static linking (2007-02-08) http://python.org/sf/1655392 opened by Marien Zwart hotshot.stats.load (2004-02-19) http://python.org/sf/900092 reopened by bcannon 3.4.1 comparison methods content (2007-02-09) CLOSED http://python.org/sf/1655683 opened by Jeffrey Miller email.Generator docs contain a bad off-site link (2007-02-08) CLOSED http://python.org/sf/1655800 opened by Forest Wilkinson Typo in Docs (2007-02-09) CLOSED http://python.org/sf/1656078 opened by Thomas Guettler I think, I have found this bug on time.mktime() (2007-02-10) http://python.org/sf/1656559 opened by S?rgio Monteiro Basto shutil.copyfileobj description is incomplete (2007-02-10) http://python.org/sf/1656578 opened by Witten tarfile.TarFile fileobject use needs clarification (2007-02-10) CLOSED http://python.org/sf/1656581 opened by Witten 'Ok' key in options dialog does nothing (2007-02-11) http://python.org/sf/1657034 opened by torhu random.betavariate(-0,5, -0,5) (2007-02-12) CLOSED http://python.org/sf/1658430 opened by Alain Spineux Bugs Closed ___________ IDLE Hung up after open script by command line... (2006-09-20) http://python.org/sf/1562193 closed by kbk HP-UX11.23: module zlib missing (2007-01-31) http://python.org/sf/1648960 deleted by jabt Nested Objects scope problem (2007-02-05) http://python.org/sf/1652387 closed by kkelchev Nested Objects scope problem (2007-02-05) http://python.org/sf/1652387 closed by gbrandl Nested Objects scope problem (2007-02-05) http://python.org/sf/1652387 closed by kkelchev Nested Objects scope problem (2007-02-05) http://python.org/sf/1652387 closed by gbrandl Quitter object masked (2006-05-01) http://python.org/sf/1479785 closed by kbk Arguments tooltip wrong if def contains tuple (2003-08-20) http://python.org/sf/791968 closed by kbk subprocess fails on GetStdHandle in interactive GUI (2005-02-17) http://python.org/sf/1124861 closed by astrand Problems in datetime.c and typeobject.c. (2007-02-07) http://python.org/sf/1653736 closed by loewis crash / abort during install (2007-02-06) http://python.org/sf/1653753 closed by sandreas41 configure does not check/warn/stop for tk/tcl (2007-02-07) http://python.org/sf/1653757 closed by loewis popen - docs - wrong order on fileobjects in tuple returned (2007-02-07) http://python.org/sf/1653940 closed by gbrandl isSequenceType returns True for dict subclasses (<> 2.3) (2006-10-11) http://python.org/sf/1575169 closed by rhettinger builtin enumerate overflows (2006-06-26) http://python.org/sf/1512504 closed by rhettinger 3.4.1 comparison methods content (2007-02-09) http://python.org/sf/1655683 closed by loewis email.Generator docs contain a bad off-site link (2007-02-09) http://python.org/sf/1655800 closed by loewis --enable-shared links extensions to libpython statically (2006-11-22) http://python.org/sf/1600860 closed by loewis Typo in Docs (2007-02-09) http://python.org/sf/1656078 closed by gbrandl tarfile.TarFile fileobject use needs clarification (2007-02-10) http://python.org/sf/1656581 closed by gustaebel decimals compare badly to floats (2007-02-01) http://python.org/sf/1650053 closed by rhettinger XMLGenerator.startElementNS dies on EMPTY_NAMESPACE attribut (2003-11-23) http://python.org/sf/847665 closed by loewis random.betavariate(-0,5, -0,5) (2007-02-12) http://python.org/sf/1658430 closed by gbrandl New / Reopened RFE __________________ [PATCH] Debuggers need a way to change the locals of a frame (2007-02-07) http://python.org/sf/1654367 opened by Fabio Zadrozny dict(key,values) initializer (2007-02-10) http://python.org/sf/1656538 opened by George Sakkis Drop-Handler for Python files (2007-02-10) http://python.org/sf/1656675 opened by Marek Kubica RFE Closed __________ itertools.count wraps around after maxint (2005-10-13) http://python.org/sf/1326277 closed by rhettinger From guido at python.org Tue Feb 13 06:14:45 2007 From: guido at python.org (Guido van Rossum) Date: Mon, 12 Feb 2007 21:14:45 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <200702130533.02892.maric@aristote.info> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <200702130533.02892.maric@aristote.info> Message-ID: On 2/12/07, Maric Michaud wrote: > Le mardi 13 f?vrier 2007 01:36, Barry Warsaw a ?crit: > > On Feb 12, 2007, at 7:32 PM, Guido van Rossum wrote: > > > Oh, now I am definitely in favor of .[]! I read it in gmail in FireFox > > > which uses a small variable-pitch font whose dot is a single pixel. > > > The .() example was hard to find; the .[] jumped out immediately. > > > (When do you ever see self[anything]?) > > > > Raymond's -> suggestion was nice. > > > I really dislikes the .[ or .( or .{ operators. > Just on my mail editor the two expressions > > a.[b] > > and > > a,[b] > > are quite hard to differentiate while completely unrelated. Yeah, so are 1.2 and 1,2. This is why the style guide (PEP 8) insists on a space after a comma but not after a period. > Why did the brace syntax wasn't even discussed ? Seems clean to me. Because they are arbitrary -- if x{y} or x.{y} would be acceptable, why not x. or x.|y| or x./y/? Or indeed why not x at y? (Not that I'm in favor of that. :-) That x.[y] and x.(y) resemble x[y] and x(y) is an *advantage* of the proposed new notation, not a disadvantage; the new operator is semantically closer to x[y] than to x(y) so x.[y] makes sense. > obj{expr} can be read as "given obj as a namespace, retrieve the name > resulting by expr in obj and enclosing namespaces (supers)". Yeah, it can just as well be read as "call obj with 2**expr as argument". Also, your reference to enclosing namespaces and supers is misplaced -- this should map to the existing __getattr__ operation, which can have many different semantics depending on the type of obj. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Tue Feb 13 07:29:58 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 07:29:58 +0100 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D0FF6B.6040104@canterbury.ac.nz> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> <45D0B11B.3080309@acm.org> <45D0FF6B.6040104@canterbury.ac.nz> Message-ID: <45D15AE6.4010302@v.loewis.de> Greg Ewing schrieb: > The other thing is that, even if some kind of migration > path could be found, Guido et al wouldn't want to follow > that path anyway -- because the end result would be too > convoluted for ordinary people to understand and maintain. That very much depends on what the end result would be. Regards, Martin From martin at v.loewis.de Tue Feb 13 07:39:18 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 07:39:18 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> Message-ID: <45D15D16.7060109@v.loewis.de> Brett Cannon schrieb: > name = 'open_' + urltype > self.type = urltype > name = name.replace('-', '_') > if not hasattr(self, name): > if proxy: > return self.open_unknown_proxy(proxy, fullurl, data) > else: > return self.open_unknown(fullurl, data) > try: > if data is None: > return self.(name)(url) > else: > return self.(name)(url, data) > except socket.error, msg: > raise IOError, ('socket error', msg), sys.exc_info()[2] Also notice that this leaves a hasattr call in, as there is no replacement proposed for that. > name = 'http_error_%d' % errcode > if hasattr(self, name): > method = self.(name) > if data is None: > result = method(url, fp, errcode, errmsg, headers) > else: > result = method(url, fp, errcode, errmsg, headers, data) > if result: return result > return self.http_error_default(url, fp, errcode, errmsg, headers) Likewise. > if attr[:12] == '_Request__r_': > name = attr[12:] > if hasattr(Request, 'get_' + name): > self.['get_' + name]() > return self.[attr] > raise AttributeError, attr And again. Apparently, people favor hasattr over catching AttributeError. I'm not sure why this is - I would probably rewrite them all to deal with AttributeError if I use the new syntax in the first place. Regards, Martin From martin at v.loewis.de Tue Feb 13 07:46:57 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 07:46:57 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D12A25.802@ronadam.com> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> <45D12A25.802@ronadam.com> Message-ID: <45D15EE1.4010700@v.loewis.de> Ron Adam schrieb: > I think it's gets a bit awkward in some situations. > > > if bar->'__%s__' % attr < -42: print 'Hello World' > > if bar.['__%s__' % attr] > -42: print 'Hello World' > > > To me it's easier to parse the second one visually. Ah, precedence. It definitly should be a bracketed form, or else people always wonder what the precedence is, and add parenthesis anyway just to be on the safe side. BTW, which of these would be correct (a).[b] (a.)[b] a.[(b)] a.([b]) a . [ b ] and what is the semantics of a.[42] Regards, Martin From martin at v.loewis.de Tue Feb 13 08:00:15 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 08:00:15 +0100 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> Message-ID: <45D161FF.8010700@v.loewis.de> Collin Winter schrieb: > 1) There are times when the _fields attribute on some AST nodes is > None; if this was done to indicate that a given node has no > AST-related attributes, it would be much easier if _fields was [] in > this case. As it is, I had to special-case "node._fields is None" in > the visitor so that I don't accidentally iterate over it. That can be done, I think. > 2) {BinOp,AugAssign,BoolOp,etc}.op is an instance of, eg, Add, Sub, > Mult, Mod, meaning you have to dispatch based on tests like > "isinstance(node.op, x)" or "type(node.op) is x". I would much, much > prefer to spell this "node.op is x", ie, use "node.op = Add" rather > than the current "node.op = Add()" when constructing the nodes. Please look at Python.asdl. This things are really belong to sum nodes, and Add, Sub etc. are really operators. I *think* they are singletons, and I don't mind making it a guarantee that they are: i.e. all operators that have no fields and where the sum has no attributes. If you want to write "is", you can also write node.op.__class__ is Add > 3) I'd like there to be an Else() node for If.orelse, While.orelse, > etc. My motivation is that I need the lineno and col_offset values of > the "else" statement for a code-coverage utility; as it is, I have to > find the last lineno of the if/while/etc suite and the first lineno of > the "else" suite and then search between those two for the "else" > statement. Notice that you cannot restore the original source code, anyway. You cannot tell whether something was else:if or elif. I don't understand why you need the line number of the else keyword (which, as I said, may not exist in the source) for code coverage: The 'else' is never executed (only the statements in it are). Regards, Martin From steve at holdenweb.com Tue Feb 13 09:19:41 2007 From: steve at holdenweb.com (Steve Holden) Date: Tue, 13 Feb 2007 08:19:41 +0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D0B599.8040608@v.loewis.de> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> <45D0B599.8040608@v.loewis.de> Message-ID: <45D1749D.6060601@holdenweb.com> Martin v. L?wis wrote: > Richard Tew schrieb: >> I can't point you to the posts, but I can point you to something >> Christian has written on the subject which may indicate why >> it was never actually submitted for inclusion. >> >> See "A Bit Of History" >> http://svn.python.org/view/stackless/trunk/Stackless/readme.txt >> >> I have not talked to Christian about it but as the current maintainer >> of Stackless I was under the impression that there was a complete >> lack of interest in even considering it for integration. > > I think this comes from a major misunderstanding of how free software > works. This is not commercial software, so there is no "company > agenda" that drives it forward. There are various contributors, > each having their own agenda; for most of us, it's "scratch your > own itches". Surely, the voices of people have varying significance > (in Python, Guido is the BDFL, for example), but "there was a complete > lack of interest in even considering" just cannot be true: clearly, > some contributors and users *were* interested. > > There were surely objections to the implementation strategy, and > I also had objections to the specific execution of this > implementation strategy, see http://tinyurl.com/27v23r > > Now, the question is whose "job" it would have been to integrate > Stackless to Python. Apparently, Christian tried to make it happen > (see http://tinyurl.com/3bqe5d), although I'm still uncertain > whether he ever got to a point where he said: "this is a patch, > please review it" (of that, I can find no archives, as I said). > > I find it understandable that nobody else took over to work > on integrating it (although there is PEP 219, so apparently > some people worked at some point on it). > > From my point of view, people suggesting to incorporate code > as-is often misunderstand that it is not sufficient for the code > to work in the applications were it is used. It must also be sound > (i.e. work in some meaningful way also in cases for which it wasn't > designed), and it must be maintainable. > > I can understand how discouraging it is when you see that your > code "works" that then people object to incorporating it as-is. > However, I believe the long-term quality of Python can only > be guaranteed when careful review is applied to every change > made, or else we will regret changes in the future (and indeed, > Python 3000 would not have been necessary if no mistakes had > been made). > >> If addition of microthreading is being considered please consider >> Stackless. One reason why Stackless is not so popular is that >> as a fork people shy away from it because it is not the main >> version. It would benefit Stackless to be integrated and I would >> be willing to do any work involved to either integrate or maintain >> it afterwards. > > This is what I'm talking about: nothing "is" ever considered. Instead, > individuals make contributions, individuals review them, and individuals > support and object to changes. There is no abstract python-dev > entity that likes or dislikes changes. Individuals have individual > opinions. > > Now, where is the SF patch that makes Stackless part of Python ?-) > (and if you are serious about it, try to break it down into multiple > pieces, each individually correct and useful; start with one where > the effort isn't too big in case it gets rejected by BDFL > pronouncement) > > I haven't reviewed the Stackless code in a while; last I looked, > I found it was possible to crash the interpreter if you use it > "incorrectly". I *personally* object to changes where it is easy > to crash the interpreter, unless there is a clear specification > when this may happen, and there is a systematic way to avoid that > (this is the ctypes clause). > Well given that ctypes has now been incorporated I suspect that the objections to Stackless might be different should a further integration "effort" materialise. I know that there's work in progress (and indeed almost complete) to put Stackless into 2.5, so it seems as though it might be practical to follow the approach you suggest. The only things that concern me are a) whether it could make sense to add Stackless in bits and pieces and b) whether the BDFL (or even the developer community en masse) would object in principle, thereby rendering such efforts useless. My (limited) understanding is that with Stackless installed *all* existing programs that don't import stackless should continue to run unchanged. If that's true then it seems to me it would be a desirable addition as long as maintainability wasn't compromised. I suppose only the patches will allow a sensible judgment on that issue. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From g.brandl at gmx.net Tue Feb 13 09:33:49 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 13 Feb 2007 09:33:49 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D15EE1.4010700@v.loewis.de> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> <45D12A25.802@ronadam.com> <45D15EE1.4010700@v.loewis.de> Message-ID: Martin v. L?wis schrieb: > Ron Adam schrieb: >> I think it's gets a bit awkward in some situations. >> >> >> if bar->'__%s__' % attr < -42: print 'Hello World' >> >> if bar.['__%s__' % attr] > -42: print 'Hello World' >> >> >> To me it's easier to parse the second one visually. > > > Ah, precedence. > > It definitly should be a bracketed form, or else people > always wonder what the precedence is, and add parenthesis > anyway just to be on the safe side. Indeed. > BTW, which of these would be correct > > (a).[b] Yes. (you can write (a).b today) > (a.)[b] No. (you can't write (a.)b today) > a.[(b)] Yes. (you can write a[(b)] today) > a.([b]) No. (you can't write a([b]) today) > a . [ b ] Yes. (you can write a . b today) In short, I'd just add a new form to Grammar's trailer: trailer: '(' [arglist] ')' | '[' subscriptlist ']' | '.' attrtrailer attrtrailer: '[' test ']' | NAME This should be consistent and unsurprising. > and what is the semantics of > > a.[42] A TypeError, just as getattr(a, 42) would be. I think I like the .[] form better, that being a +0 now. Georg From hrvoje.niksic at avl.com Tue Feb 13 09:46:40 2007 From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=) Date: Tue, 13 Feb 2007 09:46:40 +0100 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <20070212122602.ACCF.JCARLSON@uci.edu> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> Message-ID: <1171356400.32307.80.camel@localhost> On Mon, 2007-02-12 at 12:29 -0800, Josiah Carlson wrote: > Hrvoje Nik?ic wrote: > > > > I propose modifying PyString_InternInPlace to better cope with string > > subtype instances. > > Any particular reason why the following won't work for you? [... snipped a simple intern implementation ...] Because Python internals don't use that; they call PyString_InternInPlace directly. Another reason is that Python's interning mechanism is much better than such a simple implementation: it stores the interned state directly in the PyString_Object structure, so you can find out that a string is already interned without looking it up in the dictionary. This information can (and is) used by both Python core and by C extensions. Another advantage is that, as of recently, interned strings can be garbage collected, which is typically not true of simple replacements (although it could probably be emulated by using weak references, it's not trivial.) From steve at holdenweb.com Tue Feb 13 11:10:37 2007 From: steve at holdenweb.com (Steve Holden) Date: Tue, 13 Feb 2007 10:10:37 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45CFAA81.5040906@redfrontdoor.org> References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: Ben North wrote: > Hi, > > A few days ago I posted to python-ideas with a suggestion for some new > Python syntax, which would allow easier access to object attributes > where the attribute name is known only at run-time. For example: > > setattr(self, method_name, getattr(self.metadata, method_name)) > > from Lib/distutils/dist.py could be rewritten > > self.(method_name) = self.metadata.(method_name) > > The new syntax would also be usable in augmented assignments, as in > > obj.(attr_name) += 1 > > There was some discussion on python-ideas, which I've linked to in the > draft PEP below. In particular, Guido van Rossum wrote: > > > I've thought of the same syntax. I think you should submit this to the > > PEP editor and argue on Python-dev for its inclusion in Python 2.6 -- > > there's no benefit that I see of waiting until 3.0. > I don't like this. Just because an enhancement is syntactically permissible doesn't mean it's a good idea. This seems to me to take Python further away from the "Computer Programming for Everyone" arena and closer to the "Systems Programming for Clever Individuals" camp. > so here I am. Does anybody have any opinions/suggestions, particularly > on the "open questions" referred to in the draft PEP? To summarise > these open questions: > > * The draft currently allows a two-argument form, to supply a default > value if the object has no attribute of that name. This mimics the > behaviour of the three-argument form of getattr, but looks a bit wrong: > > s = obj.(attr_name, 'default string') > > I agree that it looks odd, but perhaps the extra expressive power > gained might be worth the oddness. > It looks extremely odd. Since the opening parenthesis takes us into new syntactic territory the unimaginative use of the (already heavily overloaded) comma would sound the idea's death-knell. If we *have* to have this (and I agree with Greg that many uses cases for dynamic attribute access are invalid) then why not s = obj.(attr_name: 'default-string') I presume that the default value would in fact be an arbitrary expression? > * The draft implementation (a sourceforge patch, linked to in the draft > PEP) may have a general performance penalty of around 1%, although my > initial measurements were quite noisy. Josiah Carlson thought this > would not be too substantial, but he did suggest a couple of other > implementation routes which could be explored. The general > performance hit is offset by a speed gain of around 40% for attribute > access using the new syntax over getattr etc. Is 1% "too much" for > this feature? > Yes. I believe it would decrease the sum total of Python's efficiency for a very marginal return in performance on a very marginal feature. It threatens to reduce Python's readability substantially, and readability is more important than efficiency. "Expressive power" is all very well as long as the expressiveness is comprehensible. This proposal makes the left parenthesis carry a larger burden than the introduction of generator expressions did, and it makes Python a more difficult language to understand. [provisional PEP snipped] If it's added in 2.6 I propose it should be deprecated in 2.7 and removed from 3.0 ... regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From martin at v.loewis.de Tue Feb 13 11:18:32 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 13 Feb 2007 11:18:32 +0100 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <1171356400.32307.80.camel@localhost> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> <1171356400.32307.80.camel@localhost> Message-ID: <45D19078.5000109@v.loewis.de> Hrvoje Nik?i? schrieb: > Another reason is that Python's interning mechanism is much better than > such a simple implementation: it stores the interned state directly in > the PyString_Object structure, so you can find out that a string is > already interned without looking it up in the dictionary. This > information can (and is) used by both Python core and by C extensions. > Another advantage is that, as of recently, interned strings can be > garbage collected, which is typically not true of simple replacements > (although it could probably be emulated by using weak references, it's > not trivial.) OTOH, in an application that needs unique strings, you normally know what the scope is (i.e. where the strings come from, and when they aren't used anymore). For example, in XML parsing, pyexpat supports an interning dictionary. It puts all element and attribute names into (but not element content, which typically isn't likely to be repeated). It starts with a fresh dictionary before parsing starts, and releases the dictionary when parsing is done. Regards, Martin From martin at v.loewis.de Tue Feb 13 11:22:48 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 11:22:48 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> <45D12A25.802@ronadam.com> <45D15EE1.4010700@v.loewis.de> Message-ID: <45D19178.3090105@v.loewis.de> Georg Brandl schrieb: > >> a.([b]) > No. (you can't write a([b]) today) Actually, you can, but it means something > >> a . [ b ] > Yes. (you can write a . b today) OTOH, you can't write x + = 2 or a = 2 * * 4 so it's not that obvious that .[ should be two tokens. Regards, Martin From martin at v.loewis.de Tue Feb 13 11:39:23 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 11:39:23 +0100 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D1749D.6060601@holdenweb.com> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> <45D0B599.8040608@v.loewis.de> <45D1749D.6060601@holdenweb.com> Message-ID: <45D1955B.80508@v.loewis.de> Steve Holden schrieb: > The only things that concern me are a) whether it could make sense to > add Stackless in bits and pieces and b) whether the BDFL (or even the > developer community en masse) would object in principle, thereby > rendering such efforts useless. I think I need to try again. The 'developer community en masse' does not have a single voice, so it won't object. Not sure about the BDFL, but I personally don't object 'to a change like that' 'in principle', primarily because I don't know what the change is. I know I'm almost certainly object to the change 'incorporate Stackless Python into Python as-is', because history shows that any change of that size would need multiple iterations until it was acceptable (despite the code being in-use for multiple years). Very few people can get away with getting their code unedited into Python (ctypes and elementtree being the most prominent examples); in these cases, I could accept what I consider flaws in the code because: a) the authors promised to maintain the code, and react to actual bug reports, and b) these come as extension modules, so it's easy to ignore them if you don't like them. > My (limited) understanding is that with Stackless installed *all* > existing programs that don't import stackless should continue to run > unchanged. If that's true then it seems to me it would be a desirable > addition as long as maintainability wasn't compromised. I suppose only > the patches will allow a sensible judgment on that issue. My understanding is that incorporating Stackless will reduce the portability: it cannot work on certain platforms, and even for platforms it works on, it cannot work with certain compilers or compiler switches (the Windows SEH being the primary example; the SPARC register stack another one - although this might now be supported through assembler code). On platforms where it isn't supported, it still may compile, but might crash the interpreter when used. Please understand that this is from a shallow inspection a few years ago (actually, a description of the working mechanics that Christian gave me). It would make me feel uncomfortable if Python had modules that may randomly crash the interpreter, and there is no procedure to tell beforehand whether a certain applications is supported or not. Also, I would not like to see modules that monkey-patch other modules in the standard library. If socket.connect is to behave differently, the code that makes that so should be *in* socket.connect, and the documentation of socket.connect should state that it has that modified behavior. Of course, people may object to massive library changes of the nature "if running in stackless mode, this library routine behaves differently". Regards, Martin From p.f.moore at gmail.com Tue Feb 13 11:44:21 2007 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 13 Feb 2007 10:44:21 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <200702131126.11208.anthony@interlink.com.au> References: <45D0F3EA.4090208@redfrontdoor.org> <200702131126.11208.anthony@interlink.com.au> Message-ID: <79990c6b0702130244l394f2dc8i504a573a57e808f8@mail.gmail.com> On 13/02/07, Anthony Baxter wrote: > The killer problem with backticks (to pick the syntax that currently > causes this problem the most) is with webpages and with printed > books with code. Sure, everyone can pick a font for coding that > they can read, but that's not the only way you read code. This is > my issue with the foo.(bar) syntax. The period is far far too small > and easy to miss. That's a good point, but I've been reading this thread in gmail (proportional font) and quite often using the wrong glasses (:-)) and I've found that the dot has never been that unreadable. Even where it isn't very visible, it introduces an extra bit of space, which makes me look again, and see what's going on. In practice, I find that distinguishing {}, [] and () is harder... Just a small data point. Paul. PS On the overall change, I'm +1, I don't have much opinion on exact syntax. And I'd second the comments that the way the PEP has been handled by Ben has been great - it's been very easy to follow what could have been a really messy thread. From martin at v.loewis.de Tue Feb 13 11:46:04 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 11:46:04 +0100 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <45D10E50.2080107@canterbury.ac.nz> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> <45D10E50.2080107@canterbury.ac.nz> Message-ID: <45D196EC.3050303@v.loewis.de> Greg Ewing schrieb: > That doesn't quite give you everything that real interning > does, though. The string comparison method knows when both > strings are interned, so it can compare them quickly > whether they are equal or not. No, it doesn't - see stringobject.c:string_richcompare. If they are identical, they are quickly recognized as equal, and as not not-equal. Otherwise, if eq comparison is requested, it compares the size, then the first characters, then does memcmp - so if they are unequal, it doesn't help that they are both interned. If comparison is for notequal, no special casing is performed at all (except that knowing that identical strings aren't notequal). The entire algorithm and optimization works just as fine for a user-defined interning dictionary (assuming that all relevant strings get into it). Regards, Martin From rasky at develer.com Tue Feb 13 11:46:28 2007 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 13 Feb 2007 11:46:28 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D15D16.7060109@v.loewis.de> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> <45D15D16.7060109@v.loewis.de> Message-ID: On 13/02/2007 7.39, Martin v. L?wis wrote: > And again. Apparently, people favor hasattr over catching > AttributeError. I'm not sure why this is - Because the code becomes longer, unless you want to mask other exceptions: name = 'http_error_%d' % errcode -if hasattr(self, name): - method = self.(name) +try: + method = self.(name) +except AttributeError: + pass +else: if data is None: result = method(url, fp, errcode, errmsg, headers) else: result = method(url, fp, errcode, errmsg, headers, data) if result: return result return self.http_error_default(url, fp, errcode, errmsg, headers) -- Giovanni Bajo From rasky at develer.com Tue Feb 13 11:52:42 2007 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 13 Feb 2007 11:52:42 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <200702130533.02892.maric@aristote.info> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <200702130533.02892.maric@aristote.info> Message-ID: On 13/02/2007 5.33, Maric Michaud wrote: > I really dislikes the .[ or .( or .{ operators. > Just on my mail editor the two expressions > > a.[b] > > and > > a,[b] > > are quite hard to differentiate while completely unrelated. I'll propose a new color for this bikeshed: a.[[b]] handlers = chain.get(kind, ()) for handler in handlers: func = handler.[[meth_name]] result = func(*args) if result is not None: return result Little heavy on the eye, but it seems that it's exactly what people want and can't find in the .[] syntax. -- Giovanni Bajo From martin at v.loewis.de Tue Feb 13 11:57:40 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 11:57:40 +0100 Subject: [Python-Dev] Any value in setting up pybots for py3k? In-Reply-To: <3f09d5a00702121250j73fef0f8k3baf114db6276b9f@mail.gmail.com> References: <3f09d5a00702121250j73fef0f8k3baf114db6276b9f@mail.gmail.com> Message-ID: <45D199A4.6000603@v.loewis.de> Grig Gheorghiu schrieb: > So is there any value or interest in setting up a svn notification > hook for py3k commits that would go to the pybots buildbot master? A similar question is whether there should be buildbots for Python 3 itself (i.e. running the test suite). Even for that, it was considered too early. I would expect that packages break in massive ways because of the by-design incompatibilities. It would be good to get an estimate of what the impact is, but having a build once a month might be more than enough. To fully study the problems, one has to install Python 3 locally, anyway, and then run the package in question on top of that. Regards, Martin From phd at phd.pp.ru Tue Feb 13 12:00:23 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Tue, 13 Feb 2007 14:00:23 +0300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: <20070213110023.GB11163@phd.pp.ru> On Tue, Feb 13, 2007 at 10:10:37AM +0000, Steve Holden wrote: > Python further away from the "Computer Programming for Everyone" arena > and closer to the "Systems Programming for Clever Individuals" camp. That's because Python is being developed by "Clever Individuals" and not by "Computer Programming for Everyone Committee". Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From richard.m.tew at gmail.com Tue Feb 13 13:31:34 2007 From: richard.m.tew at gmail.com (Richard Tew) Date: Tue, 13 Feb 2007 12:31:34 +0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D1955B.80508@v.loewis.de> References: <20070210212702.GH4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> <45D0B599.8040608@v.loewis.de> <45D1749D.6060601@holdenweb.com> <45D1955B.80508@v.loewis.de> Message-ID: <952d92df0702130431x38c8c62atcfdf9f854bfc45f5@mail.gmail.com> On 2/13/07, "Martin v. L?wis" wrote: > Steve Holden schrieb: > > The only things that concern me are a) whether it could make sense to > > add Stackless in bits and pieces and b) whether the BDFL (or even the > > developer community en masse) would object in principle, thereby > > rendering such efforts useless. > > I think I need to try again. The 'developer community en masse' does > not have a single voice, so it won't object. Not sure about the BDFL, > but I personally don't object 'to a change like that' 'in principle', > primarily because I don't know what the change is. > > I know I'm almost certainly object to the change 'incorporate > Stackless Python into Python as-is', because history shows that > any change of that size would need multiple iterations until it > was acceptable (despite the code being in-use for multiple years). > Very few people can get away with getting their code unedited > into Python (ctypes and elementtree being the most prominent > examples); in these cases, I could accept what I consider flaws > in the code because: > a) the authors promised to maintain the code, and react to > actual bug reports, and > b) these come as extension modules, so it's easy to ignore > them if you don't like them. This makes a lot of sense. You have given a lot of food for thought and a path forward. If I gave an impression in my mails that I expected a consensus of objection, it was because I believed objection to Stackless to be an established thing and that it didn't matter whether there was consensus or not because someone would step forward and reiterate that. I didn't want to push the issue or invest wasted effort if that proved to be the case as mistakenly expected. At no time have I expected that Stackless would be added as-is. > > My (limited) understanding is that with Stackless installed *all* > > existing programs that don't import stackless should continue to run > > unchanged. If that's true then it seems to me it would be a desirable > > addition as long as maintainability wasn't compromised. I suppose only > > the patches will allow a sensible judgment on that issue. > > My understanding is that incorporating Stackless will reduce the > portability: it cannot work on certain platforms, and even for > platforms it works on, it cannot work with certain compilers > or compiler switches (the Windows SEH being the primary example; > the SPARC register stack another one - although this might now > be supported through assembler code). On platforms where it isn't > supported, it still may compile, but might crash the interpreter > when used. If there is no Stackless 'hard switching' available (the stack switching done with assistance of assembler) for the platform it is being compiled on, then compilation proceeds without Stackless compiled into Python. The module is not available. The process of adding support for additional platforms is rather straightforward and with access to them I can do it if required. I don't know about SEH but I believe support for SPARC was added in 2002. http://svn.python.org/view/stackless/trunk/Stackless/platf/switch_sparc_sun_gcc.h > Please understand that this is from a shallow inspection a few > years ago (actually, a description of the working mechanics that > Christian gave me). It would make me feel uncomfortable if Python > had modules that may randomly crash the interpreter, and there > is no procedure to tell beforehand whether a certain applications > is supported or not. Right. I imagine that if there are still problems like these, they will be brought to light when and if patches are submitted. But for now I will address what you mention just so that people don't assume they are still the case, especially since Stackless has been rewritten since then. > Also, I would not like to see modules that monkey-patch other > modules in the standard library. If socket.connect is to > behave differently, the code that makes that so should be > *in* socket.connect, and the documentation of socket.connect > should state that it has that modified behavior. Of course, > people may object to massive library changes of the nature > "if running in stackless mode, this library routine behaves > differently". Perhaps my pursuit of better support for asynchronous calls has led to some confusion. Stackless is an implementation with minimal changes to the core. It does not include any modules which monkey patch. However I have a replacement socket module based on asyncore which is not part of the Stackless distribution. If monkey patched it allows people to use sockets suitably in their microthreads. I am experimenting with this in order to see if I can make using Stackless as close to being as straightforward as writing normal Python code as possible, while still gaining the benefits which use of microthreads should allow. It was considered and quickly dismissed to modify the socket and file support in the core at C level to do what was needed. Primarily because it would complicate the limited changes we make to the core. Thanks, Richard. From hrvoje.niksic at avl.com Tue Feb 13 14:00:07 2007 From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=) Date: Tue, 13 Feb 2007 14:00:07 +0100 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <1170859141.32307.32.camel@localhost> References: <1170859141.32307.32.camel@localhost> Message-ID: <1171371607.32307.84.camel@localhost> On Wed, 2007-02-07 at 15:39 +0100, Hrvoje Nik?i? wrote: > The patch could look like this. If there is interest in this, I can > produce a complete patch. The complete patch is now available on SourceForge: http://sourceforge.net/tracker/index.php?func=detail&aid=1658799&group_id=5470&atid=305470 From ncoghlan at gmail.com Tue Feb 13 14:37:29 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 13 Feb 2007 23:37:29 +1000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45CFAA81.5040906@redfrontdoor.org> References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: <45D1BF19.6030706@gmail.com> Taking a step back a bit... the basic issue is that we have an attribute namespace (compile time key determination) that we want to access in a dictionary style (runtime key determination). This is currently done by switching from syntactic access to the getattr/setattr/delattr builtin functions. Elsewhere in the thread, Calvin made the suggestion that, rather than introducing new syntax, this could instead be achieved with a wrapper class that automatically converted dict-style access into the appropriate calls to getattr/setattr/delattr. I've tried this out on Brett's urllib & urllib2 examples below. (calling the new builtin attrview() to emphasise the fact that it retains a reference to the original instance). I don't consider it any uglier than the proposed syntax changes, and it provides a few other benefits: - the two-argument form is naturally available as the .get() method on the resulting dict-like object (e.g. "attrview(obj).get(some_attr, None)") - hasattr() is naturally replaced by containment testing (e.g. "some_attr in attrview(obj)") - keywords/builtins are easier to look up in the documentation than symbolic syntax With this approach, performance would be attained by arranging to create the view objects once, and then performing multiple dynamic attribute accesses using those view objects. First urllib.py example:: name = 'open_' + urltype self.type = urltype name = name.replace('-', '_') self_d = attrview(self) if name in self_d: if proxy: return self.open_unknown_proxy(proxy, fullurl, data) else: return self.open_unknown(fullurl, data) try: if data is None: return self_d[name](url) else: return self_d[name](url, data) except socket.error, msg: raise IOError, ('socket error', msg), sys.exc_info()[2] Second urllib.py example:: name = 'http_error_%d' % errcode self_d = attrview(self) if name in self_d: method = self_d[name] if data is None: result = method(url, fp, errcode, errmsg, headers) else: result = method(url, fp, errcode, errmsg, headers, data) if result: return result return self.http_error_default(url, fp, errcode, errmsg, headers) First urllib.py example:: if attr[:12] == '_Request__r_': name = attr[12:] get_name = 'get_' + name if get_name in attrview(Request): self_d = attrview(self) self_d[get_name]() return self_d[attr] raise AttributeError, attr Second urllib2.py example:: handlers = chain.get(kind, ()) for handler in handlers: func = attrview(handler)[meth_name] result = func(*args) if result is not None: return result Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ben at redfrontdoor.org Tue Feb 13 14:50:01 2007 From: ben at redfrontdoor.org (Ben North) Date: Tue, 13 Feb 2007 13:50:01 +0000 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion Message-ID: <1171374601.45d1c2091fe11@imp.hosting365.ie> The support for the including the feature at all is still not unanimous. Perhaps the way forward is to try to reach consensus on the favourite (or least-unfavourite) syntax, and I'll revise the PEP and sample implementation. I think the "obj.[attr_name]" syntax has the most support. To stop this going round in circles for ages, then, I will take this as the winner. I'll mention the other contenders in the PEP, including the new "visually distinctive" suggestions obj@[foo] obj.[[foo]] and the "wrapper class" idea of Nick Coghlan: attrview(obj)[foo] (I personally think the "attrview" idea results in slightly cluttered-looking code, and prefer the visual simplicity of "obj.[foo]".) One of the new opinions against the feature at all was Steve Holden's: > This seems to me to take Python further away from the "Computer > Programming for Everyone" arena and closer to the "Systems Programming > for Clever Individuals" camp. I don't agree. People who find it clearer to write x = getattr(obj, attr_name) can continue to do so. Now, of course, a person with such a preference has no control over what other people write, so everybody will have to understand what the new syntax means, but I think it's quite mnemonic. It's a combination of the "." attribute look-up and the "[]" dictionary look-up. Steve, further down his message, continues: > It threatens to reduce Python's readability substantially I find obj.[attr_name] = other_obj.[attr_name] a much clearer expression of the "assignment" intent of this statement than setattr(obj, attr_name, getattr(other_obj, attr_name)) in the same way that I find my_dict[new_key] = new_value clearer than a hypothetical setvalue(my_dict, new_key, new_value) would be. Opinion is divided on whether it's a readability win, but I think it *is* a win. (Well, I would, I suppose). My experience in Matlab was that code became much more readable when they introduced "dynamic fields", especially in code which sets "fields" in one variable from "fields" in others. I don't know whether others on this list have worked in Matlab and have any opinions from their experience. Turning now to the performance question, Steve also writes: > > Is 1% "too much" for this feature? > > Yes. I believe it would decrease the sum total of Python's efficiency > for a very marginal return in performance on a very marginal feature. The performance question is important, certainly. Initial reaction on python-ideas was that a 1% cost would not count as substantial, but of course quicker would be better, and there were a couple of suggestions as to how the implementation could be improved. I agree that "readability is more important than efficiency", and I think a 1% efficiency loss would be "small" compared to what I think is a readability win, especially in cases like the example above. Thanks very much for all the interest in this idea. I think I've got enough of a sense of the list's reaction to update the PEP, and will do so over the next couple of days. I'll then re-post it so that everyone can check I haven't misrepresented their opinion, and take it from there. Ben. From anthony at interlink.com.au Tue Feb 13 15:09:18 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 14 Feb 2007 01:09:18 +1100 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <1171374601.45d1c2091fe11@imp.hosting365.ie> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> Message-ID: <200702140109.21004.anthony@interlink.com.au> [meta-comment: my congratulations to Ben North for managing this process as painlessly as any syntax discussion I've ever seen. Regardless of the outcome, I'd like to see this thread referenced in the appropriate places as a great example of how to propose new features in Python] I've been thinking about this on and off some more, and the only use case I can see that's even vaguely useful (imho) is method lookup. Most other cases I can think of are really better handled with dictionaries rather than objects. I'm not sure that this is really worth adding syntax for. > I think the "obj.[attr_name]" syntax has the most support. If there is to be new syntax, I prefer this over any of the other options. > and the "wrapper class" idea of Nick Coghlan: > attrview(obj)[foo] This also appeals - partly because it's not magic syntax > Steve, further down his message, continues: > > It threatens to reduce Python's readability substantially > I find > obj.[attr_name] = other_obj.[attr_name] > a much clearer expression of the "assignment" intent of this > statement than > setattr(obj, attr_name, getattr(other_obj, attr_name)) I guess that's a matter of preference - given how rarely I see constructs like this, I don't have a problem with the latter. (But see above comment about method lookup) I really cannot think of a single time where the setattr() form of this syntax is something I would have used. Anecdotally, I asked a couple of other folks for their opinions, and they were also fairly unimpressed with it (but of course the plural of anecdote is not "data" :-) so take that for what it's worth). > Opinion is divided on whether it's a readability win, but I think > it *is* a win. (Well, I would, I suppose). My experience in > Matlab was that code became much more readable when they > introduced "dynamic fields", especially in code which sets > "fields" in one variable from "fields" in others. I don't know > whether others on this list have worked in Matlab and have any > opinions from their experience. I haven't touched Matlab for many years - but why wouldn't using dictionaries be preferable here? You have much less restrictions on key names (vs attribute names), for instance. > Turning now to the performance question, Steve also writes: > > > Is 1% "too much" for this feature? > > > > Yes. I believe it would decrease the sum total of Python's > > efficiency for a very marginal return in performance on a very > > marginal feature. > > The performance question is important, certainly. Initial > reaction on python-ideas was that a 1% cost would not count as > substantial I'd disagree. Those 1% losses add up, and it takes a heck of a lot of work to claw them back. Again, this is through my filter of "marginal value". -- Anthony Baxter It's never too late to have a happy childhood. From martin at v.loewis.de Tue Feb 13 16:50:35 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 16:50:35 +0100 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <952d92df0702130431x38c8c62atcfdf9f854bfc45f5@mail.gmail.com> References: <20070210212702.GH4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> <45D0B599.8040608@v.loewis.de> <45D1749D.6060601@holdenweb.com> <45D1955B.80508@v.loewis.de> <952d92df0702130431x38c8c62atcfdf9f854bfc45f5@mail.gmail.com> Message-ID: <45D1DE4B.5030003@v.loewis.de> Richard Tew schrieb: > If there is no Stackless 'hard switching' available (the stack > switching done with assistance of assembler) for the platform it is > being compiled on, then compilation proceeds without Stackless > compiled into Python. The module is not available. The process > of adding support for additional platforms is rather straightforward > and with access to them I can do it if required. I don't think portability to all systems should be necessary. Rather, there should be a clear demarcation as to what works and what not, and that demarcation should be fully understood so that anybody using it could know what the restrictions are. > I don't know about SEH but I believe support for SPARC was added > in 2002. > http://svn.python.org/view/stackless/trunk/Stackless/platf/switch_sparc_sun_gcc.h My concern here is more of the kind "what about the next processor?" Is there some uniform way of dealing with it, if not, is there an explicit list of architectures supported? Is there, or could there be, an automated test telling whether the implementation will work? > Right. I imagine that if there are still problems like these, they > will be brought to light when and if patches are submitted. But for > now I will address what you mention just so that people don't > assume they are still the case, especially since Stackless has > been rewritten since then. This also contributes to misunderstanding. When is "then"? I'm talking about the "new" implementation, the one that bases on setjmp and longjmp, and copies slices of the stack around. It has been rewritten after that implementation strategy was implemented? > Perhaps my pursuit of better support for asynchronous calls > has led to some confusion. Stackless is an implementation > with minimal changes to the core. It does not include any modules > which monkey patch. That's good. > It was considered and quickly dismissed to modify the socket and > file support in the core at C level to do what was needed. Primarily > because it would complicate the limited changes we make to the > core. This is also something to consider: "minimize changes to the core" should not be a goal when contributing to the core (it certainly is when you maintain a fork). Feel free to make any changes you like, completely deviating from the current code base if necessary, as long as a) the resulting code still works in all cases where the old code did (if there are changes to the semantics, they need to be discussed), and b) the resulting code is still as maintainable and readable (or better in these respects) as the old code. This misconception is what kicked out the first attempt to merge setuptools: this was apparently written in a way to minimize changes to the core, and also to preserve semantics unmodified in all details. By doing so, it became less maintainable than if it had taken the liberty to change things. Of course, then you have two versions to maintain: the in-core version that nicely integrates, and the out-of-core version, that has minimal changes. It first takes effort to create the version to contribute, and then also the trick is how to keep both of them maintainable, ideally from a single source. Regards, Martin From guido at python.org Tue Feb 13 17:03:58 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Feb 2007 08:03:58 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <200702140109.21004.anthony@interlink.com.au> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> Message-ID: On 2/13/07, Anthony Baxter wrote: > [meta-comment: my congratulations to Ben North for managing this > process as painlessly as any syntax discussion I've ever seen. > Regardless of the outcome, I'd like to see this thread referenced > in the appropriate places as a great example of how to propose new > features in Python] Seconded. > I've been thinking about this on and off some more, and the only use > case I can see that's even vaguely useful (imho) is method lookup. > Most other cases I can think of are really better handled with > dictionaries rather than objects. I'm not sure that this is really > worth adding syntax for. > > > I think the "obj.[attr_name]" syntax has the most support. > > If there is to be new syntax, I prefer this over any of the other > options. Agreed again. > > and the "wrapper class" idea of Nick Coghlan: > > attrview(obj)[foo] > > This also appeals - partly because it's not magic syntax Not to me -- magic objects are harder to grok than magic syntax; the magic syntax gives you a more direct hint that something unusual is going on than a magic object. Also, Nick's examples show (conceptual) aliasing problems: after "x = attrview(y)", both x and y refer to the same object, but use a different notation to access it. > > Steve, further down his message, continues: > > > It threatens to reduce Python's readability substantially > > I find > > obj.[attr_name] = other_obj.[attr_name] > > a much clearer expression of the "assignment" intent of this > > statement than > > setattr(obj, attr_name, getattr(other_obj, attr_name)) > > I guess that's a matter of preference - given how rarely I see > constructs like this, I don't have a problem with the latter. (But > see above comment about method lookup) I really cannot think of a > single time where the setattr() form of this syntax is something I > would have used. Anecdotally, I asked a couple of other folks for > their opinions, and they were also fairly unimpressed with it (but > of course the plural of anecdote is not "data" :-) so take that for > what it's worth). This is probably moot given the infrequency. > > Opinion is divided on whether it's a readability win, but I think > > it *is* a win. (Well, I would, I suppose). My experience in > > Matlab was that code became much more readable when they > > introduced "dynamic fields", especially in code which sets > > "fields" in one variable from "fields" in others. I don't know > > whether others on this list have worked in Matlab and have any > > opinions from their experience. > > I haven't touched Matlab for many years - but why wouldn't using > dictionaries be preferable here? You have much less restrictions on > key names (vs attribute names), for instance. > > > Turning now to the performance question, Steve also writes: > > > > Is 1% "too much" for this feature? > > > > > > Yes. I believe it would decrease the sum total of Python's > > > efficiency for a very marginal return in performance on a very > > > marginal feature. > > > > The performance question is important, certainly. Initial > > reaction on python-ideas was that a 1% cost would not count as > > substantial > > I'd disagree. Those 1% losses add up, and it takes a heck of a lot > of work to claw them back. Again, this is through my filter > of "marginal value". I missed discussion of the source of the 1%. Does it slow down pystone or other benchmarks by 1%? That would be really odd, since I can't imagine that the code path changes in any way for code that doesn't use the feature. Is it that the ceval main loop slows down by having two more cases? That's extremely noisy data; I've seen cases where adding code would actually slow it up, due to the magic of cache collisions. If that's all, I am taking the 1% figure with a lot of salt. There was some discussion of grammar and priorities. IMO the '.' and '[' should remain separate tokens; the grammar for 'trailer' grows a new alternative: trailer: ... | '.' NAME | '.' '[' test ']' There are no ambiguities in this grammar. (Since it needs to evaluate to a string I don't see the need to use testlist.) Regarding hasattr(): yes, this won't have an alternative in syntax yet. IMO that's fine; it means we'll be catching AttributeError instead of using "look before you leap", which if fine with me. As for the 3-arg version of getattr(), it's still there. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Tue Feb 13 17:20:02 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 17:20:02 +0100 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <200702140109.21004.anthony@interlink.com.au> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> Message-ID: <45D1E532.1080401@v.loewis.de> Anthony Baxter schrieb: >> and the "wrapper class" idea of Nick Coghlan: >> attrview(obj)[foo] > > This also appeals - partly because it's not magic syntax I also like this. I would like to spell it attrs, and I think its specification is class attrs: def __init__(self, obj): self.obj = obj def __getitem__(self, name): return getattr(self.obj, name) def __setitem__(self, name, value): return setattr(self.obj, name, value) def __delitem__(self, name): return delattr(self, name) def __contains__(self, name): return hasattr(self, name) It's so easy people can include in their code for backwards compatibility; in Python 2.6, it could be a highly-efficient builtin (you still pay for the lookup of the name 'attrs', of course). Regards, Martin From larry at hastings.org Tue Feb 13 17:29:23 2007 From: larry at hastings.org (Larry Hastings) Date: Tue, 13 Feb 2007 08:29:23 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D1BF19.6030706@gmail.com> References: <45CFAA81.5040906@redfrontdoor.org> <45D1BF19.6030706@gmail.com> Message-ID: <45D1E763.4010500@hastings.org> Nick Coghlan wrote: > I've tried this out on Brett's urllib & urllib2 examples below. (calling > the new builtin attrview() to emphasise the fact that it retains a > reference to the original instance). Ooh, ooh! I wanna change my vote! +1 on attrview(), +0.25 on ".[]". Maybe I haven't written enough Python, but I don't think you need this specialized form of accessing attributes very often. So I've shifted to the "new syntax seems like overkill" camp. Besides, once you've got the attrview() you can use all the existing dict syntax and it looks totally clean. The original example: setattr(self, method_name, getattr(self.metadata, method_name) which became in the new syntax: self.[method_name] = self.metadata.[method_name] would be: attrview(self)[method_name] = attrview(self.metadata)[method_name] And an attrview lets you use get() and the in operator. Plus, if you were performing a lot of operations on attributes all at one go, you'd cache it in a local and then it'd look even better. Perhaps object.__attrs__() returns a view on the object's attributes, and attrview() is simply a convenience function. /larry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070213/f770d913/attachment.html From radix at twistedmatrix.com Tue Feb 13 17:52:27 2007 From: radix at twistedmatrix.com (Christopher Armstrong) Date: Tue, 13 Feb 2007 11:52:27 -0500 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D1E532.1080401@v.loewis.de> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> <45D1E532.1080401@v.loewis.de> Message-ID: <60ed19d40702130852m54df3ea1o456ecca944e768f1@mail.gmail.com> On 2/13/07, "Martin v. L?wis" wrote: > Anthony Baxter schrieb: > >> and the "wrapper class" idea of Nick Coghlan: > >> attrview(obj)[foo] > > > > This also appeals - partly because it's not magic syntax > > I also like this. Martin and Anthony are correct. We do not need more syntax for such a trivial and trivially-implemented feature. The syntax is no real benefit. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/ From exarkun at divmod.com Tue Feb 13 18:03:13 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 13 Feb 2007 12:03:13 -0500 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D1E532.1080401@v.loewis.de> Message-ID: <20070213170313.25807.1468053862.divmod.quotient.20362@ohm> On Tue, 13 Feb 2007 17:20:02 +0100, "\"Martin v. L?wis\"" wrote: >Anthony Baxter schrieb: >>> and the "wrapper class" idea of Nick Coghlan: >>> attrview(obj)[foo] >> >> This also appeals - partly because it's not magic syntax > >I also like this. I would like to spell it attrs, and >I think its specification is > >class attrs: > def __init__(self, obj): > self.obj = obj > def __getitem__(self, name): > return getattr(self.obj, name) > def __setitem__(self, name, value): > return setattr(self.obj, name, value) > def __delitem__(self, name): > return delattr(self, name) > def __contains__(self, name): > return hasattr(self, name) > >It's so easy people can include in their code for backwards >compatibility; in Python 2.6, it could be a highly-efficient >builtin (you still pay for the lookup of the name 'attrs', >of course). This looks nice. The simplicity of the implementation is great too. Jean-Paul From g.brandl at gmx.net Tue Feb 13 18:11:37 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 13 Feb 2007 18:11:37 +0100 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D1E532.1080401@v.loewis.de> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> <45D1E532.1080401@v.loewis.de> Message-ID: Martin v. L?wis schrieb: > Anthony Baxter schrieb: >>> and the "wrapper class" idea of Nick Coghlan: >>> attrview(obj)[foo] >> >> This also appeals - partly because it's not magic syntax > > I also like this. I would like to spell it attrs, and > I think its specification is > > class attrs: > def __init__(self, obj): > self.obj = obj > def __getitem__(self, name): > return getattr(self.obj, name) > def __setitem__(self, name, value): > return setattr(self.obj, name, value) > def __delitem__(self, name): > return delattr(self, name) > def __contains__(self, name): > return hasattr(self, name) > > It's so easy people can include in their code for backwards > compatibility; in Python 2.6, it could be a highly-efficient > builtin (you still pay for the lookup of the name 'attrs', > of course). I fear people will confuse vars() and attrs() then. Georg From jcarlson at uci.edu Tue Feb 13 18:45:21 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 13 Feb 2007 09:45:21 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <200702140109.21004.anthony@interlink.com.au> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> Message-ID: <20070213093904.ACEA.JCARLSON@uci.edu> Anthony Baxter wrote: > > The performance question is important, certainly. Initial > > reaction on python-ideas was that a 1% cost would not count as > > substantial > > I'd disagree. Those 1% losses add up, and it takes a heck of a lot > of work to claw them back. Again, this is through my filter > of "marginal value". When speed loss was discussed, I was the one who stated that 1% wasn't substantial. Why? I've found variations of up to 3% in benchark times that seemed to be based on whether I was drinking juice or eating a scone while working. One percent is likely noise, in my experience, and the *speed* of Python has tended to gain double-digit percentage increases in the last few releases. - Josiah From veloso at verylowsodium.com Tue Feb 13 18:44:41 2007 From: veloso at verylowsodium.com (Greg Falcon) Date: Tue, 13 Feb 2007 12:44:41 -0500 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D1E532.1080401@v.loewis.de> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> <45D1E532.1080401@v.loewis.de> Message-ID: <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> On 2/13/07, "Martin v. L?wis" wrote: > Anthony Baxter schrieb: > >> and the "wrapper class" idea of Nick Coghlan: > >> attrview(obj)[foo] > > > > This also appeals - partly because it's not magic syntax > > It's so easy people can include in their code for backwards > compatibility; in Python 2.6, it could be a highly-efficient > builtin (you still pay for the lookup of the name 'attrs', > of course). attrview() (or whatever name it ultimately gets) feels Pythonic to me. It's a lot cleaner than getattr/setattr/hasattr/delattr. It's perhaps not entirely as pretty as .[], but it's close enough that I question whether adding new syntax is worth it. The easy backwards compatibility is the huge feature here, but there are a couple of others. Because attrview() can be made to support much of the dict interface, it can clearly express things the .[] syntax cannot, like: * attrview(obj).get(key, default) * attrview(obj).setdefault(key, []).append(x) Also, because attrview objects implement a mapping, they can be used in interesting ways in places that expect a mapping. (The locals parameter to eval() comes to mind, but I'm sure there would be many other good uses we won't be able to predict here.) Guido van Rossum wrote: > Also, Nick's examples show (conceptual) > aliasing problems: after "x = attrview(y)", both x and y refer to the > same object, but use a different notation to access it. Of course there is an aliasing here, but it's being explicitly constructed, and so I fail to see what is problematic about it. You propose in PEP 3106 that Py3k's items/keys/values dictionary methods should return wrapper objects, which provide a different interface to the same data, and which can be used to mutate the underlying objects. Aren't the set of ailasing concerns in these two cases exactly the same? +1 for attrview(), +0 for .[] syntax. (-1 for .[] if attrview() is accepted.) Greg F From rrr at ronadam.com Tue Feb 13 19:11:18 2007 From: rrr at ronadam.com (Ron Adam) Date: Tue, 13 Feb 2007 12:11:18 -0600 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> <45D1E532.1080401@v.loewis.de> Message-ID: <45D1FF46.8050408@ronadam.com> Georg Brandl wrote: > Martin v. L?wis schrieb: >> Anthony Baxter schrieb: >>>> and the "wrapper class" idea of Nick Coghlan: >>>> attrview(obj)[foo] >>> This also appeals - partly because it's not magic syntax >> I also like this. I would like to spell it attrs, and >> I think its specification is >> >> class attrs: >> def __init__(self, obj): >> self.obj = obj >> def __getitem__(self, name): >> return getattr(self.obj, name) >> def __setitem__(self, name, value): >> return setattr(self.obj, name, value) >> def __delitem__(self, name): >> return delattr(self, name) >> def __contains__(self, name): >> return hasattr(self, name) >> >> It's so easy people can include in their code for backwards >> compatibility; in Python 2.6, it could be a highly-efficient >> builtin (you still pay for the lookup of the name 'attrs', >> of course). > > I fear people will confuse vars() and attrs() then. > > Georg Would it be possible for attrview to be a property? Something like... (Probably needs more than this to handle all cases.) class obj(object): def _attrview(self): return self.__dict__ attr = property(_attrview) If it was this simple we just do obj.__dict__[foo] in the first place. Right? I'm overlooking something obvious I think, but the spelling is nice. obj[foo] -> access content obj.foo -> access attribute directly obj.attr[foo] -> access attribute dynamically Ron From jcarlson at uci.edu Tue Feb 13 19:47:05 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 13 Feb 2007 10:47:05 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D1FF46.8050408@ronadam.com> References: <45D1FF46.8050408@ronadam.com> Message-ID: <20070213104321.ACF0.JCARLSON@uci.edu> Ron Adam wrote: > Georg Brandl wrote: > Would it be possible for attrview to be a property? Yes, but getting the right spelling will be hard. > Something like... (Probably needs more than this to handle all cases.) > > class obj(object): > def _attrview(self): > return self.__dict__ > attr = property(_attrview) This doesn't handle descriptors, slots, non-dynamic instance methods, or even objects without a __dict__ . Generally speaking, you need a wrapper object. - Josiah From jcarlson at uci.edu Tue Feb 13 19:47:39 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 13 Feb 2007 10:47:39 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> Message-ID: <20070213102544.ACED.JCARLSON@uci.edu> "Greg Falcon" wrote: > Guido van Rossum wrote: > > Also, Nick's examples show (conceptual) > > aliasing problems: after "x = attrview(y)", both x and y refer to the > > same object, but use a different notation to access it. > > Of course there is an aliasing here, but it's being explicitly > constructed, and so I fail to see what is problematic about it. You > propose in PEP 3106 that Py3k's items/keys/values dictionary methods > should return wrapper objects, which provide a different interface to > the same data, and which can be used to mutate the underlying objects. > Aren't the set of ailasing concerns in these two cases exactly the > same? Only if it was spelled obj.attrview() . Having personally suggested a variant of attrview to others who have asked for dynamic attribute access in the past, I can't say I'm particularly convinced, today, that attrview(obj) or an equivalent is necessarily better than obj.[...] . Why? In order for it to be effective, either you need to save it to a name (cluttering up a namespace, adding mental overhead that values X and Y are really the same thing), or you need to keep calling attrview over and over (in which case you may as well stick to the getattr/setattr/delattr). I personally like the syntax, as my non-proportional font and syntax highlighting editor can easily be taught to recognize that particular bit of syntax and _really_ highlight it if I find that my brain can't do the pattern matching. As for people who say, "but getattr, setattr, and delattr aren't used"; please do some searches of the Python standard library. In a recent source checkout of the trunk Lib, there are 100+ uses of setattr, 400+ uses of getattr (perhaps 10-20% of which being the 3 argument form), and a trivial number of delattr calls. In terms of applications where dynamic attribute access tends to happen; see httplib, urllib, smtpd, the SocketServer variants, etc. - Josiah From martin at v.loewis.de Tue Feb 13 19:48:56 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 19:48:56 +0100 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D1FF46.8050408@ronadam.com> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> <45D1E532.1080401@v.loewis.de> <45D1FF46.8050408@ronadam.com> Message-ID: <45D20818.7060706@v.loewis.de> Ron Adam schrieb: > Would it be possible for attrview to be a property? Sure. It might conflict with a proper name of an attribute, of course. > Something like... (Probably needs more than this to handle all cases.) > > class obj(object): > def _attrview(self): > return self.__dict__ > attr = property(_attrview) That wouldn't work: you really need to invoke the entire attribute lookup machinery (e.g. to find methods, invoke properties, and so on). Also, for 2.6, it wouldn't support old-style classes. Regards, Martin From aahz at pythoncraft.com Tue Feb 13 20:13:08 2007 From: aahz at pythoncraft.com (Aahz) Date: Tue, 13 Feb 2007 11:13:08 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <1171374601.45d1c2091fe11@imp.hosting365.ie> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> Message-ID: <20070213191308.GA8383@panix.com> On Tue, Feb 13, 2007, Ben North wrote: > > I think the "obj.[attr_name]" syntax has the most support. To stop this > going round in circles for ages, then, I will take this as the winner. > I'll mention the other contenders in the PEP, including the new > "visually distinctive" suggestions > > obj@[foo] > obj.[[foo]] > > and the "wrapper class" idea of Nick Coghlan: > > attrview(obj)[foo] For most cases where this is needed, why not just use a mixin class? That works perfectly well with current Python and doesn't even look funny: obj[foo] = blah print obj[foo] My company makes heavy use of this coding style, we can use obj.foo whenever appropriate. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From guido at python.org Tue Feb 13 20:22:53 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Feb 2007 11:22:53 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <20070213191308.GA8383@panix.com> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <20070213191308.GA8383@panix.com> Message-ID: I think the point of attrview() and x.[y] and getattr()/setattr() is that these should be usable regardless of whether the object is prepared for such use; they map directly to __getattr__ and __setattr__. If I had to design an object specifically for such dynamic access, sure, I'd implement __getitem__ and friends and the need wouldn't exist. For that specific object. But the use case of general objects is different. On 2/13/07, Aahz wrote: > On Tue, Feb 13, 2007, Ben North wrote: > > > > I think the "obj.[attr_name]" syntax has the most support. To stop this > > going round in circles for ages, then, I will take this as the winner. > > I'll mention the other contenders in the PEP, including the new > > "visually distinctive" suggestions > > > > obj@[foo] > > obj.[[foo]] > > > > and the "wrapper class" idea of Nick Coghlan: > > > > attrview(obj)[foo] > > For most cases where this is needed, why not just use a mixin class? > That works perfectly well with current Python and doesn't even look > funny: > > obj[foo] = blah > print obj[foo] > > My company makes heavy use of this coding style, we can use obj.foo > whenever appropriate. > -- > Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ > > "I disrespectfully agree." --SJM > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mike.klaas at gmail.com Tue Feb 13 20:27:48 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 13 Feb 2007 11:27:48 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <20070213102544.ACED.JCARLSON@uci.edu> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> Message-ID: <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> On 2/13/07, Josiah Carlson wrote: > As for people who say, "but getattr, setattr, and delattr aren't used"; > please do some searches of the Python standard library. In a recent > source checkout of the trunk Lib, there are 100+ uses of setattr, 400+ > uses of getattr (perhaps 10-20% of which being the 3 argument form), and > a trivial number of delattr calls. In terms of applications where > dynamic attribute access tends to happen; see httplib, urllib, smtpd, > the SocketServer variants, etc. Another data point: on our six-figure loc code base, we have 123 instances of getattr, 30 instances of setattr, and 0 instances of delattr. There are 5 instances of setattr( ... getattr( ... ) ) on one line (and probably a few more that grep didn't pick up that span multiple lines). As a comparison, enumerate (that I would have believed was much more frequent a priori), is used 67 times, and zip/izip 165 times. +1 on .[] notation and the idea in general. -Mike From brett at python.org Tue Feb 13 21:05:43 2007 From: brett at python.org (Brett Cannon) Date: Tue, 13 Feb 2007 12:05:43 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D1BF19.6030706@gmail.com> References: <45CFAA81.5040906@redfrontdoor.org> <45D1BF19.6030706@gmail.com> Message-ID: On 2/13/07, Nick Coghlan wrote: > Taking a step back a bit... the basic issue is that we have an attribute > namespace (compile time key determination) that we want to access in a > dictionary style (runtime key determination). > > This is currently done by switching from syntactic access to the > getattr/setattr/delattr builtin functions. > > Elsewhere in the thread, Calvin made the suggestion that, rather than > introducing new syntax, this could instead be achieved with a wrapper > class that automatically converted dict-style access into the > appropriate calls to getattr/setattr/delattr. > In other words, an object view analagous to what Py3K is doing with keys()/values()/items(). > I've tried this out on Brett's urllib & urllib2 examples below. (calling > the new builtin attrview() to emphasise the fact that it retains a > reference to the original instance). I don't consider it any uglier than > the proposed syntax changes, and it provides a few other benefits: > > - the two-argument form is naturally available as the .get() method > on the resulting dict-like object (e.g. "attrview(obj).get(some_attr, > None)") > > - hasattr() is naturally replaced by containment testing (e.g. > "some_attr in attrview(obj)") > > - keywords/builtins are easier to look up in the documentation than > symbolic syntax > Yeah, the generalization is really nice. It allows use to ditch getattr/setattr/hasattr all without losing the expressiveness of those built-ins. > With this approach, performance would be attained by arranging to create > the view objects once, and then performing multiple dynamic attribute > accesses using those view objects. > > First urllib.py example:: > > name = 'open_' + urltype > self.type = urltype > name = name.replace('-', '_') > self_d = attrview(self) > if name in self_d: > if proxy: > return self.open_unknown_proxy(proxy, fullurl, data) > else: > return self.open_unknown(fullurl, data) > try: > if data is None: > return self_d[name](url) > else: > return self_d[name](url, data) > except socket.error, msg: > raise IOError, ('socket error', msg), sys.exc_info()[2] > > Second urllib.py example:: > > name = 'http_error_%d' % errcode > self_d = attrview(self) > if name in self_d: > method = self_d[name] > if data is None: > result = method(url, fp, errcode, errmsg, headers) > else: > result = method(url, fp, errcode, errmsg, headers, data) > if result: return result > return self.http_error_default(url, fp, errcode, errmsg, headers) > > > First urllib.py example:: > > if attr[:12] == '_Request__r_': > name = attr[12:] > get_name = 'get_' + name > if get_name in attrview(Request): > self_d = attrview(self) > self_d[get_name]() > return self_d[attr] > raise AttributeError, attr > > Second urllib2.py example:: > > handlers = chain.get(kind, ()) > for handler in handlers: > func = attrview(handler)[meth_name] > result = func(*args) > if result is not None: > return result > I also think it is just as clean as doing it any of the proposed ways:: getattr(self, name) self.[name] attrview(self)[name] So my vote is for Nick's object attribute view; +1. If we are going to do the view thing, let's go all the way! =) -Brett From exarkun at divmod.com Tue Feb 13 21:07:15 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 13 Feb 2007 15:07:15 -0500 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> Message-ID: <20070213200715.25807.171679807.divmod.quotient.20504@ohm> On Tue, 13 Feb 2007 11:27:48 -0800, Mike Klaas wrote: >On 2/13/07, Josiah Carlson wrote: > >> As for people who say, "but getattr, setattr, and delattr aren't used"; >> please do some searches of the Python standard library. In a recent >> source checkout of the trunk Lib, there are 100+ uses of setattr, 400+ >> uses of getattr (perhaps 10-20% of which being the 3 argument form), and >> a trivial number of delattr calls. In terms of applications where >> dynamic attribute access tends to happen; see httplib, urllib, smtpd, >> the SocketServer variants, etc. > >Another data point: on our six-figure loc code base, we have 123 >instances of getattr, 30 instances of setattr, and 0 instances of >delattr. There are 5 instances of setattr( ... getattr( ... ) ) on >one line (and probably a few more that grep didn't pick up that span >multiple lines). Another data point: in our six-figure loc code base, we have 469 instances of getattr, 91 instances of setattr, and 0 instances of delattr. There is one instances of setattr(..., getattr(...)), and one instance of setattr(getattr(...), ...). > >+1 on .[] notation and the idea in general. > -1 on a syntax change for this. Somewhere between -0 and +0 for a builtin or library function like attrview(). Jean-Paul From skip at pobox.com Tue Feb 13 21:13:44 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 13 Feb 2007 14:13:44 -0600 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> Message-ID: <17874.7160.131350.491881@montanaro.dyndns.org> Mike> Another data point: on our six-figure loc code base, we have 123 Mike> instances of getattr, 30 instances of setattr, and 0 instances of Mike> delattr. There are 5 instances of setattr( ... getattr( ... ) ) Mike> on one line (and probably a few more that grep didn't pick up that Mike> span multiple lines). Mike> As a comparison, enumerate (that I would have believed was much Mike> more frequent a priori), is used 67 times, and zip/izip 165 times. But (get|set|has)attr has been around much longer than enumerate. I'm almost certain they existed in 1.5, and perhaps as far back as 1.0. If you really want to compare the two, go back to your code baseline before enumerate was added to Python (2.3?) and subtract from your counts all the *attr calls that existed then and then compare the adjusted counts with enumerate. Given that you have more uses of zip/izip maybe we should be discussion syntactic support for that instead. ;-) Mike> +1 on .[] notation and the idea in general. Skip From glyph at divmod.com Tue Feb 13 21:20:06 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Tue, 13 Feb 2007 20:20:06 -0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access Message-ID: <20070213202006.28203.890165486.divmod.xquotient.1451@joule.divmod.com> On 12 Feb, 11:19 pm, greg.ewing at canterbury.ac.nz wrote: >Ben North wrote: > >> Generally gently positive, with the exception of Anthony Baxter's >> "-1", which I understand to be motivated by concerns about newcomers to >> the syntax > >The more I think about it, the more I'm leaning >towards -1 as well. Adding syntax is a very big >step, and it needs a very solid reason to jusify >it. I don't see this filling a use case that's >anywhere near common enough to reach that >threshold. I also strongly dislike every syntax that has thus far been proposed, but even if I loved them, there is just no motivating use-case. New syntax is not going to make dynamic attribute access easier to understand, and it *is* going to cause even more version-compatibility headaches. I really, really wish that every feature proposal for Python had to meet some burden of proof, or submit a cost/benefit analysis. Who is this going to help? How much is this going to help them? "Who is this going to hurt" is easy, but should also be included for completeness - everyone who wants to be able to deploy new code on old Pythons. I suspect this would kill 90% of "hey wouldn't this syntax be neat" proposals on day zero, and the ones that survived would be a lot more interesting to talk about. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070213/872ade98/attachment.html From rrr at ronadam.com Tue Feb 13 20:39:36 2007 From: rrr at ronadam.com (Ron Adam) Date: Tue, 13 Feb 2007 13:39:36 -0600 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D20818.7060706@v.loewis.de> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> <45D1E532.1080401@v.loewis.de> <45D1FF46.8050408@ronadam.com> <45D20818.7060706@v.loewis.de> Message-ID: <45D213F8.8020204@ronadam.com> Martin v. L?wis wrote: > Ron Adam schrieb: >> Would it be possible for attrview to be a property? > > Sure. It might conflict with a proper name of an attribute, of course. > >> Something like... (Probably needs more than this to handle all cases.) >> >> class obj(object): >> def _attrview(self): >> return self.__dict__ >> attr = property(_attrview) > > That wouldn't work: you really need to invoke the entire attribute > lookup machinery (e.g. to find methods, invoke properties, and so > on). Also, for 2.6, it wouldn't support old-style classes. > > Regards, > Martin Yes, I thought that might be a problem. So I guess I'm for the shortened version which wouldn't conflict with a property and could be made to work in more cases. +1 obj.[foo] I don't care for the longer attrview() as it adds another level of indirectness. I feel that is a counter solution to the point of the suggested syntax. Ron From aahz at pythoncraft.com Tue Feb 13 21:39:38 2007 From: aahz at pythoncraft.com (Aahz) Date: Tue, 13 Feb 2007 12:39:38 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <20070213191308.GA8383@panix.com> Message-ID: <20070213203938.GA6136@panix.com> On Tue, Feb 13, 2007, Guido van Rossum wrote: > > I think the point of attrview() and x.[y] and getattr()/setattr() is > that these should be usable regardless of whether the object is > prepared for such use; they map directly to __getattr__ and > __setattr__. If I had to design an object specifically for such > dynamic access, sure, I'd implement __getitem__ and friends and the > need wouldn't exist. For that specific object. But the use case of > general objects is different. My point is that I suspect that general objects are not used much with getattr/setattr. Does anyone have evidence counter to that? I think that evidence should be provided before this goes much further; perhaps all that's needed is education and documentation. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From glyph at divmod.com Tue Feb 13 21:41:11 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Tue, 13 Feb 2007 20:41:11 -0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib Message-ID: <20070213204111.28203.725763507.divmod.xquotient.1479@joule.divmod.com> On 12 Feb, 05:11 pm, richard.m.tew at gmail.com wrote: >On 2/12/07, Tristan Seligmann wrote: >> * Richard Tew [2007-02-12 13:46:43 +0000]: >> > Perhaps there is a better way. And I of course have no concept of >> > how this might be done on other platforms. >> >> Building on an existing framework that does this seems better than >> reinventing the wheel, for something of this magnitude. > >This to me seems to be the low level support you would build >something like Twisted on top of. Pushing Twisted so that others >can get it seems a little over the top. It sounds like you don't really understand what Twisted is, what it does, or the difficulty involved in building that "low level" so that it's usable by something like Twisted. Tristan is correct: this should be a patch against Twisted, or perhaps as a separate library that could implement a reactor. I have no problem with other, competing event-driven mechanisms being developed; in fact, I think it's great that the style of programming is getting some attention. But this is not a robust and straightforward wrapping of some lower level. This is an entirely new, experimental project, and the place to start developing it is _NOT_ in the Python core. Another post in this thread outlined that the first thing you should do is develop something and get people in the community to use it. Please do that, start its own mailing list, and stop discussing it here. On a personal note, I believe that "those who do not understand twisted are doomed to repeat it". I predict that the ultimate outcome of this project will be that all concerned will realize that, actually, Twisted already does almost everything that it proposes, and the microthreading features being discussed here are a trivial hack somewhere in its mainloop machinery, not an entirely new subsystem that it should be implemented in terms of. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070213/78e122ec/attachment.html From larry at hastings.org Tue Feb 13 21:46:00 2007 From: larry at hastings.org (Larry Hastings) Date: Tue, 13 Feb 2007 12:46:00 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <20070213102544.ACED.JCARLSON@uci.edu> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> Message-ID: <45D22388.4010304@hastings.org> Josiah Carlson wrote: > In a recent source checkout of the trunk Lib, there are 100+ uses of setattr, 400+ uses of getattr (perhaps 10-20% of which being the 3 argument form), and a trivial number of delattr calls. I just duplicated this test on all the .py files in the Lib directory tree of a freshly updated 2.5 trunk. The results: 228 1119 18032 sets 679 3547 52532 gets 30 105 2003 dels 937 4771 72567 total Here's wc on all those .py files: 471821 1659218 16585060 total So 937 lines out of 471,821 lines called *attr() functions, or very nearly 0.2%. I'm now totally convinced: +1 on attrview(), -1 on a syntax change. This is a minor improvement for 1 out of every 500 lines, and the class arguably nicer anyway. (Of the syntax changes, I still prefer ".[]".) /larry/ From mike.klaas at gmail.com Tue Feb 13 21:56:13 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 13 Feb 2007 12:56:13 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <17874.7160.131350.491881@montanaro.dyndns.org> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> <17874.7160.131350.491881@montanaro.dyndns.org> Message-ID: <3d2ce8cb0702131256p53dd512bl8d4311d9f83e245@mail.gmail.com> On 2/13/07, skip at pobox.com wrote: > Mike> As a comparison, enumerate (that I would have believed was much > Mike> more frequent a priori), is used 67 times, and zip/izip 165 times. > > But (get|set|has)attr has been around much longer than enumerate. I'm > almost certain they existed in 1.5, and perhaps as far back as 1.0. If you > really want to compare the two, go back to your code baseline before > enumerate was added to Python (2.3?) and subtract from your counts all the > *attr calls that existed then and then compare the adjusted counts with > enumerate. The entire codebase was developed post-2.4, and I am a bit of an enumerate-nazi, so I don't think that is a concern . > Given that you have more uses of zip/izip maybe we should be discussion > syntactic support for that instead. ;-) There are even more instances of len()... len(seq) -> |seq|? -Mike From larry at hastings.org Tue Feb 13 22:38:42 2007 From: larry at hastings.org (Larry Hastings) Date: Tue, 13 Feb 2007 13:38:42 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D22388.4010304@hastings.org> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> <45D22388.4010304@hastings.org> Message-ID: <45D22FE2.4040709@hastings.org> Larry Hastings wrote: > I just duplicated this test on all the .py files in the Lib directory > tree of a freshly updated 2.5 trunk. Whoops! Sorry, bungled it again. I counted definitions of __*attr__ too. This time I used "fgrep -w getattr | fgrep 'getattr('" to cull. The corrected results: 9 27 611 dels 488 2637 38947 gets 120 539 8644 sets 617 3203 48202 total So 617 lines out of 471,821 lines called *attr() functions, or 0.13%. In other words *attr() functions are used on 1 out of every 764 lines. /larry/ From martin at v.loewis.de Tue Feb 13 22:40:17 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 13 Feb 2007 22:40:17 +0100 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <3d2ce8cb0702131256p53dd512bl8d4311d9f83e245@mail.gmail.com> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> <17874.7160.131350.491881@montanaro.dyndns.org> <3d2ce8cb0702131256p53dd512bl8d4311d9f83e245@mail.gmail.com> Message-ID: <45D23041.5020706@v.loewis.de> Mike Klaas schrieb: > The entire codebase was developed post-2.4 Can you find out what percentage of these could have used a __getitem__ if it had been implemented? As a starting point, count all 'getattr(self' invocations. Regards, Martin From collinw at gmail.com Tue Feb 13 22:46:41 2007 From: collinw at gmail.com (Collin Winter) Date: Tue, 13 Feb 2007 15:46:41 -0600 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D1BF19.6030706@gmail.com> References: <45CFAA81.5040906@redfrontdoor.org> <45D1BF19.6030706@gmail.com> Message-ID: <43aa6ff70702131346w5a2c35d9radda89e0b9a4b1f@mail.gmail.com> On 2/13/07, Nick Coghlan wrote: [snip] > I've tried this out on Brett's urllib & urllib2 examples below. (calling > the new builtin attrview() to emphasise the fact that it retains a > reference to the original instance). I don't consider it any uglier than > the proposed syntax changes, and it provides a few other benefits: > > - the two-argument form is naturally available as the .get() method > on the resulting dict-like object (e.g. "attrview(obj).get(some_attr, > None)") > > - hasattr() is naturally replaced by containment testing (e.g. > "some_attr in attrview(obj)") > > - keywords/builtins are easier to look up in the documentation than > symbolic syntax > > With this approach, performance would be attained by arranging to create > the view objects once, and then performing multiple dynamic attribute > accesses using those view objects. This changes my vote: +1 on including attrview(), -1 on the syntax proposal. Collin Winter From johann at rocholl.net Tue Feb 13 23:17:20 2007 From: johann at rocholl.net (Johann C. Rocholl) Date: Tue, 13 Feb 2007 23:17:20 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <43aa6ff70702131346w5a2c35d9radda89e0b9a4b1f@mail.gmail.com> References: <45CFAA81.5040906@redfrontdoor.org> <45D1BF19.6030706@gmail.com> <43aa6ff70702131346w5a2c35d9radda89e0b9a4b1f@mail.gmail.com> Message-ID: <8233478f0702131417v74287e6awa094a0237d7208f3@mail.gmail.com> > On 2/13/07, Nick Coghlan wrote: > [snip] > > I've tried this out on Brett's urllib & urllib2 examples below. (calling > > the new builtin attrview() to emphasise the fact that it retains a > > reference to the original instance). I don't consider it any uglier than > > the proposed syntax changes, and it provides a few other benefits: > > > > - the two-argument form is naturally available as the .get() method > > on the resulting dict-like object (e.g. "attrview(obj).get(some_attr, > > None)") > > > > - hasattr() is naturally replaced by containment testing (e.g. > > "some_attr in attrview(obj)") > > > > - keywords/builtins are easier to look up in the documentation than > > symbolic syntax > > > > With this approach, performance would be attained by arranging to create > > the view objects once, and then performing multiple dynamic attribute > > accesses using those view objects. On 2/13/07, Collin Winter wrote: > This changes my vote: +1 on including attrview(), -1 on the syntax proposal. Me too! I imagine that the addition of the proposed "obj.[name]" syntax to Python 2.6 would lead to more software that will not run on 2.5 and earlier versions. I don't think the benefits outweigh this backward incompatibility of new code. Therefore I'm -1 on including "obj.[name]" in Python 2.6. For Python 3000, I think it would be alright. With "attrview(obj)[name]" it would be possible to provide a fallback implementation in the same module for pre-2.6 users, so +1 for that. Johann C. Rocholl From arigo at tunes.org Tue Feb 13 23:18:55 2007 From: arigo at tunes.org (Armin Rigo) Date: Tue, 13 Feb 2007 23:18:55 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D01973.9000802@cs.byu.edu> References: <45CFAA81.5040906@redfrontdoor.org> <200702121533.15305.anthony@interlink.com.au> <45D01973.9000802@cs.byu.edu> Message-ID: <20070213221854.GA20499@code0.codespeak.net> Hi, On Mon, Feb 12, 2007 at 12:38:27AM -0700, Neil Toronto wrote: > obj.*str_expression x = *('variable%d' % n) f(a, b, *('keyword%d' % n) = c) class *('33strangename'): pass def *(funcname)(x, y, *(argname), *args, **kwds): pass import *modname as mymodule Sorry for the noise, Armin From fuzzyman at voidspace.org.uk Wed Feb 14 00:55:46 2007 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Tue, 13 Feb 2007 23:55:46 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <20070213221854.GA20499@code0.codespeak.net> References: <45CFAA81.5040906@redfrontdoor.org> <200702121533.15305.anthony@interlink.com.au> <45D01973.9000802@cs.byu.edu> <20070213221854.GA20499@code0.codespeak.net> Message-ID: <45D25002.1000206@voidspace.org.uk> Armin Rigo wrote: > Hi, > > On Mon, Feb 12, 2007 at 12:38:27AM -0700, Neil Toronto wrote: > >> obj.*str_expression >> > > > x = *('variable%d' % n) > > f(a, b, *('keyword%d' % n) = c) > > class *('33strangename'): > pass > > def *(funcname)(x, y, *(argname), *args, **kwds): > pass > > import *modname as mymodule > > > Are you for these uses or mocking them ? Some of them look interesting... FWIW, at Resolver (75k lines of IronPython code currently) we use getattr / setattr almost as many as 8 times. (I was very surprised at how low that was.) That said, when I explained the proposal to two of my colleagues both of them were *very* happy with the obj.[name] suggestion. Michael Foord > Sorry for the noise, > > Armin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > From anthony at interlink.com.au Wed Feb 14 00:55:40 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 14 Feb 2007 10:55:40 +1100 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <20070213203938.GA6136@panix.com> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <20070213203938.GA6136@panix.com> Message-ID: <200702141055.43228.anthony@interlink.com.au> On Wednesday 14 February 2007 07:39, Aahz wrote: > My point is that I suspect that general objects are not used much > with getattr/setattr. Does anyone have evidence counter to that? > I think that evidence should be provided before this goes much > further; perhaps all that's needed is education and > documentation. Ok. I just spent a little time looking in the top level of the stdlib. I count 102 uses of 2-arg getattr. Here's what getattr() is used for: Putting aside all of the stuff that does introspection of objects (pydoc, cgitb, copy, &c - I don't care that these have a slightly more verbose spelling, since writing this sort of introspection tool is hardly a common thing to do throughout a codebase) there's really only three use cases: - Cheap inheritence, ala def __getattr__(self, attr): return getattr(self.socket, attr) - Method lookup by name - as I suspected, this is the #1 use case. - Looking things up in modules - you're generally only going to do a small amount of this, once per module. The other uses of it make up a fairly small handful - and looking at them, there's better ways to do some of them. setattr is even less used. Most uses of it are for copying the attributes of one object en masse to another object. This can be encapsulated in a quickie function to do the work. So based on this quick survey, I _strongly_ question the need for the new syntax. New syntax should need to vault vastly higher hurdles before getting into Python - it's just not possible to write code that's backwards compatible once you add it. I just don't see that this comes even close. We're already looking at a big discontinuity with 2.x -> 3.x in terms of compatibility, I think adding another for 2.5->2.6, for what I consider a marginal use case is VERY BAD. Something like the attrview (or attrdict, the name I'd use) _can_ be done in a backwards compatible way, where people can distribute a workalike with their code. I doubt I'd use it, either, but it's there for people who need it. I again ask for examples of other compelling uses that wouldn't be better solved by using a dictionary with keys rather than an object with attributes. Anthony From greg.ewing at canterbury.ac.nz Wed Feb 14 00:56:30 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 12:56:30 +1300 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> Message-ID: <45D2502E.9060608@canterbury.ac.nz> Brett Cannon wrote: > On 2/12/07, Collin Winter wrote: > > 2) {BinOp,AugAssign,BoolOp,etc}.op > > I can't think of a reason, off the top of my head, why they can't be > singletons. Why not just use interned strings? -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 01:20:07 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 13:20:07 +1300 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D15AE6.4010302@v.loewis.de> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> <45D0B11B.3080309@acm.org> <45D0FF6B.6040104@canterbury.ac.nz> <45D15AE6.4010302@v.loewis.de> Message-ID: <45D255B7.4020209@canterbury.ac.nz> Martin v. L?wis wrote: > Greg Ewing schrieb: > > the end result would be too > > convoluted for ordinary people to understand and maintain. > > That very much depends on what the end result would be. True. What I should have said is that Guido *believes* the outcome would be too convoluted. Christian's original Stackless implementation certainly was, and to my knowledge nobody has ever suggested a way to do better. If someone could come up with a plan for a stackless interpreter that was just as straightforward as the existing one, that would be worth considering. But I won't be holding my breath. -- Greg From pje at telecommunity.com Wed Feb 14 01:27:38 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 13 Feb 2007 19:27:38 -0500 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070213204111.28203.725763507.divmod.xquotient.1479@joule .divmod.com> Message-ID: <5.1.1.6.0.20070213171302.03a98f58@sparrow.telecommunity.com> At 08:41 PM 2/13/2007 +0000, glyph at divmod.com wrote: > and the microthreading features being discussed here are a trivial hack > somewhere in its mainloop machinery, not an entirely new subsystem that > it should be implemented in terms of. It doesn't even require hacking the mainloop; it can be done entirely in userspace. Here's some code I wrote a few months ago but never got around to using it. (Or testing it, for that matter, so it may not work, or even compile!) It requires Python 2.5 and the "simplegeneric" package from the Cheeseshop, and it allows you to "spawn" generators to create independently-running pseudothreads. These pseudothreads can yield to Deferreds, "returning" the value or failure thereof. (Failures cause the error to be raises such that it appears to happen at the yield point.) There are also a few other yield targets like "Return" (to return a value to a calling co-routine), "Pause" (to delay resumption), and "with_timeout" (to wrap some other yieldable in a timeout that raises TimeoutError). Enjoy! import types from twisted.internet import reactor from twisted.internet.defer import Deferred, TimeoutError from twisted.python import failure from simplegeneric import generic from functools import partial __all__ = [ # user APIs: 'spawn', 'Pause', 'Return', 'with_timeout', # extension APIs: 'schedule', 'resume', 'throw', 'current_task', 'yield_to', 'yieldable', ] def spawn(geniter, delay=0): """Create a new task and schedule it for execution Usage:: spawn(someGeneratorFunc(args)) The given generator-iterator will be run by the Twisted reactor after `delay` seconds have passed (0 by default). The return value of this function is a "task" object that can be passed to the ``schedule()``, ``throw()``, and ``yield_to()`` APIs. """ task = [geniter] schedule(task, delay) return task def schedule(task, delay=0, value=None): """Schedule `task` to resume after `delay` seconds (w/optional `value`)""" if task: return reactor.callLater(delay, resume, task, value) # XXX warn if non-None value sent to empty task? def resume(task, value=None): """Resume `task` immediately, returning `value` for the current yield""" if task: _invoke(task, partial(task[-1].send, value)) # XXX warn if non-None value sent to empty task? def throw(task, exc): """Raise `exc` tuple in `task` and immediately resume its execution""" if not task: # Propagate exception out of a failed task raise exc[0], exc[1], exc[2] _invoke(task, partial(task[-1].throw, *exc)) def _invoke(task, method): """Invoke method() in context of `task`, yielding to the return value""" try: value = method() except StopIteration: task.pop() # it's an exit with no return value resume(task) # so send None up the stack except: task.pop() # Propagate exception up the stack throw(task, sys.exc_info()) else: # Handle a yielded value yield_to(value, task) @generic def yield_to(value, task): """Handle a yielded value To register special handlers, use ``@yield_to.when_type()`` or or ``@yield_to.when_object()``. (See the ``simplegeneric`` docs for details.) """ raise TypeError( "Unrecognized yield value (maybe missing Return()?): %r" % (value,) ) @yield_to.when_type(defer.Deferred): def _yield_to_deferred(value, task): """Return a deferred value immediately or pause until fired""" def _resume(value): if isinstance(value, failure.Failure): throw(task, (type(value), value, None)) # XXX extract error? else: resume(task, value) return value # don't alter the deferred's value # This will either fire immediately, or delay until the appropriate time value.addCallbacks(_resume, _resume) @yield_to.when_type(types.GeneratorType) def _yield_to_subgenerator(value, task): """Call a sub-generator, putting it on the task's call stack""" task.append(value) schedule(task) def yieldable(f): """Declare that a function may be yielded to Usage:: @yieldable def do_something(task): # code that will be invoked upon (yield do_something) (yield do_something) The function's return value will be ignored, and errors in it are NOT caught! It must instead use the ``resume()``, ``schedule()``, ``throw()``, or ``yield_to()`` APIs to communicate with the yielding task (which is provided as the sole argument.) """ f.__yieldable__ = True return f @yield_to.when_type(types.FunctionType) def _yield_to_function(value, task): """Invoke task-management function""" if getattr(value, '__yieldable__', None): return value(task) # function is marked as yieldable else: raise TypeError( "Function is not marked yieldable (maybe missing Return()?): %r" % (value,) ) @yieldable def current_task(task): """Yield this function (don't call it) to obtain the current task Usage: ``task = (yield current_task)`` The yielding coroutine is immediately resumed; no task switching will occur. """ resume(task, task) class Pause(object): """Object that can be yielded to temporarily pause execution Usage:: yield Pause # allow other tasks to run, then resume ASAP yield Pause(5) # don't run for at least five seconds """ seconds = 0 def __init__(self, seconds=0): self.seconds = seconds def __repr__(self): return 'Pause(%s)' % self.seconds @yield_to.when_type(Pause) @yield_to.when_object(Pause) def _yield_to_pause(value, task): """Allow other tasks to run, by moving this task to the end of the queue""" schedule(task, value.seconds) class Return(object): """Return a value to your caller Usage:: yield Return(42) # returns 42 to calling coroutine yield Return # returns None to calling coroutine """ value = None def __init__(self, value): self.value = value def __repr__(self): return 'Return(%s)' % self.value @yield_to.when_type(Return) @yield_to.when_object(Return) def _yield_to_return(value, task): """Return a value to the caller""" task.pop().close() # ensure any ``finally`` clauses are run resume(task, value.value) def with_timeout(seconds, value): """Raise TimeoutError if `value` doesn't resume before `seconds` elapse Example:: result = yield with_timeout(30, something(whatever)) This is equivalent to ``result = yield something(whatever)``, except that if the current task isn't resumed in 30 seconds or less, a ``defer.TimeoutError`` will be raised in the blocked task. (Note: if the The `value` passed to this routine may be any yieldable value, although it makes no sense to yield a value that will just be returned to a parent coroutine. Note that the ``something(whatever)`` subtask may trap the ``TimeoutError`` and misinterpret its significance, so generally speaking you should use this only for relatively low-level operations, and expose a timeout parameter to your callers, to allow them to simply specify the timeout rather than using this wrapper. In fact, this is what most Twisted APIs do, providing a ``timeout`` keyword argument that you should use instead of this wrapper, as they will then raise ``TimeoutError`` automatically after the specified time expires. So, if you're calling such an API, don't use this wrapper. """ # raise a timeout error in this task after seconds delayedCall = reactor.callLater( seconds, throw, (yield current_task), (TimeoutError, TimeoutError(), None) ) try: yield Return((yield value)) # call the subtask and return its result finally: # Ensure timeout call doesn't fire after we've exited if delayedCall.active(): delayedCall.cancel() From greg.ewing at canterbury.ac.nz Wed Feb 14 01:24:30 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 13:24:30 +1300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D15D16.7060109@v.loewis.de> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> <45D15D16.7060109@v.loewis.de> Message-ID: <45D256BE.1020105@canterbury.ac.nz> Martin v. L?wis wrote: > Apparently, people favor hasattr over catching > AttributeError. I'm not sure why this is - I would probably > rewrite them all to deal with AttributeError if I use the new > syntax in the first place. Actually, I prefer using getattr with a default value over either of those, wherever possible. AttributeError can be raised by too many things for me to feel comfortable about catching it. This suggests that any proposed getattr syntax should include a way of specifying a default value. I'm still -1 on the basic idea, though, on the grounds of YAGNIOE (You Aren't Going to Need It Often Enough). -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 01:29:06 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 13:29:06 +1300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D15EE1.4010700@v.loewis.de> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> <45D12A25.802@ronadam.com> <45D15EE1.4010700@v.loewis.de> Message-ID: <45D257D2.3040401@canterbury.ac.nz> Martin v. L?wis wrote: > BTW, which of these would be correct My thoughts would be (a).[b] Okay (a.)[b] Not okay a.[(b)] Okay a.([b]) Not okay a . [ b ] Okay > and what is the semantics of > > a.[42] The same as getattr(a, 42), which depends on a's __getattr__ implementation. -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 01:34:25 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 13:34:25 +1300 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D1749D.6060601@holdenweb.com> References: <20070210212702.GH4869@v.igoro.us> <20070211053518.GL4869@v.igoro.us> <45CEBBAE.70002@holdenweb.com> <45CF5BD9.4080109@v.loewis.de> <20070211212808.GA8197@code0.codespeak.net> <45D0141F.10904@v.loewis.de> <952d92df0702120936j4ea9a873t60fa54e7330974a1@mail.gmail.com> <45D0B599.8040608@v.loewis.de> <45D1749D.6060601@holdenweb.com> Message-ID: <45D25911.8040309@canterbury.ac.nz> Steve Holden wrote: > I know that there's work in progress (and indeed > almost complete) to put Stackless into 2.5 It might be less confusing to rename the current version of Stackless to "microthreads" or something, since it's not really stackless at all. -- Greg From anthony at interlink.com.au Wed Feb 14 01:38:13 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 14 Feb 2007 11:38:13 +1100 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> Message-ID: <200702141138.14033.anthony@interlink.com.au> On Wednesday 14 February 2007 03:03, Guido van Rossum wrote: > Not to me -- magic objects are harder to grok than magic syntax; > the magic syntax gives you a more direct hint that something > unusual is going on than a magic object. Also, Nick's examples > show (conceptual) aliasing problems: after "x = attrview(y)", > both x and y refer to the same object, but use a different > notation to access it. Just touching on this - I meant to earlier. I'm really unsure why this is a problem. We already have similar cases, for instance dict.keys()/values()/items(). The globals() and locals() builtins also provide an alternate view with "different notation to access it". Since you're creating the view explicitly, I really don't see the problem - any more than say, creating a set from a list, or a dict from a list, or the like. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From brett at python.org Wed Feb 14 01:40:56 2007 From: brett at python.org (Brett Cannon) Date: Tue, 13 Feb 2007 16:40:56 -0800 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <45D2502E.9060608@canterbury.ac.nz> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <45D2502E.9060608@canterbury.ac.nz> Message-ID: On 2/13/07, Greg Ewing wrote: > Brett Cannon wrote: > > On 2/12/07, Collin Winter wrote: > > > 2) {BinOp,AugAssign,BoolOp,etc}.op > > > > I can't think of a reason, off the top of my head, why they can't be > > singletons. > > Why not just use interned strings? > Because the AST code at the C level represent the 'op' value as basically an enumeration value, not a string. So creating an object is somewhat easier then trying to do the conversion to an interned string. Or at least I suspect; Martin can tell me I'm wrong if I am. -Brett From barry at python.org Wed Feb 14 01:41:40 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 13 Feb 2007 19:41:40 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D256BE.1020105@canterbury.ac.nz> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> <45D15D16.7060109@v.loewis.de> <45D256BE.1020105@canterbury.ac.nz> Message-ID: <21ECA71D-E1B8-4957-8A90-7471784691F7@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 13, 2007, at 7:24 PM, Greg Ewing wrote: > I'm still -1 on the basic idea, though, on the grounds of > YAGNIOE (You Aren't Going to Need It Often Enough). I can't really add much more than what's already be stated, but I echo Greg's sentiment. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRdJaynEjvBPtnXfVAQJHuQP7B7FlA5wO28dEW62q3VSOr7iUm/6FLZ/6 hoXq47S1F/Yre0bl9p1C2bOCAXeBNXoQC55oPw4a4XUC3C2G/Pf3TJnlOZ0u1M7y 5Ug8MbDxf8SvHl3efZSOjvx2S8OPv0m11teP+d9l11upXz7ASAVlwYZhKRMum6/s ECxTGOKH2tc= =ZSIW -----END PGP SIGNATURE----- From greg.ewing at canterbury.ac.nz Wed Feb 14 01:57:42 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 13:57:42 +1300 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <45D19078.5000109@v.loewis.de> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> <1171356400.32307.80.camel@localhost> <45D19078.5000109@v.loewis.de> Message-ID: <45D25E86.1010003@canterbury.ac.nz> Martin v. L?wis wrote: > OTOH, in an application that needs unique strings, you normally know > what the scope is (i.e. where the strings come from, and when they > aren't used anymore). That's true -- if you know that all relevant strings have been interned using the appropriate method, you can just use 'is' to compare them. -- Greg From collinw at gmail.com Wed Feb 14 02:00:30 2007 From: collinw at gmail.com (Collin Winter) Date: Tue, 13 Feb 2007 19:00:30 -0600 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> Message-ID: <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> On 2/12/07, Brett Cannon wrote: > They all sound reasonable. And it's nice to have a wanted feature be > found from actual use. =) I've just uploaded patch #1659410 to SF, implementing the changes I outlined: 1) I changed some of the code emitted by asdl_c.py so _fields is always a tuple. Where _fields used to be None, it's now a 0-element tuple. 2) It turned out that {BinOp, BoolOp,AugAssign,etc}.op were already singleton instances of their respective classes. I've changed asdl_c.py to no longer emit the *_singleton names and to use the corresponding *_type types in their place, which will enable me to write "node.op is _ast.Add", etc. 3) Adding an Else node was a little more involved, but it proved do-able. TryFinally.orelse, While.orelse, For.orelse and If.orelse can now be either None or an instance of Else. Else instances have a 'body' attribute, which is a sequence of statements. Thanks, Collin Winter From greg.ewing at canterbury.ac.nz Wed Feb 14 02:07:48 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 14:07:48 +1300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D19178.3090105@v.loewis.de> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> <45D12A25.802@ronadam.com> <45D15EE1.4010700@v.loewis.de> <45D19178.3090105@v.loewis.de> Message-ID: <45D260E4.5090606@canterbury.ac.nz> Martin v. L?wis wrote: > OTOH, you can't write > > x + = 2 > > or > > a = 2 * * 4 Although oddly enough you *can* write a[. . .] I guess whoever added the ellipsis couldn't be bothered defining a new token for it. It's something of an arbitrary choice, but to me it just seems that the dot and bracket *should* be separare tokens, perhaps because I would be thinking of the [x] as a special kind of argument to the dot operator, rather than there being a funny new operator called ".[". -- Greg From guido at python.org Wed Feb 14 02:12:09 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Feb 2007 17:12:09 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <21ECA71D-E1B8-4957-8A90-7471784691F7@python.org> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <000e01c74ece$a2bb1e70$9b01a8c0@RaymondLaptop1> <45D15D16.7060109@v.loewis.de> <45D256BE.1020105@canterbury.ac.nz> <21ECA71D-E1B8-4957-8A90-7471784691F7@python.org> Message-ID: This seems to be the overwhelming feedback at this point, so I'm withdrawing my support for the proposal. I hope that Ben can write up a PEP and mark it rejected, to summarize the discussion; it's been a useful lesson. Occasinoally, negative results are worth publishing! On 2/13/07, Barry Warsaw wrote: > On Feb 13, 2007, at 7:24 PM, Greg Ewing wrote: > > I'm still -1 on the basic idea, though, on the grounds of > > YAGNIOE (You Aren't Going to Need It Often Enough). > > I can't really add much more than what's already be stated, but I > echo Greg's sentiment. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Feb 14 02:15:07 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 13 Feb 2007 17:15:07 -0800 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D260E4.5090606@canterbury.ac.nz> References: <1171286983.45d06bc75b001@imp.hosting365.ie> <45D12A25.802@ronadam.com> <45D15EE1.4010700@v.loewis.de> <45D19178.3090105@v.loewis.de> <45D260E4.5090606@canterbury.ac.nz> Message-ID: On 2/13/07, Greg Ewing wrote: > Martin v. L?wis wrote: > > > OTOH, you can't write > > > > x + = 2 > > > > or > > > > a = 2 * * 4 > > Although oddly enough you *can* write > > a[. . .] > > I guess whoever added the ellipsis couldn't be bothered > defining a new token for it. But if we ever turn it into a single token (which we just may for Py3k) don't complain if your code breaks. > It's something of an arbitrary choice, but to me it just > seems that the dot and bracket *should* be separare > tokens, perhaps because I would be thinking of the > [x] as a special kind of argument to the dot operator, > rather than there being a funny new operator called ".[". Yes, that's how I was thinking of it. But it's all moot now. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Wed Feb 14 02:54:24 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 14:54:24 +1300 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> Message-ID: <45D26BD0.9070302@canterbury.ac.nz> Guido van Rossum wrote: > it means we'll be catching AttributeError > instead of using "look before you leap", which if fine with me. A thought on this: to avoid masking bugs, when catching AttributeError you really need to isolate the getattr operation so that it's on a line by itself, with no sub-expressions, i.e. try: x = getattr(obj, name) except AttributeError: ... else: ... If you're doing that, I see no significant lack of readability in the getattr call that needs to be remedied. So I'm down to -2 now. :-) -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 02:58:38 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 14:58:38 +1300 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D1FF46.8050408@ronadam.com> References: <1171374601.45d1c2091fe11@imp.hosting365.ie> <200702140109.21004.anthony@interlink.com.au> <45D1E532.1080401@v.loewis.de> <45D1FF46.8050408@ronadam.com> Message-ID: <45D26CCE.6030902@canterbury.ac.nz> Ron Adam wrote: > Would it be possible for attrview to be a property? To avoid polluting the namespace, it would need to have a double-underscore name, and then it would be ugly to use. Unless we had a function to call it for you... oh, wait, we've already got one -- it's called getattr. :-) -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 03:07:26 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 15:07:26 +1300 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <20070213102544.ACED.JCARLSON@uci.edu> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> Message-ID: <45D26EDE.6000601@canterbury.ac.nz> Josiah Carlson wrote: > In a recent > source checkout of the trunk Lib, there are 100+ uses of setattr, 400+ > uses of getattr ... and a trivial number of delattr calls. That's out of about 250,000 lines, or 0.2%. Not a huge proportion, I would have thought. -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 03:10:15 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 15:10:15 +1300 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> Message-ID: <45D26F87.40705@canterbury.ac.nz> Mike Klaas wrote: > As a comparison, enumerate (that I would have believed was much more > frequent a priori), is used 67 times, and zip/izip 165 times. By that argument, we should be considering a special syntax for zip and izip before getattr. -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 03:14:00 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 15:14:00 +1300 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <17874.7160.131350.491881@montanaro.dyndns.org> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> <17874.7160.131350.491881@montanaro.dyndns.org> Message-ID: <45D27068.8050407@canterbury.ac.nz> skip at pobox.com wrote: > Given that you have more uses of zip/izip maybe we should be discussion > syntactic support for that instead. ;-) I actually came up with an idea for that, slightly too late to get considered in the original lockstep-iteration debate: for (x in seq1, y in seq2): ... -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 03:20:13 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 15:20:13 +1300 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070213204111.28203.725763507.divmod.xquotient.1479@joule.divmod.com> References: <20070213204111.28203.725763507.divmod.xquotient.1479@joule.divmod.com> Message-ID: <45D271DD.8090301@canterbury.ac.nz> glyph at divmod.com wrote: > I have no problem with other, competing event-driven mechanisms being > developed; The need for different event-driven mechanisms to compete with each other is the very problem that needs to be addressed. If Twisted is designed so that it absolutely *has* to use its own special event mechanism, and everything else needs to be modified to suit its requirements, then it's part of the problem, not part of the solution. -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 03:28:18 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 15:28:18 +1300 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <45D2502E.9060608@canterbury.ac.nz> Message-ID: <45D273C2.9030905@canterbury.ac.nz> Brett Cannon wrote: > Because the AST code at the C level represent the 'op' value as > basically an enumeration value, not a string. So creating an object > is somewhat easier then trying to do the conversion to an interned > string. It would seem even easier (and a lot faster) to use an array to go from C enum --> some object, which could as well be an interned string as anything else. -- Greg From mike.klaas at gmail.com Wed Feb 14 03:33:07 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Tue, 13 Feb 2007 18:33:07 -0800 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D26F87.40705@canterbury.ac.nz> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> <45D26F87.40705@canterbury.ac.nz> Message-ID: <3d2ce8cb0702131833n1097595bs558e3d73feacb37a@mail.gmail.com> On 2/13/07, Greg Ewing wrote: > Mike Klaas wrote: > > > As a comparison, enumerate (that I would have believed was much more > > frequent a priori), is used 67 times, and zip/izip 165 times. > > By that argument, we should be considering a special syntax > for zip and izip before getattr. I don't really buy that. Frequency of use must be balanced against the improvement in legibility. Assuming that my figures bear some correspondence to typical usage patterns, enumerate() was introduced despite the older idiom of for i, item in zip(xrange(len(seq)), seq): being less frequent than getattr. SImilarly, I see no clamor to add syntactic support for len(). It's current usage is clear. [note: this post is not continuing to argue in favour of the proposal] -Mike From greg.ewing at canterbury.ac.nz Wed Feb 14 03:31:18 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 15:31:18 +1300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <1171286983.45d06bc75b001@imp.hosting365.ie> <45D12A25.802@ronadam.com> <45D15EE1.4010700@v.loewis.de> <45D19178.3090105@v.loewis.de> <45D260E4.5090606@canterbury.ac.nz> Message-ID: <45D27476.20507@canterbury.ac.nz> Guido van Rossum wrote: > But if we ever turn it into a single token (which we just may for > Py3k) don't complain if your code breaks. I won't. I always treat it as a single token anyway, unless I'm entering an obfuscated python competition. :-) -- Greg From greg.ewing at canterbury.ac.nz Wed Feb 14 03:32:24 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 14 Feb 2007 15:32:24 +1300 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <45D196EC.3050303@v.loewis.de> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> <45D10E50.2080107@canterbury.ac.nz> <45D196EC.3050303@v.loewis.de> Message-ID: <45D274B8.70505@canterbury.ac.nz> Martin v. L?wis wrote: > Greg Ewing schrieb: > > > The string comparison method knows when both > > strings are interned > > No, it doesn't - see stringobject.c:string_richcompare. Well, I'm surprised and confused. It's certainly possible to tell very easily whether a string is interned -- there's a PyString_CHECK_INTERNED macro that tests a field in the string object header. But, I can't find anywhere that it's used in the core, apart from the interning code itself and the string alloc/dealloc code. Can anyone shed any light on this? It seems to me that by not using this information, only half the benefit of interning is being achieved. -- Greg From skip at pobox.com Wed Feb 14 04:17:54 2007 From: skip at pobox.com (skip at pobox.com) Date: Tue, 13 Feb 2007 21:17:54 -0600 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D27068.8050407@canterbury.ac.nz> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> <17874.7160.131350.491881@montanaro.dyndns.org> <45D27068.8050407@canterbury.ac.nz> Message-ID: <17874.32610.151491.590045@montanaro.dyndns.org> Greg> I actually came up with an idea for that, slightly too late to get Greg> considered in the original lockstep-iteration debate: Greg> for (x in seq1, y in seq2): Greg> ... That's already valid syntax though. Skip From exarkun at divmod.com Wed Feb 14 04:34:03 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 13 Feb 2007 22:34:03 -0500 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <45D271DD.8090301@canterbury.ac.nz> Message-ID: <20070214033403.25807.820994126.divmod.quotient.20851@ohm> On Wed, 14 Feb 2007 15:20:13 +1300, Greg Ewing wrote: > Greg, productive discussion is not furthered by the unsupported statement of one position or another. Instead of only stating what you believe to be a problem, explain why you believe it is a problem. A sentence like: > >The need for different event-driven mechanisms to compete >with each other is the very problem that needs to be >addressed. > Invites a response which merely contradicts it (for example, "you are wrong"), an exchange which hasn't helped anyone to understand anything better. If you present supporting evidence for the position, then the validity and the weight of that evidence can be discussed, and one position or another might be shown to have greater validity. Also, show that you have fully understood the position you are arguing against. For example, if you respond to a message in which someone claims to welcome something, don't respond by saying that requiring that thing is bad. As you know, welcoming something is not the same as requiring that thing, so by making this statement alone, you give the impression of talking past the person to whom you are responding and it may seem to readers that you haven't understood the other person's position. >If Twisted is designed so that it absolutely *has* to >use its own special event mechanism, and everything else >needs to be modified to suit its requirements, then it's >part of the problem, not part of the solution. > Here, you've built on your unsupported premise to arrive at a conclusion which may be controversial. Again, instead of couching the debate in terms of what you might see as self evidence problems, explain why you hold the position you do. That way, the possibility is created for other people to come to understand why you believe the conclusion to be valid. You have presented what could be the beginning of supporting evidence here, in saying that requiring "everything else" to be modified is undesirable. This is only a place to start, not to end, though. You may want to discuss the real scope of modifications required (because "everything" is obviously hyperbole, focusing on what changes are actually necessary would be beneficial) and why you think that modifications are necessary (it may not be clear to others why they are, or it may be the case that others can correct misconceptions you have). For example, you might give a case in which you have needed to integrate Twisted (or a different event framework) with another event loop and describe difficulties you discovered. This will help advance the discussion around practical, specific concerns. Without this focus, it is hard for a discussion to be productive, since it will involve only vague handwaving. Finally, it is often beneficial to avoid bringing up phrases such as "the problem". Particularly in a context such as this, where the existing discussion is focusing on a specific issue, such as the necessity or utility of adding a new set of functionality to the Python standard library, the relevance of "the problem" may not be apparent to readers. In this case, some may not find it obvious how a third party library can be "the problem" with such new functionality. If you explicitly spell out the detrimental consequences of an action, instead of waving around "the problem", the resulting discussion can be that much more productive and focused. Thanks Jean-Paul From glyph at divmod.com Wed Feb 14 08:26:21 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Wed, 14 Feb 2007 07:26:21 -0000 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) Message-ID: <20070214072621.28203.1662509932.divmod.xquotient.1571@joule.divmod.com> On 02:20 am, greg.ewing at canterbury.ac.nz wrote: >If Twisted is designed so that it absolutely *has* to >use its own special event mechanism, and everything else >needs to be modified to suit its requirements, then it's >part of the problem, not part of the solution. I've often heard this complaint, usually of the form "that's twisted-specific". The thing is, Twisted isn't specific. Its event mechanism isn't "special". In fact it's hard to imagine how it might be made less "special" than it currently is. Whatever your event dispatch mechanism, *some* code has to be responsible for calling the OS API of select(), WaitForMultipleEvents(), g_main_loop_run(), or whatever. Twisted actually imposes very few requirements for code to participate in this, and was designed from day one specifically to be a generalized mainloop mechanism which would not limit you to one underlying multiplexing implementation, event-dispatch mechanism, or operating system if you used its API. There have even been a few hacks to integrate Twisted with the asyncore mainloop, but almost everyone who knows both asyncore and Twisted has rapidly decided to just port all of their code to Twisted rather than maintain that bridge. In fact, Twisted includes fairly robust support for threads, so if you really need it you can mix and match event-driven and blocking styles. Again, most people who try this find that it is just nicer to write straight to the Twisted API, but for those that need really it, such as Zope and other WSGI-contained applications, it is available. Aside from the perennial issue of restartable reactors (which could be resolved as part of a stdlib push), Twisted's event loop imposes very few constraints on your code. It provides a lot of tools, sure, but few of them are required. You don't even *need* to use Deferreds. Now, "Twisted", overall, can be daunting. It has a lot of conventions, a lot of different interfaces to memorize and deal with, but if you are using the main loop you don't have to necessarily care about our SSH, ECMA 48, NNTP, OSCAR or WebDAV implementation. Those are all built at a higher level. It may seem like I'm belaboring the point here, but every time a thread comes up on this list mentioning the possibility of a "standard" event mechanism, people who do not know it very well or haven't used it start in with implied FUD that Twisted is a big, complicated, inseparable hairy mess that has no place in the everyday Python programmer's life. It's tangled-up Deep Magic, only for Wizards Of The Highest Order. Alternatively, more specific to this method, it's highly specific and intricate and very tightly coupled. Nothing could be further from the truth. It is *strictly* layered to prevent pollution of the lower levels by higher level code, and all the dependencies are one-way. Maybe our introductory documentation isn't very good. Maybe event-driven programming is confusing for those expecting something else. Maybe the cute names of some of the modules are offputting. Still, all that aside, if you are looking for an event-driven networking engine, Twisted is about as straightforward as you can get without slavishly coding to one specific platform API. When you boil it down, Twisted's event loop is just a notification for "a connection was made", "some data was received on a connection", "a connection was closed", and a few APIs to listen or initiate different kinds of connections, start timed calls, and communicate with threads. All of the platform details of how data is delivered to the connections are abstracted away. How do you propose we would make a less "specific" event mechanism? I strongly believe that there is room for a portion of the Twisted reactor API to be standardized in the stdlib so that people can write simple event-driven code "out of the box" with Python, but still have the different plug-in implementations live in Twisted itself. The main thing blocking this on my end (why I am not writing PEPs, advocating for it more actively, etc) is that it is an extremely low priority, and other, higher level pieces of Twisted have more pressing issues (such as the current confusion in the "web" universe). Put simply, although it might be nice, nobody really *needs* it in the stdlib, so they're not going to spend the effort to get it there. If someone out there really had a need for an event mechanism in the standard library, though, I encourage them to look long and hard at how the existing interfaces in Twisted could be promoted to the standard library and continue to be maintained compatibly in both places. At the very least, standardizing on something very much like IProtocol would go a long way towards making it possible to write async clients and servers that could run out of the box in the stdlib as well as with Twisted, even if the specific hookup mechanism (listenTCP, listenSSL, et. al.) were incompatible - although a signature compatible callLater would probably be a must. As I said, I don't have time to write the PEPs myself, but I might fix some specific bugs if there were a clear set of issues preventing this from moving forward. Better integration with the standard library would definitely be a big win for both Twisted and Python. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/e3f6eb5b/attachment.html From martin at v.loewis.de Wed Feb 14 09:08:29 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Feb 2007 09:08:29 +0100 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <45D274B8.70505@canterbury.ac.nz> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> <45D10E50.2080107@canterbury.ac.nz> <45D196EC.3050303@v.loewis.de> <45D274B8.70505@canterbury.ac.nz> Message-ID: <45D2C37D.6010400@v.loewis.de> Greg Ewing schrieb: > It's certainly possible to tell very easily whether > a string is interned -- there's a PyString_CHECK_INTERNED > macro that tests a field in the string object header. > But, I can't find anywhere that it's used in the core, > apart from the interning code itself and the string > alloc/dealloc code. > > Can anyone shed any light on this? It seems to me that > by not using this information, only half the benefit of > interning is being achieved. You seem to assume that checking this property would improve performance. I find that questionable; as always with performance and optimization, one cannot know until it has been benchmarked. My guess is that - the cases you want to cover are comparably infrequent (compared to the total number of richcompare invocations) - checking the additional field always may cause a performance loss, not a performance gain, because of the additional tests. If you want to sure, you need to make some measurements (e.g: what is the percentage of EQ and NE invocations, in how many cases are both strings interned, and in what percentage of these cases are they not the same string). Regards, Martin From martin at v.loewis.de Wed Feb 14 09:44:07 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Feb 2007 09:44:07 +0100 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <45D2502E.9060608@canterbury.ac.nz> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <45D2502E.9060608@canterbury.ac.nz> Message-ID: <45D2CBD7.4070801@v.loewis.de> Greg Ewing schrieb: > Brett Cannon wrote: >> On 2/12/07, Collin Winter wrote: >>> 2) {BinOp,AugAssign,BoolOp,etc}.op >> I can't think of a reason, off the top of my head, why they can't be >> singletons. > > Why not just use interned strings? Because it's generated code. It has to follow rules, and it would be good if there were few exceptions. Regards, Martin From martin at v.loewis.de Wed Feb 14 10:20:07 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Feb 2007 10:20:07 +0100 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> Message-ID: <45D2D447.4080408@v.loewis.de> Collin Winter schrieb: > 2) It turned out that {BinOp, BoolOp,AugAssign,etc}.op were already > singleton instances of their respective classes. I've changed > asdl_c.py to no longer emit the *_singleton names and to use the > corresponding *_type types in their place, which will enable me to > write "node.op is _ast.Add", etc. I don't really like this - it's inconsistent. I'd rather prefer if the singletons where exposed under a name, e.g. _ast.Add.singleton, or _ast.add (if that doesn't cause conflicts). > 3) Adding an Else node was a little more involved, but it proved > do-able. TryFinally.orelse, While.orelse, For.orelse and If.orelse can > now be either None or an instance of Else. Else instances have a > 'body' attribute, which is a sequence of statements. I still can't see the need (we don't have line numbers for every token in the AST), but I can't see anything wrong with it, either (except for the change in the structure, of course, but that will happen, anyway). Regards, Martin From martin at v.loewis.de Wed Feb 14 10:23:24 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Feb 2007 10:23:24 +0100 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <45D273C2.9030905@canterbury.ac.nz> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <45D2502E.9060608@canterbury.ac.nz> <45D273C2.9030905@canterbury.ac.nz> Message-ID: <45D2D50C.6050000@v.loewis.de> Greg Ewing schrieb: > It would seem even easier (and a lot faster) to use an > array to go from C enum --> some object, which could > as well be an interned string as anything else. Have you actually looked at the code? How could it be faster than PyObject* ast2obj_boolop(boolop_ty o) { switch(o) { case And: Py_INCREF(And_singleton); return And_singleton; case Or: Py_INCREF(Or_singleton); return Or_singleton; } return NULL; /* cannot happen */ } Not sure what interned strings have to do with it. Regards, Martin From skip at pobox.com Wed Feb 14 13:02:02 2007 From: skip at pobox.com (skip at pobox.com) Date: Wed, 14 Feb 2007 06:02:02 -0600 Subject: [Python-Dev] Summary of "dynamic attribute access" discussion In-Reply-To: <45D2C2EA.6070702@canterbury.ac.nz> References: <45D1E532.1080401@v.loewis.de> <3cdcefb80702130944l7cfeb469hdd1a57a04599bcaf@mail.gmail.com> <20070213102544.ACED.JCARLSON@uci.edu> <3d2ce8cb0702131127u4ce8a81fw14dde06f6cd36be7@mail.gmail.com> <17874.7160.131350.491881@montanaro.dyndns.org> <45D27068.8050407@canterbury.ac.nz> <17874.32610.151491.590045@montanaro.dyndns.org> <45D2C2EA.6070702@canterbury.ac.nz> Message-ID: <17874.64058.622112.490718@montanaro.dyndns.org> Greg> skip at pobox.com wrote: Greg> for (x in seq1, y in seq2): >> That's already valid syntax though. Greg> No, it's not... >>>> for (x in seq1, y in seq2): Greg> File "", line 1 Greg> for (x in seq1, y in seq2): Greg> ^ Greg> SyntaxError: invalid syntax Ah, yeah. Misread it and just focused on the bit that looked like a tuple... Skip From arigo at tunes.org Wed Feb 14 12:56:19 2007 From: arigo at tunes.org (Armin Rigo) Date: Wed, 14 Feb 2007 12:56:19 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D25002.1000206@voidspace.org.uk> References: <45CFAA81.5040906@redfrontdoor.org> <200702121533.15305.anthony@interlink.com.au> <45D01973.9000802@cs.byu.edu> <20070213221854.GA20499@code0.codespeak.net> <45D25002.1000206@voidspace.org.uk> Message-ID: <20070214115619.GA19265@code0.codespeak.net> Hi Michael, On Tue, Feb 13, 2007 at 11:55:46PM +0000, Michael Foord wrote: > > x = *('variable%d' % n) > > f(a, b, *('keyword%d' % n) = c) > > class *('33strangename'): > > pass > > def *(funcname)(x, y, *(argname), *args, **kwds): > > pass > > import *modname as mymodule > > > Are you for these uses or mocking them ? Some of them look interesting... I don't actually have any strong opinion, I was just exploring all the places in the grammar that say NAME... Some of the above are definitely just absurd. In general I am failing to see much the point of syntax extensions, so I'll step out of this discussion again. Armin From ben at redfrontdoor.org Wed Feb 14 14:04:14 2007 From: ben at redfrontdoor.org (Ben North) Date: Wed, 14 Feb 2007 13:04:14 +0000 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax Message-ID: <1171458253.45d308ce00f09@imp.hosting365.ie> Guido van Rossum wrote: > This seems to be the overwhelming feedback at this point, so I'm > withdrawing my support for the proposal. I hope that Ben can write up > a PEP and mark it rejected, to summarize the discussion; it's been a > useful lesson. The feedback is clear, yes. The "new syntax seems like overkill" camp has become dominant, and I certainly think these are very valid arguments. Gentle support remains scattered here and there, but the consensus has emerged. I will summarise all this into the PEP and mark it as rejected. Still, the discussion was useful, I thought. Thanks for the interest. To respond, for the record, to some of the specific points recently raised: glyph at divmod.com: > I really, really wish that every feature proposal for Python had to meet > some burden of proof [...]. I suspect this would kill 90% of "hey > wouldn't this syntax be neat" proposals on day zero [...] This is what I understood the initial posting to python-ideas to be about. If the feedback from there had been "that's a stupid idea", then that would have been the end of it. I think it's a good thing that there's the python-ideas mechanism for speculative suggestions. I like the "attrview" or "attrs" wrapper idea, with the example given by Larry Hastings: > setattr(self, method_name, getattr(self.metadata, method_name)) > [...] would be: > attrview(self)[method_name] = attrview(self.metadata)[method_name] As others point out, it's very clean, captures the intended uses well, and has the great advantage of having easily-added backwards compatibility. I might start using it :-) If the people who suggested and refined this were to put it in a PEP, I'd be in favour. Guido van Rossum wrote: > I missed discussion of the source of the 1%. Does it slow down pystone > or other benchmarks by 1%? That would be really odd, since I can't > imagine that the code path changes in any way for code that doesn't > use the feature. Is it that the ceval main loop slows down by having > two more cases? That seems to be it, yes. I tested this by leaving the grammar, compilation, and AST changes in, and conditionally compiling just the three extra cases in the ceval main loop. Measurements were noisy though, as Josiah Carlson has also experienced: > I've found variations of up to 3% in benchark times that seemed to be > based on whether I was drinking juice or eating a scone while working. I'm afraid I can't remember what I was eating or drinking at the time I did my tests. (Thanks also for the kind words regarding my summaries etc. Having caused all the fuss in the first place I felt obliged to try to make myself a bit useful :-) Ben. From aahz at pythoncraft.com Wed Feb 14 15:22:22 2007 From: aahz at pythoncraft.com (Aahz) Date: Wed, 14 Feb 2007 06:22:22 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070214072621.28203.1662509932.divmod.xquotient.1571@joule.divmod.com> References: <20070214072621.28203.1662509932.divmod.xquotient.1571@joule.divmod.com> Message-ID: <20070214142222.GB5088@panix.com> On Wed, Feb 14, 2007, glyph at divmod.com wrote: > > As I said, I don't have time to write the PEPs myself, but I might fix > some specific bugs if there were a clear set of issues preventing this > from moving forward. Better integration with the standard library > would definitely be a big win for both Twisted and Python. Here's where I'm coming from: My first experience with Twisted was excellent: I needed a web server in fifteen minutes to do my PyCon presentation, and it Just Worked. My second experience with Twisted? Well, I didn't really have one. My first experience was Twisted 1.1, and when I tried installing 2.0 on my Mac (about 1.5 years ago), it just didn't work. Combined with the difficulty of using the documentation and the fact that I was in a hurry, I rejected the Twisted solution. (My company needed an FTP server that did a callback every time a file got uploaded -- something that I expect would be very simple for Twisted.) -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I disrespectfully agree." --SJM From steve at holdenweb.com Wed Feb 14 16:24:30 2007 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Feb 2007 15:24:30 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <20070213110023.GB11163@phd.pp.ru> References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> Message-ID: Oleg Broytmann wrote: > On Tue, Feb 13, 2007 at 10:10:37AM +0000, Steve Holden wrote: >> Python further away from the "Computer Programming for Everyone" arena >> and closer to the "Systems Programming for Clever Individuals" camp. > > That's because Python is being developed by "Clever Individuals" and not > by "Computer Programming for Everyone Committee". > I agree that the developers are Clever Individuals. So clever, in fact, that they realise Python needs to be as approachable as possible, not a language for them alone. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From thomas at python.org Wed Feb 14 16:30:19 2007 From: thomas at python.org (Thomas Wouters) Date: Wed, 14 Feb 2007 16:30:19 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <45CDBA55.2000907@v.loewis.de> References: <45CAF232.7000505@hpi.uni-potsdam.de> <45CBAD70.1040201@canterbury.ac.nz> <45CC35F8.9040308@v.loewis.de> <45CD0D4E.7050308@canterbury.ac.nz> <45CDBA55.2000907@v.loewis.de> Message-ID: <9e804ac0702140730s280cad7mfa6c459ed9583af1@mail.gmail.com> The same way += et al. are in-place: it would ask 'x' to modify itself, if it can. If not, no harm done. (It would be called as 'x = ipow(x, n, 10)' of course, just like 'x += n' is really 'x = x.__iadd__(n)') On 2/10/07, "Martin v. L?wis" wrote: > > Greg Ewing schrieb: > >> What could the syntax for that be? > > > > It wouldn't be a syntax, just a function, e.g. > > > > ipow(x, n, 10) > > In what way would that be inplace? A function cannot > rebind the variables it gets as parameters. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/thomas%40python.org > -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/ee026245/attachment.html From steve at holdenweb.com Wed Feb 14 16:49:37 2007 From: steve at holdenweb.com (Steve Holden) Date: Wed, 14 Feb 2007 15:49:37 +0000 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: <1171458253.45d308ce00f09@imp.hosting365.ie> References: <1171458253.45d308ce00f09@imp.hosting365.ie> Message-ID: Ben North wrote: [...] > Guido van Rossum wrote: >> I missed discussion of the source of the 1%. Does it slow down pystone >> or other benchmarks by 1%? That would be really odd, since I can't >> imagine that the code path changes in any way for code that doesn't >> use the feature. Is it that the ceval main loop slows down by having >> two more cases? > > That seems to be it, yes. I tested this by leaving the grammar, > compilation, and AST changes in, and conditionally compiling just the > three extra cases in the ceval main loop. Measurements were noisy > though, as Josiah Carlson has also experienced: > >> I've found variations of up to 3% in benchark times that seemed to be >> based on whether I was drinking juice or eating a scone while working. > > I'm afraid I can't remember what I was eating or drinking at the time I > did my tests. > A further data point is that modern machines seem to give timing variabilities due to CPU temperature variations even if you always eat exactly the same thing. One of the interesting facts to emerge from the Need for Speed sprint last year is that architectural complexities at many levels make it extremely difficult nowadays to build a repeatable benchmark of any kind. > (Thanks also for the kind words regarding my summaries etc. Having > caused all the fuss in the first place I felt obliged to try to make > myself a bit useful :-) > Your management of the discussion process has indeed been exemplary. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From phd at phd.pp.ru Wed Feb 14 17:27:58 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Wed, 14 Feb 2007 19:27:58 +0300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> Message-ID: <20070214162758.GB25564@phd.pp.ru> On Wed, Feb 14, 2007 at 03:24:30PM +0000, Steve Holden wrote: > Oleg Broytmann wrote: > > On Tue, Feb 13, 2007 at 10:10:37AM +0000, Steve Holden wrote: > >> Python further away from the "Computer Programming for Everyone" arena > >> and closer to the "Systems Programming for Clever Individuals" camp. > > > > That's because Python is being developed by "Clever Individuals" and not > > by "Computer Programming for Everyone Committee". > > > I agree that the developers are Clever Individuals. So clever, in fact, > that they realise Python needs to be as approachable as possible, not a > language for them alone. Anyway I don't believe it's possible to create a CP4E language without a "CP4E Steering Committee", and I think such committee it Python land is... dormant at present... Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From theller at ctypes.org Wed Feb 14 17:38:32 2007 From: theller at ctypes.org (Thomas Heller) Date: Wed, 14 Feb 2007 17:38:32 +0100 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: References: <1171458253.45d308ce00f09@imp.hosting365.ie> Message-ID: Steve Holden schrieb: > Ben North wrote: > [...] >> Guido van Rossum wrote: >>> I missed discussion of the source of the 1%. Does it slow down pystone >>> or other benchmarks by 1%? That would be really odd, since I can't >>> imagine that the code path changes in any way for code that doesn't >>> use the feature. Is it that the ceval main loop slows down by having >>> two more cases? >> >> That seems to be it, yes. I tested this by leaving the grammar, >> compilation, and AST changes in, and conditionally compiling just the >> three extra cases in the ceval main loop. Measurements were noisy >> though, as Josiah Carlson has also experienced: >> >>> I've found variations of up to 3% in benchark times that seemed to be >>> based on whether I was drinking juice or eating a scone while working. >> >> I'm afraid I can't remember what I was eating or drinking at the time I >> did my tests. >> > A further data point is that modern machines seem to give timing > variabilities due to CPU temperature variations even if you always eat > exactly the same thing. > > One of the interesting facts to emerge from the Need for Speed sprint > last year is that architectural complexities at many levels make it > extremely difficult nowadays to build a repeatable benchmark of any kind. My personal experience using a dual core machine (on WinXP) is that timing results become much more reproducible. Thomas From richard.m.tew at gmail.com Wed Feb 14 17:49:03 2007 From: richard.m.tew at gmail.com (Richard Tew) Date: Wed, 14 Feb 2007 16:49:03 +0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070213204111.28203.725763507.divmod.xquotient.1479@joule.divmod.com> References: <20070213204111.28203.725763507.divmod.xquotient.1479@joule.divmod.com> Message-ID: <952d92df0702140849w65189a0fnc8f52100888c412a@mail.gmail.com> On 2/13/07, glyph at divmod.com wrote: > Tristan is correct: this should be a patch against Twisted, or perhaps as a > separate library that could implement a reactor. I think there is some confusion here. I am not writing a competing event driven mechanism. What I was doing was feeling out whether there was any interest in better support for asynchronous calls. You are right. This should not have been on this list. It should have been on python-ideas, but I replied here because Dustin started the discussion here and I thought if a generator based microthreading solution were to be implemented, it could also benefit from better asynchronous support so brought it into the discussion. After all, if microthreading was in the Python distribution, then it makes sense to make other general improvements which make them more useful. What I meant to suggest that Twisted could be based on was the asynchronous file IO support using IOCP. Especially because Tristan implied that Twisted's own IOCP reactor was incomplete. I did not say this clearly. Yes I have looked at Twisted. It was the first place I looked, several months ago, and what drew me to it was the IOCP reactor. However as I also explained in my reply to Tristan it is not suitable for my needs. It is a large dependency and it is a secondary framework. And I did not feel the need to verify the implication that it wasn't ready because this matched my own recollections. But I hope you realise that asserting that things should be within Twisted without giving any reason, especially when the first person saying it just stated that the matching work in Twisted wasn't even ready, feels like Twisted is trying to push itself forward by bullying people to work within it whenever something can be asserted as laying within whatever domain it is which Twisted covers. Even if it doesn't suit their needs. I don't think Tristan intended that, but when followed up with your reply and JP's interesting response to Greg, it feels like it. From mithrandi-python-dev at mithrandi.za.net Wed Feb 14 18:10:06 2007 From: mithrandi-python-dev at mithrandi.za.net (Tristan Seligmann) Date: Wed, 14 Feb 2007 19:10:06 +0200 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <952d92df0702140849w65189a0fnc8f52100888c412a@mail.gmail.com> References: <20070213204111.28203.725763507.divmod.xquotient.1479@joule.divmod.com> <952d92df0702140849w65189a0fnc8f52100888c412a@mail.gmail.com> Message-ID: <20070214171005.GA6035@mithrandi.za.net> * Richard Tew [2007-02-14 16:49:03 +0000]: > I am not writing a competing event driven mechanism. What I was doing > was feeling out whether there was any interest in better support for > asynchronous calls. I interpreted your suggestions as being about enhancing asyncore with IOCP (or similar); sure, asyncore is already a "competing" event driven mechanism, but for all intents and purposes it is "dead" as there is not much in the way of active development on it, and a relatively small remaining user base. You seemed to be suggesting that asyncore be revived by extending the implementation, as well as sprinkling some generator / tasklets / channels / etc. sugar on top of it. > But I hope you realise that asserting that things should be within > Twisted without giving any reason, especially when the first person > saying it just stated that the matching work in Twisted wasn't even > ready, feels like Twisted is trying to push itself forward by bullying > people to work within it whenever something can be asserted as > laying within whatever domain it is which Twisted covers. I don't really think it's "bullying" to suggest that improving Twisted, rather than reimplementing Twisted's core with a brand new implementation slapped straight into the Python stdlib, is counterproductive. > Even if it doesn't suit their needs. I don't think Tristan intended > that, but when followed up with your reply and JP's interesting > response to Greg, it feels like it. The gist of my message was "Instead of reimplementing the Twisted reactor, probably poorly, just use Twisted", based on my understanding of what your suggestion was. -- mithrandi, i Ainil en-Balandor, a faer Ambar -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20070214/caadfaba/attachment.pgp From martin at v.loewis.de Wed Feb 14 18:11:55 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 14 Feb 2007 18:11:55 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <9e804ac0702140730s280cad7mfa6c459ed9583af1@mail.gmail.com> References: <45CAF232.7000505@hpi.uni-potsdam.de> <45CBAD70.1040201@canterbury.ac.nz> <45CC35F8.9040308@v.loewis.de> <45CD0D4E.7050308@canterbury.ac.nz> <45CDBA55.2000907@v.loewis.de> <9e804ac0702140730s280cad7mfa6c459ed9583af1@mail.gmail.com> Message-ID: <45D342DB.4010802@v.loewis.de> Thomas Wouters schrieb: > > The same way += et al. are in-place: it would ask 'x' to modify itself, > if it can. If not, no harm done. (It would be called as 'x = ipow(x, n, > 10)' of course, just like 'x += n' is really 'x = x.__iadd__(n)') I think this would violate the policy that a mutating function shouldn't give the object being modified as the result - just as list.reverse doesn't return the list, in addition to reversing it in-place. Regards, Martin From thomas at python.org Wed Feb 14 19:42:04 2007 From: thomas at python.org (Thomas Wouters) Date: Wed, 14 Feb 2007 10:42:04 -0800 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <45D342DB.4010802@v.loewis.de> References: <45CAF232.7000505@hpi.uni-potsdam.de> <45CBAD70.1040201@canterbury.ac.nz> <45CC35F8.9040308@v.loewis.de> <45CD0D4E.7050308@canterbury.ac.nz> <45CDBA55.2000907@v.loewis.de> <9e804ac0702140730s280cad7mfa6c459ed9583af1@mail.gmail.com> <45D342DB.4010802@v.loewis.de> Message-ID: <9e804ac0702141042x608a8e5ek2a9095be056210f9@mail.gmail.com> Sure, and I don't know if anyone will ever want ipow() -- but I've never seen real code use the three-argument pow() either. The fact is that all the in-place modifying hooks return the result (which may or may not be self, and may or may not be mutated) so an in-place three-argument pow() would have to do the same. I would prefer keeping the similarity between __ipow__ and __pow__, although I don't care if that means keeping the always-unused third argument to __ipow__ (which isn't really in the way, after all) or adding a new hook for the three-argument pow(). On 2/14/07, "Martin v. L?wis" wrote: > > Thomas Wouters schrieb: > > > > The same way += et al. are in-place: it would ask 'x' to modify itself, > > if it can. If not, no harm done. (It would be called as 'x = ipow(x, n, > > 10)' of course, just like 'x += n' is really 'x = x.__iadd__(n)') > > I think this would violate the policy that a mutating function shouldn't > give the object being modified as the result - just as list.reverse > doesn't return the list, in addition to reversing it in-place. > > Regards, > Martin > > -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/0587b53b/attachment.html From martin at v.loewis.de Wed Feb 14 20:02:42 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 14 Feb 2007 20:02:42 +0100 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <9e804ac0702141042x608a8e5ek2a9095be056210f9@mail.gmail.com> References: <45CAF232.7000505@hpi.uni-potsdam.de> <45CBAD70.1040201@canterbury.ac.nz> <45CC35F8.9040308@v.loewis.de> <45CD0D4E.7050308@canterbury.ac.nz> <45CDBA55.2000907@v.loewis.de> <9e804ac0702140730s280cad7mfa6c459ed9583af1@mail.gmail.com> <45D342DB.4010802@v.loewis.de> <9e804ac0702141042x608a8e5ek2a9095be056210f9@mail.gmail.com> Message-ID: <45D35CD2.3090908@v.loewis.de> Thomas Wouters schrieb: > > Sure, and I don't know if anyone will ever want ipow() -- but I've never > seen real code use the three-argument pow() either. The fact is that all > the in-place modifying hooks return the result (which may or may not be > self, and may or may not be mutated) so an in-place three-argument pow() > would have to do the same. I would prefer keeping the similarity between > __ipow__ and __pow__, although I don't care if that means keeping the > always-unused third argument to __ipow__ (which isn't really in the way, > after all) or adding a new hook for the three-argument pow(). It's in the way in the sense that slot_nb_inplace_power discards its third argument silently. That shouldn't happen, so I still prefer removing it throughout. Would you volunteer adding to fix that? Regards, Martin From jcarlson at uci.edu Wed Feb 14 20:16:28 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 14 Feb 2007 11:16:28 -0800 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <45D10E50.2080107@canterbury.ac.nz> References: <20070212122602.ACCF.JCARLSON@uci.edu> <45D10E50.2080107@canterbury.ac.nz> Message-ID: <20070214110505.AD27.JCARLSON@uci.edu> Greg Ewing wrote: > > Josiah Carlson wrote: > > def intern(st): > > ... > > > > If I remember the implementation of intern correctly, that's more or > > less what happens under the covers. > > That doesn't quite give you everything that real interning > does, though. The string comparison method knows when both > strings are interned, so it can compare them quickly > whether they are equal or not. Your version could detect > equal strings quickly, but not unequal strings. Assuming that dictionaries and the hash algorithm for strings is not hopelessly broken, I believe that one discovers quite quickly when two strings are not equal. Simple testing seems to confirm this, but I didn't work too hard to disprove it either. - Josiah From jcarlson at uci.edu Wed Feb 14 20:31:16 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 14 Feb 2007 11:31:16 -0800 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070214171005.GA6035@mithrandi.za.net> References: <952d92df0702140849w65189a0fnc8f52100888c412a@mail.gmail.com> <20070214171005.GA6035@mithrandi.za.net> Message-ID: <20070214112015.AD2A.JCARLSON@uci.edu> Tristan Seligmann wrote: > * Richard Tew [2007-02-14 16:49:03 +0000]: > > > I am not writing a competing event driven mechanism. What I was doing > > was feeling out whether there was any interest in better support for > > asynchronous calls. > > I interpreted your suggestions as being about enhancing asyncore with > IOCP (or similar); sure, asyncore is already a "competing" event driven > mechanism, but for all intents and purposes it is "dead" as there is not > much in the way of active development on it, and a relatively small > remaining user base. You seemed to be suggesting that asyncore be > revived by extending the implementation, as well as sprinkling some > generator / tasklets / channels / etc. sugar on top of it. It's not dead; much code is still being developed for it, and I recently stepped up as its new maintainer. One of the reasons it's not getting huge new feature additions is generally because it has all of the functionality necessary to get someone started in asynchronous socket programming. That's its purpose. Could it be expanded to support IOCP on Windows? Sure. Is it necessary for its survival? No. - Josiah From larry at hastings.org Wed Feb 14 20:27:59 2007 From: larry at hastings.org (Larry Hastings) Date: Wed, 14 Feb 2007 11:27:59 -0800 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <45D274B8.70505@canterbury.ac.nz> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> <45D10E50.2080107@canterbury.ac.nz> <45D196EC.3050303@v.loewis.de> <45D274B8.70505@canterbury.ac.nz> Message-ID: <45D362BF.5010203@hastings.org> Greg Ewing wrote: > Can anyone shed any light on this? It seems to me that > by not using this information, only half the benefit of > interning is being achieved. If I understand your question correctly, you're saying "why doesn't string comparison take advantage of interned strings?" If so, the answer is "it does". Examine string_richcompare() in stringobject.c, and PyUnicode_compare() in unicodeobject.c. Both functions compare the pointers of the two objects in case they are literally the same object, and if so skip all the expensive comparison code. Interned strings which have the same value are likely to be (are guaranteed to be?) the same object. And there you are. If that's not the question you are asking, I'll be quiet, 'cause it seems like other people understand what you're talking about. Cheers, /larry/ From martin at v.loewis.de Wed Feb 14 20:39:31 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 14 Feb 2007 20:39:31 +0100 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <45D362BF.5010203@hastings.org> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> <45D10E50.2080107@canterbury.ac.nz> <45D196EC.3050303@v.loewis.de> <45D274B8.70505@canterbury.ac.nz> <45D362BF.5010203@hastings.org> Message-ID: <45D36573.1070104@v.loewis.de> Larry Hastings schrieb: > If I understand your question correctly, you're saying "why doesn't > string comparison take advantage of interned strings?" If so, the > answer is "it does". Examine string_richcompare() in stringobject.c, > and PyUnicode_compare() in unicodeobject.c. Both functions compare the > pointers of the two objects in case they are literally the same object, > and if so skip all the expensive comparison code. Interned strings > which have the same value are likely to be (are guaranteed to be?) the > same object. And there you are. > > If that's not the question you are asking, I'll be quiet, 'cause it > seems like other people understand what you're talking about. The question was similar: "Why doesn't it take the maximum advantage?" Interned strings that are equal are guaranteed to be identical. So if strings are not identical and both interned, it could infer that they can't be equal, but currently doesn't infer so (instead, it compares length and contents in this case). Regards, Martin From glyph at divmod.com Wed Feb 14 21:22:33 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Wed, 14 Feb 2007 20:22:33 -0000 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax Message-ID: <20070214202233.28203.496054839.divmod.xquotient.1665@joule.divmod.com> On 01:04 pm, ben at redfrontdoor.org wrote: >glyph at divmod.com: >> I really, really wish that every feature proposal for Python had to meet >> some burden of proof [...]. I suspect this would kill 90% of "hey >> wouldn't this syntax be neat" proposals on day zero [...] > >This is what I understood the initial posting to python-ideas to be >about. If the feedback from there had been "that's a stupid idea", then >that would have been the end of it. I think it's a good thing that >there's the python-ideas mechanism for speculative suggestions. Discussion, with any audience, no matter how discerning, isn't really going to be a useful filter unless the standards of that audience are clear. What I was trying to say there is that the proposal of new ideas should not begin with "Hey, I think this might be 'good'" - that's too ill defined. It should be, "I noticed (myself/my users/my students/other open source projects) writing lots of code that looks like *this*, and I think it would be a real savings if it could look like *that*, instead". Some back-of-the-envelope math might be nice, in terms of savings of lines of code, or an explanation of the new functionality that might be enabled by a feature. More than the way proposals should work, I'm suggesting that the standards of the community in _evaluating_ to the proposals should be clearer. The cost-benefit analysis should be done in terms of programmer time, or ease of learning, or integration possibilities, or something. It doesn't *necessarily* have to be in terms of "lines of code saved", but especially for syntax proposals, that is probably a necessary start. It's fine to have some aesthetic considerations, but there's a lot more to Python than aesthetics, and right now it seems like the majority of posts coming into threads like this one are "I don't like the proposed ':==|', it's ugly. How about '?!?+' instead?" The short version of this is that any feature should describe what task it makes easier, for whom, and by how much. This description should be done up front, so that the discussion can center around whether than analysis is correct and whether it is worth the ever-present tradeoff against upgrade headaches. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/20e1528f/attachment.htm From glyph at divmod.com Wed Feb 14 21:34:21 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Wed, 14 Feb 2007 20:34:21 -0000 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) Message-ID: <20070214203421.28203.412373165.divmod.xquotient.1684@joule.divmod.com> On 03:32 pm, joachim.koenig-baltes at emesgarten.de wrote: >glyph at divmod.com wrote: >>When you boil it down, Twisted's event loop ... >But that is exactly the problem I have with Twisted. For HTTP ... ^^^^ >From that word on out, you have completely lost the plot. I am talking about Twisted's _event loop_. I specifically mentioned that the higher levels (and specifically "web") are separate, and can be discussed separately. If there were interest in porting the simpler XXXHTTPServer APIs in the stdlib to be event-driven and work with a Twisted-style event loop, it would be a relatively simple hack. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/0bb59daa/attachment.html From glyph at divmod.com Wed Feb 14 21:37:00 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Wed, 14 Feb 2007 20:37:00 -0000 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) Message-ID: <20070214203700.28203.1973851203.divmod.xquotient.1689@joule.divmod.com> On 02:22 pm, aahz at pythoncraft.com wrote: >On Wed, Feb 14, 2007, glyph at divmod.com wrote: >> >> As I said, I don't have time to write the PEPs myself, but I might fix >> some specific bugs if there were a clear set of issues preventing this >> from moving forward. Better integration with the standard library >> would definitely be a big win for both Twisted and Python. > >Here's where I'm coming from: > >My first experience with Twisted was excellent: I needed a web server in >fifteen minutes to do my PyCon presentation, and it Just Worked. > >My second experience with Twisted? Well, I didn't really have one. Thanks for the feedback, Aahz... I'm a little confused, though. Is this just a personal anecdote, or were you trying to make a larger point about Twisted's suitability for the stdlib? -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/96f20e8c/attachment.htm From theller at ctypes.org Wed Feb 14 21:52:29 2007 From: theller at ctypes.org (Thomas Heller) Date: Wed, 14 Feb 2007 21:52:29 +0100 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070214072621.28203.1662509932.divmod.xquotient.1571@joule.divmod.com> References: <20070214072621.28203.1662509932.divmod.xquotient.1571@joule.divmod.com> Message-ID: glyph at divmod.com schrieb: > On 02:20 am, greg.ewing at canterbury.ac.nz wrote: > >> If Twisted is designed so that it absolutely *has* to use its own >> special event mechanism, and everything else needs to be modified >> to suit its requirements, then it's part of the problem, not part >> of the solution. > > I've often heard this complaint, usually of the form "that's > twisted-specific". The thing is, Twisted isn't specific. Its event > mechanism isn't "special". In fact it's hard to imagine how it might > be made less "special" than it currently is. > > Whatever your event dispatch mechanism, *some* code has to be > responsible for calling the OS API of select(), > WaitForMultipleEvents(), g_main_loop_run(), or whatever. Twisted > actually imposes very few requirements for code to participate in > this, and was designed from day one specifically to be a generalized > mainloop mechanism which would not limit you to one underlying > multiplexing implementation, event-dispatch mechanism, or operating > system if you used its API. When I last looked at twisted (that is several years ago), there were several reactors - win32reactor, wxreactor, maybe even more. And they didn't even work too well. The problems I remember were that the win32reactor was limited to a only handful of handles, the wxreactor didn't process events when a wx modal dialog boy was displayed, and so on. Has this changed? Thanks, Thomas From glyph at divmod.com Wed Feb 14 22:58:41 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Wed, 14 Feb 2007 21:58:41 -0000 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib Message-ID: <20070214215841.28203.1815193832.divmod.xquotient.1838@joule.divmod.com> On 04:49 pm, richard.m.tew at gmail.com wrote: >On 2/13/07, glyph at divmod.com wrote: >>Tristan is correct: this should be a patch against Twisted, or perhaps as a >>separate library that could implement a reactor. > >I think there is some confusion here. Quite. >I am not writing a competing event driven mechanism. What I was doing >was feeling out whether there was any interest in better support for >asynchronous calls. Perhaps you could describe what you were proposing a bit better, because both to Tristan and myself, it sounded (and frankly, still sounds) like you were describing that would directly replace Twisted's mainloop core API. "asynchronous calls" is a very vague term with at least three completely separate definitions. >Yes I have looked at Twisted. It was the first place I looked, several >months ago, and what drew me to it was the IOCP reactor. However >as I also explained in my reply to Tristan it is not suitable for my >needs. As far as I can tell, you still haven't even clearly expressed what your needs are, let alone whether or not Twisted is suitable. In the reply you're citing, you said that "this" sounded like something "low level" that "twisted would be written on top of" - but the "this" you were talking about, based on your previous messages, sounded like monkeypatching the socket and asyncore modules to provide asynchronous file I/O based on the platform-specific IOCP API for Windows. Twisted can already provide that functionality in a _much_ cleaner fashion already, although you might need to edit some code or monkeypatch some objects specifically to get asynchronous file I/O, at least Twisted is *designed* for such applications. To quote you from the original message that Tristan replied to: >> I can't go into low level details because I do not have the necessary >> experience or knowledge to know which approach would be best. ... >> I of course have no concept of how this might be done on other platforms. As someone who _does_ have the necessary experience and knowledge, I can tell you that Twisted *IS* the "one stop shop" for events that you're describing. I do know how one would do such a thing on other platforms and it is not a simple task - so much so that async file I/O is still not available in Twisted today. >It is a large dependency and it is a secondary framework. Has it occurred to you that it is a "large" dependency not because we like making bloated and redundant code, but because it is doing something that is actually complex and involved? >And I did not feel the need to verify the implication that it wasn't >ready because this matched my own recollections. It meaning... Twisted? Twisted's IOCP support? Ready for... what? IOCPReactor definitely not up to the standard of much of the rest of the code in Twisted, but it's clearly ready for _something_ since BitTorrent uses it. >But I hope you realise that asserting that things should be within >Twisted without giving any reason, I have suggested that work proceed in Twisted rather than in Python because adding async file I/O to Twisted would be much easier than adding an entirely new event-loop core to Python, and then adding async file I/O to *that*. I thought that I provided several reasons before as well, but let me state them as clearly as I can here. Twisted is a large and mature framework with several years of development and an active user community. The pluggable event loop it exposes is solving a hard problem, and its implementation encodes a lot of knowledge about how such a thing might be done. It's also tested on a lot of different platforms. Writing all this from scratch - even a small subset of it - is a lot of work, and is unlikely to result in something robust enough for use by all Python programs without quite a lot of effort. It would be a lot easier to get the Twisted team involved in standardizing event-driven communications in Python. Every other effort I'm aware of is either much smaller, entirely proprietary, or both. Again, I would love to be corrected here, and shown some other large event-driven framework in Python that I was heretofore unaware of. Standardization is much easier to achieve when you have multiple interested parties with different interests to accommodate. As Yitzhak Rabin used to say, "you don't engage in API standardization with your friends, you engage in API standardization with your enemies" - or... something like that. >especially when the first person >saying it just stated that the matching work in Twisted wasn't even >ready, feels like Twisted is trying to push itself forward by bullying >people to work within it whenever something can be asserted as >laying within whatever domain it is which Twisted covers. You say that you weren't proposing an alternate implementation of an event loop core, so I may be reacting to something you didn't say at all. However, again, at least Tristan thought the same thing, so I'm not the only one. I think that *that* could be described as bullying. I -- and many others -- have a considerable investment in Twisted, and attempts to "standardize" on another, new main-loop within the standard library is an attempt to subvert that investment and pre-empt honest competition between different event-driven frameworks, by de-facto forcing all Python users (and Twisted itself) to learn and deal with what the standard library is doing. Ignoring the Twisted API while doing this adds insult to injury. There are some clear and very deep misunderstandings in this discussion about what the Twisted reactor is, how it works, and what it does, so I do really feel like it is being ignored. Not all of this is a direct reaction to things you've said, but to the continuing pattern of posts to this list which encourage development of new event-driven mechanisms or changes to asyncore to add features, while still ignoring Twisted and all the lessons learned from it. If we're talking about inclusion in the standard library, maybe Twisted's event loop *isn't* ready. Certainly all the _rest_ of Twisted isn't. I have admitted that already in this thread. That doesn't mean another from-scratch toy event loop, even one that Twisted could be implemented "on top of" _is_ ready. Stackless - I'd say that's certainly ready, although I understand the other concerns voiced in this thread. It's got a proven track-record and it, also, solves a difficult problem. Re-implementing microthreads for python from scratch, while ignoring stackless, specifically for inclusion in the standard library would also, presumably, be a bad idea. >Even if it doesn't suit their needs. If someone doesn't "have the necessary experience or knowledge to know which approach would be best", then how, exactly, do they know whether things suit their needs or not? You specifically claimed ignorance, and then said you were being "bullied" when someone tried to give you the information that you are lacking. >I don't think Tristan intended >that, but when followed up with your reply and JP's interesting >response to Greg, it feels like it. I thought JP's response to Greg was useful, since much of this discussion has been hopelessly vague. For example, your earlier claim that "it wasn't ready". I've tried to restrict my comments here to the possibility of a new event-driven core in the Python standard library. There are a lot of other features also being discussed here - a microthread scheduler API, for example - which I am _not_ advocating the use of Twisted for. Twisted could definitely make use of such a scheduler API if it existed in the stdlib. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/6756bc17/attachment.html From thomas at python.org Wed Feb 14 23:27:14 2007 From: thomas at python.org (Thomas Wouters) Date: Wed, 14 Feb 2007 14:27:14 -0800 Subject: [Python-Dev] Any value in setting up pybots for py3k? In-Reply-To: <45D199A4.6000603@v.loewis.de> References: <3f09d5a00702121250j73fef0f8k3baf114db6276b9f@mail.gmail.com> <45D199A4.6000603@v.loewis.de> Message-ID: <9e804ac0702141427q7c554b6ey14d4b506b2f1eb54@mail.gmail.com> In addition to the breakages because of Python 3.0 features and bugs, there currently isn't a way to deal with most of those breakages, except by forking your code in 2.x and 3.x versions. You can't write a non-bare except statement that works in 2.x and 3.x, for instance. I'd say it's still too early, but we're getting there -- most of the 3.0 features are in, and the 2.6-compatibility-mode is being worked on (slowly, but still.) When they're in, people could have a buildbot setup that tests the same source in 3.0-to-be and 2.6-to-be (and older released Python versions, if they really want to see a lot of errors.) such a setup would probably catch a lot of 3.0and 2.6 bugs, too ;-) On 2/13/07, "Martin v. L?wis" wrote: > > Grig Gheorghiu schrieb: > > So is there any value or interest in setting up a svn notification > > hook for py3k commits that would go to the pybots buildbot master? > > A similar question is whether there should be buildbots for Python 3 > itself (i.e. running the test suite). Even for that, it was considered > too early. > > I would expect that packages break in massive ways because of the > by-design incompatibilities. It would be good to get an estimate > of what the impact is, but having a build once a month might be > more than enough. To fully study the problems, one has to install > Python 3 locally, anyway, and then run the package in question on > top of that. > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/thomas%40python.org > -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/1b52b5ce/attachment.html From glyph at divmod.com Wed Feb 14 23:55:31 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Wed, 14 Feb 2007 22:55:31 -0000 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) Message-ID: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> On 08:52 pm, theller at ctypes.org wrote: >When I last looked at twisted (that is several years ago), there were >several reactors - win32reactor, wxreactor, maybe even more. Yes. That's intentional. Each of those systems offers its own event loop, and each reactor implements the basic operations in terms of those loops. They all have the same API. Application code does 'from twisted.internet import reactor; reactor.listenTCP', 'reactor.callLater', etc. Only the very top-most level decides which reactor the application will use. >And they >didn't even work too well. The problems I remember were that the win32reactor >was limited to a only handful of handles, the wxreactor didn't process >events when a wx modal dialog boy was displayed, and so on. Has this changed? win32eventreactor is limited to 64 handles because WaitForMultipleObjects is limited to 64 handles. wxreactor's event loop is ghastly and actually did pause when a modal dialog box is displayed (that has since been fixed). Process support on win32 now works in the default select reactor as well as the gtk reactor, so win32reactor is mostly a curiosity at this point (useful mainly if you want to implement your own GDI-based GUI, as PyUI did at one point), and its limitations are not as serious for Twisted as a whole. In other words, Twisted exposes the platform limitations in its platform-specific event loop implementations, and only works around them where it's possible to do so without masking platform functionality. For servers, the epoll, poll, and select reactors work just fine. The select reactor does have a maximum of FD_SETSIZE simultaneous sockets as well, but it is very easy to switch reactors if you need something more scalable. For clients, the best GUI toolkit for Twisted applications at this point is GTK, but WX, QT and Cocoa can all be made to work with a minimum of hassle. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/70e146d7/attachment.htm From greg.ewing at canterbury.ac.nz Thu Feb 15 00:18:33 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 12:18:33 +1300 Subject: [Python-Dev] [Python-3000] pre-PEP: Default Argument Expressions In-Reply-To: <20070213231036.AD24.JCARLSON@uci.edu> References: <45D28143.9010502@gmail.com> <20070213231036.AD24.JCARLSON@uci.edu> Message-ID: <45D398C9.7010202@canterbury.ac.nz> Another thing I don't like about this default argument proposal is that using it in any non-trivial way would tend to put implementation details of the function up in the header, which should be reserved for info describing the calling signature. The following example from the PEP is an extreme instance of this: > def bar(b=my_sum((["b"] * (2 * 3))[:4])): > print b -- Greg From greg.ewing at canterbury.ac.nz Thu Feb 15 01:29:07 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 13:29:07 +1300 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070214033403.25807.820994126.divmod.quotient.20851@ohm> References: <20070214033403.25807.820994126.divmod.quotient.20851@ohm> Message-ID: <45D3A953.4040900@canterbury.ac.nz> Jean-Paul Calderone wrote: > Greg, productive discussion is not furthered by the > unsupported statement of one position or another. Sorry, I'll expand on that a bit. The "problem" I was referring to is the turf wars between all the event-driven frameworks that all want to do things their own way. Solving that is going to require them to give up control and be willing to agree on a standard. It seems reasonable to consider making a suitably enhanced version of asyncore the standard, because it's small and it's already in the stdlib. Someone seemed to be claiming that it would be too hard to rework Twisted's event loop to use anything else, so we had better just use Twisted instead of asyncore, which sounded like a turf-war kind of statement. (That's what I meant by "part of the problem" -- I should really have said it's an *example* of the problem.) From a later post, it seems the main concern is that expanding asyncore to the point where it could support Twisted would involve reinventing most of Twisted's event loop implementation. That's a reasonable point to make, and I don't see anything wrong with examining Twisted's implementation to see what can be used. Whether to adopt Twisted's *API* is a separate issue, though (see next post). -- Greg From greg.ewing at canterbury.ac.nz Thu Feb 15 01:31:19 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 13:31:19 +1300 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> Message-ID: <45D3A9D7.5090103@canterbury.ac.nz> glyph at divmod.com wrote: > On 08:52 pm, theller at ctypes.org wrote: > > >When I last looked at twisted (that is several years ago), there were > >several reactors - win32reactor, wxreactor, maybe even more. > > Only the very top-most level decides which reactor the application will use. This is a worry, because it implies that there has to *be* a top level that knows what kind of reactor the whole application will use, and all parts of the application need to know that they will be getting their reactor from that top level. That may not be the case. For example, you want to incorporate some piece of event-driven code written by someone else into your gtk application. But it wasn't written with gtk in mind, so it doesn't know to use a gtkreactor, or how to get a reactor that it can use from your application. This is not my idea of what another poster called a "one-stop shop" -- a common API that different pieces of code can call independently without having to know about each other. To my mind, there shouldn't be a "reactor" object exposed to the application at all. There should just be functions for setting up callbacks. The choice of implementation should be made somewhere deep inside the library, based on what platform is being used. So at this point I'm skeptical that the Twisted API for these things should be adopted as-is. -- Greg From greg.ewing at canterbury.ac.nz Thu Feb 15 01:36:29 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 13:36:29 +1300 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070214215841.28203.1815193832.divmod.xquotient.1838@joule.divmod.com> References: <20070214215841.28203.1815193832.divmod.xquotient.1838@joule.divmod.com> Message-ID: <45D3AB0D.8090504@canterbury.ac.nz> glyph at divmod.com wrote: > sounded like > monkeypatching the socket and asyncore modules to provide asynchronous > file I/O based on the platform-specific IOCP API for Windows. I don't think anyone's suggesting that any long-term solution to all this would involve monkeypatching. -- Greg From steve at holdenweb.com Thu Feb 15 01:48:48 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 15 Feb 2007 00:48:48 +0000 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <20070214162758.GB25564@phd.pp.ru> References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> <20070214162758.GB25564@phd.pp.ru> Message-ID: Oleg Broytmann wrote: > On Wed, Feb 14, 2007 at 03:24:30PM +0000, Steve Holden wrote: >> Oleg Broytmann wrote: >>> On Tue, Feb 13, 2007 at 10:10:37AM +0000, Steve Holden wrote: >>>> Python further away from the "Computer Programming for Everyone" arena >>>> and closer to the "Systems Programming for Clever Individuals" camp. >>> That's because Python is being developed by "Clever Individuals" and not >>> by "Computer Programming for Everyone Committee". >>> >> I agree that the developers are Clever Individuals. So clever, in fact, >> that they realise Python needs to be as approachable as possible, not a >> language for them alone. > > Anyway I don't believe it's possible to create a CP4E language without a > "CP4E Steering Committee", and I think such committee it Python land is... > dormant at present... > Given that they say "a camel is a horse designed by a committee" I think that language design by committee is very unlikely to produce something well suited to the needs of unsophisticated users. That would require a single individual with good language design skills and the ability to veto features that were out of line with the design requirements. A lot like a BDFL, really. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From steve at holdenweb.com Thu Feb 15 01:59:48 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 15 Feb 2007 00:59:48 +0000 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3A9D7.5090103@canterbury.ac.nz> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > glyph at divmod.com wrote: >> On 08:52 pm, theller at ctypes.org wrote: >> >> >When I last looked at twisted (that is several years ago), there were >> >several reactors - win32reactor, wxreactor, maybe even more. >> >> Only the very top-most level decides which reactor the application will use. > > This is a worry, because it implies that there has to > *be* a top level that knows what kind of reactor the > whole application will use, and all parts of the > application need to know that they will be getting > their reactor from that top level. > > That may not be the case. For example, you want to > incorporate some piece of event-driven code written > by someone else into your gtk application. But it > wasn't written with gtk in mind, so it doesn't know > to use a gtkreactor, or how to get a reactor that it > can use from your application. > If the borrowed code takes a reactor parameter then presumably the top-level code can pass the appropriate reactor type in. If the borrowed code uses a fixed reactor then it's difficult to see how changes to the Twisted API could help. "Incorporating some piece of event-driven code written by someone else" implies specific assumptions about event types and delivery, surely. That's why it's difficult to port code between GUI toolkits, for example, and even more so to write code that runs on several toolkits without change. > This is not my idea of what another poster called a > "one-stop shop" -- a common API that different pieces > of code can call independently without having to know > about each other. > > To my mind, there shouldn't be a "reactor" object > exposed to the application at all. There should just > be functions for setting up callbacks. The choice of > implementation should be made somewhere deep inside > the library, based on what platform is being used. > You seem to be arguing for libraries that contain platform dependencies to handle multiple platforms. Glyph seems to prefer the ability for the library caller to pass in handlers for platform-dependent features. > So at this point I'm skeptical that the Twisted > API for these things should be adopted as-is. > Since Glyph has already stated his opinion that Twisted isn't yet ready for adoption as-is this doesn't add to the discussion. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From greg.ewing at canterbury.ac.nz Thu Feb 15 02:01:27 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 14:01:27 +1300 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <45D2D50C.6050000@v.loewis.de> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <45D2502E.9060608@canterbury.ac.nz> <45D273C2.9030905@canterbury.ac.nz> <45D2D50C.6050000@v.loewis.de> Message-ID: <45D3B0E7.6040708@canterbury.ac.nz> Martin v. L?wis wrote: > switch(o) { > case And: > Py_INCREF(And_singleton); > return And_singleton; > case Or: > Py_INCREF(Or_singleton); > return Or_singleton; Well, And_singleton and Or_singleton could be anything, including integers or interned strings. Creating a class and then a singleton instance for each one seems like overkill and unnecessary bloat to me. -- Greg From greg.ewing at canterbury.ac.nz Thu Feb 15 02:16:11 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 14:16:11 +1300 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: References: <1171458253.45d308ce00f09@imp.hosting365.ie> Message-ID: <45D3B45B.3080704@canterbury.ac.nz> Steve Holden wrote: > A further data point is that modern machines seem to give timing > variabilities due to CPU temperature variations even if you always eat > exactly the same thing. Oh, great. Now we're going to have to run our benchmarks in a temperature-controlled oven... -- Greg From greg.ewing at canterbury.ac.nz Thu Feb 15 02:20:37 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 14:20:37 +1300 Subject: [Python-Dev] Why is nb_inplace_power ternary? In-Reply-To: <45D342DB.4010802@v.loewis.de> References: <45CAF232.7000505@hpi.uni-potsdam.de> <45CBAD70.1040201@canterbury.ac.nz> <45CC35F8.9040308@v.loewis.de> <45CD0D4E.7050308@canterbury.ac.nz> <45CDBA55.2000907@v.loewis.de> <9e804ac0702140730s280cad7mfa6c459ed9583af1@mail.gmail.com> <45D342DB.4010802@v.loewis.de> Message-ID: <45D3B565.5000406@canterbury.ac.nz> Martin v. L?wis wrote: > I think this would violate the policy that a mutating function shouldn't > give the object being modified as the result Well, it's a necessary violation, given the way the inplace methods work. And it doesn't *necessarily* return the same value, it might return a new object. So the return value conveys useful information, unlike with list.sort() et al. -- Greg From guido at python.org Thu Feb 15 02:25:21 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Feb 2007 17:25:21 -0800 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: <20070214202233.28203.496054839.divmod.xquotient.1665@joule.divmod.com> References: <20070214202233.28203.496054839.divmod.xquotient.1665@joule.divmod.com> Message-ID: On 2/14/07, glyph at divmod.com wrote: > What I was trying to say there is that the proposal of new ideas should not > begin with "Hey, I think this might be 'good'" - that's too ill defined. It > should be, "I noticed (myself/my users/my students/other open source > projects) writing lots of code that looks like *this*, and I think it would > be a real savings if it could look like *that*, instead". Yup. > Some > back-of-the-envelope math might be nice, in terms of savings of lines of > code, or an explanation of the new functionality that might be enabled by a > feature. Well, savings in lines of code per se aren't high on my list; savings in conceptual difficulty understanding the code is. Sometimes savings in writing (if they don't go against the savings in understanding) are also important; especially avoiding distractions like having to stop working on the current algorithm in order to figure out how to best write some simple bookkeeping code, etc. (You seem to agree to some extent later, but I think it's worth mentioning my own standards here. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Thu Feb 15 02:35:49 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 14:35:49 +1300 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <20070214110505.AD27.JCARLSON@uci.edu> References: <20070212122602.ACCF.JCARLSON@uci.edu> <45D10E50.2080107@canterbury.ac.nz> <20070214110505.AD27.JCARLSON@uci.edu> Message-ID: <45D3B8F5.6070709@canterbury.ac.nz> Josiah Carlson wrote: > Assuming that dictionaries and the hash algorithm for strings is not > hopelessly broken, I believe that one discovers quite quickly when two > strings are not equal. You're probably right, since if there's a hash collision, the colliding strings probably differ in the first char or two. Interning will speed up the case where they are equal, where otherwise you would have to examine every character. Still, there could be extension modules out there somewhere that make use of PyString_CHECK_INTERNED, so there's a slight chance that home-grown interning might not give the same results in all cases. -- Greg From greg.ewing at canterbury.ac.nz Thu Feb 15 02:39:04 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 14:39:04 +1300 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <45D362BF.5010203@hastings.org> References: <1170859141.32307.32.camel@localhost> <20070212122602.ACCF.JCARLSON@uci.edu> <45D10E50.2080107@canterbury.ac.nz> <45D196EC.3050303@v.loewis.de> <45D274B8.70505@canterbury.ac.nz> <45D362BF.5010203@hastings.org> Message-ID: <45D3B9B8.9000100@canterbury.ac.nz> Larry Hastings wrote: > If I understand your question correctly, you're saying "why doesn't > string comparison take advantage of interned strings?" No, I understand that it takes advantage of it when the strings are equal. I was wondering about the case where they're not equal. But as has been pointed out, unless you're very unlucky, you find out pretty fast when they're not equal. So I'm happy now. -- Greg From pje at telecommunity.com Thu Feb 15 03:00:12 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 14 Feb 2007 21:00:12 -0500 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3A9D7.5090103@canterbury.ac.nz> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> Message-ID: <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> At 01:31 PM 2/15/2007 +1300, Greg Ewing wrote: >To my mind, there shouldn't be a "reactor" object >exposed to the application at all. There should just >be functions for setting up callbacks. The choice of >implementation should be made somewhere deep inside >the library, based on what platform is being used. *shudder*. I, on the other hand, prefer to assume that there is no one "top level" and certainly no requirement for a single event loop or reactor. peak.events, for example, lets you have multiple event loops running in the same or different threads. One of these can be Twisted's reactor, if you like, but the framework doesn't impose this singleton-ness on you. (IIUC, Twisted uses a singleton for at least two fairly good reasons: ease of programming, and the fact that certain platforms demand it for performance reasons. I just don't happen to agree that this limitation should be applied across *all* platforms. And there are good solutions for context-specific pseudo-singletons to address the API issue, although they might not be as high-performance as Twisted applications might prefer. Everything's a trade-off.) From thomas at python.org Thu Feb 15 02:59:42 2007 From: thomas at python.org (Thomas Wouters) Date: Wed, 14 Feb 2007 17:59:42 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3A9D7.5090103@canterbury.ac.nz> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> Message-ID: <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> On 2/14/07, Greg Ewing wrote: > > glyph at divmod.com wrote: > > On 08:52 pm, theller at ctypes.org wrote: > > > > >When I last looked at twisted (that is several years ago), there were > > >several reactors - win32reactor, wxreactor, maybe even more. > > > > Only the very top-most level decides which reactor the application will > use. > > This is a worry, because it implies that there has to > *be* a top level that knows what kind of reactor the > whole application will use, and all parts of the > application need to know that they will be getting > their reactor from that top level. > > That may not be the case. For example, you want to > incorporate some piece of event-driven code written > by someone else into your gtk application. But it > wasn't written with gtk in mind, so it doesn't know > to use a gtkreactor, or how to get a reactor that it > can use from your application. > > This is not my idea of what another poster called a > "one-stop shop" -- a common API that different pieces > of code can call independently without having to know > about each other. > > To my mind, there shouldn't be a "reactor" object > exposed to the application at all. There should just > be functions for setting up callbacks. The choice of > implementation should be made somewhere deep inside > the library, based on what platform is being used. Eh, your own example seems to argue the opposite. If the choice for reactor was made somewhere deep inside the library, how does it know to use the GTK reactor? Only the code using GTK knows that it's going to need the GTK reactor. The generic code just uses whatever reactor was selected, automatically. The reason the decision should be made at the topmost level is that this is the most visible and adaptive location. A library or framework has to handle a boundless number of variations of environment, use and co-operation with other libraries and frameworks, whereas the topmost script knows exactly what it wants to do. A library that uses PyGTK can say 'I need the GTK reactor', but it would be wrong. All it needs is a reactor that interacts nicely with GTK. If there are multiple reactors that integrate nicely with GTK, there is no reason for the library to insist on one particular implementation. And it could be in the way, if a second library needs a different kind of integration and it knows of a reactor that supports both that *and* GTK integration. And if the whole system breaks, for some reason, the first place the programmer will look is in his own code, not deep inside another library. For what it's worth, I think a single event-handling mainloop API in the standard library is a *great* idea. Using part of Twisted for it (suitably adapted) is an even greater idea: a whole lot of work and brainpower went into Twisted's core, and it wasn't wasted. Duplicating the effort would be a big mistake and a terrible waste. A from-scratch implementation of an event-handling mechanism will not solve any of the "issues" most people have with Twisted. It will have many more bugs than Twisted (all the ones Twisted had and fixed, sometimes very obscure and hard-to-find bugs), it will not be significantly less complex (Twisted is as complex as it is because it had to be, to do what it is supposed to do) and it will not require any less of a brain-warp to understand (because that's just what event-driven programming takes.) If I wasn't short on free time (spending most of it on Py3k and, soon, py3k-forward-compatibility) I would do all the necessary Twisted-integration work myself :-) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/837bdc7c/attachment.html From glyph at divmod.com Thu Feb 15 03:05:44 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 15 Feb 2007 02:05:44 -0000 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) Message-ID: <20070215020544.28203.50961406.divmod.xquotient.1935@joule.divmod.com> On 12:31 am, greg.ewing at canterbury.ac.nz wrote: >glyph at divmod.com wrote: >> On 08:52 pm, theller at ctypes.org wrote: >> >> >When I last looked at twisted (that is several years ago), there were >> >several reactors - win32reactor, wxreactor, maybe even more. >> >> Only the very top-most level decides which reactor the application will use. > >This is a worry, because it implies that there has to >*be* a top level that knows what kind of reactor the >whole application will use, and all parts of the >application need to know that they will be getting >their reactor from that top level. The default reactor is the most portable one, 'select', and if no other reactor is installed, that's the one that will be used. >That may not be the case. For example, you want to >incorporate some piece of event-driven code written >by someone else into your gtk application. But it >wasn't written with gtk in mind, so it doesn't know >to use a gtkreactor, or how to get a reactor that it >can use from your application. "from twisted.internet import reactor" is the way you get at the reactor, regardless of which one is currently installed. There can only be one reactor active at any given time, because at the very bottom of the event-handling machinery _some_ platform multiplexing API must be called, and that is mostly what the reactor represents. The GTK reactor does not have its own API. It simply allows you to use GTK APIs as well, by back-ending to the glib mainloop. That is, in fact, the whole point of having a "reactor" API in the first place. >This is not my idea of what another poster called a >"one-stop shop" -- a common API that different pieces >of code can call independently without having to know >about each other. >To my mind, there shouldn't be a "reactor" object >exposed to the application at all. There should just >be functions for setting up callbacks. That's what the Twisted "reactor" object *is*, exactly. Functions (well, methods) for setting up callbacks. >The choice of >implementation should be made somewhere deep inside >the library, based on what platform is being used. The "deep inside the library" decision is actually a policy decision made by a server's administrator, or dependent upon the GUI library being used if you need to interact with a GUI event loop. Perhaps the default selection could be better, but installing a reactor is literally one line of code, or a single command-line option to the "twistd" daemon. See: http://twistedmatrix.com/projects/core/documentation/howto/choosing-reactor.html It is completely transparent to the application, _unless_ the application wants to make use of platform-specific features. See the following for more information: http://www.cs.wustl.edu/~schmidt/PDF/reactor-siemens.pdf although technically Twisted's "reactor" is more like the slightly higher level POSA "proactor" pattern; asyncore is more like a true "reactor" in the sense discussed in that paper. Twisted exposes various APIs for "setting up callbacks" exactly as you describe: http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorTCP.html http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IReactorTime.html >So at this point I'm skeptical that the Twisted >API for these things should be adopted as-is Given that your supporting arguments were almost exactly the opposite of the truth in every case, I think this conclusion should be re-examined :). If you're interested in how a normal Twisted application looks, see this tutorial: http://twistedmatrix.com/projects/core/documentation/howto/servers.html -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070215/d8aa6049/attachment.htm From greg.ewing at canterbury.ac.nz Thu Feb 15 03:47:39 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 15:47:39 +1300 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> Message-ID: <45D3C9CB.5040203@canterbury.ac.nz> Steve Holden wrote: > If the borrowed code takes a reactor parameter then presumably the > top-level code can pass the appropriate reactor type in. Since there should only be one reactor at a time in any given application, it shouldn't have to be passed in -- it could be held in a global variable deep inside the library. Only the code which creates the reactor initially needs to know about that variable, or even that there is such a thing as a reactor. > "Incorporating some piece of event-driven code written by someone else" > implies specific assumptions about event types and delivery, surely. It requires agreement on how to specify the event types and what to do in response, but that's all it should require. The way I envisage it, setting up an event callback should be like opening a file -- there's only one way to do it, and you don't have to worry about what the rest of the application is doing. You don't have to get passed an object that knows how to open files -- it's a fundamental service provided by the system. You just use it. > That's why it's difficult to port code between GUI toolkits, for > example, and even more so to write code that runs on several toolkits > without change. Just in case it's not clear, the events I'm talking about are things like file and socket I/O, not GUI events. Trying to use two different GUIs at once is not something I'm addressing. Rather, you should be able to write code that does e.g. some async socket I/O, and embed it in a GUI app using e.g. gtk, without having to modify it to take account of the fact that it's working in a gtk environment, or having to parameterise it to allow for such things. > You seem to be arguing for libraries that contain platform dependencies > to handle multiple platforms. I'm arguing that as much of the platform dependency as possible should be in the asyncore library (or whatever replaces it). The main application code *might* have to give it a hint such as "this app uses gtk", but no more than that. And ideally, I'd prefer it not to even have to do that -- pygtk should do whatever is necessary to hook itself into asyncore if at all possible, not the other way around. > Since Glyph has already stated his opinion that Twisted isn't yet ready > for adoption as-is this doesn't add to the discussion. Okay, but one of the suggestions made seemed to be "why not just use the Twisted API". I'm putting forward a possible reason. -- Greg From greg.ewing at canterbury.ac.nz Thu Feb 15 04:18:40 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 16:18:40 +1300 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> Message-ID: <45D3D110.4020802@canterbury.ac.nz> Thomas Wouters wrote: > If the choice for > reactor was made somewhere deep inside the library, how does it know to > use the GTK reactor? In my ideal world, there wouldn't actually be a gtk reactor -- there would only be a Linux reactor, a MacOSX reactor, a Windows reactor, etc. Things like pygtk would be adapted to hook into the platform's standard reactor. Less ideal would be for pygtk to intall a gtk reactor when it gets imported. The danger with this approach is that two libraries could fight over which kind of reactor to use. You suggest that the top level could choose some other reactor that is compatible with both libraries. That seems like a rather hit-and-miss approach -- such a reactor might exist, or it might not. If not, you're out of luck. And the chance of finding a suitable reactor gets smaller as the number of libraries increases. > The reason the decision should be made at the topmost level is that this > is the most visible and adaptive location. This is where my vision is fundamentally different: you shouldn't have to *make* a decision in the first place. All event-driven libraries should be made to use the same substrate on any given platform. Then they can coexist without the need for any top-level choices. I know that will be hard to do, but it's the only way out of this mess that I can see. -- Greg From thomas at python.org Thu Feb 15 04:27:04 2007 From: thomas at python.org (Thomas Wouters) Date: Wed, 14 Feb 2007 19:27:04 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3D110.4020802@canterbury.ac.nz> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> Message-ID: <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> On 2/14/07, Greg Ewing wrote: > I know that will be hard to do, but it's the only > way out of this mess that I can see. That depends on what you consider messy about it. *I* don't like the idea of something in the Python installation deciding which reactor to use. It's my application, and I'm damn well going to tell it what to do. If that means it doesn't work as I expected, it's my own fault :-) In any case, your idea requires a lot of changes in external, non-Python code -- PyGTK simply exposes the GTK mainloop, which couldn't care less about Python's idea of a perfect event reactor model. While those issues are being settled, we'll have to cope with selecting the right reactor manually. It's not all that different from what you want, in any case. The PerfectReactor can be added later, all current reactors aliased to it, and no one would have to change a single line of code. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/994b34ae/attachment.htm From greg.ewing at canterbury.ac.nz Thu Feb 15 04:25:39 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 16:25:39 +1300 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> Message-ID: <45D3D2B3.9040806@canterbury.ac.nz> Phillip J. Eby wrote: > peak.events, for example, lets you have multiple event loops > running in the same or different threads. Different threads is okay if you're willing to use threads, but you might not. The reason you're using an event loop may well be precisely so that you *don't* have to use threads. And... how do you run multiple event loops simultaneously in the *same* thread? That sounds self-contradictory to me. -- Greg From thomas at python.org Thu Feb 15 04:32:21 2007 From: thomas at python.org (Thomas Wouters) Date: Wed, 14 Feb 2007 19:32:21 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3D2B3.9040806@canterbury.ac.nz> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> <45D3D2B3.9040806@canterbury.ac.nz> Message-ID: <9e804ac0702141932t406bb8b8j3baaa8488c2b9d76@mail.gmail.com> On 2/14/07, Greg Ewing wrote: > > Phillip J. Eby wrote: > > peak.events, for example, lets you have multiple event loops > > running in the same or different threads. > > Different threads is okay if you're willing to use threads, > but you might not. The reason you're using an event loop > may well be precisely so that you *don't* have to use > threads. > > And... how do you run multiple event loops simultaneously > in the *same* thread? That sounds self-contradictory to > me. If all (or all-but-one) of them have a 'run one iteration' method, you can call that from the 'main' mainloop. Or you can design all mainloops as coroutines and have them call each other. (I haven't looked at Phillip's approach at all, but something tells me coroutines are involved :-) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070214/c32837f1/attachment-0001.html From exarkun at divmod.com Thu Feb 15 04:35:54 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Wed, 14 Feb 2007 22:35:54 -0500 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3C9CB.5040203@canterbury.ac.nz> Message-ID: <20070215033554.25807.1945461889.divmod.quotient.21940@ohm> On Thu, 15 Feb 2007 15:47:39 +1300, Greg Ewing wrote: >Steve Holden wrote: > >> If the borrowed code takes a reactor parameter then presumably the >> top-level code can pass the appropriate reactor type in. > >Since there should only be one reactor at a time in >any given application, it shouldn't have to be passed >in -- it could be held in a global variable deep >inside the library. Only the code which creates the >reactor initially needs to know about that variable, >or even that there is such a thing as a reactor. Whether or not the premise here is accurate may be out of scope for this thread. Or it may not be. I dunno. However, I do want to point out that it is not necessarily correct that there should be only one reactor at a time in a given application. PJE has already explained that peak.events can have multiple reactors. Twisted is tied to one, but this may not always be the case. Whether there is a default reactor for applications that don't care about the ability to have more than one at a time is yet another question which may be worth examining. These are the kinds of things which should be spelled out in a PEP, including the rationale for any particular policy decisions (which should be kept to an absolute minimum) are made. > >> "Incorporating some piece of event-driven code written by someone else" >> implies specific assumptions about event types and delivery, surely. > >It requires agreement on how to specify the event types >and what to do in response, but that's all it should >require. > >The way I envisage it, setting up an event callback >should be like opening a file -- there's only one way >to do it, and you don't have to worry about what the >rest of the application is doing. You don't have to >get passed an object that knows how to open files -- >it's a fundamental service provided by the system. >You just use it. If we suppose that files and sockets are supported in roughly the same way, and we suppose that sockets are supported in the way that Twisted supports them, then there is no difficulty supporting files in this way. :) > >> That's why it's difficult to port code between GUI toolkits, for >> example, and even more so to write code that runs on several toolkits >> without change. > >Just in case it's not clear, the events I'm talking >about are things like file and socket I/O, not GUI >events. Trying to use two different GUIs at once is >not something I'm addressing. Alright, good. Getting two different GUI libraries to play together is a pretty hairy task indeed, and well worth keeping separate from this one. :) > >Rather, you should be able to write code that does >e.g. some async socket I/O, and embed it in a GUI >app using e.g. gtk, without having to modify it to >take account of the fact that it's working in a >gtk environment, or having to parameterise it to >allow for such things. Excellent. To be clear, this is how the Twisted model works, with respect to integration with GUI toolkits. I would not enjoy working with a system in which this was not the case. > >> You seem to be arguing for libraries that contain platform dependencies >> to handle multiple platforms. > >I'm arguing that as much of the platform dependency >as possible should be in the asyncore library (or >whatever replaces it). Certainly. Library code doesn't care if the event loop is driven by select or poll or epoll or /dev/poll or kqueue or aio or iocp or win32 events or realtime signals or kaio or whatever gnarly thing is hidden in gtk or whatever gnarly thing is hidden inside qt or whatever gnarly thing is hidden inside COM or whatever gnarly thing is hidden inside wxWidgets. It cares about what features are available. It requests them somehow, and uses them. If they are unavailable, then it can decide whether the lack is catastrophic and give up or if it can be worked around somehow. The way a Twisted application does this is based on interfaces. Assuming interfaces continue to not be present in the stdlib, a stdlib event loop would have to find some other API for presenting this information, but it is not a very hard problem to solve. >The main application code >*might* have to give it a hint such as "this app >uses gtk", but no more than that. And ideally, I'd >prefer it not to even have to do that -- pygtk >should do whatever is necessary to hook itself >into asyncore if at all possible, not the other >way around. There is some advantage to declaring things up front, lest you get into the situation where you are partway through using code which will suddenly begin to demand Gtk at the same time as you are partway through using code which will suddenly begin to demand Qt, at which point you are in trouble. But this is another minor point. > >> Since Glyph has already stated his opinion that Twisted isn't yet ready >> for adoption as-is this doesn't add to the discussion. > >Okay, but one of the suggestions made seemed to be >"why not just use the Twisted API". I'm putting >forward a possible reason. So far, it sounds like the objections to using the Twisted API are primarily based on misunderstandings of it, and the actual desired API looks pretty much the same as Twisted's. Jean-Paul From exarkun at divmod.com Thu Feb 15 04:42:08 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Wed, 14 Feb 2007 22:42:08 -0500 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3D110.4020802@canterbury.ac.nz> Message-ID: <20070215034208.25807.98758307.divmod.quotient.21948@ohm> On Thu, 15 Feb 2007 16:18:40 +1300, Greg Ewing wrote: > [snip] > >This is where my vision is fundamentally different: >you shouldn't have to *make* a decision in the first >place. All event-driven libraries should be made to >use the same substrate on any given platform. Then >they can coexist without the need for any top-level >choices. > >I know that will be hard to do, but it's the only >way out of this mess that I can see. > Thomas already pointed this out, but I'm repeating it anyway. This vision represents an impossible reality at present. You will not get Gtk or Qt or wxWidgets to use Python's event notification API. If you are really very interested in solving this problem, go to the developers of each platform those toolkits run on and sell them on a unified event notification API. Once they have adopted, implemented, and deployed it, you can go to the Gtk, Qt, and wxWidgets teams and tell them to port all of their code to that new API. Then, you can have a unified model in Python. Until then, the practical compromise with almost zero negative consequences (sometimes, one extra piece of configuration will be required - compare this to how the logging module works ;) is to optionally allow explicit reactor selection. Jean-Paul From jcarlson at uci.edu Thu Feb 15 05:18:57 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 14 Feb 2007 20:18:57 -0800 Subject: [Python-Dev] Interning string subtype instances In-Reply-To: <45D3B8F5.6070709@canterbury.ac.nz> References: <20070214110505.AD27.JCARLSON@uci.edu> <45D3B8F5.6070709@canterbury.ac.nz> Message-ID: <20070214201455.AD45.JCARLSON@uci.edu> Greg Ewing wrote: > > Josiah Carlson wrote: > > > Assuming that dictionaries and the hash algorithm for strings is not > > hopelessly broken, I believe that one discovers quite quickly when two > > strings are not equal. > > You're probably right, since if there's a hash collision, > the colliding strings probably differ in the first char > or two. > > Interning will speed up the case where they are equal, > where otherwise you would have to examine every character. Except that the scanning of the data needs to happen at some point anyways. You first allocate the new string, copy the data into it, compare the content against something in the interned dictionary, and if there is a match, deallocate the new object and return the interned object, otherwise return the new object. This is also the case when one uses the "just give me a string, I'll fill in the data myself" for more or less user-implemented things like ''.join(lst). > Still, there could be extension modules out there > somewhere that make use of PyString_CHECK_INTERNED, > so there's a slight chance that home-grown interning > might not give the same results in all cases. Perhaps. I just never intern. I've found that I'm much happier assuming that the Python runtime will do good enough. - Josiah From pje at telecommunity.com Thu Feb 15 05:21:08 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 14 Feb 2007 23:21:08 -0500 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3D2B3.9040806@canterbury.ac.nz> References: <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070214231553.02824e50@sparrow.telecommunity.com> At 04:25 PM 2/15/2007 +1300, Greg Ewing wrote: >Phillip J. Eby wrote: >>peak.events, for example, lets you have multiple event loops running in >>the same or different threads. > >Different threads is okay if you're willing to use threads, >but you might not. The reason you're using an event loop >may well be precisely so that you *don't* have to use >threads. > >And... how do you run multiple event loops simultaneously >in the *same* thread? When one is nested inside the other. This isn't a common need, but it's occasionally useful if you need to switch back and forth between blocking and non-blocking code. For example, suppose that you have some code that wants to offer a synchronous interface to an asynchronous library... and the synchronous code is being called from a FastCGI "accept" event loop. The inner code can't use the outer event loop, because the outer loop isn't going to proceed until the inner code is finished. From collinw at gmail.com Thu Feb 15 05:40:19 2007 From: collinw at gmail.com (Collin Winter) Date: Wed, 14 Feb 2007 22:40:19 -0600 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <45D2D447.4080408@v.loewis.de> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> <45D2D447.4080408@v.loewis.de> Message-ID: <43aa6ff70702142040t19837debu511e215f998ec70e@mail.gmail.com> On 2/14/07, "Martin v. L?wis" wrote: > Collin Winter schrieb: > > 2) It turned out that {BinOp, BoolOp,AugAssign,etc}.op were already > > singleton instances of their respective classes. I've changed > > asdl_c.py to no longer emit the *_singleton names and to use the > > corresponding *_type types in their place, which will enable me to > > write "node.op is _ast.Add", etc. > > I don't really like this - it's inconsistent. I'd rather prefer > if the singletons where exposed under a name, e.g. > _ast.Add.singleton, or _ast.add (if that doesn't cause conflicts). What's inconsistent about it? That classes are being used for the _ast.{Add,Sub,Mult,etc} names? I don't see the need for both _ast.Add and _ast.Add.singleton or _ast.add or however else it might be spelled. I'd be perfectly happy doing something like "_ast.Add = object()" (when initializing the _ast module), so long as I can write "node.op is _ast.Add", "node.op == _ast.Add", or something equally brief and to-the-point. Collin Winter From dustin at v.igoro.us Thu Feb 15 06:00:31 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Wed, 14 Feb 2007 23:00:31 -0600 Subject: [Python-Dev] microthreading vs. async io Message-ID: <20070215050031.GH6786@v.igoro.us> I've steered clear of this conversation for a while, because it drifted a little bit off my original topic. But I did want to straighten a little bit of terminology out, if only to resolve some of my own confusion over all the hubbub. I don't pretend to define the words others use; these definitions are mine, and apply to what I write below. cooperative multitasking: Dynamically executing multiple tasks in a single thread; comes in two varieties: continuations: Breaking tasks into discrete "chunks", and passing references those chunks around as a means of scheduling. microtreading: Exploiting language features to use cooperative multitasking in tasks that "read" like they are single-threaded. asynchronous IO: Performing IO to/from an application in such a way that the application does not wait for any IO operations to complete, but rather polls for or is notified of the readiness of any IO operations. Twisted is, by the above definitions, a continuation-based cooperative multitasking library that includes extensive support for asynchronous IO, all the way up the network stack for an impressive array of protocols. It does not itself implement microthreading, but Phillip provided a nice implementation of such on top of Twisted[1]. Asyncore *only* implements asynchronous IO -- any "tasks" performed in its context are the direct result of an IO operation, so it's hard to say it implements cooperative multitasking (and Josiah can correct me if I'm wrong, but I don't think it intends to). Much of the discussion here has been about creating a single, unified asynchronous IO mechanism that would support *any* kind of cooperative multitasking library. I have opinions on this ($0.02 each, bulk discounts available), but I'll keep them to myself for now. Instead, I would like to concentrate on producing a small, clean, consistent, generator-based microthreading library. I've seen several such libraries (including the one in PEP 342, which is fairly skeletal), and they all work *almost* the same way, but differ in, for example, the kinds of values that can be yielded, their handling of nested calls, and the names for the various "special" values one can yield. That similar mouldes are being written repeatedly, and presumably applications and frameworks are being built on top of those modules, seems to me to suggest a new "standard" implementation should be added to the standard library. I realize that I'm all talk and no code -- I've been busy, but I hope to rectify the imbalance tonight. Dustin [1] http://mail.python.org/pipermail/python-dev/2007-February/071076.html From greg.ewing at canterbury.ac.nz Thu Feb 15 06:00:59 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 18:00:59 +1300 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <9e804ac0702141932t406bb8b8j3baaa8488c2b9d76@mail.gmail.com> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> <45D3D2B3.9040806@canterbury.ac.nz> <9e804ac0702141932t406bb8b8j3baaa8488c2b9d76@mail.gmail.com> Message-ID: <45D3E90B.6090707@canterbury.ac.nz> Thomas Wouters wrote: > If all (or all-but-one) of them have a 'run one iteration' method, you > can call that from the 'main' mainloop. But without some way of blocking until an event arrives for *either* loop, you have to resort to some kind of busy polling, which is not elegant. -- Greg From greg.ewing at canterbury.ac.nz Thu Feb 15 06:25:35 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Feb 2007 18:25:35 +1300 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> Message-ID: <45D3EECF.7080600@canterbury.ac.nz> Thomas Wouters wrote: > *I* don't like the idea of something in the Python installation > deciding which reactor to use. I wouldn't mind if some way were provided of changing the reactor if you want. I'd just like to see a long term goal of making it unnecessary as far as possible. > In any case, your idea requires a lot of changes in external, non-Python > code -- PyGTK simply exposes the GTK mainloop, which couldn't care less > about Python's idea of a perfect event reactor model. On unix at least, I don't think it should be necessary to change gtk, only pygtk. If it can find out the file descriptor of the connection to the X server, it can plug that into the reactor, and then call gtk_main_iteration_do() whenever something comes in on it. A similar strategy ought to work for any X11-based toolkit that exposes a function to perform one iteration of its main loop. Mileage on other platforms may vary. > The PerfectReactor can be added later, all current reactors > aliased to it, and no one would have to change a single line > of code. Sure. The other side to all this is the client side, i.e. the code that installs event callbacks. At the moment there's no clear direction to take, so everyone makes their own choice -- some use asyncore, some use Twisted, some use the gtk event loop, some roll their own, etc. If it were made known that asyncore or some other thing in the stdlib was intended to become the standard, then it would give people some guidance as to how to write future event-driven code. -- Greg From djc at object-craft.com.au Thu Feb 15 06:20:27 2007 From: djc at object-craft.com.au (Dave Cole) Date: Thu, 15 Feb 2007 16:20:27 +1100 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <5.1.1.6.0.20070214231553.02824e50@sparrow.telecommunity.com> References: <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> <5.1.1.6.0.20070214231553.02824e50@sparrow.telecommunity.com> Message-ID: <45D3ED9B.5040603@object-craft.com.au> Phillip J. Eby wrote: > At 04:25 PM 2/15/2007 +1300, Greg Ewing wrote: >> Phillip J. Eby wrote: >>> peak.events, for example, lets you have multiple event loops running in >>> the same or different threads. >> Different threads is okay if you're willing to use threads, >> but you might not. The reason you're using an event loop >> may well be precisely so that you *don't* have to use >> threads. >> >> And... how do you run multiple event loops simultaneously >> in the *same* thread? > > When one is nested inside the other. This isn't a common need, but it's > occasionally useful if you need to switch back and forth between blocking > and non-blocking code. For example, suppose that you have some code that > wants to offer a synchronous interface to an asynchronous library... and > the synchronous code is being called from a FastCGI "accept" event > loop. The inner code can't use the outer event loop, because the outer > loop isn't going to proceed until the inner code is finished. I actually have some code that works a little bit like this. I have a SelectLoop class that the application knows about and explicitly attaches transport objects to, and queues events on the loop. I have two ways to perform some actions in a "blocking" way. 1) I can create a new SelectLoop and move some transports temporarily from the main loop to the temporary loop to ignore all other events for a while. 2) I can re-enter the SelectLoop run() method and keep reacting to all events. Once the right event occurs - I exit from the nested run() and continue on. This does require a lot more mental effort though. Note that my code is nowhere near as ambitious as Twisted, but I watned to have the above flexibility. The code really only has to work on AIX and Linux, and in a limited way, Windows. - Dave From rhamph at gmail.com Thu Feb 15 07:01:54 2007 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 14 Feb 2007 23:01:54 -0700 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <5.1.1.6.0.20070214231553.02824e50@sparrow.telecommunity.com> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> <45D3D2B3.9040806@canterbury.ac.nz> <5.1.1.6.0.20070214231553.02824e50@sparrow.telecommunity.com> Message-ID: On 2/14/07, Phillip J. Eby wrote: > When one is nested inside the other. This isn't a common need, but it's > occasionally useful if you need to switch back and forth between blocking > and non-blocking code. For example, suppose that you have some code that > wants to offer a synchronous interface to an asynchronous library... and > the synchronous code is being called from a FastCGI "accept" event > loop. The inner code can't use the outer event loop, because the outer > loop isn't going to proceed until the inner code is finished. This would also let you wrap sys.stdout.write() in a nested event loop so as to allow print statements to still work while you use have it set to non-blocking mode, but I could see it argued that using print statements at all is wrong at that point. -- Adam Olsen, aka Rhamphoryncus From martin at v.loewis.de Thu Feb 15 07:58:51 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Feb 2007 07:58:51 +0100 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <43aa6ff70702142040t19837debu511e215f998ec70e@mail.gmail.com> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> <45D2D447.4080408@v.loewis.de> <43aa6ff70702142040t19837debu511e215f998ec70e@mail.gmail.com> Message-ID: <45D404AB.9080808@v.loewis.de> Collin Winter schrieb: > What's inconsistent about it? That classes are being used for the > _ast.{Add,Sub,Mult,etc} names? Exactly. These aren't names - they are nodes in the tree. All nodes are instances of _ast.AST. > I don't see the need for both _ast.Add and _ast.Add.singleton or > _ast.add or however else it might be spelled. I'd be perfectly happy > doing something like "_ast.Add = object()" (when initializing the _ast > module), so long as I can write "node.op is _ast.Add", "node.op == > _ast.Add", or something equally brief and to-the-point. Would you like to do the same for Pass, Break, Continue, and Ellipsis? They are also "just names". If you make _ast.Add the entry in the tree itself, why not _ast.Break? Or, if you have a way to deal with _ast.Break, why can't the same way work for _ast.Add? Regards, Martin From martin at v.loewis.de Thu Feb 15 08:12:56 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 15 Feb 2007 08:12:56 +0100 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <9e804ac0702141932t406bb8b8j3baaa8488c2b9d76@mail.gmail.com> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <5.1.1.6.0.20070214205647.056e0eb0@sparrow.telecommunity.com> <45D3D2B3.9040806@canterbury.ac.nz> <9e804ac0702141932t406bb8b8j3baaa8488c2b9d76@mail.gmail.com> Message-ID: <45D407F8.4080303@v.loewis.de> Thomas Wouters schrieb: > If all (or all-but-one) of them have a 'run one iteration' method, you > can call that from the 'main' mainloop. That doesn't really work (and neither do variations involving coroutines. Either the 'run one iteration' method blocks until one even arrives, in which case it may block for a long time while other loops have events pending (which then won't get processed). Or each 'run one iteration' method has a polling mode (e.g. with a time-out), in which case you get busy wait (you can fiddle with the time-out value to trade business vs. responsiveness). The 'run one iteration' approach doesn't support writing applications that are single-threaded, responsive, and idle when there is nothing to do. I can't believe that coroutines help in any way here. Regards, Martin From martin at v.loewis.de Thu Feb 15 08:22:29 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Feb 2007 08:22:29 +0100 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3EECF.7080600@canterbury.ac.nz> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> Message-ID: <45D40A35.2020502@v.loewis.de> Greg Ewing schrieb: > On unix at least, I don't think it should be necessary > to change gtk, only pygtk. If it can find out the file > descriptor of the connection to the X server, it can > plug that into the reactor, and then call > gtk_main_iteration_do() whenever something comes in > on it. That is insufficient. The gtk main loop has more input sources than just the connection to X: - timers can be registered, which are called when the time comes - idle handlers can be registered which are called when there are no other events - child handlers are invoked when a child process terminates - additional file descriptors can be registered (probably used for sockets primarily) - a generalzed 'event source' can be hooked into it, with C functions for prepare, check, dispatch, and finalize See http://www.gtk.org/api/2.6/glib/glib-The-Main-Event-Loop.html Regards, Martin From martin at v.loewis.de Thu Feb 15 08:39:59 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Feb 2007 08:39:59 +0100 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <20070215050031.GH6786@v.igoro.us> References: <20070215050031.GH6786@v.igoro.us> Message-ID: <45D40E4F.9020002@v.loewis.de> dustin at v.igoro.us schrieb: > Asyncore *only* implements asynchronous IO -- any "tasks" performed in > its context are the direct result of an IO operation, so it's hard to > say it implements cooperative multitasking (and Josiah can correct me if > I'm wrong, but I don't think it intends to). I'm trying to correct you: By your definition, asyncore implements cooperative multi-tasking. You didn't define 'task', but I understand it as 'separate activity'. With asyncore, you can, for example, run a web server and an IRC server in a single thread, as separate tasks, and asyncore deals with the scheduling between these tasks. In your terminology, it is based on continuations: the chunk you specify is the event handler. Indeed, asyncore's doc string starts with # There are only two ways to have a program on a single # processor do "more than one thing at a time". and goes on suggesting that asyncore provides one of them. Regards, Martin From jcarlson at uci.edu Thu Feb 15 09:06:30 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 15 Feb 2007 00:06:30 -0800 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <45D40E4F.9020002@v.loewis.de> References: <20070215050031.GH6786@v.igoro.us> <45D40E4F.9020002@v.loewis.de> Message-ID: <20070215000154.AD4B.JCARLSON@uci.edu> "Martin v. L?wis" wrote: > > dustin at v.igoro.us schrieb: > > Asyncore *only* implements asynchronous IO -- any "tasks" performed in > > its context are the direct result of an IO operation, so it's hard to > > say it implements cooperative multitasking (and Josiah can correct me if > > I'm wrong, but I don't think it intends to). > > I'm trying to correct you: By your definition, asyncore implements > cooperative multi-tasking. You didn't define 'task', but I understand > it as 'separate activity'. With asyncore, you can, for example, run > a web server and an IRC server in a single thread, as separate tasks, > and asyncore deals with the scheduling between these tasks. > In your terminology, it is based on continuations: the chunk you specify > is the event handler. > > Indeed, asyncore's doc string starts with > > # There are only two ways to have a program on a single > # processor do "more than one thing at a time". > > and goes on suggesting that asyncore provides one of them. Well, when asyncore was written, it probably didn't have coroutines, generator trampolines, etc., what we would consider today, in this particular context, "cooperative multithreading". What that documentation /should/ have said is... # There are at least two ways to have a program on a single # processor do "more than one thing at a time". Then go on to describe threads, a 'polling for events' approach like asyncore, then left the rest for someone else to add later. I'll add it as my first patch to asyncore. - Josiah From phd at phd.pp.ru Thu Feb 15 10:31:32 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Thu, 15 Feb 2007 12:31:32 +0300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> <20070214162758.GB25564@phd.pp.ru> Message-ID: <20070215093132.GA28863@phd.pp.ru> On Thu, Feb 15, 2007 at 12:48:48AM +0000, Steve Holden wrote: > Given that they say "a camel is a horse designed by a committee" Metaphors can go that far but not farther. And, BTW, camels are very suited for their environments... I am not afraid of committees for large tasks. Well, that has to be a small committee ruling by a cleverest ruler. > require a > single individual with good language design skills and the ability to > veto features that were out of line with the design requirements. A lot > like a BDFL, really. Of course, but I don't know if "CP4E" idea is still on his agenda and with what priority. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From dalke at dalkescientific.com Thu Feb 15 10:36:22 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Thu, 15 Feb 2007 02:36:22 -0700 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070214072621.28203.1662509932.divmod.xquotient.1571@joule.divmod.com> References: <20070214072621.28203.1662509932.divmod.xquotient.1571@joule.divmod.com> Message-ID: I was the one on the Stackless list who last September or so proposed the idea of monkeypatching and I'm including that idea in my presentation for PyCon. See my early rough draft at http://www.stackless.com/pipermail/stackless/2007-February/002212.html which contains many details about using Stackless, though none on the Stackless implementation. (A lot on how to tie things together.) So people know, I am an applications programmer and not a systems programmer. Things like OS-specific event mechanisms annoy and frustrate me. If I could do away with hardware and still write useful programs I would. I have tried 3 times to learn Twisted. The first time I found and reported various problems and successes. See emails at http://www.twistedmatrix.com/pipermail/twisted-python/2003-June/thread.html The second time was to investigate a way to report upload progress: http://twistedmatrix.com/trac/ticket/288 and the third was to compare Allegra and Twisted http://www.dalkescientific.com/writings/diary/archive/2006/08/28/levels_of_abstraction.html In all three cases I've found it hard to use Twisted because the code didn't do as I expected it to do and when something went wrong I got results which were hard to interpret. I believe others have similar problems and is one reason Twisted is considered to be "a big, complicated, inseparable hairy mess." I find the Stackless code also hard to understand. Eg, I don't know where the watchdog code is for the "run()" command. It uses several layers of macros and I haven't been able get it straight in my head. However, so far I've not run into strange errors in Stackless that I have in Twisted. I find the normal Python code relatively easy to understand. Stackless only provides threadlets. It does no I/O. Richard Tew developed a "stacklesssocket" module which emulates the API for the stdlib "socket" module. I tweaked it a bit and showed that by doing the monkeypatch import stacklesssocket import sys sys.modules["socket"] = stacklesssocket then code like "urllib.urlopen" became Stackless compatible. Eg, in my PyCon talk draft I show something like import slib # must monkeypatch before any other module imports "socket" slib.use_monkeypatch() import urllib2 import time import hashlib def fetch_and_reverse(host): t1 = time.time() s = urllib2.urlopen("http://"+host+"/").read()[::-1] dt = time.time() - t1 digest = hashlib.md5(s).hexdigest() print "hash of %r/ = %s in %.2f s" % (host, digest, dt) slib.main_tasklet(fetch_and_reverse)("www.python.org") slib.main_tasklet(fetch_and_reverse)("docs.python.org") slib.main_tasklet(fetch_and_reverse)("planet.python.org") slib.run_all() where the three fetches occur in parallel. The choice of asyncore is, I think, done because 1) it prevents needing an external dependency, 2) asyncore is smaller and easier to understand than Twisted, and 3) it was for demo/proof of concept purposes. While tempting to improve that module I know that Twisted has already gone though all the platform-specific crap and I don't want to go through it again myself. I don't want to write a reactor to deal with GTK, and one for OS X, and one for ... Another reason I think Twisted is considered "tangled-up Deep Magic, only for Wizards Of The Highest Order" is because it's infused with event-based processing. I've done a lot of SAX processing and I can say that few people think that way or want to go through the process of learning how. Compare, for example, the following f = urllib2.urlopen("http://example.com/") for i, line in enumerate(f): print ("%06d" % i), repr(line) with the normal equivalent in Twisted or other async-based system. Yet by using the Stackless socket monkeypatch, this same code works in an async framework. And the underlying libraries have a much larger developer base than Twisted. Want NNTP? "import nntplib" Want POP3? "import poplib" Plenty of documentation about them too. On the Stackless mailing list I have proposed someone work on a talk for EuroPython titled "Stackless and Twisted". Andrew Francis has been looking into how to do that. All the earlier quotes were lifted from glyph. Here's another: > When you boil it down, Twisted's event loop is just a > notification for "a connection was made", "some data was > received on a connection", "a connection was closed", and > a few APIs to listen or initiate different kinds of > connections, start timed calls, and communicate with threads. > All of the platform details of how data is delivered to the > connections are abstracted away.. How do you propose we > would make a less "specific" event mechanism? What would I need to do to extract this Twisted core so I could replace asyncore? I know at minimum I need "twisted.internet" and "twisted.python" (the latter for logging) and "twisted.persisted" for "styles.Ephemeral". But I say this hesitantly recalling the frustrations I had in dealing with a connection error in Twisted, described in the aforementioned link http://www.dalkescientific.com/writings/diary/archive/2006/08/28/levels_of_abstraction.html I feel that using the phrase "just a" in the previously quoted text is an understatement. While the mechanics might be simple, there are many, many layers, as you can see in this stack trace. File "async_blast.py", line 55, in ? reactor.run() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/posixbase.py", line 218, in run self.mainLoop() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/posixbase.py", line 229, in mainLoop self.doIteration(t) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/selectreactor.py", line 133, in doSelect _logrun(selectable, _drdw, selectable, method, dict) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/python/log.py", line 53, in callWithLogger return callWithContext({"system": lp}, func, *args, **kw) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/python/log.py", line 38, in callWithContext return context.call({ILogContext: newCtx}, func, *args, **kw) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/python/context.py", line 59, in callWithContext return self.currentContext().callWithContext(ctx, func, *args, **kw) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/python/context.py", line 37, in callWithContext return func(*args,**kw) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite why = getattr(selectable, method)() File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/tcp.py", line 535, in doConnect self.failIfNotConnected(error.getConnectError((connectResult, os.strerror(connectResult)))) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/error.py", line 160, in getConnectError return klass(number, string) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/error.py", line 105, in __init__ traceback.print_stack() That feels like 6 layers too many, given that _logrun(selectable, _drdw, selectable, method, dict) return context.call({ILogContext: newCtx}, func, *args, **kw) return self.currentContext().callWithContext(ctx, func, *args, **kw) return func(*args, **kw) getattr(selectable, method()) klass(number, string) are all generic calls. (Note that I argued against the twisted.internet.error way of doing thing as it changed my error number on me and gave me a non-system-standard, non-i18n error message.) I do not think Twisted can be changed to be an async kernel of the sort I would like without making enough changes as to be incompatible with the existing code. Also, and I say this to stress the difficulties of an outsider in using Twisted, I don't understand what's meant by "IProtocol" in > At the very least, standardizing on something very much like > IProtocol would go a long way towards making it possible to > write async clients and servers There are 37 pages (according to Google) in the twistedmatrix domain which talk about IProtocol and are not "API docs" or part of a ticket. IProtocol site:twistedmatrix.com -"API docs" -"twisted-commits" None provided insight. The API doc is at http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IProtocol.html but I don't know how to use it or even why it would work. How would I add that to an asyncore-based library? What would I need to support the adaption? There's a very high barrier to entry and while I know there are end rewards like support across many platforms I also know that I only really need to support server-side Mac and Linux boxes, and no GUIs, so asyncore may be good enough for my own work. Andrew dalke at dalkescientific.com > At the very least, standardizing on something very much like IProtocol would > go a long way towards making it possible to write async clients and servers > that could run out of the box in the stdlib as well as with Twisted, even if > the specific hookup mechanism (listenTCP, listenSSL, et. al.) were > incompatible - although a signature compatible callLater would probably be a > must. > > As I said, I don't have time to write the PEPs myself, but I might fix some > specific bugs if there were a clear set of issues preventing this from > moving forward. Better integration with the standard library would > definitely be a big win for both Twisted and Python. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/andrewdalke%40gmail.com > > From bob at redivi.com Thu Feb 15 11:11:09 2007 From: bob at redivi.com (Bob Ippolito) Date: Thu, 15 Feb 2007 02:11:09 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D3EECF.7080600@canterbury.ac.nz> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> Message-ID: <6a36e7290702150211v38dbb21bpfeb59e2524dec2b0@mail.gmail.com> On 2/14/07, Greg Ewing wrote: > Thomas Wouters wrote: > > > *I* don't like the idea of something in the Python installation > > deciding which reactor to use. > > I wouldn't mind if some way were provided of changing > the reactor if you want. I'd just like to see a long > term goal of making it unnecessary as far as possible. > > > In any case, your idea requires a lot of changes in external, non-Python > > code -- PyGTK simply exposes the GTK mainloop, which couldn't care less > > about Python's idea of a perfect event reactor model. > > On unix at least, I don't think it should be necessary > to change gtk, only pygtk. If it can find out the file > descriptor of the connection to the X server, it can > plug that into the reactor, and then call > gtk_main_iteration_do() whenever something comes in > on it. > > A similar strategy ought to work for any X11-based > toolkit that exposes a function to perform one > iteration of its main loop. > > Mileage on other platforms may vary. > > > The PerfectReactor can be added later, all current reactors > > aliased to it, and no one would have to change a single line > > of code. > > Sure. > > The other side to all this is the client side, i.e. the > code that installs event callbacks. At the moment there's > no clear direction to take, so everyone makes their own > choice -- some use asyncore, some use Twisted, some use > the gtk event loop, some roll their own, etc. There is no single PerfectReactor. There are several use cases where you need to wait on >1 different event systems, which guarantees at least two OS threads (and two event loops). In general it's nice to have a single Python event loop ("the reactor") to act on said threads (e.g. something just sitting on a mutex waiting for messages) but waiting for IO to occur should *probably* happen on one or more ancillary threads -- one per event system (e.g. select, GTK, WaitForMultipleEvents, etc.) -bob From dalke at dalkescientific.com Thu Feb 15 11:29:01 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Thu, 15 Feb 2007 03:29:01 -0700 Subject: [Python-Dev] Trial balloon: microthreads library in stdlib In-Reply-To: <20070214215841.28203.1815193832.divmod.xquotient.1838@joule.divmod.com> References: <20070214215841.28203.1815193832.divmod.xquotient.1838@joule.divmod.com> Message-ID: On 2/14/07, glyph at divmod.com wrote in response to richard.m.tew at gmail.com: > As far as I can tell, you still haven't even clearly expressed what your > needs are, let alone whether or not Twisted is suitable. In the reply > you're citing, you said that "this" sounded like something "low level" that > "twisted would be written on top of" - but the "this" you were talking > about, based on your previous messages, sounded like monkeypatching the > socket and asyncore modules to provide asynchronous file I/O based on the > platform-specific IOCP API for Windows. I don't know Richard's needs nor requirements. I do know the mechanism of the monkeypatch he's talking about. I describe it a bit in my draft for my Stackless talk at PyCon http://www.stackless.com/pipermail/stackless/2007-February/002212.html It uses asyncore for the I/O and a scheduler which can figure out if there are other running tasklets and how long is needed until the next tasklet needs to wake up. Yes, this is another reactor. Yes, it's not widely cross-platform. Yes, it doesn't work with gtk and other GUI frameworks. Yes, as written it doesn't handle threads. But also yes, it's a different style of writing reactors because of how Stackless changes the control flow. I and others would like to see something like the "stacklesssocket" implemented on top of Twisted. Andrew Francis is looking in to it but I don't know to what degree of success he's had. Err, looking through the email archive, he's has had some difficulties in doing a Twisted/Stackless integration. I don't think Twisted people understand Stackless well enough (nor obviously he Twisted) or what he's trying to do. > >It is a large dependency and it is a secondary framework. > > Has it occurred to you that it is a "large" dependency not because we like > making bloated and redundant code, but because it is doing something that is > actually complex and involved? Things like Twisted's support for NNTP, POP3, etc. aren't needed with the monkeypatch approach because the standard Python libraries will work, with Stackless and the underlying asyc library collaborating under the covers. So those parts of Twisted aren't needed or relevant. Twisted is, after all, many things. > I thought that I provided several reasons before as well, but let me state > them as clearly as I can here. Twisted is a large and mature framework with > several years of development and an active user community. The pluggable > event loop it exposes is solving a hard problem, and its implementation > encodes a lot of knowledge about how such a thing might be done. It's also > tested on a lot of different platforms. > > Writing all this from scratch - even a small subset of it - is a lot of > work, and is unlikely to result in something robust enough for use by all > Python programs without quite a lot of effort. It would be a lot easier to > get the Twisted team involved in standardizing event-driven communications > in Python. Every other effort I'm aware of is either much smaller, entirely > proprietary, or both. Again, I would love to be corrected here, and shown > some other large event-driven framework in Python that I was heretofore > unaware of. Sorry for the long quote; wasn't sure how to trim it. I made this point elsewhere and above but feel it's important enough to emphasize once more. Stackless lets people write code which looks like blocking code but which is not. The blocking functions are forwarded to the reactor, which does whatever it's supposed to do, and the results returned. Because most Python networking code will work unchanged (assuming changes to the underlying APIs to work with Stackless, as we hack now through monkeypatches), a microthreads solution almost automatically and trivially gains a large working code base, documentation, and active developer base. There does not need to be "some other large event-driven framework in Python" because all that's needed is the 13kLOC of reactor code from twisted.internet and not 140+kLOC in all of Twisted. > Standardization is much easier to achieve when you have > multiple interested parties with different interests to accommodate. As > Yitzhak Rabin used to say, "you don't engage in API standardization with > your friends, you engage in API standardization with your enemies" - or... > something like that. I thought you made contracts even with -- or especially with -- your friends regarding important matters so that both sides know what they are getting into and so you don't become enemies in the future. > You say that you weren't proposing an alternate implementation of an event > loop core, so I may be reacting to something you didn't say at all. > However, again, at least Tristan thought the same thing, so I'm not the > only one. For demonstration (and for my case pedagogical reasons) we have an event core loop. I would rather pull out the parts I need from Twisted. I don't know how. I don't need to know how right now. > I think that *that* could be described as bullying. I -- and many others -- > have a considerable investment in Twisted, and attempts to "standardize" on > another, new main-loop within the standard library is an attempt to subvert > that investment and pre-empt honest competition between different I've lost the thread of this as it's too late in the night for me. I do want to point out that whatever Richard proposed cannot be done without Stackless or an equivalent microthread library. As that's unlikely to happen within the next couple of years there's no chance of bullying here. Please note that yes I am aware that this means my argument, now stressed twice in this email, is at present for intellectual amusement and contemplation only. Were Stackless, or Dustin's microthread thoughts, or some solution with libtask (http://swtch.com/libtask/) or other library added, and if there was more than about 3 days of experience with this monkeyhack, then things would be different. But at present this is only a glimmer in about 5 pairs of eyes. > I've tried to restrict my comments here to the possibility of a new > event-driven core in the Python standard library. I do not advocate one. I do advocate my going to sleep though. :) Andrew dalke at dalkescientific.com From steve at holdenweb.com Thu Feb 15 11:48:22 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 15 Feb 2007 10:48:22 +0000 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: <45D3B45B.3080704@canterbury.ac.nz> References: <1171458253.45d308ce00f09@imp.hosting365.ie> <45D3B45B.3080704@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Steve Holden wrote: > >> A further data point is that modern machines seem to give timing >> variabilities due to CPU temperature variations even if you always eat >> exactly the same thing. > > Oh, great. Now we're going to have to run our > benchmarks in a temperature-controlled oven... > ... with the fan running at constant speed :) -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From anthony at interlink.com.au Thu Feb 15 12:27:45 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 15 Feb 2007 22:27:45 +1100 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: References: <1171458253.45d308ce00f09@imp.hosting365.ie> <45D3B45B.3080704@canterbury.ac.nz> Message-ID: <200702152227.47672.anthony@interlink.com.au> On Thursday 15 February 2007 21:48, Steve Holden wrote: > Greg Ewing wrote: > > Steve Holden wrote: > >> A further data point is that modern machines seem to give > >> timing variabilities due to CPU temperature variations even if > >> you always eat exactly the same thing. > > > > Oh, great. Now we're going to have to run our > > benchmarks in a temperature-controlled oven... > > ... with the fan running at constant speed :) Unless the fans are perfectly balanced, small changes in gravity are going to affect the rate at which they spin. So I guess the position of the moon will affect it :-) From joachim.koenig-baltes at emesgarten.de Thu Feb 15 12:37:14 2007 From: joachim.koenig-baltes at emesgarten.de (=?ISO-8859-1?Q?Joachim_K=F6nig-Baltes?=) Date: Thu, 15 Feb 2007 12:37:14 +0100 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <20070215050031.GH6786@v.igoro.us> References: <20070215050031.GH6786@v.igoro.us> Message-ID: <45D445EA.50908@emesgarten.de> dustin at v.igoro.us wrote: [...] > microtreading: > Exploiting language features to use cooperative multitasking in tasks > that "read" like they are single-threaded. > > asynchronous IO: > Performing IO to/from an application in such a way that the > application does not wait for any IO operations to complete, but > rather polls for or is notified of the readiness of any IO operations. > > [...] > Asyncore *only* implements asynchronous IO -- any "tasks" performed in > its context are the direct result of an IO operation, so it's hard to > say it implements cooperative multitasking (and Josiah can correct me if > I'm wrong, but I don't think it intends to). > > Much of the discussion here has been about creating a single, unified > asynchronous IO mechanism that would support *any* kind of cooperative > multitasking library. I have opinions on this ($0.02 each, bulk > discounts available), but I'll keep them to myself for now. > Talking only about async I/O in order to write cooperative tasks that "smell" single threaded is to restricted IMO. If there are a number of cooperative tasks that "read" single-threaded (or sequential) than the goal is to avoid a _blocking operation_ in any of them because the other tasks could do useful things in the meantime. But there are a number of different blocking operations, not only async IO (which is easily handled by select()) but also: - waiting for a child process to exit - waiting for a posix thread to join() - waiting for a signal/timer - ... Kevent (kernel event) on BSD e.g. tries to provide a common infrastructure to provide a file descriptor where one can push some conditions onto and select() until one of the conditions is met. Unfortunately, thread joining is not covered by it, so one cannot wait (without some form of busy looping) until one of the conditions is true if thread joining is one of them, but for all the other cases it would be possible. There are many other similar approaches (libevent, notify, to name a few). So in order to avoid blocking in a task, I'd prefer that the task: - declaratively specifies what kind of conditions (events) it wants to wait for. (API) If that declaration is a function call, then this function could implicitely yield if the underlying implementation would be stackless or greenlet based. Kevent on BSD systems already has a usable API for defining the conditions by structures and there is also a python module for it. The important point IMO is to have an agreed API for declaring the conditions a task wants to wait for. The underlying implementation in a scheduler would be free to use whatever event library it wants to use. E.g. have a wait(events = [], timeout = -1) method would be sufficient for most cases, where an event would specify - resource type (file, process, timer, signal, ...) - resource id (fd, process id, timer id, signal number, ...) - filter/flags (when to fire, e.g. writable, readable exception for fd, ...) - ... the result could be a list of events that have "fired", more or less similar to the events in the argument list, but with added information on the exact condition. The task would return from wait(events) when at least 1 of the conditions is met. The task then knows e.g. that an fd is readable and can then do the read() on its own in the way it likes to do it, without being forced to let some uber framework do the low level IO. Just the waiting for conditions without blocking the application is important. I have implemented something like the above, based on greenlets. In addition to the event types specified by BSD kevent(2) I've added a TASK and CHANNEL resource type for the events, so that I can wait for tasks to complete or send/receive messages to/from other tasks without blocking the application. But the implementation is not the important thing, the API is, and then we can start writing competing implementations. Joachim From barry at python.org Thu Feb 15 12:56:33 2007 From: barry at python.org (Barry Warsaw) Date: Thu, 15 Feb 2007 06:56:33 -0500 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: <200702152227.47672.anthony@interlink.com.au> References: <1171458253.45d308ce00f09@imp.hosting365.ie> <45D3B45B.3080704@canterbury.ac.nz> <200702152227.47672.anthony@interlink.com.au> Message-ID: <4C8892DE-76F3-4A65-8430-5C915D7F2B01@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 15, 2007, at 6:27 AM, Anthony Baxter wrote: > On Thursday 15 February 2007 21:48, Steve Holden wrote: >> Greg Ewing wrote: >>> Steve Holden wrote: >>>> A further data point is that modern machines seem to give >>>> timing variabilities due to CPU temperature variations even if >>>> you always eat exactly the same thing. >>> >>> Oh, great. Now we're going to have to run our >>> benchmarks in a temperature-controlled oven... >> >> ... with the fan running at constant speed :) > > Unless the fans are perfectly balanced, small changes in gravity are > going to affect the rate at which they spin. So I guess the > position of the moon will affect it :-) Except on Tuesdays. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRdRKcXEjvBPtnXfVAQJUNAP/ebCrt2RV2pmKTElHUyzWIWxPFCqIiIuF FDkiSx4x/ZZtOcdXlJHOA6lBWjuPAxe1jxE9BVpNy/6qCtky+mpnM5nXqIeAlQUk XByguxKmsxF4HWSlk6GJ4hZWbZqsMdFiw8WZttCihQJwmr58rDMTUKRHss2IPOhL egl76Tg1gE4= =OTwQ -----END PGP SIGNATURE----- From steve at holdenweb.com Thu Feb 15 13:09:26 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 15 Feb 2007 12:09:26 +0000 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: <4C8892DE-76F3-4A65-8430-5C915D7F2B01@python.org> References: <1171458253.45d308ce00f09@imp.hosting365.ie> <45D3B45B.3080704@canterbury.ac.nz> <200702152227.47672.anthony@interlink.com.au> <4C8892DE-76F3-4A65-8430-5C915D7F2B01@python.org> Message-ID: <45D44D76.1040508@holdenweb.com> Barry Warsaw wrote: > -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > On Feb 15, 2007, at 6:27 AM, Anthony Baxter wrote: > >> On Thursday 15 February 2007 21:48, Steve Holden wrote: >>> Greg Ewing wrote: >>>> Steve Holden wrote: >>>>> A further data point is that modern machines seem to give >>>>> timing variabilities due to CPU temperature variations even if >>>>> you always eat exactly the same thing. >>>> >>>> Oh, great. Now we're going to have to run our >>>> benchmarks in a temperature-controlled oven... >>> >>> ... with the fan running at constant speed :) >> >> Unless the fans are perfectly balanced, small changes in gravity are >> going to affect the rate at which they spin. So I guess the >> position of the moon will affect it :-) > > Except on Tuesdays. [off-list, because "this is getting silly" ...] Anthony's antipodean antecedents (alliteration, all right?) remind me we will also have to factor Coriolis effects in. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From steve at holdenweb.com Thu Feb 15 13:18:14 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 15 Feb 2007 12:18:14 +0000 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: <45D44D76.1040508@holdenweb.com> References: <1171458253.45d308ce00f09@imp.hosting365.ie> <45D3B45B.3080704@canterbury.ac.nz> <200702152227.47672.anthony@interlink.com.au> <4C8892DE-76F3-4A65-8430-5C915D7F2B01@python.org> <45D44D76.1040508@holdenweb.com> Message-ID: Steve Holden wrote: > Barry Warsaw wrote: >> -----BEGIN PGP SIGNED MESSAGE----- >> Hash: SHA1 >> >> On Feb 15, 2007, at 6:27 AM, Anthony Baxter wrote: >> >>> On Thursday 15 February 2007 21:48, Steve Holden wrote: >>>> Greg Ewing wrote: >>>>> Steve Holden wrote: >>>>>> A further data point is that modern machines seem to give >>>>>> timing variabilities due to CPU temperature variations even if >>>>>> you always eat exactly the same thing. >>>>> Oh, great. Now we're going to have to run our >>>>> benchmarks in a temperature-controlled oven... >>>> ... with the fan running at constant speed :) >>> Unless the fans are perfectly balanced, small changes in gravity are >>> going to affect the rate at which they spin. So I guess the >>> position of the moon will affect it :-) >> Except on Tuesdays. > [off-list, because "this is getting silly" ...] Apparently the "off-list" part was a desire that didn't become reality. Sorry, I'll shut up now. regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From rhamph at gmail.com Thu Feb 15 14:17:03 2007 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 15 Feb 2007 06:17:03 -0700 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <45D445EA.50908@emesgarten.de> References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> Message-ID: On 2/15/07, Joachim K?nig-Baltes wrote: > dustin at v.igoro.us wrote: > E.g. have a wait(events = [], timeout = -1) method would be sufficient > for most cases, where an event would specify I agree with everything except this. A simple function call would have O(n) cost, thus being unacceptable for servers with many open connections. Instead you need it to maintain a set of events and let you add or remove from that set as needed. > I have implemented something like the above, based on greenlets. I assume greenlets would be an internal implementation detail, not exposed to the interface? -- Adam Olsen, aka Rhamphoryncus From exarkun at divmod.com Thu Feb 15 15:19:30 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 15 Feb 2007 09:19:30 -0500 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: Message-ID: <20070215141930.25807.412274186.divmod.quotient.22521@ohm> On Thu, 15 Feb 2007 02:36:22 -0700, Andrew Dalke wrote: >I was the one on the Stackless list who last September or so >proposed the idea of monkeypatching and I'm including that >idea in my presentation for PyCon. See my early rough draft >at http://www.stackless.com/pipermail/stackless/2007-February/002212.html >which contains many details about using Stackless, though >none on the Stackless implementation. (A lot on how to tie things together.) > >So people know, I am an applications programmer and not a >systems programmer. Things like OS-specific event mechanisms >annoy and frustrate me. If I could do away with hardware and >still write useful programs I would. What a wonderful world it would be. :) > > [snip] > >In all three cases I've found it hard to use Twisted because >the code didn't do as I expected it to do and when something >went wrong I got results which were hard to interpret. I >believe others have similar problems and is one reason Twisted >is considered to be "a big, complicated, inseparable hairy mess." > >I find the Stackless code also hard to understand. Eg, >I don't know where the watchdog code is for the "run()" >command. It uses several layers of macros and I haven't >been able get it straight in my head. However, so far >I've not run into strange errors in Stackless that I >have in Twisted. > As you point out below, however, Twisted and stackless achieve different goals. >I find the normal Python code relatively easy to understand. > >Stackless only provides threadlets. It does no I/O. >Richard Tew developed a "stacklesssocket" module which emulates >the API for the stdlib "socket" module. I tweaked it a >bit and showed that by doing the monkeypatch > > import stacklesssocket > import sys > sys.modules["socket"] = stacklesssocket > >then code like "urllib.urlopen" became Stackless compatible. >Eg, in my PyCon talk draft I show something like > It may be of interest to you to learn that a Twisted developer implement this model several years ago. It has not been further developed for a handful of reasons, at the core of which is the fact that it is very similar to pre-emptive threading in terms of application-level complexity. You gave several examples of the use of existing code which expects a blocking socket interface and which "just works" when the socket module is changed in this way. However, this is a slight simplification. Code written without expecting a context switch (exactly what happens when a socket operation is performed in this model) is not necessarily correct when context switches are suddenly introduced. Consider this extremely trivial example: x = 0 def foo(conn): global x a = x + 1 b = ord(conn.recv(1)) x = a + b return x Clearly, foo is not threadsafe. Global mutable state is a terrible, terrible thing. The point to note is that by introducing a context switch at the conn.recv(1) call, the same effect is achieved as by any other context switch: it becomes possible for foo to return an inconsistent result or otherwise corrupt its own state if another piece of code violates its assumptions and changes x while it is waiting for the recv call to complete. Is urllib2 threadsafe? I have heard complaints that it is not. I have looked at the code, and at least in its support for caching, it appears not to be. Perhaps it can be made threadsafe, but in requiring that, the advantage of having a whole suite of modules which will "just work" with a transparently context switching socket module are mostly lost. > > [snip - urllib2/tasklet example] > >The choice of asyncore is, I think, done because 1) it >prevents needing an external dependency, But if some new event loop is introduced into the standard library, then using it also will not require an external dependency. ;) >2) asyncore is >smaller and easier to understand than Twisted, While I hear this a lot, applications written with Twisted _are_ shorter and contain less irrelevant noise in the form of boilerplate than the equivalent asyncore programs. This may not mean that Twisted programs are easier to understand, but it is at least an objectively measurable metric. >and >3) it was for demo/proof of concept purposes. >While >tempting to improve that module I know that Twisted >has already gone though all the platform-specific crap >and I don't want to go through it again myself. I don't >want to write a reactor to deal with GTK, and one for >OS X, and one for ... > Now if we can only figure out a way for everyone to benefit from this without tying too many brains up in knots. :) > >Another reason I think Twisted is considered "tangled-up >Deep Magic, only for Wizards Of The Highest Order" is because >it's infused with event-based processing. I've done a lot >of SAX processing and I can say that few people think that >way or want to go through the process of learning how. > >Compare, for example, the following > > f = urllib2.urlopen("http://example.com/") > for i, line in enumerate(f): > print ("%06d" % i), repr(line) > >with the normal equivalent in Twisted or other >async-based system. > Several years ago, Christopher Armstrong (hopefully he won't get too upset at me for mentioning him here) write a Twisted/ Stackless integration library. When greenlets came out, he write a similar library for integrating Twisted with those. He also wrote a utility generally referred to as "defgen", and James Knight updated it to take advantage of the Python 2.5 changes to generators. Through all of that, however, Twisted is still taking care of all of the nitty gritty platform details. Whether one uses stackless or greenlets or generators or any other mechanism, it is important to realize that the lexical structure of the code is not inherently tied to the networking library in use. If you want to write code in the style of the above for loop, you can do so with Twisted and stackless. The problems involved are much the same as those involving urllib2/stackless, but at least Twisted is prepared to deal with context switching around network events, so any bugs you encounter are likely to be due to mistaken assumptions in your own application code. :) For what it's worth, I prefer context switches to be explicit in the style of continuation passing so that I am less likely to introduce such bugs into my own code. This is, however, entirely at my discretion, and I am not about to force anyone else to develop their applications this way. >Yet by using the Stackless socket monkeypatch, this >same code works in an async framework. And the underlying >libraries have a much larger developer base than Twisted. >Want NNTP? "import nntplib" Want POP3? "import poplib" >Plenty of documentation about them too. This is going to come out pretty harshly, for which I can only apologize in advance, but it bears mention. The quality of protocol implementations in the standard library is bad. As in "not good". Twisted's NNTP support is better (even if I do say so myself - despite only having been working on by myself, when I knew almost nothing about Twisted, and having essentially never been touched since). Twisted's POP3 support is fantastically awesome. Next to imaplib, twisted.mail.imap4 is a sparkling diamond. And each of these implements the server end of the protocol as well: you won't find that in the standard library for almost any protocol. As for the documentation, please compare these two pages: http://python.org/doc/lib/pop3-objects.html http://twistedmatrix.com/documents/current/api/twisted.mail.pop3.AdvancedPOP3Client.html I think it is fair to call them comparable. They could both stand some improvement, really. :) And if someone wants to argue that, if the POP3 client from Twisted is going to be added to the standard library, its documentation should be improved first, I'm not going to argue against that. Docs are great, more docs are greater. But let's bear in mind that at present, no one has suggested adding anything but the core Twisted event loop. > >All the earlier quotes were lifted from glyph. Here's another: >> When you boil it down, Twisted's event loop is just a >> notification for "a connection was made", "some data was >> received on a connection", "a connection was closed", and >> a few APIs to listen or initiate different kinds of >> connections, start timed calls, and communicate with threads. >> All of the platform details of how data is delivered to the >> connections are abstracted away.. How do you propose we >> would make a less "specific" event mechanism? > >What would I need to do to extract this Twisted core so >I could replace asyncore? I know at minimum I need >"twisted.internet" and "twisted.python" (the latter for >logging) and "twisted.persisted" for "styles.Ephemeral". Neither of those dependencies is a very hard one. I suspect that there would be resistence to adding a new logging system to the standard library, just for Twisted. You have the right idea though. Some portion of twisted.internet, and whatever utility code it depends on. > >But I say this hesitantly recalling the frustrations >I had in dealing with a connection error in Twisted, >described in the aforementioned link > http://www.dalkescientific.com/writings/diary/archive/2006/08/28/levels_of_abstraction.html > >I feel that using the phrase "just a" in the previously quoted >text is an understatement. I think you're right. We throw around "just" a lot in our line of work, don't we? :) Twisted does also account for a raft of platform-specific quirks and inconsistencies. I take this to be a good thing. >While the mechanics might be >simple, there are many, many layers, as you can see in this >stack trace. > > File "async_blast.py", line 55, in ? > reactor.run() > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/internet/posixbase.py", line 218, in run > self.mainLoop() > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/internet/posixbase.py", line 229, in mainLoop > self.doIteration(t) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/internet/selectreactor.py", line 133, in doSelect > _logrun(selectable, _drdw, selectable, method, dict) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/python/log.py", line 53, in callWithLogger > return callWithContext({"system": lp}, func, *args, **kw) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/python/log.py", line 38, in callWithContext > return context.call({ILogContext: newCtx}, func, *args, **kw) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/python/context.py", line 59, in callWithContext > return self.currentContext().callWithContext(ctx, func, *args, **kw) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/python/context.py", line 37, in callWithContext > return func(*args,**kw) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/internet/selectreactor.py", line 139, in _doReadOrWrite > why = getattr(selectable, method)() > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/internet/tcp.py", line 535, in doConnect > self.failIfNotConnected(error.getConnectError((connectResult, >os.strerror(connectResult)))) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/internet/error.py", line 160, in getConnectError > return klass(number, string) > File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ >site-packages/twisted/internet/error.py", line 105, in __init__ > traceback.print_stack() > >That feels like 6 layers too many, given that > _logrun(selectable, _drdw, selectable, method, dict) > return context.call({ILogContext: newCtx}, func, *args, **kw) > return self.currentContext().callWithContext(ctx, func, *args, **kw) > return func(*args, **kw) > getattr(selectable, method()) > klass(number, string) > >are all generic calls. I know function calls are expensive in Python, and method calls even more so... but I still don't understand this issue. Twisted's call stack is too deep? It is fair to say it is deep, I guess, but I don't see how that is a problem. If it is, I don't see how it is specific to this discussion. >(Note that I argued against the >twisted.internet.error way of doing thing as it changed my >error number on me and gave me a non-system-standard, non-i18n >error message.) Note that we ended up /not/ changing the error number in the case you encountered. We changed the connection setup code to handle the unexpected behavior on OS X. :) Twisted is faithfully reporting the same errno as the underlying platform is producing. Since most applications don't know or care about such things though, it is also putting it into an exception instance which indicates the category into which the error falls. These all seem like good things to me. > >I do not think Twisted can be changed to be an async >kernel of the sort I would like without making enough >changes as to be incompatible with the existing code. > >Also, and I say this to stress the difficulties of an outsider >in using Twisted, I don't understand what's meant by "IProtocol" in >> At the very least, standardizing on something very much like >> IProtocol would go a long way towards making it possible to >> write async clients and servers It is exactly these interfaces which make it possible to make changes to Twisted without breaking things. The behaviour of the APIs exposed by Twisted is defined. Even if IProtocol is not adopted verbatim, the existence of IProtocol and another interface means one can be adapted to the other in some manner, providing compatibility for existing applications. > >There are 37 pages (according to Google) in the twistedmatrix domain >which talk about IProtocol and are not "API docs" or part of a ticket. > > IProtocol site:twistedmatrix.com -"API docs" -"twisted-commits" > >None provided insight. The API doc is at > http://twistedmatrix.com/documents/current/api/twisted.internet.interfaces.IProtocol.html > Since the standard library lacks interfaces, it may be the case that IProtocol is instead adopted as an ABC or even that it won't appear in code at all, but instead be translated into non-source documentation. I'd prefer if z.i were in the stdlib, but that's a separate issue. What Glyph is saying when he talks about standardizing IProtocol is the standardization of an API, nothing more. Which is what this whole thread is about, if I am not mistaken. I apologize for writing such a long message, but I didn't have time to write a shorter one. Jean-Paul From joachim.koenig-baltes at emesgarten.de Thu Feb 15 15:49:09 2007 From: joachim.koenig-baltes at emesgarten.de (=?ISO-8859-1?Q?Joachim_K=F6nig-Baltes?=) Date: Thu, 15 Feb 2007 15:49:09 +0100 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> Message-ID: <45D472E5.6070803@emesgarten.de> Adam Olsen wrote: > I agree with everything except this. A simple function call would > have O(n) cost, thus being unacceptable for servers with many open > connections. Instead you need it to maintain a set of events and let > you add or remove from that set as needed. We can learn from kevent here, it already has EV_ADD, EV_DELETE, EV_ENABLE, EV_DISABLE, EV_ONESHOT flags. So the event-conditions would stay "in the scheduler" (per task) so that they can fire multiple times without the need to be handled over again and again. Thanks, that's exactly the discussion I'd like to see, discussing about a simple API. >> I have implemented something like the above, based on greenlets. > > I assume greenlets would be an internal implementation detail, not > exposed to the interface? Yes, you could use stackless, perhaps even Twisted, but I'm not sure if that would work because the requirement for the "reads single-threaded" is the simple wait(...) function call that does a yield (over multiple stack levels down to the function that created the task), something that is only provided by greenlet and stackless to my knowledge. Joachim From rhamph at gmail.com Thu Feb 15 16:11:43 2007 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 15 Feb 2007 08:11:43 -0700 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <45D472E5.6070803@emesgarten.de> References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <45D472E5.6070803@emesgarten.de> Message-ID: On 2/15/07, Joachim K?nig-Baltes wrote: > Adam Olsen wrote: > >> I have implemented something like the above, based on greenlets. > > > > I assume greenlets would be an internal implementation detail, not > > exposed to the interface? > Yes, you could use stackless, perhaps even Twisted, > but I'm not sure if that would work because the requirement for the > "reads single-threaded" is the simple wait(...) function call that does > a yield > (over multiple stack levels down to the function that created the task), > something that is only provided by greenlet and stackless to my knowledge. I don't think we're on the same page then. The way I see it you want a single async IO implementation shared by everything while having a collection of event loops that cooperate "just enough". The async IO itself would likely end up being done in C. -- Adam Olsen, aka Rhamphoryncus From joachim.koenig-baltes at emesgarten.de Thu Feb 15 16:28:17 2007 From: joachim.koenig-baltes at emesgarten.de (=?ISO-8859-1?Q?Joachim_K=F6nig-Baltes?=) Date: Thu, 15 Feb 2007 16:28:17 +0100 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <45D472E5.6070803@emesgarten.de> Message-ID: <45D47C11.3070108@emesgarten.de> Adam Olsen schrieb: > I don't think we're on the same page then. The way I see it you want > a single async IO implementation shared by everything while having a > collection of event loops that cooperate "just enough". The async IO > itself would likely end up being done in C. > No, I'd like to have: - An interface for a task to specifiy the events it's interested in, and waiting for at least one of the events (with a timeout). - an interface for creating a task (similar to creating a thread) - an interface for a schedular to manage the tasks When a task resumes after a wait(...) it knows which of the events it was waiting for have fired. It can then do whatever it wants, do the low level non blocking IO on its own or using something else. Of course, as these are cooperative tasks, they still must be careful not to block, e.g. not reading to much from a file descriptor that is readable, but these problems have been solved in a lot of libraries, and I would not urge the task to use a specific way to accomplish its "task". The problem solved by this approach is to allow a number of cooperating threads to wait for an event without the need to busy loop or block by delegating the waiting to a central instance, the scheduler. How this efficient waiting is implemented is the responsability of the scheduler, but the schedular would not do the (possibly blocking) io operation, it would only guaranty to continue a task, when it can do an IO operation without blocking. Joachim From amk at amk.ca Thu Feb 15 16:46:05 2007 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 15 Feb 2007 10:46:05 -0500 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070215141930.25807.412274186.divmod.quotient.22521@ohm> References: <20070215141930.25807.412274186.divmod.quotient.22521@ohm> Message-ID: <20070215154605.GA5434@localhost.localdomain> On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote: > >That feels like 6 layers too many, given that > > _logrun(selectable, _drdw, selectable, method, dict) > > return context.call({ILogContext: newCtx}, func, *args, **kw) > > return self.currentContext().callWithContext(ctx, func, *args, **kw) > > return func(*args, **kw) > > getattr(selectable, method()) > > klass(number, string) > > > >are all generic calls. > > I know function calls are expensive in Python, and method calls even more > so... but I still don't understand this issue. Twisted's call stack is too > deep? It is fair to say it is deep, I guess, but I don't see how that is a > problem. If it is, I don't see how it is specific to this discussion. It's hard to debug the resulting problem. Which level of the *12* levels in the stack trace is responsible for a bug? Which of the *6* generic calls is calling the wrong thing because a handler was set up incorrectly or the wrong object provided? The code is so 'meta' that it becomes effectively undebuggable. --amk From dustin at v.igoro.us Thu Feb 15 16:49:45 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Thu, 15 Feb 2007 09:49:45 -0600 Subject: [Python-Dev] generic async io (was: microthreading vs. async io) In-Reply-To: <45D47C11.3070108@emesgarten.de> References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <45D472E5.6070803@emesgarten.de> <45D47C11.3070108@emesgarten.de> Message-ID: <20070215154945.GK6786@v.igoro.us> On Thu, Feb 15, 2007 at 04:28:17PM +0100, Joachim K?nig-Baltes wrote: > No, I'd like to have: > > - An interface for a task to specifiy the events it's interested in, and > waiting for at least one of the events (with a timeout). > - an interface for creating a task (similar to creating a thread) > - an interface for a schedular to manage the tasks I think this discussion would be facilitated by teasing the first bullet-point from the latter two: the first deals with async IO, while the latter two deal with cooperative multitasking. It's easy to write a single package that does both, but it's much harder to write *two* fairly generic packages with a clean API between them, given the varied platform support for async IO and the varied syntax and structures (continuations vs. microthreads, in my terminology) for multitasking. Yet I think that division is exactly what's needed. Since you asked (I'll assume the check for $0.02 is in the mail), I think a strictly-async-IO library would offer the following: - a sleep queue object to which callables can be added - wrappers for all/most of the stdlib blocking IO operations which add the operation to the list of outstanding operations and return a sleep queue object - some relatively easy method of extending that for new IO operations - a poll() function (for multitasking libraries) and a serve_forever() loop (for asyncore-like uses, where all the action is IO-driven) The mechanisms for accomplishing all of that on the chosen platform would be an implementation detail, possibly with some intitialization "hinting" from the application. The library would also need to expose its platform-based limitations (can't wait on thd.join(), can only wait on 64 fd's, etc.) to the application for compatibility-checking purposes. Thoughts? Dustin From joachim.koenig-baltes at emesgarten.de Thu Feb 15 16:51:30 2007 From: joachim.koenig-baltes at emesgarten.de (=?ISO-8859-1?Q?Joachim_K=F6nig-Baltes?=) Date: Thu, 15 Feb 2007 16:51:30 +0100 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <45D47C11.3070108@emesgarten.de> References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <45D472E5.6070803@emesgarten.de> <45D47C11.3070108@emesgarten.de> Message-ID: <45D48182.5030200@emesgarten.de> Joachim K?nig-Baltes wrote: > The problem solved by this approach is to allow a number of cooperating > threads to wait for an event without the need to busy loop or block by > delegating > the waiting to a central instance, the scheduler. How this efficient > waiting is > implemented is the responsability of the scheduler, but the schedular would > not do the (possibly blocking) io operation, it would only guaranty to > continue a task, when it can do an IO operation without blocking. > > From the point of view of the task, it only has to sprinkle a number of wait(...) calls before doing blocking calls, so there is no need to use callbacks or writing the inherently sequential code upside down. That is the gain I'm interested in. The style used in asyncore, inheriting from a class and calling return in a method and being called later at a different location (different method) just interrupts the sequential flow of operations and makes it harder to understand. The same is true for all other strategies using callbacks or similar mechanisms. All this can be achieved with a multilevel yield() that is hidden in a function call. So the task does a small step down (wait) in order to jump up (yield) to the scheduler without disturbing the eye of the beholder. Joachim From exarkun at divmod.com Thu Feb 15 17:01:17 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 15 Feb 2007 11:01:17 -0500 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070215154605.GA5434@localhost.localdomain> Message-ID: <20070215160117.25807.36797535.divmod.quotient.22619@ohm> On Thu, 15 Feb 2007 10:46:05 -0500, "A.M. Kuchling" wrote: >On Thu, Feb 15, 2007 at 09:19:30AM -0500, Jean-Paul Calderone wrote: >> >That feels like 6 layers too many, given that >> > _logrun(selectable, _drdw, selectable, method, dict) >> > return context.call({ILogContext: newCtx}, func, *args, **kw) >> > return self.currentContext().callWithContext(ctx, func, *args, **kw) >> > return func(*args, **kw) >> > getattr(selectable, method()) >> > klass(number, string) >> > >> >are all generic calls. >> >> I know function calls are expensive in Python, and method calls even more >> so... but I still don't understand this issue. Twisted's call stack is too >> deep? It is fair to say it is deep, I guess, but I don't see how that is a >> problem. If it is, I don't see how it is specific to this discussion. > >It's hard to debug the resulting problem. Which level of the *12* >levels in the stack trace is responsible for a bug? Which of the *6* >generic calls is calling the wrong thing because a handler was set up >incorrectly or the wrong object provided? The code is so 'meta' that >it becomes effectively undebuggable. I've debugged plenty of Twisted applications. So it's not undebuggable. :) Application code tends to reside at the bottom of the call stack, so Python's traceback order puts it right where you're looking, which makes it easy to find. For any bug which causes something to be set up incorrectly and only later manifests as a traceback, I would posit that whether there is 1 frame or 12, you aren't going to get anything useful out of the traceback. Standard practice here is just to make exception text informative, I think, but this is another general problem with Python programs and event loops, not one specific to either Twisted itself or the particular APIs Twisted exposes. As a personal anecdote, I've never once had to chase a bug through any of the 6 "generic calls" singled out. I can't think of a case where I've helped any one else who had to do this, either. That part of Twisted is very old, it is _very_ close to bug-free, and application code doesn't have very much control over it at all. Perhaps in order to avoid scaring people, there should be a way to elide frames from a traceback (I don't much like this myself, I worry about it going wrong and chopping out too much information, but I have heard other people ask for it)? Jean-Paul From larry at hastings.org Thu Feb 15 17:02:54 2007 From: larry at hastings.org (Larry Hastings) Date: Thu, 15 Feb 2007 08:02:54 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <6a36e7290702150211v38dbb21bpfeb59e2524dec2b0@mail.gmail.com> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> <6a36e7290702150211v38dbb21bpfeb59e2524dec2b0@mail.gmail.com> Message-ID: <45D4842E.9080105@hastings.org> Bob Ippolito wrote: > There is no single PerfectReactor. There are several use cases where > you need to wait on >1 different event systems, which guarantees at > least two OS threads (and two event loops). In general it's nice to > have a single Python event loop ("the reactor") to act on said threads > (e.g. something just sitting on a mutex waiting for messages) but > waiting for IO to occur should *probably* happen on one or more > ancillary threads -- one per event system (e.g. select, GTK, > WaitForMultipleEvents, etc.) Why couldn't PerfectReactor be a reactor for other reactors? A sort of concentrator for these multiple event systems and multiple threads. You ask to listen to sockets, so it instantiates a singleton PerfectReactor which instantiates a select() reactor and listens to it directly in a single-threaded manner. If you then ask to listen to Win32 messages, the PerfectReactor instantiates a GetMessage() reactor. Then, realizing it has two "event systems", it spawns a thread for each child reactor with a listener that serializes the incoming events into the PerfectReactor's queue. Bingo, your application doesn't need to be written thread-safe, PerfectReactor is platform-agnostic, and you don't have to know in advance all the event types you might ever listen to. Sorry if this is a dumb question (or if I'm mangling the terminology), /larry/ From pje at telecommunity.com Thu Feb 15 17:32:03 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 15 Feb 2007 11:32:03 -0500 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <20070215050031.GH6786@v.igoro.us> Message-ID: <5.1.1.6.0.20070215112716.0281ad80@sparrow.telecommunity.com> At 11:00 PM 2/14/2007 -0600, dustin at v.igoro.us wrote: >Instead, I would like to concentrate on producing a small, clean, >consistent, generator-based microthreading library. I've seen several >such libraries (including the one in PEP 342, which is fairly skeletal), >and they all work *almost* the same way, but differ in, for example, the >kinds of values that can be yielded, their handling of nested calls, and >the names for the various "special" values one can yield. Which is one reason that any "standard" coroutine library would need to be extensible with respect to the handling of yielded values, e.g. by using a generic function to implement the yielding. See the 'yield_to' function in the example I posted. Actually, the example I posted would work fine as a microthreads core by adding a thread-local variable that points to some kind of "scheduler" to replace the Twisted scheduling functions I used. It would need to be a variable, because applications would have to be able to replace it, e.g. with the Twisted reactor. In other words, the code I posted isn't really depending on Twisted for anything but reactor.callLater() and the corresponding .isActive() and .cancel() methods of the objects it returns. If you add a default implementation of those features that can be replaced with the Twisted reactor, and dropped the code that deals with Deferreds and TimeoutErrors, you'd have a nice standalone library. From martin at v.loewis.de Thu Feb 15 17:35:37 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 15 Feb 2007 17:35:37 +0100 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D4842E.9080105@hastings.org> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> <6a36e7290702150211v38dbb21bpfeb59e2524dec2b0@mail.gmail.com> <45D4842E.9080105@hastings.org> Message-ID: <45D48BD9.3040608@v.loewis.de> Larry Hastings schrieb: > Bob Ippolito wrote: >> There is no single PerfectReactor. There are several use cases where >> you need to wait on >1 different event systems, which guarantees at >> least two OS threads (and two event loops). In general it's nice to >> have a single Python event loop ("the reactor") to act on said threads >> (e.g. something just sitting on a mutex waiting for messages) but >> waiting for IO to occur should *probably* happen on one or more >> ancillary threads -- one per event system (e.g. select, GTK, >> WaitForMultipleEvents, etc.) > Why couldn't PerfectReactor be a reactor for other reactors? A sort of > concentrator for these multiple event systems and multiple threads. Because you can't write a single function that blocks (idle) until an event in one of the systems occurs. None of the system is truly asynchronous. All APIs, eventually, rely on synchronous, blocking, operating system calls. In Unix, the call to make is typically select or poll, and you know it is safe to make because you have all even sources as file descriptors and you know nothing else needs to be done before one of the events occurs. Now, for these generalized event loops, it may not be possible anymore to combine all event sources into a single blocking call. If you have timeouts, you have to find out what the minimum timeout is (to make it the timeout for the blocking call, assuming that supports a timeout). If you have idle tasks, you have to invoke all idle tasks of all reactors (which you may not be able to get because there is no API to fetch them). Even more difficult: if these use different OS event mechanisms (on systems that have multiple such systems). E.g. try combinging a Win32 GetMessage/PostMessage loop with a WaitForMultipleObjects loop with a WinSock select call; then add the Win32 RPC libraries to that (i.e. try to determine the named pipe handles you have to pass to WaitForMultipleObjects). > You ask to listen to sockets, so it instantiates a singleton > PerfectReactor which instantiates a select() reactor and listens to it > directly in a single-threaded manner. If you then ask to listen to > Win32 messages, the PerfectReactor instantiates a GetMessage() reactor. > Then, realizing it has two "event systems", it spawns a thread for each > child reactor with a listener that serializes the incoming events into > the PerfectReactor's queue. Ah, threads :-( It turns out that you need to invoke GetMessage in the context of the thread in which the window was created. In a different thread, you won't get any messages. (This is indeed one of the biggest limitations of USER32, which was only overcome in Vista when the "native" GUI programming model is based on DirectX - the problem is then of course solved only for applications that use the DirectX UI API). > Bingo, your application doesn't need to be > written thread-safe, PerfectReactor is platform-agnostic, and you don't > have to know in advance all the event types you might ever listen to. > > Sorry if this is a dumb question (or if I'm mangling the terminology), Integrating with threads might be a solution in some cases, and a problem in others. You can't assume it is a universal solution. Regards, Martin From dustin at v.igoro.us Thu Feb 15 17:36:21 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Thu, 15 Feb 2007 10:36:21 -0600 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <45D48182.5030200@emesgarten.de> References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <45D472E5.6070803@emesgarten.de> <45D47C11.3070108@emesgarten.de> <45D48182.5030200@emesgarten.de> Message-ID: <20070215163621.GL6786@v.igoro.us> On Thu, Feb 15, 2007 at 04:51:30PM +0100, Joachim K?nig-Baltes wrote: > The style used in asyncore, inheriting from a class and calling return > in a method > and being called later at a different location (different method) just > interrupts the > sequential flow of operations and makes it harder to understand. The same is > true for all other strategies using callbacks or similar mechanisms. > > All this can be achieved with a multilevel yield() that is hidden in a > function call. > So the task does a small step down (wait) in order to jump up (yield) to > the scheduler > without disturbing the eye of the beholder. I agree -- I find that writing continuations or using asyncore's structure makes spaghetti out of functionality that requires multiple blocking operations inside looping or conditional statements. The best example, for me, was writing a complex site-specific web spider that had to fetch 5-10 pages in a certain sequence, where each step in that sequence depended on the results of the previous fetches. I wrote it in Twisted, but the proliferation of nested callback functions and chained deferreds made my head explode while trying to debug it. With a decent microthreading library, that could look like: def fetchSequence(...): fetcher = Fetcher() yield fetcher.fetchHomepage() firstData = yield fetcher.fetchPage('http://...') if someCondition(firstData): while True: secondData = yield fetcher.fetchPage('http://...') # ... if someOtherCondition(secondData): break else: # ... which is *much* easier to read and debug. (FWIW, after I put my head back together, I rewrote the app with threads, and it now looks like the above, without the yields. Problem is, throttlling on fetches means 99% of my threads are blocked on sleep() at any given time, which is just silly). All that said, I continue to contend that the microthreading and async IO operations are separate. The above could be implemented relatively easily in Twisted with a variant of the microthreading module Phillip posted earlier. It could also be implemented atop a bare-bones microthreading module with Fetcher using asyncore on the backend, or even scheduler urllib.urlopen() calls into OS threads. Presumably, it could run in NanoThreads and Kamaelia too, among others. What I want is a consistent syntax for microthreaded code, so that I could write my function once and run it in *all* of those circumstances. Dustin P.S. For the record -- I've written lots of other apps in Twisted with great success; this one just wasn't a good fit. From exarkun at divmod.com Thu Feb 15 17:47:27 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 15 Feb 2007 11:47:27 -0500 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <20070215163621.GL6786@v.igoro.us> Message-ID: <20070215164727.25807.382881589.divmod.quotient.22665@ohm> On Thu, 15 Feb 2007 10:36:21 -0600, dustin at v.igoro.us wrote: > [snip] > >def fetchSequence(...): > fetcher = Fetcher() > yield fetcher.fetchHomepage() > firstData = yield fetcher.fetchPage('http://...') > if someCondition(firstData): > while True: > secondData = yield fetcher.fetchPage('http://...') > # ... > if someOtherCondition(secondData): break > else: > # ... Ahem: from twisted.internet import reactor from twisted.internet.defer import inlineCallbacks from twisted.web.client importt getPage @inlineCallbacks def fetchSequence(...): homepage = yield getPage(homepage) firstData = yield getPage(anotherPage) if someCondition(firstData): while: secondData = yield getPage(wherever) if someOtherCondition(secondData): break else: ... So as I pointed out in another message in this thread, for several years it has been possible to do this with Twisted. Since Python 2.5, you can do it exactly as I have written above, which looks exactly the same as your example code. Is the only problem here that this style of development hasn't had been made visible enough? Jean-Paul From pje at telecommunity.com Thu Feb 15 18:10:25 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 15 Feb 2007 12:10:25 -0500 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <20070215164727.25807.382881589.divmod.quotient.22665@ohm> References: <20070215163621.GL6786@v.igoro.us> Message-ID: <5.1.1.6.0.20070215120339.04e64088@sparrow.telecommunity.com> At 11:47 AM 2/15/2007 -0500, Jean-Paul Calderone wrote: >Is the only problem here that this style of development hasn't had been made >visible enough? You betcha. I sure as heck wouldn't have bothered writing the module I did, if I'd known it already existed. Or at least only written whatever parts that the module doesn't do. The Twisted used by the current version of Chandler doesn't include this feature yet, though, AFAICT. But this is excellent; it means people will be able to write plugins that do network I/O without needing to grok CPS. They'll still need to be able to grok some basics (like not blocking the reactor), but this is good to know about. Now I won't have to actually test that module I wrote. ;-) You guys should be trumpeting this - it's real news and in fact a motivator for people to upgrade to Python 2.5 and whatever version of Twisted supports this. You just lifted a *major* barrier to using Twisted for client-oriented tasks, although I imagine there probably needs to be even more of it. From rhamph at gmail.com Thu Feb 15 18:33:42 2007 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 15 Feb 2007 10:33:42 -0700 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <45D47C11.3070108@emesgarten.de> References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <45D472E5.6070803@emesgarten.de> <45D47C11.3070108@emesgarten.de> Message-ID: On 2/15/07, Joachim K?nig-Baltes wrote: > Adam Olsen schrieb: > > I don't think we're on the same page then. The way I see it you want > > a single async IO implementation shared by everything while having a > > collection of event loops that cooperate "just enough". The async IO > > itself would likely end up being done in C. > > > No, I'd like to have: > > - An interface for a task to specifiy the events it's interested in, and > waiting > for at least one of the events (with a timeout). > - an interface for creating a task (similar to creating a thread) > - an interface for a schedular to manage the tasks My tasks are transient and only wait on one thing at a time (if not waiting for the scheduler to let them run!). I have my own semantics for creating tasks that incorporate exception propagation. My exception propagation (and return handling) require a scheduler with specific support for them. Net result is that I'd have to wrap all you provide, if not monkey-patching it because it doesn't provide the interfaces to let me wrap it properly. All I want is a global select.poll() object that all the event loops can hook into and each will get a turn to run after each call. Well, that, plus I want it to work around all the platform-specific peculiarities. -- Adam Olsen, aka Rhamphoryncus From rhamph at gmail.com Thu Feb 15 18:40:09 2007 From: rhamph at gmail.com (Adam Olsen) Date: Thu, 15 Feb 2007 10:40:09 -0700 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <20070215164727.25807.382881589.divmod.quotient.22665@ohm> References: <20070215163621.GL6786@v.igoro.us> <20070215164727.25807.382881589.divmod.quotient.22665@ohm> Message-ID: On 2/15/07, Jean-Paul Calderone wrote: > Is the only problem here that this style of development hasn't had been made > visible enough? Perhaps not the only problem, but definitely a big part of it. I looked for such a thing in twisted after python 2.5 came out and was unable to find it. If I had I may not have bothered to update my own microthreads to use python 2.5 (my proof-of-concept was based on real threads). -- Adam Olsen, aka Rhamphoryncus From larry at hastings.org Thu Feb 15 18:48:47 2007 From: larry at hastings.org (Larry Hastings) Date: Thu, 15 Feb 2007 09:48:47 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D48BD9.3040608@v.loewis.de> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> <6a36e7290702150211v38dbb21bpfeb59e2524dec2b0@mail.gmail.com> <45D4842E.9080105@hastings.org> <45D48BD9.3040608@v.loewis.de> Message-ID: <45D49CFF.1030107@hastings.org> Martin v. L?wis wrote: > Now, for these generalized event loops, it may not be possible anymore > to combine all event sources into a single blocking call. Right, that's why my proposal assumed that each disparate event source would need its own thread. > Ah, threads :-( It turns out that you need to invoke GetMessage in the > context of the thread in which the window was created. In a different > thread, you won't get any messages. Oof! I'm embarrassed to have forgotten that. But that's not a fatal problem. It means that on Windows the PerfectReactor must service the blocking GetMessage loop, and these other threads notify the PerfectReactor of new events by sending a message. (Either that, or, it could poll GetMessage and its incoming event queue without ever blocking. But that is obviously suboptimal.) I think I've done this sort of thing before, in fact. Of course, in the absence of any windows, the Windows PerfectReactor could fall back to a mutex. Abstract this inside PerfectReactor and its event sources wouldn't notice the difference. > Integrating with threads might be a solution in some cases, and a > problem in others. You can't assume it is a universal solution. Universal? Yeah, I doubt it too. But perhaps it would be "good enough for nearly all cases". In the cases where it wasn't, it could throw an "I can't listen to that type of event right now" exception, forcing you to fall back to preconfiguring your central reactor by hand. Anyway, like many folks I'm hoping this whole conversation results in establishing basic standard duck-typed conventions--what other languages might call "interfaces"--for event senders and receivers. In which case this non-universal threaded approach would simply be one of several to choose from. I'd be interested to hear about other situations where threading would cause a problem. My suspicion is that Windows is the hard one, and as I've shown that one is solvable. Thanks for the thoughtful reply, /larry/ From dustin at v.igoro.us Thu Feb 15 18:53:13 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Thu, 15 Feb 2007 11:53:13 -0600 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <20070215164727.25807.382881589.divmod.quotient.22665@ohm> References: <20070215163621.GL6786@v.igoro.us> <20070215164727.25807.382881589.divmod.quotient.22665@ohm> Message-ID: <20070215175313.GN6786@v.igoro.us> On Thu, Feb 15, 2007 at 11:47:27AM -0500, Jean-Paul Calderone wrote: > Is the only problem here that this style of development hasn't had been made > visible enough? Yep -- I looked pretty hard about two years ago, and although I haven't been looking for that specifically since, I haven't heard anything about it. The API docs don't provide a good way to find things like this, and the Twisted example tutorial didn't mention it at my last check. So if we have an in-the-field implementation of this style of programming (call it what you will), is it worth *standardizing* that style so that it's the same in Twisted, my library, and anyone else's library that cares to follow the standard? Dustin From nmm1 at cus.cam.ac.uk Thu Feb 15 20:46:59 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Thu, 15 Feb 2007 19:46:59 +0000 Subject: [Python-Dev] generic async io (was: microthreading vs. async io) Message-ID: dustin at v.igoro.us wrote: > > I think this discussion would be facilitated by teasing the first > bullet-point from the latter two: the first deals with async IO, while > the latter two deal with cooperative multitasking. > > It's easy to write a single package that does both, but it's much harder > to write *two* fairly generic packages with a clean API between them, > given the varied platform support for async IO and the varied syntax and > structures (continuations vs. microthreads, in my terminology) for > multitasking. Yet I think that division is exactly what's needed. Hmm. Now, please, people, don't take offence, but I don't know how to phrase this tactfully :-( The 'threading' approach to asynchronous I/O was found to be a BAD IDEA back in the 1970s, was abandoned in favour of separating asynchronous I/O from threading, and God alone knows why it was reinvented - except that most of the people with prior experience had died or retired :-( Let's go back to the days when asynchronous I/O was the norm, and I/O performance critical applications drove the devices directly. In those days, yes, that approach did make sense. But it rapidly ceased to do so with the advent of 'semi-intelligent' devices and the virtualisation of I/O by the operating system. That was in the mid-1970s. Nowadays, ALL devices are semi-intelligent and no system since Unix has allowed applications direct access to devices, except for specialised HPC and graphics. We used to get 90% of theoretical peak performance on mainframes using asynchronous I/O from clean, portable applications, but it was NOT done by treating the I/O as threads and controlling their synchronisation by hand. In fact, quite the converse! It was done by realising that asynchronous I/O and explicit threading are best separated ENTIRELY. There were two main models: Streaming, as in most languages (Fortran, C, Python, but NOT in POSIX). The key properties here are that the transfer boundaries have no significance, only heavyweight synchronisation primitives (open, close etc.) provide any constraints on when data are actually transferred and (for very high performance) buffers are unavailable from when a transfer is started to when it is checked. If copying is acceptable, the last constraint can be dropped. In the simple case, this allows the library/system to reblock and perform transfers asynchronously. In the more advanced case, the application has to use multiple buffering (at least double), but can get full performance without any form of threading. IBM MVT applications used to get up to 90% without hassle in parallel with computation and using only a single thread (well, there was only a single CPU, anyway). The other model is transactions. This has the property that there is a global commit primitive, and the order of transfers is undefined between commits. Inter alia, it means that overlapping transfers are undefined behaviour, whether in a single thread or in multiple threads. BSP uses this model. The MPI-2 design team included a lot of ex-mainframe people and specifies both models. While it is designed for parallel applications, the I/O per se is not controlled like threads. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From dustin at v.igoro.us Thu Feb 15 20:55:50 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Thu, 15 Feb 2007 13:55:50 -0600 Subject: [Python-Dev] generic async io (was: microthreading vs. async io) In-Reply-To: References: Message-ID: <20070215195550.GA26490@v.igoro.us> On Thu, Feb 15, 2007 at 07:46:59PM +0000, Nick Maclaren wrote: > dustin at v.igoro.us wrote: > > > > I think this discussion would be facilitated by teasing the first > > bullet-point from the latter two: the first deals with async IO, while > > the latter two deal with cooperative multitasking. > > > > It's easy to write a single package that does both, but it's much harder > > to write *two* fairly generic packages with a clean API between them, > > given the varied platform support for async IO and the varied syntax and > > structures (continuations vs. microthreads, in my terminology) for > > multitasking. Yet I think that division is exactly what's needed. > > The 'threading' approach to asynchronous I/O was found to be a BAD > IDEA back in the 1970s, was abandoned in favour of separating > asynchronous I/O from threading, and God alone knows why it was > reinvented - except that most of the people with prior experience > had died or retired :-( Knowing the history of something like this is very helpful, but I'm not sure what you mean by this first paragraph. I think I'm most unclear about the meaning of "The 'threading' approach to asynchronous I/O"? Its opposite ("separating asynchronous I/O from threading") doesn't illuminate it much more. Could you elaborate? Dustin From raymond.hettinger at verizon.net Thu Feb 15 20:59:38 2007 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Thu, 15 Feb 2007 11:59:38 -0800 Subject: [Python-Dev] Py2.6 ideas Message-ID: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> * Teach vars() to work with classes defining __slots__. Essentially, __slots__ are just an implementation detail designed to make instances a bit more compact. * Make the docstring writable for staticmethods, classmethods, and properties. We did this for function objects and it worked-out well. * Have staticmethods and classmethods automatically fill-in a docstring from the wrapped function. An editor's tooltips would benefit nicely. * Add a pure python named_tuple class to the collections module. I've been using the class for about a year and found that it greatly improves the usability of tuples as records. http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 * Give builtin types a __name__ attribute so that we have a uniform way of accessing type names. * Enhance doctest with a method that computes updated doctest results. If the only change I make to a matrix suite is to add commas to long numbers in the matrix repr(), then it would be nice to have an automated way to update the matrix output on all the other test cases in the module. * add an optional position argument to heapq.heapreplace to allow an arbitrary element to be updated in-place and then have the heap condition restored. I've now encountered three separate parties trying to figure-out how to do this. From nmm1 at cus.cam.ac.uk Thu Feb 15 21:18:11 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Thu, 15 Feb 2007 20:18:11 +0000 Subject: [Python-Dev] generic async io (was: microthreading vs. async io) Message-ID: dustin at v.igoro.us wrote: > > Knowing the history of something like this is very helpful, but I'm not > sure what you mean by this first paragraph. I think I'm most unclear > about the meaning of "The 'threading' approach to asynchronous I/O"? > Its opposite ("separating asynchronous I/O from threading") doesn't > illuminate it much more. Could you elaborate? I'll try. Sorry about being unclear - it is one of my failings. Here is an example draft of some interfaces: Threading --------- An I/O operation passes a buffer, length, file and action and receives a token back. This token can be queried for completion, waited on and so on, and is cancelled by waiting on it and getting a status back. I.e. it is a thread-like object. This is the POSIX-style operation, and is what I say cannot be made to work effectively. Streaming --------- An I/O operation either writes some data to a stream or reads some data from it; such actions are sequenced within a thread, but not between threads (even if the threads coordinate their I/O). Data written goes into limbo until it is read, and there is no way for a reader to find the block boundaries it was written with or whether data HAS been written. A non-blocking read merely tests if data are ready for reading, which is not the same. There are no positioning operations, and only open, close and POSSIBLY a heavyweight synchronise or rewind (both equivalent to close+open) force written data to be transferred. Think of Fortran sequential I/O without BACKSPACE or C I/O without ungetc/ungetchar/fseek/fsetpos. Transactions ------------ An I/O operation either writes some data to a file or reads some data from it. There is no synchronisation of any form until a commit. If two transfers between a pair of commits overlap (including file length changes), the behaviour is undefined. All I/O includes its own positioning, and no positioning is relative. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From python at rcn.com Thu Feb 15 22:08:36 2007 From: python at rcn.com (Raymond Hettinger) Date: Thu, 15 Feb 2007 13:08:36 -0800 Subject: [Python-Dev] [Python-3000] UserDict revamp References: <00fd01c75128$16a67d60$ea146b0a@RaymondLaptop1> Message-ID: <002701c75145$75ece080$ea146b0a@RaymondLaptop1> > On 2/15/07, Raymond Hettinger wrote: >> I would like to be the one to migrate it to Py3.0. [Steve] > No complaints here. Not that you need my permission of course. ;-) Thanks, I had already started working on this one. Of course, everyone is welcome to contribute. Raymond From joachim.koenig-baltes at emesgarten.de Thu Feb 15 21:15:32 2007 From: joachim.koenig-baltes at emesgarten.de (Joachim Koenig-Baltes) Date: Thu, 15 Feb 2007 21:15:32 +0100 Subject: [Python-Dev] generic async io In-Reply-To: <20070215154945.GK6786@v.igoro.us> References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <45D472E5.6070803@emesgarten.de> <45D47C11.3070108@emesgarten.de> <20070215154945.GK6786@v.igoro.us> Message-ID: <45D4BF64.8090805@emesgarten.de> dustin at v.igoro.us wrote: > I think this discussion would be facilitated by teasing the first > bullet-point from the latter two: the first deals with async IO, while > the latter two deal with cooperative multitasking. > > It's easy to write a single package that does both, but it's much harder > to write *two* fairly generic packages with a clean API between them, > given the varied platform support for async IO and the varied syntax and > structures (continuations vs. microthreads, in my terminology) for > multitasking. Yet I think that division is exactly what's needed. > > Since you asked (I'll assume the check for $0.02 is in the mail), I > think a strictly-async-IO library would offer the following: > > - a sleep queue object to which callables can be added > - wrappers for all/most of the stdlib blocking IO operations which > add the operation to the list of outstanding operations and return > a sleep queue object > - some relatively easy method of extending that for new IO operations > - a poll() function (for multitasking libraries) and a serve_forever() > loop (for asyncore-like uses, where all the action is IO-driven) A centralized approach of wrapping all blocking IO operation in stdlib could only work in pure python applications. What about extensions that integrate e.g. gtk2, gstreamer and other useful libraries that come with their own low level IO. Python is not the right place to solve this problem, and there are so many C-Libraries which tried it, e.g. gnu-pth tries to implement pthreads on a single-threaded OS. But none of these approaches is perfect. E.g. if you want to read 5 bytes from a fd, you can use FIONREAD on a socket and get the number of bytes available from the OS, so you can be sure to not block, but FIONREAD on a normal file fd (e.g. on a NFS mount) will not tell you, how many bytes the OS has prefetched, so you might block, even if you are reading only 1 byte. I think it's best to decide how to do the low level IO for each case in the task. It knows what's it's doing and how to avoid blocking. Therefore I propose to decouple the waiting for a condition/event from the actual blocking operation. And to avoid the blocking, there is no need to reinvent the wheel, the socket module already provides ways to avoid it for network IO and a lot of C libraries exist to do it in a portable way, but none is perfect. And based on these events it's much easier to design a schedular than to write one which also has to do the non blocking IO operations in order to give the tasks the illusion of a blocking operation. The BSD kevent is the most powerful event waiting mechanism with kernel support (as it unifies the waiting on different events on different resources like fd, process, timers, signals) but its API can be emulated to a sudden degree in the other event mechanism like notify on Linux or Niels Provos' libevent. The real showstopper for making the local event waiting easy are the missing coroutines or at least a form of non local goto like setjmp/longjump in C (that's what greenlets provides), remember that yield() only suspends the current function, so every function on the stack must be prepared to handle the yield, even if they are not interested in it (hiding this fact with decorators does not make it better IMO) Joachim From baptiste13 at altern.org Thu Feb 15 22:20:50 2007 From: baptiste13 at altern.org (Baptiste Carvello) Date: Thu, 15 Feb 2007 22:20:50 +0100 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D49CFF.1030107@hastings.org> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> <6a36e7290702150211v38dbb21bpfeb59e2524dec2b0@mail.gmail.com> <45D4842E.9080105@hastings.org> <45D48BD9.3040608@v.loewis.de> <45D49CFF.1030107@hastings.org> Message-ID: >> Ah, threads :-( It turns out that you need to invoke GetMessage in the >> context of the thread in which the window was created. In a different >> thread, you won't get any messages. > > I'd be interested to hear about other situations where threading would > cause a problem. My suspicion is that Windows is the hard one, and as > I've shown that one is solvable. > > I've tried something similar on Linux, with gtk an wx. You can run the gtk main loop in its own thread, but because gtk is not thread safe, you have to grab a mutex everytime you run gtk code outside the thread the mainloop is running in. So you have to surround your calls to the gtk api with calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of course, because they are executed in the main thread... Doable, but not fun. The same goes for wx. Then all hell breaks loose when you try to use both gtk and wx at the same time. That's because on Linux, the wx main loop calls the gtk mainloop behind the scenes. As far as I know, that problem can not be solved from python. So yes that strategy can work, but it's no silver bullet. Cheers, Baptiste From bob at redivi.com Thu Feb 15 22:26:24 2007 From: bob at redivi.com (Bob Ippolito) Date: Thu, 15 Feb 2007 13:26:24 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> <6a36e7290702150211v38dbb21bpfeb59e2524dec2b0@mail.gmail.com> <45D4842E.9080105@hastings.org> <45D48BD9.3040608@v.loewis.de> <45D49CFF.1030107@hastings.org> Message-ID: <6a36e7290702151326m3c8c43b1j25a9b7d7c001e0bb@mail.gmail.com> On 2/15/07, Baptiste Carvello wrote: > >> Ah, threads :-( It turns out that you need to invoke GetMessage in the > >> context of the thread in which the window was created. In a different > >> thread, you won't get any messages. > > > > I'd be interested to hear about other situations where threading would > > cause a problem. My suspicion is that Windows is the hard one, and as > > I've shown that one is solvable. > > > > > I've tried something similar on Linux, with gtk an wx. > > You can run the gtk main loop in its own thread, but because gtk is not thread > safe, you have to grab a mutex everytime you run gtk code outside the thread the > mainloop is running in. So you have to surround your calls to the gtk api with > calls to gtk.threads_enter and gtk.threads_leave. Except for callbacks of > course, because they are executed in the main thread... Doable, but not fun. > > The same goes for wx. Then all hell breaks loose when you try to use both gtk > and wx at the same time. That's because on Linux, the wx main loop calls the gtk > mainloop behind the scenes. As far as I know, that problem can not be solved > from python. > > So yes that strategy can work, but it's no silver bullet. And it's worse on Windows and Mac OS X where some GUI API calls *must* happen on a particular thread or they simply don't work. -bob From jcarlson at uci.edu Thu Feb 15 22:55:31 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 15 Feb 2007 13:55:31 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070215141930.25807.412274186.divmod.quotient.22521@ohm> References: <20070215141930.25807.412274186.divmod.quotient.22521@ohm> Message-ID: <20070215133744.AD62.JCARLSON@uci.edu> Jean-Paul Calderone wrote: > On Thu, 15 Feb 2007 02:36:22 -0700, Andrew Dalke wrote: [snip] > >2) asyncore is > >smaller and easier to understand than Twisted, > > While I hear this a lot, applications written with Twisted _are_ shorter and > contain less irrelevant noise in the form of boilerplate than the equivalent > asyncore programs. This may not mean that Twisted programs are easier to > understand, but it is at least an objectively measurable metric. In my experience, the boilerplate is generally incoming and outgoing buffers. If both had better (optional default) implementations, and perhaps a way of saying "use the default implementations of handle_close, etc.", then much of the boilerplate would vanish. People would likely implement a found_terminator method and be happy. > >and > >3) it was for demo/proof of concept purposes. > >While > >tempting to improve that module I know that Twisted > >has already gone though all the platform-specific crap > >and I don't want to go through it again myself. I don't > >want to write a reactor to deal with GTK, and one for > >OS X, and one for ... > > Now if we can only figure out a way for everyone to benefit from this without > tying too many brains up in knots. :) Whenever I need to deal with these kinds of things (in wxPython specifically), I usually set up a wxTimer to signal asyncore.poll(timeout=0), but I'm lazy, and rarely need significant throughput in my GUI applications. [snip] > >Yet by using the Stackless socket monkeypatch, this > >same code works in an async framework. And the underlying > >libraries have a much larger developer base than Twisted. > >Want NNTP? "import nntplib" Want POP3? "import poplib" > >Plenty of documentation about them too. > > This is going to come out pretty harshly, for which I can only apologize in > advance, but it bears mention. The quality of protocol implementations in the > standard library is bad. As in "not good". Twisted's NNTP support is better > (even if I do say so myself - despite only having been working on by myself, > when I knew almost nothing about Twisted, and having essentially never been > touched since). Twisted's POP3 support is fantastically awesome. Next to > imaplib, twisted.mail.imap4 is a sparkling diamond. And each of these > implements the server end of the protocol as well: you won't find that in the > standard library for almost any protocol. Protocol support is hit and miss. NNTP in Python could be better, but that's not an asyncore issue (being that nntplib isn't implemented using asyncore), that's an "NNTP in Python could be done better" issue. Is it worth someone's time to patch it, or should they just use Twisted? Well, if we start abandoning stdlib modules, "because they can always use Twisted", then we may as well just ship Twisted with Python. - Josiah From ben at redfrontdoor.org Thu Feb 15 23:08:53 2007 From: ben at redfrontdoor.org (Ben North) Date: Thu, 15 Feb 2007 22:08:53 +0000 Subject: [Python-Dev] Wrapping up 'dynamic attribute' discussion Message-ID: <45D4D9F5.1060006@redfrontdoor.org> I've sent an updated version of PEP 363 to the editors, which includes the following summary of the discussion. I hope I've captured the important points, but please let me know if there's something important I've left out or misrepresented. - - - - 8< - - - - Mailing Lists Discussion Initial posting of this PEP in draft form was to python-ideas on 20070209 [2], and the response was generally positive. The PEP was then posted to python-dev on 20070212 [3], and an interesting discussion ensued. A brief summary: Initially, there was reasonable (but not unanimous) support for the idea, although the precise choice of syntax had a more mixed reception. Several people thought the "." would be too easily overlooked, with the result that the syntax could be confused with a method/function call. A few alternative syntaxes were suggested: obj.(foo) obj.[foo] obj.{foo} obj{foo} obj.*foo obj->foo obj<-foo obj at foo obj.[[foo]] with "obj.[foo]" emerging as the preferred one. In this initial discussion, the two-argument form was universally disliked, so it was to be taken out of the PEP. Discussion then took a step back to whether this particular feature provided enough benefit to justify new syntax. As well as requiring coders to become familiar with the new syntax, there would also be the problem of backward compatibility --- code using the new syntax would not run on older pythons. Instead of new syntax, a new "wrapper class" was proposed, with the following specification / conceptual implementation suggested by Martin Loewis: class attrs: def __init__(self, obj): self.obj = obj def __getitem__(self, name): return getattr(self.obj, name) def __setitem__(self, name, value): return setattr(self.obj, name, value) def __delitem__(self, name): return delattr(self, name) def __contains__(self, name): return hasattr(self, name) This was considered a cleaner and more elegant solution to the original problem. The decision was made that the present PEP did not meet the burden of proof for the introduction of new syntax, a view which had been put forward by some from the beginning of the discussion. The wrapper class idea was left open as a possibility for a future PEP. From dalke at dalkescientific.com Thu Feb 15 23:22:28 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Thu, 15 Feb 2007 15:22:28 -0700 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070215160117.25807.36797535.divmod.quotient.22619@ohm> References: <20070215154605.GA5434@localhost.localdomain> <20070215160117.25807.36797535.divmod.quotient.22619@ohm> Message-ID: > On Thu, 15 Feb 2007 10:46:05 -0500, "A.M. Kuchling" wrote: > >It's hard to debug the resulting problem. Which level of the *12* > >levels in the stack trace is responsible for a bug? Which of the *6* > >generic calls is calling the wrong thing because a handler was set up > >incorrectly or the wrong object provided? The code is so 'meta' that > >it becomes effectively undebuggable. On 2/15/07, Jean-Paul Calderone wrote, > I've debugged plenty of Twisted applications. So it's not undebuggable. :) Hence the word "effectively". Or are you offering to be on-call within 5 minutes for anyone wanting to debug code? Not very scalable that. The code I was talking about took me an hour to track down and I could only find the location be inserting a "print traceback" call to figure out where I was. > Application code tends to reside at the bottom of the call stack, so Python's > traceback order puts it right where you're looking, which makes it easy to > find. As I also documented, Twisted tosses a lot of the call stack. Here is the complete and full error message I got: Error: [Failure instance: Traceback (failure with no frames): twisted.internet.error.ConnectionRefusedError: Connection was refused by other side: 22: Invalid argument. ] I wrote the essay at http://www.dalkescientific.com/writings/diary/archive/2006/08/28/levels_of_abstraction.html to, among others, show just how hard it is to figure things out in Twisted. > For any bug which causes something to be set up incorrectly and only > later manifests as a traceback, I would posit that whether there is 1 frame or > 12, you aren't going to get anything useful out of the traceback. I posit that tracebacks are useful. Consider: def blah(): make_network_request("A") make_network_request("B") where "A" and "B" are encoded as part of a HTTP POST payload to the same URI. If there's an error in the network connection - eg, the implementation for "B" on the server dies so the connection closes w/o a response - then knowning that the call for "B" failed and not "A" is helpful during debugging. The low level error message cannot report that. Yes, I could put my own try blocks around everything and contextualize all of the error messages so they are semantically correct for the given level of code. But that I would be a lot of code, hard to test, and not cost effective. > Standard practice here is just to make exception text informative, > I think, If you want to think of it as "exception text" then consider that the stack trace is "just" more text for the message. >but this is another general problem with Python programs > and event loops, not one specific to either Twisted itself or the > particular APIs Twisted exposes. The thread is "Twisted Isn't Specific", as a branch of a discussion on microthreads in the standard library. As someone experimenting with Stackless and how it can be used on top of an async library I feel competent enough to comment on the latter topic. As someone who has seen the reverse Bozo bit set by Twisted people on everyone who makes the slightest comment towards using any other async library, and has documented evidence as to just why one might do so, I also feel competent enough to comment on the branch topic. My belief is that there are too many levels of generiticity in Twisted. This makes is hard for an outsider to come in and use the system. By "use" I include 1) understanding how the parts go together, 2) diagnose problems and 3) adding new features that Twisted doesn't support. Life is trade offs. A Twisted trade off is generiticity at the cost of understandability. Remember, this is all my belief, backed by examples where I did try to understand. My experience with other networking packages have been much easier, including with asyncore and Allegra. They are not as general purpose, but it's hard for me to believe the extra layers in Twisted are needed to get that extra whatever functionality. My other belief is that async programming is hard for most people, who would rather do "normal" programming instead of "inside-out" programming. Because of this 2nd belief I am interested in something like Stackless on top of an async library. > As a personal anecdote, I've never once had to chase a bug through any of the > 6 "generic calls" singled out. I can't think of a case where I've helped any > one else who had to do this, either. That part of Twisted is very old, it is > _very_ close to bug-free, and application code doesn't have very much control > over it at all. Perhaps in order to avoid scaring people, there should be a > way to elide frames from a traceback (I don't much like this myself, I worry > about it going wrong and chopping out too much information, but I have heard > other people ask for it)? Even though I said some of this earlier I'll elaborate for clarification. The specific bug I was tracking down had *no* traceback. There was nothing to elide. Because there was no traceback I couldn't figure out where the error came from. I had to use the error message text to find the error class, from there modify the source code to generate a traceback, then work up the stack to find the code which had the actual error. Here is the tail end of the traceback. File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/tcp.py", line 535, in doConnect self.failIfNotConnected(error.getConnectError((connectResult, os.strerror(connectResult)))) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/error.py", line 160, in getConnectError return klass(number, string) File "/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ site-packages/twisted/internet/error.py", line 105, in __init__ traceback.print_stack() You can see the actual error occured at [-3] in the stack where the os.strerror() was. One layer of genericity is mapping OS-level error codes as integers into error classes, one class per integer. You previously said that my problem was resolved thusly: Note that we ended up /not/ changing the error number in the case you encountered. We changed the connection setup code to handle the unexpected behavior on OS X. :) This means that at least someone did help someone track a bug which were affected by those levels of abstraction. BTW, the ticket is at http://twistedmatrix.com/trac/ticket/2022 and fix was r18064. The final solution was to # doConnect gets called twice. The first time we actually need to # start the connection attempt. The second time we don't really # want to (SO_ERROR above will have taken care of any errors, and if # it reported none, the mere fact that doConnect was called again is # sufficient to indicate that the connection has succeeded), but it # is not /particularly/ detrimental to do so. This should get # cleaned up some day, though. and has nothing to do with changing the error number. Twisted was using the 2nd error code when it should have used the 1st. That was the reason for my getting the "wrong" number. It was the right number for a different check for an error. Note that last comment -- the double call to doConnect was the problem, and a source of my confusion. It remain, just neutered. Also note that that patch included removing code from error.py errno.ENETUNREACH: NoRouteError, errno.ECONNREFUSED: ConnectionRefusedError, errno.ETIMEDOUT: TCPTimedOutError, - # for FreeBSD - might make other unices in certain cases - # return wrong exception, alas - errno.EINVAL: ConnectionRefusedError, which was part the mixup that gave me problems. This definitely was an error in one of those levels of abstraction. It was a bad fix earlier "fixed" by incorrectly mapping an error code, probably on the justification of there being an OS error rather than a Twisted implementation problem. But that's just a wild guess based solely on seeing other fixes of that type. To bring this back into python-dev, .... none of this is a topic for python-dev. I'm reacting to what I perceive as a overly territorial response that occurs nearly every time the words "Twisted", "asynchronous I/O", "reactor" or "main event loop" is uttered. I think using microthreads/stackless/... makes an interesting and useful alternative to the Twisted approach, including different ways to structure the main event loop. I think anyone who's been involved with Python and on this list knows the work Twisted has done to understand platform problems, and needs at most a hint to look at Twisted for insight. Though I feel that such insight is obscured. That said, I resign from this thread and I'll do additional responses in private mail. Andrew dalke at dalkescientific.com From ben at redfrontdoor.org Thu Feb 15 23:24:46 2007 From: ben at redfrontdoor.org (Ben North) Date: Thu, 15 Feb 2007 22:24:46 +0000 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax Message-ID: <45D4DDAE.9010806@redfrontdoor.org> glyph at divmod.com wrote: > > > I really, really wish that every feature proposal for Python had to meet > > > some burden of proof Ben North wrote: > > This is what I understood the initial posting to python-ideas to be > > about. glyph at divmod.com wrote: > I'm suggesting that the standards of the community in _evaluating_ to > the proposals should be clearer Perhaps I didn't need to take your initial comments personally then, sorry :-) I do see what you're pointing out: the later part of the "dynamic attribute" discussion was where the question of whether python really needs new syntax for this was addressed, and the outcome made the earlier discussion of "x.[y]" vs "x.(y)" vs "x.{y}" vs "x->y" etc. irrelevant. Ben. From guido at python.org Fri Feb 16 00:03:32 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 15 Feb 2007 15:03:32 -0800 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> Message-ID: On 2/15/07, Raymond Hettinger wrote: > * Add a pure python named_tuple class to the collections module. I've been > using the class for about a year and found that it greatly improves the > usability of tuples as records. > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 Hm, but why would they still have to be tuples? Why not just have a generic 'record' class? > * Give builtin types a __name__ attribute so that we have a uniform way of > accessing type names. This already exists, unless I misunderstand you: Python 2.2.3+ (#94, Jun 4 2003, 08:24:18) [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> int.__name__ 'int' >>> > * Enhance doctest with a method that computes updated doctest results. If the > only change I make to a matrix suite is to add commas to long numbers in the > matrix repr(), then it would be nice to have an automated way to update the > matrix output on all the other test cases in the module. What I could have used for Py3k (specifically for the 2to3 transformer): mods to the DocTestParser class that allow you to exactly reproduce the input string from the return value of parse(). Unrelated, I'd like the tokenize module to be more easily extensible. E.g. I'd like to add new tokens, and I'd like to be able to change its whitespace handling. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tdelaney at avaya.com Fri Feb 16 00:12:09 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Fri, 16 Feb 2007 10:12:09 +1100 Subject: [Python-Dev] Py2.6 ideas Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> Guido van Rossum wrote: > On 2/15/07, Raymond Hettinger wrote: >> * Add a pure python named_tuple class to the collections module. >> I've been using the class for about a year and found that it greatly >> improves the usability of tuples as records. >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 > > Hm, but why would they still have to be tuples? Why not just have a > generic 'record' class? Hmm - possibilities. "record" definitely has greater connotations of heterogeneous elements than "tuple", which would put paid to the constant arguments that "a tuple is really just an immutable list". list - primarily intended for homogeneous elements record - primarily intended for heterogeneous elements, elements are (optionally?) named and have mutable and immutable versions of each. Maybe the current list syntax would then continue to create a mutable list, and the current tuple syntax would create an immutable record (with no element names) i.e. the current tuple. Tim Delaney From greg.ewing at canterbury.ac.nz Fri Feb 16 00:32:55 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Feb 2007 12:32:55 +1300 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D40A35.2020502@v.loewis.de> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> <45D40A35.2020502@v.loewis.de> Message-ID: <45D4EDA7.6020201@canterbury.ac.nz> Martin v. L?wis wrote: > That is insufficient. The gtk main loop has more input > sources than just the connection to X: > - timers > - idle handlers > - child handlers > - additional file descriptors > - a generalzed 'event source' When gtk is not the central event mechanism, there's no need to use the gtk event loop for these things -- you can just use the central event mechanism directly. The pygtk APIs for setting these up can redirect them to the appropriate place, to accommodate existing code that uses the gtk event loop for them. -- Greg From skip at pobox.com Fri Feb 16 00:41:51 2007 From: skip at pobox.com (skip at pobox.com) Date: Thu, 15 Feb 2007 17:41:51 -0600 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> Message-ID: <17876.61375.903893.846784@montanaro.dyndns.org> >> Hm, but why would they still have to be tuples? Why not just have a >> generic 'record' class? Tim> Hmm - possibilities. "record" definitely has greater connotations Tim> of heterogeneous elements than "tuple", which would put paid to the Tim> constant arguments that "a tuple is really just an immutable list". (What do you mean by "... put paid ..."? It doesn't parse for me.) Based on posts the current thread in c.l.py with the improbable subject "f---ing typechecking", lots of people refuse to believe tuples are anything other than immutable lists. Skip From greg.ewing at canterbury.ac.nz Fri Feb 16 00:40:54 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Feb 2007 12:40:54 +1300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <20070215093132.GA28863@phd.pp.ru> References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> <20070214162758.GB25564@phd.pp.ru> <20070215093132.GA28863@phd.pp.ru> Message-ID: <45D4EF86.3040709@canterbury.ac.nz> Oleg Broytmann wrote: > > Given that they say "a camel is a horse designed by a committee" > > BTW, camels are very suited for their environments... The quote is actually "a camel is a *racehorse* designed by a committee". Camels are very good at surviving in the desert, but not so good at winning a horse race (not camel race). Which is the point of the saying. -- Greg From tdelaney at avaya.com Fri Feb 16 00:50:24 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Fri, 16 Feb 2007 10:50:24 +1100 Subject: [Python-Dev] Py2.6 ideas Message-ID: <2773CAC687FD5F4689F526998C7E4E5F074465@au3010avexu1.global.avaya.com> skip at pobox.com wrote: > >> Hm, but why would they still have to be tuples? Why not just > have a >> generic 'record' class? > > Tim> Hmm - possibilities. "record" definitely has greater > connotations Tim> of heterogeneous elements than "tuple", which > would put paid to the Tim> constant arguments that "a tuple is > really just an immutable list". > > (What do you mean by "... put paid ..."? It doesn't parse for me.) > Based on posts the current thread in c.l.py with the improbable > subject "f---ing typechecking", lots of people refuse to believe > tuples are anything other than immutable lists. Sorry - "put paid to" means "to finish" ... http://www.phrases.org.uk/meanings/293200.html That thread is a perfect example why I think a "record" type should be standard in python, and "tuple" should be deprecated (and removed in 3.0). Instead, have mutable and immutable lists, and mutable and immutable records. You could add a mutable list and an immutable list (resulting always in a new mutable list I think). You could *not* add two records together (even if neither had named elements). Cheers, Tim Delaney From tdelaney at avaya.com Fri Feb 16 00:51:46 2007 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Fri, 16 Feb 2007 10:51:46 +1100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1EC9E@au3010avexu1.global.avaya.com> Greg Ewing wrote: > Oleg Broytmann wrote: > >>> Given that they say "a camel is a horse designed by a committee" >> >> BTW, camels are very suited for their environments... > > The quote is actually "a camel is a *racehorse* designed by a > committee". Camels are very good at surviving in the desert, but not > so good at winning a horse race (not camel race). Which is the point > of the saying. Speaking of which, have you ever seen a camel race? Those things go *fast* ... I think we're getting way too off-topic now ;) Tim Delaney From greg.ewing at canterbury.ac.nz Fri Feb 16 00:52:15 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Feb 2007 12:52:15 +1300 Subject: [Python-Dev] Summary: rejection of 'dynamic attribute' syntax In-Reply-To: <200702152227.47672.anthony@interlink.com.au> References: <1171458253.45d308ce00f09@imp.hosting365.ie> <45D3B45B.3080704@canterbury.ac.nz> <200702152227.47672.anthony@interlink.com.au> Message-ID: <45D4F22F.8040503@canterbury.ac.nz> Anthony Baxter wrote: > Unless the fans are perfectly balanced, small changes in gravity are > going to affect the rate at which they spin. So I guess the > position of the moon will affect it :-) A standard gravitational field could also be important to eliminate relativistic effects. So we need to standardise latitude/longitude/altitude, time of year and phase of moon. Better check atmospheric pressure and humidity, too, just to be on the safe side. -- Greg From greg.ewing at canterbury.ac.nz Fri Feb 16 01:28:01 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 16 Feb 2007 13:28:01 +1300 Subject: [Python-Dev] generic async io (was: microthreading vs. async io) In-Reply-To: References: Message-ID: <45D4FA91.6000703@canterbury.ac.nz> Nick Maclaren wrote: > Threading > --------- > > An I/O operation passes a buffer, length, file and action and receives a > token back. You seem to be using the word "threading" in a completely different way than usual here, which may be causing some confusion. -- Greg From anthony at interlink.com.au Fri Feb 16 01:35:51 2007 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri, 16 Feb 2007 11:35:51 +1100 Subject: [Python-Dev] Wrapping up 'dynamic attribute' discussion In-Reply-To: <45D4D9F5.1060006@redfrontdoor.org> References: <45D4D9F5.1060006@redfrontdoor.org> Message-ID: <200702161135.53911.anthony@interlink.com.au> On Friday 16 February 2007 09:08, Ben North wrote: > Instead of new syntax, a new "wrapper class" was proposed, > with the following specification / conceptual implementation > suggested by Martin Loewis: > > ...snip... > > This was considered a cleaner and more elegant solution to > the original problem. The decision was made that the present PEP > did not meet the burden of proof for the introduction of new > syntax, a view which had been put forward by some from the > beginning of the discussion. The wrapper class idea was left > open as a possibility for a future PEP. A good first step would be to contribute something like this to the Python Cookbook, if it isn't already there. -- Anthony Baxter It's never too late to have a happy childhood. From steve at holdenweb.com Fri Feb 16 01:44:55 2007 From: steve at holdenweb.com (Steve Holden) Date: Thu, 15 Feb 2007 19:44:55 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D4EF86.3040709@canterbury.ac.nz> References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> <20070214162758.GB25564@phd.pp.ru> <20070215093132.GA28863@phd.pp.ru> <45D4EF86.3040709@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Oleg Broytmann wrote: > >>> Given that they say "a camel is a horse designed by a committee" >> BTW, camels are very suited for their environments... > > The quote is actually "a camel is a *racehorse* designed by a committee". > Camels are very good at surviving in the desert, but not so good at > winning a horse race (not camel race). Which is the point of the saying. > As far as I know Sir Alec Issigonis, the inventor of the Mini (the car, not the Mac Mini) said this, and he used "horse", not "racehorse". The point of the saying is that a camel has properties that are completely unnecessary in a horse, such as the ability to travel many days without water. He was saying that committees tend to over-specify and add redundant features rather than designing strictly for purpose. A bit like Python 3.0 ;-) regards Steve -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From amk at amk.ca Fri Feb 16 01:51:36 2007 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 15 Feb 2007 19:51:36 -0500 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <17876.61375.903893.846784@montanaro.dyndns.org> References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <17876.61375.903893.846784@montanaro.dyndns.org> Message-ID: <20070216005136.GA28724@andrew-kuchlings-computer.local> On Thu, Feb 15, 2007 at 05:41:51PM -0600, skip at pobox.com wrote: > Tim> Hmm - possibilities. "record" definitely has greater connotations > Tim> of heterogeneous elements than "tuple", which would put paid to the > Tim> constant arguments that "a tuple is really just an immutable list". > > (What do you mean by "... put paid ..."? It doesn't parse for me.) "Put paid" usually means "to finish off"; Tim is saying this would finish the constant arguments that &c &c... --amk From python at rcn.com Fri Feb 16 02:12:30 2007 From: python at rcn.com (Raymond Hettinger) Date: Thu, 15 Feb 2007 17:12:30 -0800 Subject: [Python-Dev] Py2.6 ideas References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> Message-ID: <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> [Raymond Hettinger] >>> * Add a pure python named_tuple class to the collections module. >>> I've been using the class for about a year and found that it greatly >>> improves the usability of tuples as records. >>> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 [Delaney, Timothy] > Hmm - possibilities. "record" definitely has greater connotations of > heterogeneous elements than "tuple", which would put paid to the > constant arguments that "a tuple is really just an immutable list". No need to go so widely off-track. The idea is to have an efficient type that is directly substitutable for tuples but is a bit more self-descriptive. I like to have the doctest result cast at NamedTuple('TestResults failed attempted). The repr of that result looks like TestResult(failed=0, attempted=15) but is still accessible as a tuple and passes easily into other functions that expect a tuple. This sort of thing would be handly for things like os.stat(). http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 Raymond From rasky at develer.com Fri Feb 16 02:35:33 2007 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 16 Feb 2007 02:35:33 +0100 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> Message-ID: On 15/02/2007 20.59, Raymond Hettinger wrote: > * Add a pure python named_tuple class to the collections module. I've been > using the class for about a year and found that it greatly improves the > usability of tuples as records. > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 +1 from me too, I've been using a class with the same name and semantic (though an inferior implementation) for almost two years now, with great benefits. As suggested in the cookbook comment, please consider changing the semantic of the generated constructor so that it accepts a single iterable positional arguments (or keyword arguments). This matches tuple() (and other containers) in behaviour, and makes it easier to substitute existing uses with named tuples. -- Giovanni Bajo From exarkun at divmod.com Fri Feb 16 04:47:42 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Thu, 15 Feb 2007 22:47:42 -0500 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070215133744.AD62.JCARLSON@uci.edu> Message-ID: <20070216034742.25807.2085565503.divmod.quotient.23293@ohm> On Thu, 15 Feb 2007 13:55:31 -0800, Josiah Carlson wrote: > >Jean-Paul Calderone wrote: > [snip] >> >> Now if we can only figure out a way for everyone to benefit from this without >> tying too many brains up in knots. :) > >Whenever I need to deal with these kinds of things (in wxPython >specifically), I usually set up a wxTimer to signal >asyncore.poll(timeout=0), but I'm lazy, and rarely need significant >throughput in my GUI applications. And I guess you also don't mind that on OS X this is often noticably broken? :) > [snip] > >Protocol support is hit and miss. NNTP in Python could be better, but >that's not an asyncore issue (being that nntplib isn't implemented using >asyncore), that's an "NNTP in Python could be done better" issue. Is it >worth someone's time to patch it, or should they just use Twisted? Well, >if we start abandoning stdlib modules, "because they can always use >Twisted", then we may as well just ship Twisted with Python. > We could always replace the stdlib modules with thin compatibility layers based on the Twisted protocol implementations. It's trivial to turn an asynchronous API into a synchronous one. I think you are correct in marking this an unrelated issue, though. Jean-Paul From veloso at verylowsodium.com Fri Feb 16 05:00:01 2007 From: veloso at verylowsodium.com (Greg Falcon) Date: Thu, 15 Feb 2007 23:00:01 -0500 Subject: [Python-Dev] Wrapping up 'dynamic attribute' discussion In-Reply-To: <200702161135.53911.anthony@interlink.com.au> References: <45D4D9F5.1060006@redfrontdoor.org> <200702161135.53911.anthony@interlink.com.au> Message-ID: <3cdcefb80702152000k429f9607p50a02775c457eb3e@mail.gmail.com> On 2/15/07, Anthony Baxter wrote: > On Friday 16 February 2007 09:08, Ben North wrote: > > The wrapper class idea was left > > open as a possibility for a future PEP. > > A good first step would be to contribute something like this to the > Python Cookbook, if it isn't already there. I could not find such a class in the cookbook. (That's not to say there's not one there that I missed.) Because I think attrview() should happen, I submitted a recipe to the Python Cookbook. While it awaits editor approval, I have it posted at http://www.verylowsodium.com/attrview.py . One possibly controversial design choice here: since there is no guaranteed way to enumerate attributes in the face of __getattr__ and friends, my version of attrview() does not provide iteration or any other operation that assumes object attributes can be enumerated. Therefore, it technically does not implement a mapping. Once Python grows a __dir__ special method, it's possible I could be convinced this is the wrong choice, but I'm not sure of that. Greg F From jcarlson at uci.edu Fri Feb 16 07:48:03 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 15 Feb 2007 22:48:03 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070216034742.25807.2085565503.divmod.quotient.23293@ohm> References: <20070215133744.AD62.JCARLSON@uci.edu> <20070216034742.25807.2085565503.divmod.quotient.23293@ohm> Message-ID: <20070215210955.AD67.JCARLSON@uci.edu> Jean-Paul Calderone wrote: > > On Thu, 15 Feb 2007 13:55:31 -0800, Josiah Carlson wrote: > > > >Jean-Paul Calderone wrote: > > [snip] > >> > >> Now if we can only figure out a way for everyone to benefit from this without > >> tying too many brains up in knots. :) > > > >Whenever I need to deal with these kinds of things (in wxPython > >specifically), I usually set up a wxTimer to signal > >asyncore.poll(timeout=0), but I'm lazy, and rarely need significant > >throughput in my GUI applications. > > And I guess you also don't mind that on OS X this is often noticably broken? > :) I don't own a Mac, and so far, of the perhaps dozen or so Mac users of the software that does this, I've heard no reports of it being broken. From what I understand, wxTimers work on all supported platforms (which includes OSX), and if asyncore.poll() is broken on Macs, then someone should file a bug report. If it's asyncore's fault, assign it to me, otherwise someone with Mac experience needs to dig into it. > > [snip] > >Protocol support is hit and miss. NNTP in Python could be better, but > >that's not an asyncore issue (being that nntplib isn't implemented using > >asyncore), that's an "NNTP in Python could be done better" issue. Is it > >worth someone's time to patch it, or should they just use Twisted? Well, > >if we start abandoning stdlib modules, "because they can always use > >Twisted", then we may as well just ship Twisted with Python. > > We could always replace the stdlib modules with thin compatibility layers > based on the Twisted protocol implementations. It's trivial to turn an > asynchronous API into a synchronous one. I think you are correct in marking > this an unrelated issue, though. If the twisted folks (or anyone else) want to implement a "shim" that pretends to be nntplib, it's their business whether calling twisted.internet.monkeypatch.nntplib() does what the name suggests. :) That is to say, I don't believe anyone would be terribly distraught if there was an easy way to use Twisted without drinking the kool-aid. Then again, I do believe that it makes sense to patch the standard library whenever possible - if Twisted has better parsing of nntp, smtp, pop3, imap4, etc. responses, then perhaps we should get the original authors to sign a PSF contributor agreement, and we could translate whatever is better. - Josiah From martin at v.loewis.de Fri Feb 16 07:48:16 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Feb 2007 07:48:16 +0100 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D49CFF.1030107@hastings.org> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> <6a36e7290702150211v38dbb21bpfeb59e2524dec2b0@mail.gmail.com> <45D4842E.9080105@hastings.org> <45D48BD9.3040608@v.loewis.de> <45D49CFF.1030107@hastings.org> Message-ID: <45D553B0.1070208@v.loewis.de> Larry Hastings schrieb: > Oof! I'm embarrassed to have forgotten that. But that's not a fatal > problem. It means that on Windows the PerfectReactor must service the > blocking GetMessage loop, and these other threads notify the > PerfectReactor of new events by sending a message. ... > I'd be interested to hear about other situations where threading would > cause a problem. My suspicion is that Windows is the hard one, and as > I've shown that one is solvable. As others have reported: if you have a gtk reactor and a wx reactor to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), your approach still won't work. They all do GetMessage behind the scenes (sp?), yet you would run them in multiple threads, so they get their own messages. Plus they are not thread-safe. Speaking of which: Tcl has a "multiple interpreter" model where each interpreter is single-threaded. You can build Tcl either single- or multi-threaded; for the multi-threaded Tcl, you still can't share data across threads directly, but instead, you create an interpreter per thread. Then, all operations on that interpreter must occur in this thread, including the reactor calls. So if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl even loop in the Tcl thread, which is normally the main thread (in Tkinter, it's the thread where you originally created the tkapp object). OTOH, PerfectReactor may also need to run some Win32 event loop in the same thread. Regards, Martin From martin at v.loewis.de Fri Feb 16 08:05:09 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Feb 2007 08:05:09 +0100 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> Message-ID: <45D557A5.3040804@v.loewis.de> Raymond Hettinger schrieb: > No need to go so widely off-track. The idea is to have an efficient type that > is directly substitutable for tuples but is a bit more self-descriptive. I like > to have the doctest result cast at NamedTuple('TestResults failed attempted). > The repr of that result looks like TestResult(failed=0, attempted=15) but is > still accessible as a tuple and passes easily into other functions that expect a > tuple. This sort of thing would be handly for things like os.stat(). > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 I'd like to repeat Guido's question: Why does this still need to support the tuple interface (i.e. indexed access)? I'm not (anymore) sure that you are aware that the os.stat result *already* has named fields, in addition to the indexed access. However, the indexed access is deprecated, and only preserved for backwards compatibility. So why would a new type be handy for os.stat? And, if it's not for os.stat, what other uses does it have? Regards, Martin From jcarlson at uci.edu Fri Feb 16 08:45:25 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 15 Feb 2007 23:45:25 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D553B0.1070208@v.loewis.de> References: <45D49CFF.1030107@hastings.org> <45D553B0.1070208@v.loewis.de> Message-ID: <20070215233930.AD6E.JCARLSON@uci.edu> "Martin v. L?wis" wrote: > > Larry Hastings schrieb: > > Oof! I'm embarrassed to have forgotten that. But that's not a fatal > > problem. It means that on Windows the PerfectReactor must service the > > blocking GetMessage loop, and these other threads notify the > > PerfectReactor of new events by sending a message. > ... > > I'd be interested to hear about other situations where threading would > > cause a problem. My suspicion is that Windows is the hard one, and as > > I've shown that one is solvable. > > As others have reported: if you have a gtk reactor and a wx reactor > to support, in addition to a "regular" Win32 reactor (e.g. PythonWin), > your approach still won't work. They all do GetMessage behind the > scenes (sp?), yet you would run them in multiple threads, so they > get their own messages. Plus they are not thread-safe. At least on the wx side of things, there exist ways of scheduling calls to be executed inside its event dispatching loop (wx.CallAfter, wx.FutureCall, wx.PostEvent), which allows for worker threads to hand data back to the main thread (the creator of the wx.App instance, like Tk). I would be surprised if the other toolkits didn't have something similar, couldn't be manipulated into using Queues to communicate in some way, or couldn't be manipulated into directly sending a message on win32. - Josiah From ncoghlan at gmail.com Fri Feb 16 09:45:16 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Feb 2007 18:45:16 +1000 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <45D557A5.3040804@v.loewis.de> References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> <45D557A5.3040804@v.loewis.de> Message-ID: <45D56F1C.1070901@gmail.com> Martin v. L?wis wrote: > Raymond Hettinger schrieb: >> No need to go so widely off-track. The idea is to have an efficient type that >> is directly substitutable for tuples but is a bit more self-descriptive. I like >> to have the doctest result cast at NamedTuple('TestResults failed attempted). >> The repr of that result looks like TestResult(failed=0, attempted=15) but is >> still accessible as a tuple and passes easily into other functions that expect a >> tuple. This sort of thing would be handly for things like os.stat(). >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 > > I'd like to repeat Guido's question: Why does this still need to support > the tuple interface (i.e. indexed access)? So that it remains interoperable with existing libraries that expect a tuple? Otherwise you'd be casting (and copying) every time you needed to pass it to something that used indexed access. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From nmm1 at cus.cam.ac.uk Fri Feb 16 09:58:11 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Fri, 16 Feb 2007 08:58:11 +0000 Subject: [Python-Dev] generic async io (was: microthreading vs. async io) Message-ID: Greg Ewing wrote: > > > An I/O operation passes a buffer, length, file and action and receives a > > token back. > > You seem to be using the word "threading" in a completely > different way than usual here, which may be causing some > confusion. Not really, though I may have been unclear again. Here is why that approach is best regarded as a threading concept: Perhaps the main current approach to using threads to implement asynchronous I/O operates by the main threads doing just that, and the I/O threads transferring the data synchronously. The reason that a token is needed is to avoid a synchronous data copy that blocks both threads. My general point is that all experience is that asynchronous I/O is best done by separating it completely from threads, and defining a proper asynchronous but NOT threaded interface. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From phd at phd.pp.ru Fri Feb 16 10:02:24 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Fri, 16 Feb 2007 12:02:24 +0300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D4EF86.3040709@canterbury.ac.nz> References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> <20070214162758.GB25564@phd.pp.ru> <20070215093132.GA28863@phd.pp.ru> <45D4EF86.3040709@canterbury.ac.nz> Message-ID: <20070216090223.GE13248@phd.pp.ru> On Fri, Feb 16, 2007 at 12:40:54PM +1300, Greg Ewing wrote: > The quote is actually "a camel is a *racehorse* designed by a committee". > Camels are very good at surviving in the desert, but not so good at > winning a horse race (not camel race). Which is the point of the saying. That changes the meaning, but... have you ever tried to ride a horse designed by a group of Clever Individuals loosely connected by email? ;) I am afraid of even thinking of its ugliness and speed. (-: I think a committee is better than nothing, and I believe CP4E has been dropped from the agenda. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From martin at v.loewis.de Fri Feb 16 10:09:00 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Feb 2007 10:09:00 +0100 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <45D56F1C.1070901@gmail.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> <45D557A5.3040804@v.loewis.de> <45D56F1C.1070901@gmail.com> Message-ID: <45D574AC.3060608@v.loewis.de> Nick Coghlan schrieb: >> I'd like to repeat Guido's question: Why does this still need to >> support the tuple interface (i.e. indexed access)? > > So that it remains interoperable with existing libraries that expect a > tuple? Otherwise you'd be casting (and copying) every time you needed to > pass it to something that used indexed access. Can you give a few example, for libraries where this isn't already done? Regards, Martin From ncoghlan at gmail.com Fri Feb 16 10:28:39 2007 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 16 Feb 2007 19:28:39 +1000 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <45D574AC.3060608@v.loewis.de> References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> <45D557A5.3040804@v.loewis.de> <45D56F1C.1070901@gmail.com> <45D574AC.3060608@v.loewis.de> Message-ID: <45D57947.70704@gmail.com> Martin v. L?wis wrote: > Nick Coghlan schrieb: >>> I'd like to repeat Guido's question: Why does this still need to >>> support the tuple interface (i.e. indexed access)? >> >> So that it remains interoperable with existing libraries that expect a >> tuple? Otherwise you'd be casting (and copying) every time you needed >> to pass it to something that used indexed access. > > Can you give a few example, for libraries where this isn't already done? I don't have any specific examples of that, no - that's why I phrased it as a question. However, another aspect that occurred to me is that inheriting from tuple has significant practical benefits in terms of speed and memory consumption, at which point it doesn't seem worthwhile to *remove* the indexing capability. I suppose you *could* write a completely new C-level record class, but given that Raymond's NamedTuple class gets good performance from a Python implementation, rewriting it in C seems like wasted effort. Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From eopadoan at altavix.com Fri Feb 16 12:30:40 2007 From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan) Date: Fri, 16 Feb 2007 09:30:40 -0200 Subject: [Python-Dev] [Python-3000] UserDict revamp In-Reply-To: <002701c75145$75ece080$ea146b0a@RaymondLaptop1> References: <00fd01c75128$16a67d60$ea146b0a@RaymondLaptop1> <002701c75145$75ece080$ea146b0a@RaymondLaptop1> Message-ID: [Steve] > No complaints here. Not that you need my permission of course. ;-) Same here, obviously. [Raymond] > Thanks, I had already started working on this one. > Of course, everyone is welcome to contribute. Ok, you can count on that. -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt From eopadoan at altavix.com Fri Feb 16 12:40:13 2007 From: eopadoan at altavix.com (Eduardo "EdCrypt" O. Padoan) Date: Fri, 16 Feb 2007 09:40:13 -0200 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <20070216090223.GE13248@phd.pp.ru> References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> <20070214162758.GB25564@phd.pp.ru> <20070215093132.GA28863@phd.pp.ru> <45D4EF86.3040709@canterbury.ac.nz> <20070216090223.GE13248@phd.pp.ru> Message-ID: > I think a committee is better than nothing, and I believe CP4E has > been dropped from the agenda. The general CP4E idea is part of the "General Pythonic Ideal", whatever it may be :P -- EduardoOPadoan (eopadoan->altavix::com) Bookmarks: http://del.icio.us/edcrypt From martin at v.loewis.de Fri Feb 16 13:38:04 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Feb 2007 13:38:04 +0100 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <45D57947.70704@gmail.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> <45D557A5.3040804@v.loewis.de> <45D56F1C.1070901@gmail.com> <45D574AC.3060608@v.loewis.de> <45D57947.70704@gmail.com> Message-ID: <45D5A5AC.5000004@v.loewis.de> Nick Coghlan schrieb: > However, another aspect that occurred to me is that inheriting from > tuple has significant practical benefits in terms of speed and memory > consumption, at which point it doesn't seem worthwhile to *remove* the > indexing capability. I'm not so sure that inheriting from tuple, and giving it named fields, has significant speed and memory benefits. In particular for the memory benefits, you can use __slots__ to achieve the same effects, and more efficiently so (because it you don't store the tuple length). As for speed, I would have to see measurements to be convinced it is faster. > I suppose you *could* write a completely new C-level record class, but > given that Raymond's NamedTuple class gets good performance from a > Python implementation, rewriting it in C seems like wasted effort. It wouldn't necessarily be rewriting: In the C API, you have already the PyStructSequence machinery (see posixmodule.c:stat_result_fields for an example). It's just that this machinery isn't available to Python code, yet, and no alternative convenience library is, either (other than using __slots__, which won't directly give indexed access). Regards, Martin From martin at v.loewis.de Fri Feb 16 13:42:54 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Feb 2007 13:42:54 +0100 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <20070216090223.GE13248@phd.pp.ru> References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> <20070214162758.GB25564@phd.pp.ru> <20070215093132.GA28863@phd.pp.ru> <45D4EF86.3040709@canterbury.ac.nz> <20070216090223.GE13248@phd.pp.ru> Message-ID: <45D5A6CE.8080505@v.loewis.de> Oleg Broytmann schrieb: > That changes the meaning, but... have you ever tried to ride a horse > designed by a group of Clever Individuals loosely connected by email? ;) I > am afraid of even thinking of its ugliness and speed. (-: > I think a committee is better than nothing, and I believe CP4E has > been dropped from the agenda. Ah, this passive voice again, and again the assumption that there is an agenda of python-dev. Regards, Martin From phd at phd.pp.ru Fri Feb 16 13:54:09 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Fri, 16 Feb 2007 15:54:09 +0300 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D5A6CE.8080505@v.loewis.de> References: <45CFAA81.5040906@redfrontdoor.org> <20070213110023.GB11163@phd.pp.ru> <20070214162758.GB25564@phd.pp.ru> <20070215093132.GA28863@phd.pp.ru> <45D4EF86.3040709@canterbury.ac.nz> <20070216090223.GE13248@phd.pp.ru> <45D5A6CE.8080505@v.loewis.de> Message-ID: <20070216125409.GA13220@phd.pp.ru> On Fri, Feb 16, 2007 at 01:42:54PM +0100, "Martin v. L?wis" wrote: > Ah, this passive voice again, and again the assumption that there is an > agenda of python-dev. Exactly opposite. There is no agenda, and thus there is no pojnt in trying to prevent new features in Python language based on the idea the features are not "4E". Python, IMHO, has gone far beyond "4E" already. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From koder.mail at gmail.com Fri Feb 16 15:38:41 2007 From: koder.mail at gmail.com (KoDer) Date: Fri, 16 Feb 2007 16:38:41 +0200 Subject: [Python-Dev] urllib2 EP + decr. startup time Message-ID: <6e8b06c90702160638o7c6ce54fwcbb5af975f10982c@mail.gmail.com> Hello to all. During more than two years i widely use urllib2 to write commercial applications (almost for extracting data from web sites to excel sheets) and here is some enhanced enhanced for it: 1) Add support for 'HEAD' request (and maybe some other). This needs small changes. a)Add request_type = 'GET' to urllib2.Request class constructor. b)Then put request_type value pass to http header, except Request has data - in this case it's change to 'POST'. The results of such request will be the same as in case of 'GET' request, except zero size of body. 2)HTTH keep-alive opener. Almost complete realizations can be found in urlgrabber (http://linux.duke.edu/projects/urlgrabber)(used by yum, so tested well enough, i think). It's use urllib2 opener protocol and well integrated in urllib2 structure. They need just little change to properly support some headers. 3) Save HTTP exchange history. Now there is no suitable way to obtain all sent and received headers. Received headers are saved only for last response in redirection chain and sent headers are not saved at all. I use run-time patching of httplib to intercept of the sent and received data (may be i missed something?). Proposal is to add property 'history' to object returned from urllib2.urlopen - list of objects which contain send/recv headers for all redirect chain. 4) Add possibilities to obtain underlying socket, used for recv http data. Now it's impossible to work with http connection in async matter (or i miss something again?). If connection hangs then whole program hangs too and i don't known way to fix this. Of cause if you obtain such socket then you respond for compression and etc. Now i use following code: x = urrlib2.urlopen(.....) sock = x.fp._sock.fp._sock. There only one problem, as i know, - chunked encoding. In case of chunked encoding need to return socket-like object which do all work to assemble chunks in original stream. I already use such object for two years and it's ok. 5) And now for something completely different ;)). This is just initial proposal and it needs enhancement. May be i need put it to python-ideas list? At last Goggle SOC there was one of problem to solve - the decrease of interpreter's startup time. 'strace' command shows next: most of startup time the interpreter try to find imported modules. And most of them finished with 'not found' error, because of large size of sys.path variable. In future this time will be increase - setuptools adds many dirs to search path using pth files (to manage installed modules and eggs). I propose to add something like .so caching which used in modern *nix sytems to load shared libraries. a) Add to python interpreter --build-modules-index option. When python found this opts it scans all dirs in paths and build dictionary {module_name:module_path}. Dict will be saved in external file (save only top-dir for packages and path for one-file modules). Also it saves in this file mtime for all pth files and dirs from path and path variable. b) When interpreter is started-up it, as usually, scans all path dirs for pth files, and using data saved in cache file check is new modules or search dirs added or old modified. Then it read cache file and compares mtimes and path dirs. If it isn't modified then cache data used for fast module loading. If imported module didn't found in cache - interpreter falls back to standard scheme. Also it is necessary to add some options to control of using cache like --disable-cache, --clear-cache,disable cashing some dirs, etc. --- K.Danilov aka KoDer From pje at telecommunity.com Fri Feb 16 17:41:51 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 16 Feb 2007 11:41:51 -0500 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <45D5A5AC.5000004@v.loewis.de> References: <45D57947.70704@gmail.com> <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> <45D557A5.3040804@v.loewis.de> <45D56F1C.1070901@gmail.com> <45D574AC.3060608@v.loewis.de> <45D57947.70704@gmail.com> Message-ID: <5.1.1.6.0.20070216114041.054dbec0@sparrow.telecommunity.com> At 01:38 PM 2/16/2007 +0100, Martin v. L?wis wrote: >Nick Coghlan schrieb: > > However, another aspect that occurred to me is that inheriting from > > tuple has significant practical benefits in terms of speed and memory > > consumption, at which point it doesn't seem worthwhile to *remove* the > > indexing capability. > >I'm not so sure that inheriting from tuple, and giving it named fields, >has significant speed and memory benefits. In particular for the memory >benefits, you can use __slots__ to achieve the same effects, and more >efficiently so (because it you don't store the tuple length). As for >speed, I would have to see measurements to be convinced it is faster. For an otherwise-pure Python implementation, the performance benefit of inheriting from a tuple is in having ready-made C implementations of hashing and comparison. From pje at telecommunity.com Fri Feb 16 17:57:18 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 16 Feb 2007 11:57:18 -0500 Subject: [Python-Dev] urllib2 EP + decr. startup time In-Reply-To: <6e8b06c90702160638o7c6ce54fwcbb5af975f10982c@mail.gmail.co m> Message-ID: <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> At 04:38 PM 2/16/2007 +0200, KoDer wrote: > 'strace' command shows next: most of startup time the interpreter >try to find imported modules. > And most of them finished with 'not found' error, because of large >size of sys.path variable. > In future this time will be increase - setuptools adds many dirs to >search path > using pth files (to manage installed modules and eggs). Actually, under normal circumstances, most eggs installed are .zip files, which the interpreter already caches the indexes of. Eggs installed as directories should be increasing in rarity, except for in-development packages installed via the "develop" command. Also, I plan to make setuptools 0.7's "nest" packaging tool support managing packages the "old" way, i.e., as a single flat directory structure, using manifest files to support uninstallation and the like. So it should not really be the case that this will keep increasing over time. Also, are you aware that putting a zipped version of the standard library on sys.path already speeds up startup considerably? Python since 2.3 automatically includes an appropriate entry in sys.path: Python 2.3.4 (#53, May 25 2004, 21:17:02) [MSC v.1200 32 bit (Intel)] on win32 >>> import sys >>> sys.path ['', 'C:\\WINDOWS\\system32\\python23.zip', '', 'c:\\Python23\\DLLs', 'c:\\Pytho n23\\lib', 'c:\\Python23\\lib\\plat-win', 'c:\\Python23\\lib\\lib-tk', 'c:\\Pyth on23'] Creating the zip file that's already listed in the default sys.path will allow most startup imports to be handled without a lot of path checking. > I propose to add something like .so caching which used in modern >*nix sytems to load > shared libraries. > > a) Add to python interpreter --build-modules-index option. When python > found > this opts it scans all dirs in paths and build dictionary >{module_name:module_path}. > Dict will be saved in external file (save only top-dir for packages >and path for one-file modules). > Also it saves in this file mtime for all pth files and dirs from >path and path variable. Unless you mean something more abstract by "dirs" than just filesystem directories, this isn't going to help eggs or other zipped modules any, compared to how they are now. From koder.mail at gmail.com Fri Feb 16 18:29:41 2007 From: koder.mail at gmail.com (KoDer) Date: Fri, 16 Feb 2007 19:29:41 +0200 Subject: [Python-Dev] urllib2 EP + decr. startup time In-Reply-To: <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> References: <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> Message-ID: <6e8b06c90702160929p1ac361e5s6ab1df3c9e44a51d@mail.gmail.com> 2007/2/16, Phillip J. Eby : > At 04:38 PM 2/16/2007 +0200, KoDer wrote: ..... > > > Also, are you aware that putting a zipped version of the standard library > on sys.path already speeds up startup considerably? Python since 2.3 > automatically includes an appropriate entry in sys.path: > zipped version has one weakness - you can't put .so(or dll) files inside. In my system 19 from 25 installed egg add directories ,not archives (because it's contain dll ). But even without egg directories >> ['', 'C:\\Python25\\Scripts', 'C:\\WINDOWS\\system32\\python25.zip', 'C:\\Python25\\DLLs', 'C:\\Python25\\lib', 'C:\\Python25\\lib\\plat-win', ......... 'C:\\Python25\\lib\\site-packages\\wx-2.8-msw-unicode'] len(sys.path) == 18 (without eggs) near 18 / 2 = 9 'file not found' errors for every first module import. So improvement of setuptools will help, but not solve this problem . -- K.Danilov aka KoDer From pje at telecommunity.com Fri Feb 16 18:55:32 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 16 Feb 2007 12:55:32 -0500 Subject: [Python-Dev] urllib2 EP + decr. startup time In-Reply-To: <6e8b06c90702160929p1ac361e5s6ab1df3c9e44a51d@mail.gmail.co m> References: <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070216124921.04799d38@sparrow.telecommunity.com> At 07:29 PM 2/16/2007 +0200, KoDer wrote: >2007/2/16, Phillip J. Eby : > > At 04:38 PM 2/16/2007 +0200, KoDer wrote: >..... > > > > > > Also, are you aware that putting a zipped version of the standard library > > on sys.path already speeds up startup considerably? Python since 2.3 > > automatically includes an appropriate entry in sys.path: > > > >zipped version has one weakness - you can't put .so(or dll) files inside. >In my system 19 from 25 installed egg add directories ,not archives >(because it's contain dll ). There's something wrong there. You can put .so, .dll, or .pyd files in eggs - they get extracted to a cache directory automatically, but those directories *don't* get added to sys.path. The mere presence of such files is not enough to cause an egg to be installed unzipped. The package has to be using things like __file__, the package author has to have marked it zip-unsafe, or easy_install was invoked with a request to install unzipped. As long as none of these conditions apply, the egg should be installed zipped, with dynamic libraries automatically extracted on-demand to the $PYTHON_EGG_CACHE. >But even without egg directories >> >['', >'C:\\Python25\\Scripts', >'C:\\WINDOWS\\system32\\python25.zip', >'C:\\Python25\\DLLs', >'C:\\Python25\\lib', >'C:\\Python25\\lib\\plat-win', >......... >'C:\\Python25\\lib\\site-packages\\wx-2.8-msw-unicode'] >len(sys.path) == 18 (without eggs) near 18 / 2 = 9 'file not found' errors > for every first module import. That depends on what the module is. In the path you've shown, having a python25.zip would allow only 2 failures before each stdlib import. (Note that an failed zipfile imports are roughly equivalent to a dictionary lookup in time - they don't access the filesystem once the zipfile index is loaded). >So improvement of setuptools will help, but not solve this problem . Right -- most of your problem will be solved by creating 'C:\\WINDOWS\\system32\\python25.zip', containing the contents of C:\\Python25\\lib\\. Hm. Interesting, actually. Does anybody know why it's looking for 'C:\\WINDOWS\\system32\\python25.zip'? That seems wrong to me. From guido at python.org Fri Feb 16 19:21:25 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 16 Feb 2007 10:21:25 -0800 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <45D56F1C.1070901@gmail.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> <45D557A5.3040804@v.loewis.de> <45D56F1C.1070901@gmail.com> Message-ID: On 2/16/07, Nick Coghlan wrote: > Martin v. L?wis wrote: > > Raymond Hettinger schrieb: > >> No need to go so widely off-track. The idea is to have an efficient type that > >> is directly substitutable for tuples but is a bit more self-descriptive. I like > >> to have the doctest result cast at NamedTuple('TestResults failed attempted). > >> The repr of that result looks like TestResult(failed=0, attempted=15) but is > >> still accessible as a tuple and passes easily into other functions that expect a > >> tuple. This sort of thing would be handly for things like os.stat(). > >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 > > > > I'd like to repeat Guido's question: Why does this still need to support > > the tuple interface (i.e. indexed access)? > > So that it remains interoperable with existing libraries that expect a > tuple? Otherwise you'd be casting (and copying) every time you needed to > pass it to something that used indexed access. In the case of os.stat and friends I propose that in Py3k we drop the tuple-ness completely; it's been dual-op since 2.2. Maybe Raymond's proposed record type should have two versions: one that's also a tuple, for compatibility, and one that's just a record. The compatibility version should also support having named fields that don't show up in the tuple view -- this has proved very useful for the os.stat() result. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From larry at hastings.org Fri Feb 16 19:30:07 2007 From: larry at hastings.org (Larry Hastings) Date: Fri, 16 Feb 2007 10:30:07 -0800 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <45D553B0.1070208@v.loewis.de> References: <20070214225531.28203.555255924.divmod.xquotient.1888@joule.divmod.com> <45D3A9D7.5090103@canterbury.ac.nz> <9e804ac0702141759p40e6e2f5wb0d308577203cb17@mail.gmail.com> <45D3D110.4020802@canterbury.ac.nz> <9e804ac0702141927j2268e65ah7c8dd8a60fe5bdc6@mail.gmail.com> <45D3EECF.7080600@canterbury.ac.nz> <6a36e7290702150211v38dbb21bpfeb59e2524dec2b0@mail.gmail.com> <45D4842E.9080105@hastings.org> <45D48BD9.3040608@v.loewis.de> <45D49CFF.1030107@hastings.org> <45D553B0.1070208@v.loewis.de> Message-ID: <45D5F82F.80402@hastings.org> > As others have reported: if you have a gtk reactor and a wx reactor to > support, in addition to a "regular" Win32 reactor (e.g. PythonWin), > your approach still won't work. They all do GetMessage behind the > scenes (sp?), yet you would run them in multiple threads, so they get > their own messages. Plus they are not thread-safe. I'm not sure what your point is here. Let me see if I can clarify things, at least from my side. I suggest it is possible to implement a PerfectReactor. I define this PerfectReactor as a singleton top-level reactor which runs in the main thread of a Python application. All incoming events would be serialized to this single reactor, which would dispatch them to their appropriate listeners. When asked to monitor disparate event systems, the PerfectReactor would act as a concentrator of multiple other reactors, with various ugly things going on hidden in the implementation. The goal would be that modular libraries of event listeners could be written in an agnostic manner, simply requesting to listen to whatever events they need to, and the main application wouldn't have to know in advance what reactors to configure or how to configure them. You seem to assert it is not possible to implement such a PerfectReactor, and list counter-examples. So far so good. If you're saying "it is impossible for any implementation to simultaneously support more than one of the Win32, gtk, and wx reactors": fine, this hypothetical PerfectReactor can't do the impossible, but that's not an argument against a PerfectReactor. If it *is* possible to support more than one of these simultaneously through precise configuration, I assert that a PerfectReactor could support that configuration too, and the goal would be to have it automatically configure itself. You point out the gtk reactor and the wx reactor must be run in the main thread, because they run their own message pumps. What I had in mind was that, when you asked to start listening to Win32 events, the Win32 reactor would tell the PerfectReactor "I must be run in the main thread", and it would (re-) configure that to be the main thread's listener; the notification mechanism for other threads saying "there's a new event in your queue" would switch to using Win32 messages. If the gtk and wx reactors had the same restriction, then whichever was first introduced to the PerfectReactor first would win, and introducing subsequent reactors would throw an exception. (Again, can't do the impossible.) However, if gtk and wx expose enough of the plumbing of their message pumps, this central Win32 message pump could identify messages intended for the gtk and wx reactor message pumps and inject them. The gtk and wx reactors wouldn't call GetMessage, but they could call the rest of their message loops. Therefore they could all be run in the main thread. (And yes, "behind the scenes" is correct.) > if the PerfectReactor is to support Tcl/Tk, it needs to run the Tcl > even loop in the Tcl thread, which is normally the main thread (in > Tkinter, it's the thread where you originally created the tkapp > object). OTOH, PerfectReactor may also need to run some Win32 event > loop in the same thread. Again, I'm not sure of your point. If you're saying "it is impossible to run multithreaded Tcl and any message pump (Win32, gtk, wx) in the same application", then fine, PerfectReactor cannot do the impossible. If it is only possible to do with clever configuration, then I maintain a PerfectReactor could self-configure to support this configuration at runtime. Cheers, /larry/ From python at rcn.com Fri Feb 16 19:41:40 2007 From: python at rcn.com (Raymond Hettinger) Date: Fri, 16 Feb 2007 10:41:40 -0800 Subject: [Python-Dev] Py2.6 ideas References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> <45D557A5.3040804@v.loewis.de> Message-ID: <001701c751fa$1ea59370$ea146b0a@RaymondLaptop1> [Martin v. L?wis] >Why does this still need to support the > tuple interface (i.e. indexed access)? I used named tuples whereever I need a tuple but the number and meaning of the fields starts to tax my memory. For doctests, I return a named tuple like TestResults(failed=0, attempted=15). That named tuple can still be unpacked after a function call: f,a=results. And it can be unpacked in a function call: f(*results). It can be handed to functions that expect a tuple: 'Missed %d out of %d tests' % results. Also, the named tuple used with indexed access has the same high performance as a regular tuple; however, if an error occurs, its repr is shown in a more readable form. Likewise, when constructing the NamedTuple, an editor's tooltips reminds you of what goes in each field. Those properties have proved useful to me when modeling option contracts where each contract has to track the remaining time, interest rate, option type, underlying security, and strike price. The same applies to model results: delta, gamma, vega, theta, rho. This could also be done with attribute access, but it would be much slower and much more verbose when unpacking the model's results: d, g, v, t, r = model(somecontract) vs. m = model(somecontract) d, g, v, t, r = m.delta, m.gamma, m.vega, m.theta, m.rho > I'm not (anymore) sure that you are aware that the os.stat result *already* > has named fields, in addition to the indexed access. Of course, that specific example was solved long ago. We did not however expose a general purpose mechanism applicable where similar issues arise for other tuples. Raymond From derek.shockey at gmail.com Fri Feb 16 20:53:20 2007 From: derek.shockey at gmail.com (Derek Shockey) Date: Fri, 16 Feb 2007 14:53:20 -0500 Subject: [Python-Dev] Lack of sequential decompression in the zipfile module Message-ID: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> Though I am an avid Python programmer, I've never forayed into the area of developing Python itself, so I'm not exactly sure how all this works. I was confused (and somewhat disturbed) to discover recently that the zipfile module offers only one-shot decompression of files, accessible only via the read() method. It is my understanding that the module will handle files of up to 4 GB in size, and the idea of decompressing 4 GB directly into memory makes me a little queasy. Other related modules (zlib, tarfile, gzip, bzip2) all offer sequential decompression, but this does not seem to be the case for zipfile (even though the underlying zlib makes it easy to do). Since I was writing a script to work with potentially very large zipped files, I took it upon myself to write an extract() method for zipfile, which is essentially an adaption of the read() method modeled after tarfile's extract(). I feel that this is something that should really be provided in the zipfile module to make it more usable. I'm wondering if this has been discussed before, or if anyone has ever viewed this as a problem. I can post the code I wrote as a patch, though I'm not sure if my file IO handling is as robust as it needs to be for the stdlib. I'd appreciate any insight into the issue or direction on where I might proceed from here so as to fix what I see as a significant problem. Thanks, Derek -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070216/58127a75/attachment.htm From p.f.moore at gmail.com Fri Feb 16 21:52:29 2007 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 16 Feb 2007 20:52:29 +0000 Subject: [Python-Dev] urllib2 EP + decr. startup time In-Reply-To: <5.1.1.6.0.20070216124921.04799d38@sparrow.telecommunity.com> References: <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> <5.1.1.6.0.20070216124921.04799d38@sparrow.telecommunity.com> Message-ID: <79990c6b0702161252i47201e09k8319ccbc6b5f8dfd@mail.gmail.com> On 16/02/07, Phillip J. Eby wrote: > Hm. Interesting, actually. Does anybody know why it's looking for > 'C:\\WINDOWS\\system32\\python25.zip'? That seems wrong to me. It looks alongside python25.dll, which is installed in windows\system32 by default. If you then ask why the DLL goes in the system directory, I believe this is to help with Python COM objects, which don't otherwise have the Python directory on their PATH (and so wouldn't find python25.dll if it were there). Paul. From brett at python.org Fri Feb 16 22:09:10 2007 From: brett at python.org (Brett Cannon) Date: Fri, 16 Feb 2007 13:09:10 -0800 Subject: [Python-Dev] Lack of sequential decompression in the zipfile module In-Reply-To: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> References: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> Message-ID: On 2/16/07, Derek Shockey wrote: > Though I am an avid Python programmer, I've never forayed into the area of > developing Python itself, so I'm not exactly sure how all this works. > > I was confused (and somewhat disturbed) to discover recently that the > zipfile module offers only one-shot decompression of files, accessible only > via the read() method. It is my understanding that the module will handle > files of up to 4 GB in size, and the idea of decompressing 4 GB directly > into memory makes me a little queasy. Other related modules (zlib, tarfile, > gzip, bzip2) all offer sequential decompression, but this does not seem to > be the case for zipfile (even though the underlying zlib makes it easy to > do). > > Since I was writing a script to work with potentially very large zipped > files, I took it upon myself to write an extract() method for zipfile, which > is essentially an adaption of the read() method modeled after tarfile's > extract(). I feel that this is something that should really be provided in > the zipfile module to make it more usable. I'm wondering if this has been > discussed before, Not that I know of, but searching Google would better answer that question. > or if anyone has ever viewed this as a problem. Not that I know of. > I can post > the code I wrote as a patch, though I'm not sure if my file IO handling is > as robust as it needs to be for the stdlib. I'd appreciate any insight into > the issue or direction on where I might proceed from here so as to fix what > I see as a significant problem. > Best way is to post it as a patch to the SF tracker for Python (http://sourceforge.net/patch/?group_id=5470). Then hopefully someone will eventually get to it and have a look. Just please understand that it might be a while as it requires someone to take an interest in your patch to put the time and effort to make sure it up to including. To help your chances of getting it included, make sure you do the following: 1. Make it match PEP 7/8 style guidelines. 2. Have unit tests. 3. Have proper documentation. It is okay if it is not in LaTeX if you don't already know the language. -Brett > Thanks, > Derek > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > > From ark at acm.org Fri Feb 16 22:22:31 2007 From: ark at acm.org (Andrew Koenig) Date: Fri, 16 Feb 2007 16:22:31 -0500 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: Message-ID: <004501c75210$948d48b0$6402a8c0@arkdesktop> > Maybe Raymond's proposed record type should have two versions: one > that's also a tuple, for compatibility, and one that's just a record. FWIW, ML unifies tuples and records by defining a tuple to be a record whose component names are all consecutive integers starting with 1. For example, in ML, the literal { name = "ark", state = "NJ" } represents a record with type { name: string, state: string }. The identifiers "name" and "state" are bound during compilation, ML being a statically typed language. In ML, one extracts a component named foo by applying a function named #foo. So, for example, the value of #state { name = "ark", state = "NJ" } is "NJ", and trying to evaluate #foo { name = "ark", state = "NJ" } results in a compilation error because of type-checking failure. Component names can be either identifiers or integers. So, for example, { name = "spells", 1 = "xyzzy", 2 = "plugh" } is a record of type {1: string, 2: string, name: string }. So here is the point. If the component names of a record are all positive integers with no gaps, the record is *also* a tuple. So, for example { 2 = "plugh", 1 = "xyzzy" } has exactly the same meaning--including the same type--as { "xyzzy", "plugh" } In both cases, the compiler normalizes the display, both of the value (i.e. it prints {"xyzzy", "plugh"} instead of { 2 = "plugh", 1 = "xyzzy" }, and it prints the type as string * string instead of (the equivalent) { 1: string, 2: string } So in ML, tuple types aren't really anything special -- they're just abbreviations for elements of a particular subset of record types. From martin at v.loewis.de Fri Feb 16 23:12:32 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Feb 2007 23:12:32 +0100 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <001701c751fa$1ea59370$ea146b0a@RaymondLaptop1> References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> <45D557A5.3040804@v.loewis.de> <001701c751fa$1ea59370$ea146b0a@RaymondLaptop1> Message-ID: <45D62C50.2030502@v.loewis.de> Raymond Hettinger schrieb: > d, g, v, t, r = model(somecontract) I find that line quite unreadable, and find it likely that I would not be able to remember the standard order of the fields. You almost "had me" with the two fields example, but this makes me think "-1" again. Is it really that you need all these values in the following computation? For stat, this was never the case: you would only need some field normaly (especially when the more esoteric, platform-dependent fields got added). If you absolutely want tuple unpacking on a record-like object, I'd suggest to support explicit tuple conversion, like d, g, v, t, r = model(somecontract).to_tuple() Or, if you absolutely cannot stand the explicit tuple creation, add def __getitem__(self, index): return getattr(self, self.__slots__[index]) # or is it self.[self.__slots__[index]] :-? No need to inherit from tuple for that. > Of course, that specific example was solved long ago. We did not > however expose a general purpose mechanism applicable where > similar issues arise for other tuples. As you've explained now, your use case is not similar. For os.stat, it's a means of transition and backwards compatibility. For your code, it seems you want it a permanent feature in your code. Regards, Martin From falcon at intercable.ru Wed Feb 14 13:09:13 2007 From: falcon at intercable.ru (Sokolov Yura) Date: Wed, 14 Feb 2007 15:09:13 +0300 Subject: [Python-Dev] Alternative Python VM Message-ID: <45D2FBE9.3090906@intercable.ru> It could be interesting. """ - pyvm is * 2 times * faster than Python 2.4. In the source code there is a collection of benchmarks which includes 65 python scripts collected from the internet. At average if Python 2.4 needs 24 hours to do some job, pyvm can do it in 12 hours. - pyvm is a virtual machine that can run Python 2.3/2.4 bytecode. There is a compiler written in python (the 'pyc' program) which is executed by the vm to compile source code to bytecode. It is very easy to do advanced optimizations in python and the compiler produces bytecode of high quality (speed and size). """ http://students.ceid.upatras.gr/~sxanth/pyvm/ Sokolov Yura (funny_falcon) From joachim.koenig-baltes at emesgarten.de Wed Feb 14 16:32:10 2007 From: joachim.koenig-baltes at emesgarten.de (=?UTF-8?B?Sm9hY2hpbSBLw7ZuaWctQmFsdGVz?=) Date: Wed, 14 Feb 2007 16:32:10 +0100 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) In-Reply-To: <20070214072621.28203.1662509932.divmod.xquotient.1571@joule.divmod.com> References: <20070214072621.28203.1662509932.divmod.xquotient.1571@joule.divmod.com> Message-ID: <45D32B7A.1000206@emesgarten.de> glyph at divmod.com wrote: > When you boil it down, Twisted's event loop is just a notification for > "a connection was made", "some data was received on a connection", "a > connection was closed", and a few APIs to listen or initiate different > kinds of connections, start timed calls, and communicate with threads. > All of the platform details of how data is delivered to the > connections are abstracted away. How do you propose we would make a > less "specific" event mechanism? But that is exactly the problem I have with Twisted. For HTTP it creates its own set of notifications instead of structuring the code similar to SocketServer (UDP and TCP), BaseHTTPServer, SimpleHTTPServer etc which are well understood in the python community and e.g. used by medusa and asyncore. Having to completely restructure one's own code is a high price. Giving control away into a big framework that calls my own code for not so easy to understand reasons (for a twisted noob) does not give me a warm feeling. It's o.k. for complex applications like web servers but for small networking applications I'd like to have a chance to understand what's going on. Asyncore is so simple that it's easy to follow when I let it do the select() for me. That said, I conclude that the protocol implementations are superb but unfortunately to tightly coupled to the Twisted philosophy, sitting in the middle, trying to orchestrate instead of being easy to integrate. Joachim From mithrandi at mithrandi.za.net Mon Feb 12 11:41:05 2007 From: mithrandi at mithrandi.za.net (Tristan Seligmann) Date: Mon, 12 Feb 2007 12:41:05 +0200 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45CFAA81.5040906@redfrontdoor.org> References: <45CFAA81.5040906@redfrontdoor.org> Message-ID: <20070212104105.GA14464@mithrandi.za.net> * Ben North [2007-02-11 23:45:05 +0000]: > Dynamic attribute access is currently possible using the "getattr" > and "setattr" builtins. The present PEP suggests a new syntax to > make such access easier, allowing the coder for example to write > > x.('foo_%d' % n) += 1 > > z = y.('foo_%d' % n).('bar_%s' % s) > > instead of > > attr_name = 'foo_%d' % n > setattr(x, attr_name, getattr(x, attr_name) + 1) > > z = getattr(getattr(y, 'foo_%d' % n), 'bar_%s' % s) Clipper (and other members of the "xBase" family) have something vaguely similar, and from working with that syntax, my feeling is that this is too "subtle" given the mixing of levels going on here (a "literal" name vs. an expression evaluating to a name). That is, it's too easy to not properly notice the presence / absence of the () and become confused between foo.bar and foo.(bar) or similar. (For the curious, the Clipper syntax is generally used in "commands" which are a kind of statement that is macro-evaluated; so, for example, something like this: USE clients will be transformed into something like the following expression: dbUseArea("clients") during the preprocessor pass, care of a "builtin" definition of USE; this function has the effect of opening a database file on disk for use. Now, in order to use the command syntax with a database name stored in a variable, one does something like the following: USE (dbName) which is transformed into something like: dbUseArea(dbname) with similar effects as before.) -- mithrandi, i Ainil en-Balandor, a faer Ambar -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 197 bytes Desc: Digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20070212/439fd0c8/attachment.pgp From sergio.correia at gmail.com Mon Feb 12 17:30:20 2007 From: sergio.correia at gmail.com (Sergio Correia) Date: Mon, 12 Feb 2007 11:30:20 -0500 Subject: [Python-Dev] New syntax for 'dynamic' attribute access In-Reply-To: <45D08B3E.5020105@egenix.com> References: <1171291253.45d07c75297b5@imp.hosting365.ie> <45D08B3E.5020105@egenix.com> Message-ID: A few of you have expressed concern about how would that look to a newbie. Being one, this is what I think: - The idea sounds good. Setattr and getattr seems both unpythonic and illegible. - please.(dont_torture) = me(with_dots,that_look,like.(function),calls). Ok, so the dot _is_ needed in order to indicate that we are talking about objects. But if I see something like spam.(eggs) , I would feel tempted to delete the dot thinking it's a typo. - Square brackets have enough overloading. - Braces feel good. I think they are the best choice of the ones proposed. Because spam{eggs} doesn't mean anything, then there would be no confusion with a typo in spam.{eggs} As someone said before, the problem is that it would limit braces for serving further purposes in the future. If the chances of requiring using braces in the next few versions seem low to you, I would say go for it. Sergio On 2/12/07, M.-A. Lemburg wrote: > On 2007-02-12 16:19, Georg Brandl wrote: > >> Tim Delaney asked in particular: > >>> Have you checked if [the existing uses of getattr, where "getattr" in > >>> that scope is a function argument with default value the built-in > >>> "getattr"] are intended to bring the "getattr" name into local scope > >>> for fast lookup, or to force a binding to the builtin "gettattr" at > >>> compile time (two common (ab)uses of default arguments)? If they are, > >>> they would be better served by the new syntax. > >> They're all in Lib/codecs.py, and are of the form: > >> > >> class StreamRecoder: > >> def __getattr__(self, name, > >> getattr=getattr): > >> > >> """ Inherit all other methods from the underlying stream. > >> """ > >> return getattr(self.stream, name) > >> > >> Without digging deeper into that code I'm afraid I can't say precisely > >> what is going on. > > > > Since that is a special method and ought to have the signature > > __getattr__(self, name), I think it's safe to assume that that's meant > > as an optimization. > > I can confirm that: it's a case of fast-local-lookup optimization. > > You can add a -1 from me to the list as well: I don't think that > dynamic lookups are common enough to warrant new syntax. > > Even if you do add a new syntax for this, using parenthesis is > a poor choice IMHO as the resulting code looks too much like a > function call (e.g. "callable.(variable)"). > > Other choices would be square brackets [], but these have the > same problem as they are in use for indexing. > > The only brackets that are not yet overloaded in the context > of applying them to an object are curly brackets, so > "callable.{variable}" would cause enough raising eyebrows > to not think of a typo. > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Services directly from the Source (#1, Feb 12 2007) > >>> Python/Zope Consulting and Support ... http://www.egenix.com/ > >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ > ________________________________________________________________________ > > :::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! :::: > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/sergio.correia%2Bpydev%40gmail.com > From vancleef at lostwells.net Wed Feb 14 02:09:43 2007 From: vancleef at lostwells.net (vancleef at lostwells.net) Date: Tue, 13 Feb 2007 18:09:43 -0700 (MST) Subject: [Python-Dev] Porting Python 2.4.4. and 2.5 to Sparc Solaris 9 and 10 Message-ID: <200702140109.l1E19hPD026904@julie.lostwells.net> Dear Python Developers. I posted the following to the python-help yesterday. While I received some interesting responses, they indicated that I'd already gone beyond that group's knowledge, and suggested that I mail the developers directly on this list. By way of background, I'm a retired hardware-software engineer with substantial experience in kernel programming, device drivers, porting between hardware and software architectures, so am probably more prepared than the average systems administrator to get into program details. I have downloaded the tars for Python 2.4.4 and 2.5, and built both of them on two Solaris systems, using a variety of compilers. The original objective was to get a usable Python to support a Mailman 2.1.9 installation. While it was clear that a build of 2.5, using the Sun Studio 11 development system, was incomplete and did not pass all of the regression tests, it did prove to be adequate to support Mailman. I have now come back to audit building Python on Sparc Solaris and to get some idea of what needs to be done to get a clean and reasonably complete build. My attempts to find some support through the Python web site seemed quite inconclusive. I spent part of a day trying to weave my way through the Sourceforge bug tracker, but was unable to find anything that seemed meaningful vis-a-vis what my audits were showing. Accordingly, I chose to devote some time and effort to determining where the build problems lay, and what might be needed to fix some of them. Build Platform Data: System A: Sun Ultra 60, Solaris 10 11/06, with Sun Studio 11 development system (/opt/SUNWspro). A Solaris 10 11/06 full distribution install includes gcc-3.4.3, Python 2.3.3, along with libraries and header files for such things as Tcl/Tk and openssl. System B: Sun Ultra 10, Solaris 9 9/05. Development systems are Sun Studio 11 and a new local build of gcc-4.1.1 (c and c++ only). The O/S is patched with the mid-January Sun patch cluster. Solaris 9 does not include a Python or a gcc, but does include Tcl/Tk libraries and include files. In both cases, the Studio 11 programs have been patched to current. The cc patch level is 121015-03. Both O/S installs are recent "full system" installs on bare machines, plus the development systems noted. I consider these representative of what an average systems administrator would have for resources "out of the box" for building and installing Python from source. On both systems, the non-Solaris software resides in the /usr/sfw directory tree. After having tried several configure/make/make test runs, it appears to me that there are two problem areas. 1. There is no way in the distribution setup to get the build to find the /opt/sfw include libraries for building extension modules. It appears to me that this search is done by setup.py, and that configuring the appropriate paths in the main Makefile does not propogate out to the module build. If the configure script has a way to find these libraries and to set the build up for them, I can't see what that might be. This problem affects both 2.4.4 and 2.5. I have taken the time to add the search code for the Solaris /opt/sfw directories to setup.py. There were other changes needed to existing code to get it to work properly. I include a diff between the code I am using with success and the 2.5 distribution version at the end of this note. It may not be the most appropriate for distribution, but it works on Solaris 9 and 10. I had to revise the setup.py in the 2.4.4 distribution similarly. 2. On Python 2.5, the ctypes and libffi code is not only GNU-specific, but gcc-version-specific as well. The problem with the __attribute__ specification in ffi.h seems to be well-understood, and may be amenable to some sort of workaround. However, attempting to build with the gcc-3.4.3 included with Solaris 10 failed with a linker misalignment error. 3. Regression test failures: test test_ioctl failed -- errors occurred in test.test_ioctl.IoctlTests I get this output when running make test on any build (Sun cc or any gcc) on both 2.4.4 and 2.5. Running the ioctl test standalone does not produce a failure. At this point, I wonder if I am reinventing the wheel or finding things that need to be addressed for Solaris builds. I am new to Python, and while I'd like to use it to make some local changes to Mailman, I feel concerned over reliability and robustness. Just how are you getting full Solaris builds? Hank van Cleef vancleef at lostwells.net ==================================================================== The test summary and setup.py diffs follow: 280 tests OK. 1 test failed: test_ioctl 38 tests skipped: test_aepack test_al test_applesingle test_bsddb test_bsddb185 test_bsddb3 test_cd test_cl test_codecmaps_cn test_codecmaps_hk test_codecmaps_jp test_codecmaps_kr test_codecmaps_tw test_ctypes test_curses test_gdbm test_gl test_imgfile test_linuxaudiodev test_macfs test_macostools test_nis test_normalization test_ossaudiodev test_pep277 test_plistlib test_scriptpackages test_socket_ssl test_socketserver test_sqlite test_startfile test_timeout test_unicode_file test_urllib2net test_urllibnet test_winreg test_winsound test_zipfile64 2 skips unexpected on sunos5: test_ctypes test_nis (nis is not enabled on these systems) Diffs for setup.py: *** setup.py.dist Thu Feb 8 11:39:42 2007 --- setup.py Mon Feb 12 11:04:44 2007 *************** *** 306,311 **** --- 306,316 ---- platform = self.get_platform() (srcdir,) = sysconfig.get_config_vars('srcdir') + # Add search directories for Solaris distribution add-ons + if platform == 'sunos5': + inc_dirs += ['/usr/sfw/include'] + lib_dirs += ['/usr/sfw/lib'] + # Check for AtheOS which has libraries in non-standard locations if platform == 'atheos': lib_dirs += ['/system/libs', '/atheos/autolnk/lib'] *************** *** 520,529 **** depends = ['socketmodule.h']) ) # Detect SSL support for the socket module (via _ssl) search_for_ssl_incs_in = [ '/usr/local/ssl/include', '/usr/contrib/ssl/include/' ] ! ssl_incs = find_file('openssl/ssl.h', inc_dirs, search_for_ssl_incs_in ) if ssl_incs is not None: --- 525,535 ---- depends = ['socketmodule.h']) ) # Detect SSL support for the socket module (via _ssl) search_for_ssl_incs_in = [ + '/usr/sfw/include', '/usr/local/ssl/include', '/usr/contrib/ssl/include/' ] ! ssl_incs = find_file('/openssl/ssl.h', inc_dirs, search_for_ssl_incs_in ) if ssl_incs is not None: *************** *** 531,541 **** ['/usr/kerberos/include']) if krb5_h: ssl_incs += krb5_h ! ssl_libs = find_library_file(self.compiler, 'ssl',lib_dirs, ['/usr/local/ssl/lib', '/usr/contrib/ssl/lib/' ] ) - if (ssl_incs is not None and ssl_libs is not None): exts.append( Extension('_ssl', ['_ssl.c'], --- 537,546 ---- ['/usr/kerberos/include']) if krb5_h: ssl_incs += krb5_h ! ssl_libs = find_library_file(self.compiler, 'ssl.so',lib_dirs, ['/usr/local/ssl/lib', '/usr/contrib/ssl/lib/' ] ) if (ssl_incs is not None and ssl_libs is not None): exts.append( Extension('_ssl', ['_ssl.c'], *************** *** 1245,1251 **** --- 1250,1258 ---- # Check for various platform-specific directories if platform == 'sunos5': include_dirs.append('/usr/openwin/include') + include_dirs.append('/usr/sfw/include') added_lib_dirs.append('/usr/openwin/lib') + added_lib_dirs.append('/usr/sfw/lib') elif os.path.exists('/usr/X11R6/include'): include_dirs.append('/usr/X11R6/include') added_lib_dirs.append('/usr/X11R6/lib64') From brett at python.org Fri Feb 16 23:43:24 2007 From: brett at python.org (Brett Cannon) Date: Fri, 16 Feb 2007 14:43:24 -0800 Subject: [Python-Dev] Alternative Python VM In-Reply-To: <45D2FBE9.3090906@intercable.ru> References: <45D2FBE9.3090906@intercable.ru> Message-ID: On 2/14/07, Sokolov Yura wrote: > It could be interesting. > > """ > - pyvm is * 2 times * faster than Python 2.4. In the source code there > is a collection of benchmarks which includes 65 python scripts collected > from the internet. At average if Python 2.4 needs 24 hours to do some > job, pyvm can do it in 12 hours. > - pyvm is a virtual machine that can run Python 2.3/2.4 bytecode. > There is a compiler written in python (the 'pyc' program) which is > executed by the vm to compile source code to bytecode. It is very easy > to do advanced optimizations in python and the compiler produces > bytecode of high quality (speed and size). > """ > > http://students.ceid.upatras.gr/~sxanth/pyvm/ > I love how the first line says this project "most of the times produce the expected output". =) Until this thing can run parrotbench and the entire regression test suite for Python source modules I will consider the speed claims suspect. -Brett From martin at v.loewis.de Fri Feb 16 23:57:41 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 16 Feb 2007 23:57:41 +0100 Subject: [Python-Dev] Lack of sequential decompression in the zipfile module In-Reply-To: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> References: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> Message-ID: <45D636E5.9030603@v.loewis.de> Derek Shockey schrieb: > Since I was writing a script to work with potentially very large zipped > files, I took it upon myself to write an extract() method for zipfile, > which is essentially an adaption of the read() method modeled after > tarfile's extract(). I feel that this is something that should really be > provided in the zipfile module to make it more usable. I'm wondering if > this has been discussed before, or if anyone has ever viewed this as a > problem. I can post the code I wrote as a patch, though I'm not sure if > my file IO handling is as robust as it needs to be for the stdlib. I'd > appreciate any insight into the issue or direction on where I might > proceed from here so as to fix what I see as a significant problem. I think something like this is patch #1121142. Regards, Martin From python at rcn.com Sat Feb 17 00:06:14 2007 From: python at rcn.com (Raymond Hettinger) Date: Fri, 16 Feb 2007 15:06:14 -0800 Subject: [Python-Dev] Py2.6 ideas References: <2773CAC687FD5F4689F526998C7E4E5FF1EC9B@au3010avexu1.global.avaya.com> <014f01c75167$8a909140$ea146b0a@RaymondLaptop1> <45D557A5.3040804@v.loewis.de> <001701c751fa$1ea59370$ea146b0a@RaymondLaptop1> <45D62C50.2030502@v.loewis.de> Message-ID: <010c01c7521f$122ef6c0$ea146b0a@RaymondLaptop1> > Raymond Hettinger schrieb: >> d, g, v, t, r = model(somecontract) [MvL] > I find that line quite unreadable Of course, I can't give you the fully spelled-out line from proprietary code. But at this point we're just talking about the use cases for tuples with or without named attributes. Some functions return multiple values and some calls to those functions do tuple unpacking. That is ubiquitous throughout Python. If the tuple also happens to be a NamedTuple, you get tooltips for it (reminding you which fields are which) and any error messages will show the full repr with both the names and values. If not unpacked, then the attribute access is helpful. Something like contract.vega or testresult.failures or somesuch. Essentially, I'm proposing a variant of tuple that has self-documenting extra features: traditional positional arguments construction or option keyword argument construction annotated repr: Contract(type='put', strike=45, security='IBM', expirymonth=4) instead of: ('put', 45, 'IBM, 4) optional attribute access: contract.strike nice docstring for tooltips: 'Contract(type, strike, security, expirymonth)' The use cases are the same as the ones for tuples. The new type is just more self-documenting. That's all there is to it. FWIW, I've been using NamedTuples for at least six months and have found them to be a nice improvement over straight-tuples in situations where I can't easily remember what each tuple position represents. If added to the collections module, I think NamedTuples will become quite popular. > If you absolutely want tuple unpacking on a record-like object, > I'd suggest to support explicit tuple conversion, like > > d, g, v, t, r = model(somecontract).to_tuple() Entirely unnecessary. The goal is to have better tuples with low overhead and a near zero learning curve. Raymond From jcarlson at uci.edu Sat Feb 17 00:33:23 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 16 Feb 2007 15:33:23 -0800 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <010c01c7521f$122ef6c0$ea146b0a@RaymondLaptop1> References: <45D62C50.2030502@v.loewis.de> <010c01c7521f$122ef6c0$ea146b0a@RaymondLaptop1> Message-ID: <20070216152601.AD7E.JCARLSON@uci.edu> "Raymond Hettinger" wrote: > > Raymond Hettinger schrieb: > >> d, g, v, t, r = model(somecontract) > > [MvL] > > I find that line quite unreadable > > Of course, I can't give you the fully spelled-out line from proprietary code. > But at this point we're just talking about the use cases for tuples with or > without named attributes. Some functions return multiple values and > some calls to those functions do tuple unpacking. That is ubiquitous > throughout Python. If the tuple also happens to be a NamedTuple, > you get tooltips for it (reminding you which fields are which) and > any error messages will show the full repr with both the names > and values. > > If not unpacked, then the attribute access is helpful. Something > like contract.vega or testresult.failures or somesuch. For what it's worth, I've actually been using a similar approach with lists and global names of list indices because I needed a mutable structure, the list instance was significantly smaller than an object with __slots__ (by a factor of 3), and because using global constants was actually competitive with a __slots__ name lookup. After having seen your tuple recipe, I've been planning on converting it to a list-based recipe for the same benefits (except for unpacking) in my own code. Then again, I'm also looking forward to adding the tuple-based recipe to my own library for all of the reasons you outlined. - Josiah From collinw at gmail.com Sat Feb 17 02:01:20 2007 From: collinw at gmail.com (Collin Winter) Date: Fri, 16 Feb 2007 17:01:20 -0800 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <45D404AB.9080808@v.loewis.de> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> <45D2D447.4080408@v.loewis.de> <43aa6ff70702142040t19837debu511e215f998ec70e@mail.gmail.com> <45D404AB.9080808@v.loewis.de> Message-ID: <43aa6ff70702161701t5434b433v4e3961a35bdd05f5@mail.gmail.com> On 2/14/07, "Martin v. L?wis" wrote: > Collin Winter schrieb: > > What's inconsistent about it? That classes are being used for the > > _ast.{Add,Sub,Mult,etc} names? > > Exactly. These aren't names - they are nodes in the tree. All nodes > are instances of _ast.AST. > > > I don't see the need for both _ast.Add and _ast.Add.singleton or > > _ast.add or however else it might be spelled. I'd be perfectly happy > > doing something like "_ast.Add = object()" (when initializing the _ast > > module), so long as I can write "node.op is _ast.Add", "node.op == > > _ast.Add", or something equally brief and to-the-point. > > Would you like to do the same for Pass, Break, Continue, and Ellipsis? > > They are also "just names". If you make _ast.Add the entry in the > tree itself, why not _ast.Break? Or, if you have a way to deal with > _ast.Break, why can't the same way work for _ast.Add? But Pass, Break, Continue and Ellipsis aren't in the same category as Add, Mult, Div, etc.. The former stand alone, while the only time you'll ever encounter Add, Mult, Mod and co. is in the context of a BoolOp, BinOp or whatever. I could of course add separate visitor methods for each op type to my code generator, but that's a lot of boilerplate code for very little gain. I could use a dispatch table in visitBoolOp, and dispatch like "dispatch[type(node.op)]", but why? In this scenario, node.op doesn't convey any information other than its type. Add, Mult, etc are just values in an enumeration, so why not treat them that way and be able to write "dispatch[node.op]". So: what if _ast.Add and co. were singleton instances of _ast.AST or even of a subclass of AST, like _ast.Op? That way, we keep the consistency that all AST nodes are instances of _ast.AST and I get the nice property I'm looking for, ie, "node.op is _ast.Mult". Collin Winter From brett at python.org Sat Feb 17 02:11:08 2007 From: brett at python.org (Brett Cannon) Date: Fri, 16 Feb 2007 17:11:08 -0800 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <43aa6ff70702161701t5434b433v4e3961a35bdd05f5@mail.gmail.com> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> <45D2D447.4080408@v.loewis.de> <43aa6ff70702142040t19837debu511e215f998ec70e@mail.gmail.com> <45D404AB.9080808@v.loewis.de> <43aa6ff70702161701t5434b433v4e3961a35bdd05f5@mail.gmail.com> Message-ID: On 2/16/07, Collin Winter wrote: > On 2/14/07, "Martin v. L?wis" wrote: > > Collin Winter schrieb: > > > What's inconsistent about it? That classes are being used for the > > > _ast.{Add,Sub,Mult,etc} names? > > > > Exactly. These aren't names - they are nodes in the tree. All nodes > > are instances of _ast.AST. > > > > > I don't see the need for both _ast.Add and _ast.Add.singleton or > > > _ast.add or however else it might be spelled. I'd be perfectly happy > > > doing something like "_ast.Add = object()" (when initializing the _ast > > > module), so long as I can write "node.op is _ast.Add", "node.op == > > > _ast.Add", or something equally brief and to-the-point. > > > > Would you like to do the same for Pass, Break, Continue, and Ellipsis? > > > > They are also "just names". If you make _ast.Add the entry in the > > tree itself, why not _ast.Break? Or, if you have a way to deal with > > _ast.Break, why can't the same way work for _ast.Add? > > But Pass, Break, Continue and Ellipsis aren't in the same category as > Add, Mult, Div, etc.. The former stand alone, while the only time > you'll ever encounter Add, Mult, Mod and co. is in the context of a > BoolOp, BinOp or whatever. I could of course add separate visitor > methods for each op type to my code generator, but that's a lot of > boilerplate code for very little gain. I could use a dispatch table in > visitBoolOp, and dispatch like "dispatch[type(node.op)]", but why? In > this scenario, node.op doesn't convey any information other than its > type. Add, Mult, etc are just values in an enumeration, so why not > treat them that way and be able to write "dispatch[node.op]". > > So: what if _ast.Add and co. were singleton instances of _ast.AST or > even of a subclass of AST, like _ast.Op? That way, we keep the > consistency that all AST nodes are instances of _ast.AST and I get the > nice property I'm looking for, ie, "node.op is _ast.Mult". > If you look back in the history of this thread I think Martin suggested this: "I'd rather prefer if the singletons where exposed under a name, e.g. _ast.Add.singleton, or _ast.add (if that doesn't cause conflicts)." Since the both of you have proposed this I think this is the best way to move forward. -Brett From collinw at gmail.com Sat Feb 17 02:19:24 2007 From: collinw at gmail.com (Collin Winter) Date: Fri, 16 Feb 2007 17:19:24 -0800 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> <45D2D447.4080408@v.loewis.de> <43aa6ff70702142040t19837debu511e215f998ec70e@mail.gmail.com> <45D404AB.9080808@v.loewis.de> <43aa6ff70702161701t5434b433v4e3961a35bdd05f5@mail.gmail.com> Message-ID: <43aa6ff70702161719x7da767edjd4cf57a52586a0d@mail.gmail.com> On 2/16/07, Brett Cannon wrote: > On 2/16/07, Collin Winter wrote: > > On 2/14/07, "Martin v. L?wis" wrote: > > > Collin Winter schrieb: > > > > What's inconsistent about it? That classes are being used for the > > > > _ast.{Add,Sub,Mult,etc} names? > > > > > > Exactly. These aren't names - they are nodes in the tree. All nodes > > > are instances of _ast.AST. > > > > > > > I don't see the need for both _ast.Add and _ast.Add.singleton or > > > > _ast.add or however else it might be spelled. I'd be perfectly happy > > > > doing something like "_ast.Add = object()" (when initializing the _ast > > > > module), so long as I can write "node.op is _ast.Add", "node.op == > > > > _ast.Add", or something equally brief and to-the-point. > > > > > > Would you like to do the same for Pass, Break, Continue, and Ellipsis? > > > > > > They are also "just names". If you make _ast.Add the entry in the > > > tree itself, why not _ast.Break? Or, if you have a way to deal with > > > _ast.Break, why can't the same way work for _ast.Add? > > > > But Pass, Break, Continue and Ellipsis aren't in the same category as > > Add, Mult, Div, etc.. The former stand alone, while the only time > > you'll ever encounter Add, Mult, Mod and co. is in the context of a > > BoolOp, BinOp or whatever. I could of course add separate visitor > > methods for each op type to my code generator, but that's a lot of > > boilerplate code for very little gain. I could use a dispatch table in > > visitBoolOp, and dispatch like "dispatch[type(node.op)]", but why? In > > this scenario, node.op doesn't convey any information other than its > > type. Add, Mult, etc are just values in an enumeration, so why not > > treat them that way and be able to write "dispatch[node.op]". > > > > So: what if _ast.Add and co. were singleton instances of _ast.AST or > > even of a subclass of AST, like _ast.Op? That way, we keep the > > consistency that all AST nodes are instances of _ast.AST and I get the > > nice property I'm looking for, ie, "node.op is _ast.Mult". > > > > If you look back in the history of this thread I think Martin > suggested this: "I'd rather prefer > if the singletons where exposed under a name, e.g. _ast.Add.singleton, > or _ast.add (if that doesn't cause conflicts)." > > Since the both of you have proposed this I think this is the best way > to move forward. That's what I get for letting this thread drop for a few days. I read Martin's original proposal as wanting to have _ast.add (singleton instance) *in addition to* _ast.Add (presumably the class of _ast.add). To be clear, I want to replace the current _ast.Add (a class, subclassing AST) with an instance of _ast.Op or something comparable; making _ast.{Load,Store,Del,AugLoad,AugStore,Param} into instances of an _ast.Context class would be a part of this. Collin Winter From glyph at divmod.com Sat Feb 17 04:34:51 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Sat, 17 Feb 2007 03:34:51 -0000 Subject: [Python-Dev] Twisted Isn't Specific (was Re: Trial balloon: microthreads library in stdlib) Message-ID: <20070217033451.28203.298672314.divmod.xquotient.2313@joule.divmod.com> On 16 Feb, 06:30 pm, larry at hastings.org wrote: >I suggest it is possible to implement a PerfectReactor. I don't think this constitutes a sufficient existence proof. Perhaps you could write a prototype? There are a bunch of existing reactors you could either import or copy/paste from to bootstrap it, so you wouldn't necessarily need to do many of the platform-specific hacks it has already implemented for those loops. Personally I am highly skeptical that this is possible, and I didn't find any of your arguments convincing, but I would be ecstatic if it worked; it isn't *fun* maintaining 12 or so subtly incompatible implementations of the same interface :). I also think this discussion would be more appropriate to continue on the Twisted list, as we have veered pretty far afield of stdlib enhancements and are now talking about multiplexing method implementations, but I'll follow it wherever it continues. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070217/35c96ecf/attachment.htm From michele.simionato at gmail.com Sat Feb 17 08:10:13 2007 From: michele.simionato at gmail.com (Michele Simionato) Date: Sat, 17 Feb 2007 07:10:13 +0000 (UTC) Subject: [Python-Dev] Lack of sequential decompression in the zipfile module References: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> Message-ID: Derek Shockey gmail.com> writes: > > Though I am an avid Python programmer, I've never forayed into the area of developing Python itself, so I'm not exactly sure how all this works.I was confused (and somewhat disturbed) to discover recently that the zipfile module offers only one-shot decompression of files, accessible only via the read() method. It is my understanding that the module will handle files of up to 4 GB in size, and the idea of decompressing 4 GB directly into memory makes me a little queasy. Other related modules (zlib, tarfile, gzip, bzip2) all offer sequential decompression, but this does not seem to be the case for zipfile (even though the underlying zlib makes it easy to do). > Since I was writing a script to work with potentially very large zipped files, I took it upon myself to write an extract() method for zipfile, which is essentially an adaption of the read() method modeled after tarfile's extract(). I feel that this is something that should really be provided in the zipfile module to make it more usable. I'm wondering if this has been discussed before, or if anyone has ever viewed this as a problem. I can post the code I wrote as a patch, though I'm not sure if my file IO handling is as robust as it needs to be for the stdlib. I'd appreciate any insight into the issue or direction on where I might proceed from here so as to fix what I see as a significant problem. This is definitely a significant problem. We had to face it at work, and at the end we decided to use zipstream (http://www.doxdesk.com/software/py/zipstream.html) instead of zipfile, but of course having the functionality in the standard library would be much better. Michele Simionato From martin at v.loewis.de Sat Feb 17 09:15:26 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 17 Feb 2007 09:15:26 +0100 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <43aa6ff70702161701t5434b433v4e3961a35bdd05f5@mail.gmail.com> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> <45D2D447.4080408@v.loewis.de> <43aa6ff70702142040t19837debu511e215f998ec70e@mail.gmail.com> <45D404AB.9080808@v.loewis.de> <43aa6ff70702161701t5434b433v4e3961a35bdd05f5@mail.gmail.com> Message-ID: <45D6B99E.1030607@v.loewis.de> Collin Winter schrieb: > But Pass, Break, Continue and Ellipsis aren't in the same category as > Add, Mult, Div, etc.. The former stand alone That's not true. Pass, Break, Continue don't stand alone; they are members of the body sequence of some other statement (in particular for Break and Continue), you need some kind of loop around it. In any case, they are in the same category as they have no child nodes, which is a prerequisite for not creating objects. I can't see why the property "stand alone" should impact whether objects are created or not. > So: what if _ast.Add and co. were singleton instances of _ast.AST or > even of a subclass of AST, like _ast.Op? That way, we keep the > consistency that all AST nodes are instances of _ast.AST and I get the > nice property I'm looking for, ie, "node.op is _ast.Mult". I like this better. The base class you are looking for is _ast.operator; it should be already there (please do take a look at Python.asdl to see how I came to this conclusion, without studying the _ast module again). Regards, Martin From koder.mail at gmail.com Sat Feb 17 12:25:03 2007 From: koder.mail at gmail.com (KoDer) Date: Sat, 17 Feb 2007 13:25:03 +0200 Subject: [Python-Dev] urllib2 EP + decr. startup time In-Reply-To: <5.1.1.6.0.20070216124921.04799d38@sparrow.telecommunity.com> References: <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> <5.1.1.6.0.20070216124921.04799d38@sparrow.telecommunity.com> Message-ID: <6e8b06c90702170325m3a1ee9fbmc8a23ab4b62f3498@mail.gmail.com> > Right -- most of your problem will be solved by creating > 'C:\\WINDOWS\\system32\\python25.zip', containing the contents of > C:\\Python25\\lib\\. C:\\Python25\\lib\\. contain *many* packages with .dll files - i can't just zip it. wxPython,pyOpenGL,PIL,tk and so on. On Fedora 6 more than 40% dirs of /usr/lib/site-packages contained .so files. Some of them add dirs to path (wx,PIL,Gtk,...). yum,apt and other will bee very angry if i zip site-packages directory. I don't known any package manager which can properly work with packages installed in archive. Are setuptools can work properly with packages packed in one big zip archive (i really don't known)? And finally - if it's so easy why this don't done already? Python widely used in many linux distros and i don't known any one which can install even standard library in zip archive. Most of users can't done it(because they don't known about python at all). Or this just because lack of time? Yesterday i test some programs with strace and receive follow results: command num of sys_calls num of FILE_NOT_FOUND python -c "pass" 2807 619 ~20% yum 20263 11282 >50% pychecker 6181 2527 ~40% meld(nice GUI merge util) 16075 8024 50% ipython < exit.txt 16448 8957 >50% (exit txt contain "exit()\n") (if filter some of FILE_NOT_FOUND which are not produced by python modules search) BTW. In trace results many call chain like this: open("/usr/lib/python2.4/site-packages/Durus-3.6-py2.4-linux-i686.egg", O_RDONLY|O_LARGEFILE) = 6 ...... _llseek(6, 98304, [98304], SEEK_SET) = 0 read(6, "\340\377\224\322\373C\200\177.\245\367\205\0\307x\207\r"..., 4096) = 4096 _llseek(6, 102400, [102400], SEEK_SET) = 0 _llseek(6, 102400, [102400], SEEK_SET) = 0 _llseek(6, 102400, [102400], SEEK_SET) = 0 ..... and so on. As i understand all _llseek(6, 102400, [102400], SEEK_SET) = 0 calls after first are just heating air. -- K.Danilov aka KoDer From martin at v.loewis.de Sat Feb 17 15:39:31 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 17 Feb 2007 15:39:31 +0100 Subject: [Python-Dev] urllib2 EP + decr. startup time In-Reply-To: <6e8b06c90702170325m3a1ee9fbmc8a23ab4b62f3498@mail.gmail.com> References: <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> <5.1.1.6.0.20070216124921.04799d38@sparrow.telecommunity.com> <6e8b06c90702170325m3a1ee9fbmc8a23ab4b62f3498@mail.gmail.com> Message-ID: <45D713A3.5030603@v.loewis.de> KoDer schrieb: > open("/usr/lib/python2.4/site-packages/Durus-3.6-py2.4-linux-i686.egg", > O_RDONLY|O_LARGEFILE) = 6 > ...... > _llseek(6, 98304, [98304], SEEK_SET) = 0 > read(6, "\340\377\224\322\373C\200\177.\245\367\205\0\307x\207\r"..., > 4096) = 4096 > _llseek(6, 102400, [102400], SEEK_SET) = 0 > _llseek(6, 102400, [102400], SEEK_SET) = 0 > _llseek(6, 102400, [102400], SEEK_SET) = 0 > ..... > and so on. As i understand all > _llseek(6, 102400, [102400], SEEK_SET) = 0 > calls after first are just heating air. If you want to implement a patch to eliminate unnecessary system calls, please submit it to sf.net/projects/python. Regards, Martin From dustin at v.igoro.us Sat Feb 17 17:29:09 2007 From: dustin at v.igoro.us (dustin at v.igoro.us) Date: Sat, 17 Feb 2007 10:29:09 -0600 Subject: [Python-Dev] generic async io (was: microthreading vs. async io) In-Reply-To: <45D4FA91.6000703@canterbury.ac.nz> References: <45D4FA91.6000703@canterbury.ac.nz> Message-ID: <20070217162909.GB5718@v.igoro.us> On Fri, Feb 16, 2007 at 01:28:01PM +1300, Greg Ewing wrote: > Nick Maclaren wrote: > > > Threading > > --------- > > > > An I/O operation passes a buffer, length, file and action and receives a > > token back. > > You seem to be using the word "threading" in a completely > different way than usual here, which may be causing some > confusion. According to subsequent clarification, the kind of IO Nick is talking about is the sort of thing described recently on kerneltrap: http://kerneltrap.org/node/7728 (although he was referring specifically to POSIX async IO) Dustin From pje at telecommunity.com Sat Feb 17 19:02:45 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 17 Feb 2007 13:02:45 -0500 Subject: [Python-Dev] urllib2 EP + decr. startup time In-Reply-To: <6e8b06c90702170325m3a1ee9fbmc8a23ab4b62f3498@mail.gmail.co m> References: <5.1.1.6.0.20070216124921.04799d38@sparrow.telecommunity.com> <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> <5.1.1.6.0.20070216124921.04799d38@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20070217125913.037e0ae8@sparrow.telecommunity.com> At 01:25 PM 2/17/2007 +0200, KoDer wrote: >C:\\Python25\\lib\\. contain *many* packages with .dll files - i can't >just zip it. >wxPython,pyOpenGL,PIL,tk and so on. On Fedora 6 more than 40% dirs of >/usr/lib/site-packages contained .so files. Some of them add dirs to path >(wx,PIL,Gtk,...). I'm not talking about the subdirectories under lib, just lib itself. That is, the Python standard library. Stdlib DLLs are in a separate subdirectory ('DLLs' on Windows, 'lib-dynload' most other places). >Are setuptools can work properly with packages packed in one big zip archive >(i really don't known)? I don't follow you; this has nothing to do with setuptools. It's a feature of Python since version 2.3, but as far as I know nobody's ever set up the build machinery to create the necessary zipfile when Python is installed. Perhaps that would be a nice place to begin your patch: a script to create a stdlib zipfile in the platform-appropriate location, that can run after the bytecode compilation of the stdlib modules, or that users can run on older versions of Python to do the same thing. From collinw at gmail.com Sat Feb 17 19:16:13 2007 From: collinw at gmail.com (Collin Winter) Date: Sat, 17 Feb 2007 10:16:13 -0800 Subject: [Python-Dev] Recent experience with the _ast module In-Reply-To: <45D6B99E.1030607@v.loewis.de> References: <43aa6ff70702121627p5dbb91c0r7ec5d9339c29bf33@mail.gmail.com> <43aa6ff70702131700k757acc85y1faf4b1e6bf97d24@mail.gmail.com> <45D2D447.4080408@v.loewis.de> <43aa6ff70702142040t19837debu511e215f998ec70e@mail.gmail.com> <45D404AB.9080808@v.loewis.de> <43aa6ff70702161701t5434b433v4e3961a35bdd05f5@mail.gmail.com> <45D6B99E.1030607@v.loewis.de> Message-ID: <43aa6ff70702171016n12bb8909rb7af26798cb7c9ed@mail.gmail.com> On 2/17/07, "Martin v. L?wis" wrote: > Collin Winter schrieb: > > But Pass, Break, Continue and Ellipsis aren't in the same category as > > Add, Mult, Div, etc.. The former stand alone > > That's not true. Pass, Break, Continue don't stand alone; they are > members of the body sequence of some other statement (in particular > for Break and Continue), you need some kind of loop around it. > > In any case, they are in the same category as they have no child > nodes, which is a prerequisite for not creating objects. I can't > see why the property "stand alone" should impact whether objects > are created or not. I phrased that poorly. If Pass, Break and Continue were made into singleton instances, they wouldn't be able to be handled by the same visitor dispatch routine as the other statement types. Taking Demo/parser/unparse.py as an example, Unparser.dispatch would have to change to handle Break, Pass and Continue specially from the others. Changing the operator and context nodes to be singleton instances involves just modifying the dispatch tables used by _BoolOp, _BinOp, etc. Collin Winter From pedronis at openendsystems.com Sat Feb 17 20:24:41 2007 From: pedronis at openendsystems.com (Samuele Pedroni) Date: Sat, 17 Feb 2007 20:24:41 +0100 Subject: [Python-Dev] pypy-0.99.0: new object spaces, optimizations, configuration ... Message-ID: <45D75679.3040709@openendsystems.com> ================================================================= pypy-0.99.0: new object spaces, optimizations, configuration ... ================================================================= Welcome to the PyPy 0.99.0 release - a major snapshot and milestone of the last 8 months of work and contributions since PyPy-0.9.0 came out in June 2006! Main entry point for getting-started/download and documentation: http://codespeak.net/pypy/dist/pypy/doc/index.html Further below you'll find some notes about PyPy, the 0.99.0 highlights and our aims for PyPy 1.0. have fun, the PyPy team, Samuele Pedroni, Carl Friedrich Bolz, Armin Rigo, Michael Hudson, Maciej Fijalkowski, Anders Chrigstroem, Holger Krekel, Guido Wesdorp and many others: http://codespeak.net/pypy/dist/pypy/doc/contributor.html What is PyPy? ================================ Technically, PyPy is both a Python Interpreter implementation and an advanced Compiler, actually a framework for implementing dynamic languages and generating virtual machines for them. The Framework allows for alternative frontends and for alternative backends, currently C, LLVM and .NET. For our main target "C", we can can "mix in" different Garbage Collectors and threading models, including micro-threads aka "Stackless". The inherent complexity that arises from this ambitious approach is mostly kept away from the Python interpreter implementation, our main frontend. Socially, PyPy is a collaborative effort of many individuals working together in a distributed and sprint-driven way since 2003. PyPy would not have gotten as far without the coding, feedback and general support from numerous people. Formally, many of the current developers are involved in executing an EU contract with the goal of exploring and researching new approaches to Language/Compiler development and software engineering. This contract's duration is about to end March 2007 and we are working and preparing the according final review which is scheduled for May 2007. Key 0.99.0 Features ===================== * new object spaces: - Tainting: a 270-line proxy object space tracking and boxing sensitive information within an application. A tainted object is completely barred from crossing an I/O barrier, such as writing to files, databases or sockets. This allows to significantly reduce the effort of e.g. security reviews to the few places where objects are "declassified" in order to send information across I/O barriers. - Transparent proxies: allow to customize both application and builtin objects from application level code. Works as an addition to the Standard Object Space (and is translatable). For details see http://codespeak.net/pypy/dist/pypy/doc/proxy.html * optimizations: - Experimental new optimized implementations for various built in Python types (strings, dicts, lists) - Optimized builtin lookups to not require any dictionary lookups if the builtin is not shadowed by a name in the global dictionary. - Improved inlining (now also working for higher level backends) and malloc removal. - twice the speed of the 0.9 release, overall 2-3 slower than CPython * High level backends: - It is now possible to translate the PyPy interpreter to run on the .NET platform, which gives a very compliant (but somewhat slow) Python interpreter. - the JavaScript backend has evolved to a point where it can be used to write AJAX web applications with it. This is still an experimental technique, though. For demo applications see: http://play1.codespeak.net:8008/ * new configuration system: There is a new comprehensive configuration system that allows fine-grained configuration of the PyPy standard interpreter and the translation process. * new and improved modules: Since the last release, the signal, mmap, bz2 and fcntl standard library modules have been implemented for PyPy. The socket, _sre and os modules have been greatly improved. In addition we added a the pypymagic module that contains PyPy-specific functionality. * improved file implementation: Our file implementation was ported to RPython and is therefore faster (and not based on libc). * The stability of stackless features was greatly improved. For more details see: http://codespeak.net/pypy/dist/pypy/doc/stackless.html * RPython library: The release contains our emerging RPython library that tries to make programming in RPython more pleasant. It contains an experimental parser generator framework. For more details see: http://codespeak.net/pypy/dist/pypy/doc/rlib.html * improved documentation: - extended documentation about stackless features: http://codespeak.net/pypy/dist/pypy/doc/stackless.html - PyPy video documentation: eight hours of talks, interviews and features: http://codespeak.net/pypy/dist/pypy/doc/video-index.html - technical reports about various aspects of PyPy: http://codespeak.net/pypy/dist/pypy/doc/index-report.html The entry point to all our documentation is: http://codespeak.net/pypy/dist/pypy/doc/index.html What about 1.0? ====================== In the last week leading up to the release, we decided to go for tagging the release as 0.99.0, mainly because we have some efforts pending to integrate and complete research and coding work: * the JIT Compiler Generator is ready, but not fully integrated with the PyPy interpreter. As a result, the JIT does not give actual speed improvements yet, so we chose to leave it out of the 0.99 release: the result doesn't meet yet the speed expectations that we set for ourselves - and which some blogs and people have chosen as the main criterium for looking at PyPy. * the extension enabling runtime changes of the Python grammar is not yet integrated. This will be used to provide Aspect-Oriented Programming extensions and Design by Contract facilities in PyPy. * the Logic object space, which provides Logic Variables in PyPy, needs to undergo a bit more testing. A constraint problem solver extension module is ready, and needs to be integrated with the codebase. PyPy 0.99 is the start for getting to 1.0 end of March 2007, which we intend to become a base for a longer (and more relaxed :) time to come. Funding partners and organisations ===================================================== PyPy development and activities happen as an open source project and with the support of a consortium partially funded by a 28 months European Union IST research grant. The full partners of that consortium are: Heinrich-Heine University (Germany), Open End (Sweden) merlinux GmbH (Germany), tismerysoft GmbH (Germany) Logilab Paris (France), DFKI GmbH (Germany) ChangeMaker (Sweden), Impara (Germany) From alan.mcintyre at gmail.com Sat Feb 17 20:48:38 2007 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Sat, 17 Feb 2007 14:48:38 -0500 Subject: [Python-Dev] Lack of sequential decompression in the zipfile module In-Reply-To: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> References: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> Message-ID: <1d36917a0702171148v1a118a8i8bc98dbb6f6d40a2@mail.gmail.com> On 2/16/07, Derek Shockey wrote: > Since I was writing a script to work with potentially very large zipped > files, I took it upon myself to write an extract() method for zipfile, which > is essentially an adaption of the read() method modeled after tarfile's > extract(). I feel that this is something that should really be provided in > the zipfile module to make it more usable. I'm wondering if this has been > discussed before, or if anyone has ever viewed this as a problem. I can post > the code I wrote as a patch, though I'm not sure if my file IO handling is > as robust as it needs to be for the stdlib. I'd appreciate any insight into > the issue or direction on where I might proceed from here so as to fix what > I see as a significant problem. I ran into the same thing and made a patch a long while ago (the one Martin mentioned): https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 I am actually working on it this weekend; if you'd like to exchange code/test cases/whatever feel free to send me your stuff. I'll try to get a patch that works against the trunk posted today or tomorrow if you want to try it out. Cheers, Alan From martin at v.loewis.de Sun Feb 18 10:35:22 2007 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 18 Feb 2007 10:35:22 +0100 Subject: [Python-Dev] NOARGS_NULL Message-ID: <45D81DDA.5060905@v.loewis.de> Patch #1648268 corrects a huge load of errors in Python wrt. incorrect function pointers (i.e. functions called with a different signature then declared, through pointers). The rationale for this patch is that the submitter apparently has a platform where Python crashes in the face of these errors. I believe the patch is correct, and would like to apply it. The patch also renames many function arguments: parameters in a METH_NOARGS function get consistently named NOARGS_NULL (currently often called 'unused' or 'noargs'); the second parameter to getters gets consistently named 'closure' (it's called closure in many places already, and 'unused' in others). I would also apply this part of the change, and both to the trunk and Python 2.5. Objections? Regards, Martin From koder.mail at gmail.com Sun Feb 18 14:12:04 2007 From: koder.mail at gmail.com (KoDer) Date: Sun, 18 Feb 2007 15:12:04 +0200 Subject: [Python-Dev] urllib2 EP + decr. startup time In-Reply-To: <5.1.1.6.0.20070217125913.037e0ae8@sparrow.telecommunity.com> References: <5.1.1.6.0.20070216114230.03ae7b00@sparrow.telecommunity.com> <5.1.1.6.0.20070216124921.04799d38@sparrow.telecommunity.com> <5.1.1.6.0.20070217125913.037e0ae8@sparrow.telecommunity.com> Message-ID: <6e8b06c90702180512q6d73c9b5x491a7f4a2e3a6a93@mail.gmail.com> 2007/2/17, Phillip J. Eby : > > I don't follow you; this has nothing to do with setuptools. It's a feature > of Python since version 2.3, > I mean install/update/delete package to exist zip archive, which may contain many other packages(some time it's hart to understand what i write not on native language , sorry ). > but as far as I know nobody's ever set up the > build machinery to create the necessary zipfile when Python is > installed. Perhaps that would be a nice place to begin your patch: a > script to create a stdlib zipfile in the platform-appropriate location, > that can run after the bytecode compilation of the stdlib modules, or that > users can run on older versions of Python to do the same thing. I already work on this ). -- K.Danilov aka KoDer From anugupta at pu.ac.in Mon Feb 19 10:45:44 2007 From: anugupta at pu.ac.in (Anu Gupta DCSA) Date: Mon, 19 Feb 2007 15:15:44 +0530 Subject: [Python-Dev] A Survey on Defect Management Practices in Free/Open Source Software Message-ID: <20070219094514.M12603@pu.ac.in> Hi Python developers I seek help from designers, developers, testers,defect fixers,project managers or playing any other key role in Free/Open Source software development or maintenence in carrying out a study to identify practices and problems of defect management in various Free/Open Source Software projects. The insights gained from the study can further help us to extract publicly accessible defect data and determine impact of defect management practices on software quality. Please spend a few minutes of your precious time to fill up the Questionnaire. The most of the questions follow multiple choice formats and are quite easy to answer. To have the Online Questionnaire, please visit: http://anu.puchd.ac.in/phpESP/public/survey.php?name=FOSS_Defect_Survey (You can also copy and paste this link into your browser, and hit the 'Return' key.) I hope you will find all the questions interesting and thought-provoking. Your answers will be kept anonymous.The data thus collected will only be used for research purpose.It would be nice if you may further refer this mail to others actively engaged with Free/Open Source Software development. If you have any query or suggestions then feel free to contact. Thank You With regards, Anu Gupta Senior Lecturer Department of Computer Science and Applications, Panjab University, Chandigarh. INDIA In case of any problem in accessing/using the above mentioned link please contact: E-mail: anugupta at pu.ac.in anugupta98 at gmail.com From ben at redfrontdoor.org Mon Feb 19 14:03:08 2007 From: ben at redfrontdoor.org (Ben North) Date: Mon, 19 Feb 2007 13:03:08 +0000 Subject: [Python-Dev] Wrapping up 'dynamic attribute' discussion Message-ID: <1171890188.45d9a00c2dc29@imp.hosting365.ie> I wrote: > I've sent an updated version of PEP 363 to the editors, which > includes the following summary of the discussion. I hope I've > captured the important points, but please let me know if there's > something important I've left out or misrepresented. There were a couple of points made off-list, which I've incorporated. The final version of the PEP is now with the editors. Ben. From michele.simionato at gmail.com Tue Feb 20 06:25:24 2007 From: michele.simionato at gmail.com (Michele Simionato) Date: Tue, 20 Feb 2007 05:25:24 +0000 (UTC) Subject: [Python-Dev] Py2.6 ideas References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> Message-ID: Raymond Hettinger verizon.net> writes: > * Add a pure python named_tuple class to the collections module. I've been > using the class for about a year and found that it greatly improves the > usability of tuples as records. > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 The implementation of this recipe is really clean and I like it a lot (I even think of including it in our codebase), but there are a few issues that I would like to point out. 1. I find the camelcase confusing and I think that NamedTuple should be spelled namedtuple, since is a function, not a class. The fact that it returns classes does not count ;) 2. I agree with Giovanni Bajo, the constructor signature should be consistent with regular tuples. For instance I want to be able to map a named tuple over a record set as returned by fetchall. 3. I would like to pass the list of the fields as a sequence, not as a string. It would be more consistent and it would make easier the programmatic creation of NamedTuple classes at runtime. 4. I want help(MyNamedTuple) to work well; in particular it should display the right module name. That means that in the m dictionary you should add a __module__ attribute: __module__ = sys._getframe(1).f_globals['__name__'] 5. The major issue is that pickle does work with named tuples since the __module__ attribute is wrong. The suggestion in #4 would solve even this issue for free. 6. The ability to pass a show function to the __repr__ feems over-engineering to me. In short, here is how I would change the recipe: import sys from operator import itemgetter def namedtuple(f): """Returns a new subclass of tuple with named fields. >>> Point = namedtuple('Point x y'.split()) >>> Point.__doc__ # docstring for the new class 'Point(x, y)' >>> p = Point((11,), y=22) # instantiate with positional args or keywords >>> p[0] + p[1] # works just like the tuple (11, 22) 33 >>> x, y = p # unpacks just like a tuple >>> x, y (11, 22) >>> p.x + p.y # fields also accessable by name 33 >>> p # readable __repr__ with name=value style Point(x=11, y=22) """ typename, field_names = f[0], f[1:] nargs = len(field_names) def __new__(cls, args=(), **kwds): if kwds: try: args += tuple(kwds[name] for name in field_names[len(args):]) except KeyError, name: raise TypeError( '%s missing required argument: %s' % (typename, name)) if len(args) != nargs: raise TypeError( '%s takes exactly %d arguments (%d given)' % (typename, nargs, len(args))) return tuple.__new__(cls, args) template = '%s(%s)' % ( typename, ', '.join('%s=%%r' % name for name in field_names)) def __repr__(self): return template % self m = dict(vars(tuple)) # pre-lookup superclass methods (for faster lookup) m.update(__doc__= '%s(%s)' % (typename, ', '.join(field_names)), __slots__ = (), # no per-instance dict __new__ = __new__, __repr__ = __repr__, __module__ = sys._getframe(1).f_globals['__name__'], ) m.update((name, property(itemgetter(index))) for index, name in enumerate(field_names)) return type(typename, (tuple,), m) if __name__ == '__main__': import doctest TestResults = namedtuple(['TestResults', 'failed', 'attempted']) print TestResults(doctest.testmod()) From alan.mcintyre at gmail.com Tue Feb 20 06:40:25 2007 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Tue, 20 Feb 2007 00:40:25 -0500 Subject: [Python-Dev] Lack of sequential decompression in the zipfile module In-Reply-To: <1d36917a0702171148v1a118a8i8bc98dbb6f6d40a2@mail.gmail.com> References: <5728eb900702161153t27f98db2ub06ec892b6039519@mail.gmail.com> <1d36917a0702171148v1a118a8i8bc98dbb6f6d40a2@mail.gmail.com> Message-ID: <1d36917a0702192140q7d08f9a4gd440b1e688a5fa@mail.gmail.com> On 2/17/07, Alan McIntyre wrote: > I ran into the same thing and made a patch a long while ago (the one > Martin mentioned): > > https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 > > I am actually working on it this weekend; if you'd like to exchange > code/test cases/whatever feel free to send me your stuff. I'll try to > get a patch that works against the trunk posted today or tomorrow if > you want to try it out. Derek mentioned something that hadn't occurred to me: adding ZipFile.open() is helpful, but if we don't provide something analogous to TarFile.extract, anybody wanting to just extract a file from an archive to disk will have to write their own "get and write chunks from the file object" loop. Since this is likely to be a common task, it makes sense to me to provide this capability in the ZipFile class with an extract method that takes an optional path argument (defaulting to the working directory + path for file in archive). I'll add this to the patch unless somebody greatly disagrees or has a better idea. From python at rcn.com Tue Feb 20 07:47:34 2007 From: python at rcn.com (Raymond Hettinger) Date: Mon, 19 Feb 2007 22:47:34 -0800 Subject: [Python-Dev] Py2.6 ideas References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> Message-ID: <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> Thanks for the comments. Will experiment with them and get back to you.. The alternate naming is fine. At work, we call it ntuple(). The constructor signature has been experimented with several time and had best results in its current form which allows the *args for casting a record set returned by SQL or by the CSV module as in Point(*fetchall(s)), and it allows for direct construction with Point(2,3) without the slower and weirder form: Point((2,3)). Also, the current signature works better with keyword arguments: Point(x=2, y=3) or Point(2, y=3) which wouldn't be common but would be consistent with the relationship between keyword arguments and positional arguments in other parts of the language. The string form for the named tuple factory was arrived at because it was easier to write, read, and alter than its original form with a list of strings: Contract = namedtuple('Contract stock strike volatility expiration rate iscall') vs. Contract = namedtuple('Contract', 'stock', 'strike', 'volatility', 'expiration', 'rate', 'iscall') That former is easier to edit and to re-arrange. Either form is trivial to convert programmatically to the other and the definition step only occurs once while the use of the new type can appear many times throughout the code. Having experimented with both forms, I've found the string form to be best thought it seems a bit odd. Yet, the decision isn't central to the proposal and is still an open question. The __module__ idea is nice. Will add it. Likewise with pickling. The 'show' part of __repr__ was done for speed (localized access versus accessing the enclosing scope. Will rename it to _show to make it clear that it is not part of the API. Thanks for the accolades and the suggestions. Raymond ----- Original Message ----- From: "Michele Simionato" To: Sent: Monday, February 19, 2007 9:25 PM Subject: Re: [Python-Dev] Py2.6 ideas > Raymond Hettinger verizon.net> writes: >> * Add a pure python named_tuple class to the collections module. I've been >> using the class for about a year and found that it greatly improves the >> usability of tuples as records. >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 > > The implementation of this recipe is really clean and I like it a lot > (I even think of including it in our codebase), but there are a few issues > that I would like to point out. > > 1. I find the camelcase confusing and I think that NamedTuple should be > spelled namedtuple, since is a function, not a class. The fact that it > returns classes does not count ;) > > 2. I agree with Giovanni Bajo, the constructor signature should be consistent > with regular tuples. For instance I want to be able to map a named tuple > over a record set as returned by fetchall. > > 3. I would like to pass the list of the fields as a sequence, not as > a string. It would be more consistent and it would make easier > the programmatic creation of NamedTuple classes at runtime. > > 4. I want help(MyNamedTuple) to work well; in particular it should > display the right module name. That means > that in the m dictionary you should add a __module__ attribute: > > __module__ = sys._getframe(1).f_globals['__name__'] > > 5. The major issue is that pickle does work with named tuples since the > __module__ attribute is wrong. The suggestion in #4 would solve even > this issue for free. > > 6. The ability to pass a show function to the __repr__ feems over-engineering > to me. > > In short, here is how I would change the recipe: > > import sys > from operator import itemgetter > > def namedtuple(f): > """Returns a new subclass of tuple with named fields. > > >>> Point = namedtuple('Point x y'.split()) > >>> Point.__doc__ # docstring for the new class > 'Point(x, y)' > >>> p = Point((11,), y=22) # instantiate with positional args or keywords > >>> p[0] + p[1] # works just like the tuple (11, 22) > 33 > >>> x, y = p # unpacks just like a tuple > >>> x, y > (11, 22) > >>> p.x + p.y # fields also accessable by name > 33 > >>> p # readable __repr__ with name=value style > Point(x=11, y=22) > > """ > typename, field_names = f[0], f[1:] > nargs = len(field_names) > > def __new__(cls, args=(), **kwds): > if kwds: > try: > args += tuple(kwds[name] for name in field_names[len(args):]) > except KeyError, name: > raise TypeError( > '%s missing required argument: %s' % (typename, name)) > if len(args) != nargs: > raise TypeError( > '%s takes exactly %d arguments (%d given)' % > (typename, nargs, len(args))) > return tuple.__new__(cls, args) > > template = '%s(%s)' % ( > typename, ', '.join('%s=%%r' % name for name in field_names)) > > def __repr__(self): > return template % self > > m = dict(vars(tuple)) # pre-lookup superclass methods (for faster lookup) > m.update(__doc__= '%s(%s)' % (typename, ', '.join(field_names)), > __slots__ = (), # no per-instance dict > __new__ = __new__, > __repr__ = __repr__, > __module__ = sys._getframe(1).f_globals['__name__'], > ) > m.update((name, property(itemgetter(index))) > for index, name in enumerate(field_names)) > > return type(typename, (tuple,), m) > > > if __name__ == '__main__': > import doctest > TestResults = namedtuple(['TestResults', 'failed', 'attempted']) > print TestResults(doctest.testmod()) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/python%40rcn.com From python at rcn.com Tue Feb 20 08:58:58 2007 From: python at rcn.com (Raymond Hettinger) Date: Mon, 19 Feb 2007 23:58:58 -0800 Subject: [Python-Dev] Py2.6 ideas References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> Message-ID: <006e01c754c4$fab77a00$6d00a8c0@RaymondLaptop1> More thoughts on named tuples after trying-out all of Michele's suggestions: * The lowercase 'namedtuple' seemed right only because it's a function, but as a factory function, it is somewhat class-like. In use, 'NamedTuple' more closely matches my mental picture of what is happening and distinguishes what it does from the other two entries in collections, 'deque' and 'defaultdict' which are used to create instances instead of new types. * I remembered why the __repr__ function had a 'show' argument. I've changed the name now to make it more clear and added a docstring. The idea was the some use cases require that the repr exactly match the default style for tuples and the optional argument allowed for that possiblity with almost no performance hit. The updated recipe is at http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 Raymond From michele.simionato at gmail.com Tue Feb 20 09:52:53 2007 From: michele.simionato at gmail.com (Michele Simionato) Date: Tue, 20 Feb 2007 08:52:53 +0000 (UTC) Subject: [Python-Dev] Py2.6 ideas References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> <006e01c754c4$fab77a00$6d00a8c0@RaymondLaptop1> Message-ID: Raymond Hettinger rcn.com> writes: > > More thoughts on named tuples after trying-out all of Michele's suggestions: > > * The lowercase 'namedtuple' seemed right only because it's a function, but > as a factory function, it is somewhat class-like. In use, 'NamedTuple' more > closely matches my mental picture of what is happening and distinguishes > what it does from the other two entries in collections, 'deque' and > 'defaultdict' > which are used to create instances instead of new types. This is debatable. I remember Guido using lowercase for metaclasses in the famous descrintro essay. I still like more the lowercase for class factories. But I will not fight on this ;) > * I remembered why the __repr__ function had a 'show' argument. I've > changed the name now to make it more clear and added a docstring. > The idea was the some use cases require that the repr exactly match > the default style for tuples and the optional argument allowed for that > possiblity with almost no performance hit. But what about simply changing the __repr__? In [2]: Point = NamedTuple('Point','x','y') In [3]: Point(1,2) Out[3]: Point(x=1, y=2) In [4]: Point.__repr__ = tuple.__repr__ In [5]: Point(1,2) Out[5]: (1, 2) It feels clearer to me. Michele Simionato From python at rcn.com Tue Feb 20 09:57:56 2007 From: python at rcn.com (Raymond Hettinger) Date: Tue, 20 Feb 2007 00:57:56 -0800 Subject: [Python-Dev] Py2.6 ideas References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1><003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1><006e01c754c4$fab77a00$6d00a8c0@RaymondLaptop1> Message-ID: <001c01c754cd$3856ccf0$6d00a8c0@RaymondLaptop1> >> * I remembered why the __repr__ function had a 'show' argument. I've >> changed the name now to make it more clear and added a docstring. >> The idea was the some use cases require that the repr exactly match >> the default style for tuples and the optional argument allowed for that >> possiblity with almost no performance hit. > > But what about simply changing the __repr__? . . . > In [4]: Point.__repr__ = tuple.__repr__ Okay, that is the better way. Raymond From michele.simionato at gmail.com Tue Feb 20 10:45:47 2007 From: michele.simionato at gmail.com (Michele Simionato) Date: Tue, 20 Feb 2007 09:45:47 +0000 (UTC) Subject: [Python-Dev] Py2.6 ideas References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> Message-ID: Raymond Hettinger wrote: > The constructor signature has been experimented with > several time and had best results in its current form > which allows the *args for casting a record set returned > by SQL or by the CSV module as in Point(*fetchall(s)), I think you mean something like [Point(*tup) for tup in fetchall(s)], which I don't like for the reasons explained later. > and it allows for direct construction with Point(2,3) without the > slower and weirder form: Point((2,3)). Also, the current signature > works better with keyword arguments: Point(x=2, y=3) or > Point(2, y=3) which wouldn't be common but would be > consistent with the relationship between keyword arguments > and positional arguments in other parts of the language. I don't buy this argument. Yes, Point(2,3) is nicer than Point((2,3)) in the interactive interpreter and in the doctests, but in real life one has always tuples coming as return values from functions. Consider your own example, TestResults(*doctest.testmod()). I will argue that the * does not feel particularly good and that it would be better to just write TestResults(doctest.testmod()). Moreover I believe that having a subclass constructor incompatible with the base class constructor is very evil. First of all, you must be consistent with the tuple constructor, not with "other parts of the language". Finally I did some timing of code like this:: from itertools import imap Point = namedtuple('Point x y'.split()) lst = [(i, i*i) for i in range(500)] def with_imap(): for _ in imap(Point, lst): pass def with_star(): for _ in (Point(*t) for t in lst): pass and as expected the performances are worse with the * notation. In short, I don't feel any substantial benefit coming from the *args constructor. > The string form for the named tuple factory was arrived at > because it was easier to write, read, and alter than its original > form with a list of strings: > Contract = namedtuple('Contract stock strike volatility > expiration rate > iscall') > vs. > Contract = namedtuple('Contract', 'stock', 'strike', 'volatility', > 'expiration', 'rate', 'iscall') > That former is easier to edit and to re-arrange. Either form is trivial to > convert > programmatically to the other and the definition step only occurs > once while the > use of the new type can appear many times throughout the code. > Having experimented with both forms, I've found the string form to > be best thought it seems a bit odd. Yet, the decision isn't central to > the proposal and is still an open question. ``Contract = namedtuple('Contract stock strike volatility expiration rate iscall'.split())`` is not that bad either, but I agree that this is a second order issue. Michele Simionato From raymond.hettinger at verizon.net Tue Feb 20 10:47:01 2007 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue, 20 Feb 2007 01:47:01 -0800 Subject: [Python-Dev] Py_ssize_t Message-ID: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> After thinking more about Py_ssize_t, I'm surprised that we're not hearing about 64 bit users having a couple of major problems. If I'm understanding what was done for dictionaries, the hash table can grow larger than the range of hash values. Accordingly, I would expect large dictionaries to have an unacceptably large number of collisions. OTOH, we haven't heard a single complaint, so perhaps my understanding is off. The other area where I expected to hear wailing and gnashing of teeth is users compiling with third-party extensions that haven't been updated to a Py_ssize_t API and still use longs. I would have expected some instability due to the size mismatches in function signatures -- the difference would only show-up with giant sized data structures -- the bigger they are, the harder they fall. OTOH, there have not been any compliants either -- I would have expected someone to submit a patch to pyport.h that allowed a #define to force Py_ssize_t back to a long so that the poster could make a reliable build that included non-updated third-party extensions. In the absence of a bug report, it's hard to know whether there is a real problem. Have all major third-party extensions adopted Py_ssize_t or is some divine force helping unconverted extensions work with converted Python code? Maybe the datasets just haven't gotten big enough yet. Raymond From ronaldoussoren at mac.com Tue Feb 20 11:00:38 2007 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 20 Feb 2007 11:00:38 +0100 Subject: [Python-Dev] Py_ssize_t In-Reply-To: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> Message-ID: <85E05AE1-838D-4B97-963F-61FF3764F66E@mac.com> On 20 Feb, 2007, at 10:47, Raymond Hettinger wrote: > > The other area where I expected to hear wailing and gnashing of > teeth is users > compiling with third-party extensions that haven't been updated to > a Py_ssize_t > API and still use longs. I would have expected some instability > due to the size > mismatches in function signatures -- the difference would only show- > up with > giant sized data structures -- the bigger they are, the harder they > fall. OTOH, > there have not been any compliants either -- I would have expected > someone to > submit a patch to pyport.h that allowed a #define to force > Py_ssize_t back to a > long so that the poster could make a reliable build that included > non-updated > third-party extensions. Maybe that's because most sane 64-bit systems use LP64 and therefore don't have any problems with mixing Py_ssize_t and long. AFAIK Windows is the only major platform that doesn't use the LP64 model and 64-bit windows isn't used a lot. Ronald From fuzzyman at voidspace.org.uk Tue Feb 20 11:17:49 2007 From: fuzzyman at voidspace.org.uk (Fuzzyman) Date: Tue, 20 Feb 2007 10:17:49 +0000 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> Message-ID: <45DACACD.30805@voidspace.org.uk> Michele Simionato wrote: >Raymond Hettinger verizon.net> writes: > > >>* Add a pure python named_tuple class to the collections module. I've been >>using the class for about a year and found that it greatly improves the >>usability of tuples as records. >>http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 >> >> >[snip..] >4. I want help(MyNamedTuple) to work well; in particular it should > display the right module name. That means > that in the m dictionary you should add a __module__ attribute: > > __module__ = sys._getframe(1).f_globals['__name__'] > > > Hello all, If this is being considered for inclusion in the standard library, using _getframe' hackery will guarantee that it doesn't work with alternative implementations of Python (like IronPython at least which doesn't have Python stack frames). At least wrapping it in a try/except would be helpful. All the best, Michael Foord From martin at v.loewis.de Tue Feb 20 12:16:23 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 20 Feb 2007 12:16:23 +0100 Subject: [Python-Dev] Py_ssize_t In-Reply-To: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> Message-ID: <45DAD887.9050700@v.loewis.de> Raymond Hettinger schrieb: > If I'm understanding what was done for dictionaries, the hash table can grow > larger than the range of hash values. Accordingly, I would expect large > dictionaries to have an unacceptably large number of collisions. OTOH, we > haven't heard a single complaint, so perhaps my understanding is off. I think this would happen, but users don't have enough memory to notice it. For a dictionary with more than 4GEntries, you need 72GiB memory (8 byte for each key, value, and cached-hash). So you are starting to see massive collisions only when you have that much memory - plus in that dictionary, you would also need space for keys and values. Very few people have machines with 128+GiB main memory, so no complaints yet. But you are right: extending the hash value to be a 64-bit quantity was "forgotten", mainly because it isn't a count of something - and being "count of something" was the primary criterion for the 2.5 changes. > The other area where I expected to hear wailing and gnashing of teeth is users > compiling with third-party extensions that haven't been updated to a Py_ssize_t > API and still use longs. I would have expected some instability due to the size > mismatches in function signatures -- the difference would only show-up with > giant sized data structures -- the bigger they are, the harder they fall. OTOH, > there have not been any compliants either -- I would have expected someone to > submit a patch to pyport.h that allowed a #define to force Py_ssize_t back to a > long so that the poster could make a reliable build that included non-updated > third-party extensions. On most 64-bit systems, there is also an option to run 32-bit programs (atleast on AMD64, Sparc-64, and PPC64 there is). So people are more likely to do that when they run into problems, rather than recompiling the 64-bit Python. > In the absence of a bug report, it's hard to know whether there is a real > problem. Have all major third-party extensions adopted Py_ssize_t or is some > divine force helping unconverted extensions work with converted Python code? I know Matthias Klose has fixed all extension modules in the entire Debian source to compile without warnings on 64-bit machines. They may not work all yet, but yes, for all modules in Debian, it has been fixed. Not sure whether Matthias is a divine force, but working for Canonical comes fairly close :-) > Maybe the datasets just haven't gotten big enough yet. Primarily that. We still have a few years ahead to find all bugs before people would start complaining that Python is unstable on 64-bit systems. By the time people would actually see problems, hopefully they all have been resolved. Regards, Martin From juancarlosuarez at ciudad.com.ar Tue Feb 20 12:59:22 2007 From: juancarlosuarez at ciudad.com.ar (Juan Carlos Suarez) Date: Tue, 20 Feb 2007 08:59:22 -0300 Subject: [Python-Dev] Welcome to the "Python-Dev" mailing list Message-ID: <006701c754e6$8f2d7240$dd70d4c9@user61af7958a3> Good morning, thanks a lot for your answer, I confirmed by this means my subscription. What I really wish and need is to be able to use Mailman as a freely distributer , as a distributer of my own mailing lists ,as a mailing list manager , to be able to send my mails by Mailman and to be able to open my own mail address. How should I do this. I need your instructions and help. I'm waiting for your answer soon Thanks, my regards, Juan Carlos Suarez You wrote: Welcome to the Python-Dev at python.org mailing list! If you are a new subscriber, please take the time to introduce yourself briefly in your first post. It is appreciated if you lurk around for a while before posting! :-) Additional information on Python's development process can be found in the Python Developer's Guide: http://www.python.org/dev/ To post to this list, send your email to: python-dev at python.org General information about the mailing list is at: http://mail.python.org/mailman/listinfo/python-dev If you ever want to unsubscribe or change your options (eg, switch to or from digest mode, change your password, etc.), visit your subscription page at: http://mail.python.org/mailman/options/python-dev/juancarlosuarez%40ciudad.com.ar You can also make such adjustments via email by sending a message to: Python-Dev-request at python.org with the word `help' in the subject or body (don't include the quotes), and you will get back a message with instructions. You must know your password to change your options (including changing the password, itself) or to unsubscribe. It is: casandra07 Normally, Mailman will remind you of your python.org mailing list passwords once every month, although you can disable this if you prefer. This reminder will also include instructions on how to unsubscribe or change your account options. There is also a button on your options page that will email your current password to you. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070220/081718e2/attachment.html From steve at holdenweb.com Tue Feb 20 14:00:09 2007 From: steve at holdenweb.com (Steve Holden) Date: Tue, 20 Feb 2007 08:00:09 -0500 Subject: [Python-Dev] Welcome to the "Python-Dev" mailing list In-Reply-To: <006701c754e6$8f2d7240$dd70d4c9@user61af7958a3> References: <006701c754e6$8f2d7240$dd70d4c9@user61af7958a3> Message-ID: <45DAF0D9.8010804@holdenweb.com> Juan Carlos: Please note that the python-dev list is for the developers of the Python language, rather than for Python application developer, so your query isn't really appropriate for this list. I would suggest that you read the mailman license carefully, as it will tell you what you are allowed to do with the software. Since Mailman is distributed under the GPL it is likely you can do all the things you want to. If you have any further queries then there is a Mailman mailing list to which you should refer. Please see http://www.gnu.org/software/mailman/ for further details. Good luck with your project. regards Steve Juan Carlos Suarez wrote: > > *Good morning*, thanks a lot for your answer, I confirmed by this means > my subscription. > What *I really wish and need* is to be able to use Mailman as a freely > distributer , as a distributer of my own mailing lists ,as a mailing list > manager , to be able to send my mails by Mailman and to be able to open > my own mail address. > How should I do this. I need your instructions and help. > I'm waiting for your answer soon > Thanks, my regards, *Juan Carlos Suarez* > > > > > You wrote: > Welcome to the Python-Dev at python.org mailing list! If you are a new > subscriber, please take the time to introduce yourself briefly in your > first post. It is appreciated if you lurk around for a while before > posting! :-) > > Additional information on Python's development process can be found in > the Python Developer's Guide: > > http://www.python.org/dev/ > > To post to this list, send your email to: > > python-dev at python.org > > General information about the mailing list is at: > > http://mail.python.org/mailman/listinfo/python-dev > > If you ever want to unsubscribe or change your options (eg, switch to > or from digest mode, change your password, etc.), visit your > subscription page at: > > > http://mail.python.org/mailman/options/python-dev/juancarlosuarez%40ciudad.com.ar > > > You can also make such adjustments via email by sending a message to: > > Python-Dev-request at python.org > > with the word `help' in the subject or body (don't include the > quotes), and you will get back a message with instructions. > > You must know your password to change your options (including changing > the password, itself) or to unsubscribe. It is: > > casandra07 > > Normally, Mailman will remind you of your python.org mailing list > passwords once every month, although you can disable this if you > prefer. This reminder will also include instructions on how to > unsubscribe or change your account options. There is also a button on > your options page that will email your current password to you. > > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From steve at holdenweb.com Tue Feb 20 14:00:09 2007 From: steve at holdenweb.com (Steve Holden) Date: Tue, 20 Feb 2007 08:00:09 -0500 Subject: [Python-Dev] Welcome to the "Python-Dev" mailing list In-Reply-To: <006701c754e6$8f2d7240$dd70d4c9@user61af7958a3> References: <006701c754e6$8f2d7240$dd70d4c9@user61af7958a3> Message-ID: <45DAF0D9.8010804@holdenweb.com> Juan Carlos: Please note that the python-dev list is for the developers of the Python language, rather than for Python application developer, so your query isn't really appropriate for this list. I would suggest that you read the mailman license carefully, as it will tell you what you are allowed to do with the software. Since Mailman is distributed under the GPL it is likely you can do all the things you want to. If you have any further queries then there is a Mailman mailing list to which you should refer. Please see http://www.gnu.org/software/mailman/ for further details. Good luck with your project. regards Steve Juan Carlos Suarez wrote: > > *Good morning*, thanks a lot for your answer, I confirmed by this means > my subscription. > What *I really wish and need* is to be able to use Mailman as a freely > distributer , as a distributer of my own mailing lists ,as a mailing list > manager , to be able to send my mails by Mailman and to be able to open > my own mail address. > How should I do this. I need your instructions and help. > I'm waiting for your answer soon > Thanks, my regards, *Juan Carlos Suarez* > > > > > You wrote: > Welcome to the Python-Dev at python.org mailing list! If you are a new > subscriber, please take the time to introduce yourself briefly in your > first post. It is appreciated if you lurk around for a while before > posting! :-) > > Additional information on Python's development process can be found in > the Python Developer's Guide: > > http://www.python.org/dev/ > > To post to this list, send your email to: > > python-dev at python.org > > General information about the mailing list is at: > > http://mail.python.org/mailman/listinfo/python-dev > > If you ever want to unsubscribe or change your options (eg, switch to > or from digest mode, change your password, etc.), visit your > subscription page at: > > > http://mail.python.org/mailman/options/python-dev/juancarlosuarez%40ciudad.com.ar > > > You can also make such adjustments via email by sending a message to: > > Python-Dev-request at python.org > > with the word `help' in the subject or body (don't include the > quotes), and you will get back a message with instructions. > > You must know your password to change your options (including changing > the password, itself) or to unsubscribe. It is: > > casandra07 > > Normally, Mailman will remind you of your python.org mailing list > passwords once every month, although you can disable this if you > prefer. This reminder will also include instructions on how to > unsubscribe or change your account options. There is also a button on > your options page that will email your current password to you. > > > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org -- Steve Holden +44 150 684 7255 +1 800 494 3119 Holden Web LLC/Ltd http://www.holdenweb.com Skype: holdenweb http://del.icio.us/steve.holden Blog of Note: http://holdenweb.blogspot.com See you at PyCon? http://us.pycon.org/TX2007 From barry at python.org Tue Feb 20 14:18:45 2007 From: barry at python.org (Barry Warsaw) Date: Tue, 20 Feb 2007 08:18:45 -0500 Subject: [Python-Dev] Py_ssize_t In-Reply-To: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> Message-ID: <58A0F359-90D6-4365-9175-9E1E0771AEDC@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 20, 2007, at 4:47 AM, Raymond Hettinger wrote: > The other area where I expected to hear wailing and gnashing of > teeth is users > compiling with third-party extensions that haven't been updated to > a Py_ssize_t > API and still use longs. I would have expected some instability > due to the size > mismatches in function signatures -- the difference would only show- > up with > giant sized data structures -- the bigger they are, the harder they > fall. OTOH, > there have not been any compliants either -- I would have expected > someone to > submit a patch to pyport.h that allowed a #define to force > Py_ssize_t back to a > long so that the poster could make a reliable build that included > non-updated > third-party extensions. When I did an experimental port of our big embedded app to Python 2.5, that's (almost) exactly what I did. I didn't add the #define to a Python header file, but to our own and it worked pretty well, IIRC. I never went farther than the experimental phase though. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRdr1QHEjvBPtnXfVAQIO0wP5Adr7c467NFn5fjmvcAemtvjg+3Tri0qV SHI6LF88tSYkxKLezTojXPFQ+kYTjgz1yLa1KuQ6W9Q8dhiKGUVu7ZqFT12IGcIV n6Yf0htkpGmq/3G7m7D7yWHQrQE3Ce3+f6tI/4aL5eQ3mgdo1y828sY/sCCc4fTC Ln2gSad6g/M= =QQ5y -----END PGP SIGNATURE----- From selliott4 at austin.rr.com Tue Feb 20 16:07:45 2007 From: selliott4 at austin.rr.com (Steven Elliott) Date: Tue, 20 Feb 2007 09:07:45 -0600 Subject: [Python-Dev] Making builtins more efficient In-Reply-To: <4410BE69.3080004@canterbury.ac.nz> References: <1141879691.11091.78.camel@grey> <440FF9CB.5030407@gmail.com> <79990c6b0603090400h25dd2c7ev3d5c379f6529f3c2@mail.gmail.com> <1141915806.11091.127.camel@grey> <4410BE69.3080004@canterbury.ac.nz> Message-ID: <1171984065.22648.47.camel@grey> I'm finally getting back into this. I'd like to take one more shot at it with a revised version of what I proposed before. For those of you that did not see the original thread it was about ways that accessing builtins could be more efficient. It's a bit much to summarize again now, but you should be able to find it in the archive with this subject and a date of 2006-03-08. On Fri, 2006-03-10 at 12:46 +1300, Greg Ewing wrote: > Steven Elliott wrote: > > One way of handling it is to > > alter STORE_ATTR (op code for assigning to mod.str) to always check to > > see if the key being assigned is one of the default builtins. If it is, > > then the module's indexed array of builtins is assigned to. > > As long as you're going to all that trouble, it > doesn't seem like it would be much harder to treat > all global names that way, instead of just a predefined > set. The compiler already knows all of the names that > are used as globals in the module's code. What I have in mind may be close to what you are suggesting above. My thought now is that builtins are a set of tokens that typically, but don't necessarily, point to the same objects in all modules. Such tokens, which I'll refer to as "global tokens", can be roughly broken into two sets: 1) Global tokens that typically point to the same object in all modules. 2) Global tokens that that are likely to point to the different objects (or be undefined) in different modules. Set 1) is pretty much the the builtins. "True" and "len" are likely to point to the same objects in all modules, but not necessarily. Set 2) might be things like "os" and "sys" which are often defined (imported) in modules, but not necessarily. Access to the globals of a module, including the current module, is done with one of three opcodes (LOAD_GLOBAL, LOAD_ATTR and LOAD_NAME). For each of these opcodes the following snippet of code from ceval.c (for LOAD_GLOBAL) is relevant to this discussion: /* This is the un-inlined version of the code above */ x = PyDict_GetItem(f->f_globals, w); if (x == NULL) { x = PyDict_GetItem(f->f_builtins, w); if (x == NULL) { load_global_error: format_exc_check_arg( PyExc_NameError, GLOBAL_NAME_ERROR_MSG, w); break; } } So, to avoid the hash table lookups above maybe the global tokens could be assigned an index value that is fixed for any given version of the interpreter and that is the same for all modules (that "True" is always index 7, "len" is always index 3, etc.) Once a set of indexes have been determined a new opcode, that I'll call "LOAD_GTOKEN", could be created that avoids the hash table lookup by functioning in a way that is similar to LOAD_FAST (pull a local variable value out of an array). For example, static references to "True" could always be compiled to LOAD_GTOKEN 7 (True) As to set 1) and set 2) that I mentioned above - there is only a need to distinguish between the two sets if a copy-on-write mechanism is used. That way global tokens that are likely to have their value changed (group 2) ) can all be together in one group so that only that group needs to be copied when one of the global tokens is written to. For example code such as: True = 1 print True would be compiled into something like: 1 LOAD_CONST 1 (1) STORE_GTOKEN1 7 (True) 2 LOAD_GTOKEN1 7 (True) PRINT_ITEM PRINT_NEWLINE Note that "1" has been appended to "STORE_GTOKEN" to indicate that group 1) is being worked with. The store command will copy the array of pointers once, the first time it is called. Just as a new opcode is needed for LOAD_GLOBAL one would be needed for LOAD_ATTR. Perhaps "LOAD_ATOKEN" would work. For example: amodule.len = my_len print amodule.len would be compiled into something like: 1 LOAD_GLOBAL 0 (my_len) LOAD_GLOBAL 1 (amodule) STORE_ATOKEN1 3 (len) 2 LOAD_GLOBAL 1 (amodule) LOAD_ATOKEN1 3 (len) PRINT_ITEM PRINT_NEWLINE LOAD_CONST 0 (None) RETURN_VALUE Note that it looks almost identical to the code that is currently generated, but the oparg "3" shown for the "LOAD_ATOKEN1" above indexes into an array (like LOAD_FAST) to get at the attribute directly whereas the oparg that would be shown for LOAD_ATTR is an index into an array of constants/strings which is then used to retrieve the attribute from the module's global hash table. > > That's great, but I'm curious if additional gains can be > > made be focusing just on builtins. > > As long as builtins can be shadowed, I can't see how > to make any extra use of the fact that it's a builtin. > A semantic change would be needed, such as forbidding > shadowing of builtins, or at least forbidding this > from outside the module. I now think that it best not to think of builtins as being a special case. What really matters is that if all modules agree on a way of indexing global tokens (possible since they are all compiled with the same compiler) then it should be possible to avoid any hash table lookups when those global tokens are read or written to. So while True: can be just as efficient as while 1: Is it reasonable to require the compiler to be aware of the set of builtins and their indexes at the time the bytecode is produced? That is one cost to consider - a less clean separation between the compiler and the meaning behind the code it compiles. One cost to consider is the memory consumed by unsused slots in the array of global tokens for a module if that module uses a small number of builtins, but also writes to one of them so that the array needs to be copied. But hopefully that won't be too bad. -- ----------------------------------------------------------------------- | Steven Elliott | selliott4 at austin.rr.com | ----------------------------------------------------------------------- From guido at python.org Tue Feb 20 16:48:21 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Feb 2007 07:48:21 -0800 Subject: [Python-Dev] Making builtins more efficient In-Reply-To: <1171984065.22648.47.camel@grey> References: <1141879691.11091.78.camel@grey> <440FF9CB.5030407@gmail.com> <79990c6b0603090400h25dd2c7ev3d5c379f6529f3c2@mail.gmail.com> <1141915806.11091.127.camel@grey> <4410BE69.3080004@canterbury.ac.nz> <1171984065.22648.47.camel@grey> Message-ID: If this is not a replay of an old message, please move the discussion to python-ideas. On 2/20/07, Steven Elliott wrote: > I'm finally getting back into this. I'd like to take one more shot at > it with a revised version of what I proposed before. > > For those of you that did not see the original thread it was about ways > that accessing builtins could be more efficient. It's a bit much to > summarize again now, but you should be able to find it in the archive > with this subject and a date of 2006-03-08. > > On Fri, 2006-03-10 at 12:46 +1300, Greg Ewing wrote: > > Steven Elliott wrote: > > > One way of handling it is to > > > alter STORE_ATTR (op code for assigning to mod.str) to always check to > > > see if the key being assigned is one of the default builtins. If it is, > > > then the module's indexed array of builtins is assigned to. > > > > As long as you're going to all that trouble, it > > doesn't seem like it would be much harder to treat > > all global names that way, instead of just a predefined > > set. The compiler already knows all of the names that > > are used as globals in the module's code. > > What I have in mind may be close to what you are suggesting above. My > thought now is that builtins are a set of tokens that typically, but > don't necessarily, point to the same objects in all modules. Such > tokens, which I'll refer to as "global tokens", can be roughly broken > into two sets: > 1) Global tokens that typically point to the same object in all > modules. > 2) Global tokens that that are likely to point to the different > objects (or be undefined) in different modules. > Set 1) is pretty much the the builtins. "True" and "len" are likely to > point to the same objects in all modules, but not necessarily. Set 2) > might be things like "os" and "sys" which are often defined (imported) > in modules, but not necessarily. > > Access to the globals of a module, including the current module, is done > with one of three opcodes (LOAD_GLOBAL, LOAD_ATTR and LOAD_NAME). For > each of these opcodes the following snippet of code from ceval.c (for > LOAD_GLOBAL) is relevant to this discussion: > /* This is the un-inlined version of the code above */ > x = PyDict_GetItem(f->f_globals, w); > if (x == NULL) { > x = PyDict_GetItem(f->f_builtins, w); > if (x == NULL) { > load_global_error: > format_exc_check_arg( > PyExc_NameError, > GLOBAL_NAME_ERROR_MSG, w); > break; > } > } > > So, to avoid the hash table lookups above maybe the global tokens could > be assigned an index value that is fixed for any given version of the > interpreter and that is the same for all modules (that "True" is always > index 7, "len" is always index 3, etc.) > > Once a set of indexes have been determined a new opcode, that I'll call > "LOAD_GTOKEN", could be created that avoids the hash table lookup by > functioning in a way that is similar to LOAD_FAST (pull a local variable > value out of an array). For example, static references to "True" could > always be compiled to > LOAD_GTOKEN 7 (True) > > As to set 1) and set 2) that I mentioned above - there is only a need to > distinguish between the two sets if a copy-on-write mechanism is used. > That way global tokens that are likely to have their value changed > (group 2) ) can all be together in one group so that only that group > needs to be copied when one of the global tokens is written to. For > example code such as: > True = 1 > print True > would be compiled into something like: > 1 LOAD_CONST 1 (1) > STORE_GTOKEN1 7 (True) > 2 LOAD_GTOKEN1 7 (True) > PRINT_ITEM > PRINT_NEWLINE > Note that "1" has been appended to "STORE_GTOKEN" to indicate that group > 1) is being worked with. The store command will copy the array of > pointers once, the first time it is called. > > Just as a new opcode is needed for LOAD_GLOBAL one would be needed for > LOAD_ATTR. Perhaps "LOAD_ATOKEN" would work. For example: > amodule.len = my_len > print amodule.len > would be compiled into something like: > 1 LOAD_GLOBAL 0 (my_len) > LOAD_GLOBAL 1 (amodule) > STORE_ATOKEN1 3 (len) > > 2 LOAD_GLOBAL 1 (amodule) > LOAD_ATOKEN1 3 (len) > PRINT_ITEM > PRINT_NEWLINE > LOAD_CONST 0 (None) > RETURN_VALUE > > Note that it looks almost identical to the code that is currently > generated, but the oparg "3" shown for the "LOAD_ATOKEN1" above indexes > into an array (like LOAD_FAST) to get at the attribute directly whereas > the oparg that would be shown for LOAD_ATTR is an index into an array of > constants/strings which is then used to retrieve the attribute from the > module's global hash table. > > > > That's great, but I'm curious if additional gains can be > > > made be focusing just on builtins. > > > > As long as builtins can be shadowed, I can't see how > > to make any extra use of the fact that it's a builtin. > > A semantic change would be needed, such as forbidding > > shadowing of builtins, or at least forbidding this > > from outside the module. > > I now think that it best not to think of builtins as being a special > case. What really matters is that if all modules agree on a way of > indexing global tokens (possible since they are all compiled with the > same compiler) then it should be possible to avoid any hash table > lookups when those global tokens are read or written to. So > while True: > can be just as efficient as > while 1: > > Is it reasonable to require the compiler to be aware of the set of > builtins and their indexes at the time the bytecode is produced? That > is one cost to consider - a less clean separation between the compiler > and the meaning behind the code it compiles. > > One cost to consider is the memory consumed by unsused slots in the > array of global tokens for a module if that module uses a small number > of builtins, but also writes to one of them so that the array needs to > be copied. But hopefully that won't be too bad. > > -- > ----------------------------------------------------------------------- > | Steven Elliott | selliott4 at austin.rr.com | > ----------------------------------------------------------------------- > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Feb 20 16:57:48 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Feb 2007 07:57:48 -0800 Subject: [Python-Dev] Py_ssize_t In-Reply-To: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> Message-ID: On 2/20/07, Raymond Hettinger wrote: > After thinking more about Py_ssize_t, I'm surprised that we're not hearing about > 64 bit users having a couple of major problems. > > If I'm understanding what was done for dictionaries, the hash table can grow > larger than the range of hash values. Accordingly, I would expect large > dictionaries to have an unacceptably large number of collisions. OTOH, we > haven't heard a single complaint, so perhaps my understanding is off. Not until the has table has 4 billion entries. I believe that would be 96 GB just for the hash table; plus probably at least that for that many unique key strings. Not to mention the values (but those needn't be unique). I think the benefit of 64-bit architecture start above using 2 or 3 GB of RAM, so there's quite a bit of expansion space for 64-bit users before they run into this theoretical problem. > The other area where I expected to hear wailing and gnashing of teeth is users > compiling with third-party extensions that haven't been updated to a Py_ssize_t > API and still use longs. I would have expected some instability due to the size > mismatches in function signatures -- the difference would only show-up with > giant sized data structures -- the bigger they are, the harder they fall. OTOH, > there have not been any compliants either -- I would have expected someone to > submit a patch to pyport.h that allowed a #define to force Py_ssize_t back to a > long so that the poster could make a reliable build that included non-updated > third-party extensions. > > In the absence of a bug report, it's hard to know whether there is a real > problem. Have all major third-party extensions adopted Py_ssize_t or is some > divine force helping unconverted extensions work with converted Python code? > Maybe the datasets just haven't gotten big enough yet. My suspicion is that building Python for an 64-bit address space is still a somewhat academic exercise. I know we don't do this at Google (we switch to other languages long before the datasets become so large we'd need a 64-bit address space for Python). What's your experience at EWT? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Tue Feb 20 17:11:29 2007 From: python at rcn.com (Raymond Hettinger) Date: Tue, 20 Feb 2007 08:11:29 -0800 Subject: [Python-Dev] Py_ssize_t References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> Message-ID: <003501c75509$cd21cec0$7400a8c0@RaymondLaptop1> > My suspicion is that building Python for an 64-bit address space is > still a somewhat academic exercise. I know we don't do this at Google > (we switch to other languages long before the datasets become so large > we'd need a 64-bit address space for Python). What's your experience > at EWT? Two people had some difficulty building non-upgraded third-party modules with Py2.5 on 64-bit machines (I think wxPython was one of the problems) but they either gave up or switched machines before we could isolate the problem and say for sure whether Py_ssize_t was the culprit. I had remembered the PEP saying that there might be some issues for non-upgraded third-party modules and have wondered whether others were similarly affected. Raymond From jimjjewett at gmail.com Tue Feb 20 18:24:41 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 20 Feb 2007 12:24:41 -0500 Subject: [Python-Dev] NamedTuple (was: Py2.6 ideas) Message-ID: Raymond Hettinger explained: > The constructor signature ... Point(*fetchall(s)), > and it allows for direct construction with Point(2,3) without the > slower and weirder form: Point((2,3)). If you were starting from scratch, I would agree whole-heartedly; this is one of my most frequent mistakes. The question is whether it makes sense to "fix" NamedTuple without also fixing regular tuple, list, set, etc. Assuming this goes to collections rather than builtins, would it be reasonable to include both versions? > Also, the current signature > works better with keyword arguments: Point(x=2, y=3) or > Point(2, y=3) which wouldn't be common but would be > consistent with the relationship between keyword arguments > and positional arguments in other parts of the language. Yes and no. One important trait of (pre 2.6) keyword arguments is that they have defaults. When I'm using a record format that someone else defined, it is pretty common for large portions of most records to be essentially wasted. It would be nice if I could create a record by just specifying the fields that have non-default values. I can do this with a data class (since attribute lookup falls back to the class), but not with this tuple factory. > The string form for the named tuple factory was arrived at > because it was easier to write, read, and alter than its original > form with a list of strings: > Contract = namedtuple('Contract stock strike volatility expiration rate iscall') > vs. > Contract = namedtuple('Contract', 'stock', 'strike', 'volatility', > 'expiration', 'rate', 'iscall') The type name is somehow different. Do either of these affect your decision? # separate arguments for type name and field names Contract = namedtuple('Contract', "stock strike volatility expiration rate iscall") # no explicit assignment, and bad hackery to get it namedtuple('Contract', "stock strike volatility expiration rate iscall") -jJ From pje at telecommunity.com Tue Feb 20 18:57:32 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 20 Feb 2007 12:57:32 -0500 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <45DACACD.30805@voidspace.org.uk> References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> Message-ID: <5.1.1.6.0.20070220125439.04ca2eb8@sparrow.telecommunity.com> At 10:17 AM 2/20/2007 +0000, Fuzzyman wrote: >Michele Simionato wrote: > > >Raymond Hettinger verizon.net> writes: > > > > > >>* Add a pure python named_tuple class to the collections module. I've > been > >>using the class for about a year and found that it greatly improves the > >>usability of tuples as records. > >>http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261 > >> > >> > >[snip..] > >4. I want help(MyNamedTuple) to work well; in particular it should > > display the right module name. That means > > that in the m dictionary you should add a __module__ attribute: > > > > __module__ = sys._getframe(1).f_globals['__name__'] > > > > > > >Hello all, > >If this is being considered for inclusion in the standard library, using >_getframe' hackery will guarantee that it doesn't work with alternative >implementations of Python (like IronPython at least which doesn't have >Python stack frames). Here's a way that doesn't need it: @namedtuple def Point(x,y): """The body of this function is ignored -- but this docstring will be used for the Point class""" This approach also gets rid of the messy string stuff, and it also allows one to specify default values. The created type can be given the function's __name__, __module__, and __doc__. From larry at hastings.org Tue Feb 20 18:56:28 2007 From: larry at hastings.org (Larry Hastings) Date: Tue, 20 Feb 2007 09:56:28 -0800 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> Message-ID: <45DB364C.1070907@hastings.org> Michele Simionato wrote: > ``Contract = namedtuple('Contract stock strike volatility expiration rate > iscall'.split())`` is not that bad either, but I agree that this is a > second order issue. > That also nicely makes another point: this form accepts not just a list of strings but any iterable. That sounds nice. I'd vote against the one-string approach; it seems wholly un-Pythonic to me. Python already has a parser, so don't invent your own. Speaking of un-Pythonic-ness, the whole concept of a "named tuple" strikes me as un-Pythonic. It's the only data structure I can think of where every item inside has two names. (Does TOOWTDI apply to member lookup?) I'd prefer a lightweight frozen dict, let's call it a "record" after one of the suggestions in this thread. That achieves symmetry: mutable & immutable & heavyweight lightweight +-------------------------- positional | list tuple keyword | dict record For new code, I can't think of a single place I'd want to use a "NamedTuple" where a "record" wouldn't be arguably more suitable. The "NamedTuple" makes sense only for "fixing" old APIs like os.stat. My final bit of feedback: why is it important that a NamedTuple have a class name? In the current implementation, the first identifier split from the argument string is used as the "name" of the NamedTuple, and the following arguments are names of fields. Dicts and sets are anonymous, not to mention lists and tuples; why do NamedTuples need names? My feeling is: if you want a class name, use a class. /larry/ From pje at telecommunity.com Tue Feb 20 18:59:33 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 20 Feb 2007 12:59:33 -0500 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <45DB364C.1070907@hastings.org> References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> Message-ID: <5.1.1.6.0.20070220125909.027252b8@sparrow.telecommunity.com> At 09:56 AM 2/20/2007 -0800, Larry Hastings wrote: >My final bit of feedback: why is it important that a NamedTuple have a >class name? In a word: pickling. From python at rcn.com Tue Feb 20 18:59:08 2007 From: python at rcn.com (python at rcn.com) Date: Tue, 20 Feb 2007 12:59:08 -0500 (EST) Subject: [Python-Dev] NamedTuple (was: Py2.6 ideas) Message-ID: <20070220125908.AYT33183@ms09.lnh.mail.rcn.net> [Raymond Hettinger] > The constructor signature ... Point(*fetchall(s)), > and it allows for direct construction with Point(2,3) without > the slower and weirder form: Point((2,3)). [Jim Jewett] >> If you were starting from scratch, I would agree >> whole-heartedly; this is one of my most frequent >> mistakes.The question is whether it makes sense to >> "fix" NamedTuple without also fixing regular tuple, list, Yes. Tuples and lists both have syntactic support for direct construction and NamedTuples aspire to that functionality: vec = (dx*3.0, dy*dx/dz, dz) # Regular tuple vec = Vector(dx*3.0, dy*dx/dz, dz) # Named tuple I've worked with the current version of the recipe for a long time and after a while the wisdom of the signature becomes self-evident. We REALLY don't want: vec = Vector((dx*3.0, dy*dx/dz, dz)) # yuck For conversion from other iterables, it is REALLY easy to write: vec = Vector(*partial_derivatives) Remember, list() and tuple() are typically used as casts, not as direct constructors. How often do you write: dlist = list((dx*3.0, dy*dx/dz, dz)) That is usually written: dlist = [dx*3.0, dy*dx/dz, dz] I think the Vec((dx, dy, dz)) sysntax falls into the category of foolish consistency. Raymond From tim.peters at gmail.com Tue Feb 20 19:28:17 2007 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 20 Feb 2007 13:28:17 -0500 Subject: [Python-Dev] Py_ssize_t In-Reply-To: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> Message-ID: <1f7befae0702201028k7d37535cy3b0b22e4dab595bf@mail.gmail.com> [Raymond Hettinger] > After thinking more about Py_ssize_t, I'm surprised that we're not hearing about > 64 bit users having a couple of major problems. > > If I'm understanding what was done for dictionaries, the hash table can grow > larger than the range of hash values. Accordingly, I would expect large > dictionaries to have an unacceptably large number of collisions. OTOH, we > haven't heard a single complaint, so perhaps my understanding is off. > ... As others have noted, it would require a truly gigantic dict for anyone to notice, and nobody yet has enough RAM to build something that large. I added this comment to dictobject.c for 2.5: Theoretical Python 2.5 headache: hash codes are only C "long", but sizeof(Py_ssize_t) > sizeof(long) may be possible. In that case, and if a dict is genuinely huge, then only the slots directly reachable via indexing by a C long can be the first slot in a probe sequence. The probe sequence will still eventually reach every slot in the table, but the collision rate on initial probes may be much higher than this scheme was designed for. Getting a hash code as fat as Py_ssize_t is the only real cure. But in practice, this probably won't make a lick of difference for many years (at which point everyone will have terabytes of RAM on 64-bit boxes). Ironically, IIRC we /have/ had a complaint in the other direction: someone on SF claims to have a box where sizeof(Py_ssize_t) < sizeof(long). Something else breaks as a result of that. I think I always implicitly assumed sizeof(Py_ssize_t) >= sizeof(long) would hold. In any case, hash codes are defined to be of type "long" in the C API, so there appears no painless way to boost their size on boxes where sizeof(Py_ssize_t) > sizeof(long). From guido at python.org Tue Feb 20 19:35:57 2007 From: guido at python.org (Guido van Rossum) Date: Tue, 20 Feb 2007 10:35:57 -0800 Subject: [Python-Dev] Py_ssize_t In-Reply-To: <1f7befae0702201028k7d37535cy3b0b22e4dab595bf@mail.gmail.com> References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> <1f7befae0702201028k7d37535cy3b0b22e4dab595bf@mail.gmail.com> Message-ID: On 2/20/07, Tim Peters wrote: > In any case, hash codes are defined to be of type "long" in the C API, > so there appears no painless way to boost their size on boxes where > sizeof(Py_ssize_t) > sizeof(long). But that would only be on Windows; I believe other vendors have a 64-bit long on 64-bit machines. I suppose the pain wouldn't be any greater than the pain of turning int into Py_ssize_t. Perhaps less so in Py3k since there the issue that PyInt only holds a C long is solved (by equating it to PyLong :). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From larry at hastings.org Tue Feb 20 20:16:13 2007 From: larry at hastings.org (Larry Hastings) Date: Tue, 20 Feb 2007 11:16:13 -0800 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: <45DB364C.1070907@hastings.org> References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> <45DB364C.1070907@hastings.org> Message-ID: <45DB48FD.2090801@hastings.org> Larry Hastings wrote: > I'd prefer a lightweight frozen dict, let's call it a "record" > after one of the suggestions in this thread. That achieves symmetry: > mutable & immutable & > heavyweight lightweight > +-------------------------- > positional | list tuple > keyword | dict record > I knocked together a quick prototype of a "record" to show what I had in mind. Here goes: ------ import exceptions class record(dict): __classname__ = "record" def __repr__(self): s = [self.__classname__, "("] comma = "" for name in self.__names__: s += (comma, name, "=", repr(self[name])) comma = ", " s += ")" return "".join(s) def __delitem__(self, name): raise exceptions.TypeError("object is read-only") def __setitem__(self, name, value): self.__delitem__(name) def __getattr__(self, name): if name in self: return self[name] return object.__getattr__(self, name) def __hasattr__(self, name): if name in self.__names__: return self[name] return object.__hasattr__(self, name) def __setattr__(self, name, value): # hack: allow setting __classname__ and __names__ if name in ("__classname__", "__names__"): super(record, self).__setattr__(name, value) else: # a shortcut to throw our exception self.__delitem__(name) def __init__(self, **args): names = [] for name, value in args.iteritems(): names += name super(record, self).__setitem__(name, value) self.__names__ = tuple(names) if __name__ == "__main__": r = record(a=1, b=2, c="3") print r print r.a print r.b print r.c # this throws a TypeError # r.c = 123 # r.d = 456 # the easy way to define a "subclass" of record def Point(x, y): return record(x = x, y = y) # you can use hack-y features to make your "subclasses" more swell def Point(x, y): x = record(x = x, y = y) # a hack to print the name "Point" instead of "record" x.__classname__ = "Point" # a hack to impose an ordering on the repr() display x.__names__ = ("x", "y") return x p = Point(3, 5) q = Point(2, y=5) r = Point(y=2, x=4) print p, q, r # test pickling import pickle pikl = pickle.dumps(p) pp = pickle.loads(pikl) print pp print pp == p # test that the output repr works to construct s = repr(p) print repr(s) peval = eval(s) print peval print p == peval ------ Yeah, I considered using __slots__, but that was gonna take too long. Cheers, /larry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070220/33b6b9a9/attachment.html From steven.bethard at gmail.com Wed Feb 21 00:16:31 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 20 Feb 2007 16:16:31 -0700 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: <45DB48FD.2090801@hastings.org> References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> <45DB364C.1070907@hastings.org> <45DB48FD.2090801@hastings.org> Message-ID: On 2/20/07, Larry Hastings wrote: > # the easy way to define a "subclass" of record > def Point(x, y): > return record(x = x, y = y) > > # you can use hack-y features to make your "subclasses" more swell > def Point(x, y): > x = record(x = x, y = y) > # a hack to print the name "Point" instead of "record" > x.__classname__ = "Point" > # a hack to impose an ordering on the repr() display > x.__names__ = ("x", "y") > return x > > p = Point(3, 5) > q = Point(2, y=5) > r = Point(y=2, x=4) > print p, q, r > > # test pickling > import pickle > pikl = pickle.dumps(p) > pp = pickle.loads(pikl) > print pp > print pp == p > > # test that the output repr works to construct > s = repr(p) > print repr(s) > peval = eval(s) > print peval > print p == peval > ------ > > Yeah, I considered using __slots__, but that was gonna take too long. Here's a simple implementation using __slots__: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/502237 And you don't have to hack anything to get good help() and a nice repr(). Declare a simple class for your type and you're ready to go:: >>> class Point(Record): ... __slots__ = 'x', 'y' ... >>> Point(3, 4) Point(x=3, y=4) STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From turnbull at sk.tsukuba.ac.jp Wed Feb 21 03:25:29 2007 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 21 Feb 2007 11:25:29 +0900 Subject: [Python-Dev] Py_ssize_t In-Reply-To: <003501c75509$cd21cec0$7400a8c0@RaymondLaptop1> References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> <003501c75509$cd21cec0$7400a8c0@RaymondLaptop1> Message-ID: <87hctgmeli.fsf@uwakimon.sk.tsukuba.ac.jp> Raymond Hettinger writes: > Two people had some difficulty building non-upgraded third-party modules > with Py2.5 on 64-bit machines (I think wxPython was one of the problems) In my experience wxPython is problematic, period. It's extremely tightly bound to internal details of everything around it. In particular, on every package system I've (tried to) build it, the first thing the package does is to check for its own version of Python, and pull it in if it's not there. From jcarlson at uci.edu Wed Feb 21 05:20:52 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 20 Feb 2007 20:20:52 -0800 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <45DB364C.1070907@hastings.org> References: <45DB364C.1070907@hastings.org> Message-ID: <20070220201052.ADD4.JCARLSON@uci.edu> Larry Hastings wrote: > For new code, I can't think of a single place I'd want to use a > "NamedTuple" where a "record" wouldn't be arguably more suitable. The > "NamedTuple" makes sense only for "fixing" old APIs like os.stat. I disagree. def add(v1, v2) return Vector(i+j for i,j in izip(v1,v2)) x,y,z = point And more are examples where not having a defined ordering (as would be the case with a 'static dict') would reduce its usefulness. Having a defined ordering (as is the case of lists and tuples) implies indexability (I want the ith item in this sequence!). It also allows one to use a NamedTuple without change otherwise anywhere you would have previously used a tuple (except for doctests, which would need to be changed). This was all discussed earlier. - Josiah From michele.simionato at gmail.com Wed Feb 21 05:55:35 2007 From: michele.simionato at gmail.com (Michele Simionato) Date: Wed, 21 Feb 2007 04:55:35 +0000 (UTC) Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> <45DB364C.1070907@hastings.org> <45DB48FD.2090801@hastings.org> Message-ID: Steven Bethard gmail.com> writes: > Here's a simple implementation using __slots__: > > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/502237 That's pretty cool! Two suggestions: 1. rename the _items method to __iter__, so that you have easy casting to tuple and lists; 2. put a check in the metaclass such as ``assert '__init__' not in bodydict`` to make clear to the users that they cannot override the __init__ method, that's the metaclass job. Great hack! Michele Simionato From larry at hastings.org Wed Feb 21 06:31:54 2007 From: larry at hastings.org (Larry Hastings) Date: Tue, 20 Feb 2007 21:31:54 -0800 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> <45DB364C.1070907@hastings.org> <45DB48FD.2090801@hastings.org> Message-ID: <45DBD94A.6040807@hastings.org> Steven Bethard wrote: > On 2/20/07, Larry Hastings wrote: >> I considered using __slots__, but that was gonna take too long. > Here's a simple implementation using __slots__: Thanks for steering me to it. However, your implementation and Mr. Hettinger's original NamedTuple both requires you to establish a type at the onset; with my prototype, you can create records ad-hoc as you can with dicts and tuples. I haven't found a lot of documentation on how to use __slots__, but I'm betting it wouldn't mesh well with my breezy ad-hoc records type. Cheers, /larry/ From jcarlson at uci.edu Wed Feb 21 07:04:48 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 20 Feb 2007 22:04:48 -0800 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: <45DBD94A.6040807@hastings.org> References: <45DBD94A.6040807@hastings.org> Message-ID: <20070220215842.ADDA.JCARLSON@uci.edu> Larry Hastings wrote: > > Steven Bethard wrote: > > On 2/20/07, Larry Hastings wrote: > >> I considered using __slots__, but that was gonna take too long. > > Here's a simple implementation using __slots__: > > Thanks for steering me to it. However, your implementation and Mr. > Hettinger's original NamedTuple both requires you to establish a type at > the onset; with my prototype, you can create records ad-hoc as you can > with dicts and tuples. I haven't found a lot of documentation on how to > use __slots__, but I'm betting it wouldn't mesh well with my breezy > ad-hoc records type. If it helps, you can think of Steven's and Raymond's types as variations of a C struct. They are fixed at type definition time, but that's more or less the point. Also, you can find more than you ever wanted to know about __slots__ by searching on google for 'python __slots__' (without quotes), but it would work *just fine* with your ad-hoc method, though one thing to note with your method - you can't guarantee the order of the attributes as they are being displayed. Your example would have the same issues as dict does here: >>> dict(b=1, a=2) {'a': 2, 'b': 1} Adding more attributes could further arbitrarily rearrange them. - Josiah From larry at hastings.org Wed Feb 21 07:49:15 2007 From: larry at hastings.org (Larry Hastings) Date: Tue, 20 Feb 2007 22:49:15 -0800 Subject: [Python-Dev] Py2.6 ideas In-Reply-To: <20070220201052.ADD4.JCARLSON@uci.edu> References: <45DB364C.1070907@hastings.org> <20070220201052.ADD4.JCARLSON@uci.edu> Message-ID: <45DBEB6B.8030503@hastings.org> Josiah Carlson wrote: > Larry Hastings wrote: > >> For new code, I can't think of a single place I'd want to use a >> "NamedTuple" where a "record" wouldn't be arguably more suitable. The >> "NamedTuple" makes sense only for "fixing" old APIs like os.stat. >> > I disagree. > > def add(v1, v2) > return Vector(i+j for i,j in izip(v1,v2)) > > x,y,z = point > I realize I'm in the minority here, but I prefer to use tuples only as lightweight frozen lists of homogeneous objects. I understand the concept that a tuple is a frozen container of heterogeneous objects referenced by position, it's considered a perfectly Pythonic idiom, it has a long-standing history in mathematical notation... I *get* it, I just don't *agree* with it. Better to make explicit the association of data with its name. x.dingus is simply better than remembering "the ith element of x is the 'dingus'". The examples you cite don't sway me; I'd be perfectly happy to use a positionless "record" that lacked the syntactic shortcuts you suggest. I stand by my assertion above. At least now you know "where I'm coming from", /larry/ From larry at hastings.org Wed Feb 21 08:12:36 2007 From: larry at hastings.org (Larry Hastings) Date: Tue, 20 Feb 2007 23:12:36 -0800 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: <20070220215842.ADDA.JCARLSON@uci.edu> References: <45DBD94A.6040807@hastings.org> <20070220215842.ADDA.JCARLSON@uci.edu> Message-ID: <45DBF0E4.5090907@hastings.org> Josiah Carlson wrote: > one thing to note with your method - you can't guarantee the order of the attributes as they are being displayed. > Actually, my record type *can*; see the hack using the __names__ field. It won't preserve that order during iteration--but it's only a prototype so far, and it could be fixed if there was interest. /larry/ From jcarlson at uci.edu Wed Feb 21 09:28:12 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 21 Feb 2007 00:28:12 -0800 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: <45DBF0E4.5090907@hastings.org> References: <20070220215842.ADDA.JCARLSON@uci.edu> <45DBF0E4.5090907@hastings.org> Message-ID: <20070221002351.ADE0.JCARLSON@uci.edu> Larry Hastings wrote: > > Josiah Carlson wrote: > > one thing to note with your method - you can't guarantee the order > > of the attributes as they are being displayed. > > > Actually, my record type *can*; see the hack using the __names__ field. > It won't preserve that order during iteration--but it's only a prototype > so far, and it could be fixed if there was interest. Actually, it *can't*. The ordering of the dict produced by the **kwargs arguments is exactly same as a regular dictionary. >>> def foo(**k): ... return k ... >>> foo(b=1, a=2) {'a': 2, 'b': 1} >>> foo(hello=1, world=2) {'world': 2, 'hello': 1} After construction you can preserve the order it has, but you've already lost the order provided in the call, so the order you get is arbitrarily defined by implementation details of dictionaries and hash(str). - Josiah From michele.simionato at gmail.com Wed Feb 21 09:33:00 2007 From: michele.simionato at gmail.com (Michele Simionato) Date: Wed, 21 Feb 2007 08:33:00 +0000 (UTC) Subject: [Python-Dev] Py2.6 ideas References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> Message-ID: Michele Simionato gmail.com> writes: > Finally I did some timing of code like this:: > > from itertools import imap > Point = namedtuple('Point x y'.split()) > > lst = [(i, i*i) for i in range(500)] > > def with_imap(): > for _ in imap(Point, lst): > pass > > def with_star(): > for _ in (Point(*t) for t in lst): > pass > > and as expected the performances are worse with the * notation BTW, I take back this point. It is true that the generation expression is slower, but the direct equivalent of imap is starmap and using that I don't see a significant difference in execution times. So it seems that starmap is smart enough to avoid unnecessary tuple unpacking (as you probably know ;-). I still don't like for a subclass to have an incompatible signature with the parent class, but that's just me. Michele Simionato From juancarlosuarez at ciudad.com.ar Wed Feb 21 11:59:14 2007 From: juancarlosuarez at ciudad.com.ar (Juan Carlos Suarez) Date: Wed, 21 Feb 2007 07:59:14 -0300 Subject: [Python-Dev] Welcome to the "Python-Dev" mailing list References: <006701c754e6$8f2d7240$dd70d4c9@user61af7958a3> <45DAF0D9.8010804@holdenweb.com> Message-ID: <00a301c755a7$534293c0$dd70d4c9@user61af7958a3> Hello Steve Holden, I'm already registered and I want to be a list administrator, where to go? How to do this? I need to have my own mail address to administer my mailing list, do you understand me? I've been trying to do this and I'm lost, I don't know where exactly to go, to which address to be registered for this purposes, what are the steps, I have already read averything, please help me!! I wish I would have a proper guide this time and sorry to bother you. my regards, Juan Carlos ----- Original Message ----- From: "Steve Holden" To: Cc: Sent: Tuesday, February 20, 2007 10:00 AM Subject: Re: [Python-Dev] Welcome to the "Python-Dev" mailing list > Juan Carlos: > > Please note that the python-dev list is for the developers of the Python > language, rather than for Python application developer, so your query > isn't really appropriate for this list. > > I would suggest that you read the mailman license carefully, as it will > tell you what you are allowed to do with the software. Since Mailman is > distributed under the GPL it is likely you can do all the things you > want to. If you have any further queries then there is a Mailman mailing > list to which you should refer. Please see > > http://www.gnu.org/software/mailman/ > > for further details. Good luck with your project. > > regards > Steve > > Juan Carlos Suarez wrote: >> >> *Good morning*, thanks a lot for your answer, I confirmed by this means >> my subscription. >> What *I really wish and need* is to be able to use Mailman as a freely >> distributer , as a distributer of my own mailing lists ,as a mailing list >> manager , to be able to send my mails by Mailman and to be able to open >> my own mail address. >> How should I do this. I need your instructions and help. >> I'm waiting for your answer soon >> Thanks, my regards, *Juan Carlos Suarez* >> >> >> >> >> You wrote: >> Welcome to the Python-Dev at python.org mailing list! If you are a new >> subscriber, please take the time to introduce yourself briefly in your >> first post. It is appreciated if you lurk around for a while before >> posting! :-) >> >> Additional information on Python's development process can be found in >> the Python Developer's Guide: >> >> http://www.python.org/dev/ >> >> To post to this list, send your email to: >> >> python-dev at python.org >> >> General information about the mailing list is at: >> >> http://mail.python.org/mailman/listinfo/python-dev >> >> If you ever want to unsubscribe or change your options (eg, switch to >> or from digest mode, change your password, etc.), visit your >> subscription page at: >> >> >> http://mail.python.org/mailman/options/python-dev/juancarlosuarez%40ciudad.com.ar >> >> >> You can also make such adjustments via email by sending a message to: >> >> Python-Dev-request at python.org >> >> with the word `help' in the subject or body (don't include the >> quotes), and you will get back a message with instructions. >> >> You must know your password to change your options (including changing >> the password, itself) or to unsubscribe. It is: >> >> casandra07 >> >> Normally, Mailman will remind you of your python.org mailing list >> passwords once every month, although you can disable this if you >> prefer. This reminder will also include instructions on how to >> unsubscribe or change your account options. There is also a button on >> your options page that will email your current password to you. >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > > > -- > Steve Holden +44 150 684 7255 +1 800 494 3119 > Holden Web LLC/Ltd http://www.holdenweb.com > Skype: holdenweb http://del.icio.us/steve.holden > Blog of Note: http://holdenweb.blogspot.com > See you at PyCon? http://us.pycon.org/TX2007 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/juancarlosuarez%40ciudad.com.ar > > From juancarlosuarez at ciudad.com.ar Wed Feb 21 11:59:14 2007 From: juancarlosuarez at ciudad.com.ar (Juan Carlos Suarez) Date: Wed, 21 Feb 2007 07:59:14 -0300 Subject: [Python-Dev] Welcome to the "Python-Dev" mailing list References: <006701c754e6$8f2d7240$dd70d4c9@user61af7958a3> <45DAF0D9.8010804@holdenweb.com> Message-ID: <00a301c755a7$534293c0$dd70d4c9@user61af7958a3> Hello Steve Holden, I'm already registered and I want to be a list administrator, where to go? How to do this? I need to have my own mail address to administer my mailing list, do you understand me? I've been trying to do this and I'm lost, I don't know where exactly to go, to which address to be registered for this purposes, what are the steps, I have already read averything, please help me!! I wish I would have a proper guide this time and sorry to bother you. my regards, Juan Carlos ----- Original Message ----- From: "Steve Holden" To: Cc: Sent: Tuesday, February 20, 2007 10:00 AM Subject: Re: [Python-Dev] Welcome to the "Python-Dev" mailing list > Juan Carlos: > > Please note that the python-dev list is for the developers of the Python > language, rather than for Python application developer, so your query > isn't really appropriate for this list. > > I would suggest that you read the mailman license carefully, as it will > tell you what you are allowed to do with the software. Since Mailman is > distributed under the GPL it is likely you can do all the things you > want to. If you have any further queries then there is a Mailman mailing > list to which you should refer. Please see > > http://www.gnu.org/software/mailman/ > > for further details. Good luck with your project. > > regards > Steve > > Juan Carlos Suarez wrote: >> >> *Good morning*, thanks a lot for your answer, I confirmed by this means >> my subscription. >> What *I really wish and need* is to be able to use Mailman as a freely >> distributer , as a distributer of my own mailing lists ,as a mailing list >> manager , to be able to send my mails by Mailman and to be able to open >> my own mail address. >> How should I do this. I need your instructions and help. >> I'm waiting for your answer soon >> Thanks, my regards, *Juan Carlos Suarez* >> >> >> >> >> You wrote: >> Welcome to the Python-Dev at python.org mailing list! If you are a new >> subscriber, please take the time to introduce yourself briefly in your >> first post. It is appreciated if you lurk around for a while before >> posting! :-) >> >> Additional information on Python's development process can be found in >> the Python Developer's Guide: >> >> http://www.python.org/dev/ >> >> To post to this list, send your email to: >> >> python-dev at python.org >> >> General information about the mailing list is at: >> >> http://mail.python.org/mailman/listinfo/python-dev >> >> If you ever want to unsubscribe or change your options (eg, switch to >> or from digest mode, change your password, etc.), visit your >> subscription page at: >> >> >> http://mail.python.org/mailman/options/python-dev/juancarlosuarez%40ciudad.com.ar >> >> >> You can also make such adjustments via email by sending a message to: >> >> Python-Dev-request at python.org >> >> with the word `help' in the subject or body (don't include the >> quotes), and you will get back a message with instructions. >> >> You must know your password to change your options (including changing >> the password, itself) or to unsubscribe. It is: >> >> casandra07 >> >> Normally, Mailman will remind you of your python.org mailing list >> passwords once every month, although you can disable this if you >> prefer. This reminder will also include instructions on how to >> unsubscribe or change your account options. There is also a button on >> your options page that will email your current password to you. >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > > > -- > Steve Holden +44 150 684 7255 +1 800 494 3119 > Holden Web LLC/Ltd http://www.holdenweb.com > Skype: holdenweb http://del.icio.us/steve.holden > Blog of Note: http://holdenweb.blogspot.com > See you at PyCon? http://us.pycon.org/TX2007 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/juancarlosuarez%40ciudad.com.ar > > From fredrik at pythonware.com Wed Feb 21 15:32:32 2007 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 21 Feb 2007 15:32:32 +0100 Subject: [Python-Dev] Py_ssize_t References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> Message-ID: Guido van Rossum wrote: > My suspicion is that building Python for an 64-bit address space is > still a somewhat academic exercise. arbitrary 64-bit systems, perhaps. the first Python system I ever built was deployed on an LP64 system back in 1995. it's still running, and is still being maintained. From selliott4 at austin.rr.com Wed Feb 21 16:34:44 2007 From: selliott4 at austin.rr.com (Steven Elliott) Date: Wed, 21 Feb 2007 09:34:44 -0600 Subject: [Python-Dev] Making builtins more efficient In-Reply-To: References: <1141879691.11091.78.camel@grey> <440FF9CB.5030407@gmail.com> <79990c6b0603090400h25dd2c7ev3d5c379f6529f3c2@mail.gmail.com> <1141915806.11091.127.camel@grey> <4410BE69.3080004@canterbury.ac.nz> <1171984065.22648.47.camel@grey> Message-ID: <1172072084.22648.50.camel@grey> On Tue, 2007-02-20 at 07:48 -0800, Guido van Rossum wrote: > If this is not a replay of an old message, please move the discussion > to python-ideas. It's a modified version of an old idea, so I wasn't sure where to post it since previously it was discussed here. I'll look into python-ideas. -- ----------------------------------------------------------------------- | Steven Elliott | selliott4 at austin.rr.com | ----------------------------------------------------------------------- From martin at v.loewis.de Wed Feb 21 19:31:37 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 21 Feb 2007 19:31:37 +0100 Subject: [Python-Dev] Py_ssize_t In-Reply-To: References: <004501c754d4$132573d0$6d00a8c0@RaymondLaptop1> Message-ID: <45DC9009.7060301@v.loewis.de> Fredrik Lundh schrieb: >> My suspicion is that building Python for an 64-bit address space is >> still a somewhat academic exercise. > > arbitrary 64-bit systems, perhaps. the first Python system I ever built was deployed > on an LP64 system back in 1995. it's still running, and is still being maintained. I can see that you would run a 64-bit version of Python on a system where no 32-bit mode is available, or where certain libraries are only available for 64-bit mode, or where the performance of the 32-bit mode is inadequate. However, I would expect that you "normally" cannot exercise the "large collections" feature of Python 2.5 on such an installation, because you have not enough memory (in particular if the system was built in 1995). Regards, Martin From steven.bethard at gmail.com Wed Feb 21 19:40:05 2007 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 21 Feb 2007 11:40:05 -0700 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: References: <000b01c7513b$d4f4ecd0$ea146b0a@RaymondLaptop1> <003c01c754bb$029a9090$6d00a8c0@RaymondLaptop1> <45DB364C.1070907@hastings.org> <45DB48FD.2090801@hastings.org> Message-ID: On 2/20/07, Steven Bethard wrote: > Declare a simple class for your type and you're ready to go:: > > >>> class Point(Record): > ... __slots__ = 'x', 'y' > ... > >>> Point(3, 4) > Point(x=3, y=4) Here's a brief comparison between Raymond's NamedTuple factory (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/500261) and my Record class (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/502237). Given a ``records`` module containing the recipes and the following code:: class RecordPoint(Record): __slots__ = 'x', 'y' NamedTuplePoint = NamedTuple('NamedTuplePoint x y') I got the following times from timeit:: $ python -m timeit -s "import records; Point = records.RecordPoint" "Point(1, 2)" 1000000 loops, best of 3: 0.856 usec per loop $ python -m timeit -s "import records; Point = records.NamedTuplePoint" "Point(1, 2)" 1000000 loops, best of 3: 1.59 usec per loop $ python -m timeit -s "import records; point = records.RecordPoint(1, 2)" "point.x; point.y" 10000000 loops, best of 3: 0.209 usec per loop $ python -m timeit -s "import records; point = records.NamedTuplePoint(1, 2)" "point.x; point.y" 1000000 loops, best of 3: 0.551 usec per loop $ python -m timeit -s "import records; point = records.RecordPoint(1, 2)" "x, y = point" 1000000 loops, best of 3: 1.47 usec per loop $ python -m timeit -s "import records; point = records.NamedTuplePoint(1, 2)" "x, y = point" 10000000 loops, best of 3: 0.155 usec per loop In short, for object construction and attribute access, the Record class is faster (because __init__ is a simple list of assignments and attribute access is through C-level slots). For unpacking and iteration, the NamedTuple factory is faster (because it's using the C-level tuple iteration instead of a Python-level generator). STeVe -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From brett at python.org Wed Feb 21 20:05:39 2007 From: brett at python.org (Brett Cannon) Date: Wed, 21 Feb 2007 11:05:39 -0800 Subject: [Python-Dev] Welcome to the "Python-Dev" mailing list In-Reply-To: <00a301c755a7$534293c0$dd70d4c9@user61af7958a3> References: <006701c754e6$8f2d7240$dd70d4c9@user61af7958a3> <45DAF0D9.8010804@holdenweb.com> <00a301c755a7$534293c0$dd70d4c9@user61af7958a3> Message-ID: As Steve said, this is the wrong place to be asking for help about this. We have *nothing* to do with mailing lists or Mailman. Please read Steve's email again and follow the link in it. The people on this list cannot help you with what you are asking for. -Brett On 2/21/07, Juan Carlos Suarez wrote: > Hello Steve Holden, I'm already registered and I want to be a list > administrator, where to go? How to do this? I need to have my own mail > address to administer my mailing list, do you understand me? I've been > trying to do this and I'm lost, I don't know where exactly to go, to which > address to be registered for this purposes, what are the steps, I have > already read averything, please help me!! > I wish I would have a proper guide this time and sorry to bother you. > my regards, Juan Carlos > ----- Original Message ----- > From: "Steve Holden" > To: > Cc: > Sent: Tuesday, February 20, 2007 10:00 AM > Subject: Re: [Python-Dev] Welcome to the "Python-Dev" mailing list > > > > Juan Carlos: > > > > Please note that the python-dev list is for the developers of the Python > > language, rather than for Python application developer, so your query > > isn't really appropriate for this list. > > > > I would suggest that you read the mailman license carefully, as it will > > tell you what you are allowed to do with the software. Since Mailman is > > distributed under the GPL it is likely you can do all the things you > > want to. If you have any further queries then there is a Mailman mailing > > list to which you should refer. Please see > > > > http://www.gnu.org/software/mailman/ > > > > for further details. Good luck with your project. > > > > regards > > Steve > > > > Juan Carlos Suarez wrote: > >> > >> *Good morning*, thanks a lot for your answer, I confirmed by this means > >> my subscription. > >> What *I really wish and need* is to be able to use Mailman as a freely > >> distributer , as a distributer of my own mailing lists ,as a mailing list > >> manager , to be able to send my mails by Mailman and to be able to open > >> my own mail address. > >> How should I do this. I need your instructions and help. > >> I'm waiting for your answer soon > >> Thanks, my regards, *Juan Carlos Suarez* > >> > >> > >> > >> > >> You wrote: > >> Welcome to the Python-Dev at python.org mailing list! If you are a new > >> subscriber, please take the time to introduce yourself briefly in your > >> first post. It is appreciated if you lurk around for a while before > >> posting! :-) > >> > >> Additional information on Python's development process can be found in > >> the Python Developer's Guide: > >> > >> http://www.python.org/dev/ > >> > >> To post to this list, send your email to: > >> > >> python-dev at python.org > >> > >> General information about the mailing list is at: > >> > >> http://mail.python.org/mailman/listinfo/python-dev > >> > >> If you ever want to unsubscribe or change your options (eg, switch to > >> or from digest mode, change your password, etc.), visit your > >> subscription page at: > >> > >> > >> http://mail.python.org/mailman/options/python-dev/juancarlosuarez%40ciudad.com.ar > >> > >> > >> You can also make such adjustments via email by sending a message to: > >> > >> Python-Dev-request at python.org > >> > >> with the word `help' in the subject or body (don't include the > >> quotes), and you will get back a message with instructions. > >> > >> You must know your password to change your options (including changing > >> the password, itself) or to unsubscribe. It is: > >> > >> casandra07 > >> > >> Normally, Mailman will remind you of your python.org mailing list > >> passwords once every month, although you can disable this if you > >> prefer. This reminder will also include instructions on how to > >> unsubscribe or change your account options. There is also a button on > >> your options page that will email your current password to you. > >> > >> > >> ------------------------------------------------------------------------ > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> http://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > > > > > > -- > > Steve Holden +44 150 684 7255 +1 800 494 3119 > > Holden Web LLC/Ltd http://www.holdenweb.com > > Skype: holdenweb http://del.icio.us/steve.holden > > Blog of Note: http://holdenweb.blogspot.com > > See you at PyCon? http://us.pycon.org/TX2007 > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/juancarlosuarez%40ciudad.com.ar > > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From jcarlson at uci.edu Wed Feb 21 20:35:41 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 21 Feb 2007 11:35:41 -0800 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: References: Message-ID: <20070221110629.ADFC.JCARLSON@uci.edu> "Steven Bethard" wrote: [snip] > In short, for object construction and attribute access, the Record > class is faster (because __init__ is a simple list of assignments and > attribute access is through C-level slots). For unpacking and > iteration, the NamedTuple factory is faster (because it's using the > C-level tuple iteration instead of a Python-level generator). Somewhere in the back of my mind something is telling me - if you can get a tuple with attributes based on __slots__, you just duplicate the references as both an attribute AND in the standard tuple PyObject* array, which could offer good speed in both cases, at the cost of twice as many pointers as the other two options. But alas, it isn't possible right now... >>> class foo(tuple): ... __slots__ = 'a' ... Traceback (most recent call last): File "", line 1, in TypeError: Error when calling the metaclass bases nonempty __slots__ not supported for subtype of 'tuple' Ah well. Anyone have any intuition as to what they will be doing with named tuples/record objects enough to know which case is better? Also, it doesn't seem like it would be terribly difficult to use metaclasses to get the ease of creation from the Record type for the NamedTuple variant, if desired. - Josiah From greg.ewing at canterbury.ac.nz Wed Feb 21 23:41:07 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 22 Feb 2007 11:41:07 +1300 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: <20070221110629.ADFC.JCARLSON@uci.edu> References: <20070221110629.ADFC.JCARLSON@uci.edu> Message-ID: <45DCCA83.7040706@canterbury.ac.nz> Josiah Carlson wrote: > Somewhere in the back of my mind something is telling me - if you can > get a tuple with attributes based on __slots__, you just duplicate the > references as both an attribute AND in the standard tuple PyObject* > array, It should be possible to start with a tuple subclass and add descriptors to provide named access. If the descriptor is implemented in C it should be just as efficient as using __slots__, but with no duplication of the references. Given a suitable descriptor type, it shouldn't be hard to write a factory function to assemble such classes. -- Greg From rasky at develer.com Thu Feb 22 01:26:05 2007 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 22 Feb 2007 01:26:05 +0100 Subject: [Python-Dev] Making builtins more efficient In-Reply-To: <1171984065.22648.47.camel@grey> References: <1141879691.11091.78.camel@grey> <440FF9CB.5030407@gmail.com> <79990c6b0603090400h25dd2c7ev3d5c379f6529f3c2@mail.gmail.com> <1141915806.11091.127.camel@grey> <4410BE69.3080004@canterbury.ac.nz> <1171984065.22648.47.camel@grey> Message-ID: On 20/02/2007 16.07, Steven Elliott wrote: > I'm finally getting back into this. I'd like to take one more shot at > it with a revised version of what I proposed before. > > For those of you that did not see the original thread it was about ways > that accessing builtins could be more efficient. It's a bit much to > summarize again now, but you should be able to find it in the archive > with this subject and a date of 2006-03-08. Are you aware of this patch, which is still awaiting review? https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1616125&group_id=5470 -- Giovanni Bajo From kbk at shore.net Thu Feb 22 08:01:39 2007 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu, 22 Feb 2007 02:01:39 -0500 (EST) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200702220701.l1M71d59016556@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 408 open ( -9) / 3585 closed (+20) / 3993 total (+11) Bugs : 968 open ( +8) / 6505 closed ( +7) / 7473 total (+15) RFE : 267 open ( +1) / 251 closed ( +0) / 518 total ( +1) New / Reopened Patches ______________________ Handle requests to intern string subtype instances (2007-02-13) http://python.org/sf/1658799 opened by Hrvoje Nik?i? Minor pasting patch (2007-02-13) http://python.org/sf/1659326 opened by Tal Einat Minor AST tweaks (2007-02-13) http://python.org/sf/1659410 opened by Collin Winter functools.compose to chain functions together (2007-02-14) CLOSED http://python.org/sf/1660179 opened by Chris AtLee IDLE doesn't save files on closing (2007-02-15) CLOSED http://python.org/sf/1660202 opened by Toni Hide iteration variable in list comprehensions (2007-02-15) http://python.org/sf/1660500 opened by Nick Coghlan ftplib passive ftp problem on multihomed clients (2007-02-16) http://python.org/sf/1661754 opened by Tim Baum setuptools: avoid sets module for python>2.3 (2007-02-19) http://python.org/sf/1663226 opened by Amit Aronovitch enable "python -m doctest FILE" to run tests in FILE (2007-02-19) http://python.org/sf/1663234 opened by Stefan Behnel Fix for urllib.ftpwrapper.retrfile() and none existing files (2007-02-20) http://python.org/sf/1664522 opened by Phil Knirsch Patches Closed ______________ imputil must respect __path__ for package submodules (2003-02-12) http://python.org/sf/685268 closed by loewis Scalable zipfile extension (2003-09-27) http://python.org/sf/813436 closed by loewis zipfile and big zipped file (2004-07-17) http://python.org/sf/992750 closed by loewis ZipFile - support for file decryption (2003-03-06) http://python.org/sf/698833 closed by loewis Bug 1514451: zipfile "append" mode should create a file ... (2006-07-06) http://python.org/sf/1517891 closed by loewis socketmodule fails to build because missing NETLINK_DNRTMSG (2007-02-11) http://python.org/sf/1657276 closed by loewis gzip.GzipFile has no name attribute (2007-01-29) http://python.org/sf/1647484 closed by gustaebel Missing HCI sockets in bluetooth code from socketmodule (2006-02-15) http://python.org/sf/1432399 closed by loewis Patch to support lots of file descriptors (2006-02-19) http://python.org/sf/1434657 closed by loewis isapi.samples.advanced.py fix (2005-02-17) http://python.org/sf/1126187 closed by loewis Add IEEE Float support to wave.py (2005-02-19) http://python.org/sf/1144504 closed by loewis Make urllib2.OpenerDirector instances pickle-able (2005-02-20) http://python.org/sf/1144636 closed by jjlee functools.compose to chain functions together (2007-02-15) http://python.org/sf/1660179 closed by loewis IDLE doesn't save files on closing (2007-02-15) http://python.org/sf/1660202 closed by gbrandl decorator module (2006-03-12) http://python.org/sf/1448297 closed by gbrandl dictnotes.txt (2006-01-05) http://python.org/sf/1397848 closed by loewis Documentation for new Struct object (2006-05-24) http://python.org/sf/1494140 closed by gbrandl Detect incomplete readline implementation (2006-02-21) http://python.org/sf/1435651 closed by loewis add os.chflags() and os.lchflags() where available (2006-05-17) http://python.org/sf/1490190 closed by loewis Fix for bug #1486663 mutable types check kwargs in tp_new (2006-05-20) http://python.org/sf/1491939 closed by zseil New / Reopened Bugs ___________________ I think, I have found this bug on time.mktime() (2007-02-10) CLOSED http://python.org/sf/1656559 reopened by sergiomb This shouldn't be there: Note that this code that uses... (2007-02-13) CLOSED http://python.org/sf/1658794 opened by Alf Lerv?g os.wait child process fail when under stress (2007-02-13) http://python.org/sf/1658959 opened by The Groff Calling tparm from extension lib fails in Python 2.4 (2007-02-13) http://python.org/sf/1659171 opened by Richard B. Kreckel Calling tparm from extension lib fails in Python 2.4 (2007-02-13) CLOSED http://python.org/sf/1659173 opened by Richard B. Kreckel Python extension problems after re-install (2007-02-14) http://python.org/sf/1659705 opened by elf continuing problem with httplib multiple set-cookie headers (2007-02-14) http://python.org/sf/1660009 opened by David Margrave base64.urlsafe_b64encode() shouldn't use the = character (2007-02-16) http://python.org/sf/1661108 opened by Ryan Barrett Misleading behavior for [] and {} default arguments (2007-02-16) CLOSED http://python.org/sf/1661603 opened by Matthijs finditer stuck in infinite loop (2007-02-16) http://python.org/sf/1661745 opened by Milan [2.5 regression?] failure to import the ORBit extension (2007-02-17) http://python.org/sf/1662529 opened by Matthias Klose the re module can perform poorly: O(2**n) versus O(n**2) (2007-02-17) http://python.org/sf/1662581 opened by Gregory P. Smith Over-zealous keyword-arguments check for built-in set class (2006-05-11) CLOSED http://python.org/sf/1486663 reopened by rhettinger subprocess/popen close_fds perform poor if SC_OPEN_MAX is hi (2007-02-19) http://python.org/sf/1663329 opened by H. von Bargen PyType_IsSubtype segfaults (2007-02-19) http://python.org/sf/1663392 opened by navtej singh crash in exec statement if uncode filename cannot be decoded (2007-02-21) http://python.org/sf/1664966 opened by Stefan Schukat Hangup when using cgitb in a thread while still in import (2007-02-21) http://python.org/sf/1665206 opened by hoffie Documentation missing for OptionGroup class in optparse (2007-02-21) http://python.org/sf/1665333 opened by LunarYorn Bugs Closed ___________ I think, I have found this bug on time.mktime() (2007-02-10) http://python.org/sf/1656559 closed by loewis I think, I have found this bug on time.mktime() (2007-02-10) http://python.org/sf/1656559 closed by loewis zipfile "append" mode should create a file if not present (2006-06-29) http://python.org/sf/1514451 closed by loewis This shouldn't be there: Note that this code that uses... (2007-02-13) http://python.org/sf/1658794 closed by loewis Calling tparm from extension lib fails in Python 2.4 (2007-02-13) http://python.org/sf/1659173 closed by gbrandl Misleading behavior for [] and {} default arguments (2007-02-16) http://python.org/sf/1661603 closed by gbrandl logging formatter %(lineno)d does not work (2007-02-05) http://python.org/sf/1652788 closed by vsajip Over-zealous keyword-arguments check for built-in set class (2006-05-11) http://python.org/sf/1486663 closed by rhettinger Double free/corruption? (2007-02-06) http://python.org/sf/1653121 closed by sf-robot New / Reopened RFE __________________ Datetime enhancements (2007-02-21) http://python.org/sf/1665292 opened by Christian Heimes From anugupta at pu.ac.in Thu Feb 22 11:01:33 2007 From: anugupta at pu.ac.in (Anu Gupta DCSA) Date: Thu, 22 Feb 2007 15:31:33 +0530 Subject: [Python-Dev] A Reminder : A Survey on Defect Management Practices in Free/Open Source Software Message-ID: <20070222100111.M46883@pu.ac.in> Sir/Madam I seek help from designers, developers, testers,defect fixers,project managers or playing any other key role in Free/Open Source software development or maintenence in carrying out a study to identify practices and problems of defect management in various Free/Open Source Software projects. The insights gained from the study can further help us to extract publicly accessible defect data and determine impact of defect management practices on software quality. Please spend a few minutes of your precious time to fill up the Questionnaire. The most of the questions follow multiple choice formats and are quite easy to answer. To have the Online Questionnaire, please visit: http://anu.puchd.ac.in/phpESP/public/survey.php?name=FOSS_Defect_Survey (You can also copy and paste this link into your browser, and hit the 'Return' key.) I hope you will find all the questions interesting and thought-provoking. Your answers will be kept anonymous.The data thus collected will only be used for research purpose.It would be nice if you may further refer this mail to others actively engaged with Free/Open Source Software development. If you have any query or suggestions then feel free to contact. Thank You With regards, Anu Gupta Senior Lecturer Department of Computer Science and Applications, Panjab University, Chandigarh. INDIA In case of any problem in accessing/using the above mentioned link please contact: E-mail: anugupta at pu.ac.in anugupta98 at gmail.com From adurdin at gmail.com Thu Feb 22 21:29:58 2007 From: adurdin at gmail.com (Andrew Durdin) Date: Thu, 22 Feb 2007 20:29:58 +0000 Subject: [Python-Dev] Bundling decimal.py with Django Message-ID: <59e9fd3a0702221229m4b66b6c7m372b1299a568feb4@mail.gmail.com> Apologies if this is a little off-topic for python-dev, but it seemed like the best place to ask and get the attention of those needed. I am championing a patch to improve Django's support for numeric types by using Decimals with numeric columns and floats with double precision columns, rather than the current rather unpredictable behaviour (which depends on the particular python version and database backend being used). It has been proposed[1] to include decimal.py from the Python 2.5 distribution with Django, to ensure consistent and compatible behaviour between Python 2.3 and Python >= 2.4[2]. Jacob Kaplan-Moss wrote in response to this proposal """ In order for this to be accepted, we need to know: * What is the license for the decimal module, and is said license compatible with the BSD license that Django uses? * Are the authors of the decimal module OK with us distributing it with Django? Our policy for including third party libraries requires an answer to both those questions. We won't include anything if it's not legal to do so, and we won't include anything that the original author doesn't want included. """ I'm addressing the second of Jacob's points here; would it be acceptable if the decimal module was distributed with Django? Andrew [1] http://groups.google.com/group/django-developers/msg/230892a9f9a71472 [2] Python's _threading_local.py module is already included with Django for the same compatibility reasons. From python at rcn.com Thu Feb 22 22:27:20 2007 From: python at rcn.com (python at rcn.com) Date: Thu, 22 Feb 2007 16:27:20 -0500 (EST) Subject: [Python-Dev] Bundling decimal.py with Django Message-ID: <20070222162720.AZB21106@ms09.lnh.mail.rcn.net> > What is the license for the decimal module, > and is said license compatible > with the BSD license that Django uses? It is the the same license as Python itself: http://www.python.org/psf/license/ > Are the authors of the decimal module OK > with usdistributing it with Django? You don't need my okay, but you've got it anyway. Per the source file, the listed authors are: # Written by Eric Price # and Facundo Batista # and Raymond Hettinger # and Aahz # and Tim Peters From ndbecker2 at gmail.com Fri Feb 23 01:31:41 2007 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 22 Feb 2007 19:31:41 -0500 Subject: [Python-Dev] bool conversion wart? Message-ID: >>> int ('4') 4 >>> bool ('False') True From jml at mumak.net Fri Feb 23 02:08:29 2007 From: jml at mumak.net (Jonathan Lange) Date: Fri, 23 Feb 2007 12:08:29 +1100 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: Message-ID: On 2/23/07, Neal Becker wrote: > >>> bool ('False') > True > Non-empty strings are considered True, empty strings are considered False. This is not a wart, as the behaviour matches that of other sequences. cheers, jml From ndbecker2 at gmail.com Fri Feb 23 02:13:06 2007 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 22 Feb 2007 20:13:06 -0500 Subject: [Python-Dev] bool conversion wart? References: Message-ID: Jonathan Lange wrote: > On 2/23/07, Neal Becker wrote: >> >>> bool ('False') >> True >> > > Non-empty strings are considered True, empty strings are considered > False. This is not a wart, as the behaviour matches that of other > sequences. > Well consider this: >>>str (4) '4' >>>int(str (4)) 4 >>>str (False) 'False' >>>bool(str(False)) True Doesn't this seem a bit inconsisent? From mike.klaas at gmail.com Fri Feb 23 02:39:48 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Thu, 22 Feb 2007 17:39:48 -0800 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: Message-ID: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> On 2/22/07, Neal Becker wrote: > Well consider this: > >>>str (4) > '4' > >>>int(str (4)) > 4 > >>>str (False) > 'False' > > >>>bool(str(False)) > True > > Doesn't this seem a bit inconsisent? Virtually no python objects accept a stringified version of themselves in their constructor: >>> str({}) '{}' >>> dict('{}') Traceback (most recent call last): File "", line 1, in ValueError: dictionary update sequence element #0 has length 1; 2 is required >>> str([]) '[]' >>> list('[]') ['[', ']'] Python is not Perl. -Mike From ndbecker2 at gmail.com Fri Feb 23 02:44:00 2007 From: ndbecker2 at gmail.com (Neal Becker) Date: Thu, 22 Feb 2007 20:44:00 -0500 Subject: [Python-Dev] bool conversion wart? References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> Message-ID: Mike Klaas wrote: > On 2/22/07, Neal Becker wrote: > >> Well consider this: >> >>>str (4) >> '4' >> >>>int(str (4)) >> 4 >> >>>str (False) >> 'False' >> >> >>>bool(str(False)) >> True >> >> Doesn't this seem a bit inconsisent? > > Virtually no python objects accept a stringified version of themselves > in their constructor: > >>>> str({}) > '{}' >>>> dict('{}') > Traceback (most recent call last): > File "", line 1, in > ValueError: dictionary update sequence element #0 has length 1; 2 is > required >>>> str([]) > '[]' >>>> list('[]') > ['[', ']'] > > Python is not Perl. > Except, all the numeric types do, including int, float, and complex. But not bool. In fact, this is not just academic. The fact that other numeric types act this way leaves a reasonable expectation that bool will. Instead, bool fails in _the worst possible way_: it silently gives a _wrong result_. From mike.klaas at gmail.com Fri Feb 23 02:56:47 2007 From: mike.klaas at gmail.com (Mike Klaas) Date: Thu, 22 Feb 2007 17:56:47 -0800 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> Message-ID: <3d2ce8cb0702221756x34d932dfn3e41c73fe1a2a832@mail.gmail.com> On 2/22/07, Neal Becker wrote: > Except, all the numeric types do, including int, float, and complex. But > not bool. Oh? In [5]: str(complex(1, 2)) Out[5]: '(1+2j)' In [6]: complex(str(complex(1, 2))) --------------------------------------------------------------------------- : complex() arg is a malformed string > In fact, this is not just academic. The fact that other numeric > types act this way leaves a reasonable expectation that bool will. > Instead, bool fails in _the worst possible way_: it silently gives a _wrong > result_. I'd debate the assertion that 'bool' is a numeric type (despite being a subclass of int). For bool() to return anything other than the value of the python expression evaluated in boolean context would be _lunacy_ and there is absolutely no chance it that will be changed. -Mike From ben at 666.com Fri Feb 23 03:18:33 2007 From: ben at 666.com (Ben Wing) Date: Thu, 22 Feb 2007 20:18:33 -0600 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> Message-ID: On 2/22/07, Neal Becker wrote: > > Mike Klaas wrote: > > > On 2/22/07, Neal Becker wrote: > > > >> Well consider this: > >> >>>str (4) > >> '4' > >> >>>int(str (4)) > >> 4 > >> >>>str (False) > >> 'False' > >> > >> >>>bool(str(False)) > >> True > >> > >> Doesn't this seem a bit inconsisent? > > > > Virtually no python objects accept a stringified version of themselves > > in their constructor: > > > >>>> str({}) > > '{}' > >>>> dict('{}') > > Traceback (most recent call last): > > File "", line 1, in > > ValueError: dictionary update sequence element #0 has length 1; 2 is > > required > >>>> str([]) > > '[]' > >>>> list('[]') > > ['[', ']'] > > > > Python is not Perl. > > > Except, all the numeric types do, including int, float, and complex. But > not bool. In fact, this is not just academic. The fact that other > numeric > types act this way leaves a reasonable expectation that bool will. > Instead, bool fails in _the worst possible way_: it silently gives a > _wrong > result_. i agree with mike; it would just be asking for trouble. (have you ever been bitten by the Perl behavior where the string '0' is considered false? it's a nasty, nasty problem to debug.) neal, you may be confusing the concepts of "convert data from one type to another" and "read the printed representation of data". ben -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070222/025eff8f/attachment.htm From larry at hastings.org Fri Feb 23 03:19:38 2007 From: larry at hastings.org (Larry Hastings) Date: Thu, 22 Feb 2007 18:19:38 -0800 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> Message-ID: <45DE4F3A.3000502@hastings.org> Neal Becker wrote: > Instead, bool fails in _the worst possible way_: it silently gives a > _wrong result_. I disagree with the word "fail" there; Python is working correctly. The behavior of converting expressions to a boolean is well-defined: http://docs.python.org/ref/Booleans.html Perhaps the behavior is unexpected or unwelcome--but it is *correct*. It's just one of those things about Python that one has to get used to; this is a side-effect of another, much more important principle. If you really want to turn the string "True" into True, use "eval". eval("4") -> 4 eval("True") -> True eval("{}") -> {} Perhaps it would be safest to cast the output of eval to bool: bool(eval("True")) -> True bool(eval("")) -> False Cheers, /larry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070222/15e6eb76/attachment-0001.html From turnbull at sk.tsukuba.ac.jp Fri Feb 23 03:31:15 2007 From: turnbull at sk.tsukuba.ac.jp (Stephen J. Turnbull) Date: Fri, 23 Feb 2007 11:31:15 +0900 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: Message-ID: <87fy8xli4s.fsf@uwakimon.sk.tsukuba.ac.jp> Neal Becker writes: > Well consider this: > >>>str (4) > '4' > >>>int(str (4)) > 4 > >>>str (False) > 'False' > > >>>bool(str(False)) > True > > Doesn't this seem a bit inconsisent? The former case is a *conversion* from an expression that *does not* have an interpretation in a numerical context to an integer. The latter case is a *canonicalization* from an expression that *does* have an interpretation in a boolean context to the equivalent boolean constant. I don't have a problem with that. YMMV. From rrr at ronadam.com Fri Feb 23 04:09:05 2007 From: rrr at ronadam.com (Ron Adam) Date: Thu, 22 Feb 2007 21:09:05 -0600 Subject: [Python-Dev] bool conversion wart? In-Reply-To: <45DE4F3A.3000502@hastings.org> References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DE4F3A.3000502@hastings.org> Message-ID: <45DE5AD1.6090009@ronadam.com> Larry Hastings wrote: > Neal Becker wrote: >> Instead, bool fails in _the worst possible way_: it silently gives a >> _wrong result_. > > I disagree with the word "fail" there; Python is working correctly. The > behavior of converting expressions to a boolean is well-defined: > http://docs.python.org/ref/Booleans.html > Perhaps the behavior is unexpected or unwelcome--but it is *correct*. > It's just one of those things about Python that one has to get used to; > this is a side-effect of another, much more important principle. > > If you really want to turn the string "True" into True, use "eval". > eval("4") -> 4 > eval("True") -> True > eval("{}") -> {} > Perhaps it would be safest to cast the output of eval to bool: > bool(eval("True")) -> True > bool(eval("")) -> False > > > Cheers, > > /larry/ In most cases like this you would not use string "True" or "False" but instead values True or False. In cases where you do have such strings You would probably be doing string comparisons anyway because there will most likely be other values to consider. if S == "True": .... Or use a dictionary to get predetermined values of expected strings. if svalues_dict["True"]: ... To Neal: Instead of being consistent with numeric types, bool is consistent in a different way. Think of it as len() having precedence over value if that helps. >>> bool('') False >>> bool([]) False >>> bool(()) False >>> bool({}) False >>> bool('anything') True >>> bool(['anything']) True >>> bool(('anything',)) True >>> bool({'any':'thing'}) True This is really useful because you can do simple if tests to see if containers have anything in them. if somestring: ... if sometuple: ... if somelist: ... if somedict: ... If they do have contents of some some type, you then need to test the contents to see what is in them. Which might be a True or False values of some type. I suggest you take this to comp.lang.python to get further help with using bool. It is a good idea to test thoughts like this out there first. Cheers, Ron From guido at python.org Fri Feb 23 04:33:54 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Feb 2007 21:33:54 -0600 Subject: [Python-Dev] [Python-checkins] r53860 - peps/trunk/pep-0000.txt peps/trunk/pep-0358.txt In-Reply-To: References: <20070222235747.3546F1E4003@bag.python.org> Message-ID: [-python-checkins, +python-dev] On 2/22/07, Jim Jewett wrote: > > __setitem__ > > __setslice__ > > append > > count > > + decode > > + endswith > > extend > > + find > > index > > insert > > + join > > + partition > > remove > > + replace > > + rindex > > + rpartition > > + split > > + startswith > > + rfind > > + rindex > > + rsplit > > + translate > > What sort of arguments do they take? You should be able to infer this from what the corresponding str or list methods do -- always substituting bytes for those, and int for the single element. > Other bytes objects? (Then the literal is more important.) > > Unicode? With an extra decoding argument? The only way to use unicode is to use bytes(, ). The others methods barf on Unicode. > Sequences of integers? Is startswith(500) False or a ValueException? TypeError. > Single integers? startswith(ord('A')) TypeError (this is the same as the previous.) > + Note the conspicuous absence of .isupper(), .upper(), and friends. > > This does force the use of more regular expressions, by ruling out > > data.upper().startswith("ERROR:") Yeah, over dinner we started leaning towards perhaps supporting at least lower() and upper() (but acting only on the 2x26 ASCII letters). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Feb 23 04:37:59 2007 From: guido at python.org (Guido van Rossum) Date: Thu, 22 Feb 2007 21:37:59 -0600 Subject: [Python-Dev] bool conversion wart? In-Reply-To: <45DE5AD1.6090009@ronadam.com> References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DE4F3A.3000502@hastings.org> <45DE5AD1.6090009@ronadam.com> Message-ID: This won't change so just get used to it. Please move any further discussion to c.l.py. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Fri Feb 23 11:52:51 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 23 Feb 2007 23:52:51 +1300 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> Message-ID: <45DEC783.8040601@canterbury.ac.nz> Neal Becker wrote: > The fact that other numeric > types act this way leaves a reasonable expectation that bool will. But bool isn't really a numeric type in the same way that the others are. It's only a subtype of int for historical reasons. If it had been a part of Python from the beginning, it probably would have been a separate type altogether. Hmmm... is that something that should be revisited in 3.0? -- Greg From meine at informatik.uni-hamburg.de Fri Feb 23 15:36:50 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Fri, 23 Feb 2007 15:36:50 +0100 Subject: [Python-Dev] [PATCH] Handling of scripts / substitution of python executable path Message-ID: <200702231536.51556.meine@informatik.uni-hamburg.de> Hi! For a long time, I have been annoyed by distutils behavior concerning "scripts": I always put #!/usr/bin/env python into the first line in order to let the incredibly useful "env" program start the right python version. I know that it is quite evil to hardcode /usr/bin/python or /usr/local/bin/python; I have seen dozens of #! hacks for finding e.g. perl, and I was delighted to find /usr/bin/env, which solves the problem once and for all. (And yes - it is always in /usr/bin/! ;-) ) Now distutils tries to be intelligent and "destroys" my nice #! lines, which can be quite evil in complex setups, e.g. when you share your home directory via NFS (or rsync/unison) between several environments with different python installations. Furthermore, we are using the "module" system here at our university, so that I can dynamically choose between half a dozen python versions ("module" manages your PATH variables). Replacing the python path turns nice, pure python scrips into platform-specific programs as you can see here: meine at kogspc12:~ head -n1 ~/vigra/interactive/build/scripts-2.4/pyterm #!/software/python-2.4.4/SuSE-9.0/bin/python Note the SuSE-9.0 exec-prefix in the path; we are using several Linux and Solaris versions here: meine at kogspc12:~/tmp/vi3build -> ls -1 /software/python-2.4.4/*/bin/python /software/python-2.4.4/SuSE-10.0/bin/python* /software/python-2.4.4/SuSE-9.0/bin/python* /software/python-2.4.4/SunOS-5.8/bin/python* I see that distutils as it is now does the right thing * on Windows systems, * on any system where /usr/bin/env is missing, or * when the source file has a #! with a broken path. What I propose is a minimal invasive change which keeps /usr/bin/env iff it is in the #! line *and* exists on the current system. (A more brave change would be to always use /usr/bin/env if it exists, but I think that is a topic open for discussion.) Attached you'll find a patch which implements this; I did not yet update tests/test_build_scripts.py however. Comments? (I first posted this to distutils-sig but was told that distutils is a bit neglected there, so I decided to try to push these simple patches in via python-dev.) Ciao, / / /--/ / / ANS -------------- next part -------------- A non-text attachment was scrubbed... Name: distutils_env.diff Type: text/x-diff Size: 2348 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070223/8410bcaa/attachment.bin From meine at informatik.uni-hamburg.de Fri Feb 23 15:37:24 2007 From: meine at informatik.uni-hamburg.de (Hans Meine) Date: Fri, 23 Feb 2007 15:37:24 +0100 Subject: [Python-Dev] Old bug still unfixed? [ python-Patches-1183712 ] package_data chops off first char of default package Message-ID: <200702231537.25070.meine@informatik.uni-hamburg.de> Hi! (I first posted this to distutils-sig but was told that distutils is a bit neglected there, so I decided to try to push these simple patches in via python-dev.) Lately, a student came to me complaining about an old distutils bug which bit him: http://mail.python.org/pipermail/patches/2006-August/020642.html The patch is quite trivial and cannot possibly break something AFAICS, so could someone please apply it? (Hmm, now that I write this; we are still using python 2.4.4 here, so maybe it has been fixed in between?) Ciao, / / /--/ / / ANS -------------- next part -------------- A non-text attachment was scrubbed... Name: distutils-rootpackage.diff Type: text/x-diff Size: 501 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20070223/ab6fb366/attachment.bin From jimjjewett at gmail.com Fri Feb 23 16:47:02 2007 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 23 Feb 2007 10:47:02 -0500 Subject: [Python-Dev] [Python-checkins] r53860 - peps/trunk/pep-0000.txt peps/trunk/pep-0358.txt In-Reply-To: References: <20070222235747.3546F1E4003@bag.python.org> Message-ID: On 2/22/07, Guido van Rossum wrote: > [-python-checkins, +python-dev] > > On 2/22/07, Jim Jewett wrote: > > > __setitem__ > > > __setslice__ > > > append > > > count > > > + decode > > > + endswith > > > extend > > > + find > > > index > > > insert > > > + join > > > + partition > > > remove > > > + replace > > > + rindex > > > + rpartition > > > + split > > > + startswith > > > + rfind > > > + rindex > > > + rsplit > > > + translate > > What sort of arguments do they take? > You should be able to infer this from what the corresponding str or > list methods do -- always substituting bytes for those, and int for > the single element. ... > > Single integers? startswith(ord('A')) > > TypeError (this is the same as the previous.) >>> "asdf".index("df") == "asdf".index("d") Assuming : >>> data = bytes("asdf", 'ASCII') Are you saying that, even for the single-char call, I must write: >>> data.index(bytes("d", 'ASCII')) instead of: >>> data.index("d") or even: >>> data.index(ord("d")) -jJ From exarkun at divmod.com Fri Feb 23 16:55:56 2007 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Fri, 23 Feb 2007 10:55:56 -0500 Subject: [Python-Dev] [PATCH] Handling of scripts / substitution of python executable path In-Reply-To: <200702231536.51556.meine@informatik.uni-hamburg.de> Message-ID: <20070223155556.25807.1573444003.divmod.quotient.30452@ohm> On Fri, 23 Feb 2007 15:36:50 +0100, Hans Meine wrote: >Hi! > > [snip - distutils should leave #!/usr/bin/env python alone] > >Comments? (I first posted this to distutils-sig but was told that distutils >is a bit neglected there, so I decided to try to push these simple patches in >via python-dev.) How about a distutils installation command line option to tell it to use this behavior? People with unusual environments can select it. Jean-Paul From guido at python.org Fri Feb 23 16:57:01 2007 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Feb 2007 09:57:01 -0600 Subject: [Python-Dev] [Python-checkins] r53860 - peps/trunk/pep-0000.txt peps/trunk/pep-0358.txt In-Reply-To: References: <20070222235747.3546F1E4003@bag.python.org> Message-ID: The latest version of the PEP clarifies that both are allowed. On 2/23/07, Jim Jewett wrote: > On 2/22/07, Guido van Rossum wrote: > > [-python-checkins, +python-dev] > > > > On 2/22/07, Jim Jewett wrote: > > > > __setitem__ > > > > __setslice__ > > > > append > > > > count > > > > + decode > > > > + endswith > > > > extend > > > > + find > > > > index > > > > insert > > > > + join > > > > + partition > > > > remove > > > > + replace > > > > + rindex > > > > + rpartition > > > > + split > > > > + startswith > > > > + rfind > > > > + rindex > > > > + rsplit > > > > + translate > > > > What sort of arguments do they take? > > > You should be able to infer this from what the corresponding str or > > list methods do -- always substituting bytes for those, and int for > > the single element. > ... > > > Single integers? startswith(ord('A')) > > > > TypeError (this is the same as the previous.) > > >>> "asdf".index("df") == "asdf".index("d") > > Assuming : > >>> data = bytes("asdf", 'ASCII') > > Are you saying that, even for the single-char call, I must write: > >>> data.index(bytes("d", 'ASCII')) > > instead of: > >>> data.index("d") > > or even: > >>> data.index(ord("d")) > > -jJ > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Fri Feb 23 20:26:52 2007 From: barry at python.org (Barry Warsaw) Date: Fri, 23 Feb 2007 13:26:52 -0600 Subject: [Python-Dev] [PATCH] Handling of scripts / substitution of python executable path In-Reply-To: <20070223155556.25807.1573444003.divmod.quotient.30452@ohm> References: <20070223155556.25807.1573444003.divmod.quotient.30452@ohm> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 23, 2007, at 9:55 AM, Jean-Paul Calderone wrote: > On Fri, 23 Feb 2007 15:36:50 +0100, Hans Meine > wrote: >> Hi! >> >> [snip - distutils should leave #!/usr/bin/env python alone] >> >> Comments? (I first posted this to distutils-sig but was told that >> distutils >> is a bit neglected there, so I decided to try to push these simple >> patches in >> via python-dev.) > > How about a distutils installation command line option to tell it > to use this > behavior? People with unusual environments can select it. This would be best I think. There's a related problem here. Many operating systems (e.g. various Linux distros) provide system scripts written in Python and they incorrectly use /usr/bin/env. The problem is that if you have your own version of Python earlier in your $PATH, you can break these scripts. This happens when say your /usr/local/ bin/python doesn't have the packages that your /usr/bin/python expects. I've hit this quite a few times on Gentoo for example. In these cases, you really do want the #! line to point to the OS's version of Python rather than whatever random installation is on your $PATH. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBRd8//XEjvBPtnXfVAQLZKQP+NvcjMIT4tCZtOIPWo8Kj1JMp5iX0Nqib BBXPgDtncJ6VvITclbYLAmitPhwkDddHXgNIHoj2lY9HJildfOkDQxVpDRTvrEvd hlIW4z7S6SmlrvxJYSB2jGSpkNkVRo8UpZKsmU0BWFBE9t6Tnw+ofItuKIF1TlWS unSo4LceXFo= =wiLG -----END PGP SIGNATURE----- From martin at v.loewis.de Fri Feb 23 22:22:55 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 23 Feb 2007 22:22:55 +0100 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: Message-ID: <45DF5B2F.8090805@v.loewis.de> Neal Becker schrieb: > Doesn't this seem a bit inconsisent? Please don't ask such questions. Instead, formulate them as "I would have expected the result to be X instead. Why is the result Y?" (if that is actually the question you meant to ask) IOW, what does it help you if somebody answers "yes" to your question? Regards, Martin From martin at v.loewis.de Fri Feb 23 22:27:12 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 23 Feb 2007 22:27:12 +0100 Subject: [Python-Dev] bool conversion wart? In-Reply-To: <45DEC783.8040601@canterbury.ac.nz> References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DEC783.8040601@canterbury.ac.nz> Message-ID: <45DF5C30.6060103@v.loewis.de> Greg Ewing schrieb: > But bool isn't really a numeric type in the same way > that the others are. It's only a subtype of int for > historical reasons. If it had been a part of Python > from the beginning, it probably would have been a > separate type altogether. > > Hmmm... is that something that should be revisited > in 3.0? I specifically asked the question when unifying ints and longs for 3.0 (as int went away, the question was what True and False ought to be). Guido pronounced that it is deliberate that they are "integer-like", and should continue to inherit from the integer type. One idiom that people use a lot is foo[b], where b is a boolean. Regards, Martin From martin at v.loewis.de Fri Feb 23 22:33:01 2007 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 23 Feb 2007 22:33:01 +0100 Subject: [Python-Dev] [PATCH] Handling of scripts / substitution of python executable path In-Reply-To: <200702231536.51556.meine@informatik.uni-hamburg.de> References: <200702231536.51556.meine@informatik.uni-hamburg.de> Message-ID: <45DF5D8D.40400@v.loewis.de> Hans Meine schrieb: > For a long time, I have been annoyed by distutils behavior concerning > "scripts": I always put > #!/usr/bin/env python > into the first line in order to let the incredibly useful "env" program start > the right python version. > > I know that it is quite evil to hardcode /usr/bin/python No. The current distutils behaviour is very deliberate, intentional, and has undergone a number of iterations to arrive at this point. While it is true that you would normally use /usr/bin/env for scripts that you *distribute*, it's not true that you should use that for scripts that you *install*. Instead, the script that you install will likely depend on the specific installation of Python: the script will likely use modules that are only installed in a single location. So the installed scripts need to hard-code the path to the interpreter, or else they break if somebody changes the path and suddenly picks up a different Python. Notice that it may not just be that the Python is a different version (on which the script won't work); it may also be that the it won't work on a different installation of the *same* Python version, since it requires libraries not available in this other installation. So the default must stay as it is. Regards, Martin From martin at v.loewis.de Fri Feb 23 22:35:56 2007 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 23 Feb 2007 22:35:56 +0100 Subject: [Python-Dev] Old bug still unfixed? [ python-Patches-1183712 ] package_data chops off first char of default package In-Reply-To: <200702231537.25070.meine@informatik.uni-hamburg.de> References: <200702231537.25070.meine@informatik.uni-hamburg.de> Message-ID: <45DF5E3C.7040204@v.loewis.de> Hans Meine schrieb: > (I first posted this to distutils-sig but was told that distutils is a bit > neglected there, so I decided to try to push these simple patches in via > python-dev.) Lately, a student came to me complaining about an old distutils > bug which > bit him: > http://mail.python.org/pipermail/patches/2006-August/020642.html > > The patch is quite trivial and cannot possibly break something AFAICS, so > could someone please apply it? (Hmm, now that I write this; we are still > using python 2.4.4 here, so maybe it has been fixed in between?) See https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1183712&group_id=5470 The last comment is from jimjjewet, and he claims that the patch is still incomplete. There is no response to that claim so far, by either the patch author nor anybody else. I still believe that the patch is unnecessary, i.e. it fixes a problem which shouldn't arise in the first place. Regards, Martin From pje at telecommunity.com Fri Feb 23 23:09:50 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 23 Feb 2007 17:09:50 -0500 Subject: [Python-Dev] Old bug still unfixed? [ python-Patches-1183712 ] package_data chops off first char of default package In-Reply-To: <45DF5E3C.7040204@v.loewis.de> References: <200702231537.25070.meine@informatik.uni-hamburg.de> <200702231537.25070.meine@informatik.uni-hamburg.de> Message-ID: <5.1.1.6.0.20070223165631.0776caa0@sparrow.telecommunity.com> At 10:35 PM 2/23/2007 +0100, Martin v. L?wis wrote: >Hans Meine schrieb: > > > (I first posted this to distutils-sig but was told that distutils is a bit > > neglected there, so I decided to try to push these simple patches in via > > python-dev.) Lately, a student came to me complaining about an old > distutils > > bug which > > bit him: > > http://mail.python.org/pipermail/patches/2006-August/020642.html > > > > The patch is quite trivial and cannot possibly break something AFAICS, so > > could someone please apply it? (Hmm, now that I write this; we are still > > using python 2.4.4 here, so maybe it has been fixed in between?) > >See > >https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1183712&group_id=5470 > >The last comment is from jimjjewet, and he claims that the patch is >still incomplete. There is no response to that claim so far, by >either the patch author nor anybody else. I still believe that the >patch is unnecessary, i.e. it fixes a problem which shouldn't >arise in the first place. I believe you're correct. If anything, the fix should be to disallow an empty string being listed in the 'packages' argument to 'setup()'. I've added some notes to the bug to also show why Jim's concerns about absolute paths and trailing /'s don't actually come into play for this code. From greg.ewing at canterbury.ac.nz Sat Feb 24 00:45:05 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 24 Feb 2007 12:45:05 +1300 Subject: [Python-Dev] bool conversion wart? In-Reply-To: <45DF5C30.6060103@v.loewis.de> References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DEC783.8040601@canterbury.ac.nz> <45DF5C30.6060103@v.loewis.de> Message-ID: <45DF7C81.80705@canterbury.ac.nz> Martin v. L?wis wrote: > One idiom that people use a lot is foo[b], where > b is a boolean. Could that be addressed using the new __index__ mechanism? -- Greg From thomas at python.org Sat Feb 24 01:27:24 2007 From: thomas at python.org (Thomas Wouters) Date: Fri, 23 Feb 2007 16:27:24 -0800 Subject: [Python-Dev] NOARGS_NULL In-Reply-To: <45D81DDA.5060905@v.loewis.de> References: <45D81DDA.5060905@v.loewis.de> Message-ID: <9e804ac0702231627h4c2f9434v15e038b2de67b6b5@mail.gmail.com> My only objection (which is a minor one) is with the 'NOARGS_NULL' name. Caps don't fit the normal style rules, and 'noargs_null' doesn't make much sense to me. 'unused' strikes me as a clearer name (or 'noargs_unused' or 'args_unused' or such.) It should be fine to fix this in 2.5 as well (as long as arguments previously ignored don't suddenly raise exceptions, but it doesn't sound like that's happening at all.) On 2/18/07, "Martin v. L?wis" wrote: > > Patch #1648268 corrects a huge load of errors in Python wrt. > incorrect function pointers (i.e. functions called with a > different signature then declared, through pointers). > The rationale for this patch is that the submitter apparently > has a platform where Python crashes in the face of these errors. > > I believe the patch is correct, and would like to apply it. > > The patch also renames many function arguments: parameters > in a METH_NOARGS function get consistently named NOARGS_NULL > (currently often called 'unused' or 'noargs'); the second parameter > to getters gets consistently named 'closure' (it's called > closure in many places already, and 'unused' in others). > > I would also apply this part of the change, and both to > the trunk and Python 2.5. Objections? > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/thomas%40python.org > -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070223/44ad2f43/attachment.htm From mateusz.rukowicz at vp.pl Sat Feb 24 02:49:51 2007 From: mateusz.rukowicz at vp.pl (Mateusz Rukowicz) Date: Sat, 24 Feb 2007 02:49:51 +0100 Subject: [Python-Dev] C decimal project. Message-ID: <45DF99BF.5060108@vp.pl> Hi, Last holidays I was writting C decimal during summer of code 2006, I planned to take a breath for 1-2 months and then continue work, but well, things didn't go the way I planned. Now I am ready to continue work, I mainly plan to optimize it a little bit (top priority would be cut some copying) and add C api, so C decimal would eventually leave sandbox. My question is - am I permitted to just continue work on svn? Or there is something I have to do before that? Any sugestions etc. would be appreciated. Thanks in advance and best regards, Mateusz Rukowicz. From josepharmbruster at gmail.com Sat Feb 24 04:09:08 2007 From: josepharmbruster at gmail.com (Joseph Armbruster) Date: Fri, 23 Feb 2007 22:09:08 -0500 Subject: [Python-Dev] Unicode command line parameter missing Message-ID: <938f42d70702231909o21b6449cpdecd06b4295b7911@mail.gmail.com> Python Versions: 2.5 and trunk Browsing through the code this evening and noticed a discrepancy between the command line parameters listed with --help and those truly available in the code. Namely with the -U Unicode flag.. The -U option was added back in 2000, rev 15296 and mention of it was removed in rev 25288. Should the -U option still be handled from the command line? Python 2.5 Session without flag: >python Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> a = "monty" >>> a 'monty' >>> Python 2.5 Session with flag: >python -U Python 2.5 (r25:51908, Sep 19 2006, 09:52:17) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> a = "monty" >>> a u'monty' >>> Joseph Armbruster -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070223/77bcf28e/attachment.htm From larry at hastings.org Sat Feb 24 05:36:30 2007 From: larry at hastings.org (Larry Hastings) Date: Fri, 23 Feb 2007 20:36:30 -0800 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: <20070221002351.ADE0.JCARLSON@uci.edu> References: <20070220215842.ADDA.JCARLSON@uci.edu> <45DBF0E4.5090907@hastings.org> <20070221002351.ADE0.JCARLSON@uci.edu> Message-ID: <45DFC0CE.1070305@hastings.org> Josiah Carlson wrote: > Larry Hastings wrote: > >> Josiah Carlson wrote: >> >>> one thing to note with your method - you can't guarantee the order >>> of the attributes as they are being displayed. >>> >> Actually, my record type *can*; see the hack using the __names__ field. > Actually, it *can't*. The ordering of the dict produced by the **kwargs > arguments is exactly same as a regular dictionary. Just to set the record.py straight, more for posterity than anything else: Actually, it *does*, because my prototype has explicit support for imposing such an ordering. That's what the "__names__" field is used for--it's an optional array of field names, in the order you want them displayed from __repr__(). Mr. Carlson's posting, while correct on general principles, was just plain wrong about my code; I suspect he hadn't bothered to read it, and instead based his reply on speculation about how it "probably" worked. My "record" prototype has no end of failings, but an inability to "guarantee the order of the attributes as they are being displayed" is simply not one of them. /larry/ From jcarlson at uci.edu Sat Feb 24 06:44:25 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 23 Feb 2007 21:44:25 -0800 Subject: [Python-Dev] A "record" type (was Re: Py2.6 ideas) In-Reply-To: <45DFC0CE.1070305@hastings.org> References: <20070221002351.ADE0.JCARLSON@uci.edu> <45DFC0CE.1070305@hastings.org> Message-ID: <20070223213736.AE40.JCARLSON@uci.edu> Larry Hastings wrote: > > Josiah Carlson wrote: > > Larry Hastings wrote: > > > >> Josiah Carlson wrote: > >> > >>> one thing to note with your method - you can't guarantee the order > >>> of the attributes as they are being displayed. > >>> > >> Actually, my record type *can*; see the hack using the __names__ field. > > Actually, it *can't*. The ordering of the dict produced by the **kwargs > > arguments is exactly same as a regular dictionary. > > Just to set the record.py straight, more for posterity than anything else: > > Actually, it *does*, because my prototype has explicit support for > imposing such an ordering. That's what the "__names__" field is used > for--it's an optional array of field names, in the order you want them > displayed from __repr__(). > > Mr. Carlson's posting, while correct on general principles, was just > plain wrong about my code; I suspect he hadn't bothered to read it, and > instead based his reply on speculation about how it "probably" worked. No, I just didn't notice the portion of your code (in the 'if __name__ == "__main__" block) that modified the __names__ field after record construction. In the other record types that were offered by Steven and Raymond, the point is to have an ordering that is fixed. I didn't notice it because I expected that it was like Raymond and Steven's; static layout after construction - but that is not the case. - Josiah From martin at v.loewis.de Sat Feb 24 08:11:41 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 Feb 2007 08:11:41 +0100 Subject: [Python-Dev] bool conversion wart? In-Reply-To: <45DF7C81.80705@canterbury.ac.nz> References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DEC783.8040601@canterbury.ac.nz> <45DF5C30.6060103@v.loewis.de> <45DF7C81.80705@canterbury.ac.nz> Message-ID: <45DFE52D.7050206@v.loewis.de> Greg Ewing schrieb: >> One idiom that people use a lot is foo[b], where >> b is a boolean. > > Could that be addressed using the new __index__ mechanism? That would be possible, yes. Martin From martin at v.loewis.de Sat Feb 24 08:17:12 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 24 Feb 2007 08:17:12 +0100 Subject: [Python-Dev] NOARGS_NULL In-Reply-To: <9e804ac0702231627h4c2f9434v15e038b2de67b6b5@mail.gmail.com> References: <45D81DDA.5060905@v.loewis.de> <9e804ac0702231627h4c2f9434v15e038b2de67b6b5@mail.gmail.com> Message-ID: <45DFE678.5010202@v.loewis.de> Thomas Wouters schrieb: > My only objection (which is a minor one) is with the 'NOARGS_NULL' name. > Caps don't fit the normal style rules, and 'noargs_null' doesn't make > much sense to me. 'unused' strikes me as a clearer name (or > 'noargs_unused' or 'args_unused' or such.) The point clearly is that the submitter wants the code to explain what the meaning of the parameter is, and why it is unused. NOARGS_NULL is intended to express that the argument will always be NULL (and thus obviously irrelevant). It's easy to see from the code that the parameter is unused, so it doesn't give much information calling it unused; OTOH it may not be so obvious what information the caller will provide (i.e. whether it's an argument tuple, or the "closure" field). > It should be fine to fix this > in 2.5 as well (as long as arguments previously ignored don't suddenly > raise exceptions, but it doesn't sound like that's happening at all.) That change does not change behaviour at all. Regards, Martin From martin at v.loewis.de Sat Feb 24 08:21:14 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 Feb 2007 08:21:14 +0100 Subject: [Python-Dev] C decimal project. In-Reply-To: <45DF99BF.5060108@vp.pl> References: <45DF99BF.5060108@vp.pl> Message-ID: <45DFE76A.4020100@v.loewis.de> Mateusz Rukowicz schrieb: > Last holidays I was writting C decimal during summer of code 2006, I > planned to take a breath for 1-2 months and then continue work, but > well, things didn't go the way I planned. Now I am ready to continue > work, I mainly plan to optimize it a little bit (top priority would be > cut some copying) and add C api, so C decimal would eventually leave > sandbox. > > My question is - am I permitted to just continue work on svn? Or there > is something I have to do before that? Any sugestions etc. would be > appreciated. As long as you restrict yourself to making only those changes that you originally wanted to make, you can keep the commit privilege as long as you need it. It would be nice if you would let us (Neal Norwitz or myself, or whoever manages svn accounts then) know if you find that you won't contribute to the project anymore, but there are many committers who have "silently" withdrawn, and that's really no problem, either. Regards, Martin From martin at v.loewis.de Sat Feb 24 08:23:27 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 24 Feb 2007 08:23:27 +0100 Subject: [Python-Dev] Unicode command line parameter missing In-Reply-To: <938f42d70702231909o21b6449cpdecd06b4295b7911@mail.gmail.com> References: <938f42d70702231909o21b6449cpdecd06b4295b7911@mail.gmail.com> Message-ID: <45DFE7EF.2010506@v.loewis.de> Joseph Armbruster schrieb: > The -U option was added back in 2000, rev 15296 and mention of it was > removed > in rev 25288. Should the -U option still be handled from the command line? Yes. The original goal of the -U command line (find out what happens if all strings were unicode) still remains at the moment. It may be sensible to remove it when Python 3 has solved the issue in some other way, but then, removing it may cause unnecessary breakage, so it can just as well stay throughout 2.x. Regards, Martin From guido at python.org Sat Feb 24 14:56:42 2007 From: guido at python.org (Guido van Rossum) Date: Sat, 24 Feb 2007 07:56:42 -0600 Subject: [Python-Dev] bool conversion wart? In-Reply-To: <45DFE52D.7050206@v.loewis.de> References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DEC783.8040601@canterbury.ac.nz> <45DF5C30.6060103@v.loewis.de> <45DF7C81.80705@canterbury.ac.nz> <45DFE52D.7050206@v.loewis.de> Message-ID: Can anyone who is in favor of changing this please come up with a spec for the new bool() signature? What would you do for the most common use case of bool(), which is converting an arbitrary value to its Boolean equivalent without using an if test or the "not not x" hack? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From g.brandl at gmx.net Sat Feb 24 19:42:41 2007 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 24 Feb 2007 19:42:41 +0100 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DEC783.8040601@canterbury.ac.nz> <45DF5C30.6060103@v.loewis.de> <45DF7C81.80705@canterbury.ac.nz> <45DFE52D.7050206@v.loewis.de> Message-ID: Guido van Rossum schrieb: > Can anyone who is in favor of changing this please come up with a spec > for the new bool() signature? What would you do for the most common > use case of bool(), which is converting an arbitrary value to its > Boolean equivalent without using an if test or the "not not x" hack? I think this subthread doesn't propose to change bool(), but only to stop inheriting bool from int, while giving bools an __index__ method that returns 0 or 1 to still allow them being used as list indices. The obvious consequence would be that arithmetic wouldn't work any longer with bools involved. Also, mapping access or set membership wouldn't work like now, >>> d = {1: 2} >>> d[True] 2 >>> True in set([1]) True but that would only be logical. Georg From skip at pobox.com Sat Feb 24 23:02:04 2007 From: skip at pobox.com (skip at pobox.com) Date: Sat, 24 Feb 2007 16:02:04 -0600 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> Message-ID: <17888.46556.163061.795918@montanaro.dyndns.org> Neal> Except, all the numeric types do, including int, float, and Neal> complex. But not bool. The fact that bool is a subclass of int is more historic than necessary. If not for Python's long usage of 0 and 1 to be the canonical False and True, I suspect that bool might have been implemented as a new standalone type. Skip From pje at telecommunity.com Sun Feb 25 02:45:52 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 24 Feb 2007 20:45:52 -0500 Subject: [Python-Dev] [Distutils] unicode bug in distutils In-Reply-To: <45E0C012.7090801@palladion.com> References: <94bdd2610702241306q60b1a10rb91dff4919fdae13@mail.gmail.com> <94bdd2610702241247t568a942dw2fe1b10883b62d20@mail.gmail.com> <200702242309.46022.pogonyshev@gmx.net> <94bdd2610702241306q60b1a10rb91dff4919fdae13@mail.gmail.com> Message-ID: <5.1.1.6.0.20070224203115.0270a5a8@sparrow.telecommunity.com> At 02:47 PM 2/24/2007 -0600, Tarek Ziad? wrote: >I have created a setup.py file for distirbution and I bumped into >a small bug when i tried to set my name in the contact field (Tarek Ziad?) > >Using string (utf8 file): > >setup( > maintainer="Tarek Ziad?" >) > >leads to: > > File ".../lib/python2.5/distutils/command/register.py", line 162, in > send_metadata > auth) > File ".../lib/python2.5/distutils/command/register.py", line 257, in > post_to_server > value = unicode(value).encode("utf-8") >UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: >ordinal not in range(128) > > >Using unicode: > >setup( > maintainer=u"Tarek Ziad?" >) > >leads to: > > File ".../lib/python2.5/distutils/dist.py", line 1094, in write_pkg_file > file.write('Author: %s\n' % self.get_contact() ) >UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in >position 18: ordinal not in range(128) > >I would propose a patch for this problem but i don't know what would be >the best input (i guess unicode > for names) At 05:45 PM 2/24/2007 -0500, Tres Seaver wrote: >Don't you still need to tell Python about the encoding of your string >literals [1] [2] ? E.g.:: That's not the problem, it's that the code that writes the PKG-INFO file doesn't handle Unicode. See distutils.dist.DistributionMetadata.write_pkg_info(). It needs to use a file with encoding support, if it's doing unicode However, there's currently no standard, as far as I know, for what encoding the PKG-INFO file should use. Meanwhile, the 'register' command accepts Unicode, but is broken in handling it. Essentially, the problem is that Python 2.5 broke this by adding a unicode *requirement* to the "register" command. Previously, register simply sent whatever you gave it, and the PKG-INFO writing code still does. Unfortunately, this means that there is no longer any one value that you can use for your name that will be accepted by both "register" and anything that writes a PKG-INFO file. Both register and write_pkg_info() are arguably broken here, and should be able to work with either strings or unicode, and degrade gracefully in the event of non-ASCII characters in a string. (Because even though "register" is only run by the package's author, users may run other commands that require a PKG-INFO, so a package prepared using Python <2.5 must still be usable with Python 2.5 distutils, and Python <2.5 allows 8-bit maintainer names.) Unfortunately, this isn't fixable until there's a new 2.5.x release. For previous Python versions, both register and write_pkg_info() accepted 8-bit strings and passed them on as-is, so the only workaround for this issue at the moment is to revert to Python 2.4 or less. From guido at python.org Sun Feb 25 06:24:32 2007 From: guido at python.org (Guido van Rossum) Date: Sat, 24 Feb 2007 23:24:32 -0600 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DEC783.8040601@canterbury.ac.nz> <45DF5C30.6060103@v.loewis.de> <45DF7C81.80705@canterbury.ac.nz> <45DFE52D.7050206@v.loewis.de> Message-ID: On 2/24/07, Georg Brandl wrote: > Guido van Rossum schrieb: > > Can anyone who is in favor of changing this please come up with a spec > > for the new bool() signature? What would you do for the most common > > use case of bool(), which is converting an arbitrary value to its > > Boolean equivalent without using an if test or the "not not x" hack? > > I think this subthread doesn't propose to change bool(), but only to > stop inheriting bool from int, while giving bools an __index__ method > that returns 0 or 1 to still allow them being used as list indices. > > The obvious consequence would be that arithmetic wouldn't work any > longer with bools involved. > > Also, mapping access or set membership wouldn't work like now, > > >>> d = {1: 2} > >>> d[True] > 2 > >>> True in set([1]) > True > > but that would only be logical. How would this change be helpful? I'm utterly mystified by these suggestions that bool would be more useful if it didn't behave like an int in arithmetic. On 2/24/07, skip at pobox.com wrote: > > Neal> Except, all the numeric types do, including int, float, and > Neal> complex. But not bool. > > The fact that bool is a subclass of int is more historic than necessary. If > not for Python's long usage of 0 and 1 to be the canonical False and True, I > suspect that bool might have been implemented as a new standalone type. Not necessarily. I really like the idea that bool is embedded in int, just like int is embedded in float (real), and real is embedded in complex. The construction signature is a different issue -- some people seem to forget that constructor signatures don't need to match their base class; it's not required by Liskov substitutability. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From gabriel.becedillas at corest.com Thu Feb 22 21:11:36 2007 From: gabriel.becedillas at corest.com (Gabriel Becedillas) Date: Thu, 22 Feb 2007 17:11:36 -0300 Subject: [Python-Dev] Bug in PyErr_WriteUnraisable ? Message-ID: <45DDF8F8.5090002@corest.com> I'd hit an access violation inside PyErr_WriteUnraisable when a non-exception instance was raised. The call to PyExceptionClass_Name with a non-exception instance is yielding an invalid pointer. We are embedding Python 2.5 and a string instance is raised using PyThreadState_SetAsyncExc. I can fix that in my code, by raising an appropiate exception instance, but I think PyErr_WriteUnraisable lacks some checks. -- Gabriel Becedillas Developer CORE SECURITY TECHNOLOGIES From sydney.pang at gmail.com Fri Feb 23 13:24:19 2007 From: sydney.pang at gmail.com (Sydney Pang) Date: Fri, 23 Feb 2007 04:24:19 -0800 Subject: [Python-Dev] Embedded Python:: C/Python: Is Py_Finalize() necessary between embedded function calls? Message-ID: <45DEDCF3.9070608@gmail.com> Hi, I am developing an application where I have Python embedded in C functions using the C/Python API to execute commands. My question stems from my need to preserve a PyObject to pass between these Python embedded C functions. My question is: do I have to call Py_Finalize() at the end of each Python embedded C function? The abstract structure of my code looks like this: PyObject *po; main() { po = py_function_1(); // some C code in the middle here... po = py_function_2(po); // some more C code here... } In my implementation of py_function_1 and py_function_2 do I have to call Py_Initialize() and Py_Finalize at the beginning and end of each of the two functions? Or can I just call Py_Initialize at the beginning of py_function_1 and then call Py_Finalize() at the end of py_function_2? Thank You. SP From phd at phd.pp.ru Sun Feb 25 15:48:52 2007 From: phd at phd.pp.ru (Oleg Broytmann) Date: Sun, 25 Feb 2007 17:48:52 +0300 Subject: [Python-Dev] Embedded Python:: C/Python: Is Py_Finalize() necessary between embedded function calls? In-Reply-To: <45DEDCF3.9070608@gmail.com> References: <45DEDCF3.9070608@gmail.com> Message-ID: <20070225144852.GA2813@phd.pp.ru> On Fri, Feb 23, 2007 at 04:24:19AM -0800, Sydney Pang wrote: > do I have to call Py_Finalize() at > the end of each Python embedded C function? As far I understand you only need to call Py_Initialize() and Py_Finalize() once per every embedded interpreter, not for an every function. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From brett at python.org Sun Feb 25 16:14:58 2007 From: brett at python.org (Brett Cannon) Date: Sun, 25 Feb 2007 09:14:58 -0600 Subject: [Python-Dev] Bug in PyErr_WriteUnraisable ? In-Reply-To: <45DDF8F8.5090002@corest.com> References: <45DDF8F8.5090002@corest.com> Message-ID: On 2/22/07, Gabriel Becedillas wrote: > I'd hit an access violation inside PyErr_WriteUnraisable when a > non-exception instance was raised. The call to PyExceptionClass_Name > with a non-exception instance is yielding an invalid pointer. > We are embedding Python 2.5 and a string instance is raised using > PyThreadState_SetAsyncExc. I can fix that in my code, by raising an > appropiate exception instance, but I think PyErr_WriteUnraisable lacks > some checks. > Please use the the bug tracker at http://sourceforge.net/tracker/?group_id=5470&atid=105470 to report issues with Python. -Brett > -- > > > Gabriel Becedillas > Developer > CORE SECURITY TECHNOLOGIES > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org > From arigo at tunes.org Sun Feb 25 18:18:46 2007 From: arigo at tunes.org (Armin Rigo) Date: Sun, 25 Feb 2007 18:18:46 +0100 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> Message-ID: <20070225171846.GA11989@code0.codespeak.net> Hi Adam, On Thu, Feb 15, 2007 at 06:17:03AM -0700, Adam Olsen wrote: > > E.g. have a wait(events = [], timeout = -1) method would be sufficient > > for most cases, where an event would specify > > I agree with everything except this. A simple function call would > have O(n) cost, thus being unacceptable for servers with many open > connections. Instead you need it to maintain a set of events and let > you add or remove from that set as needed. I just realized that this is not really true in the present context. If the goal is to support programs that "look like" they are multi-threaded, i.e. don't use callbacks, as I think is Joachim's goal, then most of the time the wait() function would be only called with a *single* event, rarely two or three, never more. Indeed, in this model a large server is implemented with many microthreads: at least one per client. Each of them blocks in a separate call to wait(). In each such call, only the events revelant to that client are mentioned. In other words, the cost is O(n), but n is typically 1 or 2. It is not the total number of events that the whole application is currently waiting on. Indeed, the scheduler code doing the real OS call (e.g. to select()) can collect the events in internal dictionaries, or in Poll objects, or whatever, and update these dictionaries or Poll objects with the 1 or 2 new events that a call to wait() introduces. In this respect, the act of *calling* wait() already means "add these events to the set of all events that need waiting for", without the need for a separate API for doing that. [Actually, I think that the simplicity of the wait(events=[]) interface over any add/remove/callback APIs is an argument in favor of the "microthread-looking" approach in general, though I know that it's a very subjective topic.] [I have experimented myself with a greenlet-based system giving wrapper functions for os.read()/write() and socket.recv()/send(), and in this style of code we tend to simply spawn new greenlets all the time. Each one looks like an infinite loop doing a single simple job: read some data, process it, write the result somewhere else, start again. (The loops are not really infinite; e.g. if sockets are closed, an exception is generated, and it causes the greenlet to exit.) So far I've managed to always wait on a *single* event in each greenlet, but sometimes it was a bit contrieved and being able to wait on 2-3 events would be handy.] A bientot, Armin. From martin at v.loewis.de Sun Feb 25 19:01:30 2007 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sun, 25 Feb 2007 19:01:30 +0100 Subject: [Python-Dev] NOARGS_NULL In-Reply-To: <45E0C5E3.4030002@tao-group.com> References: <45D81DDA.5060905@v.loewis.de> <9e804ac0702231627h4c2f9434v15e038b2de67b6b5@mail.gmail.com> <45E0C5E3.4030002@tao-group.com> Message-ID: <45E1CEFA.2070601@v.loewis.de> K D schrieb: > I thought it would be nice to have the code marked somehow so that it > was obvious what the additional (and obviously unused) parameter was. > The two references (NULL and METH_NOARGS) are both in upper-case and as > C is a case-sensitive language I think it's probably usual to do > case-sensitive greps etc when trying to understand code. Therefore, I > thought NOARGS_NULL as a name had enough information in it for someone > to "find their way" when looking at the code and wondering what that > unused parameter was declared for. Kev, The style-guide in C is that macro names are spelled in all-caps, so if you see an all-caps identifier, you expect it to be a macro name. Regards, Martin From jeremy at alum.mit.edu Sun Feb 25 20:48:42 2007 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Sun, 25 Feb 2007 13:48:42 -0600 Subject: [Python-Dev] dinner at Standard in Dallas Message-ID: I'm organizing a trip to Standard in downtown Dallas for dinner tonight (Sunday night). It's about a 10 minute cab ride to Standard. We can share cabs and get there without too much trouble. The restaurant is on the expensive side. I'm thinking we should leave from the hotal around 6:30pm. http://www.standarddallas.com/ http://restaurants.dallasobserver.com/search/restaurants.php?oid=27399 A large group of us went for dinner last year and had a great time. The food was excellent, short ribs and mashed potatoes with truffles stand out in my mind. They also serve a tangerita cocktail--margerita made with Tang--that was memorable. Let me know if you'd like to come and I'll make reservations and arrange for cabs. Jeremy [Apologies if you get two copies of this message. The first one looks like it got lost or stuck.] From thomas at python.org Sun Feb 25 22:12:51 2007 From: thomas at python.org (Thomas Wouters) Date: Sun, 25 Feb 2007 13:12:51 -0800 Subject: [Python-Dev] Python-3000 upgrade path Message-ID: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> I'm sending this to python-dev instead of python-3000 for two reasons: It's not about features to be added to Python 3.0, and it's not really about 3.0at all -- it's about 2.6 and later. It's about how we get Python 2.x to 3.0, and howmuch of 3.0we put into 2.6 and later. So here at PyCon, Guido did his usual talk about Py3K and how it's going to look. He also covered the tool he's writing to do the necessary syntactic translations, and short bits about how to write code to accommodate the other changes. One thing Guido didn't cover -- because it isn't his area of expertise and (correctly) expects others to step up and do it -- is the upgrade path for third party Python applications and libraries. I happen to care a lot about that, and so do a few other core Python developers, and we *will* make it happen in the smoothest way possible. We could use some community guidance on this, though. Here's my current thinking on the subject (and I talked about this with a couple of different people at the conference and in email): 1) Anything we can backport from Python 3.0 to Python 2.6 without breaking existing code, will be backported. So far, this means all the new features. 2) A few of the things going away in 3.0 will be warned about in 2.6 by default: things that have had better alternative for ages and are generally bad: backticks and input() are the only ones that come to mind right now. 3) The rest of the things that will change or go away in 3.0 *and* have alternatives in 2.6 will be warned about optionally. Neal is rewriting the warnings module to C, which should speed up warning generation a lot, and if necessary we'll use a global 'warn-for-py3k' flag to reduce the overhead for these warnings to the absolute minimum. 4) The new features of 3.0 that we can't backport (things changing semantics, instead of being added or going away) will be made available in 2.6 (as much as possible), using future statements. Right now, I cannot think of a single thing that cannot be made available that way. They will be, in essence, backward-compatibility hacks, but they'll work and the performance impact will be minimal. Of course, 4 is somewhat of a sweeping statement, even with the parenthesised reservation, considering some of the 3.0 features haven't been implemented yet. I'm pretty confident we can do this without comprimising 3.0, though. If we cannot, we might need to add an extra 'migration feature' in Python 2.7, or 2.8 or even 2.9 (but as it is now, I'm not sure that's specifically necessary. We may still want to release Python 2.7 and later for other reasons, of course.) The end result will be that you can write your Python code so it works in 2.6-or-later and 3.0, or -- just like now -- in 2.x-or-later (where 'x' depends on the features you use) but not 3.0. If you write careful code, you may get away with writing 2.2-or-later code that can be run in 3.0 with a single translation run. It will be fairly unlikely you can write code for 2.5-or-earlier *and* 3.0 without using the translation step, unless you live in a world blissfully unaware of anything non-ASCII. In any case, while we aren't working to make that possible, we certainly won't go out of the way to prevent it (but there still won't be any compromises in the 3.0 language just for code compatibility.) As I said, I've talked with a few people about this, both python core developers and people who develop their own software. I would like to hear from people who have concrete doubts about this upgrade path. I don't mean doubts about Python 3.0 -- this mail isn't about 3.0 at all, and I can only suggest you try out the py3k branch when the dubious feature(s) are implemented. I also don't mean doubts about how we're going to keep the performance impact minimal or how we're going to handle dict views and what not. I mean doubts about this procedure of having transitional releases between 2.5 and 3.0, and the *community* impact of 2.6+ and 3.0 this way. One thing in particular I wonder about is the warning about mixing tabs and spaces. Should it be in category 2) (on by default) or 3) (still off by default, but part of -Wpy3k)? -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070225/c498f01d/attachment-0001.htm From brett at python.org Sun Feb 25 22:22:58 2007 From: brett at python.org (Brett Cannon) Date: Sun, 25 Feb 2007 15:22:58 -0600 Subject: [Python-Dev] status of switch to Roundup off of SF Message-ID: When I gave the PSF members an update on the work to move the python-dev tracker over to Roundup Andrew Kuchling asked me to also send an email to python-dev since October/November. As of right now the biggest thing holding up the transition is documentation. A doc needs to get written explaining basic workflow of how issues should be handled. Also need to go through and remove SF references and make them more general (e.g., mention an "issue tracker" when possible). After that some redirects need to be tweaked. www.python.org/sf/ needs to redirect to the Roundup installation (which is keeping SF bug numbers). bugs.python.org will also be set to point to the Roundup server. After all of that we will find out from Anthony and Neal when a release is going to be so as to do the switch when it won't run into a release (before or after). The current test tracker is at http://psf.upfronthosting.co.za/roundup/tracker/ . The meta tracker for dealing with this stuff is at http://psf.upfronthosting.co.za/roundup/meta/ . Feel free to have a look, but please don't go overboard with the suggestions on changes. It is not difficult to change the tracker after it launches. The last thing any of us want to see is the tracker launch delayed because someone wants a feature that takes a while to implement. Unless there is a critical issue you find it would be appreciated if you file the suggestion and just let us push after the launch if it is deemed it will take too much time to implement. The mailing list where all discussions take place is http://mail.python.org/mailman/listinfo/tracker-discuss . Fairly low-traffic although all updates to the meta tracker get sent there. Obviously if you have questions feel free to ask. And if you want to help with getting the switch to happen just speak up. =) -Brett From jeremy at alum.mit.edu Sun Feb 25 20:03:28 2007 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Sun, 25 Feb 2007 13:03:28 -0600 Subject: [Python-Dev] dinner at Standard in Dallas Message-ID: I'm organizing a trip to Standard in downtown Dallas for dinner tonight (Sunday night). It's about a 10 minute cab ride to Standard. We can share cabs and get there without too much trouble. The restaurant is on the expensive side. I'm thinking we should leave from the hotal around 6:30pm. http://www.standarddallas.com/ http://restaurants.dallasobserver.com/search/restaurants.php?oid=27399 A large group of us went for dinner last year and had a great time. The food was excellent, short ribs and mashed potatoes with truffles stand out in my mind. They also serve a tangerita cocktail--margerita made with Tang--that was memorable. Let me know if you'd like to come and I'll make reservations and arrange for cabs. Jeremy From nnorwitz at gmail.com Sun Feb 25 22:49:41 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 25 Feb 2007 15:49:41 -0600 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: On 2/25/07, Thomas Wouters wrote: > > It's about how we get Python 2.x to 3.0, and howmuch of 3.0 we put into 2.6 and later. I've also talked to a bunch of people at PyCon, including Thomas. There seems to be much concern (rightfully so!) about the upgrade path from 2.x to 3.x. I don't think it will be easy to navigate, but I have confidence we can do a great job. My goal is to rip out as much cruft from the code base for 3k to make it easier to maintain. In order for users to adopt 3k, it must provide real benefit. In addition, it as painless as possible for third party developers to support both 2.x and 3.x. All the developers I talked to expressed relief that we weren't forgetting about them. There's still unease and will be until we demonstrate what we plan to do. They were satisfied with our general approach. The time schedules in PEP 361 (2.6 release schedule) and what Guido has said for 3k (from what I remember) are roughly: April 2007 - 3.0 PEPs and features accepted/decided June 2007 - 3.0a1 - basic (most) features implemented Dec 2007 - 2.6a1 Apr 2008 - 2.6 final July 2008 - 3.0 final Even if we slip these schedules, as long as we keep them in relative order, we can keep 2.6 robust, while also offering many of the 3k features. n From greg.ewing at canterbury.ac.nz Sun Feb 25 23:00:13 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 26 Feb 2007 11:00:13 +1300 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DEC783.8040601@canterbury.ac.nz> <45DF5C30.6060103@v.loewis.de> <45DF7C81.80705@canterbury.ac.nz> <45DFE52D.7050206@v.loewis.de> Message-ID: <45E206ED.2080708@canterbury.ac.nz> Guido van Rossum wrote: > How would this change be helpful? I'm utterly mystified by these > suggestions that bool would be more useful if it didn't behave like an > int in arithmetic. I think there's a desire by some people to get rid of unnecessary conceptual baggage left over for historical reasons. Those people are mystified as to why anyone would *want* a boolean to behave like an int. Personally I'm +0 on this. I wouldn't object if it happened, but I'm not pushing for it. -- Greg From nas at arctrix.com Mon Feb 26 00:05:47 2007 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 25 Feb 2007 23:05:47 +0000 (UTC) Subject: [Python-Dev] Python-3000 upgrade path References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: Neal Norwitz wrote: > The time schedules in PEP 361 (2.6 release schedule) and what Guido > has said for 3k (from what I remember) are roughly: > > April 2007 - 3.0 PEPs and features accepted/decided > June 2007 - 3.0a1 - basic (most) features implemented Any talk at PyCon regarding the new IO system? That looks like the biggest piece of unfinished Py3k work. Neil From nnorwitz at gmail.com Mon Feb 26 00:10:33 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 25 Feb 2007 17:10:33 -0600 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: On 2/25/07, Neil Schemenauer wrote: > Neal Norwitz wrote: > > The time schedules in PEP 361 (2.6 release schedule) and what Guido > > has said for 3k (from what I remember) are roughly: > > > > April 2007 - 3.0 PEPs and features accepted/decided > > June 2007 - 3.0a1 - basic (most) features implemented > > Any talk at PyCon regarding the new IO system? That looks like the > biggest piece of unfinished Py3k work. AFAIK, there hasn't been any work on the new IO system or str/unicode unification. Guido asked for help on these, but I don't know if there were any takers. Sprints will be held over the next several days, so hopefully there will be some work in these areas. n From thomas at python.org Mon Feb 26 00:13:36 2007 From: thomas at python.org (Thomas Wouters) Date: Sun, 25 Feb 2007 15:13:36 -0800 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: <9e804ac0702251513m44678d25y2cba85c3ad2a9057@mail.gmail.com> Just the "it's not there yet" part :) There's some prototype code and email conversations archived, but no recent work that I'm aware of. On 2/25/07, Neil Schemenauer wrote: > > Neal Norwitz wrote: > > The time schedules in PEP 361 (2.6 release schedule) and what Guido > > has said for 3k (from what I remember) are roughly: > > > > April 2007 - 3.0 PEPs and features accepted/decided > > June 2007 - 3.0a1 - basic (most) features implemented > > Any talk at PyCon regarding the new IO system? That looks like the > biggest piece of unfinished Py3k work. > > Neil > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/thomas%40python.org > -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070225/79d9302d/attachment.html From guido at python.org Mon Feb 26 00:37:08 2007 From: guido at python.org (Guido van Rossum) Date: Sun, 25 Feb 2007 17:37:08 -0600 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: On 2/25/07, Neal Norwitz wrote: > On 2/25/07, Neil Schemenauer wrote: > > Any talk at PyCon regarding the new IO system? That looks like the > > biggest piece of unfinished Py3k work. > > AFAIK, there hasn't been any work on the new IO system or str/unicode > unification. Guido asked for help on these, but I don't know if there > were any takers. Sprints will be held over the next several days, so > hopefully there will be some work in these areas. Right. To be honest, I consider the str/unicode unification a much bigger project than the new I/O library. I plan to do the new I/O library first (mostly in Python) first, hopefully at the PyCon sprint, since it can be done with the bytes and unicode objects; the old I/O library will break once we do the unification so the I/O library must be replaced before the unification can be started. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nas at arctrix.com Mon Feb 26 01:26:26 2007 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 25 Feb 2007 18:26:26 -0600 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: <20070226002625.GA3067@python.ca> On Sun, Feb 25, 2007 at 05:37:08PM -0600, Guido van Rossum wrote: > Right. To be honest, I consider the str/unicode unification a much > bigger project than the new I/O library. I was more concerned about IO because it would seem to require your time for design work. The str/unicode work could be farmed out to a bunch of workers. Neil From guido at python.org Mon Feb 26 01:37:41 2007 From: guido at python.org (Guido van Rossum) Date: Sun, 25 Feb 2007 18:37:41 -0600 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <20070226002625.GA3067@python.ca> References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> <20070226002625.GA3067@python.ca> Message-ID: On 2/25/07, Neil Schemenauer wrote: > On Sun, Feb 25, 2007 at 05:37:08PM -0600, Guido van Rossum wrote: > > Right. To be honest, I consider the str/unicode unification a much > > bigger project than the new I/O library. > > I was more concerned about IO because it would seem to require your > time for design work. The str/unicode work could be farmed out to a > bunch of workers. I was more thinking of pulling a few all-nighters -- seriously, since it needs to be done as quickly as possible once it is started. The number of volunteers who want to do C code is also dwindling... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy at alum.mit.edu Mon Feb 26 01:47:55 2007 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Sun, 25 Feb 2007 18:47:55 -0600 Subject: [Python-Dev] Renaming Include/object.h In-Reply-To: <459BD46D.6090505@v.loewis.de> References: <459BD46D.6090505@v.loewis.de> Message-ID: On 1/3/07, "Martin v. L?wis" wrote: > In #1626545, Anton Tropashko requests that object.h should be > renamed, because it causes conflicts with other software. > > I would like to comply with this requests for 2.6, assuming there > shouldn't be many problems with existing software as object.h > shouldn't be included directly, anyway. I like the idea of renaming the header files, but I expect there is a lot more opportunity for breaking code that you estimate. I did a search on google.com/codesearch and found seven external packages that include Python.h and object.h (before I gave up). I assume we expect people won't include it, because it is also included in object.h. But they do it. Is there documentation that says you shouldn't import it? Jeremy > What do you think? > > Regards, > Martin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > From greg.ewing at canterbury.ac.nz Mon Feb 26 03:46:24 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 26 Feb 2007 15:46:24 +1300 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: <45E24A00.4080208@canterbury.ac.nz> Thomas Wouters wrote: > One thing in particular I wonder about > is the warning about mixing tabs and spaces. Should it be in category 2) > (on by default) or 3) (still off by default, but part of -Wpy3k)? For my part, it wouldn't bother me at all if you turn it on by default any time you want. I've been careful to avoid tab-space mixing ever since my earliest uses of Python. -- Greg From barry at python.org Mon Feb 26 05:22:58 2007 From: barry at python.org (Barry Warsaw) Date: Sun, 25 Feb 2007 22:22:58 -0600 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Feb 25, 2007, at 3:49 PM, Neal Norwitz wrote: > The time schedules in PEP 361 (2.6 release schedule) and what Guido > has said for 3k (from what I remember) are roughly: > > April 2007 - 3.0 PEPs and features accepted/decided > June 2007 - 3.0a1 - basic (most) features implemented > Dec 2007 - 2.6a1 > Apr 2008 - 2.6 final > July 2008 - 3.0 final > > Even if we slip these schedules, as long as we keep them in relative > order, we can keep 2.6 robust, while also offering many of the 3k > features. I said semi-jokingly that the first release of Py3k should be / literally/ called Python 3000. Then each successive stabilizing release should drop a zero -- e.g. Python 3000, then Python 300, then Python 30. By the time we get to Python 3 all we should basically have to do is change the defaults of whatever Python 2.x version is out to complete the transition. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.5 (Darwin) iQCVAwUBReJgp3EjvBPtnXfVAQIZQgP+K5iWRTYHYvqb3cAUFJw/yDDiz5JPG94x XdMEnCUXwJVyU30q3FGSNeaz3hwQq7xgJuH5DBRHnGkxp7H/K42ft/KXnJVGnkt4 Kai8e26+zBXmCSCTVdJyKhpAiiFAqKTw26+L+a1jyJdSbUnly3coBAvRaOS9oQn6 QcVfx5vCOsM= =o7Ux -----END PGP SIGNATURE----- From stephen at xemacs.org Mon Feb 26 07:32:11 2007 From: stephen at xemacs.org (Stephen J. Turnbull) Date: Mon, 26 Feb 2007 15:32:11 +0900 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: <87irdpjuok.fsf@uwakimon.sk.tsukuba.ac.jp> Barry Warsaw writes: > I said semi-jokingly that the first release of Py3k should be / > literally/ called Python 3000. Then each successive stabilizing > release should drop a zero -- e.g. Python 3000, then Python 300, then > Python 30. RKN = reverse Knuth numbering? From jordangreenberg at gmail.com Mon Feb 26 04:36:13 2007 From: jordangreenberg at gmail.com (Jordan Greenberg) Date: Sun, 25 Feb 2007 22:36:13 -0500 Subject: [Python-Dev] bool conversion wart? In-Reply-To: References: <3d2ce8cb0702221739r2fbca4ffy4825d02361748e13@mail.gmail.com> <45DEC783.8040601@canterbury.ac.nz> <45DF5C30.6060103@v.loewis.de> <45DF7C81.80705@canterbury.ac.nz> <45DFE52D.7050206@v.loewis.de> Message-ID: <45E255AD.4060104@gmail.com> Guido van Rossum wrote: > How would this change be helpful? I'm utterly mystified by these > suggestions that bool would be more useful if it didn't behave like an > int in arithmetic. I don't think anyones really saying it would be more useful, it obviously wouldn't, since like Greg said, it wouldn't work in mapping access or set membership like it does now. I think it would be more readable though, since thinks like this: >>> True in set([1]) True or that 5+True is equal to 6 aren't necessarily obvious to someone who doesn't know that True evaluates to 1. >> The fact that bool is a subclass of int is more historic than necessary. If >> not for Python's long usage of 0 and 1 to be the canonical False and True, I >> suspect that bool might have been implemented as a new standalone type. > > Not necessarily. I really like the idea that bool is embedded in int, > just like int is embedded in float (real), and real is embedded in > complex. It is elegant to embed bool in int, but to me it screams C. To me, it makes more sense to use an int if you want an int, and to reserve boolean for logic. From an elegance/usefulness standpoint, maybe bool should evaluate to 0 and 1. But IMHO, it would eliminate some gotchas and improve readability if it didn't. -Jordan Greenberg, not a Py-Dev, just an interested luser^H^H^Hrker. From joachim.koenig-baltes at emesgarten.de Mon Feb 26 10:50:27 2007 From: joachim.koenig-baltes at emesgarten.de (=?ISO-8859-1?Q?Joachim_K=F6nig-Baltes?=) Date: Mon, 26 Feb 2007 10:50:27 +0100 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <20070225171846.GA11989@code0.codespeak.net> References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <20070225171846.GA11989@code0.codespeak.net> Message-ID: <45E2AD63.208@emesgarten.de> Armin Rigo wrote: > I just realized that this is not really true in the present context. > If the goal is to support programs that "look like" they are > multi-threaded, i.e. don't use callbacks, as I think is Joachim's goal, > then most of the time the wait() function would be only called with a > *single* event, rarely two or three, never more. Indeed, in this model > a large server is implemented with many microthreads: at least one per > client. Each of them blocks in a separate call to wait(). In each such > call, only the events revelant to that client are mentioned. > Yes exactly. > In other words, the cost is O(n), but n is typically 1 or 2. It is not > the total number of events that the whole application is currently > waiting on. Indeed, the scheduler code doing the real OS call (e.g. to > select()) can collect the events in internal dictionaries, or in Poll > objects, or whatever, and update these dictionaries or Poll objects with > the 1 or 2 new events that a call to wait() introduces. In this > respect, the act of *calling* wait() already means "add these events to > the set of all events that need waiting for", without the need for a > separate API for doing that. > But as I'd like to make the event structure similar to the BSD-kevent structure, we could use a flag in the event structure that tells the schedular to consider it only once or keep it in its dictionary, than the task would not need to supply the event each time. > [I have experimented myself with a greenlet-based system giving wrapper > functions for os.read()/write() and socket.recv()/send(), and in this > style of code we tend to simply spawn new greenlets all the time. Each > one looks like an infinite loop doing a single simple job: read some > data, process it, write the result somewhere else, start again. (The > loops are not really infinite; e.g. if sockets are closed, an exception > is generated, and it causes the greenlet to exit.) So far I've managed > to always wait on a *single* event in each greenlet, but sometimes it > was a bit contrieved and being able to wait on 2-3 events would be > handy.] > I do not spawn new greenlets all the time. Instead, my tasks either wait(....) or call wrappers for read/write/send/recv... that implicitely call wait(...) until enough data is available, and the wait(...) does the yield to the scheduler that can either continue other tasks or call kevent/poll/select if no task is runnable. What I'd like to see in an API/library: * a standard schedular that is easily extensible * event structure/class that's easily extensible E.g. I've extended the kevent structure for the scheduler to also include channels similar to stackless. These are python only communication structures, so there is no OS support for blocking on them, but the scheduler can decide if there is something available for a task that waits on a channel, so the channels are checked first in the schedular to see if a task can continue and only if no channel event is available the schedular calls kevent/select/poll. While the scheluler blocks in kevent/select/poll, nothing happens on the channels as no task is running, so the scheduler never blocks (inside the OS) unnecessarily. Joachim From rhamph at gmail.com Mon Feb 26 15:02:44 2007 From: rhamph at gmail.com (Adam Olsen) Date: Mon, 26 Feb 2007 07:02:44 -0700 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: <20070225171846.GA11989@code0.codespeak.net> References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <20070225171846.GA11989@code0.codespeak.net> Message-ID: On 2/25/07, Armin Rigo wrote: > Hi Adam, > > On Thu, Feb 15, 2007 at 06:17:03AM -0700, Adam Olsen wrote: > > > E.g. have a wait(events = [], timeout = -1) method would be sufficient > > > for most cases, where an event would specify > > > > I agree with everything except this. A simple function call would > > have O(n) cost, thus being unacceptable for servers with many open > > connections. Instead you need it to maintain a set of events and let > > you add or remove from that set as needed. > > I just realized that this is not really true in the present context. > If the goal is to support programs that "look like" they are > multi-threaded, i.e. don't use callbacks, as I think is Joachim's goal, > then most of the time the wait() function would be only called with a > *single* event, rarely two or three, never more. Indeed, in this model > a large server is implemented with many microthreads: at least one per > client. Each of them blocks in a separate call to wait(). In each such > call, only the events revelant to that client are mentioned. > > In other words, the cost is O(n), but n is typically 1 or 2. It is not > the total number of events that the whole application is currently > waiting on. Indeed, the scheduler code doing the real OS call (e.g. to > select()) can collect the events in internal dictionaries, or in Poll > objects, or whatever, and update these dictionaries or Poll objects with > the 1 or 2 new events that a call to wait() introduces. In this > respect, the act of *calling* wait() already means "add these events to > the set of all events that need waiting for", without the need for a > separate API for doing that. That would depend on whether Joachim's wait() refers to the individual tasks' calls or the scheduler's call. I assumed it referred to the scheduler. In the basic form it would literally be select.select(), which has O(n) cost and often fairly large n. -- Adam Olsen, aka Rhamphoryncus From joachim.koenig-baltes at emesgarten.de Mon Feb 26 15:22:59 2007 From: joachim.koenig-baltes at emesgarten.de (=?ISO-8859-1?Q?Joachim_K=F6nig-Baltes?=) Date: Mon, 26 Feb 2007 15:22:59 +0100 Subject: [Python-Dev] microthreading vs. async io In-Reply-To: References: <20070215050031.GH6786@v.igoro.us> <45D445EA.50908@emesgarten.de> <20070225171846.GA11989@code0.codespeak.net> Message-ID: <45E2ED43.5050908@emesgarten.de> Adam Olsen wrote: > That would depend on whether Joachim's wait() refers to the individual > tasks' calls or the scheduler's call. I assumed it referred to the > scheduler. In the basic form it would literally be select.select(), > which has O(n) cost and often fairly large n. The wait(events, timeout) call of a task would only mention the events that the task is interested in. The wait() call yields that list to the scheduler. The scheduler then analyzes the list of events that tasks are waiting for and compares it to it's last call to select/poll/kevent and continues tasks in a round robin fashion until all events have been scheduled to the waiting tasks. Only when the scheduler has no events to deliver (e.g. all tasks are waiting) a new select/poll/kevent OS call is made by the scheduler, with a computed timeout to the lowest timeout value of all the tasks, so that a timeout can be delivered at the right time. Joachim From nnorwitz at gmail.com Mon Feb 26 15:27:26 2007 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 26 Feb 2007 08:27:26 -0600 Subject: [Python-Dev] Renaming Include/object.h In-Reply-To: References: <459BD46D.6090505@v.loewis.de> Message-ID: On 2/25/07, Jeremy Hylton wrote: > On 1/3/07, "Martin v. L?wis" wrote: > > In #1626545, Anton Tropashko requests that object.h should be > > renamed, because it causes conflicts with other software. > > > > I would like to comply with this requests for 2.6, assuming there > > shouldn't be many problems with existing software as object.h > > shouldn't be included directly, anyway. > > I like the idea of renaming the header files, but I expect there is a > lot more opportunity for breaking code that you estimate. I did a > search on google.com/codesearch and found seven external packages that > include Python.h and object.h (before I gave up). So rather than a simple rename, we should probably rename, change all references in the core to use the new name, and make a simple object.h that only does: #include "new_object.h" > I assume we expect people won't include it, because it is also > included in object.h. But they do it. Is there documentation that > says you shouldn't import it? I thought somewhere (embedding/extending doc?) it mentions that only Python.h should be included, but if we have a naming convention/directory structure, this becomes more obvious. Doc/ext/extending.tex: To support extensions, the Python API (Application Programmers Interface) defines a set of functions, macros and variables that provide access to most aspects of the Python run-time system. The Python API is incorporated in a C source file by including the header \code{"Python.h"}. (But it may not say nothing else should be included.) n From brett at python.org Mon Feb 26 16:35:27 2007 From: brett at python.org (Brett Cannon) Date: Mon, 26 Feb 2007 09:35:27 -0600 Subject: [Python-Dev] Bug in PyErr_WriteUnraisable ? In-Reply-To: <45E2F6DA.6030307@corest.com> References: <45DDF8F8.5090002@corest.com> <45E2F6DA.6030307@corest.com> Message-ID: Thanks! Hopefully someone will be able to get to it in the near future. -Brett On 2/26/07, Gabriel Becedillas wrote: > Brett Cannon wrote: > > On 2/22/07, Gabriel Becedillas wrote: > >> I'd hit an access violation inside PyErr_WriteUnraisable when a > >> non-exception instance was raised. The call to PyExceptionClass_Name > >> with a non-exception instance is yielding an invalid pointer. > >> We are embedding Python 2.5 and a string instance is raised using > >> PyThreadState_SetAsyncExc. I can fix that in my code, by raising an > >> appropiate exception instance, but I think PyErr_WriteUnraisable lacks > >> some checks. > >> > > > > Please use the the bug tracker at > > http://sourceforge.net/tracker/?group_id=5470&atid=105470 to report > > issues with Python. > > > > -Brett > > > >> -- > >> > >> > >> Gabriel Becedillas > >> Developer > >> CORE SECURITY TECHNOLOGIES > >> > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> http://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> http://mail.python.org/mailman/options/python-dev/brett%40python.org > >> > > > > I've submitted the bug report with a snippet of code to reproduce the crash. > http://sourceforge.net/tracker/index.php?func=detail&aid=1669182&group_id=5470&atid=105470 > > -- > > > Gabriel Becedillas > Developer > CORE SECURITY TECHNOLOGIES > > From jcarlson at uci.edu Mon Feb 26 19:34:29 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 26 Feb 2007 10:34:29 -0800 Subject: [Python-Dev] bool conversion wart? In-Reply-To: <45E255AD.4060104@gmail.com> References: <45E255AD.4060104@gmail.com> Message-ID: <20070226093805.AE54.JCARLSON@uci.edu> Jordan Greenberg wrote: > Guido van Rossum wrote: > > How would this change be helpful? I'm utterly mystified by these > > suggestions that bool would be more useful if it didn't behave like an > > int in arithmetic. > > I don't think anyones really saying it would be more useful, it > obviously wouldn't, since like Greg said, it wouldn't work in mapping > access or set membership like it does now. I think it would be more > readable though, since thinks like this: > >>> True in set([1]) > True > > or that 5+True is equal to 6 aren't necessarily obvious to someone who > doesn't know that True evaluates to 1. Maybe, but 30-some years of C semantics and another 15 years of Python semantics die hard. a==b for builtin types used to return 1 or 0, now they return True or False. Changing such semantics because some users haven't ever used C or older Pythons would be a bit like changing 'def' to 'fcn' because some users have never used 'def' to define a function. Learn the semantic and move on. > It is elegant to embed bool in int, but to me it screams C. To me, it > makes more sense to use an int if you want an int, and to reserve > boolean for logic. You say "screams C" as if it were necessarily a bad thing. Certainly C isn't a perfect language (which is why I use Python), but there are many reasons why I (and others) use (or not) C, but I can just about guarantee you that the semantics of booleans in C *help* rather than hurt its adoption. > From an elegance/usefulness standpoint, maybe bool should evaluate to 0 > and 1. But IMHO, it would eliminate some gotchas and improve readability > if it didn't. You are, of course, entitled to your opinion. From what I understand, there is perhaps one major "gotcha" in the current semantics: bool(str()) isn't correct for anything except for '' and u''. Then again, that isn't a guarantee with str or bool (or list, tuple, dict, deque, set, etc.) - Josiah From dalke at dalkescientific.com Mon Feb 26 23:38:05 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Mon, 26 Feb 2007 15:38:05 -0700 Subject: [Python-Dev] with_traceback Message-ID: Guido's talk at PyCon said: > Use raise E(arg).with_traceback(tb) > instead of raise E, arg, tb That seems strange to me because of the mutability. Looking through the back discussions on this list I see Guido commented: http://mail.python.org/pipermail/python-3000/2007-February/005689.html > Returning the mutated object is acceptable here > because the *dominant* use case is creating and raising an exception > in one go: > > raise FooException().with_traceback() The 3 argument raise statement is rarely used, in my experience. I believe most don't even know it exists, excepting mostly advanced Python programmers and language lawyers. My concern when I saw Guido's keynote was the worry that people do/might write code like this NO_END_OF_RECORD = ParserError("Cannot find end of record") def parse_record(input_file): ... raise NO_END_OF_RECORD ... That is, create instances at the top of the module, to be used later. This code assume that the NO_END_OF_RECORD exception instance is never modified. If the traceback is added to its __traceback__ attribute then I see two problems if I were to write code like the above: - the traceback stays around "forever" - the code is no longer thread-safe. As an example of code which is affected by this, see pyparsing, which has code that looks like class Token(ParserElement): """Abstract ParserElement subclass, for defining atomic matching patterns."" " def __init__( self ): super(Token,self).__init__( savelist=False ) self.myException = ParseException("",0,"",self) .... class Literal(Token): .... def parseImpl( self, instring, loc, doActions=True ): if (instring[loc] == self.firstMatchChar and (self.matchLen==1 or instring.startswith(self.match,loc)) ): return loc+self.matchLen, self.match #~ raise ParseException( instring, loc, self.errmsg ) exc = self.myException exc.loc = loc exc.pstr = instring raise exc The "Literal" and other token classes are part of the grammar definition and usually exist in module scope. There's another question I came up with. If the exception already has a __traceback__, will that traceback be overwritten if the instance is reraised? Consider this code, loosly derived from os._execvpe import os, sys _PATH = ["here", "there", "elsewhere"] def open_file_on_path(name): # If nothing exists, raises an exception based on the # first attempt saved_err = None saved_tb = None for dirname in _PATH: try: return open(os.path.join(dirname, name)) except Exception, err: if not saved_err: saved_err = err saved_tb = sys.exc_info()[2] raise saved_err.__class__, saved_err, saved_tb open_file_on_path("spam") which generates this Traceback (most recent call last): File "raise.py", line 19, in open_file_on_path("spam") File "raise.py", line 11, in open_file_on_path return open(os.path.join(dirname, name)) IOError: [Errno 2] No such file or directory: 'here/spam' What is the correct way to rewrite this for use with "with_traceback"? Is it def open_file_on_path(name): # If nothing exists, raises an exception based on the # first attempt saved_err = None for dirname in _PATH: try: return open(os.path.join(dirname, name)) except Exception, err: if not saved_err: saved_err = err saved_tb = sys.exc_info()[2] raise saved_err.with_traceback(saved_err.__traceback__) One alternative, btw, is raise saved_err.with_traceback() to have it use the existing __traceback__ (and raising its own exception if __traceback__ is None?) Andrew dalke at dalkescientific.com From thomas at python.org Tue Feb 27 00:10:52 2007 From: thomas at python.org (Thomas Wouters) Date: Mon, 26 Feb 2007 15:10:52 -0800 Subject: [Python-Dev] with_traceback In-Reply-To: References: Message-ID: <9e804ac0702261510s565384ecuf96e054788fac7f3@mail.gmail.com> On 2/26/07, Andrew Dalke wrote: > My concern when I saw Guido's keynote was the worry that > people do/might write code like this > > NO_END_OF_RECORD = ParserError("Cannot find end of record") > > def parse_record(input_file): > ... > raise NO_END_OF_RECORD > ... FWIW, you can remove 'might' from that sentence: I've seen more than a few people use that kind of code (and I share your concerns.) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20070226/4171161c/attachment.htm From greg.ewing at canterbury.ac.nz Tue Feb 27 00:26:22 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Feb 2007 12:26:22 +1300 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <87irdpjuok.fsf@uwakimon.sk.tsukuba.ac.jp> References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> <87irdpjuok.fsf@uwakimon.sk.tsukuba.ac.jp> Message-ID: <45E36C9E.4040401@canterbury.ac.nz> Stephen J. Turnbull wrote: > RKN = reverse Knuth numbering? No, for RKN you'd have to start with 3141.5926... (an infinite number of digits following) and drop one off the end each time... although it would take rather a long time to get to the final release then. :-( -- Greg From pje at telecommunity.com Tue Feb 27 00:41:27 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 26 Feb 2007 18:41:27 -0500 Subject: [Python-Dev] with_traceback In-Reply-To: Message-ID: <5.1.1.6.0.20070226182128.02651388@sparrow.telecommunity.com> At 03:38 PM 2/26/2007 -0700, Andrew Dalke wrote: >Guido's talk at PyCon said: > > > Use raise E(arg).with_traceback(tb) > > instead of raise E, arg, tb > >That seems strange to me because of the mutability. Looking through >the back discussions on this list I see Guido commented: > http://mail.python.org/pipermail/python-3000/2007-February/005689.html > > > Returning the mutated object is acceptable here > > because the *dominant* use case is creating and raising an exception > > in one go: > > > > raise FooException().with_traceback() > >The 3 argument raise statement is rarely used, in my experience. >I believe most don't even know it exists, excepting mostly advanced >Python programmers and language lawyers. > >My concern when I saw Guido's keynote was the worry that >people do/might write code like this > >NO_END_OF_RECORD = ParserError("Cannot find end of record") > >def parse_record(input_file): > ... > raise NO_END_OF_RECORD > ... > > >That is, create instances at the top of the module, to be used >later. This code assume that the NO_END_OF_RECORD >exception instance is never modified. > >If the traceback is added to its __traceback__ attribute then >I see two problems if I were to write code like the above: > > - the traceback stays around "forever" > - the code is no longer thread-safe. Then don't do that, as it's bad style for Python 3.x. ;-) But do note that 3-argument raise should NOT be implemented this way in Python 2.x. 2.6 and other 2.x revisions should still retain the existing raise machinery, it's just that *catching* an exception using 3.x style ("except foo as bar:") should call with_traceback() at the time of the catch. This does mean you won't be able to port your code to 3.x style until you've gotten rid of shared exception instances from all your dependencies, but 3.x porting requires all your dependencies to be ported anyway. It should be sufficient in both 2.x and 3.x for with_traceback() to raise an error if the exception already has a traceback -- this should catch any exception instance reuse. >What is the correct way to rewrite this for use >with "with_traceback"? Is it > >def open_file_on_path(name): > # If nothing exists, raises an exception based on the > # first attempt > saved_err = None > for dirname in _PATH: > try: > return open(os.path.join(dirname, name)) > except Exception, err: > if not saved_err: > saved_err = err > saved_tb = sys.exc_info()[2] > raise saved_err.with_traceback(saved_err.__traceback__) No, it's more like this: try: for dirname in ... try: return ... except Exception as err: saved_err = err raise saved_err finally: del saved_err I've added the outer try-finally block to minimize the GC impact of the *original* code you showed, as the `saved_tb` would otherwise have created a cycle. That is, the addition is not because of the porting, it's just something that you should've had to start with. Anyway, the point here is that in 3.x style, most uses of 3-argument raise just disappear altogether. If you hold on to an exception instance, you have to be careful about it for GC, but no more so than in current Python. The "save one instance and use it forever" use case is new to me - I've never seen nor written code that uses it before now. It's definitely incompatible with 3.x style, though. From greg.ewing at canterbury.ac.nz Tue Feb 27 01:37:21 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 27 Feb 2007 13:37:21 +1300 Subject: [Python-Dev] with_traceback In-Reply-To: <5.1.1.6.0.20070226182128.02651388@sparrow.telecommunity.com> References: <5.1.1.6.0.20070226182128.02651388@sparrow.telecommunity.com> Message-ID: <45E37D41.8000005@canterbury.ac.nz> Phillip J. Eby wrote: > At 03:38 PM 2/26/2007 -0700, Andrew Dalke wrote: > > > NO_END_OF_RECORD = ParserError("Cannot find end of record") > > Then don't do that, as it's bad style for Python 3.x. ;-) I don't like that answer. I can think of legitimate reasons for wanting to pre-create exceptions, e.g. if I'm intending to raise and catch a particular exception frequently and I don't want the overhead of creating a new instance each time. For me, this is casting serious doubt on the whole idea of attaching the traceback to the exception. -- Greg From dalke at dalkescientific.com Tue Feb 27 01:52:37 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Mon, 26 Feb 2007 17:52:37 -0700 Subject: [Python-Dev] with_traceback In-Reply-To: <5.1.1.6.0.20070226182128.02651388@sparrow.telecommunity.com> References: <5.1.1.6.0.20070226182128.02651388@sparrow.telecommunity.com> Message-ID: PJE: > Then don't do that, as it's bad style for Python 3.x. ;-) It's bad style for 3.x only if Python goes with this interface. If it stays with the 2.x style then there's no problem. There may also be solutions which are cleaner and which don't mutate the exception instance. I am not proposing such a syntax. I have ideas I am not a language designer and have long given up the idea that I might be good at it. > This does mean you won't be able to port your code to 3.x style until > you've gotten rid of shared exception instances from all your dependencies, > but 3.x porting requires all your dependencies to be ported anyway. What can be done to minimize the number of dependencies which need to be changed? > It should be sufficient in both 2.x and 3.x for with_traceback() to raise > an error if the exception already has a traceback -- this should catch any > exception instance reuse. That would cause a problem in my example where I save then reraise the exception, as raise saved_err.with_traceback(saved_err.__traceback__) > >What is the correct way to rewrite this for use > >with "with_traceback"? Is it [...] > No, it's more like this: > > try: > for dirname in ... > try: > return ... > except Exception as err: > saved_err = err > raise saved_err > finally: > del saved_err I don't get it. The "saved_err" has a __traceback__ attached to it, and is reraised. Hence it gets the old stack, right? Suppose I wrote ERR = Exception("Do not do that") try: f(x) except Exception: raise ERR try: f(x*2) except Exception: raise ERR Yes it's bad style, but people will write it. The ERR gets the traceback from the first time there's an error, and that traceback is locked in ... since raise won't change the __traceback__ if one exists. (Based on what you said it does.) > I've added the outer try-finally block to minimize the GC impact of the > *original* code you showed, as the `saved_tb` would otherwise have created > a cycle. That is, the addition is not because of the porting, it's just > something that you should've had to start with. Like I said, I used code based on os._execvpe. Here's the code saved_exc = None saved_tb = None for dir in PATH: fullname = path.join(dir, file) try: func(fullname, *argrest) except error, e: tb = sys.exc_info()[2] if (e.errno != ENOENT and e.errno != ENOTDIR and saved_exc is None): saved_exc = e saved_tb = tb if saved_exc: raise error, saved_exc, saved_tb raise error, e, tb I see similar use in atexit._run_exitfuncs, though as Python is about to exit it won't make a real difference. doctest shows code like >>> exc_info = failure.exc_info >>> raise exc_info[0], exc_info[1], exc_info[2] SimpleXMLRPCServer does things like except: # report exception back to server exc_type, exc_value, exc_tb = sys.exc_info() response = xmlrpclib.dumps( xmlrpclib.Fault(1, "%s:%s" % (exc_type, exc_value)), encoding=self.encoding, allow_none=self.allow_none, ) I see threading.py gets it correctly. My point here is that most Python code which uses the traceback term doesn't break the cycle, so must be caught by the gc. While there might be a more correct way to do it, it's too complicated for most to get it right. > Anyway, the point here is that in 3.x style, most uses of 3-argument raise > just disappear altogether. If you hold on to an exception instance, you > have to be careful about it for GC, but no more so than in current Python. Where people already make a lot of mistakes. But my concern is not in the gc, it's in the mutability of the exception causing hard to track down problems in code which is written by beginning to intermediate users. > The "save one instance and use it forever" use case is new to me - I've > never seen nor written code that uses it before now. It's definitely > incompatible with 3.x style, though. I pointed out an example in pyparsing. Thomas W. says he's seen other code. I've been looking for another real example but as this is relatively uncommon code, I don't have a wide enough corpus for the search. I also don't know of a good tool for searching for this sort of thing. (Eg, www.koders.com doesn't help.) It's a low probability occurance. So is the use of the 3 arg raise. Hence it's hard to get good intuition about problems which might arise. Andrew dalke at dalkescientific.com From gabriel.becedillas at corest.com Mon Feb 26 16:03:54 2007 From: gabriel.becedillas at corest.com (Gabriel Becedillas) Date: Mon, 26 Feb 2007 12:03:54 -0300 Subject: [Python-Dev] Bug in PyErr_WriteUnraisable ? In-Reply-To: References: <45DDF8F8.5090002@corest.com> Message-ID: <45E2F6DA.6030307@corest.com> Brett Cannon wrote: > On 2/22/07, Gabriel Becedillas wrote: >> I'd hit an access violation inside PyErr_WriteUnraisable when a >> non-exception instance was raised. The call to PyExceptionClass_Name >> with a non-exception instance is yielding an invalid pointer. >> We are embedding Python 2.5 and a string instance is raised using >> PyThreadState_SetAsyncExc. I can fix that in my code, by raising an >> appropiate exception instance, but I think PyErr_WriteUnraisable lacks >> some checks. >> > > Please use the the bug tracker at > http://sourceforge.net/tracker/?group_id=5470&atid=105470 to report > issues with Python. > > -Brett > >> -- >> >> >> Gabriel Becedillas >> Developer >> CORE SECURITY TECHNOLOGIES >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> http://mail.python.org/mailman/options/python-dev/brett%40python.org >> > I've submitted the bug report with a snippet of code to reproduce the crash. http://sourceforge.net/tracker/index.php?func=detail&aid=1669182&group_id=5470&atid=105470 -- Gabriel Becedillas Developer CORE SECURITY TECHNOLOGIES From anugupta at pu.ac.in Tue Feb 27 09:15:22 2007 From: anugupta at pu.ac.in (Anu Gupta DCSA) Date: Tue, 27 Feb 2007 13:45:22 +0530 Subject: [Python-Dev] A Survey on Defect Management Practices in Free/Open Source Software Message-ID: <20070227081513.M67577@pu.ac.in> Sir/Madam I seek help from designers, developers, testers,defect fixers,project managers or playing any other key role in Free/Open Source software development or maintenence in carrying out a study to identify practices and problems of defect management in various Free/Open Source Software projects. The insights gained from the study can further help us to extract publicly accessible defect data and determine impact of defect management practices on software quality. Please spend a few minutes of your precious time to fill up the Questionnaire. The most of the questions follow multiple choice formats and are quite easy to answer. To have the Online Questionnaire, please visit: http://anu.puchd.ac.in/phpESP/public/survey.php?name=FOSS_Defect_Survey (You can also copy and paste this link into your browser, and hit the 'Return' key.) I hope you will find all the questions interesting and thought-provoking. Your answers will be kept anonymous.The data thus collected will only be used for research purpose.It would be nice if you may further refer this mail to others actively engaged with Free/Open Source Software development. If you have any query or suggestions then feel free to contact. Thank You With regards, Anu Gupta Senior Lecturer Department of Computer Science and Applications, Panjab University, Chandigarh. INDIA In case of any problem in accessing/using the above mentioned link please contact: E-mail: anugupta at pu.ac.in anugupta98 at gmail.com From anugupta at pu.ac.in Tue Feb 27 09:04:40 2007 From: anugupta at pu.ac.in (Anu Gupta DCSA) Date: Tue, 27 Feb 2007 13:34:40 +0530 Subject: [Python-Dev] A Reminder : A Survey on Defect Management Practices in Free/Open Source Software Message-ID: <20070227080423.M60394@pu.ac.in> Sir/Madam I seek help from designers, developers, testers,defect fixers,project managers or playing any other key role in Free/Open Source software development or maintenence in carrying out a study to identify practices and problems of defect management in various Free/Open Source Software projects. The insights gained from the study can further help us to extract publicly accessible defect data and determine impact of defect management practices on software quality. Please spend a few minutes of your precious time to fill up the Questionnaire. The most of the questions follow multiple choice formats and are quite easy to answer. To have the Online Questionnaire, please visit: http://anu.puchd.ac.in/phpESP/public/survey.php?name=FOSS_Defect_Survey (You can also copy and paste this link into your browser, and hit the 'Return' key.) I hope you will find all the questions interesting and thought-provoking. Your answers will be kept anonymous.The data thus collected will only be used for research purpose.It would be nice if you may further refer this mail to others actively engaged with Free/Open Source Software development. If you have any query or suggestions then feel free to contact. Thank You With regards, Anu Gupta Senior Lecturer Department of Computer Science and Applications, Panjab University, Chandigarh. INDIA In case of any problem in accessing/using the above mentioned link please contact: E-mail: anugupta at pu.ac.in anugupta98 at gmail.com From facundo at taniquetil.com.ar Tue Feb 27 13:09:35 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Tue, 27 Feb 2007 12:09:35 +0000 (UTC) Subject: [Python-Dev] Adding timeout option to httplib...connect() References: <17868.31888.60854.91462@montanaro.dyndns.org> <200702091353.25069.fdrake@acm.org> <17868.57918.906768.490015@montanaro.dyndns.org> <45CDBE8E.2080805@v.loewis.de> <17870.5847.223422.932006@montanaro.dyndns.org> Message-ID: skip at pobox.com wrote: > Guido, I looked at urllib2 and quickly gave up. I have no idea how that > code works (where is a lower level library's connection object instantiated, > for example?). I presume with timeouts in the lower level libraries someone > who knows how urllib2 works will be able to graft timeouts onto it without > too much work. I have in my TODO list an item that says "put timout to urllib2.urlopen". I once digged in urllib2, trying to figure out something, and in a lightning shock I understood it. I took a look at it again a few weeks ago, but this time wasn't able to follow it, :(. Anyway, I'll check it once again, hope to understand it, and try to came up with a patch to be able to call urllib2.urlopen with "timeout" parameter, for all its protocols. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From alan.mcintyre at gmail.com Wed Feb 28 01:48:08 2007 From: alan.mcintyre at gmail.com (Alan McIntyre) Date: Tue, 27 Feb 2007 19:48:08 -0500 Subject: [Python-Dev] zipfile patch (1121142) updated Message-ID: <1d36917a0702271648l2124600cu4659d751328d86c@mail.gmail.com> I posted what I hope can be the final update to patch 1121142 (adding an open() method to ZipFile that returns a read-only file-like object). If anybody has time to review and apply it I would appreciate it. https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1121142&group_id=5470 If I find time later I may start another patch to add a ZipFile method analogous to TarFile.extract, and any neat features from the module Glyph suggested, but I figured it was best to stick to the scope of the original patch and get it wrapped up. Thanks, Alan From greg.ewing at canterbury.ac.nz Wed Feb 28 04:34:58 2007 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 28 Feb 2007 16:34:58 +1300 Subject: [Python-Dev] Integer division operator can give float result? Message-ID: <45E4F862.20401@canterbury.ac.nz> Is this intentional? I would have expected the // operator to always give an integer result. Python 2.3 (#1, Aug 5 2003, 15:52:30) [GCC 3.1 20020420 (prerelease)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 7.0 // 2 3.0 -- Greg From fdrake at acm.org Wed Feb 28 05:30:40 2007 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue, 27 Feb 2007 23:30:40 -0500 Subject: [Python-Dev] Integer division operator can give float result? In-Reply-To: <45E4F862.20401@canterbury.ac.nz> References: <45E4F862.20401@canterbury.ac.nz> Message-ID: <200702272330.41201.fdrake@acm.org> On Tuesday 27 February 2007 22:34, Greg Ewing wrote: > Is this intentional? I would have expected the > // operator to always give an integer result. Think "floor division", not "integer division". The result (r) may be a float, but it'll hold to the constraint: r == int(r) -Fred -- Fred L. Drake, Jr. From glyph at divmod.com Wed Feb 28 06:49:23 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Wed, 28 Feb 2007 00:49:23 -0500 Subject: [Python-Dev] with_traceback In-Reply-To: <45E37D41.8000005@canterbury.ac.nz> Message-ID: <20070228054923.17852.893320900.divmod.quotient.712@ohm> On Tue, 27 Feb 2007 13:37:21 +1300, Greg Ewing wrote: >I don't like that answer. I can think of legitimate >reasons for wanting to pre-create exceptions, e.g. if >I'm intending to raise and catch a particular exception >frequently and I don't want the overhead of creating >a new instance each time. This seems like kind of a strange micro-optimization to have an impact on a language change discussion. Wouldn't it be better just to optimize instance creation overhead? Or modify __new__ on your particular heavily-optimized exception to have a free-list, so it can be both correct (you can still mutate exceptions) and efficient (you'll only get a new exception object if you really need it). >For me, this is casting serious doubt on the whole >idea of attaching the traceback to the exception. I'm sorry if this has been proposed already in this discussion (I searched around but did not find it), but has anyone thought of adding methods to Exception to handle these edge cases and *not* attempting to interact with the 'raise' keyword at all? This is a strawman, but: except Foo as foo: if foo.some_property: foo.reraise(modify_stack=False) This would make the internal implementation details less important, since the 'raise' keyword machinery will have to respect some internal state of the exception object in either case, but the precise thing being raised need not be the result of the method, nor the exception itself. From glyph at divmod.com Wed Feb 28 07:28:40 2007 From: glyph at divmod.com (glyph at divmod.com) Date: Wed, 28 Feb 2007 01:28:40 -0500 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> Message-ID: <20070228062840.17852.238152490.divmod.quotient.727@ohm> On Sun, 25 Feb 2007 13:12:51 -0800, Thomas Wouters wrote: >I'm sending this to python-dev instead of python-3000 for two reasons: It's >not about features to be added to Python 3.0, and it's not really >about 3.0at all -- it's about >2.6 and later. It's about how we get Python 2.x to 3.0, and howmuch of >3.0we put into >2.6 and later. ... >I also don't mean doubts about how we're going to keep the >performance impact minimal or how we're going to handle dict views and what >not. I mean doubts about this procedure of having transitional releases >between 2.5 and 3.0, and the *community* impact of 2.6+ and 3.0 this way. I'm pretty tired at the moment, fried from PyCon, and have a lot of work to catch up on, so I'll have to spend a while collecting my thoughts to respond in more depth. However, especially since I've been a vocal proponent of a strategy like this for a while, I would like to say that I'm basically happy with this. Not only that, I'm significantly relieved after PyCon about the difficulty of the migration, and confident that, if my current understanding is correct, Twisted _will_ support 3.0 and beyond. Effectively, what this plan means is that the most *basic* aspects of transitioning to Python 3.0 should work like the transition to any other new Python release. More changes may be necessary, but it should be possible to add a few restrictions to your codebase, and continue to supporting older releases with little trouble. Transitioning to 2.5 was a fairly rocky upgrade for Twisted as well, though, and we made that one reasonably well. The one concern I have is that there are things the source translator won't do correctly. That's fine, but I think there are two important technical features it really needs to make the social aspects of this upgrade work well. 2to3 should take great care _tell_ you when it fails. One concern I have is that the source translation may subtly alter the *semantics* of unit test code, so that the tests are no longer effective and do not provide adequate coverage. I think this is an extremely unlikely edge case, but a very dangerous one in the event that it does happen. I don't know of any bugs in 2to3 that will do this at the moment, but then it's hardly the final release. Also, the limitations of 2to3 should be very well documented, so that when it does tell you how it has failed, it's clear to the developer what changes to make to the *2.6 source* to cause it to translate correctly, not how to fix up the translated 3.0 code. Thanks *very* much to all the python core developers and community members who have been working to make this happen, especially to Neal, Thomas, and Guido for listening intently to all of our concerns at PyCon. From collinw at gmail.com Wed Feb 28 08:41:49 2007 From: collinw at gmail.com (Collin Winter) Date: Wed, 28 Feb 2007 01:41:49 -0600 Subject: [Python-Dev] Python-3000 upgrade path In-Reply-To: <20070228062840.17852.238152490.divmod.quotient.727@ohm> References: <9e804ac0702251312ma7f1fb8w7d8d6f8426af920d@mail.gmail.com> <20070228062840.17852.238152490.divmod.quotient.727@ohm> Message-ID: <43aa6ff70702272341l75f75b34y71e6c100764644c9@mail.gmail.com> On 2/28/07, glyph at divmod.com wrote: [snip] > 2to3 should take great care _tell_ you when it fails. One concern I have is that the > source translation may subtly alter the *semantics* of unit test code, so that the tests > are no longer effective and do not provide adequate coverage. I think this is an > extremely unlikely edge case, but a very dangerous one in the event that it does > happen. I don't know of any bugs in 2to3 that will do this at the moment, but then it's > hardly the final release. We're trying hard to avoid semantic changes, and anywhere that semantic changes might be introduced, 2to3 will throw up all kinds of warning messages. 2to3 generally tries to limit itself to safe, syntax-only changes ("apply(f, v, k)" -> "(f)(*v, **k)"), though there are pathological cases where even that could go astray ("d.has_key(k)" -> "k in d", where d.has_key() and d.__contains__ differ). 2to3 should be approached as a conversion guide, not as a converter itself. > Also, the limitations of 2to3 should be very well documented, so that when it does tell > you how it has failed, it's clear to the developer what changes to make to the *2.6 > source* to cause it to translate correctly, not how to fix up the translated 3.0 code. This is something I'm working on at the moment, compiling a list of "things 2to3 can't catch". Some are obvious (m = d.has_key; if m(k): ...), others less so ("raise E, V" where V is an exception instance). We're hoping that 2to3 will be able to convert 90% of Python 2.x automatically and offer a solid, step-by-step plan for manually recoding the remaining 10%. Collin Winter From dalke at dalkescientific.com Wed Feb 28 09:50:54 2007 From: dalke at dalkescientific.com (Andrew Dalke) Date: Wed, 28 Feb 2007 01:50:54 -0700 Subject: [Python-Dev] with_traceback In-Reply-To: <20070228054923.17852.893320900.divmod.quotient.712@ohm> References: <45E37D41.8000005@canterbury.ac.nz> <20070228054923.17852.893320900.divmod.quotient.712@ohm> Message-ID: Glyph: > This seems like kind of a strange micro-optimization to have an impact > on a language change discussion. Just as a reminder, my concern is that people reuse exceptions (rarely) and that the behavior of the "with_exceptions()" method is ambiguous when that happens. It has nothing to do with optimization. The two solutions of: 1. always replace an existing __traceback__ 2. never replace an existing __traceback__ both seem to lead to problems. Here are minimal examples for thought: # I know PJE says this is bad style for 3.0. Will P3K help # identify this problem? If it's allowable, what will it do? # (Remember, I found existing code which reuses exceptions # so this isn't purely theoretical, only rare.) BAD = Exception("that was bad") try: raise BAD except Exception: pass raise BAD # what traceback will be shown here? (Oh, and what would a debugger report?) # 2nd example; reraise an existing exception instance. # It appears that any solution which reports a problem # for #1 will not allow one or both of the following. try: raise Exception("bad") except Exception as err: first_err = err try: raise Exception("bad") except Exception: raise first_err # what traceback will be shown here? # 3rd example, like the 2nd but end it with raise first_err.with_exception(first_err.__traceback__) # what traceback will be shown here? > I'm sorry if this has been proposed already in this discussion (I > searched around but did not find it), I saw references to a PEP about it but could not find the PEP. Nor could I find much discussion either. I would like to know the details. I assume that "raise" adds the __traceback__ if it's not None, hence there's no way it can tell if the __traceback__ on the instance was created with "with_traceback()" from an earlier "raise" or from the with_traceback. But in the current examples it appears that the Exception class could attach a traceback during instantiation and "with_traceback" simply replaces that. I doubt this version, but cannot be definitive. While variant method/syntax may improve matters, I think people will write code as above -- all of which are valid Python 2.x and 3.x -- and end up with strange and hard to interpret tracebacks. Andrew dalke at dalkescientific.com From nmm1 at cus.cam.ac.uk Wed Feb 28 10:00:03 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Wed, 28 Feb 2007 09:00:03 +0000 Subject: [Python-Dev] Class destructor Message-ID: I am gradually making progress with my binary floating-point software, but have had to rewrite several times as I have forgotten most of the details of how to do it! After 30 years, I can't say I am surprised. But I need to clean up workspace when a class (not object) is deallocated. I can't easily use attributes, as people suggested, because there is no anonymous storage built-in type. I could subvert one of the existing storage types (buffer, string etc.), but that is unclean. And I could write one, but that is excessive. So far, I have been unable to track down how to get something called when a class is destroyed. The obvious attempts all didn't work, in a variety of ways. Surely there must be a method? This could be in either Python or C. Thanks. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From facundo at taniquetil.com.ar Wed Feb 28 13:22:36 2007 From: facundo at taniquetil.com.ar (Facundo Batista) Date: Wed, 28 Feb 2007 12:22:36 +0000 (UTC) Subject: [Python-Dev] Integer division operator can give float result? References: <45E4F862.20401@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Is this intentional? I would have expected the > // operator to always give an integer result. > > Python 2.3 (#1, Aug 5 2003, 15:52:30) > [GCC 3.1 20020420 (prerelease)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > >>> 7.0 // 2 > 3.0 >From the Language Reference: The / (division) and // (floor division) operators yield the quotient of their arguments. The numeric arguments are first converted to a common type. Plain or long integer division yields an integer of the same type; the result is that of mathematical division with the `floor' function applied to the result. So, first the arguments are converted to a common type, in this case float. Then the division is made. Then the 'floor' function is applied. Result: 3.0. So yes, it's intentional. Regards, -- . Facundo . Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From rhamph at gmail.com Wed Feb 28 15:33:35 2007 From: rhamph at gmail.com (Adam Olsen) Date: Wed, 28 Feb 2007 07:33:35 -0700 Subject: [Python-Dev] with_traceback In-Reply-To: <20070228054923.17852.893320900.divmod.quotient.712@ohm> References: <45E37D41.8000005@canterbury.ac.nz> <20070228054923.17852.893320900.divmod.quotient.712@ohm> Message-ID: On 2/27/07, glyph at divmod.com wrote: > On Tue, 27 Feb 2007 13:37:21 +1300, Greg Ewing wrote: > > >I don't like that answer. I can think of legitimate > >reasons for wanting to pre-create exceptions, e.g. if > >I'm intending to raise and catch a particular exception > >frequently and I don't want the overhead of creating > >a new instance each time. > > This seems like kind of a strange micro-optimization to have an impact on a language change discussion. Wouldn't it be better just to optimize instance creation overhead? Or modify __new__ on your particular heavily-optimized exception to have a free-list, so it can be both correct (you can still mutate exceptions) and efficient (you'll only get a new exception object if you really need it). It sounds like we should always copy the exception given to raise, and that not doing so is an optimization (albeit a commonly hit one). Not arguing for or against, just making an observation. On second thought, we could check that the refcount is 1 and avoid copying in the common case of "raise Foo()". Is reraising common enough that we need to optimize it? -- Adam Olsen, aka Rhamphoryncus From pje at telecommunity.com Wed Feb 28 17:40:53 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 28 Feb 2007 11:40:53 -0500 Subject: [Python-Dev] Class destructor In-Reply-To: Message-ID: <5.1.1.6.0.20070228113417.046cb008@sparrow.telecommunity.com> At 09:00 AM 2/28/2007 +0000, Nick Maclaren wrote: >I am gradually making progress with my binary floating-point software, >but have had to rewrite several times as I have forgotten most of the >details of how to do it! After 30 years, I can't say I am surprised. > >But I need to clean up workspace when a class (not object) is >deallocated. I can't easily use attributes, as people suggested, >because there is no anonymous storage built-in type. I could subvert >one of the existing storage types (buffer, string etc.), but that is >unclean. And I could write one, but that is excessive. > >So far, I have been unable to track down how to get something called >when a class is destroyed. The obvious attempts all didn't work, in >a variety of ways. Surely there must be a method? This could be in >either Python or C. Have you tried a PyCObject? This is pretty much what they're for: http://www.python.org/doc/1.5.2/api/cObjects.html And yes, they're still around today: http://www.python.org/doc/2.5/api/cObjects.html (with an extra PyCObject_SetVoidPtr API added in in 2.4). From nmm1 at cus.cam.ac.uk Wed Feb 28 18:24:57 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Wed, 28 Feb 2007 17:24:57 +0000 Subject: [Python-Dev] Class destructor Message-ID: "Phillip J. Eby" wrote: > > >But I need to clean up workspace when a class (not object) is > >deallocated. I can't easily use attributes, as people suggested, > >because there is no anonymous storage built-in type. I could subvert > >one of the existing storage types (buffer, string etc.), but that is > >unclean. And I could write one, but that is excessive. > > > >So far, I have been unable to track down how to get something called > >when a class is destroyed. The obvious attempts all didn't work, in > >a variety of ways. Surely there must be a method? This could be in > >either Python or C. > > Have you tried a PyCObject? This is pretty much what they're for: Oh, yes, I use them in several places, but they don't really help. Their first problem is that they take a 'void *' and not a request for space, so I have to allocate and deallocate the space manually. Now, I could add a destructor to each of them and do that, but it isn't really much prettier than subverting one of the semi-generic storage types for an improper purpose! It would be a heck of a lot cleaner to deallocate all of my space in exactly the converse way that I allocate and initialise it. It would also all me to collect and log statistics, should I so choose. This could be VERY useful for tuning! I haven't done that, yet, but might well do so. All in all, what I need is some way to get a callback when a class object is destroyed. Well, actually, any time from its last use for object work and the time that its space is reclaimed - I don't need any more precise time than that. I suppose that I could add a C object as an attribute that points to a block of memory that contains copies of all my workspace pointers, and use the object deallocator to clean up. If all else fails, I will try that, but it seems a hell of a long way round for what I would have thought was a basic requirement. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From pje at telecommunity.com Wed Feb 28 18:51:16 2007 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 28 Feb 2007 12:51:16 -0500 Subject: [Python-Dev] Class destructor In-Reply-To: Message-ID: <5.1.1.6.0.20070228124910.04984540@sparrow.telecommunity.com> At 05:24 PM 2/28/2007 +0000, Nick Maclaren wrote: >I suppose that I could add a C object as an attribute that points to >a block of memory that contains copies of all my workspace pointers, >and use the object deallocator to clean up. If all else fails, I >will try that, but it seems a hell of a long way round for what I >would have thought was a basic requirement. Well, you could use a custom metaclass with a tp_dealloc or whatever. But I just mainly meant that a PyCObject is almost as good as a weakref for certain purposes -- i.e. it's got a pointer and a callback. You could of course also use weak references, but that's a bit more awkward as well. From shane.holloway at ieee.org Wed Feb 28 18:43:02 2007 From: shane.holloway at ieee.org (Shane Holloway) Date: Wed, 28 Feb 2007 10:43:02 -0700 Subject: [Python-Dev] with_traceback In-Reply-To: References: <45E37D41.8000005@canterbury.ac.nz> <20070228054923.17852.893320900.divmod.quotient.712@ohm> Message-ID: <07DDF6B3-E329-4005-9E91-291ECFD866C3@ieee.org> On Feb 28, 2007, at 1:50 AM, Andrew Dalke wrote: > Glyph: >> This seems like kind of a strange micro-optimization to have an >> impact >> on a language change discussion. > > Just as a reminder, my concern is that people reuse exceptions > (rarely) > and that the behavior of the "with_exceptions()" method is ambiguous > when that happens. It has nothing to do with optimization. > > The two solutions of: > 1. always replace an existing __traceback__ > 2. never replace an existing __traceback__ > both seem to lead to problems. I may be strange, or in left field, or both. Since the traceback is the object that is always created, it would seem natural to me that the traceback have a reference to the exception and not the other way around. It would also seem to be a good place to attach a nested traceback which intern has it's own reference to its exception. I never really thought about it when they were just peer objects traveling up the stack. Just an idea from a different seat ;) -Shane Holloway From nmm1 at cus.cam.ac.uk Wed Feb 28 19:00:55 2007 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Wed, 28 Feb 2007 18:00:55 +0000 Subject: [Python-Dev] Class destructor Message-ID: "Phillip J. Eby" wrote: > > Well, you could use a custom metaclass with a tp_dealloc or whatever. Yes, I thought of that, but a custom metaclass to provide one callback is pretty fair overkill! > But I just mainly meant that a PyCObject is almost as good as a weakref > for certain purposes -- i.e. it's got a pointer and a callback. Ah. Yes. Thanks for suggesting it - if it is the simplest way, I may as well do it. > You could of course also use weak references, but that's a bit more > awkward as well. Yes. And they aren't a technology I have used (in Python), so I would have to find out about them. Attributes etc. I have already played with. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From jackdied at jackdied.com Wed Feb 28 22:17:57 2007 From: jackdied at jackdied.com (Jack Diederich) Date: Wed, 28 Feb 2007 16:17:57 -0500 Subject: [Python-Dev] PEP 306 changes (How to Change Python's Grammar) Message-ID: <20070228211757.GE5537@performancedrivers.com> As I found when writing the class decorator patch PEP 306 hasn't been updated since the new AST was added. Here is a suggested replacement block for the Checklist section. AST hackers feel free to make suggestions. Checklist __ Grammar/Grammar: OK, you'd probably worked this one out :) __ Parser/Python.asdl may need changes to match the Grammar. Use Parser/asdl_c.py to regenerate Include/Python-ast.h __ Python/Python-ast.c may need changes to create the AST objects involved with the Grammar change. Lib/compiler/ast.py will need matching changes to the pure-python AST objects. __ Parser/pgen needs to be rerun to regenerate Include/graminit.h and Include/graminit.c __ Python/compile.c: You will need to create or modify the compiler_* functions for your productions. __ You may need to regenerate Lib/symbol.py and/or Lib/token.py and/or Lib/keyword.py __ The parser module. Add some of your new syntax to test_parser, bang on parsermodule.c until it passes. __ The compiler package. A good test is to compile the standard library and test suite with the compiler package and then check it runs. You did add some of your new syntax to the test suite, didn't you? There's a script in Tools/compiler that does this. __ If you've gone so far as to change the token structure of Python, then the tokenizer library module will need to be changed. __ Certain changes may require tweaks to the library module pyclbr. __ Documentation must be written! From chris at atlee.ca Wed Feb 28 22:44:10 2007 From: chris at atlee.ca (Chris AtLee) Date: Wed, 28 Feb 2007 16:44:10 -0500 Subject: [Python-Dev] Windows compiler for Python 2.6+ Message-ID: <7790b6530702281344n6122637fg1de2daf52e3ec67e@mail.gmail.com> I just got bitten by the runtime library incompatibility problem on windows when I tried to load a C extension compiled with MSVC 2005 (64-bit) into Python 2.5. I realize that Python2.5 will continue to use MSVC 2003 for compatibility reasons, but I was curious if any thought had been given to the future of the 2.x series. Cheers, Chris From guido at python.org Wed Feb 28 22:44:55 2007 From: guido at python.org (Guido van Rossum) Date: Wed, 28 Feb 2007 15:44:55 -0600 Subject: [Python-Dev] Integer division operator can give float result? In-Reply-To: References: <45E4F862.20401@canterbury.ac.nz> Message-ID: Also consider this example: >>> 7.2 // 0.5 14.0 >>> On 2/28/07, Facundo Batista wrote: > Greg Ewing wrote: > > > Is this intentional? I would have expected the > > // operator to always give an integer result. > > > > Python 2.3 (#1, Aug 5 2003, 15:52:30) > > [GCC 3.1 20020420 (prerelease)] on darwin > > Type "help", "copyright", "credits" or "license" for more information. > > >>> 7.0 // 2 > > 3.0 > > >From the Language Reference: > > The / (division) and // (floor division) operators yield the quotient > of their arguments. The numeric arguments are first converted to a > common type. Plain or long integer division yields an integer of the > same type; the result is that of mathematical division with the > `floor' function applied to the result. > > So, first the arguments are converted to a common type, in this case > float. Then the division is made. Then the 'floor' function is applied. > Result: 3.0. > > So yes, it's intentional. > > Regards, > > -- > . Facundo > . > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Wed Feb 28 22:58:57 2007 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 28 Feb 2007 13:58:57 -0800 Subject: [Python-Dev] Windows compiler for Python 2.6+ In-Reply-To: <7790b6530702281344n6122637fg1de2daf52e3ec67e@mail.gmail.com> References: <7790b6530702281344n6122637fg1de2daf52e3ec67e@mail.gmail.com> Message-ID: <20070228135629.AE80.JCARLSON@uci.edu> "Chris AtLee" wrote: > I just got bitten by the runtime library incompatibility problem on > windows when I tried to load a C extension compiled with MSVC 2005 > (64-bit) into Python 2.5. I would guess it is more an issue of 32bit + 64bit dynamic linking having issues, but I could certainly be wrong. > I realize that Python2.5 will continue to use MSVC 2003 for > compatibility reasons, but I was curious if any thought had been given > to the future of the 2.x series. IIUC, there exists a project file in PCBUILD8 for compiling with MSVC 2005. You should be able to recompile Python 2.5 with that compiler, though you may need to change some things (I've never tried myself). - Josiah From chris at atlee.ca Wed Feb 28 23:24:29 2007 From: chris at atlee.ca (Chris AtLee) Date: Wed, 28 Feb 2007 17:24:29 -0500 Subject: [Python-Dev] Windows compiler for Python 2.6+ In-Reply-To: <20070228135629.AE80.JCARLSON@uci.edu> References: <7790b6530702281344n6122637fg1de2daf52e3ec67e@mail.gmail.com> <20070228135629.AE80.JCARLSON@uci.edu> Message-ID: <7790b6530702281424j5bd269u3a1e88dd844dbf1@mail.gmail.com> On 2/28/07, Josiah Carlson wrote: > "Chris AtLee" wrote: > > I just got bitten by the runtime library incompatibility problem on > > windows when I tried to load a C extension compiled with MSVC 2005 > > (64-bit) into Python 2.5. > > I would guess it is more an issue of 32bit + 64bit dynamic linking > having issues, but I could certainly be wrong. I don't think so, this was the 64bit version of Python 2.5. When I recompiled with the 2003 compiler it worked fine. > > I realize that Python2.5 will continue to use MSVC 2003 for > > compatibility reasons, but I was curious if any thought had been given > > to the future of the 2.x series. > > IIUC, there exists a project file in PCBUILD8 for compiling with MSVC > 2005. You should be able to recompile Python 2.5 with that compiler, > though you may need to change some things (I've never tried myself). That is kind of a last-resort for me...I'd like for my code to work with all the other python extensions out there, which is why I switched to the 2003 compiler for now. From martin at v.loewis.de Wed Feb 28 23:31:13 2007 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 28 Feb 2007 23:31:13 +0100 Subject: [Python-Dev] Windows compiler for Python 2.6+ In-Reply-To: <7790b6530702281344n6122637fg1de2daf52e3ec67e@mail.gmail.com> References: <7790b6530702281344n6122637fg1de2daf52e3ec67e@mail.gmail.com> Message-ID: <45E602B1.2000003@v.loewis.de> Chris AtLee schrieb: > I just got bitten by the runtime library incompatibility problem on > windows when I tried to load a C extension compiled with MSVC 2005 > (64-bit) into Python 2.5. > > I realize that Python2.5 will continue to use MSVC 2003 for > compatibility reasons, but I was curious if any thought had been given > to the future of the 2.x series. You are mistaken. The 64-bit versions (AMD64 and Itanium) are not compiled using MSVC 2003 - this product does not even include compilers for this platform. Instead, they are compiled with the SDK compilers. Regards, Martin From swarren at wwwdotorg.org Wed Feb 28 08:20:57 2007 From: swarren at wwwdotorg.org (Stephen Warren) Date: Wed, 28 Feb 2007 00:20:57 -0700 Subject: [Python-Dev] GeneratorExit inheriting from Exception References: ca471dc20603250743sf4d8fbp940c07fd2b2adb40@mail.gmail.com Message-ID: <1172647259.10236.TMDA@tmda.severn.wwwdotorg.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Re: the discussion in: http://mail.python.org/pipermail/python-dev/2006-March/062823.html Just as an FYI, the tlslite package (http://trevp.net/tlslite/) got broken in Python 2.5 and needed the exact fix quoted in the URL above. It was an easy fix, but the argument isn't hypothetical any more! A little late to bother changing anything, though. -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFF5S1Zhk3bo0lNTrURAjIuAKC1ASOfx0L2+hf+3EKa2hktZYRjEgCeNRAn n395GwS11yM2AMSK67b5oNA= =+iBp -----END PGP SIGNATURE-----