From python-dev at zesty.ca Sat Jul 1 00:00:26 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri, 30 Jun 2006 17:00:26 -0500 (CDT) Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <001201c69c65$b3869750$6402a8c0@arkdesktop> References: <001201c69c65$b3869750$6402a8c0@arkdesktop> Message-ID: On Fri, 30 Jun 2006, Andrew Koenig wrote: > I saw messages out of sequence and did not realize that this would be a > change in behavior from 2.4. Sigh. Yes, this is not a good time to change it. > I hope Py3000 has lexical scoping a la Scheme... Me too -- that would be really nice. -- ?!ng From python-dev at zesty.ca Sat Jul 1 00:05:25 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri, 30 Jun 2006 17:05:25 -0500 (CDT) Subject: [Python-Dev] For sandboxing: alternative to crippling file() In-Reply-To: References: <20060630175205.GA17748@code0.codespeak.net> Message-ID: On Fri, 30 Jun 2006, Brett Cannon wrote: > On 6/30/06, Armin Rigo wrote: > > >>> object.__subclasses__() > > [..., ] > > > > Maybe this one won't work if __subclasses__ is forbidden, but in general > > I think there *will* be a way to find this object. > > Yeah, that's been my (what I thought was paranoid) feeling. Glad I am not > the only one who thinks that hiding file() is near impossible. If you want to do this right, it should be about *making* hiding possible. If you can't hide things, it will be hard to get very far. I realize that may be difficult for Python 2.x, but hiding is pretty essential for security. It would be really good to keep this in mind for the design of Python 3k. (It doesn't mean we can't have introspection, just that we need to agree on some discipline for how to do it.) -- ?!ng From jimjjewett at gmail.com Sat Jul 1 00:13:37 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 30 Jun 2006 18:13:37 -0400 Subject: [Python-Dev] traceback regression Message-ID: python.org/sf/1515343 fixes python.org/sf/1515163, which is a new-in-2.5 regression. On the one hand, the regression only affects >>> raise "string1", "string2" which is both obscure and deprecated. On the other hand, it is a regression, and it is something I bumped into while working with unittest. [Note that I probably won't be checking email again for a week, so I'm afraid I won't be very responsive to comments.] -jJ From guido at python.org Sat Jul 1 00:24:04 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Jun 2006 15:24:04 -0700 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: References: <001201c69c65$b3869750$6402a8c0@arkdesktop> Message-ID: On 6/30/06, Ka-Ping Yee wrote: > On Fri, 30 Jun 2006, Andrew Koenig wrote: > > I saw messages out of sequence and did not realize that this would be a > > change in behavior from 2.4. Sigh. > > Yes, this is not a good time to change it. > > > I hope Py3000 has lexical scoping a la Scheme... > > Me too -- that would be really nice. That's not a very constructive proposal (especially since I don't know Scheme). Perhaps you could elaborate on what needs to change? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Sat Jul 1 00:33:20 2006 From: brett at python.org (Brett Cannon) Date: Fri, 30 Jun 2006 15:33:20 -0700 Subject: [Python-Dev] For sandboxing: alternative to crippling file() In-Reply-To: References: <20060630175205.GA17748@code0.codespeak.net> Message-ID: On 6/30/06, Ka-Ping Yee wrote: > > On Fri, 30 Jun 2006, Brett Cannon wrote: > > On 6/30/06, Armin Rigo wrote: > > > >>> object.__subclasses__() > > > [..., ] > > > > > > Maybe this one won't work if __subclasses__ is forbidden, but in > general > > > I think there *will* be a way to find this object. > > > > Yeah, that's been my (what I thought was paranoid) feeling. Glad I am > not > > the only one who thinks that hiding file() is near impossible. > > If you want to do this right, it should be about *making* hiding > possible. If you can't hide things, it will be hard to get very far. Well, this is only file() we are worrying about leaking out. Stuff from import are the worry here. I realize that may be difficult for Python 2.x, but hiding is pretty > essential for security. It would be really good to keep this in mind > for the design of Python 3k. (It doesn't mean we can't have > introspection, > just that we need to agree on some discipline for how to do it.) That's fine; I personally have no issue with tweaking the security model for Py3K. But I am worrying about 2.x here. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060630/172e6d14/attachment-0001.htm From python-dev at zesty.ca Sat Jul 1 01:03:14 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri, 30 Jun 2006 18:03:14 -0500 (CDT) Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: References: <001201c69c65$b3869750$6402a8c0@arkdesktop> Message-ID: On Fri, 30 Jun 2006, Guido van Rossum wrote: > On 6/30/06, Ka-Ping Yee wrote: > > On Fri, 30 Jun 2006, Andrew Koenig wrote: > > > I hope Py3000 has lexical scoping a la Scheme... > > > > Me too -- that would be really nice. > > That's not a very constructive proposal (especially since I don't know > Scheme). Perhaps you could elaborate on what needs to change? Sorry! I should have been more clear. Right now, Python supports the existence of nested scopes: a = 3 def f(): a = 4 def g(): a = 5 print a # prints 5 g() print a # prints 4 f() print a # prints 3 The above example shows that there are three distinct scopes, and that each one has a distinct binding named 'a' -- assigning to one doesn't affect the others. a = 3 def f(): b = 4 def g(): c = 5 print a, b, c # i can see all three g() f() The above example shows that all of the scopes can be *read*. But in today's Python, not all of the scopes can be *written*. a = 3 def f(): b = 4 def g(): c = 5 a, b, c = 0, 1, 2 # changes local c, not outer a and b g() f() The code in g() can affect its own local, 'c', and it can affect the global variable 'a' if it declares 'global a', but no matter what you write in g(), it cannot assign to 'b' (or to any other intermediate scope). This is a strange limitation and it would be nice to remove it. The asymmetry comes from Python having one foot in the new paradigm of nested lexical scopes and one foot still in the older paradigm of only two scopes, local and global. Most other languages that support lexical scoping (including Scheme, JavaScript, Ruby, Perl, E, Java, Smalltalk) provide a uniform way to read and write to scopes at all levels. This is done by letting programmers specify the scope in which they want a variable bound (usually with a keyword like "var" in JavaScript, "my" in Perl, or "define" in E). So here are some thoughts on how Python might be adjusted to support this. I'm not saying these would be the only ways, but at least they're some ideas to start with. In JavaScript, the "var" keyword is required whenever you want to declare a local variable. Anything without "var" is assumed to be a global name. The cleanest and most consistent solution that comes to mind would be to adopt exactly this for Python. Without "var": a = 3 # global def f(): b = 4 # global def g(): c = 5 # global a, b, c = 0, 1, 2 # changes all three globals g() f() print a, b, c # prints 0, 1, 2 With "var": var a = 3 # global def f(): var b = 4 # local to f def g(): var c = 5 # local to g a, b, c = 0, 1, 2 # changes outer a, outer b, and c print c # prints 2 g() print b # prints 1 f() print a # prints 0 print b # no such name print c # no such name But that is a big change. Perhaps it would be too unpythonic to suddenly require declarations for all local variables. So an alternative would be to retain the default assumption that undeclared variables are local. Here's what we'd get: Without "var": a = 3 def f(): b = 4 def g(): c = 5 a, b, c = 0, 1, 2 # changes local c, not outer a and b g() f() With "var": var a = 3 def f(): var b = 4 def g(): var c = 5 a, b, c = 0, 1, 2 # changes outer a, outer b, and c g() f() Now i think this is a little bit weird, because the statement "var b = 4" in an outer scope changes the meaning of "b" in an inner scope. But it does have the virtue of retaining behaviour compatible with today's Python, while offering a way to get proper lexical scopes for those who want to use them. Thoughts? Other ideas? -- ?!ng From ark at acm.org Sat Jul 1 01:04:53 2006 From: ark at acm.org (Andrew Koenig) Date: Fri, 30 Jun 2006 19:04:53 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: Message-ID: <000d01c69c99$9c023ec0$6402a8c0@arkdesktop> > That's not a very constructive proposal (especially since I don't know > Scheme). Perhaps you could elaborate on what needs to change? The fundamental principle is that the binding of every name is determined during compilation, not during execution. This property does not quite apply to Python at present. For example: x = 42 def f(): y = x x = 123 return y f() This example fails with "local variable 'x' referenced before assignment" because the compiler sees that f contains an assignment to x, so it makes x a local variable, and then when you try to assign x to y during execution, it fails. This behavior is consistent with the notion of lexical scoping. However, if I write def g(): return x x = 42 g() the result is 42. With lexical scoping, I believe it should be undefined. The reason is that when the compiler encounters the definition of g, variable x is not yet bound, and there is nothing in the body of g that would bind it. Therefore, g requires a binding to exist at the time it is compiled; because no such binding exists, this example would be an error (at compile time) under lexical scoping. From scott+python-dev at scottdial.com Sat Jul 1 01:13:58 2006 From: scott+python-dev at scottdial.com (Scott Dial) Date: Fri, 30 Jun 2006 19:13:58 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: References: <001201c69c65$b3869750$6402a8c0@arkdesktop> Message-ID: <44A5B036.7030905@scottdial.com> Guido van Rossum wrote: > On 6/30/06, Ka-Ping Yee wrote: >> On Fri, 30 Jun 2006, Andrew Koenig wrote: >>> I hope Py3000 has lexical scoping a la Scheme... >> Me too -- that would be really nice. > > That's not a very constructive proposal (especially since I don't know > Scheme). Perhaps you could elaborate on what needs to change? I believe the essence of their request for lexical scope boils down to allowing rebinding. Such code like the following is legal in Scheme: def f(x): def incr(): x = x + 1 return x def decr(): x = x - 1 return x return (incr, decr) (incr, decr) = f(1) print incr() # 2 print incr() # 3 print decr() # 2 print decr() # 1 -- FWIW, the Scheme equivalent would be something like: (define f (lambda (x) (list (lambda () (set! x (+ x 1)) x) (lambda () (set! x (- x 1)) x)))) (let ([fs (f 1)]) (let ([incr (car fs)] [decr (cadr fs)]) (display (incr)) (newline) ; 2 (display (incr)) (newline) ; 3 (display (decr)) (newline) ; 2 (display (decr)) (newline))) ; 1 As a more personal aside, I can't imagine where I would use this in any python program I have ever wrote. I actually never noticed that rebinding wasn't allowed until recently. -- Scott Dial scott at scottdial.com scodial at indiana.edu From tim.peters at gmail.com Sat Jul 1 01:23:56 2006 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 30 Jun 2006 19:23:56 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: References: <001201c69c65$b3869750$6402a8c0@arkdesktop> Message-ID: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com> [Andrew Koenig] >>> I saw messages out of sequence and did not realize that this would be a >>> change in behavior from 2.4. Sigh. [Ka-Ping Yee] >> Yes, this is not a good time to change it. >>> I hope Py3000 has lexical scoping a la Scheme... >> Me too -- that would be really nice. [Guido] > That's not a very constructive proposal (especially since I don't know > Scheme). Perhaps you could elaborate on what needs to change? It's effectively the opposite of Python <0.1 wink>: a name is local to a scope in Scheme if and only if a declaration says it is. For example, the "let" family of forms is often used for this purpose, and (let ((x 2) (y 3)) # declares x and y as local to this `let`, and gives initial values (let ((x 7) (z (+ x y))) # x comes from outer `let` so is 2, and z is 2+3=5 (* z x))) # x comes from inner `let`, so this is 5*7=35 If you use `let*` instead of `let` in the inner one, z picks up the inner x=7, so that z is 7+3=10, and the result is 7*10 = 70 instead. The bindings in a `let` "happen" in an undefined order. In `let*`, a binding is visible "to its right" within the `let*`. Then there's `letrec`, which allows establishing mutually recursive bindings. While the `let` family is entirely about declaration, there are lots of other forms that mix in some declaration as part of their purpose (for example, the names in a lambda's argument list are local to the lambda's body), but they're all explicit in Scheme. I read "a la Scheme" here as "actually nothing like Scheme, except I want a non-tricky way to rebind a name from an enclosing scope within an enclosed scope". In Scheme, the scope a name x belongs to is found by searching enclosing scopes until you hit the first with an explicit "x belongs to me" declaration (OK, there's a hokey fallback to "top level" definitions too). Searching "textually up" always suffices (there's no form of delayed declaration -- a name must be declared before use). Scheme's assignment of assignment: (set! variable expression) has nothing to do with establishing the scope of `variable`. From ark at acm.org Sat Jul 1 01:27:06 2006 From: ark at acm.org (Andrew Koenig) Date: Fri, 30 Jun 2006 19:27:06 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com> Message-ID: <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> > That sounds like a bug, not a feature. It's frequently useful to have > forward references in function bodies to names that are not yet globally > bound, e.g. for classes, or mutually-recursive functions. The trouble is that you don't necessarily know in what scope they will be defined, so I think that forcing you to be explicit about it is useful. Can you show me an example of where you think it isn't? Incidentally, I think that lexical scoping would also deal with the problem that people often encounter in which they have to write things like "lambda x=x:" where one would think "lambda x:" would suffice. From ark at acm.org Sat Jul 1 01:29:45 2006 From: ark at acm.org (Andrew Koenig) Date: Fri, 30 Jun 2006 19:29:45 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com> Message-ID: <001401c69c9d$12d53900$6402a8c0@arkdesktop> > I read "a la Scheme" here as "actually nothing like Scheme, except I > want a non-tricky way to rebind a name from an enclosing scope within > an enclosed scope". Almost. What I really want is for it to be possible to determine the binding of every name by inspecting the source text of the program. Right now, it is often possible to do so, but sometimes it isn't. From pje at telecommunity.com Sat Jul 1 01:09:41 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 30 Jun 2006 19:09:41 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <000d01c69c99$9c023ec0$6402a8c0@arkdesktop> References: Message-ID: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com> At 07:04 PM 6/30/2006 -0400, Andrew Koenig wrote: >However, if I write > > def g(): > return x > x = 42 > g() > >the result is 42. With lexical scoping, I believe it should be undefined. > >The reason is that when the compiler encounters the definition of g, >variable x is not yet bound, and there is nothing in the body of g that >would bind it. Therefore, g requires a binding to exist at the time it is >compiled; because no such binding exists, this example would be an error (at >compile time) under lexical scoping. That sounds like a bug, not a feature. It's frequently useful to have forward references in function bodies to names that are not yet globally bound, e.g. for classes, or mutually-recursive functions. From tim.peters at gmail.com Sat Jul 1 01:39:03 2006 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 30 Jun 2006 19:39:03 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com> <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> Message-ID: <1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com> [Andrew Koenig] > ... > Incidentally, I think that lexical scoping would also deal with the problem > that people often encounter in which they have to write things like "lambda > x=x:" where one would think "lambda x:" would suffice. They _shouldn't_ encounter that at all anymore. For example, >>> def f(x): ... return lambda: x+1 >>> f(3)() 4 works fine in modern Pythons. Earlier Python's had a no-exceptions 3-scope implementation (local, global, builtin), and in those the "x" in the lambda body was "not local" (was either global or builtin, although the compiler couldn't tell which of those two it was). In _those_ Pythons people had to write "lambda x=x: x+1" instead, to suck the binding of the outer x into the lambda body, but if people are still doing that they're confused. Modern Pythons do have lexical scoping + global + builtin, although there's no way to spell "rebind a name local to an outer scope from within an inner scope". From guido at python.org Sat Jul 1 01:44:02 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Jun 2006 16:44:02 -0700 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <001401c69c9d$12d53900$6402a8c0@arkdesktop> References: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com> <001401c69c9d$12d53900$6402a8c0@arkdesktop> Message-ID: On 6/30/06, Andrew Koenig wrote: > > I read "a la Scheme" here as "actually nothing like Scheme, except I > > want a non-tricky way to rebind a name from an enclosing scope within > > an enclosed scope". > > Almost. What I really want is for it to be possible to determine the > binding of every name by inspecting the source text of the program. Right > now, it is often possible to do so, but sometimes it isn't. Then your example def f(): return x x = 42 print f() is entirely well-defined -- x is a global and the compiler in fact generates code that benefits from knowing that it's not a local. Python knows which locals there are; also which locals there are in surrounding function scopes. It *could* also know which globals and builtins there are, except the language currently allows dynamic rebinding of module-level variables so that they replace builtins. E.g. def f(): return len([]) print f() # prints 0 def len(x): return "booh" print f() # prints "booh" del len print f() # prints 0 again Worse, instead if explicitly overriding len in the module, it could have been an assignment to __main__.len in some other module. We've been thinking on how to deal with this for years, since nobody really likes it in all its freedom. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Sat Jul 1 01:56:58 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 30 Jun 2006 16:56:58 -0700 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: References: Message-ID: <20060630163625.10D1.JCARLSON@uci.edu> Ka-Ping Yee wrote: [snip lexical scoping option] > Now i think this is a little bit weird, because the statement > "var b = 4" in an outer scope changes the meaning of "b" in an > inner scope. But it does have the virtue of retaining behaviour > compatible with today's Python, while offering a way to get proper > lexical scopes for those who want to use them. > > Thoughts? Other ideas? Using a keyword in an outer scope to state that a variable could be used in a nested scope is counter to the current method for accessing a parent scope with 'global'. Using 'var' as the equivalent of 'global', only for nested scopes, would be a more reasonable approach. However, I'm -1 on the feature now, for the same reasons I've been -1 on the feature for the last 2 times it has come up. In many of the cases where lexically nested scopes have been used to solve problems in Python, and programmers have run into a 'limitation' where not being able to modify a value in a parent scope has hindered them, the problem could have been better solved with another method that was more readable, more extensible, etc. What I asked before, and what I'd like to ask again, is if there are any _nontrivial uses_ of lexically nested scopes which are made cumbersome by our inability to write to parent scopes. If there aren't, then I'm going to again have to argue against new syntax, keywords, and their use. If there are, then we'll see how compelling such uses are. - Josiah From python-dev at zesty.ca Sat Jul 1 01:55:12 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri, 30 Jun 2006 18:55:12 -0500 (CDT) Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <000d01c69c99$9c023ec0$6402a8c0@arkdesktop> References: <000d01c69c99$9c023ec0$6402a8c0@arkdesktop> Message-ID: On Fri, 30 Jun 2006, Andrew Koenig wrote: > The fundamental principle is that the binding of every name is determined > during compilation, not during execution. This property does not quite > apply to Python at present. I think this property does apply. In your example: > def g(): > return x > x = 42 > g() > > the result is 42. It is already known at compile time that the "return x" in g() refers to an 'x' in the outer scope. 'x' cannot be a local variable to g() because there are no statements in g() that bind 'x'. Regardless of whether the binding itself exists yet, you (the reader) and the compiler can know which scope to look in for the binding at runtime. Have i understood your desired property correctly? -- ?!ng From tim.peters at gmail.com Sat Jul 1 01:55:38 2006 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 30 Jun 2006 19:55:38 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <001401c69c9d$12d53900$6402a8c0@arkdesktop> References: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com> <001401c69c9d$12d53900$6402a8c0@arkdesktop> Message-ID: <1f7befae0606301655i2425c0a1lf8d956cad0f028b7@mail.gmail.com> [Andrew Koenig] > Almost. What I really want is for it to be possible to determine the > binding of every name by inspecting the source text of the program. Right > now, it is often possible to do so, but sometimes it isn't. Local names are always determined at compile-time in Python. What you can't always determine is whether a non-local (to any enclosing scope) name will end up getting resolved from the module globals or from __builtin__. The runtime error in: def f(): y = x x = 1 f() doesn't occur because Python doesn't know "x" is local to "f" at compile-time (it does know that), it's because Python's compiler doesn't do any flow analysis to detect potential use-before-definition. Instead the runtime initalizes locals to a special "not bound yet" value that the LOAD_FAST (really "load local") opcode special-cases. Note that this is quite unlike Scheme, in which declaration must appear before use (ignoring fancy letrec cases), and declaration must also supply an initial binding (Scheme has no "unbound local" problem because there's no way to create an uninitialized local). From rasky at develer.com Sat Jul 1 02:01:00 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 1 Jul 2006 02:01:00 +0200 Subject: [Python-Dev] 2.5 and beyond References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com><001201c69c9c$b6531fd0$6402a8c0@arkdesktop> <1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com> Message-ID: <020c01c69ca1$754c7310$d1b12997@bagio> Tim Peters wrote: >> ... >> Incidentally, I think that lexical scoping would also deal with the >> problem >> that people often encounter in which they have to write things like >> "lambda >> x=x:" where one would think "lambda x:" would suffice. > > They _shouldn't_ encounter that at all anymore. For example, > >>>> def f(x): > ... return lambda: x+1 >>>> f(3)() > 4 > > works fine in modern Pythons. Yes but: >>> a = [] >>> for i in range(10): ... a.append(lambda: i) ... >>> print [x() for x in a] [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] This subtle semantic of lambda is quite confusing, and still forces people to use the "i=i" trick. Giovanni Bajo From tim.peters at gmail.com Sat Jul 1 02:15:51 2006 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 30 Jun 2006 20:15:51 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <020c01c69ca1$754c7310$d1b12997@bagio> References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com> <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> <1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com> <020c01c69ca1$754c7310$d1b12997@bagio> Message-ID: <1f7befae0606301715m6fe7044eue4e5c8d74a7c8a22@mail.gmail.com> [Giovanni Bajo] > Yes but: > > >>> a = [] > >>> for i in range(10): > ... a.append(lambda: i) > ... > >>> print [x() for x in a] > [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] > > This subtle semantic of lambda is quite confusing, and still forces people to > use the "i=i" trick. So stay away from excruciating abuses of lexical scoping you don't understand ;-) What do you _expect_ `i` to refer to? "Oh, it should guess that I didn't really mean to defer evaluation of the lambda body at all, but instead evaluate the lambda body at the time I define the lambda and then synthesize some other function that captures the specific outer bindings in effect at lambda-definition time" doesn't really cut it. Try spelling what you think you want here in Scheme. Before it works, you'll probably end up with some equally "atrocious" (let ((i i)) ...) gimmick to force capturing each binding for `i` as it flies by. Else Scheme will also use the outer binding for `i` in effect at the time the lambdas are _executed_. This isn't typical use for lambda, and I don't think it's what Andrew had in mind. From bjourne at gmail.com Sat Jul 1 03:11:59 2006 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Sat, 1 Jul 2006 03:11:59 +0200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: References: <001201c69c65$b3869750$6402a8c0@arkdesktop> Message-ID: <740c3aec0606301811i6b323bei3abcfb05cde5b747@mail.gmail.com> > With "var": > > var a = 3 > def f(): > var b = 4 > def g(): > var c = 5 > a, b, c = 0, 1, 2 # changes outer a, outer b, and c > g() > f() > > Now i think this is a little bit weird, because the statement > "var b = 4" in an outer scope changes the meaning of "b" in an > inner scope. But it does have the virtue of retaining behaviour > compatible with today's Python, while offering a way to get proper > lexical scopes for those who want to use them. > > Thoughts? Other ideas? Maybe an object, like self, for referring to enclosing scopes? a = 3 def f(): b = 4 def g(): c = 5 outer.outer.a, outer.b, c = 0, 1, 2 # changes outer a, outer b, and c g() f() Chaining the keyword looks a little weird, but it is not often that you have to refer to variables in the enclosing scope of the enclosing scope. I have often wanted something similar to that for global variables, instead of the global declaration: cache = None def init(): if not global.cache: global.cache = init_cache() -- mvh Bj?rn From janssen at parc.com Sat Jul 1 03:31:12 2006 From: janssen at parc.com (Bill Janssen) Date: Fri, 30 Jun 2006 18:31:12 PDT Subject: [Python-Dev] 2.5 and beyond In-Reply-To: Your message of "Fri, 30 Jun 2006 17:01:00 PDT." <020c01c69ca1$754c7310$d1b12997@bagio> Message-ID: <06Jun30.183119pdt."58641"@synergy1.parc.xerox.com> > >>> a = [] > >>> for i in range(10): > ... a.append(lambda: i) > ... > >>> print [x() for x in a] > [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] Isn't this exactly what you'd expect? Maybe I've been writing Python for too long... :-). Bill From rasky at develer.com Sat Jul 1 03:43:01 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 1 Jul 2006 03:43:01 +0200 Subject: [Python-Dev] 2.5 and beyond References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com> <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> <1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com> <020c01c69ca1$754c7310$d1b12997@bagio> <1f7befae0606301715m6fe7044eue4e5c8d74a7c8a22@mail.gmail.com> Message-ID: <027f01c69caf$b1c20450$d1b12997@bagio> [Giovanni Bajo] > Yes but: > >>>> a = [] >>>> for i in range(10): > ... a.append(lambda: i) > ... >>>> print [x() for x in a] > [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] > > This subtle semantic of lambda is quite confusing, and still forces people to > use the "i=i" trick. [Tim Peters] > So stay away from excruciating abuses of lexical scoping you don't > understand What do you expect `i` to refer to? "Oh, it should > guess that I didn't really mean to defer evaluation of the lambda body > at all, but instead evaluate the lambda body at the time I define the > lambda and then synthesize some other function that captures the > specific outer bindings in effect at lambda-definition time" doesn't > really cut it. I think I understand what happens, I just don't know whether this can be "fixed" or not. Unless you are saying that the above behaviour is not only a complex side-effect the way things are, but the way things should be. Do you agree that it would be ideal if the above code generated range(10) instead of [9]*10, or you believe that the current behaviour is more sound (and if so, why)? As for actual implementing this change of semantic, the fact that `i` is a local variable in the outer scope (assuming it's all within a function), doesn't make it possible for Python to early-bound it, by realizing that, since `i` is not an argument of the lambda, and it's a local of the outer scope? At worse, couldn't Python do the "i=i" trick by itself when it sees that `i` is a local in the outer scope? Right now I can't think off-hand of a case in which this would break things. [Tim Peters] > This isn't typical use for lambda, Yes, maybe it's not the most used idiom and Andrew wasn't referring to this, but it happens quite often to me (where 'often' means 'many times' among my rare usages of lambda). For instance, in GUI code, it's common to do things like: for b in self.buttons: self.setEventCallback(b, "clicked", lambda: self.label.setText("I pressed button %r" % b)) ... which of course won't work, as written above. Giovanni Bajo -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 135 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/ebb3a2b4/attachment.gif From guido at python.org Sat Jul 1 04:55:03 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Jun 2006 19:55:03 -0700 Subject: [Python-Dev] ImportWarning flood In-Reply-To: <44A5D5DA.2060601@hathawaymix.org> References: <20060630194331.29014.836424634.divmod.quotient.16780@ohm> <44A5D5DA.2060601@hathawaymix.org> Message-ID: It's up to the release manager now to decide whether the pitchforks at Google or the pitchforks in the larger Python community are sharper. ;-) --Guido (ducks) On 6/30/06, Shane Hathaway wrote: > Guido van Rossum wrote: > > On 6/30/06, Jean-Paul Calderone wrote: > >> How about if someone grovels through import.c and figures out how to make > >> the warning information only show up if the import actually fails? > > > > That would work I think. But it's not easy. > > I just posted a patch intended to solve this. With my patch, > find_module() collects the warnings in a list and only generates > ImportWarning if no package is found. It works, but I have not done > extensive testing. > > I also discovered and tried to resolve what appear to be memory leaks > involving the "copy" variable. If the ImportWarning patch doesn't fly, > I'll be happy to post a different patch that only fixes the leaks. > > The patch is #1515361: > > https://sourceforge.net/tracker/index.php?func=detail&aid=1515361&group_id=5470&atid=305470 > > Shane > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Sat Jul 1 05:32:01 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 30 Jun 2006 22:32:01 -0500 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 Message-ID: <17573.60593.759293.549583@montanaro.dyndns.org> Just upgraded my Mac to OSX 10.4.7 yesterday. svn up'd Python trunk, then "make clean ; configure ; make" and I see that building the zlib module fails: gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I/Users/skip/src/python-svn/trunk/./Include -I/Users/skip/src/python-svn/trunk/./Mac/Include -I/Users/skip/local/include -I../Include -I. -I/usr/local/include -I/Users/skip/src/python-svn/trunk/Include -I/Users/skip/src/python-svn/trunk/build -c /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c -o build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c: In function 'PyZlib_uncopy': /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c:724: warning: implicit declaration of function 'inflateCopy' gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o -L/Users/skip/local/lib -L/usr/local/lib -lz -o build/lib.macosx-10.3-ppc-2.5/zlib.so -Wl,-search_paths_first *** WARNING: renaming "zlib" since importing it failed: dlopen(build/lib.macosx-10.3-ppc-2.5/zlib.so, 2): Symbol not found: _inflateCopy Referenced from: build/lib.macosx-10.3-ppc-2.5/zlib.so Expected in: dynamic lookup Anybody else seen this? I checked the buildbot trunk osx 10.4. It seemed to have no trouble. And what's with the "10.3" bit in the directory names? Skip From greg.ewing at canterbury.ac.nz Sat Jul 1 05:45:02 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Jul 2006 15:45:02 +1200 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: <043001c69b45$35742290$100a0a0a@enfoldsystems.local> References: <043001c69b45$35742290$100a0a0a@enfoldsystems.local> Message-ID: <44A5EFBE.9050300@canterbury.ac.nz> Mark Hammond wrote: > that helps "mozilla the platform" more than it helps "firebox the browser" ^^^^^^^ Firebox - the sandfoxed web browser! -- Greg From tjreedy at udel.edu Sat Jul 1 06:27:16 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 1 Jul 2006 00:27:16 -0400 Subject: [Python-Dev] 2.5 and beyond References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com><001201c69c9c$b6531fd0$6402a8c0@arkdesktop><1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com> <020c01c69ca1$754c7310$d1b12997@bagio> Message-ID: "Giovanni Bajo" wrote in message news:020c01c69ca1$754c7310$d1b12997 at bagio... > Yes but: > >>>> a = [] >>>> for i in range(10): > ... a.append(lambda: i) > ... >>>> print [x() for x in a] > [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] > > This subtle semantic of lambda is quite confusing, and still forces > people to > use the "i=i" trick. The 'subtle sematic' had nothing to do with lambda but with Python functions. The above is exactly equivalent (except the different .funcname) to a = [] for i in range(10): def f(): return i a.append(f) del f That should be equally confusing (or not), and equally requires the 'i=i' trick (or not). As is, either function definitiion is a constant and the loop makes useless duplicates. Either form would have the same effect is hoisted out of the loop. Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sat Jul 1 06:59:19 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Jul 2006 16:59:19 +1200 Subject: [Python-Dev] For sandboxing: alternative to crippling file() In-Reply-To: References: Message-ID: <44A60127.3030706@canterbury.ac.nz> Brett Cannon wrote: > 1) Is removing 'file' from the builtins dict in PyInterpreterState (and > maybe some other things) going to be safe enough to sufficiently hide > 'file' confidently (short of someone being stupid in their C extension > module and exposing 'file' directly)? > > 2) Changing open() to return C-implemented delegate objects for files > (and thus won't type check, but this is Python so I am not worried about > that too much) and delegate socket objects for IP and URL addresses. My suggestion is to change things so that the constructor of the file type doesn't open files (at least in restricted mode). Then it wouldn't matter if untrusted code had real file objects, as they couldn't use them to get access to any other files. -- Greg From tjreedy at udel.edu Sat Jul 1 07:00:40 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 1 Jul 2006 01:00:40 -0400 Subject: [Python-Dev] 2.5 and beyond References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com><001201c69c9c$b6531fd0$6402a8c0@arkdesktop><1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com><020c01c69ca1$754c7310$d1b12997@bagio><1f7befae0606301715m6fe7044eue4e5c8d74a7c8a22@mail.gmail.com> <027f01c69caf$b1c20450$d1b12997@bagio> Message-ID: "Giovanni Bajo" wrote in message news:027f01c69caf$b1c20450$d1b12997 at bagio... > [Giovanni Bajo] >> Yes but: >> >>>>> a = [] >>>>> for i in range(10): >> ... a.append(lambda: i) >> ... >>>>> print [x() for x in a] >> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] >. Do you agree that it would be ideal if the above code > generated range(10) instead of [9]*10, No. You are trying to reify an optical illusion resulting from putting a constant function definition inside a loop. Making the meaning of 'def f(): return i' depend on the definition-time context by partially and variably evaluating the body would make code much harder to read and understand. Consider: if a: i=666 def f(): return i > At > worse, couldn't Python do the "i=i" trick by itself when it sees that `i` > is a > local in the outer scope? Right now I can't think off-hand of a case in > which > this would break things. It would make code more fragile. for i in range(666): print name[i] ... ... def total(num): return cost[item]*num Now someone decides first loop should have more expressive loop var name and changes the first line to for item in range(666): print name[item] and the meaning of total is completely changed. Adding such long-range coupling between language statements strikes me as a poor idea. Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sat Jul 1 07:39:05 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Jul 2006 17:39:05 +1200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: References: <001201c69c65$b3869750$6402a8c0@arkdesktop> Message-ID: <44A60A79.8090802@canterbury.ac.nz> Ka-Ping Yee wrote: > while offering a way to get proper > lexical scopes for those who want to use them. I don't disagree with anything you said, but I think it would be a good idea to avoid using phrases like "proper lexical scopes", which is likely to set people off on a tangent. The issue isn't lexicality, it's writeability. -- Greg From nnorwitz at gmail.com Sat Jul 1 07:53:26 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 30 Jun 2006 22:53:26 -0700 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <17573.60593.759293.549583@montanaro.dyndns.org> References: <17573.60593.759293.549583@montanaro.dyndns.org> Message-ID: Maybe do a make distclean. There was a problem where old versions of zlib (those without inflateCopy) weren't supported. They are now, but it's a configure check. That coupled with the upgrade and the 10.3 in the pathname, seems like it's just something didn't get cleaned up properly. You could always rm -rf build/ n -- On 6/30/06, skip at pobox.com wrote: > Just upgraded my Mac to OSX 10.4.7 yesterday. svn up'd Python trunk, then > "make clean ; configure ; make" and I see that building the zlib module > fails: > > gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I/Users/skip/src/python-svn/trunk/./Include -I/Users/skip/src/python-svn/trunk/./Mac/Include -I/Users/skip/local/include -I../Include -I. -I/usr/local/include -I/Users/skip/src/python-svn/trunk/Include -I/Users/skip/src/python-svn/trunk/build -c /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c -o build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o > /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c: In function 'PyZlib_uncopy': > /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c:724: warning: implicit declaration of function 'inflateCopy' > gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o -L/Users/skip/local/lib -L/usr/local/lib -lz -o build/lib.macosx-10.3-ppc-2.5/zlib.so -Wl,-search_paths_first > *** WARNING: renaming "zlib" since importing it failed: dlopen(build/lib.macosx-10.3-ppc-2.5/zlib.so, 2): Symbol not found: _inflateCopy > Referenced from: build/lib.macosx-10.3-ppc-2.5/zlib.so > Expected in: dynamic lookup > > Anybody else seen this? I checked the buildbot trunk osx 10.4. It seemed > to have no trouble. And what's with the "10.3" bit in the directory names? > > Skip > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From greg.ewing at canterbury.ac.nz Sat Jul 1 08:14:28 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Jul 2006 18:14:28 +1200 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> References: <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> Message-ID: <44A612C4.6020908@canterbury.ac.nz> Andrew Koenig wrote: > Incidentally, I think that lexical scoping would also deal with the problem > that people often encounter in which they have to write things like "lambda > x=x:" where one would think "lambda x:" would suffice. This is another red herring. Python's problem here is not because its scoping isn't lexical (it is). It's because Scheme implicitly introduces new scopes in various places where the equivalent Python constructs don't. -- Greg From greg.ewing at canterbury.ac.nz Sat Jul 1 08:14:54 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Jul 2006 18:14:54 +1200 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <1f7befae0606301655i2425c0a1lf8d956cad0f028b7@mail.gmail.com> References: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com> <001401c69c9d$12d53900$6402a8c0@arkdesktop> <1f7befae0606301655i2425c0a1lf8d956cad0f028b7@mail.gmail.com> Message-ID: <44A612DE.1080503@canterbury.ac.nz> Tim Peters wrote: > Note that this is quite unlike Scheme, in which declaration must > appear before use (ignoring fancy letrec cases), I think that's overstating things a bit -- mutually recursive functions are quite easy to write in Scheme and don't look at all "fancy" (unless you object for some reason to using (define ...)). > and declaration must > also supply an initial binding (Scheme has no "unbound local" problem > because there's no way to create an uninitialized local). That much is true. -- Greg From greg.ewing at canterbury.ac.nz Sat Jul 1 08:15:04 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Jul 2006 18:15:04 +1200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <20060630163625.10D1.JCARLSON@uci.edu> References: <20060630163625.10D1.JCARLSON@uci.edu> Message-ID: <44A612E8.8050205@canterbury.ac.nz> Josiah Carlson wrote: > What I asked before, and what I'd like to ask again, is if there are any > _nontrivial uses_ of lexically nested scopes which are made cumbersome > by our inability to write to parent scopes. The trouble with taking that position is that the very cases which would benefit are very *simple* ones, where it would be cumbersome to refactor it to use a class, or mutable object in the outer scope, etc. So you've effectively set up your acceptance criteria to be unmeetable. > If there aren't, then I'm > going to again have to argue against new syntax, keywords, and their use. There's one very simple way we could do this in Py3k without requiring any new syntax or keywords: just redefine the meaning of "global" to mean "not local". -- Greg From greg.ewing at canterbury.ac.nz Sat Jul 1 08:15:29 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Jul 2006 18:15:29 +1200 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <020c01c69ca1$754c7310$d1b12997@bagio> References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com> <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> <1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com> <020c01c69ca1$754c7310$d1b12997@bagio> Message-ID: <44A61301.2070805@canterbury.ac.nz> Giovanni Bajo wrote: >>>>a = [] >>>>for i in range(10): > > ... a.append(lambda: i) > ... > >>>>print [x() for x in a] > > [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] > > This subtle semantic of lambda is quite confusing, and still forces people to > use the "i=i" trick. This has *nothing* to do with the semantics of lambda! It's because Python's for-loop doesn't put its control variable in a new scope, the way Scheme's equivalent construct does. *That's* what needs to be addressed to fix this problem. I've made a suggestion about that before, but Guido rejected it, so I won't repeat it here. -- Greg From tim.peters at gmail.com Sat Jul 1 09:28:35 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 1 Jul 2006 03:28:35 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <44A612DE.1080503@canterbury.ac.nz> References: <1f7befae0606301623q4072f138vb4b7c24ca7bbff08@mail.gmail.com> <001401c69c9d$12d53900$6402a8c0@arkdesktop> <1f7befae0606301655i2425c0a1lf8d956cad0f028b7@mail.gmail.com> <44A612DE.1080503@canterbury.ac.nz> Message-ID: <1f7befae0607010028q5da09c6ahb2f8c6358dbcd510@mail.gmail.com> [Tim Peters] >> Note that this is quite unlike Scheme, in which declaration must >> appear before use (ignoring fancy letrec cases), [Greg Ewing] > I think that's overstating things a bit -- So do I :-), but I don't really care about Scheme here. > mutually recursive functions are quite easy to write in > Scheme and don't look at all "fancy" (unless you object for > some reason to using (define ...)). In this context, yes, I object to using "define", because the semantics of internal definitions are defined in terms of an equivalent (letrec ...) form. The "fancy" gimmick is that letrec views all its bindings as occurring simultaneously, so strains a natural, linear understanding of "no use before declaration". But none of this appears to have any relevance to Python, so I'm happiest _here_ just calling that "fancy" and ignoring the details. Ditto "top level" definitions, which have unique rules of their own. From anthony at interlink.com.au Sat Jul 1 09:57:59 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat, 1 Jul 2006 17:57:59 +1000 Subject: [Python-Dev] ImportWarning flood In-Reply-To: References: <44A5D5DA.2060601@hathawaymix.org> Message-ID: <200607011758.03756.anthony@interlink.com.au> On Saturday 01 July 2006 12:55, Guido van Rossum wrote: > It's up to the release manager now to decide whether the pitchforks > at Google or the pitchforks in the larger Python community are > sharper. ;-) At this point, I think removing the warning code is the prudent course. If someone wanted to find an easy and safe way to make it only be triggered when the import fails, it could stay in. I'm not convinced that _anything_ in import.c is easy and safe. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From ncoghlan at gmail.com Sat Jul 1 10:01:29 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 01 Jul 2006 18:01:29 +1000 Subject: [Python-Dev] ImportWarning flood In-Reply-To: References: <20060630194331.29014.836424634.divmod.quotient.16780@ohm> <44A5D5DA.2060601@hathawaymix.org> Message-ID: <44A62BD9.7000906@gmail.com> Guido van Rossum wrote: > It's up to the release manager now to decide whether the pitchforks at > Google or the pitchforks in the larger Python community are sharper. > ;-) > > --Guido (ducks) I vaguely recall one of the reasons we went with the warning approach was to find out whether or not dropping __init__.py would cause serious problems - I think we have our answer to that question now :) How does this sound for a way forward?: 2.5b2: - ignore ImportWarning by default (like PendingDeprecationWarning) - include in What's New instructions to enable it via the command line or Python code 2.6: - only show ImportWarning if the import ultimately fails - enable ImporWarning by default - consider allowing a .py extension on a directory name as an alternative to an __init__.py file. Google could then change their sitecustomize.py to enable the warning by default when they roll out 2.5 :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From anthony at interlink.com.au Sat Jul 1 10:02:21 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Sat, 1 Jul 2006 18:02:21 +1000 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <44A57954.7030905@v.loewis.de> References: <86FF939A-5D9F-42C0-B633-95837FD7C991@fuhm.net> <44A57954.7030905@v.loewis.de> Message-ID: <200607011802.23996.anthony@interlink.com.au> On Saturday 01 July 2006 05:19, Martin v. L?wis wrote: > James Y Knight wrote: > > I just submitted http://python.org/sf/1515169 for the > > ImportWarning issue previously discussed here. IMO it's > > important. > > At the moment (i.e. without an acceptable alternative > implementation) it's primarily a policy issue. There really isn't > any bug here; (to speak with Microsoft's words): This behavior is > by design. > > Only the release manager or the BDFL could revert the feature, and > Guido already stated that the warning stays until Python 3, and > probably even after that. I personally believe the only chance to > get this changed now is a well-designed alternative implementation > (although this is no promise that such an alternative would > actually be accepted). given the number of people and ways that this can emit a spurious warning, I think it should be reverted for 2.5. At _best_ we could maybe have a new -W switch to make it be generated, but this should be off by default. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From tim.peters at gmail.com Sat Jul 1 10:08:39 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 1 Jul 2006 04:08:39 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <44A61301.2070805@canterbury.ac.nz> References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com> <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> <1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com> <020c01c69ca1$754c7310$d1b12997@bagio> <44A61301.2070805@canterbury.ac.nz> Message-ID: <1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com> [Giovanni Bajo] >> >>> a = [] >> >>> for i in range(10): >> >> ... a.append(lambda: i) >> ... >> >> >>> print [x() for x in a] >> >> [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] >> >> This subtle semantic of lambda is quite confusing, and still forces people to >> use the "i=i" trick. [Greg Ewing] > This has *nothing* to do with the semantics of lambda! > It's because Python's for-loop doesn't put its control > variable in a new scope, the way Scheme's equivalent > construct does. I don't think I follow that. Scheme has no loops in Python's sense -- things like "do" are shorthand for expressing stylized recursion, where each conceptual iteration gets a fresh set of "loop variables". When people talk about giving a Python for-loop vrbl its own scope, they generally don't mean a new scope on _each iteration_, they just mean that, e.g., i = 5 for i in range(10): # do stuff print i prints 5 intead of 9, about the same as creating a nested block with its own autos in C. The Scheme way is more like: i = 5 def step(i): # do stuff if i < 9: step(i+1) step(0) print i except with tail-recursion elimination. That also prints 5, but does a hell of a lot more than _just_ arrange for that. > *That's* what needs to be addressed to fix this problem. > I've made a suggestion about that before, but Guido > rejected it, so I won't repeat it here. Don't recall what that was, but creating a new scope on each iteration sounds hard to explain in Python. If Giovanni wants the Scheme way ;-), it's available: """ a = [] def step(i): a.append(lambda: i) if i < 9: step(i+1) step(0) print [x() for x in a] """ prints [0, 1, 2, 3, 4, 5, 6, 7, 8, 9], although it's more sanely written in Python with a loop: """ def make_lambda(i): return lambda: i a = [] for i in range(10): a.append(make_lambda(i)) print [x() for x in a] """ Abusing the default-argument machinery to capture current bindings is never necessary, and _is_ abuse. Of course I do it too -- but rarely :-) From ncoghlan at gmail.com Sat Jul 1 10:35:02 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 01 Jul 2006 18:35:02 +1000 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <020c01c69ca1$754c7310$d1b12997@bagio> References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com><001201c69c9c$b6531fd0$6402a8c0@arkdesktop> <1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com> <020c01c69ca1$754c7310$d1b12997@bagio> Message-ID: <44A633B6.3020403@gmail.com> Giovanni Bajo wrote: > Yes but: > >>>> a = [] >>>> for i in range(10): > ... a.append(lambda: i) > ... >>>> print [x() for x in a] > [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] > > This subtle semantic of lambda is quite confusing, and still forces people to > use the "i=i" trick. If you'd like each function instance to have a separate closure scope, then *give* each function a separate closure scope, instead of making them all share the same one the way you have above: >>> def make_f(i): ... def f(): ... return i ... return f ... >>> a = [] >>> for i in range(10): ... a.append(make_f(i)) ... >>> print [x() for x in a] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sat Jul 1 10:37:37 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 01 Jul 2006 18:37:37 +1000 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <200607011802.23996.anthony@interlink.com.au> References: <86FF939A-5D9F-42C0-B633-95837FD7C991@fuhm.net> <44A57954.7030905@v.loewis.de> <200607011802.23996.anthony@interlink.com.au> Message-ID: <44A63451.6010909@gmail.com> Anthony Baxter wrote: > given the number of people and ways that this can emit a spurious > warning, I think it should be reverted for 2.5. At _best_ we could > maybe have a new -W switch to make it be generated, but this should > be off by default. Last line of warnings.py Copy, paste, s/PendingDeprecationWarning/ImportWarning -Wd to enable it again :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From arigo at tunes.org Sat Jul 1 10:49:00 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat, 1 Jul 2006 10:49:00 +0200 Subject: [Python-Dev] Cleanup of test harness for Python In-Reply-To: <4dab5f760606300705l41c208c8tfb83f09f74badf2e@mail.gmail.com> References: <4dab5f760606300705l41c208c8tfb83f09f74badf2e@mail.gmail.com> Message-ID: <20060701084900.GB17748@code0.codespeak.net> Hi all, On Fri, Jun 30, 2006 at 10:05:14AM -0400, Frank Wierzbicki wrote: > some checks for CPython internal tests that should be excluded from > Jython I know Frank already knows about this, but I take the occasion to remind us that http://codespeak.net/svn/pypy/dist/lib-python/modified-2.4.1/test already shows which tests we had to modify for PyPy to make them less implementation-detail-dependent, and which changes were made. A possible first step here would be to find a consistent way to check, in the test, which implementation we are running on top of, so that we can (re-)write the tests accordingly. A bientot, Armin From arigo at tunes.org Sat Jul 1 10:52:21 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat, 1 Jul 2006 10:52:21 +0200 Subject: [Python-Dev] sys.settrace() in Python 2.3 vs. 2.4 In-Reply-To: <20060630131620.10CE.JCARLSON@uci.edu> References: <20060630094140.10C8.JCARLSON@uci.edu> <44A582AE.7010902@v.loewis.de> <20060630131620.10CE.JCARLSON@uci.edu> Message-ID: <20060701085221.GC17748@code0.codespeak.net> Hi Josiah, On Fri, Jun 30, 2006 at 01:27:24PM -0700, Josiah Carlson wrote: > I'll just have to gracefully degrade functionality for older Pythons. More precisely, the bug shows up because in while 1: pass the current line remains on the 'pass' forever. It works for a loop like that: while 1: sys sys but it's admittedly quite obscure. Armin From jcarlson at uci.edu Sat Jul 1 11:01:12 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 01 Jul 2006 02:01:12 -0700 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <44A612E8.8050205@canterbury.ac.nz> References: <20060630163625.10D1.JCARLSON@uci.edu> <44A612E8.8050205@canterbury.ac.nz> Message-ID: <20060701013710.10D7.JCARLSON@uci.edu> Greg Ewing wrote: > > Josiah Carlson wrote: > > > What I asked before, and what I'd like to ask again, is if there are any > > _nontrivial uses_ of lexically nested scopes which are made cumbersome > > by our inability to write to parent scopes. > > The trouble with taking that position is that the very > cases which would benefit are very *simple* ones, where > it would be cumbersome to refactor it to use a class, > or mutable object in the outer scope, etc. So you've > effectively set up your acceptance criteria to be > unmeetable. If the only code that benefits from such changes are "very *simple*", then I think that says something about its necessity. That is, if anything more complicated than those that are "very *simple*" generally don't benefit, then I don't believe that such a modification would be beneficial to the language overall. Further, a simple namespace factory can handle much of the current issues, without needing to create or change keywords. def namespace(**kwds): class namespace(object): __slots__ = kwds.keys() def __init__(self): for i,j in kwds.iteritems(): setattr(self, i,j) return namespace() def trivial_counter(start): ns = namespace(current=start-1) def next(): ns.current += 1 return ns.current return next Maybe a variant of the above namespace factory should make it into the collections module. > > If there aren't, then I'm > > going to again have to argue against new syntax, keywords, and their use. > > There's one very simple way we could do this in Py3k > without requiring any new syntax or keywords: just > redefine the meaning of "global" to mean "not local". I would probably be a solid -0 on such a proposal; I still don't think it's really necessary, but I've never used (or really seen) global more than one level deep, so would guess its impact would be low. - Josiah From g.brandl at gmx.net Sat Jul 1 11:00:13 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 01 Jul 2006 11:00:13 +0200 Subject: [Python-Dev] Bug in stringobject? In-Reply-To: References: Message-ID: Georg Brandl wrote: > In string_replace, there is > > if (PyString_Check(from)) { > /* Can this be made a '!check' after the Unicode check? */ > } > #ifdef Py_USING_UNICODE > if (PyUnicode_Check(from)) > return PyUnicode_Replace((PyObject *)self, > from, to, count); > #endif > else if (PyObject_AsCharBuffer(from, &tmp_s, &tmp_len)) > return NULL; > > [the same check with "to"] > > return (PyObject *)replace((PyStringObject *) self, > (PyStringObject *) from, > (PyStringObject *) to, count); > > > Can this be correct if from or to isn't a string object, but a > char buffer compatible object? May I note that this is still unresolved? I can submit a bug report and add it to PEP 356, too... Georg From fredrik at pythonware.com Sat Jul 1 11:29:56 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat, 01 Jul 2006 11:29:56 +0200 Subject: [Python-Dev] Bug in stringobject? In-Reply-To: References: Message-ID: Georg Brandl wrote: >> Can this be correct if from or to isn't a string object, but a >> char buffer compatible object? > > May I note that this is still unresolved? I can submit a bug report > and add it to PEP 356, too... it's already on my todo list, but that list is full of stuff, so having it on the official todo list is probably a good idea. if you do add it, assign it to me. From g.brandl at gmx.net Sat Jul 1 11:41:53 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 01 Jul 2006 11:41:53 +0200 Subject: [Python-Dev] Bug in stringobject? In-Reply-To: References: Message-ID: Fredrik Lundh wrote: > Georg Brandl wrote: > >>> Can this be correct if from or to isn't a string object, but a >>> char buffer compatible object? >> >> May I note that this is still unresolved? I can submit a bug report >> and add it to PEP 356, too... > > it's already on my todo list, but that list is full of stuff, so having > it on the official todo list is probably a good idea. if you do add it, > assign it to me. Done. #1515471. Georg From andrewdalke at gmail.com Sat Jul 1 11:59:00 2006 From: andrewdalke at gmail.com (Andrew Dalke) Date: Sat, 1 Jul 2006 11:59:00 +0200 Subject: [Python-Dev] PEP 328 and PEP 338, redux In-Reply-To: <018e01c69b77$a9172e40$d503030a@trilan> References: <44A11EA1.1000605@iinet.net.au> <5.1.1.6.0.20060627120926.02021fe0@sparrow.telecommunity.com> <033001c69a07$44140890$d503030a@trilan> <44A1CE4A.2000900@canterbury.ac.nz> <039201c69ad7$4dc91df0$d503030a@trilan> <44A3B648.2060104@gmail.com> <018e01c69b77$a9172e40$d503030a@trilan> Message-ID: Giovanni Bajo wrote: > Real-world usage case for import __main__? Otherwise, I say screw it :) I have used it as a workaround for timeit.py's requirement that I pass it strings instead of functions. >>> def compute(): ... 1+1 ... >>> import timeit >>> t = timeit.Timer("__main__.compute()", "import __main__") >>> t.timeit() 1.9755008220672607 >>> You can argue (as many have) that timeit.py needs a better API for this. That's a different world than the existing real one. Andrew dalke at dalkescientific.com From tomerfiliba at gmail.com Sat Jul 1 15:49:46 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 1 Jul 2006 15:49:46 +0200 Subject: [Python-Dev] weakattr Message-ID: <1d85506f0607010649y65c9fb24mbc5c636360970e52@mail.gmail.com> weakattr (weak attributes) are attributes that are weakly referenced by their containing object. they are very useful for cyclic references -- an object that holds a reference to itself. when a cyclic reference is found by the GC, the memory may be freed, but __del__ is not called, because it's impossible to tell which __del__ to call first. this is an awkward asymmetry with no clean solution: most such objects provide a "close" or "dispose" method that must be called explicitly. weakattrs to solve this problem, by providing a "magical" attribute that "disappears" when the attribute is no longer strongly-referenced. you can find the code, as well as some examples, on this link http://sebulba.wikispaces.com/recipe+weakattr since the stdlib already comes with weakref.py, which provides higher level concepts over the builtin _weakref module, i'd like to make weakattr a part of it. it's only ~20 lines of code, and imho saves the trouble of explicitly releasing the resource of un__del__able objects. i think it's useful. here's a snippet: >>> from weakref import weakattr >>> >>> class blah(object): ... yada = weakref() ... >>> o1 = blah() >>> o2 = blah() >>> o1.yada = o2 >>> o2.yada = o1 o1.yada is a *weakref* to o2, so that when o2 is no longer strongly-referenced... >>> del o2 o1.yada "magically" disappears as well. >>> o1.yada ... AttributeError(...) since the programmer explicitly defined "yada" as a weakatt, he/she knows it might "disappear". it might look awkward at first, but that's exactly the *desired* behavior (otherwise we'd just use the regular strong attributes). another thing to note is that weakattrs are likely to be gone only when the object's __del__ is already invoked, so the only code that needs to take such precautions is __del__ (which already has some constraints) for example: >>> class blah(object): ... me = weakattr() ... ... def __init__(self): ... self.me = self ... ... def something(self): ... # we can rest assure me exists at this stage ... print self.me ... ... def __del__(self): ... # by the time __del__ is called, "me" is removed ... print "me exists?", hasattr(self, "me") ... >>> b = blah() >>> b.me <__main__.blah object at 0x00C0EC10> >>> b.something() <__main__.blah object at 0x00C0EC10> >>> del b >>> import gc >>> gc.collect() me exists? False 0 >>> -tomer -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060701/b174e77e/attachment.htm From nas at arctrix.com Sat Jul 1 17:41:29 2006 From: nas at arctrix.com (Neil Schemenauer) Date: Sat, 1 Jul 2006 15:41:29 +0000 (UTC) Subject: [Python-Dev] Lexical scoping in Python 3k References: <001201c69c65$b3869750$6402a8c0@arkdesktop> Message-ID: Ka-Ping Yee wrote: > Most other languages that support lexical scoping (including Scheme, > JavaScript, Ruby, Perl, E, Java, Smalltalk) provide a uniform way > to read and write to scopes at all levels. This is done by letting > programmers specify the scope in which they want a variable bound > (usually with a keyword like "var" in JavaScript, "my" in Perl, or > "define" in E). That's not the Python way, IMO. I think the right way (assuming we actually want to allow it) is to introduce a pure assignment statement in addition to the assignment/declaration statement that we already have. For example: a = 1 def f(): b = 2 a := 2 def g(): b := 3 print a, b, c g() f() would print "2 3 4". The := would assign but not declare a variable in the current scope. Neil From ronaldoussoren at mac.com Sat Jul 1 18:24:15 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 1 Jul 2006 18:24:15 +0200 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <17573.60593.759293.549583@montanaro.dyndns.org> References: <17573.60593.759293.549583@montanaro.dyndns.org> Message-ID: <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> On Jul 1, 2006, at 5:32 AM, skip at pobox.com wrote: > Just upgraded my Mac to OSX 10.4.7 yesterday. svn up'd Python > trunk, then > "make clean ; configure ; make" and I see that building the zlib > module > fails: > > gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno- > fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I/Users/ > skip/src/python-svn/trunk/./Include -I/Users/skip/src/python-svn/ > trunk/./Mac/Include -I/Users/skip/local/include -I../Include -I. -I/ > usr/local/include -I/Users/skip/src/python-svn/trunk/Include -I/ > Users/skip/src/python-svn/trunk/build -c /Users/skip/src/python-svn/ > trunk/Modules/zlibmodule.c -o build/temp.macosx-10.3-ppc-2.5/Users/ > skip/src/python-svn/trunk/Modules/zlibmodule.o > /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c: In > function 'PyZlib_uncopy': > /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c:724: > warning: implicit declaration of function 'inflateCopy' > gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.3- > ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o -L/ > Users/skip/local/lib -L/usr/local/lib -lz -o build/lib.macosx-10.3- > ppc-2.5/zlib.so -Wl,-search_paths_first > *** WARNING: renaming "zlib" since importing it failed: dlopen > (build/lib.macosx-10.3-ppc-2.5/zlib.so, 2): Symbol not found: > _inflateCopy > Referenced from: build/lib.macosx-10.3-ppc-2.5/zlib.so > Expected in: dynamic lookup > > Anybody else seen this? I checked the buildbot trunk osx 10.4. It > seemed > to have no trouble. And what's with the "10.3" bit in the > directory names? Are you sure you're building on a 10.4 box? Both the macosx-10.3 thingy and lack of inflateCopy seem to indicate that you're running on 10.3. Ronald > > Skip > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ > ronaldoussoren%40mac.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2157 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/f51f75e9/attachment.bin From almann.goo at gmail.com Sat Jul 1 18:42:27 2006 From: almann.goo at gmail.com (Almann T. Goo) Date: Sat, 1 Jul 2006 12:42:27 -0400 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <20060701013710.10D7.JCARLSON@uci.edu> References: <20060630163625.10D1.JCARLSON@uci.edu> <44A612E8.8050205@canterbury.ac.nz> <20060701013710.10D7.JCARLSON@uci.edu> Message-ID: <7e9b97090607010942r599bfed9u4bc02a564becaf2b@mail.gmail.com> On 7/1/06, Josiah Carlson wrote: > > There's one very simple way we could do this in Py3k > > without requiring any new syntax or keywords: just > > redefine the meaning of "global" to mean "not local". > > I would probably be a solid -0 on such a proposal; I still don't think > it's really necessary, but I've never used (or really seen) global more > than one level deep, so would guess its impact would be low. > This has been discussed at length in the following thread that I started in February and at least one time before that. http://mail.python.org/pipermail/python-dev/2006-February/061568.html I think using the "global" keyword is probably the lowest impact form and has the least amount of backwards incompatibility. Below is the part of the last thread that I talked about changing the meaning of "global." http://mail.python.org/pipermail/python-dev/2006-February/061852.html Having the "global" keyword semantics changed to be "lexically global" would break in the cases that "global" is used on a name within a nested scope that has an enclosing scope with the same name. I would suppose that actual instances in real code of this would be rare. Consider: >>>* x = 1 *>>>* def f() : *... x = 2 ... def inner() : ... global x ... print x ... inner() ... >>>* f() *1 Under the proposed rules: >>>* f() *2 PEP 227 also had backwards incompatibilities that were similar and I suggest handling them the same way by issuing a warning in these cases when the new semantics are not being used (i.e. no "from __future__"). Most people probably think that this is a low impact "wart" on the Python language that not really worth fixing as there are workarounds (i.e. mutable objects) or other ways to express (i.e. use classes) such things, but it does trip people up from time to time as warts typically do--I guess that's why this gets brought up now and again. Best regards, Almann -- Almann T. Goo almann.goo at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060701/7f3f113c/attachment.html From skip at pobox.com Sat Jul 1 18:57:34 2006 From: skip at pobox.com (skip at pobox.com) Date: Sat, 1 Jul 2006 11:57:34 -0500 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> References: <17573.60593.759293.549583@montanaro.dyndns.org> <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> Message-ID: <17574.43390.560231.425494@montanaro.dyndns.org> Ronald> Are you sure you're building on a 10.4 box? Both the Ronald> macosx-10.3 thingy and lack of inflateCopy seem to indicate that Ronald> you're running on 10.3. Well, yeah, pretty sure. Let's see. The box with the disk says "Mac OS X Tiger - Version 10.4" on the spine. The "About This Mac" popup says "10.4.7". It used to run 10.3 though. Is there some possibility the update from 10.3 to 10.4 had problems? Note that the compile log on the buildbot 10.4 box also has "10.3" in its directory names. If I remember correctly, it came from Apple with 10.4 installed. Skip From sergey at optimaltec.com Sat Jul 1 18:52:19 2006 From: sergey at optimaltec.com (Sergey A. Lipnevich) Date: Sat, 01 Jul 2006 12:52:19 -0400 Subject: [Python-Dev] ImportWarning flood In-Reply-To: <44A282E3.1000009@v.loewis.de> References: <20060628122558.19551.qmail@web31505.mail.mud.yahoo.com> <44A282E3.1000009@v.loewis.de> Message-ID: All, I tried to implement Jean-Paul Calderone's idea for the following patch, plagiarizing Ralf W. Grosse-Kunstleve's error text. It delays import warning until end of search for modules, but remembers how many potential modules (candidates without __init__.py) it didn't import. I didn't really try to analyze any conditions, instead I simply assumed that wherever ImportWarning would be issued, we have a suitable candidate, and saved it on the stack. If nothing is found, Python emits ImportWarning right before ImportError, and explains what happened. Please let me know if this would work and if anything needs to be done for this patch to be accepted. Thank you! Sergey. -------------- next part -------------- A non-text attachment was scrubbed... Name: import_warning.diff Type: text/x-patch Size: 1867 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/94ab9c43/attachment-0001.bin From aahz at pythoncraft.com Sat Jul 1 19:31:00 2006 From: aahz at pythoncraft.com (Aahz) Date: Sat, 1 Jul 2006 10:31:00 -0700 Subject: [Python-Dev] ImportWarning flood In-Reply-To: References: <20060628122558.19551.qmail@web31505.mail.mud.yahoo.com> <44A282E3.1000009@v.loewis.de> Message-ID: <20060701173100.GA22185@panix.com> On Sat, Jul 01, 2006, Sergey A. Lipnevich wrote: > > Please let me know if this would work and if anything needs to be done > for this patch to be accepted. The first thing you need to do for ANY patch to be considered is to post it so SourceForge (or at least post to python-dev explaining that you'll post to SF as soon as it comes back). -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I saw `cout' being shifted "Hello world" times to the left and stopped right there." --Steve Gonedes From martin at v.loewis.de Sat Jul 1 19:32:29 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 01 Jul 2006 19:32:29 +0200 Subject: [Python-Dev] ImportWarning flood In-Reply-To: References: <20060628122558.19551.qmail@web31505.mail.mud.yahoo.com> <44A282E3.1000009@v.loewis.de> Message-ID: <44A6B1AD.6020204@v.loewis.de> Sergey A. Lipnevich wrote: > I tried to implement Jean-Paul Calderone's idea for the following patch, > plagiarizing Ralf W. Grosse-Kunstleve's error text. It delays import > warning until end of search for modules, but remembers how many > potential modules (candidates without __init__.py) it didn't import. I > didn't really try to analyze any conditions, instead I simply assumed > that wherever ImportWarning would be issued, we have a suitable > candidate, and saved it on the stack. If nothing is found, Python emits > ImportWarning right before ImportError, and explains what happened. > Please let me know if this would work and if anything needs to be done > for this patch to be accepted. Please notice that there is also python.org/sf/1515361 I had no time to compare this with your patch, yet. Regards, Martin From ronaldoussoren at mac.com Sat Jul 1 19:45:17 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 1 Jul 2006 19:45:17 +0200 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <17574.43390.560231.425494@montanaro.dyndns.org> References: <17573.60593.759293.549583@montanaro.dyndns.org> <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> <17574.43390.560231.425494@montanaro.dyndns.org> Message-ID: <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> On Jul 1, 2006, at 6:57 PM, skip at pobox.com wrote: > > Ronald> Are you sure you're building on a 10.4 box? Both the > Ronald> macosx-10.3 thingy and lack of inflateCopy seem to > indicate that > Ronald> you're running on 10.3. > > Well, yeah, pretty sure. Let's see. The box with the disk says > "Mac OS X > Tiger - Version 10.4" on the spine. The "About This Mac" popup says > "10.4.7". That gets the easy solution out of the way ;-) > It used to run 10.3 though. Is there some possibility the update > from 10.3 to 10.4 had problems? > > Note that the compile log on the buildbot 10.4 box also has "10.3" > in its > directory names. If I remember correctly, it came from Apple with > 10.4 > installed. /me slaps head. Having 10.3 in the directory names is intentional, the version in the directory name is the value of MACOSX_DEPLOYMENT_TARGET, with is defaulted to 10.3 in the configure script. What I don't understand yet is why your copy of libz doesn't have inflateCopy. What does /usr/lib/libz.dylib point to on your system? On my 10.4 box it is a symlink that points to libz.1.2.3.dylib and there is an older version of libz (libz.1.1.3.dylib) in /usr/lib as well. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2157 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/d2da4e4b/attachment.bin From sergey at optimaltec.com Sat Jul 1 20:10:42 2006 From: sergey at optimaltec.com (Sergey A. Lipnevich) Date: Sat, 01 Jul 2006 14:10:42 -0400 Subject: [Python-Dev] ImportWarning flood In-Reply-To: <44A6B1AD.6020204@v.loewis.de> References: <20060628122558.19551.qmail@web31505.mail.mud.yahoo.com> <44A282E3.1000009@v.loewis.de> <44A6B1AD.6020204@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Sergey A. Lipnevich wrote: >> I tried to implement Jean-Paul Calderone's idea for the following patch, >> plagiarizing Ralf W. Grosse-Kunstleve's error text. It delays import ... > Please notice that there is also python.org/sf/1515361 > > I had no time to compare this with your patch, yet. Thanks! I made python.org/sf/1515609. The difference (also documented in the description) is that my patch saves memory and some CPU cycles by not trying to collect all directories Python did not import because of missing __init__.py. It only reports how many such directories there are and what is the first one. Sergey. From s.percivall at chello.se Sat Jul 1 20:27:47 2006 From: s.percivall at chello.se (Simon Percivall) Date: Sat, 1 Jul 2006 20:27:47 +0200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <7e9b97090607010942r599bfed9u4bc02a564becaf2b@mail.gmail.com> References: <20060630163625.10D1.JCARLSON@uci.edu> <44A612E8.8050205@canterbury.ac.nz> <20060701013710.10D7.JCARLSON@uci.edu> <7e9b97090607010942r599bfed9u4bc02a564becaf2b@mail.gmail.com> Message-ID: <8B4078DF-F730-4E46-9F2C-04807B178471@chello.se> What about doing something similar to how import was changed? .a = 5 # this scope (self might be too magical ..a = 3 # up one scope ...a # up three Of course, this looks ... perhaps a bit strange. Also, counting is a bother. //Simon From martin at v.loewis.de Sat Jul 1 20:46:44 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 01 Jul 2006 20:46:44 +0200 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> References: <17573.60593.759293.549583@montanaro.dyndns.org> <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> <17574.43390.560231.425494@montanaro.dyndns.org> <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> Message-ID: <44A6C314.2000709@v.loewis.de> Ronald Oussoren wrote: > What I don't understand yet is why your copy of libz doesn't have > inflateCopy. What I don't understand is that configure does not detect that. Regards, Martin From ark at acm.org Sat Jul 1 21:03:19 2006 From: ark at acm.org (Andrew Koenig) Date: Sat, 1 Jul 2006 15:03:19 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <44A61301.2070805@canterbury.ac.nz> Message-ID: <000601c69d41$073dbf00$6402a8c0@arkdesktop> > a = [] > for i in range(10): > a.append(lambda: i) > print [x() for x in a] > > [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] Aha! -- Thank you for jogging my memory. You seem to be right -- the problem is not that Python is lexically scoped, but that when you define a variable with =, it leaks out into the surrounding function scope. Here's an example: If True: y = 123 print y It may be obvious that this should print 123, but that's only because = combines properties of assignment and definition. In particular, if we were to write y = 42 if True: y = 123 print y it would be very surprising if this example were to print anything but 123. Here is a corresponding fragment in C++: int y = 42; if (true) { y = 123; } std::cout << y << "\n"; The "int" in the first line means that the variable y is being defined. Its lack in the third line means that y refers to a variable defined in an outer scope. So both instances of y here refer to the same variable, as they do in Python. But because definition and assignment are separated in C++, we can also write int y = 42; if (true) { int y = 123; } std::cout << y << "\n"; and the fragment will print 42. In this example, there are two distinct variables, both named y. So the problem, as I see it, is indeed that in Python there are suites that look to me as if they should define scopes, but don't. Indeed, if I write if (foo): y = 123 I can't even determine by inspecting the program whether y is defined at all. I might argue that y is always defined, by virtue of appearing before = somewhere in this scope, but the compiler tells me "name 'y' is not defined" if I try it, so I guess that's the right way to treat it. So here's how I understand what Greg was saying. Suppose I write x = [] for i in range(10): x.append(lambda:i) print [f() for f in x] This example will print [9, 9, 9, 9, 9, 9, 9, 9, 9, 9], which I think is wildly unintuitive. My intuition in this matter is partly formed by C++, but it is also formed by other languages going as far back as Algol 68. That intuition says that because the suite controlled by a "for" statement is executed any number of times, potentially including zero, it should be considered as its own scope, and any variables defined in that scope should stay there. In particular, the variable "i" should be defined in the scope of the "for", which implies that each time through the loop, the name "i" should be (re)bound to a different object. What surprises me even more is that if I try to define such a variable explicitly, it still doesn't work: x = [] for i in range(10): j = i x.append(lambda:j) print [f() for f in x] This example still prints [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]. If I understand the reason correctly, it is because even though j is defined only in the body of the loop, loop bodies are not scopes, so the variable's definition is hoisted out into the surrounding function scope. To convince myself of this behavior, I defined an extra function scope, the purpose of which is to localize j: x = [] for i in range(10): def foo(): j = i return lambda:j x.append(foo()) print [f() for f in x] Indeed, this example prints [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]. The example also points up the fact that x.append(lambda:i) and def foo(): j = i return lambda:j x.append(foo()) behave differently, where my intuition (and, I suspect, many other people's as well) would be that they would be equivalent. Finally, I observe that this second example above is also equivalent to x.append(lambda i=i: i) which is what explains the fairly common idiom x = [] for i in range(10): x.append(lambda i=i:i) print [f() for f in x] So maybe what I meant when I asked for lexical scopes was two things: 1) Every indentation level should be a scope; 2) In general, variable definitions should not leak into surrounding scopes. I realize that (2) is too simplistic. Someone who writes if x < 0: y = -x else: y = x will expect y to be defined in the scope surrounding the "if" even if it was not already defined there. On the other hand, I think that the subtle pitfalls that come from allowing "for" variables to leak into the surrounding scopes are much harder to deal with and understand than would be the consequences of restricting their scopes as outlined above. From ark at acm.org Sat Jul 1 21:06:38 2006 From: ark at acm.org (Andrew Koenig) Date: Sat, 1 Jul 2006 15:06:38 -0400 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <8B4078DF-F730-4E46-9F2C-04807B178471@chello.se> Message-ID: <000701c69d41$7dc7a9b0$6402a8c0@arkdesktop> > What about doing something similar to how import was changed? > > .a = 5 # this scope (self might be too magical > ..a = 3 # up one scope > ...a # up three > > Of course, this looks ... perhaps a bit strange. Also, counting is a > bother. I'd rather see a simpler rule: = never defines a variable in a surrounding scope. If you want to affect the binding of such a variable, you have to define it explicitly in the scope in which you want it. Example: x = 42 def f(): x = 123 # rebinds x as defined above y = 123 # defines local variable f() print x # prints 123 print y # error -- y not defined Yes, I know that rule is too simplistic. But I think I'd still prefer it to the way things are now. From ronaldoussoren at mac.com Sat Jul 1 21:12:24 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 1 Jul 2006 21:12:24 +0200 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <44A6C314.2000709@v.loewis.de> References: <17573.60593.759293.549583@montanaro.dyndns.org> <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> <17574.43390.560231.425494@montanaro.dyndns.org> <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> <44A6C314.2000709@v.loewis.de> Message-ID: <0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com> On Jul 1, 2006, at 8:46 PM, Martin v. L?wis wrote: > Ronald Oussoren wrote: >> What I don't understand yet is why your copy of libz doesn't have >> inflateCopy. > > What I don't understand is that configure does not detect that. You may be onto something there. Skip, do you have another copy of libz somewhere? Given the link line in your first message either in / usr/local/lib or /Users/skip/local/lib. And if you have, is that a static library (libz.a) instead of a dylib? As background to my question: the linker on OSX behaves slightly different than the one on most other unix-y systems. It first searches the entire linker path for shared libraries (dylibs) before looking for static libraries. I added a flag to the link flags for the zlib extension a while back that changes the search order into a more traditional one: look in every directory on the linker path for either a dylib or static library. The new flag is -Wl,- search_paths_first. If skip does indeed have libz somewhere else we'll either have to make a matching update to configure, or roll back my change. If the latter I'll have to tweak the build script for the binary installer for OSX because I want to link that using a static copy of libz for binary compatibility with OSX 10.3.9. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2157 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20060701/e0a2a627/attachment-0001.bin From bob at redivi.com Sat Jul 1 21:19:58 2006 From: bob at redivi.com (Bob Ippolito) Date: Sat, 1 Jul 2006 12:19:58 -0700 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> References: <17573.60593.759293.549583@montanaro.dyndns.org> <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> <17574.43390.560231.425494@montanaro.dyndns.org> <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> Message-ID: On Jul 1, 2006, at 10:45 AM, Ronald Oussoren wrote: > > On Jul 1, 2006, at 6:57 PM, skip at pobox.com wrote: > >> >> Ronald> Are you sure you're building on a 10.4 box? Both the >> Ronald> macosx-10.3 thingy and lack of inflateCopy seem to >> indicate that >> Ronald> you're running on 10.3. >> >> Well, yeah, pretty sure. Let's see. The box with the disk says >> "Mac OS X >> Tiger - Version 10.4" on the spine. The "About This Mac" popup says >> "10.4.7". > > That gets the easy solution out of the way ;-) > >> It used to run 10.3 though. Is there some possibility the update >> from 10.3 to 10.4 had problems? >> >> Note that the compile log on the buildbot 10.4 box also has "10.3" >> in its >> directory names. If I remember correctly, it came from Apple with >> 10.4 >> installed. > > /me slaps head. > > Having 10.3 in the directory names is intentional, the version in > the directory name is the value of MACOSX_DEPLOYMENT_TARGET, with > is defaulted to 10.3 in the configure script. > > What I don't understand yet is why your copy of libz doesn't have > inflateCopy. What does /usr/lib/libz.dylib point to on your system? > On my 10.4 box it is a symlink that points to libz.1.2.3.dylib and > there is an older version of libz (libz.1.1.3.dylib) in /usr/lib as > well. Maybe Skip didn't upgrade to the latest version of Xcode? Perhaps he's still got an old SDK? -bob From rasky at develer.com Sat Jul 1 21:24:04 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 1 Jul 2006 21:24:04 +0200 Subject: [Python-Dev] 2.5 and beyond References: <000601c69d41$073dbf00$6402a8c0@arkdesktop> Message-ID: <024901c69d43$ea5a6480$d503030a@trilan> Andrew Koenig wrote: > Suppose I write > > x = [] > for i in range(10): > x.append(lambda:i) > print [f() for f in x] > > This example will print [9, 9, 9, 9, 9, 9, 9, 9, 9, 9], which I think > is wildly unintuitive. That is my point: to me, it's counter-intuitive just like the infamous "except NameError, TypeError". I believe that names in lambdas/nested-functions referring to local names in the outer scope should really be bound at function definition time (much like default arguments are). > What surprises me even more is that if I try to define such a variable > explicitly, it still doesn't work: > > x = [] > for i in range(10): > j = i > x.append(lambda:j) > print [f() for f in x] > > This example still prints [9, 9, 9, 9, 9, 9, 9, 9, 9, 9]. If I > understand the reason correctly, it is because even though j is > defined only in the body of the loop, loop bodies are not scopes, so > the variable's definition is hoisted out into the surrounding > function scope. Yes. And by itself, I like this fact because it's very handy in many cases. And it's also handy that the iteration variable of the for loop is accessible after the for loop is terminated (in fact, this specific behaviour is already listed among the wont-change for Py3k). > On the other hand, I think that > the subtle pitfalls that come from allowing "for" variables to leak > into the surrounding scopes are much harder to deal with and > understand than would be the consequences of restricting their scopes > as outlined above. As I said, to me there's nothing wrong with the way Python variables leak out of the suites; or, in other words, with the fact that Python has only two namespaces, the function-local and the global namespace. What I don't like is that the lookup of lambda's names are fully deferred at execution time. This behaviour is already not fully followed for local variables in functions, since: >>> y = 0 >>> def foo(): ... print y ... y = 2 ... >>> foo() Traceback (most recent call last): File "", line 1, in ? File "", line 2, in foo UnboundLocalError: local variable 'y' referenced before assignment which means that Python users *already* know that a variable is not really looked up only at run-time, but there's "something" going on even at function definition time. I don't see anything wrong if lambdas (or nested scopes) did the same for names provably coming from the outer scope. -- Giovanni Bajo From noamraph at gmail.com Sat Jul 1 22:06:28 2006 From: noamraph at gmail.com (Noam Raphael) Date: Sat, 1 Jul 2006 23:06:28 +0300 Subject: [Python-Dev] Empty Subscript PEP on Wiki - keep or toss? In-Reply-To: References: <17573.33939.473091.920283@montanaro.dyndns.org> Message-ID: Hello, I posted it as a "pre-PEP", in the hope that it may become a PEP and be accepted. As it happened, Guido said "no" at the end, so I stopped pushing the subject. I think that the main reason for the "no" was that my use case wasn't convincing enough - the objections were that this wasn't useful enough, not that it does anything harmful*. As the one who does think it's useful, I have the tiniest hope that if, in the future, people will become familiar with the package and see the usefulness of allowing empty subscript list, the decision will change. If the only result of me posting it as a PEP is a final "rejected" status that will prevent any chance of that happening, I don't think I'll bother to make it a PEP. If it's not the case, then I'll make it a PEP and post it. Have a good week, Noam * Yes, I know that adding unneeded feature to the language can be considered "harmful". You may not agree with my distinction in this case. As it is, I barely consider this as an "added feature" - I would say it's mostly "a small generalization". 2006/6/30, Georg Brandl : > skip at pobox.com wrote: > > Noam Raphael posted an empty subscript PEP on the Python Wiki: > > > > http://wiki.python.org/moin/EmptySubscriptListPEP > > > > It's not linked to by any other pages on the wiki. Is there a reason it > > wasn't added to the peps repository? > > Perhaps the author forgot to submit it to the PEP editor, or he decided > to abandon it after the mostly negative discussion here. > > Georg > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/noamraph%40gmail.com > From pedronis at strakt.com Sat Jul 1 22:10:40 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Sat, 01 Jul 2006 22:10:40 +0200 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: References: <44A2BF2E.4020706@activestate.com> Message-ID: <44A6D6C0.70408@strakt.com> Brett Cannon wrote: > > I don't know how JavaScript is doing it yet. The critical thing for me > for this month was trying to come up with a security model. > > And if you don't think it is going to scale, how do you think it should > be done? if I remember correctly, the boundary/granularity of mutual isolation is practically web domains, pages from the same domain can liberally access each other data, javascript state. From brett at python.org Sat Jul 1 22:27:51 2006 From: brett at python.org (Brett Cannon) Date: Sat, 1 Jul 2006 13:27:51 -0700 Subject: [Python-Dev] For sandboxing: alternative to crippling file() In-Reply-To: <44A60127.3030706@canterbury.ac.nz> References: <44A60127.3030706@canterbury.ac.nz> Message-ID: On 6/30/06, Greg Ewing wrote: > > Brett Cannon wrote: > > > 1) Is removing 'file' from the builtins dict in PyInterpreterState (and > > maybe some other things) going to be safe enough to sufficiently hide > > 'file' confidently (short of someone being stupid in their C extension > > module and exposing 'file' directly)? > > > > 2) Changing open() to return C-implemented delegate objects for files > > (and thus won't type check, but this is Python so I am not worried about > > that too much) and delegate socket objects for IP and URL addresses. > > My suggestion is to change things so that the constructor > of the file type doesn't open files (at least in restricted > mode). Then it wouldn't matter if untrusted code had real > file objects, as they couldn't use them to get access to > any other files. So require use of open() to open a file and then put the access restrictions in open() while turning off the constructor for file? Seems reasonable. It basically shifts the access restrictions to open() instead of 'file'. For some reason this proposal makes me want to remove the checks in read/write methods as well. That way there is only open() that needs to do the checks and 'file' can have the constructor crippled and that be it. Really minimize the impact of code on 'file' itself. Do people think that having the restriction checks for every read/write method is necessary? I originally thought of doing that so that if an open file object leaked into a restricted interpreter by accident there would still be proper protections, but perhaps that should not be the resopnsibility of 'file' and instead should be more up to modules not passing back exposed 'file' objects into code. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060701/1087426f/attachment.html From talin at acm.org Sat Jul 1 22:32:02 2006 From: talin at acm.org (Talin) Date: Sat, 01 Jul 2006 13:32:02 -0700 Subject: [Python-Dev] More Switch: Explicit freezing Message-ID: <44A6DBC2.5080302@acm.org> Here's another stab at the "explicit freezing" school of thought on the switch semantics. The idea is to borrow the freeze protocol and apply it to functions. In this scheme, the default behavior of switch is to rebuild the dictionary each time the switch is executed. However, by calling freeze(), you can get a 'frozen' version of the function in which all switch dictionaries contained within the function are precalculated: # Emulating 'freeze at function definition time' def myfunc( x ): switch y: case a: ... case b: ... myfunc = freeze( myfunc ) This of course lends itself well to decorator syntax: @freeze def myfunc( x ): switch y: case a: ... case b: ... You can also get 'freeze on first use' via the appropriate decorator function, although that's a litte harder to white (essentially, you need a way to test if the function is already frozen.) Each time you call freeze(), you get a new copy of the function object with the switch dictionaries bound to the values that were in the scope of the call to 'freeze'. This means that you can call freeze several times and get several different versions of the function: def myfunc( x ): switch y: case a: ... case b: ... a = 1 b = 2 f1 = freeze( myfunc ) a = 3 b = 4 f2 = freeze( myfunc ) Now we have two versions of the function, each having a different switch dictionary. Note that 'switch' is still usable without 'freeze', it just won't run as fast. This means that the folks who are interested in a switch statement purely for its improved expressiveness can simply not bother with freezing the function. -- Talin From skip at pobox.com Sun Jul 2 00:26:17 2006 From: skip at pobox.com (skip at pobox.com) Date: Sat, 1 Jul 2006 17:26:17 -0500 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> References: <17573.60593.759293.549583@montanaro.dyndns.org> <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> <17574.43390.560231.425494@montanaro.dyndns.org> <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> Message-ID: <17574.63113.771422.208090@montanaro.dyndns.org> Ronald> What does /usr/lib/libz.dylib point to on your system? % cd /usr/lib % ls -l libz.* lrwxr-xr-x 1 root wheel 12 Feb 12 00:32 libz.1.1.3.dylib -> libz.1.dylib -rwxr-xr-x 1 root wheel 72588 Jun 29 18:36 libz.1.2.3.dylib lrwxr-xr-x 1 root wheel 16 Feb 12 00:32 libz.1.dylib -> libz.1.2.3.dylib lrwxr-xr-x 1 root wheel 16 Feb 12 00:32 libz.dylib -> libz.1.2.3.dylib Looks like everything on my system winds up at 1.2.3. Ronald> What I don't understand yet is why your copy of libz doesn't Ronald> have inflateCopy. It appears to: % nm libz.1.2.3.dylib | egrep -i inflate U _inflate U _inflateEnd U _inflateInit2_ U _inflateReset U _inflate U _inflateEnd U _inflateInit_ libz.1.2.3.dylib(inflate.o): 9110ea18 T _inflate 911168cc T _inflateCopy 9110e5d8 T _inflateEnd 91116694 t _inflateGetHeader 9110dc84 T _inflateInit2_ 9110e680 T _inflateInit_ 91116524 t _inflatePrime 9110dddc T _inflateReset 91116584 T _inflateSetDictionary 91116744 T _inflateSync 91116888 T _inflateSyncPoint u _inflate_fast u _inflate_table 91116b90 T _inflateBack 91117a6c T _inflateBackEnd 91116a38 T _inflateBackInit_ u _inflate_fast u _inflate_table 91118334 s _inflate_copyright 9111047c t _inflate_table 91110930 t _inflate_fast However, even though the zlib.so is linked with -lz, there's no evidence of it in the otool -L output: % make case $MAKEFLAGS in \ *-s*) CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python.exe -E ../setup.py -q build;; \ *) CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup' OPT='-DNDEBUG -g -O3 -Wall -Wstrict-prototypes' ./python.exe -E ../setup.py build;; \ esac running build running build_ext db.h: found (4, 2) in /sw/include/db4 db lib: using (4, 2) db-4.2 sqlite: found /usr/include/sqlite3.h /usr/include/sqlite3.h: version 3.1.3 building 'zlib' extension gcc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes -I. -I/Users/skip/src/python-svn/trunk/./Include -I/Users/skip/src/python-svn/trunk/./Mac/Include -I/Users/skip/local/include -I../Include -I. -I/usr/local/include -I/Users/skip/src/python-svn/trunk/Include -I/Users/skip/src/python-svn/trunk/build -c /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c -o build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c: In function 'PyZlib_uncopy': /Users/skip/src/python-svn/trunk/Modules/zlibmodule.c:724: warning: implicit declaration of function 'inflateCopy' gcc -bundle -undefined dynamic_lookup build/temp.macosx-10.3-ppc-2.5/Users/skip/src/python-svn/trunk/Modules/zlibmodule.o -L/Users/skip/local/lib -L/usr/local/lib -lz -o build/lib.macosx-10.3-ppc-2.5/zlib.so -Wl,-search_paths_first *** WARNING: renaming "zlib" since importing it failed: dlopen(build/lib.macosx-10.3-ppc-2.5/zlib.so, 2): Symbol not found: _inflateCopy Referenced from: build/lib.macosx-10.3-ppc-2.5/zlib.so Expected in: dynamic lookup running build_scripts montanaro:build% otool -L build/lib.macosx-10.3-ppc-2.5/zlib_failed.so build/lib.macosx-10.3-ppc-2.5/zlib_failed.so: /usr/lib/libmx.A.dylib (compatibility version 1.0.0, current version 93.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 88.1.6) Ah, found it! There was an antique libz.a in /usr/local/lib dating from 2003. It's all better now. Skip From skip at pobox.com Sun Jul 2 00:44:33 2006 From: skip at pobox.com (skip at pobox.com) Date: Sat, 1 Jul 2006 17:44:33 -0500 Subject: [Python-Dev] Empty Subscript PEP on Wiki - keep or toss? In-Reply-To: References: <17573.33939.473091.920283@montanaro.dyndns.org> Message-ID: <17574.64209.730168.967635@montanaro.dyndns.org> Noam> If the only result of me posting it as a PEP is a final "rejected" Noam> status that will prevent any chance of that happening, I don't Noam> think I'll bother to make it a PEP. If it's not the case, then Noam> I'll make it a PEP and post it. Even if it's ultimately rejected, it still serves as useful documentation of the process. I'd go ahead and update it to reflect the latest discussions, submit it, then let the chips fall where they may. That would also get it out of the wiki, orphaned page that it is. Skip From python-dev at zesty.ca Sun Jul 2 00:50:22 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Sat, 1 Jul 2006 17:50:22 -0500 (CDT) Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <44A60A79.8090802@canterbury.ac.nz> References: <001201c69c65$b3869750$6402a8c0@arkdesktop> <44A60A79.8090802@canterbury.ac.nz> Message-ID: On Sat, 1 Jul 2006, Greg Ewing wrote: > I don't disagree with anything you said, but I think it > would be a good idea to avoid using phrases like "proper > lexical scopes", which is likely to set people off on > a tangent. The issue isn't lexicality, it's writeability. "Fully functional" lexical scopes, then? Python's scopes are lexical (except for builtins) but currently somewhat hamstrung. -- ?!ng From ark at acm.org Sun Jul 2 01:42:28 2006 From: ark at acm.org (Andrew Koenig) Date: Sat, 1 Jul 2006 19:42:28 -0400 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: Message-ID: <001301c69d68$070e5fe0$6402a8c0@arkdesktop> > "Fully functional" lexical scopes, then? Fine-grained scopes? From ark at acm.org Sun Jul 2 02:22:46 2006 From: ark at acm.org (Andrew Koenig) Date: Sat, 1 Jul 2006 20:22:46 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com> Message-ID: <001401c69d6d$a7cd8d20$6402a8c0@arkdesktop> > Don't recall what that was, but creating a new scope on each iteration > sounds hard to explain in Python. I don't think it's particularly hard to explain. For example, one way to explain it is to say that for i in <>: body is equivalent to for <> in <>: local i = <> body This explanation doesn't need to rest on recursion. From tim.peters at gmail.com Sun Jul 2 03:08:25 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 1 Jul 2006 21:08:25 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <001401c69d6d$a7cd8d20$6402a8c0@arkdesktop> References: <1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com> <001401c69d6d$a7cd8d20$6402a8c0@arkdesktop> Message-ID: <1f7befae0607011808x7f7b2661h7f96a57f4bedc672@mail.gmail.com> [Tim] >> Don't recall what that was, but creating a new scope on each iteration >> sounds hard to explain in Python. [Andrew Koenig] > I don't think it's particularly hard to explain. For example, one way to explain it is > to say that > > for i in <>: > body > > is equivalent to > > for <> in <>: > local i = <> > body > > This explanation doesn't need to rest on recursion. Sorry, but as a Python programmer that explanation makes little sense to me. In effect, it pushes the mystery into what "local" is supposed to mean, but there's nothing _already_ in Python that acts the way you need "local" to act. Scope in Python is defined wrt "blocks", so you need to phrase this in terms of blocks, and there are very few kinds of blocks in Python's execution model: A block is a piece of Python program text that is executed as a unit. The following are blocks: a module, a function body, and a class definition. Each command typed interactively is a block. A script file (a file given as standard input to the interpreter or specified on the interpreter command line the first argument) is a code block. A script command (a command specified on the interpreter command line with the `-c' option) is a code block. The file read by the built-in function execfile() is a code block. The string argument passed to the built-in function eval() and to the exec statement is a code block. The expression read and evaluated by the built-in function input() is a code block. That's from section "Naming and binding" of the Python Reference Manual. I expect most Python programmers have "module, function, class ... plus some weird stuff I don't much care about" in mind. Python's execution model also has a one-to-one correspondence between active blocks and execution frames (see the rest of that section), which would need to be snapped to consider a finer-grained notion of block that didn't have its own execution frame. In short, it's only easy to define this in Python, without invoking nested functions, if you don't have Python's execution model in mind to begin with. From fdrake at acm.org Sun Jul 2 04:04:47 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sat, 1 Jul 2006 22:04:47 -0400 Subject: [Python-Dev] =?iso-8859-1?q?how_long_to_wait_for_expat_to_incorpo?= =?iso-8859-1?q?rate_a_fix_to=09prevent_a_crasher=3F?= In-Reply-To: <44A583A6.6010602@v.loewis.de> References: <44A583A6.6010602@v.loewis.de> Message-ID: <200607012204.47631.fdrake@acm.org> On Friday 30 June 2006 16:03, Martin v. L?wis wrote: > If you have a patch, you should commit it to our copy. Make sure you > activate the test case, so that somebody incorporating the next Expat > release doesn't mistakenly roll back your change. A modified version of Brett's patch has been committed to Expat, along with regression tests for two specific cases that it handles (only one of which is relevant to Python). The patch to xmlparse.c has also been committed to Python's copy, and the crasher test has been moved to the regular xml.parsers.expat tests. > Of course, you might wait a few days to see whether Fred creates another > release that we could incorporate without introducing new features. I'm not ready to push for an Expat release, since I've not had much time to pay attention to that project over the past year. I'm trying to catch up on that project's email, but don't expect it to be quick. Once I've had time to discuss this with the current principal maintainer, it shouldn't be difficult to get a 2.0.1 release out the door. Once that's done, it'll be time to sync with the Expat release again. -Fred -- Fred L. Drake, Jr. From ncoghlan at gmail.com Sun Jul 2 05:26:48 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 02 Jul 2006 13:26:48 +1000 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: References: <001201c69c65$b3869750$6402a8c0@arkdesktop> Message-ID: <44A73CF8.2080300@gmail.com> Neil Schemenauer wrote: > The := would assign but not declare a variable > in the current scope. There are other benefits to such a statement, too, since we can make it similar to other augmented assignments by letting the object being assigned to interfere with the process. a := 2 could translate to something like: a = a.__assign__(2) with the default behaviour of __assign__ simply being: def __assign__(rhs) return rhs This gives you: - runtime checking for typos (you can't accidentally declare a new variable with := when you really meant to assign to an existing one) - if/when control flow analysis is added to the AST compiler, it will be picked up as an error at compile time along with the other augmented assignments - the object being assigned to can validate/modify its replacement (e.g. automatically wrapping it in a weakref proxy, or checking that it has the correct type) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From jcarlson at uci.edu Sun Jul 2 05:56:07 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 01 Jul 2006 20:56:07 -0700 Subject: [Python-Dev] sys.settrace() in Python 2.3 vs. 2.4 In-Reply-To: <20060701085221.GC17748@code0.codespeak.net> References: <20060630131620.10CE.JCARLSON@uci.edu> <20060701085221.GC17748@code0.codespeak.net> Message-ID: <20060701205520.10DD.JCARLSON@uci.edu> Armin Rigo wrote: > > Hi Josiah, > > On Fri, Jun 30, 2006 at 01:27:24PM -0700, Josiah Carlson wrote: > > I'll just have to gracefully degrade functionality for older Pythons. > > More precisely, the bug shows up because in > > while 1: > pass > > the current line remains on the 'pass' forever. It works for a loop > like that: > > while 1: > sys > sys > > but it's admittedly quite obscure. That is good to know, thank you Armin. - Josiah From jcarlson at uci.edu Sun Jul 2 06:50:11 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 01 Jul 2006 21:50:11 -0700 Subject: [Python-Dev] weakattr In-Reply-To: <1d85506f0607010649y65c9fb24mbc5c636360970e52@mail.gmail.com> References: <1d85506f0607010649y65c9fb24mbc5c636360970e52@mail.gmail.com> Message-ID: <20060701213745.10E0.JCARLSON@uci.edu> "tomer filiba" wrote: > weakattr (weak attributes) are attributes that are weakly referenced > by their containing object. they are very useful for cyclic references -- > an object that holds a reference to itself. I like the added functionality offered with weakattrs as defined. I'm not terribly in love with the syntax of their creation, and I'm curious as to how it plays with __slots__ (not quite having the time to look at its implementation right now), but it is quite explicit, so I can get past that. It would allow us to say, "stop using __del__, use weakattrs", but I'm not sure how well that would work, generally. Toss it out in python-list, I think some people over there would be able to offer more feedback. - Josiah From jcarlson at uci.edu Sun Jul 2 07:46:57 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 01 Jul 2006 22:46:57 -0700 Subject: [Python-Dev] More Switch: Explicit freezing In-Reply-To: <44A6DBC2.5080302@acm.org> References: <44A6DBC2.5080302@acm.org> Message-ID: <20060701215239.10E3.JCARLSON@uci.edu> Talin wrote: > Here's another stab at the "explicit freezing" school of thought on the > switch semantics. The idea is to borrow the freeze protocol and apply it > to functions. -1 . Freezing was previously questionably useful in the realm of general data structures. Using switch/case as a use-case for this functionality, I think, is a non-starter. - Josiah From arigo at tunes.org Sun Jul 2 13:23:44 2006 From: arigo at tunes.org (Armin Rigo) Date: Sun, 2 Jul 2006 13:23:44 +0200 Subject: [Python-Dev] LOAD_CONST POP_TOP In-Reply-To: References: Message-ID: <20060702112344.GA30481@code0.codespeak.net> Hi Georg, On Fri, Jun 30, 2006 at 08:39:13PM +0200, Georg Brandl wrote: > + case LOAD_CONST: > + cumlc = lastlc + 1; > + /* Skip over LOAD_CONST POP_TOP */ > + if (codestr[i+3] == POP_TOP) { This is missing a ISBASICBLOCK() check. It makes the following example segfault: a = 5 for i in range(1000): a or 4 Attached an updated patch to the SF tracker. A bientot, Armin. From jan-python at maka.demon.nl Sun Jul 2 13:58:55 2006 From: jan-python at maka.demon.nl (jan-python at maka.demon.nl) Date: Sun, 02 Jul 2006 13:58:55 +0200 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: References: Message-ID: <20060702135855.bdk6r8kugw0c0w0o@webmail.nl.demon.net> Hi everyone, Even though I'm new on this list I think I've got something sensible to say on this one. (I've been following this list a bit through the archives) Andrew Koenig wrote: > You seem to be right -- the problem is not that Python is lexically scoped, > but that when you define a variable with =, it leaks out into the > surrounding function scope. > So maybe what I meant when I asked for lexical scopes was two things: > > 1) Every indentation level should be a scope; > 2) In general, variable definitions should not leak into > surrounding scopes. > > I realize that (2) is too simplistic. Someone who writes I believe the problem has nothing to do with how many scopes a block/function definition has, but with what the lambda does with the scope it's given. Currently it remembers the block and looks up the nescessary variables in it when it's invoked. I think it shoud should have just taken the values of the needed variables and rememberd those as it's own local variables. So the closed over variables become just local variables initialised to the value they have in the outer scope. Without having any blocks to be confused about, I think this is counterintuitive as well: >>> x = 1 >>> f = lambda: x >>> x = 2 >>> g = lambda: x >>> f() 2 >>> g() 2 I think it should have been: .... >>> f() 1 >>> g() 2 Using the lambda x=x: x trick gives exactly this behaviour because it apparently does copy the value of x. As far as I can see it also solves the >>> for i in range(10): ... a.append(lambda: i) case, and other similar examples. (However, this would probably be a to big change for 2.5) From pje at telecommunity.com Sun Jul 2 16:07:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 02 Jul 2006 10:07:57 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <20060702135855.bdk6r8kugw0c0w0o@webmail.nl.demon.net> References: Message-ID: <5.1.1.6.0.20060702100646.01f1b878@sparrow.telecommunity.com> At 01:58 PM 7/2/2006 +0200, jan-python at maka.demon.nl wrote: >I believe the problem has nothing to do with how many scopes a block/function >definition has, but with what the lambda does with the scope it's given. >Currently it remembers the block and looks up the nescessary variables in it >when it's invoked. I think it shoud should have just taken the values of the >needed variables and rememberd those as it's own local variables. So >the closed >over variables become just local variables initialised to the value >they have in >the outer scope. That won't work. Consider this code, that's perfectly valid Python today: def foo(): def bar(): print x for x in range(10): bar() From amk at amk.ca Sun Jul 2 18:00:44 2006 From: amk at amk.ca (A.M. Kuchling) Date: Sun, 2 Jul 2006 12:00:44 -0400 Subject: [Python-Dev] Another 2.5 bug candidate? Message-ID: <20060702160044.GA21217@rogue.amk.ca> http://www.python.org/sf/1488934 argues that Python's use of fwrite() has incorrect error checking; this most affects file.write(), but there are other uses of fwrite() in the core. It seems fwrite() can return N bytes written even if an error occurred, and the code needs to also check ferror(f->fp). At the last sprint I tried to assemble a small test case to exhibit the problem but failed. The reporter's test case uses SSH, and I did verify that Python does loop infinitely if executed under SSH, but a test case would need to work without SSH. Should this be fixed in 2.5? I'm nervous about such a change to error handling without a test case to add; maybe it'll cause problems on one of our platforms. --amk From nmm1 at cus.cam.ac.uk Sun Jul 2 20:08:19 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Sun, 02 Jul 2006 19:08:19 +0100 Subject: [Python-Dev] Another 2.5 bug candidate? Message-ID: "A.M. Kuchling" wrote: > > http://www.python.org/sf/1488934 argues that Python's use of fwrite() > has incorrect error checking; this most affects file.write(), but > there are other uses of fwrite() in the core. It seems fwrite() can > return N bytes written even if an error occurred, and the code needs > to also check ferror(f->fp). > > At the last sprint I tried to assemble a small test case to exhibit > the problem but failed. The reporter's test case uses SSH, and I did > verify that Python does loop infinitely if executed under SSH, but a > test case would need to work without SSH. > > Should this be fixed in 2.5? I'm nervous about such a change to error > handling without a test case to add; maybe it'll cause problems on one > of our platforms. So would assembling a test case. NOTHING will cause ferror to return True that isn't classed as undefined behaviour, and therefore may fail on some platforms. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From python-dev at zesty.ca Mon Jul 3 00:07:43 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Sun, 2 Jul 2006 17:07:43 -0500 (CDT) Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <000701c69d41$7dc7a9b0$6402a8c0@arkdesktop> References: <000701c69d41$7dc7a9b0$6402a8c0@arkdesktop> Message-ID: On Sat, 1 Jul 2006, Andrew Koenig wrote: > I'd rather see a simpler rule: = never defines a variable in a surrounding > scope. If you want to affect the binding of such a variable, you have to > define it explicitly in the scope in which you want it. > > Example: > > x = 42 > def f(): > x = 123 # rebinds x as defined above > y = 123 # defines local variable > f() > print x # prints 123 > print y # error -- y not defined > > Yes, I know that rule is too simplistic. But I think I'd still prefer it to > the way things are now. I agree with you that this is a nicer and more consistent rule. What do you think of the proposal for a keyword to say "don't rebind"? It would achieve the same distinction you're aiming for above, but without the drastic incompatibility with today's Python. This has been previously discussed as "change the meaning of 'global' to mean 'not local'": http://mail.python.org/pipermail/python-dev/2006-February/061568.html http://mail.python.org/pipermail/python-dev/2006-July/066908.html I support this proposal, though i would prefer a clearer keyword such as "outer x" or "nonlocal x". If we can't agree on another keyword (or can't afford to spend one more keyword), i'm willing to support "global" for this purpose. -- ?!ng From greg.ewing at canterbury.ac.nz Mon Jul 3 03:11:43 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Jul 2006 13:11:43 +1200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <20060701013710.10D7.JCARLSON@uci.edu> References: <20060630163625.10D1.JCARLSON@uci.edu> <44A612E8.8050205@canterbury.ac.nz> <20060701013710.10D7.JCARLSON@uci.edu> Message-ID: <44A86ECF.4060705@canterbury.ac.nz> Josiah Carlson wrote: > If the only code that benefits from such changes are "very *simple*", > then I think that says something about its necessity. The point is that they're only "very simple" if you can write them using access to an outer scope. Without that ability, they become less simple, less efficient, more convoluted, harder to follow, etc. Also I don't buy the argument that something has to be useful for big, complicated things in order to be worth having in the language. -- Greg From greg.ewing at canterbury.ac.nz Mon Jul 3 03:36:31 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Jul 2006 13:36:31 +1200 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <024901c69d43$ea5a6480$d503030a@trilan> References: <000601c69d41$073dbf00$6402a8c0@arkdesktop> <024901c69d43$ea5a6480$d503030a@trilan> Message-ID: <44A8749F.2000801@canterbury.ac.nz> Giovanni Bajo wrote: > I believe that names in > lambdas/nested-functions referring to local names in the outer scope should > really be bound at function definition time No, you don't want that, because it would make functions that call each other very awkward to arrange. > And it's also handy that the iteration variable of the for loop is > accessible after the for loop is terminated (in fact, this specific > behaviour is already listed among the wont-change for Py3k). I'd just like to point out that the create-a-new-cell behaviour that I have proposed for loop variables *preserves* this ability! for new i in range(10): ... print i will still print 9. -- Greg From greg.ewing at canterbury.ac.nz Mon Jul 3 03:50:54 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Jul 2006 13:50:54 +1200 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com> References: <5.1.1.6.0.20060630190709.01ef5858@sparrow.telecommunity.com> <001201c69c9c$b6531fd0$6402a8c0@arkdesktop> <1f7befae0606301639u6f8c968cr29afce0ee7fbb43b@mail.gmail.com> <020c01c69ca1$754c7310$d1b12997@bagio> <44A61301.2070805@canterbury.ac.nz> <1f7befae0607010108hb99c574l9e0d7e3cbf378355@mail.gmail.com> Message-ID: <44A877FE.6040004@canterbury.ac.nz> Tim Peters wrote: > Scheme has no loops in Python's sense -- > things like "do" are shorthand for expressing stylized recursion But it does have foreach and map, which are the moral equivalent of Python's for-loops and list comprehensions. The body is a lambda which takes the loop variable as a parameter, thus providing the extra level of scope. Recursion isn't needed. > When people talk about giving a Python for-loop vrbl its own scope, > they generally don't mean a new scope on _each iteration_, But that's exactly what you *do* need in order for a for-loop with a lambda in it to behave intuitively. If that's not what people mean, it's because they don't fully understand what they really mean to mean. :-) BTW, I'm not suggesting that a new stack frame gets allocated for every iteration -- all you need is a cell. > about the same as creating a nested block with > its own autos in C. Analogies with C aren't very helpful here, because it doesn't have closures, so it's only a matter of visibility, not lifetime. > creating a new scope on each iteration > sounds hard to explain in Python. But is it harder to explain than the reason someone's loop-with-a-lambda doesn't do what they expect? BTW, I wouldn't explain it by saying it creates a new scope, I'd say it creates a new binding on each iteration, or something like that. In my earlier proposal, you would actually say that explicitly, with something like for new i in range(10): ... > Abusing the default-argument machinery to capture current bindings is > never necessary, and _is_ abuse. But it's abuse that still happens, because although scoping has been fixed, other parts of the story are still missing. -- Greg From steven.bethard at gmail.com Mon Jul 3 01:28:08 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 2 Jul 2006 17:28:08 -0600 Subject: [Python-Dev] DRAFT: python-dev summary for 2006-06-01 to 2006-06-15 Message-ID: Here's the summary for the first half of June. Thanks in advance for your comments and corrections! ============= Announcements ============= ------------------- Python 2.5 schedule ------------------- Python 2.5 is moving steadily towards its next release. See `PEP 356`_ for more details and the full schedule. .. _PEP 356: http://www.python.org/dev/peps/pep-0356/ Contributing threads: - `beta1 coming real soon `__ - `2.5 issues need resolving in a few days `__ ----------------------------------------------- Request for Bug Trackers to replace SourceForge ----------------------------------------------- The Python Software Foundation's Infrastructure committee asked for suggestions for tracker systems that could replace SourceForge. The minimum requirements are: * Can import SourceForge data * Can export data * Has an email interface and if you'd like to suggest a particular tracker system all you need to do is: * Install a test tracker * Import the `SourceForge data dump`_ * Make the `Infrastructure committee members`_ administrators of the tracker * Add your tracker to the `wiki page`_ * Email `the Infrastructure committee`_ Be sure to check the `wiki page`_ for additional information. .. _SourceForge data dump: http://effbot.org/zone/sandbox-sourceforge.htm .. _Infrastructure committee members: http://wiki.python.org/moin/PythonSoftwareFoundationCommittees#infrastructure-committee-ic .. _wiki page: http://wiki.python.org/moin/CallForTrackers .. _the Infrastructure committee: infrastructure at python.org Contributing thread: - `Request for trackers to evaluate as SF replacement for Python development `__ ========= Summaries ========= -------------------------------------------- Getting more comparable results from pybench -------------------------------------------- Skip Montanaro mentioned that the NeedForSpeed_ folks had some trouble with the pybench_ string and unicode tests. In some discussions both on the checkins list and off-list, Fredrik Lundh had concluded that stringbench more reliably reported performance than pybench. There was then a long discussion about how to improve pybench including: * Using time.clock() on Windows and time.time() on Linux. This was accompanied by a long debate about whether to use wall-time or process time. Both wall time and process time can see interference from other programs running at the same time; wall time because the time consumed by other programs running at the same time is also counted, and process time because it is sampled so that other processes can charge their time to the running process by using less than a full time slice. In general, the answer was to use the timer with the best resolution. * Using the minimum time rather than the average. Andrew Dalke explained that timing results do not have a Gaussian distribution (they have more of a gamma distribution) and provided some graphs generated on his machine to demonstrate this. Since the slower runs are typically caused by other things running at the same time (which is pretty much unpredictable), it's much better to report the fastest run, which should more consistently approximate the best possible time. * Making sure to use an appropriate warp factor. Marc-Andre Lemburg explained that each testing round of pybench is expected to take around 20-50 seconds. If rounds are much shorter than this, pybench's warp factor should be adjusted until they are long enough. At the end of the thread, Marc-Andre checked in pybench_ 2.0, which included the improvements suggested above. .. _NeedForSpeed: http://wiki.python.org/moin/NeedForSpeed .. _pybench: http://svn.python.org/view/python/trunk/Tools/pybench/ .. _stringbench: http://svn.python.org/view/sandbox/trunk/stringbench/ Contributing threads: - `Python Benchmarks `__ - `Python Benchmarks `__ --------------------------------------- PEP 360: Externally Maintained Packages --------------------------------------- After checking wsgiref into the Python repository, Phillip J. Eby requested in `PEP 360`_ that patches to wsgiref be passed to him before being committed on the trunk. After a number of changes were committed to the trunk and he had to go through a complicated two-way merge, he complained that people were not following the posted procedures. Guido suggested that `PEP 360`_ was a mistake, and that whenever possible, development for any module in the stdlib should be done in the Python trunk, not externally. He also requested that the PEP indicate that even for externally maintained modules, bugfixes and ordinary maintenance should be allowed on the trunk so that bugs in external modules don't hold up Python core development. A number of solutions were discussed for authors of code that is also distributed standalone. Using svn:externals is mostly undesirable because svn is much slower at checking whether or not an svn:externals directory is updated, and because upgrading to a newer version would require making sure that no changes made by Python developers were lost in the new version. Phillip suggested adding an "Externals" directory and modifying Python's setup to invoke all the ``Externals/*/setup.py`` scripts, though this would mean having some Python code that lives outside of the Lib/ subtree. Barry Warsaw explained that for the email package, he maintains a directory in the sandbox with all the distutils and documentation stuff needed for the standalone releases as well as the email package from the Python repository through svn:externals. This means having to create some extra directories (since svn:externals doesn't work with individual files) and having one checkout per version of Python supported, but seemed to work pretty well for Barry. People seemed to like Phillip's Externals idea (possibly renamed to Packages), but work on that was postponed for Python 2.6. One of the side benefits of these discussions was that Thomas Heller generously offered to move ctypes development fully into the Python repository. .. _PEP 360: http://www.python.org/dev/peps/pep-0360/ Contributing threads: - `wsgiref documentation `__ - `wsgiref doc draft; reviews/patches wanted `__ - `[Web-SIG] wsgiref doc draft; reviews/patches wanted `__ - `FYI: wsgiref is now checked in `__ - `Please stop changing wsgiref on the trunk `__ - `Dropping externally maintained packages (Was: Please stop changing wsgiref on the trunk) `__ - `External Package Maintenance (was Re: Please stop changing wsgiref on the trunk) `__ - `External Package Maintenance `__ - `rewording PEP 360 `__ - `Updating packages from external ? `__ -------------------------------------- Universally unique identifiers (UUIDs) -------------------------------------- Ka-Ping Yee was looking to put his `uuid module`_ into Python 2.5. He addressed a number of requests from the last round of discussions, including making UUIDs immutable, removing curly braces from the UUID string and adding the necessary tests to the test suite. Then he asked about how best to address the fact that ``uuid1()`` required looking up a MAC address, a potentially slow procedure. At the suggestion of Fredrik Lundh, he changed the API to allow a MAC address to be passed in if it was already known. If a MAC address is not passed in to ``uuid1()``, the ``getnode()`` utility function is called, which searches for the MAC address through a variety of routes, including some quicker paths through ctypes that Thomas Heller and others helped Ka-Ping with. The code was checked into the Python trunk. .. _uuid module: http://zesty.ca/python/uuid.py Contributing thread: - `UUID module `__ ------------------------------------- PEP 275: Switching on Multiple Values ------------------------------------- Thomas Lee offered up a `patch implementing the switch statement`_ from `PEP 275`_. People brought up a number of concerns with the implementation (and the switch statement in general). The implementation didn't allow for any way of allowing multiple values to be mapped to the same case (without repeating the code in the case). The implementation also made the switch statement essentially syntactic sugar for a series of if/elif/else statements, and people were concerned that just adding another way to write if/elif/else was not much of a gain for Python. The discussion continued on into the next fortnight. .. _patch implementing the switch statement: http://bugs.python.org/1504199 .. _PEP 275: http://www.python.org/dev/peps/pep-0275/ Contributing thread: - `Switch statement `__ --------------------------------------------------------- The period of the random module's random number generator --------------------------------------------------------- Alex Martelli noticed a note in random.shuffle.__doc__ which said that most permutations of a long sequence would never be generated due to the short period of the random number generator. This turned out to be an artifact from back when Python used the Whichman-Hill generator instead of the Mersenne Twister generator it uses currently. There was some discussion as to whether the comment should be removed or updated, and Robert Kern pointed out that at sequence lengths of 2081 or greater, the comment was still true. Tim Peters decided it was best to just remove the comment, explaining that "anyone sophisticated enough to *understand* an accurate warning correctly would have no need to be warned". Contributing thread: - `a note in random.shuffle.__doc__ ... `__ ------------------------------------------------------- Pre-PEP: Allow Empty Subscript List Without Parentheses ------------------------------------------------------- Noam Raphael presented a `pre-PEP for empty subscript lists`_ in getitem-style access to objects. This would allow zero-dimensional arrays to work in a similar manner to all other N dimensional arrays, and make all of the following equivalences hold:: x[i, j] <--> x[(i, j)] x[i,] <--> x[(i,)] x[i] <--> x[(i)] x[] <--> x[()] Most people felt that zero-dimensional arrays were uncommon enough that either they could be replaced with simple names, e.g. ``x``, or could use the currently available syntax, i.e. ``x[()]``. Zero-dimensional arrays are even uncommon in numpy_ where, after `rehashing the issue`_ innumerable times, zero-dimensional arrays have been almost entirely replaced with scalars. .. _pre-PEP for empty subscript lists: http://wiki.python.org/moin/EmptySubscriptListPEP .. _numpy: http://numeric.scipy.org/ .. _rehashing the issue: http://projects.scipy.org/scipy/numpy/wiki/ZeroRankArray Contributing thread: - `Pre-PEP: Allow Empty Subscript List Without Parentheses `__ ---------------------------------------------- PEP 337: Logging Usage in the Standard Library ---------------------------------------------- For the `Google Summer of Code`_, Jackilyn Hoxworth has been working on implementing parts of `PEP 337`_ to use the logging module in parts of the stdlib. When Jim Jewett, who is mentoring her, brought up a few issues, people got concerned that this work was being done at all, being that `PEP 337`_ has not been approved. Jim and A.M. Kuchling clarified that the goal of Jackilyn's work is to both clarify the PEP (e.g. determine exactly which modules would benefit from logging) and to provide an implementation that can be tweaked as necessary if the PEP is accepted. For the first draft at least, it looked like Jackilyn would keep things simple -- using "py." + __name__ for the logger name, not adding any new logging messages, not changing any message formats, and generally aiming only to give stderr and stdout messages across different modules a common choke point. .. _Google Summer of Code: http://code.google.com/soc/ .. _PEP 337: http://www.python.org/dev/peps/pep-0337/ Contributing thread: - `Stdlib Logging questions (PEP 337 SoC) `__ ------------------- inspect.isgenerator ------------------- Michele Simionato asked for a new function in the inspect module that would identify a function as being a generator function. Phillip J. Eby pointed out that any function can return a generator-iterator (though generator functions are of course guaranteed to do so) and suggested that the perceived need for this inspect function was misguided. Michele agreed and withdrew the proposal. Contributing threads: - `feature request: inspect.isgenerator `__ - `feature request: inspect.isgenerator `__ -------------------------------- Unescaping entities with sgmllib -------------------------------- Sam Ruby asked why sgmllib unescapes entities selectively, not all or nothing (which would be easier to work around), and Fred L. Drake, Jr. explained that sgmllib is really only intended as support for htmllib. Sam suggested isolating the code that attempts to resolve character references into a single method so that subclasses could override this behavior as needed. Martin v. L?wis agreed that this seemed reasonable, though he suggested two functions, one for character references and one for entity references. Sam implemented the suggested behavior and provided a `patch to sgmllib`_. .. _patch to sgmllib: http://bugs.python.org/1504676 Contributing thread: - `sgmllib Comments `__ ------------------------------------------------------------------------- Scoping vs augmented assignment vs sets (Re: 'fast locals' in Python 2.5) ------------------------------------------------------------------------- A bug in Python 2.5 that did not detect augmented assignment as creating a local name allowed code like the following to work:: >>> g = 1 >>> def f1(): ... g += 1 ... >>> f1() >>> g 2 This of course started the usual discussion about giving Python a way to rebind names in enclosing scopes. Boris Borcic in particular was hoping that the bug could be considered a feature, but Terry Reedy explained that Python was not willing to give up the near equivalence between ``x = x + 1`` and ``x += 1``. Since the former creates a local name, the latter ought to do the same thing. The thread seemed like it might drift on further until Guido cut it off, pronouncing that the behavior of augmented assignments creating local names was not going to change. Contributing threads: - `'fast locals' in Python 2.5 `__ - `Scoping vs augmented assignment vs sets (Re: 'fast locals' in Python 2.5) `__ - `Comparing closures and arguments (was Re: Scoping vs augmented assignment vs sets (Re: 'fast locals' in Python 2.5) `__ - `The baby and the bathwater (Re: Scoping, augmented assignment, 'fast locals' - conclusion) `__ --------------------------------------- Checking out an older version of Python --------------------------------------- Skip Montanaro asked about checking out a particular version of Python. Oleg Broytmann and Tim Peters explained that tags are just directories in Subversion, and you can vew all the existing ones and their corresponding revision numbers at http://svn.python.org/projects/python/tags/. Oleg also explained that the difference between:: svn switch svn+ssh://pythondev at svn.python.org/python/tags/r242 and noting that the r242 tag corresponds to revision 39619 and doing:: svn up -r 39619 is that with the latter, commits will go to the trunk (assuming the update was performed on a trunk checkout), while with the former, updates will go to the appropriate tag or branch. Giovanni Bajo provided a nice explanation of this, describing Subversion's 2D coordinate system of [url, revision] and Skip added the explanation to the `Development FAQ`_. .. _Development FAQ: http://www.python.org/dev/faq/ Contributing thread: - `Subversion repository question - back up to older versions `__ -------------------- Source control tools -------------------- In the externally maintained packages discussion, Guido suggested offhand that some other version control project might make it easier to resolve some of the issues. Thomas Wouters put forward a number of considerations. On the negative side of changing to one of the newer version control systems: * Workflow would have to change somewhat to use most of the new branch-oriented systems. * Everyone would have to download the whole repository (at least once) since with the newer systems everyone usually has their own repository. But on the positive side: * History can be preserved for merges of brances (unlike Subversion), which is a big gain for when the trunk is switched to 3.0. Thomas tried importing the Python repository into a number of different systems, and after playing around with them, concluded that in the short term, none of the other version control systems were quite ready yet, though he seemed optimistic for them in the next few years. He also promised to publish imports of the Python repository into Git, Darcs, Mercurial, Bazaar-NG and Monotone somewhere once he was able to successfully import them all. Contributing thread: - `Source control tools `__ ---------------------------------------- Is implicit underscore assignment buggy? ---------------------------------------- Raymond Hettinger noted that in the interactive interpreter, an expression that returns None not only suppresses the printing of that None, but also suppresses the assignment to ``_``. Raymond asked if this was intentional as it makes code like the following break:: >>> import re, string >>> re.search('lmnop', string.letters) <_sre.SRE_Match object at 0xb6f2c480> >>> re.search('pycon', string.letters) >>> if _ is not None: ... print _.group() lmnop Fredrik Lundh pointed out that users just need to recognize that the ``_`` holds the most recently *printed* result. Guido pronounced that this would not change. Terry Reedy suggested adding some documentation for this behavior to either Language Reference 2.3.2 Reserved Classes of Identifiers and/or to Tutorial 2.1.2 Interactive Mode, but it was unclear if any doc changes were committed. Contributing thread: - `Is implicit underscore assignment buggy? `__ ----------------------- Removing MAC OS 9 cruft ----------------------- A number of old MAC OS 9 bits and pieces that are no longer used were removed: * IDE scripts * MPW * Tools/IDE * Tools/macfreeze * Unsupported/mactcp/dnrglue.c * Wastemods This should solve some problems for Windows checkouts where files with trailing dots are not supported. Contributing threads: - `Removing Mac OS 9 cruft `__ - `Mac/wastemodule build failing `__ ------------------------------------------ Fixing buffer object's char buffer support ------------------------------------------ Brett Cannon found that ``import array; int(buffer(array.array('c')))`` caused the interpreter to segfault because buffer objects were redirecting tp_as_buffer->bf_getcharbuffer to the wrong tp_as_buffer slot. Brett fixed the bug and updated the docs a bit to clarify what was intended for the implementation, but kept changes pretty minimal as Python 3.0 will ditch buffer for the bytes type anyway. Contributing threads: - `How to fix the buffer object's broken char buffer support `__ - `Is "t#" argument format meant to be char buffer, or just read-only? `__ ------------------------------- Importing subpackages in Jython ------------------------------- In Jython 2.1, importing a module makes all subpackages beneath it available, unlike in regular Python, where subpackages must be imported separately. Samuele Pedroni explained that this was intentional so that imports in Jython would work like imports in Java do. Guido suggested that having imports work this way in Jython was fine as long as a Java package was being imported, but when a Python package was being imported, Jython should use the Python semantics. Contributing thread: - `Import semantics `__ --------------------------------------------- RFC 3986: Uniform Resource Identifiers (URIs) --------------------------------------------- There was some continued discussion of Paul Jimenez's proposed `uriparse module`_ which more faithfully implements `RFC 3986`_ than the current urlparse module. Nick Coghlan submitted an `alternate implementation`_ that kept all parsed URIs as (scheme, authority, path, query, fragment) tuples by allowing some of these elements to be non-strings, e.g. authority could be a (user, password, host, port) tuple, and path could be a (user, host) tuple. People seemed to like Nick's implementation, but no final decision on the module was made. .. _uriparse module: http://bugs.python.org/1462525 .. _RFC 3986: http://www.ietf.org/rfc/rfc3986.txt .. _alternate implementation: http://bugs.python.org/1500504 Contributing thread: - `Some more comments re new uriparse module, patch 1462525 `__ ----------------------------------------------------- False instead of TypeError for frozenset.__contains__ ----------------------------------------------------- Collin Winter suggested that code like ``{} in frozenset([1, 2, 3])`` should return False instead of raising a TypeError. Guido didn't like the idea because he thought it would mask bugs where, say, a user-defined __hash__() method accidentally raised a TypeError. Contributing thread: - `Unhashable objects and __contains__() `__ -------------------------------------------- IOError or ValueError for invalid file modes -------------------------------------------- Kristj?n V. J?nsson asked why open()/file() throws an IOError for an invalid mode string instead of a ValueError. Georg Brandl explained that either an IOError or a ValueError can be raised depending on whether the invalid mode was detected in Python's code or in the OS's fopen call. Guido suggested that this couldn't really be fixed until Python gets rid of its stdio-based implementation in Python 3.0. Contributing thread: - `file() `__ --------------------------------------------------------------- [Python-checkins] r46603 - python/trunk/Lib/test/test_struct.py --------------------------------------------------------------- Martin Blais checked in un-unittestification of test_struct, and a number of people questioned whether that was a wise thing to do. Thomas Wouters suggested that unittest should merge as many features from py.test_ as possible. This would reduce some of the class-based boilerplate currently required, and also allow some nice additional features like test cases generated on the fly. He didn't get much of a response though, so it was unclear what the plans for Python 2.6 were. .. _py.test: http://codespeak.net/py/current/doc/test.html Contributing thread: - `[Python-checkins] r46603 - python/trunk/Lib/test/test_struct.py `__ ----------------------------------------------- Should hex() yield 'L' suffix for long numbers? ----------------------------------------------- Ka-Ping Yee asked why hex() and oct() still produced an 'L' suffix for long numbers even now that ints and longs have basically been unified. `PEP 237`_ had mentioned the removal of this suffix, but not given it a specific release for removal, so people decided it was best to wait until Python 3.0 when the 'L' suffix will also be removed from repr(). .. _PEP 237: http://www.python.org/dev/peps/pep-0237/ Contributing thread: - `Should hex() yield 'L' suffix for long numbers? `__ --------------------------------- Adding an index of Python symbols --------------------------------- Terry Reedy suggested adding a page to the Python Language Reference index that would list each symbol in Python (e.g. ``()``, ``[]`` and ``@``) along with the places in the documentation where it was discussed. Terry promised to submit a plain-text version in time for the Python 2.5 release, so that someone could convert it to LaTeX and merge it into the docs. Contributing thread: - `Symbol page for Language Reference Manual Index `__ ------------------------------------------ Behavior of searching for empty substrings ------------------------------------------ Fredrik Lundh resolved the issues discussed previously with searching for an empty substring at a position past the end of the string. The current behavior looks like:: >>> "ab".find("") 0 >>> "ab".find("", 1) 1 >>> "ab".find("", 2) 2 >>> "ab".find("", 3) -1 Both Tim Peters and Guido applauded the final resolution. Contributing thread: - `Search for empty substrings (was Re: Let's stop eating exceptions in dict lookup) `__ ----------------- subprocess.IGNORE ----------------- Martin Blais asked about adding subprocess.IGNORE along the lines of subprocess.PIPE which would ignore the child's output without being susceptible to buffer deadlock problems. Under Unix, IGNORE could be implemented as ``open('/dev/null', 'w')``, and on Windows, ``open('nul:', 'w')``. People seemed to think this was a useful feature, but at the time of this summary, no patch had yet been provided. Contributing thread: - `subprocess.Popen(.... stdout=IGNORE, ...) `__ ================ Deferred Threads ================ - `Improve error msgs? `__ - `Keeping interned strings in a set `__ - `Documentation enhancement: "MS free compiler"? `__ - `Code coverage reporting. `__ - `Numerical robustness, IEEE etc. `__ ================== Previous Summaries ================== - `Let's stop eating exceptions in dict lookup `__ - `ssize_t question: longs in header files `__ - `ssize_t: ints in header files `__ - `zlib module doesn't build - inflateCopy() not found `__ =============== Skipped Threads =============== - `Segmentation fault of Python if build on Solaris 9 or10 with Sun Studio 11 `__ - `Possible bug in complexobject.c (still in Python 2.5) `__ - `[Python-checkins] r46300 - in python/trunk: Lib/socket.py Lib/test/test_socket.py Lib/test/test_struct.py Modules/_struct.c Modules/arraymodule.c Modules/socketmodule.c `__ - `test_struct failure on 64 bit platforms `__ - `string inconsistency `__ - `S/390 buildbot URLs problematic `__ - `SF patch #1473257: "Add a gi_code attr to generators" `__ - `test_unicode failure on MIPS `__ - `valgrind report `__ - `test_ctypes failures on ppc64 debian `__ - `Request for patch review `__ - `patch #1454481 vs buildbot `__ - `Seeking Core Developers for Vancouver Python Workshop `__ - `[Python-checkins] Python Regression Test Failures refleak (1) `__ - `Include/structmember.h, Py_ssize_t `__ - `DC Python sprint on July 29th `__ - `tarfile and unicode filenames in windows `__ - `[Python-checkins] buildbot warnings in hppa Ubuntu dapper trunk `__ - `-Wi working for anyone else? `__ - `Inject some tracing ... `__ - `Segmentation fault in collections.defaultdict `__ - `Add pure python PNG writer module to stdlib? `__ - `crash in dict on gc collect `__ - `"can't unpack IEEE 754 special value on non-IEEE platform" `__ - `socket._socketobject.close() doesn't really close sockets `__ - `DRAFT: python-dev summary for 2006-04-16 to 2006-04-30 `__ - `[Python-checkins] r46795 - in python/trunk: Doc/lib/libstdtypes.tex Lib/test/string_tests.py Misc/NEWS Objects/stringobject.c Objects/unicodeobject.c `__ - `xrange vs. int.__getslice__ `__ - `request for review: patch 1446489 (zip64 extensions in zipfile) `__ - `DRAFT: python-dev summary for 2006-05-01 to 2006-05-15 `__ - `pychecker warnings in Lib/encodings `__ - `Moving PEP 343 to Final `__ - `Python sprint at Google Aug. 21-24 `__ - `Long options support `__ - `High Level Virtual Machine `__ - `sqlite3 test errors - was : Re: [Python-checkins] r46936 - in python/trunk: Lib/sqlite3/test/regression.py Lib/sqlite3/test/types.py Lib/sqlite3/test/userfunctions.py Modules/_sqlite/connection.c Modules/_sqlite/cursor.c Modules/_sqlite/module.c Modules/_sqlite/module.h `__ - `[Python-checkins] sqlite3 test errors - was : Re: r46936 - in python/trunk: Lib/sqlite3/test/regression.py Lib/sqlite3/test/types.py Lib/sqlite3/test/userfunctions.py Modules/_sqlite/connection.c Modules/_sqlite/cursor.c Modules/_sqlite/module.c Modules/_sqlite/module.h `__ - `[Python-checkins] sqlite3 test errors - was : Re: r46936 - in python/trunk: Lib/sqlite3/test/regression.py Lib/sqlite3/test/types.py Lib/sqlite3/test/userfunctions.py Modules/_sqlite/connection.c Modules/_sqlite/cursor.c Modules/_sqlite/module.c `__ - `[Python-checkins] sqlite3 test errors - was : Re: r46936 - in python/trunk: Lib/sqlite3/test/regression.py Lib/sqlite3/test/types.py Lib/sqlite3/test/userfunctions.py Modules/_sqlite/connection.c Modules/_sqlite/cursor.c Modules/_sql `__ - `Last-minute curses patch `__ - `DRAFT: python-dev summary for 2006-05-16 to 2006-05-31 `__ - `Bug: xml.dom.pulldom never gives you END_DOCUMENT events with an Expat parser `__ - `Misleading error message from PyObject_GenericSetAttr `__ - `About dynamic module loading `__ From jcarlson at uci.edu Mon Jul 3 07:15:39 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 02 Jul 2006 22:15:39 -0700 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <44A86ECF.4060705@canterbury.ac.nz> References: <20060701013710.10D7.JCARLSON@uci.edu> <44A86ECF.4060705@canterbury.ac.nz> Message-ID: <20060702211647.10F1.JCARLSON@uci.edu> Greg Ewing wrote: > Josiah Carlson wrote: > > If the only code that benefits from such changes are "very *simple*", > > then I think that says something about its necessity. > > The point is that they're only "very simple" if you > can write them using access to an outer scope. Without > that ability, they become less simple, less efficient, > more convoluted, harder to follow, etc. As is known and has been stated, assigning to a parent scope can be emulated in various ways, either through an explicit namespace object, or through a namespace list. > Also I don't buy the argument that something has to > be useful for big, complicated things in order to be > worth having in the language. I never claimed that something needed to be useful for "big, complicated things" in order to be worth having in the language. To be explicit, if nontrivial code isn't improved, that doesn't necessarily mean that the feature is useless. However, if the feature is really only useful for generally trivial cases *without* the feature, then making them even more trivial, I think, is a bit of over optimization. - Josiah From guido at python.org Mon Jul 3 07:25:21 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Jul 2006 07:25:21 +0200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <20060702211647.10F1.JCARLSON@uci.edu> References: <20060701013710.10D7.JCARLSON@uci.edu> <44A86ECF.4060705@canterbury.ac.nz> <20060702211647.10F1.JCARLSON@uci.edu> Message-ID: On 7/3/06, Josiah Carlson wrote: > > Greg Ewing wrote: > > Josiah Carlson wrote: > > > If the only code that benefits from such changes are "very *simple*", > > > then I think that says something about its necessity. > > > > The point is that they're only "very simple" if you > > can write them using access to an outer scope. Without > > that ability, they become less simple, less efficient, > > more convoluted, harder to follow, etc. > > As is known and has been stated, assigning to a parent scope can be > emulated in various ways, either through an explicit namespace object, or > through a namespace list. And the fact that this desire and need remains, even amongst people who should know better, suggests that it may be worth supporting it more directly, as the current work-arounds ain't pretty. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Mon Jul 3 08:08:33 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 02 Jul 2006 23:08:33 -0700 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: References: <20060702211647.10F1.JCARLSON@uci.edu> Message-ID: <20060702225058.10F4.JCARLSON@uci.edu> "Guido van Rossum" wrote: > > On 7/3/06, Josiah Carlson wrote: > > > > Greg Ewing wrote: > > > Josiah Carlson wrote: > > > > If the only code that benefits from such changes are "very *simple*", > > > > then I think that says something about its necessity. > > > > > > The point is that they're only "very simple" if you > > > can write them using access to an outer scope. Without > > > that ability, they become less simple, less efficient, > > > more convoluted, harder to follow, etc. > > > > As is known and has been stated, assigning to a parent scope can be > > emulated in various ways, either through an explicit namespace object, or > > through a namespace list. > > And the fact that this desire and need remains, even amongst people > who should know better, suggests that it may be worth supporting it > more directly, as the current work-arounds ain't pretty. Perhaps not pretty, but not wholly ugly either. Or expressed another way, it's a wart, but the wart isn't 1" across on a forehead, it's fairly small and tucked away on an elbow. I had hoped that there would be a response to my second (and I believe more applicable statement); "if the feature is really only useful for generally trivial cases *without* the feature, then making them even more trivial, I think, is a bit of over optimization." As for a solution, I find the "global means 'not local'" proposition is the least undesireable of the possibilities. It suffers from a change in semantics and potential name masking issues, but I don't believe these are any more serious than normal global masking for the former, and the latter is solvable with a __future__ (at least for 2.6). I'm a solid -0 on this particular proposition, which is far better than the -1 I am on all of the other recent lexical scoping propositions. - Josiah From martin at v.loewis.de Mon Jul 3 08:19:14 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 03 Jul 2006 08:19:14 +0200 Subject: [Python-Dev] Proposal to eliminate PySet_Fini In-Reply-To: <20060627184404.GL10485@performancedrivers.com> References: <20060627184404.GL10485@performancedrivers.com> Message-ID: <44A8B6E2.2040003@v.loewis.de> Jack Diederich wrote: > PyObject_MALLOC does a good job of reusing small allocations but it > can't quite manage the same speed as a free list, especially for things that > have some extra setup involved (tuples have a free list for each length). I would question that statement, for any practical purposed. The cost of tuple comes from setting the elements to NULL, and that has to be done regardless of whether they were allocated new or came from the list. Likewise, the GC management has to be done regardless. So I expect that the speedup is rather minor, and not worth it. Regards, Martin From talin at acm.org Mon Jul 3 09:27:04 2006 From: talin at acm.org (Talin) Date: Mon, 03 Jul 2006 00:27:04 -0700 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <20060702225058.10F4.JCARLSON@uci.edu> References: <20060702211647.10F1.JCARLSON@uci.edu> <20060702225058.10F4.JCARLSON@uci.edu> Message-ID: <44A8C6C8.3030401@acm.org> Josiah Carlson wrote: > I had hoped that there would be a response to my second (and I believe > more applicable statement); "if the feature is really only useful for > generally trivial cases *without* the feature, then making them even > more trivial, I think, is a bit of over optimization." It really depends on how common the trivial case is. In other words, multiply the savings for each occurance times the number of occurances. (Unfortunately, I don't know what units to measure said savings in - is there a unit of 'mental disconnect' or unintuitiveness? :) In an idealy world, the language would allow everything to be said in the most comprehensible way possible. Longer and more verbose ways of stating something are at an inherent disadvantage in this, simply because of the time it takes to scan and absorb the information by the human brain. However, losing excess syntax has to be done in a way that doesn't also lose information. Highly compressed representations of a concept may require such a level of abstraction that it is as much work to puzzle out their meaning as it would be to read the longer version and more. To put it another way - I am an advocate of applying Claude Shannon's theory of information to language design. The highest level of compression should be used for expressions that occur the most frequently. > As for a solution, I find the "global means 'not local'" proposition is > the least undesireable of the possibilities. It suffers from a change > in semantics and potential name masking issues, but I don't believe > these are any more serious than normal global masking for the former, > and the latter is solvable with a __future__ (at least for 2.6). I'm a > solid -0 on this particular proposition, which is far better than the -1 > I am on all of the other recent lexical scoping propositions. I'd say that the more common case is where you want global to really mean global - that is, you want to be able to write to some module-level variable, regardless of how deeply nested your function scope is. While being able to access the 'next outer scope' is occasionally useful, it's not all that common. So changing the behavior of 'global' in this case would be both confusing (since it no longer means 'global'), and less useful (because it doesn't match the most common case.) (This assumes that I haven't completely understood the meaning of the phrase 'not local' - I assumed that it means 'not defined in this scope') Of course, the reason why it's not all that common may be because of the fact that it's not as easy to do, and so people tend to (consciously or otherwise) avoid that pattern in their designs. That being said, I don't think that's necessarily such a bad thing. Python isn't Scheme, and the scoping rules of Python are IMHO more oriented towards practicality and common sense than theoretical purity. This is why I'm not bothered by the fact that Python doesn't create a new scope for loop statements and such. Most of the time, this is what you want. It does mean that you need to name all of your variables uniquely, but that's good programming style in any case. The same is true for local variables not needing to be specially declared as 'my' or 'var' - most of the time, a local variable is what you want. On the other hand, the thing about theoretical purity is that it can be so mouth-wateringly powerful at times. For example, a language that supports closures is, IMHO, at least twice as powerful as a language that doesn't -- because you can use them in so many different and interesting ways. OK, so about the lexical scoping issue - let me brainstorm a moment: One idea would be to introduce the keyword 'local' which would have the effect of capturing any 'global' statements in any enclosing scope. So for example: f = 1 def a(): local f f = 2 def b(): global f f = 3 So in this case, the 'global' statement, which would normally associate 'f' with the outermost (module-level) scope, would instead associate 'f' with the innermost 'local' declaration of that variable. So in the above example, assigning 3 to f assigns it to the middle scope, but does not affect the module-level definition. Admittedly, that's a bit confusing and also verbose, considering that you are not only adding an extra keyword, but also using two statements to specify the home of one variable. Another alternative would be a way to declare an explicitly scoped variable. Lets use the keyword 'my' to indicate this: f = 1 def a(): my f = 2 def b(): f = 3 In this case, what the 'my' statement is doing is indicating that this scope 'owns' the definition of 'f' -- in other words, the definition is hoisted out of any enclosed scopes. So again, in the above example, the innermost assignment will be to the definition of 'f' in the middle scope. What's interesting about this is that you can use the same method with globals: my f = 1 def a(): f = 2 a() print f # prints '2' So again, you are indicating that the global scope 'owns' the definition of 'f', and any enclosed scopes should use that definition, and not create their own. Of course, if you really *do* need to have your own version, you can always override the 'my' statement with another 'my' statement: my f = 1 def a(): my f = 2 a() print f # prints '1' The 'my' statement essentially changes the scoping rules for all variables of that name, within the defining scope and all enclosed scopes. Of course, you can also override this behavior using the 'global' statement, which does exactly what it does now - makes the reference global (i.e. module-level): my f = 1 def a(): global f f = 2 a() print f # prints '2' All right, I'm pretty happy with that. Brainstorming done. :) -- Talin From tim.peters at gmail.com Mon Jul 3 10:14:42 2006 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 3 Jul 2006 04:14:42 -0400 Subject: [Python-Dev] Proposal to eliminate PySet_Fini In-Reply-To: <44A8B6E2.2040003@v.loewis.de> References: <20060627184404.GL10485@performancedrivers.com> <44A8B6E2.2040003@v.loewis.de> Message-ID: <1f7befae0607030114h49fb33edwa61bc5e55ec3b569@mail.gmail.com> [Jack Diederich] >> PyObject_MALLOC does a good job of reusing small allocations but it >> can't quite manage the same speed as a free list, especially for things that >> have some extra setup involved (tuples have a free list for each length). [Martin v. L?wis] > I would question that statement, for any practical purposed. The cost of > tuple comes from setting the elements to NULL, and that has to be done > regardless of whether they were allocated new or came from the list. Except _that_ overhead is trivial for small tuples, and small tuples are the only kind the free lists cache. There are many other overheads. If a tuple is taken off a free list, we get to skip integer multiplication and division checking for overflow before calling PyObject_GC_NewVar. We also get to skip the call to PyObject_GC_NewVar. That in turns skips another integer multiplication in the _PyObject_VAR_SIZE macro, and a call to _PyObject_GC_Malloc. That it turn skips a call to PyObject_MALLOC, and conditionals checking whether it's time to trigger a gc collection. All of that is highly significant compared to the cost of setting at most a handful of slots to NULL inline. > Likewise, the GC management has to be done regardless. _PyObject_GC_TRACK expands to 5 inlined simple stores, and a predictable branch, so it is often more expensive than setting the tuple slots to NULL. But, as above, we get to skip three layers of function call and "will it overflow?" arithmetic in the service of _setting up_ an object for gc initially. Only the gc track/untrack pair remains for tuples in a free list. > So I expect that the speedup is rather minor, and not worth it. Depends on the app :-) Here's a test case that gets supernatural benefit from small-tuple caching: """ def doit(): N1000 = [None] * 1000 basetup = (5,) for i in N1000: tups = [] push = tups.append for j in xrange(10): for k in N1000: push(basetup * j) from time import clock as now times = [] for i in range(3): start = now() doit() finish = now() times.append(finish - start) print sorted(times) """ With current trunk that printed [2.9363677646013846, 2.9489729031005703, 2.9689538729183949] After changing #define MAXSAVEDTUPLES 2000 to #define MAXSAVEDTUPLES 0 the times zoomed to [4.5894824930441587, 4.6023111649343242, 4.629560027293957] That's pretty dramatic. OTOH, I don't have any apps that do that <0.5 wink>, and there's another downside: on SF recently, someone complained that the 2.5 obmalloc work to release unused arenas wasn't doing much in his (perhaps equally artificial -- don't know) test case. It surprised me too, so I dug into it. The problem turned out to be that piles of arenas were being kept "artificially" alive because obmalloc was the original source of thousands of tuple objects being kept alive (from obmalloc's POV) in tupleobject.c's free lists. If you're unlucky, it only takes one tiny tuple in one free list to keep an entire 256KB arena alive -- and if you're very unlucky, you manage to allocate objects in such a way that this happens repeatedly. His test also created lots of dicts along the way, and each arena got mostly filled up with dict objects and a relative handful of small tuples. By the time all of this became trash, the tuples were spread over a few hundred areans, which effectively became immortal. While it doesn't make any real sense, I've seen repeatedly that users _try_ calling gc.collect() in such cases (doesn't make sense because it has nothing to do with cyclic gc). But that suggests it could be a _pragmatic_ win to add an internal "clear all known free lists" function, which gc could call at times, or even just be forced by the user via a `gc` entry point or gc.collect() option. The immortal & unbounded int and float free lists don't cause obmalloc arenas to stay alive unduly, because they get their memory in chunks straight from the system malloc(). But they have to be the cause of the most frequent source of complaints from newbies and Zope users ;-), who do things like range(10000000) and then marvel that some 120MB of VM is still being used after nuking the list and doing a gc.collect(). I get weary of explaining that one :-(. From greg.ewing at canterbury.ac.nz Mon Jul 3 11:19:14 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Jul 2006 21:19:14 +1200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <20060702225058.10F4.JCARLSON@uci.edu> References: <20060702211647.10F1.JCARLSON@uci.edu> <20060702225058.10F4.JCARLSON@uci.edu> Message-ID: <44A8E112.1050309@canterbury.ac.nz> Josiah Carlson wrote: > I had hoped that there would be a response to my second (and I believe > more applicable statement); "if the feature is really only useful for > generally trivial cases *without* the feature, then making them even > more trivial, I think, is a bit of over optimization." I don't think "trivial" is the right word to use here, since it implies something that's of so little importance that it can be ignored. But the simple cases are precisely the ones where this wart hurts the most, so we can't ignore them. Arguments that a feature is undesirable because this or that workaround exists seem like post-hoc justifications to me. Think about it the other way around -- if writing to outer scopes had been straightforward from the beginning, would you be arguing for the *removal* of that ability? Would it even have occurred to anyone to do such a thing? -- Greg From greg.ewing at canterbury.ac.nz Mon Jul 3 11:50:06 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Jul 2006 21:50:06 +1200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <44A8C6C8.3030401@acm.org> References: <20060702211647.10F1.JCARLSON@uci.edu> <20060702225058.10F4.JCARLSON@uci.edu> <44A8C6C8.3030401@acm.org> Message-ID: <44A8E84E.2040806@canterbury.ac.nz> Talin wrote: > To put it another way - I am an advocate of applying Claude Shannon's > theory of information to language design. The highest level of > compression should be used for expressions that occur the most frequently. I believe the proposal in question would cause no net worsening in this information content, and may actually improve it slightly, due to allowing a few things to be written in a shorter and clearer way, while allowing the vast majority of existing things to be written in exactly the same way. > I'd say that the more common case is where you want global to really > mean global - that is, you want to be able to write to some module-level > variable, regardless of how deeply nested your function scope is. It would still mean that, except in the (expected to be *extremely* rare) case where you happened to have a variable with the same name assigned in some intermediate scope. Such cases would be easily fixed by renaming the intermediate variable -- using a name of shorter or equal length, if you like, to keep the information content up. :-) > So changing the behavior of 'global' in this case > would be both confusing (since it no longer means 'global'), An alternative would be to change the keyword as well, to something like 'outer', which would better match its semantics. But if that were done, I would argue for the *removal* of the existing 'global' keyword, which would then be almost completely redundant. This would break large amounts of existing code, however, and it's highly dubious whether that would be worth the small increase in pendantic accuracy, even in Py3k. We're not supposed to be *gratuitously* breaking things in Py3k, after all. > (This assumes that I haven't completely understood the meaning of the > phrase 'not local' - I assumed that it means 'not defined in this scope') Yes, the new meaning would be "in the next outermost scope where there is an assignment to this name, or the module scope if you get that far". > Python isn't Scheme, and the scoping rules of Python are > IMHO more oriented towards practicality and common sense > than theoretical purity. Yes, but I find it hard to regard being *forbidden* from assigning to intermediate scopes as something driven by practical need rather than just being a historical accident. Back when there were strictly two scopes, many people argued themselves blue in the face that this was actually a *good* thing, even if you didn't realise it, and that Python was doing you a favour by enforcing it. Eventually a measure of sanity prevailed, and we got something a lot more like traditional lexical scoping. But one remnant of the old system remained, like a vestigial organ -- the 'global' statement that reaches all the way out to the module scope, regardless of what exists in between. To someone used to lexical scoping in almost any other language that has it, this is a *very* strange and unintuitive thing. Looking back, I think the meaning of 'global' should have been redefined right then at the same time. That would have been the logical and consistent thing to do, and in my opinion would have resulted in a scoping model that was simpler, more useful and no less practical. The most theoretically pure thing would have been to change it to 'outer' at the same time, but that would have broken too much code, and would therefore not have been practical. See, I'm not immune to practicality arguments. :-) > One idea would be to introduce the keyword 'local' which would have the > effect of capturing any 'global' statements in any enclosing scope. That seems unnecessary to me. Or at least not necessary enough to be worth the extra complexity in the scoping model. -- Greg From skip at pobox.com Mon Jul 3 15:01:45 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 3 Jul 2006 08:01:45 -0500 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <20060702225058.10F4.JCARLSON@uci.edu> References: <20060702211647.10F1.JCARLSON@uci.edu> <20060702225058.10F4.JCARLSON@uci.edu> Message-ID: <17577.5433.301972.602166@montanaro.dyndns.org> Josiah> As for a solution, I find the "global means 'not local'" Josiah> proposition is the least undesireable of the possibilities. It Josiah> suffers from a change in semantics and potential name masking Josiah> issues... Pychecker and PyLint both already identify cases where builtins are masked by locals or module globals (and may identify cases where locals mask module globals - I don't recall). I suspect both could be generalized in this regard without a huge effort. That's probably the best place for this sort of warning. Skip From ark at acm.org Mon Jul 3 17:08:18 2006 From: ark at acm.org (Andrew Koenig) Date: Mon, 3 Jul 2006 11:08:18 -0400 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <44A8E112.1050309@canterbury.ac.nz> Message-ID: <003801c69eb2$872dc990$6402a8c0@arkdesktop> > I don't think "trivial" is the right word to use here, > since it implies something that's of so little importance > that it can be ignored. But the simple cases are precisely > the ones where this wart hurts the most, so we can't > ignore them. I'd like to inject an example that might help make this discussion more concrete. Consider the following function: def for_each(seq, f): for i in seq: f(i) I'm sure I've seen more than one instance of someone on comp.lang.python trying to do the equivalent of using a function such as this one to compute the sum of the elements of a sequence as follows: def sum(seq): result = 0 def accum(i): result += i for_each(seq, accum) return result and wonder why it doesn't work. Still odder, why it doesn't work and the following does: def sum(seq): result = [0] def accum(i): result[0] += i for_each(seq, accum) return result[0] Transforming the first definition of sum above into the second may be trivial, but only if you've encountered the technique before. Moreover, the first version of sum uses a technique that is more than 45 years old (!), as it was available to Algol 60 programmers. From martin at v.loewis.de Mon Jul 3 17:10:16 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 03 Jul 2006 17:10:16 +0200 Subject: [Python-Dev] Proposal to eliminate PySet_Fini In-Reply-To: <1f7befae0607030114h49fb33edwa61bc5e55ec3b569@mail.gmail.com> References: <20060627184404.GL10485@performancedrivers.com> <44A8B6E2.2040003@v.loewis.de> <1f7befae0607030114h49fb33edwa61bc5e55ec3b569@mail.gmail.com> Message-ID: <44A93358.3040000@v.loewis.de> Tim Peters wrote: > With current trunk that printed > > [2.9363677646013846, 2.9489729031005703, 2.9689538729183949] > > After changing > > #define MAXSAVEDTUPLES 2000 > > to > > #define MAXSAVEDTUPLES 0 > > the times zoomed to > > [4.5894824930441587, 4.6023111649343242, 4.629560027293957] > > That's pretty dramatic. Interesting. I ran this through gprof, and found the following changes to the number of function calls with-cache without-cache PyObject_Malloc 59058 24055245 tupletraverse 33574 67863194 visit_decref 131333 197199417 visit_reachable 131333 197199417 collect 17 33006 (for reference:) tuplerepeat 30000000 30000000 According to gprof, these functions (excluding tuplerepeat) together account for 40% of the execution time in the without-cache (i.e. MAXSAVEDTUPLES 0) case. So it appears that much of the slowdown in disabling the fast tuple allocator is due to the higher frequency of garbage collection in your example. Can you please re-run the example with gc disabled? Of course, it's really no surprise that GC is called more often: if the tuples are allocated from the cache, that doesn't count as an allocation wrt. GC. It so happens that your example just triggers gc a few times in its inner loop; I wouldn't attribute that overhead to obmalloc per se. Regards, Martin From guido at python.org Mon Jul 3 17:45:49 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Jul 2006 17:45:49 +0200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <003801c69eb2$872dc990$6402a8c0@arkdesktop> References: <44A8E112.1050309@canterbury.ac.nz> <003801c69eb2$872dc990$6402a8c0@arkdesktop> Message-ID: On 7/3/06, Andrew Koenig wrote: > > I don't think "trivial" is the right word to use here, > > since it implies something that's of so little importance > > that it can be ignored. But the simple cases are precisely > > the ones where this wart hurts the most, so we can't > > ignore them. > > I'd like to inject an example that might help make this discussion more > concrete. > > Consider the following function: > > def for_each(seq, f): > for i in seq: > f(i) > > I'm sure I've seen more than one instance of someone on comp.lang.python > trying to do the equivalent of using a function such as this one to compute > the sum of the elements of a sequence as follows: > > def sum(seq): > result = 0 > def accum(i): > result += i > for_each(seq, accum) > return result > > and wonder why it doesn't work. Still odder, why it doesn't work and the > following does: > > def sum(seq): > result = [0] > def accum(i): > result[0] += i > for_each(seq, accum) > return result[0] > > Transforming the first definition of sum above into the second may be > trivial, but only if you've encountered the technique before. Moreover, the > first version of sum uses a technique that is more than 45 years old (!), as > it was available to Algol 60 programmers. Much though the Algol 60 tickles my nostalgia (it was my first programming language!) I don't think that it's a particularly strong argument. I like to think that we have better ways these days. I think you need to come up with a better motivating example; the above is particular un-idiomatic Python. It starts by defining a higher-order function for_each that has little to offer over writing an explicit for loop, and then uses this to motivate writing a simple operation (result += i) as a function instead so that it fits in the inconvenient for_each() API. I understand that both for_each() and accum() are just examples of more complicated functions, but I can't help thinking that the problem here only occurs for very *simple* functions in the place of accum(); a more complicated form of accum would likely be a bound method of a class instance which carries the state. A better way to decompose these kinds of problems is probably by using generators. The equivalent of for_each() would not take a function parameter but *yield* the successive values instead of calling f() with successive values; e.g.: def for_each(seq): for i in seq: yield i Then the sum() function could be written like this: def sum(seq): result = 0 for i in for_each(seq): result += i return result -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ark at acm.org Mon Jul 3 17:56:02 2006 From: ark at acm.org (Andrew Koenig) Date: Mon, 3 Jul 2006 11:56:02 -0400 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: Message-ID: <006c01c69eb9$323f37a0$6402a8c0@arkdesktop> > Much though the Algol 60 tickles my nostalgia (it was my first > programming language!) I don't think that it's a particularly strong > argument. I like to think that we have better ways these days. Even if so, that's not the point I was trying to make. The point is that there is a programming technique that is widely used, works in many languages, and has been around for 45 years; and when you try to use it in Python, it fails. I believe that such failures, even if there are alternative ways of solving the problems that engender them, are barriers to learning that should be removed if it is possible to do so without substantial cost. From theller at python.net Mon Jul 3 18:19:02 2006 From: theller at python.net (Thomas Heller) Date: Mon, 03 Jul 2006 18:19:02 +0200 Subject: [Python-Dev] Moving the ctypes repository to python.org In-Reply-To: <44A57D6D.70701@v.loewis.de> References: <20060623145602.GB10250@niemeyer.net> <449C5157.5050004@v.loewis.de> <44A57D6D.70701@v.loewis.de> Message-ID: <44A94376.2030004@python.net> Martin v. L?wis schrieb: > Thomas Heller wrote: >> - Do I need special rights to call 'svnadmin load' to import this dumpfile >> into Python SVN, or are the normal commit rights sufficient? > > It's called "svnadmin" for a reason :-) > > Neal Norwitz or myself will have to do that; we need to do it on the > repository machine locally. I would likely take subversion write > access offline for the time of the import, so that I can rollback > the entire repository in case of an operator mistake. Please tell me when you or Neal have time for the import. >> What exactly is the URL/PATH where it should be imported (some sandbox, >> I assume)? > > My view is that this is the "projects" repository; with ctypes being a > project, it should go into the root directory (i.e. as a sibling to > python, peps, distutils, stackless, ...). If you prefer to see it in > sandbox, this could work as well. Having it in "projects" is fine, this matches the directory structure that cvs2svn creates (ctypes/trunk, ctypes/tags, ctypes/branches). >> - What about the Python trunk? Should changes from the sandbox be merged >> into Modules/_ctypes and Lib/ctypes, or would it be better (or possible at all) >> to use the external mechanism? > > I would prefer to see two-way merges going on, at least until 2.5 is > released (i.e. no changes to Modules/ctypes except for bug fixes). Ok. > Using svn:external is a risky thing wrt. to branching/tagging: > > When we tag the Python tree, we want to tag the entire source tree. > With svn:external, only the external link would be in the tag, > i.e. later changes to the external link would modify old tags. > This is undesirable. > > This problem could be solved with a versioned external link; > this would mean that ctypes could not be edited directly, but > that one would have to go through the original repository > URL to perform modifications, and then update the external > link. > > So I think I still would prefer two-way merges. There are > tools to make the merges pretty mechanic. I have no experience at all with svn:external, so that's fine with me too. Thanks, Thomas From theller at python.net Mon Jul 3 18:20:09 2006 From: theller at python.net (Thomas Heller) Date: Mon, 03 Jul 2006 18:20:09 +0200 Subject: [Python-Dev] Moving the ctypes repository to python.org In-Reply-To: <44A57D6D.70701@v.loewis.de> References: <20060623145602.GB10250@niemeyer.net> <449C5157.5050004@v.loewis.de> <44A57D6D.70701@v.loewis.de> Message-ID: <44A943B9.9080708@python.net> Martin v. L?wis schrieb: > Thomas Heller wrote: >> - Do I need special rights to call 'svnadmin load' to import this dumpfile >> into Python SVN, or are the normal commit rights sufficient? > > It's called "svnadmin" for a reason :-) > > Neal Norwitz or myself will have to do that; we need to do it on the > repository machine locally. I would likely take subversion write > access offline for the time of the import, so that I can rollback > the entire repository in case of an operator mistake. Please tell me when you or Neal have time for the import. >> What exactly is the URL/PATH where it should be imported (some sandbox, >> I assume)? > > My view is that this is the "projects" repository; with ctypes being a > project, it should go into the root directory (i.e. as a sibling to > python, peps, distutils, stackless, ...). If you prefer to see it in > sandbox, this could work as well. Having it in "projects" is fine, this matches the directory structure that cvs2svn creates (ctypes/trunk, ctypes/tags, ctypes/branches). >> - What about the Python trunk? Should changes from the sandbox be merged >> into Modules/_ctypes and Lib/ctypes, or would it be better (or possible at all) >> to use the external mechanism? > > I would prefer to see two-way merges going on, at least until 2.5 is > released (i.e. no changes to Modules/ctypes except for bug fixes). Ok. > Using svn:external is a risky thing wrt. to branching/tagging: > > When we tag the Python tree, we want to tag the entire source tree. > With svn:external, only the external link would be in the tag, > i.e. later changes to the external link would modify old tags. > This is undesirable. > > This problem could be solved with a versioned external link; > this would mean that ctypes could not be edited directly, but > that one would have to go through the original repository > URL to perform modifications, and then update the external > link. > > So I think I still would prefer two-way merges. There are > tools to make the merges pretty mechanic. I have no experience at all with svn:external, so that's fine with me too. Thanks, Thomas From tomerfiliba at gmail.com Mon Jul 3 19:00:09 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Mon, 3 Jul 2006 19:00:09 +0200 Subject: [Python-Dev] weakattr In-Reply-To: <20060701213745.10E0.JCARLSON@uci.edu> References: <1d85506f0607010649y65c9fb24mbc5c636360970e52@mail.gmail.com> <20060701213745.10E0.JCARLSON@uci.edu> Message-ID: <1d85506f0607031000q578c8dccka762b13ba3413de7@mail.gmail.com> > I like the added functionality offered with weakattrs as defined. I'm > not terribly in love with the syntax of their creation, and I'm curious > as to how it plays with __slots__ weakattrs are data descriptors, just like properties etc. they are part of the class, not the instance, so there shouldn't be any trouble with mixing those with __slots__ moreover, adding those to stdlib is very staight-forward. we don't even need to introduce a new module. if people show interest, i'll write a bit of a longer doc string and add some unit tests (although there's not much to test :) ) > Toss it out in python-list, I think some people over there would be able > to offer more feedback. will do... although i doubt they will offer any -tomer On 7/2/06, Josiah Carlson wrote: > > "tomer filiba" wrote: > > weakattr (weak attributes) are attributes that are weakly referenced > > by their containing object. they are very useful for cyclic references -- > > an object that holds a reference to itself. > > I like the added functionality offered with weakattrs as defined. I'm > not terribly in love with the syntax of their creation, and I'm curious > as to how it plays with __slots__ (not quite having the time to look at > its implementation right now), but it is quite explicit, so I can get > past that. It would allow us to say, "stop using __del__, use weakattrs", > but I'm not sure how well that would work, generally. > > Toss it out in python-list, I think some people over there would be able > to offer more feedback. > > - Josiah > > From facundobatista at gmail.com Mon Jul 3 20:07:20 2006 From: facundobatista at gmail.com (Facundo Batista) Date: Mon, 3 Jul 2006 15:07:20 -0300 Subject: [Python-Dev] Time-out in URL Open Message-ID: I need a timeout in urlopen, just to be able to make: >>> urllib2.urlopen("http://no.host.org", timeout=2) This is actually not possible, but I'll make it work. I want to know, please, if this is useful in general, for me to post a patch in SF. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From fdrake at acm.org Mon Jul 3 20:12:22 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon, 3 Jul 2006 14:12:22 -0400 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: <200607031412.22874.fdrake@acm.org> On Monday 03 July 2006 14:07, Facundo Batista wrote: > I want to know, please, if this is useful in general, for me to post a > patch in SF. It seems like something that should be easy, and lots of people need to consider this for applications. -Fred -- Fred L. Drake, Jr. From martin at v.loewis.de Mon Jul 3 22:27:56 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 03 Jul 2006 22:27:56 +0200 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: <44A97DCC.2040202@v.loewis.de> Facundo Batista wrote: >>>> urllib2.urlopen("http://no.host.org", timeout=2) > > This is actually not possible, but I'll make it work. > > I want to know, please, if this is useful in general, for me to post a > patch in SF. While it might be useful, it can only be added to Python 2.6 now. So take your time with that patch. Regards, Martin From guido at python.org Mon Jul 3 23:00:38 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Jul 2006 23:00:38 +0200 Subject: [Python-Dev] Lexical scoping in Python 3k In-Reply-To: <006c01c69eb9$323f37a0$6402a8c0@arkdesktop> References: <006c01c69eb9$323f37a0$6402a8c0@arkdesktop> Message-ID: On 7/3/06, Andrew Koenig wrote: > > Much though the Algol 60 tickles my nostalgia (it was my first > > programming language!) I don't think that it's a particularly strong > > argument. I like to think that we have better ways these days. > > Even if so, that's not the point I was trying to make. The point is that > there is a programming technique that is widely used, works in many > languages, and has been around for 45 years; and when you try to use it in > Python, it fails. That's true for lots of things that have been around for a long time. Can you provide a better example? (The use of += is not particularly relevant to the example; it could just as well have said "result = result + i".) > I believe that such failures, even if there are alternative ways of solving > the problems that engender them, are barriers to learning that should be > removed if it is possible to do so without substantial cost. And that is of course the crucial question. Probably the only proposal that has any chance of succeeding is to extend the 'global' statement so that it also applies to variables in intermediate outer scopes; or perhaps a new keyword (since "global" is not a very good name for the extended semantics). We would have to decide what this example would do: def outer(): def inner1(x): global a a = x def inner2(): return a return inner1, inner2 f1, f2 = outer() g1, g2 = outer() f1(42) g1(0) print f2() # Does it print 0 or 42 ??? In current Python this prints 0: there's only one (global) variable a, and the call to g1(0) overrides the value that was stored by f1(42). If global were changed to mean "nonlocal" what should it do? The question the example poses is that a is not initialized except in inner1() -- we somehow have to decide whether this is an error, or whether it chooses some well-defined outer scope, with the choices being the nearest enclosing scope or the outermost (truly global) scope. We have one guideline: if there is exactly one outer scope that defines a variable named a, we would like it to be referenced (by the 'global a') statement and the variable references governed by it automatically. Also, of there's more than one such scope, we'd like it to reference the innermost one. But this doesn't have a natural extension to what should happen if there's no such scope! Perhaps the best solution would be to make it an error if there wasn't a visible variable named a in an outer scope. That would suit me fine because I'd like to migrate towards more static analysis of variables anyway. If that means equipping modues with traps for attempts to store arbitrary variables into their namespaces, that's fine with me (as long as there's some escape -- and of course instancs and classes remain fully dynamic). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Jul 3 23:06:02 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Jul 2006 23:06:02 +0200 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: To fake things like this, socket.setdefaulttimeout() was added, though I don't know if it actually works. Have you tried that? --Guido On 7/3/06, Facundo Batista wrote: > I need a timeout in urlopen, just to be able to make: > > >>> urllib2.urlopen("http://no.host.org", timeout=2) > > This is actually not possible, but I'll make it work. > > I want to know, please, if this is useful in general, for me to post a > patch in SF. > > Regards, > > -- > . Facundo > > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Mon Jul 3 23:30:11 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 3 Jul 2006 16:30:11 -0500 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: <17577.35939.965788.902096@montanaro.dyndns.org> Facundo> I need a timeout in urlopen, just to be able to make: >>>> urllib2.urlopen("http://no.host.org", timeout=2) Facundo> This is actually not possible, but I'll make it work. Facundo> I want to know, please, if this is useful in general, for me to Facundo> post a patch in SF. As others have posted, yes, it would be useful for 2.6. However, you should consider how that might be applied to the other Internet service modules (ftplib, telnetlib, urllib, etc). Skip From skip at pobox.com Mon Jul 3 23:31:12 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 3 Jul 2006 16:31:12 -0500 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: <17577.36000.58814.231162@montanaro.dyndns.org> Guido> To fake things like this, socket.setdefaulttimeout() was added, Guido> though I don't know if it actually works. Have you tried that? I'm pretty sure it does, but is a rather blunt instrument for the task, as it affects all socket connections the app might make. Skip From skip at pobox.com Mon Jul 3 23:31:12 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 3 Jul 2006 16:31:12 -0500 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: <17577.36000.58814.231162@montanaro.dyndns.org> Guido> To fake things like this, socket.setdefaulttimeout() was added, Guido> though I don't know if it actually works. Have you tried that? I'm pretty sure it does, but is a rather blunt instrument for the task, as it affects all socket connections the app might make. Skip From jjl at pobox.com Mon Jul 3 23:57:36 2006 From: jjl at pobox.com (John J Lee) Date: Mon, 3 Jul 2006 21:57:36 +0000 (UTC) Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: On Mon, 3 Jul 2006, Guido van Rossum wrote: > To fake things like this, socket.setdefaulttimeout() was added, though > I don't know if it actually works. Have you tried that? [...] It works. I think there's some issue with SSL, though (can't seem to find the issue now). Of course, feeding through the timeout to the individual protocol modules would be a good thing. John From billchi at microsoft.com Tue Jul 4 01:17:36 2006 From: billchi at microsoft.com (Bill Chiles) Date: Mon, 3 Jul 2006 16:17:36 -0700 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <06Jun30.183119pdt."58641"@synergy1.parc.xerox.com> Message-ID: <9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com> For Common Lispers and probably Schemers, Python has some surprising semantics around scope and lifetime extent of variables. Three that leap out at me are: * function parameters with default values are NOT new bindings for each invocation, so a default value of [] changes if you destructively modify this list object in the function * loop variables are NOT distinct lexical variables. The binding gloms on to a variable in the function's scope, both changing that lexical binding and not creating a new one for the loop (that goes away when the loop's scope ends) * loop variables are NOT distinct bindings per iteration, leading to the surprising results below Bill -----Original Message----- From: python-dev-bounces+billchi=microsoft.com at python.org [mailto:python-dev-bounces+billchi=microsoft.com at python.org] On Behalf Of Bill Janssen Sent: Friday, June 30, 2006 6:31 PM To: Giovanni Bajo Cc: Phillip J. Eby; Ka-Ping Yee; Guido van Rossum; Tim Peters; python-dev at python.org Subject: Re: [Python-Dev] 2.5 and beyond > >>> a = [] > >>> for i in range(10): > ... a.append(lambda: i) > ... > >>> print [x() for x in a] > [9, 9, 9, 9, 9, 9, 9, 9, 9, 9] Isn't this exactly what you'd expect? Maybe I've been writing Python for too long... :-). Bill _______________________________________________ Python-Dev mailing list Python-Dev at python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/billchi%40microsoft.co m From jcarlson at uci.edu Tue Jul 4 02:20:28 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 03 Jul 2006 17:20:28 -0700 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com> References: <06Jun30.183119pdt."58641"@synergy1.parc.xerox.com> <9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com> Message-ID: <20060703171546.1108.JCARLSON@uci.edu> "Bill Chiles" wrote: > > For Common Lispers and probably Schemers, Python has some surprising > semantics around scope and lifetime extent of variables. Three that > leap out at me are: One thing to remember is that Python is not Scheme/Lisp. It borrows some ideas from Scheme/Lisp, but that borrowing does not necessitate a it also use a completely equivalent scoping mechanism. From what I have been hearing about Python 2.6, and 3.0, the three "surprises" you describe are not going to be "fixed" (with respect to expected Scheme/Lisp semantics). Feel free to argue as to why they should be "fixed" in Py3k (unless Guido says, "you're dreaming"), but please do so in the py3k list. - Josiah > * function parameters with default values are NOT new bindings for each > invocation, so a > default value of [] changes if you destructively modify this list > object in the function > * loop variables are NOT distinct lexical variables. The binding gloms > on to a variable in the > function's scope, both changing that lexical binding and not creating > a new one for the > loop (that goes away when the loop's scope ends) > * loop variables are NOT distinct bindings per iteration, leading to > the surprising results > below From guido at python.org Tue Jul 4 06:56:41 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Jul 2006 06:56:41 +0200 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <20060703171546.1108.JCARLSON@uci.edu> References: <9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com> <20060703171546.1108.JCARLSON@uci.edu> Message-ID: On 7/4/06, Josiah Carlson wrote: > One thing to remember is that Python is not Scheme/Lisp. It borrows > some ideas from Scheme/Lisp, I can say it stronger. Any resemblance between Python and Scheme or Lisp is purely a coincidence. Neither language is in Python's ancestry, at least not explicitly; I'd never used or tried to learn Scheme when I started Python (still haven't) and my Lisp experience was limited to copying Emacs startup code from friends (still is). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From g.brandl at gmx.net Tue Jul 4 09:03:44 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 04 Jul 2006 09:03:44 +0200 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: <17577.35939.965788.902096@montanaro.dyndns.org> References: <17577.35939.965788.902096@montanaro.dyndns.org> Message-ID: skip at pobox.com wrote: > Facundo> I need a timeout in urlopen, just to be able to make: > > >>>> urllib2.urlopen("http://no.host.org", timeout=2) > > Facundo> This is actually not possible, but I'll make it work. > > Facundo> I want to know, please, if this is useful in general, for me to > Facundo> post a patch in SF. > > As others have posted, yes, it would be useful for 2.6. However, you should > consider how that might be applied to the other Internet service modules > (ftplib, telnetlib, urllib, etc). There was one patch that did this: http://python.org/sf/723312. Georg From talin at acm.org Tue Jul 4 09:59:50 2006 From: talin at acm.org (Talin) Date: Tue, 04 Jul 2006 00:59:50 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) Message-ID: <44AA1FF6.1050501@acm.org> This is sort of a re-do of an earlier proposal which seems to have gotten lost in the shuffle of the larger debate. I propose to create a new type of scoping rule, which I will call "explicit" lexical scoping, that will co-exist with the current "implicit" scoping rule that exists in Python today. Definitions: Implicit scoping is what we have now - a variable is defined within a scope implicitly by assignment. More specifically, when a name is assigned, the name is defined at the innermost function-level scope from where the assignment took place. Explicit scoping is where the programmer explicitly specifies which scope the variable should be defined in. Unlike implicit scoping, assignments to a named variable do not automatically redefine that variable within the current scope. Syntax: Borrowing from Perl, the keyword 'my' is used to declare an explicitly scoped variable: def f1(): my x = 1 def f2(): x = 2 # Does not create a new x In the above example, the statement 'my x = 1' declares that the scope of the variable 'x' is the outer function f1. Any assignment to x will modify the existing x, rather than creating a new definition. Note that the 'my' prefix can be combined with an assignment operation. It is anticipated that the 'my' prefix will be used quite frequently (and encouraged), so it makes sense to cut down on the number of statements by combining declaration and assignment. Explicitly scoped variables can also be declared at the module level: my x = 1 def f1(): x = 2 # Modifies the global X Declaring a module-level variable with an explicit scope eliminates the need for a 'global' statement for that variable. Nested Scopes: Each occurance of the keyword 'my' creates a new scope which hides any outer definitions of the name. So for example: my x = 1 def f1(): my x = 2 # This is a different 'x' than the global def f2(): x = 3 # This is the 'x' defined within f1() Interaction between explicit scoping and globals: The 'global' statement, when used with explicitly scoped variables, means exactly the same as it does with implicitly scoped variables: It allows access to the outermost scope, overriding any intermediate definitions in surrounding scopes: x = 1 def f1(): my x = 2 def f2(): global x x = 3 # This is the module-level 'x' Explicit scoping and code block structure: Implicitly scoped variables are always defined at the nearest enclosing function scope, even if they are created within a code block. It might be worth considering allowing explicitly scoped variables to be defined within other scopes. For example, we might choose to allow explicit scope declarations to be limited to the current suite: def f1(): for x in range(0,10): my y = x*x # A new definition of y for each iteration Note that this is a speculation only, and not a core part of the proposal (so please don't reject the proposal on this one point.) Formal definition: When a value is assigned to a local variable name, the rules for determining which scope the variable will be defined in are as follows: 1) Starting with the current (innermost) scope, examine all of the currently active scopes: 1a) If the current scope contains a 'global' statement for the given name, then set the result scope to the outermost (module-level) scope. 1b) If the current scope contains a 'my' statement for the given name, then set the result scope to the scope in which the 'my' statement occurred. 2) Otherwise, continue until we run out of scopes. If neither a 'global' or 'my' declaration was discovered, then use the innermost scope as the result scope. How is this different from 'outer'? The explicit scope proposal requires that the scope be specified at the place where the variable is *defined* as opposed to where it is *used*. This definition is inherited by all inner scopes. This allows a finer degree of control, for less typing, than the 'outer' proposal. With explicit scoping, there is no confusion as to which scope is being considered; And explicit scoping allows a single declaration of a variable to be shared by many different inner scopes, which would otherwise require a separate 'outer' statement for each one. Explicit scoping and static analysis: It should be easier to do static analysis of code with explicit scoping, since you always know what scope a variable is defined in (as opposed to implicit scoping, where a variable may switch from global to local as a result of an assignment.) Note that this implies that the creation of the scope does not occur at the time of the assignment, but rather at the time the function is entered. Thus: x = 1 def f1(): print x # Error, unassigned value my x = 2 In the above example, even though the 'my' statement occurs after the print, the scope created by the 'my' statement is in effect for the entire function, although the actual *assignment* takes place after the print. The reason for this is that the scope creation is actually done by the compiler. -- Talin From python-dev at zesty.ca Tue Jul 4 12:32:00 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Tue, 4 Jul 2006 05:32:00 -0500 (CDT) Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: References: Message-ID: Hi Brett, Here are some comments on the description of the restricted execution model that you posted. > When referring to the state of an interpreter, it is either "trusted" or > "untrusted". A trusted interpreter has no restrictions imposed upon any > resource. An untrusted interpreter has at least one, possibly more, resource > with a restriction placed upon it. In response to Guido's comment about confusing the words "trusted" and "untrusted", how about "empowered" and "restricted"? > When the Interpreter Is Embedded > ================================ > > Single Untrusted Interpreter > ---------------------------- > > This use case is when an application embeds the interpreter and never has more > than one interpreter running. > > The main security issue to watch out for is not having default abilities be > provided to the interpreter by accident. I'd rather rephrase this in the opposite direction. The onus shouldn't be on the application to hunt down each possible dangerous authority and deactivate them all one by one. The main security issue is to let the application choose which abilities it wants the restricted interpreter to have, and then ensure that the restricted interpreter gets only those abilities. > Multiple Untrusted Interpreters > ------------------------------- > > When multiple interpreters, all untrusted at varying levels, need to be > running within a single application. This is the key use case that this > proposed design is targetted for. > > On top of the security issues from a single untrusted interpreter, > there is one additional worry. Resources cannot end up being leaked > into other interpreters where they are given escalated rights. What is your model here for communication between interpreters? If two interpreters can communicate, any attempt to "prevent leakage" of resources is meaningless. When you say "leaked into other interpreters" are you talking about a Python object leaking or something else at a lower level? Suppose for example that the application wants to embed two interpreters, P and Q, and that the application wants P to be able to write files but Q to be restricted against writing files. When you say "leaked" above, that suggests to me that you want to prevent something like # code running in P import spam f = open('/home/doofus/.ssh/authorized_keys', 'a') spam.f = f # code running in Q import spam spam.f.write('blargh') The above example supposes that P and Q can communicate through a shared module, spam, where they can pass Python objects. But notice that even if you prevent them from passing Python objects like open files, any form of communication is sufficient to leak resources: # code running in P def add_key(key): f = open('/home/doofus/.ssh/authorized_keys', 'a') f.write(key + '\n') f.close() import socket s = socket.socket() s.bind(('', 6666)) s.listen(1) ns, addr = s.accept() add_key(ns.recv(100)) # code running in Q import webbrowser webbrowser.open('http://localhost:6666/zebra') As long as P can listen for instructions from Q, it can give Q the power to write to the filesystem. > Filesystem > =================== > > The most obvious facet of a filesystem to protect is reading from it. > One does not want what is stored in ``/etc/passwd`` to get out. And > one also does not want writing to the disk unless explicitly allowed > for basically the same reason; if someone can write ``/etc/passwd`` > then they can set the password for the root account. There's a big difference between modifying (or erasing) an existing file and writing a new file (e.g. for temporary storage). If i give you a little filesystem of your own to play in, and it starts out empty, you can put whatever you want in it without violating my secrecy or the integrity of my files. I think you should be talking about this in terms of specifically what abilities you want to be able to allow, based on examples of real-life applications. > Physical Resources > =================== > > Memory should be protected. It is a limited resource on the system > that can have an impact on other running programs if it is exhausted. > Being able to restrict the use of memory would help alleviate issues > from denial-of-service (DoS) attacks. > Networking > =================== > > Networking is somewhat like the filesystem in terms of wanting similar > protections. You do not want to let untrusted code make tons of socket > connections or accept them to do possibly nefarious things (e.g., acting > as a zombie). > > You also want to prevent finding out information about the network you are > connected to. This includes doing DNS resolution since that allows one > to find out what addresses your intranet has or what subnets you use. Again, it's risky to describe only individual cases of things to prevent. What networking abilities are safe or necessary for the kinds of applications you have in mind? Start from nothing and work up from there. > Interpreter > =================== > > One must make sure that the interpreter is not harmed in any way. > There are several ways to possibly do this. One is generating > hostile bytecode. Another is some buffer overflow. In general any > ability to crash the interpreter is unacceptable. This is hard for me to understand. What exactly do you trust and not trust? It seems to me that crashing an interpreter is only a problem if a single interpreter is running both trusted and untrusted code -- then if the untrusted code crashes the interpreter, the trusted code suffers. But there doesn't seem to be any such thing in your model. Each interpreter is either trusted or untrusted. If the interpreter is trusted, and the code running in it causes it to crash, i assume you would consider that to be the code's "own fault", right? And if the interpreter is untrusted, and the code running in it causes it to crash, then the code has only harmed itself. It seems to me that we need only be concerned about crashing when the crash of an embedded interpreter will bring down its host application, or there are multiple interpreters embedded at once and one interpreter causes another interpreter to crash. > Resource Hiding > ============================= [...] > This can be viewed as a passive system for security. [...] > Resource Crippling > ============================= > Another approach to security is to provide constant, proactive security > checking of rights to use a resource. I think you have this backwards. Resource hiding is proactive: before untrusted code has a chance to abuse anything, you decide what you want to allow it to do. It defaults to no access, and only gets access to resources you have proactively decided to provide. Resource crippling is the opposite: it begins by giving carte blanche to the untrusted code, then you run around trying to plug holes by stopping everything you don't want. This is a lot more work, and it is also much more dangerous. If you forget to plug even one hole, you're hosed. Back to what you wrote about resource hiding: > This can be viewed as a passive system for security. Once a resource > has been given to code there are no more checks to make sure the > security model is being violated. This last sentence doesn't make any sense. If you decided to give the resource, how is using the resource a violation? Either you want to enable the resource or you don't. If you want to enable it, give it; if you don't, don't give it. As a criticism of the resource hiding approach, it's a red herring -- there's no way to interpret this sentence that doesn't make it also an unfalsifiable criticism of any possible security model. > The most common implementation of resource hiding is capabilities. > In this type of system a resource's reference acts as a ticket that > represents the right to use the resource. Once code has a reference > it is considered to have full use of that resource it represents and > no further security checks are performed. Same thing. What "further security checks" are we worried about? Woult it check to see whether we've authorized the interpreter to have access to the resource ... which we already know to be true? > To allow customizable restrictions one can pass references to wrappers of > resources. This allows one to provide custom security to resources instead of > requiring an all-or-nothing approach. The ability to customize security restrictions is an important advantage of the resource hiding approach, since resource crippling requires that the architect of the security model anticipate every possible security restriction that future programmers might need. Using resource crippling is analogous to removing "def" from the language and requiring Python programmers to only use functions that are provided in the built-in modules instead of writing their own functions. > To use an analogy, imagine you are providing security for your home. > With capabilities, security came from not having any way to know > where your house is without being told where it was; a reference > to its location. You might be able to ask a guard (e.g., Java's > ClassLoader) for a map, but if they refuse there is no way for you > to guess its location without being told. But once you knew where > it was, you had complete use of the house. This analogy is only fair if you compare it to the same analogy for the resource crippling approach. Resource crippling doesn't get you any finer-grained control either! The comparison story is: With resource crippling, security comes from having a guard at the door to your house. When a Python interpreter comes up to the door, the guard checks to see if the interpreter has permission to enter the house, and if it does, then it gets complete use of the house. Why is the granularity of control described as the whole house in the resource-hiding story, but as each door in the house in the resource-crippling story? > And that complete access is an issue with a capability system. > If someone played a little loose with a reference for a resource > then you run the risk of it getting out. Could you be more specific about what you mean by "it getting out"? If you mean getting from a trusted interpreter to an untrusted interpreter -- then how is a resource going to travel between interpreters? Or if not, then are you thinking of a situation in which one piece of code is trusted with the resource, but another piece of code is not, and both are running in the same interpreter? -- ?!ng From fuzzyman at voidspace.org.uk Tue Jul 4 13:08:21 2006 From: fuzzyman at voidspace.org.uk (Fuzzyman) Date: Tue, 04 Jul 2006 12:08:21 +0100 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: References: Message-ID: <44AA4C25.3020706@voidspace.org.uk> Ka-Ping Yee wrote: >Hi Brett, > >Here are some comments on the description of the restricted execution >model that you posted. > > [snip...] > >>Filesystem >>=================== >> >>The most obvious facet of a filesystem to protect is reading from it. >>One does not want what is stored in ``/etc/passwd`` to get out. And >>one also does not want writing to the disk unless explicitly allowed >>for basically the same reason; if someone can write ``/etc/passwd`` >>then they can set the password for the root account. >> >> > >There's a big difference between modifying (or erasing) an existing file >and writing a new file (e.g. for temporary storage). If i give you a >little filesystem of your own to play in, and it starts out empty, you >can put whatever you want in it without violating my secrecy or the >integrity of my files. > >I think you should be talking about this in terms of specifically >what abilities you want to be able to allow, based on examples of >real-life applications. > > > > As an adjunct to this, one of the barriers to Javascript applications is the lack of client-side data persistence. This makes (amongst other things) offline, or entirely clientside, applications very difficult. All sorts of novel ways round this have been found [ http://codinginparadise.org/weblog/2006/04/now-in-browser-near-you-offline-access.html ]. If a 'standard' interpreter running in the browser had sandboxed access to the filesystem, this would be great. Of course Mozilla would probably disable it by default, and only provide horrific means for users to re-enable it [ http://www.mozilla.org/editor/midasdemo/securityprefs.html ]. All the best, Michael Foord http://www.voidspace.org.uk/python/index.shtml > > >-- ?!ng >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > From theller at python.net Tue Jul 4 13:35:13 2006 From: theller at python.net (Thomas Heller) Date: Tue, 04 Jul 2006 13:35:13 +0200 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: References: Message-ID: Neal Norwitz schrieb: > I'm glad to see Anthony ratcheting down. At this point, we need to be > fixing bugs and improving doc. Maybe Anthony and I should have a > contest to see who can revert the most changes. :-) > Neal (and/or Anthony), I would like to ask about the possibility to add some improvements to ctypes in Python 2.5, although the feature freeze is now in effect. Hopefully former third-party libraries can have the freeze relaxed somewhat;-). I intend to do these changes, the first is a small and trivial one, but allows a lot of flexibility: - Remove the restriction that the argtypes attribute of foreign functions must be ctypes types. Instead they are only required to implement a .from_param class method. The advantage is that custom objects can be used as function parameters. One usecase is to allow numpy arrays as function parameters without any conversion - this change at least allows to code this in Python. The patch is attached as from_param.patch. The second one is more involved, and not yet complete. I can post the patch or a link to it for review when it is implemented completely: - Implement the __array_struct__ attribute as describes by the numpy pep at http://numpy.scipy.org/array_interface.html. The properties needed to implement the __array_struct__ attribute could be calculated from a given ctypes array type, however, it would be more efficient to calculate them at type creation time. This requires the StgDSictObject that holds information about the ctypes type to grow a few fields: 'int nd' - contains the number of dimensions, 'char typekind' - a struct-like character for the item type, and 'Py_intptr_t *shape' - an array of size 'nd' containing shape information. Thanks for investigating this, Thomas -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: from_param.patch Url: http://mail.python.org/pipermail/python-dev/attachments/20060704/3c0500e4/attachment.pot From ncoghlan at iinet.net.au Tue Jul 4 14:32:38 2006 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Tue, 04 Jul 2006 22:32:38 +1000 Subject: [Python-Dev] Proposed beta 2 changes (Q for Anthony/Neal) Message-ID: <44AA5FE6.2030801@iinet.net.au> I've got a couple of changes ready to go for beta 2, but need a go ahead from one of the release managers before committing either of them: 1. Finishing the __module_name__ workaround to allow relative imports from the main module when using -m. I'd really like to finish this, because having PEP 328 and 338 not playing well together is a wart that's quite visible to end users. I'd rather not have people's first encounter with the features provided by either PEP involve discovering that they're "broken". The patch to fix this also takes care of adding a couple of paragraphs to the tutorial about explicit relative imports (which aren't currently written up in the main documentation). The patch is attached directly to the beta 1 bug report about the problem [1]. (Guido gave a +1 to the concept, but explicitly deferred to Anthony and Neal as to whether or not the fix should go in for beta 2) 2. Adding an 'ignore' filter for ImportWarning at the end of warnings.py This is a safe and easy fix to silence import warning spam for people that don't want it. I don't believe there are any Pending Deprecation Warnings at the moment, so -Wd at the command line would be sufficient to enable ImportWarning for people that want to see it. Adding the line "warnings.simplefilter('default', ImportWarning)" to sitecustomize.py would be sufficient for organisations to turn the warning on across the board if they so chose. The reason I haven't checked this in directly is that there's no point if Anthony and/or Neal intend to accept one of the patches that tries to make the import machinery more intelligent about missing __init__.py files. Cheers, Nick. [1] http://www.python.org/sf/1510172 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From rasky at develer.com Tue Jul 4 14:43:40 2006 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 4 Jul 2006 14:43:40 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) References: <44AA1FF6.1050501@acm.org> Message-ID: <010601c69f67$7a535f90$d503030a@trilan> Talin wrote: > This is sort of a re-do of an earlier proposal which seems to have > gotten lost in the shuffle of the larger debate. > > I propose to create a new type of scoping rule, which I will call > "explicit" lexical scoping, that will co-exist with the current > "implicit" scoping rule that exists in Python today. Interesting. What if for-loops implicitally used "my" on the iteration variable? That would solve the binding problem we were discussing and make lambdas "Do The Right Thing"(TM) when used in loops. -- Giovanni Bajo From ark at acm.org Tue Jul 4 16:16:59 2006 From: ark at acm.org (Andrew Koenig) Date: Tue, 4 Jul 2006 10:16:59 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44AA1FF6.1050501@acm.org> Message-ID: <001901c69f74$86a28b60$6402a8c0@arkdesktop> > Borrowing from Perl, the keyword 'my' is used to declare an explicitly > scoped variable: > > def f1(): > my x = 1 > def f2(): > x = 2 # Does not create a new x > > In the above example, the statement 'my x = 1' declares that the scope > of the variable 'x' is the outer function f1. Any assignment to x will > modify the existing x, rather than creating a new definition. -1, for this reason: def f() x = 2 # Does this create a local variable? Today, the answer is yes. Under this proposal, you can't answer the question without inspecting the entire context in which f is defined. For that reason, I would much rather have the first assignment in a block say explicitly whether it is intended to create a local variable: def f1(): x = 1 def f2(): global x x = 2 # Does not create a new x This might even be abbreviated: def f1(): x = 1 def f2(): global x = 2 # Equivalent to the last example above From guido at python.org Tue Jul 4 16:28:30 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Jul 2006 16:28:30 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44AA1FF6.1050501@acm.org> References: <44AA1FF6.1050501@acm.org> Message-ID: Please move this to the python-3000 list. Also please explain what problem you are solving before proposing a solution. I note that we are seeing quite a flurry of language change proposals. I have to recommend restraint; I *don't* want to turn the entire language upside down. That's not a comment on this particular proposal, but on the issue of too many proposals. From actual users of the language I get more complaints about the breakneck speed of Python's evolution than about the brokenness of the current language. --Guido On 7/4/06, Talin wrote: > This is sort of a re-do of an earlier proposal which seems to have > gotten lost in the shuffle of the larger debate. > > I propose to create a new type of scoping rule, which I will call > "explicit" lexical scoping, that will co-exist with the current > "implicit" scoping rule that exists in Python today. > > > Definitions: > > Implicit scoping is what we have now - a variable is defined within a > scope implicitly by assignment. More specifically, when a name is > assigned, the name is defined at the innermost function-level scope from > where the assignment took place. > > Explicit scoping is where the programmer explicitly specifies which > scope the variable should be defined in. Unlike implicit scoping, > assignments to a named variable do not automatically redefine that > variable within the current scope. > > > Syntax: > > Borrowing from Perl, the keyword 'my' is used to declare an explicitly > scoped variable: > > def f1(): > my x = 1 > def f2(): > x = 2 # Does not create a new x > > In the above example, the statement 'my x = 1' declares that the scope > of the variable 'x' is the outer function f1. Any assignment to x will > modify the existing x, rather than creating a new definition. > > Note that the 'my' prefix can be combined with an assignment operation. > It is anticipated that the 'my' prefix will be used quite frequently > (and encouraged), so it makes sense to cut down on the number of > statements by combining declaration and assignment. > > Explicitly scoped variables can also be declared at the module level: > > my x = 1 > def f1(): > x = 2 # Modifies the global X > > Declaring a module-level variable with an explicit scope eliminates the > need for a 'global' statement for that variable. > > > Nested Scopes: > > Each occurance of the keyword 'my' creates a new scope which hides any > outer definitions of the name. So for example: > > my x = 1 > def f1(): > my x = 2 # This is a different 'x' than the global > def f2(): > x = 3 # This is the 'x' defined within f1() > > > Interaction between explicit scoping and globals: > > The 'global' statement, when used with explicitly scoped variables, > means exactly the same as it does with implicitly scoped variables: It > allows access to the outermost scope, overriding any intermediate > definitions in surrounding scopes: > > x = 1 > def f1(): > my x = 2 > def f2(): > global x > x = 3 # This is the module-level 'x' > > > Explicit scoping and code block structure: > > Implicitly scoped variables are always defined at the nearest enclosing > function scope, even if they are created within a code block. > > It might be worth considering allowing explicitly scoped variables to be > defined within other scopes. For example, we might choose to allow > explicit scope declarations to be limited to the current suite: > > def f1(): > for x in range(0,10): > my y = x*x # A new definition of y for each iteration > > Note that this is a speculation only, and not a core part of the > proposal (so please don't reject the proposal on this one point.) > > > Formal definition: > > When a value is assigned to a local variable name, the rules for > determining which scope the variable will be defined in are as follows: > > 1) Starting with the current (innermost) scope, examine all of the > currently active scopes: > 1a) If the current scope contains a 'global' statement for the > given name, then set the result scope to the outermost (module-level) scope. > 1b) If the current scope contains a 'my' statement for the given > name, then set the result scope to the scope in which the 'my' statement > occurred. > 2) Otherwise, continue until we run out of scopes. If neither a > 'global' or 'my' declaration was discovered, then use the innermost > scope as the result scope. > > > How is this different from 'outer'? > > The explicit scope proposal requires that the scope be specified at the > place where the variable is *defined* as opposed to where it is *used*. > This definition is inherited by all inner scopes. > > This allows a finer degree of control, for less typing, than the 'outer' > proposal. With explicit scoping, there is no confusion as to which scope > is being considered; And explicit scoping allows a single declaration of > a variable to be shared by many different inner scopes, which would > otherwise require a separate 'outer' statement for each one. > > > Explicit scoping and static analysis: > > It should be easier to do static analysis of code with explicit scoping, > since you always know what scope a variable is defined in (as opposed to > implicit scoping, where a variable may switch from global to local as a > result of an assignment.) > > Note that this implies that the creation of the scope does not occur at > the time of the assignment, but rather at the time the function is > entered. Thus: > > x = 1 > def f1(): > print x # Error, unassigned value > my x = 2 > > In the above example, even though the 'my' statement occurs after the > print, the scope created by the 'my' statement is in effect for the > entire function, although the actual *assignment* takes place after the > print. The reason for this is that the scope creation is actually done > by the compiler. > > -- Talin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From facundobatista at gmail.com Tue Jul 4 17:43:35 2006 From: facundobatista at gmail.com (Facundo Batista) Date: Tue, 4 Jul 2006 12:43:35 -0300 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: 2006/7/3, Guido van Rossum : > To fake things like this, socket.setdefaulttimeout() was added, though > I don't know if it actually works. Have you tried that? This affect all the sockets. And I hit the problem when servicing information with a web service (TCPServer), and I need to timeout the connection of the URLOpen *only*. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From fperez.net at gmail.com Tue Jul 4 22:14:43 2006 From: fperez.net at gmail.com (Fernando Perez) Date: Tue, 04 Jul 2006 14:14:43 -0600 Subject: [Python-Dev] 2.5 and beyond References: Message-ID: Thomas Heller wrote: > I would like to ask about the possibility to add some improvements to > ctypes > in Python 2.5, although the feature freeze is now in effect. Hopefully > former third-party libraries can have the freeze relaxed somewhat;-). > > I intend to do these changes, the first is a small and trivial one, but > allows a lot of flexibility: [...] I'd just like to provide a bit of context for Thomas' request (disclaimer: he did NOT ask me to write this, nor did anyone else). I understand the release managers' need to be strict with the freeze, but perhaps knowing what's behind this particular change will help them make a more informed decision. Numpy (http://numpy.scipy.org/) is the new python array package for numerical computing, which has been developed at enormous effort by Travis Oliphant (with community help) over the last year, as a way to unify the old Numeric package (written by Jim Hugunin, of Jython and IronPython fame) and Numarray (written by the Hubble telescope team). The effect of numpy in the community, even in its current pre-1.0 form, has been tremendous. There is a real, pressing need in the scientific world for open source and technically superior replacements to Matlab and IDL, the propietary 800-lb gorillas of the field. Many major research institutions across the world are seriously looking at python as fulfilling this role, but the previous situation of a divided library (Numeric/numarray) was keeping a lot of people on the fence. With Travis' effort and numpy maturing towards a 1.0 release right around the time of python 2.5, a LOT of people have come out of the woodwork to contribute code, ideas, documentation, etc. There is a real sense that the combination of python2.5 (with better 64-bit and __index__ support) and numpy will provide a significant advancement for scientific computing with modern, high-level tools. In this particular community, the need to interface with low-level existing libraries is probably much more common than in other fields. There are literally millions of lines of C/C++ code for scientific work which we have to use efficiently, and this is an everyday need for us. While there are a number of tools for this (SWIG, Boost::Python, pyrex, scipy.weave,...), very recently people have discovered how useful ctypes can be for this task. One of the Google SoC projects (http://2006.planet-soc.com/blog/140) started trying to wrap libsvm with SWIG and a week of frustrated efforts led nowhere. Albert then discovered ctypes and in a few hours was up and running. This has generated a lot of interest in the numpy crowd for ctypes, and people would really, really like to see python2.5 come 'out of the box' with as solid a support as possible from ctypes for numpy array interfacing. Ultimately the decision is up to the release team, I know that. But at least with this info, I hope you can understand: 1. why this is important to this community 2. why the timing isn't ideal: it is only /very/ recently that the numpy team 'discovered' how much ctypes could truly help with a necessary (and often very unpleasant) task in the numerical/python world. Thanks for reading, f From nnorwitz at gmail.com Tue Jul 4 23:21:55 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 4 Jul 2006 14:21:55 -0700 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com> References: <17573.60593.759293.549583@montanaro.dyndns.org> <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> <17574.43390.560231.425494@montanaro.dyndns.org> <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> <44A6C314.2000709@v.loewis.de> <0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com> Message-ID: Ronald, Bob, I know Skip found and fixed his problem, however, is this problem likely to affect other users? Is there anything we can do to help alleviate/diagnose this problem? n -- On 7/1/06, Ronald Oussoren wrote: > > On Jul 1, 2006, at 8:46 PM, Martin v. L?wis wrote: > > > Ronald Oussoren wrote: > >> What I don't understand yet is why your copy of libz doesn't have > >> inflateCopy. > > > > What I don't understand is that configure does not detect that. > > You may be onto something there. Skip, do you have another copy of > libz somewhere? Given the link line in your first message either in / > usr/local/lib or /Users/skip/local/lib. And if you have, is that a > static library (libz.a) instead of a dylib? > > As background to my question: the linker on OSX behaves slightly > different than the one on most other unix-y systems. It first > searches the entire linker path for shared libraries (dylibs) before > looking for static libraries. I added a flag to the link flags for > the zlib extension a while back that changes the search order into a > more traditional one: look in every directory on the linker path for > either a dylib or static library. The new flag is -Wl,- > search_paths_first. > > If skip does indeed have libz somewhere else we'll either have to > make a matching update to configure, or roll back my change. If the > latter I'll have to tweak the build script for the binary installer > for OSX because I want to link that using a static copy of libz for > binary compatibility with OSX 10.3.9. > > Ronald > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > > > > From talin at acm.org Tue Jul 4 23:39:44 2006 From: talin at acm.org (Talin) Date: Tue, 04 Jul 2006 14:39:44 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> Message-ID: <44AAE020.5060204@acm.org> Guido van Rossum wrote: > Please move this to the python-3000 list. > > Also please explain what problem you are solving before proposing a > solution. > > I note that we are seeing quite a flurry of language change proposals. > I have to recommend restraint; I *don't* want to turn the entire > language upside down. That's not a comment on this particular > proposal, but on the issue of too many proposals. From actual users of > the language I get more complaints about the breakneck speed of > Python's evolution than about the brokenness of the current language. > > --Guido Actually, the "problem" I am trying to solve is the debate on the mailing list. That is, I listen to what people are asking for, and what disagreements they have, and then I try to provide a solution that resolves the debate. In this case, there was a lot of discussion about lexical scoping, and various people proposing solutions (such as redefining the behavior of 'global') that I thought were (a) problematic, and (b) not a lot of bang for the buck (i.e. the disruption vs. utility tradeoff was poor IMHO.) To be honest, I really have no stake in this proposal, and I don't intend to spend any time defending it other than to correct misperceptions - however, I offer it as a potential starting point for people who are interested in the whole lexical scoping issue. If someone feels that this proposal gives them what they want, then great - otherwise I'll drop it. -- Talin From guido at python.org Tue Jul 4 23:58:04 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Jul 2006 23:58:04 +0200 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: On 7/4/06, Facundo Batista wrote: > 2006/7/3, Guido van Rossum : > > > To fake things like this, socket.setdefaulttimeout() was added, though > > I don't know if it actually works. Have you tried that? > > This affect all the sockets. So, assuming your app is single-threaded, set the timeout, call urlopen(), and reset the timeout to None. > And I hit the problem when servicing > information with a web service (TCPServer), and I need to timeout the > connection of the URLOpen *only*. That's not so easy even if you were to have a timeout parameter to urlopen(). You'd have to implement that peculiarity in all the layers (in this case, urllib and httplib; and possibly ftplib, gopherlib etc. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Jul 5 00:18:13 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 00:18:13 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44AAE020.5060204@acm.org> References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> Message-ID: On 7/4/06, Talin wrote: > Guido van Rossum wrote: > > Also please explain what problem you are solving before proposing a > > solution. > Actually, the "problem" I am trying to solve is the debate on the > mailing list. That is, I listen to what people are asking for, and what > disagreements they have, and then I try to provide a solution that > resolves the debate. > > In this case, there was a lot of discussion about lexical scoping, and > various people proposing solutions (such as redefining the behavior of > 'global') that I thought were (a) problematic, and (b) not a lot of bang > for the buck (i.e. the disruption vs. utility tradeoff was poor IMHO.) > > To be honest, I really have no stake in this proposal, and I don't > intend to spend any time defending it other than to correct > misperceptions - however, I offer it as a potential starting point for > people who are interested in the whole lexical scoping issue. If someone > feels that this proposal gives them what they want, then great - > otherwise I'll drop it. Thanks; I appreciate the attempt. I just think that we're not quite ready for more proposals (and certainly not for radical ones). Instead, I'd like to go back to review the needs and desires first. I think the needs are actually pretty simple. Python currently doesn't allow assignment to variables in an outer non-global scope, and people have shown by their behavior that they cannot get used to this (otherwise the debate would have fizzled by now). There are two fundamentally different mechanisms seen in programming languages to control the binding of such variables. The most common approach is to require declaration of variables in the scope to which they belong. But Python doesn't do this, and I think it would be a shame if we had to start doing this now -- the objections against your proposal clearly show the problems if we try to mix this with Python's traditional "assignment is declaration" philosophy. The other approach is an extension of what Python already does for variables in the global scope. ABC did this too (the SHARE command, see http://homepages.cwi.nl/~steven/abc/qr.html#HOWTOs). I think we have to continue to search for a solution that extends the idea of global declarations. I've proposed extending its meaning to refer to the nearest outer scope where the variable is set; if there is no such scope it's an error. This will break a small number of program but probably not very many; still, it'll require a future statement or waiting until Python 3.0. The downside is that "global" is not a very intuitive word for this new meaning. (Maybe ABC's SHARE would have been better.) We could use a different keyword instead, e.g. 'outer'. I believe I've also seen proposals in the past that used a number to indicate how many scopes to go out; I don't like that at all. I don't see anything else that's attractive. The realistic options are: 1. do nothing 2. extend global's meaning 3. add outer keyword Personally I think I'd vote for (2) since it doesn't require a new keyword. But that's only a slight preference over the other two. Personally it's not a burning need; by the time you start feeling the need to modify variables in an outer scope you should probably consider refactoring using an explicit class to hold the state. But I used the same argument to keep the current form of nested scopes (can we say "closures"? But what exactly is the closure?) out of the door and I lost that argument. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Jul 5 00:25:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 04 Jul 2006 18:25:38 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AAE020.5060204@acm.org> <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> Message-ID: <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> At 12:18 AM 7/5/2006 +0200, Guido van Rossum wrote: >I don't see anything else that's attractive. The realistic options are: > >1. do nothing >2. extend global's meaning >3. add outer keyword Did you also consider and reject: * Alternate binding operators (e.g. ":=", ".=", etc.) * Alternate spelling of outer names when binding (e.g. ".x = whatever" to bind an outer x) If so, then these should probably be added to the "rejected alternatives" for Py3K so they don't get rehashed. From nnorwitz at gmail.com Wed Jul 5 01:49:13 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 4 Jul 2006 16:49:13 -0700 Subject: [Python-Dev] User's complaints Message-ID: On 7/4/06, Guido van Rossum wrote: > > From actual users of > the language I get more complaints about the breakneck speed of > Python's evolution than about the brokenness of the current language. Guido, I'm really interested in your perspective here. I assume you hear far more "average" complaints from Joe Random User. Can you help give the rest of us an idea about the top 10 complaints/problems people have? I realize this will be subjective, that's ok. Perhaps we should try to focus our energies on some of these issues. For example, we heard grumblings about the releases coming too often. Once we went to an 18 month release schedule, there was minimal complaining. It should be fairly safe to assume this silence means people think we are doing a good job. What are the things that could be fixed that would silence the most number of user's complaints? n -- PS. One thing I tend to talk to users about is stability of the interpreter. When I talk about crashing the interpreter, the most common first reaction I get is "you can crash the interpreter? How do you do that?" I take that answer as a good sign. :-) From aahz at pythoncraft.com Wed Jul 5 05:04:29 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 4 Jul 2006 20:04:29 -0700 Subject: [Python-Dev] 2.5b1 Windows install In-Reply-To: <449F7CA9.9010105@v.loewis.de> References: <20060626004726.GA24988@panix.com> <449F7CA9.9010105@v.loewis.de> Message-ID: <20060705030429.GA18105@panix.com> On Mon, Jun 26, 2006, "Martin v. L?wis" wrote: > Aahz wrote: >> >> Has anyone else tried doing an admin install with "compile .py files" >> checked? It's causing my install to blow up, but I'd prefer to assume >> it's some weird Windows config/bug unless other people also have it, in >> which case I'll file an SF report. > > It works fine for me. One way for it to fail is if you uncompilable > modules in the target directory. Currently, it invokes > > [TARGETDIR]python.exe -Wi [TARGETDIR]Lib\compileall.py -f -x > bad_coding|badsyntax|site-packages [TARGETDIR]Lib > > where TARGETDIR is, well, the target directory of the installation. > You could try to run this after you installed Python without pyc > compilation, to see whether it succeeds. Ah-ha! I haven't actually tested this directly, but I bet I know what's going on: this isn't properly quoted and fails with TARGETDIR of "C:\Program Files\Python25" because of the space. I did test to see that it works fine with "C:\Python25" Shall I file a bug? Or do you want to just document this as a limitation? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I saw `cout' being shifted "Hello world" times to the left and stopped right there." --Steve Gonedes From guido at python.org Wed Jul 5 05:49:10 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 05:49:10 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> Message-ID: On 7/5/06, Phillip J. Eby wrote: > At 12:18 AM 7/5/2006 +0200, Guido van Rossum wrote: > >I don't see anything else that's attractive. The realistic options are: > > > >1. do nothing > >2. extend global's meaning > >3. add outer keyword > > Did you also consider and reject: > > * Alternate binding operators (e.g. ":=", ".=", etc.) Brr. > * Alternate spelling of outer names when binding (e.g. ".x = whatever" to > bind an outer x) We looked at and rejected "globals.x = whatever". I think the same reasoning applies here. > If so, then these should probably be added to the "rejected alternatives" > for Py3K so they don't get rehashed. Georgbot? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Wed Jul 5 07:23:54 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 05 Jul 2006 07:23:54 +0200 Subject: [Python-Dev] 2.5b1 Windows install In-Reply-To: <20060705030429.GA18105@panix.com> References: <20060626004726.GA24988@panix.com> <449F7CA9.9010105@v.loewis.de> <20060705030429.GA18105@panix.com> Message-ID: <44AB4CEA.9080705@v.loewis.de> Aahz wrote: > Ah-ha! I haven't actually tested this directly, but I bet I know what's > going on: this isn't properly quoted and fails with TARGETDIR of > "C:\Program Files\Python25" because of the space. I did test to see that > it works fine with "C:\Python25" > > Shall I file a bug? Or do you want to just document this as a > limitation? If this is indeed the problem, it should be fixed. Before filing the bug report, please confirm that this actually is a problem. Regards, Martin From ronaldoussoren at mac.com Wed Jul 5 07:35:25 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 5 Jul 2006 07:35:25 +0200 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: References: <17573.60593.759293.549583@montanaro.dyndns.org> <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> <17574.43390.560231.425494@montanaro.dyndns.org> <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> <44A6C314.2000709@v.loewis.de> <0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com> Message-ID: <738A121B-3529-411D-89A7-EBD8C6B42A29@mac.com> On Jul 4, 2006, at 11:21 PM, Neal Norwitz wrote: > Ronald, Bob, > > I know Skip found and fixed his problem, however, is this problem > likely to affect other users? Is there anything we can do to help > alleviate/diagnose this problem? I'll either enhance configure or roll back my change to setup.py. I'd prefer to do the former, but if beta2 gets too close I'll just change setup.py. Ronald -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2157 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20060705/256a01dc/attachment.bin From pje at telecommunity.com Wed Jul 5 08:26:25 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 05 Jul 2006 02:26:25 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> At 05:49 AM 7/5/2006 +0200, Guido van Rossum wrote: >>* Alternate spelling of outer names when binding (e.g. ".x = whatever" to >>bind an outer x) > >We looked at and rejected "globals.x = whatever". I think the same >reasoning applies here. I thought the 'globals.x' proposal required that 'x' always be accessed using 'globals', even if it wasn't being rebound. I don't see a problem with requiring '.x' to be used for both reading and writing of outer-scope names; it just shouldn't be required for an outer-scope name that you don't rebind in the current scope. That symmetry requirement can't be implemented with the 'globals.x' approach unless 'globals' is treated specially by the compiler. Using the classic nonsense example: def counter(num): def inc(): .num += 1 return .num return inc If inc() only needed to *read* num, it could just use 'num' without the '.', and be nicely backward compatible with today's Python. (Note: It should be illegal to use both '.num' and 'num' in the same scope, whether writing or reading the value, to prevent readers from becoming confused about what variable you mean. It should also be required that the compiler can see a definition of 'num' in an outer scope if you use the '.num' syntax, so that misspelling a name doesn't create a global variable.) I personally think this approach could be the overall least-intrusive solution as far as syntax goes. It also allows for dropping the 'global' keyword in 3.x, and it has a nice EIBTI feel to it, as it allows you to highlight closure variables in an inner function by using the '.'. It's often not obvious when an inner function (such as a decorator returned by a decorator factory) is using variables that were defined in the outer scope; the leading '.' would make them stand out, and so could be considered the recommended code style when referring to outer variables. In addition, there's a nice symmetry between nested functions and top-level functions, e.g. in this global version of the counter example: num = 0 def inc(): .num += 1 return .num The principle downside taht I see is that it uses semi-random punctuation in place of keywords. OTOH, we are already using more-or-less this syntax for relative imports, so reusing it to mean "relative variables" seems to at least avoid creating any entirely new principles. :) Anyway, I won't argue this one further; I just wanted to make sure it had been considered, as I'm not sure that you were reading the thread where it was first brought up (possibly as long as a few months ago). From guido at python.org Wed Jul 5 10:12:53 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 10:12:53 +0200 Subject: [Python-Dev] Import semantics In-Reply-To: <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> References: <448D1F0D.7000405@strakt.com> <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> Message-ID: On 6/25/06, Frank Wierzbicki wrote: > Sorry for the untrimmed conversation, but I've cc'ed jython-dev, my > comments are at the bottom. > > On 6/12/06, Guido van Rossum wrote: > > On 6/12/06, Samuele Pedroni wrote: > > > Fabio Zadrozny wrote: > > > > Python and Jython import semantics differ on how sub-packages should be > > > > accessed after importing some module: > > > > > > > > Jython 2.1 on java1.5.0 (JIT: null) > > > > Type "copyright", "credits" or "license" for more information. > > > > >>> import xml > > > > >>> xml.dom > > > > > > > > > > > > Python 2.4.2 (#67, Sep 28 2005, 12:41:11) [MSC v.1310 32 bit (Intel)] on > > > > win32 > > > > Type "help", "copyright", "credits" or "license" for more information. > > > > >>> import xml > > > > >>> xml.dom > > > > Traceback (most recent call last): > > > > File "", line 1, in ? > > > > AttributeError: 'module' object has no attribute 'dom' > > > > >>> from xml.dom import pulldom > > > > >>> xml.dom > > > > > > > > > > > > Note that in Jython importing a module makes all subpackages beneath it > > > > available, whereas in python, only the tokens available in __init__.py > > > > are accessible, but if you do load the module later even if not getting > > > > it directly into the namespace, it gets accessible too -- this seems > > > > more like something unexpected to me -- I would expect it to be > > > > available only if I did some "import xml.dom" at some point. > > > > > > > > My problem is that in Pydev, in static analysis, I would only get the > > > > tokens available for actually imported modules, but that's not true for > > > > Jython, and I'm not sure if the current behaviour in Python was expected. > > > > > > > > So... which would be the right semantics for this? > > > > > > the difference in Jython is deliberate. I think the reason was to mimic > > > more the Java style for this, in java fully qualified names always work. > > > In jython importing the top level packages is enough to get a similar > > > effect. > > > > > > This is unlikely to change for backward compatibility reasons, at least > > > from my POV. > > > > IMO it should do this only if the imported module is really a Java > > package. If it's a Python package it should stick to python semantics > > if possible. > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > This is a tough one since the BDFL and Samuele disagree here. Perhaps > we should document the Java import behavior as permanent, but document > the Python imports in Jython as being deprecated but available until > some future release? I believe we would keep it at least through > Jython 2.3. Hi Frank, Have you and/or the Jython community made up your mind about this? The thread seems to have disappeared after you posted (or perhaps it continued only on jython-dev, which I don't read?). Also, I just realized that you're the new Jython maintainer. Is *that* official? I'd like to offer you my congratulations, and, more importantly, any support you might need. I find Jython an important part for Python's long-term stategy. I'm asked occasionally what the status of Jython is; people point out that the last release was 2.1 many years ago and the website has no news since early 2005; thy're afraid that Jython is dying and that it's not a viable choice for new projects. I'm very happy to be able to tell them that soon there will be a 2.3 release and yes there *is* continued support... So if you need anything from me or from the PSF, please let me know! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Wed Jul 5 10:17:59 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 5 Jul 2006 18:17:59 +1000 Subject: [Python-Dev] Import semantics In-Reply-To: References: <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> Message-ID: <200607051818.02742.anthony@interlink.com.au> On Wednesday 05 July 2006 18:12, Guido van Rossum wrote: > I'm asked > occasionally what the status of Jython is; people point out that > the last release was 2.1 many years ago and the website has no news > since early 2005; thy're afraid that Jython is dying and that it's > not a viable choice for new projects. I'm very happy to be able to > tell them that soon there will be a 2.3 release and yes there *is* > continued support... So if you need anything from me or from the > PSF, please let me know! In that case, why not post a news item saying this? The website is probably the first place people look... Anthony -- Anthony Baxter It's never too late to have a happy childhood. From guido at python.org Wed Jul 5 10:21:00 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 10:21:00 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> Message-ID: On 7/5/06, Phillip J. Eby wrote: > At 05:49 AM 7/5/2006 +0200, Guido van Rossum wrote: > >>* Alternate spelling of outer names when binding (e.g. ".x = whatever" to > >>bind an outer x) > > > >We looked at and rejected "globals.x = whatever". I think the same > >reasoning applies here. > > I thought the 'globals.x' proposal required that 'x' always be accessed > using 'globals', even if it wasn't being rebound. I don't see a problem > with requiring '.x' to be used for both reading and writing of outer-scope > names; it just shouldn't be required for an outer-scope name that you don't > rebind in the current scope. That symmetry requirement can't be > implemented with the 'globals.x' approach unless 'globals' is treated > specially by the compiler. > > Using the classic nonsense example: > > def counter(num): > def inc(): > .num += 1 > return .num > return inc > > If inc() only needed to *read* num, it could just use 'num' without the > '.', and be nicely backward compatible with today's Python. > > (Note: It should be illegal to use both '.num' and 'num' in the same scope, > whether writing or reading the value, to prevent readers from becoming > confused about what variable you mean. It should also be required that the > compiler can see a definition of 'num' in an outer scope if you use the > '.num' syntax, so that misspelling a name doesn't create a global variable.) > > I personally think this approach could be the overall least-intrusive > solution as far as syntax goes. It also allows for dropping the 'global' > keyword in 3.x, and it has a nice EIBTI feel to it, as it allows you to > highlight closure variables in an inner function by using the '.'. It's > often not obvious when an inner function (such as a decorator returned by a > decorator factory) is using variables that were defined in the outer scope; > the leading '.' would make them stand out, and so could be considered the > recommended code style when referring to outer variables. > > In addition, there's a nice symmetry between nested functions and top-level > functions, e.g. in this global version of the counter example: > > num = 0 > def inc(): > .num += 1 > return .num > > The principle downside taht I see is that it uses semi-random punctuation > in place of keywords. OTOH, we are already using more-or-less this syntax > for relative imports, so reusing it to mean "relative variables" seems to > at least avoid creating any entirely new principles. :) > > Anyway, I won't argue this one further; I just wanted to make sure it had > been considered, as I'm not sure that you were reading the thread where it > was first brought up (possibly as long as a few months ago). Thanks for bringing this up. I'm not sure what I think of it yet. One problem I see is that there might end up being two ways to reference variables in outer scopes: .num if you plan to assign to it, or just num if you only reference it. I find that the most disurbing issue so far; modified global declarations or outer declarations don't have this problem. Would this also use ..num to refer to num in an outer scope two levels removed? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Jul 5 10:22:16 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 10:22:16 +0200 Subject: [Python-Dev] Import semantics In-Reply-To: <200607051818.02742.anthony@interlink.com.au> References: <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> <200607051818.02742.anthony@interlink.com.au> Message-ID: On 7/5/06, Anthony Baxter wrote: > On Wednesday 05 July 2006 18:12, Guido van Rossum wrote: > > I'm asked > > occasionally what the status of Jython is; people point out that > > the last release was 2.1 many years ago and the website has no news > > since early 2005; thy're afraid that Jython is dying and that it's > > not a viable choice for new projects. I'm very happy to be able to > > tell them that soon there will be a 2.3 release and yes there *is* > > continued support... So if you need anything from me or from the > > PSF, please let me know! > > In that case, why not post a news item saying this? The website is > probably the first place people look... I'm all for that; but I don't have webmaster privileges (nor do I want them :-). Frank? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Wed Jul 5 10:25:16 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 5 Jul 2006 18:25:16 +1000 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> Message-ID: <200607051825.17720.anthony@interlink.com.au> On Wednesday 05 July 2006 18:21, Guido van Rossum wrote: > Would this also use ..num to refer to num in an outer scope two > levels removed? Ew! I don't want to even think about debugging ...x vs ....x Anthony -- Anthony Baxter It's never too late to have a happy childhood. From guido at python.org Wed Jul 5 10:28:45 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 10:28:45 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <200607051825.17720.anthony@interlink.com.au> References: <44AA1FF6.1050501@acm.org> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> <200607051825.17720.anthony@interlink.com.au> Message-ID: Sorry, I should have added a ... :-) On 7/5/06, Anthony Baxter wrote: > On Wednesday 05 July 2006 18:21, Guido van Rossum wrote: > > Would this also use ..num to refer to num in an outer scope two > > levels removed? > > Ew! > > I don't want to even think about debugging > > ...x > vs > ....x > > Anthony > -- > Anthony Baxter > It's never too late to have a happy childhood. > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From g.brandl at gmx.net Wed Jul 5 10:29:27 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 05 Jul 2006 10:29:27 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> Message-ID: Guido van Rossum wrote: > On 7/5/06, Phillip J. Eby wrote: >> At 12:18 AM 7/5/2006 +0200, Guido van Rossum wrote: >> >I don't see anything else that's attractive. The realistic options are: >> > >> >1. do nothing >> >2. extend global's meaning >> >3. add outer keyword >> >> Did you also consider and reject: >> >> * Alternate binding operators (e.g. ":=", ".=", etc.) > > Brr. > >> * Alternate spelling of outer names when binding (e.g. ".x = whatever" to >> bind an outer x) > > We looked at and rejected "globals.x = whatever". I think the same > reasoning applies here. > >> If so, then these should probably be added to the "rejected alternatives" >> for Py3K so they don't get rehashed. > > Georgbot? I added the alternative binding operators. The discussion about ".x" seems to be still in progress. Georg From scott+python-dev at scottdial.com Wed Jul 5 11:40:40 2006 From: scott+python-dev at scottdial.com (Scott Dial) Date: Wed, 05 Jul 2006 05:40:40 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> Message-ID: <44AB8918.1040308@scottdial.com> Guido van Rossum wrote: > Would this also use ..num to refer to num in an outer scope two levels removed? I realize this was a wink, but it is a valid problem with the "dot"-proposal. def foo(n): def bar(n): def baz(): return .n So, which 'n' outer 'n' is being referenced? Seems like you need to either be able to do multiple dots (ugly, hard to read) or only do a single-step outwards reference. But then that has it's own problems, if I meant the 'n' passed into 'foo', then I have to resort to such nonsense as: def foo(n): def bar(n): foon = .n def baz(): return .foon It would almost be cute if you could do something like ".foo.n" to get to the correct variable. If python maintains it's current scoping rules, then it seems like it works out, but I haven't thought this one all the way through. def foo(n): def bar(n): def baz(): return .foo.n + .bar.n -- Scott Dial scott at scottdial.com scodial at indiana.edu From and-dev at doxdesk.com Wed Jul 5 11:18:32 2006 From: and-dev at doxdesk.com (Andrew Clover) Date: Wed, 05 Jul 2006 18:18:32 +0900 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> Message-ID: <44AB83E8.2070303@doxdesk.com> Guido van Rossum wrote: > 1. do nothing > 2. extend global's meaning > 3. add outer keyword 2.5. extend global syntax to cover both [really global] and [innermost matching scope]. eg. global x, y outer # trailing non-keyword global in x, y # re-use keyword not global x # ceci n'est pas un global ... # something less ugly? > Personally it's not a burning need Agreed. Inability to write as well as read nested scopes is more of an aesthetic wart than a practical one IMO. -- And Clover mailto:and at doxdesk.com http://www.doxdesk.com/ -- And Clover mailto:and at doxdesk.com http://www.doxdesk.com/ From just at letterror.com Wed Jul 5 11:46:50 2006 From: just at letterror.com (Just van Rossum) Date: Wed, 5 Jul 2006 11:46:50 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: Message-ID: Guido van Rossum wrote: > On 7/5/06, Phillip J. Eby wrote: > > Did you also consider and reject: > > > > * Alternate binding operators (e.g. ":=", ".=", etc.) > > Brr. That's too bad :( I still find a rebinding operator (":=" being my favorite) much, *much* more appealing than any of the alternative proposals. It's beautifully symmetrical with "assignment means local". It also pretty much makes the global statement redundant. The only downside I see is that it may cause a fairly big shift in style: I for one would use := for rebinding local names. While I think that would be an improvement (eg. by catching typo's earlier), it's *different*. Just -- Change is bad. We fear change. -- Garth Algar From guido at python.org Wed Jul 5 12:02:16 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 12:02:16 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: On 7/5/06, Just van Rossum wrote: > Guido van Rossum wrote: > > > On 7/5/06, Phillip J. Eby wrote: > > > Did you also consider and reject: > > > > > > * Alternate binding operators (e.g. ":=", ".=", etc.) > > > > Brr. > > That's too bad :( > > I still find a rebinding operator (":=" being my favorite) much, *much* > more appealing than any of the alternative proposals. It's beautifully > symmetrical with "assignment means local". It also pretty much makes the > global statement redundant. > > The only downside I see is that it may cause a fairly big shift in > style: I for one would use := for rebinding local names. While I think > that would be an improvement (eg. by catching typo's earlier), it's > *different*. Hallo broer! :-) I wonder what this should mean then: def outer(): def inner(): x := 1 What is x's scope? Also, a := operator allows all sorts of left-hand sides that don't necessarily make sense, e.g. x.foo := 1 x[0] := 1 -- --Guido van Rossum (home page: http://www.python.org/~guido/) From imbaczek at gmail.com Wed Jul 5 12:17:16 2006 From: imbaczek at gmail.com (=?ISO-8859-2?Q?Marek_"Baczek"_Baczy=F1ski?=) Date: Wed, 5 Jul 2006 12:17:16 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com> 2006/7/5, Just van Rossum : > Guido van Rossum wrote: > > > On 7/5/06, Phillip J. Eby wrote: > > > Did you also consider and reject: > > > > > > * Alternate binding operators (e.g. ":=", ".=", etc.) > > > > Brr. > > That's too bad :( > > I still find a rebinding operator (":=" being my favorite) much, *much* > more appealing than any of the alternative proposals. It's beautifully > symmetrical with "assignment means local". It also pretty much makes the > global statement redundant. > > The only downside I see is that it may cause a fairly big shift in > style: I for one would use := for rebinding local names. While I think > that would be an improvement (eg. by catching typo's earlier), it's > *different*. I suggest <- as an assignment operator instead of := - it's used in OCaml and it looks *very* different, yet still makes sense. x = 0 print x def f(): x = 1 # bind locally print x def g(): x <- 42 # assign "lexically" print x f() print x g() print x prints 0 1 0 42 42 From barry at python.org Wed Jul 5 12:48:25 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 5 Jul 2006 06:48:25 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com> References: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com> Message-ID: <2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 PCTotD (pre-coffee thought of the day): On Jul 5, 2006, at 6:17 AM, Marek "Baczek" Baczy?ski wrote: > 2006/7/5, Just van Rossum : >> Guido van Rossum wrote: >> >>> On 7/5/06, Phillip J. Eby wrote: >>>> Did you also consider and reject: >>>> >>>> * Alternate binding operators (e.g. ":=", ".=", etc.) >>> >>> Brr. Clearly we need the "as if in" operator: x = 0 def foo(): x = 1 def bar(): x = 2 def baz(): x as if in foo = 3 x as if in global += 1 (Personally, I've never really needed this much, but if you have to have it, be explicit! :) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRKuY+XEjvBPtnXfVAQI7WAQAp9fp8SZU8lJdyllrrbFhR9lWxF+pbZ9d +OBv0VEFiFD3aLuKc7y7rAyCWfRVMsoO3FsapeC0/CFJDpRoDDRB7qLzX7He9Fyf JXh3g489RdokxwZClIl38RVgM7WNVK8lzFr8Kf8eTjEUItJoJjk5m2vhymQpfWYG /OeU5xCABxw= =OBRd -----END PGP SIGNATURE----- From just at letterror.com Wed Jul 5 12:54:07 2006 From: just at letterror.com (Just van Rossum) Date: Wed, 5 Jul 2006 12:54:07 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: Message-ID: Guido van Rossum wrote: > Hallo broer! :-) Yo :) > I wonder what this should mean then: > > def outer(): > def inner(): > x := 1 > > What is x's scope? UnboundVariableError: variable 'x' referenced before assignment Or a SyntaxError if the compiler can detect it. > Also, a := operator allows all sorts of left-hand sides that don't > necessarily make sense, e.g. > > x.foo := 1 > x[0] := 1 True, although maybe they could be made to make sense by defining special methods: __rebindattr__ __rebinditem__ <0.5 wink> Just From s.percivall at chello.se Wed Jul 5 14:18:00 2006 From: s.percivall at chello.se (Simon Percivall) Date: Wed, 5 Jul 2006 14:18:00 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: I know this is very similar to the "global.x =" syntax, which was already shot down?, but wouldn't allowing access to a functions locals from within, by prefixing the name, be a good way to disambiguate what happens (instead of any operator to indicate outer scope, like .x = 3 or the like)? I guess this necessitates "global.x =" as well, though. def foo(): def bar(): foo.x = 3 print x # prints 3 I seem to recall that this syntax has been proposed before, though not in this discussion. But my memory is murky. //Simon From just at letterror.com Wed Jul 5 14:23:15 2006 From: just at letterror.com (Just van Rossum) Date: Wed, 5 Jul 2006 14:23:15 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com> Message-ID: Marek "Baczek" Baczy?ski wrote: > I suggest <- as an assignment operator instead of := - it's used in > OCaml and it looks *very* different, yet still makes sense. Except it's currently valid Python syntax: >>> x = 0 >>> x <- 42 False >>> Just From skip at pobox.com Wed Jul 5 15:39:05 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed, 5 Jul 2006 08:39:05 -0500 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org> References: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com> <2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org> Message-ID: <17579.49401.682772.29091@montanaro.dyndns.org> Barry> Clearly we need the "as if in" operator: Why not be more direct? x = 0 def foo(): x = 1 def bar(): x = 2 def baz(): x in foo = 3 x in global += 1 By naming the function in which the binding is to occur you avoid problems of someone coming along and adding or deleting functions between the assignment (in baz) and the target of the assignment (x in foo) but then forgetting to increment or decrement the counters that refer to a fixed number of levels above the current function. Barry> (Personally, I've never really needed this much, but if you have Barry> to have it, be explicit! :) Nor I. I can't think of any situations in my programming where I've used nested functions, but I was never a LISPer... Skip From s.percivall at chello.se Wed Jul 5 16:04:43 2006 From: s.percivall at chello.se (Simon Percivall) Date: Wed, 5 Jul 2006 16:04:43 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44AB8918.1040308@scottdial.com> References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> <44AB8918.1040308@scottdial.com> Message-ID: <6A5A1492-EDD0-4FAF-9A2C-B4A59AF19053@chello.se> On 5 jul 2006, at 11.40, Scott Dial wrote: > Guido van Rossum wrote: >> Would this also use ..num to refer to num in an outer scope two >> levels removed? > > I realize this was a wink, but it is a valid problem with the > "dot"-proposal. > > def foo(n): > def bar(n): > def baz(): > return .n > > So, which 'n' outer 'n' is being referenced? Seems like you need to > either be able to do multiple dots (ugly, hard to read) or only do a > single-step outwards reference. But then that has it's own > problems, if > I meant the 'n' passed into 'foo', then I have to resort to such > nonsense as: No, it's actually not a problem. foo()'s "n" should just be hidden. If you don't want it to be hidden, don't write your function that way. If you find you need deeply nested functions where local names shadow names in outer scopes that you need to access you might want to think of another way to solve your problem. //Simon From just at letterror.com Wed Jul 5 16:16:15 2006 From: just at letterror.com (Just van Rossum) Date: Wed, 5 Jul 2006 16:16:15 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <6A5A1492-EDD0-4FAF-9A2C-B4A59AF19053@chello.se> Message-ID: Simon Percivall wrote: > On 5 jul 2006, at 11.40, Scott Dial wrote: > > Guido van Rossum wrote: > >> Would this also use ..num to refer to num in an outer scope two > >> levels removed? > > > > I realize this was a wink, but it is a valid problem with the > > "dot"-proposal. > > > > def foo(n): > > def bar(n): > > def baz(): > > return .n > > > > So, which 'n' outer 'n' is being referenced? Seems like you need to > > either be able to do multiple dots (ugly, hard to read) or only do a > > single-step outwards reference. But then that has it's own > > problems, if > > I meant the 'n' passed into 'foo', then I have to resort to such > > nonsense as: > > > No, it's actually not a problem. foo()'s "n" should just be > hidden. If you don't want it to be hidden, don't write your > function that way. If you find you need deeply nested functions > where local names shadow names in outer scopes that you need to > access you might want to think of another way to solve your > problem. +1, YAGNI. Just From barry at python.org Wed Jul 5 16:21:33 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 5 Jul 2006 10:21:33 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <17579.49401.682772.29091@montanaro.dyndns.org> References: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com> <2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org> <17579.49401.682772.29091@montanaro.dyndns.org> Message-ID: <7CAE772C-8050-4ED2-8799-1ACB15A0131C@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 5, 2006, at 9:39 AM, skip at pobox.com wrote: > > Barry> Clearly we need the "as if in" operator: > > Why not be more direct? Sure, why not? :) Then we can reserve the "as if" operator for those things that Guido has rejected, but that we sneak in while he's not looking. like-omg-gag-me-with-a-spoon-ly y'rs, - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRKvK8nEjvBPtnXfVAQLbswQArfvIoCdCHmryk3qkOvG6BE0Q1iW7dk0O eI178nG1tY+02JLyrPb1RcjdJG0W0wPwugvVNwVlz29cNkt048uEme6ZBfv3wCt/ bQSWTnDym/OWtQhUtsaw7V5K1o/bP5noqS2MQAcafk4lARv7TAWbBNkPqpk/yFmp 2yNhIfngjts= =thYd -----END PGP SIGNATURE----- From brad.doctor at gmail.com Wed Jul 5 17:21:10 2006 From: brad.doctor at gmail.com (Brad Doctor) Date: Wed, 5 Jul 2006 09:21:10 -0600 Subject: [Python-Dev] Patch for commands.py to provide callback Message-ID: Greetings all, I have attached a patch for commands.py to provide a callback ability. Example use: ------------------------------------- import commands cmd = 'top -b -n2' def fancy(out): print 'GOT(%s)' % out.strip() commands.cb = fancy (s,o) = commands.getstatusoutput(cmd) print 'OUTPUT (%s)' % o ------------------------------------- I am not sure if this is the proper forum or means to submit something like this, so please forgive me and advise accordingly if I am in error. The basic idea is obvious, to allow long-running commands to call back whenever there is output. This is against python 2.4. Please let me know if you have any questions or suggestions. thanks!! -brad -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/62b26e37/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: python_commandsCallBack.diff Type: text/x-patch Size: 891 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20060705/62b26e37/attachment-0001.bin From fredrik at pythonware.com Wed Jul 5 17:24:33 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 05 Jul 2006 17:24:33 +0200 Subject: [Python-Dev] Patch for commands.py to provide callback In-Reply-To: References: Message-ID: Brad Doctor wrote: > I am not sure if this is the proper forum or means to submit something > like this, so please forgive me and advise accordingly if I am in error. to make sure that they don't just disappear under a zillion other mails, patches should be submitted to the patch tracker: http://sourceforge.net/patch/?group_id=5470 From brad.doctor at gmail.com Wed Jul 5 17:26:44 2006 From: brad.doctor at gmail.com (Brad Doctor) Date: Wed, 5 Jul 2006 09:26:44 -0600 Subject: [Python-Dev] Patch for commands.py to provide callback In-Reply-To: References: Message-ID: Cool, thank you Fredrik -- going there now. -brad On 7/5/06, Fredrik Lundh wrote: > > Brad Doctor wrote: > > > I am not sure if this is the proper forum or means to submit something > > like this, so please forgive me and advise accordingly if I am in error. > > to make sure that they don't just disappear under a zillion other mails, > patches should be submitted to the patch tracker: > > http://sourceforge.net/patch/?group_id=5470 > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brad.doctor%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/1c9a90e8/attachment.htm From guido at python.org Wed Jul 5 17:31:57 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 17:31:57 +0200 Subject: [Python-Dev] Patch for commands.py to provide callback In-Reply-To: References: Message-ID: Since commands.getstatusoutput() is such a trivial wrapper around os.popen(), why bother patching commands.py? On 7/5/06, Brad Doctor wrote: > Greetings all, > > I have attached a patch for commands.py to provide a callback ability. > Example use: > > ------------------------------------- > import commands > > cmd = 'top -b -n2' > > def fancy(out): > print 'GOT(%s)' % out.strip() > > commands.cb = fancy > > (s,o) = commands.getstatusoutput(cmd) > print 'OUTPUT (%s)' % o > ------------------------------------- > > I am not sure if this is the proper forum or means to submit something like > this, so please forgive me and advise accordingly if I am in error. The > basic idea is obvious, to allow long-running commands to call back whenever > there is output. This is against python 2.4. Please let me know if you > have any questions or suggestions. > > thanks!! > -brad > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brad.doctor at gmail.com Wed Jul 5 17:35:50 2006 From: brad.doctor at gmail.com (Brad Doctor) Date: Wed, 5 Jul 2006 09:35:50 -0600 Subject: [Python-Dev] Patch for commands.py to provide callback In-Reply-To: References: Message-ID: Because it is a great way to consistently use popen(). Rather than write something specific each time, our site/company prefers to use commands to keep it all consistent. -brad On 7/5/06, Guido van Rossum wrote: > > Since commands.getstatusoutput() is such a trivial wrapper around > os.popen(), why bother patching commands.py? > > On 7/5/06, Brad Doctor wrote: > > Greetings all, > > > > I have attached a patch for commands.py to provide a callback ability. > > Example use: > > > > ------------------------------------- > > import commands > > > > cmd = 'top -b -n2' > > > > def fancy(out): > > print 'GOT(%s)' % out.strip() > > > > commands.cb = fancy > > > > (s,o) = commands.getstatusoutput(cmd) > > print 'OUTPUT (%s)' % o > > ------------------------------------- > > > > I am not sure if this is the proper forum or means to submit something > like > > this, so please forgive me and advise accordingly if I am in error. The > > basic idea is obvious, to allow long-running commands to call back > whenever > > there is output. This is against python 2.4. Please let me know if you > > have any questions or suggestions. > > > > thanks!! > > -brad > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/5eb2f554/attachment.html From fwierzbicki at gmail.com Wed Jul 5 17:47:03 2006 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Wed, 5 Jul 2006 11:47:03 -0400 Subject: [Python-Dev] Import semantics In-Reply-To: <200607051818.02742.anthony@interlink.com.au> References: <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> <200607051818.02742.anthony@interlink.com.au> Message-ID: <4dab5f760607050847s6a128435v8ad31262e697ceef@mail.gmail.com> > In that case, why not post a news item saying this? The website is > probably the first place people look... > I think any news other than "here is the beta -- follow this link to download it" would be kind of a waste at this point. And that will come when I finish __slots__, subclassable "type", and figure out how to put a release together, which will be "real soon now" -- but really. Thanks, -Frank From guido at python.org Wed Jul 5 17:50:10 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 17:50:10 +0200 Subject: [Python-Dev] Patch for commands.py to provide callback In-Reply-To: References: Message-ID: Hm. It sounds like your company would be better off developing a library of handy tools that it uses over and over, instead of on the one hand using only standard library, and then if the standard library doesn't provide the feature you need, proposing a patch. I wouldn't be so critical if I thought others would benefit from your patch. But it seems a rather arbitrary place to insert a debugging hook (if that's what you need it for), and I don't think I've seen others with the same need. Or perhaps you could start by explaining the problem you are trying to solve before proposing a specific patch? --Guido On 7/5/06, Brad Doctor wrote: > Because it is a great way to consistently use popen(). Rather than write > something specific each time, our site/company prefers to use commands to > keep it all consistent. > > -brad > > > On 7/5/06, Guido van Rossum wrote: > > Since commands.getstatusoutput() is such a trivial wrapper around > > os.popen(), why bother patching commands.py? > > > > On 7/5/06, Brad Doctor wrote: > > > Greetings all, > > > > > > I have attached a patch for commands.py to provide a callback ability. > > > Example use: > > > > > > ------------------------------------- > > > import commands > > > > > > cmd = 'top -b -n2' > > > > > > def fancy(out): > > > print 'GOT(%s)' % out.strip() > > > > > > commands.cb = fancy > > > > > > (s,o) = commands.getstatusoutput(cmd) > > > print 'OUTPUT (%s)' % o > > > ------------------------------------- > > > > > > I am not sure if this is the proper forum or means to submit something > like > > > this, so please forgive me and advise accordingly if I am in error. The > > > basic idea is obvious, to allow long-running commands to call back > whenever > > > there is output. This is against python 2.4. Please let me know if you > > > have any questions or suggestions. > > > > > > thanks!! > > > -brad > > > > > > _______________________________________________ > > > Python-Dev mailing list > > > Python-Dev at python.org > > > http://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > > > > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > > > > > > > > > > > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From facundobatista at gmail.com Wed Jul 5 17:51:47 2006 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 5 Jul 2006 12:51:47 -0300 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: 2006/7/4, Guido van Rossum : > > This affect all the sockets. > > So, assuming your app is single-threaded, set the timeout, call > urlopen(), and reset the timeout to None. No, it's multithreaded, :D > > And I hit the problem when servicing > > information with a web service (TCPServer), and I need to timeout the > > connection of the URLOpen *only*. > > That's not so easy even if you were to have a timeout parameter to > urlopen(). You'd have to implement that peculiarity in all the layers > (in this case, urllib and httplib; and possibly ftplib, gopherlib etc. > :-) Yes, it's not SO easy, because, as you said, you have to dig into the layers until you hit the actual socket creation and modify the timeout for that socket only. That's why I think that this should be handled in the standard library and not left to implement to whoever will need it, :) -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From pje at telecommunity.com Wed Jul 5 17:53:50 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 05 Jul 2006 11:53:50 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060705113910.04698e50@sparrow.telecommunity.com> At 10:21 AM 7/5/2006 +0200, Guido van Rossum wrote: >Thanks for bringing this up. I'm not sure what I think of it yet. One >problem I see is that there might end up being two ways to reference >variables in outer scopes: .num if you plan to assign to it, or just >num if you only reference it. I find that the most disurbing issue so >far; modified global declarations or outer declarations don't have >this problem. Well, you could make it mandatory in Py3K I suppose, though I'm not sure I like it being mandatory, due to the frequent need to reference top-level names that would cause an awful lot of dots to start popping up. But for existing Python, the optional nature of the '.' allows existing code to run unchanged. And for versions that support the new syntax, using a leading '.' for all non-global, non-locals should be considered "good style" since it highlights the dependency and practically shouts "tricky stuff here, pay attention." Ironically, having *only* one way to refer to outer variables makes it impossible to communicate this distinction in the common read-only case. >Would this also use ..num to refer to num in an outer scope two levels >removed? I think that's unnecessary; it would be much better to use variables with distinct names. By the way, an interesting thought for Py3K is that you could maybe use this syntax to do away with explicit 'self', if you consider the class' namespace to be part of a function's closure. E.g.: class Foo: whee = 42 def bar(baz): print .whee Consider this: if Foo were a function rather than a class, each invocation of Foo would yield a new namespace in which 'whee' is defined. However, each invocation of a *class* also yields a new namespace. So there's a definite symmetry in using .whee to refer to an instance attribute of Foo. The big problem that comes to mind with that idea is that it makes it impossible to have argument names that are the same as attribute names, unless the 'whee'/'.whee' prohibition were relaxed. :( But it's an intriguing thought, nonetheless. From guido at python.org Wed Jul 5 17:56:25 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 17:56:25 +0200 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: OK, you've convinced me. Now where's that SF patch you were promising? :-) --Guido On 7/5/06, Facundo Batista wrote: > 2006/7/4, Guido van Rossum : > > > > This affect all the sockets. > > > > So, assuming your app is single-threaded, set the timeout, call > > urlopen(), and reset the timeout to None. > > No, it's multithreaded, :D > > > > > And I hit the problem when servicing > > > information with a web service (TCPServer), and I need to timeout the > > > connection of the URLOpen *only*. > > > > That's not so easy even if you were to have a timeout parameter to > > urlopen(). You'd have to implement that peculiarity in all the layers > > (in this case, urllib and httplib; and possibly ftplib, gopherlib etc. > > :-) > > Yes, it's not SO easy, because, as you said, you have to dig into the > layers until you hit the actual socket creation and modify the timeout > for that socket only. > > That's why I think that this should be handled in the standard library > and not left to implement to whoever will need it, :) > > -- > . Facundo > > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Jul 5 17:57:03 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 05 Jul 2006 11:57:03 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44AB8918.1040308@scottdial.com> References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060705115525.045c4cb0@sparrow.telecommunity.com> At 05:40 AM 7/5/2006 -0400, Scott Dial wrote: >Guido van Rossum wrote: > > Would this also use ..num to refer to num in an outer scope two levels > removed? > >I realize this was a wink, but it is a valid problem with the >"dot"-proposal. Actually, it isn't. :) See below. >def foo(n): > def bar(n): > def baz(): > return .n > >So, which 'n' outer 'n' is being referenced? Notice that this is a made-up example. Existing Python scoping rules don't allow this! Thus your example is not a bug, it's a feature request. And I say we say "no" to adding this feature. ;-) From guido at python.org Wed Jul 5 17:58:15 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 17:58:15 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5.1.1.6.0.20060705113910.04698e50@sparrow.telecommunity.com> References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> <5.1.1.6.0.20060705113910.04698e50@sparrow.telecommunity.com> Message-ID: On 7/5/06, Phillip J. Eby wrote: > By the way, an interesting thought for Py3K is that you could maybe use > this syntax to do away with explicit 'self', if you consider the class' > namespace to be part of a function's closure. Sorry, but now I am *definitely* -1. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fwierzbicki at gmail.com Wed Jul 5 18:21:31 2006 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Wed, 5 Jul 2006 12:21:31 -0400 Subject: [Python-Dev] Import semantics In-Reply-To: References: <448D1F0D.7000405@strakt.com> <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> Message-ID: <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com> On 7/5/06, Guido van Rossum wrote: > Hi Frank, > > Have you and/or the Jython community made up your mind about this? The > thread seems to have disappeared after you posted (or perhaps it > continued only on jython-dev, which I don't read?). The thread pretty much stopped there. I think a modification of the import semantics won't really come up again until 2.3 is out, since releases and reviving Jython take priority over backwards incompatible features. For my part I would like to think that a goal for Jython should be: Pure Python code developed first on Jython should run without change on CPython. Which would mean we will eventually need to change the import semantics for importing Python code (though I doubt we will change the semantics for importing Java packages any time soon). Whether that can be done in 2.x, or if this change is so incompatible that we need to think about it in a "Jython 3000" way, I don't really know. > > Also, I just realized that you're the new Jython maintainer. Is *that* > official? It is official, at least in the unofficial way that Jython appears to work: Brian handed the baton to me after (I presume) Samuele Pedroni had handed the baton to Brian. Brian's email is here: http://sourceforge.net/mailarchive/message.php?msg_id=13859029 That said, I still regard Samuele Pedroni as the ultimate authority on Jython and give him pretty much full veto power. He fortunately continues to watch the checkins and prods me when I go in the wrong direction. > I'd like to offer you my congratulations, and, more > importantly, any support you might need. Thanks! > I find Jython an important > part for Python's long-term stategy. That's good to know. > I'm asked occasionally what the > status of Jython is; people point out that the last release was 2.1 > many years ago and the website has no news since early 2005; they're > afraid that Jython is dying and that it's not a viable choice for new > projects. > I'm very happy to be able to tell them that soon there will > be a 2.3 release and yes there *is* continued support... Perhaps for large values of "soon" -- but seriously, I am working hard to polish the coming release, make it easier for new developers to read the code, and when I have a chance, update the website. Jython is my first serious plunge into open source contributions (it is really too bad that I couldn't have been a journeyman for another year or so first, but circumstances did not allow that). I have suffered from some over-optimism when asked for release dates, so I'm really afraid to give too sharp of a definition for "soon". That said, I believe a 2.2 version will be out sometime this summer, and a 2.3 should follow relatively quickly (maybe 6 months or so) The 2.2->2.3 will (hopefully) be relatively quick because 2.2 is an unfortunate but at this point unavoidable mix of 2.2 and 2.3 features, with heavy favoring of 2.3 features. Jython has had a history of having a very small number of developers, and a history of too much off-list discussions so that new developers have a very hard time catching up. I'm trying hard to change that culture. There are 4 or 5 developers contributing patches lately -- so I'm actually spending more time trying to help them along instead of concentrating 100% on a release -- I really think they are the most important predictor of the future of Jython. I have high hopes for the project's future health. Anyhow, this post is getting long, so to sum up: Jython is alive, and we aren't going to let it die. Regards, -Frank From aleaxit at gmail.com Wed Jul 5 18:42:54 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Wed, 5 Jul 2006 09:42:54 -0700 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: What about doing it with a per-thread-timeout in TLS (overriding the global one if a thread does have it set in its TLS)? Not as clean, but perhaps far easier to implement than patching dozens of modules/functions/classes to provide timeout= options everywhere... Alex On 7/5/06, Guido van Rossum wrote: > OK, you've convinced me. Now where's that SF patch you were promising? :-) > > --Guido > > On 7/5/06, Facundo Batista wrote: > > 2006/7/4, Guido van Rossum : > > > > > > This affect all the sockets. > > > > > > So, assuming your app is single-threaded, set the timeout, call > > > urlopen(), and reset the timeout to None. > > > > No, it's multithreaded, :D > > > > > > > > And I hit the problem when servicing > > > > information with a web service (TCPServer), and I need to timeout the > > > > connection of the URLOpen *only*. > > > > > > That's not so easy even if you were to have a timeout parameter to > > > urlopen(). You'd have to implement that peculiarity in all the layers > > > (in this case, urllib and httplib; and possibly ftplib, gopherlib etc. > > > :-) > > > > Yes, it's not SO easy, because, as you said, you have to dig into the > > layers until you hit the actual socket creation and modify the timeout > > for that socket only. > > > > That's why I think that this should be handled in the standard library > > and not left to implement to whoever will need it, :) > > > > -- > > . Facundo > > > > Blog: http://www.taniquetil.com.ar/plog/ > > PyAr: http://www.python.org/ar/ > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com > From skip at pobox.com Wed Jul 5 18:49:22 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed, 5 Jul 2006 11:49:22 -0500 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: <17579.60818.203745.712478@montanaro.dyndns.org> Guido> OK, you've convinced me. Now where's that SF patch you were Guido> promising? :-) A starting point is probably the patch Georg referred to a couple days ago: Georg> There was one patch that did this: http://python.org/sf/723312. Alas, it's assigned to me and I let it get so stale that Martin asked the author to update it for 2.5 a couple months ago. Skip From skip at pobox.com Wed Jul 5 18:53:13 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed, 5 Jul 2006 11:53:13 -0500 Subject: [Python-Dev] Import semantics In-Reply-To: <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com> References: <448D1F0D.7000405@strakt.com> <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com> Message-ID: <17579.61049.476643.960849@montanaro.dyndns.org> Frank> That said, I still regard Samuele Pedroni as the ultimate Frank> authority on Jython and give him pretty much full veto power. He Frank> fortunately continues to watch the checkins and prods me when I Frank> go in the wrong direction. Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) Skip From aleaxit at gmail.com Wed Jul 5 18:54:43 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Wed, 5 Jul 2006 09:54:43 -0700 Subject: [Python-Dev] Import semantics In-Reply-To: <17579.61049.476643.960849@montanaro.dyndns.org> References: <448D1F0D.7000405@strakt.com> <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com> <17579.61049.476643.960849@montanaro.dyndns.org> Message-ID: In Italian that would be DBAV (Dittatore benevolo a vita)...;-) Alex On 7/5/06, skip at pobox.com wrote: > > Frank> That said, I still regard Samuele Pedroni as the ultimate > Frank> authority on Jython and give him pretty much full veto power. He > Frank> fortunately continues to watch the checkins and prods me when I > Frank> go in the wrong direction. > > Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) > > Skip > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com > From fwierzbicki at gmail.com Wed Jul 5 19:13:39 2006 From: fwierzbicki at gmail.com (Frank Wierzbicki) Date: Wed, 5 Jul 2006 13:13:39 -0400 Subject: [Python-Dev] Import semantics In-Reply-To: <17579.61049.476643.960849@montanaro.dyndns.org> References: <448D1F0D.7000405@strakt.com> <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com> <17579.61049.476643.960849@montanaro.dyndns.org> Message-ID: <4dab5f760607051013k3d29f39er2583d075225e7635@mail.gmail.com> On 7/5/06, skip at pobox.com wrote: > > Frank> That said, I still regard Samuele Pedroni as the ultimate > Frank> authority on Jython and give him pretty much full veto power. He > Frank> fortunately continues to watch the checkins and prods me when I > Frank> go in the wrong direction. > > Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) > > Skip > I wonder if Samuele even wants that role? Anyway, I believe Samuele has more experience with Jython as it is currently implemented than anyone else, so I must take anything he says about Jython very seriously. However, when it comes to pure *Python* matters (no C and no J) which includes half of Jython's import semantics, I think it is still Guido's opinion that matters most. Without that, there is too much chaos. -Frank From mcherm at mcherm.com Wed Jul 5 19:18:21 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 05 Jul 2006 10:18:21 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) Message-ID: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> Guido writes: [discussion of how to fix the can't-bind-outer-scope-vars wart] > I think we have to continue to search for a solution that extends the > idea of global declarations. > > I've proposed extending its meaning to refer to the nearest outer > scope where the variable is set; if there is no such scope it's an > error. This will break a small number of program but probably not very > many; still, it'll require a future statement or waiting until Python > 3.0. The downside is that "global" is not a very intuitive word for > this new meaning. I disagree with your last statement -- I think "global" _is_ a very intuitive word for this. As I understand it, in programming "global" has two meanings, closely intertwined. One is "universal, same throughout the system". For instance, "The singleton pattern is used to create a single, global instance of a type." The second meaning is the term "global variable". This term developed (I believe) in languages that had only two scopes: local-to-current-function and global-to-entire-program. But the term "global variable" refers to any variable whose assignment is a "side effect", regardless of whether that variable is global-to-entire-program, global-to-module, or even global-to-enclosing-function. I have even heard the term "global variable" (mis)used to refer to any kind of side effect. Anyhow, in Python only builtins is _really_ global -- even today's global keyword only refers to module scope. So I believe that it would be a very reasonable interpretation of "global" to mean "not local", and implement as "search enclosing scopes in order to find the binding". -- Michael Chermside From skip at pobox.com Wed Jul 5 19:19:53 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed, 5 Jul 2006 12:19:53 -0500 Subject: [Python-Dev] Import semantics In-Reply-To: References: <448D1F0D.7000405@strakt.com> <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com> <17579.61049.476643.960849@montanaro.dyndns.org> Message-ID: <17579.62649.924278.973929@montanaro.dyndns.org> Skip> Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) Alex> In Italian that would be DBAV (Dittatore benevolo a vita)...;-) Damn Google Translator. File a bug report for me please Alex (or Guido or Jeremy or Neal or ...). ;-) Skip From guido at python.org Wed Jul 5 19:27:41 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 19:27:41 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> Message-ID: On 7/5/06, Michael Chermside wrote: > Guido writes: > [discussion of how to fix the can't-bind-outer-scope-vars wart] > > I think we have to continue to search for a solution that extends the > > idea of global declarations. > > > > I've proposed extending its meaning to refer to the nearest outer > > scope where the variable is set; if there is no such scope it's an > > error. This will break a small number of program but probably not very > > many; still, it'll require a future statement or waiting until Python > > 3.0. The downside is that "global" is not a very intuitive word for > > this new meaning. > > I disagree with your last statement -- I think "global" _is_ a very > intuitive word for this. As I understand it, in programming "global" > has two meanings, closely intertwined. One is "universal, same > throughout the system". For instance, "The singleton pattern is used > to create a single, global instance of a type." The second meaning is > the term "global variable". This term developed (I believe) in > languages that had only two scopes: local-to-current-function and > global-to-entire-program. But the term "global variable" refers to > any variable whose assignment is a "side effect", regardless of > whether that variable is global-to-entire-program, global-to-module, > or even global-to-enclosing-function. I have even heard the term > "global variable" (mis)used to refer to any kind of side effect. > > Anyhow, in Python only builtins is _really_ global -- even today's > global keyword only refers to module scope. So I believe that it > would be a very reasonable interpretation of "global" to mean > "not local", and implement as "search enclosing scopes in order > to find the binding". I really wish I could agree with you, because that would make the choice so much easier. However I still don't believe "global" has the stretchiness in its meaning that you claim it has. Have you ever heard a Python programmer talking about closures use the word "global variable"? Are there any other native speakers who side with Michael? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Jul 5 19:29:59 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Jul 2006 19:29:59 +0200 Subject: [Python-Dev] Import semantics In-Reply-To: <4dab5f760607051013k3d29f39er2583d075225e7635@mail.gmail.com> References: <448D1F0D.7000405@strakt.com> <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com> <17579.61049.476643.960849@montanaro.dyndns.org> <4dab5f760607051013k3d29f39er2583d075225e7635@mail.gmail.com> Message-ID: On 7/5/06, Frank Wierzbicki wrote: > On 7/5/06, skip at pobox.com wrote: > > > > Frank> That said, I still regard Samuele Pedroni as the ultimate > > Frank> authority on Jython and give him pretty much full veto power. He > > Frank> fortunately continues to watch the checkins and prods me when I > > Frank> go in the wrong direction. > > > > Does that make Samele the DBPV (Dictator benevolo per vita)? ;-) > > > > Skip > > > I wonder if Samuele even wants that role? > > Anyway, I believe Samuele has more experience with Jython as it is > currently implemented than anyone else, so I must take anything he > says about Jython very seriously. However, when it comes to pure > *Python* matters (no C and no J) which includes half of Jython's > import semantics, I think it is still Guido's opinion that matters > most. Without that, there is too much chaos. Actually, I started this discussion because Samuele told me that there was a new Jython maintainer, and I jumped out of my chair of joy. Samuele is happy to give advice and otherwise help out when asked, but he also seems happy to delegate everything to you, which sounds like a great endorsement. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mcherm at mcherm.com Wed Jul 5 19:31:23 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 05 Jul 2006 10:31:23 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) Message-ID: <20060705103123.fvv54v7e3s0ko8wo@login.werra.lunarpages.com> Phillip Eby writes: > I don't see a problem with requiring '.x' to be used for both > reading and writing of outer-scope names; it just shouldn't be > required for an outer-scope name that you don't rebind in the > current scope. > > def counter(num): > def inc(): > .num += 1 > return .num > return inc I am reminded of Tim Peter's declaration in response to a similar proposal some time ago: Syntax should not look like grit on my monitor. (Sorry, no reference... but I swear it's word-for-word accurate because the quote burned itself into my memory.) -- Michael Chermside From mcherm at mcherm.com Wed Jul 5 19:31:25 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 05 Jul 2006 10:31:25 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) Message-ID: <20060705103125.4rc33rnnzg8w0w4c@login.werra.lunarpages.com> Phillip Eby writes: > The big problem that comes to mind with that idea is that it makes > it impossible to have argument names that are the same as attribute > names, unless the 'whee'/'.whee' prohibition were relaxed. :( But > it's an intriguing thought, nonetheless. My three-year-old has been working on that 'whee'/'.whee' prohibition, but he hasn't mastered it yet. Gotta-go-wash-another-load-of-underpants -lly yours, Michael Chermside From pje at telecommunity.com Wed Jul 5 19:43:55 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 05 Jul 2006 13:43:55 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> Message-ID: <5.1.1.6.0.20060705133753.026a7698@sparrow.telecommunity.com> At 07:27 PM 7/5/2006 +0200, Guido van Rossum wrote: >However I still don't believe "global" has the stretchiness in its >meaning that you claim it has. Have you ever heard a Python programmer >talking about closures use the word "global variable"? > >Are there any other native speakers who side with Michael? I don't, and am -1 on using "global" to mean "outer". Of course, I also don't really care for using "global" to refer to a module level variable, or it being the only non-executable statement in Python, but these are very minor things and relatively easily explained to the programmers I've worked with. Using "global" to mean "outer", on the other hand, would surely lead to much wailing and grinding of teeth in response to any attempt at rational explanation. :) From nick at craig-wood.com Wed Jul 5 19:57:33 2006 From: nick at craig-wood.com (Nick Craig-Wood) Date: Wed, 5 Jul 2006 18:57:33 +0100 Subject: [Python-Dev] Time-out in URL Open In-Reply-To: References: Message-ID: <20060705175733.3A06114C1BD@irishsea.home.craig-wood.com> Alex Martelli wrote: > What about doing it with a per-thread-timeout in TLS (overriding the > global one if a thread does have it set in its TLS)? Not as clean, > but perhaps far easier to implement than patching dozens of > modules/functions/classes to provide timeout= options everywhere... Yes please! I wrote a sketch of a module which did this on c.l.py recently http://groups.google.com/group/comp.lang.python/browse_thread/thread/d897c00b67cadca5/fd2ceb4e014de7ce?lnk=st&q=TimeoutError&rnum=2&hl=en#fd2ceb4e014de7ce It would be much better if it had help from the core though. -- Nick Craig-Wood -- http://www.craig-wood.com/nick From brett at python.org Wed Jul 5 20:56:09 2006 From: brett at python.org (Brett Cannon) Date: Wed, 5 Jul 2006 11:56:09 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: References: Message-ID: On 7/4/06, Ka-Ping Yee wrote: > > Hi Brett, > > Here are some comments on the description of the restricted execution > model that you posted. > > > When referring to the state of an interpreter, it is either "trusted" or > > "untrusted". A trusted interpreter has no restrictions imposed upon any > > resource. An untrusted interpreter has at least one, possibly more, > resource > > with a restriction placed upon it. > > In response to Guido's comment about confusing the words "trusted" and > "untrusted", how about "empowered" and "restricted"? Maybe. I am really starting to lean towards trusted and sandboxed. > When the Interpreter Is Embedded > > ================================ > > > > Single Untrusted Interpreter > > ---------------------------- > > > > This use case is when an application embeds the interpreter and never > has more > > than one interpreter running. > > > > The main security issue to watch out for is not having default abilities > be > > provided to the interpreter by accident. > > I'd rather rephrase this in the opposite direction. The onus shouldn't > be on the application to hunt down each possible dangerous authority and > deactivate them all one by one. The main security issue is to let the > application choose which abilities it wants the restricted interpreter > to have, and then ensure that the restricted interpreter gets only those > abilities. Right. I am thinking more of an implementation screw up that somehow provides access to an object that has escalated rights. > Multiple Untrusted Interpreters > > ------------------------------- > > > > When multiple interpreters, all untrusted at varying levels, need to be > > running within a single application. This is the key use case that this > > proposed design is targetted for. > > > > On top of the security issues from a single untrusted interpreter, > > there is one additional worry. Resources cannot end up being leaked > > into other interpreters where they are given escalated rights. > > What is your model here for communication between interpreters? If two > interpreters can communicate, any attempt to "prevent leakage" of > resources is meaningless. When you say "leaked into other interpreters" > are you talking about a Python object leaking or something else at a > lower level? I am talking about Python objects. As for communication, I was planning on something included directly in globals or some custom object to handle that. I have not been focusing on that aspect so far. Suppose for example that the application wants to embed two interpreters, > P and Q, and that the application wants P to be able to write files but > Q to be restricted against writing files. When you say "leaked" above, > that suggests to me that you want to prevent something like > > # code running in P > import spam > f = open('/home/doofus/.ssh/authorized_keys', 'a') > spam.f = f > > # code running in Q > import spam > spam.f.write('blargh') > > The above example supposes that P and Q can communicate through a > shared module, spam, where they can pass Python objects. Right. But Python modules are separate per interpreter and only C extension modules are in any way shared between interpreters. But sharing an open file like that is bad and why C extension modules must be whitelisted to be used. But notice that even if you prevent them from passing Python objects > like open files, any form of communication is sufficient to leak > resources: > > # code running in P > def add_key(key): > f = open('/home/doofus/.ssh/authorized_keys', 'a') > f.write(key + '\n') > f.close() > > import socket > s = socket.socket() > s.bind(('', 6666)) > s.listen(1) > ns, addr = s.accept() > add_key(ns.recv(100)) > > > # code running in Q > import webbrowser > webbrowser.open('http://localhost:6666/zebra') > > As long as P can listen for instructions from Q, it can give Q > the power to write to the filesystem. Right, which is why sockets and files are restricted and turned off by default. You have to give explicit permission to use either resource. > Filesystem > > =================== > > > > The most obvious facet of a filesystem to protect is reading from it. > > One does not want what is stored in ``/etc/passwd`` to get out. And > > one also does not want writing to the disk unless explicitly allowed > > for basically the same reason; if someone can write ``/etc/passwd`` > > then they can set the password for the root account. > > There's a big difference between modifying (or erasing) an existing file > and writing a new file (e.g. for temporary storage). If i give you a > little filesystem of your own to play in, and it starts out empty, you > can put whatever you want in it without violating my secrecy or the > integrity of my files. > > I think you should be talking about this in terms of specifically > what abilities you want to be able to allow, based on examples of > real-life applications. Fair enough. But since you have the ability to only list files specifically, you can give temporary file access by giving access to such a non-existent file for writing. If you don't like an existing file then you don't get access to it. > Physical Resources > > =================== > > > > Memory should be protected. It is a limited resource on the system > > that can have an impact on other running programs if it is exhausted. > > Being able to restrict the use of memory would help alleviate issues > > from denial-of-service (DoS) attacks. > > > Networking > > =================== > > > > Networking is somewhat like the filesystem in terms of wanting similar > > protections. You do not want to let untrusted code make tons of socket > > connections or accept them to do possibly nefarious things (e.g., acting > > as a zombie). > > > > You also want to prevent finding out information about the network you > are > > connected to. This includes doing DNS resolution since that allows one > > to find out what addresses your intranet has or what subnets you use. > > Again, it's risky to describe only individual cases of things to > prevent. What networking abilities are safe or necessary for the > kinds of applications you have in mind? Start from nothing and > work up from there. That's the plan. I am planning to go through socket function by function and explicitly allow access as warranted and block everything else. It is not going to be "let's block DNS and allow everything else". Sorry if that wasn't clear. This is mostly just to say "I plan on restricting this kind of stuff, here is an example". > Interpreter > > =================== > > > > One must make sure that the interpreter is not harmed in any way. > > There are several ways to possibly do this. One is generating > > hostile bytecode. Another is some buffer overflow. In general any > > ability to crash the interpreter is unacceptable. > > This is hard for me to understand. What exactly do you trust and > not trust? It seems to me that crashing an interpreter is only a > problem if a single interpreter is running both trusted and untrusted > code -- then if the untrusted code crashes the interpreter, the > trusted code suffers. > > But there doesn't seem to be any such thing in your model. Each > interpreter is either trusted or untrusted. If the interpreter is > trusted, and the code running in it causes it to crash, i assume > you would consider that to be the code's "own fault", right? > And if the interpreter is untrusted, and the code running in it > causes it to crash, then the code has only harmed itself. > > It seems to me that we need only be concerned about crashing when > the crash of an embedded interpreter will bring down its host > application, or there are multiple interpreters embedded at once > and one interpreter causes another interpreter to crash. Right. But being embedded, won't any segfault of an interpreter bring down the embedded application? But you are correct, I am only concerned with preventing a crash of a sandboxed interperter. > Resource Hiding > > ============================= > [...] > > This can be viewed as a passive system for security. > [...] > > Resource Crippling > > ============================= > > Another approach to security is to provide constant, proactive security > > checking of rights to use a resource. > > I think you have this backwards. Resource hiding is proactive: > before untrusted code has a chance to abuse anything, you decide > what you want to allow it to do. It defaults to no access, and > only gets access to resources you have proactively decided to provide. I am using "proactive" as in constantly checking the security model. Resource crippling is the opposite: it begins by giving carte blanche > to the untrusted code, then you run around trying to plug holes > by stopping everything you don't want. This is a lot more work, > and it is also much more dangerous. If you forget to plug even > one hole, you're hosed. Yeah, I know, which is why I am only bothering with 'file' and 'socket'. Back to what you wrote about resource hiding: > > > This can be viewed as a passive system for security. Once a resource > > has been given to code there are no more checks to make sure the > > security model is being violated. > > This last sentence doesn't make any sense. If you decided to give > the resource, how is using the resource a violation? Either you > want to enable the resource or you don't. If you want to enable > it, give it; if you don't, don't give it. As a criticism of the > resource hiding approach, it's a red herring -- there's no way > to interpret this sentence that doesn't make it also an > unfalsifiable criticism of any possible security model. Yeah, I figured that out after I wrote this. > The most common implementation of resource hiding is capabilities. > > In this type of system a resource's reference acts as a ticket that > > represents the right to use the resource. Once code has a reference > > it is considered to have full use of that resource it represents and > > no further security checks are performed. > > Same thing. What "further security checks" are we worried about? > Woult it check to see whether we've authorized the interpreter to > have access to the resource ... which we already know to be true? > > > To allow customizable restrictions one can pass references to wrappers > of > > resources. This allows one to provide custom security to resources > instead of > > requiring an all-or-nothing approach. > > The ability to customize security restrictions is an important > advantage of the resource hiding approach, since resource crippling > requires that the architect of the security model anticipate every > possible security restriction that future programmers might need. > > Using resource crippling is analogous to removing "def" from the > language and requiring Python programmers to only use functions > that are provided in the built-in modules instead of writing their > own functions. > > > To use an analogy, imagine you are providing security for your home. > > With capabilities, security came from not having any way to know > > where your house is without being told where it was; a reference > > to its location. You might be able to ask a guard (e.g., Java's > > ClassLoader) for a map, but if they refuse there is no way for you > > to guess its location without being told. But once you knew where > > it was, you had complete use of the house. > > This analogy is only fair if you compare it to the same analogy for > the resource crippling approach. Resource crippling doesn't get you > any finer-grained control either! The comparison story is: > > With resource crippling, security comes from having a guard > at the door to your house. When a Python interpreter comes > up to the door, the guard checks to see if the interpreter > has permission to enter the house, and if it does, then it > gets complete use of the house. > > Why is the granularity of control described as the whole house > in the resource-hiding story, but as each door in the house in > the resource-crippling story? Because, as you said above, if you want someone to have the resource (the house, or in more concrete terms, a 'file') you just give it to them. If you cripple it, though, you might provide a 'file' object but restrict how many bytes are written. But I also realize that resource hiding handles this by providing a wrapper that provides the protection. > And that complete access is an issue with a capability system. > > If someone played a little loose with a reference for a resource > > then you run the risk of it getting out. > > Could you be more specific about what you mean by "it getting out"? Out of a trusted interpreter and ending up in a sandboxed interpreter some how. If you mean getting from a trusted interpreter to an untrusted > interpreter -- then how is a resource going to travel between > interpreters? Beats me, but I am always scared of Armin and Samuele. =) It seems that your criticisms are aimed at resource crippling being a "plug holes as needed but if you foul up you are screwed" with resource hiding being more "fix the fundamental issues and just don't present access to resources you don't want to give access to (or wrap accordingly)". And in general I agree with this assessment. But I also realize that Python was not designed for security in mind and there seems to be new ways to get access to 'file'. If I felt confident that I could find and hide 'file' as needed, I would go that route immediately. But I don't think I can (and Armin has said this as well). If you think you can help figure out every place a reference to 'file' can be found through the standard interpreter, then fine, let's go that way. I just don't have faith this can be done effectively. -Brett Or if not, then are you thinking of a situation in which one > piece of code is trusted with the resource, but another piece of > code is not, and both are running in the same interpreter? > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/8fe64cfc/attachment.htm From brett at python.org Thu Jul 6 00:12:33 2006 From: brett at python.org (Brett Cannon) Date: Wed, 5 Jul 2006 15:12:33 -0700 Subject: [Python-Dev] branch for sandox work created: bcannon-sandboxing Message-ID: I have created a branch in svn off of HEAD for holding the sandboxing work. It's called bcannon-sandboxing and I have checked in the design doc in the root as sandboxing_design_doc.txt . You can keep an eye on the checkout message for incremental changes, but I will email the list once I have gone through at least one thorough revision. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/101ed16e/attachment.htm From mcherm at mcherm.com Thu Jul 6 00:47:50 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 05 Jul 2006 15:47:50 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python Message-ID: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> Ka-Ping Yee writes: > If you mean getting from a trusted interpreter to an untrusted > interpreter -- then how is a resource going to travel between > interpreters? Brett Cannon responds: > Beats me, but I am always scared of Armin and Samuele. =) Okay, those two scare me also, but I would still rather not spread FUD. Your proposal contains lots of details about how to address the danger that Python objects can cross from one interpreter to another. Could we instead attack that straight-on and try to find a convincing proof that objects cannot possibly cross the interpreter barrier? If so, it would simplify a bit of your proposal, and make me feel a little less worried. -- Michael Chermside From mcherm at mcherm.com Thu Jul 6 00:51:49 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 05 Jul 2006 15:51:49 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] Message-ID: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> In response to Ka-Ping's comments on the subject of "Resource Hiding" vs "Resource Crippling", Brett says: > It seems that your criticisms are aimed at resource crippling > being a "plug holes as needed but if you foul up you are screwed" > with resource hiding being more "fix the fundamental issues and > just don't present access to resources you don't want to give > access to (or wrap accordingly)". And in general I agree with > this assessment. But I also realize that Python was not designed > for security in mind and there seems to be new ways to get access > to 'file'. If I felt confident that I could find and hide 'file' > as needed, I would go that route immediately. But I don't think I > can (and Armin has said this as well). I agree completely. "Resource Hiding" (specifically, Capabilities) has the cleanest concept, yet there are valid reasons to worry about implementing it in Python. However, I would like to point out one other advantage that capabilities has over "Resource Crippling". Resource Crippling implements the restrictions as changes in the underlying C-level objects capable of performing dangerous operations. That means that the restrictions are implemented by the interpreter. The advantage is obvious: we can trust the interpreter. But the disadvantages are (1) it's slow to fix (requires a bugfix release followed by everyone in the world upgrading), and (2) it cannot be extended by the user. With resource crippling, you will need to decide just what kind of restrictions the file type will implement. I believe you are planning to restrict to a list of known filenames and known directories for reading and for writing. (Actually, you said mode string, but I presume that you won't maintain separate lists for 'r' and 'rb' modes.) Then there was discussion of whether the directories ought to be recursive, whether the total number of files opened ought to be restricted, whether the total size written should be restricted, and even whether the size should be measured in bytes or blocks. Such conversations could go on for a long time, and in the end you must make some compromises. If you were using capabilities, you would need to ensure that restricted interpreters could only get the file object that they were given. But then _all_ of these fancy versions of the restrictions would be immediately supported: it would be up to the users to create secure wrappers implementing the specific restrictions desired. I really like this feature of capabilities: that they can be extended (well, restricted) by the user, not just by the language implementer. That's a powerful feature, and I don't want to give it up. But on the other hand, I don't quite see how to maintain it -- here are my best ideas, perhaps they will help. Python already has one essential ingredient for capabilities: unforgable references. But it fails in two other ways: having secure data, and having no auxiliary means of accessing objects. Python's powerful introspection and visible implementation (eg: __dict__) make it impossible to encapsulate data in an object in a way that prevents malicious users from accessing it. But that is actually surprisingly easy to fix. Just create a new type (maybe a new metaclass), implemented in C, which contains private data and a means to control access to it. You would provide a dict which would be stored privately without access from Python, and then provide methods and attributes along with a Python function for evaluating access to each. The type would ensure that the access test was evaluated with access to the private dict before any method or attribute was accessed. Such a construct is simple enough that I believe we could implement it and be reasonably confident that it was reliably secure. (I have played with this in the past and been reasonably pleased with the results.) Adding restrictions would then incur some performance penalties, but that seems unproblematic. That leaves the other problem: auxiliary means of accessing objects. There are things like gc.get_objects(). In the special case of file, which is a type that's also dangerous, there are tricks like "object().__class__.__subclasses__()". I would love to believe that we could plug all of these holes, but experience (rexec) proves otherwise. For something like sockets, I am fairly sure that there's a clear bottleneck (the low-level socket module), but still numerous existing libraries that use this low-level module without being handed a capability. So this is where my alternative plan starts to fall apart. Your (Brett's) plan to use resource crippling for these kinds of restrictions involves putting C code around all direct access to files, sockets, or whatever resource is being accessed. Perhaps instead of your C code doing the security checks directly, it could make sure that the objects returned were contained within the correct secure wrappers. That's OK so far as it goes, but the C checks are per interpreter instance, so how do we get them to apply the correct wrappers? Maybe the interpreter maintains (in C) a stack of wrappers per-thread and provides a means for stack frames (functions) to register wrappers? We would wind up wrapping the wrappers, but that's just the nature of capabilities. Perhaps it would look something like this: def invoke_user_function(): PyXXX.set_file_wrapper(ReadOnlyRestriction) PyXXX.set_socket_wrapper( SingleDomainRestriction('example.com')) untrusted_object.do_stuff() ... To sum up: I agree that you cannot rely on prevent all the possible "python tricks", but I still think that capabilities are a superior solution. I'd like to find a way to achieve the user-customizability of capabilities without throwing out the entire standard library -- maybe some hybrid of "resource hiding" and "resource crippling". I can almost see a way to achieve it, but there are still a few flaws. -- Michael Chermside From brett at python.org Thu Jul 6 01:11:14 2006 From: brett at python.org (Brett Cannon) Date: Wed, 5 Jul 2006 16:11:14 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> Message-ID: On 7/5/06, Michael Chermside wrote: > > Ka-Ping Yee writes: > > If you mean getting from a trusted interpreter to an untrusted > > interpreter -- then how is a resource going to travel between > > interpreters? > > Brett Cannon responds: > > Beats me, but I am always scared of Armin and Samuele. =) > > Okay, those two scare me also, but I would still rather not > spread FUD. I don't consider it FUD. Armin in an email said that he thought it was a losing battle to try to hide 'file' from an interpreter. That is what I am worried about, period. Everythign else can be protected through resource hiding. Your proposal contains lots of details about how to > address the danger that Python objects can cross from one > interpreter to another. Could we instead attack that straight-on > and try to find a convincing proof that objects cannot possibly > cross the interpreter barrier? If so, it would simplify a bit > of your proposal, and make me feel a little less worried. As I said to Ping, if people *really* think this is doable and are willing to help out with this, then fine, I am willing to give this a shot. But I know I don't personally know enough about every random corner of the code base like Armin and Samuele know in order to feel comfortable in claiming I can pull this off by myself. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/db937623/attachment.htm From brett at python.org Thu Jul 6 01:26:08 2006 From: brett at python.org (Brett Cannon) Date: Wed, 5 Jul 2006 16:26:08 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> Message-ID: On 7/5/06, Michael Chermside wrote: > > In response to Ka-Ping's comments on the subject of "Resource Hiding" > vs "Resource Crippling", Brett says: > > > It seems that your criticisms are aimed at resource crippling > > being a "plug holes as needed but if you foul up you are screwed" > > with resource hiding being more "fix the fundamental issues and > > just don't present access to resources you don't want to give > > access to (or wrap accordingly)". And in general I agree with > > this assessment. But I also realize that Python was not designed > > for security in mind and there seems to be new ways to get access > > to 'file'. If I felt confident that I could find and hide 'file' > > as needed, I would go that route immediately. But I don't think I > > can (and Armin has said this as well). > > I agree completely. "Resource Hiding" (specifically, Capabilities) > has the cleanest concept, yet there are valid reasons to worry > about implementing it in Python. However, I would like to point out > one other advantage that capabilities has over "Resource Crippling". > > Resource Crippling implements the restrictions as changes in the > underlying C-level objects capable of performing dangerous operations. > That means that the restrictions are implemented by the interpreter. > The advantage is obvious: we can trust the interpreter. But the > disadvantages are (1) it's slow to fix (requires a bugfix release > followed by everyone in the world upgrading), and (2) it cannot be > extended by the user. > > With resource crippling, you will need to decide just what kind of > restrictions the file type will implement. I believe you are > planning to restrict to a list of known filenames and known > directories for reading and for writing. (Actually, you said mode > string, but I presume that you won't maintain separate lists for > 'r' and 'rb' modes.) Then there was discussion of whether the > directories ought to be recursive, whether the total number of > files opened ought to be restricted, whether the total size written > should be restricted, and even whether the size should be measured > in bytes or blocks. Such conversations could go on for a long time, > and in the end you must make some compromises. > > If you were using capabilities, you would need to ensure that > restricted interpreters could only get the file object that they > were given. But then _all_ of these fancy versions of the > restrictions would be immediately supported: it would be up to the > users to create secure wrappers implementing the specific > restrictions desired. I agree. I would prefer this way of doing it. But as I have said, making sure that 'file' does not get out into the wild is tough. I really like this feature of capabilities: that they can be > extended (well, restricted) by the user, not just by the language > implementer. That's a powerful feature, and I don't want to give > it up. But on the other hand, I don't quite see how to maintain > it -- here are my best ideas, perhaps they will help. > > Python already has one essential ingredient for capabilities: > unforgable references. But it fails in two other ways: having > secure data, and having no auxiliary means of accessing > objects. Right. Private attributes only exist at the C level. Python's powerful introspection and visible implementation > (eg: __dict__) make it impossible to encapsulate data in an > object in a way that prevents malicious users from accessing > it. But that is actually surprisingly easy to fix. Just create > a new type (maybe a new metaclass), implemented in C, which > contains private data and a means to control access to it. You > would provide a dict which would be stored privately without > access from Python, and then provide methods and attributes > along with a Python function for evaluating access to each. The > type would ensure that the access test was evaluated with > access to the private dict before any method or attribute was > accessed. Such a construct is simple enough that I believe we > could implement it and be reasonably confident that it was > reliably secure. (I have played with this in the past and been > reasonably pleased with the results.) Adding restrictions would > then incur some performance penalties, but that seems > unproblematic. Right, but as you mention below this still does not protect the C level objects such as 'file'. If you can prevent references to 'file' from getting out, you can change open() to return wrapped instances of 'file' with the desired security measures in place. That leaves the other problem: auxiliary means of accessing > objects. There are things like gc.get_objects(). In the special > case of file, which is a type that's also dangerous, there are > tricks like "object().__class__.__subclasses__()". I would love > to believe that we could plug all of these holes, but experience > (rexec) proves otherwise. For something like sockets, I am > fairly sure that there's a clear bottleneck (the low-level > socket module), but still numerous existing libraries that use > this low-level module without being handed a capability. Right, but for modules that cheat, you just don't add to the whiltelist. This is why you should never blindly add modules to the whitelist of extension modules that you can import. So this is where my alternative plan starts to fall apart. Your > (Brett's) plan to use resource crippling for these kinds of > restrictions involves putting C code around all direct access > to files, sockets, or whatever resource is being accessed. Just 'file' and sockets, nothing else. Everything else is protected by not allowing importation of the module. Perhaps instead of your C code doing the security checks > directly, it could make sure that the objects returned were > contained within the correct secure wrappers. That's OK so far > as it goes, but the C checks are per interpreter instance, so > how do we get them to apply the correct wrappers? Maybe the > interpreter maintains (in C) a stack of wrappers per-thread and > provides a means for stack frames (functions) to register > wrappers? We would wind up wrapping the wrappers, but that's > just the nature of capabilities. Perhaps it would look > something like this: > > def invoke_user_function(): > PyXXX.set_file_wrapper(ReadOnlyRestriction) > PyXXX.set_socket_wrapper( > SingleDomainRestriction('example.com')) > untrusted_object.do_stuff() > > ... > > To sum up: I agree that you cannot rely on prevent all the > possible "python tricks", but I still think that capabilities > are a superior solution. And I have never disagreed with this. It is just a "practicality vs. purity" thing. I'd like to find a way to achieve > the user-customizability of capabilities without throwing > out the entire standard library -- maybe some hybrid of > "resource hiding" and "resource crippling". I can almost see > a way to achieve it, but there are still a few flaws. It's already a hybrid solution with the import protections for capabilities and the 'file' constructor lock-out for crippling. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/5fb3b1cb/attachment-0001.html From brett at python.org Thu Jul 6 02:01:48 2006 From: brett at python.org (Brett Cannon) Date: Wed, 5 Jul 2006 17:01:48 -0700 Subject: [Python-Dev] what can we do to hide the 'file' type? Message-ID: To make sure I don't unfairly block out capabilities as a complete security model instead of just crippling 'file's constructor (I do like capabilities and think it is a good model, really!), let's discuss how one can get to the 'file' type without importing any extension modules (that can be protected at the import level so I am ignoring the 'gc' module trick and such). First, it's in __builtin__. That reference can just be left out of the dict at the PyInterpreterState stuct's dict for built-ins. But we all know it isn't that simple. Second, there is __subclasses__(). That method could just not be allowed to be in the 'type' class at the Python level (hiding it, crippling it, whatever), but would that break much code? I don't know, but I doubt it. Third, for any wrappers returned by open(), it cannot be a subclass because chaining __class__ attribute, mro() (or any of the other methods provided on 'object' or 'type'), or type() will get you to the original 'file' type. The actual 'file' reference will need to be stored at the C struct level for the wrapper and not accessed except by the wrapper directly which would be implemented in C. Can anyone think of any other way to gain access to 'file' without importing a module? At that point one would need to be *very* careful about what an extension module exported to the world, but I can live with that (as that is already part of the plan). Please seriously try to think of ways to get to 'file' everybody. If we really cannot come up with anything beyond these three ways, then I am totally willing to go with a much more complete capabilities system for security in Python and really minimize any crippling. I just need to be convinced that we won't be plugging holes in how to hide 'file' rather than plugging holes from crippling 'file' (which, at this point, I am not convinced of). And if Armin and/or Samuele sign off that what we find is most likely (with "most likely" equalling 99% chance) all there is, then bonus points and I will *really* be convinced. =) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/60204925/attachment.html From greg.ewing at canterbury.ac.nz Thu Jul 6 02:07:37 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Jul 2006 12:07:37 +1200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com> References: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com> Message-ID: <44AC5449.5020104@canterbury.ac.nz> Marek "Baczek" Baczy?ski wrote: > I suggest <- as an assignment operator instead of := - it's used in > OCaml and it looks *very* different, yet still makes sense. But assigning to an outer scope isn't *very* different, it's only slightly different. -- "And now for something slightly different..." Greg From greg.ewing at canterbury.ac.nz Thu Jul 6 02:30:54 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Jul 2006 12:30:54 +1200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: <44AC59BE.2020503@canterbury.ac.nz> Simon Percivall wrote: > def foo(): > def bar(): > foo.x = 3 That already had a different meaning - it assigns to an attribute of the function object created by executing def foo(). -- Greg From greg.ewing at canterbury.ac.nz Thu Jul 6 02:33:39 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Jul 2006 12:33:39 +1200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <17579.49401.682772.29091@montanaro.dyndns.org> References: <5f3d2c310607050317t51d796d1g348fc2d036798128@mail.gmail.com> <2FD28BA2-32D9-49C4-8D86-92B0A53346A6@python.org> <17579.49401.682772.29091@montanaro.dyndns.org> Message-ID: <44AC5A63.2020602@canterbury.ac.nz> skip at pobox.com wrote: > By naming the function in which the binding is to occur you avoid problems > of someone coming along and adding or deleting functions between the > assignment (in baz) and the target of the assignment (x in foo) but then > forgetting to increment or decrement the counters that refer to a fixed > number of levels above the current function. But it doesn't do anything for the (I expect much more common) case of factoring out something in a function body and making it a nested function -- you'd still have to change the form of all the references to the name in that case. Better not to have a scheme that uses counters or scope names at all, I think. -- Greg From greg.ewing at canterbury.ac.nz Thu Jul 6 02:59:21 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Jul 2006 12:59:21 +1200 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> Message-ID: <44AC6069.3050905@canterbury.ac.nz> Michael Chermside wrote: > That leaves the other problem: auxiliary means of accessing > objects. There are things like gc.get_objects(). In the special > case of file, which is a type that's also dangerous, there are > tricks like "object().__class__.__subclasses__()". My approach to that would be to not provide access to these kinds of things via attributes, but via builtin functions. E.g there wouldn't be a __subclasses__ attribute, but a subclasses() function. Then that capability can be denied by not providing that function. -- Greg From greg.ewing at canterbury.ac.nz Thu Jul 6 03:01:04 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Jul 2006 13:01:04 +1200 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> Message-ID: <44AC60D0.1040508@canterbury.ac.nz> Brett Cannon wrote: > Armin in an email said that he thought it was > a losing battle to try to hide 'file' from an interpreter. And I would change file() so that it didn't open files. Then it would be harmless for code to have access to the file class. -- Greg From brett at python.org Thu Jul 6 03:09:54 2006 From: brett at python.org (Brett Cannon) Date: Wed, 5 Jul 2006 18:09:54 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <44AC6069.3050905@canterbury.ac.nz> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC6069.3050905@canterbury.ac.nz> Message-ID: On 7/5/06, Greg Ewing wrote: > > Michael Chermside wrote: > > > That leaves the other problem: auxiliary means of accessing > > objects. There are things like gc.get_objects(). In the special > > case of file, which is a type that's also dangerous, there are > > tricks like "object().__class__.__subclasses__()". > > My approach to that would be to not provide access to > these kinds of things via attributes, but via builtin > functions. E.g there wouldn't be a __subclasses__ > attribute, but a subclasses() function. Then that > capability can be denied by not providing that > function. __subclasses__ is a function. And yes, if we go this route, that is what would happen most likely. The trick is figuring out any and all ways one can get to 'file' from a standard interpreter prompt. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/796de055/attachment.htm From anthony at interlink.com.au Thu Jul 6 05:24:16 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 6 Jul 2006 13:24:16 +1000 Subject: [Python-Dev] ImportWarning decision Message-ID: <200607061324.19000.anthony@interlink.com.au> So there's 3 choices here: a) revert the importwarning entirely b) make it suppressed by default c) more complicated code in import.c to only emit the warning if the import fails. After a bit of a chat with Neal, I think the best combination of prudence and functionality is (b). (a) also works for me. (c) is a bit too scary, post-beta. import.c is a bad place where programmers go to die - I'd rather not mess about with it at this stage of the release cycle. Unless I hear screams in the near future, (b) is what's going to be done for beta2, and therefore 2.5 final. We can re-evaluate the situation for 2.6 - maybe a more complex solution like (c) can be done for that. This means Google can just turn it on in sitecustomize.py and Guido can avoid the hordes of peasants with pitchforks and burning torches. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Thu Jul 6 05:35:19 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 6 Jul 2006 13:35:19 +1000 Subject: [Python-Dev] Proposed beta 2 changes (Q for Anthony/Neal) In-Reply-To: <44AA5FE6.2030801@iinet.net.au> References: <44AA5FE6.2030801@iinet.net.au> Message-ID: <200607061335.21179.anthony@interlink.com.au> On Tuesday 04 July 2006 22:32, Nick Coghlan wrote: > 1. Finishing the __module_name__ workaround to allow relative > imports from the main module when using -m. > > I'd really like to finish this, because having PEP 328 and 338 > not playing well together is a wart that's quite visible to end > users. I'd rather not have people's first encounter with the > features provided by either PEP involve discovering that they're > "broken". > > The patch to fix this also takes care of adding a couple of > paragraphs to the tutorial about explicit relative imports (which > aren't currently written up in the main documentation). The patch > is attached directly to the beta 1 bug report about the problem > [1]. > > (Guido gave a +1 to the concept, but explicitly deferred to > Anthony and Neal as to whether or not the fix should go in for beta > 2) I have some nervousness about this. Are there cases in the stdlib where this is an issue, today? Are there any cases which can't be addressed by using absolute imports? For 2.5, wouldn't it be better to simply say "use absolute imports in this case"? My nervousness is that I don't want a late change to introduce a language misfeature that we'll all regret later. > 2. Adding an 'ignore' filter for ImportWarning at the end of > warnings.py Already covered this one in another email... Yes, this seems the best approach for 2.5. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From nnorwitz at gmail.com Thu Jul 6 05:48:56 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 5 Jul 2006 20:48:56 -0700 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: References: Message-ID: On 7/4/06, Thomas Heller wrote: > > I would like to ask about the possibility to add some improvements to ctypes > in Python 2.5, although the feature freeze is now in effect. Hopefully former > third-party libraries can have the freeze relaxed somewhat;-). Ok, former third-party libraries get a 1e-308 reprieve. :-) I think the general answer is to plan some way that you could make an external release to support the other communities like numpy. I don't know the details, but I believe Barry does this with email. This would allow Python to be more stable, but allow additional features for the communities that are interested in installing an additional ctypes release. > I intend to do these changes, the first is a small and trivial one, but allows > a lot of flexibility: > > - Remove the restriction that the argtypes attribute of foreign functions must > be ctypes types. Instead they are only required to implement a .from_param > class method. This patch is ok. I will add comments to the patch on SF. > The second one is more involved, and not yet complete. I can post the patch > or a link to it for review when it is implemented completely: > > - Implement the __array_struct__ attribute as describes by the numpy pep at > http://numpy.scipy.org/array_interface.html. This is too much to add to a beta. Perhaps you could work on a branch to add these features and integrate the branch back in after 2.5 has been released. You could make an external release from the branch. n From pje at telecommunity.com Thu Jul 6 06:32:11 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Jul 2006 00:32:11 -0400 Subject: [Python-Dev] ImportWarning decision In-Reply-To: <200607061324.19000.anthony@interlink.com.au> Message-ID: <5.1.1.6.0.20060706002836.01f1a4b0@sparrow.telecommunity.com> At 01:24 PM 7/6/2006 +1000, Anthony Baxter wrote: >This means Google can just turn it on in sitecustomize.py and Guido >can avoid the hordes of peasants with pitchforks and burning torches. Is that really true? It seems to me that Guido indicated a sitecustomize-solution wasn't possible, in which case suppression is no better than elimination. By the way, while I haven't formally reviewed any of the patches to delay the warning, the one I saw that went by on Python-Dev looked quite straightforward and minimal. Until I saw it, I also believed that trying to implement something like that would be difficult and delicate. But the patch I saw looked very simple and direct. From nnorwitz at gmail.com Thu Jul 6 06:39:08 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 5 Jul 2006 21:39:08 -0700 Subject: [Python-Dev] import screwiness Message-ID: In import.c starting around line 1210 (I removed a bunch of code that doesn't matter for the problem): if (PyUnicode_Check(v)) { copy = PyUnicode_Encode(PyUnicode_AS_UNICODE(v), PyUnicode_GET_SIZE(v), Py_FileSystemDefaultEncoding, NULL); v = copy; } len = PyString_GET_SIZE(v); if (len + 2 + namelen + MAXSUFFIXSIZE >= buflen) { Py_XDECREF(copy); continue; /* Too long */ } strcpy(buf, PyString_AS_STRING(v)); *** So if v is originally unicode, then copy is unicode from the second line, right? Then we assign v to copy, so v is still unicode. Then later on we do PyString_GET_SIZE and PyString_AS_STRING. That doesn't work, does it? What am I missing? n From tim.peters at gmail.com Thu Jul 6 06:53:54 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 6 Jul 2006 00:53:54 -0400 Subject: [Python-Dev] import screwiness In-Reply-To: References: Message-ID: <1f7befae0607052153r70563202lc3f6a464b4e91a44@mail.gmail.com> [Neal Norwitz] > In import.c starting around line 1210 (I removed a bunch of code that > doesn't matter for the problem): > > if (PyUnicode_Check(v)) { > copy = PyUnicode_Encode(PyUnicode_AS_UNICODE(v), > PyUnicode_GET_SIZE(v), > Py_FileSystemDefaultEncoding, NULL); > v = copy; > } > len = PyString_GET_SIZE(v); > if (len + 2 + namelen + MAXSUFFIXSIZE >= buflen) { > Py_XDECREF(copy); > continue; /* Too long */ > } > strcpy(buf, PyString_AS_STRING(v)); > > *** > So if v is originally unicode, then copy is unicode from the second > line, right? No. An encoded unicode string is of type str, and PyUnicode_Encode() returns an encoded string. Like so: >>> u"\u1122".encode('utf-8') '\xe1\x84\xa2' >>> type(_) > Then we assign v to copy, so v is still unicode. Almost ;-) > Then later on we do PyString_GET_SIZE and PyString_AS_STRING. That doesn't > work, does it? What am I missing? The conceptual type of the object returned by PyUnicode_Encode(). From nnorwitz at gmail.com Thu Jul 6 06:59:49 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 5 Jul 2006 21:59:49 -0700 Subject: [Python-Dev] import screwiness In-Reply-To: <1f7befae0607052153r70563202lc3f6a464b4e91a44@mail.gmail.com> References: <1f7befae0607052153r70563202lc3f6a464b4e91a44@mail.gmail.com> Message-ID: On 7/5/06, Tim Peters wrote: > > > Then later on we do PyString_GET_SIZE and PyString_AS_STRING. That doesn't > > work, does it? What am I missing? > > The conceptual type of the object returned by PyUnicode_Encode(). Phew, I sure am glad I was missing that. :-) I saw as the first line in PyUnicode_Encode that it was creating a unicode string. I then went to look at PyString_Encode and saw the same. There I realized that the string created initially wasn't the string returned (it's the object from _AsEncodedString). Which naturally tells me what you just did. Thanks. Now can you fix test_subprocess hanging? :-) Please, I'll even buy you lunch and a coke at the sprints at Google. You are coming, right? n From python-dev at zesty.ca Thu Jul 6 07:02:19 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Thu, 6 Jul 2006 00:02:19 -0500 (CDT) Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> Message-ID: On Wed, 5 Jul 2006, Guido van Rossum wrote: > On 7/5/06, Phillip J. Eby wrote: > > Using the classic nonsense example: > > > > def counter(num): > > def inc(): > > .num += 1 > > return .num > > return inc > > > Would this also use ..num to refer to num in an outer scope two > levels removed? I don't think there's any need for that. I see '.num' as just another way of saying "num, but don't make a new binding". I agree with Guido that the best proposals so far are converging on the idea that it's more Pythonic to say "don't make a new binding" when a variable is used, than to declare "this is the scope for this binding" ahead of time. Of those there are two kinds: (a) State once (anywhere in a scope, but preferably at the beginning) that a variable is non-local. This is like the "global" keyword works now, and this category includes: - Change the meaning of 'global'. - Add a new keyword 'outer' or 'nonlocal', etc. (b) Indicate, when mentioning a variable, that the variable is non-local. This category includes: - Say 'global.x' or 'outer.x' instead of 'x'. - Say '.x' instead of 'x'. My favourite so far is to use a new keyword -- i think changing the meaning of 'global' would be misleading. '.x' is probably my next favourite, though i share Guido's concern about allowing both 'x' and '.x' to refer to the same thing. I see that 'outer' is used as an identifier in hmac.py in the standard library and also as a variable in test_set*.py. On the other hand 'nonlocal' does not appear anywhere in the standard library. Thus, 'nonlocal' is the best option that i've seen so far; it's less likely to break anything and it says exactly what it means. I can't think of a more accurate keyword. -- ?!ng From talin at acm.org Thu Jul 6 07:13:07 2006 From: talin at acm.org (Talin) Date: Wed, 05 Jul 2006 22:13:07 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> Message-ID: <44AC9BE3.4000301@acm.org> Brett Cannon wrote: > On 7/5/06, Michael Chermside wrote: >> If you were using capabilities, you would need to ensure that >> restricted interpreters could only get the file object that they >> were given. But then _all_ of these fancy versions of the >> restrictions would be immediately supported: it would be up to the >> users to create secure wrappers implementing the specific >> restrictions desired. > > I agree. I would prefer this way of doing it. But as I have said, making > sure that 'file' does not get out into the wild is tough. I seem to recall someone mentioned earlier in this discussion the notion of somehow throwing an exception when sandboxed code attempts to push a file reference onto the interpreter stack. I'm not an expert in these matters, so perhaps what I am going to say will make no sense, but here goes: What if there were two copies of the evaluator function. One copy would be a slightly slower 'checked' function, that would test all objects for a 'check' bit. Any attempt to evaluate a reference to an object with a check bit set would throw an exception. The other eval function would be the 'unchecked' version that would run at full speed, just like it does today. Transitioning from the checked to the unchecked state could only be done via C code. So the 'file' wrapper, for example, would switch over to the unchecked interpreter before calling the actual methods of 'file'. That C wrapper might also check the current permission state to see what operations were legal. Note that whenever a C function sets the interpreter state to 'unchecked', that fact is saved on the stack, so that when the function returns, the previous state is restored. The function for setting the interpreter state is something like PyCall_Unchecked( ... ), which restores the interpreter state back to where it was. Transitioning from unchecked to checked is trickier. Specifically, you don't want to ever run sandboxed code in the unchecked state - this is a problem for generators, callbacks, and so on. I can think of two approaches to handling this: First approach is to mark all sandboxed code with a bit indicating the code is untrusted. Any attempt to call or otherwise invoke a function that has this bit set would throw the interpreter into the 'checked' state. (Note that transitioning the other way is *not* automatic - i.e. calling trusted code does not automatically transition to unchecked state.) The approach is good because it means that if you have intermediary code between the wrapper and the sandboxed code, the interpreter still does the right thing - it sets the interpreter into checked state. One problem is how to restore the 'unchecked' state when a function call returns. Probably you would have to build this into the code that does the state transition. If marking the sandboxed code isn't feasible, then you'd have to have the wrapper objects wrap all of the callbacks with code that goes to checked state before calling the callbacks. This means finding all the possible holes - however, I suspect that there are far fewer such holes than trying to hide all possible 'file' methods. However, one advantage of doing this is that restoring the 'unchecked' state after the call returns is fairly straightforward. The advantage of the this whole approach is that once you set the 'check' bit on 'file', it doesn't matter whether 'file' gets loose or not - you can't do anything with it without throwing an exception. Moreover, it also doesn't matter what code path you go through to access it. Only C code that flips the interpreter into unchecked state can call methods on 'file' without blowing up. So essentially, what I propose is to define a simple security primitive - which essentially comes down to checking a single bit - and use that as a basis to create more complex and subtle security mechanisms. -- Talin From guido at python.org Thu Jul 6 07:22:21 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Jul 2006 07:22:21 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> Message-ID: +1 on nonlocal. I think that the := operator is also in case (b), but as I don't like it I'm find with not mentioning it. :-) Could someone write a PEP for this? Doesn't have to be very long but I'd like it to summarize the main options proposed and discuss them, like I did for the switch PEP. It's a p3yk PEP. (We really need to move this to the py3k list...) --Guido On 7/6/06, Ka-Ping Yee wrote: > On Wed, 5 Jul 2006, Guido van Rossum wrote: > > On 7/5/06, Phillip J. Eby wrote: > > > Using the classic nonsense example: > > > > > > def counter(num): > > > def inc(): > > > .num += 1 > > > return .num > > > return inc > > > > > Would this also use ..num to refer to num in an outer scope two > > levels removed? > > I don't think there's any need for that. I see '.num' as just another > way of saying "num, but don't make a new binding". > > I agree with Guido that the best proposals so far are converging on > the idea that it's more Pythonic to say "don't make a new binding" > when a variable is used, than to declare "this is the scope for this > binding" ahead of time. > > Of those there are two kinds: > > (a) State once (anywhere in a scope, but preferably at the > beginning) that a variable is non-local. This is like > the "global" keyword works now, and this category includes: > > - Change the meaning of 'global'. > - Add a new keyword 'outer' or 'nonlocal', etc. > > (b) Indicate, when mentioning a variable, that the variable > is non-local. This category includes: > > - Say 'global.x' or 'outer.x' instead of 'x'. > - Say '.x' instead of 'x'. > > My favourite so far is to use a new keyword -- i think changing the > meaning of 'global' would be misleading. '.x' is probably my next > favourite, though i share Guido's concern about allowing both 'x' > and '.x' to refer to the same thing. > > I see that 'outer' is used as an identifier in hmac.py in the > standard library and also as a variable in test_set*.py. On the > other hand 'nonlocal' does not appear anywhere in the standard > library. Thus, 'nonlocal' is the best option that i've seen so far; > it's less likely to break anything and it says exactly what it means. > I can't think of a more accurate keyword. > > > -- ?!ng > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Thu Jul 6 07:24:03 2006 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Thu, 06 Jul 2006 07:24:03 +0200 Subject: [Python-Dev] what can we do to hide the 'file' type? In-Reply-To: References: Message-ID: <44AC9E73.7030304@v.loewis.de> Brett Cannon wrote: > Can anyone think of any other way to gain access to 'file' without > importing a module? In principle, it might be possible to find file in the func_defaults or func_globals of some function, which might be defined as orig_file = file def file(...): ... I couldn't find any such function in the standard library, though. Regards, Martin From guido at python.org Thu Jul 6 07:28:27 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Jul 2006 07:28:27 +0200 Subject: [Python-Dev] ImportWarning decision In-Reply-To: <5.1.1.6.0.20060706002836.01f1a4b0@sparrow.telecommunity.com> References: <200607061324.19000.anthony@interlink.com.au> <5.1.1.6.0.20060706002836.01f1a4b0@sparrow.telecommunity.com> Message-ID: On 7/6/06, Phillip J. Eby wrote: > At 01:24 PM 7/6/2006 +1000, Anthony Baxter wrote: > >This means Google can just turn it on in sitecustomize.py and Guido > >can avoid the hordes of peasants with pitchforks and burning torches. > > Is that really true? It seems to me that Guido indicated a > sitecustomize-solution wasn't possible, in which case suppression is no > better than elimination. Don't worry about it. It's irrelevant anyway since we're not upgrading to 2.5 any time soon. BTW The pitchforks pitch was widely misunderstood -- it was a dramatization to emphasize what I still consider a general problem; if I believed the problem was limited to Google I would have proposed an internal solution. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python-dev at zesty.ca Thu Jul 6 07:39:53 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Thu, 6 Jul 2006 00:39:53 -0500 (CDT) Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: References: Message-ID: On Wed, 5 Jul 2006, Brett Cannon wrote: > On 7/4/06, Ka-Ping Yee wrote: > > In response to Guido's comment about confusing the words "trusted" and > > "untrusted", how about "empowered" and "restricted"? > > Maybe. I am really starting to lean towards trusted and sandboxed. It can be risky to use words of the form '*trust*' when talking about security because 'trust' can have different meanings in different contexts. (Examples: 'X is trusted' might mean 'X is something i feel safe running without restrictions' or 'X *is* in fact allowed to run without restrictions' or 'i need to run X without restrictions in order to accomplish my task' or 'X is something i am relying on for the security of my system'.) The most common context for 'trusted' that i seem to come across is in the phrase 'trusted computing base' (TCB), which refers to 'the thing that is supposed to be enforcing security restrictions' as opposed to 'something i'm willing to let run unrestricted'. So when i read 'trusted code' what comes to mind is the C implementation of the Python interpreter, and it may be a good idea to reserve that phrase for that purpose, if it's to be used at all. In any case, let's try to nail down clear names for the different pieces we're talking about here, and i gently advise avoiding '*trust*' words or using them with very limited meaning. -- ?!ng From tim.peters at gmail.com Thu Jul 6 07:55:01 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 6 Jul 2006 01:55:01 -0400 Subject: [Python-Dev] import screwiness In-Reply-To: References: <1f7befae0607052153r70563202lc3f6a464b4e91a44@mail.gmail.com> Message-ID: <1f7befae0607052255p16254c7fr95c2de8e39a4d8f6@mail.gmail.com> [Neal] >>> Then later on we do PyString_GET_SIZE and PyString_AS_STRING. That doesn't >>> work, does it? What am I missing? [Tim] >> The conceptual type of the object returned by PyUnicode_Encode(). [Neal] > Phew, I sure am glad I was missing that. :-) > > I saw as the first line in PyUnicode_Encode that it was creating a > unicode string. I then went to look at PyString_Encode and saw the > same. There I realized that the string created initially wasn't the > string returned (it's the object from _AsEncodedString). Which > naturally tells me what you just did. Thanks. Good! You're welcome. > Now can you fix test_subprocess hanging? :-) Sure: break it on Windows and I'll fix it tonight :-) > Please, I'll even buy you lunch and a coke at the sprints at Google. You > are coming, right? Sorry, doesn't look like it. I'm living on investment income this year, and the markets haven't been kind so far -- I don't want to go more in the hole than is truly necessary. I even switched from Campbell's soup to the store generic brand last week ;-) From python-dev at zesty.ca Thu Jul 6 08:11:14 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Thu, 6 Jul 2006 01:11:14 -0500 (CDT) Subject: [Python-Dev] Restricted execution: what's the threat model? In-Reply-To: References: Message-ID: After reading the messages on this thread i'm starting to think that it would be good to clarify what kinds of threats we are trying to defend against, and specify what invariants we are intending to preserve. For example, here are a few things Brett mentioned: > Right. I am thinking more of an implementation screw up that somehow > provides access to an object that has escalated rights. > But you are correct, I am only concerned with preventing a crash of a > sandboxed interperter. [on what is meant by "it getting out"] > Out of a trusted interpreter and ending up in a sandboxed interpreter > some how. So here are a couple of questions for clarification (some with my guesses as to their answers): 1. When we say "restricted/untrusted/ interpreter" we don't really mean that the *interpreter* is untrusted, right? We mean that the Python code that runs in that interpreter is untrusted (i.e. to be prevented from doing harm), right? 2. I'm assuming that the implementation of the Python interpreter is always trusted. As a starting point it seems to me we have to draw the line somewhere -- around at least the C code that implements the Python interpreter, and possibly more. What do we take the Trusted Computing Base to include? The Python VM implementation -- plus all the builtin objects and C modules? Plus the whole standard library? ("trusted" = "behaves safely because it's our job to write it correctly, not due to something else imposing restrictions upon it"; "untrusted" = "we wish to be able to impose restrictions on it") 3. Is it part of the plan that we want to protect Python code from other Python code? For example, should a Python program/function X be able to say "i want to launch/call program/function Y with *these* parameters and have it run under *these* limitations?" This has a big impact on the model. And here are some possible goals or invariants to consider. It will be helpful to decide on some of these so that, when someone points to what they think is a flaw in the security implementation, we can say "yes, that is our responsibility" or "no, it isn't". We want to be able to guarantee that... A. The interpreter will not crash no matter what Python code it is given to execute. B. Python programs running in different interpreters embedded in the same process cannot communicate with each other. C. Python programs running in different interpreters embedded in the same process cannot access each other's Python objects. D. A given piece of Python code cannot access or communicate with certain Python objects in the same interpreter. E. A given piece of Python code can access only a limited set of Python objects in the same interpreter. I think in order to get truly useful restricted interpreters we will end up wanting to make guarantees of all of these kinds. There may be others i haven't thought of -- feel free to edit or add others. -- ?!ng From pedronis at strakt.com Thu Jul 6 09:24:45 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Thu, 06 Jul 2006 09:24:45 +0200 Subject: [Python-Dev] Import semantics In-Reply-To: <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com> References: <448D1F0D.7000405@strakt.com> <4dab5f760606251334o718323adt409294e2952e954a@mail.gmail.com> <4dab5f760607050921h548cf2fdx9f125599ccbe8bfc@mail.gmail.com> Message-ID: <44ACBABD.6020300@strakt.com> Frank Wierzbicki wrote: > On 7/5/06, Guido van Rossum wrote: > >>Hi Frank, >> >>Have you and/or the Jython community made up your mind about this? The >>thread seems to have disappeared after you posted (or perhaps it >>continued only on jython-dev, which I don't read?). > > The thread pretty much stopped there. I think a modification of the > import semantics won't really come up again until 2.3 is out, since > releases and reviving Jython take priority over backwards incompatible > features. For my part I would like to think that a goal for Jython > should be: > > Pure Python code developed first on Jython should run without change on CPython. > > Which would mean we will eventually need to change the import > semantics for importing Python code (though I doubt we will change the > semantics for importing Java packages any time soon). Whether that > can be done in 2.x, or if this change is so incompatible that we need > to think about it in a "Jython 3000" way, I don't really know. Changing the behavior of getattr for python modules and leave it as it is for java packages seems a reasonable compromise. > > >>Also, I just realized that you're the new Jython maintainer. Is *that* >>official? > > It is official, at least in the unofficial way that Jython appears to > work: Brian handed the baton to me after (I presume) Samuele Pedroni > had handed the baton to Brian. > > Brian's email is here: > http://sourceforge.net/mailarchive/message.php?msg_id=13859029 > > That said, I still regard Samuele Pedroni as the ultimate authority on > Jython and give him pretty much full veto power. He fortunately > continues to watch the checkins and prods me when I go in the wrong > direction. > as things stand this a reasonable way to go about things, I'm the only person with a bit historical prospective, a bit of time to help (less than I would like still), knowing both python semantics well and Java and knowing the Jython code base, also I designed how the new-style classes implemantion work in Jython atm, so I should know how things are supposed to hang together in that respect, also for things that involve the Java glueing aspects of Jython one needs to grow some language design experience and sense, and I'm around since enough time for Jython and on python-dev to have some of that :) regards. From theller at python.net Thu Jul 6 11:15:24 2006 From: theller at python.net (Thomas Heller) Date: Thu, 06 Jul 2006 11:15:24 +0200 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: References: Message-ID: <44ACD4AC.4070704@python.net> Neal Norwitz schrieb: > On 7/4/06, Thomas Heller wrote: >> >> I would like to ask about the possibility to add some improvements to ctypes >> in Python 2.5, although the feature freeze is now in effect. Hopefully former >> third-party libraries can have the freeze relaxed somewhat;-). > > Ok, former third-party libraries get a 1e-308 reprieve. :-) > > I think the general answer is to plan some way that you could make an > external release to support the other communities like numpy. I don't > know the details, but I believe Barry does this with email. This > would allow Python to be more stable, but allow additional features > for the communities that are interested in installing an additional > ctypes release. > >> I intend to do these changes, the first is a small and trivial one, but allows >> a lot of flexibility: >> >> - Remove the restriction that the argtypes attribute of foreign functions must >> be ctypes types. Instead they are only required to implement a .from_param >> class method. > > This patch is ok. I will add comments to the patch on SF. Patch 1517790 is now committed, thanks. >> The second one is more involved, and not yet complete. I can post the patch >> or a link to it for review when it is implemented completely: >> >> - Implement the __array_struct__ attribute as describes by the numpy pep at >> http://numpy.scipy.org/array_interface.html. > > This is too much to add to a beta. Perhaps you could work on a branch > to add these features and integrate the branch back in after 2.5 has > been released. You could make an external release from the branch. We will see how this works out. Not what I expected to hear, but I'm sure it is a wise decision. Thanks, Thomas From ronaldoussoren at mac.com Thu Jul 6 12:14:34 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 6 Jul 2006 12:14:34 +0200 Subject: [Python-Dev] zlib module build failure on Mac OSX 10.4.7 In-Reply-To: <738A121B-3529-411D-89A7-EBD8C6B42A29@mac.com> References: <17573.60593.759293.549583@montanaro.dyndns.org> <130298B1-7336-480A-A90F-97B68327BE4A@mac.com> <17574.43390.560231.425494@montanaro.dyndns.org> <315DECAC-D582-4692-A0CA-13D3C7BF992C@mac.com> <44A6C314.2000709@v.loewis.de> <0006F35E-FD5F-4BAD-A608-642039BBE7E8@mac.com> <738A121B-3529-411D-89A7-EBD8C6B42A29@mac.com> Message-ID: <31F39BDC-A396-4EFD-B6C6-6E93458F3F7D@mac.com> On Jul 5, 2006, at 7:35 AM, Ronald Oussoren wrote: > > On Jul 4, 2006, at 11:21 PM, Neal Norwitz wrote: > >> Ronald, Bob, >> >> I know Skip found and fixed his problem, however, is this problem >> likely to affect other users? Is there anything we can do to help >> alleviate/diagnose this problem? > > I'll either enhance configure or roll back my change to setup.py. > I'd prefer to do the former, but if beta2 gets too close I'll just > change setup.py. I have checked in a patch that fixes this issue (that is, configure will correctly set HAVE_ZLIB_COPY even if you have an old static library of zlib in /usr/local/lib) in revision 47267. I don't really like that patch, it doesn't smell right. I wonder if we should just add '-Wl,-search_path_first' to the default LDFLAGS on OSX, that way the linker would behave more like the one on other unix- y platforms and special-casing for several libraries could be dropped. To recap the problem I'm trying to solve: python uses several 3th- party libraries like libz and libbz2 to build extensions. Some of these libraries are shipped with OSX as shared libraries. I want to make it possible to link with local copies of those libraries, mostly because that makes it possible to run the resulting binary on older versions of the OS but also because some of the libraries on OSX are pretty old. I also want to link using static libraries instead of dynamic libraries. To do that you must use -Wl,-search_paths_first because otherwise the OSX linker will look for a dylib anywhere on the path before looking for a static library. Ronald > > Ronald > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/ > ronaldoussoren%40mac.com -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2157 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20060706/d2dfc651/attachment.bin From arigo at tunes.org Thu Jul 6 13:03:12 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu, 6 Jul 2006 13:03:12 +0200 Subject: [Python-Dev] what can we do to hide the 'file' type? In-Reply-To: References: Message-ID: <20060706110312.GB15690@code0.codespeak.net> Hi Brett, On Wed, Jul 05, 2006 at 05:01:48PM -0700, Brett Cannon wrote: > And if Armin and/or Samuele sign off that what we find is most likely (with > "most likely" equalling 99% chance) all there is, then bonus points and I > will *really* be convinced. =) I don't think I can "sign off" that. Really hiding Python objects is quite hard IMHO. Armin From ncoghlan at gmail.com Thu Jul 6 13:43:56 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 06 Jul 2006 21:43:56 +1000 Subject: [Python-Dev] Proposed beta 2 changes (Q for Anthony/Neal) In-Reply-To: <200607061335.21179.anthony@interlink.com.au> References: <44AA5FE6.2030801@iinet.net.au> <200607061335.21179.anthony@interlink.com.au> Message-ID: <44ACF77C.5030404@gmail.com> Anthony Baxter wrote: > On Tuesday 04 July 2006 22:32, Nick Coghlan wrote: >> 1. Finishing the __module_name__ workaround to allow relative >> imports from the main module when using -m. > > I have some nervousness about this. Are there cases in the stdlib > where this is an issue, today? Are there any cases which can't be > addressed by using absolute imports? For 2.5, wouldn't it be better > to simply say "use absolute imports in this case"? > My nervousness is that I don't want a late change to introduce a > language misfeature that we'll all regret later. I actually come to agree with this POV - particularly after spending some time thinking about how I'd like to it to look in Py3k (with a simple boolean check replacing the __name__ == "__main__" idiom). I'll revert the change that added __module_name__, too. I'd love to fix this properly for 2.5 but the timing of the discovery of the limitation is unfortunately lousy :( I'll add something to PEP 338, and possibly elsewhere, noting that the -m switch adds the current directory to sys.path, so absolute imports will work fine so long as you invoke Python from the directory containing the relevant top level package directory, even if it isn't anywhere else on sys.path. Some time between now and late 2007 I'll write something up about the interaction between __name__, sys.path[0] and implicit relative imports that actually makes implicit relative imports appear to work when a module inside a package is executed directly instead of via -m (probably when I get around to writing a Python 2.6 PEP on the topic). Heh. I was going to be clever with the PEP number relating to this for 2.6, and I think I've discovered that trying to mix PEP 328 with 338 is going to be a doomed effort - add the PEP numbers together to find out why ;) >> 2. Adding an 'ignore' filter for ImportWarning at the end of >> warnings.py > > Already covered this one in another email... Yes, this seems the best > approach for 2.5. OK, I'll make the change to warnings.py, and update the warnings module documentation. AMK will still need to update What's New. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Thu Jul 6 13:50:32 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 06 Jul 2006 21:50:32 +1000 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5.1.1.6.0.20060705133753.026a7698@sparrow.telecommunity.com> References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <5.1.1.6.0.20060705133753.026a7698@sparrow.telecommunity.com> Message-ID: <44ACF908.1020403@gmail.com> Phillip J. Eby wrote: > At 07:27 PM 7/5/2006 +0200, Guido van Rossum wrote: >> However I still don't believe "global" has the stretchiness in its >> meaning that you claim it has. Have you ever heard a Python programmer >> talking about closures use the word "global variable"? >> >> Are there any other native speakers who side with Michael? > > I don't, and am -1 on using "global" to mean "outer". Another -1 on stretching the meaning of global here. I quite like 'nonlocal', though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From tom at vector-seven.com Thu Jul 6 10:51:32 2006 From: tom at vector-seven.com (Thomas Lee) Date: Thu, 06 Jul 2006 18:51:32 +1000 Subject: [Python-Dev] zipfile.ZipFile('foo.zip', 'a'): file not found -> create? Message-ID: <44ACCF14.1020901@vector-seven.com> Hi all, In reference to: http://sourceforge.net/tracker/index.php?func=detail&aid=1514451&group_id=5470&atid=105470 I wrote a patch for this "bug", but a valid point was raised by Ronald Oussorren: this borders on being more of a "feature" than a bug fix, although - IMHO - this fix improves consistency with the rest of the Python standard library. Can I get some opinions on this? My patch for this issue currently lives here: http://sourceforge.net/tracker/index.php?func=detail&aid=1517891&group_id=5470&atid=305470 Cheers, Tom From mcherm at mcherm.com Thu Jul 6 14:15:43 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 06 Jul 2006 05:15:43 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python Message-ID: <20060706051543.wxi1nukehvfkk4c4@login.werra.lunarpages.com> I wrote: > I would still rather not spread FUD. Brett Cannon responds: > I don't consider it FUD. Armin in an email said that he thought it > was a losing battle to try to hide 'file' from an interpreter. That > is what I am worried about, period. Everythign else can be > protected through resource hiding. I never intended to say that "we can't hide 'file' from the user" was FUD. Only objects crossing between interpreters. And it *might* not be FUD, but so far I haven't heard anyone say that they thought objects *could* cross between interpreters. I think your proposal would read better with a categorical statement that objects cannot cross between interpreters (right along with your categorical statement that Python code cannot directly access dangerous resources without help from the C level), so I present the following challenge: can anyone think of a way that an object could "cross" interpreters? For instance, are certain basic objects shared (str and int for instance) and objects could be passed between interpreters by attaching them as attributes of these core objects? I just don't know enough about multiple interpreters to be sure -- but somehow I thought they had separate object pools. -- Michael Chermside From mcherm at mcherm.com Thu Jul 6 14:37:56 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 06 Jul 2006 05:37:56 -0700 Subject: [Python-Dev] Restricted execution: what's the threat model? Message-ID: <20060706053756.ovcm9a05kko0o008@login.werra.lunarpages.com> Ka-Ping Yee writes: > i'm starting to think > that it would be good to clarify what kinds of threats we are > trying to defend against, and specify what invariants we are > intending to preserve. Yes! > So here are a couple of questions for clarification (some with my > guesses as to their answers): Okay, I'll throw in my thoughts also. > 1. When we say "restricted/untrusted/ interpreter" we > don't really mean that the *interpreter* is untrusted, right? > We mean that the Python code that runs in that interpreter is > untrusted (i.e. to be prevented from doing harm), right? Agreed. My interpretation of the proposal was that interpreters were either "sandboxed" or "trusted". "Sandboxed" means that there are security restrictions imposed at some level (perhaps even NO restrictions). "Trusted" means that the interpreter implements no security restrictions (beyond what CPython already implements, which isn't much) and thus runs faster. > 2. I'm assuming that the implementation of the Python interpreter > is always trusted Sure... it's got to be. > What do > we take the Trusted Computing Base to include? The Python VM > implementation -- plus all the builtin objects and C modules? > Plus the whole standard library? My interpretation of Brett's proposal is that the CPython developers would try to ensure that Python VM had no "security holes" when running in sandboxed mode. Of course, we also "try" to ensure no crashes are possible also, and while we're quite good, we're not perfect. Beyond that, all pure-python modules with source available (whether in the stdlib or not) can be "trusted" because they run in a sandboxed VM. All C modules are *up to the user*. Brett proposes to provide a default list of useful-but-believed-to-be-safe modules in the stdlib, but the user can configure the C-module whitelist to whatever she desires. > 3. Is it part of the plan that we want to protect Python code from > other Python code? For example, should a Python program/function > X be able to say "i want to launch/call program/function Y with > *these* parameters and have it run under *these* limitations?" > This has a big impact on the model. Now *that* is a good question. I would say the answer is a partial "no", because there are pieces of Brett's security model that are tied to the interpreter instance. Python code cannot launch another interpreter (but perhaps it *should* be able to?), so it cannot modify those restrictions for new Python code it launches. However, I would rather like to allow Python code to execute other code with greater restrictions, although I would accept all kinds of limitations and performance penalties to do so. I would be satisfied if the caller could restrict certain things (like web and file access) but not others (like memory limits or use of stdout). I would satisfied if the caller paid huge overhead costs of launching a separate interpreter -- heck, even a separate process. And if it is willing to launch a separate process, then Brett's model works just fine: allow the calling code to start a new (restricted) Python VM. > We want to be able to guarantee that... > > A. The interpreter will not crash no matter what Python code > it is given to execute. Agreed. We already want to guarantee that, with the caveat that the guarantee doesn't apply to a few special modules (like ctypes). > B. Python programs running in different interpreters embedded > in the same process cannot communicate with each other. I don't want to guarantee this, does someone else? It's astonishingly hard... there are all kinds of clever "knock on the walls" tricks. For instance, communicate by varying your CPU utilization up and down in regular patterns. I'd be satisfied if they could pass information (perhaps even someday provide a library making it *easy* to do so), but could not pass unforgable items like Python object references, open file descriptors, and so forth. > C. Python programs running in different interpreters embedded > in the same process cannot access each other's Python objects. I strengthen that slightly to all "unforgable" items, not just object references. > D. A given piece of Python code cannot access or communicate > with certain Python objects in the same interpreter. > > E. A given piece of Python code can access only a limited set > of Python objects in the same interpreter. Hmmm. I'm not sure. -- Michael Chermside From mwh at python.net Thu Jul 6 14:39:32 2006 From: mwh at python.net (Michael Hudson) Date: Thu, 06 Jul 2006 13:39:32 +0100 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <20060705103123.fvv54v7e3s0ko8wo@login.werra.lunarpages.com> (Michael Chermside's message of "Wed, 05 Jul 2006 10:31:23 -0700") References: <20060705103123.fvv54v7e3s0ko8wo@login.werra.lunarpages.com> Message-ID: <2m64ia6h6j.fsf@starship.python.net> Michael Chermside writes: > Phillip Eby writes: >> I don't see a problem with requiring '.x' to be used for both >> reading and writing of outer-scope names; it just shouldn't be >> required for an outer-scope name that you don't rebind in the >> current scope. >> >> def counter(num): >> def inc(): >> .num += 1 >> return .num >> return inc > > I am reminded of Tim Peter's declaration in response to a similar > proposal some time ago: > > Syntax should not look like grit on my monitor. > > (Sorry, no reference... but I swear it's word-for-word accurate because > the quote burned itself into my memory.) I think it was Anthony: http://mail.python.org/pipermail/python-dev/2005-July/054581.html Cheers, mwh -- SCSI is not magic. There are fundamental technical reasons why it is necessary to sacrifice a young goat to your SCSI chain now and then. -- John Woods From tim.hochberg at ieee.org Thu Jul 6 15:38:54 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 06 Jul 2006 06:38:54 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> Message-ID: Ka-Ping Yee wrote: > On Wed, 5 Jul 2006, Guido van Rossum wrote: >> On 7/5/06, Phillip J. Eby wrote: >>> Using the classic nonsense example: >>> >>> def counter(num): >>> def inc(): >>> .num += 1 >>> return .num >>> return inc >>> >> Would this also use ..num to refer to num in an outer scope two >> levels removed? > > I don't think there's any need for that. I see '.num' as just another > way of saying "num, but don't make a new binding". > > I agree with Guido that the best proposals so far are converging on > the idea that it's more Pythonic to say "don't make a new binding" > when a variable is used, than to declare "this is the scope for this > binding" ahead of time. > > Of those there are two kinds: > > (a) State once (anywhere in a scope, but preferably at the > beginning) that a variable is non-local. This is like > the "global" keyword works now, and this category includes: > > - Change the meaning of 'global'. > - Add a new keyword 'outer' or 'nonlocal', etc. > > (b) Indicate, when mentioning a variable, that the variable > is non-local. This category includes: > > - Say 'global.x' or 'outer.x' instead of 'x'. > - Say '.x' instead of 'x'. > > My favourite so far is to use a new keyword -- i think changing the > meaning of 'global' would be misleading. '.x' is probably my next > favourite, though i share Guido's concern about allowing both 'x' > and '.x' to refer to the same thing. > > I see that 'outer' is used as an identifier in hmac.py in the > standard library and also as a variable in test_set*.py. On the > other hand 'nonlocal' does not appear anywhere in the standard > library. Thus, 'nonlocal' is the best option that i've seen so far; > it's less likely to break anything and it says exactly what it means. > I can't think of a more accurate keyword. 'extant' (= already existing) is probably pretty rare in existing code, and has pretty close to exactly the correct meaning, but may be too obscure. -tim > > > -- ?!ng > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > From jan-python at maka.demon.nl Thu Jul 6 16:55:39 2006 From: jan-python at maka.demon.nl (jan-python at maka.demon.nl) Date: Thu, 06 Jul 2006 16:55:39 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> So.. are we only thinking about implementing this outer scope assignment because there's lots of talk about it on the list, or are there actually use cases that would become clearer if assigning to an outer scope variable was allowed? I tend to think that almost _any_ piece of code that could be written using outer scope assignment would become less clear by doing so as opposed to some other way. So, I'm not keen on the whole idea. If we must, however, I think a declaration like 'nonlocal' would be best (well, least-bad, that is). From skip at pobox.com Thu Jul 6 17:05:42 2006 From: skip at pobox.com (skip at pobox.com) Date: Thu, 6 Jul 2006 10:05:42 -0500 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> References: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> Message-ID: <17581.9926.915438.200819@montanaro.dyndns.org> jan-python> So.. are we only thinking about implementing this outer jan-python> scope assignment because there's lots of talk about it on jan-python> the list, ... :-) jan-python> ... or are there actually use cases that would become jan-python> clearer if assigning to an outer scope variable was allowed? I think full lexical scoping will only be of use to people who use nested scopes heavily. The more typical user will be happy to just refer to values in outser scopes without modifying them and rely on classes to save changed state across calls. I think it's almost a YAGNI, but I'm sure others will disagree. Skip Repeat after me: Python is not Lisp... From mcherm at mcherm.com Thu Jul 6 17:31:27 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 06 Jul 2006 08:31:27 -0700 Subject: [Python-Dev] what can we do to hide the 'file' type? Message-ID: <20060706083127.csxctx0sahr4kwk8@login.werra.lunarpages.com> Armin Rigo writes: > I don't think I can "sign off" [on hiding the file type]. Really hiding > Python objects is quite hard IMHO. I agree. But we don't have to give up yet. How about instead of hiding file, we cripple it. Completely. Modify the file type so that when executing on a sandboxed interpreter, all of the dangerous methods and attributes of file throw exceptions. Then we create a separate thing (in C) called a "SecureFileWrapper". It has methods that are passed a reference to a file object and can invoke the methods without error. We provide a means for obtaining a SecureFileWrapper bound to a given file (perhaps open()). Essentially, we give up on hiding file, which is a frequently-used type, and very hard to hide, and instead we rely on our ability to write a reliably secure "SecureFileWrapper" class (in C). -- Michael Chermside From ark at acm.org Thu Jul 6 17:40:23 2006 From: ark at acm.org (Andrew Koenig) Date: Thu, 6 Jul 2006 11:40:23 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: Message-ID: <001301c6a112$81daa0f0$6402a8c0@arkdesktop> > However I still don't believe "global" has the stretchiness in its > meaning that you claim it has. Have you ever heard a Python programmer > talking about closures use the word "global variable"? I guess the term I've heard most often is "free variable," but I wouldn't be surprised if I saw the term "global" used to describe a free variable. However, I should point out that Dijkstra used "global" in a similar way in his 1971 book "A Discipline of Programming." The program examples in that book are in a language he devised for the purpose; one of the language's features is that every variable used in every block must be explicitly declared, even if it is taken from a surrounding block. If I remember correctly, the terms he used for variables taken from a surrounding block and variables defined and used in the same block were "global" and "private," respectively. From almann.goo at gmail.com Thu Jul 6 17:49:16 2006 From: almann.goo at gmail.com (Almann T. Goo) Date: Thu, 6 Jul 2006 11:49:16 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> <44AAE020.5060204@acm.org> <5.1.1.6.0.20060704182302.0207e890@sparrow.telecommunity.com> <5.1.1.6.0.20060705020306.0207f890@sparrow.telecommunity.com> Message-ID: <7e9b97090607060849j42d62d8aqd6e6a15475df590c@mail.gmail.com> On 7/6/06, Guido van Rossum wrote: > > +1 on nonlocal. > > I think that the := operator is also in case (b), but as I don't like > it I'm find with not mentioning it. :-) > > Could someone write a PEP for this? Doesn't have to be very long but > I'd like it to summarize the main options proposed and discuss them, > like I did for the switch PEP. It's a p3yk PEP. (We really need to > move this to the py3k list...) > Drat, too bad this wasn't back in February when I was all for writing the PEP--sadly, I don't have time to do this, maybe later if no one steps up to the plate... For reference, here is a link to the other, rather large, thread on this back then: http://thread.gmane.org/gmane.comp.python.devel/76532/focus=76532 My option one, is essentially "nonlocal", though I spelled it as "use". Really, I am personally agreeable to almost any spelling of such a keyword. I am +1 on "nonlocal", though I know the biggest dissent against it is adding another "global"-like keyword that is not "pythonic". Also in regard to the "prefix-dot" notation (i.e. ".x"), I am -1 for the reason that it introduces a subtle alternate way of spelling local variables. The rules for the usage would have to be strict enough to prevent subtle code obscurity (like what is the semantics for using ".x" and "x" in a scope--are both spellings allowed, and if so, what is the meaning if assignment is involved or is not involved). Of course, if the semantics were well defined such that it would be difficult for users to trap themselves with different spellings of local variables, then I would be +0 for it. Best Regards, Almann -- Almann T. Goo almann.goo at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/db27bf8e/attachment.htm From pje at telecommunity.com Thu Jul 6 18:28:12 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Jul 2006 12:28:12 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <17581.9926.915438.200819@montanaro.dyndns.org> References: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> Message-ID: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com> At 10:05 AM 7/6/2006 -0500, skip at pobox.com wrote: > jan-python> So.. are we only thinking about implementing this outer > jan-python> scope assignment because there's lots of talk about it on > jan-python> the list, ... > >:-) > > jan-python> ... or are there actually use cases that would become > jan-python> clearer if assigning to an outer scope variable was allowed? > >I think full lexical scoping will only be of use to people who use nested >scopes heavily. The more typical user will be happy to just refer to values >in outser scopes without modifying them and rely on classes to save changed >state across calls. I think it's almost a YAGNI, but I'm sure others will >disagree. Here's the reason I think this keeps coming up, and why Guido's "just use a class" argument doesn't really address the actual problem that's taking place. When you are writing some function, and you find yourself using a nested function because it's the closest match for something you're doing, there occasionally comes a point at which you realize that, gosh, you need to mutate something in an outer scope. At that point, your choices are to either kludge it with a mutable, or to reorganize the whole thing. Both of these *feel* like warts because they're making you do less-than-optimal things. Your mental "flow" is disrupted because the language is forcing you to work around it, rather than having a way of working with it. This is a user experience issue, not a technical one. The fact that you can say, "you should've done it differently in the first place" doesn't do anything for your flow. In theory, you could design cars without any brakes, because people could just coast to a stop if they planned well enough in advance. ;-) In practice, you need the brakes because people often don't discover their need to stop until much later in the process. This is a flow issue that's specific to *incremental* development. What happens is that first you refactor the code in a function to include nested functions. The variable references don't change, you're just indenting some code. *Then*, it later comes up that you need to rebind a variable, and now you have to globally change the variable to make it work as a mutable instead, or else you have to refactor all the variables to be 'self.' references and put a class in somewhere. This destroys the flow of incrementally developing whatever it is you're developing, and makes you stop to do excise work. That's why being able to rebind a variable without redefining it is important. In short: in *theory*, a rebinding operator or "nonlocal" declaration is unnecessary. In *practice*, having one seems quite useful every time you wander down the path that leads to having to rewrite your code just because the language won't let you do that one tiny thing -- or so it feels like to the person who's experiencing it. From brett at python.org Thu Jul 6 18:59:37 2006 From: brett at python.org (Brett Cannon) Date: Thu, 6 Jul 2006 09:59:37 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <44AC9BE3.4000301@acm.org> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> Message-ID: On 7/5/06, Talin wrote: > > Brett Cannon wrote: > > On 7/5/06, Michael Chermside wrote: > >> If you were using capabilities, you would need to ensure that > >> restricted interpreters could only get the file object that they > >> were given. But then _all_ of these fancy versions of the > >> restrictions would be immediately supported: it would be up to the > >> users to create secure wrappers implementing the specific > >> restrictions desired. > > > > I agree. I would prefer this way of doing it. But as I have said, > making > > sure that 'file' does not get out into the wild is tough. > > I seem to recall someone mentioned earlier in this discussion the notion > of somehow throwing an exception when sandboxed code attempts to push a > file reference onto the interpreter stack. That was AMK. I'm not an expert in these matters, so perhaps what I am going to say > will make no sense, but here goes: > > What if there were two copies of the evaluator function. One copy would > be a slightly slower 'checked' function, that would test all objects for > a 'check' bit. Any attempt to evaluate a reference to an object with a > check bit set would throw an exception. The other eval function would be the 'unchecked' version that would run > at full speed, just like it does today. > > Transitioning from the checked to the unchecked state could only be done > via C code. So the 'file' wrapper, for example, would switch over to the > unchecked interpreter before calling the actual methods of 'file'. That > C wrapper might also check the current permission state to see what > operations were legal. So add the proper checks in Python/ceval.c:call_function() to check for this flag on every object passed in that is called? Note that whenever a C function sets the interpreter state to > 'unchecked', that fact is saved on the stack, so that when the function > returns, the previous state is restored. The function for setting the > interpreter state is something like PyCall_Unchecked( ... ), which > restores the interpreter state back to where it was. > > Transitioning from unchecked to checked is trickier. Specifically, you > don't want to ever run sandboxed code in the unchecked state - this is a > problem for generators, callbacks, and so on. I can think of two > approaches to handling this: > > First approach is to mark all sandboxed code with a bit indicating the > code is untrusted. Any attempt to call or otherwise invoke a function > that has this bit set would throw the interpreter into the 'checked' > state. (Note that transitioning the other way is *not* automatic - i.e. > calling trusted code does not automatically transition to unchecked > state.) > > The approach is good because it means that if you have intermediary code > between the wrapper and the sandboxed code, the interpreter still does > the right thing - it sets the interpreter into checked state. > > One problem is how to restore the 'unchecked' state when a function call > returns. Probably you would have to build this into the code that does > the state transition. > > If marking the sandboxed code isn't feasible, then you'd have to have > the wrapper objects wrap all of the callbacks with code that goes to > checked state before calling the callbacks. This means finding all the > possible holes - however, I suspect that there are far fewer such holes > than trying to hide all possible 'file' methods. However, one advantage > of doing this is that restoring the 'unchecked' state after the call > returns is fairly straightforward. > > The advantage of the this whole approach is that once you set the > 'check' bit on 'file', it doesn't matter whether 'file' gets loose or > not - you can't do anything with it without throwing an exception. > Moreover, it also doesn't matter what code path you go through to access > it. Only C code that flips the interpreter into unchecked state can call > methods on 'file' without blowing up. > > So essentially, what I propose is to define a simple security primitive > - which essentially comes down to checking a single bit - and use that > as a basis to create more complex and subtle security mechanisms. Right, but it does require that the proper verification function be turned on so that the permission bit on 'file' is checked. It kind of seems like 'rexec' and its f_restricted flag it set on execution frames, except you are adding an object-level flag as well. Either way, the trick is not fouling up switching between the two checking functions. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/72e33923/attachment-0001.html From brett at python.org Thu Jul 6 19:15:32 2006 From: brett at python.org (Brett Cannon) Date: Thu, 6 Jul 2006 10:15:32 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: <20060706051543.wxi1nukehvfkk4c4@login.werra.lunarpages.com> References: <20060706051543.wxi1nukehvfkk4c4@login.werra.lunarpages.com> Message-ID: On 7/6/06, Michael Chermside wrote: > > I wrote: > > I would still rather not spread FUD. > > Brett Cannon responds: > > I don't consider it FUD. Armin in an email said that he thought it > > was a losing battle to try to hide 'file' from an interpreter. That > > is what I am worried about, period. Everythign else can be > > protected through resource hiding. > > I never intended to say that "we can't hide 'file' from the user" > was FUD. Only objects crossing between interpreters. And it *might* > not be FUD, but so far I haven't heard anyone say that they thought > objects *could* cross between interpreters. I think your proposal > would read better with a categorical statement that objects cannot > cross between interpreters (right along with your categorical > statement that Python code cannot directly access dangerous resources > without help from the C level), so I present the following challenge: > can anyone think of a way that an object could "cross" interpreters? > For instance, are certain basic objects shared (str and int for > instance) and objects could be passed between interpreters by > attaching them as attributes of these core objects? I just don't > know enough about multiple interpreters to be sure -- but somehow I > thought they had separate object pools. C extension module attributes will share those objects across interpreters. Because the initializing function is not called again on extension modules when imported into another interpreter anything that is exposed by the module is the same across all interpreters. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/c1f33aaf/attachment.htm From brett at python.org Thu Jul 6 19:16:37 2006 From: brett at python.org (Brett Cannon) Date: Thu, 6 Jul 2006 10:16:37 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: References: Message-ID: On 7/5/06, Ka-Ping Yee wrote: > > On Wed, 5 Jul 2006, Brett Cannon wrote: > > On 7/4/06, Ka-Ping Yee wrote: > > > In response to Guido's comment about confusing the words "trusted" and > > > "untrusted", how about "empowered" and "restricted"? > > > > Maybe. I am really starting to lean towards trusted and sandboxed. > > It can be risky to use words of the form '*trust*' when talking > about security because 'trust' can have different meanings in > different contexts. (Examples: 'X is trusted' might mean 'X is > something i feel safe running without restrictions' or 'X *is* > in fact allowed to run without restrictions' or 'i need to run X > without restrictions in order to accomplish my task' or 'X is > something i am relying on for the security of my system'.) > > The most common context for 'trusted' that i seem to come across > is in the phrase 'trusted computing base' (TCB), which refers to > 'the thing that is supposed to be enforcing security restrictions' > as opposed to 'something i'm willing to let run unrestricted'. > So when i read 'trusted code' what comes to mind is the C implementation > of the Python interpreter, and it may be a good idea to reserve that > phrase for that purpose, if it's to be used at all. > > In any case, let's try to nail down clear names for the different > pieces we're talking about here, and i gently advise avoiding > '*trust*' words or using them with very limited meaning. For the next draft I am going with "trusted" and "sandboxed" just because I have already revised a decent amount and it is what my brain is defaulting to right now, but I can change the wording later. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/fa853fe3/attachment.html From mcherm at mcherm.com Thu Jul 6 19:24:12 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 06 Jul 2006 10:24:12 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) Message-ID: <20060706102412.7wdlblbikhs4cos8@login.werra.lunarpages.com> I quoted this unwritten bit of Python Zen, attributing it to Tim: > Syntax should not look like grit on my monitor. mwh writes: > I think it was Anthony: > > http://mail.python.org/pipermail/python-dev/2005-July/054581.html But that's not the original. Turns out, it WAS Anthony, and I had misquoted. The correct source appears to be: http://mail.python.org/pipermail/python-dev/2004-August/047179.html And the correct quote (for the record) is: Syntax Shall Not Look Like Grit On Tim's Monitor. From brett at python.org Thu Jul 6 19:27:29 2006 From: brett at python.org (Brett Cannon) Date: Thu, 6 Jul 2006 10:27:29 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: <44AC60D0.1040508@canterbury.ac.nz> References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> <44AC60D0.1040508@canterbury.ac.nz> Message-ID: On 7/5/06, Greg Ewing wrote: > > Brett Cannon wrote: > > > Armin in an email said that he thought it was > > a losing battle to try to hide 'file' from an interpreter. > > And I would change file() so that it didn't open > files. Then it would be harmless for code to have > access to the file class. Right, that is essentially what I proposed initially with the whole crippling idea. What the capabilities supporters are saying is that if we go that route we will be constantly finding objects that require similar crippling. It will degenerate into this constant chasing of our tail to plug security holes where an object that we did not account for leaked out and wreaked havoc. What they are saying is that if we harden Python so that references don't get out without us knowing about it we won't have this run-around. But then my question has been what makes us trying to cripple objects any less of a run-around then finding new ways to get at references of 'file' or any other object? I have been suggesting the former requires less running around than the latter. That is why I have asked that people see how many ways they can come up with to get to 'file' from a standard interpreter prompt so we can gauge how bad hiding references might be. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/9a96853e/attachment.htm From tomerfiliba at gmail.com Thu Jul 6 19:22:51 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Thu, 6 Jul 2006 19:22:51 +0200 Subject: [Python-Dev] introducing __dir__? Message-ID: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com> i'd guess this has been brought up before, but i'm not aware of it. anyway -- by what i've gathered, today's dir() "collects" the attributes of the object from __dict__, __methods__, __members__, and ultimately __class__. also, if i understood correctly, __methods__ and __members__ are deprecated. moreover, they are sequence objects, rather than callbale functions. that means my code can't my suggestion is simple -- replace this mechanism with a __dir__ - a special method that returns the list of attributes of the object. rationale: * remove deprecated __methods__, etc. * symmetry -- just like hex() calls __hex__, etc. * __methods__ and __members__ are lists rather than callable objects, which means they cannot be updated on-demand and the reason for bringing this up in the first place -- allowing proxy objects to show the attributes of the internal object. for example, in rpyc, i had to override __builtin__.dir to achieve this functionallity, which is of course not the optimal sollution. implementation: very straight forward and similar to the rest of the special methods: * dir() - returns the current scope * dir(obj) - retuns obj.__dir__(); if the object does not provide __dir__, uses the current mechanism as a fallback, or something similar the ultimate object type would grow a default __dir__ method that uses the same modus operandi as today's dir, which deriving classes could override to provide custom listings. therefore, i don't expect any backward-compatibility issues. aye or nay? -tomer From brett at python.org Thu Jul 6 19:40:38 2006 From: brett at python.org (Brett Cannon) Date: Thu, 6 Jul 2006 10:40:38 -0700 Subject: [Python-Dev] Restricted execution: what's the threat model? In-Reply-To: <20060706053756.ovcm9a05kko0o008@login.werra.lunarpages.com> References: <20060706053756.ovcm9a05kko0o008@login.werra.lunarpages.com> Message-ID: [replying to both Ping and Michael in the same email] On 7/6/06, Michael Chermside wrote: > > Ka-Ping Yee writes: > > i'm starting to think > > that it would be good to clarify what kinds of threats we are > > trying to defend against, and specify what invariants we are > > intending to preserve. > > Yes! > > > So here are a couple of questions for clarification (some with my > > guesses as to their answers): > > Okay, I'll throw in my thoughts also. > > > 1. When we say "restricted/untrusted/ interpreter" we > > don't really mean that the *interpreter* is untrusted, right? > > We mean that the Python code that runs in that interpreter is > > untrusted (i.e. to be prevented from doing harm), right? > > Agreed. My interpretation of the proposal was that interpreters > were either "sandboxed" or "trusted". "Sandboxed" means that there > are security restrictions imposed at some level (perhaps even NO > restrictions). "Trusted" means that the interpreter implements > no security restrictions (beyond what CPython already implements, > which isn't much) and thus runs faster. Yep. > 2. I'm assuming that the implementation of the Python interpreter > > is always trusted > > Sure... it's got to be. Yep. > What do > > we take the Trusted Computing Base to include? The Python VM > > implementation -- plus all the builtin objects and C modules? > > Plus the whole standard library? > > My interpretation of Brett's proposal is that the CPython developers > would try to ensure that Python VM had no "security holes" when > running in sandboxed mode. Of course, we also "try" to ensure no > crashes are possible also, and while we're quite good, we're not > perfect. > > Beyond that, all pure-python modules with source available (whether > in the stdlib or not) can be "trusted" because they run in a > sandboxed VM. All C modules are *up to the user*. Brett proposes > to provide a default list of useful-but-believed-to-be-safe modules > in the stdlib, but the user can configure the C-module whitelist > to whatever she desires. Michael has it on the money. > 3. Is it part of the plan that we want to protect Python code from > > other Python code? For example, should a Python program/function > > X be able to say "i want to launch/call program/function Y with > > *these* parameters and have it run under *these* limitations?" > > This has a big impact on the model. > > Now *that* is a good question. I would say the answer is a partial > "no", because there are pieces of Brett's security model that are > tied to the interpreter instance. Python code cannot launch another > interpreter (but perhaps it *should* be able to?), so it cannot > modify those restrictions for new Python code it launches. > > However, I would rather like to allow Python code to execute other > code with greater restrictions, although I would accept all kinds > of limitations and performance penalties to do so. I would be > satisfied if the caller could restrict certain things (like web > and file access) but not others (like memory limits or use of > stdout). I would satisfied if the caller paid huge overhead costs > of launching a separate interpreter -- heck, even a separate > process. And if it is willing to launch a separate process, then > Brett's model works just fine: allow the calling code to start > a new (restricted) Python VM. The plan is that there is no sandboxed eval() that runs unsafe code from a trusted interpreter within its namespace. I hope to provide Python code access to running a sandboxed interpreter where you can pass in a string to be executed, but the namespace for that sandboxed interpreter will be fresh and will not carry over in any way from the trusted interpreter. > We want to be able to guarantee that... > > > > A. The interpreter will not crash no matter what Python code > > it is given to execute. > > Agreed. We already want to guarantee that, with the caveat that the > guarantee doesn't apply to a few special modules (like ctypes). Right, which is why I have been trying to plug the various known crashers that do not rely upon a specific extension module from being imported. > B. Python programs running in different interpreters embedded > > in the same process cannot communicate with each other. > > I don't want to guarantee this, does someone else? It's > astonishingly hard... there are all kinds of clever "knock on the > walls" tricks. For instance, communicate by varying your CPU > utilization up and down in regular patterns. > > I'd be satisfied if they could pass information (perhaps even > someday provide a library making it *easy* to do so), but could > not pass unforgable items like Python object references, open file > descriptors, and so forth. Or at least cannot communicate without explicit allowances to do so. As for knocking on the walls, if you protect access to that kind of information well, it shouldn't be a problem. > C. Python programs running in different interpreters embedded > > in the same process cannot access each other's Python objects. > > I strengthen that slightly to all "unforgable" items, not just > object references. I would change that to add the caveat that what is exposed by a C extension module attribute will be shared. That is an implementation detail of multiple interpreters. > D. A given piece of Python code cannot access or communicate > > with certain Python objects in the same interpreter. > > > > E. A given piece of Python code can access only a limited set > > of Python objects in the same interpreter. > > Hmmm. I'm not sure. Not quite sure what you are getting at here, Ping. Are you saying to run code within an interpreter (sandboxed and not) and restricted even more beyond what the interpreter has been given by the security settings? These emails have convinced me to add a "Threat Model" section for the next draft of the design doc. -Brett -- Michael Chermside > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/97850f3c/attachment.htm From fdrake at acm.org Thu Jul 6 19:41:56 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu, 6 Jul 2006 13:41:56 -0400 Subject: [Python-Dev] introducing __dir__? In-Reply-To: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com> References: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com> Message-ID: <200607061341.56945.fdrake@acm.org> On Thursday 06 July 2006 13:22, tomer filiba wrote: > my suggestion is simple -- replace this mechanism with a __dir__ - > a special method that returns the list of attributes of the object. > > rationale: > * remove deprecated __methods__, etc. > * symmetry -- just like hex() calls __hex__, etc. > * __methods__ and __members__ are lists rather than callable > objects, which means they cannot be updated on-demand +1 -Fred -- Fred L. Drake, Jr. From brett at python.org Thu Jul 6 19:48:39 2006 From: brett at python.org (Brett Cannon) Date: Thu, 6 Jul 2006 10:48:39 -0700 Subject: [Python-Dev] what can we do to hide the 'file' type? In-Reply-To: <20060706083127.csxctx0sahr4kwk8@login.werra.lunarpages.com> References: <20060706083127.csxctx0sahr4kwk8@login.werra.lunarpages.com> Message-ID: On 7/6/06, Michael Chermside wrote: > > Armin Rigo writes: > > I don't think I can "sign off" [on hiding the file type]. Really hiding > > Python objects is quite hard IMHO. I agree. But we don't have to give up yet. How about instead of hiding > file, we cripple it. Completely. Modify the file type so that when > executing on a sandboxed interpreter, all of the dangerous methods > and attributes of file throw exceptions. This is basically what I proposed in the first place! Then we create a separate thing (in C) called a "SecureFileWrapper". > It has methods that are passed a reference to a file object and > can invoke the methods without error. We provide a means for obtaining > a SecureFileWrapper bound to a given file (perhaps open()). Yeah, it would be through open() if we returned wrappers instead of performing the checks directly in file itself. Essentially, we give up on hiding file, which is a frequently-used > type, and very hard to hide, and instead we rely on our ability to > write a reliably secure "SecureFileWrapper" class (in C). That is another possibility. Should simplify the code as well by having less checks and just have pure PySandbox_IsTrusted() checks in 'file' itself in unsafe places instead of a ton checks that the file being accessed is allowed. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/68c72935/attachment.htm From brett at python.org Thu Jul 6 19:51:40 2006 From: brett at python.org (Brett Cannon) Date: Thu, 6 Jul 2006 10:51:40 -0700 Subject: [Python-Dev] what can we do to hide the 'file' type? In-Reply-To: <20060706110312.GB15690@code0.codespeak.net> References: <20060706110312.GB15690@code0.codespeak.net> Message-ID: On 7/6/06, Armin Rigo wrote: > > Hi Brett, > > On Wed, Jul 05, 2006 at 05:01:48PM -0700, Brett Cannon wrote: > > And if Armin and/or Samuele sign off that what we find is most likely > (with > > "most likely" equalling 99% chance) all there is, then bonus points and > I > > will *really* be convinced. =) > > I don't think I can "sign off" that. Really hiding Python objects is > quite hard IMHO. =) That's fine. I didn't expect you to, especially without people either finding more instances of ways to get to 'file' or stating that they really tried and couldn't find any way to get to it. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/acbb3b0b/attachment.html From mcherm at mcherm.com Thu Jul 6 21:07:51 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 06 Jul 2006 12:07:51 -0700 Subject: [Python-Dev] what can we do to hide the 'file' type? Message-ID: <20060706120751.kd6pvgex6jtsgg4k@login.werra.lunarpages.com> Me: > I agree. But we don't have to give up yet. How about instead of hiding > file, we cripple it. Completely. Modify the file type so that when > executing on a sandboxed interpreter, all of the dangerous methods > and attributes of file throw exceptions. Brett: > This is basically what I proposed in the first place! in circles, pulling at his hair like a crazy man> Not quite. Your original proposal had the file type throwing exceptions when the user did something they weren't allowed to (access a file not on the list, etc). This version proposes that it *always* fails, and creates a separate beast (the SecureFileWrapper) for applying the restrictions. Why? Because if the C code in file enforces the rules, then all possible rules need to be written in advance, and you have to hold long arguments about whether to allow subdirectories, restrict file sizes, etc. Whereas SecureFileWrapper could delegate its restrictions to Python functions provided by the USER and then USERS could design whatever level of restriction they wanted. Imagine we want some code to be able to open files only if they contain the letter 'w': # get my own wrapper from __builtins__ myFileWrapper = file # define a function to constrain my callers def filenameContainsLetterW(path, mode): filename = os.path.basename(path) if 'w' not in filename and 'W' not in filename: raise PyXXX_SecurityException # create more restrictive wrapper class MustHaveW_FileWrapper: __metaclass__ = SecureFileWrapperMeta wrapped_file = init_condition = filenameContainsLetterW # register the wrapper so it applies to any code # in this stack frame or lower. The restriction is # automatically lifted when the current stack # frame exits. PyXXX_RegisterFileWrapper( MustHaveW_FileWrapper ) # Invoke less-trusted code with restrictions in place less_trusted_code() If the code fragment shown above ALREADY received a wrapped form of file which was restricted to read-only access, then less_trusted_code() would be restricted to read-only access to files containing 'w'. Okay, the syntax needs work, but the idea is that I can defin restrictions *in python* and apply them to other code. Using the stack to enforce wrappers is something Python code cannot get around (although it does prevent passing more-powerful callbacks to later stackframes). It all depends on whether we think that a simple set of restrictions implementable in C (such as 4 lists: read-only files, read-write files, read-only dirs, write-only dirs) will be sufficient, or if it is valuable to allow end users to fine-tune the restrictions. -- Michael Chermside From martin at v.loewis.de Thu Jul 6 21:52:27 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 06 Jul 2006 21:52:27 +0200 Subject: [Python-Dev] Subversion outage Friday 15:00 GMT Message-ID: <44AD69FB.7040002@v.loewis.de> I plan to do some subversion administration tomorrow; in order to be able to roll back changes, I have to disable write access during these changes. The outage shouldn't last longer than one hour; most likely, it will be much faster. Regards, Martin From brett at python.org Thu Jul 6 22:32:16 2006 From: brett at python.org (Brett Cannon) Date: Thu, 6 Jul 2006 13:32:16 -0700 Subject: [Python-Dev] what can we do to hide the 'file' type? In-Reply-To: <20060706120751.kd6pvgex6jtsgg4k@login.werra.lunarpages.com> References: <20060706120751.kd6pvgex6jtsgg4k@login.werra.lunarpages.com> Message-ID: On 7/6/06, Michael Chermside wrote: > > Me: > > I agree. But we don't have to give up yet. How about instead of hiding > > file, we cripple it. Completely. Modify the file type so that when > > executing on a sandboxed interpreter, all of the dangerous methods > > and attributes of file throw exceptions. > > Brett: > > This is basically what I proposed in the first place! > in circles, pulling at his hair like a crazy man> > > Not quite. Your original proposal had the file type throwing > exceptions when the user did something they weren't allowed to > (access a file not on the list, etc). This version proposes that > it *always* fails, and creates a separate beast (the > SecureFileWrapper) for applying the restrictions. Why? Because > if the C code in file enforces the rules, then all possible > rules need to be written in advance, and you have to hold long > arguments about whether to allow subdirectories, restrict file > sizes, etc. Whereas SecureFileWrapper could delegate its > restrictions to Python functions provided by the USER and then > USERS could design whatever level of restriction they wanted. Ah, OK, that makes more sense. I was not thinking about allowing for specifying a factory function to return the specific object to use when using open(). That could be rather handy and cool. I will definitely see if I can work it into the API in a reasonable way. -Brett Imagine we want some code to be able to open files only if they > contain the letter 'w': > > # get my own wrapper from __builtins__ > myFileWrapper = file > > # define a function to constrain my callers > def filenameContainsLetterW(path, mode): > filename = os.path.basename(path) > if 'w' not in filename and 'W' not in filename: > raise PyXXX_SecurityException > > # create more restrictive wrapper > class MustHaveW_FileWrapper: > __metaclass__ = SecureFileWrapperMeta > wrapped_file = > init_condition = filenameContainsLetterW > > # register the wrapper so it applies to any code > # in this stack frame or lower. The restriction is > # automatically lifted when the current stack > # frame exits. > PyXXX_RegisterFileWrapper( MustHaveW_FileWrapper ) > > # Invoke less-trusted code with restrictions in place > less_trusted_code() > > If the code fragment shown above ALREADY received > a wrapped form of file which was restricted to read-only > access, then less_trusted_code() would be restricted > to read-only access to files containing 'w'. > > Okay, the syntax needs work, but the idea is that I can > defin restrictions *in python* and apply them to other > code. Using the stack to enforce wrappers is something > Python code cannot get around (although it does prevent > passing more-powerful callbacks to later stackframes). > > It all depends on whether we think that a simple set > of restrictions implementable in C (such as 4 lists: > read-only files, read-write files, read-only dirs, > write-only dirs) will be sufficient, or if it is > valuable to allow end users to fine-tune the > restrictions. > > -- Michael Chermside > > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/52bc6eb9/attachment.htm From python-dev at zesty.ca Fri Jul 7 00:17:44 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Thu, 6 Jul 2006 17:17:44 -0500 (CDT) Subject: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript) In-Reply-To: References: Message-ID: On Wed, 21 Jun 2006, Brett Cannon wrote: > I have been working on a design doc for restricted execution of Python > as part of my dissertation for getting Python into Firefox to replace > JavaScript on the web. I've been doing a bunch of Firefox extension programming in Javascript and suddenly a few of the recent topics here came together in my head in a silent kapow of thoughts. This is kind of a side note to the security discussion, but they're all interconnected: network programming, concurrency, lexical scoping, security. Client-side web scripting tends to have a callback/continuation-ish concurrency style because it has to deal with network transactions (which can stall for long periods of time) in a user interface that is expected to stay always responsive. The Firefox API is full of listeners/observers, events, and continuation-like things. So one thing to consider is that, when Python is used for these purposes, it may be written in a specialized style. As i write JavaScript in this style i find i use nested functions a lot. When i want to set up a callback that uses variables in the current context, the natural thing to do is to define a new function in the local namespace. And if that function has to also provide a callback, then it has another function nested within it and so on. function spam() { var local_A = do_work(); do_network_transaction( new function(result_1) { var local_B = do_work(result_1); do_network_transaction( new function(result_2) { do_work(local_A, local_B, result_1, result_2); ... } ); } ); } So it is a possible consequence of embedding Python in Firefox that people will be using nested functions and lexical scoping in Python more often, which makes the recent discussion about access to enclosing scopes more significant. This is even related to security as well. Namespaces and lexical scoping are a natural and visually apparent way to limit access. If, for example, result_1 and result_2 in the above example are security-sensitive objects like secret keys or networking functions, you can see just by inspection that they cannot leak outside of spam() except by directly being passed in an outgoing function call. The standard Pythonic response to nested functions is to translate them into classes. But the nested function style has two advantages: 1. Variables are more appropriately scoped; they exist only where they are meaningful. (In a class, all the variables would be mixed into one namespace, where some of them would be invalid some of the time.) 2. Local variables are private. (Class instances don't get their own private namespaces to play in.) The first becomes more significant when in a more continuation-y style because it helps keep the various continuations from interfering with each other. The second becomes more significant if you care about restricting untrusted Python code. -- ?!ng From brett at python.org Thu Jul 6 00:07:23 2006 From: brett at python.org (Brett Cannon) Date: Wed, 5 Jul 2006 15:07:23 -0700 Subject: [Python-Dev] About a month until PSF call for test trackers closes! Message-ID: Back at the beginning of June, the Python Software Foundation's Infrastructure committee sent out an email requesting people to help us find a replacement tracker for SourceForge (the original announcement can be found at http://wiki.python.org/moin/CallForTrackers ). We asked that people put test trackers loaded with a data dump of Python's SourceForge tracker data online for us to play with. We said that we would close acceptance of test trackers as of August 7. That close date is a little over a month away! If you want to help get your favorite tracker up and going for the Infrastructure committee to evaluate, please visit http://wiki.python.org/moin/CallForTrackers and see if you can help out! We already have a tracker for JIRA up and loaded with the SF data to poke around with. There are also Trac and Roundup trackers in the planning stages that could use some help in getting the SF data imported and getting the servers up. In order for a tracker to be considered it *must* have the SF data loaded up. This will be a necessary step if the tracker is accepted, plus it lets us see how well the tracker lets us manage a large number of bugs. If you have questions or concerns, please email infrastructure at python.org(it is subscription-protected, but your email will be accepted; the subscription page is at http://mail.python.org/mailman/listinfo/infrastructure ). Thank you to those that have helped so far and those that do in the future. -Brett Cannon Chairman, PSF Infrastructure Committee -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060705/2cc322e8/attachment.html From pje at telecommunity.com Fri Jul 7 01:04:04 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Jul 2006 19:04:04 -0400 Subject: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript) In-Reply-To: References: Message-ID: <5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com> At 05:17 PM 7/6/2006 -0500, Ka-Ping Yee wrote: >On Wed, 21 Jun 2006, Brett Cannon wrote: > > I have been working on a design doc for restricted execution of Python > > as part of my dissertation for getting Python into Firefox to replace > > JavaScript on the web. > >I've been doing a bunch of Firefox extension programming in Javascript >and suddenly a few of the recent topics here came together in my head >in a silent kapow of thoughts. This is kind of a side note to the >security discussion, but they're all interconnected: network >programming, concurrency, lexical scoping, security. > >Client-side web scripting tends to have a callback/continuation-ish >concurrency style because it has to deal with network transactions >(which can stall for long periods of time) in a user interface that >is expected to stay always responsive. The Firefox API is full of >listeners/observers, events, and continuation-like things. So one >thing to consider is that, when Python is used for these purposes, >it may be written in a specialized style. > >As i write JavaScript in this style i find i use nested functions >a lot. When i want to set up a callback that uses variables in the >current context, the natural thing to do is to define a new function >in the local namespace. And if that function has to also provide a >callback, then it has another function nested within it and so on. > > function spam() { > var local_A = do_work(); > do_network_transaction( > new function(result_1) { > var local_B = do_work(result_1); > do_network_transaction( > new function(result_2) { > do_work(local_A, local_B, result_1, result_2); > ... > } > ); > } > ); > } > >So it is a possible consequence of embedding Python in Firefox that >people will be using nested functions and lexical scoping in Python >more often, which makes the recent discussion about access to >enclosing scopes more significant. As much as I'd love to have the nested scope feature, I think it's only right to point out that the above can be rewritten as something like this in Python 2.5: def spam(): local_A = do_work() result_1 = yield do_network_transaction() local_B = do_work(result_1) result_2 = yield do_network_transaction() do_work(local_A, local_B, result_1, result_2) ... All you need is an appropriate trampoline (possibly just a decorator) that takes the objects yielded by the function, and uses them up to set up callbacks that resume the generator with the returned result. >This is even related to security as well. Namespaces and lexical >scoping are a natural and visually apparent way to limit access. >If, for example, result_1 and result_2 in the above example are >security-sensitive objects like secret keys or networking functions, >you can see just by inspection that they cannot leak outside of >spam() except by directly being passed in an outgoing function call. > >The standard Pythonic response to nested functions is to translate >them into classes. But the nested function style has two advantages: > > 1. Variables are more appropriately scoped; they exist > only where they are meaningful. (In a class, all the > variables would be mixed into one namespace, where > some of them would be invalid some of the time.) > > 2. Local variables are private. (Class instances don't > get their own private namespaces to play in.) > >The first becomes more significant when in a more continuation-y >style because it helps keep the various continuations from >interfering with each other. The second becomes more significant >if you care about restricting untrusted Python code. These are all legitimate benefits of a functional style; it's just not necessarily the case that they also argue in favor of write access to outer function scopes, since this isn't necessary for writing callbacks, as in your original example. Co-routines are clearly superior to nested-function callbacks when the intended control flow is relatively linear. This is, if I recall correctly, why we added them. :) From evan at 4-am.com Fri Jul 7 01:08:22 2006 From: evan at 4-am.com (Evan Simpson) Date: Thu, 06 Jul 2006 18:08:22 -0500 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44AA1FF6.1050501@acm.org> References: <44AA1FF6.1050501@acm.org> Message-ID: Talin wrote: > I propose to create a new type of scoping rule, which I will call > "explicit" lexical scoping, that will co-exist with the current > "implicit" scoping rule that exists in Python today. I'd like to toss one more variant into the mix. If we really need to address variables in an intermediate scope, the most explicit way that I can think of doing so is to write (using Philip's example): def counter(num): scope as outer # "outer" is an arbitrary identifier def inc(): outer.num += 1 return outer.num return inc This is somewhat similar to the unworkable "use the function name" method that's been suggested. The "scope as X" statement associates the name "X" with the namespace of local variables in the scope in which it is executed. Such names are "lexically scoped", and only allow access to or rebinding of existing names from the originating scope (i.e. no "del outer.num" allowed). Cheers, Evan @ 4-am From jan-python at maka.demon.nl Fri Jul 7 01:00:03 2006 From: jan-python at maka.demon.nl (Jan Kanis) Date: Fri, 07 Jul 2006 01:00:03 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com> References: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com> Message-ID: On Thu, 06 Jul 2006 18:28:12 +0200, Phillip J. Eby wrote: > Here's the reason I think this keeps coming up, and why Guido's "just > use a class" argument doesn't really address the actual problem that's > taking place. I agree this argument is not generally applicable in every case, but why not in this specific situation? > In short: in *theory*, a rebinding operator or "nonlocal" declaration is > unnecessary. In *practice*, having one seems quite useful every time > you wander down the path that leads to having to rewrite your code just > because the language won't let you do that one tiny thing - I think this argument is a too general one. To me it is too close to "let's add every possible feature we can find, because it might be usefull to someone" :) One of the things I like about python is that it doesn't do this, and therefore the manual stays relatively small and I don't have to remember all kinds of rarely used features to make best use of the language. (I assume this is not a point of debate. repeating: Python is not Lisp ;-) ) Most of the arguments I've seen on the list are about 'how can we implement this', I'd like to see more arguments on whether this should be implemented at all. I was hoping someone would come up with a good example, so does anyone have one?? > - or so it feels like to the person who's experiencing it. Have you ever been that person, or come across such a situation? O, and I don't think the inc() example is a good one. In this incrementer the function call is all about the side effects, it's even in the name 'increment'. Incrementing is useless unless you increment /something/, so this should be better implemented as a class. From pje at telecommunity.com Fri Jul 7 01:25:19 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Jul 2006 19:25:19 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com> <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060706190445.0200a960@sparrow.telecommunity.com> At 01:00 AM 7/7/2006 +0200, Jan Kanis wrote: >On Thu, 06 Jul 2006 18:28:12 +0200, Phillip J. Eby >wrote: > >>Here's the reason I think this keeps coming up, and why Guido's "just >>use a class" argument doesn't really address the actual problem that's >>taking place. > >I agree this argument is not generally applicable in every case, but why >not in this specific situation? I'm saying that "just use a class" doesn't help in the incremental case. If you happen to know *ahead of time* that you will need a callback to modify state, you can perhaps do this, just like you can always coast your car to a stop if you have enough advance notice. However, sometimes you really want to have brakes. :) >>In short: in *theory*, a rebinding operator or "nonlocal" declaration is >>unnecessary. In *practice*, having one seems quite useful every time >>you wander down the path that leads to having to rewrite your code just >>because the language won't let you do that one tiny thing - > >I think this argument is a too general one. To me it is too close to >"let's add every possible feature we can find, because it might be usefull >to someone" :) Not at all. It's an argument regarding the incremental evolution of code. Python tries to keep the growth of complexity in a code base relatively flat -- incremental effort should be rewarded with incremental benefit. Discontinuities of complexity are to be avoided. The classic example is "Hello world", where in Java you must learn about six or eight different things, but in Python it is just 'print "Hello world"'. If you want to then make it a function, you indent it and add a 'def'. It is all very minimal and incremental, and for the most part, the things that people want to add to Python tend to be wherever discontinuities can occur. It is also true that, because Python is already so well-designed, that these remaining feature areas tend to lean toward the minor and/or obscure: coroutines, conditional expressions, etc. etc. >One of the things I like about python is that it doesn't do this, and >therefore the manual stays relatively small and I don't have to remember >all kinds of rarely used features to make best use of the language. (I >assume this is not a point of debate. repeating: Python is not Lisp ;-) ) And one of the things that makes it not Lisp is that not everyone is free to go around and actually add the new syntax to support their use cases. Guido must first be convinced. Discussions like this one are how we convince him. ;) >>- or so it feels like to the person who's experiencing it. > >Have you ever been that person, or come across such a situation? Many times. The hard thing about trying to provide use cases for this is that of course you can always find another way to write it. It's just that sometimes the nested function is a perfect solution at point in time A, and then at point in time B, a change in the program requires that the nested function mutate a bit of state, resulting in either a rewrite to do it the "right" way, or hacking mutable objects. Both are user-experience discontinuities: a sudden sharp rise in effort compared to reward. This then produces the perception of a "wart". Yes, it's purely a superficial problem. That's why it's called a "wart". :) >O, and I don't think the inc() example is a good one. In this incrementer >the function call is all about the side effects, it's even in the name >'increment'. Incrementing is useless unless you increment /something/, so >this should be better implemented as a class. Any example that's intended to motivate lexical scope modification is going to suffer from either being bogus due to oversimplification, or confusing due to application-specificity. That doesn't make it something that doesn't happen, it's just that the circumstances of it happening are somewhat more organic. By the way, I'm leaving out the people who have backgrounds in Lisp or Scheme and are trying to write it in Python. I don't really care for Lisp, myself, although my Python has become increasingly functional over time. Not because of Lisp (which I've never used apart from a few Emacs hacks 12 years ago, when I didn't even know what functional programming *was*), but because it's often the easiest way to do something. At least, until you bump into the rebinding problem. Personally, the idea to use a simple namespace object to store mutable variables as attributes seems rather appealing as a workaround solution in the absence of a rebinding operator. The other solution I thought of was providing a function called 'rebind()' that could be used like this: rebind(x=23) to change 'x' in the nearest enclosing scope. This solves the problem without adding any syntax, and it's incremental at the point of use. From jan-python at maka.demon.nl Fri Jul 7 01:47:50 2006 From: jan-python at maka.demon.nl (Jan Kanis) Date: Fri, 07 Jul 2006 01:47:50 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5.1.1.6.0.20060706190445.0200a960@sparrow.telecommunity.com> References: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com> <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> <5.1.1.6.0.20060706190445.0200a960@sparrow.telecommunity.com> Message-ID: On Fri, 07 Jul 2006 01:25:19 +0200, Phillip J. Eby wrote: > >>> - or so it feels like to the person who's experiencing it. >> >> Have you ever been that person, or come across such a situation? > > Many times. The hard thing about trying to provide use cases for this > is that of course you can always find another way to write it. It's > just that sometimes the nested function is a perfect solution at point > in time A, and then at point in time B, a change in the program requires > that the nested function mutate a bit of state, resulting in either a > rewrite to do it the "right" way, or hacking mutable objects. Well, not an actual example, but close enough. Leaving the rest to Guido then. From python-dev at zesty.ca Fri Jul 7 02:04:18 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Thu, 6 Jul 2006 19:04:18 -0500 (CDT) Subject: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript) In-Reply-To: <5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com> References: <5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com> Message-ID: On Thu, 6 Jul 2006, Phillip J. Eby wrote: > As much as I'd love to have the nested scope feature, I think it's only > right to point out that the above can be rewritten as something like this > in Python 2.5: > > def spam(): > local_A = do_work() > result_1 = yield do_network_transaction() > local_B = do_work(result_1) > result_2 = yield do_network_transaction() > do_work(local_A, local_B, result_1, result_2) > ... > > All you need is an appropriate trampoline (possibly just a decorator) that > takes the objects yielded by the function, and uses them up to set up > callbacks that resume the generator with the returned result. Clever! Could you help me understand what goes on in do_network_transaction() when you write it this way? In the Firefox/JavaScript world, the network transaction is fired off in another thread, and when it's done it posts an event back to the JavaScript thread, which triggers the callback. And what happens if you want to supply more than one continuation? In my JavaScript code i'm setting up two continuations per step -- one for success and one for failure, since with a network you never know what might happen. -- ?!ng From guido at python.org Fri Jul 7 02:10:59 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Jul 2006 02:10:59 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com> References: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> <17581.9926.915438.200819@montanaro.dyndns.org> <5.1.1.6.0.20060706120457.01f1bab8@sparrow.telecommunity.com> Message-ID: On 7/6/06, Phillip J. Eby wrote: > Here's the reason I think this keeps coming up, and why Guido's "just use a > class" argument doesn't really address the actual problem that's taking place. (And note that I've recently gone on record as doubting that argument myself.) > When you are writing some function, and you find yourself using a nested > function because it's the closest match for something you're doing, there > occasionally comes a point at which you realize that, gosh, you need to > mutate something in an outer scope. At that point, your choices are to > either kludge it with a mutable, or to reorganize the whole thing. Both of > these *feel* like warts because they're making you do less-than-optimal > things. Your mental "flow" is disrupted because the language is forcing > you to work around it, rather than having a way of working with it. Right. I'm actually not a big fan of the word "flow" -- it often doesn't apply to the way I work myself, and it tends to emphasize the needs of the writer at the expense of the reader. (Perl programmers have excellent flow due to the many ways of doing things in that language. 'nuff said.) But I agree that the need for a refactoring like this to be a simple, local thing is important. (Compare my argument for making print a function -- which by the way received an applause at EuroPython once I explained it, after getting boohs when the slide first went up.) > This is a user experience issue, not a technical one. The fact that you > can say, "you should've done it differently in the first place" doesn't do > anything for your flow. In theory, you could design cars without any > brakes, because people could just coast to a stop if they planned well > enough in advance. ;-) In practice, you need the brakes because people > often don't discover their need to stop until much later in the process. I don't like the brakeless car example either. It is too extreme and opens your argument up to easy objections (the similarity between the cases isn't that great anyway). > This is a flow issue that's specific to *incremental* development. What > happens is that first you refactor the code in a function to include nested > functions. The variable references don't change, you're just indenting > some code. *Then*, it later comes up that you need to rebind a variable, > and now you have to globally change the variable to make it work as a > mutable instead, or else you have to refactor all the variables to be > 'self.' references and put a class in somewhere. > > This destroys the flow of incrementally developing whatever it is you're > developing, and makes you stop to do excise work. That's why being able to > rebind a variable without redefining it is important. Quite apart from the coder's flow, if the code is being developed incrementally, there are probably readers reviewing it before and after the refactoring. It's a lot easier to verify that a refactoring is correct when it doesn't choose a completely different approach. > In short: in *theory*, a rebinding operator or "nonlocal" declaration is > unnecessary. In *practice*, having one seems quite useful every time you > wander down the path that leads to having to rewrite your code just because > the language won't let you do that one tiny thing -- or so it feels like to > the person who's experiencing it. There really is hardly any need to argue the point any more; I've said that I will accept the feature if we can find a mutually acceptable syntax. I still only see these two alternatives as viable: a new keyword (nonlocal being my current favorite) or stretching the meaning of global. After Andrew Koenig's Dijkstra reference I'm beginning to think that the latter may actually be fine. (Oh, and somebody can add "get rid of global" to PEP 3099 -- that ain't gonna happen. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From bob at redivi.com Fri Jul 7 02:22:57 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 6 Jul 2006 17:22:57 -0700 Subject: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript) In-Reply-To: References: <5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com> Message-ID: <8C44EB52-F933-4169-A7F4-45544DE68C83@redivi.com> On Jul 6, 2006, at 5:04 PM, Ka-Ping Yee wrote: > On Thu, 6 Jul 2006, Phillip J. Eby wrote: >> As much as I'd love to have the nested scope feature, I think it's >> only >> right to point out that the above can be rewritten as something >> like this >> in Python 2.5: >> >> def spam(): >> local_A = do_work() >> result_1 = yield do_network_transaction() >> local_B = do_work(result_1) >> result_2 = yield do_network_transaction() >> do_work(local_A, local_B, result_1, result_2) >> ... >> >> All you need is an appropriate trampoline (possibly just a >> decorator) that >> takes the objects yielded by the function, and uses them up to set up >> callbacks that resume the generator with the returned result. > > Clever! Could you help me understand what goes on in > do_network_transaction() when you write it this way? In the > Firefox/JavaScript world, the network transaction is fired off > in another thread, and when it's done it posts an event back > to the JavaScript thread, which triggers the callback. > > And what happens if you want to supply more than one continuation? > In my JavaScript code i'm setting up two continuations per step -- > one for success and one for failure, since with a network you never > know what might happen. When you have a failure the yield expression raises an exception instead of returning a result. -bob From bioinformed at gmail.com Fri Jul 7 03:56:06 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Thu, 6 Jul 2006 21:56:06 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <44AA1FF6.1050501@acm.org> Message-ID: <2e1434c10607061856t6ef10e92lb8bec4f879cbfcc8@mail.gmail.com> On 7/6/06, Evan Simpson wrote: > > Talin wrote: > > I propose to create a new type of scoping rule, which I will call > > "explicit" lexical scoping, that will co-exist with the current > > "implicit" scoping rule that exists in Python today. > > I'd like to toss one more variant into the mix. If we really need to > address variables in an intermediate scope, the most explicit way that I > can think of doing so is to write (using Philip's example): > > def counter(num): > scope as outer # "outer" is an arbitrary identifier > def inc(): > outer.num += 1 > return outer.num > return inc > Why not extend the interface to the locals builtin and add a __getitem__ that returns a proxy to access locals defined in other lexical scopes via __{get/set/del}attr_: def counter(num): num = 1 def inc(): locals[1].num += 1 return outer.num return inc Where, for CPython, locals[n] gives access to NamespaceProxy(sys._getframe(n).f_locals). In addition to having a relatively pleasing and explicit syntax, this may be a feasible method for allowing portable introspection into outer scopes without having to export the whole frame object a la sys._getframe(n). I strongly suspect that Jython, IronPython, and PyPy would have little difficulty supporting (and optimizing) this construct. Lacking core language support, it is easy to roll an object that does just what I suggest. Actual implementation is left to a more motivated reader, of course. Just another crazy idea to throw into the pot. -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060706/3e422aae/attachment.html From pje at telecommunity.com Fri Jul 7 03:59:09 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Jul 2006 21:59:09 -0400 Subject: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript) In-Reply-To: References: <5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com> <5.1.1.6.0.20060706185558.01f1fe40@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060706215215.01f1f938@sparrow.telecommunity.com> At 07:04 PM 7/6/2006 -0500, Ka-Ping Yee wrote: >On Thu, 6 Jul 2006, Phillip J. Eby wrote: > > As much as I'd love to have the nested scope feature, I think it's only > > right to point out that the above can be rewritten as something like this > > in Python 2.5: > > > > def spam(): > > local_A = do_work() > > result_1 = yield do_network_transaction() > > local_B = do_work(result_1) > > result_2 = yield do_network_transaction() > > do_work(local_A, local_B, result_1, result_2) > > ... > > > > All you need is an appropriate trampoline (possibly just a decorator) that > > takes the objects yielded by the function, and uses them up to set up > > callbacks that resume the generator with the returned result. > >Clever! Could you help me understand what goes on in >do_network_transaction() when you write it this way? In the >Firefox/JavaScript world, the network transaction is fired off >in another thread, and when it's done it posts an event back >to the JavaScript thread, which triggers the callback. The only difference here is that the callback is to the generator instance's send() method. How the actual callback machinery works, and whether threads are involved are orthogonal to the code itself, making it an ideal implementation-independent way to express asynchronous algorithms. >And what happens if you want to supply more than one continuation? >In my JavaScript code i'm setting up two continuations per step -- >one for success and one for failure, since with a network you never >know what might happen. Errors can be passed back via the generator instance's throw() method. See also peak.events (which could be considered a prototype implementation of PEP 342 and associated trampoline features) and Twisted's Deferred machinery (which allows sending either a result or a "failure" (where failures can be converted back into thrown exceptions). From talin at acm.org Fri Jul 7 08:12:31 2006 From: talin at acm.org (Talin) Date: Thu, 06 Jul 2006 23:12:31 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> Message-ID: <44ADFB4F.5070207@acm.org> Brett Cannon wrote: > On 7/5/06, Talin wrote: >> Transitioning from the checked to the unchecked state could only be done >> via C code. So the 'file' wrapper, for example, would switch over to the >> unchecked interpreter before calling the actual methods of 'file'. That >> C wrapper might also check the current permission state to see what >> operations were legal. > > So add the proper checks in Python/ceval.c:call_function() to check for > this > flag on every object passed in that is called? Right. I also realized that you would need to add code that propagates the checked bit from the class to any instances of the class. So whenever you call a class to create an object, if the class has the checked bit, the instance will have it set as well. >> So essentially, what I propose is to define a simple security primitive >> - which essentially comes down to checking a single bit - and use that >> as a basis to create more complex and subtle security mechanisms. > > Right, but it does require that the proper verification function be turned > on so that the permission bit on 'file' is checked. It kind of seems like > 'rexec' and its f_restricted flag it set on execution frames, except you > are > adding an object-level flag as well. > > Either way, the trick is not fouling up switching between the two checking > functions. > > -Brett I wasn't aware of how rexec worked, but that seems correct to me. Given a 'restricted' flag on a stack frame, and one on the object as well, then the code for checking for permission violations is nothing more than: if (object.restricted && exec_frame.restricted) raise SecurityException In particular, there's no need to call a function to check a "permission level" or "access rights" or anything of the sort - all that stuff is implemented at a higher level. By making the check very simple, it can also be made very fast. And by making it fast, we can afford to call it a lot - for every operation in fact. And if we can call it for every operation, then we don't have to spend time hunting down all of the possible loopholes and ways in which 'file' or other restricted objects might be accessed. Originally I had thought to simply add a check like the above into the interpreter. However, that would mean that *all* code, whether restricted or not, would have to pay the (slight) performance penalty of checking that flag. So instead, I thought it might be more efficient to have two different code paths, one with the check and one without. But all this is based on profound ignorance of the interpreter - there might be a hundred other, better ways to do this without having to create two versions of ceval. Another interesting think about the check bit idea is that you can set it on any kind of object. For example, you could set it on individual methods of a class rather than the class as a whole. However, that's probably needlessly elaborate, since fine-grained access control will be much more elegantly achieved via trusted wrappers. -- Talin From nnorwitz at gmail.com Fri Jul 7 08:12:47 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 6 Jul 2006 23:12:47 -0700 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: <20060630124911.29014.360583188.divmod.quotient.16329@ohm> References: <20060630124911.29014.360583188.divmod.quotient.16329@ohm> Message-ID: On 6/30/06, Jean-Paul Calderone wrote: > > > >There are at least 6 bugs that really, really need to be fixed before > >release. Several of these are AST bugs. Jeremy knows about them and > >plans to fix them once he's back from vacation. Anyone else wanna > >help out? One is for a socket problem and another is for doc. The > >current list of serious bugs are in the PEP: > > > > http://www.python.org/dev/peps/pep-0356/ > > > > Please add #1494314 to the list. > > http://sourceforge.net/tracker/index.php?func=detail&aid=1494314&group_id=5470&atid=105470 I updated the patch attached to the bug report. Can people test (or review) it and provide comments to the report? Thanks, n From rasky at develer.com Fri Jul 7 09:35:42 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 7 Jul 2006 09:35:42 +0200 Subject: [Python-Dev] Switch and static, redux References: Message-ID: <018501c6a197$f3bdbe70$d3402597@bagio> Guido van Rossum wrote: > So, my proposal is to give up on static, accept PEP 3103 with the > following options: > - Syntax alternative 2+B (unindented cases, 'case in ...' for > multiple cases). > - Semantics option 3 (def-time freezing) I know it's only a bikeshed issue here, but wouldn't it be the first case where a statement ending with ":" does not introduce an indented suite? Is it really worth to create this non-orthogonality in the language? IMHO, if we went for indentend cases, we could teach editors to indent cases *only* 1 or 2 spaces. That would preserve orthogonality of the language, and allow not to consume too much horizontal space. Or, what about optionally indented cases? That is, allow both forms as correct syntax. It would make it just a matter of style at that point (and Python will finally have its first religious wars over indentation.... AT LAST! :) Giovanni Bajo From greg.ewing at canterbury.ac.nz Fri Jul 7 09:48:52 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 07 Jul 2006 19:48:52 +1200 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> <44AC60D0.1040508@canterbury.ac.nz> Message-ID: <44AE11E4.9040809@canterbury.ac.nz> Brett Cannon wrote: > On 7/5/06, *Greg Ewing* And I would change file() so that it didn't open > files. Then it would be harmless for code to have > access to the file class. > Right, that is essentially what I proposed initially with the whole > crippling idea. > > What the capabilities supporters are saying is that if we go that route > we will be constantly finding objects that require similar crippling. We've got our wires crossed somewhere. I *am* a capabilities supporter. But for a capability model to work, the capabilities need to be isolated and encapsulated in functions or objects that can be independently provided or not provided. In the case of file(), that means separating the capability of being able to open a file from the capability of accessing an already-opened file. These two things are currently conflated in the file class. BTW, I object to the term "crippling" in this particular case. If you have access to open(), there's no need for file() to be able to do the same thing. And if you don't have access to open(), it's because someone doesn't want you to be able to open files. -- Greg From greg.ewing at canterbury.ac.nz Fri Jul 7 09:50:46 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 07 Jul 2006 19:50:46 +1200 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> <44AC60D0.1040508@canterbury.ac.nz> Message-ID: <44AE1256.1000509@canterbury.ac.nz> Another thing I perhaps should point out is that I'm proposing the separation of open() and file() for *all* code, not just restricted code. So it's not a matter of "crippling" file() specially for restricted code. -- Greg From just at letterror.com Fri Jul 7 12:09:26 2006 From: just at letterror.com (Just van Rossum) Date: Fri, 7 Jul 2006 12:09:26 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: Message-ID: Evan Simpson wrote: > I'd like to toss one more variant into the mix. If we really need to > address variables in an intermediate scope, the most explicit way > that I can think of doing so is to write (using Philip's example): > > def counter(num): > scope as outer # "outer" is an arbitrary identifier > def inc(): > outer.num += 1 > return outer.num > return inc > > This is somewhat similar to the unworkable "use the function name" > method that's been suggested. The "scope as X" statement associates > the name "X" with the namespace of local variables in the scope in > which it is executed. Such names are "lexically scoped", and only > allow access to or rebinding of existing names from the originating > scope (i.e. no "del outer.num" allowed). Why couldn't at least augmented assignment be implicitly rebinding? It has been suggested before (in the context of a rebinding operator), but I'm wondering, is this also off the table? def counter(num): def inc(): num += 1 return num return inc Reads very natural to me. It's likely the most frequent example of what people try before they learn that rebinding to outer scopes isn't allowed. It could Just Work. Just From ncoghlan at gmail.com Fri Jul 7 12:29:21 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 07 Jul 2006 20:29:21 +1000 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <17581.9926.915438.200819@montanaro.dyndns.org> References: <20060706165539.sl1zeof09w0sks0o@webmail.nl.demon.net> <17581.9926.915438.200819@montanaro.dyndns.org> Message-ID: <44AE3781.7080906@gmail.com> skip at pobox.com wrote: > jan-python> So.. are we only thinking about implementing this outer > jan-python> scope assignment because there's lots of talk about it on > jan-python> the list, ... > > :-) > > jan-python> ... or are there actually use cases that would become > jan-python> clearer if assigning to an outer scope variable was allowed? > > I think full lexical scoping will only be of use to people who use nested > scopes heavily. The more typical user will be happy to just refer to values > in outser scopes without modifying them and rely on classes to save changed > state across calls. I think it's almost a YAGNI, but I'm sure others will > disagree. I think it falls into the same category as Guido's ultimate acceptance of PEP 308. There are assorted ways to live *without* conditional expressions, but each of the workarounds for its absence had issues. Switching to a statement worked properly, but meant you didn't have a single expression any more. Use the and-or trick kept the single expression characteristic, but was easy to get wrong. Hence, PEP 308: One Obvious Way to do it, assuming you want to do it in the first place. I think writing to outer scopes is similar. You can box the variable, or make it an attribute of an object, but either approach requires you to refactor *all* uses of the variable, rather than just the one you currently care about. Hence, 'nonlocal': One Obvious Way to do it, assuming you want to do it in the first place. That way, when you're *reading* the code of someone who likes to use such tricks, you only need to know how to read the one obvious way, rather than having to decipher whichever method they've chosen to work around the limitation. The perennial accumulator example still takes 6 lines, though: def accumulator(n): def increment(i): nonlocal n n += i return n return increment Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From fredrik at pythonware.com Fri Jul 7 12:38:43 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 07 Jul 2006 12:38:43 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: Just van Rossum wrote: > Why couldn't at least augmented assignment be implicitly rebinding? It > has been suggested before (in the context of a rebinding operator), but > I'm wondering, is this also off the table? > > def counter(num): > def inc(): > num += 1 > return num > return inc > > Reads very natural to me. It's likely the most frequent example of what > people try before they learn that rebinding to outer scopes isn't > allowed. It could Just Work. note that most examples of this type already work, if the target type is mutable, and implement the right operations: def counter(num): num = mutable_int(num) def inc(): num += 1 return num return inc maybe we should consider adding mutable strings and mutable numbers to Python 3.0 ? and a "mutable" built-in, that does the opposite of the "freeze" stuff: def counter(num): num = mutable(num) def inc(): num += 1 return num return inc (what is this thread doing on python-dev, btw? shouldn't it be over at the 3000 list, so I can enjoy my vacation without being drawn into yet another endless discussion thread ;-) From ncoghlan at gmail.com Fri Jul 7 13:45:04 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 07 Jul 2006 21:45:04 +1000 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: <44AE1256.1000509@canterbury.ac.nz> References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> <44AC60D0.1040508@canterbury.ac.nz> <44AE1256.1000509@canterbury.ac.nz> Message-ID: <44AE4940.5020603@gmail.com> Greg Ewing wrote: > Another thing I perhaps should point out is that > I'm proposing the separation of open() and file() > for *all* code, not just restricted code. So it's > not a matter of "crippling" file() specially for > restricted code. What would the signature of the file constructor be in that case? Would it accept a single CObject instance, with open() bypassing the normal constructor, and handing the file pointer directly to the file object instance? It seems like a reasonable approach for making 'dangerous' objects like file and socket much easier to secure - have a separate factory function to create *new* instances at the C level, so that you can hand the objects over without worrying about providing access to the constructor (because the constructor *accepts* the OS-level object as an argument, rather than creating it anew). Alternatively, using a "no-Python-level-introspection" metaclass might be another way to achieve the same effect in a more universal fashion. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Fri Jul 7 13:55:45 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 07 Jul 2006 21:55:45 +1000 Subject: [Python-Dev] introducing __dir__? In-Reply-To: <200607061341.56945.fdrake@acm.org> References: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com> <200607061341.56945.fdrake@acm.org> Message-ID: <44AE4BC1.1000203@gmail.com> Fred L. Drake, Jr. wrote: > On Thursday 06 July 2006 13:22, tomer filiba wrote: > > my suggestion is simple -- replace this mechanism with a __dir__ - > > a special method that returns the list of attributes of the object. > > > > rationale: > > * remove deprecated __methods__, etc. > > * symmetry -- just like hex() calls __hex__, etc. > > * __methods__ and __members__ are lists rather than callable > > objects, which means they cannot be updated on-demand > > +1 +1 here, too. It would also allow objects which override __getattribute__ and/or __getattr__ to make dir() provide a sane answer (or raise an exception to indicate that a sane answer isn't possible). (This was something that actually came up when trying to implement a namespace object that *didn't* automatically fall back to its class namespace for Python level attribute access) For backwards compatibility, dir() could still fall back to the current mechanism if __dir__ isn't found. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From martin at v.loewis.de Fri Jul 7 17:28:23 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 07 Jul 2006 17:28:23 +0200 Subject: [Python-Dev] SVN write access is back Message-ID: <44AE7D97.6070201@v.loewis.de> I just turned the subversion write access back on. Unfortunately, I did not manage to perform the changes I wanted (import ctypes), so I'll have to retry later when the open issues have been clarified. Regards, Martin From theller at python.net Fri Jul 7 18:19:58 2006 From: theller at python.net (Thomas Heller) Date: Fri, 07 Jul 2006 18:19:58 +0200 Subject: [Python-Dev] test_ctypes failure on Mac OS X/PowerPC 10.3.9 (Panther) In-Reply-To: <3B96F3FF-B847-4E57-ACC2-F7D979DCA5BA@mac.com> References: <44982ADE.5070404@activestate.com> <4498393E.1020101@python.net> <71C6D0BF-B569-48BD-9F9D-05D3660FF2AA@mac.com> <3B96F3FF-B847-4E57-ACC2-F7D979DCA5BA@mac.com> Message-ID: <44AE89AE.1040105@python.net> Ronald Oussoren schrieb: > On 20-jun-2006, at 20:50, Ronald Oussoren wrote: > >> >> On 20-jun-2006, at 20:06, Thomas Heller wrote: >> >>> Trent Mick schrieb: >>>> Thomas and others, >>>> >>>> Has anyone else seen failures in test_ctypes on older Mac OS X/ >>>> PowerPC? >>>> Results are below. This is running a build of the trunk from last >>>> night: >>>> >>>> ./configure && make && ./python.exe Lib/test/test_ctypes.py >>>> >>>> Note that the test does NOT fail on the Mac OS X/x86 10.4.6 box >>>> that I have. >>> >>> It also works on 10.4.?? Power PC. I guess the fix has to wait until >>> I'm able to install 10.3 on my mac, I have the DVDs already but >>> have not >>> yet had the time. If anyone is willing to give me ssh access to a >>> 10.3 >>> box I can try to fix this earlier. >> >> I had some problems with my 10.3-capable box, but happily enough it >> decided to come alive again. I'm currently booted into 10.3.9 and >> will have a look. > > It is a platform bug, RTLD_LOCAL doesn't work on 10.3. The following > C snippet fails with the same error as ctypes: FAIL: dlcompat: unable > to open this file with RTLD_LOCAL. This seems to be confirmed by this > sourcet test file from darwin: http://darwinsource.opendarwin.org/ > 10.4.1/dyld-43/unit-tests/test-cases/dlopen-RTLD_LOCAL/main.c. I think that this problem still exists. From what I have found out so far (still I don't have access to osx 10.3 myself), it seems that there are three possibilities: - ignore this issue ;-) - make RTLD_GLOBAL the default mode on osx: either on 10.3 only, or on all versions (how can I determine 10.3 from 10.4?) - change the *test* only so that RTLD_GLOBAL is used on osx. I do not know how other ctypes users on osx are affected by this problem, however http://codespeak.net/pypy/dist/pypy/doc/rctypes.html#ctypes-version-and-platform-notes recommends to patch ctypes/__init__.py on osx 10.3. Thanks, Thomas From martin at v.loewis.de Fri Jul 7 18:30:08 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 07 Jul 2006 18:30:08 +0200 Subject: [Python-Dev] Extended Subversion outage: Friday 16:40 GMT In-Reply-To: <44AD69FB.7040002@v.loewis.de> References: <44AD69FB.7040002@v.loewis.de> Message-ID: <44AE8C10.2040008@v.loewis.de> Martin v. L?wis wrote: > I plan to do some subversion administration > tomorrow; in order to be able to roll back changes, > I have to disable write access during these > changes. I'm going to make a second attempt ten minutes from now. Regards, Martin From martin at v.loewis.de Fri Jul 7 18:54:12 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 07 Jul 2006 18:54:12 +0200 Subject: [Python-Dev] Extended Subversion outage: Friday 16:40 GMT In-Reply-To: <44AE8C10.2040008@v.loewis.de> References: <44AD69FB.7040002@v.loewis.de> <44AE8C10.2040008@v.loewis.de> Message-ID: <44AE91B4.9070903@v.loewis.de> Martin v. L?wis wrote: > Martin v. L?wis wrote: >> I plan to do some subversion administration >> tomorrow; in order to be able to roll back changes, >> I have to disable write access during these >> changes. > > I'm going to make a second attempt ten minutes from > now. I completed importing the ctypes history, so the repository is now back online. Regards, Martin From brett at python.org Fri Jul 7 18:56:33 2006 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jul 2006 09:56:33 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <44ADFB4F.5070207@acm.org> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> Message-ID: On 7/6/06, Talin wrote: > > Brett Cannon wrote: > > On 7/5/06, Talin wrote: > >> Transitioning from the checked to the unchecked state could only be > done > >> via C code. So the 'file' wrapper, for example, would switch over to > the > >> unchecked interpreter before calling the actual methods of 'file'. That > >> C wrapper might also check the current permission state to see what > >> operations were legal. > > > > So add the proper checks in Python/ceval.c:call_function() to check for > > this > > flag on every object passed in that is called? > > Right. I also realized that you would need to add code that propagates > the checked bit from the class to any instances of the class. So > whenever you call a class to create an object, if the class has the > checked bit, the instance will have it set as well. > > >> So essentially, what I propose is to define a simple security primitive > >> - which essentially comes down to checking a single bit - and use that > >> as a basis to create more complex and subtle security mechanisms. > > > > Right, but it does require that the proper verification function be > turned > > on so that the permission bit on 'file' is checked. It kind of seems > like > > 'rexec' and its f_restricted flag it set on execution frames, except you > > are > > adding an object-level flag as well. > > > > Either way, the trick is not fouling up switching between the two > checking > > functions. > > > > -Brett > > I wasn't aware of how rexec worked, but that seems correct to me. > > Given a 'restricted' flag on a stack frame, and one on the object as > well, then the code for checking for permission violations is nothing > more than: > > if (object.restricted && exec_frame.restricted) > raise SecurityException > > In particular, there's no need to call a function to check a "permission > level" or "access rights" or anything of the sort - all that stuff is > implemented at a higher level. > > By making the check very simple, it can also be made very fast. And by > making it fast, we can afford to call it a lot - for every operation in > fact. > > And if we can call it for every operation, then we don't have to spend > time hunting down all of the possible loopholes and ways in which 'file' > or other restricted objects might be accessed. Not true. You have to set this object restriction flag, right? What happens if you don't set it on all of the proper classes/types? You end up in the exact same situation you are with crippling; making sure you cover your ass with what you flag as unsafe else you risk having something get passed you. Originally I had thought to simply add a check like the above into the > interpreter. However, that would mean that *all* code, whether > restricted or not, would have to pay the (slight) performance penalty of > checking that flag. So instead, I thought it might be more efficient to > have two different code paths, one with the check and one without. But > all this is based on profound ignorance of the interpreter - there might > be a hundred other, better ways to do this without having to create two > versions of ceval. Yeah, keep it simple, especially when it comes to ceval.c . Another interesting think about the check bit idea is that you can set > it on any kind of object. For example, you could set it on individual > methods of a class rather than the class as a whole. However, that's > probably needlessly elaborate, since fine-grained access control will be > much more elegantly achieved via trusted wrappers. Yeah, that seems a bit extreme. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/caeaa100/attachment.html From evan at 4-am.com Fri Jul 7 18:56:01 2006 From: evan at 4-am.com (Evan Simpson) Date: Fri, 07 Jul 2006 11:56:01 -0500 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <2e1434c10607061856t6ef10e92lb8bec4f879cbfcc8@mail.gmail.com> References: <44AA1FF6.1050501@acm.org> <2e1434c10607061856t6ef10e92lb8bec4f879cbfcc8@mail.gmail.com> Message-ID: <44AE9221.9020704@4-am.com> Kevin Jacobs wrote: > Why not extend the interface to the locals builtin and add a __getitem__ > that returns a proxy to access locals defined in other lexical scopes > via __{get/set/del}attr_: > > def counter(num): > num = 1 > def inc(): > locals[1].num += 1 > return outer.num > return inc > > Where, for CPython, locals[n] gives access to > NamespaceProxy(sys._getframe(n).f_locals). Two nits: First, I suspect that you meant to write "return locals[1].num". Second, sys._getframe doesn't do what you want, here. It reaches back up the call chain, not out into lexically containing scopes. That said, I like the idea of giving "locals[]" the meaning you intended. It has the advantage of not adding any new keywords or syntactic constructs, but the disadvantage of not explicitly signaling that locals in a given scope will be twiddled elsewhere. Also, for efficiency's sake, it might be desirable to only allow a literal integer as the index, and to deal with use of "locals[]" at compile time rather than dynamically. I'm not sure how much overhead would be involved in enabling dynamic lookup of arbitrary lexically containing scopes, but I would *not* want to enable stuff like "locals[i * 2 - 1]". Cheers, Evan @ 4-am From theller at python.net Fri Jul 7 18:57:40 2006 From: theller at python.net (Thomas Heller) Date: Fri, 07 Jul 2006 18:57:40 +0200 Subject: [Python-Dev] Extended Subversion outage: Friday 16:40 GMT In-Reply-To: <44AE91B4.9070903@v.loewis.de> References: <44AD69FB.7040002@v.loewis.de> <44AE8C10.2040008@v.loewis.de> <44AE91B4.9070903@v.loewis.de> Message-ID: Martin v. L?wis schrieb: > Martin v. L?wis wrote: >> Martin v. L?wis wrote: >>> I plan to do some subversion administration >>> tomorrow; in order to be able to roll back changes, >>> I have to disable write access during these >>> changes. >> >> I'm going to make a second attempt ten minutes from >> now. > > I completed importing the ctypes history, so the > repository is now back online. Thanks, Martin. Thomas From bborcic at gmail.com Fri Jul 7 19:02:52 2006 From: bborcic at gmail.com (Boris Borcic) Date: Fri, 07 Jul 2006 19:02:52 +0200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> Message-ID: Guido van Rossum wrote: > On 7/5/06, Michael Chermside wrote: >> Guido writes: >> [discussion of how to fix the can't-bind-outer-scope-vars wart] ... > > Are there any other native speakers who side with Michael? > A bit OT, but why should native speakers (eg of English) have special authority ? I mean this neither as an attack on native speakers nor as a rethorical question. I recently re-read the "sum()" discussion to figure out how it came about that sum() half bails out on sequences_of_sequences (by requiring me to specify the start value) except when the nested sequences are strings, in which cases it 90% bails out : by first asking for an adequate start value, and *then* - once I've provided it - telling me it understands my intent but that I should use ''.join instead. Is this a programming language shell or a text-based adventure game ? <0.8 wink> AFAICT, the cause of this funny strangeness is that a very respected member of pydev and native speaker of English with an truly unique sense of humor, found early in the discussion that sum(sequence_of_strings) "made his brain hurt"; and while this person is the very opposite of a newbie, his judgment was combined with concerns relating to newbies to result in the current behavior. I believe that in this case native linguistic intuition made the decision... and that a non-native intuition should have reversed it by noting in due time that the cognitive dissonance that native speakers could find to sum(seq_of_str), isn't in fact distinct from the cognitive dissonance that any newbie should find to the use of the + operator to concatenate strings or sequences rather than summing numbers. To restate, as a non-native speaker, and inasmuch "it hurts the brain" is the real reason against sum(seq_of_str) calling ''.join(seq_of_str) directly, I believe this is a case of excess authority being given to native linguistic intuition (of english speaking python experts) with the result of /adding/ to the cognitive dissonance of using + to concatenate strings (for all programming newbies). Best regards to all, Boris Borcic -- "On na?t tous les m?tres du m?me monde" From brett at python.org Fri Jul 7 19:03:40 2006 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jul 2006 10:03:40 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: <44AE11E4.9040809@canterbury.ac.nz> References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> <44AC60D0.1040508@canterbury.ac.nz> <44AE11E4.9040809@canterbury.ac.nz> Message-ID: On 7/7/06, Greg Ewing wrote: > > Brett Cannon wrote: > > On 7/5/06, *Greg Ewing* > > And I would change file() so that it didn't open > > files. Then it would be harmless for code to have > > access to the file class. > > > Right, that is essentially what I proposed initially with the whole > > crippling idea. > > > > What the capabilities supporters are saying is that if we go that route > > we will be constantly finding objects that require similar crippling. > > We've got our wires crossed somewhere. I *am* a capabilities > supporter. But for a capability model to work, the capabilities > need to be isolated and encapsulated in functions or objects > that can be independently provided or not provided. In the > case of file(), that means separating the capability of > being able to open a file from the capability of accessing > an already-opened file. These two things are currently > conflated in the file class. OK, putting that way makes sense for labelling it as capabilities support. It just seems like everyone else who has been calling for more capabilities have not been wanting to change file() in any way. BTW, I object to the term "crippling" in this particular > case. If you have access to open(), there's no need for > file() to be able to do the same thing. And if you don't > have access to open(), it's because someone doesn't want > you to be able to open files. But, from the perspective of what file() can do now, it is losing an ability that it once had and not quite as powerful as it was. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/5f5280f6/attachment.htm From brett at python.org Fri Jul 7 19:04:55 2006 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jul 2006 10:04:55 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: <44AE1256.1000509@canterbury.ac.nz> References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> <44AC60D0.1040508@canterbury.ac.nz> <44AE1256.1000509@canterbury.ac.nz> Message-ID: On 7/7/06, Greg Ewing wrote: > > Another thing I perhaps should point out is that > I'm proposing the separation of open() and file() > for *all* code, not just restricted code. So it's > not a matter of "crippling" file() specially for > restricted code. Well, that's fine with me since I use open() for opening all of my files anyway. But I don't know if people want to make 'file' the only built-in type who lacks a constructor on the type itself. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/50760867/attachment.html From pje at telecommunity.com Fri Jul 7 19:25:43 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Jul 2006 13:25:43 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <2e1434c10607061856t6ef10e92lb8bec4f879cbfcc8@mail.gmail.co m> References: <44AA1FF6.1050501@acm.org> Message-ID: <5.1.1.6.0.20060706235548.01f22388@sparrow.telecommunity.com> At 09:56 PM 7/6/2006 -0400, Kevin Jacobs wrote: >Why not extend the interface to the locals builtin and add a __getitem__ >that returns a proxy to access locals defined in other lexical scopes via >__{get/set/del}attr_: > >def counter(num): > num = 1 > def inc(): > locals[1].num += 1 > return outer.num > return inc > > >Where, for CPython, locals[n] gives access to >NamespaceProxy(sys._getframe(n).f_locals). That doesn't actually work, because sys._getframe(1) will give you inc()'s caller, *not* the frame where it was defined. > In addition to having a relatively pleasing and explicit syntax, this > may be a feasible method for allowing portable introspection into outer > scopes without having to export the whole frame object a la > sys._getframe(n). I strongly suspect that Jython, IronPython, and PyPy > would have little difficulty supporting (and optimizing) this construct. > >Lacking core language support, it is easy to roll an object that does just >what I suggest. Actual implementation is left to a more motivated reader, >of course. While I hesitate to describe anything as "impossible", I will note that an implementation with the syntax you suggest is not likely to be possible without resorting to ctypes or a C module, because the frame doesn't have a reference to the function, only the code object, and it's the function objects that own the closure cells. I think the only way you can reasonably accomplish rebinding in Python right now is to have a rebind(func, var=value) function, e.g.: def counter(num): def inc(): rebind(inc, num=num+1) return num return inc It doesn't allow augmented assignment, of course, and it also creates a circular reference between 'inc' and itself, requiring garbage collection to clean up. Anyway, the actual implementation of such a rebind function would basically be a bit of syntax sugar over this cookbook recipe: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440515 You'd want to just use the keyword argument names to figure out which cells of the function needed changing. And the whole thing would be really slow and complex. You'd probably be better off using function attributes: def counter(num): def inc(): inc.num+=1 return inc.num inc.num = num return inc It's not ideal, but *this* should probably be the pattern we recommend for rebinding, rather than "create a class" or "use an anonymous namespace argument". It can even be prettied up a little bit with a decorator to set the function attributes. def counter(num): @uses(num=num) def inc(): inc.num+=1 return inc.num return inc But that's about as good as it gets, which is nowhere near as good as: def counter(num): def inc(): nonlocal num num+=1 return num return inc On the other hand, the rebind() syntax actually is slightly cleaner in some respects: def counter(num): def inc(): rebind(num = num+1) return num return inc Maybe we should just make rebind() a fast builtin. ;) From talin at acm.org Fri Jul 7 20:17:52 2006 From: talin at acm.org (Talin) Date: Fri, 07 Jul 2006 11:17:52 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> Message-ID: <44AEA550.1030204@acm.org> Brett Cannon wrote: > On 7/6/06, Talin wrote: >> And if we can call it for every operation, then we don't have to spend >> time hunting down all of the possible loopholes and ways in which 'file' >> or other restricted objects might be accessed. > > Not true. You have to set this object restriction flag, right? What > happens if you don't set it on all of the proper classes/types? You end up > in the exact same situation you are with crippling; making sure you cover > your ass with what you flag as unsafe else you risk having something get > passed you. But that's a much simpler problem. With the restricted flag, it isn't just *your* code that is prevented from using 'file' - it's *all* code. Only approved gateways that remove the restriction (by setting the interpreter state) can perform operations on file objects without blowing up. This means that if you call some random library function that attempts to open a file, it won't work, because the random library function is still running in restricted mode. Similarly, if you have a reference to some externally created object that has a reference to a file (or the file class) somewhere in it's inheritance hierarchy, any attempt to access that object will fail. Without this, you would have to chase down every bit of library code that opens file, or has a reference to a file. What I am proposing shares some aspects of both the crippling and the capability model: It's similar to crippling in the sense that you're protecting the object itself, not access to the object. So you avoid the problem of trying to figure out all of the possible ways an object can be accessed. However, where it resembles capabilities is that its an 'all or nothing' approach - that is, you either have access to file, or you don't. Unlike the crippling model where fine-grained access control is implemented by modifying individual methods of the crippled object, in this scheme we cripple the object *entirely*, and then provide fine-grained access control via wrappers. Those wrappers, in turn, act just like capabilities - you can have different wrappers that have different sets of access permissions. So it provides the advantage of the capability approach in that the set of restrictions can be extended or modified by writing new wrappers. Thus, by providing an extremely simple but unbreakable check at the interpreter level, we can then write classes that build on top of that a richer and more sophisticated set of permissions, while still maintaining a strong barrier to unauthorized actions. -- Talin From brett at python.org Fri Jul 7 21:39:43 2006 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jul 2006 12:39:43 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <44AEA550.1030204@acm.org> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> Message-ID: On 7/7/06, Talin wrote: > > Brett Cannon wrote: > > On 7/6/06, Talin wrote: > >> And if we can call it for every operation, then we don't have to spend > >> time hunting down all of the possible loopholes and ways in which > 'file' > >> or other restricted objects might be accessed. > > > > Not true. You have to set this object restriction flag, right? What > > happens if you don't set it on all of the proper classes/types? You end > up > > in the exact same situation you are with crippling; making sure you > cover > > your ass with what you flag as unsafe else you risk having something > get > > passed you. > > But that's a much simpler problem. > > With the restricted flag, it isn't just *your* code that is prevented > from using 'file' - it's *all* code. Only approved gateways that remove > the restriction (by setting the interpreter state) can perform > operations on file objects without blowing up. > > This means that if you call some random library function that attempts > to open a file, it won't work, because the random library function is > still running in restricted mode. Right, but that happens with either approach being proposed anyway. Similarly, if you have a reference to some externally created object > that has a reference to a file (or the file class) somewhere in it's > inheritance hierarchy, any attempt to access that object will fail. > > Without this, you would have to chase down every bit of library code > that opens file, or has a reference to a file. Not true. With the library code running under a sandboxed interpreter the checks by either implementation will still be in effect. And if a proxy is returned by open() instead of 'file' with stuff crippled then the worries alleviated even further. What I am proposing shares some aspects of both the crippling and the > capability model: > > It's similar to crippling in the sense that you're protecting the object > itself, not access to the object. So you avoid the problem of trying to > figure out all of the possible ways an object can be accessed. > > However, where it resembles capabilities is that its an 'all or nothing' > approach - that is, you either have access to file, or you don't. Unlike > the crippling model where fine-grained access control is implemented by > modifying individual methods of the crippled object, in this scheme we > cripple the object *entirely*, and then provide fine-grained access > control via wrappers. Those wrappers, in turn, act just like > capabilities - you can have different wrappers that have different sets > of access permissions. > > So it provides the advantage of the capability approach in that the set > of restrictions can be extended or modified by writing new wrappers. > > Thus, by providing an extremely simple but unbreakable check at the > interpreter level, we can then write classes that build on top of that a > richer and more sophisticated set of permissions, while still > maintaining a strong barrier to unauthorized actions. I guess I am just not seeing where your approach is better than preventing the constructor in 'file' and having open() return the 'file' object or proxy object. With your approach 'file' would be flagged, but with the other you just put the same check in 'file's constructor. With both you would probably also want open() to be a factory function anyway. So I don't see where you gain simplicity or more security. It seems like you are pushing the check into the eval loop, but you still require the flagging of objects as unsafe. Going with the other two proposals you don't burden the eval loop with the check but the objects that you would have flagged in the first place. It just seems like we are pushing around the flagging of unsafe stuff and that doesn't feel like it buys us much. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/698a6016/attachment.htm From guido at python.org Fri Jul 7 22:08:52 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Jul 2006 22:08:52 +0200 Subject: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript) In-Reply-To: References: Message-ID: On 7/7/06, Ka-Ping Yee wrote: > I've been doing a bunch of Firefox extension programming in Javascript > and suddenly a few of the recent topics here came together in my head > in a silent kapow of thoughts. This is kind of a side note to the > security discussion, but they're all interconnected: network > programming, concurrency, lexical scoping, security. Hm... I wonder if this style has become so popular in JS because it's all they have? I find callback-style programming pretty inscrutable pretty soon. > Client-side web scripting tends to have a callback/continuation-ish > concurrency style because it has to deal with network transactions > (which can stall for long periods of time) in a user interface that > is expected to stay always responsive. The Firefox API is full of > listeners/observers, events, and continuation-like things. So one > thing to consider is that, when Python is used for these purposes, > it may be written in a specialized style. > > As i write JavaScript in this style i find i use nested functions > a lot. When i want to set up a callback that uses variables in the > current context, the natural thing to do is to define a new function > in the local namespace. And if that function has to also provide a > callback, then it has another function nested within it and so on. > > function spam() { > var local_A = do_work(); > do_network_transaction( > new function(result_1) { > var local_B = do_work(result_1); > do_network_transaction( > new function(result_2) { > do_work(local_A, local_B, result_1, result_2); > ... > } > ); > } > ); > } How can you ever keep track of when a '}' must be followed by a ';' ? > So it is a possible consequence of embedding Python in Firefox that > people will be using nested functions and lexical scoping in Python > more often, which makes the recent discussion about access to > enclosing scopes more significant. > > This is even related to security as well. Namespaces and lexical > scoping are a natural and visually apparent way to limit access. > If, for example, result_1 and result_2 in the above example are > security-sensitive objects like secret keys or networking functions, > you can see just by inspection that they cannot leak outside of > spam() except by directly being passed in an outgoing function call. > > The standard Pythonic response to nested functions is to translate > them into classes. But the nested function style has two advantages: > > 1. Variables are more appropriately scoped; they exist > only where they are meaningful. (In a class, all the > variables would be mixed into one namespace, where > some of them would be invalid some of the time.) This doesn't strike me as very important. The same can happen in a local scope and doesn 't seem to bother anyone there. > 2. Local variables are private. (Class instances don't > get their own private namespaces to play in.) Hmm... I wouldn't be so sure of that. Exception handlers have access to locals of stack frames above and below the current frame. > The first becomes more significant when in a more continuation-y > style because it helps keep the various continuations from > interfering with each other. The second becomes more significant > if you care about restricting untrusted Python code. I would rather come up with a more Pythonic style of asynchronous event handling, e.g. based on yield as shown by Phillip. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Jul 7 22:18:35 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Jul 2006 22:18:35 +0200 Subject: [Python-Dev] introducing __dir__? In-Reply-To: <44AE4BC1.1000203@gmail.com> References: <1d85506f0607061022v76806e5dt6b59f5771ef3f1e3@mail.gmail.com> <200607061341.56945.fdrake@acm.org> <44AE4BC1.1000203@gmail.com> Message-ID: +1 here too. This could be added easily to Python 2.6. --Guido On 7/7/06, Nick Coghlan wrote: > Fred L. Drake, Jr. wrote: > > On Thursday 06 July 2006 13:22, tomer filiba wrote: > > > my suggestion is simple -- replace this mechanism with a __dir__ - > > > a special method that returns the list of attributes of the object. > > > > > > rationale: > > > * remove deprecated __methods__, etc. > > > * symmetry -- just like hex() calls __hex__, etc. > > > * __methods__ and __members__ are lists rather than callable > > > objects, which means they cannot be updated on-demand > > > > +1 > > +1 here, too. > > It would also allow objects which override __getattribute__ and/or __getattr__ > to make dir() provide a sane answer (or raise an exception to indicate that a > sane answer isn't possible). (This was something that actually came up when > trying to implement a namespace object that *didn't* automatically fall back > to its class namespace for Python level attribute access) > > For backwards compatibility, dir() could still fall back to the current > mechanism if __dir__ isn't found. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > http://www.boredomandlaziness.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Jul 7 22:24:22 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Jul 2006 22:24:22 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: On 7/7/06, Just van Rossum wrote: > Why couldn't at least augmented assignment be implicitly rebinding? Well, personally I'm for allowing full rebinding semantics but only when a 'global' (or 'nonlocal') statement is used first. Making augmented assignment automatically imply 'global' etc. seems too magical to me. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Jul 7 22:43:19 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Jul 2006 22:43:19 +0200 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> Message-ID: On 7/7/06, Brett Cannon wrote: > I guess I am just not seeing where your approach is better than preventing > the constructor in 'file' and having open() return the 'file' object or > proxy object. With your approach 'file' would be flagged, but with the > other you just put the same check in 'file's constructor. With both you > would probably also want open() to be a factory function anyway. So I don't > see where you gain simplicity or more security. It seems like you are > pushing the check into the eval loop, but you still require the flagging of > objects as unsafe. Going with the other two proposals you don't burden the > eval loop with the check but the objects that you would have flagged in the > first place. > > It just seems like we are pushing around the flagging of unsafe stuff and > that doesn't feel like it buys us much. At the risk of repeating someone's point or making no sense (I'm only following this with half an ear) I would like to point out that as long as there's C code involved, we can have the equivalent of private constructors in C++/Java. This means classes that cannot be constructed by calling the class from Python. C code has to use some other API to create an instance, bypassing this check. It seems reasonable to me, even if most popular types *can* be constructed. There are other types that have this property, e.g. list iterators. Try type(iter(list()))(). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Fri Jul 7 22:55:35 2006 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jul 2006 13:55:35 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> Message-ID: On 7/7/06, Guido van Rossum wrote: > > On 7/7/06, Brett Cannon wrote: > > I guess I am just not seeing where your approach is better than > preventing > > the constructor in 'file' and having open() return the 'file' object or > > proxy object. With your approach 'file' would be flagged, but with the > > other you just put the same check in 'file's constructor. With both you > > would probably also want open() to be a factory function anyway. So I > don't > > see where you gain simplicity or more security. It seems like you are > > pushing the check into the eval loop, but you still require the flagging > of > > objects as unsafe. Going with the other two proposals you don't burden > the > > eval loop with the check but the objects that you would have flagged in > the > > first place. > > > > It just seems like we are pushing around the flagging of unsafe stuff > and > > that doesn't feel like it buys us much. > > At the risk of repeating someone's point or making no sense (I'm only > following this with half an ear) I would like to point out that as > long as there's C code involved, we can have the equivalent of private > constructors in C++/Java. This means classes that cannot be > constructed by calling the class from Python. C code has to use some > other API to create an instance, bypassing this check. It seems > reasonable to me, even if most popular types *can* be constructed. > There are other types that have this property, e.g. list iterators. > Try type(iter(list()))(). Good point. C code could circumvent the bit check by doing all of the work behind the scenes without pushing the object on the stack. But if the check is in the C code for the object itself it is much harder to get around. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/fa8a7838/attachment-0001.html From python-dev at zesty.ca Fri Jul 7 23:45:13 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri, 7 Jul 2006 16:45:13 -0500 (CDT) Subject: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript) In-Reply-To: References: Message-ID: On Fri, 7 Jul 2006, Guido van Rossum wrote: > On 7/7/06, Ka-Ping Yee wrote: > > I've been doing a bunch of Firefox extension programming in Javascript > > and suddenly a few of the recent topics here came together in my head > > in a silent kapow of thoughts. This is kind of a side note to the > > security discussion, but they're all interconnected: network > > programming, concurrency, lexical scoping, security. > > Hm... I wonder if this style has become so popular in JS because it's > all they have? I find callback-style programming pretty inscrutable > pretty soon. Yes, i would love to be able to use generators instead. Maybe this will be a strong selling point for Python when it's ready for browser scripting. (There may be a fair amount of glue required to cope with the existing callback-based Firefox APIs though.) > How can you ever keep track of when a '}' must be followed by a ';' ? Hee hee -- dizzying, eh? Actually there are no occurrences of "};" in that example, just ");". In JavaScript the semicolons are optional anyway; my personal style is to use semicolons after statements but not after function declarations. > > 1. Variables are more appropriately scoped; they exist > > only where they are meaningful. (In a class, all the > > variables would be mixed into one namespace, where > > some of them would be invalid some of the time.) > > This doesn't strike me as very important. The same can happen in a > local scope and doesn't seem to bother anyone there. That's true, but then we're talking about existing coding styles where people aren't concerned about minimizing security risks. Scopes are a very powerful tool for limiting access, but you might not notice what you're missing if you haven't been faced with a security problem and tried using them to solve it. (It's a bit like generators before we had them and got used to how handy they were: substitute scopes => generators, limiting access => flow control, security problem => concurrency problem.) > > 2. Local variables are private. (Class instances don't > > get their own private namespaces to play in.) > > Hmm... I wouldn't be so sure of that. Exception handlers have access > to locals of stack frames above and below the current frame. True. It would be more accurate to say that local variables are usually thought of as private, and therefore could be enforced as private without breaking (hardly) anything. Making instance members private would break tons of stuff. -- ?!ng From skip at pobox.com Sat Jul 8 00:10:04 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 7 Jul 2006 17:10:04 -0500 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: <17582.56252.360308.228430@montanaro.dyndns.org> Guido> Well, personally I'm for allowing full rebinding semantics but Guido> only when a 'global' (or 'nonlocal') statement is used Guido> first. Making augmented assignment automatically imply 'global' Guido> etc. seems too magical to me. So, if I understand correctly, in the presence of a global statement search just goes up the lexical chain looking for the first occurrence of the variable to modify? x = 0 def f(): x = 1 def g(): global x x = 2 print x g() print x f() print x Today it prints 2 1 2 You're suggesting it will print 2 2 0 ? Sounds reasonable to me. If we're talking py3k I'd chuck "global" as a keyword though and replace it with something like "outer". Skip From bob at redivi.com Sat Jul 8 00:22:53 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 7 Jul 2006 15:22:53 -0700 Subject: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript) In-Reply-To: References: Message-ID: <03951EDD-187C-4751-B158-4C39C358CC3F@redivi.com> On Jul 7, 2006, at 1:08 PM, Guido van Rossum wrote: > On 7/7/06, Ka-Ping Yee wrote: >> I've been doing a bunch of Firefox extension programming in >> Javascript >> and suddenly a few of the recent topics here came together in my head >> in a silent kapow of thoughts. This is kind of a side note to the >> security discussion, but they're all interconnected: network >> programming, concurrency, lexical scoping, security. > > Hm... I wonder if this style has become so popular in JS because it's > all they have? I find callback-style programming pretty inscrutable > pretty soon. You really don't have any choice without continuations or some built- in concurrency primitive. Callbacks are slightly less painful in JavaScript because you can define them in-line instead of naming it first. >> Client-side web scripting tends to have a callback/continuation-ish >> concurrency style because it has to deal with network transactions >> (which can stall for long periods of time) in a user interface that >> is expected to stay always responsive. The Firefox API is full of >> listeners/observers, events, and continuation-like things. So one >> thing to consider is that, when Python is used for these purposes, >> it may be written in a specialized style. >> >> As i write JavaScript in this style i find i use nested functions >> a lot. When i want to set up a callback that uses variables in the >> current context, the natural thing to do is to define a new function >> in the local namespace. And if that function has to also provide a >> callback, then it has another function nested within it and so on. >> >> function spam() { >> var local_A = do_work(); >> do_network_transaction( >> new function(result_1) { >> var local_B = do_work(result_1); >> do_network_transaction( >> new function(result_2) { >> do_work(local_A, local_B, result_1, >> result_2); >> ... >> } >> ); >> } >> ); >> } > > How can you ever keep track of when a '}' must be followed by a ';' ? "}\n" is the same as "};" as far as the JavaScript spec goes, you can do either or both. -bob From scott+python-dev at scottdial.com Sat Jul 8 00:42:03 2006 From: scott+python-dev at scottdial.com (Scott Dial) Date: Fri, 07 Jul 2006 18:42:03 -0400 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: References: Message-ID: <44AEE33B.2020702@scottdial.com> Neal Norwitz wrote: > The current list of serious bugs are in the PEP: > > http://www.python.org/dev/peps/pep-0356/ > > If there are any bugs you think should be considered show stoppers, > mail them to the list and I will update the PEP. http://www.python.org/sf/1519018 I believe this regression in syntax checking needs to be addressed. -- Scott Dial scott at scottdial.com scodial at indiana.edu From greg.ewing at canterbury.ac.nz Sat Jul 8 01:32:00 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 08 Jul 2006 11:32:00 +1200 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: <44AE4940.5020603@gmail.com> References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> <44AC60D0.1040508@canterbury.ac.nz> <44AE1256.1000509@canterbury.ac.nz> <44AE4940.5020603@gmail.com> Message-ID: <44AEEEF0.5030001@canterbury.ac.nz> Nick Coghlan wrote: > What would the signature of the file constructor be in that case? If it's possible to call it at all, I think it would have to take a file descriptor, or whatever the platform's OS-level representation of an open file is. The other possibility is to just raise an exception if you call file() from Python code. -- Greg From greg.ewing at canterbury.ac.nz Sat Jul 8 01:40:34 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 08 Jul 2006 11:40:34 +1200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> Message-ID: <44AEF0F2.5040007@canterbury.ac.nz> Boris Borcic wrote: > I believe that in this case native linguistic intuition made the decision... The reason has nothing to do with language. Guido didn't want sum() to become an attractive nuisance by *appearing* to be an obvious way of joining a list of strings, while actually being a very inefficient way of doing that. Considerable effort was put into trying to make sum() smart enough to detect when you were using it on a list of strings and do "".join behind the scenes, but Guido decided in the end that it wasn't worth the trouble, given that he only ever intended sum() to be used on numbers in the first place. -- Greg From brett at python.org Sat Jul 8 01:54:37 2006 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jul 2006 16:54:37 -0700 Subject: [Python-Dev] doc for new restricted execution design for Python In-Reply-To: <44AEEEF0.5030001@canterbury.ac.nz> References: <20060705154750.xzn3ft0hq2gw0koc@login.werra.lunarpages.com> <44AC60D0.1040508@canterbury.ac.nz> <44AE1256.1000509@canterbury.ac.nz> <44AE4940.5020603@gmail.com> <44AEEEF0.5030001@canterbury.ac.nz> Message-ID: On 7/7/06, Greg Ewing wrote: > > Nick Coghlan wrote: > > > What would the signature of the file constructor be in that case? > > If it's possible to call it at all, I think it would > have to take a file descriptor, or whatever the > platform's OS-level representation of an open file > is. > > The other possibility is to just raise an exception > if you call file() from Python code. I am going with raising an exception. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/09f05947/attachment.html From ncoghlan at gmail.com Sat Jul 8 03:39:04 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 08 Jul 2006 11:39:04 +1000 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> Message-ID: <44AF0CB8.5010607@gmail.com> Brett Cannon wrote: > Good point. C code could circumvent the bit check by doing all of the > work behind the scenes without pushing the object on the stack. But if > the check is in the C code for the object itself it is much harder to > get around. C code can circumvent the bit check by calling fopen() directly and pushing something onto the stack that isn't even recognised by the interpreter as a file object :) You *have* to trust C code completely before importing it, because it has access to the platform C library and can do whatever the heck it wants. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From brett at python.org Sat Jul 8 04:01:27 2006 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jul 2006 19:01:27 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <44AF0CB8.5010607@gmail.com> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> <44AF0CB8.5010607@gmail.com> Message-ID: On 7/7/06, Nick Coghlan wrote: > > Brett Cannon wrote: > > Good point. C code could circumvent the bit check by doing all of the > > work behind the scenes without pushing the object on the stack. But if > > the check is in the C code for the object itself it is much harder to > > get around. > > C code can circumvent the bit check by calling fopen() directly and > pushing > something onto the stack that isn't even recognised by the interpreter as > a > file object :) Right, but you can take measures to prevent accidental circumvention. You *have* to trust C code completely before importing it, because it has > access to the platform C library and can do whatever the heck it wants. Yep. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/2a96118f/attachment.htm From guido at python.org Sat Jul 8 05:08:40 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Jul 2006 05:08:40 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <17582.56252.360308.228430@montanaro.dyndns.org> References: <17582.56252.360308.228430@montanaro.dyndns.org> Message-ID: On 7/8/06, skip at pobox.com wrote: > > Guido> Well, personally I'm for allowing full rebinding semantics but > Guido> only when a 'global' (or 'nonlocal') statement is used > Guido> first. Making augmented assignment automatically imply 'global' > Guido> etc. seems too magical to me. > > So, if I understand correctly, in the presence of a global statement search > just goes up the lexical chain looking for the first occurrence of the > variable to modify? > > x = 0 > def f(): > x = 1 > def g(): > global x > x = 2 > print x > g() > print x > f() > print x > > Today it prints > > 2 > 1 > 2 > > You're suggesting it will print > > 2 > 2 > 0 > > ? Right. And if the search finds no scope that defines x, it's a compile-time error (that's also new). > Sounds reasonable to me. If we're talking py3k I'd chuck "global" as a > keyword though and replace it with something like "outer". This is still under debate. I don't think we ought to change this in Python 2.x until we've settled on the 3.x syntax and semantics; eventually (in Python 2.9 or so :-) we can backport it with a __future__ statement. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From talin at acm.org Sat Jul 8 05:29:40 2006 From: talin at acm.org (Talin) Date: Fri, 07 Jul 2006 20:29:40 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> Message-ID: <44AF26A4.2040201@acm.org> Brett Cannon wrote: > On 7/7/06, Guido van Rossum wrote: > >> >> On 7/7/06, Brett Cannon wrote: >> > I guess I am just not seeing where your approach is better than >> preventing >> > the constructor in 'file' and having open() return the 'file' object or >> > proxy object. With your approach 'file' would be flagged, but with the >> > other you just put the same check in 'file's constructor. With both >> you >> > would probably also want open() to be a factory function anyway. So I >> don't >> > see where you gain simplicity or more security. It seems like you are >> > pushing the check into the eval loop, but you still require the >> flagging >> of >> > objects as unsafe. Going with the other two proposals you don't burden >> the >> > eval loop with the check but the objects that you would have flagged in >> the >> > first place. >> > >> > It just seems like we are pushing around the flagging of unsafe stuff >> and >> > that doesn't feel like it buys us much. >> >> At the risk of repeating someone's point or making no sense (I'm only >> following this with half an ear) I would like to point out that as >> long as there's C code involved, we can have the equivalent of private >> constructors in C++/Java. This means classes that cannot be >> constructed by calling the class from Python. C code has to use some >> other API to create an instance, bypassing this check. It seems >> reasonable to me, even if most popular types *can* be constructed. >> There are other types that have this property, e.g. list iterators. >> Try type(iter(list()))(). > > > > Good point. C code could circumvent the bit check by doing all of the work > behind the scenes without pushing the object on the stack. But if the > check > is in the C code for the object itself it is much harder to get around. > > -Brett I may be confused (I usually am), but I think you are misinterpreting Guido's point. I think what he is saying is not "you should beware of C code getting around the checks" but rather he is pointing out that C code that gets around the checks can be a useful part of the system - thus the notion of "private constructors", in other words methods for creating an object that are normally inaccessible to Python code. As to your point: I think I am beginning to understand, so let me reiterate to see if I have it right. In the scenario you describe, the open() function is replaced by a function that returns a proxy that has limits on what it is allowed to do. (Ideally, it would return one of several different types of proxies based on the input file path - so a file handle to /tmp would have a different set of restrictions than a file handle to /home/me.) Somewhere inside this proxy is the 'real' file handle. We need to insure that the proxy is air-tight, so that it can't 'leak' to the outside world. (The original motivation for my scheme was the belief that air-tightness couldn't be achieved, however the point that Guido makes above is beginning to make me believe otherwise.) The proxy also has to be able to support all of the methods that the regular file handle can - because we're going to want to pass that proxy over to other subsystems that don't know that they are dealing with a proxy - such as XML parsers or config file readers. Because the permission checks are implemented in the proxy, this means that library code using the proxy has exactly the same access restrictions as the sandboxed code. If you want any library code to be able to have additional privileges beyond what the sandboxed code can do, you'll need to pass them the 'real' file handle, something which can only be done by either the proxy, or by a C 'gateway' function. This is not a problem as long as the library code doesn't attempt to hold on to the file handle (otherwise then the sandboxed code could potentially grab it from the library's objects.) So for any case where a library needs additional privileges, you'll need to add additional methods to the proxy or create the gateway functions to deal with that. So if it truly is the case that the file handle cannot leak into the outside world, then you are correct - there's no need to put a check into the interpreter. My motivation was based on the belief that there would always be a way to get around that, so instead the notion was to 'poison' these protected objects so that even if they did leak out, attempting to use them in a restricted environment would fail. Moreover, there would be no need to even bother preventing such leakage - you wouldn't have to change the behavior of __subclasses__ and so on - which means that the 'restricted' environment would look nearly identical to the normal Python programming environment, i.e. you are free to inspect and use __subclasses__ or any other inspection feature as long as the result doesn't contain any poisoned objects. (BTW, I'm going to dub my idea the 'poisoned object' scheme, just so we have a label.) While I was typing this, I did realize a drawback to poisoned objects, which I will illustrate by the following example: Suppose we want to grant to the sandboxed program permission to read and write cofiguration properties. We don't want to give them arbitrary write access to the file, instead we want to force the sandbox code to only access that file by setting and getting properties. This is an example where a subsystem would require elevated privileges compared to the main program - the config file reader / writer needs to be able to read & write the file as a text stream, but we don't want to allow the sandboxed program to just write arbitrary data to it. The only way to enforce this restriction is to re-wrap the 'real' file handle - in other words, replace the 'file-like object' wrapper with a 'config-like object' wrapper. Merely passing the poisoned file handle to 'config' doesn't work, because 'config' doesn't know how to safely handle it (only the C gateway code can shift the interpreter into a state where poisoned objects can be handled safely.) Passing the file-like proxy to 'config' doesn't work either, because the proxy doesn't allow arbitrary writes. The only thing that you really can do is write another wrapper - which is exactly what you would have to do in the non-poison case. -- Talin From python-dev at zesty.ca Sat Jul 8 06:28:06 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri, 7 Jul 2006 23:28:06 -0500 (CDT) Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <44AF26A4.2040201@acm.org> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> <44AF26A4.2040201@acm.org> Message-ID: On Fri, 7 Jul 2006, Talin wrote: > While I was typing this, I did realize a drawback to poisoned objects, > which I will illustrate by the following example: > > Suppose we want to grant to the sandboxed program permission to read and > write cofiguration properties. We don't want to give them arbitrary > write access to the file, instead we want to force the sandbox code to > only access that file by setting and getting properties. > > This is an example where a subsystem would require elevated privileges > compared to the main program - the config file reader / writer needs to > be able to read & write the file as a text stream, but we don't want to > allow the sandboxed program to just write arbitrary data to it. The situation you're describing here is a classic case of one component keeping a closely held authority while using it to provide some limited capability to some other component. This comes up quite often when you're trying to write secure code. If you want to be able to write that subsystem in Python, then we will need a way to create airtight Python objects (i.e. objects that only leak what they explicitly choose to leak). So this goes back to the big question of goals: Do we want to be able to protect one piece of Python code from another piece of Python code? I'd like the answer to be yes. It sounded for a while like this was not part of Brett's plan, though. Now i'm not so sure. It sounds like you're also interested in having the answer be yes? Let's keep talking about and playing with more examples -- i think they'll help us understand what goals we should aim for and what pitfalls to anticipate before we nail down too many details. -- ?!ng From guido at python.org Sat Jul 8 06:45:08 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Jul 2006 06:45:08 +0200 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> <44AF26A4.2040201@acm.org> Message-ID: On 7/8/06, Ka-Ping Yee wrote: > The situation you're describing here is a classic case of one > component keeping a closely held authority while using it to > provide some limited capability to some other component. This > comes up quite often when you're trying to write secure code. > > If you want to be able to write that subsystem in Python, then > we will need a way to create airtight Python objects (i.e. objects > that only leak what they explicitly choose to leak). > > So this goes back to the big question of goals: > > Do we want to be able to protect one piece of Python code > from another piece of Python code? > > I'd like the answer to be yes. It sounded for a while like this > was not part of Brett's plan, though. Now i'm not so sure. It > sounds like you're also interested in having the answer be yes? > > Let's keep talking about and playing with more examples -- i think > they'll help us understand what goals we should aim for and what > pitfalls to anticipate before we nail down too many details. I'd like the answer to be no, because I don't believe that we can trust the VM to provide sufficient barriers. The old pre-2.2 restricted execution mode tried to do this but 2.2 punched a million holes in it. Python isn't designed for this (it doesn't even enforce private attributes). I guess this is also the main reason I'm skeptical about capabilities for Python. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From shane at hathawaymix.org Sat Jul 1 17:09:25 2006 From: shane at hathawaymix.org (Shane Hathaway) Date: Sat, 01 Jul 2006 09:09:25 -0600 Subject: [Python-Dev] ImportWarning flood In-Reply-To: <200607011758.03756.anthony@interlink.com.au> References: <44A5D5DA.2060601@hathawaymix.org> <200607011758.03756.anthony@interlink.com.au> Message-ID: <44A69025.9030503@hathawaymix.org> Anthony Baxter wrote: > On Saturday 01 July 2006 12:55, Guido van Rossum wrote: >> It's up to the release manager now to decide whether the pitchforks >> at Google or the pitchforks in the larger Python community are >> sharper. ;-) > > At this point, I think removing the warning code is the prudent > course. If someone wanted to find an easy and safe way to make it > only be triggered when the import fails, it could stay in. I created a patch (#1515361) intended to do just that. https://sourceforge.net/tracker/index.php?func=detail&aid=1515361&group_id=5470&atid=305470 Unfortunately, something appears to be misconfigured in my ISP's mail server, and as a result, my messages aren't making it to this list. So you may not have noticed Guido's re-posting of my message announcing the patch. Shane From tomerfiliba at gmail.com Sat Jul 8 18:49:46 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 8 Jul 2006 18:49:46 +0200 Subject: [Python-Dev] exception too expensive? Message-ID: <1d85506f0607080949v1853d38u28c952954afb02d1@mail.gmail.com> i thought avoiding a second dict lookup should be faster, but it turned out to be completely wrong. it's only marginally faster, but if an exception occurs, it's x10 slower. # # the code # >>> import time >>> b = dict((i, 7) for i in range(1000)) >>> def try_lookup(k): ... try: ... return b[k] ... except KeyError: ... return 17 ... >>> def if_lookup(k): ... if k in b: ... return b[k] ... return 17 ... >>> def test(f, k, count=100000): ... t=time.time() ... while count: ... count -=1 ... f(k) ... return time.time()-t ... # # try_lookup with valid key # >>> test(try_lookup, 56, 1000000) 0.6099998950958252 >>> test(try_lookup, 56, 1000000) 0.60899996757507324 >>> test(try_lookup, 56, 1000000) 0.6099998950958252 ~0.61 sec # # if_lookup with valid key # >>> test(if_lookup, 56, 1000000) 0.68799996376037598 >>> test(if_lookup, 56, 1000000) 0.68700003623962402 >>> test(if_lookup, 56, 1000000) 0.67200016975402832 ~0.68 sec # # try_lookup with invalid key # >>> test(try_lookup, 9456, 1000000) 7.062000036239624 >>> test(try_lookup, 9456, 1000000) 6.812000036239624 >>> test(try_lookup, 9456, 1000000) 6.8440001010894775 ~6.90 sec # # if_lookup with invalid key # >>> test(if_lookup, 9456, 1000000) 0.625 >>> test(if_lookup, 9456, 1000000) 0.64100003242492676 >>> test(if_lookup, 9456, 1000000) 0.65599989891052246 ~0.64 sec ====== before someone says "why don't you use dict.get", it's not applicable in my code. i have a cache -- either object already exists in the cache, or i create and store it in the cache -- so get() and setdefault() are not useful. so first of all, i'd recommend not using the try_lookup method; use the if_lookup instead... unless you can be sure the changes a KeyError will be raised are marginal. second, isn't there anything that can be done to improve the performance of exceptions? imo, exceptions should be cheap. i understand it has to do with creating the exception instance everytime, but can't something be done to improve that? for example, hold a cache exception instances and just memcpy them when needed (just a wild idea)? -tomer -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060708/74dcbdd6/attachment.htm From fredrik at pythonware.com Sat Jul 8 18:54:50 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat, 08 Jul 2006 18:54:50 +0200 Subject: [Python-Dev] exception too expensive? In-Reply-To: <1d85506f0607080949v1853d38u28c952954afb02d1@mail.gmail.com> References: <1d85506f0607080949v1853d38u28c952954afb02d1@mail.gmail.com> Message-ID: tomer filiba wrote: > i thought avoiding a second dict lookup should be faster, but it turned out > to be completely wrong. it's only marginally faster, but if an exception > occurs, it's x10 slower. maybe you should take this to comp.lang.python instead... From raymond.hettinger at verizon.net Sat Jul 8 23:37:38 2006 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sat, 08 Jul 2006 21:37:38 -0000 Subject: [Python-Dev] exception too expensive? References: <1d85506f0607080949v1853d38u28c952954afb02d1@mail.gmail.com> Message-ID: <003401c68b43$b4ddeaf0$dc00000a@RaymondLaptop1> > i thought avoiding a second dict lookup should be faster, but it turned out > to be completely wrong. Unless you have an expensive hash function, it is almost never worth it to try to avoid a second lookup. Because of memory cache effects, the second lookup is dirt cheap (a cache miss is about as expensive as a floating point division). See dictnotes.txt for a few thoughts on the subject. Raymond From tim.peters at gmail.com Sun Jul 9 03:31:26 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 8 Jul 2006 21:31:26 -0400 Subject: [Python-Dev] "Missing" 2.5 feature Message-ID: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> Back in: http://mail.python.org/pipermail/python-dev/2005-March/051856.html I made a pitch for adding: sys._current_frames() to 2.5, which would return a dict mapping each thread's id to that thread's current (Python) frame. As noted there, an extension module exists along these lines that's used at least by the popular Zope DeadlockDebugger product, but it's not possible to do it correctly in an extension. The latter is why it needs to be in the core (so long as it's in an extension, it risks segfaulting, because the core's internal `head_mutex` lock isn't exposed). I forgot about this but was recently reminded. How much opposition would there be to sneaking this into 2.5b2? It would consist of adding a relatively simple new function, docs, and tests; since it wouldn't _change_ any existing code, it would have a hard time breaking anything that currently works. From brett at python.org Sun Jul 9 04:44:40 2006 From: brett at python.org (Brett Cannon) Date: Sat, 8 Jul 2006 19:44:40 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <44AF26A4.2040201@acm.org> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AC9BE3.4000301@acm.org> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> <44AF26A4.2040201@acm.org> Message-ID: On 7/7/06, Talin wrote: > > Brett Cannon wrote: > > On 7/7/06, Guido van Rossum wrote: > > > >> > >> On 7/7/06, Brett Cannon wrote: > >> > I guess I am just not seeing where your approach is better than > >> preventing > >> > the constructor in 'file' and having open() return the 'file' object > or > >> > proxy object. With your approach 'file' would be flagged, but with > the > >> > other you just put the same check in 'file's constructor. With both > >> you > >> > would probably also want open() to be a factory function anyway. So > I > >> don't > >> > see where you gain simplicity or more security. It seems like you > are > >> > pushing the check into the eval loop, but you still require the > >> flagging > >> of > >> > objects as unsafe. Going with the other two proposals you don't > burden > >> the > >> > eval loop with the check but the objects that you would have flagged > in > >> the > >> > first place. > >> > > >> > It just seems like we are pushing around the flagging of unsafe stuff > >> and > >> > that doesn't feel like it buys us much. > >> > >> At the risk of repeating someone's point or making no sense (I'm only > >> following this with half an ear) I would like to point out that as > >> long as there's C code involved, we can have the equivalent of private > >> constructors in C++/Java. This means classes that cannot be > >> constructed by calling the class from Python. C code has to use some > >> other API to create an instance, bypassing this check. It seems > >> reasonable to me, even if most popular types *can* be constructed. > >> There are other types that have this property, e.g. list iterators. > >> Try type(iter(list()))(). > > > > > > > > Good point. C code could circumvent the bit check by doing all of the > work > > behind the scenes without pushing the object on the stack. But if the > > check > > is in the C code for the object itself it is much harder to get around. > > > > -Brett > > I may be confused (I usually am), but I think you are misinterpreting > Guido's point. I think what he is saying is not "you should beware of C > code getting around the checks" but rather he is pointing out that C > code that gets around the checks can be a useful part of the system - > thus the notion of "private constructors", in other words methods for > creating an object that are normally inaccessible to Python code. > > As to your point: I think I am beginning to understand, so let me > reiterate to see if I have it right. > > In the scenario you describe, the open() function is replaced by a > function that returns a proxy that has limits on what it is allowed to > do. (Ideally, it would return one of several different types of proxies > based on the input file path - so a file handle to /tmp would have a > different set of restrictions than a file handle to /home/me.) Yep. Somewhere inside this proxy is the 'real' file handle. We need to insure > that the proxy is air-tight, so that it can't 'leak' to the outside > world. (The original motivation for my scheme was the belief that > air-tightness couldn't be achieved, however the point that Guido makes > above is beginning to make me believe otherwise.) The proxy is air-tight by being written in C and holding on to the reference to the file object in the struct of the proxy. The proxy also has to be able to support all of the methods that the > regular file handle can - because we're going to want to pass that proxy > over to other subsystems that don't know that they are dealing with a > proxy - such as XML parsers or config file readers. Because the > permission checks are implemented in the proxy, this means that library > code using the proxy has exactly the same access restrictions as the > sandboxed code. Yep. Or at least the methods that will be needed (e.g., a read-only proxy only needs the read()-like methods). If you want any library code to be able to have additional privileges > beyond what the sandboxed code can do, you'll need to pass them the > 'real' file handle, something which can only be done by either the > proxy, or by a C 'gateway' function. This is not a problem as long as > the library code doesn't attempt to hold on to the file handle > (otherwise then the sandboxed code could potentially grab it from the > library's objects.) So for any case where a library needs additional > privileges, you'll need to add additional methods to the proxy or create > the gateway functions to deal with that. Basically. A library shouldn't need additional abilities. If they do they are not following the security restrictions so it shouldn't work. It should be a rarity that a library really needs to sneak around the security of a proxy to the point of we don't even consider it. Duck typing should be enough for this not to be a problem. So if it truly is the case that the file handle cannot leak into the > outside world, then you are correct - there's no need to put a check > into the interpreter. My motivation was based on the belief that there > would always be a way to get around that, so instead the notion was to > 'poison' these protected objects so that even if they did leak out, > attempting to use them in a restricted environment would fail. At some point we have to trust something. If you control the constructor, you shouldn't have to worry about anything getting out since you won't be able to make anything out of it. Moreover, there would be no need to even bother preventing such leakage > - you wouldn't have to change the behavior of __subclasses__ and so on - > which means that the 'restricted' environment would look nearly > identical to the normal Python programming environment, i.e. you are > free to inspect and use __subclasses__ or any other inspection feature > as long as the result doesn't contain any poisoned objects. Yep. I would like to minimize the impact on Python code and what abilities they have. (BTW, I'm going to dub my idea the 'poisoned object' scheme, just so we > have a label.) > > While I was typing this, I did realize a drawback to poisoned objects, > which I will illustrate by the following example: > > Suppose we want to grant to the sandboxed program permission to read and > write cofiguration properties. We don't want to give them arbitrary > write access to the file, instead we want to force the sandbox code to > only access that file by setting and getting properties. > > This is an example where a subsystem would require elevated privileges > compared to the main program - the config file reader / writer needs to > be able to read & write the file as a text stream, but we don't want to > allow the sandboxed program to just write arbitrary data to it. > > The only way to enforce this restriction is to re-wrap the 'real' file > handle - in other words, replace the 'file-like object' wrapper with a > 'config-like object' wrapper. Yep. But that is just a general issue that you are going to have with any security system. It all comes down to where you put your foot down in terms or restricting access to a resource. Merely passing the poisoned file handle to 'config' doesn't work, > because 'config' doesn't know how to safely handle it (only the C > gateway code can shift the interpreter into a state where poisoned > objects can be handled safely.) > > Passing the file-like proxy to 'config' doesn't work either, because the > proxy doesn't allow arbitrary writes. > > The only thing that you really can do is write another wrapper - which > is exactly what you would have to do in the non-poison case. Yep. -Brett -- Talin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060708/7564ef57/attachment-0001.htm From brett at python.org Sun Jul 9 04:48:38 2006 From: brett at python.org (Brett Cannon) Date: Sat, 8 Jul 2006 19:48:38 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> <44AF26A4.2040201@acm.org> Message-ID: On 7/7/06, Guido van Rossum wrote: > > On 7/8/06, Ka-Ping Yee wrote: > > The situation you're describing here is a classic case of one > > component keeping a closely held authority while using it to > > provide some limited capability to some other component. This > > comes up quite often when you're trying to write secure code. > > > > If you want to be able to write that subsystem in Python, then > > we will need a way to create airtight Python objects (i.e. objects > > that only leak what they explicitly choose to leak). > > > > So this goes back to the big question of goals: > > > > Do we want to be able to protect one piece of Python code > > from another piece of Python code? > > > > I'd like the answer to be yes. It sounded for a while like this > > was not part of Brett's plan, though. Now i'm not so sure. It > > sounds like you're also interested in having the answer be yes? > > > > Let's keep talking about and playing with more examples -- i think > > they'll help us understand what goals we should aim for and what > > pitfalls to anticipate before we nail down too many details. > > I'd like the answer to be no, because I don't believe that we can > trust the VM to provide sufficient barriers. The old pre-2.2 > restricted execution mode tried to do this but 2.2 punched a million > holes in it. Python isn't designed for this (it doesn't even enforce > private attributes). I guess this is also the main reason I'm > skeptical about capabilities for Python. My plan is no. As Guido said, getting this right is feasibly questionable. I do not plan on trying to have security proxies or such implemented in Python code; it will need to be in C. If someone comes along and manages to find a way to make Python work without significantly changing the languages, great, and we can toss out my security implementation for that. But as of right now, I am not planning on making Python code safe to run in Python code. -Brett -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060708/1edabd3c/attachment.html From brett at python.org Sun Jul 9 05:01:32 2006 From: brett at python.org (Brett Cannon) Date: Sat, 8 Jul 2006 20:01:32 -0700 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> Message-ID: On 7/8/06, Tim Peters wrote: > > Back in: > > http://mail.python.org/pipermail/python-dev/2005-March/051856.html > > I made a pitch for adding: > > sys._current_frames() > > to 2.5, which would return a dict mapping each thread's id to that > thread's current (Python) frame. As noted there, an extension module > exists along these lines that's used at least by the popular Zope > DeadlockDebugger product, but it's not possible to do it correctly in > an extension. The latter is why it needs to be in the core (so long > as it's in an extension, it risks segfaulting, because the core's > internal `head_mutex` lock isn't exposed). > > I forgot about this but was recently reminded. How much opposition > would there be to sneaking this into 2.5b2? It would consist of > adding a relatively simple new function, docs, and tests; since it > wouldn't _change_ any existing code, it would have a hard time > breaking anything that currently works. Well, my understanding is that beta is feature freeze, so I am going to say we shouldn't do this. Obviously this can go in day one when 2.6 is started. Anyway, this is a true test of how rigid the rules are! Can Neal and Anthony going to say no to Uncle Timmy? =) -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060708/a8240a7a/attachment.htm From talin at acm.org Sun Jul 9 05:15:18 2006 From: talin at acm.org (Talin) Date: Sat, 08 Jul 2006 20:15:18 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44ADFB4F.5070207@acm.org> <44AEA550.1030204@acm.org> <44AF26A4.2040201@acm.org> Message-ID: <44B074C6.3070701@acm.org> Brett Cannon wrote: > On 7/7/06, Guido van Rossum wrote: >> On 7/8/06, Ka-Ping Yee wrote: >> > I'd like the answer to be yes. It sounded for a while like this >> > was not part of Brett's plan, though. Now i'm not so sure. It >> > sounds like you're also interested in having the answer be yes? >> > >> > Let's keep talking about and playing with more examples -- i think >> > they'll help us understand what goals we should aim for and what >> > pitfalls to anticipate before we nail down too many details. >> >> I'd like the answer to be no, because I don't believe that we can >> trust the VM to provide sufficient barriers. The old pre-2.2 >> restricted execution mode tried to do this but 2.2 punched a million >> holes in it. Python isn't designed for this (it doesn't even enforce >> private attributes). I guess this is also the main reason I'm >> skeptical about capabilities for Python. > > My plan is no. As Guido said, getting this right is feasibly > questionable. I do not plan on trying to have security proxies or such > implemented in Python code; it will need to be in C. If someone comes > along > and manages to find a way to make Python work without significantly > changing > the languages, great, and we can toss out my security implementation for > that. > > But as of right now, I am not planning on making Python code safe to run in > Python code. It might be possible for the code *outside* the sandbox to create new security policies written in Python. Lets start with the concept of a generic "protection" wrapper - its a C proxy object which can wrap around any Python object, and which can restrict access to a specific set of methods. So for example: protected_object = protect(myObject, methods=set('open','close')) 'protect' creates a C proxy which restricts access to the object, allowing only those methods listed to be called. Now, lets create a security policy, written in Python. The policy is essentially a factory which creates wrapped objects: class MyPolicy: # Ask the policy to create a file object def file( path, perms ): if perms == 'r': # Trivial example, a real proxy would be more # sophisticated, and probably configurable. return protect( file( path, perms ), methods=set('open', 'read', 'close') ) raise SecurityException Now, when we create our sandbox, we pass in the policy: sb = Sandbox( MyPolicy() ) The sandbox calls 'protect' on the policy object, preventing it from being inspected or called inappropriately. -- Talin From raymond.hettinger at verizon.net Sun Jul 9 05:16:34 2006 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun, 09 Jul 2006 03:16:34 -0000 Subject: [Python-Dev] "Missing" 2.5 feature References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> Message-ID: <00b601c68b73$15849c30$dc00000a@RaymondLaptop1> > I forgot about this but was recently reminded. How much opposition > would there be to sneaking this into 2.5b2? It would consist of > adding a relatively simple new function, docs, and tests; since it > wouldn't _change_ any existing code, it would have a hard time > breaking anything that currently works. +1 Raymond From g.brandl at gmx.net Sun Jul 9 08:38:42 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 09 Jul 2006 08:38:42 +0200 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> Message-ID: Tim Peters wrote: > Back in: > > http://mail.python.org/pipermail/python-dev/2005-March/051856.html > > I made a pitch for adding: > > sys._current_frames() > > to 2.5, which would return a dict mapping each thread's id to that > thread's current (Python) frame. As noted there, an extension module > exists along these lines that's used at least by the popular Zope > DeadlockDebugger product, but it's not possible to do it correctly in > an extension. The latter is why it needs to be in the core (so long > as it's in an extension, it risks segfaulting, because the core's > internal `head_mutex` lock isn't exposed). > > I forgot about this but was recently reminded. How much opposition > would there be to sneaking this into 2.5b2? It would consist of > adding a relatively simple new function, docs, and tests; since it > wouldn't _change_ any existing code, it would have a hard time > breaking anything that currently works. If you just check it in, using an appropriate log message[1] I think nobody would object. Georg [1] Thinking of "Whitespace normalization"... From anthony at interlink.com.au Sun Jul 9 09:05:14 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Sun, 9 Jul 2006 17:05:14 +1000 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> Message-ID: <200607091705.16027.anthony@interlink.com.au> On Sunday 09 July 2006 11:31, Tim Peters wrote: > Back in: > > > http://mail.python.org/pipermail/python-dev/2005-March/051856.html > > I made a pitch for adding: > > sys._current_frames() > > to 2.5, which would return a dict mapping each thread's id to that > thread's current (Python) frame. As noted there, an extension > module exists along these lines that's used at least by the popular > Zope DeadlockDebugger product, but it's not possible to do it > correctly in an extension. The latter is why it needs to be in the > core (so long as it's in an extension, it risks segfaulting, > because the core's internal `head_mutex` lock isn't exposed). > > I forgot about this but was recently reminded. How much opposition > would there be to sneaking this into 2.5b2? It would consist of > adding a relatively simple new function, docs, and tests; since it > wouldn't _change_ any existing code, it would have a hard time > breaking anything that currently works. Hm. Would it be a smaller change to expose head_mutex so that the external module could use it? I'm really not keen on this seeming tide of new features that seem to be popping up. We're only a few days away from the second and final planned beta - it's getting _awfully_ late to be slotting in new features. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From richard at commonground.com.au Sun Jul 9 09:57:54 2006 From: richard at commonground.com.au (Richard Jones) Date: Sun, 9 Jul 2006 09:57:54 +0200 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <200607091705.16027.anthony@interlink.com.au> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> Message-ID: <0171D5BD-AAA8-4E58-ADE6-C62F033415F7@commonground.com.au> On 09/07/2006, at 9:05 AM, Anthony Baxter wrote: > I'm really not keen on this seeming tide of new features that > seem to be popping up. We're only a few days away from the second and > final planned beta - it's getting _awfully_ late to be slotting in > new features. And besides, one person has already been told that there's absolutely no new features added after the beta freeze, ever. Richard From g.brandl at gmx.net Sun Jul 9 11:24:36 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 09 Jul 2006 11:24:36 +0200 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <0171D5BD-AAA8-4E58-ADE6-C62F033415F7@commonground.com.au> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <0171D5BD-AAA8-4E58-ADE6-C62F033415F7@commonground.com.au> Message-ID: Richard Jones wrote: > On 09/07/2006, at 9:05 AM, Anthony Baxter wrote: >> I'm really not keen on this seeming tide of new features that >> seem to be popping up. We're only a few days away from the second and >> final planned beta - it's getting _awfully_ late to be slotting in >> new features. > > And besides, one person has already been told that there's absolutely > no new features added after the beta freeze, ever. Last time I looked, features approved by the BDFL and/or release manager were OK to go in. Besides, quite a few bugfixes are likely to introduce more "new features" than this ;) Cheers, Georg From mwh at python.net Sun Jul 9 11:56:25 2006 From: mwh at python.net (Michael Hudson) Date: Sun, 09 Jul 2006 10:56:25 +0100 Subject: [Python-Dev] Musings on concurrency and scoping ("replacing" Javascript) In-Reply-To: (Ka-Ping Yee's message of "Thu, 6 Jul 2006 17:17:44 -0500 (CDT)") References: Message-ID: <2modvz3xva.fsf@starship.python.net> Ka-Ping Yee writes: > Client-side web scripting tends to have a callback/continuation-ish > concurrency style because it has to deal with network transactions > (which can stall for long periods of time) in a user interface that > is expected to stay always responsive. The Firefox API is full of > listeners/observers, events, and continuation-like things. So one > thing to consider is that, when Python is used for these purposes, > it may be written in a specialized style. You could even call it a Twisted style :-) Cheers, mwh -- Considering that this thread is completely on-topic in the way only c.l.py threads can be, I think I can say that you should replace "Oblivion" with "Gravity", and increase your Radiohead quotient. -- Ben Wolfson, comp.lang.python From tim.peters at gmail.com Sun Jul 9 11:56:52 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 9 Jul 2006 05:56:52 -0400 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <200607091705.16027.anthony@interlink.com.au> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> Message-ID: <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> [Anthony Baxter] > Hm. Would it be a smaller change to expose head_mutex so that the > external module could use it? No, in part because `head_mutex` may not exist (depends on the build type). What an external module would actually need is 3 new public C API functions, callable workalikes for pystate.c's internal HEAD_{INIT, LOCK, UNLOCK} macros. Then the only clear use for those would be to help implement the proposed new function -- bad tradeoff. > I'm really not keen on this seeming tide of new features that > seem to be popping up. Well, suck it up, cuz that never ends ;-) > We're only a few days away from the second and final planned beta - it's > getting _awfully_ late to be slotting in new features. Yup! OTOH, I'm not proposing to change the behavior of existing code in any way, just adding a new C function comprising about a dozen lines of straightforward code (it can be that simple in the core because it has direct access to the relevant internals then). The biggest risk I see is that I might screw up the LaTeX -- which is a risk. There's no realistic chance that this would suffer platform-specific compiler or runtime problems, or break other code (it only needs read access to the relevant internals, it never writes to them). Of course that doesn't mean you can't say no. It only means you shouldn't :-) From ark at acm.org Sun Jul 9 17:07:56 2006 From: ark at acm.org (Andrew Koenig) Date: Sun, 9 Jul 2006 11:07:56 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <17582.56252.360308.228430@montanaro.dyndns.org> Message-ID: <001d01c6a369$78a59950$6402a8c0@arkdesktop> > So, if I understand correctly, in the presence of a global statement > search > just goes up the lexical chain looking for the first occurrence of the > variable to modify? > > x = 0 > def f(): > x = 1 > def g(): > global x > x = 2 > print x > g() > print x > f() > print x > > Today it prints > > 2 > 1 > 2 > > You're suggesting it will print > > 2 > 2 > 0 > > ? Sounds right to me. > Sounds reasonable to me. If we're talking py3k I'd chuck "global" as a > keyword though and replace it with something like "outer". I must say that I don't like "outer" any more than I like "global." The problem is that in both cases we are selecting the *inner*most definition that isn't in the current scope. From brett at python.org Sun Jul 9 19:52:35 2006 From: brett at python.org (Brett Cannon) Date: Sun, 9 Jul 2006 10:52:35 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <44B074C6.3070701@acm.org> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AEA550.1030204@acm.org> <44AF26A4.2040201@acm.org> <44B074C6.3070701@acm.org> Message-ID: On 7/8/06, Talin wrote: > > Brett Cannon wrote: > > On 7/7/06, Guido van Rossum wrote: > >> On 7/8/06, Ka-Ping Yee wrote: > >> > I'd like the answer to be yes. It sounded for a while like this > >> > was not part of Brett's plan, though. Now i'm not so sure. It > >> > sounds like you're also interested in having the answer be yes? > >> > > >> > Let's keep talking about and playing with more examples -- i think > >> > they'll help us understand what goals we should aim for and what > >> > pitfalls to anticipate before we nail down too many details. > >> > >> I'd like the answer to be no, because I don't believe that we can > >> trust the VM to provide sufficient barriers. The old pre-2.2 > >> restricted execution mode tried to do this but 2.2 punched a million > >> holes in it. Python isn't designed for this (it doesn't even enforce > >> private attributes). I guess this is also the main reason I'm > >> skeptical about capabilities for Python. > > > > My plan is no. As Guido said, getting this right is feasibly > > questionable. I do not plan on trying to have security proxies or such > > implemented in Python code; it will need to be in C. If someone comes > > along > > and manages to find a way to make Python work without significantly > > changing > > the languages, great, and we can toss out my security implementation for > > that. > > > > But as of right now, I am not planning on making Python code safe to run > in > > Python code. > > It might be possible for the code *outside* the sandbox to create new > security policies written in Python. > > Lets start with the concept of a generic "protection" wrapper - its a C > proxy object which can wrap around any Python object, and which can > restrict access to a specific set of methods. So for example: > > protected_object = protect(myObject, methods=set('open','close')) > > 'protect' creates a C proxy which restricts access to the object, > allowing only those methods listed to be called. > > Now, lets create a security policy, written in Python. The policy is > essentially a factory which creates wrapped objects: > > class MyPolicy: > # Ask the policy to create a file object > def file( path, perms ): > if perms == 'r': > # Trivial example, a real proxy would be more > # sophisticated, and probably configurable. > return protect( file( path, perms ), > methods=set('open', 'read', 'close') ) > raise SecurityException > > Now, when we create our sandbox, we pass in the policy: > > sb = Sandbox( MyPolicy() ) > > The sandbox calls 'protect' on the policy object, preventing it from > being inspected or called inappropriately. Using a factory method callback, one could store the PyCodeObject in a C proxy object that just acts as a complete delegate, forwarding all method calls to the internally stored PyCodeObject. That would work. For this initial implementation, though, I am not going to try to support this. We can always add support like this later since it doesn't fundamentally break or change anything that is already planned. Let's focus on getting even more basic stuff working before we start to get too fancy. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060709/c6bbbb68/attachment.html From python-dev at zesty.ca Sun Jul 9 20:50:00 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Sun, 9 Jul 2006 13:50:00 -0500 (CDT) Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <001d01c6a369$78a59950$6402a8c0@arkdesktop> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> Message-ID: On Sun, 9 Jul 2006, Andrew Koenig wrote: > > Sounds reasonable to me. If we're talking py3k I'd chuck "global" as a > > keyword though and replace it with something like "outer". > > I must say that I don't like "outer" any more than I like "global." The > problem is that in both cases we are selecting the *inner*most definition > that isn't in the current scope. That's why "nonlocal" is a better choice in my opinion. -- ?!ng From kbk at shore.net Sun Jul 9 21:55:43 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Sun, 9 Jul 2006 15:55:43 -0400 (EDT) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200607091955.k69Jthl5015458@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 393 open (+15) / 3315 closed (+17) / 3708 total (+32) Bugs : 908 open (+22) / 5975 closed (+49) / 6883 total (+71) RFE : 223 open ( -1) / 229 closed ( +2) / 452 total ( +1) New / Reopened Patches ______________________ test_grp.py doesn't skip special NIS entry, fails (2006-06-22) http://python.org/sf/1510987 opened by Mat Martineau socket.gethostbyaddr fix for machines with incomplete DNS (2006-06-23) http://python.org/sf/1511317 opened by Charles Schwieters Speed up decimal object construction (2006-06-25) CLOSED http://python.org/sf/1511853 opened by Nick Coghlan ast branch changed interactive module name (2006-06-25) http://python.org/sf/1512007 opened by Jp Calderone Improves an error message from setattr (2006-06-26) http://python.org/sf/1512942 opened by Alexander Belopolsky [FIX][Bug 1512695] cPickle.loads seg. faults (2006-06-27) CLOSED http://python.org/sf/1513206 opened by Faik Uygur Remove reduce() (2006-06-27) CLOSED http://python.org/sf/1513249 opened by Collin Winter Clean up usage of map() in the stdlib (2006-06-27) http://python.org/sf/1513299 opened by Collin Winter new turtle module (2006-06-28) http://python.org/sf/1513695 opened by Lingl docs for xturtle.py (2006-06-28) CLOSED http://python.org/sf/1513699 opened by Lingl Move reduce() to functools (2006-06-28) http://python.org/sf/1513870 opened by Collin Winter mailbox (Maildir): avoid losing messages on name clash (2006-06-29) http://python.org/sf/1514543 opened by David Watson mailbox: use fsync() to ensure data is really on disk (2006-06-29) http://python.org/sf/1514544 opened by David Watson proposal for a new circle method (2006-06-29) CLOSED http://python.org/sf/1514737 opened by Lingl 1515163 fix - traceback and str exc (2006-06-30) http://python.org/sf/1515343 opened by Jim Jewett urllib2 redirection fix (2006-07-02) http://python.org/sf/1515745 opened by Petr Gladkikh tarfile.py fix for #1471427 and updates (2006-05-09) http://python.org/sf/1484695 reopened by gustaebel tarfile.py fix for #1471427 and updates (2006-05-09) http://python.org/sf/1484695 reopened by gustaebel Remove deprecated functions from operator (2006-07-03) http://python.org/sf/1516309 opened by Collin Winter Module uuid: reduce pickle footprint (2006-07-03) http://python.org/sf/1516327 opened by Michael Amrhein Module uuid: functions for retrieving MAC addres (2006-07-03) http://python.org/sf/1516330 opened by Michael Amrhein Remove sys.exitfunc (2006-07-03) http://python.org/sf/1516375 opened by Collin Winter OpenVMS patches Modules directory (2006-07-04) http://python.org/sf/1516912 opened by Pi?ronne Jean-Fran?ois Fix for Lib/test/crashers/gc_inspection.py (2006-07-04) http://python.org/sf/1517042 opened by Collin Winter Patch to commands.py to enable callback support (2006-07-05) http://python.org/sf/1517586 opened by Brad Doctor Patch for ctypes to enable custom objects in function calls (2006-07-05) http://python.org/sf/1517790 opened by Thomas Heller Patch for bug #1514451 (2006-07-06) http://python.org/sf/1517891 opened by Thomas Lee assert for NULL in Py_INCREF Py_DECREF (2006-07-06) http://python.org/sf/1517947 opened by Rene Dudfield ext/win-cookbook.html has a broken link to distutils (2006-07-07) http://python.org/sf/1518772 opened by Johannes Gijsbers New ver. of 1102879: Fix for 926423: socket timeouts (2006-07-07) http://python.org/sf/1519025 opened by Tony Nelson turtle.py: correcting begin_fill (2006-07-09) http://python.org/sf/1519566 opened by Lingl Patches Closed ______________ MS Toolkit Compiler no longer available (2006-06-20) http://python.org/sf/1509163 closed by loewis Patch for 1506758 - popen2/subprocess MAXFD MemoryErrors (2006-06-15) http://python.org/sf/1506760 closed by astrand Speed up decimal object construction (2006-06-25) http://python.org/sf/1511853 closed by ncoghlan [FIX][Bug 1512695] cPickle.loads seg. faults (2006-06-27) http://python.org/sf/1513206 closed by gbrandl Remove reduce() (2006-06-27) http://python.org/sf/1513249 closed by collinwinter docs for xturtle.py (2006-06-28) http://python.org/sf/1513699 closed by loewis Make thread stack size runtime tunable (03/20/06) http://python.org/sf/1454481 closed by sf-robot proposal for a new circle method (2006-06-30) http://python.org/sf/1514737 closed by loewis 1515169 fix (ImportWarning flood) (2006-06-30) http://python.org/sf/1515361 closed by nnorwitz Alternative fix for ImportWarning (fix for 1515169) (2006-07-01) http://python.org/sf/1515609 closed by nnorwitz telnetlib timeout fix (bug 822974) (2003-10-17) http://python.org/sf/825417 closed by loewis UUID module for Python (2005-11-29) http://python.org/sf/1368955 closed by gbrandl IDLE L&F on MacOSX (2006-05-19) http://python.org/sf/1491759 closed by ronaldoussoren 'import foo as bar' not checking for 'as' (2006-07-07) http://python.org/sf/1518569 closed by twouters Fix for 926423: socket timeouts + Ctrl-C don't play nice (2005-01-15) http://python.org/sf/1102879 closed by gbrandl New / Reopened Bugs ___________________ Setting category fails with -W switch (2006-06-22) CLOSED http://python.org/sf/1510580 opened by A.M. Kuchling logging fileConfig swallows handler exception (2006-06-18) CLOSED http://python.org/sf/1508253 reopened by tdir method format of logging.Formatter caches incorrectly (2006-06-06) CLOSED http://python.org/sf/1501699 reopened by blorbeer CSV regression in 2.5a1: multi-line cells (2006-04-06) CLOSED http://python.org/sf/1465014 reopened by andrewmcnamara 2.5b1 windows won't install or admit failure (2006-06-22) http://python.org/sf/1510984 opened by Jim Jewett codec_getstreamcodec passes extra None (2006-06-24) CLOSED http://python.org/sf/1511381 opened by Hye-Shik Chang xml.sax.expatreader is missing (2006-06-23) http://python.org/sf/1511497 opened by Wummel Python gettext doesn't support libglade (again) (2006-06-24) CLOSED http://python.org/sf/1511736 opened by Shmyrev Nick vague xref in description of sorted() builtin (2006-06-24) CLOSED http://python.org/sf/1511911 opened by rurpy Can't use sockets in 2.5b1 (2006-06-24) CLOSED http://python.org/sf/1511964 opened by Bryan O'Sullivan Glitches in What's New for beta 1 (2006-06-25) CLOSED http://python.org/sf/1511998 opened by Nick Coghlan OSX: debugger hangs IDLE (2006-06-25) http://python.org/sf/1512124 opened by Aahz mailbox (2.5b1): locking doesn't work (esp. on FreeBSD) (2006-06-25) http://python.org/sf/1512163 opened by David Watson builtin enumerate overflows (2006-06-26) http://python.org/sf/1512504 opened by James Harlow Install on Windows Vista locks up (2006-06-26) http://python.org/sf/1512604 opened by Michael Lucas cPickle.loads() seg.faults when interrupted with keyboard (2006-06-26) CLOSED http://python.org/sf/1512695 opened by Faik Uygur module wave does no rounding (2006-06-26) http://python.org/sf/1512791 opened by Greg Kochanski Incorrect lineno's in code objects (2006-06-26) http://python.org/sf/1512814 opened by Thomas Wouters 'make install' failure on FreeBSD 5.3 (2006-06-26) CLOSED http://python.org/sf/1513032 opened by David Watson socket close() hangs client if used without prior shutdown() (2006-06-27) CLOSED http://python.org/sf/1513223 opened by Irmen de Jong xml.sax.ParseException weirdness in python 2.5b1 (2006-06-27) http://python.org/sf/1513611 opened by Marien Zwart os.access reports incorrect write permissions in Windows (2006-06-27) CLOSED http://python.org/sf/1513646 opened by Yi S. Ding Change the name of a variable causes an exception (2006-06-28) http://python.org/sf/1513802 opened by Arun mimetools message's To field can't be changed (2006-06-28) http://python.org/sf/1513913 opened by Albert Strasheim Typo in signal.alarm() description (2006-06-29) CLOSED http://python.org/sf/1514404 opened by Stephen Shirley Missing module code does spurious file search (2006-06-29) http://python.org/sf/1514420 opened by Nick Maclaren NaN comparison in Decimal broken (2006-06-29) http://python.org/sf/1514428 opened by Nick Maclaren zipfile "append" mode should create a file if not present (2006-06-29) http://python.org/sf/1514451 opened by Ritesh Raj Sarraf String Methods 2.3.6.1 not in TOC (2006-06-29) http://python.org/sf/1514540 opened by Terry J. Reedy radians() doesn't work correctly (2006-06-29) CLOSED http://python.org/sf/1514685 opened by Lingl setup in Pen.__init__() (2006-06-29) CLOSED http://python.org/sf/1514703 opened by Lingl turtle.py: setup() call in Pen.__init__() (2006-06-29) CLOSED http://python.org/sf/1514725 opened by Lingl turtle.py docs out of date (2006-06-29) CLOSED http://python.org/sf/1514730 opened by Lingl site.py can break the location of the python library (2006-06-29) http://python.org/sf/1514734 opened by Mike Meyer sgmllib should recover from unmatched quotes (2006-06-30) http://python.org/sf/1515142 opened by Sam Ruby traceback now masks some string exceptions (2006-06-30) http://python.org/sf/1515163 opened by Jim Jewett version says beta (2006-06-30) http://python.org/sf/1515164 opened by Jim Jewett ImportWarning should be removed (2006-06-30) http://python.org/sf/1515169 opened by James Y Knight wrong handling of character buffers in stringobject (2006-07-01) http://python.org/sf/1515471 opened by Georg Brandl Exponential behavior in regular expression (2006-07-02) CLOSED http://python.org/sf/1515829 opened by Erik Demaine socket timeout inheritance on accept (2006-07-02) http://python.org/sf/1515839 opened by Jari Kirma 2.3.6.4 Mutable Sequence Types clarification (2006-07-02) CLOSED http://python.org/sf/1515932 opened by Alan bdist_msi fails (egg-info) (2006-07-02) CLOSED http://python.org/sf/1515998 opened by Paul Moore Under OS X, compiling against local readline fails (2006-07-02) http://python.org/sf/1516068 opened by Nelson Arzola inspect.py: still infinite recursion inspecting frames (2006-07-03) http://python.org/sf/1516184 opened by Don Quijote Decimal should allow leading or trailing spaces. (2006-07-03) http://python.org/sf/1516613 opened by Ronaldo Francisco Maia warnings.py still refers to OverflowWarning (2006-07-04) CLOSED http://python.org/sf/1516910 opened by John Machin test_logging fails with reference count-checking (2006-07-04) http://python.org/sf/1516995 opened by Collin Winter Getting an error message import site failed -v traceback. (2006-07-05) http://python.org/sf/1517370 opened by brijesh kumar memory leak threading or socketserver module (2006-07-05) http://python.org/sf/1517495 opened by Ids van der Molen filter() implementation does not match docs (2006-07-05) CLOSED http://python.org/sf/1517509 opened by Collin Winter Interpreter crash: filter() + gc.get_referrers() (2006-07-05) http://python.org/sf/1517663 opened by Collin Winter IDLE on macosx keybindings need work (2006-07-06) http://python.org/sf/1517990 opened by Ronald Oussoren IDLE: config-main.def contains windows-specific settings (2006-07-06) http://python.org/sf/1517993 opened by Ronald Oussoren IDLE (macosx): Class and Path browsers show Tk menu (2006-07-06) http://python.org/sf/1517996 opened by Ronald Oussoren c_void_pointer should accept a long pointer > 0x7fffffff (2006-07-06) http://python.org/sf/1518190 opened by Thomas Heller cookielib doku (2006-07-06) CLOSED http://python.org/sf/1518346 opened by Christoph Zwerschke re '\' char interpretation problem (2006-07-06) CLOSED http://python.org/sf/1518406 opened by ollie oldham '\' problem in re.sub (2006-07-06) CLOSED http://python.org/sf/1518453 opened by ilan29 'import foo as bar' not checking for 'as' (2006-07-07) CLOSED http://python.org/sf/1518569 opened by Thomas Wouters Website updating part of PEP101/102 out of date (2006-07-07) http://python.org/sf/1518617 opened by Johannes Gijsbers optparse.parse_args() ret value seems to be a dict but isn't (2006-07-07) http://python.org/sf/1518621 opened by rhunger import_as_name and dotted_as_name fail to validate syntax (2006-07-07) CLOSED http://python.org/sf/1519018 opened by Scott Dial incorrect locale.strcoll() return in Windows (2006-07-07) http://python.org/sf/1519069 opened by Brian Matherly too many files? (2006-07-09) http://python.org/sf/1519452 opened by Joe Brown turtle.py Docs still incomplete (2006-07-09) http://python.org/sf/1519571 opened by Lingl Unmatched Group issue (2006-07-09) http://python.org/sf/1519638 opened by Bobby Xiao Bugs Closed ___________ Setting category fails with -W switch (2006-06-22) http://python.org/sf/1510580 closed by bcannon logging fileConfig swallows handler exception (2006-06-18) http://python.org/sf/1508253 closed by vsajip Logging doesn't report the correct filename or line numbers (2006-04-14) http://python.org/sf/1470422 closed by vsajip method format of logging.Formatter caches incorrectly (06/06/06) http://python.org/sf/1501699 closed by sf-robot method format of logging.Formatter caches incorrectly (2006-06-06) http://python.org/sf/1501699 closed by vsajip CSV regression in 2.5a1: multi-line cells (2006-04-05) http://python.org/sf/1465014 closed by goodger If MAXFD too high, popen2/subprocess produce MemoryErrors (2006-06-15) http://python.org/sf/1506758 closed by astrand IP6 address parsing i sgmllib (2003-12-03) http://python.org/sf/853506 closed by fdrake codec_getstreamcodec passes extra None (2006-06-24) http://python.org/sf/1511381 closed by perky Python gettext doesn't support libglade (again) (2006-06-24) http://python.org/sf/1511736 closed by loewis vague xref in description of sorted() builtin (2006-06-24) http://python.org/sf/1511911 closed by akuchling Can't use sockets in 2.5b1 (2006-06-24) http://python.org/sf/1511964 closed by bos Glitches in What's New for beta 1 (2006-06-25) http://python.org/sf/1511998 closed by akuchling package manager page outdated link (2004-03-24) http://python.org/sf/922690 closed by ronaldoussoren "-framework Python" for building modules is bad (2004-01-29) http://python.org/sf/887242 closed by ronaldoussoren PackageManager does not clean up after itself (2003-11-07) http://python.org/sf/838144 closed by ronaldoussoren PackageManager maintainer missing documentation (2003-07-28) http://python.org/sf/779154 closed by ronaldoussoren IDE preferences cleanup (2003-06-22) http://python.org/sf/758566 closed by ronaldoussoren cPickle.loads() seg.faults when interrupted with keyboard (2006-06-26) http://python.org/sf/1512695 closed by nnorwitz 'make install' failure on FreeBSD 5.3 (2006-06-26) http://python.org/sf/1513032 closed by nnorwitz socket close() hangs client if used without prior shutdown() (2006-06-27) http://python.org/sf/1513223 closed by loewis os.access reports incorrect write permissions in Windows (2006-06-28) http://python.org/sf/1513646 closed by loewis OS/X on i386 framework build (03/16/06) http://python.org/sf/1451717 closed by sf-robot Typo in signal.alarm() description (2006-06-30) http://python.org/sf/1514404 closed by quiver sgmllib should allow angle brackets in quoted values (2006-06-11) http://python.org/sf/1504333 closed by fdrake radians() doesn't work correctly (2006-06-29) http://python.org/sf/1514685 closed by loewis setup in Pen.__init__() (2006-06-29) http://python.org/sf/1514703 closed by loewis turtle.py: setup() call in Pen.__init__() (2006-06-30) http://python.org/sf/1514725 closed by loewis turtle.py docs out of date (2006-06-30) http://python.org/sf/1514730 closed by loewis incorrect LOAD/STORE_GLOBAL generation (2006-06-06) http://python.org/sf/1501934 closed by nascheme Exponential behavior in regular expression (2006-07-02) http://python.org/sf/1515829 closed by tim_one 2.3.6.4 Mutable Sequence Types clarification (2006-07-02) http://python.org/sf/1515932 closed by akuchling bdist_msi fails (egg-info) (2006-07-02) http://python.org/sf/1515998 closed by loewis float/atof have become locale aware (2006-01-29) http://python.org/sf/1417699 closed by loewis Bug in dbm - long strings in keys and values (2003-10-21) http://python.org/sf/827760 closed by loewis Telnet.read_until() timeout parameter misleading (2003-10-13) http://python.org/sf/822974 closed by loewis Python segfaults when freeing "deep" objects (2004-04-01) http://python.org/sf/927248 closed by loewis warnings.py still refers to OverflowWarning (2006-07-04) http://python.org/sf/1516910 closed by gbrandl 2.5 rev 45972 fails build on Mac-Intel (2006-05-11) http://python.org/sf/1487105 closed by nnorwitz filter() implementation does not match docs (2006-07-05) http://python.org/sf/1517509 closed by rhettinger zipfile appends broken in 2.5b1 (2006-07-06) http://python.org/sf/1517887 deleted by krumms sqlite3.dll only installed with Tcl/Tk (2006-07-05) http://python.org/sf/1517388 closed by loewis cookielib doku port cookie-attribute (2006-07-06) http://python.org/sf/1518346 closed by gbrandl re '\' char interpretation problem (2006-07-06) http://python.org/sf/1518406 closed by niemeyer '\' problem in re.sub (2006-07-07) http://python.org/sf/1518453 closed by gbrandl import_as_name and dotted_as_name fail to validate syntax (2006-07-07) http://python.org/sf/1519018 closed by nnorwitz New / Reopened RFE __________________ Add Windows 9x/ME (lack of) support information to README.TX (2006-06-22) http://python.org/sf/1510853 opened by ajaksu RFE Closed __________ Add some dicts to datetime module (2006-06-14) http://python.org/sf/1506340 closed by nnorwitz Add str.partition (2006-01-08) http://python.org/sf/1399936 closed by nnorwitz From nas at arctrix.com Sun Jul 9 22:54:56 2006 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 9 Jul 2006 20:54:56 +0000 (UTC) Subject: [Python-Dev] Unary minus bug Message-ID: The bug was reported by Armin in SF #1333982: the literal -2147483648 (i.e. the value of -sys.maxint-1) gives a long in 2.5, but an int in <= 2.4. I have a fix but I wonder if it's the right thing to do. I suppose returning a long has the chance of breaking someone code. Here's the test we currently have for a related case: [...] # 32-bit machine all_one_bits = '0xffffffff' self.assertEqual(eval(all_one_bits), 4294967295L) self.assertEqual(eval("-" + all_one_bits), -4294967295L) I guess I would add self.assertTrue(isinstance(eval("-2147483648"), int)) to that set of tests. Neil From guido at python.org Sun Jul 9 23:20:07 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 9 Jul 2006 23:20:07 +0200 Subject: [Python-Dev] Unary minus bug In-Reply-To: References: Message-ID: I think it ought to be an int, like before. --Guido On 7/9/06, Neil Schemenauer wrote: > The bug was reported by Armin in SF #1333982: > > the literal -2147483648 (i.e. the value of -sys.maxint-1) gives > a long in 2.5, but an int in <= 2.4. > > I have a fix but I wonder if it's the right thing to do. I suppose > returning a long has the chance of breaking someone code. Here's > the test we currently have for a related case: > > [...] > # 32-bit machine > all_one_bits = '0xffffffff' > self.assertEqual(eval(all_one_bits), 4294967295L) > self.assertEqual(eval("-" + all_one_bits), -4294967295L) > > I guess I would add self.assertTrue(isinstance(eval("-2147483648"), int)) > to that set of tests. > > Neil > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Sun Jul 9 23:54:53 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 9 Jul 2006 17:54:53 -0400 Subject: [Python-Dev] Unary minus bug In-Reply-To: References: Message-ID: <1f7befae0607091454g4945e866ifb36ff6292fa489@mail.gmail.com> [Neil Schemenauer] > The bug was reported by Armin in SF #1333982: > > the literal -2147483648 (i.e. the value of -sys.maxint-1) gives > a long in 2.5, but an int in <= 2.4. That actually depends on how far back you go. It was also a long "at the start". IIRC, Fred or I added hackery to make it an int, because back then there was a sharp distinction between int and long and a surprising number of people complained about this case. Strictly speaking, -2147483648 is not an integer literal (see the grammar -- Python has only positive integer literals). > I have a fix but I wonder if it's the right thing to do. Practicality beats purity here, IMO -- +1. > I suppose returning a long has the chance of breaking someone code. Here's > the test we currently have for a related case: > > [...] > # 32-bit machine > all_one_bits = '0xffffffff' > self.assertEqual(eval(all_one_bits), 4294967295L) > self.assertEqual(eval("-" + all_one_bits), -4294967295L) > > I guess I would add self.assertTrue(isinstance(eval("-2147483648"), int)) > to that set of tests. Or, more obscure but maybe better (since this also tests the similar endcase on sizeof(long) == 8 boxes): minint = -sys.maxint-1 self.assert_(isinstance(minint, int)) self.assert_(isinstance(eval(str(minint)), int)) From nnorwitz at gmail.com Mon Jul 10 00:02:06 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 9 Jul 2006 15:02:06 -0700 Subject: [Python-Dev] Unary minus bug In-Reply-To: References: Message-ID: Do we care about this (after your checkin and with my fix to make 32-63 bit values ints rather than longs): # 64 bit box >>> minint = str(-sys.maxint - 1) >>> minint '-9223372036854775808' >>> eval(minint) -9223372036854775808 >>> eval('-(%s)' % minint[1:]) -9223372036854775808L n -- On 7/9/06, Neil Schemenauer wrote: > The bug was reported by Armin in SF #1333982: > > the literal -2147483648 (i.e. the value of -sys.maxint-1) gives > a long in 2.5, but an int in <= 2.4. > > I have a fix but I wonder if it's the right thing to do. I suppose > returning a long has the chance of breaking someone code. Here's > the test we currently have for a related case: > > [...] > # 32-bit machine > all_one_bits = '0xffffffff' > self.assertEqual(eval(all_one_bits), 4294967295L) > self.assertEqual(eval("-" + all_one_bits), -4294967295L) > > I guess I would add self.assertTrue(isinstance(eval("-2147483648"), int)) > to that set of tests. > > Neil > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From nas at arctrix.com Mon Jul 10 00:18:36 2006 From: nas at arctrix.com (Neil Schemenauer) Date: Sun, 9 Jul 2006 16:18:36 -0600 Subject: [Python-Dev] Unary minus bug In-Reply-To: References: Message-ID: <20060709221836.GA20195@mems-exchange.org> On Sun, Jul 09, 2006 at 03:02:06PM -0700, Neal Norwitz wrote: > Do we care about this (after your checkin and with my fix to make > 32-63 bit values ints rather than longs): > > # 64 bit box > >>>minint = str(-sys.maxint - 1) > >>>minint > '-9223372036854775808' > >>>eval(minint) > -9223372036854775808 > >>>eval('-(%s)' % minint[1:]) > -9223372036854775808L I don't think we care. Python <= 2.4 has the same behavior (i.e. adding parens defeats the constant folding since it's done at the syntax tree level). Where's your fix regarding 32-63 bit ints? I'm not familiar with that issue but my recent check-in appears to have broke the 64-bit buildbots. If no one can explain what's going on there then I guess I will have to find a login on a 64-bit machine at debug it myself. Neil From nnorwitz at gmail.com Mon Jul 10 00:23:26 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 9 Jul 2006 15:23:26 -0700 Subject: [Python-Dev] Unary minus bug In-Reply-To: <20060709221836.GA20195@mems-exchange.org> References: <20060709221836.GA20195@mems-exchange.org> Message-ID: On 7/9/06, Neil Schemenauer wrote: > On Sun, Jul 09, 2006 at 03:02:06PM -0700, Neal Norwitz wrote: > > Do we care about this (after your checkin and with my fix to make > > 32-63 bit values ints rather than longs): > > > > # 64 bit box > > >>>minint = str(-sys.maxint - 1) > > >>>minint > > '-9223372036854775808' > > >>>eval(minint) > > -9223372036854775808 > > >>>eval('-(%s)' % minint[1:]) > > -9223372036854775808L > > I don't think we care. Python <= 2.4 has the same behavior (i.e. > adding parens defeats the constant folding since it's done at the > syntax tree level). Ok, unless someone care's enough to make a patch, I'm removing this from the outstanding issues for 2.5. > Where's your fix regarding 32-63 bit ints? I'm not familiar with I just had to finish the test. It's checked in now and should fix the buildbot problems. Your test exposed the issue, but didn't cause it. It was caused by the optimization of converting strings to ints. My approach to fix it was kinda hacky (just add the proper table for 64-bit machines). I guess ideally we would generate this table based off sizeof(long). Probably should have added a TODO. n From tim.peters at gmail.com Mon Jul 10 00:44:27 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 9 Jul 2006 18:44:27 -0400 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> Message-ID: <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> Just to make life harder ;-), I should note that code, docs and tests for sys._current_frames() are done, on the tim-current_frames branch. All tests pass, and there are no leaks in the new code. It's just a NEWS blurb away from being just another hectic release memory :-) From talin at acm.org Mon Jul 10 01:27:27 2006 From: talin at acm.org (Talin) Date: Sun, 09 Jul 2006 16:27:27 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> Message-ID: <44B190DF.3000805@acm.org> Ka-Ping Yee wrote: > On Sun, 9 Jul 2006, Andrew Koenig wrote: > >>>Sounds reasonable to me. If we're talking py3k I'd chuck "global" as a >>>keyword though and replace it with something like "outer". >> >>I must say that I don't like "outer" any more than I like "global." The >>problem is that in both cases we are selecting the *inner*most definition >>that isn't in the current scope. > > > That's why "nonlocal" is a better choice in my opinion. Some alternatives: use x using x with x -- recycle a keyword? reuse x use extant x share x common x same x borrow x existing x Although, to be perfectly honest, the longer this discussion goes on, the more that I find that I'm not buying Guido's argument about it being better to define this at the point of use rather than at the point of definition. I agree with him that "point of use" is more Pythonic, but I'm also beginning to believe that there are some good reasons why many other languages do it the other way. Part of the reason why its so hard to name this feature is that it's real name is something like "Hey, Python, you know that cool funky thing you do with defining variables in the same scope as they are assigned? Well, don't do that here." -- Talin From talin at acm.org Mon Jul 10 01:51:45 2006 From: talin at acm.org (Talin) Date: Sun, 09 Jul 2006 16:51:45 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44B190DF.3000805@acm.org> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> Message-ID: <44B19691.4050509@acm.org> Talin wrote: > Some alternatives: > > use x > using x > with x -- recycle a keyword? > reuse x > use extant x > share x > common x > same x > borrow x > existing x > > Although, to be perfectly honest, the longer this discussion goes on, > the more that I find that I'm not buying Guido's argument about it being > better to define this at the point of use rather than at the point of > definition. I agree with him that "point of use" is more Pythonic, but > I'm also beginning to believe that there are some good reasons why many > other languages do it the other way. > > Part of the reason why its so hard to name this feature is that it's > real name is something like "Hey, Python, you know that cool funky thing > you do with defining variables in the same scope as they are assigned? > Well, don't do that here." (Followup to my own comment) There are really 3 places where you can indicate that a variable is to be reused instead of redefined: 1) The point of definition in the outer scope, 2) A declaration in the inner scope, and 3) The actual point of assignment. #1 is what I've been pushing for, #2 is what most of the discussion has been about, #3 has been talked about a little bit in the context of an augmented assignment operator. I actually like #3 a little better than #2, but not with a new operator. I'm thinking more along the lines of a keyword that modifies and assignment statement: rebind x = 10 Other possible keywords are: modify, mutate, change, update, change, etc... My gut feeling is that most code that wants to use this feature only wants to use it in a few places. A good example is fcgi.py (implements WSGI for FastCGI), where they use a mutable array to store a flag indicating whether or not the headers have already been sent. -- Talin From guido at python.org Mon Jul 10 05:39:01 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 9 Jul 2006 20:39:01 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44B19691.4050509@acm.org> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> Message-ID: On 7/9/06, Talin wrote: > Talin wrote: > > Some alternatives: > > > > use x > > using x > > with x -- recycle a keyword? > > reuse x > > use extant x > > share x > > common x > > same x > > borrow x > > existing x Of these, I like reuse, share, common slightly better than the rest. > > Although, to be perfectly honest, the longer this discussion goes on, > > the more that I find that I'm not buying Guido's argument about it being > > better to define this at the point of use rather than at the point of > > definition. I agree with him that "point of use" is more Pythonic, but > > I'm also beginning to believe that there are some good reasons why many > > other languages do it the other way. Well, I still don't like point of definition. It means that something far away can change the interpretation of something local. The reason other languages do it differently is that variable declarations are obligatory. It's really unacceptable that when I see def foo(): x = 12 I would have to search the entire module for a possible definition of a global named 'x' to see whether the x assigned to here is local or not. > > Part of the reason why its so hard to name this feature is that it's > > real name is something like "Hey, Python, you know that cool funky thing > > you do with defining variables in the same scope as they are assigned? > > Well, don't do that here." Sorry, that sounds like a B.S. argument. You can translate most language constructs into long sentences if you want to. > (Followup to my own comment) > > There are really 3 places where you can indicate that a variable is to > be reused instead of redefined: 1) The point of definition in the outer > scope, 2) A declaration in the inner scope, and 3) The actual point of > assignment. > > #1 is what I've been pushing for, #2 is what most of the discussion has > been about, #3 has been talked about a little bit in the context of an > augmented assignment operator. > > I actually like #3 a little better than #2, but not with a new operator. > I'm thinking more along the lines of a keyword that modifies and > assignment statement: > > rebind x = 10 > > Other possible keywords are: modify, mutate, change, update, change, etc... A problem with this is the ambiguity if some assignments to x use the rebind keyword and others don't. Then there's a local x and also a nonlocal x. When x is used in an expression (even on the RHS of a rebind statement!) it would be interpreted as the local one. > My gut feeling is that most code that wants to use this feature only > wants to use it in a few places. A good example is fcgi.py (implements > WSGI for FastCGI), where they use a mutable array to store a flag > indicating whether or not the headers have already been sent. Then let's allow nonlocal x = 12 as a shortcut for nonlocal x x = 12 Bash users should be familiar with this from e.g. export FOO=bar. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nnorwitz at gmail.com Mon Jul 10 05:45:05 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 9 Jul 2006 20:45:05 -0700 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> Message-ID: On 7/9/06, Tim Peters wrote: > Just to make life harder ;-), I should note that code, docs and tests > for sys._current_frames() are done, on the tim-current_frames branch. > All tests pass, and there are no leaks in the new code. It's just a > NEWS blurb away from being just another hectic release memory :-) There hasn't been much positive response (in the original thread or here). Given you forgot about it for over a year, how important can it be? :-) What's the justifcation to add this feature, but none of the others? Can we defer this to 2.6? n From nnorwitz at gmail.com Mon Jul 10 06:00:49 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 9 Jul 2006 21:00:49 -0700 Subject: [Python-Dev] xml issue in 2.5 Message-ID: http://python.org/sf/1513611 xml.sax.ParseException weirdness in python 2.5b1. The following code doesn't work: from xml.sax import make_parser, SAXParseException parser = make_parser() try: parser.parse(StringIO('invalid')) except SAXParseException: print 'caught it!' Any comments? n From martin at v.loewis.de Mon Jul 10 08:09:22 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 10 Jul 2006 08:09:22 +0200 Subject: [Python-Dev] xml issue in 2.5 In-Reply-To: References: Message-ID: <44B1EF12.3010603@v.loewis.de> Neal Norwitz wrote: > http://python.org/sf/1513611 > xml.sax.ParseException weirdness in python 2.5b1. The following code > doesn't work: > > from xml.sax import make_parser, SAXParseException > > parser = make_parser() > try: > parser.parse(StringIO('invalid')) > except SAXParseException: > print 'caught it!' > > Any comments? The problem can be simplified to this: py> import xml.sax,xmlcore.sax,sys py> sys.modules['xml.sax'] is sys.modules['xmlcore.sax'] False One way to fix this would be to never refer to "xmlcore" explicitly (i.e. import from xml.sax._exceptions in expatreader), but I guess that would defeat the purpose of the xmlcore renaming. Regards, Martin From anthony at interlink.com.au Mon Jul 10 09:32:26 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon, 10 Jul 2006 17:32:26 +1000 Subject: [Python-Dev] TRUNK FREEZE for 2.5b2, tomorrow Tuesday 11th, 00:00 UTC Message-ID: <200607101732.28316.anthony@interlink.com.au> The trunk is frozen from 00:00 UTC Tuesday the 11th of July. This is in about 16 hours or so time. As usual, unless you're a member of the release team, please don't checkin past that time until I send an email that the release is done. Thanks, Anthony -- Anthony Baxter It's never too late to have a happy childhood. From scott+python-dev at scottdial.com Mon Jul 10 11:25:33 2006 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 10 Jul 2006 05:25:33 -0400 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> Message-ID: <44B21D0D.5090603@scottdial.com> Tim Peters wrote: > Just to make life harder ;-), I should note that code, docs and tests > for sys._current_frames() are done, on the tim-current_frames branch. > All tests pass, and there are no leaks in the new code. It's just a > NEWS blurb away from being just another hectic release memory :-) Wouldn't this function be better named sys._getframes since we already have a sys._getframe for getting the current frame? -- Scott Dial scott at scottdial.com scodial at indiana.edu From 2006a at usenet.alexanderweb.de Mon Jul 10 13:11:42 2006 From: 2006a at usenet.alexanderweb.de (Alexander Schremmer) Date: Mon, 10 Jul 2006 13:11:42 +0200 Subject: [Python-Dev] "Missing" 2.5 feature References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> Message-ID: On Sun, 9 Jul 2006 20:45:05 -0700, Neal Norwitz wrote: > There hasn't been much positive response (in the original thread or > here). Given you forgot about it for over a year, how important can > it be? :-) For me it would be very important because I often wonder where the threads are currently working in my multithreaded apps. As a partial solution, I wrote a "thread monitor" that tracks all thread's frames using a trace function and generates tracebacks on demand. This slows down program execution of course. With that function, it would be much simpler to see the current state of the program (which is trivial in environments like the JVM for example). Kind regards, Alexander From skip at pobox.com Mon Jul 10 14:49:39 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 10 Jul 2006 07:49:39 -0500 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> Message-ID: <17586.19683.920761.293494@montanaro.dyndns.org> >> > Part of the reason why its so hard to name this feature is that >> > it's real name is something like "Hey, Python, you know that cool >> > funky thing you do with defining variables in the same scope as >> > they are assigned? Well, don't do that here." Guido> Sorry, that sounds like a B.S. argument. You can translate most Guido> language constructs into long sentences if you want to. I think Talin's got a point though. It seems hard to find one short English word that captures the essence of the desired behavior. None of the words in his list seem strongly suggestive of the meaning to me. I suspect that means one's ultimately as good (or as bad) as the rest. how-about-"lookout"?-ly, y'rs, Skip From mattjfleming at googlemail.com Mon Jul 10 14:50:22 2006 From: mattjfleming at googlemail.com (Matt Fleming) Date: Mon, 10 Jul 2006 13:50:22 +0100 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> Message-ID: <5ff4a1e50607100550j6280929dpd27f18676945125a@mail.gmail.com> On 10/07/06, Alexander Schremmer <2006a at usenet.alexanderweb.de> wrote: > On Sun, 9 Jul 2006 20:45:05 -0700, Neal Norwitz wrote: > > > There hasn't been much positive response (in the original thread or > > here). Given you forgot about it for over a year, how important can > > it be? :-) > I'm the SoC student working on the improvements for pdb, one of the improvements is allowing thread debugging. I was in fact, going to use the threadframe module if it was available on the system, having this method in the Python core is an even better solution. cheering-for-tim-ly yr's, Matt -- http://mattssanctuary.blogspot.com From arigo at tunes.org Mon Jul 10 17:13:53 2006 From: arigo at tunes.org (Armin Rigo) Date: Mon, 10 Jul 2006 17:13:53 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: References: Message-ID: <20060710151353.GA30654@code0.codespeak.net> Hi, On Tue, Jul 04, 2006 at 04:49:13PM -0700, Neal Norwitz wrote: > On 7/4/06, Guido van Rossum wrote: > > > > From actual users of > > the language I get more complaints about the breakneck speed of > > Python's evolution than about the brokenness of the current language. I'd like to report another (subjective) experience in favor of the "Python is complete enough already" camp, from last year's EuroPython, during Guido's keynote. He announced he accepted two of the major 2.5 PEPs: the 'yield' extension, and I think the 'with' statement. This didn't draw much applause. It certainly gave me the impression that many changes in Python are advocated and welcomed by only a small fraction of users. I cannot be objective here, though, being myself firmly of the impression that there are only so many syntactic features you can put in a language before it stops being elegant and starts promoting obscure code... > PS. One thing I tend to talk to users about is stability of the > interpreter. When I talk about crashing the interpreter, the most > common first reaction I get is "you can crash the interpreter? How do > you do that?" I take that answer as a good sign. :-) Indeed :-) Getting some more python-dev discussions about Lib/test/crashers/*.py would be nice too, though. A bientot, Armin From theller at python.net Mon Jul 10 18:07:50 2006 From: theller at python.net (Thomas Heller) Date: Mon, 10 Jul 2006 18:07:50 +0200 Subject: [Python-Dev] Fix for Lib/test/leakers/test_gestalt.py Message-ID: I think this fixes the leak in Lib/test/leakers/test_gestalt.py. Index: Python/mactoolboxglue.c =================================================================== --- Python/mactoolboxglue.c (revision 50522) +++ Python/mactoolboxglue.c (working copy) @@ -60,8 +60,9 @@ strncpy(buf, input, sizeof(buf) - 1); buf[sizeof(buf) - 1] = '\0'; } + Py_DECREF(rv); } - + Py_XDECREF(m); return buf; } I'm less sure about this one, but it looks like the DECREF is also required: Index: Python/mactoolboxglue.c =================================================================== --- Python/mactoolboxglue.c (revision 50522) +++ Python/mactoolboxglue.c (working copy) @@ -60,8 +60,9 @@ strncpy(buf, input, sizeof(buf) - 1); buf[sizeof(buf) - 1] = '\0'; } + Py_DECREF(rv); } - + Py_XDECREF(m); return buf; } Thomas From tim.peters at gmail.com Mon Jul 10 18:30:30 2006 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 10 Jul 2006 12:30:30 -0400 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> Message-ID: <1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com> [Neal Norwitz] > There hasn't been much positive response (in the original thread or > here). Do note that there was little response of any kind, but all it got was positive. It's not sexy, but is essential for debugging deadlocks. If you ask for positive response, you'll get some -- the use is obvious for those who need it. I didn't (and don't) think this needs a "sales job". > Given you forgot about it for over a year, how important can it be? :-) Important enough that I stopped beating my wife to write the code ;-) > What's the justifcation to add this feature, but none of the others? Don't know which others you have in mind. This is very simple code, changes no existing behaviors, and has no potential to break other code. There is an extension module that tries to do the same, but can't do so wholly safely. It needs to be in the core to be done correctly. The new code, docs and tests are 100% ready to go. How do the others you have in mind stack up against those, point by point? > Can we defer this to 2.6? Of course, although waiting another ~18 months given the above seems needlessly wasteful. From brett at python.org Mon Jul 10 20:17:54 2006 From: brett at python.org (Brett Cannon) Date: Mon, 10 Jul 2006 11:17:54 -0700 Subject: [Python-Dev] Discussion on Lib/test/crashers/ Message-ID: As I am sure some have noticed, as part of my dissertation I have been trying to fix the various crashers. I currently have some patches in SF for some of the crashers. The other ones Armin and I have been talking, while others I have not started yet. Review for the patches or help with fixing the ones I have not gotten to would be great. http://www.python.org/sf/1202533 has a potential fix for the infinite_rec_*.py crashers. PyObject_Call() basically needs a recursion check. But there were some issues with PyErr_NormalizeException() blowing up while trying to normalize the exception, that also needs to watch out for hitting the recursion limit and returning a RuntimeError that does not need normalization itself. http://www.python.org/sf/1377858 has a patch for weakref_in_del.py . Turns out if you create a new weakref in a __del__ for self the weakref is never notified that self is gone and it no longer has a valid weakref to anything. Added a check after __del__ is called to make sure that if any weakrefs were created that they get cleared. http://www.python.org/sf/1303614 is a discussion currently between Armin and I on how to handle del___dict__.py . Looks like more careful reference counting is needed, but Armin is worried about a performance hit. There is also a patch there to fix loosing_dict_ref.py . dangerous_subclassing.py, modify_dict_attr.py, and nasty_eq_vs_dict.py I have not gotten to yet. I am ignoring bogus_code_obj.py, gc_inspection.py, and recursive_call.py since they either require setting the recursion depth higher or involve an extension module. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/496b5c1f/attachment.htm From tim.peters at gmail.com Mon Jul 10 20:23:11 2006 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 10 Jul 2006 14:23:11 -0400 Subject: [Python-Dev] Add new PyErr_WarnEx() to 2.5? In-Reply-To: <1f7befae0605310742y7ac1f7b3i6e9c2e5cc20affbe@mail.gmail.com> References: <1f7befae0605310742y7ac1f7b3i6e9c2e5cc20affbe@mail.gmail.com> Message-ID: <1f7befae0607101123x539c3b5fnf35f15ce042af51b@mail.gmail.com> As noted in http://mail.python.org/pipermail/python-dev/2006-May/065478.html it looks like we need a new Python C API function to make new warnings from the struct module non-useless. For example, runnning test_zipfile on Windows now yields """ test_zipfile C:\Code\python\lib\struct.py:63: DeprecationWarning: struct integer overflow masking is deprecated return o.pack(*args) """ This is useless because the message points _into_ struct.py, not to the code calling struct.pack(). Consequently I have no idea where the offending pack() call is, and neither do you ;-) The current PyErr_Warn() doesn't allow for passing a non-zero `stacklevel` argument to warnings.warn(), so there's little the C code in _struct.c can do to make the warning more useful. From rhettinger at ewtllc.com Mon Jul 10 22:11:10 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Mon, 10 Jul 2006 13:11:10 -0700 Subject: [Python-Dev] Discussion on Lib/test/crashers/ In-Reply-To: References: Message-ID: <44B2B45E.8030209@ewtllc.com> Brett Cannon wrote: > As I am sure some have noticed, as part of my dissertation I have been > trying to fix the various crashers. Nice project. One quick thought: Any crasher that relies on gc.get_referrers() should not be considered a bug. The codebase should not be convoluted, complexified, obfucated, altered, touched, or slowed down just prevent perverse, sadistic safe-cracking using this tool. The docs are clear on this subject: "Care must be taken when using objeccts returned by get_referrers() becasue some of them could still be under construction and hence in a temporarily invalid state. Avoid using get_referrers() for any purpose other than debugging." Raymond From brett at python.org Mon Jul 10 22:15:30 2006 From: brett at python.org (Brett Cannon) Date: Mon, 10 Jul 2006 13:15:30 -0700 Subject: [Python-Dev] Discussion on Lib/test/crashers/ In-Reply-To: <44B2B45E.8030209@ewtllc.com> References: <44B2B45E.8030209@ewtllc.com> Message-ID: On 7/10/06, Raymond Hettinger wrote: > > Brett Cannon wrote: > > > As I am sure some have noticed, as part of my dissertation I have been > > trying to fix the various crashers. > > Nice project. > > One quick thought: Any crasher that relies on gc.get_referrers() should > not be considered a bug. Right. That is on the list of crashers I am ignoring. The codebase should not be convoluted, > complexified, obfucated, altered, touched, or slowed down just prevent > perverse, sadistic safe-cracking using this tool. The docs are clear on > this subject: > > "Care must be taken when using objeccts returned by get_referrers() > becasue some of them could still be under construction and hence in a > temporarily invalid state. Avoid using get_referrers() for any purpose > other than debugging." And I am trying to make sure all the fixes are not superfluous but fix actual issues in the code and behaviour of Python. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/b76f248c/attachment.htm From rhettinger at ewtllc.com Mon Jul 10 22:16:06 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Mon, 10 Jul 2006 13:16:06 -0700 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> <1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com> Message-ID: <44B2B586.1030102@ewtllc.com> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/2cf7e4a5/attachment.htm From guido at python.org Mon Jul 10 22:20:38 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Jul 2006 13:20:38 -0700 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <44B2B586.1030102@ewtllc.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> <1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com> <44B2B586.1030102@ewtllc.com> Message-ID: +1 On 7/10/06, Raymond Hettinger wrote: > > Tim Peters wrote: > [Neal Norwitz] > > > There hasn't been much positive response (in the original thread or > here). > > Do note that there was little response of any kind, but all it got was > positive. It's not sexy, but is essential for debugging deadlocks. > If you ask for positive response, you'll get some -- the use is > obvious for those who need it. I didn't (and don't) think this needs > a "sales job". > > > > FWIW, I think this patch should go in. The benefits are obvious and real. > But, the imagined costs of a new feature during beta are illusory. > > In this case, practicality beats pedantry. For the users, it is a net win > if this goes in. > > > Raymond > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From python-dev at zesty.ca Mon Jul 10 22:34:55 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Mon, 10 Jul 2006 15:34:55 -0500 (CDT) Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <17586.19683.920761.293494@montanaro.dyndns.org> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: On Mon, 10 Jul 2006 skip at pobox.com wrote: > I think Talin's got a point though. It seems hard to find one short English > word that captures the essence of the desired behavior. None of the words > in his list seem strongly suggestive of the meaning to me. I suspect that > means one's ultimately as good (or as bad) as the rest. What's wrong with "nonlocal"? I don't think i've seen an argument against that one so far (from Talin or others). -- ?!ng From amk at amk.ca Mon Jul 10 22:48:59 2006 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 10 Jul 2006 16:48:59 -0400 Subject: [Python-Dev] User's complaints In-Reply-To: <20060710151353.GA30654@code0.codespeak.net> References: <20060710151353.GA30654@code0.codespeak.net> Message-ID: <20060710204859.GA11451@rogue.amk.ca> On Mon, Jul 10, 2006 at 05:13:53PM +0200, Armin Rigo wrote: > didn't draw much applause. It certainly gave me the impression that > many changes in Python are advocated and welcomed by only a small > fraction of users. The benefits of changes are usually clear, but I don't think the costs of changes are fully assessed. python-dev considers the cost of changes to CPython's implementation, but I don't think the cost changes to Jython, IronPython, or PyPy are taken into account here. PyPy probably has enough representatives here who would squawk if something was difficult, but I don't think Jython or IronPython do. I think if we assessed those costs fully, the bar for changes to the language would be a good deal higher. --amk From jeremy at alum.mit.edu Mon Jul 10 22:43:58 2006 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon, 10 Jul 2006 16:43:58 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: On 7/10/06, Ka-Ping Yee wrote: > On Mon, 10 Jul 2006 skip at pobox.com wrote: > > I think Talin's got a point though. It seems hard to find one short English > > word that captures the essence of the desired behavior. None of the words > > in his list seem strongly suggestive of the meaning to me. I suspect that > > means one's ultimately as good (or as bad) as the rest. > > What's wrong with "nonlocal"? I don't think i've seen an argument > against that one so far (from Talin or others). It's a made-up word. You won't find it in the dictionary and the google define: query sends me to a wikipedia page about quantum mechanics. It also expresses itself in the negative form "not local" as opposed to the positive form like global "this is a global." Finally, I think it sounds yucky. To express this email in the positive form: 1. Reserved words should be real words. 2. The meaning of the word should be clear. 3. "Put statements in positive form." (Strunk & White) 4. The word should sound good. global meets all of these requirements. "free" was the word I remember preferring from earlier discussions, but I think it fails #2. (Too much confusion about freeing memory, for example.) Jeremy > > > -- ?!ng > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > From anthony at interlink.com.au Mon Jul 10 22:45:02 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue, 11 Jul 2006 06:45:02 +1000 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <44B2B586.1030102@ewtllc.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com> <44B2B586.1030102@ewtllc.com> Message-ID: <200607110645.07875.anthony@interlink.com.au> On Tuesday 11 July 2006 06:16, Raymond Hettinger wrote: > FWIW, I think this patch should go in. The benefits are > obvious and real. Yep. I'm going to check it in, unless someone else beats me to it in the next couple of hours before the b2 freeze. > But, the imagined costs of a new feature > during beta are illusory. This, I cannot agree with. The costs and risks of just continuing to add new features all through the release process are high. The chances of us releasing a broken Python increases dramatically. Then we have to do emergency bugfix releases. And I'm sorry, but the release process is not something that's zero work. Sure, it's zero work _for_ _you_ , but not for Martin, Fred and myself. > In this case, practicality beats pedantry. I don't think trying to produce the most stable and bugfree Python possible could in _anyway_ be considered "pedantry", and it makes me quite grumpy to have it described in that way. > For the users, it > is a net win if this goes in. In the case of this feature, that's true. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From mbk.lists at gmail.com Mon Jul 10 22:51:58 2006 From: mbk.lists at gmail.com (Mike Krell) Date: Mon, 10 Jul 2006 13:51:58 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: > What's wrong with "nonlocal"? I don't think i've seen an argument > against that one so far (from Talin or others). It sounds a bit awkward to me. Also, it would be nice if the keyword indicated which scope was operative. If I've followed the discussions correctly, I think the parent scope would be operative, so I humbly suggest "parent". Mike From tim.peters at gmail.com Mon Jul 10 22:52:30 2006 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 10 Jul 2006 16:52:30 -0400 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <200607110645.07875.anthony@interlink.com.au> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com> <44B2B586.1030102@ewtllc.com> <200607110645.07875.anthony@interlink.com.au> Message-ID: <1f7befae0607101352j6ff60371m988dae24266740@mail.gmail.com> [Raymond] >> FWIW, I think this patch should go in. The benefits are >> obvious and real. [Anthony Baxter] > Yep. I'm going to check it in, unless someone else beats me to it in > the next couple of hours before the b2 freeze. I'll merge it from my branch right after I send this email. It still needs a NEWS blurb, which I'll write. Thanks! You made the right decision :-) >> But, the imagined costs of a new feature during beta are illusory. > This, I cannot agree with. The costs and risks of just continuing to > add new features all through the release process are high. The > chances of us releasing a broken Python increases dramatically. Then > we have to do emergency bugfix releases. And I'm sorry, but the > release process is not something that's zero work. Sure, it's zero > work _for_ _you_ , but not for Martin, Fred and myself. I agree. For example, even here (i.e., a pure, simple "new feature"), as I noted from the start, it's possible that my doc changes broke the LaTeX somehow, and that would be disruptive. >> In this case, practicality beats pedantry. > I don't think trying to produce the most stable and bugfree Python > possible could in _anyway_ be considered "pedantry", and it makes me > quite grumpy to have it described in that way. He meant that "no new features", while a useful guideline, can be counterproductive if followed slavishly. >> For the users, it is a net win if this goes in. > In the case of this feature, that's true. More to the point, at this stage in the release process this specific little new feature carries almost no risk of loss for anyone. From anthony at interlink.com.au Mon Jul 10 23:02:02 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue, 11 Jul 2006 07:02:02 +1000 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <1f7befae0607101352j6ff60371m988dae24266740@mail.gmail.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607110645.07875.anthony@interlink.com.au> <1f7befae0607101352j6ff60371m988dae24266740@mail.gmail.com> Message-ID: <200607110702.04021.anthony@interlink.com.au> On Tuesday 11 July 2006 06:52, Tim Peters wrote: > > I don't think trying to produce the most stable and bugfree > > Python possible could in _anyway_ be considered "pedantry", and > > it makes me quite grumpy to have it described in that way. > > He meant that "no new features", while a useful guideline, can be > counterproductive if followed slavishly. I'm not taking a slavish "no new features" line. I _am_ saying that any new features, post beta, require a good justification and a clear understanding of the risks that are added by the new code. In this case, the tradeoff is fine. Simply saying code is very low risk isn't enough - there also has to be a positive reason for the code going in. The ability to debug deadlocks is a good thing, and the clincher (once I sat and thought about it a bit) is that there is _already_ a module out there that attempts to do this, albeit in a buggy fashion. This is pretty clear indication that there is a demand for the feature. Similarly, the PyErr_WarnEx() is _probably_ a good thing to add in, because otherwise we can't do anything about the struct warning. But that really will have to wait until post-beta2 at this point. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From brett at python.org Mon Jul 10 23:10:07 2006 From: brett at python.org (Brett Cannon) Date: Mon, 10 Jul 2006 14:10:07 -0700 Subject: [Python-Dev] Klocwork analysis of source if we want it Message-ID: http://www.klocwork.com/company/releases/06_26_06.asp Looks like Klocowork is doing the same thing as Coverity and providing free static analysis of source for open source projects. Doubt we want this *and* Coverity, but figured wouldn't hurt to let people know about it. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/7d11306d/attachment.html From rhettinger at ewtllc.com Mon Jul 10 23:12:32 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Mon, 10 Jul 2006 14:12:32 -0700 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <200607110645.07875.anthony@interlink.com.au> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <1f7befae0607100930m10fb4ee3xc20d1bf9d410667c@mail.gmail.com> <44B2B586.1030102@ewtllc.com> <200607110645.07875.anthony@interlink.com.au> Message-ID: <44B2C2C0.3020305@ewtllc.com> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/e8ba99fd/attachment.htm From theller at python.net Mon Jul 10 23:51:28 2006 From: theller at python.net (Thomas Heller) Date: Mon, 10 Jul 2006 23:51:28 +0200 Subject: [Python-Dev] Fix for Lib/test/leakers/test_gestalt.py In-Reply-To: References: Message-ID: Thomas Heller schrieb: > I think this fixes the leak in Lib/test/leakers/test_gestalt.py. > > Index: Python/mactoolboxglue.c > =================================================================== > --- Python/mactoolboxglue.c (revision 50522) > +++ Python/mactoolboxglue.c (working copy) > @@ -60,8 +60,9 @@ > strncpy(buf, input, sizeof(buf) - 1); > buf[sizeof(buf) - 1] = '\0'; > } > + Py_DECREF(rv); > } > - > + Py_XDECREF(m); > return buf; > } > > > I'm less sure about this one, but it looks like the DECREF > is also required: (Here's the patch that I *really* meant) Index: Mac/Modules/macosmodule.c =================================================================== --- Mac/Modules/macosmodule.c (Revision 50515) +++ Mac/Modules/macosmodule.c (Arbeitskopie) @@ -375,6 +375,7 @@ /* And try again... */ h = GetResource('Estr', err); } + Py_DECREF(m); } } /* Thomas From python-dev at zesty.ca Mon Jul 10 23:53:01 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Mon, 10 Jul 2006 16:53:01 -0500 (CDT) Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: On 7/10/06, Ka-Ping Yee wrote: > What's wrong with "nonlocal"? I don't think i've seen an argument > against that one so far (from Talin or others). On Mon, 10 Jul 2006, Jeremy Hylton wrote: > It's a made-up word. You won't find it in the dictionary and the > google define: query sends me to a wikipedia page about quantum > mechanics. Two million Google hits for "nonlocal" seems like plenty. The explanation of "nonlocal" is pretty straightforward -- If the definition of f() contains an assignment to x, then x is a local variable in f, unless x is declared nonlocal. > To express this email in the positive form: > 1. Reserved words should be real words. > 2. The meaning of the word should be clear. > 3. "Put statements in positive form." (Strunk & White) > 4. The word should sound good. > > global meets all of these requirements. But it's the wrong word. "Purple" also meets all of these requirements. I'd rather accurately express the concept in the negative (the true meaning really is in the negative: "don't make a new binding"), than choose a simple-sounding word that is essentially a lie. x = 1 def f(x): print x def g(): global x x = 3 print x The "x" used in g is not global at all -- it belongs to f. -- ?!ng From guido at python.org Tue Jul 11 00:24:29 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Jul 2006 15:24:29 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <20060710204859.GA11451@rogue.amk.ca> References: <20060710151353.GA30654@code0.codespeak.net> <20060710204859.GA11451@rogue.amk.ca> Message-ID: Not to mention the cost to documentation and books everywhere -- updating our own docs is only the tip of the iceberg. And then updating the users' brains is an even bigger job... (Though at least it is highly parallellizable. :-) --Guido On 7/10/06, A.M. Kuchling wrote: > On Mon, Jul 10, 2006 at 05:13:53PM +0200, Armin Rigo wrote: > > didn't draw much applause. It certainly gave me the impression that > > many changes in Python are advocated and welcomed by only a small > > fraction of users. > > The benefits of changes are usually clear, but I don't think the costs > of changes are fully assessed. python-dev considers the cost of > changes to CPython's implementation, but I don't think the cost > changes to Jython, IronPython, or PyPy are taken into account here. > PyPy probably has enough representatives here who would squawk if > something was difficult, but I don't think Jython or IronPython do. > > I think if we assessed those costs fully, the bar for changes to the > language would be a good deal higher. > > --amk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From almann.goo at gmail.com Tue Jul 11 01:11:27 2006 From: almann.goo at gmail.com (Almann T. Goo) Date: Mon, 10 Jul 2006 19:11:27 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: <7e9b97090607101611k6e1718bbi13edf82b865e8406@mail.gmail.com> On 7/10/06, Jeremy Hylton wrote: > > On 7/10/06, Ka-Ping Yee wrote: > > On Mon, 10 Jul 2006 skip at pobox.com wrote: > > > I think Talin's got a point though. It seems hard to find one short > English > > > word that captures the essence of the desired behavior. None of the > words > > > in his list seem strongly suggestive of the meaning to me. I suspect > that > > > means one's ultimately as good (or as bad) as the rest. > > > > What's wrong with "nonlocal"? I don't think i've seen an argument > > against that one so far (from Talin or others). > > It's a made-up word. You won't find it in the dictionary and the > google define: query sends me to a wikipedia page about quantum > mechanics. It also expresses itself in the negative form "not local" > as opposed to the positive form like global "this is a global." > Finally, I think it sounds yucky. > > To express this email in the positive form: > 1. Reserved words should be real words. > 2. The meaning of the word should be clear. > 3. "Put statements in positive form." (Strunk & White) > 4. The word should sound good. > > global meets all of these requirements. "free" was the word I > remember preferring from earlier discussions, but I think it fails #2. > (Too much confusion about freeing memory, for example.) I remember previous discussions also referring to spelling this as "outer" which IMO passes #2 as well as the other, although arguably #4 is subjective ;-). -Almann -- Almann T. Goo almann.goo at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060710/5d24e320/attachment.html From greg.ewing at canterbury.ac.nz Tue Jul 11 01:32:22 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Jul 2006 11:32:22 +1200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> Message-ID: <44B2E386.50101@canterbury.ac.nz> Guido van Rossum wrote: > Then let's allow > > nonlocal x = 12 > > as a shortcut for > > nonlocal x > x = 12 I thought you didn't like that, because in nonlocal x = 12 x = 42 it's not clear whether these are talking about the same x or not. -- Greg From greg.ewing at canterbury.ac.nz Tue Jul 11 01:50:35 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Jul 2006 11:50:35 +1200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: <44B2E7CB.7040405@canterbury.ac.nz> Mike Krell wrote: > If I've followed the discussions correctly, I think the parent scope > would be operative, so I humbly suggest "parent". -1, this is far too commonly used as a variable name. -- Greg From tim.peters at gmail.com Tue Jul 11 02:46:16 2006 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 10 Jul 2006 20:46:16 -0400 Subject: [Python-Dev] "Missing" 2.5 feature In-Reply-To: <44B21D0D.5090603@scottdial.com> References: <1f7befae0607081831x4ff28307j8778f552c63be95@mail.gmail.com> <200607091705.16027.anthony@interlink.com.au> <1f7befae0607090256h22cf00b7q1a4a6d099588d300@mail.gmail.com> <1f7befae0607091544l7966feecsaaca705b3eeefaa7@mail.gmail.com> <44B21D0D.5090603@scottdial.com> Message-ID: <1f7befae0607101746x14f52816jb64dc1f7abfdf900@mail.gmail.com> [Scott Dial] > Wouldn't this function be better named sys._getframes since we already > have a sys._getframe for getting the current frame? http://mail.python.org/pipermail/python-dev/2005-March/051887.html The first & only name suggested won. As it says there, I usually have no appetite for naming arguments. From barry at python.org Tue Jul 11 03:52:34 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 10 Jul 2006 21:52:34 -0400 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Patch #1520294 adds support for attributes defined with PyGetSetDef in extension modules to pydoc, specifically so things like help (array.array.typecode) gives something useful, like the attribute's docstring for instance. Along the way, I added types.GetSetterType and inspect.isgetsetter(), along with tests and docs. Now I definitely think that the pydoc change should go into Python 2.5 and be backported to Python 2.4, but it's a bit of a question whether the types.py and inspect.py changes should go into Python 2.5 now that we're in beta. I personal think they're worthwhile changes to go it, but I won't argue too hard for them since the pydoc fixes can be implemented with local changes to pydoc.py alone. Thoughts? - -Barry https://sourceforge.net/tracker/? func=detail&atid=305470&aid=1520294&group_id=5470 -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRLMEYnEjvBPtnXfVAQKB8gQAmkk0V0B0xdRD/Xgko4vD0JmvFiXZL3EO bEzE9Wtn8u54JB/EXLdocjeFLjlOzlSzalLKGvxgQHyKGP0F8CSBN6mIreTsztct wl58FyPro3mkwsamUrkPzf/xSZk/4tO81oXPkuzqANbv+0WEnD5MmdSpVqz7jQIo qqbAqah2ha4= =kxSp -----END PGP SIGNATURE----- From jeremy at alum.mit.edu Tue Jul 11 04:54:22 2006 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon, 10 Jul 2006 22:54:22 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: On 7/10/06, Ka-Ping Yee wrote: > On 7/10/06, Ka-Ping Yee wrote: > > What's wrong with "nonlocal"? I don't think i've seen an argument > > against that one so far (from Talin or others). > > On Mon, 10 Jul 2006, Jeremy Hylton wrote: > > It's a made-up word. You won't find it in the dictionary and the > > google define: query sends me to a wikipedia page about quantum > > mechanics. > > Two million Google hits for "nonlocal" seems like plenty. > The explanation of "nonlocal" is pretty straightforward -- > > If the definition of f() contains an assignment to x, then > x is a local variable in f, unless x is declared nonlocal. > > > To express this email in the positive form: > > 1. Reserved words should be real words. > > 2. The meaning of the word should be clear. > > 3. "Put statements in positive form." (Strunk & White) > > 4. The word should sound good. > > > > global meets all of these requirements. > > But it's the wrong word. "Purple" also meets all of these requirements. No. Don't be silly. The current use of global meets the requirements. The meaning there is clear, because global means global namespace. If purple were a keyword, I wouldn't know what it means. I was not proposing the use of global for some other meaning (and thought I made that clear in the remainder of the message). Jeremy > > I'd rather accurately express the concept in the negative (the > true meaning really is in the negative: "don't make a new binding"), > than choose a simple-sounding word that is essentially a lie. > > x = 1 > > def f(x): > print x > > def g(): > global x > x = 3 > print x > > The "x" used in g is not global at all -- it belongs to f. > > > -- ?!ng > From mbarnes at redhat.com Tue Jul 11 05:52:28 2006 From: mbarnes at redhat.com (Matthew Barnes) Date: Mon, 10 Jul 2006 23:52:28 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: <1152589949.14276.16.camel@localhost.localdomain> On Mon, 2006-07-10 at 16:43 -0400, Jeremy Hylton wrote: > To express this email in the positive form: > 1. Reserved words should be real words. > 2. The meaning of the word should be clear. > 3. "Put statements in positive form." (Strunk & White) > 4. The word should sound good. As I've been following this thread I find that the word "extern" keeps coming to mind. It's debatable whether "extern" passes #1, although my dictionary has an entry for it. But more importantly, there seems to be fairly strong parallels between what we're discussing here and its meaning in C/C++ (i.e. the symbol is defined outside of the current scope). So I think "extern" easily passes #2 for C/C++ programmers, and I think others can probably guess that extern == external (my wife did, at least). I haven't seen this suggested yet so I thought I'd just throw it out there. Matt From python-dev at zesty.ca Tue Jul 11 05:58:32 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Mon, 10 Jul 2006 22:58:32 -0500 (CDT) Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: On Mon, 10 Jul 2006, Jeremy Hylton wrote: > The current use of global meets the requirements. The meaning there > is clear, because global means global namespace. If purple were a > keyword, I wouldn't know what it means. I was not proposing the use > of global for some other meaning (and thought I made that clear in the > remainder of the message). My apologies. I misunderstood you -- i thought you were suggesting it as an alternative to "nonlocal". -- ?!ng From benji at zope.com Tue Jul 11 05:36:07 2006 From: benji at zope.com (Benji York) Date: Mon, 10 Jul 2006 23:36:07 -0400 Subject: [Python-Dev] Doctest and Footnotes Message-ID: <44B31CA7.7080906@zope.com> A coworker of mine (Gary Poster) had a really good idea a couple weeks ago: teach doctest about ReST-style footnotes. I implemented it over the weekend and brought it to Tim Peter's attention today. Tim generally liked the idea and suggested I bring it up here. Here's the idea: when a footnote is referenced in prose, execute the code associated with the footnote at that point. For example: """ After initializing the system [#init]_ it is possible to retrieve status information: >>> system.status() 'good to go' [snip some of the doctest] .. [#init] Initialize the system: >>> system = System() >>> system.init() """ The footnote code is executed every time the footnote is referenced, and is /not/ executed at any other time (i.e. it's not executed at the point the footnote is defined). A warning is generated if a footnote (that includes code) is defined but never used. This would allow moving repetitive or verbose code (e.g. tests for corner cases) into footnotes so they won't hinder the documentation aspect of a test. It also allows defining reusable bits of setup code, freeing the doctest author to structure the prose as they wish instead of being constrained by having to place bits of code with common environmental needs together. I've implemented this in a branch of zope.testing (which contains a copy of a post-Python 2.4.3 version of doctest (http://tinyurl.com/nekam). The behavior is controlled by an optionflag, much like ELLIPSIS or NORMALIZE_WHITESPACE. Tim has given me a few pointers on improvements to make, which I'll work on this week. Thoughts/questions? -- Benji York Senior Software Engineer Zope Corporation From skip at pobox.com Tue Jul 11 06:44:25 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 10 Jul 2006 23:44:25 -0500 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: <17587.11433.405626.356611@montanaro.dyndns.org> >> What's wrong with "nonlocal"? I don't think i've seen an argument >> against that one so far (from Talin or others). Mike> It sounds a bit awkward to me. Also, it would be nice if the Mike> keyword indicated which scope was operative. Sounds awkward to me as well. Couldn't put my finger on it until seeing Jeremy's post though. I don't think the keyword should indicate a scope. I'd prefer it if LOAD_ just percolated its way up the chain of cells (or could be identified at compile time by inspecting the AST as I think Guido intends) without the programmer having to name the binding scope. Maybe if it names the function containing the variable to bind. Nothing static (like "up 3 levels") though. Mike> If I've followed the discussions correctly, I think the parent Mike> scope would be operative, so I humbly suggest "parent". Since it might not just be in the immediate parent scope, how about "ancestor"? <0.5 wink> Skip From pje at telecommunity.com Tue Jul 11 06:47:21 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 00:47:21 -0400 Subject: [Python-Dev] Doctest and Footnotes In-Reply-To: <44B31CA7.7080906@zope.com> Message-ID: <5.1.1.6.0.20060711004320.01eed7d0@sparrow.telecommunity.com> At 11:36 PM 7/10/2006 -0400, Benji York wrote: >The footnote code is executed every time the footnote is referenced, and >is /not/ executed at any other time (i.e. it's not executed at the point >the footnote is defined). A warning is generated if a footnote (that >includes code) is defined but never used. > >This would allow moving repetitive or verbose code (e.g. tests for >corner cases) into footnotes so they won't hinder the documentation >aspect of a test. It also allows defining reusable bits of setup code, >freeing the doctest author to structure the prose as they wish instead >of being constrained by having to place bits of code with common >environmental needs together. > >I've implemented this in a branch of zope.testing (which contains a copy >of a post-Python 2.4.3 version of doctest (http://tinyurl.com/nekam). >The behavior is controlled by an optionflag, much like ELLIPSIS or >NORMALIZE_WHITESPACE. Tim has given me a few pointers on improvements >to make, which I'll work on this week. > >Thoughts/questions? It would be nice if tracebacks in the footnote show the invoking context, so that if a footnote is invoked from more than one place, you'll be able to tell which one from any errors that occur. I haven't looked at the code, so perhaps you've already done this. My other thought would be that having a patch that works against the 2.5 version of doctest would be good, as 2.5 has fixes to make doctest work with zipped (or egged) doctest files. Apart from those two comments, it sounds like an interesting idea. From mbk.lists at gmail.com Tue Jul 11 07:51:32 2006 From: mbk.lists at gmail.com (Mike Krell) Date: Mon, 10 Jul 2006 22:51:32 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <17587.11433.405626.356611@montanaro.dyndns.org> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> <17587.11433.405626.356611@montanaro.dyndns.org> Message-ID: On 7/10/06, skip at pobox.com wrote: > I don't think the keyword should indicate a scope. > I'd prefer it if LOAD_ just percolated its way up the chain of > cells (or could be identified at compile time by inspecting the AST as I > think Guido intends) without the programmer having to name the binding > scope. I agree completely. I realize that probably wasn't clear when I said "it would be nice if the keyword indicated which scope was operative". I was really only trying to proffer a keyword which would hopefully suggest to a newbie that they should percolate up the chain of scopes. In this sense, the word "outer" works for me, but I'm sympathetic to Andrew Koenig's argument that "outer" is confusing because the innermost binding is actually affected. > Since it might not just be in the immediate parent scope, how about > "ancestor"? <0.5 wink> That thought had occurred to me, but then I beat it down with a stick :-) My rationale was that the affected binding is the one seen by the parent, even if the parent did not create the binding itself. Greg Ewing's point about parent being a common variable name is well-taken, though. Mike From talin at acm.org Tue Jul 11 08:05:19 2006 From: talin at acm.org (Talin) Date: Mon, 10 Jul 2006 23:05:19 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: <44B33F9F.2000804@acm.org> Ka-Ping Yee wrote: > On Mon, 10 Jul 2006 skip at pobox.com wrote: > >>I think Talin's got a point though. It seems hard to find one short English >>word that captures the essence of the desired behavior. None of the words >>in his list seem strongly suggestive of the meaning to me. I suspect that >>means one's ultimately as good (or as bad) as the rest. > > > What's wrong with "nonlocal"? I don't think i've seen an argument > against that one so far (from Talin or others). Well, I just think that a fix for "an aesthetic wart" should be, well, aesthetic :) I also think that it won't be a complete disaster if we do nothing at all - there *are* existing ways to deal with this problem; there are even some which aren't hackish and non-obvious. For example, its easy enough to create an object which acts as an artificial scope: def x(): scope = object() scope.x = 1 def y(): scope.x = 2 To my mind, the above code looks about as elegant and efficient as most of the proposals put forward so far, and it already works. How much are we really saving here by building this feature into the language? -- Talin From talin at acm.org Tue Jul 11 08:29:10 2006 From: talin at acm.org (Talin) Date: Mon, 10 Jul 2006 23:29:10 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AEA550.1030204@acm.org> <44AF26A4.2040201@acm.org> <44B074C6.3070701@acm.org> Message-ID: <44B34536.7040907@acm.org> Brett Cannon wrote: > Using a factory method callback, one could store the PyCodeObject in a C > proxy object that just acts as a complete delegate, forwarding all method > calls to the internally stored PyCodeObject. That would work. > > > For this initial implementation, though, I am not going to try to support > this. We can always add support like this later since it doesn't > fundamentally break or change anything that is already planned. Let's > focus > on getting even more basic stuff working before we start to get too fancy. > > -Brett Thinking about this some more, I've come up with a simpler idea for a generic wrapper class. The wrapper consists of two parts: a decorator to indicate that a given method is 'public', and a C 'guard' wrapper that insures that only 'public' members can be accessed. So for example: from Sandbox import guard, public class FileProxy: # Public method, no decorator @public def read( length ): ... @public def seek( offset ): ... # Private method, no decorator def open( path ): ... # Construct an instance of FileProxy, and wrap a # guard object around it. Any attempt to access a non-public # attribute will raise an exception (or perhaps simply report # that the attribute doesn't exist.) fh = guard( FileProxy() ) Now, from my point of view this *is* 'the basic stuff'. In other words, this right here is the fundamental sandbox mechanism, and everything else is built on top of it. Now, the C 'guard' function is only a low-level means to insure that no-one can mess with the object; It is not intended to be the actual restriction policy itself. Those are placed in the wrapper classes, just as in your proposed scheme. (This goes back to my basic premise, that a simple "yes/no" security feature can be used to build up much more complex and subtle security features.) The only real complexity of this, as I see it, is that methods to references will have to be themselves wrapped. In other words, if I say 'fh.read' without the (), what I get back can't be the actual read function object - that would be too easy to fiddle with. What I'd have to get is a wrapped version of the method that is a callable. One relatively simply way to deal with this is to have the 'public' decorator create a C wrapper for the function object, and store it as an attribute of the function. The class wrapper then simply looks up the attribute, and if it has an attribute wrapper, returns that, otherwise it fails. (Although, I've often wished for Python to have a variant of __call__ that could be used to override individual methods, i.e.: __call_method__( self, methodname, *args ) This would make the guard wrapper much easier to construct, since we could restrict the methods only to being called, and not allow references to methods to be taken.) -- Talin From martin at v.loewis.de Tue Jul 11 08:30:06 2006 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 11 Jul 2006 08:30:06 +0200 Subject: [Python-Dev] Klocwork analysis of source if we want it In-Reply-To: References: Message-ID: <44B3456E.3000109@v.loewis.de> Brett Cannon wrote: > http://www.klocwork.com/company/releases/06_26_06.asp > > Looks like Klocowork is doing the same thing as Coverity and providing > free static analysis of source for open source projects. Doubt we want > this *and* Coverity, but figured wouldn't hurt to let people know about it. See http://python.org/sf/1484556 ISTM that it generates too many false positives to be useful. Regards, Martin From fredrik at pythonware.com Tue Jul 11 08:42:02 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 11 Jul 2006 08:42:02 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: Jeremy Hylton wrote: > To express this email in the positive form: > 1. Reserved words should be real words. > 2. The meaning of the word should be clear. > 3. "Put statements in positive form." (Strunk & White) > 4. The word should sound good. agreed. a word should describe what a thing is, not what it isn't. not entirely sure about "global", though; things like "outer" and "extern(al)" might be better (especially if we could ignore C, which I don't think we can). cannot think of prior art here, but there has to be some. anyone ? From fredrik at pythonware.com Tue Jul 11 08:45:49 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 11 Jul 2006 08:45:49 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44B33F9F.2000804@acm.org> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> <44B33F9F.2000804@acm.org> Message-ID: Talin wrote: > I also think that it won't be a complete disaster if we do nothing at > all - there *are* existing ways to deal with this problem; there are > even some which aren't hackish and non-obvious. For example, its easy > enough to create an object which acts as an artificial scope: > > def x(): > scope = object() > scope.x = 1 > def y(): > scope.x = 2 > > To my mind, the above code looks about as elegant and efficient as most > of the proposals put forward so far, and it already works. > > How much are we really saving here by building this feature into the > language? I don't think anyone commented on my "mutable" post, but as I mentioned in that post, if you look at the use cases for this, what most use cases really need is *mutation*, not *rebinding*. From talin at acm.org Tue Jul 11 08:53:35 2006 From: talin at acm.org (Talin) Date: Mon, 10 Jul 2006 23:53:35 -0700 Subject: [Python-Dev] easy_install Message-ID: <44B34AEF.8030008@acm.org> Here's something to discuss: First, let me say that I love easy_install. I absolutely "just works" and does what I want, and makes it really simple to install whatever bit of Python code I need. At the same time, however, I get kind of scared when I hear people on the list discussing the various hacks needed to get setuputils and distutils et al all playing nice with each other (monkeypatching, etc.) Having done a small bit of fiddling with distutils myself (as a user, I mean), I can see that while there's a terrific amount of effort put into it, its also not for the feignt of heart. That's not entirely distutil's fault - I gather that it's dealing with a lot of accumulated cruft (I imagine things like different and strange ways of archiving modules, dynamic modifications to path, all that sort of thing.) It seems to me that if someone was going to spend some energy on this list coming up with proposals to improve Python, the thing that would have the most positive benefit in the long run (with the possible exception of Brett's work on rexec) would be a unified and clean vision of the whole import / package / download architecture. Now, speaking from complete ignorance here, I might be way off base - it may be that this matter is well in hand, perhaps on some other mailing list. I don't know. In any case, I wanted to throw this out there... -- Talin From nnorwitz at gmail.com Tue Jul 11 08:54:28 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 10 Jul 2006 23:54:28 -0700 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: On 7/10/06, Fredrik Lundh wrote: > Jeremy Hylton wrote: > > > To express this email in the positive form: > > 1. Reserved words should be real words. > > 2. The meaning of the word should be clear. > > 3. "Put statements in positive form." (Strunk & White) > > 4. The word should sound good. > > agreed. a word should describe what a thing is, not what it isn't. Just looking at a thesaurus, came up with expose. Not to mention outward, remote, peripheral and without. We already have a 'with' keyword, why not 'without'? Oh, I guess it fails #3...and #2...and #4 too. Screw it, just go with alien. :-) http://thesaurus.reference.com/browse/outer n From astrand at cendio.se Tue Jul 11 09:05:47 2006 From: astrand at cendio.se (=?iso-8859-1?Q?Peter_=C5strand?=) Date: Tue, 11 Jul 2006 09:05:47 +0200 (CEST) Subject: [Python-Dev] subprocess.CalledProcessError.errno (#1223937) Message-ID: I intend to fix bug #1223937: subprocess.py abuse of errno. I thought this was going to to tricky, to maintain backwards compatibility, but then I realized that check_call() and CalledProcessError() are not available in any released version of Python, so I guess it's safe to change them. See http://www.python.org/sf?id=1223937 for my suggested patch. Any chance this can go into 2.5? Yeah, I know I'm late :) Regards, -- Peter ?strand ThinLinc Chief Developer Cendio http://www.cendio.se Teknikringen 3 583 30 Link?ping Phone: +46-13-21 46 00 From fredrik at pythonware.com Tue Jul 11 09:15:20 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 11 Jul 2006 09:15:20 +0200 Subject: [Python-Dev] subprocess.CalledProcessError.errno (#1223937) In-Reply-To: References: Message-ID: Peter ?strand wrote: > I intend to fix bug #1223937: subprocess.py abuse of errno. I thought > this was going to to tricky, to maintain backwards compatibility, but > then I realized that check_call() and CalledProcessError() are not > available in any released version of Python, so I guess it's safe to > change them. > > See http://www.python.org/sf?id=1223937 for my suggested patch. Any > chance this can go into 2.5? Yeah, I know I'm late :) that's up to Anthony to decide, but I'd say this this qualifies as a bug fix. From fuzzyman at voidspace.org.uk Tue Jul 11 10:14:22 2006 From: fuzzyman at voidspace.org.uk (Fuzzyman) Date: Tue, 11 Jul 2006 09:14:22 +0100 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44B33F9F.2000804@acm.org> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> <44B33F9F.2000804@acm.org> Message-ID: <44B35DDE.5000405@voidspace.org.uk> Talin wrote: >Ka-Ping Yee wrote: > > >>On Mon, 10 Jul 2006 skip at pobox.com wrote: >> >> >> >>>I think Talin's got a point though. It seems hard to find one short English >>>word that captures the essence of the desired behavior. None of the words >>>in his list seem strongly suggestive of the meaning to me. I suspect that >>>means one's ultimately as good (or as bad) as the rest. >>> >>> >>What's wrong with "nonlocal"? I don't think i've seen an argument >>against that one so far (from Talin or others). >> >> > >Well, I just think that a fix for "an aesthetic wart" should be, well, >aesthetic :) > >I also think that it won't be a complete disaster if we do nothing at >all - there *are* existing ways to deal with this problem; there are >even some which aren't hackish and non-obvious. For example, its easy >enough to create an object which acts as an artificial scope: > > def x(): > scope = object() > scope.x = 1 > def y(): > scope.x = 2 > >To my mind, the above code looks about as elegant and efficient as most >of the proposals put forward so far, and it already works. > >How much are we really saving here by building this feature into the >language? > > >>> def x(): ...: scope = object() ...: scope.x = 1 ...: def y(): ...: scope.x = 2 >>> x() --------------------------------------------------------------------------- exceptions.AttributeError Traceback (most recent call last) I've often found it a nuisance that you can't instantiate an 'object', to use as a mutable 'namespace', but instead have to define an arbitrary empty class. What happened to the 'namespace' proposal ? Michael Foord http://www.voidspace.org.uk/python/index.shtml >-- Talin >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > From mwh at python.net Tue Jul 11 10:21:12 2006 From: mwh at python.net (Michael Hudson) Date: Tue, 11 Jul 2006 09:21:12 +0100 Subject: [Python-Dev] User's complaints In-Reply-To: <20060710204859.GA11451@rogue.amk.ca> (A. M. Kuchling's message of "Mon, 10 Jul 2006 16:48:59 -0400") References: <20060710151353.GA30654@code0.codespeak.net> <20060710204859.GA11451@rogue.amk.ca> Message-ID: <2m3bd84knb.fsf@starship.python.net> "A.M. Kuchling" writes: > On Mon, Jul 10, 2006 at 05:13:53PM +0200, Armin Rigo wrote: >> didn't draw much applause. It certainly gave me the impression that >> many changes in Python are advocated and welcomed by only a small >> fraction of users. > > The benefits of changes are usually clear, but I don't think the costs > of changes are fully assessed. python-dev considers the cost of > changes to CPython's implementation, but I don't think the cost > changes to Jython, IronPython, or PyPy are taken into account here. I don't want to attempt to speak for Armin, but I don't think that was quite the point he was trying to make. The cost of implementation isn't really that high -- most of the changes from 2.3 to 2.4 were implemented for PyPy in about a week and we have implementations of most of the 2.5 features already, and you can also consider the decorators discussion where each of the proposed syntaxes had solid tested implementations. At least from my POV, the cost of bad design, whether through simple lack of sufficient thought or aggregation of features into cruft, is much much higher. > PyPy probably has enough representatives here who would squawk if > something was difficult, but I don't think Jython or IronPython do. People implementing Python should accept the fact that Python is a changing language, IMHO. > I think if we assessed those costs fully, the bar for changes to the > language would be a good deal higher. I think if you assessed these other costs fully, the bar would be higher still. Cheers, mwh -- This is the fixed point problem again; since all some implementors do is implement the compiler and libraries for compiler writing, the language becomes good at writing compilers and not much else! -- Brian Rogoff, comp.lang.functional From sreeram at tachyontech.net Tue Jul 11 10:56:54 2006 From: sreeram at tachyontech.net (K.S.Sreeram) Date: Tue, 11 Jul 2006 14:26:54 +0530 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44B33F9F.2000804@acm.org> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> <44B33F9F.2000804@acm.org> Message-ID: <44B367D6.9030600@tachyontech.net> Talin wrote: > def x(): > scope = object() > scope.x = 1 > def y(): > scope.x = 2 'object' doesn't let you set custom attributes. Here's what I normally use in my code... class Data : pass def x() : d = Data() d.x = 1 def y() : d.x += 1 Regards Sreeram -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20060711/eb5ca5be/attachment.pgp From ncoghlan at gmail.com Tue Jul 11 11:44:36 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 11 Jul 2006 19:44:36 +1000 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <44B35DDE.5000405@voidspace.org.uk> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> <44B33F9F.2000804@acm.org> <44B35DDE.5000405@voidspace.org.uk> Message-ID: <44B37304.9090106@gmail.com> Fuzzyman wrote: > I've often found it a nuisance that you can't instantiate an 'object', > to use as a mutable 'namespace', but instead have to define an arbitrary > empty class. > > What happened to the 'namespace' proposal ? The code and the pre-PEP [1] are still out there, but Carlos, Steve and I all got distracted by other things. It really needs to go back to c.l.p to thrash out some of the name collision prblems (e.g. should *all* access to methods, even special methods, be via type(x)?). Cheers, Nick. [1] http://namespace.python-hosting.com/wiki/NamespacePep -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From talin at acm.org Tue Jul 11 12:36:09 2006 From: talin at acm.org (Talin) Date: Tue, 11 Jul 2006 03:36:09 -0700 Subject: [Python-Dev] Capabilities / Restricted Execution Message-ID: <44B37F19.7090703@acm.org> I thought a little bit more about Guido's comment that you can hide Python objects in a C wrapper class. However, as I was trying to sleep, I realized that you don't even need C to do it. The trick is to store the object reference as a closure variable. Assuming that there's no mechanism to directly access such variables, you can effectively have 'private' variables. Here's an pure-Python implementation of a wrapper that restricts access to all class members except those marked with the 'public' decorator. Note in the test cases below that even '__class__' and '__dict__' are inaccessible. ------------------------------------------------------------------ # Exception to throw when we violate restrictions class GuardException( Exception ): pass # Decorator function def public(func): func.f_public = True return func # Wrapper function def guard(content): class Guard(object): def __getattribute__(self, name): # This will throw an AttributeException if the # attribute doesn't exist, which is what # we want. attr = object.__getattribute__(content,name) if hasattr( attr, 'im_func' ): if hasattr( attr.im_func, 'f_public' ): # We can't return the bound method directly, # instead we create a proxy for it. def proxy_attr( *args, **kwargs ): return attr( *args, **kwargs ) return proxy_attr # Attribute exists but has no proxy raise GuardException( name ) return Guard() # Test class class Test(object): def __init__(self,name): self.name = name @public def get_name(self): return self.name def set_name(self,name): self,name = name # Test objects. First is unwrapped, second is wrapped. t1 = Test("alpha") t2 = guard(Test("beta")) # These functions work correctly print t1.name print t2.get_name() # This gets AttributeError because there's no such attribute print t2.unk() # These generate GuardException print t2.set_name() print t2.__dict__ print t2.__class__ print t2.name ------------------------------------------------------ -- Talin From benji at benjiyork.com Tue Jul 11 13:33:10 2006 From: benji at benjiyork.com (Benji York) Date: Tue, 11 Jul 2006 07:33:10 -0400 Subject: [Python-Dev] Doctest and Footnotes In-Reply-To: <5.1.1.6.0.20060711004320.01eed7d0@sparrow.telecommunity.com> References: <5.1.1.6.0.20060711004320.01eed7d0@sparrow.telecommunity.com> Message-ID: <44B38C76.5050502@benjiyork.com> Phillip J. Eby wrote: > It would be nice if tracebacks in the footnote show the invoking context Yep. Someone (Jim Fulton I think) had suggested that to me. I'll look into it. > My other thought would be that having a patch that works against the 2.5 > version of doctest would be good My intent is to update zope.testing to the 2.5 version of doctest and generate a new patch against that. It would be even better if there were a stand-alone release of doctest instead, or (another Jim suggestion) if doctest were made extensible. I have another doctest/ReST idea I'd like to try some time (using ReST tables for FIT), so if there's interest perhaps I'll try my hand at a plugin system of some sort. (wish I could use zope.component :) -- Benji York From gmccaughan at synaptics-uk.com Tue Jul 11 13:09:51 2006 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Tue, 11 Jul 2006 12:09:51 +0100 Subject: [Python-Dev] 2.5 and beyond In-Reply-To: References: <9B706ACDA57B23438AAE4822369672AF04062AD1@RED-MSG-10.redmond.corp.microsoft.com> <20060703171546.1108.JCARLSON@uci.edu> Message-ID: <200607111209.55177.gmccaughan@synaptics-uk.com> > I can say it stronger. Any resemblance between Python and Scheme or > Lisp is purely a coincidence. Neither language is in Python's > ancestry, at least not explicitly; I'd never used or tried to learn > Scheme when I started Python (still haven't) and my Lisp experience > was limited to copying Emacs startup code from friends (still is). Python resembles Lisp like an octopus eye resembles a mammalian eye: they have lots in common because they're both pretty good solutions to similar problems. Deciding whether it's Python or Lisp that has the retina fitted back-to-front is left as an exercise for the reader. -- g From bborcic at gmail.com Tue Jul 11 14:43:52 2006 From: bborcic at gmail.com (Boris Borcic) Date: Tue, 11 Jul 2006 14:43:52 +0200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: <44AEF0F2.5040007@canterbury.ac.nz> References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Boris Borcic wrote: > >> I believe that in this case native linguistic intuition made the decision... > > The reason has nothing to do with language. Guido didn't > want sum() to become an attractive nuisance by *appearing* > to be an obvious way of joining a list of strings, while > actually being a very inefficient way of doing that. sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of chaining strings in a list (without actually being one). > Considerable effort was put into trying to make sum() > smart enough to detect when you were using it on a > list of strings and do "".join behind the scenes, but > Guido decided in the end that it wasn't worth the > trouble, given that he only ever intended sum() to > be used on numbers in the first place. That's not quite conform to the record. According to py-dev archives it happened on April 2003 with a thread "Fwd: summing a bunch of numbers (or "whatevers")" initiated by Alex Martelli where he actually proposed a working implementation of sum() in C, that did short-circuit the case of strings to ''.join. That was Sat 19th of April. Debate ensued, and by late Sunday 20th around 11PM;, the honorable author of the Zen of Python had killed that use case for sum() with "sum(sequence_of_strings) hurts my brain". (Hello Tim, so what about < sqrt(':(') > ?) Guido's first intervention in the thread was the next morning, and the two very first lines of his intervention where : "OK, let me summarize and pronounce. sum(sequence_of_strings) is out...." I admit that there is a step of arguable interpretation from these recorded facts to my diagnostic, but the latter is compatible with the facts. Your version otoh looks more robust in the role of eg creation myth. Best Regards, Boris Borcic -- "assert 304 in 340343, P424D15E_M15M47CH" From fredrik at pythonware.com Tue Jul 11 15:06:09 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 11 Jul 2006 15:06:09 +0200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> Message-ID: Boris Borcic wrote: >>> I believe that in this case native linguistic intuition made the decision... >> >> The reason has nothing to do with language. Guido didn't >> want sum() to become an attractive nuisance by *appearing* >> to be an obvious way of joining a list of strings, while >> actually being a very inefficient way of doing that. > > sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of > chaining strings in a list (without actually being one). in what language the word "sum" an appropriate synonym for "concatenate" ? From gmccaughan at synaptics-uk.com Tue Jul 11 15:29:21 2006 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Tue, 11 Jul 2006 14:29:21 +0100 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> Message-ID: <200607111429.22964.gmccaughan@synaptics-uk.com> (Attention conservation notice: the following is concerned almost entirely with exegesis of an old python-dev thread. Those interested in improving Python and not in history and exegesis should probably ignore it.) On Tuesday 2006-07-11 13:43, Boris Borcic wrote: > >> I believe that in this case native linguistic intuition made the decision... > > > > The reason has nothing to do with language. Guido didn't > > want sum() to become an attractive nuisance by *appearing* > > to be an obvious way of joining a list of strings, while > > actually being a very inefficient way of doing that. > > sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of > chaining strings in a list (without actually being one). It is a *short-term* attractive nuisance; anyone who tries it will quickly find that it can't be used to "sum" strings; whereas if this functionality of sum() had been retained then users would much more readily have been led to use it *and leave it in their programs*, to the detriment of their efficiency. > > Considerable effort was put into trying to make sum() > > smart enough to detect when you were using it on a > > list of strings and do "".join behind the scenes, but > > Guido decided in the end that it wasn't worth the > > trouble, given that he only ever intended sum() to > > be used on numbers in the first place. > > That's not quite conform to the record. According to py-dev archives it happened > on April 2003 with a thread "Fwd: summing a bunch of numbers (or "whatevers")" > initiated by Alex Martelli where he actually proposed a working implementation > of sum() in C, that did short-circuit the case of strings to ''.join. That was > Sat 19th of April. > > Debate ensued, and by late Sunday 20th around 11PM;, the honorable author of the > Zen of Python had killed that use case for sum() with "sum(sequence_of_strings) > hurts my brain". (Hello Tim, so what about < sqrt(':(') > ?) Hardly "killed", since discussion of sum(stringseq) continued after Tim's comment, and since Tim was neither the only, nor the first, nor (I think) the last, person to object to sum(stringseq). > Guido's first intervention in the thread was the next morning, and the two very > first lines of his intervention where : > > "OK, let me summarize and pronounce. > > sum(sequence_of_strings) is out...." > > I admit that there is a step of arguable interpretation from these recorded > facts to my diagnostic, but the latter is compatible with the facts. Your > version otoh looks more robust in the role of eg creation myth. Your interpretation is only "compatible with the facts" by means of the hypothesis that Guido, despite saying "OK, let me summarize and pronounce" (which, incidentally, was the *actual* opening of that message, which was not Guido's first intervention in the thread, but who cares about facts?), had only read Tim Peters's message and not the rest of the thread. Including, for instance, one of the last messages before Guido's, from Alex, reporting that his sum() concatenated strings twice as slowly as ''.join. It also requires one to ignore the fact that Guido said (in that same message that you described as his "first intervention") | OTOH if we provide *any* way of providing a | different starting point, some creative newbie is going to use | sum(list_of_strings, "") instead of "".join(), and be hurt by the | performance months later. and the fact that in Guido's *actual* "first intervention" he said that he found sum(stringseq) "weird" and would continue to use ''.join. (Remark: Guido is not a native speaker of English, though he's a very competent one.) For that matter, Tim's comment about sum(stringseq) hurting his brain was in direct response to another part of that same message from Guido in which he said that "the name sum() strongly suggests that it's summing up numbers". (Remark: Guido still isn't a native speaker of English.) Perhaps the fact that "sum" isn't a natural English term for concatenation of strings -- which, indeed, it isn't -- was one of his reasons; it demonstrably wasn't the only one; and it seems to have been the non-native speaker Guido himself who introduced that consideration to the discussion. It appears to me that your interpretation is difficult to reconcile with the facts and is based on an incorrect and narrow reading of the original texts. What was that you were saying about creation myths? (I agree that Greg's interpretation is also not well supported by that thread; I don't know whether Guido later said anything that would determine how much truth there is in it. Since sum() was Alex Martelli's invention, it seems unlikely that Greg's quite right.) -- g From bborcic at gmail.com Tue Jul 11 15:47:25 2006 From: bborcic at gmail.com (Boris Borcic) Date: Tue, 11 Jul 2006 15:47:25 +0200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: <200607111429.22964.gmccaughan@synaptics-uk.com> References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> <200607111429.22964.gmccaughan@synaptics-uk.com> Message-ID: Gareth McCaughan wrote: > (Attention conservation notice: the following is concerned almost entirely > with exegesis of an old python-dev thread. Those interested in improving Python > and not in history and exegesis should probably ignore it.) > > On Tuesday 2006-07-11 13:43, Boris Borcic wrote: > >>>> I believe that in this case native linguistic intuition made the decision... >>> The reason has nothing to do with language. Guido didn't >>> want sum() to become an attractive nuisance by *appearing* >>> to be an obvious way of joining a list of strings, while >>> actually being a very inefficient way of doing that. >> sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of >> chaining strings in a list (without actually being one). > > It is a *short-term* attractive nuisance; anyone who tries it > will quickly find that it can't be used to "sum" strings; > whereas if this functionality of sum() had been retained > then users would much more readily have been led to use it > *and leave it in their programs*, to the detriment of their > efficiency. > >>> Considerable effort was put into trying to make sum() >>> smart enough to detect when you were using it on a >>> list of strings and do "".join behind the scenes, but >>> Guido decided in the end that it wasn't worth the >>> trouble, given that he only ever intended sum() to >>> be used on numbers in the first place. >> That's not quite conform to the record. According to py-dev archives it happened >> on April 2003 with a thread "Fwd: summing a bunch of numbers (or "whatevers")" >> initiated by Alex Martelli where he actually proposed a working implementation >> of sum() in C, that did short-circuit the case of strings to ''.join. That was >> Sat 19th of April. >> >> Debate ensued, and by late Sunday 20th around 11PM;, the honorable author of the >> Zen of Python had killed that use case for sum() with "sum(sequence_of_strings) >> hurts my brain". (Hello Tim, so what about < sqrt(':(') > ?) > > Hardly "killed", since discussion of sum(stringseq) continued after > Tim's comment, and since Tim was neither the only, nor the first, nor > (I think) the last, person to object to sum(stringseq). > >> Guido's first intervention in the thread was the next morning, and the two very >> first lines of his intervention where : >> >> "OK, let me summarize and pronounce. >> >> sum(sequence_of_strings) is out...." >> >> I admit that there is a step of arguable interpretation from these recorded >> facts to my diagnostic, but the latter is compatible with the facts. Your >> version otoh looks more robust in the role of eg creation myth. > > Your interpretation is only "compatible with the facts" by means > of the hypothesis that Guido, despite saying "OK, let me summarize > and pronounce" (which, incidentally, was the *actual* opening of > that message, which was not Guido's first intervention in the > thread, but who cares about facts?), had only read Tim Peters's > message and not the rest of the thread. Including, for instance, > one of the last messages before Guido's, from Alex, reporting that > his sum() concatenated strings twice as slowly as ''.join. It also > requires one to ignore the fact that Guido said (in that same > message that you described as his "first intervention") > > | OTOH if we provide *any* way of providing a > | different starting point, some creative newbie is going to use > | sum(list_of_strings, "") instead of "".join(), and be hurt by the > | performance months later. > > and the fact that in Guido's *actual* "first intervention" he > said that he found sum(stringseq) "weird" and would continue to > use ''.join. (Remark: Guido is not a native speaker of English, > though he's a very competent one.) For that matter, Tim's > comment about sum(stringseq) hurting his brain was in direct > response to another part of that same message from Guido in > which he said that "the name sum() strongly suggests that it's > summing up numbers". (Remark: Guido still isn't a native speaker > of English.) > > Perhaps the fact that "sum" isn't a natural English term for > concatenation of strings -- which, indeed, it isn't -- was one > of his reasons; it demonstrably wasn't the only one; and it > seems to have been the non-native speaker Guido himself who > introduced that consideration to the discussion. > > It appears to me that your interpretation is difficult to reconcile > with the facts and is based on an incorrect and narrow reading of the > original texts. What was that you were saying about creation myths? > > (I agree that Greg's interpretation is also not well supported > by that thread; I don't know whether Guido later said anything > that would determine how much truth there is in it. Since sum() > was Alex Martelli's invention, it seems unlikely that Greg's > quite right.) > From bborcic at gmail.com Tue Jul 11 15:54:24 2006 From: bborcic at gmail.com (Boris Borcic) Date: Tue, 11 Jul 2006 15:54:24 +0200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: <200607111429.22964.gmccaughan@synaptics-uk.com> References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> <200607111429.22964.gmccaughan@synaptics-uk.com> Message-ID: I wish to apologize for mistakenly pushing the "send" button on an untouched copy of Gereth McGaughan's reply, in the case my early cancel at gmane did not stop the propagation. I'll profit just to add (bringing this to a conclusion) Gareth McCaughan wrote: > (...was not Guido's first intervention in the > thread, but who cares about facts?) I do, and I stand corrected. Best regards, Boris Borcic -- "On na?t tous les m?tres du m?me monde" From stefan.rank at ofai.at Tue Jul 11 15:55:46 2006 From: stefan.rank at ofai.at (Stefan Rank) Date: Tue, 11 Jul 2006 15:55:46 +0200 Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt Message-ID: <44B3ADE2.7040108@ofai.at> Hi, urllib.quote fails on unicode strings and in an unhelpful way:: Python 2.4.3 (#69, Mar 29 2006, 17:35:34) [MSC v.1310 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> import urllib >>> urllib.quote('a\xf1a') 'a%F1a' >>> urllib.quote(u'ana') 'ana' >>> urllib.quote(u'a\xf1a') Traceback (most recent call last): File "", line 1, in ? File "C:\Python24\lib\urllib.py", line 1117, in quote res = map(safe_map.__getitem__, s) KeyError: u'\xf1' There is a (closed) tracker item, dated 2000-10-12, http://sourceforge.net/tracker/?group_id=5470&atid=105470&aid=216716&func=detail and there was a note added to PEP-42 by Guido. According to a message I found on quixote-users, http://mail.mems-exchange.org/durusmail/quixote-users/5363/ it might have worked prior to 2.4.2. (I guess that this changed because of ascii now being the default encoding?) BTW, a patch by rhettinger from 8 months or so ago allows urllib.unquote to operate transparently on unicode strings:: >>> urllib.unquote('a%F1a') 'a\xf1a' >>> urllib.unquote(u'a%F1a') u'a\xf1a' I suggest to add (after 2.5 I assume) one of the following to the beginning of urllib.quote to either fail early and consistently on unicode arguments and improve the error message:: if isinstance(s, unicode): raise TypeError("quote needs a byte string argument, not unicode," " use `argument.encode('utf-8')` first.") or to do The Right Thing (tm), which is utf-8 encoding:: if isinstance(s, unicode): s = s.encode('utf-8') as suggested in http://www.w3.org/International/O-URL-code.html and rfc3986. cheers, stefan From bborcic at gmail.com Tue Jul 11 16:07:20 2006 From: bborcic at gmail.com (Boris Borcic) Date: Tue, 11 Jul 2006 16:07:20 +0200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> Message-ID: Fredrik Lundh wrote: > > in what language the word "sum" an appropriate synonym for "concatenate" ? any that admits a+b to mean ''.join([a,b]), I'd say. - BB From fredrik at pythonware.com Tue Jul 11 16:16:49 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 11 Jul 2006 16:16:49 +0200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> Message-ID: Boris Borcic wrote: >> in what language [is] the word "sum" an appropriate synonym for "concatenate" ? > > any that admits a+b to mean ''.join([a,b]), I'd say. and what human language would that be ? From skip at pobox.com Tue Jul 11 16:35:55 2006 From: skip at pobox.com (skip at pobox.com) Date: Tue, 11 Jul 2006 09:35:55 -0500 Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt In-Reply-To: <44B3ADE2.7040108@ofai.at> References: <44B3ADE2.7040108@ofai.at> Message-ID: <17587.46923.677564.898184@montanaro.dyndns.org> Stefan> According to a message I found on quixote-users, Stefan> http://mail.mems-exchange.org/durusmail/quixote-users/5363/ it Stefan> might have worked prior to 2.4.2. Confirmed with 2.3.5. Stefan> if isinstance(s, unicode): Stefan> s = s.encode('utf-8') Stefan> as suggested in Stefan> http://www.w3.org/International/O-URL-code.html Stefan> and rfc3986. Seems like the right way to do it to me. Skip From robinbryce at gmail.com Tue Jul 11 16:46:44 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Tue, 11 Jul 2006 15:46:44 +0100 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: outbound x = 1 x = 2 evaluating using Jeremy Hilton's' list: 1. is a real word 2. For me - in python - it would mean: Is found in 'outer' scope and is already bound. And the literal meaning of 'outbound 'headed away' [1] is pretty darn close to what I mean when I spell the usual mutables kluge. 3 statement is positive form 4. I like it could not find a use of outbound in python source (2.4.3) [1] http://dictionary.reference.com/search?q=outbound Robin From fuzzyman at voidspace.org.uk Tue Jul 11 16:55:20 2006 From: fuzzyman at voidspace.org.uk (Fuzzyman) Date: Tue, 11 Jul 2006 15:55:20 +0100 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> Message-ID: <44B3BBD8.7050307@voidspace.org.uk> Robin Bryce wrote: >outbound x = 1 > > FWINW, to me 'nonlocal' clearly and immediately tells you what you need to know about the variable. 'outbound' has no programming associations for me (it makes me think of 'outward bound' and roaming the great outdoors). So negative or not I'm +1 on nonlocal if we really need this... (Hey, and I'm a native.) Michael Foord http://www.voidspace.org.uk/python/index.shtml >x = 2 > >evaluating using Jeremy Hilton's' list: > >1. is a real word >2. For me - in python - it would mean: Is found in 'outer' scope and >is already bound. > And the literal meaning of 'outbound 'headed away' [1] is pretty >darn close to what I mean when I spell the usual mutables kluge. > >3 statement is positive form >4. I like it > >could not find a use of outbound in python source (2.4.3) > >[1] http://dictionary.reference.com/search?q=outbound > > >Robin >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > > From pje at telecommunity.com Tue Jul 11 16:56:22 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 10:56:22 -0400 Subject: [Python-Dev] Capabilities / Restricted Execution In-Reply-To: <44B37F19.7090703@acm.org> Message-ID: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com> At 03:36 AM 7/11/2006 -0700, Talin wrote: >I thought a little bit more about Guido's comment that you can hide >Python objects in a C wrapper class. However, as I was trying to sleep, >I realized that you don't even need C to do it. > >The trick is to store the object reference as a closure variable. >Assuming that there's no mechanism to directly access such variables, >you can effectively have 'private' variables. A function's func_closure contains cell objects that hold the variables. These are readable if you can set the func_closure of some function of your own. If the overall plan includes the ability to restrict func_closure setting (or reading) in a restricted interpreter, then you might be okay. From anthony at python.org Tue Jul 11 17:30:35 2006 From: anthony at python.org (Anthony Baxter) Date: Wed, 12 Jul 2006 01:30:35 +1000 Subject: [Python-Dev] Subject: RELEASED Python 2.5 (beta 2) Message-ID: <200607120130.49083.anthony@python.org> On behalf of the Python development team and the Python community, I'm happy to announce the second BETA release of Python 2.5. This is an *beta* release of Python 2.5. As such, it is not suitable for a production environment. It is being released to solicit feedback and hopefully discover bugs, as well as allowing you to determine how changes in 2.5 might impact you. If you find things broken or incorrect, please log a bug on Sourceforge. In particular, note that changes to improve Python's support of 64 bit systems might require authors of C extensions to change their code. More information (as well as source distributions and Windows installers) are available from the 2.5 website: http://www.python.org/2.5/ A Universal Mac OSX Installer will be available shortly - in the meantime, Mac users can build from the source tarballs. Since the first beta, a large number of bug fixes have been made to Python 2.5 - see the release notes (available from the 2.5 webpage) for the full details. There has been one very small new feature added - the sys._current_frames() function was added. This is extremely useful for tracking down deadlocks and related problems - a similar technique is already used in the popular DeadlockDebugger extension for Zope. It is not possible to do this sort of debugging from outside the Python core safely and robustly, which is why we've snuck this in after the feature freeze. As of this release, Python 2.5 is now in *feature freeze*. Unless absolutely necessary, no functionality changes will be made between now and the final release of Python 2.5. The plan is for this to be the final beta release. We should now move to one or more release candidates, leading to a 2.5 final release early August. PEP 356 includes the schedule and will be updated as the schedule evolves. At this point, any testing you can do would be greatly, greatly appreciated. The new features in Python 2.5 are described in Andrew Kuchling's What's New In Python 2.5. It's available from the 2.5 web page. Amongst the language features added include conditional expressions, the with statement, the merge of try/except and try/finally into try/except/finally, enhancements to generators to produce a coroutine kind of functionality, and a brand new AST-based compiler implementation. New modules added include hashlib, ElementTree, sqlite3, wsgiref and ctypes. In addition, a new profiling module "cProfile" was added. Enjoy this new release, Anthony Anthony Baxter anthony at python.org Python Release Manager (on behalf of the entire python-dev team) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 191 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20060712/d60b96c4/attachment.pgp From bborcic at gmail.com Tue Jul 11 17:35:10 2006 From: bborcic at gmail.com (Boris Borcic) Date: Tue, 11 Jul 2006 17:35:10 +0200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains Message-ID: Fredrik Lundh wrote: > Boris Borcic wrote: > >>> in what language [is] the word "sum" an appropriate synonym for "concatenate" ? >> any that admits a+b to mean ''.join([a,b]), I'd say. > > and what human language would that be ? Let's admit the answer is 'none' (and I apologize for accusing only English), what is the impact on the idea that natural language intuition is a first rank suspect, to explain the double standard in the Python treatment of str1+str2 and sum([str1,str2]) ? As for 'concatenate'. To my linguistic intuition it is an ugly elitist jargon word made up for the function, like "we mean to say 'to chain' but don't want to say it to your ears, unless you had latin classes". Admitting such a word as the legal standard eliminates the possibility of synonyms. Makes me think, maybe *that* word is the root of the problem, for being too ugly to find its way into Python in the first place. Cheers, BB From bborcic at gmail.com Tue Jul 11 18:21:20 2006 From: bborcic at gmail.com (Boris Borcic) Date: Tue, 11 Jul 2006 18:21:20 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: Fredrik Lundh wrote: > Just van Rossum wrote: > >> Why couldn't at least augmented assignment be implicitly rebinding? It >> has been suggested before (in the context of a rebinding operator), but >> I'm wondering, is this also off the table? >> >> def counter(num): >> def inc(): >> num += 1 >> return num >> return inc >> >> Reads very natural to me. It's likely the most frequent example of what >> people try before they learn that rebinding to outer scopes isn't >> allowed. It could Just Work. > > note that most examples of this type already work, if the target type is > mutable, and implement the right operations: > > def counter(num): > num = mutable_int(num) > def inc(): > num += 1 > return num > return inc I agree with you (and argued it in "scopes vs augmented assignment vs sets" recently) that mutating would be sufficient /if/ the compiler would view augmented assignment as mutations operators : which it doesn't as far as concerns scopes where a variable appears as target only of /augmented/ assignments. Currently, the code you propose above will not work, and whatever your mutable_int() it will result in UnboundLocalError: local variable 'num' referenced before assignment What probably trips you is that the compiler thus makes a choice of interpretation that has no use cases. Cheers, BB From guido at python.org Tue Jul 11 18:35:48 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Jul 2006 09:35:48 -0700 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: References: Message-ID: Please end this thread. Now. Really. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Tue Jul 11 19:03:56 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 12 Jul 2006 03:03:56 +1000 Subject: [Python-Dev] TRUNK is UNFROZEN. Message-ID: <200607120303.59555.anthony@interlink.com.au> beta2 is done, so trunk is unfrozen. Remember, we're still in feature freeze, so new features need approval before being committed. Thanks! Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Tue Jul 11 19:05:44 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 12 Jul 2006 03:05:44 +1000 Subject: [Python-Dev] Minor: Unix icons for 2.5? Message-ID: <200607120305.45714.anthony@interlink.com.au> There's an open PEP-356 issue for "update the icons to the newer shinier ones" for Unix. As far as I can see, there's the 14x15 GIF images used for Idle and the documentation. Note that for me at least, idle comes up without an icon _anyway_. Are there any others I missed? Anthony -- Anthony Baxter It's never too late to have a happy childhood. From g.brandl at gmx.net Tue Jul 11 19:25:49 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Jul 2006 19:25:49 +0200 Subject: [Python-Dev] Minor: Unix icons for 2.5? In-Reply-To: <200607120305.45714.anthony@interlink.com.au> References: <200607120305.45714.anthony@interlink.com.au> Message-ID: Anthony Baxter wrote: > There's an open PEP-356 issue for "update the icons to the newer > shinier ones" for Unix. As far as I can see, there's the 14x15 GIF > images used for Idle and the documentation. Note that for me at least, > idle comes up without an icon _anyway_. > > Are there any others I missed? In case we add a Python .desktop file (as proposed in patch #1353344), we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in is attached. Georg Index: Misc/python.desktop.in =================================================================== --- Misc/python.desktop.in (Revision 0) +++ Misc/python.desktop.in (Revision 0) @@ -0,0 +1,14 @@ +[Desktop Entry] +Encoding=UTF-8 +Name=Python Programming Language +Name[sv]=Programspr??ket Python +Name[de]=Die Programmiersprache Python +Comment=Program in Python +Comment[sv]=Programmera med Python +Comment[de]=Programmieren in Python +Exec=python_VER_ %f +Terminal=true +Type=Application +Icon=pycon.png +Categories=Application;Development;ConsoleOnly +MimeType=text/x-python;application/x-python Index: Makefile.pre.in =================================================================== --- Makefile.pre.in (Revision 47289) +++ Makefile.pre.in (Arbeitskopie) @@ -89,6 +89,8 @@ INCLUDEDIR= @includedir@ CONFINCLUDEDIR= $(exec_prefix)/include SCRIPTDIR= $(prefix)/lib +DESKTOPDIR= $(prefix)/share/applications +ICONDIR= $(prefix)/share/icons # Detailed destination directories BINLIBDEST= $(LIBDIR)/python$(VERSION) @@ -614,7 +616,7 @@ $(TESTPYTHON) $(TESTPROG) $(MEMTESTOPTS) # Install everything -install: @FRAMEWORKINSTALLFIRST@ altinstall bininstall maninstall @FRAMEWORKINSTALLLAST@ +install: @FRAMEWORKINSTALLFIRST@ altinstall bininstall maninstall desktopinstall @FRAMEWORKINSTALLLAST@ # Install almost everything without disturbing previous versions altinstall: @FRAMEWORKALTINSTALLFIRST@ altbininstall libinstall inclinstall libainstall \ @@ -687,6 +689,25 @@ $(INSTALL_DATA) $(srcdir)/Misc/python.man \ $(DESTDIR)$(MANDIR)/man1/python.1 +# Install the .desktop file and the icons +desktopinstall: + @for i in $(DESKTOPDIR) $(ICONDIR) $(ICONDIR)/hicolor $(ICONDIR)/hicolor/16x16 $(ICONDIR)/hicolor/32x32 $(ICONDIR)/hicolor/48x48; \ + do \ + if test ! -d $(DESTDIR)$$i; then \ + echo "Creating directory $$i"; \ + $(INSTALL) -d -m $(DIRMODE) $(DESTDIR)$$i; \ + else true; \ + fi; \ + done + sed 's/_VER_/$(VERSION)$(EXE)/' $(srcdir)/Misc/python.desktop.in > $(srcdir)/Misc/python.desktop + $(INSTALL_DATA) $(srcdir)/Misc/python.desktop $(DESTDIR)$(DESKTOPDIR)/python.desktop + for i in 16x16 32x32 48x48; \ + do \ + $(INSTALL_DATA) $(srcdir)/Misc/py-$$i.png $(DESTDIR)$(ICONDIR)/hicolor/$$i/py.png; \ + $(INSTALL_DATA) $(srcdir)/Misc/pyc-$$i.png $(DESTDIR)$(ICONDIR)/hicolor/$$i/pyc.png; \ + $(INSTALL_DATA) $(srcdir)/Misc/pycon-$$i.png $(DESTDIR)$(ICONDIR)/hicolor/$$i/pycon.png; \ + done + # Install the library PLATDIR= plat-$(MACHDEP) EXTRAPLATDIR= @EXTRAPLATDIR@ @@ -1076,7 +1097,7 @@ # Declare targets that aren't real files .PHONY: all sharedmods oldsharedmods test quicktest memtest .PHONY: install altinstall oldsharedinstall bininstall altbininstall -.PHONY: maninstall libinstall inclinstall libainstall sharedinstall +.PHONY: maninstall libinstall inclinstall libainstall sharedinstall desktopinstall .PHONY: frameworkinstall frameworkinstallframework frameworkinstallstructure .PHONY: frameworkinstallmaclib frameworkinstallapps frameworkinstallunixtools .PHONY: frameworkaltinstallunixtools recheck autoconf clean clobber distclean From scott+python-dev at scottdial.com Tue Jul 11 19:30:17 2006 From: scott+python-dev at scottdial.com (Scott Dial) Date: Tue, 11 Jul 2006 13:30:17 -0400 Subject: [Python-Dev] Capabilities / Restricted Execution In-Reply-To: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com> References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com> Message-ID: <44B3E029.5070604@scottdial.com> Phillip J. Eby wrote: > A function's func_closure contains cell objects that hold the > variables. These are readable if you can set the func_closure of some > function of your own. If the overall plan includes the ability to restrict > func_closure setting (or reading) in a restricted interpreter, then you > might be okay. Except this function (__getattribute__) has been trapped inside of a class which does not expose it as an attribute. So, you shouldn't be able to get to the func_closure attribute of the __getattribute__ function for an instance of the Guard class. I can't come up with a way to defeat this protection, at least. If you have a way, then I'd be interested to hear it. -- Scott Dial scott at scottdial.com scodial at indiana.edu From rowen at cesmail.net Tue Jul 11 19:32:01 2006 From: rowen at cesmail.net (Russell E. Owen) Date: Tue, 11 Jul 2006 10:32:01 -0700 Subject: [Python-Dev] get for lists and tuples? Message-ID: I'd like to have the get method available for lists and tuples. (I figured this must have been discussed before but can't recall it and didn't turn anything up on google). It's obviously not a use-all-the-time method (or it'd already be there), but I find myself wanting it often enough to justify it in my own mind (and curse this omission, relative to dict). Basically I run into it when parsing data of variable length (where the extra elements have some obvious default value), including config files, sys.argv (for simple command scripts), that sort of thing. Yes a 4-liner does the job, but "get" would be a much clearer way to write it. Anyway, I'm just testing the waters. If it's not heresy then I'd like to do what I can to make it happen. -- Russell From fdrake at acm.org Tue Jul 11 19:33:07 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue, 11 Jul 2006 13:33:07 -0400 Subject: [Python-Dev] Minor: Unix icons for 2.5? In-Reply-To: References: <200607120305.45714.anthony@interlink.com.au> Message-ID: <200607111333.07453.fdrake@acm.org> Anthony Baxter wrote: > There's an open PEP-356 issue for "update the icons to the newer > shinier ones" for Unix. As far as I can see, there's the 14x15 GIF > images used for Idle and the documentation. Note that for me at least, > idle comes up without an icon _anyway_. A pyfav.(gif|png) replacement would be quite welcome! On Tuesday 11 July 2006 13:25, Georg Brandl wrote: > In case we add a Python .desktop file (as proposed in patch #1353344), > we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in > is attached. I know the .desktop files have become fairly standard, but are these our responsibility or does that rest with the distributions/integrators? (I'm not objecting, but I'm not sure what the right thing really is since Python is an interpreter, not a desktop application.) -Fred -- Fred L. Drake, Jr. From alexander.belopolsky at gmail.com Tue Jul 11 19:34:49 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 11 Jul 2006 17:34:49 +0000 (UTC) Subject: [Python-Dev] Doctest and Footnotes References: <44B31CA7.7080906@zope.com> Message-ID: Benji York zope.com> writes: > > Here's the idea: when a footnote is referenced in prose, execute the > code associated with the footnote at that point. For example: > Another natural place for the referenced code is the __test__ dictionary. Using that has an advantage of not clobbering the display in the default pydoc viewer. From g.brandl at gmx.net Tue Jul 11 19:39:56 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Jul 2006 19:39:56 +0200 Subject: [Python-Dev] get for lists and tuples? In-Reply-To: References: Message-ID: Russell E. Owen wrote: > I'd like to have the get method available for lists and tuples. (I > figured this must have been discussed before but can't recall it and > didn't turn anything up on google). > > It's obviously not a use-all-the-time method (or it'd already be there), > but I find myself wanting it often enough to justify it in my own mind > (and curse this omission, relative to dict). > > Basically I run into it when parsing data of variable length (where the > extra elements have some obvious default value), including config files, > sys.argv (for simple command scripts), that sort of thing. > > Yes a 4-liner does the job, but "get" would be a much clearer way to > write it. A 4-liner? x = (list[i:i+1] or (default,))[0] is just one line ;) Honestly, often enough you can work around with slices in one or the other way. > Anyway, I'm just testing the waters. If it's not heresy then I'd like to > do what I can to make it happen. IMO there's almost no chance this can go into 2.5. Georg From g.brandl at gmx.net Tue Jul 11 19:42:53 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Jul 2006 19:42:53 +0200 Subject: [Python-Dev] Minor: Unix icons for 2.5? In-Reply-To: <200607111333.07453.fdrake@acm.org> References: <200607120305.45714.anthony@interlink.com.au> <200607111333.07453.fdrake@acm.org> Message-ID: Fred L. Drake, Jr. wrote: > Anthony Baxter wrote: > > There's an open PEP-356 issue for "update the icons to the newer > > shinier ones" for Unix. As far as I can see, there's the 14x15 GIF > > images used for Idle and the documentation. Note that for me at least, > > idle comes up without an icon _anyway_. > > A pyfav.(gif|png) replacement would be quite welcome! > > On Tuesday 11 July 2006 13:25, Georg Brandl wrote: > > In case we add a Python .desktop file (as proposed in patch #1353344), > > we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in > > is attached. > > I know the .desktop files have become fairly standard, but are these our > responsibility or does that rest with the distributions/integrators? (I'm > not objecting, but I'm not sure what the right thing really is since Python > is an interpreter, not a desktop application.) I'm also not very sure, as I myself would never start python via a menu entry. But for newcomers, it could be as it is for Windows users: A menu entry that starts the interpreter in a terminal window, and maybe one for IDLE too. Georg From anthony at interlink.com.au Tue Jul 11 19:50:47 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 12 Jul 2006 03:50:47 +1000 Subject: [Python-Dev] get for lists and tuples? In-Reply-To: References: Message-ID: <200607120350.51728.anthony@interlink.com.au> On Wednesday 12 July 2006 03:39, Georg Brandl wrote: > > Anyway, I'm just testing the waters. If it's not heresy then I'd > > like to do what I can to make it happen. > > IMO there's almost no chance this can go into 2.5. "almost"? I'll go you one better. No way at all will it be in 2.5. And I'd be a firm -1 on adding in to 2.6 as well. It encourages bad coding. You shouldn't be searching lists and tuples like that unless you know what you're doing. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From benji at zope.com Tue Jul 11 19:57:23 2006 From: benji at zope.com (Benji York) Date: Tue, 11 Jul 2006 13:57:23 -0400 Subject: [Python-Dev] Doctest and Footnotes In-Reply-To: References: <44B31CA7.7080906@zope.com> Message-ID: <44B3E683.2060004@zope.com> Alexander Belopolsky wrote: > Benji York zope.com> writes: >>Here's the idea: when a footnote is referenced in prose, execute the >>code associated with the footnote at that point. For example: >> > > Another natural place for the referenced code is the __test__ dictionary. > Using that has an advantage of not clobbering the display in the default > pydoc viewer. I'm not quite sure what you're suggesting. A guess: put the code that isn't to be seen in the __test__ dict with a string key being the name of the footnote? I don't think a ReST processor would like that much. It would see references to footnotes that are never defined. Or perhaps you're suggesting a non-ReST mechanism for the references? Also, for many of the use-cases we have, we do want the code in the test, just not in such a prominent place, and not repeated more than once. -- Benji York Senior Software Engineer Zope Corporation From brett at python.org Tue Jul 11 19:56:47 2006 From: brett at python.org (Brett Cannon) Date: Tue, 11 Jul 2006 10:56:47 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <44B34536.7040907@acm.org> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AF26A4.2040201@acm.org> <44B074C6.3070701@acm.org> <44B34536.7040907@acm.org> Message-ID: On 7/10/06, Talin wrote: > > Brett Cannon wrote: > > Using a factory method callback, one could store the PyCodeObject in a C > > proxy object that just acts as a complete delegate, forwarding all > method > > calls to the internally stored PyCodeObject. That would work. > > > > > > For this initial implementation, though, I am not going to try to > support > > this. We can always add support like this later since it doesn't > > fundamentally break or change anything that is already planned. Let's > > focus > > on getting even more basic stuff working before we start to get too > fancy. > > > > -Brett > > Thinking about this some more, I've come up with a simpler idea for a > generic wrapper class. The wrapper consists of two parts: a decorator to > indicate that a given method is 'public', and a C 'guard' wrapper that > insures that only 'public' members can be accessed. > > So for example: > > from Sandbox import guard, public > > class FileProxy: > # Public method, no decorator Is that supposed to say "no decorator", even though there is one? @public > def read( length ): > ... > > @public > def seek( offset ): > ... > > # Private method, no decorator > def open( path ): > ... > > # Construct an instance of FileProxy, and wrap a > # guard object around it. Any attempt to access a non-public > # attribute will raise an exception (or perhaps simply report > # that the attribute doesn't exist.) > fh = guard( FileProxy() ) But wouldn't you still need to protect things like object.__metaclasses__() so that you can't get access to the class and mutate it? And if you use a custom metaclass to avoid having object expose the reference you then would need to protect the metaclass. I guess I would need to see an implementation to see if there is any way to get around security protections since it is still Python code. Now, from my point of view this *is* 'the basic stuff'. In other words, > this right here is the fundamental sandbox mechanism, and everything > else is built on top of it. > > Now, the C 'guard' function is only a low-level means to insure that > no-one can mess with the object; It is not intended to be the actual > restriction policy itself. Those are placed in the wrapper classes, just > as in your proposed scheme. > > (This goes back to my basic premise, that a simple "yes/no" security > feature can be used to build up much more complex and subtle security > features.) Yeah, this tends to be true. -Brett The only real complexity of this, as I see it, is that methods to > references will have to be themselves wrapped. > > In other words, if I say 'fh.read' without the (), what I get back can't > be the actual read function object - that would be too easy to fiddle > with. What I'd have to get is a wrapped version of the method that is a > callable. > > One relatively simply way to deal with this is to have the 'public' > decorator create a C wrapper for the function object, and store it as an > attribute of the function. The class wrapper then simply looks up the > attribute, and if it has an attribute wrapper, returns that, otherwise > it fails. > > (Although, I've often wished for Python to have a variant of __call__ > that could be used to override individual methods, i.e.: > > __call_method__( self, methodname, *args ) > > This would make the guard wrapper much easier to construct, since we > could restrict the methods only to being called, and not allow > references to methods to be taken.) > > -- Talin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060711/b11958de/attachment.html From pje at telecommunity.com Tue Jul 11 20:09:37 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 14:09:37 -0400 Subject: [Python-Dev] Capabilities / Restricted Execution In-Reply-To: <44B3E029.5070604@scottdial.com> References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com> <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060711140809.03aa2818@sparrow.telecommunity.com> At 01:30 PM 7/11/2006 -0400, Scott Dial wrote: >Phillip J. Eby wrote: > > A function's func_closure contains cell objects that hold the > > variables. These are readable if you can set the func_closure of some > > function of your own. If the overall plan includes the ability to > restrict > > func_closure setting (or reading) in a restricted interpreter, then you > > might be okay. > >Except this function (__getattribute__) has been trapped inside of a >class which does not expose it as an attribute. So, you shouldn't be >able to get to the func_closure attribute of the __getattribute__ >function for an instance of the Guard class. That doesn't matter, because it's the *returned* function's func_closure that's at issue. That is, proxy_attr.func_closure[0] is the cell for the 'attr' value. From talin at acm.org Tue Jul 11 20:10:34 2006 From: talin at acm.org (Talin) Date: Tue, 11 Jul 2006 11:10:34 -0700 Subject: [Python-Dev] Capabilities / Restricted Execution In-Reply-To: <44B3E029.5070604@scottdial.com> References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com> <44B3E029.5070604@scottdial.com> Message-ID: <44B3E99A.4080000@acm.org> Scott Dial wrote: > Phillip J. Eby wrote: > >>A function's func_closure contains cell objects that hold the >>variables. These are readable if you can set the func_closure of some >>function of your own. If the overall plan includes the ability to restrict >>func_closure setting (or reading) in a restricted interpreter, then you >>might be okay. > > > Except this function (__getattribute__) has been trapped inside of a > class which does not expose it as an attribute. So, you shouldn't be > able to get to the func_closure attribute of the __getattribute__ > function for an instance of the Guard class. I can't come up with a way > to defeat this protection, at least. If you have a way, then I'd be > interested to hear it. I've thought of several ways to break it already. Some are repairable, I'm not sure that they all are. For example, neither of the following statements blows up: print t2.get_name.func_closure[0] print object.__getattribute__( t2, '__dict__' ) Still, its perhaps a useful basis for experimentation. -- Talin From alexander.belopolsky at gmail.com Tue Jul 11 20:12:19 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 11 Jul 2006 14:12:19 -0400 Subject: [Python-Dev] Doctest and Footnotes In-Reply-To: <44B3E683.2060004@zope.com> References: <44B31CA7.7080906@zope.com> <44B3E683.2060004@zope.com> Message-ID: On 7/11/06, Benji York wrote: [snip] > I'm not quite sure what you're suggesting. A guess: put the code that > isn't to be seen in the __test__ dict with a string key being the name > of the footnote? That's right. > I don't think a ReST processor would like that much. > It would see references to footnotes that are never defined. Or perhaps > you're suggesting a non-ReST mechanism for the references? > I don't know how ReST processor is used. If you just filter the output of pydoc through a ReST processor, then you are right about undefined references. If, however, ReST processing is implemented inside pydoc, I don't see any problem in implementing __test__ lookup. > Also, for many of the use-cases we have, we do want the code in the > test, just not in such a prominent place, and not repeated more than once. I my use-cases, testing code is clobbering the documentation, but there is no easy way to move it outside of the docstrings without breaking the order of evaluation. I don't use a ReST processor, by I can read ReST formatted text with little difficulty. I would greatly appreciate if I could clean the docstrings without loosing the tests. BTW, another feature that I would greatly appreciate would be a unittest wrapper which makes each docstring in a separate test case. Also __new__ and __init__ method docstrings is the natural place to put set-up code. From fred at zope.com Tue Jul 11 20:17:00 2006 From: fred at zope.com (Fred Drake) Date: Tue, 11 Jul 2006 14:17:00 -0400 Subject: [Python-Dev] Doctest and Footnotes In-Reply-To: References: <44B31CA7.7080906@zope.com> <44B3E683.2060004@zope.com> Message-ID: <200607111417.01500.fred@zope.com> On Tuesday 11 July 2006 14:12, Alexander Belopolsky wrote: > Also __new__ and __init__ method docstrings is the natural place to > put set-up code. Maybe, if all the tests required the same setup code. That's often not the case. -Fred -- Fred L. Drake, Jr. Zope Corporation From g.brandl at gmx.net Tue Jul 11 20:17:35 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Jul 2006 20:17:35 +0200 Subject: [Python-Dev] get for lists and tuples? In-Reply-To: <200607120350.51728.anthony@interlink.com.au> References: <200607120350.51728.anthony@interlink.com.au> Message-ID: Anthony Baxter wrote: > On Wednesday 12 July 2006 03:39, Georg Brandl wrote: >> > Anyway, I'm just testing the waters. If it's not heresy then I'd >> > like to do what I can to make it happen. >> >> IMO there's almost no chance this can go into 2.5. > > "almost"? > > I'll go you one better. No way at all will it be in 2.5. And I'd be a > firm -1 on adding in to 2.6 as well. It encourages bad coding. You > shouldn't be searching lists and tuples like that unless you know > what you're doing. NB: I did not support adding that method, only notified the OP of the 2.5 situation. Georg From rhettinger at ewtllc.com Tue Jul 11 20:19:48 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Tue, 11 Jul 2006 11:19:48 -0700 Subject: [Python-Dev] get for lists and tuples? In-Reply-To: References: Message-ID: <44B3EBC4.3090703@ewtllc.com> Russell E. Owen wrote: >I'd like to have the get method available for lists and tuples. (I >figured this must have been discussed before but can't recall it and >didn't turn anything up on google). > >It's obviously not a use-all-the-time method (or it'd already be there), >but I find myself wanting it often enough to justify it in my own mind >(and curse this omission, relative to dict). > >Basically I run into it when parsing data of variable length (where the >extra elements have some obvious default value), including config files, >sys.argv (for simple command scripts), that sort of thing. > > How about: optarg = argv[4] if len(argv)>4 else 'default' >Yes a 4-liner does the job, but "get" would be a much clearer way to >write it. > >Anyway, I'm just testing the waters. If it's not heresy then I'd like to >do what I can to make it happen. > > Perhaps in an alternate universe ;-) Raymond From pje at telecommunity.com Tue Jul 11 20:29:35 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 14:29:35 -0400 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <44B34536.7040907@acm.org> <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AF26A4.2040201@acm.org> <44B074C6.3070701@acm.org> <44B34536.7040907@acm.org> Message-ID: <5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com> At 10:56 AM 7/11/2006 -0700, Brett Cannon wrote: >On 7/10/06, Talin <talin at acm.org> wrote: >>(Although, I've often wished for Python to have a variant of __call__ >>that could be used to override individual methods, i.e.: >> >> __call_method__( self, methodname, *args ) As with so many other things in this discussion, this was already invented by the Zope folks just shy of a decade ago. ;-) __call_method__ is actually a feature of ExtensionClasses, although you can of course implement it yourself now atop new-style classes. For other things that Zope has already done in the area of restricted execution R&D, see: http://svn.zope.org/Zope3/trunk/src/zope/security/untrustedinterpreter.txt?view=auto The surrounding directory has other documents regarding their approach. I haven't been following this discussion closely, but a lot of the things mentioned in this thread seem to have a lot in common with stuff the Zope folks have had in production for "untrusted" Python execution for some time now, working with current versions of Python. It would be a shame to reinvent all the same wheels, especially since their code is nicely documented complete with extensive doctests and explanations of the approach. From pje at telecommunity.com Tue Jul 11 20:32:17 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 14:32:17 -0400 Subject: [Python-Dev] Doctest and Footnotes In-Reply-To: References: <44B3E683.2060004@zope.com> <44B31CA7.7080906@zope.com> <44B3E683.2060004@zope.com> Message-ID: <5.1.1.6.0.20060711143002.03e9c800@sparrow.telecommunity.com> At 02:12 PM 7/11/2006 -0400, Alexander Belopolsky wrote: >On 7/11/06, Benji York wrote: >[snip] > > I'm not quite sure what you're suggesting. A guess: put the code that > > isn't to be seen in the __test__ dict with a string key being the name > > of the footnote? > >That's right. > > > I don't think a ReST processor would like that much. > > It would see references to footnotes that are never defined. Or perhaps > > you're suggesting a non-ReST mechanism for the references? > > > >I don't know how ReST processor is used. If you just filter the >output of pydoc through a ReST processor, then you are right about >undefined references. A common use case for doctests in Zope is separate text files that are used as documentation. I also use this approach for generating documentation myself, e.g.: http://chandler.osafoundation.org/docs/0.7/parcel-schema-guide.html http://chandler.osafoundation.org/docs/0.7/running-code-at-startup.html http://peak.telecommunity.com/DevCenter/BytecodeAssembler To name just a few. From brett at python.org Tue Jul 11 20:35:23 2006 From: brett at python.org (Brett Cannon) Date: Tue, 11 Jul 2006 11:35:23 -0700 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: <5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com> References: <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AF26A4.2040201@acm.org> <44B074C6.3070701@acm.org> <44B34536.7040907@acm.org> <5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com> Message-ID: On 7/11/06, Phillip J. Eby wrote: > > At 10:56 AM 7/11/2006 -0700, Brett Cannon wrote: > >On 7/10/06, Talin <talin at acm.org> wrote: > >>(Although, I've often wished for Python to have a variant of __call__ > >>that could be used to override individual methods, i.e.: > >> > >> __call_method__( self, methodname, *args ) > > As with so many other things in this discussion, this was already invented > by the Zope folks just shy of a decade ago. ;-) __call_method__ is > actually a feature of ExtensionClasses, although you can of course > implement it yourself now atop new-style classes. > > For other things that Zope has already done in the area of restricted > execution R&D, see: > > > http://svn.zope.org/Zope3/trunk/src/zope/security/untrustedinterpreter.txt?view=auto > > The surrounding directory has other documents regarding their approach. > > I haven't been following this discussion closely, but a lot of the things > mentioned in this thread seem to have a lot in common with stuff the Zope > folks have had in production for "untrusted" Python execution for some > time > now, working with current versions of Python. It would be a shame to > reinvent all the same wheels, especially since their code is nicely > documented complete with extensive doctests and explanations of the > approach. Taking a proxy approach is just being discussed; it has not been decided as the proper solution. I have been considering just preventing direct 'file' access and using open() to act as a delegate for opening files. That approach has nothing to do with proxies. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060711/88cfec06/attachment.html From alexander.belopolsky at gmail.com Tue Jul 11 20:37:37 2006 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 11 Jul 2006 14:37:37 -0400 Subject: [Python-Dev] Doctest and Footnotes In-Reply-To: <200607111417.01500.fred@zope.com> References: <44B31CA7.7080906@zope.com> <44B3E683.2060004@zope.com> <200607111417.01500.fred@zope.com> Message-ID: On 7/11/06, Fred Drake wrote: > On Tuesday 11 July 2006 14:12, Alexander Belopolsky wrote: > > Also __new__ and __init__ method docstrings is the natural place to > > put set-up code. > > Maybe, if all the tests required the same setup code. That's often not the > case. That's true, but you cannot test an object method without creating the object first. For the main cases you would want the object definition close to the test for the benefit of people who do pydoc Foo.bar, but for corner cases it is better to have a predefined set of exotic objects available under descriptive names. I am also advocating __init__ docstring because in my use-cases it rarely much of documentation because object construction is documented in the class docstring, therefore test set-up code placed in __init__ docstring is unlikely to clobber often-used documentation. From jjl at pobox.com Tue Jul 11 20:43:22 2006 From: jjl at pobox.com (John J Lee) Date: Tue, 11 Jul 2006 19:43:22 +0100 (GMT Standard Time) Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt In-Reply-To: <44B3ADE2.7040108@ofai.at> References: <44B3ADE2.7040108@ofai.at> Message-ID: On Tue, 11 Jul 2006, Stefan Rank wrote: > urllib.quote fails on unicode strings and in an unhelpful way:: [...] > >>> urllib.quote(u'a\xf1a') > Traceback (most recent call last): > File "", line 1, in ? > File "C:\Python24\lib\urllib.py", line 1117, in quote > res = map(safe_map.__getitem__, s) > KeyError: u'\xf1' More helpful than silently producing the wrong answer. [...] > I suggest to add (after 2.5 I assume) one of the following to the > beginning of urllib.quote to either fail early and consistently on > unicode arguments and improve the error message:: > > if isinstance(s, unicode): > raise TypeError("quote needs a byte string argument, not unicode," > " use `argument.encode('utf-8')` first.") Won't this break existing code that catches the KeyError, for no big benefit? If nobody is yet sure what the Right Thing is (see below), I think we should not change this yet. > or to do The Right Thing (tm), which is utf-8 encoding:: > > if isinstance(s, unicode): > s = s.encode('utf-8') > > as suggested in > http://www.w3.org/International/O-URL-code.html > and rfc3986. You seem quite confident of that. You may be correct, but have you read all of the following? (not trying to claim superior knowledge by asking that, I just dunno what the right thing is yet: I haven't yet read RFC 2617 or got my head around what the unicode issues are or how they should apply to the Python stdlib) http://www.ietf.org/rfc/rfc2617.txt http://www.ietf.org/rfc/rfc2616.txt http://en.wikipedia.org/wiki/Percent-encoding http://mail.python.org/pipermail/python-dev/2004-September/048944.html Also note the recent discussions here about a module named "uriparse" or "urischemes", which fits in to this somewhere. It would be good to make all the following changes in a single Python release (2.6, with luck): - extend / modify urllib and urllib2 to handle unicode input - address the urllib.quote issue you raise above (+ consider the other utility functions in that module) - add the urischemes module In summary, I agree that your suggested fix (and all of the rest I refer to above) should wait for 2.6, unless somebody (Martin?) who understands all these issues is quite confident your suggested change is OK. Presumably the release managers wouldn't allow it in 2.5 anyway. John From fred at zope.com Tue Jul 11 20:52:29 2006 From: fred at zope.com (Fred Drake) Date: Tue, 11 Jul 2006 14:52:29 -0400 Subject: [Python-Dev] Doctest and Footnotes In-Reply-To: References: <44B31CA7.7080906@zope.com> <200607111417.01500.fred@zope.com> Message-ID: <200607111452.30118.fred@zope.com> On Tuesday 11 July 2006 14:37, Alexander Belopolsky wrote: > That's true, but you cannot test an object method without creating the > object first. True. How the object is created can vary; if the creation affects the expected behavior in any way, you'll need be careful about how the constructor is called for that test. In fact, __init__ isn't always the desired constructor; some other class method might be used. > For the main cases you would want the object definition > close to the test for the benefit of people who do pydoc Foo.bar, but > for corner cases it is better to have a predefined set of exotic > objects available under descriptive names. This is subjective. That's certainly one way of organizing the tests, and I've no problem with it, but some of us use a more narrative approach, and we don't want our test structure to be constrained by the layout of the source code. Gary's idea about using footnotes to aid in structuring tests is only about enabling a particular approach to structure, not about requiring it. If it doesn't fit your needs, there's certainly no need to use it. For more elaborate test structures, I suspect it will prove quite useful. I also expect it will be more helpful for tests in separate text files than for tests embedded in source code. -Fred -- Fred L. Drake, Jr. Zope Corporation From pje at telecommunity.com Tue Jul 11 20:59:10 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 11 Jul 2006 14:59:10 -0400 Subject: [Python-Dev] In defense of Capabilities [was: doc for new restricted execution design for Python] In-Reply-To: References: <5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com> <20060705155149.eipxm6j6oycgkcs8@login.werra.lunarpages.com> <44AF26A4.2040201@acm.org> <44B074C6.3070701@acm.org> <44B34536.7040907@acm.org> <5.1.1.6.0.20060711141257.03e8de00@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060711145232.03c8c0e0@sparrow.telecommunity.com> At 11:35 AM 7/11/2006 -0700, Brett Cannon wrote: >On 7/11/06, Phillip J. Eby ><pje at telecommunity.com> wrote: >>At 10:56 AM 7/11/2006 -0700, Brett Cannon wrote: >> >On 7/10/06, Talin >> <talin at acm.org> wrote: >> >>(Although, I've often wished for Python to have a variant of __call__ >> >>that could be used to override individual methods, i.e.: >> >> >> >> __call_method__( self, methodname, *args ) >> >>As with so many other things in this discussion, this was already invented >>by the Zope folks just shy of a decade ago. ;-) __call_method__ is >>actually a feature of ExtensionClasses, although you can of course >>implement it yourself now atop new-style classes. >> >>For other things that Zope has already done in the area of restricted >>execution R&D, see: >> >>http://svn.zope.org/Zope3/trunk/src/zope/security/untrustedinterpreter.txt?view=auto >> >>The surrounding directory has other documents regarding their approach. >> >>I haven't been following this discussion closely, but a lot of the things >>mentioned in this thread seem to have a lot in common with stuff the Zope >>folks have had in production for "untrusted" Python execution for some time >>now, working with current versions of Python. It would be a shame to >>reinvent all the same wheels, especially since their code is nicely >>documented complete with extensive doctests and explanations of the approach. > >Taking a proxy approach is just being discussed; it has not been decided >as the proper solution. My point is more to the study of the requirements for untrusted execution, and what options are open within the existing CPython architecture. Many aspects of the Zope untrusted interpreter do *not* involve proxies; there are quite a few simple types that Zope's untrusted interpreter allows direct access to. See the section on "Basic objects" in the link above. From jimjjewett at gmail.com Tue Jul 11 22:40:10 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 11 Jul 2006 16:40:10 -0400 Subject: [Python-Dev] PEP 356: python.org/sf/1515343 resolution Message-ID: python.org/sf/1515343 fixes a regression against 2.4, which masks exceptions, and should therefore go in before release. The tracker has a patch to the test cases, and two alternative fixes. (One minimal, the other less ugly.) Since this only affects string exceptions with a value, it might be acceptable to just say "won't ever fix", but then it should be added to the upgrade news. -jJ From martin at v.loewis.de Tue Jul 11 23:16:21 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Jul 2006 23:16:21 +0200 Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt In-Reply-To: <44B3ADE2.7040108@ofai.at> References: <44B3ADE2.7040108@ofai.at> Message-ID: <44B41525.3040004@v.loewis.de> Stefan Rank wrote: > I suggest to add (after 2.5 I assume) one of the following to the > beginning of urllib.quote to either fail early and consistently on > unicode arguments and improve the error message:: > > if isinstance(s, unicode): > raise TypeError("quote needs a byte string argument, not unicode," > " use `argument.encode('utf-8')` first.") > > or to do The Right Thing (tm), which is utf-8 encoding:: The right thing to do is IRIs. This is more complicated than encoding the Unicode string as UTF-8, though: for the host part of the URL, you have to encode it with IDNA (and there are additional complicated rules in place, e.g. when the Unicode string already contains %). Contributions are welcome, as long as they fix this entire issue "for good" (i.e. in all URL-processing code, and considering all relevant RFCs). Regards, Martin From tjreedy at udel.edu Tue Jul 11 23:23:51 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 11 Jul 2006 17:23:51 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) References: Message-ID: "Boris Borcic" wrote in message news:e90j6o$fij$1 at sea.gmane.org... > I agree with you (and argued it in "scopes vs augmented assignment vs > sets" > recently) that mutating would be sufficient /if/ the compiler would view > augmented assignment as mutations operators : Mutation is an operation on objects. Binding is an operation on namespaces. The difference between objects and namespaces (and the actions thereupon) is fundamental to Python. Asking the interpreter to view one thing as something else which it isn't can only lead to more confusion. In particular, asking that arithmetic operations on immutable numbers be seen as mutations seems wacky to me. > which it doesn't as far as concerns scopes where a variable > appears as target only of /augmented/ assignments. The interpreter/compiler, as far as I can think, never views binding as mutation, nor should it. The request that it do so makes me wonder whether it might have been a mistake to allow mutable objects in an augmented assignment to choose to implement the associated operation as an in-place mutation. Terry Jan Reedy From martin at v.loewis.de Tue Jul 11 23:28:58 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Jul 2006 23:28:58 +0200 Subject: [Python-Dev] Minor: Unix icons for 2.5? In-Reply-To: References: <200607120305.45714.anthony@interlink.com.au> Message-ID: <44B4181A.1070506@v.loewis.de> Georg Brandl wrote: > In case we add a Python .desktop file (as proposed in patch #1353344), > we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in > is attached. Independent of whether this should be done at all, I have a comment on the patch. Instead of invoking sed, I would use autoconf machinery here > +Exec=python_VER_ %f This would become Exec=python at VERSION@@EXEEXT@ %f > + sed 's/_VER_/$(VERSION)$(EXE)/' $(srcdir)/Misc/python.desktop.in > > $(srcdir)/Misc/python.desktop This could go away, and configure should grow AC_CONFIG_FILES(Makefile.pre Modules/Setup.config Misc/python.desktop) Regards, Martin From bjourne at gmail.com Tue Jul 11 23:39:33 2006 From: bjourne at gmail.com (=?ISO-8859-1?Q?BJ=F6rn_Lindqvist?=) Date: Tue, 11 Jul 2006 23:39:33 +0200 Subject: [Python-Dev] Minor: Unix icons for 2.5? In-Reply-To: <200607111333.07453.fdrake@acm.org> References: <200607120305.45714.anthony@interlink.com.au> <200607111333.07453.fdrake@acm.org> Message-ID: <740c3aec0607111439h691c4450i9aa52fa00eb73072@mail.gmail.com> > I know the .desktop files have become fairly standard, but are these our > responsibility or does that rest with the distributions/integrators? (I'm > not objecting, but I'm not sure what the right thing really is since Python > is an interpreter, not a desktop application.) The same anal argument could of course be made for every piece of software. Is Emacs a desktop application or a lisp interpreter? Is vim a desktop app? Are games? There is no reason add yet another task to to multiple distributors/integrators (that they won't do) when the problem can be fixed at the source. -- mvh Bj?rn From g.brandl at gmx.net Tue Jul 11 23:43:24 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 11 Jul 2006 23:43:24 +0200 Subject: [Python-Dev] Minor: Unix icons for 2.5? In-Reply-To: <44B4181A.1070506@v.loewis.de> References: <200607120305.45714.anthony@interlink.com.au> <44B4181A.1070506@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Georg Brandl wrote: >> In case we add a Python .desktop file (as proposed in patch #1353344), >> we'll need some PNGs in /usr/share/icons. A patch for Makefile.pre.in >> is attached. > > Independent of whether this should be done at all, I have a comment on > the patch. Instead of invoking sed, I would use autoconf machinery here > >> +Exec=python_VER_ %f > > This would become > > Exec=python at VERSION@@EXEEXT@ %f > >> + sed 's/_VER_/$(VERSION)$(EXE)/' $(srcdir)/Misc/python.desktop.in > >> $(srcdir)/Misc/python.desktop > > This could go away, and configure should grow > > AC_CONFIG_FILES(Makefile.pre Modules/Setup.config Misc/python.desktop) Ah, thanks. I'm not very fit with autoconf magic. Georg From michael at ellerman.id.au Wed Jul 12 02:30:17 2006 From: michael at ellerman.id.au (Michael Ellerman) Date: Wed, 12 Jul 2006 10:30:17 +1000 Subject: [Python-Dev] User's complaints In-Reply-To: References: Message-ID: On 7/5/06, Neal Norwitz wrote: > On 7/4/06, Guido van Rossum wrote: > > > > From actual users of > > the language I get more complaints about the breakneck speed of > > Python's evolution than about the brokenness of the current language. > > Guido, > > I'm really interested in your perspective here. I assume you hear far > more "average" complaints from Joe Random User. Can you help give the > rest of us an idea about the top 10 complaints/problems people have? > I realize this will be subjective, that's ok. Perhaps we should try > to focus our energies on some of these issues. Well here's one I stumbled across the other day. I don't know if it's legit, but it's still bad PR: http://www.gbch.net/gjb/blog/software/discuss/python-sucks.html For the impatient, he's not at all bothered about the lack of obscure language feature X. cheers From brett at python.org Sat Jul 8 00:51:11 2006 From: brett at python.org (Brett Cannon) Date: Fri, 7 Jul 2006 15:51:11 -0700 Subject: [Python-Dev] second draft of sandboxing design doc Message-ID: OK, lots of revisions. The approach to handling 'file' I have left up in the air. Biggest change is switching to "unprotected" and "sandboxed" for terms when referring to the interpreter. Also added a Threat Model section to explain assumptions about the basics of the interpreter. Hopefully it is also more clear about the competing approaches for dealing with 'file'. I am planning on starting work next week on implementation, but I will start with the least controversial and work my way up. One thing that is open that I would like some feedback on immediately is whether people would rather pass in PyObjects or C level types to the API. The latter makes implementing the Python wrapper much easier, but makes embedding a more heavy-handed. People have a preference or think that the Python API will be used more often than the C one? I am leaning towards making the C API simpler by using C level types (const char *, etc.) and just deal with the Python wrappings requiring more rejiggering between types. Once again, I have a branch going (bcannon-sandboxing) where the work is going to be done and where this doc lives. I don't plan on doing another post of this doc until another major revision. --------------------------------------------------------------------------------------------------------------- Restricted Execution for Python ####################################### About This Document ============================= This document is meant to lay out the general design for re-introducing a sandboxing model for Python. This document should provide one with enough information to understand the goals for sandboxing, what considerations were made for the design, and the actual design itself. Design decisions should be clear and explain not only why they were chosen but possible drawbacks from taking a specific approach. If any of the above is found not to be true, please email me at brett at python.org and let me know what problems you are having with the document. XXX TO DO ============================= * threading needs protection? * python-dev convince me that hiding 'file' possible? + based on that, handle code objects + also decide how to handle sockets + perhaps go with crippling but try best effort on hiding reference and if best effort holds up eventually shift over to capabilities system * resolve to IP at call time to prevent DNS man-in-the-middle attacks when allowing a specific host name? * what network info functions are allowed by default? * does the object.__subclasses__() trick work across interpreters, or is it unique per interpreter? * figure out default whitelist of extension modules * check default accessible objects for file path exposure * helper functions to get at StringIO instances for stdin, stdout, and friends? * decide on what type of objects (e.g., PyStringObject or const char *) are to be passed in * all built-ins properly protected? * exactly how to tell whether argument to open() is a path, IP, or host name (third argument, 'n' prefix for networking, format of path, ...) * API at the Python level * for extension module protection, allow for wildcard allowance (e.g., ``xml.*``) Goal ============================= A good sandboxing model provides enough protection to prevent malicious harm to come to the system, and no more. Barriers should be minimized so as to allow most code that does not do anything that would be regarded as harmful to run unmodified. But the protections need to be thorough enough to prevent any unintended changes or information of the system to come about. An important point to take into consideration when reading this document is to realize it is part of my (Brett Cannon's) Ph.D. dissertation. This means it is heavily geared toward sandboxing when the interpreter is working with Python code embedded in a web page as viewed in Firefox. While great strides have been taken to keep the design general enough so as to allow all previous uses of the 'rexec' module [#rexec]_ to be able to use the new design, it is not the focused goal. This means if a design decision must be made for the embedded use case compared to sandboxing Python code in a pure Python application, the former will win out over the latter. Throughout this document, the term "resource" is used to represent anything that deserves possible protection. This includes things that have a physical representation (e.g., memory) to things that are more abstract and specific to the interpreter (e.g., sys.path). When referring to the state of an interpreter, it is either "unprotected" or "sandboxed". A unprotected interpreter has no restrictions imposed upon any resource. A sandboxed interpreter has at least one, possibly more, resource with restrictions placed upon it to prevent unsafe code that is running within the interpreter to cause harm to the system. .. contents:: Use Cases ///////////////////////////// All use cases are based on how many sandboxed interpreters are running in a single process and whether an unprotected interpreter is also running. The use cases can be broken down into two categories: when the interpreter is embedded and only using sandboxed interpreters, and when pure Python code is running in an unprotected interpreter and uses sandboxed interpreters. When the Interpreter Is Embedded ================================ Single Sandboxed Interpreter ---------------------------- This use case is when an application embeds the interpreter and never has more than one interpreter running which happens to be sandboxed. Multiple Sandboxed Interpreters ------------------------------- When multiple interpreters, all sandboxed at varying levels, need to be running within a single application. This is the key use case that this proposed design is targeted for. Stand-Alone Python ============================= When someone has written a Python program that wants to execute Python code in an sandboxed interpreter(s). This is the use case that 'rexec' attempted to fulfill. Issues to Consider ============================= Common to all use cases, resources that the interpreter requires to function at a level below user code cannot be exposed to a sandboxed interpreter. For instance, the interpreter might need to stat a file to see if it is possible to import. If the ability to stat a file is not allowed to a sandboxed interpreter, it should not be allowed to perform that action, regardless of whether the interpreter at a level below user code needs that ability. When multiple interpreters are involved (sandboxed or not), not allowing an interpreter to gain access to resources available in other interpreters without explicit permission must be enforced. Resources to Protect ///////////////////////////// It is important to make sure that the proper resources are protected from a sandboxed interpreter. If you don't there is no point to sandboxing. Filesystem =================== All facets of the filesystem must be protected. This means restricting reading and writing to the filesystem (e.g., files, directories, etc.). It should be allowed in controlled situations where allowing access to the filesystem is desirable, but that should be an explicit allowance. There must also be protection to prevent revealing any information about the filesystem. Disclosing information on the filesystem could allow one to infer what OS the interpreter is running on, for instance. Memory =================== Memory should be protected. It is a limited resource on the system that can have an impact on other running programs if it is exhausted. Being able to restrict the use of memory would help alleviate issues from denial-of-service (DoS) attacks on the system. Networking =================== Networking is somewhat like the filesystem in terms of wanting similar protections. You do not want to let unsafe code make socket connections unhindered or accept them to do possibly nefarious things. You also want to prevent finding out information about the network your are connected to. Interpreter =================== One must make sure that the interpreter is not harmed in any way from sandboxed code. This usually takes the form of crashing the program that the interpreter is embedded in or the unprotected interpreter that started the sandbox interpreter. Executing hostile bytecode that might lead to undesirable effects is another possible issue. There is also the issue of taking it over. One should not able to gain escalated privileges in any way without explicit permission. Types of Security /////////////////////////////////////// As with most things, there are multiple approaches one can take to tackle a problem. Security is no exception. In general there seem to be two approaches to protecting resources. Resource Hiding ============================= By never giving code a chance to access a resource, you prevent it from being (ab)used. This is the idea behind resource hiding; you can't misuse something you don't have in the first place. The most common implementation of resource hiding is capabilities. In this type of system a resource's reference acts as a ticket that represents the right to use the resource. Once code has a reference it is considered to have full use of resource that reference represents and no further security checks are directly performed (using delegates and other structured ways one can actually have a security check for each access of a resource, but this is not a default behaviour). As an example, consider the 'file' type as a resource we want to protect. That would mean that we did not want a reference to the 'file' type to ever be accessible without explicit permission. If one wanted to provide read-only access to a temp file, you could have open() perform a check on the permissions of the current interpreter, and if it is allowed to, return a proxy object for the file that only allows reading from it. The 'file' instance for the proxy would need to be properly hidden so that the reference was not reachable from outside so that 'file' access could still be controlled. Python, as it stands now, unfortunately does not work well for a pure capabilities system. Capabilities require the prohibition of certain abilities, such as "direct access to another's private state" [#paradigm regained]_. This obviously is not possible in Python since, at least at the Python level, there is no such thing as private state that is persistent (one could argue that local variables that are not cell variables for lexical scopes are private, but since they do not survive after a function call they are not usable for keeping persistent state). One can hide references at the C level by storing it in the struct for the instance of a type and not providing a function to access that attribute. Python's introspection abilities also do not help make implementing capabilities that much easier. Consider how one could access 'file' even when it is deleted from __builtin__. You can still get to the reference for 'file' through the sequence returned by ``object.__subclasses__()``. Resource Crippling ============================= Another approach to security is to not worry about controlling access to the reference of a resource. One can have a resource perform a security check every time someone tries to use a method on that resource. This pushes the security check to a lower level; from a reference level to the method level. By performing the security check every time a resource's method is called the worry of a specific resource's reference leaking out to insecure code is alleviated. This does add extra overhead, though, by having to do so many security checks. It also does not handle the situation where an unexpected exposure of a type occurs that has not been properly crippled. FreeBSD's jail system provides a protection scheme similar to this. Various system calls allow for basic usage, but knowing or having access to the system call is not enough to grant usage. Every call to a system call requires checking that the proper rights have been granted to the use in order to allow for the system call to perform its action. An even better example in FreeBSD's jail system is its protection of sockets. One can only bind a single IP address to a jail. Any attempt to do more or perform uses with the one IP address that is granted is prevented. The check is performed at every call involving the one granted IP address. Using 'file' as the example again, one could cripple the type so that instantiation is not possible for the type in Python. One could also provide a permission check on each call to a unsafe method call and thus allow the type to be used in normal situations (such as type checking), but still feel safe that illegal operations are not performed. Regardless of which approach you take, you do not need to worry about a reference to the type being exposed unexpectedly since the reference is not the security check but the actual method calls. Comparison of the Two Approaches ================================ >From the perspective of Python, the two approaches differ on what would be the most difficult thing to analyze from a security standpoint: all of the ways to gain access to various types from a sandboxed interpreter with no imports, or finding all of the types that can lead to possibly dangerous actions and thus need to be crippled. Some Python developers, such as Armin Rigo, feel that truly hiding objects in Python is "quite hard" [#armin-hiding]_. This sentiment means that making a pure capabilities system in Python that is secure is not possible as people would continue to find new ways to get a hold of the reference to a protected resource. Others feel that by not going the capabilities route we will be constantly chasing down new types that require crippling. The thinking is that if we cannot control the references for 'file', how are we to know what other types might become exposed later on and thus require more crippling? It essentially comes down to what is harder to do: find all the ways to access the types in Python in a sandboxed interpreter with no imported modules, or to go through the Python code base and find all types that should be crippled? The 'rexec' Module /////////////////////////////////////// The 'rexec' module [#rexec]_ was the original attempt at providing a sandbox environment for Python code to run in. It's design was based on Safe-Tcl which was essentially a capabilities system [#safe-tcl]_. Safe-Tcl allowed you to launch a separate interpreter where its global functions were specified at creation time. This prevented one from having any abilities that were not explicitly provided. For 'rexec', the Safe-Tcl model was tweaked to better match Python's situation. An RExec object represented a sandboxed environment. Imports were checked against a whitelist of modules. You could also restrict the type of modules to import based on whether they were Python source, bytecode, or C extensions. Built-ins were allowed except for a blacklist of built-ins to not provide. One could restrict whether stdin, stdout, and stderr were provided or not on a per-RExec basis. Several other protections were provided; see documentation for the complete list. The ultimate undoing of the 'rexec' module was how access to objects that in normal Python require no imports to reach was handled. Importing modules requires a direct action, and thus can be protected against directly in the import machinery. But for built-ins, they are accessible by default and require no direct action to access in normal Python; you just use their name since they are provided in all namespaces. For instance, in a sandboxed interpreter, one only had to ``del __builtins__`` to gain access to the full set of built-ins. Another way is through using the gc module: ``gc.get_referrers(''.__class__.__bases__[0])[6]['file']``. While both of these could be fixed (the former was a bug in 'rexec' that was fixed and the latter could be handled by not allowing 'gc' to be imported), they are examples of things that do not require proactive actions on the part of the programmer in normal Python to gain access to a resource. This was an unfortunate side-effect of having all of that wonderful reflection in Python. There is also the issue that 'rexec' was written in Python which provides its own problems based on reflection and the ability to modify the code at run-time without security protection. Much has been learned since 'rexec' was written about how Python tends to be used and where security issues tend to appear. Essentially Python's dynamic nature does not lend itself very well to a security implementation that does not require a constant checking of permissions. Threat Model /////////////////////////////////////// Below is a list of what the security implementation assumes, along with what section of this document that addresses that part of the security model (if not already true in Python by default). The term "bare" when in regards to an interpreter means an interpreter that has not performed a single import of a module. Also, all comments refer to a sandboxed interpreter unless otherwise explicitly stated. This list does not address specifics such as how 'file' will be protected or whether memory should be protected. This list is meant to make clear at a more basic level what the security model is assuming is true. * The Python interpreter itself is always trusted. * The Python interpreter cannot be crashed by valid Python source code in a bare interpreter. * Python source code is always considered safe. * Python bytecode is always considered dangerous [`Hostile Bytecode`_]. * C extension modules are inherently considered dangerous [`Extension Module Importation`_]. + Explicit trust of a C extension module is possible. * Sandboxed interpreters running in the same process inherently cannot communicate with each other. + Communication through C extension modules is possible because of the technical need to share extension module instances between interpreters. * Sandboxed interpreters running in the same process inherently cannot share objects. + Sharing objects through C extension modules is possible because of the technical need to share extension module instances between interpreters. * When starting a sandboxed interpreter, it starts with a fresh built-in and global namespace that is not shared with the interpreter that started it. * Objects in the default built-in namespace should be safe to use [`Reading/Writing Files`_, `Stdin, Stdout, and Stderr`_]. + Either hide the dangerous ones or cripple them so they can cause no harm. There are also some features that might be desirable, but are not being addressed by this security model. * Communication in any direction between an unprotected interpreter and a sandboxed interpreter it created. The Proposed Approach /////////////////////////////////////// In light of where 'rexec' succeeded and failed along with what is known about the two main approaches to security and how Python tends to operate, the following is a proposal on how to secure Python for sandboxing. Implementation Details =============================== Support for sandboxed interpreters will require a compilation flag. This allows the more common case of people not caring about protections to not take a performance hit. And even when Python is compiled for sandboxed interpreter restrictions, when the running interpreter *is* unprotected, there will be no accidental triggers of protections. This means that developers should be liberal with the security protections without worrying about there being issues for interpreters that do not need/want the protection. At the Python level, the __sandboxed__ built-in will be set based on whether the interpreter is sandboxed or not. This will be set for *all* interpreters, regardless of whether sandboxed interpreter support was compiled in or not. For setting what is to be protected, the PyThreadState for the sandboxed interpreter must be passed in. This makes the protection very explicit and helps make sure you set protections for the exact interpreter you mean to. All functions that set protections begin with the prefix ``PySandbox_Set*()``. These functions are meant to only work with sandboxed interpreters that have not been used yet to execute any Python code. The calls must be made by the code creating and handling the sandboxed interpreter *before* the sandboxed interpreter is used to execute any Python code. The functions for checking for permissions are actually macros that take in at least an error return value for the function calling the macro. This allows the macro to return on behalf of the caller if the check fails and cause the SandboxError exception to be propagated automatically. This helps eliminate any coding errors from incorrectly checking a return value on a rights-checking function call. For the rare case where this functionality is disliked, just make the check in a utility function and check that function's return value (but this is strongly discouraged!). Functions that check that an operation is allowed implicitly operate on the currently running interpreter as returned by ``PyInterpreter_Get()`` and are to be used by any code (the interpreter, extension modules, etc.) that needs to check for permission to execute. They have the common prefix of `PySandbox_Allowed*()``. API -------------- * PyThreadState* PySandbox_NewInterpreter() Return a new interpreter that is considered sandboxed. There is no corresponding ``PySandbox_EndInterpreter()`` as ``Py_EndInterpreter()`` will be taught how to handle sandboxed interpreters. ``NULL`` is returned on error. * PySandbox_Allowed(error_return) Macro that has the caller return with 'error_return' if the interpreter is unprotected, otherwise do nothing. Memory ============================= Protection -------------- A memory cap will be allowed. Modification to pymalloc will be needed to properly keep track of the allocation and freeing of memory. Same goes for the macros around the system malloc/free system calls. This provides a platform-independent system for protection of memory instead of relying on the operating system to provide a service for capping memory usage of a process. It also allows the protection to be at the interpreter level instead of at the process level. Why -------------- Protecting excessive memory usage allows one to make sure that a DoS attack against the system's memory is prevented. Possible Security Flaws ----------------------- If code makes direct calls to malloc/free instead of using the proper ``PyMem_*()`` macros then the security check will be circumvented. But C code is *supposed* to use the proper macros or pymalloc and thus this issue is not with the security model but with code not following Python coding standards. API -------------- * int PySandbox_SetMemoryCap(PyThreadState *, integer) Set the memory cap for an sandboxed interpreter. If the interpreter is not running an sandboxed interpreter, return a false value. * PySandbox_AllowedMemoryAlloc(integer, error_return) Macro to increase the amount of memory that is reported that the running sandboxed interpreter is using. If the increase puts the total count passed the set limit, raise an SandboxError exception and cause the calling function to return with the value of 'error_return', otherwise do nothing. * PySandbox_AllowedMemoryFree(integer, error_return) Macro to decrease the current running interpreter's allocated memory. If this puts the memory used to below 0, raise a SandboxError exception and return 'error_return', otherwise do nothing. Reading/Writing Files ============================= Protection -------------- XXX To open a file, one will have to use open(). This will make open() a factory function that controls reference access to the 'file' type in terms of creating new instances. When an attempted file opening fails (either because the path does not exist or of security reasons), SandboxError will be raised. The same exception must be raised to prevent filesystem information being gleaned from the type of exception returned (i.e., returning IOError if a path does not exist tells the user something about that file path). What open() returns may not be an instance of 'file' but a proxy that provides the security measures needed. While this might break code that uses type checking to make sure a 'file' object is used, taking a duck typing approach would be better. This is not only more Pythonic but would also allow the code to use a StringIO instance. It has been suggested to allow for a passed-in callback to be called when a specific path is to be opened. While this provides good flexibility in terms of allowing custom proxies with more fine-grained security (e.g., capping the amount of disk write), this has been deemed unneeded in the initial security model and thus is not being considered at this time. Why -------------- Allowing anyone to be able to arbitrarily read, write, or learn about the layout of your filesystem is extremely dangerous. It can lead to loss of data or data being exposed to people whom should not have access. Possible Security Flaws ----------------------- XXX API -------------- * int PySandbox_SetAllowedFile(PyThreadState *, string path, string mode) Add a file that is allowed to be opened in 'mode' by the 'file' object. If the interpreter is not sandboxed then return a false value. * PySandbox_AllowedPath(string path, string mode, error_return) Macro that causes the caller to return with 'error_return' and raise SandboxError as the exception if the specified path with 'mode' is not allowed, otherwise do nothing. Extension Module Importation ============================ Protection -------------- A whitelist of extension modules that may be imported must be provided. A default set is given for stdlib modules known to be safe. A check in the import machinery will check that a specified module name is allowed based on the type of module (Python source, Python bytecode, or extension module). Python bytecode files are never directly imported because of the possibility of hostile bytecode being present. Python source is always considered safe based on the assumption that all resource harm is eventually done at the C level, thus Python source code directly cannot cause harm without help of C extension modules. Thus only C extension modules need to be checked against the whitelist. The requested extension module name is checked in order to make sure that it is on the whitelist if it is a C extension module. If the name is not correct a SandboxError exception is raised. Otherwise the import is allowed. Even if a Python source code module imports a C extension module in an unprotected interpreter it is not a problem since the Python source code module is reloaded in the sandboxed interpreter. When that Python source module is freshly imported the normal import check will be triggered to prevent the C extension module from becoming available to the sandboxed interpreter. For the 'os' module, a special sandboxed version will be used if the proper C extension module providing the correct abilities is not allowed. This will default to '/' as the path separator and provide as much reasonable abilities as possible from a pure Python module. The 'sys' module is specially addressed in `Changing the Behaviour of the Interpreter`_. By default, the whitelisted modules are: * XXX Why -------------- Because C code is considered unsafe, its use should be regulated. By using a whitelist it allows one to explicitly decide that a C extension module is considered safe. Possible Security Flaws ----------------------- If a whitelisted C extension module imports a non-whitelisted C extension module and makes it an attribute of the whitelisted module there will be a breach in security. Luckily this a rarity in extension modules. There is also the issue of a C extension module calling the C API of a non-whitelisted C extension module. Lastly, if a whitelisted C extension module is loaded in an unprotected interpreter and then loaded into a sandboxed interpreter then there is no checks during module initialization for possible security issues in the sandboxed interpreter that would have occurred had the sandboxed interpreter done the initial import. All of these issues can be handled by never blindly whitelisting a C extension module. Added support for dealing with C extension modules comes in the form of `Extension Module Crippling`_. API -------------- * int PySandbox_SetModule(PyThreadState *, string module_name) Allow the sandboxed interpreter to import 'module_name'. If the interpreter is not sandboxed, return a false value. Absolute import paths must be specified. * int PySandbox_BlockModule(PyThreadState *, string module_name) Remove the specified module from the whitelist. Used to remove modules that are allowed by default. Return a false value if called on an unprotected interpreter. * PySandbox_AllowedModule(string module_name, error_return) Macro that causes the caller to return with 'error_return' and sets the exception SandboxError if the specified module cannot be imported, otherwise does nothing. Extension Module Crippling ========================== Protection -------------- By providing a C API for checking for allowed abilities, modules that have some useful functionality can do proper security checks for those functions that could provide insecure abilities while allowing safe code to be used (and thus not fully deny importation). Why -------------- Consider a module that provides a string processing ability. If that module provides a single convenience function that reads its input string from a file (with a specified path), the whole module should not be blocked from being used, just that convenience function. By whitelisting the module but having a security check on the one problem function, the user can still gain access to the safe functions. Even better, the unsafe function can be allowed if the security checks pass. Possible Security Flaws ----------------------- If a C extension module developer incorrectly implements the security checks for the unsafe functions it could lead to undesired abilities. API -------------- Use PySandbox_Allowed() to protect unsafe code from being executed. Hostile Bytecode ============================= Protection -------------- XXX Why -------------- Without implementing a bytecode verification tool, there is no way of making sure that bytecode does not jump outside its bounds, thus possibly executing malicious code. It also presents the possibility of crashing the interpreter. Possible Security Flaws ----------------------- None known. API -------------- N/A Changing the Behaviour of the Interpreter ========================================= Protection -------------- Only a subset of the 'sys' module will be made available to sandboxed interpreters. Things to allow from the sys module: * byteorder (?) * copyright * displayhook * excepthook * __displayhook__ * __excepthook__ * exc_info * exc_clear * exit * getdefaultencoding * _getframe (?) * hexversion * last_type * last_value * last_traceback * maxint (?) * maxunicode (?) * modules * stdin # See `Stdin, Stdout, and Stderr`_. * stdout * stderr * version Why -------------- Filesystem information must be removed. Any settings that could possibly lead to a DoS attack (e.g., sys.setrecursionlimit()) or risk crashing the interpreter must also be removed. Possible Security Flaws ----------------------- Exposing something that could lead to future security problems (e.g., a way to crash the interpreter). API -------------- None. Socket Usage ============================= Protection -------------- Allow sending and receiving data to/from specific IP addresses on specific ports. open() is to be used as a factory function to open a network connection. If the connection is not possible (either because of an invalid address or security reasons), SandboxError is raised. A socket object may not be returned by the call. A proxy to handle security might be returned instead. XXX Why -------------- Allowing arbitrary sending of data over sockets can lead to DoS attacks on the network and other machines. Limiting accepting data prevents your machine from being attacked by accepting malicious network connections. It also allows you to know exactly where communication is going to and coming from. Possible Security Flaws ----------------------- If someone managed to influence the used DNS server to influence what IP addresses were used after a DNS lookup. API -------------- * int PySandbox_SetIPAddress(PyThreadState *, string IP, integer port) Allow the sandboxed interpreter to send/receive to the specified 'IP' address on the specified 'port'. If the interpreter is not sandboxed, return a false value. * PySandbox_AllowedIPAddress(string IP, integer port, error_return) Macro to verify that the specified 'IP' address on the specified 'port' is allowed to be communicated with. If not, cause the caller to return with 'error_return' and SandboxError exception set, otherwise do nothing. * int PySandbox_SetHost(PyThreadState *, string host, integer port) Allow the sandboxed interpreter to send/receive to the specified 'host' on the specified 'port'. If the interpreter is not sandboxed, return a false value. * PySandbox_AllowedHost(string host, integer port, error_return) Check that the specified 'host' on the specified 'port' is allowed to be communicated with. If not, set a SandboxError exception and cause the caller to return 'error_return', otherwise do nothing. Network Information ============================= Protection -------------- Limit what information can be gleaned about the network the system is running on. This does not include restricting information on IP addresses and hosts that are have been explicitly allowed for the sandboxed interpreter to communicate with. XXX Why -------------- With enough information from the network several things could occur. One is that someone could possibly figure out where your machine is on the Internet. Another is that enough information about the network you are connected to could be used against it in an attack. Possible Security Flaws ----------------------- As long as usage is restricted to only what is needed to work with allowed addresses, there are no security issues to speak of. API -------------- * int PySandbox_SetNetworkInfo(PyThreadState *) Allow the sandboxed interpreter to get network information regardless of whether the IP or host address is explicitly allowed. If the interpreter is not sandboxed, return a false value. * PySandbox_AllowedNetworkInfo(error_return) Macro that will return 'error_return' for the caller and set a SandboxError exception if the sandboxed interpreter does not allow checking for arbitrary network information, otherwise do nothing. Filesystem Information ============================= Protection -------------- Do not allow information about the filesystem layout from various parts of Python to be exposed. This means blocking exposure at the Python level to: * __file__ attribute on modules * __path__ attribute on packages * co_filename attribute on code objects * XXX Why -------------- Exposing information about the filesystem is not allowed. You can figure out what operating system one is on which can lead to vulnerabilities specific to that operating system being exploited. Possible Security Flaws ----------------------- Not finding every single place where a file path is exposed. API -------------- * int PySandbox_SetFilesystemInfo(PyThreadState *) Allow the sandboxed interpreter to expose filesystem information. If the passed-in interpreter is not sandboxed, return NULL. * PySandbox_AllowedFilesystemInfo(error_return) Macro that checks if exposing filesystem information is allowed. If it is not, cause the caller to return with the value of 'error_return' and raise SandboxError, otherwise do nothing. Stdin, Stdout, and Stderr ============================= Protection -------------- By default, sys.__stdin__, sys.__stdout__, and sys.__stderr__ will be set to instances of StringIO. Explicit allowance of the process' stdin, stdout, and stderr is possible. This will protect the 'print' statement, and the built-ins input() and raw_input(). Why -------------- Interference with stdin, stdout, or stderr should not be allowed unless desired. No one wants uncontrolled output sent to their screen. Possible Security Flaws ----------------------- Unless StringIO instances can be used maliciously, none to speak of. API -------------- * int PySandbox_SetTrueStdin(PyThreadState *) int PySandbox_SetTrueStdout(PyThreadState *) int PySandbox_SetTrueStderr(PyThreadState *) Set the specific stream for the interpreter to the true version of the stream and not to the default instance of StringIO. If the interpreter is not sandboxed, return a false value. Adding New Protections ============================= .. note:: This feature has the lowest priority and thus will be the last feature implemented (if ever). Protection -------------- Allow for extensibility in the security model by being able to add new types of checks. This allows not only for Python to add new security protections in a backwards-compatible fashion, but to also have extension modules add their own as well. An extension module can introduce a group for its various values to check, with a type being a specific value within a group. The "Python" group is specifically reserved for use by the Python core itself. Why -------------- We are all human. There is the possibility that a need for a new type of protection for the interpreter will present itself and thus need support. By providing an extensible way to add new protections it helps to future-proof the system. It also allows extension modules to present their own set of security protections. That way one extension module can use the protection scheme presented by another that it is dependent upon. Possible Security Flaws ------------------------ Poor definitions by extension module users of how their protections should be used would allow for possible exploitation. API -------------- + Bool * int PySandbox_SetExtendedFlag(PyThreadState *, string group, string type) Set a group-type to be true. Expected use is for when a binary possibility of something is needed and that the default is to not allow use of the resource (e.g., network information). Returns a false value if used on an unprotected interpreter. * PySandbox_AllowedExtendedFlag(string group, string type, error_return) Macro that if the group-type is not set to true, cause the caller to return with 'error_return' with SandboxError exception raised. For unprotected interpreters the check does nothing. + Numeric Range * int PySandbox_SetExtendedCap(PyThreadState *, string group, string type, integer cap) Set a group-type to a capped value, 'cap', with the initial allocated value set to 0. Expected use is when a resource has a capped amount of use (e.g., memory). Returns a false value if the interpreter is not sandboxed. * PySandbox_AllowedExtendedAlloc(integer increase, error_return) Macro to raise the amount of a resource is used by 'increase'. If the increase pushes the resource allocation past the set cap, then return 'error_return' and set SandboxError as the exception, otherwise do nothing. * PySandbox_AllowedExtendedFree(integer decrease, error_return) Macro to lower the amount a resource is used by 'decrease'. If the decrease pushes the allotment to below 0 then have the caller return 'error_return' and set SandboxError as the exception, otherwise do nothing. + Membership * int PySandbox_SetExtendedMembership(PyThreadState *, string group, string type, string member) Add a string, 'member', to be considered a member of a group-type (e.g., allowed file paths). If the interpreter is not an sandboxed interpreter, return a false value. * PySandbox_AllowedExtendedMembership(string group, string type, string member, error_return) Macro that checks 'member' is a member of the values set for the group-type. If it is not, then have the caller return 'error_return' and set an exception for SandboxError, otherwise does nothing. + Specific Value * int PySandbox_SetExtendedValue(PyThreadState *, string group, string type, string value) Set a group-type to 'value'. If the interpreter is not sandboxed, return NULL. * PySandbox_AllowedExtendedValue(string group, string type, string value, error_return) Macro to check that the group-type is set to 'value'. If it is not, then have the caller return 'error_return' and set an exception for SandboxError, otherwise do nothing. Python API ============================= __sandboxed__ -------------- A built-in that flags whether the interpreter currently running is sandboxed or not. Set to a 'bool' value that is read-only. To mimic working of __debug__. sandbox module -------------- XXX References /////////////////////////////////////// .. [#rexec] The 'rexec' module (http://docs.python.org/lib/module-rexec.html) .. [#safe-tcl] The Safe-Tcl Security Model (http://research.sun.com/technical-reports/1997/abstract-60.html) .. [#ctypes] 'ctypes' module (http://docs.python.org/dev/lib/module-ctypes.html) .. [#paradigm regained] "Paradigm Regained: Abstraction Mechanisms for Access Control" (http://erights.org/talks/asian03/paradigm-revised.pdf) .. [#armin-hiding] [Python-Dev] what can we do to hide the 'file' type? (http://mail.python.org/pipermail/python-dev/2006-July/067076.html) -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060707/a32678f1/attachment-0001.html From christopher.perkins at pw.utc.com Mon Jul 10 19:36:02 2006 From: christopher.perkins at pw.utc.com (Perkins, Christopher) Date: Mon, 10 Jul 2006 13:36:02 -0400 Subject: [Python-Dev] Autoloading? (Making Queue.Queue easier to use) Message-ID: <4B9D477017AEEC4188C18CB8DA8364B606156A02@pusehe0l.eh.pweh.com> John, I see what you are doing with the algorithm now, and I can easily re-factor it. What I am having issues with is how structured it is. 5 minute windows? Then running hours will always be recorded in 1/12th time steps. Would it not be more accurate to record engine time where the breaker was closed? -chris From skip at pobox.com Wed Jul 12 02:54:03 2006 From: skip at pobox.com (skip at pobox.com) Date: Tue, 11 Jul 2006 19:54:03 -0500 Subject: [Python-Dev] User's complaints In-Reply-To: References: Message-ID: <17588.18475.171696.506911@montanaro.dyndns.org> Michael> Well here's one I stumbled across the other day. I don't know Michael> if it's legit, but it's still bad PR: Michael> http://www.gbch.net/gjb/blog/software/discuss/python-sucks.html Michael> For the impatient, he's not at all bothered about the lack of Michael> obscure language feature X. The way I used to format dates using time.strftime does indeed no longer work. Python 2.3: >>> import time >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6) '2005-06-04' Python 2.4 or 2.5: >>> import time >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6) Traceback (most recent call last): File "", line 1, in ? ValueError: day of year out of range >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (1,)*6) '2005-06-04' I don't actually run into this problem as I've pretty much converted to use datetime in new code. I also realize that's not documented as the way it should be done, but I'm fairly certain it was common usage before the datetime module came along. Still, it is a bit annoying that the (undocumented, but I think de facto) commonly used idiom no longer works. (In fact, it always bothered me a bit that I had to even provide the unused values.) Skip From brett at python.org Wed Jul 12 03:05:21 2006 From: brett at python.org (Brett Cannon) Date: Tue, 11 Jul 2006 18:05:21 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <17588.18475.171696.506911@montanaro.dyndns.org> References: <17588.18475.171696.506911@montanaro.dyndns.org> Message-ID: On 7/11/06, skip at pobox.com wrote: > > > Michael> Well here's one I stumbled across the other day. I don't know > Michael> if it's legit, but it's still bad PR: > > Michael> > http://www.gbch.net/gjb/blog/software/discuss/python-sucks.html > > Michael> For the impatient, he's not at all bothered about the lack of > Michael> obscure language feature X. That whole entry is a little overblown. I mean the guy has a single issue with a change and he gives up on the language after only the second one? Try using any language that is under active development and show me stuff that won't break. I think it shows how hard we work to not break things that this guy only got bit twice, and at least the second time was minor and had a legit reason for the change. The way I used to format dates using time.strftime does indeed no longer > work. > > Python 2.3: > > >>> import time > >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6) > '2005-06-04' > > Python 2.4 or 2.5: > > >>> import time > >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6) > Traceback (most recent call last): > File "", line 1, in ? > ValueError: day of year out of range > >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (1,)*6) > '2005-06-04' That was done to fix buffer overflow issues when libc implementations didn't do bound checks on the arguments to strftime() and would index too far (e.g., setting the month as 20 indexes too far and setting to 0 can be an issue as well since the array indexing for the month name will be January, and thus will most likely do ``mon - 1`` to get the index into the array). I don't actually run into this problem as I've pretty much converted to use > datetime in new code. I also realize that's not documented as the way it > should be done, It is the last point in the first paragraph on time.strftime() discussing what changed in Python 2.4 as to what the change was. It's also in Misc/NEWS . Basically the guy didn't read the release notes or the docs to see why that changed and that it was legitimate and needed for stability. but I'm fairly certain it was common usage before the > datetime module came along. Still, it is a bit annoying that the > (undocumented, but I think de facto) commonly used idiom no longer works. As I said, stability called for the need to make the change. It was discussed on python-dev when it came up. (In fact, it always bothered me a bit that I had to even provide the unused > values.) That would probably be fine to add in 2.6 . Should be a huge issue. You could also change the function to detect values of 0 when provided and then automatically use defaults in those cases that are within the proper range. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060711/6cb60fa7/attachment.html From skip at pobox.com Wed Jul 12 03:57:18 2006 From: skip at pobox.com (skip at pobox.com) Date: Tue, 11 Jul 2006 20:57:18 -0500 Subject: [Python-Dev] User's complaints In-Reply-To: References: <17588.18475.171696.506911@montanaro.dyndns.org> Message-ID: <17588.22270.591885.831508@montanaro.dyndns.org> Brett> That whole entry is a little overblown. Well, sure. Think of it as a bug report with attitude. ;-) Brett> That was done to fix buffer overflow issues when libc Brett> implementations didn't do bound checks on the arguments to Brett> strftime() and would index too far... That special case could simply be recognized and converted into one that works couldn't it? Documented or not, I believe it was the standard idiom for formatting just a date before 2.4. http://python.org/sf/1520914 Keep or toss as you see fit. Seems like breakage that could have been avoided to me though. Skip From anthony at interlink.com.au Wed Jul 12 04:42:09 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 12 Jul 2006 12:42:09 +1000 Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt In-Reply-To: <44B41525.3040004@v.loewis.de> References: <44B3ADE2.7040108@ofai.at> <44B41525.3040004@v.loewis.de> Message-ID: <200607121242.12323.anthony@interlink.com.au> On Wednesday 12 July 2006 07:16, Martin v. L?wis wrote: > Stefan Rank wrote: > > I suggest to add (after 2.5 I assume) one of the following to the > > beginning of urllib.quote to either fail early and consistently > > on unicode arguments and improve the error message:: > > > > if isinstance(s, unicode): > > raise TypeError("quote needs a byte string argument, not > > unicode," " use `argument.encode('utf-8')` first.") > > > > or to do The Right Thing (tm), which is utf-8 encoding:: > > The right thing to do is IRIs. This is more complicated than > encoding the Unicode string as UTF-8, though: for the host part of > the URL, you have to encode it with IDNA (and there are additional > complicated rules in place, e.g. when the Unicode string already > contains %). > > Contributions are welcome, as long as they fix this entire issue > "for good" (i.e. in all URL-processing code, and considering all > relevant RFCs). For 2.5, should we at least detect that it's unicode and raise a useful error? -- Anthony Baxter It's never too late to have a happy childhood. From brett at python.org Wed Jul 12 04:50:05 2006 From: brett at python.org (Brett Cannon) Date: Tue, 11 Jul 2006 19:50:05 -0700 Subject: [Python-Dev] changing time.strftime() to accept 0s (was: User's complaints) Message-ID: On 7/11/06, skip at pobox.com wrote: > > Brett> That whole entry is a little overblown. > > Well, sure. Think of it as a bug report with attitude. ;-) > > Brett> That was done to fix buffer overflow issues when libc > Brett> implementations didn't do bound checks on the arguments to > Brett> strftime() and would index too far... > > That special case could simply be recognized and converted into one that > works couldn't it? Documented or not, I believe it was the standard idiom > for formatting just a date before 2.4. > > http://python.org/sf/1520914 > > Keep or toss as you see fit. Seems like breakage that could have been > avoided to me though. Right, but that would have required realizing how to prevent it at the time. =) I can change it so that 0 is an acceptable value and internally handles defaulting to something reasonable (accepting less than 9 values in the tuple is a separate thing that I don't feel like bothering to implement). It does possibly hide bugs where 0 was not meant to be passed, though. If people think this is a reasonable thing to change, then Neal and Anthony, do you want it to go into 2.5? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060711/7593f880/attachment.html From greg.ewing at canterbury.ac.nz Wed Jul 12 05:05:37 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Jul 2006 15:05:37 +1200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: <1152589949.14276.16.camel@localhost.localdomain> References: <001d01c6a369$78a59950$6402a8c0@arkdesktop> <44B190DF.3000805@acm.org> <44B19691.4050509@acm.org> <17586.19683.920761.293494@montanaro.dyndns.org> <1152589949.14276.16.camel@localhost.localdomain> Message-ID: <44B46701.3040305@canterbury.ac.nz> Matthew Barnes wrote: > its > meaning in C/C++ (i.e. the symbol is defined outside of the current > scope). It means more than that -- it means defined outside the current *file*. That's much too drastic for what we want. -- Greg From ncoghlan at gmail.com Wed Jul 12 05:20:27 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 12 Jul 2006 13:20:27 +1000 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> Message-ID: <44B46A7B.8060509@gmail.com> Fredrik Lundh wrote: > Boris Borcic wrote: > >>> in what language [is] the word "sum" an appropriate synonym for "concatenate" ? >> any that admits a+b to mean ''.join([a,b]), I'd say. > > and what human language would that be ? Python :) Sure-computers-can-speak-it-but-so-can-humans'ly yours, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From greg.ewing at canterbury.ac.nz Wed Jul 12 05:31:13 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Jul 2006 15:31:13 +1200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> Message-ID: <44B46D01.4090809@canterbury.ac.nz> Boris Borcic wrote: > sum() *is* exactly an attractive nuisance by *appearing* to be an obvious way of > chaining strings in a list (without actually being one). But at least it fails immediately, prompting you to look in another direction. > I admit that there is a step of arguable interpretation from these recorded > facts to my diagnostic, but the latter is compatible with the facts. Your > version otoh looks more robust in the role of eg creation myth. I suppose you could call that a linguistic matter, but I don't think it's exclusively a *native* one. I suspect that only someone with a programmer's warped mind would make the leap from "sum" to "string concatenation" -- whether they were a native English speaker or not. Also I don't see that my version of events is inconsistent with the messages you quoted either -- at least not so much as to be relegated to a myth! -- Greg From greg.ewing at canterbury.ac.nz Wed Jul 12 05:44:32 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Jul 2006 15:44:32 +1200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: <200607111429.22964.gmccaughan@synaptics-uk.com> References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> <200607111429.22964.gmccaughan@synaptics-uk.com> Message-ID: <44B47020.8050906@canterbury.ac.nz> Gareth McCaughan wrote: > (I agree that Greg's interpretation is also not well supported > by that thread; I was perhaps a bit excessive in claiming that language had nothing to do with it. What I meant was that it wasn't the *only* consideration. If there hadn't been any disadvantages, quite possibly the ability to use sum() on strings would have been supported, or at least not actively blocked. But because of the disadvantage I mentioned, *together* with the fact that it's an odd meaning for "sum" (whether you're a native English speaker or not), the decision was made to disallow it. Does that version sound less like a creation myth?-) -- Greg From greg.ewing at canterbury.ac.nz Wed Jul 12 05:45:21 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Jul 2006 15:45:21 +1200 Subject: [Python-Dev] [slighly OT] Native speakers and hurting brains In-Reply-To: References: <20060705101821.7sqnxo7lu5cgg4ks@login.werra.lunarpages.com> <44AEF0F2.5040007@canterbury.ac.nz> Message-ID: <44B47051.9090608@canterbury.ac.nz> Boris Borcic wrote: > Fredrik Lundh wrote: > >>in what language the word "sum" an appropriate synonym for "concatenate" ? > > any that admits a+b to mean ''.join([a,b]), I'd say. Not the same thing. "a + b" is usually pronounced "a plus b". Now, "plus" has a somewhat wider meaning than "sum". It sounds quite in order to say things like "I have a sandwich plus an apple in my lunch box", but it would be odd to say "I have the sum of a sandwich and an apple in my lunch box". -- Greg From martin at v.loewis.de Wed Jul 12 07:53:24 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 12 Jul 2006 07:53:24 +0200 Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt In-Reply-To: <200607121242.12323.anthony@interlink.com.au> References: <44B3ADE2.7040108@ofai.at> <44B41525.3040004@v.loewis.de> <200607121242.12323.anthony@interlink.com.au> Message-ID: <44B48E54.3060908@v.loewis.de> Anthony Baxter wrote: >> The right thing to do is IRIs. > > For 2.5, should we at least detect that it's unicode and raise a > useful error? That can certainly be done, sure. Martin From anthony at interlink.com.au Wed Jul 12 07:57:14 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed, 12 Jul 2006 15:57:14 +1000 Subject: [Python-Dev] changing time.strftime() to accept 0s (was: User's complaints) In-Reply-To: References: Message-ID: <200607121557.20462.anthony@interlink.com.au> Making strftime accept 0s is fine to be checked in, since it's a regression (an understandable one, but what the hell). Making it accept less than 9 items and have useful defaults should wait for 2.6 From tjreedy at udel.edu Wed Jul 12 08:29:32 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 12 Jul 2006 02:29:32 -0400 Subject: [Python-Dev] changing time.strftime() to accept 0s (was: User'scomplaints) References: <200607121557.20462.anthony@interlink.com.au> Message-ID: "Anthony Baxter" wrote in message news:200607121557.20462.anthony at interlink.com.au... > Making strftime accept 0s is fine to be checked in, since it's a > regression (an understandable one, but what the hell). Once it is, someone should put a note back on the complainant's blog page. From martin at v.loewis.de Wed Jul 12 08:34:49 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 12 Jul 2006 08:34:49 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: <17588.18475.171696.506911@montanaro.dyndns.org> References: <17588.18475.171696.506911@montanaro.dyndns.org> Message-ID: <44B49809.5050707@v.loewis.de> skip at pobox.com wrote: > The way I used to format dates using time.strftime does indeed no longer > work. > > Python 2.3: > > >>> import time > >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6) > '2005-06-04' Is there any specific reason you couldn't write "%d-%02d-%02d" % (2005, 6, 4) (i.e. not use strftime at all)? It seems strange to fake a time tuple just to use that function, in particular if the time formatting should not use any locale-specific output. > I don't actually run into this problem as I've pretty much converted to use > datetime in new code. I also realize that's not documented as the way it > should be done, but I'm fairly certain it was common usage before the > datetime module came along. Still, it is a bit annoying that the > (undocumented, but I think de facto) commonly used idiom no longer works. I guess this was caused by this change: /* Checks added to make sure strftime() does not crash Python by indexing blindly into some array for a textual representation by some bad index (fixes bug #897625). No check for year since handled in gettmarg(). */ So this was changed in response to a bug report about a crash. Regards, Martin From stefan.rank at ofai.at Wed Jul 12 10:27:32 2006 From: stefan.rank at ofai.at (Stefan Rank) Date: Wed, 12 Jul 2006 10:27:32 +0200 Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt In-Reply-To: <44B48E54.3060908@v.loewis.de> References: <44B3ADE2.7040108@ofai.at> <44B41525.3040004@v.loewis.de> <200607121242.12323.anthony@interlink.com.au> <44B48E54.3060908@v.loewis.de> Message-ID: <44B4B274.4050602@ofai.at> on 12.07.2006 07:53 Martin v. L?wis said the following: > Anthony Baxter wrote: >>> The right thing to do is IRIs. >> For 2.5, should we at least detect that it's unicode and raise a >> useful error? > > That can certainly be done, sure. > > Martin That would be great. And I agree that updating urllib.quote for unicode should be part of a grand plan that updates all of urllib[2] and introduces an irilib / urischemes / uriparse module in 2.6 as Martin and John J Lee suggested. =) cheers, stefan From skip at pobox.com Wed Jul 12 13:13:47 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed, 12 Jul 2006 06:13:47 -0500 Subject: [Python-Dev] User's complaints In-Reply-To: <44B49809.5050707@v.loewis.de> References: <17588.18475.171696.506911@montanaro.dyndns.org> <44B49809.5050707@v.loewis.de> Message-ID: <17588.55659.727250.840456@montanaro.dyndns.org> >> Python 2.3: >> >> >>> import time >> >>> time.strftime("%Y-%m-%d", (2005, 6, 4) + (0,)*6) >> '2005-06-04' Martin> Is there any specific reason you couldn't write Martin> "%d-%02d-%02d" % (2005, 6, 4) Martin> (i.e. not use strftime at all)? Sure, but that was just me being lazy typing at the interactive prompt. %Y-%m-%d is just about the only date format I can ever remember without consulting the strftime(3) man page. ;-) Suppose I had used >>> time.strftime("%b %d, %Y", (2005, 6, 4) + (1,)*6) 'Jun 04, 2005' instead (switching to the all-ones default that still works)? Martin> So this was changed in response to a bug report about a crash. Yeah, but it broke common (at the time) usage. Skip From g.brandl at gmx.net Wed Jul 12 13:55:31 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 12 Jul 2006 13:55:31 +0200 Subject: [Python-Dev] Long options support In-Reply-To: References: Message-ID: Fredrik Lundh wrote: > Georg Brandl wrote: > >> I've just closed a bug report wishing for long option support, >> pointing to a patch sitting in the patch tracker implementing >> this. >> >> Should we accept at least the very common options "--help" and >> "--version" in 2.5? > > Guido pronounced on this in May Late it comes, but here is a patch for getopt.c implementing this pronouncement. I think there's no need to wait for 2.6 with it, or is there? Cheers, Georg Index: Python/getopt.c =================================================================== --- Python/getopt.c (Revision 50599) +++ Python/getopt.c (Arbeitskopie) @@ -24,6 +24,9 @@ * davegottner at delphi.com. *---------------------------------------------------------------------------*/ +/* Modified to support --help and --version, as well as /? on Windows + * by Georg Brandl. */ + #include #include @@ -43,15 +46,35 @@ if (*opt_ptr == '\0') { - if (_PyOS_optind >= argc || argv[_PyOS_optind][0] != '-' || - argv[_PyOS_optind][1] == '\0' /* lone dash */ ) + if (_PyOS_optind >= argc) return -1; +#ifdef MS_WINDOWS + else if (strcmp(argv[_PyOS_optind], "/?") == 0) { + ++_PyOS_optind; + return 'h'; + } +#endif + else if (argv[_PyOS_optind][0] != '-' || + argv[_PyOS_optind][1] == '\0' /* lone dash */ ) + return -1; + else if (strcmp(argv[_PyOS_optind], "--") == 0) { ++_PyOS_optind; return -1; } + else if (strcmp(argv[_PyOS_optind], "--help") == 0) { + ++_PyOS_optind; + return 'h'; + } + + else if (strcmp(argv[_PyOS_optind], "--version") == 0) { + ++_PyOS_optind; + return 'V'; + } + + opt_ptr = &argv[_PyOS_optind++][1]; } @@ -62,7 +85,7 @@ if (_PyOS_opterr) fprintf(stderr, "Unknown option: -%c\n", option); - return '?'; + return '_'; } if (*(ptr + 1) == ':') { @@ -76,7 +99,7 @@ if (_PyOS_opterr) fprintf(stderr, "Argument expected for the -%c option\n", option); - return '?'; + return '_'; } _PyOS_optarg = argv[_PyOS_optind++]; Index: Modules/main.c =================================================================== --- Modules/main.c (Revision 50599) +++ Modules/main.c (Arbeitskopie) @@ -40,7 +40,7 @@ static int orig_argc; /* command line options */ -#define BASE_OPTS "c:dEhim:OQ:StuUvVW:xX" +#define BASE_OPTS "c:dEhim:OQ:StuUvVW:xX?" #ifndef RISCOS #define PROGRAM_OPTS BASE_OPTS @@ -62,7 +62,7 @@ -c cmd : program passed in as string (terminates option list)\n\ -d : debug output from parser (also PYTHONDEBUG=x)\n\ -E : ignore environment variables (such as PYTHONPATH)\n\ --h : print this help message and exit\n\ +-h : print this help message and exit (also --help)\n\ -i : inspect interactively after running script, (also PYTHONINSPECT=x)\n\ and force prompts, even if stdin does not appear to be a terminal\n\ "; @@ -78,7 +78,7 @@ static char *usage_3 = "\ see man page for details on internal buffering relating to '-u'\n\ -v : verbose (trace import statements) (also PYTHONVERBOSE=x)\n\ --V : print the Python version number and exit\n\ +-V : print the Python version number and exit (also --version)\n\ -W arg : warning control (arg is action:message:category:module:lineno)\n\ -x : skip first line of source, allowing use of non-Unix forms of #!cmd\n\ file : program read from script file\n\ @@ -313,6 +313,7 @@ Py_UnicodeFlag++; break; case 'h': + case '?': help++; break; case 'V': From fredrik at pythonware.com Wed Jul 12 15:41:04 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Wed, 12 Jul 2006 15:41:04 +0200 Subject: [Python-Dev] Long options support References: Message-ID: Georg Brandl wrote: > Late it comes, but here is a patch for getopt.c implementing > this pronouncement. I think there's no need to wait for 2.6 with it, > or is there? check it in already. From bioinformed at gmail.com Wed Jul 12 16:33:28 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Wed, 12 Jul 2006 10:33:28 -0400 Subject: [Python-Dev] Behavior change in subprocess.py Message-ID: <2e1434c10607120733h3b2a0d86l1908518593d2eb49@mail.gmail.com> During my testing of Python 2.5b2, I've found something that may be worthy of discussion. I suspect that recent GC and finalization changes have altered the behavior of the Popen object in subprocess.py. I am now getting many many many finalization warnings in my code like: Exception exceptions.AttributeError: "'NoneType' object has no attribute 'append'" in > ignored Is this a bug or a feature? Personally, I'd like to see these messages silenced, since it is being generated during interpreter shutdown. The following patch does the trick for me: --- /usr/local/lib/python2.5/subprocess.py 2006-07-11 14:11: 59.000000000 -0400 +++ subprocess.py 2006-07-12 10:17:09.000000000 -0400 @@ -613,7 +613,7 @@ return # In case the child hasn't been waited on, check if it's done. self.poll(_deadstate=sys.maxint) - if self.returncode is None: + if self.returncode is None and _active is not None: # Child is still running, keep us alive until we can wait on it. _active.append(self) Note that popen.py does something similar, though I am not convinced that the test is right or if it is doing something more subtle: def __del__(self): # In case the child hasn't been waited on, check if it's done. self.poll(_deadstate=sys.maxint) if self.sts < 0: if _active: # Child is still running, keep us alive until we can wait on it. _active.append(self) Regards, -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/6b697648/attachment.htm From bborcic at gmail.com Wed Jul 12 16:36:28 2006 From: bborcic at gmail.com (Boris Borcic) Date: Wed, 12 Jul 2006 16:36:28 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: Terry Reedy wrote: > "Boris Borcic" wrote in message > news:e90j6o$fij$1 at sea.gmane.org... >> I agree with you (and argued it in "scopes vs augmented assignment vs >> sets" >> recently) that mutating would be sufficient /if/ the compiler would view >> augmented assignment as mutations operators : > > Mutation is an operation on objects. Binding is an operation on > namespaces. The difference between objects and namespaces (and the actions > thereupon) is fundamental to Python. If you want to put it that way, but I think it muddies the relevant water to confuse scopes with namespaces, or compiler with interpreter responsibilities. Deciding whether an augmented assignment results in a bona-fide mutation or in the rebinding of an already bound name in an unchanging namespace : that's the responsibility of the interpreter. [Well, nowa-pypy-days I guess we should say, not even of the interpreter, but of the object space]. Deciding how names bind to lexical scopes, that's the responsibility of the compiler. When faced with such code as below [ while mutable_int() might as well equal (lambda x : x) as far as the compiler is concerned ] > def counter(num): > > num = mutable_int(num) > > def inc(): > > num += 1 > > return num > > return inc It is disingenuous that the /scope-binding phase/ of the compiler design feigns to believe that my "num += 1" is really more like "num = 2" than it is like a mutation operator, given that : whether the statement ultimately results in REbinding or mutation is /irrelevant/ to the fact that any augmented assignment requires an autonomous initial binding of the variable to ever function. - what is exactly the sort of assertions independent of run-time details that compilers are made to reason about. > Asking the interpreter to view one > thing as something else which it isn't can only lead to more confusion. I think its impossible to be more confused than setting things up so that the programmer of counter() above gets : UnboundLocalError: local variable 'num' referenced before assignment When what is really meant is (your drift no ?) : << We think allowing you what you want would make a=b=c ; a += d ; b = b+d ; assert a == b more difficult for newbies to understand and predict in context >> > In > particular, asking that arithmetic operations on immutable numbers be seen > as mutations seems wacky to me. Nobody really asked for this (I invite you to check), but even so I feel an exact analysis of the impact on use-cases would be worth its while. > >> which it doesn't as far as concerns scopes where a variable >> appears as target only of /augmented/ assignments. > > The interpreter/compiler, as far as I can think, never views binding as > mutation, nor should it. The request that it do so The request is not for the "compiler/interpreter" to view binding as mutation, it is for the /compiler/ to view /REbinding/ just like mutation - which it is as far as compiler responsibilities are concerned imo. > makes me wonder whether > it might have been a mistake to allow mutable objects in an augmented > assignment to choose to implement the associated operation as an in-place > mutation. I agree with you that removing this possibility would provide better consistency to your position. Best regards, Boris Borcic -- "C++ is a contradiction in terms" - Lorentz, Einstein, Poincar? From guido at python.org Wed Jul 12 17:26:15 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 12 Jul 2006 08:26:15 -0700 Subject: [Python-Dev] Autoloading? (Making Queue.Queue easier to use) In-Reply-To: <4B9D477017AEEC4188C18CB8DA8364B606156A02@pusehe0l.eh.pweh.com> References: <4B9D477017AEEC4188C18CB8DA8364B606156A02@pusehe0l.eh.pweh.com> Message-ID: Why do I have the feeling you sent this to the wrong list? On 7/10/06, Perkins, Christopher wrote: > John, > > I see what you are doing with the algorithm now, and I can easily re-factor > it. What I am having issues with is how structured it is. 5 minute > windows? Then running hours will always be recorded in 1/12th time steps. > > Would it not be more accurate to record engine time where the breaker was > closed? > > -chris > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Wed Jul 12 17:25:38 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 13 Jul 2006 01:25:38 +1000 Subject: [Python-Dev] Long options support In-Reply-To: References: Message-ID: <200607130125.44037.anthony@interlink.com.au> On Wednesday 12 July 2006 21:55, Georg Brandl wrote: > >> Should we accept at least the very common options "--help" and > >> "--version" in 2.5? > > > > Guido pronounced on this in May > > Late it comes, but here is a patch for getopt.c implementing > this pronouncement. I think there's no need to wait for 2.6 with > it, or is there? Check it in. Anthony From barry at python.org Wed Jul 12 20:56:36 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 12 Jul 2006 14:56:36 -0400 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 10, 2006, at 9:52 PM, Barry Warsaw wrote: > Patch #1520294 adds support for attributes defined with PyGetSetDef > in extension modules to pydoc, specifically so things like help > (array.array.typecode) gives something useful, like the attribute's > docstring for instance. > > Along the way, I added types.GetSetterType and inspect.isgetsetter > (), along with tests and docs. Now I definitely think that the > pydoc change should go into Python 2.5 and be backported to Python > 2.4, but it's a bit of a question whether the types.py and > inspect.py changes should go into Python 2.5 now that we're in beta. > > I personal think they're worthwhile changes to go it, but I won't > argue too hard for them since the pydoc fixes can be implemented > with local changes to pydoc.py alone. Turns out there is at least one other type that I think should be better supported, but which I'm having a hard time figuring out the right way to do this. Member descriptors are one such type, an example of which is datetime.timedelta.days >>> import datetime >>> type(datetime.timedelta.days) Do a help(datetime.timedelta.days) and you get nothing useful. Now, I started down a similar path for solving this as I did the other day for getset descriptors, but I've run into a bit of a roadblock. What you basically want to do is add a constant to the types module, then add an is-predicate to inspect.py, then add the necessary glue to pydoc.py. The problem is that you have to have something readily available to call type() on to get a member descriptor type, and for that you have to go hunting around in Python's C layer. Basically you're looking for a type initialized with PyObject_HEAD_INIT(NULL) /and/ has a tp_members slot. There are a few of them around, such as the datetime.timedelta type mentioned above, but they all have a pretty serious problem. You cannot import their module in types.py. I'm not exactly sure why this is, but if you put import datetime in types.py, you'll get the "import site" failed error. Run with -v and you get buried in the output: 'import site' failed; traceback: Traceback (most recent call last): File "/Users/barry/projects/python/Lib/site.py", line 62, in import os File "/Users/barry/projects/python/Lib/os.py", line 690, in import copy_reg as _copy_reg File "/Users/barry/projects/python/Lib/copy_reg.py", line 7, in from types import ClassType as _ClassType File "/Users/barry/projects/python/Lib/types.py", line 90, in import datetime ImportError: No module named datetime I think the problem is that the platform-specific extension module directory isn't on sys.path yet, so it can't e.g. find datetime.so. Of course, datetime is perfectly importable later on. As far as I can tell, there does not seem to be any candidate member descriptors that are built into the interpreter and guaranteed to be there early enough to import into types, but maybe I'm missing something. I can think of a number of ways to address this, with varying levels of ickyness, but I wanted to see what you all thought. For example, I could change inspect locally so that it gets the type of datetime.timedelta.days without adding a constant to types.py. Or I could patch pydoc.py directly and leave even inspect.py out of it. Or I could create some stupid internal type in some stupid internal module who's only purpose would be to have a handle on member descriptors. Or I could change datetime to be built-in. (see what I mean about levels of ickyness? :). I'm up for suggestions. I think this would be worthwhile to address in Python 2.5 since I think it would be good to have an accurate representation of Python's built-in types in types.py. Ultimately, I really care about teaching pydoc.help() about instances of these types so that users can get better help when they encounter them (such as might be the case in 3rd party extension modules). Suggestions are welcome. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRLVF6XEjvBPtnXfVAQJVmAP5ATS56TtrdUamo+CxdNHrP6j7zxwEef2t fCM3XuEjv/Y+UH3goQIm34ENySjqYgC/whQqXGPWa/5GVc1MkhZo3W2BsA4NJ8yq Za8+6+KsWaR625qj/mh/Fbw5I/2vOE3MClATuQ75aCJjLpSAwhxYqleYPTD7tBiF P4zc+2OYedU= =ikCK -----END PGP SIGNATURE----- From arigo at tunes.org Wed Jul 12 22:29:08 2006 From: arigo at tunes.org (Armin Rigo) Date: Wed, 12 Jul 2006 22:29:08 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: References: <17588.18475.171696.506911@montanaro.dyndns.org> Message-ID: <20060712202908.GA13303@code0.codespeak.net> Hi Brett, On Tue, Jul 11, 2006 at 06:05:21PM -0700, Brett Cannon wrote: > It is the last point in the first paragraph on time.strftime() discussing > what changed in Python 2.4 as to what the change was. It's also in > Misc/NEWS . Basically the guy didn't read the release notes or the docs to > see why that changed and that it was legitimate and needed for stability. Surely everybody should read and think carefully about each (longish) NEWS file for each software package whenever they update their machines or switch to one with newer software than they last used. Or if they cannot bother, surely they should read at least Python's? I guess I'm going to side with Greg Black on his blog entry. Only two breakages is certainly nice, and I know that we all try quite hard to minimize that; that's probably still two breakages too much. Armin From nnorwitz at gmail.com Wed Jul 12 23:09:23 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 12 Jul 2006 14:09:23 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <20060712202908.GA13303@code0.codespeak.net> References: <17588.18475.171696.506911@montanaro.dyndns.org> <20060712202908.GA13303@code0.codespeak.net> Message-ID: On 7/12/06, Armin Rigo wrote: > > Only two breakages is certainly nice, and I know that we all try quite > hard to minimize that; that's probably still two breakages too much. I agree, but some of this responsibility has to fall to users. Sometimes these breakages are bugs, pure and simple. Our tests don't catch everything. This is why it's really, really important to get as many alpha/beta testers as possible. Had the issues been brought up before 2.4 was released, we could have addressed them. Basically, if you care about these things, test before we release. That way the issues can be addressed. If you don't care enough to test and get bitten by a problem, quit whining, you had your chance. I have no sympathy for someone who is only willing to whine after the fact, but unwilling to do any work. That's really not helpful. n -- PS People do this all the time in politics too. They complain about someone, when they couldn't even be bothered to vote. From jimjjewett at gmail.com Wed Jul 12 23:23:35 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 12 Jul 2006 17:23:35 -0400 Subject: [Python-Dev] Restricted execution: what's the threat model? Message-ID: Ka-Ping Yee writes: > A. The interpreter will not crash no matter what Python code > it is given to execute. Why? We don't want it to crash the embedding app (which might be another python interpreter), but if the sandboxed interpreter itself crashes, is that so bad? The embedding app should just act as though that interpreter exited, possibly with a status code. > B. Python programs running in different interpreters embedded > in the same process cannot communicate with each other. Why not? Can't eavesdrop, yes. Can't force a connection, so that the other interpreter is free to ignore them. Maybe even make it lockable, like sockets -- but it isn't something worth promising. > C. Python programs running in different interpreters embedded > in the same process cannot access each other's Python objects. Note that Brett's assumption of shared extension modules violates this -- but I'm not sure why he needs to assume that. (Because of the init-only-once semantics, I'm not even sure it is a good idea to share them.) > D. A given piece of Python code cannot access or communicate > with certain Python objects in the same interpreter. Why not? Is this just a way of allowing lightweight subinterpreters? Or do you really mean that they can't replace or modify certain objects, such as the permission-controlling code? > E. A given piece of Python code can access only a limited set > of Python objects in the same interpreter. Does this include objects it creates? Or are you just saying that it will behave as if its builtins were segregated, and not see changes made by another interpreter? -jJ From bob at redivi.com Wed Jul 12 23:50:13 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 12 Jul 2006 14:50:13 -0700 Subject: [Python-Dev] Restricted execution: what's the threat model? In-Reply-To: References: Message-ID: On Jul 12, 2006, at 2:23 PM, Jim Jewett wrote: > Ka-Ping Yee writes: > >> A. The interpreter will not crash no matter what Python code >> it is given to execute. > > Why? > > We don't want it to crash the embedding app (which might be another > python interpreter), but if the sandboxed interpreter itself crashes, > is that so bad? The embedding app should just act as though that > interpreter exited, possibly with a status code. When he says crash, I'd have to imagine that he means of the segfault variety. Good luck saving the embedding app after that. >> C. Python programs running in different interpreters embedded >> in the same process cannot access each other's Python objects. > > Note that Brett's assumption of shared extension modules violates this > -- but I'm not sure why he needs to assume that. (Because of the > init-only-once semantics, I'm not even sure it is a good idea to share > them.) Well if you don't share them, you can't have them at all other than in the main trusted interpreter. C extensions can only be safely initialized once and they often cache objects in static variables... lots of C modules aren't even safe to use when combined with multiple interpreters and threads (e.g. PyGILState API), so I guess that perhaps the C API should be refined anyway. -bob From fredrik at pythonware.com Thu Jul 13 00:13:51 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 13 Jul 2006 00:13:51 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: Boris Borcic wrote: >> note that most examples of this type already work, if the target type is >> mutable, and implement the right operations: >> >> def counter(num): >> num = mutable_int(num) >> def inc(): >> num += 1 >> return num >> return inc > > I agree with you (and argued it in "scopes vs augmented assignment vs sets" > recently) that mutating would be sufficient /if/ the compiler would view > augmented assignment as mutations operators feel free to replace that += with an .add(1) method call; the point wasn't the behaviour of augmented assigment, the point was that that the most common use pattern involves *mutation* of the target object. the syntax isn't that important, really. From python-dev at zesty.ca Thu Jul 13 00:38:44 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Wed, 12 Jul 2006 17:38:44 -0500 (CDT) Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: On Thu, 13 Jul 2006, Fredrik Lundh wrote: > Boris Borcic wrote: > > >> note that most examples of this type already work, if the target type is > >> mutable, and implement the right operations: > >> > >> def counter(num): > >> num = mutable_int(num) > >> def inc(): > >> num += 1 > >> return num > >> return inc [...] > feel free to replace that += with an .add(1) method call; the point > wasn't the behaviour of augmented assigment, the point was that that the > most common use pattern involves *mutation* of the target object. I don't think that's clear. In: a = 1 b = a a += 1 it doesn't seem common to me that one would expect b to become 2. Sometimes you want mutation, sometimes you don't. I don't think that necessarily follows from whether you are accessing a free variable. -- ?!ng From brett at python.org Thu Jul 13 00:51:45 2006 From: brett at python.org (Brett Cannon) Date: Wed, 12 Jul 2006 15:51:45 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <20060712202908.GA13303@code0.codespeak.net> References: <17588.18475.171696.506911@montanaro.dyndns.org> <20060712202908.GA13303@code0.codespeak.net> Message-ID: On 7/12/06, Armin Rigo wrote: > > Hi Brett, > > On Tue, Jul 11, 2006 at 06:05:21PM -0700, Brett Cannon wrote: > > It is the last point in the first paragraph on time.strftime() > discussing > > what changed in Python 2.4 as to what the change was. It's also in > > Misc/NEWS . Basically the guy didn't read the release notes or the docs > to > > see why that changed and that it was legitimate and needed for > stability. > > Surely everybody should read and think carefully about each (longish) > NEWS file for each software package whenever they update their machines > or switch to one with newer software than they last used. > > Or if they cannot bother, surely they should read at least Python's? Obviously I don't expect people to read all of the docs; I know I don't. But when something I am using starts to act differently in a program (and especially a programming language), I do do at least a little investigating. I don't expect people to read Misc/NEWS upfront, but if something suddnely starts to act differently I would expect them to at least try to figure it out and hopefully report a bug if it is one. I guess I'm going to side with Greg Black on his blog entry. > Only two breakages is certainly nice, and I know that we all try quite > hard to minimize that; that's probably still two breakages too much. As Neal said, we are not perfect; bugs happen. If we all gave up on a piece of software after two bugs we would not be able to turn our computers. =) -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/f1d7446b/attachment.htm From brett at python.org Thu Jul 13 00:58:50 2006 From: brett at python.org (Brett Cannon) Date: Wed, 12 Jul 2006 15:58:50 -0700 Subject: [Python-Dev] Restricted execution: what's the threat model? In-Reply-To: References: Message-ID: On 7/12/06, Jim Jewett wrote: > > Ka-Ping Yee writes: > > > A. The interpreter will not crash no matter what Python code > > it is given to execute. > > Why? > > We don't want it to crash the embedding app (which might be another > python interpreter), but if the sandboxed interpreter itself crashes, > is that so bad? The embedding app should just act as though that > interpreter exited, possibly with a status code. As Bob said, "crash" means segfaulting the Python proceess. Can't exactly save yourself from that one easily. =) > B. Python programs running in different interpreters embedded > > in the same process cannot communicate with each other. > > Why not? Can't eavesdrop, yes. Can't force a connection, so that > the other interpreter is free to ignore them. Maybe even make it > lockable, like sockets -- but it isn't something worth promising. >From an initial design point it is. It is an assumption I want to be able to make about the design. If we can come up with a reasonable way for it to work, great. But to begin with, I am assuming objects created within an interpreter will not be passed into another one. > C. Python programs running in different interpreters embedded > > in the same process cannot access each other's Python objects. > > Note that Brett's assumption of shared extension modules violates this > -- but I'm not sure why he needs to assume that. (Because of the > init-only-once semantics, I'm not even sure it is a good idea to share > them.) Security reasons. If I can get a Python object in other interpreter with more rights to do something for me I have a serious problem. Do realize that things assumed might have to be made true as much as possible. And in my Threat Model list, I have the caveat that C extension modules are exempt from this. > D. A given piece of Python code cannot access or communicate > > with certain Python objects in the same interpreter. > > Why not? Is this just a way of allowing lightweight subinterpreters? > Or do you really mean that they can't replace or modify certain > objects, such as the permission-controlling code? This one is not in my Threat Model. > E. A given piece of Python code can access only a limited set > > of Python objects in the same interpreter. > > Does this include objects it creates? Or are you just saying that it > will behave as if its builtins were segregated, and not see changes > made by another interpreter? I am going with your latter assumption in my design. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/98b621f6/attachment.html From guido at python.org Thu Jul 13 03:34:48 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 12 Jul 2006 18:34:48 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <20060712202908.GA13303@code0.codespeak.net> References: <17588.18475.171696.506911@montanaro.dyndns.org> <20060712202908.GA13303@code0.codespeak.net> Message-ID: On 7/12/06, Armin Rigo wrote: > I guess I'm going to side with Greg Black on his blog entry. I seem to recall that that particular one wass *not* an accidental bug. I believe I fell over exactly the problem that Greg Black complained about (or almost the same; maybe my problem was setting the month and day to zero and formatting only the time :-) and tried to convince the author to change it; but was told that the new behavior was never documented and that the new behavior was somehow better; and I didn't fight it further then. Do I remember this correctly? Does anybody else recall? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Thu Jul 13 03:42:49 2006 From: brett at python.org (Brett Cannon) Date: Wed, 12 Jul 2006 18:42:49 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: References: <17588.18475.171696.506911@montanaro.dyndns.org> <20060712202908.GA13303@code0.codespeak.net> Message-ID: On 7/12/06, Guido van Rossum wrote: > > On 7/12/06, Armin Rigo wrote: > > I guess I'm going to side with Greg Black on his blog entry. > > I seem to recall that that particular one wass *not* an accidental > bug. I believe I fell over exactly the problem that Greg Black > complained about (or almost the same; maybe my problem was setting the > month and day to zero and formatting only the time :-) and tried to > convince the author to change it; but was told that the new behavior > was never documented and that the new behavior was somehow better; and > I didn't fight it further then. Do I remember this correctly? Does > anybody else recall? My recollection is that we found a way to cause a crash if improper values were used. We never said 0s were allowed in the docs and that it could mask bugs if you did use them and we supported them (e.g., setting 0 for January instead of 1 if you were thinking in terms of indexing at the time). So we said that values should be within the proper range and not above or below them. The python-dev Summary coverage can be found at http://www.python.org/dev/summary/2004-02-01_2004-02-29/#time-strftime-now-checks-its-argument-s-bounds. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/afe7da0c/attachment.htm From guido at python.org Thu Jul 13 06:46:21 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 12 Jul 2006 21:46:21 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: References: <17588.18475.171696.506911@montanaro.dyndns.org> <20060712202908.GA13303@code0.codespeak.net> Message-ID: On 7/12/06, Brett Cannon wrote: > On 7/12/06, Guido van Rossum wrote: > > > On 7/12/06, Armin Rigo wrote: > > > I guess I'm going to side with Greg Black on his blog entry. > > > > I seem to recall that that particular one wass *not* an accidental > > bug. I believe I fell over exactly the problem that Greg Black > > complained about (or almost the same; maybe my problem was setting the > > month and day to zero and formatting only the time :-) and tried to > > convince the author to change it; but was told that the new behavior > > was never documented and that the new behavior was somehow better; and > > I didn't fight it further then. Do I remember this correctly? Does > > anybody else recall? > > > My recollection is that we found a way to cause a crash if improper values > were used. We never said 0s were allowed in the docs and that it could mask > bugs if you did use them and we supported them ( e.g., setting 0 for January > instead of 1 if you were thinking in terms of indexing at the time). So we > said that values should be within the proper range and not above or below > them. > > The python-dev Summary coverage can be found at > http://www.python.org/dev/summary/2004-02-01_2004-02-29/#time-strftime-now-checks-its-argument-s-bounds Thanks for confirming memory! So it was an intentional regression; "bugs happen" doesn't apply in this case. And an unfortunate regression at that -- not because one guy writes a silly blog entry about it, but because it breaks real code -- undocumented be damned. Are we going to fix it, without allowing those crashes again? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Thu Jul 13 06:56:10 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 13 Jul 2006 14:56:10 +1000 Subject: [Python-Dev] User's complaints In-Reply-To: References: Message-ID: <200607131456.16309.anthony@interlink.com.au> On Thursday 13 July 2006 14:46, Guido van Rossum wrote: > Thanks for confirming memory! So it was an intentional regression; > "bugs happen" doesn't apply in this case. And an unfortunate > regression at that -- not because one guy writes a silly blog entry > about it, but because it breaks real code -- undocumented be > damned. > > Are we going to fix it, without allowing those crashes again? I already said "yes" for 2.5. Brett, can you work up a patch? Anthony -- Anthony Baxter It's never too late to have a happy childhood. From brett at python.org Thu Jul 13 07:06:25 2006 From: brett at python.org (Brett Cannon) Date: Wed, 12 Jul 2006 22:06:25 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <200607131456.16309.anthony@interlink.com.au> References: <200607131456.16309.anthony@interlink.com.au> Message-ID: On 7/12/06, Anthony Baxter wrote: > > On Thursday 13 July 2006 14:46, Guido van Rossum wrote: > > Thanks for confirming memory! So it was an intentional regression; > > "bugs happen" doesn't apply in this case. And an unfortunate > > regression at that -- not because one guy writes a silly blog entry > > about it, but because it breaks real code -- undocumented be > > damned. > > > > Are we going to fix it, without allowing those crashes again? > > I already said "yes" for 2.5. Brett, can you work up a patch? Yep, it's on my todo list and will get done. Might have to wait until next week, though (if that is okay), since my girlfriend is leaving town Sunday and thus I will have a ton more time after that. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060712/3aae1f34/attachment.html From ashemedai at gmail.com Thu Jul 13 07:43:30 2006 From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven) Date: Thu, 13 Jul 2006 07:43:30 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: References: Message-ID: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> On 7/5/06, Neal Norwitz wrote: > For example, we heard grumblings about the releases coming too often. > Once we went to an 18 month release schedule, there was minimal > complaining. It should be fairly safe to assume this silence means > people think we are doing a good job. What are the things that could > be fixed that would silence the most number of user's complaints? I frequent a lot of IRC channels either devoted to Python or heaving people use Python a lot next to my own use so I think I can, hopefully, give some comments. Of course, things are highly subjective. When some of us first saw what PEP 3000 suggested we were thinking: shit, there goes Python. It incredibly felt like Python's best measures were being dumbed down. The release cycle is not bothering people from what I gather, especially not given the fact that the world is still changing when it comes to things like XML. What is bothering people is how you have to reinstall all your site-packages when you go from one major version to another. I understand this might be a difficult problem to tackle, but I can also understand the hassle if people have more than 10-15 modules installed. Another point is how the module documentation is very terse in a lot of areas. Especially when it concerns what kind of exceptions a particular function can/will raise. As a professional, part-time, technical writer I wonder how to best tackle the entirety of the documentation we have. (Note that this is in no way any jibe towards Fred Drake, since I bet it's quite a bulk of documentation to work with/on every time.) Also it can get quite confusing when you need the introductory manual, when the language reference and when the module documentation. Things that struck me as peculiar is the old: if __name__ == "__main__": whatever() This is so out of tune with the rest of python it becomes a nuisance. > PS. One thing I tend to talk to users about is stability of the > interpreter. When I talk about crashing the interpreter, the most > common first reaction I get is "you can crash the interpreter? How do > you do that?" I take that answer as a good sign. :-) I've come relatively late to the Python scene, but ever since 2.2 I never once managed to crash the interpreter. I'll ask around and see what people are reporting to me as their top 3 or 5 Python complaints though. -- Jeroen Ruigrok van der Werven From nnorwitz at gmail.com Thu Jul 13 08:24:22 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 12 Jul 2006 23:24:22 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> Message-ID: On 7/12/06, Jeroen Ruigrok van der Werven wrote: > On 7/5/06, Neal Norwitz wrote: > > For example, we heard grumblings about the releases coming too often. > > Once we went to an 18 month release schedule, there was minimal > > complaining. It should be fairly safe to assume this silence means > > people think we are doing a good job. What are the things that could > > be fixed that would silence the most number of user's complaints? > > I frequent a lot of IRC channels either devoted to Python or heaving > people use Python a lot next to my own use so I think I can, > hopefully, give some comments. Of course, things are highly > subjective. Jeroen, Thank you very much for your feedback. It helps. > The release cycle is not bothering people from what I gather, > especially not given the fact that the world is still changing when it > comes to things like XML. What is bothering people is how you have to > reinstall all your site-packages when you go from one major version to > another. I understand this might be a difficult problem to tackle, but > I can also understand the hassle if people have more than 10-15 > modules installed. If it's pure python, why don't people just copy everything under site-packages after installing? They could/should run compileall after that to recompile the .pyc files. With 2.5 on 64-bit machines, C extension modules *must* be recompiled due to lots of internal changes. One thing you didn't mention that I've heard from time to time is the stdlib should be improved. For example, cleaning up old modules. Though I'm not really sure everyone has the same thing in mind when it comes to improving the stdlib. > Another point is how the module documentation is very terse in a lot > of areas. Especially when it concerns what kind of exceptions a > particular function can/will raise. As a professional, part-time, > technical writer I wonder how to best tackle the entirety of the > documentation we have. (Note that this is in no way any jibe towards > Fred Drake, since I bet it's quite a bulk of documentation to work > with/on every time.) > Also it can get quite confusing when you need the introductory manual, > when the language reference and when the module documentation. Do you think you could help with the doc? How can we get people, especially tech writers, interested in improving the doc? Most people agree it's important, but few make time to really improve the doc. We've talked about making it easier for people to contribute to the docs, perhaps adding something like a wiki/comments. Do you think that would help? > Things that struck me as peculiar is the old: > > if __name__ == "__main__": > whatever() > > This is so out of tune with the rest of python it becomes a nuisance. I'm not sure I understand your point. Can you provide more info about what you dislike/expect instead? > I'll ask around and see what people are reporting to me as their top 3 > or 5 Python complaints though. Great. Thanks! n From tds333+pydev at gmail.com Thu Jul 13 09:37:45 2006 From: tds333+pydev at gmail.com (Wolfgang Langner) Date: Thu, 13 Jul 2006 09:37:45 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> Message-ID: <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> On 7/13/06, Jeroen Ruigrok van der Werven wrote: > Things that struck me as peculiar is the old: > > if __name__ == "__main__": > whatever() > > This is so out of tune with the rest of python it becomes a nuisance. It is not beautiful but very useful. In Python 3000 we can replace it with: @main def whatever(): ... to mark this function as main function if module executed directly. -- bye by Wolfgang From python-dev at zesty.ca Thu Jul 13 09:44:41 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Thu, 13 Jul 2006 02:44:41 -0500 (CDT) Subject: [Python-Dev] User's complaints In-Reply-To: <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> Message-ID: On Thu, 13 Jul 2006, Wolfgang Langner wrote: > On 7/13/06, Jeroen Ruigrok van der Werven wrote: > > Things that struck me as peculiar is the old: > > > > if __name__ == "__main__": > > whatever() > > > > This is so out of tune with the rest of python it becomes a nuisance. > > It is not beautiful but very useful. > In Python 3000 we can replace it with: > > @main > def whatever(): > ... > > to mark this function as main function if module executed directly. Why not simply: def __main__(): ... or even pass in the command-line arguments: def __main__(*args): ... Having to 'import sys' to get at the command-line arguments always seemed awkward to me. 'import sys' feels like it should be a privileged operation (access to interpreter internals), and getting the command-line args isn't privileged. -- ?!ng From ashemedai at gmail.com Thu Jul 13 09:52:21 2006 From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven) Date: Thu, 13 Jul 2006 09:52:21 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> Message-ID: <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> On 7/13/06, Neal Norwitz wrote: > On 7/12/06, Jeroen Ruigrok van der Werven wrote: > Thank you very much for your feedback. It helps. With apologies in advance if my own level of understanding is, of course, lacking of advanced constructs. > If it's pure python, why don't people just copy everything under > site-packages after installing? They could/should run compileall > after that to recompile the .pyc files. With 2.5 on 64-bit machines, > C extension modules *must* be recompiled due to lots of internal > changes. I wasn't even aware of the compileall step, can you elaborate since this is the first time I see it being mentioned. > One thing you didn't mention that I've heard from time to time is the > stdlib should be improved. For example, cleaning up old modules. > Though I'm not really sure everyone has the same thing in mind when it > comes to improving the stdlib. How do you envision cleaning up old modules? > Do you think you could help with the doc? How can we get people, > especially tech writers, interested in improving the doc? Most people > agree it's important, but few make time to really improve the doc. > We've talked about making it easier for people to contribute to the > docs, perhaps adding something like a wiki/comments. Do you think > that would help? Wiki/comments are useful, sure, the only problem is that if you have to sift through every single page to spot a useful comment it is not helping. I never liked PHP's comment system to take an example. Personally, but that's just me of course, I prefer a periodic evaluation of comments and reintegration of said comments into the documentation itself. I've been looking at Python's docstrings in the main codebase and that use alone is inconsistent, some areas feature a lot of docstrings, some have none. But with alone docstrings you will not cut it I think. I hope to get around in the coming time to work up some documentation changes/patches and submit these. [if __name__ == "__main__":] > I'm not sure I understand your point. Can you provide more info about > what you dislike/expect instead? Not the above construct at least. :) To me it just feels like a kludge, perhaps due to the fact that the rest of Python just flows from your fingers when writing/programming. Perhaps one could even get away with it in the form of: def __main__(): ...main program... Some other complaints/wishes from a hard core pythonista: - There's no support for real macros. - Writable closures would be nice. - Open classes would be nice. -- Jeroen Ruigrok van der Werven From bob at redivi.com Thu Jul 13 09:58:08 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 13 Jul 2006 00:58:08 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> Message-ID: <6713F515-C684-486E-8FFF-EAC60C83E069@redivi.com> On Jul 13, 2006, at 12:37 AM, Wolfgang Langner wrote: > On 7/13/06, Jeroen Ruigrok van der Werven wrote: >> Things that struck me as peculiar is the old: >> >> if __name__ == "__main__": >> whatever() >> >> This is so out of tune with the rest of python it becomes a nuisance. > > It is not beautiful but very useful. > In Python 3000 we can replace it with: > > @main > def whatever(): > ... > > to mark this function as main function if module executed directly. It would probably need to be called something else, because main is often the name of the main function... but you could write such a decorator now if you really wanted to. def mainfunc(fn): if fn.func_globals.get('__name__') == '__main__': # ensure the function is in globals fn.func_globals[fn.__name__] = fn fn() return fn @mainfunc def main(): print 'this is in __main__' -bob From mike at skew.org Thu Jul 13 10:26:43 2006 From: mike at skew.org (Mike Brown) Date: Thu, 13 Jul 2006 02:26:43 -0600 (MDT) Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt In-Reply-To: <44B4B274.4050602@ofai.at> Message-ID: <200607130826.k6D8QhHA077212@chilled.skew.org> Stefan Rank wrote: > on 12.07.2006 07:53 Martin v. L?wis said the following: > > Anthony Baxter wrote: > >>> The right thing to do is IRIs. > >> For 2.5, should we at least detect that it's unicode and raise a > >> useful error? > > > > That can certainly be done, sure. > > > > Martin > > That would be great. > > And I agree that updating urllib.quote for unicode should be part of a > grand plan that updates all of urllib[2] and introduces an irilib / > urischemes / uriparse module in 2.6 as Martin and John J Lee suggested. > =) > > cheers, > stefan Put me down as +1 on raising a useful error instead of a KeyError or whatever, and +1 on having an irilib, but -1 on working toward accepting unicode in the URI-oriented urllib.quote(), because (a.) user expectations for strings that contain non-ASCII-range characters will vary, and (b.) percent-encoding is supposed to only operate on a byte-encoded view of non-URI information, not the information itself. Longer explanation: I, too, initially thought that quote() was outdated since it choked on unicode input, but eventually I came to realize that it's wise to reject such input, because to attempt to percent-encode characters, rather than bytes, reflects a fundamental misunderstanding of the level at which percent-encoding is intended to operate. This is one of the hardest aspects of URI processing to grok, and I'm not very good at explaining it, even though I've tried my best in the Wikipedia articles. It's basically these 3 points: 1. A URI can only consist of 'unreserved' characters, as I'm sure you know. It's a specific set that has varied slightly over the years, and is a subset of printable ASCII. 2. A URI scheme is essentially a mapping of non-URI information to a sequence of URI characters. That is, it is a method of producing a URI from non-URI information within a particular information domain ...and vice-versa. 3. A URI scheme should (though may not do so very clearly, especially the older it is!) tell you that the way to represent a particular bit of non-URI information, 'info', in a URI is to convert_to_bytes(info), and then, as per STD 66, make the bytes that correspond, in ASCII, to unreserved characters manifest as those characters, and all others manifest as their percent-encoded equivalents. In urllib parlance, this step is 'quoting' the bytes. 3.1. [This isn't crucial to my argument, but has to be mentioned to complete the explanation of percent-encoding.] In addition, those bytes corresponding, in ASCII, to some 'reserved' characters are exempt from needing to be percent-encoded, so long as they're not being used for their reserved purpose (if any) in whatever URI component they're going into -- Semantically, there's no difference between such bytes when expressed in the URI as a literal reserved character or as a percent-encoded byte. URI scheme specs vary greatly in how they deal with this nuance. In any case, urllib.quote() has the 'safe' argument which can be used to specify the exempt reserved characters. In the days when the specs that urllib was based on were relevant, 99% of the time, the bytes being 'quoted' were ASCII-encoded strings representing ASCII character-based non-URI information, so quite a few of us, including many URI scheme authors, were tempted to think that what was being 'quoted'/percent-encoded *was* the original non-URI information, rather than a bytewise view of it mandated by a URI scheme. That's what I was doing when I thought that quote(some_unicode_path) should 'work', especially in light of Python's "treat all strings alike" guideline. But if you accept all of the above, which is what I believe the standard requires, then unicode input is a very different situation from str input; it's unclear whether and how the caller wants the input to be converted to bytes, if they even understand what they're doing at all. See, right now, quote('abc 123%') returns 'abc%20123%25', as you would expect. Similarly, everyone would probably expect u'abc 123%' to return u'abc%20123%25', and if we were to implement that, there'd probably be no harm done. But look at quote('\xb7'), which, assuming you accept everything I've said above is correct, rightfully returns '%B7'. What would someone expect quote(u'\xb7') to return? Some might want u'%B7' because they want the same result type as the input they gave, with no other changes from how it would normally be handled. Some might want u'%C2%B7' because they're conflating the levels of abstraction and expect, say, UTF-8 conversion to be done on their input. Some (like me) might want a TypeError or ValueError because we shouldn't be handing such ambiguous data to quote() in the first place. And then there's the u'\u0100'-and-up input to worry about; what does a user expect to be done with that? I would prefer to see quote() always reject unicode input with a TypeError. Alternatively, if it accepts unicode, it should produce unicode, and since it can only reasonably assume what the user wants done with ASCII-range characters, it should only accept input < u'\x80'. In any case, quote() should be better documented to explain what it accepts ( a byte sequence ) and why ( it is intended to be used at the stage of URI production where non-URI info, such as a unicode filesystem path, has already been converted to bytes according to the requirements of a URI scheme, and now needs to be represented as a URI-safe character sequence ) and exactly what it produces ( a str representing URI character s). Mike From bingham at cenix-bioscience.com Thu Jul 13 10:07:44 2006 From: bingham at cenix-bioscience.com (Aaron Bingham) Date: Thu, 13 Jul 2006 10:07:44 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> Message-ID: <44B5FF50.3060000@cenix-bioscience.com> Ka-Ping Yee wrote: >On Thu, 13 Jul 2006, Wolfgang Langner wrote: > > >>On 7/13/06, Jeroen Ruigrok van der Werven wrote: >> >> >>>Things that struck me as peculiar is the old: >>> >>>if __name__ == "__main__": >>> whatever() >>> >>>This is so out of tune with the rest of python it becomes a nuisance. >>> >>> >>It is not beautiful but very useful. >>In Python 3000 we can replace it with: >> >>@main >>def whatever(): >> ... >> >>to mark this function as main function if module executed directly. >> >> > >Why not simply: > > def __main__(): > ... > >or even pass in the command-line arguments: > > def __main__(*args): > ... > >Having to 'import sys' to get at the command-line arguments always >seemed awkward to me. 'import sys' feels like it should be a >privileged operation (access to interpreter internals), and getting >the command-line args isn't privileged. > > +1, seems a lot more elegant than "if __name__ == '__main__'" Regards, -- -------------------------------------------------------------------- Aaron Bingham Senior Software Engineer Cenix BioScience GmbH -------------------------------------------------------------------- From greg.ewing at canterbury.ac.nz Thu Jul 13 10:49:37 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 13 Jul 2006 20:49:37 +1200 Subject: [Python-Dev] User's complaints In-Reply-To: <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> Message-ID: <44B60921.9010300@canterbury.ac.nz> Wolfgang Langner wrote: > @main > def whatever(): > ... This seems like replacing one unpythonic feature with another. (I *still* can't get used to that @ syntax -- it looks like an intruder from Rubyland...) -- Greg From bingham at cenix-bioscience.com Thu Jul 13 10:53:59 2006 From: bingham at cenix-bioscience.com (Aaron Bingham) Date: Thu, 13 Jul 2006 10:53:59 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: <44B5FF50.3060000@cenix-bioscience.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B5FF50.3060000@cenix-bioscience.com> Message-ID: <44B60A27.2040303@cenix-bioscience.com> Aaron Bingham wrote: >Ka-Ping Yee wrote: > > > >>Why not simply: >> >> def __main__(): >> ... >> >>or even pass in the command-line arguments: >> >> def __main__(*args): >> ... >> >>Having to 'import sys' to get at the command-line arguments always >>seemed awkward to me. 'import sys' feels like it should be a >>privileged operation (access to interpreter internals), and getting >>the command-line args isn't privileged. >> >> >> >> >+1, seems a lot more elegant than "if __name__ == '__main__'" > > Just to clarify, I was writing in support of Ping's suggestion of def __main__() or def __main__(*args). The decorator-based solution is ugly. Regards, -- -------------------------------------------------------------------- Aaron Bingham Senior Software Engineer Cenix BioScience GmbH -------------------------------------------------------------------- From greg.ewing at canterbury.ac.nz Thu Jul 13 10:59:06 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 13 Jul 2006 20:59:06 +1200 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> Message-ID: <44B60B5A.2080309@canterbury.ac.nz> Ka-Ping Yee wrote: > Having to 'import sys' to get at the command-line arguments always > seemed awkward to me. 'import sys' feels like it should be a > privileged operation (access to interpreter internals), and getting > the command-line args isn't privileged. Would it help if sys were pre-imported into the builtins? Or do you think that args shouldn't live in sys at all? Recently I've come to appreciate the ability to get at the args from anywhere, instead of having to catch them from a call to main() and pass them around. So I'd like to still be able to import them from somewhere if I want (doesn't have to be sys, though). And while we're on the subject, anyone think it would be a good idea to drop the silly feature of having the program name as args[0]? You almost *never* want to treat it the same way as the rest of the args, so the first thing you always do is args[1:]. It's not so bad in C, where it's just as easy to start indexing argv from 1 instead of 0. But it makes no sense in Python, IMO. It would be much more sensible to move it into a separate attribute of whatever module we decide to put args in. -- Greg From greg.ewing at canterbury.ac.nz Thu Jul 13 11:02:07 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 13 Jul 2006 21:02:07 +1200 Subject: [Python-Dev] User's complaints In-Reply-To: <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> Message-ID: <44B60C0F.7060405@canterbury.ac.nz> Jeroen Ruigrok van der Werven wrote: > - Open classes would be nice. What do you mean by "open classes"? Python classes already seem pretty open to me, by the standards of other languages! -- Greg From skip at pobox.com Thu Jul 13 12:18:29 2006 From: skip at pobox.com (skip at pobox.com) Date: Thu, 13 Jul 2006 05:18:29 -0500 Subject: [Python-Dev] User's complaints In-Reply-To: <20060712202908.GA13303@code0.codespeak.net> References: <17588.18475.171696.506911@montanaro.dyndns.org> <20060712202908.GA13303@code0.codespeak.net> Message-ID: <17590.7669.73105.356443@montanaro.dyndns.org> Armin> On Tue, Jul 11, 2006 at 06:05:21PM -0700, Brett Cannon wrote: >> It is the last point in the first paragraph on time.strftime() >> discussing what changed in Python 2.4 as to what the change was. >> It's also in Misc/NEWS . Basically the guy didn't read the release >> notes or the docs to see why that changed and that it was legitimate >> and needed for stability. Armin> Surely everybody should read and think carefully about each Armin> (longish) NEWS file for each software package whenever they Armin> update their machines or switch to one with newer software than Armin> they last used. I'll add one further point here. I exchanged a couple emails with Greg Black. What happened to him was that some of his customers upgraded their operating systems and got Python 2.4 in the bargain. His software started throwing off errors. At that point he read the NEWS file and saw the problem. He clearly wasn't fully master of the environment in which his customers ran his software, so I think it's understandable that he was caught by surprise by this change. Skip From fredrik at pythonware.com Thu Jul 13 12:21:55 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 13 Jul 2006 12:21:55 +0200 Subject: [Python-Dev] Proposal: Add Sudoku Solver To The "this" Module Message-ID: given that java has beaten us with some 60 bytes: http://programming.reddit.com/info/9xha/comments/c9y8b and in order to further improve Python's Kolmogorov rating: http://en.wikipedia.org/wiki/Kolmogorov_complexity how about adding Peter Norvig's constraint-based solver to the Python library: http://www.norvig.com/sudoku.html ? Anthony ? ("import this;this.suduko()" is 25 characters, and we could always optimize this a bit more in Python 2.6, if needed). [\F] From skip at pobox.com Thu Jul 13 12:23:44 2006 From: skip at pobox.com (skip at pobox.com) Date: Thu, 13 Jul 2006 05:23:44 -0500 Subject: [Python-Dev] User's complaints In-Reply-To: References: <17588.18475.171696.506911@montanaro.dyndns.org> <20060712202908.GA13303@code0.codespeak.net> Message-ID: <17590.7984.990599.47651@montanaro.dyndns.org> Neal> I agree, but some of this responsibility has to fall to users. Neal> Sometimes these breakages are bugs, pure and simple. Our tests Neal> don't catch everything. This is why it's really, really important Neal> to get as many alpha/beta testers as possible. Had the issues Neal> been brought up before 2.4 was released, we could have addressed Neal> them. You're of course preaching to the choir here. Somehow we need to do a better job convincing the Python-using public to pay attention to alphas and betas, especially software consultants like Greg Black. Another area where this particular process broke down was that there simply wasn't already a test case for the fairly common case of calling strftime with a six-zeroes pad to format a simple date. Had it been there already, it would have been discovered when Brett added the range checks. Skip From bob at redivi.com Thu Jul 13 12:25:16 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 13 Jul 2006 03:25:16 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <44B60C0F.7060405@canterbury.ac.nz> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> Message-ID: <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> On Jul 13, 2006, at 2:02 AM, Greg Ewing wrote: > Jeroen Ruigrok van der Werven wrote: > >> - Open classes would be nice. > > What do you mean by "open classes"? Python > classes already seem pretty open to me, by > the standards of other languages! I'm guessing he's talking about being like Ruby or Objective-C where you can add methods to any other class in the runtime. Basically we'd have that if the built-in classes were mutable, but that just really encourages fragile code. The big problem you run into with open classes is that you end up depending on two libraries that have a different idea of what the "foo" method on string objects should do. Adding open classes would make it easier to develop DSLs, but you'd only be able to reasonably do one per interpreter (unless you mangled the class in a "with" block or something). -bob From fredrik at pythonware.com Thu Jul 13 12:25:22 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 13 Jul 2006 12:25:22 +0200 Subject: [Python-Dev] User's complaints References: <17588.18475.171696.506911@montanaro.dyndns.org><20060712202908.GA13303@code0.codespeak.net> <17590.7669.73105.356443@montanaro.dyndns.org> Message-ID: skip at pobox.com wrote: > He clearly wasn't fully master of the environment in which his > customers ran his software, so I think it's understandable that he was > caught by surprise by this change. a programmer that's surprised that code that relies on undocumented behaviour might behave differently if run on a different platform ? *that's* surprising. From sreeram at tachyontech.net Thu Jul 13 12:39:37 2006 From: sreeram at tachyontech.net (K.S.Sreeram) Date: Thu, 13 Jul 2006 16:09:37 +0530 Subject: [Python-Dev] Proposal: Add Sudoku Solver To The "this" Module In-Reply-To: References: Message-ID: <44B622E9.4000802@tachyontech.net> Fredrik Lundh wrote: > given that java has beaten us with some 60 bytes: > and in order to further improve Python's Kolmogorov rating: > how about adding Peter Norvig's constraint-based solver to the Python library: lol! (just waiting for somebody to give a serious explanation on why this is a bad idea!) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-dev/attachments/20060713/4e491363/attachment.pgp From fredrik at pythonware.com Thu Jul 13 12:36:07 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 13 Jul 2006 12:36:07 +0200 Subject: [Python-Dev] User's complaints References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com><3e1553560607130052m66899441t50065786495928fe@mail.gmail.com><44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> Message-ID: Bob Ippolito wrote: >> What do you mean by "open classes"? Python >> classes already seem pretty open to me, by >> the standards of other languages! > > I'm guessing he's talking about being like Ruby or Objective-C where > you can add methods to any other class in the runtime. wouldn't a standard multiargument dispatch mechanism solve this, for most (all?) practical use cases. (and while we're at it, wouldn't a standard multiargument dispatch be nice replacement for the instance-oriented lookup we're using today? dispatching on a single value is so last century ;-) From fredrik at pythonware.com Thu Jul 13 12:49:40 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 13 Jul 2006 12:49:40 +0200 Subject: [Python-Dev] Proposal: Add Sudoku Solver To The "this" Module References: <44B622E9.4000802@tachyontech.net> Message-ID: "K.S.Sreeram" wrote: > (just waiting for somebody to give a serious explanation on why this is > a bad idea!) \F might have to post this to comp.lang.python first... From skip at pobox.com Thu Jul 13 12:53:28 2006 From: skip at pobox.com (skip at pobox.com) Date: Thu, 13 Jul 2006 05:53:28 -0500 Subject: [Python-Dev] User's complaints In-Reply-To: References: <17588.18475.171696.506911@montanaro.dyndns.org> <20060712202908.GA13303@code0.codespeak.net> <17590.7669.73105.356443@montanaro.dyndns.org> Message-ID: <17590.9768.952125.349646@montanaro.dyndns.org> >> He clearly wasn't fully master of the environment in which his >> customers ran his software, so I think it's understandable that he >> was caught by surprise by this change. Fredrik> a programmer that's surprised that code that relies on Fredrik> undocumented behaviour might behave differently if run on a Fredrik> different platform ? *that's* surprising. Not necessarily. He may not have realized his customers were going to upgrade their operating systems or that the upgrades would involve a new version of Python. Not everyone tracks every Python release. In this case, a) Python 2.4 had been out long enough to turn up in a new (presumably Linux) OS release, and b) since Mr. Black appears to program in Python for a living he should probably have been more attentive to new releases. Maybe Mr. Black hadn't gotten around to testing his software with 2.4 yet. Maybe the software was written on-site for a client and he didn't have access to it to even test in his own environment. Skip From ashemedai at gmail.com Thu Jul 13 14:02:22 2006 From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven) Date: Thu, 13 Jul 2006 14:02:22 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> Message-ID: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> Hi Bob, On 7/13/06, Bob Ippolito wrote: > Adding open classes would make it easier to develop DSLs, but you'd > only be able to reasonably do one per interpreter (unless you mangled > the class in a "with" block or something). The person whose 'complaints' I was stating says that DSLs (Domain Specific Languages for those who, like me, were confused about the acronym) are a big part of what he is after and one per interpreter is fine by him. He also realises that the application(s) he needs them for might be unusual. He doesn't specifically need the builtin types to be extendable. It's just nice to be able to define a single class in multiple modules. Even C++ allows this to some extent (but not as much as he'd like). He understands the implications of allowing open classes (import vs. no import changes semantics, etc.). Personally, he doesn't care *too* much about newbie safety since he's not a newbie. To quote verbatim: "give me the big guns :-)" And while we're at it, he also stated: "[...] add multiple dispatch to your list of improvements for Python". I hope this clarifies it a bit for other people. -- Jeroen Ruigrok van der Werven From ncoghlan at gmail.com Thu Jul 13 14:57:30 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 13 Jul 2006 22:57:30 +1000 Subject: [Python-Dev] User's complaints In-Reply-To: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> Message-ID: <44B6433A.1080705@gmail.com> Jeroen Ruigrok van der Werven wrote: > Hi Bob, > > On 7/13/06, Bob Ippolito wrote: >> Adding open classes would make it easier to develop DSLs, but you'd >> only be able to reasonably do one per interpreter (unless you mangled >> the class in a "with" block or something). > > The person whose 'complaints' I was stating says that DSLs (Domain > Specific Languages for those who, like me, were confused about the > acronym) are a big part of what he is after and one per interpreter is > fine by him. He also realises that the application(s) he needs them > for might be unusual. He doesn't specifically need the builtin types > to be extendable. It's just nice to be able to define a single class > in multiple modules. Even C++ allows this to some extent (but not as > much as he'd like). I'm somewhat confused as to how Python's classes aren't open. Sure, types like the builtin types that don't have a __dict__ aren't open because there isn't anywhere to put the extensions, but metaclassing lets you do whatever you want to any other class: def extends(orig_cls): if not hasattr(orig_cls, "__dict__"): raise TypeError("Cannot extend %r" % cls) class ExtendMeta(type): def __new__(mcl, name, bases, ns): if len(bases) != 1: raise TypeError("Can only extend single class") if bases[0] is object: return type.__new__(mcl, name, bases, ns) for key, value in ns.iteritems(): if key not in ("__metaclass__", "__dict__"): setattr(orig_cls, key, value) return orig_cls class ExtendCls(object): __metaclass__ = ExtendMeta return ExtendCls >>> class A1(object): ... def method1(self): ... print "Hi, I'm method 1!" ... >>> class A2(extends(A1)): ... def method2(self): ... print "Hi, I'm method 2!" ... >>> x = A1() >>> x.method1() Hi, I'm method 1! >>> x.method2() Hi, I'm method 2! >>> y = A2() >>> y.method1() Hi, I'm method 1! >>> y.method2() Hi, I'm method 2! >>> A1 is A2 True Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From fredrik at pythonware.com Thu Jul 13 15:05:49 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 13 Jul 2006 15:05:49 +0200 Subject: [Python-Dev] User's complaints References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com><3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> <44B6433A.1080705@gmail.com> Message-ID: Nick Coghlan wrote: >> The person whose 'complaints' I was stating says that DSLs (Domain >> Specific Languages for those who, like me, were confused about the >> acronym) are a big part of what he is after and one per interpreter is >> fine by him. He also realises that the application(s) he needs them >> for might be unusual. He doesn't specifically need the builtin types >> to be extendable. It's just nice to be able to define a single class >> in multiple modules. Even C++ allows this to some extent (but not as >> much as he'd like). > > I'm somewhat confused as to how Python's classes aren't open. Sure, types like > the builtin types that don't have a __dict__ aren't open because there isn't > anywhere to put the extensions, but metaclassing lets you do whatever you want > to any other class: you don't even need metaclass trickery to deal with the "define a single class in multiple modules" problem; just use multiple inheritance to bring all the component classes together. From ncoghlan at gmail.com Thu Jul 13 15:15:46 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 13 Jul 2006 23:15:46 +1000 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc In-Reply-To: References: Message-ID: <44B64782.9010706@gmail.com> Barry Warsaw wrote: > For example, I could change inspect locally so that it gets the type > of datetime.timedelta.days without adding a constant to types.py. Or > I could patch pydoc.py directly and leave even inspect.py out of it. > Or I could create some stupid internal type in some stupid internal > module who's only purpose would be to have a handle on member > descriptors. Or I could change datetime to be built-in. (see what I > mean about levels of ickyness? :). > > I'm up for suggestions. I think this would be worthwhile to address > in Python 2.5 since I think it would be good to have an accurate > representation of Python's built-in types in types.py. Ultimately, I > really care about teaching pydoc.help() about instances of these > types so that users can get better help when they encounter them > (such as might be the case in 3rd party extension modules). > > Suggestions are welcome. Could you include a "look up late-breaking types" function in types.py that site.py calls after it finishes setting up the standard library path? Still a little hackish, I know, but it seems less icky than your suggestions above, as it means that fixing anything similar that comes up in the future will only require modification of types.py. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Thu Jul 13 15:21:43 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 13 Jul 2006 23:21:43 +1000 Subject: [Python-Dev] User's complaints In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com><3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> <44B6433A.1080705@gmail.com> Message-ID: <44B648E7.3050403@gmail.com> Fredrik Lundh wrote: > Nick Coghlan wrote: > >>> The person whose 'complaints' I was stating says that DSLs (Domain >>> Specific Languages for those who, like me, were confused about the >>> acronym) are a big part of what he is after and one per interpreter is >>> fine by him. He also realises that the application(s) he needs them >>> for might be unusual. He doesn't specifically need the builtin types >>> to be extendable. It's just nice to be able to define a single class >>> in multiple modules. Even C++ allows this to some extent (but not as >>> much as he'd like). >> I'm somewhat confused as to how Python's classes aren't open. Sure, types like >> the builtin types that don't have a __dict__ aren't open because there isn't >> anywhere to put the extensions, but metaclassing lets you do whatever you want >> to any other class: > > you don't even need metaclass trickery to deal with the "define a single > class in multiple modules" problem; just use multiple inheritance to bring > all the component classes together. I didn't mean to say that I thought what I posted was a sensible thing to do - I was mainly curious as to how close I could get to Ruby's class extension syntax by using metaclasses (with the answer being "very", IMO). If someone really wants to extend a Python class, they can just define a function that does what they want and assign it to whatever method names on whatever mutable classes they feel like. No fancy syntax needed :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Thu Jul 13 16:20:33 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 13 Jul 2006 07:20:33 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> Message-ID: Somebody whose name doesn't matter (it's not about him) wrote: > When some of us first saw what PEP 3000 suggested we were thinking: > shit, there goes Python. [...] And later in the same message the same person wrote: > Things that struck me as peculiar is the old: > > if __name__ == "__main__": > whatever() > > This is so out of tune with the rest of python it becomes a nuisance. This is an illustration of the dilemma of maintaining a popular language: Everybody hates change (me too!) but everybody also has one thing that's bothering them so much they absolutely want it to be changed. If you were to implement all those personal pet peeves, you'd get a language that's more different from Python than Python is from Fortran. So where's the middle ground? I believe it's established without a doubt that in biological evolution, changes comes in spurts: A species may change hardly at all for millions of years, and then suddenly, due to not quite understood mechanisms, it starts to change rapidly until a completely new species (or several) has evolved, which again remains stable for a long time. I don't want to adopt this time scale for Python (:-), but I do think it's useful to think of language evolution as a kind of fractal movement -- at every time scale, there are small jumps and the occasional large jump. Python 2.2 was a fairly large jump (new-style classes, iterators, generators, decorators). Python 3000 will be the largest jump so far. There will be larger jumps yet in the distant future. But in between, there will be long periods of (relative) stability. Will it hurt? You bet! But for many species, sooner or later it's evolve or become extinct. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From stefan.rank at ofai.at Thu Jul 13 16:29:15 2006 From: stefan.rank at ofai.at (Stefan Rank) Date: Thu, 13 Jul 2006 16:29:15 +0200 Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt In-Reply-To: <200607130826.k6D8QhHA077212@chilled.skew.org> References: <44B4B274.4050602@ofai.at> <200607130826.k6D8QhHA077212@chilled.skew.org> Message-ID: <44B658BB.5090308@ofai.at> on 13.07.2006 10:26 Mike Brown said the following: > Stefan Rank wrote: >> on 12.07.2006 07:53 Martin v. L?wis said the following: >>> Anthony Baxter wrote: >>>>> The right thing to do is IRIs. >>>> For 2.5, should we at least detect that it's unicode and raise a >>>> useful error? >>> That can certainly be done, sure. > Put me down as +1 on raising a useful error instead of a KeyError or whatever, > and +1 on having an irilib, but -1 on working toward accepting unicode in the > URI-oriented urllib.quote(), because > See, right now, quote('abc 123%') returns 'abc%20123%25', as you would expect. > Similarly, everyone would probably expect u'abc 123%' to return > u'abc%20123%25', and if we were to implement that, there'd probably be no harm > done. Well, originally, I would have expected it to return a byte str(ing), BUT I am now converted and think it is best to raise a TypeError for unicode, and leave the encoding decisions to higher level code. So I'll repeat the "patch" #1, slightly modified:: if isinstance(s, unicode): raise TypeError("quote expects an encoded byte string as argument") Is it safe to assume that code that breaks because of this change was already broken? stefan From barry at python.org Thu Jul 13 18:12:28 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 13 Jul 2006 12:12:28 -0400 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc In-Reply-To: <44B64782.9010706@gmail.com> References: <44B64782.9010706@gmail.com> Message-ID: <7FE62713-24B0-47CE-BA94-41150DB683F4@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 13, 2006, at 9:15 AM, Nick Coghlan wrote: > Could you include a "look up late-breaking types" function in > types.py that site.py calls after it finishes setting up the > standard library path? > > Still a little hackish, I know, but it seems less icky than your > suggestions above, as it means that fixing anything similar that > comes up in the future will only require modification of types.py. I've come to the conclusion that it's much better to add a "helper" module that defines a few extension types solely for the purposes of exposing them in types.py, than it is to rely on accidents of implementation discovered in the existing interpreter. To that end, I've implemented a _typesmodule.c that exposes a getset descriptor and a member descriptor on a type that can't be instantiated. That module gets built into Python and provides a central place to add any future C extension types. It also provides a handy place for Jython or IronPython to hang their own equivalents, if and when they ever get them. Then in types.py, we import _types and pull the appropriate types from there. types.py will not define GetSetType or MemberDescriptorType if that module can't be imported so that it's robust for other Python implementations. I like not defining them better than setting them to None in that case. Then inspect.py defines isgetset() and ismemberdescriptor() in terms of those types, but it always returns False if those types don't exist. So inspect.py too is robust for other Python implementations. I've updated SF patch #1520294 and assigned it back to Georg for another quick review. Assuming he likes the updated patch and Anthony approves, please assign it back to me for final checkin. In thinking about this more, re: back porting to Python 2.4, I don't think we can expose GetSetType, MemberDescriptorType, isgetset(), or ismemberdescriptor() in the appropriate modules. That smells too much like the True/False thing. However, I believe I can weedle a fix into pydoc for those types without exposing these new names, so I'd like to do that if there are no objections. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRLZw8XEjvBPtnXfVAQLkIAP+O0WXh+xO902KNah/kIYy/GSPsO/CfRUO +YchtF2HdbyEXORwhsLXJVTKVROYgohCQrZp7LUX5vNUah0J0ycIPXhzpqrB0Gmc 8AaRChkFO8EGofzEGOvtunGUGdl9733u2mNUUAhGk15YFs8GDyiuzkL8SitcJQ3q OHnCsdNRuSU= =pbA8 -----END PGP SIGNATURE----- From jeremy at alum.mit.edu Thu Jul 13 18:14:50 2006 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu, 13 Jul 2006 12:14:50 -0400 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: On 7/12/06, Fredrik Lundh wrote: > Boris Borcic wrote: > > >> note that most examples of this type already work, if the target type is > >> mutable, and implement the right operations: > >> > >> def counter(num): > >> num = mutable_int(num) > >> def inc(): > >> num += 1 > >> return num > >> return inc > > > > I agree with you (and argued it in "scopes vs augmented assignment vs sets" > > recently) that mutating would be sufficient /if/ the compiler would view > > augmented assignment as mutations operators > > feel free to replace that += with an .add(1) method call; the point > wasn't the behaviour of augmented assigment, the point was that that the > most common use pattern involves *mutation* of the target object. > > the syntax isn't that important, really. Mutation is different from rebinding. A tuple is immutable, but you can rebind the variable that refers to the tuple. I think we will confuse users if we use the term mutation to refer to name binding. Name binding is already a subtle issue, so I think the risk is significant. Jeremy From bob at redivi.com Thu Jul 13 18:49:31 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 13 Jul 2006 09:49:31 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> Message-ID: <89F00269-C67E-474A-9AF0-282EE343FC18@redivi.com> On Jul 13, 2006, at 5:02 AM, Jeroen Ruigrok van der Werven wrote: > Hi Bob, > > On 7/13/06, Bob Ippolito wrote: >> Adding open classes would make it easier to develop DSLs, but you'd >> only be able to reasonably do one per interpreter (unless you mangled >> the class in a "with" block or something). > > The person whose 'complaints' I was stating says that DSLs (Domain > Specific Languages for those who, like me, were confused about the > acronym) are a big part of what he is after and one per interpreter is > fine by him. He also realises that the application(s) he needs them > for might be unusual. He doesn't specifically need the builtin types > to be extendable. It's just nice to be able to define a single class > in multiple modules. Even C++ allows this to some extent (but not as > much as he'd like). > > He understands the implications of allowing open classes (import vs. > no import changes semantics, etc.). Personally, he doesn't care *too* > much about newbie safety since he's not a newbie. To quote verbatim: > "give me the big guns :-)" > > And while we're at it, he also stated: "[...] add multiple dispatch to > your list of improvements for Python". > > I hope this clarifies it a bit for other people. Well, if this person really weren't a newbie then of course they'd know how to define a metaclass that can be used to extend a (non- built-in) class from another module. They'd probably also know of two or three different implementations of multiple dispatch (or equivalent, such as generic functions) available, and could probably write their own if they had to ;) The only valid complaint, really, is that built-in classes are read- only. I doubt anyone wants to change that. If they want to write things in the style of Ruby, why not just use it? -bob From theller at python.net Thu Jul 13 18:52:33 2006 From: theller at python.net (Thomas Heller) Date: Thu, 13 Jul 2006 18:52:33 +0200 Subject: [Python-Dev] The buffer() function Message-ID: I just answered a question on comp.lang.python for someone who was asking about how to convert the internal buffer of a ctypes instance into base64 coding, without too much copying: "The conversion calls in the base64 module expect strings as input, so right now I'm converting the binary blocks to strings first, then converting the resulting string to base64. This seems highly inefficient and I'd like to just go straight from binary to a base64 string." Naturally I tried to call base64.encodestring(buffer(ctypes_instance)) and it worked, so that was my answer. As it turns out the guy (unusal!) tried to look up information about the buffer function in the documentation. What he found is this: "Python programmers, trainers, students and bookwriters should feel free to bypass these functions without concerns about missing something important". Not an encouragement to use this function. So, what is the real state of the buffer() function, and the buffer object? IIUC, the buffer object was broken some time ago, but I think it has been fixed. Can the 'status' of the buffer function be changed? To quote the next question from the OP: "Is buffer safe to use? Is there an alternative?" My thinking is that it *is* safe to use, and that there is no alternative (but imo also no alternative is needed). Thanks, Thomas From brett at python.org Thu Jul 13 18:54:22 2006 From: brett at python.org (Brett Cannon) Date: Thu, 13 Jul 2006 09:54:22 -0700 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: <44B60B5A.2080309@canterbury.ac.nz> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> Message-ID: On 7/13/06, Greg Ewing wrote: > > Ka-Ping Yee wrote: > > > Having to 'import sys' to get at the command-line arguments always > > seemed awkward to me. 'import sys' feels like it should be a > > privileged operation (access to interpreter internals), and getting > > the command-line args isn't privileged. > > Would it help if sys were pre-imported into the builtins? > Or do you think that args shouldn't live in sys at all? > > Recently I've come to appreciate the ability to get at > the args from anywhere, instead of having to catch them > from a call to main() and pass them around. So I'd > like to still be able to import them from somewhere > if I want (doesn't have to be sys, though). > > And while we're on the subject, anyone think it would > be a good idea to drop the silly feature of having > the program name as args[0]? You almost *never* want > to treat it the same way as the rest of the args, > so the first thing you always do is args[1:]. > > It's not so bad in C, where it's just as easy to > start indexing argv from 1 instead of 0. But it > makes no sense in Python, IMO. It would be much > more sensible to move it into a separate attribute > of whatever module we decide to put args in. Makes sense to me. Ruby does this and it makes working with arguments a little bit nicer. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060713/41f25f26/attachment.html From fredrik at pythonware.com Thu Jul 13 18:57:46 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 13 Jul 2006 18:57:46 +0200 Subject: [Python-Dev] The buffer() function In-Reply-To: References: Message-ID: Thomas Heller wrote: > Naturally I tried to call base64.encodestring(buffer(ctypes_instance)) > and it worked, so that was my answer. does ctypes_instance implement the buffer API ? if it does, is the buffer() call even necessary ? From theller at python.net Thu Jul 13 19:03:22 2006 From: theller at python.net (Thomas Heller) Date: Thu, 13 Jul 2006 19:03:22 +0200 Subject: [Python-Dev] The buffer() function In-Reply-To: References: Message-ID: Fredrik Lundh schrieb: > Thomas Heller wrote: > >> Naturally I tried to call base64.encodestring(buffer(ctypes_instance)) >> and it worked, so that was my answer. > > does ctypes_instance implement the buffer API ? if it does, is the > buffer() call even necessary ? Yes, in both cases. Thomas From fredrik at pythonware.com Thu Jul 13 19:20:53 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 13 Jul 2006 19:20:53 +0200 Subject: [Python-Dev] The buffer() function In-Reply-To: References: Message-ID: Thomas Heller wrote: >>> Naturally I tried to call base64.encodestring(buffer(ctypes_instance)) >>> and it worked, so that was my answer. >> >> does ctypes_instance implement the buffer API ? if it does, is the >> buffer() call even necessary ? > > Yes, in both cases. are you sure? does it implement the bf_getreadbuffer slot? afaik, afaicr, and from what I can tell by looking briefly at the sources, the binascii.b2a_base64 function uses the the s# marker, and that argument type accepts strings *and* read only buffers, right away. no extra crud should be necessary. From theller at python.net Thu Jul 13 19:38:28 2006 From: theller at python.net (Thomas Heller) Date: Thu, 13 Jul 2006 19:38:28 +0200 Subject: [Python-Dev] The buffer() function In-Reply-To: References: Message-ID: Fredrik Lundh schrieb: > Thomas Heller wrote: > >>>> Naturally I tried to call base64.encodestring(buffer(ctypes_instance)) >>>> and it worked, so that was my answer. > >> >>> does ctypes_instance implement the buffer API ? if it does, is the >>> buffer() call even necessary ? >> >> Yes, in both cases. > > are you sure? does it implement the bf_getreadbuffer slot? afaik, > afaicr, and from what I can tell by looking briefly at the sources, the > binascii.b2a_base64 function uses the the s# marker, and that argument > type accepts strings *and* read only buffers, right away. no extra crud > should be necessary. Yes, for binascii.b2a_base64 you are right: bf_getreadbuffer is sufficient. For binascii.b2a_hex (for example), it is not sufficient. But that was not the question. What about the status of the buffer function? All the above functions accept a buffer object. Thomas From glyph at divmod.com Thu Jul 13 20:03:22 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 13 Jul 2006 14:03:22 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: 0 Message-ID: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> On Wed, 12 Jul 2006 10:30:17 +1000, Michael Ellerman wrote: >Well here's one I stumbled across the other day. I don't know if it's >legit, but it's still bad PR: > >http://www.gbch.net/gjb/blog/software/discuss/python-sucks.html Having been exhorted (or maybe I mean "excoriated") by your friendly release manager earlier this week to post my comments and criticisms about Python here rather than vent in random IRC chatter, I feel morally compelled to comment. I see some responses to that post which indicate that the specific bug will be fixed, and that's good, but there is definitely a pattern he's talking about here, not just one issue. I think there is a general pattern of small, difficult to detect breakages in Python. Twisted (and the various other Divmod projects that I maintain) are thoroughly unit-tested, but there are still significant issues in each Python release that require changes. Unlike the jerk who posted that "python sucks" rant, I'm not leaving Python because one function changed in a major release; I do expect to have to maintain my projects myself, and major releases are "major" for a reason. I only wish that upgrading all my other dependencies were as easy as upgrading Python! I do see that the developers are working with some caution to avoid breaking code, and attempting to consider the cost of each change. However, although I've seen lots of discussions of what "average" code might break when exposed to new versions of Python, these discussions tend to be entirely hypothetical. Do the core Python developers typically run the test suites of major Python applications when considering major changes, such as Zope, Plone, TurboGears, and of course Twisted? Not that breakages would be considered unacceptable -- some gain is surely worth the cost -- but to establish some empirical level of burden on the community? I would like to propose, although I certainly don't have time to implement, a program by which Python-using projects could contribute buildslaves which would run their projects' tests with the latest Python trunk. This would provide two useful incentives: Python code would gain a reputation as generally well-tested (since there is a direct incentive to write tests for your project: get notified when core python changes might break it), and the core developers would have instant feedback when a "small" change breaks more code than it was expected to. I can see that certain Python developers expect that some of this work is the responsibility of the user community, and to some extent that's true, but at least half of the work needs to be done _before_ the changes are made. If some Python change breaks half of Twisted, I would like to know about it in time to complain about the implementation, rather than flailing around once the Python feature-freeze has set in and hoping that it's nothing too serious. For example, did anyone here know that the new-style exceptions stuff in 2.5 caused hundreds of unit-test failures in Twisted? I am glad the change was made, and one of our users did catch it, so the process isn't fatally broken, but it is still worrying. Another problem is simply that the Python developers don't have the same very public voice that other language developers do. It doesn't necessarily have to be a blog, but python-dev is fast-paced and intimidating, and a vehicle for discussion among those in the know, rather than dissimenation to the community at large. It's a constant source of anxiety to me that I might miss some key feature addition to Python which breaks or distorts some key bit of Twisted functionality (as the new-style exceptions, or recent ImportWarning almost did) because I don't have enough time to follow all the threads here. I really appreciate the work that Steve Bethard et. al. are doing on the python-dev summaries, but they're still pretty dry and low level. While the new python.org is very nice, I do note that there's no "blogs" entry on the front page, something which has become a fixture on almost every other website I visit regularly. The news page is not very personal, mainly a listing of releases and events. There's no forum for getting the community _excited_ about new features (especially new features which are explicitly enabled by potential breakages), and selling them on the cool uses. Who knows, maybe I'll even start using decorators syntax all over the place if I see them presented somewhere by someone who is very excited about the feature and thinks it's worthwhile, rather than as a defense on a mailing list against a criticism, or a simple announcement of the feature's existence. I've seen the other side of this problem as well, so I can appreciate that it is quite difficult to get this kind of thing right: lots of applications using Twisted break when we change broken or deprecated APIs. Twisted is lucky though; it has numerous subprojects, and I maintain a half-dozen unrelated projects in a different repository. Twisted itself is a fairly representative "application using Twisted", so when something breaks "average" Twisted code, I know about it right away. Python is less an application using Python: the standard library is rather passive (there is no "end-to-end" application path to test, only individual functions and classes), and the test coverage of included Python modules is rather spotty. I hope someone finds some of these suggestions helpful. I only wish I had the time and energy to act on some of them. From mwh at python.net Thu Jul 13 20:19:08 2006 From: mwh at python.net (Michael Hudson) Date: Thu, 13 Jul 2006 19:19:08 +0100 Subject: [Python-Dev] Community buildbots In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> (glyph@divmod.com's message of "Thu, 13 Jul 2006 14:03:22 -0400") References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> Message-ID: <2mfyh52wrn.fsf@starship.python.net> No real time to respond in detail here, but one small comment. glyph at divmod.com writes: > I see some responses to that post which indicate that the specific bug will be > fixed, and that's good, but there is definitely a pattern he's talking about > here, not just one issue. I think there is a general pattern of small, > difficult to detect breakages in Python. Remember these words... > For example, did anyone here know that the new-style exceptions stuff in 2.5 > caused hundreds of unit-test failures in Twisted? I am glad the change was > made, and one of our users did catch it, so the process isn't fatally broken, > but it is still worrying. When implementing this stuff, I could have merely made it possible for exceptions to be new-style, and left the builtin exceptions as classic classes. This didn't seem to be an especially good idea, as it would leave code working _most_ of the time, only to break horribly when confronted with a rare new-style exception. So I made the decision (and I don't think I ever got around to explicitly discussing this with anyone, nor if the people who actually updated my patch and checked it in thought about it at all) to make the built in exceptions new-style, precisely to make it a screamingly obvious change. I didn't _know_ when I was doing it that I'd break Twisted unit tests, but I was hardly a surprise. I think the idea of a buildbot running various projects tests suites with svn HEAD is a great idea -- back when I was doing 2.2.1 I did run a few third party test suites (zodb comes to mind) but as always with these things, automation is a good idea. Cheers, mwh -- > Why are we talking about bricks and concrete in a lisp newsgroup? After long experiment it was found preferable to talking about why Lisp is slower than C++... -- Duane Rettig & Tim Bradshaw, comp.lang.lisp From jcarlson at uci.edu Thu Jul 13 20:30:20 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 13 Jul 2006 11:30:20 -0700 Subject: [Python-Dev] The buffer() function In-Reply-To: References: Message-ID: <20060713112730.0181.JCARLSON@uci.edu> Thomas Heller wrote: > But that was not the question. What about the status of the buffer function? From what I understand, it is relatively safe as long as you don't mutate an object while there is a buffer attached to it. That is: import array a = array.array(...) b = buffer(a) for i in xrange(...): a.extend(a[:]) print str(b) ... may cause you some problems (the a[:] bit was to pointer movement movement on realloc). Those problems will depend on your platform. - Josiah From aahz at pythoncraft.com Thu Jul 13 20:29:16 2006 From: aahz at pythoncraft.com (Aahz) Date: Thu, 13 Jul 2006 11:29:16 -0700 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> Message-ID: <20060713182916.GA12832@panix.com> On Thu, Jul 13, 2006, glyph at divmod.com wrote: > > I would like to propose, although I certainly don't have time to > implement, a program by which Python-using projects could contribute > buildslaves which would run their projects' tests with the latest > Python trunk. This would provide two useful incentives: Python code > would gain a reputation as generally well-tested (since there is a > direct incentive to write tests for your project: get notified when > core python changes might break it), and the core developers would > have instant feedback when a "small" change breaks more code than it > was expected to. There's been some recent discussion in the PSF wondering where it would make sense to throw some money to remove grit in the wheels; do you think this is a case where that would help? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "I saw `cout' being shifted "Hello world" times to the left and stopped right there." --Steve Gonedes From amk at amk.ca Thu Jul 13 20:42:55 2006 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 13 Jul 2006 14:42:55 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> Message-ID: <20060713184255.GA3929@rogue.amk.ca> On Thu, Jul 13, 2006 at 02:03:22PM -0400, glyph at divmod.com wrote: > I would like to propose, although I certainly don't have time to implement, > a program by which Python-using projects could contribute buildslaves which > would run their projects' tests with the latest Python trunk. An excellent idea! > It's a constant source of anxiety to me > that I might miss some key feature addition to Python which breaks > or distorts some key bit of Twisted functionality (as the new-style > exceptions, or recent ImportWarning almost did) because I don't have > enough time to follow all the threads here. So don't read threads -- just try the alphas and betas! The "What's New" documents have a 'porting' section that flags changes that may require application changes, but items are added only if I think of them or if someone suggests them (e.g. generator.gi_frame can be None in Python 2.5 -- I would never have thought people would have code that broke because of this). > While the new python.org is very nice, I do note that there's no "blogs" > entry > on the front page, something which has become a fixture on almost every > other website I visit regularly. A 'Blogs' link could be trivially added by linking to planet.python.org, though the blogs collected there are not in any way 'the Python developers', but a jumble of developers and users. I don't think enough core developers have weblogs (or write about Python) to make a 'python-dev only' planet very useful. > mainly a listing of releases and events. There's no forum for getting the > community _excited_ about new features (especially new features which are > explicitly enabled by potential breakages), and selling them on the cool > uses. I think the venue for this would be weblog entries or news sites such as oreillynet.com. I do worry that we don't have enough people writing articles and things, but have no idea how to encourage people to write more. --amk From foom at fuhm.net Thu Jul 13 20:36:59 2006 From: foom at fuhm.net (James Y Knight) Date: Thu, 13 Jul 2006 14:36:59 -0400 Subject: [Python-Dev] The buffer() function In-Reply-To: References: Message-ID: <1AE30CC8-3CD9-421D-98AD-C8DF4145A6F0@fuhm.net> On Jul 13, 2006, at 12:52 PM, Thomas Heller wrote: > IIUC, the buffer object was broken some time ago, but I think it has > been fixed. Can the 'status' of the buffer function be changed? > To quote the next question from the OP: > > "Is buffer safe to use? Is there an alternative?" > > My thinking is that it *is* safe to use, and that there is > no alternative (but imo also no alternative is needed). I believe it's safe, except when used on an array.array object. However, that's not buffer's fault, but rather a bug in the array class. The buffer interface requires that, as long as a reference to a python object is alive, pointers into its buffer will not become invalidated. Array breaks that guarantee. To fix this, array ought to make a sub-object that this guarantee _does_ hold for. And when it needs more storage, simply make a new sub-object with more storage. Then, the buffer's reference would be to the refcounted sub-object, and thus the associated memory wouldn't go away until the buffer was done with it. James From python-dev at zesty.ca Thu Jul 13 20:41:10 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Thu, 13 Jul 2006 13:41:10 -0500 (CDT) Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: <44B60B5A.2080309@canterbury.ac.nz> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> Message-ID: On Thu, 13 Jul 2006, Greg Ewing wrote: > Would it help if sys were pre-imported into the builtins? > Or do you think that args shouldn't live in sys at all? I feel like the command-line arguments don't really belong in sys, and i'd rather not have 'sys' pre-imported into the builtins. I think of 'sys' as the place for sensitive interpreter internals and the builtins as the place for harmless functions. They're at opposite ends of the scale -- 'sys' is the most privileged; builtins are the least privileged. From a security point of view, 'import' is the main way to get dangerous new functionality. I realize 'open' breaks this concept, but it's nice to be able to say, to a rough approximation, that preventing 'import' gives you an interpreter that can do computations but isn't a security risk. These are just my feelings -- i'm not sure how consistent this is with the original intent of 'sys'. I think it would be a quite useful distinction to have, though. > And while we're on the subject, anyone think it would > be a good idea to drop the silly feature of having > the program name as args[0]? You almost *never* want > to treat it the same way as the rest of the args, > so the first thing you always do is args[1:]. Starting the args array with the first argument (and putting the program name somewhere else) seems Pythonic to me. -- ?!ng From theller at python.net Thu Jul 13 20:51:42 2006 From: theller at python.net (Thomas Heller) Date: Thu, 13 Jul 2006 20:51:42 +0200 Subject: [Python-Dev] The buffer() function In-Reply-To: <20060713112730.0181.JCARLSON@uci.edu> References: <20060713112730.0181.JCARLSON@uci.edu> Message-ID: Josiah Carlson schrieb: > Thomas Heller wrote: >> But that was not the question. What about the status of the buffer function? > >>From what I understand, it is relatively safe as long as you don't > mutate an object while there is a buffer attached to it. > > That is: > > import array > a = array.array(...) > b = buffer(a) > for i in xrange(...): > a.extend(a[:]) > print str(b) > > ... may cause you some problems (the a[:] bit was to pointer movement > movement on realloc). Those problems will depend on your platform. AFAIK, the buffer object now does not hold a pointer into the object it has been constructed from, it only gets it when its needed. IMO Objects/bufferobject.c, revision 35400 is considered safe. The checkin comment (by nascheme) was, more than 2 years ago: "Make buffer objects based on mutable objects (like array) safe." Thomas From guido at python.org Thu Jul 13 21:25:50 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 13 Jul 2006 12:25:50 -0700 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> Message-ID: On 7/13/06, Ka-Ping Yee wrote: > On Thu, 13 Jul 2006, Greg Ewing wrote: > > Would it help if sys were pre-imported into the builtins? > > Or do you think that args shouldn't live in sys at all? > > I feel like the command-line arguments don't really belong in sys, > and i'd rather not have 'sys' pre-imported into the builtins. > > I think of 'sys' as the place for sensitive interpreter internals > and the builtins as the place for harmless functions. They're at > opposite ends of the scale -- 'sys' is the most privileged; > builtins are the least privileged. From a security point of view, > 'import' is the main way to get dangerous new functionality. > I realize 'open' breaks this concept, but it's nice to be able to > say, to a rough approximation, that preventing 'import' gives you > an interpreter that can do computations but isn't a security risk. > > These are just my feelings -- i'm not sure how consistent this is > with the original intent of 'sys'. I think it would be a quite > useful distinction to have, though. sys is a grab-bag. Note that it also has a good deal of harmless read-only data: maxint, version, platform and many more. But it seems to specialize in things that you may modify to change global behavior: path, modules, stdout, ps1, displayhook and others. I'nm afraid if we were to split it by functionality we'd have to split it 5-way or so... > > And while we're on the subject, anyone think it would > > be a good idea to drop the silly feature of having > > the program name as args[0]? You almost *never* want > > to treat it the same way as the rest of the args, > > so the first thing you always do is args[1:]. > > Starting the args array with the first argument (and putting > the program name somewhere else) seems Pythonic to me. As long as it's called args, not argv, I agree. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Thu Jul 13 21:33:23 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 13 Jul 2006 12:33:23 -0700 Subject: [Python-Dev] The buffer() function In-Reply-To: References: <20060713112730.0181.JCARLSON@uci.edu> Message-ID: <20060713123144.0184.JCARLSON@uci.edu> Thomas Heller wrote: > AFAIK, the buffer object now does not hold a pointer into the object > it has been constructed from, it only gets it when its needed. > > IMO Objects/bufferobject.c, revision 35400 is considered safe. > > The checkin comment (by nascheme) was, more than 2 years ago: > "Make buffer objects based on mutable objects (like array) safe." It's good to hear that bug has been squished. I believe buffer() is generally safe then, according to my somewhat limited experience in its use and knowledge about its implementation. - Josiah From nmm1 at cus.cam.ac.uk Thu Jul 13 21:55:37 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Thu, 13 Jul 2006 20:55:37 +0100 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) Message-ID: On systems that are not Unix-derived (which, nowadays, are rare), there is commonly no such thing as a program name in the first place. It is possible to get into that state on some Unices - i.e. ones which have a form of exec that takes a file descriptor, inode number or whatever. This is another argument for separating off argv[0] and allowing the program name to be None if there IS no program name. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From barry at python.org Thu Jul 13 22:05:47 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 13 Jul 2006 16:05:47 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> Message-ID: <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 13, 2006, at 2:03 PM, glyph at divmod.com wrote: > Having been exhorted (or maybe I mean "excoriated") by your > friendly release > manager earlier this week to post my comments and criticisms about > Python here > rather than vent in random IRC chatter, I feel morally compelled to > comment. And I'm glad you did because you made some excellent comments. > However, although I've seen lots of discussions of what "average" > code might > break when exposed to new versions of Python, these discussions > tend to be > entirely hypothetical. Do the core Python developers typically run > the test > suites of major Python applications when considering major changes, > such as > Zope, Plone, TurboGears, and of course Twisted? Not that breakages > would be > considered unacceptable -- some gain is surely worth the cost -- > but to > establish some empirical level of burden on the community? I do plan on very soon running Mailman's meager test suite and running more manual functional tests on Python 2.5b2. Compared to Twisted, Mailman is pretty simple and doesn't do that much fancy stuff so I suspect that it'll mostly work, but you never know until you try it. I'd support adding Mailman to a community buildbot army, although I don't personally have the cycles to build out such a beast. I also plan on testing my Real Job's embedded application against Python 2.5b2 sometime this week or next. We're on Python 2.4.2 atm and that is much more complicated because of all the C API we use. I'm less sanguine about breakage there, but I'll post here if I find anything egregious (fortunately, we have an extensive test suite to rely upon). > Twisted itself is a fairly > representative "application using Twisted", so when something > breaks "average" > Twisted code, I know about it right away. Python is less an > application using > Python: the standard library is rather passive (there is no "end-to- > end" > application path to test, only individual functions and classes), > and the test > coverage of included Python modules is rather spotty. This really is an excellent point and makes me think that we may want to consider elaborating on the Python release cycle to include a gamma phase or a longer release candidate cycle. OT1H I think there will always be people or projects that won't try anything until the gold release, and that if we've broken anything in their code we simply won't know it until after that, no matter how diligent we are. OTOH, a more formal gamma phase would allow us to say "absolutely no changes are allowed now unless it's to fix backward compatibility". No more sneaking in new sys functions or types module constants during the gamma phase. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRLanoHEjvBPtnXfVAQJB6gP/SwVtejenN0/7tszePbJ4O20l98k2Z/7N rs9dfF2+Jcy/OLzSCcW/OW4iVf+iPJMWUNqHEPFDQO/+nVifh8pjjWGTQJMc8ynm 7HNCk/ZciZyNqeGbmGvzHoywmrldbDPkx5Y6Fo13yel0slw2Qo6gC6W8aygi7giP boJiovnaKH0= =wRxy -----END PGP SIGNATURE----- From pierrebai at hotmail.com Thu Jul 13 22:10:11 2006 From: pierrebai at hotmail.com (Pierre Baillargeon) Date: Thu, 13 Jul 2006 20:10:11 +0000 (UTC) Subject: [Python-Dev] Partial support for dlmodule.c in 64-bits OSes Message-ID: Currently, many 64-bits Oses cannot uses the dlmodule due to the conflicts between the sizes of int, long and char *. That is well. The check is made as run-time, which is also very well. The problem is that the Python configuration script (setup.py) also makes the check and plainly excludes dlmodule.c from being built and deployed. That is not so well. The reason is that we use the dlmodule solely to get access to the various flags (RTLD_NOW, RTLD_GLOBAL, etc), so that we can do some magic with loaded shared libraries, such as over-ridding the import mechanism so that the default load flags get changed (via sys.setdlopenflags()) to force some semantics. Currently this doesn't work on most 64-bits OSes because the dl module doesn't exists, so it cannot be imported and its RTLD_* symbols are not accessible. So I ask if it would be possible that the test for sys.maxint == 0x7fffffff in setup.py be dropped in future releases. Here's a relevant diff from the current setup.py in SVN: 1025,1030c1025,1029 + # This requires sizeof(int) == sizeof(long) == sizeof(char*) + # but it will be tested at run-time: give access to the dl module so + # that RTDL_* symbols can be accessed. + dl_inc = find_file('dlfcn.h', [], inc_dirs) + if (dl_inc is not None) and (platform not in ['atheos']): + exts.append( Extension('dl', ['dlmodule.c']) ) --- - if sys.maxint == 0x7fffffff: - # This requires sizeof(int) == sizeof(long) == sizeof(char*) - dl_inc = find_file('dlfcn.h', [], inc_dirs) - if (dl_inc is not None) and (platform not in ['atheos']): - exts.append( Extension('dl', ['dlmodule.c']) ) From glyph at divmod.com Thu Jul 13 22:41:46 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 13 Jul 2006 16:41:46 -0400 Subject: [Python-Dev] Community buildbots In-Reply-To: <2mfyh52wrn.fsf@starship.python.net> Message-ID: <20060713204146.29014.283506497.divmod.quotient.35352@ohm> On Thu, 13 Jul 2006 19:19:08 +0100, Michael Hudson wrote: >glyph at divmod.com writes: >> For example, did anyone here know that the new-style exceptions stuff in 2.5 >> caused hundreds of unit-test failures in Twisted? I am glad the change was >> made, and one of our users did catch it, so the process isn't fatally broken, >> but it is still worrying. > >When implementing this stuff, I could have (... snip ...) To be clear, I agree with the decision you made in this particular case. I just would have appreciated the opportunity to participate in the discussion before the betas were out and the featureset frozen. (Of course I *can* always do that, and some other Twisted devs watch python-dev a bit more closely than I do, but the point is that the amount of effort required to do this is prohibitive for the average Python hacker, whereas the time to set up an individual buildbot might not be.) (Aside: IMHO, the sooner we can drop old-style classes entirely, the better. That is one bumpy Python upgrade process that I will be _very_ happy to do. There's no way to have documentation that expresses the requirement that an implementation of an interface be new-style or old-style without reference to numerous historical accidents, which are bewildering and upsetting to people reading documentation for the first time.) From rasky at develer.com Thu Jul 13 22:49:57 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 13 Jul 2006 22:49:57 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> Message-ID: <058b01c6a6bd$e6b6b080$d503030a@trilan> Barry Warsaw wrote: > OTOH, a more formal gamma phase would allow us to say > "absolutely no changes are allowed now unless it's to fix backward > compatibility". No more sneaking in new sys functions or types > module constants during the gamma phase. This is pretty common in project management. For instance, GCC has a rather complex 4-stage release process, whose last phase (beginning at the point the release is branched in SVN) is made of commits only for regressions. -- Giovanni Bajo From brett at python.org Thu Jul 13 22:52:50 2006 From: brett at python.org (Brett Cannon) Date: Thu, 13 Jul 2006 13:52:50 -0700 Subject: [Python-Dev] Community buildbots In-Reply-To: <20060713204146.29014.283506497.divmod.quotient.35352@ohm> References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> Message-ID: On 7/13/06, glyph at divmod.com wrote: > > On Thu, 13 Jul 2006 19:19:08 +0100, Michael Hudson wrote: > >glyph at divmod.com writes: > > >> For example, did anyone here know that the new-style exceptions stuff > in 2.5 > >> caused hundreds of unit-test failures in Twisted? I am glad the change > was > >> made, and one of our users did catch it, so the process isn't fatally > broken, > >> but it is still worrying. > > > >When implementing this stuff, I could have (... snip ...) > > To be clear, I agree with the decision you made in this particular > case. I > just would have appreciated the opportunity to participate in the > discussion before the betas were out and the featureset frozen. (Of > course I > *can* always do that, and some other Twisted devs watch python-dev a bit > more > closely than I do, but the point is that the amount of effort required to > do > this is prohibitive for the average Python hacker, whereas the time to set > up an > individual buildbot might not be.) One way to try to track python-dev is the python-dev Summaries. While Steven is only human and thus cannot always get them out immediately, they do happen frequently enough that major decisions will be covered before betas are hit. -Brett (Aside: IMHO, the sooner we can drop old-style classes entirely, the better. > That is one bumpy Python upgrade process that I will be _very_ happy to > do. > There's no way to have documentation that expresses the requirement that > an > implementation of an interface be new-style or old-style without reference > to > numerous historical accidents, which are bewildering and upsetting to > people > reading documentation for the first time.) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060713/9de6c971/attachment.html From rasky at develer.com Thu Jul 13 22:53:02 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 13 Jul 2006 22:53:02 +0200 Subject: [Python-Dev] Community buildbots References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> Message-ID: <05af01c6a6be$54b653b0$d503030a@trilan> glyph at divmod.com wrote: > (Aside: IMHO, the sooner we can drop old-style classes entirely, the > better. > That is one bumpy Python upgrade process that I will be _very_ happy > to do. I think python should have a couple more of future imports. "from __future__ import new_classes" and "from __future__ import unicode_literals" would be really welcome, and would smooth the Py3k migration process -- Giovanni Bajo From radix at twistedmatrix.com Thu Jul 13 23:06:04 2006 From: radix at twistedmatrix.com (Christopher Armstrong) Date: Thu, 13 Jul 2006 17:06:04 -0400 Subject: [Python-Dev] Community buildbots In-Reply-To: <05af01c6a6be$54b653b0$d503030a@trilan> References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> <05af01c6a6be$54b653b0$d503030a@trilan> Message-ID: <60ed19d40607131406u498b0901oc84296f4e631ae5e@mail.gmail.com> On 7/13/06, Giovanni Bajo wrote: > I think python should have a couple more of future imports. "from __future__ > import new_classes" and "from __future__ import unicode_literals" would be > really welcome, and would smooth the Py3k migration process Along similar lines as glyph, after complaining about this for a long time "offline" with my friends, I decided it's probably a good idea to get involved and vote that the __future__ import for unicode_literals be implemented. python -U is a failure for obvious reasons and a __future__ import is clearly better. Does anyone want to pair on this? -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/ From bob at redivi.com Thu Jul 13 23:07:07 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 13 Jul 2006 14:07:07 -0700 Subject: [Python-Dev] Community buildbots In-Reply-To: <05af01c6a6be$54b653b0$d503030a@trilan> References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> <05af01c6a6be$54b653b0$d503030a@trilan> Message-ID: On Jul 13, 2006, at 1:53 PM, Giovanni Bajo wrote: > glyph at divmod.com wrote: > >> (Aside: IMHO, the sooner we can drop old-style classes entirely, the >> better. >> That is one bumpy Python upgrade process that I will be _very_ happy >> to do. > > I think python should have a couple more of future imports. "from > __future__ > import new_classes" and "from __future__ import unicode_literals" > would be > really welcome, and would smooth the Py3k migration process "from __future__ import new_classes" exists, but the syntax is different: __metaclass__ = type -bob From glyph at divmod.com Fri Jul 14 00:30:24 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 13 Jul 2006 18:30:24 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <20060713182916.GA12832@panix.com> Message-ID: <20060713223024.29014.1648933869.divmod.quotient.35468@ohm> On Thu, 13 Jul 2006 11:29:16 -0700, Aahz wrote: >There's been some recent discussion in the PSF wondering where it would >make sense to throw some money to remove grit in the wheels; do you think >this is a case where that would help? Most likely yes. It's not a huge undertaking, and there are a lot of people out there in the community with the knowledge of Buildbot to make this happen. From radix at twistedmatrix.com Fri Jul 14 00:32:45 2006 From: radix at twistedmatrix.com (Christopher Armstrong) Date: Thu, 13 Jul 2006 18:32:45 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <20060713223024.29014.1648933869.divmod.quotient.35468@ohm> References: <20060713182916.GA12832@panix.com> <20060713223024.29014.1648933869.divmod.quotient.35468@ohm> Message-ID: <60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com> On 7/13/06, glyph at divmod.com wrote: > On Thu, 13 Jul 2006 11:29:16 -0700, Aahz wrote: > > >There's been some recent discussion in the PSF wondering where it would > >make sense to throw some money to remove grit in the wheels; do you think > >this is a case where that would help? > > Most likely yes. It's not a huge undertaking, and there are a lot of people out > there in the community with the knowledge of Buildbot to make this happen. I'm at least willing to set up the buildbots for projects I care about (Twisted, pydoctor, whatever), and perhaps help out with the setting up the buildmaster. -- Christopher Armstrong International Man of Twistery http://radix.twistedmatrix.com/ http://twistedmatrix.com/ http://canonical.com/ From greg.ewing at canterbury.ac.nz Fri Jul 14 02:36:29 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 14 Jul 2006 12:36:29 +1200 Subject: [Python-Dev] User's complaints In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> Message-ID: <44B6E70D.9010508@canterbury.ac.nz> Fredrik Lundh wrote: > (and while we're at it, wouldn't a standard multiargument dispatch be > nice replacement for the instance-oriented lookup we're using today? > dispatching on a single value is so last century ;-) That's highly debatable, and as I'm sure you remember, has been highly debated. :-) -- Greg From greg.ewing at canterbury.ac.nz Fri Jul 14 02:41:25 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 14 Jul 2006 12:41:25 +1200 Subject: [Python-Dev] User's complaints In-Reply-To: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> Message-ID: <44B6E835.30208@canterbury.ac.nz> Jeroen Ruigrok van der Werven wrote: > It's just nice to be able to define a single class > in multiple modules. It *seems* nice until you want to track down which source file the definition of some method comes from. Those used to the "one huge global namespace" of C and C++ likely don't see this as a problem. But since I've come to appreciate the benefits of Python's module system, I don't want to go back to that nightmare. -- Greg From terry at jon.es Fri Jul 14 03:15:14 2006 From: terry at jon.es (Terry Jones) Date: Fri, 14 Jul 2006 03:15:14 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: Your message at 12:41:25 on Friday, 14 July 2006 References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> <44B6E835.30208@canterbury.ac.nz> Message-ID: <17590.61474.301502.95204@terry.jones.tc> >>>>> "Greg" == Greg Ewing writes: Greg> Jeroen Ruigrok van der Werven wrote: >> It's just nice to be able to define a single class >> in multiple modules. Greg> It *seems* nice until you want to track down which Greg> source file the definition of some method comes Greg> from. Greg> Those used to the "one huge global namespace" of Greg> C and C++ likely don't see this as a problem. But Greg> since I've come to appreciate the benefits of Greg> Python's module system, I don't want to go back Greg> to that nightmare. While I think there are good arguments both ways, I don't think that finding source definitions of methods or classes counts as one - let alone as a nightmare. Tools like tags (in vi and emacs and lisp environments) work quickly and accurately, are easy to use (one keystroke in vi and M-. or similar in emacs to go to a defn, and then a tags pop to come back), work in an identical way on many source languages, and they have been around for literally decades. That's to say nothing of IDE or CASE tools that support finding definitions, callers, etc. Regards, Terry From fdrake at acm.org Fri Jul 14 03:21:27 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu, 13 Jul 2006 21:21:27 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> Message-ID: <200607132121.27521.fdrake@acm.org> On Thursday 13 July 2006 16:05, Barry Warsaw wrote: > This really is an excellent point and makes me think that we may want > to consider elaborating on the Python release cycle to include a > gamma phase or a longer release candidate cycle. OT1H I think there ... > "absolutely no changes are allowed now unless it's to fix backward > compatibility". No more sneaking in new sys functions or types > module constants during the gamma phase. +42 It feels like the release cycle from alpha1 to final has gotten increasingly rushed. While I'm sure some of that is just me having my attention elsewhere, I suspect a longer tail on the cycle to do gammas (or release candidates, or whatever) would definately encourage more testing with applications and the larger frameworks. No, it won't catch everything, but I think it would help. -Fred -- Fred L. Drake, Jr. From nnorwitz at gmail.com Fri Jul 14 05:02:13 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 13 Jul 2006 20:02:13 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> Message-ID: On 7/13/06, Jeroen Ruigrok van der Werven wrote: > > > If it's pure python, why don't people just copy everything under > > site-packages after installing? They could/should run compileall > > after that to recompile the .pyc files. With 2.5 on 64-bit machines, > > C extension modules *must* be recompiled due to lots of internal > > changes. > > I wasn't even aware of the compileall step, can you elaborate since > this is the first time I see it being mentioned. python -mcompileall -h It's in the stdlib. It is used by the build process to recursively traverse directories and compile all the .py files into .pyc (or .pyo) files. So if you wanted to copy everything from site-package in 2.4 to 2.5, you could do something like: cp -r /usr/lib/python2.{4,5}/site-packages /usr/bin/python2.5 -mcompileall -d /usr/lib/python2.5/site-packages That should really be all that's required. It of course doesn't verify the packages are correct or that you still want to keep all the files. This generally works for .so's too. However a warning will be generated each time you import the .so since it was built on an old version of Python.. On 2.5 in 64-bit systems, this will definitely not work. > > One thing you didn't mention that I've heard from time to time is the > > stdlib should be improved. For example, cleaning up old modules. > > Though I'm not really sure everyone has the same thing in mind when it > > comes to improving the stdlib. > > How do you envision cleaning up old modules? For me, manually. Do you still need moduleX? There's no way for a computer to tell you the answer, it depends on what you will use with the new version of Python. n From anthony at interlink.com.au Fri Jul 14 05:43:36 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri, 14 Jul 2006 13:43:36 +1000 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> Message-ID: <200607141343.38424.anthony@interlink.com.au> FWIW, I tend to run a few project(*) test suites when doing minor releases (2.x.y), just to make sure I don't break things. For major releases, it's a fair bit more work - something like Twisted or Zope3 play at such a low level with the Python interfaces that there's nearly always breakages or massive warning dumps (from memory, zope.interface uses 'with' as an argument name, a lot). The "community buildbot" idea is a good one, although it should just be possible to write something for buildbot that checks out and builds the latest Python SVN, then installs it to a temporary location, then adds that location to the front of the path. Then each project could just add a "Python SVN buildbot" to their exists bbot install. Anthony (*) Typically includes Zope3 and Twisted, because I have these just lying around. From skip at pobox.com Fri Jul 14 06:27:56 2006 From: skip at pobox.com (skip at pobox.com) Date: Thu, 13 Jul 2006 23:27:56 -0500 Subject: [Python-Dev] Community buildbots In-Reply-To: <20060713204146.29014.283506497.divmod.quotient.35352@ohm> References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> Message-ID: <17591.7500.168119.701223@montanaro.dyndns.org> glyph> *can* always do that, and some other Twisted devs watch glyph> python-dev a bit more closely than I do, but the point is that glyph> the amount of effort required to do this is prohibitive for the glyph> average Python hacker, whereas the time to set up an individual glyph> buildbot might not be.) If you're concerned about noticing when a new release train is pulling out of the station I think it would be sufficient to subscribe to the low-volume python-announce mailing list. You will see announcements about alphas and betas there. Even lower volume would be a subscription to the RSS feed of Python announcements on the python.org front page (scroll to the bottom). The buildbot idea sounds excellent. Skip From skip at pobox.com Fri Jul 14 06:32:52 2006 From: skip at pobox.com (skip at pobox.com) Date: Thu, 13 Jul 2006 23:32:52 -0500 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <200607132121.27521.fdrake@acm.org> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> <200607132121.27521.fdrake@acm.org> Message-ID: <17591.7796.553081.425160@montanaro.dyndns.org> Fred> It feels like the release cycle from alpha1 to final has gotten Fred> increasingly rushed. Same here. I believe there was some shortening of the 2.5 release cycle two or three months ago. I don't recall why or by how much, but I think the acceleration has resulted in a lot of the "can't we please squeeze this one little change in?" that's been happening. Shortening a micro release a bit seems reasonably easy to accommodate, but since minor releases occur so infrequently, I think it would be better to stretch them out if necessary. Skip From greg.ewing at canterbury.ac.nz Fri Jul 14 06:55:34 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 14 Jul 2006 16:55:34 +1200 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> Message-ID: <44B723C6.9010309@canterbury.ac.nz> Ka-Ping Yee wrote: > I think of 'sys' as the place for sensitive interpreter internals Well, it seems to be rather a mixture at the moment. I suppose you could regard sys.modules as fairly sensitive, since messing with it can have big effects on the behaviour of the whole program, and changing sys.stdout or sys.stderr also has global effects. But it's also how you do things like getting info about the current exception, which doesn't seem particularly sensitive to me. Maybe sys needs to be split into two modules, with the non-sensitive one pre-imported (so that the importless interpreter you suggest wouldn't be unnecessarily crippled). -- Greg From greg.ewing at canterbury.ac.nz Fri Jul 14 07:04:00 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 14 Jul 2006 17:04:00 +1200 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> Message-ID: <44B725C0.5000202@canterbury.ac.nz> Guido van Rossum wrote: > I'nm afraid if we > were to split it by functionality we'd have to split it 5-way or so... What about just splitting it into "mutable" and "immutable" parts? That would be a fairly clear division, I think. -- Greg From greg.ewing at canterbury.ac.nz Fri Jul 14 07:09:40 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 14 Jul 2006 17:09:40 +1200 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: Message-ID: <44B72714.6030502@canterbury.ac.nz> Nick Maclaren wrote: > On systems that are not Unix-derived (which, nowadays, are rare), > there is commonly no such thing as a program name in the first place. > It is possible to get into that state on some Unices - i.e. ones which > have a form of exec that takes a file descriptor, inode number or > whatever. I don't think that applies to the Python args[] though, since its args[0] isn't the path of the OS-level executable, it's the path of the main Python script. But you could still end up without one, if the main script comes from somewhere other than a file. -- Greg From glyph at divmod.com Fri Jul 14 07:11:57 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 14 Jul 2006 01:11:57 -0400 Subject: [Python-Dev] Community buildbots In-Reply-To: <17591.7500.168119.701223@montanaro.dyndns.org> Message-ID: <20060714051157.29014.803759518.divmod.quotient.35955@ohm> On Thu, 13 Jul 2006 23:27:56 -0500, skip at pobox.com wrote: >The buildbot idea sounds excellent. Thanks. If someone can set this up, it pretty much addresses my concerns. >If you're concerned about noticing when a new release train is pulling out >of the station I think it would be sufficient to subscribe to the low-volume >python-announce mailing list. You will see announcements about alphas and >betas there. Even lower volume would be a subscription to the RSS feed of >Python announcements on the python.org front page (scroll to the bottom). I am aware of when new releases come out :). What I'm not aware of is what features (may) have broken my code, and why. As long as Python's trunk at HEAD continues to run the test suites cleanly, I am mostly unconcerned. When it breaks, though, I want a chance to look at the cause of the breakage, *before* there is an alpha or beta Python release out and people are starting to write code that depends on its new features. Most importantly, python-dev is extremely context-sensitive. I want to be able to participate in the discussion when my concerns are relevant and still reasonable to act upon. Additionally I would like to know about these changes so that I can modify code to support Python releases and release a compatible version of Twisted _in advance_ of the Python release that's going to break them, so assuming users are keeping relatively current, there won't be a window where their most recent Twisted release will not work with the most recent Python release. From talin at acm.org Fri Jul 14 07:15:27 2006 From: talin at acm.org (Talin) Date: Thu, 13 Jul 2006 22:15:27 -0700 Subject: [Python-Dev] Community buildbots In-Reply-To: <05af01c6a6be$54b653b0$d503030a@trilan> References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> <05af01c6a6be$54b653b0$d503030a@trilan> Message-ID: <44B7286F.7070406@acm.org> Giovanni Bajo wrote: > glyph at divmod.com wrote: > > I think python should have a couple more of future imports. "from __future__ > import new_classes" and "from __future__ import unicode_literals" would be > really welcome, and would smooth the Py3k migration process Actually - can we make new-style classes the default, but allow a way to switch to old-style classes if needed? Perhaps a command-line argument to set the default back to old-style? -- Talin From greg.ewing at canterbury.ac.nz Fri Jul 14 07:20:41 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 14 Jul 2006 17:20:41 +1200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> Message-ID: <44B729A9.3060705@canterbury.ac.nz> Barry Warsaw wrote: > we may want > to consider elaborating on the Python release cycle to include a > gamma phase or a longer release candidate cycle. Maybe there could be an "unstable" release phase that lasts for a whole release cycle. So you'd first release version 2.n as "unstable", and keep 2.(n-1) as the current "stable" release. Then when 2.(n+1) is ready, 2.n would become "stable" and 2.(n+1) would become the new "unstable". (Feel free to substitute other terms for "stable" and "unstable" if you don't like them.) That would give people plenty of warning and time to try things out with the new version. This wouldn't actually be much different to what is done now, except for the naming. But by not officially blessing the latest release as current for a while, it might give less of an impression that stuff is being sprung on the community unawares. > OT1H I think there > will always be people or projects that won't try anything until the > gold release, Well, at least they've had a chance to try it. If they don't take that chance, they don't have much ground for complaint. -- Greg From fredrik at pythonware.com Fri Jul 14 07:26:39 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 14 Jul 2006 07:26:39 +0200 Subject: [Python-Dev] Community buildbots In-Reply-To: <44B7286F.7070406@acm.org> References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> <05af01c6a6be$54b653b0$d503030a@trilan> <44B7286F.7070406@acm.org> Message-ID: Talin wrote: >> I think python should have a couple more of future imports. "from __future__ >> import new_classes" and "from __future__ import unicode_literals" would be >> really welcome, and would smooth the Py3k migration process > > Actually - can we make new-style classes the default, but allow a way to > switch to old-style classes if needed? Perhaps a command-line argument > to set the default back to old-style? so people complain about new releases breaking too much code, and your response to that is break even more code ? From greg.ewing at canterbury.ac.nz Fri Jul 14 07:27:57 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 14 Jul 2006 17:27:57 +1200 Subject: [Python-Dev] Community buildbots In-Reply-To: References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> <05af01c6a6be$54b653b0$d503030a@trilan> Message-ID: <44B72B5D.9050905@canterbury.ac.nz> Bob Ippolito wrote: > "from __future__ import new_classes" exists, but the syntax is > different: > > __metaclass__ = type Although it's not a very obvious spelling, particularly to the casual reader who may not be familiar with the intricacies of classes and metaclasses. I don't think it would hurt to have it available as a __future__ import as well. There's also the advantage that all of a module's future assumptions could then be documented uniformly in one place, i.e. in a __future__ import at the top. -- Greg From fdrake at acm.org Fri Jul 14 07:39:38 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 14 Jul 2006 01:39:38 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <17591.7796.553081.425160@montanaro.dyndns.org> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607132121.27521.fdrake@acm.org> <17591.7796.553081.425160@montanaro.dyndns.org> Message-ID: <200607140139.38384.fdrake@acm.org> On Friday 14 July 2006 00:32, skip at pobox.com wrote: > Same here. I believe there was some shortening of the 2.5 release cycle > two or three months ago. I don't recall why or by how much, but I think > the acceleration has resulted in a lot of the "can't we please squeeze > this one little change in?" that's been happening. Shortening a micro > release a bit seems reasonably easy to accommodate, but since minor > releases occur so infrequently, I think it would be better to stretch them > out if necessary. The squeezing of the releases isn't where the problem is, I think. It's that, once squeezed, more releases aren't being added to compensate. We really need to determine what time we need to go from beta1 to (gamma|rc)1, and then from (gamma|rc)1 to final. Plenty of interim releases in the beta phase is good. -Fred -- Fred L. Drake, Jr. From fredrik at pythonware.com Fri Jul 14 07:45:24 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 14 Jul 2006 07:45:24 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <200607132121.27521.fdrake@acm.org> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> <200607132121.27521.fdrake@acm.org> Message-ID: Fred L. Drake, Jr. wrote: > It feels like the release cycle from alpha1 to final has gotten increasingly > rushed. python 2.2: ~5 months python 2.3: ~7 months python 2.4: ~5 months python 2.5: ~4 months I think the biggest problem is the time between "basically stable beta" and "final"; it was ~90 days in 2.3 and is now ~50 days, most of which is vacation time in my part of the world. we could extend the beta period by playing silly games with naming, but as long as the "big things" go in *before* the beta period starts, I think it's easier to simply extend the beta period by adding another month to it. I'd prefer something like 90 days. From fdrake at acm.org Fri Jul 14 07:50:13 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 14 Jul 2006 01:50:13 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607132121.27521.fdrake@acm.org> Message-ID: <200607140150.13563.fdrake@acm.org> On Friday 14 July 2006 01:45, Fredrik Lundh wrote: > I'd prefer something like 90 days. +1 -Fred -- Fred L. Drake, Jr. From anthony at interlink.com.au Fri Jul 14 07:51:00 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri, 14 Jul 2006 15:51:00 +1000 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> Message-ID: <200607141551.06701.anthony@interlink.com.au> On Friday 14 July 2006 06:05, Barry Warsaw wrote: > This really is an excellent point and makes me think that we may > want to consider elaborating on the Python release cycle to include > a gamma phase or a longer release candidate cycle. OT1H I think > there will always be people or projects that won't try anything > until the gold release, and that if we've broken anything in their > code we simply won't know it until after that, no matter how > diligent we are. OTOH, a more formal gamma phase would allow us to > say "absolutely no changes are allowed now unless it's to fix > backward compatibility". No more sneaking in new sys functions or > types module constants during the gamma phase. alpha 1: April 5, 2006 [completed] alpha 2: April 27, 2006 [completed] beta 1: June 20, 2006 [completed] beta 2: July 11, 2006 [completed] rc 1: August 1, 2006 [planned] final: August 8, 2006 [planned] Four months would seem to me to be quite long enough as a release cycle. Extending it means far more work for everyone - either we have special checkin rules for the trunk for a longer period of time (requiring extra monitoring by people like myself and Neal), or we branch earlier, requiring double commits to the trunk and the branch for bugfixes. I also strongly doubt that making a longer "release candidate" cycle would lead to any significant additional testing by end-users. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From greg.ewing at canterbury.ac.nz Fri Jul 14 07:52:05 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 14 Jul 2006 17:52:05 +1200 Subject: [Python-Dev] Community buildbots In-Reply-To: <44B7286F.7070406@acm.org> References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> <05af01c6a6be$54b653b0$d503030a@trilan> <44B7286F.7070406@acm.org> Message-ID: <44B73105.70707@canterbury.ac.nz> Talin wrote: > Actually - can we make new-style classes the default, but allow a way to > switch to old-style classes if needed? That sounds dangerously like a "from __past__" kind of feature, and Guido has said that there will never be a __past__ module. Also, this is probably not something that should be a program-wide setting. -- Greg From nnorwitz at gmail.com Fri Jul 14 07:55:04 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 13 Jul 2006 22:55:04 -0700 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <17591.7796.553081.425160@montanaro.dyndns.org> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> <200607132121.27521.fdrake@acm.org> <17591.7796.553081.425160@montanaro.dyndns.org> Message-ID: On 7/13/06, skip at pobox.com wrote: > > Fred> It feels like the release cycle from alpha1 to final has gotten > Fred> increasingly rushed. I think that's just because you are getting older and time goes by faster the less time you've got left. :-) It seems to be going quite quick, but I'm not so sure it's really different. > Same here. I believe there was some shortening of the 2.5 release cycle two > or three months ago. I don't recall why or by how much, but I think the > acceleration has resulted in a lot of the "can't we please squeeze this one > little change in?" that's been happening. Shortening a micro release a bit > seems reasonably easy to accommodate, but since minor releases occur so > infrequently, I think it would be better to stretch them out if necessary. Not exactly. The PEP was created Feb 7 (r42266) with a very relaxed schedule to allow enough time for ssize_t branch to solidify. Martin was ready sooner, so on Feb 26 (r42577) sped up the schedule to something more reasonable: - alpha 1: April 1, 2006 [planned] - alpha 2: April 29, 2006 [planned] - beta 1: June 24, 2006 [planned] - beta 2: July 15, 2006 [planned] - rc 1: August 5, 2006 [planned] - final: August 19, 2006 [planned] The current schedule is: + alpha 1: April 5, 2006 [completed] + alpha 2: April 27, 2006 [completed] + beta 1: June 20, 2006 [completed] + beta 2: July 11, 2006 [completed] + rc 1: August 1, 2006 [planned] + final: August 8, 2006 [planned] So beta1 was 4 days earlier than the date set at the end of Feb. The first beta was about 19 months after 2.4 was released if I'm counting correctly. In 2.4 beta1 to release was 1.5 months. For 2.5 it is about the same. The total 2.4 schedule (from a1 to release) was 4.5 months which included 3 alphas. For 2.5 we only had 2 alphas and the total schedule is just over 4 months. So 2.5 is just a little faster. But 2.5 has had a ton more testing than any previous released due to the buildbots, not to mention Coverity and now Klocwork. I definitely pushed the schedule, but I don't think it was really all that different from in the past. Given that several people here think we should lengthen the schedule in some way, I suspect we will do something. I'm not really against it, but I don't think it will provide much benefit either. A few more bugs will be fixed since we have more time. What I think might really make a difference if we could leverage the buildbots and copy the builds (especially WIndows) up to python.org and make them available for download. That might make it easier for people to try stuff out. n From nnorwitz at gmail.com Fri Jul 14 07:59:48 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 13 Jul 2006 22:59:48 -0700 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com> References: <20060713182916.GA12832@panix.com> <20060713223024.29014.1648933869.divmod.quotient.35468@ohm> <60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com> Message-ID: On 7/13/06, Christopher Armstrong wrote: > On 7/13/06, glyph at divmod.com wrote: > > On Thu, 13 Jul 2006 11:29:16 -0700, Aahz wrote: > > > > >There's been some recent discussion in the PSF wondering where it would > > >make sense to throw some money to remove grit in the wheels; do you think > > >this is a case where that would help? > > > > Most likely yes. It's not a huge undertaking, and there are a lot of people out > > there in the community with the knowledge of Buildbot to make this happen. > > I'm at least willing to set up the buildbots for projects I care about > (Twisted, pydoctor, whatever), and perhaps help out with the setting > up the buildmaster. It would be really great to have someone setup buildbots for other important/large python projects to test python svn HEAD with. I can't even stay on top of the python buildbots, so I'm not volunteering for more work. I will greatly appreciate and thank whoever takes this on though. :-) The harder problem is keeping it running. Setting it up, in some ways is a lot easier. It's at least well defined. Maintaining it for several months is a different story. n From fredrik at pythonware.com Fri Jul 14 08:15:31 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 14 Jul 2006 08:15:31 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> <200607132121.27521.fdrake@acm.org> <17591.7796.553081.425160@montanaro.dyndns.org> Message-ID: Neal Norwitz wrote: > Given that several people here think we should lengthen the schedule > in some way, I suspect we will do something. I'm not really against > it, but I don't think it will provide much benefit either. A few more > bugs will be fixed since we have more time. you're still missing the point of this thread: it isn't bugs in the core Python distribution that's the problem, it's compatibility with third- party modules and applications. I'm quite sure that we could ship the current beta2 as 2.5 final and have a Python release that, in itself, is more stable than all the ones before it, but since 2.5 introduces lots of new stuff, that stability doesn't necessarily translate to a better experience for people who wants to use their existing applications and libraries. a longer beta period gives *external* developers more time to catch up, and results in less work for the end users. From nnorwitz at gmail.com Fri Jul 14 08:39:06 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 13 Jul 2006 23:39:06 -0700 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> <200607132121.27521.fdrake@acm.org> <17591.7796.553081.425160@montanaro.dyndns.org> Message-ID: On 7/13/06, Fredrik Lundh wrote: > Neal Norwitz wrote: > > > Given that several people here think we should lengthen the schedule > > in some way, I suspect we will do something. I'm not really against > > it, but I don't think it will provide much benefit either. A few more > > bugs will be fixed since we have more time. > > you're still missing the point of this thread: it isn't bugs in the core > Python distribution that's the problem, it's compatibility with third- > party modules and applications. ... > a longer beta period gives *external* developers more time to catch up, > and results in less work for the end users. This is the part I don't get. For the external developers, if they care about compatibility, why aren't they testing periodically, regardless of alpha/beta releases? How often is the python build broken or otherwise unusable? On Unix there's no more or less work to grab a tarball whether it's a nightly or a release. For Windows it's definitely easier to wait for a release. If there was an installable windows version made by the buildbots, would that mean there would be more testing? Is part of your point that these developers only care about something called "release" and they won't start testing before then? If that's the case why don't we start making (semi-)automated alpha releases every month? That will give people lots more time. Somehow I don't think it will make a difference. People mostly work on schedules or should I say avoid schedules. Give them 5 months and they will still wait to the last minute. Remember I also tried to push for more features to go in early? That would have given more time for external testing. Still features are coming in. Python developers weren't happy about having to get things in earlier. I don't see a practical way to implement what you propose (see Anthony's comments). n From fredrik at pythonware.com Fri Jul 14 09:00:20 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 14 Jul 2006 09:00:20 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> <200607132121.27521.fdrake@acm.org> <17591.7796.553081.425160@montanaro.dyndns.org> Message-ID: Neal Norwitz wrote: > This is the part I don't get. For the external developers, if they > care about compatibility, why aren't they testing periodically, > regardless of alpha/beta releases? because they don't want to waste time and effort on testing against releases that are not necessarily a close approximation of the full release? testing takes time, and time can be used for many different things. From anthony at interlink.com.au Fri Jul 14 09:10:56 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri, 14 Jul 2006 17:10:56 +1000 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> Message-ID: <200607141711.03069.anthony@interlink.com.au> On Friday 14 July 2006 17:00, Fredrik Lundh wrote: > because they don't want to waste time and effort on testing against > releases that are not necessarily a close approximation of the full > release? testing takes time, and time can be used for many > different things. Oddly enough, so does release management. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Fri Jul 14 09:14:06 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri, 14 Jul 2006 17:14:06 +1000 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> Message-ID: <200607141714.09304.anthony@interlink.com.au> On Friday 14 July 2006 16:39, Neal Norwitz wrote: > Remember I also tried to push for more features to go in early? > That would have given more time for external testing. Still > features are coming in. Python developers weren't happy about > having to get things in earlier. I don't see a practical way to > implement what you propose (see Anthony's comments). Following up on this point: Given the number of "just-this-one-more-thing-please" we've _already_ had since the b1 feature freeze, do you really except that 90 days of feature freeze is feasible? And if there's not going to be a feature freeze, there's hardly any point to people doing testing until there _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. Oops. 2.5b9 added a new feature that broke my code, but I didn't test with that. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From nmm1 at cus.cam.ac.uk Fri Jul 14 10:42:45 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Fri, 14 Jul 2006 09:42:45 +0100 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: Your message of "Fri, 14 Jul 2006 17:09:40 +1200." <44B72714.6030502@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > > > On systems that are not Unix-derived (which, nowadays, are rare), > > there is commonly no such thing as a program name in the first place. > > It is possible to get into that state on some Unices - i.e. ones which > > have a form of exec that takes a file descriptor, inode number or > > whatever. > > I don't think that applies to the Python args[] though, > since its args[0] isn't the path of the OS-level > executable, it's the path of the main Python script. Oh, yes, it does! The file descriptor or inode number could refer to the script just as well as it could to the interpreter binary. > But you could still end up without one, if the main > script comes from somewhere other than a file. I didn't want to include that, to avoid confusing people who haven't used systems with such features. Several systems have had the ability to exec to a memory segment, for example. But, yes. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From fredrik at pythonware.com Fri Jul 14 10:55:32 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 14 Jul 2006 10:55:32 +0200 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: <44B72714.6030502@canterbury.ac.nz> Message-ID: Nick Maclaren wrote: >> I don't think that applies to the Python args[] though, >> since its args[0] isn't the path of the OS-level >> executable, it's the path of the main Python script. > > Oh, yes, it does! The file descriptor or inode number could refer to > the script just as well as it could to the interpreter binary. (insert sound of head hitting desk here) From rasky at develer.com Fri Jul 14 11:06:17 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 14 Jul 2006 11:06:17 +0200 Subject: [Python-Dev] Community buildbots References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> <05af01c6a6be$54b653b0$d503030a@trilan> <44B72B5D.9050905@canterbury.ac.nz> Message-ID: <091801c6a724$c3ee1860$12472597@bagio> Greg Ewing wrote: >> "from __future__ import new_classes" exists, but the syntax is >> different: >> >> __metaclass__ = type > > Although it's not a very obvious spelling, > particularly to the casual reader who may not be > familiar with the intricacies of classes and > metaclasses. I don't think it would hurt to have > it available as a __future__ import as well. > > There's also the advantage that all of a > module's future assumptions could then be > documented uniformly in one place, i.e. in > a __future__ import at the top. +1. Giovanni Bajo From mwh at python.net Fri Jul 14 11:15:11 2006 From: mwh at python.net (Michael Hudson) Date: Fri, 14 Jul 2006 10:15:11 +0100 Subject: [Python-Dev] Community buildbots In-Reply-To: <20060713204146.29014.283506497.divmod.quotient.35352@ohm> (glyph@divmod.com's message of "Thu, 13 Jul 2006 16:41:46 -0400") References: <20060713204146.29014.283506497.divmod.quotient.35352@ohm> Message-ID: <2mbqrs35uo.fsf@starship.python.net> glyph at divmod.com writes: > I just would have appreciated the opportunity to participate in the > discussion before the betas were out and the featureset frozen. I think something that has happened to some extent with this release is that there was a long-ish period where stuff got discussed and implemented (PEPs 343 and 344, new-style exceptions, the new compiler, the ssize_t branch) and then suddenly we went "crap! we have to release this!" and then the whole alpha/beta/release part has happened much more quickly. Maybe we should have had a more explicit discussion on what was going to be in 2.5 before beta1, but that kind of thing is difficult to make happen (it's entirely possible that it did happen and I just missed it by being snowed under, but I don't think so). Cheers, mwh -- Have you considered downgrading your arrogance to a reasonable level? -- Erik Naggum, comp.lang.lisp, to yet another C++-using troll From ncoghlan at gmail.com Fri Jul 14 11:31:11 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Jul 2006 19:31:11 +1000 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: <44B72714.6030502@canterbury.ac.nz> References: <44B72714.6030502@canterbury.ac.nz> Message-ID: <44B7645F.6040900@gmail.com> Greg Ewing wrote: > Nick Maclaren wrote: >> On systems that are not Unix-derived (which, nowadays, are rare), >> there is commonly no such thing as a program name in the first place. >> It is possible to get into that state on some Unices - i.e. ones which >> have a form of exec that takes a file descriptor, inode number or >> whatever. > > I don't think that applies to the Python args[] though, > since its args[0] isn't the path of the OS-level > executable, it's the path of the main Python script. > > But you could still end up without one, if the main > script comes from somewhere other than a file. sys.argv[0] can end up being None in Python 2.5 - zipimporter objects don't currently expose the info runpy.run_module needs to set __file__ and sys.argv[0] correctly, so they'll both end up being None if you use -m to run a module from a zipfile. PEP 302 went a fairly long way towards decoupling imports from the filesystem lay out, but it isn't all the way there just yet. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From glingl at aon.at Fri Jul 14 11:33:13 2006 From: glingl at aon.at (Gregor Lingl) Date: Fri, 14 Jul 2006 11:33:13 +0200 Subject: [Python-Dev] IDLE - firewall warning Message-ID: <44B764D8.5000101@aon.at> I have posted the following message to idle-dev, but no reply yet. Just a suggestion: The firewall warning message in the Pythonshell window was introduced in Python 2.3 (IDLE 1.0 or something similar?) I remember well the problems, to which it was the answer. (If I remember correctly I was involved in the discussion thread which led to the introduction of that message.) Since more than three years I've used IDLE on many different systems and I didn't encounter a single Firewall warning since then. (It seems that firewalls nowadays are not offended by the use of 127.0.0.1) Therefore, and because the message is long and ugly, I'd like to suggest to delete it from the Python Shell window and - if considered necessary - for instance to put it into the IDLE Help - "About IDLE" submenu or in a special IDLE-Firewall warning submenu of IDLE-Help. Please observe that it pops up thousands of times and it's read at most once. Or perhaps a single-line message like === subprocesses use internal loopback interface === would do it, just to signal where you are as does the respective line in -n mode.? Regards, Gregor Lingl From rasky at develer.com Fri Jul 14 12:00:07 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 14 Jul 2006 12:00:07 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm><65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org><200607132121.27521.fdrake@acm.org><17591.7796.553081.425160@montanaro.dyndns.org> Message-ID: <09f901c6a72c$495f2690$12472597@bagio> Neal Norwitz wrote: >> a longer beta period gives *external* developers more time to catch >> up, and results in less work for the end users. > > This is the part I don't get. For the external developers, if they > care about compatibility, why aren't they testing periodically, > regardless of alpha/beta releases? Because it is a cost, and I don't want to waste my time if the trunk is unstable or whatnot. There is no official statement of the kind "all the real development is done in branches, the trunk is always very stable, feel free to grab it". Thus, we (external developers) assume that it's better to wait for the first alpha or beta before starting doing any testing. Personally, I won't touch something before the first beta: there are already enough known bugs when the alphas are shipped that I prefer to wait a little bit more. In my case, the beta period is too short. It's already a great luck if, during the beta cycle, I'm not deep into some milestone or release cycle for my own software. It might well happen that I have barely time to notice a Python beta is out, but I can't afford spending any time on it because of time constraint. By the time I'm done with my own deadlines, Python final is already out. But for the sake of the argument and the rest of this mail, let's assume I have tons of spare time to dedicate to Python 2.5 beta testing. My applications have several external dependencies (I wouldn't classify those as "many", but still around 6-7 libraries in average). For each of those libraries, there won't be Py2.5-ready RPM or Windows installer to grab and build. Most of the time, I have to download the source package, and try to build it. This also means hunting down and fixing Py2.5-related bugs in all those libraries *before* I can even boot my own application (which is what I would care about). Then I occasionally hit a core Python bug while doing so, which is good, but I have to sit and wait. Some libraries have very complex build process which are still distutils-based, but might require many other external C/C++ libraries, which need to be fetched and compiled just to setup the build environment. Alternatively, I could cross my fingers and wait for all the maintainers of the external libraries to have spare time, and dedicate to Python 2.5 upgrading. If I'm lucky, by the time RC1 is out, most of them might have binary packages available for download, or at least have their (unstable) CVS/SVN trunk fixed for Python 2.5 (which means that I'll have to fetch that unstable version and basically perform a forced upgrade of the library, which might trigger another class of compatibility/feature/regression bugs in my application, not related at all to Python 2.5 by itself, but still needed to be dealt). So I think it's useless to ask people to rush testing beta releases: it takes months to get the community synched, and will always take. It's also useless to point the finger to external developers if they don't test Python and there is a bug in a .0 release. Bugs happen. It's software that evolves. My own suggestion is that it's useless to stall a release for months to give external developers time to fix things. I think it's much better to "release early - release often", and just get the damn release out of the door. Usually, it's at that point that the whole community start working on it, and discover bugs. And so what? For production usage, .0 releases of libraries (let alone the interpreter of the language) are a no-go for all software, and I know that for a long time already. I don't ship an application of mine with a Python .0 release no matter what, no matter even if the whole testsuite passes and everything seems to work. I don't have enough benefits for the risks, so I'll wait for .1 or .2 release anyway. It's *very* fine by me if .0 release is actually the first "official" "beta" release for the community, when the core developers say "we're done" and external developers really start get things going. If you really care about .0 stability from a purely commercial point-of-view (it's bad advertisement if many people complain if a major release is broken, etc. etc.), I might suggest you to coordinate with a small group of selected external developers maintaing the larger external packages (Twisted and others). You could put a list of those packages in the release PEP as showstoppers for the release: you should not get a .0 out if those packages are broken. I think this could help smooth out the process. If important external libraries work, many applications relying on it will *mostly* work as well. I personally don't think it's such a big problem if one has to fix a couple of things in a 100K-line application to adjust it to the new .0 release, even if it's actually because of a bug in Python itself. Giovanni Bajo From ncoghlan at gmail.com Fri Jul 14 12:09:02 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Jul 2006 20:09:02 +1000 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: <44B723C6.9010309@canterbury.ac.nz> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> <44B723C6.9010309@canterbury.ac.nz> Message-ID: <44B76D3E.90503@gmail.com> Greg Ewing wrote: > Maybe sys needs to be split into two modules, with > the non-sensitive one pre-imported (so that the > importless interpreter you suggest wouldn't be > unnecessarily crippled). Maybe not splitting it, but providing a read-only mechanism of getting at certain elements, with "import sys" still being the way of *modifying* any of these things. Relatively safe items (could be provided as attributes and methods of a read-only class instance in builtins instead of as a module): argv (as a tuple instead of a list) byteorder maxint maxunicode builtin_module_names copyright exc_info() exec_prefix executable exit([arg]) getdefaultencoding() getfilesystemencoding() getwindowsversion() hexversion platform prefix stdin stdout stderr version version_info winver Arguably privileged information (no real reason for non-privileged code to know this stuff): subversion getcheckinterval() getdlopenflags() dllhandle _current_frames() getrefcount(object) getrecursionlimit() _getframe([depth]) __displayhook__ __excepthook__ __stdin__ __stdout__ __stderr__ api_version warnoptions tracebacklimit displayhook(value) excepthook(type, value, traceback) ps1 ps2 Definitely privileged operations: Actually *setting* any of the above to something different modules path exc_clear() setcheckinterval(interval) setdefaultencoding(name) setdlopenflags(n) setprofile(profilefunc) setrecursionlimit(limit) settrace(tracefunc) settscdump(on_flag) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Fri Jul 14 12:45:14 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Jul 2006 20:45:14 +1000 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <200607141714.09304.anthony@interlink.com.au> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141714.09304.anthony@interlink.com.au> Message-ID: <44B775BA.1070905@gmail.com> Anthony Baxter wrote: > On Friday 14 July 2006 16:39, Neal Norwitz wrote: >> Remember I also tried to push for more features to go in early? >> That would have given more time for external testing. Still >> features are coming in. Python developers weren't happy about >> having to get things in earlier. I don't see a practical way to >> implement what you propose (see Anthony's comments). > > Following up on this point: > Given the number of "just-this-one-more-thing-please" we've _already_ > had since the b1 feature freeze, do you really except that 90 days of > feature freeze is feasible? And if there's not going to be a feature > freeze, there's hardly any point to people doing testing until there > _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. > Oops. 2.5b9 added a new feature that broke my code, but I didn't test > with that. I think this is where release candidates can come into their own - the beta's can flush out the "just-one-more-thing" pleases, the maintenance branch is forked for rc1, and then any changes on the branch are purely to fix regressions. As far as the idea of a suite of buildbots running the unit tests of large Python applications against the current SVN trunk goes, one problem is that failures in those buildbots will come from two sources: - Python itself fails (broken checkin) - the application unit tests fail (backwards incompatibility) To weed out the false alarms, the slaves will need to be set up to run the Python unit tests first and then the application unit tests only if the Python test run is successful. When the slave suffers a real failure due to a backwards incompatibility, it will take a developer of the application to figure out what it was that broke the application's tests. So while I think it's a great idea, I also think it will need significant support from the application developers in debugging any buildbot failures to really make it work. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Fri Jul 14 12:56:22 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Jul 2006 20:56:22 +1000 Subject: [Python-Dev] IDLE - firewall warning In-Reply-To: <44B764D8.5000101@aon.at> References: <44B764D8.5000101@aon.at> Message-ID: <44B77856.8080106@gmail.com> Gregor Lingl wrote: > I have posted the following message to idle-dev, > but no reply yet. Just a suggestion: > > The firewall warning message in the Pythonshell window > was introduced in Python 2.3 (IDLE 1.0 or something similar?) > > I remember well the problems, to which it was the answer. > (If I remember correctly I was involved in the discussion > thread which led to the introduction of that message.) > > Since more than three years I've used IDLE on many different > systems and I didn't encounter a single Firewall warning since > then. (It seems that firewalls nowadays are not offended by > the use of 127.0.0.1) Unless that firewall is ZoneAlarm on Windows XP :) Something that can easily happen to suppress the firewall warnings for IDLE is that someone runs a Python application and grants it access to the internet. For an executable file signature based outbound firewall like ZoneAlarm, this actually grants passage through the firewall to *all* Python applications, since it is the interpreter binary that gets registered (ZA has a similar lack of granularity when it comes to Java applications). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From amk at amk.ca Fri Jul 14 13:21:37 2006 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 14 Jul 2006 07:21:37 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <09f901c6a72c$495f2690$12472597@bagio> References: <09f901c6a72c$495f2690$12472597@bagio> Message-ID: <20060714112137.GA891@Andrew-iBook2.local> On Fri, Jul 14, 2006 at 12:00:07PM +0200, Giovanni Bajo wrote: > unstable or whatnot. There is no official statement of the kind "all > the real development is done in branches, the trunk is always very > stable, feel free to grab it". Thus, we (external developers) assume > that it's better to wait for the first alpha or beta before starting > doing any testing. Where could we put such a statement? http://www.python.org/dev/tools/, in a discussion of checkin policies, does say: The Python source tree is managed for stability, meaning that if you make a checkout at a random point in time the tree will almost always compile and be quite stable. Large sets of changes, such as the 2.2 type/class rewrite, are usually tested on a branch first and merged into the main stream of development once they're believed to be complete. but this is buried pretty deeply. Maybe some text should be added to the release announcements about this. > I'm lucky, by the time RC1 is out, most of them might have binary packages > available for download, or at least have their (unstable) CVS/SVN trunk fixed > for Python 2.5 (which means that I'll have to fetch that unstable version and And many people won't make binary packages until 2.5 is final, so you're probably not often lucky. --amk From skip at pobox.com Fri Jul 14 13:28:31 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 14 Jul 2006 06:28:31 -0500 Subject: [Python-Dev] Community buildbots In-Reply-To: <20060714051157.29014.803759518.divmod.quotient.35955@ohm> References: <17591.7500.168119.701223@montanaro.dyndns.org> <20060714051157.29014.803759518.divmod.quotient.35955@ohm> Message-ID: <17591.32735.270029.593594@montanaro.dyndns.org> >> If you're concerned about noticing when a new release train is >> pulling out of the station ... glyph> I am aware of when new releases come out :). What I'm not aware glyph> of is what features (may) have broken my code, and why. Hmmm... Well, you could subscribe to the python-checkins mailing list. Most of the time you'd probably decide rather quickly to delete the messages, but the stuff you care about should be in the checkin comments. Skip From skip at pobox.com Fri Jul 14 13:30:36 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 14 Jul 2006 06:30:36 -0500 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <44B729A9.3060705@canterbury.ac.nz> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> <44B729A9.3060705@canterbury.ac.nz> Message-ID: <17591.32860.610966.442590@montanaro.dyndns.org> Greg> Maybe there could be an "unstable" release phase that lasts for a Greg> whole release cycle. So you'd first release version 2.n as Greg> "unstable", and keep 2.(n-1) as the current "stable" release. Then Greg> when 2.(n+1) is ready, 2.n would become "stable" and 2.(n+1) would Greg> become the new "unstable". In GCC don't they do an odd (stable)/even (unstable) release schedule? Same for Linux kernels? Would that help? Skip From skip at pobox.com Fri Jul 14 13:32:48 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 14 Jul 2006 06:32:48 -0500 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <200607140139.38384.fdrake@acm.org> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607132121.27521.fdrake@acm.org> <17591.7796.553081.425160@montanaro.dyndns.org> <200607140139.38384.fdrake@acm.org> Message-ID: <17591.32992.90867.191048@montanaro.dyndns.org> >> I believe there was some shortening of the 2.5 release cycle two or >> three months ago. Fred> The squeezing of the releases isn't where the problem is, I think. Had we been in alpha a bit longer (officially before feature freeze) I think there'd have been less back-and-forth about a couple otherwise noncontroversial checkins. Skip From fredrik at pythonware.com Fri Jul 14 13:37:28 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 14 Jul 2006 13:37:28 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <20060713184255.GA3929@rogue.amk.ca> Message-ID: "A.M. Kuchling" wrote: >> While the new python.org is very nice, I do note that there's no "blogs" >> entry on the front page, something which has become a fixture on almost >> every other website I visit regularly. > > A 'Blogs' link could be trivially added by linking to > planet.python.org, though the blogs collected there are not in any way > 'the Python developers', but a jumble of developers and users. I > don't think enough core developers have weblogs (or write about > Python) to make a 'python-dev only' planet very useful. on the other hand, the material you're producing for the "what's new" document would be a rather excellent development blog in itself. just post new sections to the blog when they're ready... (add PEP announcements and python-dev summary items to the mix, and you have a high-quality development blog generated entirely from existing content) (hmm. maybe this could be put together by a robot? time to start hacking on "The Daily URL 2.0 Beta", I suppose...) From skip at pobox.com Fri Jul 14 13:46:55 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 14 Jul 2006 06:46:55 -0500 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> <200607132121.27521.fdrake@acm.org> <17591.7796.553081.425160@montanaro.dyndns.org> Message-ID: <17591.33839.641795.66854@montanaro.dyndns.org> Neal> How often is the python build broken or otherwise unusable? Not very often. I use the head branch of the repository as the source of the interpreter I run on my laptop. It generally takes just a couple minutes on my now-aging PowerBook to svn up and reinstall. I can only recall one time in the past where I had to temporarily fall back to 2.4 because of some change that broke an application. Admittedly, I'm not as sophisticated a user as Fredrik or Glyph, but I suspect that my usage of the language isn't all that different from most Python developers out there. Neal> Is part of your point that these developers only care about Neal> something called "release" and they won't start testing before Neal> then? If that's the case why don't we start making Neal> (semi-)automated alpha releases every month? How would that be any easier than a user setting up a read-only repository and svn-up-ing it once a month then using that as the default interpreter on that person's development machine? I maintain interpreters for 2.3, 2.4 and bleeding edge at the moment. If I need to it's fairly trivial (a symlink change) to fall back to the latest stable release. Glyph, would that sort of scheme work for you? Skip From skip at pobox.com Fri Jul 14 13:49:24 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 14 Jul 2006 06:49:24 -0500 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org> <200607132121.27521.fdrake@acm.org> <17591.7796.553081.425160@montanaro.dyndns.org> Message-ID: <17591.33988.996995.204882@montanaro.dyndns.org> >> This is the part I don't get. For the external developers, if they >> care about compatibility, why aren't they testing periodically, >> regardless of alpha/beta releases? Fredrik> because they don't want to waste time and effort on testing Fredrik> against releases that are not necessarily a close approximation Fredrik> of the full release? testing takes time, and time can be used Fredrik> for many different things. How is that a waste of time? You've got the latest stable 2.4 installed I presume. Do the svn-up/make dance (at least on Unix-y systems - can it be much more difficult on Windows?). If it breaks something, either figure out why or drop back to 2.4 for a week or two and try again. If application breakage remains, investigate. Skip From bborcic at gmail.com Fri Jul 14 14:38:22 2006 From: bborcic at gmail.com (Boris Borcic) Date: Fri, 14 Jul 2006 14:38:22 +0200 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> Message-ID: Guido van Rossum wrote: ... > > This is an illustration of the dilemma of maintaining a popular > language: Everybody hates change (me too!) but everybody also has one > thing that's bothering them so much they absolutely want it to be > changed. If you were to implement all those personal pet peeves, you'd > get a language that's more different from Python than Python is from > Fortran. > > So where's the middle ground? I feel some freedom could be reclaimed with a solution in the spirit of Turing equivalence. Or, to take a less grandiose comparison, web style sheets - separation of content and presentation. Suppose the standard required a (possibly empty) style-defining file prefix that constrains the python source code in the file, and concurrently defined (mostly) reversible and transparent source-to-source transforms that would map any source code file to an equivalent source code file with an arbitrary chosen prefix. Then users could chose their style of Python and either transform all source files they install to their own style, or setup their editor to do it back-and-forth for them. The choice of python presentation style would then become a private choice. To illustrate the idea, this already exists in very embryonic form with unicode encoding modelines. The current standard allows to imagine a Python editor that would permit to set a "local standard encoding modeline" and then present any source file as if it had been written while taking maximal profit from the chosen encoding. Which may also be simple ascii. Cheers, BB -- "C++ is a contradiction in terms" - Lorentz, Einstein, Poincar? From jeremy at alum.mit.edu Fri Jul 14 14:45:04 2006 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri, 14 Jul 2006 08:45:04 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <200607141714.09304.anthony@interlink.com.au> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141714.09304.anthony@interlink.com.au> Message-ID: On 7/14/06, Anthony Baxter wrote: > On Friday 14 July 2006 16:39, Neal Norwitz wrote: > > Remember I also tried to push for more features to go in early? > > That would have given more time for external testing. Still > > features are coming in. Python developers weren't happy about > > having to get things in earlier. I don't see a practical way to > > implement what you propose (see Anthony's comments). > > Following up on this point: > Given the number of "just-this-one-more-thing-please" we've _already_ > had since the b1 feature freeze, do you really except that 90 days of > feature freeze is feasible? And if there's not going to be a feature > freeze, there's hardly any point to people doing testing until there > _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. > Oops. 2.5b9 added a new feature that broke my code, but I didn't test > with that. Maybe the basic question is right, but the emphasis needs to be changed. If we had a rule that said the final release was 90 days after the last submission that wasn't to fix a regression, we'd ask "Is this feature important enough to warrant delaying the release until three months from now?" I'm not sure what I think, but it doesn't seem like an implausible policy. Jeremy > > Anthony > -- > Anthony Baxter > It's never too late to have a happy childhood. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > From jeremy at alum.mit.edu Fri Jul 14 14:48:07 2006 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri, 14 Jul 2006 08:48:07 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <44B775BA.1070905@gmail.com> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141714.09304.anthony@interlink.com.au> <44B775BA.1070905@gmail.com> Message-ID: On 7/14/06, Nick Coghlan wrote: > Anthony Baxter wrote: > > On Friday 14 July 2006 16:39, Neal Norwitz wrote: > >> Remember I also tried to push for more features to go in early? > >> That would have given more time for external testing. Still > >> features are coming in. Python developers weren't happy about > >> having to get things in earlier. I don't see a practical way to > >> implement what you propose (see Anthony's comments). > > > > Following up on this point: > > Given the number of "just-this-one-more-thing-please" we've _already_ > > had since the b1 feature freeze, do you really except that 90 days of > > feature freeze is feasible? And if there's not going to be a feature > > freeze, there's hardly any point to people doing testing until there > > _is_ a feature freeze, is there? Oh, great, my code works with 2.5b1. > > Oops. 2.5b9 added a new feature that broke my code, but I didn't test > > with that. > > I think this is where release candidates can come into their own - the beta's > can flush out the "just-one-more-thing" pleases, the maintenance branch is > forked for rc1, and then any changes on the branch are purely to fix regressions. > > As far as the idea of a suite of buildbots running the unit tests of large > Python applications against the current SVN trunk goes, one problem is that > failures in those buildbots will come from two sources: > - Python itself fails (broken checkin) > - the application unit tests fail (backwards incompatibility) > > To weed out the false alarms, the slaves will need to be set up to run the > Python unit tests first and then the application unit tests only if the Python > test run is successful. > > When the slave suffers a real failure due to a backwards incompatibility, it > will take a developer of the application to figure out what it was that broke > the application's tests. > > So while I think it's a great idea, I also think it will need significant > support from the application developers in debugging any buildbot failures to > really make it work. These buildbots should run the tests from stable, released versions of external packages, assuming that those packages don't ship releases with failing tests. If you ran the test suite for a Zope release and had a test failure, I think there would be a reasonable expectation that it was a Python bug. I'd definitely avoiding mixing a development version of Python and a development version of the test software in the automated build environment. Jeremy > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > http://www.boredomandlaziness.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/jeremy%40alum.mit.edu > From rasky at develer.com Fri Jul 14 15:06:06 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 14 Jul 2006 15:06:06 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm><65ECC6D7-7E46-463C-92D9-37AF2BF15234@python.org><44B729A9.3060705@canterbury.ac.nz> <17591.32860.610966.442590@montanaro.dyndns.org> Message-ID: <015d01c6a746$44808320$d503030a@trilan> skip at pobox.com wrote: > Greg> Maybe there could be an "unstable" release phase that lasts > for a Greg> whole release cycle. So you'd first release version > 2.n as Greg> "unstable", and keep 2.(n-1) as the current "stable" > release. Then Greg> when 2.(n+1) is ready, 2.n would become > "stable" and 2.(n+1) would Greg> become the new "unstable". > > In GCC don't they do an odd (stable)/even (unstable) release > schedule? Same for Linux kernels? Would that help? No. Linux kernel releases are more aggressive because of the fact that all the patches are mostly developed in different branch/repositories and get merged when they are already well tested and incorporated. Linus can merge literally hundreds of patches daily into his *stable* tree, and do releases from it even weekly, because most destabilizing works are being done in large branches carried on for months before they even are evaluated for being merged; or because patches were settled in the -mm tree for months. Linus' tree is kind-of a release branch, with the difference that he is the BDFL and does what he wants with his tree :) To keep this into perspective, remember also that they don't have *any* kind of testsuite (nor a debugger, if I might say). GCC has a more "old-fashioned" release process, where the trunk evolves through 3 stages: Stage 1 is open for all kind of changes (eg: from simple polishing/refactoring, to merging of large branches containing work of several man-years). Stage 2 is still open for new features, but not for big merges. Stage 3 is feature-freezed, bug-fixing only. Then, the trunk is branched into the new release branch, and the trunk gets back to Stage 1. Nominally, a stage lasts 3 months, but Stage 3 often stretches up to 6 months. The release branches are open for *only* regression fixes (that is, fixes that correct things that used to work in previous releases but do not work anymore). Any regression fix (with a corresponding Bugzilla entry, where it's marked and confirmed as "regression") is applied to trunk *and* the open release branches where the regression exists. For convoluted or dangerous regression fixes, usually maintainers prefer to wait 1 week for the patch to settle down on the trunk before applying it to the release branches. The release manager pulls dot releases from the release branch. Usually, the first release (.0) happens 3-4 months *after* the release branch was created, that is after several months of regression-only patches being merged to it (while new work is being done on the trunk in parallel, in its aggressive Stage 1). The 3-Stage work in the trunk is streamlined by the release manager. At the beginning of Stage 1, a detailed techinical list of on-going "projects" (new features) is presented to the mailing list, explaining the current status of the work and its ETA, and the release manager then publishes a work-plan for Stage 1 and 2, telling which projects will be merged when. This avoids multiple large projects to hit the trunk at the same time, causing may headaches to all the other developers. The work-plan is managed and updated in the GCC Wiki (which is off-line right now, but I'll post a link as example when it's back). -- Giovanni Bajo From amk at amk.ca Fri Jul 14 15:26:54 2006 From: amk at amk.ca (A.M. Kuchling) Date: Fri, 14 Jul 2006 09:26:54 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <20060713184255.GA3929@rogue.amk.ca> Message-ID: <20060714132654.GA4858@localhost.localdomain> On Fri, Jul 14, 2006 at 01:37:28PM +0200, Fredrik Lundh wrote: > (add PEP announcements and python-dev summary items to the mix, and you > have a high-quality development blog generated entirely from existing content) > (hmm. maybe this could be put together by a robot? time to start hacking on > "The Daily URL 2.0 Beta", I suppose...) There's already an RSS feed for the python-dev summaries, so that could go into an aggregator. There isn't a feed for the PEPs, but those events are rarer and could be handled manually. --amk From bborcic at gmail.com Fri Jul 14 15:46:26 2006 From: bborcic at gmail.com (Boris Borcic) Date: Fri, 14 Jul 2006 15:46:26 +0200 Subject: [Python-Dev] Explicit Lexical Scoping (pre-PEP?) In-Reply-To: References: Message-ID: [Fredrik Lundh] >>>> >>>> def counter(num): >>>> num = mutable_int(num) >>>> def inc(): >>>> num += 1 >>>> return num >>>> return inc >> >> feel free to replace that += with an .add(1) method call; the point >> wasn't the behaviour of augmented assigment, the point was that that the >> most common use pattern involves *mutation* of the target object. >> >> the syntax isn't that important, really. No it isn't, but I believe it deserves notice that it's easy enough, currently, to mock your mutable_int replacing the "+=" by a simple "+", while the "+=" itself is not possible even though it reflects the intention more precisely. [Jeremy Hylton] > > Mutation is different from rebinding.A tuple is immutable, but you > can rebind the variable that refers to the tuple. It's more than just rebinding, it's rebinding to a function of the currently bound value. And /that/ has clear features in common with mutation (witness most of the current semantics and implementation of augmented assignments, ie, how much and far it manages to postpone the decision as to which is actually the case). > I think we will > confuse users if we use the term mutation to refer to name binding. > Name binding is already a subtle issue, so I think the risk is > significant. I'd tend to agree, but imho there is a complementary risk of propagating confusion by denying what augmented assignments clearly show : that rebinding to a function of the currently bound value has so much in common with mutation that augmented assignments allow to confuse both cases in source code. Except for the scoping issue under discussion, that is. Cheers, BB -- "C++ is a contradiction in terms" - Lorentz, Einstein, Poincar? From theller at python.net Fri Jul 14 17:09:37 2006 From: theller at python.net (Thomas Heller) Date: Fri, 14 Jul 2006 17:09:37 +0200 Subject: [Python-Dev] Partial support for dlmodule.c in 64-bits OSes In-Reply-To: References: Message-ID: Pierre Baillargeon schrieb: > Currently, many 64-bits Oses cannot uses the dlmodule due to the conflicts > between the sizes of int, long and char *. That is well. The check is made as > run-time, which is also very well. > > The problem is that the Python configuration script (setup.py) also makes the > check and plainly excludes dlmodule.c from being built and deployed. That is not > so well. > > The reason is that we use the dlmodule solely to get access to the various flags > (RTLD_NOW, RTLD_GLOBAL, etc), so that we can do some magic with loaded shared > libraries, such as over-ridding the import mechanism so that the default load > flags get changed (via sys.setdlopenflags()) to force some semantics. > > Currently this doesn't work on most 64-bits OSes because the dl module doesn't > exists, so it cannot be imported and its RTLD_* symbols are not accessible. > > So I ask if it would be possible that the test for sys.maxint == 0x7fffffff in > setup.py be dropped in future releases. I don't know if your patch is acceptable or not, but if it is applied Lib/test/test_dl.py crashes on 64-bit platforms, so this must be changed as well. Further, patches should be uploaded to the SF tracker so they don't get lost or forgotten. OTOH, the RTLD_ constants are possibly available in the DLFCN module, as the documentation explains. Thomas From ncoghlan at gmail.com Fri Jul 14 17:12:29 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Jul 2006 01:12:29 +1000 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141714.09304.anthony@interlink.com.au> <44B775BA.1070905@gmail.com> Message-ID: <44B7B45D.1070709@gmail.com> Jeremy Hylton wrote: >> When the slave suffers a real failure due to a backwards >> incompatibility, it >> will take a developer of the application to figure out what it was >> that broke >> the application's tests. >> >> So while I think it's a great idea, I also think it will need significant >> support from the application developers in debugging any buildbot >> failures to >> really make it work. > > These buildbots should run the tests from stable, released versions of > external packages, assuming that those packages don't ship releases > with failing tests. If you ran the test suite for a Zope release and > had a test failure, I think there would be a reasonable expectation > that it was a Python bug. Definitely, but there's a difference between "bug that broke Python's own unit tests" and "change to a behaviour that package X depended on". It's the latter cases that the external buildbots would be looking for - and while some of those will be shallow enough that the regression is obvious from the unit test error message and the recent Python checkins, the non-obvious ones will require a collaborative resolution. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From glyph at divmod.com Fri Jul 14 17:26:36 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 14 Jul 2006 11:26:36 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <17591.33839.641795.66854@montanaro.dyndns.org> Message-ID: <20060714152636.29014.1693386533.divmod.quotient.36584@ohm> On Fri, 14 Jul 2006 06:46:55 -0500, skip at pobox.com wrote: > > Neal> How often is the python build broken or otherwise unusable? > >Not very often. I have to agree. The effort I'm talking about is not in fixing large numbers of problems, but simply gearing up to properly test to see if there *are* problems. Keep in mind though: just because the problems are small or easy to fix doesn't mean they're not severe. One tiny bug can prevent a program from even starting up. >Admittedly, I'm not as sophisticated a user as Fredrik or Glyph, but I >suspect that my usage of the language isn't all that different from most >Python developers out there. A huge percentage of Python developers are working with Zope, which means that although *their* code might not be terribly "sophisticated", it is nevertheless sitting on top of a rather large and intricate pile of implicit dependencies on interpreter behavior. > Neal> Is part of your point that these developers only care about > Neal> something called "release" and they won't start testing before > Neal> then? If that's the case why don't we start making > Neal> (semi-)automated alpha releases every month? >How would that be any easier than a user setting up a read-only repository >and svn-up-ing it once a month then using that as the default interpreter on >that person's development machine? I maintain interpreters for 2.3, 2.4 and >bleeding edge at the moment. If I need to it's fairly trivial (a symlink >change) to fall back to the latest stable release. >Glyph, would that sort of scheme work for you? No. I really think you're underestimating the sort of effort required to upgrade Python. First of all, I do have a job :) and it'd be very hard to make the case to an investor that it was worth tracking down every potential bug I had to determine whether it was a problem because I was working with an unreleased version of Python. This is the same reason I don't use beta versions of the kernel or libc for development. For that matter, I've had to avoid posting to this mailing list because even *this* is stretching my already overburdened schedule :). Secondly, I have to work with a few extension modules. To name a few: * ctypes * PyPAM * pysqlite2 * zope.interface (C optimizations) * xapian * pylucene * dspam * PyOpenSSL * PIL * PyCrypto * pyexpat * pygtk * pyvte Recompiling all of these is a project that takes a day. PyLucene, in fact, I've never managed to build myself, and I can only run because there happen to be debian packages which work (with some fiddling) on Ubuntu. There's no interactive warning during 'svn up' that it's time to recompile everything, either, so I don't even know when the ABIs are going to have changed. Even if everything works perfectly, and were perfectly automated, the compile itself would take a few hours. I also test work on Windows on occasion and recompiling these things _there_ is the work of a week and a half, not to mention it requires that I be sitting at the one machine where I have my Microsoft? Developer? Tools? installed. I made the buildbot recommendation specifically because it would centralize the not inconsiderable effort of integrating these numerous dependencies (presuming that I could submit a Divmod buildbot as well as a Twisted one). From glyph at divmod.com Fri Jul 14 17:27:41 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 14 Jul 2006 11:27:41 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: Message-ID: <20060714152741.29014.392313751.divmod.quotient.36585@ohm> On Thu, 13 Jul 2006 23:39:06 -0700, Neal Norwitz wrote: >On 7/13/06, Fredrik Lundh wrote: >> a longer beta period gives *external* developers more time to catch up, >> and results in less work for the end users. >This is the part I don't get. For the external developers, if they >care about compatibility, why aren't they testing periodically, >regardless of alpha/beta releases? How often is the python build >broken or otherwise unusable? How often do you test new builds of Python against the most recent alpha of, e.g. the Linux kernel? This isn't just a hypothetical question: Twisted has broken because of changes to Linux as often as it has broken due to changes in Python :). In Linux's case we're all lucky because *any* regressions with existing software are considered bugs, whereas in Python's case, some breakagaes are considered acceptable since it's more feasible to have multiple builds of Python installed more than multiple kernels for different applications. From jcarlson at uci.edu Fri Jul 14 18:21:47 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 14 Jul 2006 09:21:47 -0700 Subject: [Python-Dev] IDLE - firewall warning In-Reply-To: <44B764D8.5000101@aon.at> References: <44B764D8.5000101@aon.at> Message-ID: <20060714091004.0194.JCARLSON@uci.edu> Gregor Lingl wrote: > I have posted the following message to idle-dev, > but no reply yet. Just a suggestion: > > The firewall warning message in the Pythonshell window > was introduced in Python 2.3 (IDLE 1.0 or something similar?) Speaking of 'features that would have been nice to get into 2.5'; had we gotten the asynchronous subprocess support in, we could have done away with the socket support, though it would have required a bit of modification to the XML-RPC implementation. - Josiah From steven.bethard at gmail.com Fri Jul 14 18:24:56 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 14 Jul 2006 10:24:56 -0600 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <20060714132654.GA4858@localhost.localdomain> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <20060713184255.GA3929@rogue.amk.ca> <20060714132654.GA4858@localhost.localdomain> Message-ID: On 7/14/06, A.M. Kuchling wrote: > On Fri, Jul 14, 2006 at 01:37:28PM +0200, Fredrik Lundh wrote: > > (add PEP announcements and python-dev summary items to the mix, and you > > have a high-quality development blog generated entirely from existing content) > > (hmm. maybe this could be put together by a robot? time to start hacking on > > "The Daily URL 2.0 Beta", I suppose...) > > There's already an RSS feed for the python-dev summaries, so that > could go into an aggregator. There isn't a feed for the PEPs, but > those events are rarer and could be handled manually. I believe that's broken at the moment, at least in the sense that it's no longer updated: http://psf.pollenation.net/cgi-bin/trac.cgi/ticket/366 I don't have enough time right now to figure out the new website and how to fix it. Maybe when I get back from Sydney in a couple of weeks. Steve -- I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a tiny blip on the distant coast of sanity. --- Bucky Katt, Get Fuzzy From guido at python.org Fri Jul 14 21:44:21 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Jul 2006 12:44:21 -0700 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> Message-ID: You must be misunderstanding. The root problem is that people (rightly) complain that the language changes too much. And you want to "fix" this by adding a deep and fundamental change to the language? What planet are you from? It reminds me of Jini, which was presented as a new standard to address the problem of too many conflicting standard. Get it? :-) --Guido On 7/14/06, Boris Borcic wrote: > Guido van Rossum wrote: > ... > > > > This is an illustration of the dilemma of maintaining a popular > > language: Everybody hates change (me too!) but everybody also has one > > thing that's bothering them so much they absolutely want it to be > > changed. If you were to implement all those personal pet peeves, you'd > > get a language that's more different from Python than Python is from > > Fortran. > > > > So where's the middle ground? > > I feel some freedom could be reclaimed with a solution in the spirit of Turing > equivalence. Or, to take a less grandiose comparison, web style sheets - > separation of content and presentation. > > Suppose the standard required a (possibly empty) style-defining file prefix that > constrains the python source code in the file, and concurrently defined (mostly) > reversible and transparent source-to-source transforms that would map any source > code file to an equivalent source code file with an arbitrary chosen prefix. > Then users could chose their style of Python and either transform all source > files they install to their own style, or setup their editor to do it > back-and-forth for them. The choice of python presentation style would then > become a private choice. > > To illustrate the idea, this already exists in very embryonic form with unicode > encoding modelines. The current standard allows to imagine a Python editor that > would permit to set a "local standard encoding modeline" and then present any > source file as if it had been written while taking maximal profit from the > chosen encoding. Which may also be simple ascii. > > Cheers, BB > -- > "C++ is a contradiction in terms" - Lorentz, Einstein, Poincar? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Jul 14 21:48:56 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Jul 2006 12:48:56 -0700 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: <44B76D3E.90503@gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> <44B723C6.9010309@canterbury.ac.nz> <44B76D3E.90503@gmail.com> Message-ID: Whoa, whoa. What's the *problem* we're trying to solve here? On 7/14/06, Nick Coghlan wrote: > Greg Ewing wrote: > > Maybe sys needs to be split into two modules, with > > the non-sensitive one pre-imported (so that the > > importless interpreter you suggest wouldn't be > > unnecessarily crippled). > > Maybe not splitting it, but providing a read-only mechanism of getting at > certain elements, with "import sys" still being the way of *modifying* any of > these things. > > Relatively safe items (could be provided as attributes and methods of a > read-only class instance in builtins instead of as a module): > > argv (as a tuple instead of a list) > byteorder > maxint > maxunicode > builtin_module_names > copyright > exc_info() > exec_prefix > executable > exit([arg]) > getdefaultencoding() > getfilesystemencoding() > getwindowsversion() > hexversion > platform > prefix > stdin > stdout > stderr > version > version_info > winver > > > Arguably privileged information (no real reason for non-privileged code to > know this stuff): > subversion > getcheckinterval() > getdlopenflags() > dllhandle > _current_frames() > getrefcount(object) > getrecursionlimit() > _getframe([depth]) > __displayhook__ > __excepthook__ > __stdin__ > __stdout__ > __stderr__ > api_version > warnoptions > tracebacklimit > displayhook(value) > excepthook(type, value, traceback) > ps1 > ps2 > > > Definitely privileged operations: > Actually *setting* any of the above to something different > modules > path > exc_clear() > setcheckinterval(interval) > setdefaultencoding(name) > setdlopenflags(n) > setprofile(profilefunc) > setrecursionlimit(limit) > settrace(tracefunc) > settscdump(on_flag) > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > http://www.boredomandlaziness.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Fri Jul 14 23:30:17 2006 From: brett at python.org (Brett Cannon) Date: Fri, 14 Jul 2006 14:30:17 -0700 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> <44B723C6.9010309@canterbury.ac.nz> <44B76D3E.90503@gmail.com> Message-ID: On 7/14/06, Guido van Rossum wrote: > > Whoa, whoa. What's the *problem* we're trying to solve here? I have a use case for sandboxing. I am already having to plan to have a mini-sys module in a sandbox so that they cannot get access to dangerous things. -Brett On 7/14/06, Nick Coghlan wrote: > > Greg Ewing wrote: > > > Maybe sys needs to be split into two modules, with > > > the non-sensitive one pre-imported (so that the > > > importless interpreter you suggest wouldn't be > > > unnecessarily crippled). > > > > Maybe not splitting it, but providing a read-only mechanism of getting > at > > certain elements, with "import sys" still being the way of *modifying* > any of > > these things. > > > > Relatively safe items (could be provided as attributes and methods of a > > read-only class instance in builtins instead of as a module): > > > > argv (as a tuple instead of a list) > > byteorder > > maxint > > maxunicode > > builtin_module_names > > copyright > > exc_info() > > exec_prefix > > executable > > exit([arg]) > > getdefaultencoding() > > getfilesystemencoding() > > getwindowsversion() > > hexversion > > platform > > prefix > > stdin > > stdout > > stderr > > version > > version_info > > winver > > > > > > Arguably privileged information (no real reason for non-privileged code > to > > know this stuff): > > subversion > > getcheckinterval() > > getdlopenflags() > > dllhandle > > _current_frames() > > getrefcount(object) > > getrecursionlimit() > > _getframe([depth]) > > __displayhook__ > > __excepthook__ > > __stdin__ > > __stdout__ > > __stderr__ > > api_version > > warnoptions > > tracebacklimit > > displayhook(value) > > excepthook(type, value, traceback) > > ps1 > > ps2 > > > > > > Definitely privileged operations: > > Actually *setting* any of the above to something different > > modules > > path > > exc_clear() > > setcheckinterval(interval) > > setdefaultencoding(name) > > setdlopenflags(n) > > setprofile(profilefunc) > > setrecursionlimit(limit) > > settrace(tracefunc) > > settscdump(on_flag) > > > > Cheers, > > Nick. > > > > -- > > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > --------------------------------------------------------------- > > http://www.boredomandlaziness.org > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > http://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060714/42f411c8/attachment-0001.html From guido at python.org Fri Jul 14 23:32:52 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Jul 2006 14:32:52 -0700 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> <44B723C6.9010309@canterbury.ac.nz> <44B76D3E.90503@gmail.com> Message-ID: On 7/14/06, Brett Cannon wrote: > On 7/14/06, Guido van Rossum wrote: > > Whoa, whoa. What's the *problem* we're trying to solve here? > > I have a use case for sandboxing. I am already having to plan to have a > mini-sys module in a sandbox so that they cannot get access to dangerous > things. OK, then I propose that we wait to see which things you end up having to provide to sandboxed code, rather than trying to analyze it to death in abstracto. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Fri Jul 14 23:36:07 2006 From: brett at python.org (Brett Cannon) Date: Fri, 14 Jul 2006 14:36:07 -0700 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <44B60B5A.2080309@canterbury.ac.nz> <44B723C6.9010309@canterbury.ac.nz> <44B76D3E.90503@gmail.com> Message-ID: On 7/14/06, Guido van Rossum wrote: > > On 7/14/06, Brett Cannon wrote: > > On 7/14/06, Guido van Rossum wrote: > > > Whoa, whoa. What's the *problem* we're trying to solve here? > > > > I have a use case for sandboxing. I am already having to plan to have a > > mini-sys module in a sandbox so that they cannot get access to dangerous > > things. > > OK, then I propose that we wait to see which things you end up having > to provide to sandboxed code, rather than trying to analyze it to > death in abstracto. Fine by me. My design doc already has a preliminary list going that people can take a look at. But it could still change if security analysis shows possible problems with exposing some of them. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060714/0dbcccc5/attachment.html From martin at v.loewis.de Fri Jul 14 23:38:52 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 14 Jul 2006 23:38:52 +0200 Subject: [Python-Dev] Community buildbots In-Reply-To: <60ed19d40607131406u498b0901oc84296f4e631ae5e@mail.gmail.com> References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> <05af01c6a6be$54b653b0$d503030a@trilan> <60ed19d40607131406u498b0901oc84296f4e631ae5e@mail.gmail.com> Message-ID: <44B80EEC.1030306@v.loewis.de> Christopher Armstrong wrote: > python -U is a failure for obvious reasons and a > __future__ import is clearly better. I disagree. Regards, Martin From martin at v.loewis.de Fri Jul 14 23:42:30 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 14 Jul 2006 23:42:30 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com> References: <20060713182916.GA12832@panix.com> <20060713223024.29014.1648933869.divmod.quotient.35468@ohm> <60ed19d40607131532g3c0a5cc1n3177a22d90999dab@mail.gmail.com> Message-ID: <44B80FC6.4090006@v.loewis.de> Christopher Armstrong wrote: > I'm at least willing to set up the buildbots for projects I care about > (Twisted, pydoctor, whatever), and perhaps help out with the setting > up the buildmaster. If you need trigger calls from subversion's post-commit hooks, please let me know, and I can set them up. Regards, Martin From tjreedy at udel.edu Sat Jul 15 04:12:44 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 14 Jul 2006 22:12:44 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) References: <09f901c6a72c$495f2690$12472597@bagio> <20060714112137.GA891@Andrew-iBook2.local> Message-ID: "A.M. Kuchling" wrote in message news:20060714112137.GA891 at Andrew-iBook2.local... > http://www.python.org/dev/tools/, in a discussion of checkin policies, > does say: > > The Python source tree is managed for stability, meaning that > if you make a checkout at a random point in time the tree will almost > always compile and be quite stable. That is the goal, but when I watched the buildbot results last spring, the degree of stability (greenness) appeared to vary. Is it possible to tag particular versions as a 'green' version, or the 'most recent green version' worth playing with? tjr From glyph at divmod.com Sat Jul 15 04:47:26 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Fri, 14 Jul 2006 22:47:26 -0400 Subject: [Python-Dev] Community buildbots In-Reply-To: <44B80EEC.1030306@v.loewis.de> Message-ID: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm> On Fri, 14 Jul 2006 23:38:52 +0200, "\"Martin v. L?wis\"" wrote: >Christopher Armstrong wrote: >> python -U is a failure for obvious reasons and a >> __future__ import is clearly better. > >I disagree. I am surprised that you do, since I thought that Chris's conclusion was pretty obvious. Python -U doesn't work, even on the standard library. For example, glyph at legion:~% python -S -U -c 'import pickletools' Traceback (most recent call last): File "", line 1, in ? File "/usr/lib/python2.4/pickletools.py", line 219, in ? doc="One-byte unsigned integer.") File "/usr/lib/python2.4/pickletools.py", line 187, in __init__ assert isinstance(name, str) AssertionError This was just the first convenient example. There are others. A __future__ import would allow these behaviors to be upgraded module-by-module. Right now, all -U provides is an option that can't be used on any realistically sized program, so I don't see what the utility is. From tjreedy at udel.edu Sat Jul 15 06:13:35 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 15 Jul 2006 00:13:35 -0400 Subject: [Python-Dev] Community buildbots References: <17591.7500.168119.701223@montanaro.dyndns.org> <20060714051157.29014.803759518.divmod.quotient.35955@ohm> Message-ID: wrote in message news:20060714051157.29014.803759518.divmod.quotient.35955 at ohm... > On Thu, 13 Jul 2006 23:27:56 -0500, skip at pobox.com wrote: > >>The buildbot idea sounds excellent. > > Thanks. If someone can set this up, it pretty much addresses my > concerns. > ... > I am aware of when new releases come out :). What I'm not aware of is > what > features (may) have broken my code, and why. As long as Python's > trunk at HEAD > continues to run the test suites cleanly, I am mostly unconcerned. When > it > breaks, though, I want a chance to look at the cause of the breakage, > *before* > there is an alpha or beta Python release out and people are starting to > write > code that depends on its new features. Is the following something like what you are suggesting? A Python Application Testing (PAT) machine is set up with buildbot and any needed custom scripts. Sometime after that and after 2.5 is released, when you have a version of, for instance, Twisted that passes its automated test suite when run on 2.5, you send it (or a URL) and an email address to PAT. Other developers do the same. Periodically (once a week?), when PAT is free and a new green development version of either the 2.5.x or 2.6 branches is available, PAT runs the test suites against that version. An email is sent for any that fail, perhaps accompanied by the concatenation of the relevant checkin message. Some possible options are to select just one of the branches for testing, to have more than one stable version being tested, and to receive pass emails. Terry Jan Reedy From ncoghlan at gmail.com Sat Jul 15 06:35:08 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 15 Jul 2006 14:35:08 +1000 Subject: [Python-Dev] Community buildbots In-Reply-To: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm> References: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm> Message-ID: <44B8707C.3050207@gmail.com> glyph at divmod.com wrote: > A __future__ import would allow these behaviors to be upgraded module-by-module. No it wouldn't. __future__ works solely on the semantics of different pieces of syntax, because any syntax changes are purely local to the current module. This doesn't work for datatypes, because data types can cross module boundaries - other modules may still be using the old behaviour, and there's nothing the current module can do about it. Changing all the literals in a module to be unicode instances instead of str instances is merely scratching the surface of the problem - such a module would still cause severe problems for any non-Unicode aware applications that expected it to return strings. > Right now, all -U provides is an option that can't be used on any realistically > sized program, so I don't see what the utility is. There's never been a concerted effort to make even the standard library work under -U. Maybe that should be a goal for Python 2.6 - figure out what tools or changes are needed to make code -U safe, add them, and then update the standard library to use them. PEP 349 (currently deferred, although I don't recall why) discusses some of the issues involved in making code unicode-safe, while still supporting non-unicode safe applications as clients. Cheers, Nick. [1] http://www.python.org/dev/peps/pep-0349/ -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From nmm1 at cus.cam.ac.uk Sat Jul 15 08:59:53 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Sat, 15 Jul 2006 07:59:53 +0100 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) Message-ID: "Guido van Rossum" wrote: > > OK, then I propose that we wait to see which things you end up having > to provide to sandboxed code, rather than trying to analyze it to > death in abstracto. However, the ORIGINAL proposal in this thread (to split off argv[0] and/or make that and the arguments read-only) is entirely different. That is purely a matter of convenience, cleanliness of specification or whatever you call it. I can't imagine any good reason to separate argv[0] from argv[1:] by a sandbox (either way). Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From ashemedai at gmail.com Sat Jul 15 09:43:00 2006 From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven) Date: Sat, 15 Jul 2006 09:43:00 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <200607141343.38424.anthony@interlink.com.au> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141343.38424.anthony@interlink.com.au> Message-ID: <3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com> On 7/14/06, Anthony Baxter wrote: > The "community buildbot" idea is a good one, although it should just > be possible to write something for buildbot that checks out and > builds the latest Python SVN, then installs it to a temporary > location, then adds that location to the front of the path. Then each > project could just add a "Python SVN buildbot" to their exists bbot > install. This is what Xenofarm for Pike has done for a long time now. See for example: http://pike.ida.liu.se/development/pikefarm/7.7.xml This is also what Bitten (http://bitten.cmlenz.net/) has implemented for Trac (which is one of the bug/incident trackers for Python's call for trackers). -- Jeroen Ruigrok van der Werven From martin at v.loewis.de Sat Jul 15 10:36:28 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 15 Jul 2006 10:36:28 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <09f901c6a72c$495f2690$12472597@bagio> <20060714112137.GA891@Andrew-iBook2.local> Message-ID: <44B8A90C.6070309@v.loewis.de> Terry Reedy wrote: > That is the goal, but when I watched the buildbot results last spring, the > degree of stability (greenness) appeared to vary. Is it possible to tag > particular versions as a 'green' version, or the 'most recent green > version' worth playing with? Don't get confused by these colors. The tree compiled nearly all of the time, even if some tests were failing for an extended period of time on some platforms. Regards, Martin From martin at v.loewis.de Sat Jul 15 10:40:26 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 15 Jul 2006 10:40:26 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141343.38424.anthony@interlink.com.au> <3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com> Message-ID: <44B8A9FA.90209@v.loewis.de> Jeroen Ruigrok van der Werven wrote: > This is what Xenofarm for Pike has done for a long time now. See for > example: http://pike.ida.liu.se/development/pikefarm/7.7.xml > > This is also what Bitten (http://bitten.cmlenz.net/) has implemented > for Trac (which is one of the bug/incident trackers for Python's call > for trackers). Are you aware of http://www.python.org/dev/buildbot/ ? We are not just talking about buildbots here (which the links you refer to seem to be); we are talking about buildbots that don't test Python, but test Python applications. We do know how to run a buildbot. Regards, Martin From martin at v.loewis.de Sat Jul 15 10:43:22 2006 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 15 Jul 2006 10:43:22 +0200 Subject: [Python-Dev] Community buildbots In-Reply-To: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm> References: <20060715024726.29014.1894639716.divmod.quotient.37353@ohm> Message-ID: <44B8AAAA.70304@v.loewis.de> glyph at divmod.com wrote: >>> python -U is a failure for obvious reasons and a >>> __future__ import is clearly better. >> I disagree. > > I am surprised that you do, since I thought that Chris's conclusion was > pretty obvious. Python -U doesn't work, even on the standard library. Sure, it doesn't work. That doesn't mean it's "a failure". It just hasn't been completed yet. > A __future__ import would allow these behaviors to be upgraded module-by-module. > Right now, all -U provides is an option that can't be used on any realistically > sized program, so I don't see what the utility is. People can use it to improve the Unicode support in the Python standard library. When they find that something doesn't work, they can study the problem, and ponder possible solutions. Then, they can contribute patches. -U has worked fine for me in the past, I contributed various patches to make it work better. It hasn't failed for me at all. Regards, Martin From stephen.thorne at gmail.com Sat Jul 15 10:44:18 2006 From: stephen.thorne at gmail.com (Stephen Thorne) Date: Sat, 15 Jul 2006 18:44:18 +1000 Subject: [Python-Dev] list.__init__() vs. dict.__init__() behaviour Message-ID: <3e8ca5c80607150144r6b3990ccq7b61660ebfeb7858@mail.gmail.com> Hi, When testing some 'real world' code using pypy, an inconsistancy with the way __init__ works between lists and dicts. The assumption was made when implementing __init__ for pypy that list.__init__ and dict.__init__ would both wipe the contents of the objects, but it seems that in cpython, this isn't precisely the case. >>> l = [2,3] >>> list.__init__(l) >>> l [] >>> d = {2: 3} >>> dict.__init__(d) >>> d {2: 3} dict.__init__(mydict) does not wipe the keys. list.__init__(mylist) wipes the lists contents. https://codespeak.net/issue/pypy-dev/issue240 Is there a good reason for this behaviour? It has broken my code (a subclass of dict that populates a key before calling the superclasses constructer, in the twisted codebase). -- Stephen Thorne "Give me enough bandwidth and a place to sit and I will move the world." --Jonathan Lange From ashemedai at gmail.com Sat Jul 15 10:57:30 2006 From: ashemedai at gmail.com (Jeroen Ruigrok van der Werven) Date: Sat, 15 Jul 2006 10:57:30 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <44B8A9FA.90209@v.loewis.de> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141343.38424.anthony@interlink.com.au> <3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com> <44B8A9FA.90209@v.loewis.de> Message-ID: <3e1553560607150157o2e521e64m677016161741ffc@mail.gmail.com> On 7/15/06, "Martin v. L?wis" wrote: > Are you aware of http://www.python.org/dev/buildbot/ ? Yes. And it does not seem to be open for all, but then again, any documentation with regard to it seems to be very sparse or hidden so I can very well be wrong here. Ah, hidden away on the wiki: http://wiki.python.org/moin/BuildBot > We are not just talking about buildbots here (which the links you refer > to seem to be); we are talking about buildbots that don't test Python, > but test Python applications. Then I would dare to say you haven't fully investigated the links fully, Bitten, for example, also runs the unit-tests for any target you configure, saves all the outputs and provides quick overviews as well as coverage statistics. So that would fall perfectly into that category. See http://bitten.cmlenz.net/build/0.5.x/762 for an example. > We do know how to run a buildbot. How relevant was this comment in the entire? I am merely trying to help out here pointing out other similar projects when the community participating building issue was raised. -- Jeroen Ruigrok van der Werven From martin at v.loewis.de Sat Jul 15 12:25:19 2006 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Sat, 15 Jul 2006 12:25:19 +0200 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <3e1553560607150157o2e521e64m677016161741ffc@mail.gmail.com> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141343.38424.anthony@interlink.com.au> <3e1553560607150043j603889fco95a993a2ca587419@mail.gmail.com> <44B8A9FA.90209@v.loewis.de> <3e1553560607150157o2e521e64m677016161741ffc@mail.gmail.com> Message-ID: <44B8C28F.1090207@v.loewis.de> Jeroen Ruigrok van der Werven wrote: >> Are you aware of http://www.python.org/dev/buildbot/ ? > > Yes. And it does not seem to be open for all Ah, ok. It indeed isn't open for anonymous participation; the test results are open for all, though. > >> We are not just talking about buildbots here (which the links you refer >> to seem to be); we are talking about buildbots that don't test Python, >> but test Python applications. > > Then I would dare to say you haven't fully investigated the links > fully, Bitten, for example, also runs the unit-tests for any target > you configure I don't understand that comment. Bitten seems to be a software package, similar to buildbot. It doesn't do anything useful until I install and configure it, unlike, say, SourceForge, which is not (just) a soft package, but a running installation of it also. Right? The effort is in installing and configuring it, given that many such packages are already written. Distributed testing frameworks typically support running arbitrary target comments, so Bitten is not really special here (buildbot can also run about anything if you configure it to). >> We do know how to run a buildbot. > > How relevant was this comment in the entire? I am merely trying to > help out here pointing out other similar projects when the community > participating building issue was raised. It would have been helpful if you had stated *why* you think these URLs are relevant in the context of the discussion. To me, it seemed you are pointing to the existence of distributed testing frameworks. I was pointing out that we (at least, I) am aware of the existence of such frameworks, and that we were doing it for quite some time now also, just like Pike. Regards, Martin From guido at python.org Sat Jul 15 16:45:37 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 15 Jul 2006 07:45:37 -0700 Subject: [Python-Dev] Handling of sys.args (Re: User's complaints) In-Reply-To: References: Message-ID: On 7/14/06, Nick Maclaren wrote: > "Guido van Rossum" wrote: > > > > OK, then I propose that we wait to see which things you end up having > > to provide to sandboxed code, rather than trying to analyze it to > > death in abstracto. > > However, the ORIGINAL proposal in this thread (to split off argv[0] > and/or make that and the arguments read-only) is entirely different. > That is purely a matter of convenience, cleanliness of specification > or whatever you call it. I can't imagine any good reason to separate > argv[0] from argv[1:] by a sandbox (either way). Sure. But that proposal has already been discussed (and found little opposition). The thread then dispersed in a much less focused discussion about the true purpose of the sys module.. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Jul 15 16:50:31 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 15 Jul 2006 07:50:31 -0700 Subject: [Python-Dev] list.__init__() vs. dict.__init__() behaviour In-Reply-To: <3e8ca5c80607150144r6b3990ccq7b61660ebfeb7858@mail.gmail.com> References: <3e8ca5c80607150144r6b3990ccq7b61660ebfeb7858@mail.gmail.com> Message-ID: On 7/15/06, Stephen Thorne wrote: > When testing some 'real world' code using pypy, an inconsistancy with > the way __init__ works between lists and dicts. > > The assumption was made when implementing __init__ for pypy that > list.__init__ and dict.__init__ would both wipe the contents of the > objects, but it seems that in cpython, this isn't precisely the case. > > >>> l = [2,3] > >>> list.__init__(l) > >>> l > [] > > >>> d = {2: 3} > >>> dict.__init__(d) > >>> d > {2: 3} > > dict.__init__(mydict) does not wipe the keys. list.__init__(mylist) > wipes the lists contents. I think it's an accident of implementation. I never thought about this from the POV of a subclass. Apparently dict.__init__() shares some code with dict.update(). Or does it do a merge without overwriting existing keys? > https://codespeak.net/issue/pypy-dev/issue240 > > Is there a good reason for this behaviour? It has broken my code (a > subclass of dict that populates a key before calling the superclasses > constructer, in the twisted codebase). I think your code was skating on awfully thin ice by making any assumptions whatsoever about the base class constructor. But I also think that this ought to be specified in the language spec. I'm not sure that consistency between list and dict is important to me, but they probably should both answer to some higher principle which I cannot formulate ATM. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Sat Jul 15 17:00:52 2006 From: aahz at pythoncraft.com (Aahz) Date: Sat, 15 Jul 2006 08:00:52 -0700 Subject: [Python-Dev] Community buildbots In-Reply-To: <44B7286F.7070406@acm.org> References: <2mfyh52wrn.fsf@starship.python.net> <20060713204146.29014.283506497.divmod.quotient.35352@ohm> <05af01c6a6be$54b653b0$d503030a@trilan> <44B7286F.7070406@acm.org> Message-ID: <20060715150052.GA5194@panix.com> On Thu, Jul 13, 2006, Talin wrote: > > Actually - can we make new-style classes the default, but allow a > way to switch to old-style classes if needed? Perhaps a command-line > argument to set the default back to old-style? Nope. Python 3.0 will have new-style classes as the default, but there will probably not be any way to use old-style classes. As has been said before, if you want to make new-style classes the default for any module, just put __metaclass__ = type at the top. Please, just drop this subject. Guido has long since Pronounced. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian W. Kernighan From glyph at divmod.com Sat Jul 15 20:19:52 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Sat, 15 Jul 2006 14:19:52 -0400 Subject: [Python-Dev] Community buildbots In-Reply-To: Message-ID: <20060715181952.29014.1007814129.divmod.quotient.38416@ohm> On Sat, 15 Jul 2006 00:13:35 -0400, Terry Reedy wrote: >Is the following something like what you are suggesting? Something like it, but... >A Python Application Testing (PAT) machine is set up with buildbot and any >needed custom scripts. Sometime after that and after 2.5 is released, when >you have a version of, for instance, Twisted that passes its automated test >suite when run on 2.5, you send it (or a URL) and an email address to PAT. >Other developers do the same. Periodically (once a week?), when PAT is ^ "once per checkin to Python trunk" >free and a new green development version of either the 2.5.x or 2.6 >branches is available, PAT runs the test suites against that version. An >email is sent for any that fail, perhaps accompanied by the concatenation >of the relevant checkin message. Some possible options are to select just >one of the branches for testing, to have more than one stable version being >tested, and to receive pass emails. Sending email also isn't really necessary; I would just like a web page I can look at (and draw the attention of the python core developers to). From glyph at divmod.com Sat Jul 15 20:37:34 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Sat, 15 Jul 2006 14:37:34 -0400 Subject: [Python-Dev] Community buildbots In-Reply-To: <44B8707C.3050207@gmail.com> Message-ID: <20060715183734.29014.989719061.divmod.quotient.38438@ohm> On Sat, 15 Jul 2006 14:35:08 +1000, Nick Coghlan wrote: >glyph at divmod.com wrote: >>A __future__ import would allow these behaviors to be upgraded module-by- >>module. > >No it wouldn't. Yes it would! :) >__future__ works solely on the semantics of different pieces of syntax, >because any syntax changes are purely local to the current module. ... >Changing all the literals in a module to be unicode instances instead of str >instances is merely scratching the surface of the problem - such a module >would still cause severe problems for any non-Unicode aware applications >that expected it to return strings. A module with the given __future__ import could be written to expect that literals are unicode instances instead of str, and encode them appropriately when passing to modules that expect str. This doesn't solve the problem, but unlike -U, you can make fixes which will work persistently without having to treat the type of every string literal as unknown. The obvious way to write code that works under -U and still works in normal Python is to .encode('charmap') every value intended to be an octet, and put 'u' in front of every string intended to be unicode. That would seem to defeat the purpose of changing the default literal type. From jdahlin at async.com.br Sat Jul 15 20:38:04 2006 From: jdahlin at async.com.br (Johan Dahlin) Date: Sat, 15 Jul 2006 15:38:04 -0300 Subject: [Python-Dev] Dynamic module namspaces Message-ID: In an effort to reduce the memory usage used by GTK+ applications written in python I've recently added a feature that allows attributes to be lazy loaded in a module namespace. The gtk python module contains quite a few attributes (around 850) of which many are classes or interfaces (150+) The changes to PyGTK I had to make can not be considered anything but a hack; I had to put a subclass of a ModuleType in sys.modules and override __getattribute__ to be able to get old code which accessed gtk.__dict__ directly to still work (PyModule_GetDict requires that). However, even if I didn't have to use __getattribute__ overriding sys.modules is rather unpleasent and I'm afraid it'll cause problems in the future. My point is that I consider this to be a valid use case, the amount of saved memory is significan, and I could not find another way of doing it and still keep the gtk interface (import gtk; gtk.Button) to still be backwards compatible. What I want to ask, is it possible to have a sanctioned way to implement a dynamic module/namespace in python? For instance, it could be implemented to allow you to replace the __dict__ attribute in a module with a user provided object which implements the dictionary protocol. Johan From glyph at divmod.com Sat Jul 15 20:40:45 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Sat, 15 Jul 2006 14:40:45 -0400 Subject: [Python-Dev] Community buildbots In-Reply-To: <44B8AAAA.70304@v.loewis.de> Message-ID: <20060715184045.29014.1300881616.divmod.quotient.38443@ohm> On Sat, 15 Jul 2006 10:43:22 +0200, "\"Martin v. L?wis\"" wrote: >People can use [-U] to improve the Unicode support in the Python standard >library. When they find that something doesn't work, they can study the >problem, and ponder possible solutions. Then, they can contribute >patches. -U has worked fine for me in the past, I contributed various >patches to make it work better. It hasn't failed for me at all. I guess it makes more sense as a development tool to work on zero-dependency tools like the standard library. Still, -Q has both a __future__ import and a command-line option, why not -U? From rasky at develer.com Sat Jul 15 20:48:20 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 15 Jul 2006 20:48:20 +0200 Subject: [Python-Dev] Dynamic module namspaces References: Message-ID: <07e001c6a83f$3dd96060$d503030a@trilan> Johan Dahlin wrote: > My point is that I consider this to be a valid use case, the amount of > saved memory is significan, and I could not find another way of doing > it and still keep the gtk interface (import gtk; gtk.Button) to still be > backwards compatible. You may want to have a look at SIP/PyQt. They implement the full Qt interface which is rather large, but import time is blazingly fast and memory occupation grows only of 4-5 Mb at import-time. The trick is that methods are "generated" dynamically at their first usage somehow (but dir() and introspection still works...). SIP is free and generic btw, you may want to consider it as a tool. -- Giovanni Bajo From pje at telecommunity.com Sat Jul 15 20:53:06 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 15 Jul 2006 14:53:06 -0400 Subject: [Python-Dev] Dynamic module namspaces In-Reply-To: Message-ID: <5.1.1.6.0.20060715144208.02864158@sparrow.telecommunity.com> At 03:38 PM 7/15/2006 -0300, Johan Dahlin wrote: >In an effort to reduce the memory usage used by GTK+ applications >written in python I've recently added a feature that allows attributes >to be lazy loaded in a module namespace. The gtk python module contains >quite a few attributes (around 850) of which many are classes or >interfaces (150+) > >The changes to PyGTK I had to make can not be considered anything but a >hack; I had to put a subclass of a ModuleType in sys.modules and >override __getattribute__ to be able to get old code which accessed >gtk.__dict__ directly to still work (PyModule_GetDict requires that). Just as a point of reference, the Importing package does something very similar, to support "weak" and "lazy" imports: http://cheeseshop.python.org/pypi/Importing >However, even if I didn't have to use __getattribute__ overriding >sys.modules is rather unpleasent and I'm afraid it'll cause problems in >the future. The things most likely to be problems are tools like pydoc or other inspect-using code that expects modules to be exactly ModuleType and don't use isinstance(). Apart from that, I've been using the technique since the early days of Python 2.2 without encountering any problems until the PEP 302 "reload()" bug came along, but that was fixed in 2.3.5. I haven't seen any other problems since. On the other hand, the Importing package takes a much more conservative approach than you are doing; it simply runs reload() on a module when __getattribute__ is called (after restoring the old version of __getattribute__). So, as soon as you touch a lazily loaded module, it ceases to be particularly special, and it has a real __dict__. It's possible that what you're doing could have more side-effects than what I'm doing. >What I want to ask, is it possible to have a sanctioned way to implement >a dynamic module/namespace in python? That would be nice, but I think that what you and I are doing are probably the One Obvious Ways to do the respective things we're doing. From mal at egenix.com Sat Jul 15 21:15:04 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 15 Jul 2006 21:15:04 +0200 Subject: [Python-Dev] Community buildbots In-Reply-To: <20060715183734.29014.989719061.divmod.quotient.38438@ohm> References: <20060715183734.29014.989719061.divmod.quotient.38438@ohm> Message-ID: <44B93EB8.5020908@egenix.com> glyph at divmod.com wrote: > On Sat, 15 Jul 2006 14:35:08 +1000, Nick Coghlan wrote: >> glyph at divmod.com wrote: >>> A __future__ import would allow these behaviors to be upgraded module-by- >>> module. >> No it wouldn't. > > Yes it would! :) > >> __future__ works solely on the semantics of different pieces of syntax, >> because any syntax changes are purely local to the current module. > ... >> Changing all the literals in a module to be unicode instances instead of str >> instances is merely scratching the surface of the problem - such a module >> would still cause severe problems for any non-Unicode aware applications >> that expected it to return strings. > > A module with the given __future__ import could be written to expect that > literals are unicode instances instead of str, and encode them appropriately > when passing to modules that expect str. This doesn't solve the problem, but > unlike -U, you can make fixes which will work persistently without having to > treat the type of every string literal as unknown. > > The obvious way to write code that works under -U and still works in normal > Python is to .encode('charmap') every value intended to be an octet, and put > 'u' in front of every string intended to be unicode. That would seem to > defeat the purpose of changing the default literal type. Think of it this way: with the u'' literal prefix you can upgrade the behavior on a per-literal basis ;-) Seriously, it really does makes more sense to upgrade to Unicode on a per-literal basis than on a per-module or per- process basis. When upgrading to Unicode you have to carefully consider the path each changed object will take through the code. Note that it also helps setting the default encoding to 'unknown'. That way you disable the coercion of strings to Unicode and all the places where this implicit conversion takes place crop up, allowing you to take proper action (i.e. explicit conversion or changing of the string to Unicode as appropriate). Later on, you can then simply remove the u'' prefix when moving to Py3k. In fact, Py3k could simply ignore the prefix for you (and maybe generate an optional warning) to make the transition easier. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From foom at fuhm.net Sat Jul 15 22:00:35 2006 From: foom at fuhm.net (James Y Knight) Date: Sat, 15 Jul 2006 16:00:35 -0400 Subject: [Python-Dev] Community buildbots In-Reply-To: <44B93EB8.5020908@egenix.com> References: <20060715183734.29014.989719061.divmod.quotient.38438@ohm> <44B93EB8.5020908@egenix.com> Message-ID: On Jul 15, 2006, at 3:15 PM, M.-A. Lemburg wrote: > Note that it also helps setting the default encoding > to 'unknown'. That way you disable the coercion of strings > to Unicode and all the places where this implicit conversion > takes place crop up, allowing you to take proper action (i.e. > explicit conversion or changing of the string to Unicode > as appropriate). I've tried that before to verify no such conversion issues occurred in Twisted, but, as the python stdlib isn't usable like that, it's hard to use it to find bugs in any other libraries. (in particular, the re module is badly broken, some other stuff was too). James From mike at skew.org Sat Jul 15 22:27:15 2006 From: mike at skew.org (Mike Brown) Date: Sat, 15 Jul 2006 14:27:15 -0600 (MDT) Subject: [Python-Dev] urllib.quote and unicode bug resuscitation attempt In-Reply-To: <44B658BB.5090308@ofai.at> Message-ID: <200607152027.k6FKRFGO053680@chilled.skew.org> Stefan Rank wrote: > Well, originally, I would have expected it to return a byte str(ing), I'd assume unicode in, unicode out, and str in, str out, but since it's always going to produce ASCII-range characters, it wouldn't matter. Moot point anyway. > BUT > I am now converted and think it is best to raise a TypeError for > unicode, and leave the encoding decisions to higher level code. > > So I'll repeat the "patch" #1, slightly modified:: > > if isinstance(s, unicode): > raise TypeError("quote expects an encoded byte string as argument") > > Is it safe to assume that code that breaks because of this change was > already broken? Yes. The patch seems fine to me, although maybe if not isinstance(s, str) would be better? From mal at egenix.com Sat Jul 15 23:01:56 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Sat, 15 Jul 2006 23:01:56 +0200 Subject: [Python-Dev] Community buildbots In-Reply-To: References: <20060715183734.29014.989719061.divmod.quotient.38438@ohm> <44B93EB8.5020908@egenix.com> Message-ID: <44B957C4.2010307@egenix.com> James Y Knight wrote: > On Jul 15, 2006, at 3:15 PM, M.-A. Lemburg wrote: >> Note that it also helps setting the default encoding >> to 'unknown'. That way you disable the coercion of strings >> to Unicode and all the places where this implicit conversion >> takes place crop up, allowing you to take proper action (i.e. >> explicit conversion or changing of the string to Unicode >> as appropriate). > > I've tried that before to verify no such conversion issues occurred in > Twisted, but, as the python stdlib isn't usable like that, it's hard to > use it to find bugs in any other libraries. (in particular, the re > module is badly broken, some other stuff was too). True: it breaks a lot of code that was written to work with both strings and Unicode (or does so by accident ;-). The stdlib isn't too well prepared for Unicode yet, so if your code relies a lot on it, then the above may not be the right strategy for you. Perhaps a new ASCII codec that issues warnings for all these cases would help ?! (one that still converts to Unicode assuming ASCII, but issues a warning pointing to the location in the code where the conversion happend) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 15 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From martin at v.loewis.de Sun Jul 16 01:38:04 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 16 Jul 2006 01:38:04 +0200 Subject: [Python-Dev] Community buildbots In-Reply-To: <20060715183734.29014.989719061.divmod.quotient.38438@ohm> References: <20060715183734.29014.989719061.divmod.quotient.38438@ohm> Message-ID: <44B97C5C.6070001@v.loewis.de> glyph at divmod.com wrote: > A module with the given __future__ import could be written to expect that > literals are unicode instances instead of str, and encode them appropriately > when passing to modules that expect str. Such changes might have to be reverted in Python 3, since the module might then expect character strings instead of byte strings, and then might complain when it gets byte strings (which .encode will produce). So declaring that all string literals are Unicode objects might not help in the future, contrary to what the future import suggests. > The obvious way to write code that works under -U and still works in normal > Python is to .encode('charmap') every value intended to be an octet, and put > 'u' in front of every string intended to be unicode. That would seem to > defeat the purpose of changing the default literal type. The not so obvious way is to change the modules/methods receiving these strings to work with either string type if that is reasonable. Regards, Martin From nnorwitz at gmail.com Sun Jul 16 03:35:24 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sat, 15 Jul 2006 18:35:24 -0700 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <44B8A90C.6070309@v.loewis.de> References: <09f901c6a72c$495f2690$12472597@bagio> <20060714112137.GA891@Andrew-iBook2.local> <44B8A90C.6070309@v.loewis.de> Message-ID: On 7/15/06, "Martin v. L?wis" wrote: > Terry Reedy wrote: > > That is the goal, but when I watched the buildbot results last spring, the > > degree of stability (greenness) appeared to vary. Is it possible to tag > > particular versions as a 'green' version, or the 'most recent green > > version' worth playing with? > > Don't get confused by these colors. The tree compiled nearly all of the > time, even if some tests were failing for an extended period of time on > some platforms. Right. It's very rare that the interpreter or stdlib is broken. There are 19 buildbots currently. 7 are not green. 5 of those are offline, most with known (to me and the person that donated the machines) hardware issues ranging from lack of air conditioning, to a bad router, to no power. 1 is cygwin which is running an old version. I just (an hour or so ago) got an account on a cygwin box to help diagnose the status and figure out if anything within Python or the tests are broken. That leaves 1 unexplained failure on a Windows bot. This started happening recently after being down for a while. I haven't had time to investigate. The reason why it was not very green in spring was because that's when we were getting it set up! The master looks like it was installed at the end of 2005/beginning of 2006. It took several months to get rid of many testing issues. Tests that couldn't be run in a particular order, tests that couldn't be run simultaneously (socket), bad compilation of sqlite/bsddb modules (not in python), misconfigured systems, tests verifying something that was system dependent, signed addresses, etc. Of all the problems there were, I only remember a single problem in Python. (There were probably more, I just remember one.) That was in test code (xxmodule or something like that. There were a bunch of problems with ctypes and/or sqlite when they got integrated having to deal with these new platforms. That may be what you are recalling. Right before alpha 2 was a particularly bad time. What we mean by stable is that at any time/any stage of development, if a change goes in that breaks the tests, it is likely to be fixed or reverted within hours. The amount of time the build or tests are broken on the majority of platforms is quite small. It took a while to get to this point due to a bunch of flaky tests, but those have mostly been fixed. The only known problems are mentioned on the wiki. When the buildbots fail, we get mail to python-checkins. Unfortunately, that's mostly spam at this point. I hope to fix that at some point. I also hope to change the main page to give more info in less space. n From tim.peters at gmail.com Sun Jul 16 05:47:50 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 15 Jul 2006 23:47:50 -0400 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <09f901c6a72c$495f2690$12472597@bagio> <20060714112137.GA891@Andrew-iBook2.local> <44B8A90C.6070309@v.loewis.de> Message-ID: <1f7befae0607152047u43993a15ue5180b990f9a530f@mail.gmail.com> [Neal Norwitz] > ... > That leaves 1 unexplained failure on a Windows bot. It wasn't my Windows bot, but I believe test_profile has failed (rarely) on several of the bots, and in the same (or very similar) way. Note that the failure went away on the Windows bot in question the next time the tests were run on it. That's typical of test_profile failures. Unfortunately, because test_profile works by comparing stdout against a canned expected-output file under Lib/test/output/, when we re-run the test in verbose mode at the end, that comparison isn't done, so there's isn't a simple way to know whether the test passed or failed during its second run. I _bet_ it actually passed, but don't know. This problem is obscured because when you run regrtest.py "by hand" with -v, you get this message at the end: """ CAUTION: stdout isn't compared in verbose mode: a test that passes in verbose mode may fail without it. """ which is intended to alert you to the possible problem. But when we re-run failing tests in verbose mode "by magic" (via passing -w), that warning isn't produced. BTW, the best solution to this is to convert all the tests away from regrtest's original "compare stdout against a canned output/TEST_NAME file" scheme. That won't fix test_profile, but would make it less of a puzzle to figure out what's wrong with it. From tjreedy at udel.edu Sun Jul 16 06:29:29 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 16 Jul 2006 00:29:29 -0400 Subject: [Python-Dev] Community buildbots References: <20060715181952.29014.1007814129.divmod.quotient.38416@ohm> Message-ID: wrote in message news:20060715181952.29014.1007814129.divmod.quotient.38416 at ohm... > > > On Sat, 15 Jul 2006 00:13:35 -0400, Terry Reedy wrote: >>Other developers do the same. Periodically (once a week?), when PAT is > ^ > "once per checkin to Python trunk" If the average rate of checkins is equal to or greater than the maximum rate of test runs then that is physically impossible. (Indeed, queueing theory suggests that keeping the queue size small requires that the checkin be more like half the test rate.) This appears to be currently true for the Python trunk buildbot and I would expect running the test of numerous large applications would take even longer, making the test rate even lower. If you want something like twisted run against multiple Python versions a day, I suspect you will have to do so on a system you control. Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sun Jul 16 10:12:11 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 16 Jul 2006 20:12:11 +1200 Subject: [Python-Dev] Problem with super() usage Message-ID: <44B9F4DB.3030004@canterbury.ac.nz> For about the third time in my life, I thought I might have found a use for cooperative super calls, but I've run into another problem with the concept. Consider: class A(object): def m(self): print "A.m" class B(object): def m(self): print "B.m" super(B, self).m() class C(B, A): def m(self): print "C.m" super(C, self).m() >>> c = C() >>> c.m() C.m B.m A.m Okay so far, but... what if I want to use class B on its own, or in combination with other classes that don't have an m() method? >>> b = B() >>> b.m() B.m Traceback (most recent call last): File "", line 1, in ? File "", line 4, in m AttributeError: 'super' object has no attribute 'm' In general, how is one supposed to use super() in a class which may or may not be at the end of the mro with respect to a particular method? The only thing I can think of is to write all my super()-using methods defensively like this: def m(self): ... s = super(C, self) if hasattr(s, 'm'): s.m() which seems awfully tedious. Does the Theory of Cooperative Method Calls have anything to say about this? -- Greg From scott+python-dev at scottdial.com Sun Jul 16 11:42:30 2006 From: scott+python-dev at scottdial.com (Scott Dial) Date: Sun, 16 Jul 2006 05:42:30 -0400 Subject: [Python-Dev] Capabilities / Restricted Execution In-Reply-To: <44B3E99A.4080000@acm.org> References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com> <44B3E029.5070604@scottdial.com> <44B3E99A.4080000@acm.org> Message-ID: <44BA0A06.40101@scottdial.com> Talin wrote: > Scott Dial wrote: >> Phillip J. Eby wrote: >> >>> A function's func_closure contains cell objects that hold the >>> variables. These are readable if you can set the func_closure of some >>> function of your own. If the overall plan includes the ability to restrict >>> func_closure setting (or reading) in a restricted interpreter, then you >>> might be okay. >> >> Except this function (__getattribute__) has been trapped inside of a >> class which does not expose it as an attribute. So, you shouldn't be >> able to get to the func_closure attribute of the __getattribute__ >> function for an instance of the Guard class. I can't come up with a way >> to defeat this protection, at least. If you have a way, then I'd be >> interested to hear it. > > I've thought of several ways to break it already. Some are repairable, > I'm not sure that they all are. > > For example, neither of the following statements blows up: > > print t2.get_name.func_closure[0] > print object.__getattribute__( t2, '__dict__' ) > > Still, its perhaps a useful basis for experimentation. > > -- Talin I quickly poked around it in python and realized that in 2.5 (as opposed to the 2.4 python I was playing in) the cell object exposes cell_contents.. blargh. So, yes, you can defeat the protection because the wrapped instance is exposed. print t2.get_name() t2.get_name.func_closure[0].cell_contents.im_self.name = 'poop' print t2.get_name() Although, your second example with using the object.__getattribute__ doesn't seem to really be an issue. You retrieved the __dict__ for the Guard class which is empty and is something we should not feel concerned about being leaked. Only way I see this as viable is if in "restricted" mode cell_contents was removed from cell objects. -- Scott Dial scott at scottdial.com scodial at indiana.edu From foom at fuhm.net Sun Jul 16 19:20:43 2006 From: foom at fuhm.net (James Y Knight) Date: Sun, 16 Jul 2006 13:20:43 -0400 Subject: [Python-Dev] Capabilities / Restricted Execution In-Reply-To: <44BA0A06.40101@scottdial.com> References: <5.1.1.6.0.20060711105343.01eed878@sparrow.telecommunity.com> <44B3E029.5070604@scottdial.com> <44B3E99A.4080000@acm.org> <44BA0A06.40101@scottdial.com> Message-ID: <36ABCB42-2560-466B-A18B-E831036BCFCF@fuhm.net> On Jul 16, 2006, at 5:42 AM, Scott Dial wrote: > Talin wrote: >> Scott Dial wrote: >>> Phillip J. Eby wrote: >>> >>>> A function's func_closure contains cell objects that hold the >>>> variables. These are readable if you can set the func_closure >>>> of some >>>> function of your own. If the overall plan includes the ability >>>> to restrict >>>> func_closure setting (or reading) in a restricted interpreter, >>>> then you >>>> might be okay. >>> >>> Except this function (__getattribute__) has been trapped inside of a >>> class which does not expose it as an attribute. So, you shouldn't be >>> able to get to the func_closure attribute of the __getattribute__ >>> function for an instance of the Guard class. I can't come up with >>> a way >>> to defeat this protection, at least. If you have a way, then I'd be >>> interested to hear it. >> >> I've thought of several ways to break it already. Some are >> repairable, >> I'm not sure that they all are. >> >> For example, neither of the following statements blows up: >> >> print t2.get_name.func_closure[0] >> print object.__getattribute__( t2, '__dict__' ) >> >> Still, its perhaps a useful basis for experimentation. >> >> -- Talin > > I quickly poked around it in python and realized that in 2.5 (as > opposed > to the 2.4 python I was playing in) the cell object exposes > cell_contents.. blargh. So, yes, you can defeat the protection because > the wrapped instance is exposed. > > print t2.get_name() > t2.get_name.func_closure[0].cell_contents.im_self.name = 'poop' > print t2.get_name() > > Although, your second example with using the object.__getattribute__ > doesn't seem to really be an issue. You retrieved the __dict__ for the > Guard class which is empty and is something we should not feel > concerned > about being leaked. > > Only way I see this as viable is if in "restricted" mode cell_contents > was removed from cell objects. Similarly to how function attributes aren't accessible in restricted mode. In older versions of python, it's always been possible to get the closure variables in non-restricted mode, via mutating func_code... def get_closure_contents(fun): num = len(fun.func_closure) vars = ["x%d" % n for n in range(num)] defines = ' = '.join(vars) + " = None" returns = ', '.join(vars)+',' exec """ def b(): %s def bb(): return %s return bb """ % (defines, returns) old_code = fun.func_code fun.func_code = b().func_code result = fun() fun.func_code = old_code return dict(zip(old_code.co_freevars, result)) def make_secret(x,y): def g(): return x*y return g >>> secret = f(5,7) >>> secret() 35 >>> get_closure_contents(secret) {'y': 7, 'x': 5} From barry at python.org Mon Jul 17 00:18:50 2006 From: barry at python.org (Barry Warsaw) Date: Sun, 16 Jul 2006 18:18:50 -0400 Subject: [Python-Dev] Pronouncement on SF #1520294 sought Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Just as 2.5b2 was being release, I updated SF patch #1520294: https://sourceforge.net/tracker/index.php? func=detail&aid=1520294&group_id=5470&atid=305470 This fixes the pydoc, inspect, and types modules for built-in types like getset and member descriptors. I'm not sure if Georg has had a chance to look over the latest rev of the patch (which is currently assigned to him). I'm basically just looking for an RM pronouncement one way or the other. Can this be added to 2.5 or does it have to wait for 2.6? Thanks! - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRLq7SnEjvBPtnXfVAQLCDAP/UWVSiX9zTnG4qMZJaHHbJDFqxZZnURkx m7OXiCo0H9u4+mY59+FJRos1QA+CldEQ815n+AdxfJQFl0pQqOLaFrTBQtRTzbRU wLV06bUT0OZLOW6mrLxCGNVjAZeua4bcNJBBet8uwj9tx+dftbUnQR921bDjrz9V ZFUjUVXVirs= =7utP -----END PGP SIGNATURE----- From qiuyingbo at gmail.com Mon Jul 17 05:29:37 2006 From: qiuyingbo at gmail.com (Yingbo Qiu) Date: Mon, 17 Jul 2006 11:29:37 +0800 Subject: [Python-Dev] I have submitted a patch that implement IrDA socket support . Message-ID: I have uploaded it at https://sourceforge.net/tracker/download.php?group_id=5470&atid=305470&file_id=184896&aid=1522400. socketmodule.c already have a bluetooth socket, so i think we could put irda socket support in it. Now I can run a simple obex session through IrDA socket layer. my reference include: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnirda/html/irdawp.asp ftp://ftp.ravioli.pasta.cs.uit. no/pub/Ravioli/pyobex/patch-py152-irda3 Will anyone have interest in the patch? From foom at fuhm.net Mon Jul 17 06:33:23 2006 From: foom at fuhm.net (James Y Knight) Date: Mon, 17 Jul 2006 00:33:23 -0400 Subject: [Python-Dev] Dynamic module namspaces In-Reply-To: References: Message-ID: On Jul 15, 2006, at 2:38 PM, Johan Dahlin wrote: > What I want to ask, is it possible to have a sanctioned way to > implement > a dynamic module/namespace in python? > > For instance, it could be implemented to allow you to replace the > __dict__ attribute in a module with a user provided object which > implements the dictionary protocol. I'd like this, as well, although my use case is different: I'd like to be able to deprecate attributes in a module. That is, if I have: foo.py: SOME_CONSTANT = 5 I'd like to be able to do something such that any time anyone accessed foo.SOME_CONSTANT, it'd emit a DeprecationWarning. James From andrew-pythondev at puzzling.org Mon Jul 17 07:15:44 2006 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Mon, 17 Jul 2006 15:15:44 +1000 Subject: [Python-Dev] Dynamic module namspaces In-Reply-To: References: Message-ID: <20060717051544.GE21600@steerpike.home.puzzling.org> On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote: > In an effort to reduce the memory usage used by GTK+ applications > written in python I've recently added a feature that allows attributes > to be lazy loaded in a module namespace. The gtk python module contains > quite a few attributes (around 850) of which many are classes or > interfaces (150+) Have you seen the "demandload" hack that Mercurial uses? You can find it here: http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py You can see an example use of it here: http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py The advantage for an interactive command line tool isn't so much memory consumption as speed. Why waste hundreds of milliseconds importing code that isn't used? There's an experimental branch to use the same demandload code in bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root' from 400ms => 200ms, etc." (according to http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967) Over half the runtime wasted on importing unused code! There's a definite need for a nice solution to this, and it should be included in the standard batteries that come with Python. If we can address related problems at the same time, like emitting deprecation warnings for accessing certain module attributes, then even better! -Andrew. From jcarlson at uci.edu Mon Jul 17 08:52:48 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 16 Jul 2006 23:52:48 -0700 Subject: [Python-Dev] Dynamic module namspaces In-Reply-To: <20060717051544.GE21600@steerpike.home.puzzling.org> References: <20060717051544.GE21600@steerpike.home.puzzling.org> Message-ID: <20060716234046.0CA7.JCARLSON@uci.edu> Andrew Bennetts wrote: > On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote: > > In an effort to reduce the memory usage used by GTK+ applications > > written in python I've recently added a feature that allows attributes > > to be lazy loaded in a module namespace. The gtk python module contains > > quite a few attributes (around 850) of which many are classes or > > interfaces (150+) > > Have you seen the "demandload" hack that Mercurial uses? You can find it here: > http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py > > You can see an example use of it here: > http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py The problem with that particular method is that it requires that you use a string to describe the set of modules you would like to import, rather than a name. In the py3k mailing list I recently described a mechanism (though not implementation) for a somewhat related method to support automatic reloading when a file has changed (for things like web frameworks), but by removing that file change check, you get the equivalent to what you describe, though you don't need to use strings, you can use names... from dynamicreload import DR and used like... DR.os.path.join(...) As long as you use the DR.-prefixed name, you get automatic reloading (if desired) on access. > The advantage for an interactive command line tool isn't so much memory > consumption as speed. Why waste hundreds of milliseconds importing code that > isn't used? There's an experimental branch to use the same demandload code in > bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root' > from 400ms => 200ms, etc." (according to > http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967) > > Over half the runtime wasted on importing unused code! There's a definite need > for a nice solution to this, and it should be included in the standard > batteries that come with Python. Well, just starting up Python without loading any modules can be time consuming; perhaps even dwarfing the few hundred ms saved by dynamic loading. > If we can address related problems at the same time, like emitting deprecation > warnings for accessing certain module attributes, then even better! __deprecated__ = ['name1', ...] Make your dynamic load/reload aware of the __deprecated__ attribute, and you are done. - Josiah From martin at v.loewis.de Mon Jul 17 09:21:56 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 17 Jul 2006 09:21:56 +0200 Subject: [Python-Dev] I have submitted a patch that implement IrDA socket support . In-Reply-To: References: Message-ID: <44BB3A94.1040607@v.loewis.de> Yingbo Qiu wrote: > Will anyone have interest in the patch? This patch comes to late for Python 2.5. It might be considered for 2.6, if anybody has time to review it. Regards, Martin From anthony at interlink.com.au Mon Jul 17 10:20:16 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon, 17 Jul 2006 18:20:16 +1000 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141714.09304.anthony@interlink.com.au> Message-ID: <200607171820.19936.anthony@interlink.com.au> On Friday 14 July 2006 22:45, Jeremy Hylton wrote: > Maybe the basic question is right, but the emphasis needs to be > changed. If we had a rule that said the final release was 90 days > after the last submission that wasn't to fix a regression, we'd ask > "Is this feature important enough to warrant delaying the release > until three months from now?" I'm not sure what I think, but it > doesn't seem like an implausible policy. I really really doubt that this would work. There's always pressure for "just one more feature" - and as the release drags on, this would increase, not decrease. Note that dragging the release process out has it's own costs, as mentioned previously - either the trunk is in some sort of near-frozen state for an extended period, or else we end up in the double-applying-bugfixes state by forking much earlier. This approach would also make it extremely difficult to plan for releases. I know that my free time varies across the course of the year, and I need _some_ sort of plan for when the release will happen so I can make sure I have the time free to spend on it. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From andrew-pythondev at puzzling.org Mon Jul 17 10:57:15 2006 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Mon, 17 Jul 2006 18:57:15 +1000 Subject: [Python-Dev] Dynamic module namspaces In-Reply-To: <20060716234046.0CA7.JCARLSON@uci.edu> References: <20060717051544.GE21600@steerpike.home.puzzling.org> <20060716234046.0CA7.JCARLSON@uci.edu> Message-ID: <20060717085715.GJ21600@steerpike.home.puzzling.org> On Sun, Jul 16, 2006 at 11:52:48PM -0700, Josiah Carlson wrote: > Andrew Bennetts wrote: [...] > > > > Have you seen the "demandload" hack that Mercurial uses? You can find it here: > > http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py > > > > You can see an example use of it here: > > http://selenic.com/repo/hg?f=d276571f2c4b;file=mercurial/commands.py > > The problem with that particular method is that it requires that you use > a string to describe the set of modules you would like to import, rather > than a name. I agree, it's ugly. I'd like there to be a nicer solution. > In the py3k mailing list I recently described a mechanism (though not > implementation) for a somewhat related method to support automatic > reloading when a file has changed (for things like web frameworks), but > by removing that file change check, you get the equivalent to what you > describe, though you don't need to use strings, you can use names... > > from dynamicreload import DR > > and used like... > > DR.os.path.join(...) > > As long as you use the DR.-prefixed name, you get automatic reloading > (if desired) on access. Aside from also being ugly in its own way, a magic prefix like this would add a small performance penalty to places that use it. I believe demandload has the nice feature that once a demandloaded object is accessed, it is replaced with the actual object rather than the demandload gunk, so there's only a one-off performance penalty. > > The advantage for an interactive command line tool isn't so much memory > > consumption as speed. Why waste hundreds of milliseconds importing code that > > isn't used? There's an experimental branch to use the same demandload code in > > bzr, the reported results are "400ms for 'bzr rocks' down to 100ms, 'bzr root' > > from 400ms => 200ms, etc." (according to > > http://permalink.gmane.org/gmane.comp.version-control.bazaar-ng.general/13967) > > > > Over half the runtime wasted on importing unused code! There's a definite need > > for a nice solution to this, and it should be included in the standard > > batteries that come with Python. > > Well, just starting up Python without loading any modules can be time > consuming; perhaps even dwarfing the few hundred ms saved by dynamic > loading. Well, it's only about 10ms on my laptop running Ubuntu (it varies up to 90ms, but I expect that's just noise), and the "-S" switch makes no obvious difference (tested with "python -c ''"). 10ms overhead to start python I can live with. It takes about that long run "svn --version". > > If we can address related problems at the same time, like emitting deprecation > > warnings for accessing certain module attributes, then even better! > > __deprecated__ = ['name1', ...] > > Make your dynamic load/reload aware of the __deprecated__ attribute, and > you are done. I'm fine with that syntax. But regardless of how it's spelled, I'd really like the standard library or interpreter to have support for it, rather than having a non-standard 3rd-party system for it. I want there to "be one-- and preferably only one --obvious way to do it". -Andrew. From bborcic at gmail.com Mon Jul 17 14:51:17 2006 From: bborcic at gmail.com (Boris Borcic) Date: Mon, 17 Jul 2006 14:51:17 +0200 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> Message-ID: Guido van Rossum wrote: > You must be misunderstanding. I don't think so. You appeared to say that the language changes too much because everyone wants different changes - that accumulate. I suggested a mechanism allowing people to see only the changes they want - or none at all - might be devised. Regards, BB > The root problem is that people > (rightly) complain that the language changes too much. And you want to > "fix" this by adding a deep and fundamental change to the language? > What planet are you from? It reminds me of Jini, which was presented > as a new standard to address the problem of too many conflicting > standard. Get it? :-) > > --Guido > > On 7/14/06, Boris Borcic wrote: >> Guido van Rossum wrote: >> ... >>> This is an illustration of the dilemma of maintaining a popular >>> language: Everybody hates change (me too!) but everybody also has one >>> thing that's bothering them so much they absolutely want it to be >>> changed. If you were to implement all those personal pet peeves, you'd >>> get a language that's more different from Python than Python is from >>> Fortran. >>> >>> So where's the middle ground? >> I feel some freedom could be reclaimed with a solution in the spirit of Turing >> equivalence. Or, to take a less grandiose comparison, web style sheets - >> separation of content and presentation. >> >> Suppose the standard required a (possibly empty) style-defining file prefix that >> constrains the python source code in the file, and concurrently defined (mostly) >> reversible and transparent source-to-source transforms that would map any source >> code file to an equivalent source code file with an arbitrary chosen prefix. >> Then users could chose their style of Python and either transform all source >> files they install to their own style, or setup their editor to do it >> back-and-forth for them. The choice of python presentation style would then >> become a private choice. >> >> To illustrate the idea, this already exists in very embryonic form with unicode >> encoding modelines. The current standard allows to imagine a Python editor that >> would permit to set a "local standard encoding modeline" and then present any >> source file as if it had been written while taking maximal profit from the >> chosen encoding. Which may also be simple ascii. >> >> Cheers, BB >> -- >> "C++ is a contradiction in terms" - Lorentz, Einstein, Poincar? >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > From g.brandl at gmx.net Mon Jul 17 15:11:36 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 17 Jul 2006 15:11:36 +0200 Subject: [Python-Dev] Pronouncement on SF #1520294 sought In-Reply-To: References: Message-ID: Barry Warsaw wrote: > Just as 2.5b2 was being release, I updated SF patch #1520294: > > https://sourceforge.net/tracker/index.php? > func=detail&aid=1520294&group_id=5470&atid=305470 > > This fixes the pydoc, inspect, and types modules for built-in types > like getset and member descriptors. I'm not sure if Georg has had a > chance to look over the latest rev of the patch (which is currently > assigned to him). I have now reviewed it and added a comment to the patch. Georg From jdahlin at async.com.br Mon Jul 17 15:41:30 2006 From: jdahlin at async.com.br (Johan Dahlin) Date: Mon, 17 Jul 2006 10:41:30 -0300 Subject: [Python-Dev] Dynamic module namspaces In-Reply-To: <20060717051544.GE21600@steerpike.home.puzzling.org> References: <20060717051544.GE21600@steerpike.home.puzzling.org> Message-ID: <44BB938A.1040400@async.com.br> Andrew Bennetts wrote: > On Sat, Jul 15, 2006 at 03:38:04PM -0300, Johan Dahlin wrote: >> In an effort to reduce the memory usage used by GTK+ applications >> written in python I've recently added a feature that allows attributes >> to be lazy loaded in a module namespace. The gtk python module contains >> quite a few attributes (around 850) of which many are classes or >> interfaces (150+) > > Have you seen the "demandload" hack that Mercurial uses? You can find it here: > http://selenic.com/repo/hg?f=cb4715847a81;file=mercurial/demandload.py It seems quite similar to Philips Importer, it's not completely solving the problem I'm having, since I do something like this: class LazyNamespace(ModuleType): def __init__(self, realmodule, locals): attributes = {} for attr in realmodule._get_lazy_attribute_names(): attributes[attr] = None def __getattribute__(_, name): if name in attributes: value = realmodule._construct_lazy_attribute(name) ... return value There are almost 1000 symbols in the gtk namespace, creating all of them at import time wastes memory and speed, while I've been mainly looking at the memory consumption speed will also benefit. Importing gtk at this point quite fast actually, less than 200ms on my fairly old box. GUI programs does not need to be as responsive as command line applications as hg & bzr, users seems to accept that it takes a second or a few to start up a GUI application. -- Johan Dahlin Async Open Source From andymac at bullseye.apana.org.au Mon Jul 17 12:27:28 2006 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Mon, 17 Jul 2006 21:27:28 +1100 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <200607171820.19936.anthony@interlink.com.au> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607141714.09304.anthony@interlink.com.au> <200607171820.19936.anthony@interlink.com.au> Message-ID: <44BB6610.7080307@bullseye.apana.org.au> Anthony Baxter wrote: > On Friday 14 July 2006 22:45, Jeremy Hylton wrote: >> Maybe the basic question is right, but the emphasis needs to be >> changed. If we had a rule that said the final release was 90 days >> after the last submission that wasn't to fix a regression, we'd ask >> "Is this feature important enough to warrant delaying the release >> until three months from now?" I'm not sure what I think, but it >> doesn't seem like an implausible policy. > > I really really doubt that this would work. There's always pressure > for "just one more feature" - and as the release drags on, this would > increase, not decrease. Note that dragging the release process out > has it's own costs, as mentioned previously - either the trunk is in > some sort of near-frozen state for an extended period, or else we end > up in the double-applying-bugfixes state by forking much earlier. > > This approach would also make it extremely difficult to plan for > releases. I know that my free time varies across the course of the > year, and I need _some_ sort of plan for when the release will happen > so I can make sure I have the time free to spend on it. On the face of it, it seems to me that branching a new major release at the 1st beta would be one way of managing this. The trunk is not frozen for an extended period, and any "features" and bug fixes could probably be backported from the trunk on clearance from the RE with no more pain than is currently endured. One down side would be the possibility of a loss in emphasis in finishing the job on the release to be. I'm sure there are others... ------------------------------------------------------------------------- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac at bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac at pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From jdahlin at async.com.br Mon Jul 17 15:31:44 2006 From: jdahlin at async.com.br (Johan Dahlin) Date: Mon, 17 Jul 2006 10:31:44 -0300 Subject: [Python-Dev] Dynamic module namspaces In-Reply-To: References: Message-ID: <44BB9140.5050602@async.com.br> James Y Knight wrote: > > On Jul 15, 2006, at 2:38 PM, Johan Dahlin wrote: >> What I want to ask, is it possible to have a sanctioned way to implement >> a dynamic module/namespace in python? >> >> For instance, it could be implemented to allow you to replace the >> __dict__ attribute in a module with a user provided object which >> implements the dictionary protocol. > > I'd like this, as well, although my use case is different: I'd like to > be able to deprecate attributes in a module. That is, if I have: > > foo.py: > SOME_CONSTANT = 5 > > I'd like to be able to do something such that any time anyone accessed > foo.SOME_CONSTANT, it'd emit a DeprecationWarning. Agreed, this would be another nice feature to have. I've did something similar a time ago for PyGTK aswell, while less elegant than your proposed solution, it seems that it's working fairly well: DeprecatedConstant can be found here: http://cvs.gnome.org/viewcvs/pygtk/gtk/deprecation.py?view=markup -- Johan Dahlin Async Open Source From jdahlin at async.com.br Mon Jul 17 15:29:22 2006 From: jdahlin at async.com.br (Johan Dahlin) Date: Mon, 17 Jul 2006 10:29:22 -0300 Subject: [Python-Dev] Dynamic module namspaces In-Reply-To: <5.1.1.6.0.20060715144208.02864158@sparrow.telecommunity.com> References: <5.1.1.6.0.20060715144208.02864158@sparrow.telecommunity.com> Message-ID: <44BB90B2.5010907@async.com.br> Phillip J. Eby wrote: > Just as a point of reference, the Importing package does something very > similar, to support "weak" and "lazy" imports: > > http://cheeseshop.python.org/pypi/Importing Interesting, I was not aware of that, thanks for the pointer. Another reason for including this feature in the standard library ;-) > The things most likely to be problems are tools like pydoc or other > inspect-using code that expects modules to be exactly ModuleType and > don't use isinstance(). Apart from that, I've been using the technique > since the early days of Python 2.2 without encountering any problems > until the PEP 302 "reload()" bug came along, but that was fixed in > 2.3.5. I haven't seen any other problems since. I'd argue that pydoc & friends are broken if they assume that a module will always be of ModuleType and not a subclass of it. > On the other hand, the Importing package takes a much more conservative > approach than you are doing; it simply runs reload() on a module when > __getattribute__ is called (after restoring the old version of > __getattribute__). So, as soon as you touch a lazily loaded module, it > ceases to be particularly special, and it has a real __dict__. It's > possible that what you're doing could have more side-effects than what > I'm doing. This is an interesting approach, I thought of using that but I didn't quite manage to find the time to implement it properly. However, for the gtk namespace this won't be enough, since I want to avoid creating all the types when the first one is accessed; when gtk.Button is accessed it, gtk.Window will still not be created. >> What I want to ask, is it possible to have a sanctioned way to implement >> a dynamic module/namespace in python? > > That would be nice, but I think that what you and I are doing are > probably the One Obvious Ways to do the respective things we're doing. I consider __getattribute__ a hack, being able to override __dict__ is less hackish, IMHO. -- Johan Dahlin Async Open Source From anthony at interlink.com.au Mon Jul 17 18:54:23 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue, 18 Jul 2006 02:54:23 +1000 Subject: [Python-Dev] Community buildbots (was Re: User's complaints) In-Reply-To: <44BB6610.7080307@bullseye.apana.org.au> References: <20060713180322.29014.1145514133.divmod.quotient.35181@ohm> <200607171820.19936.anthony@interlink.com.au> <44BB6610.7080307@bullseye.apana.org.au> Message-ID: <200607180254.26666.anthony@interlink.com.au> On Monday 17 July 2006 20:27, Andrew MacIntyre wrote: > On the face of it, it seems to me that branching a new major > release at the 1st beta would be one way of managing this. The > trunk is not frozen for an extended period, and any "features" and > bug fixes could probably be backported from the trunk on clearance > from the RE with no more pain than is currently endured. > > One down side would be the possibility of a loss in emphasis in > finishing the job on the release to be. I'm sure there are > others... The major one is the doubling of the workload for bugfixes. SVN is better than CVS, but it's still not great when it comes to backporting fixes between branches. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From glyph at divmod.com Mon Jul 17 18:59:23 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Mon, 17 Jul 2006 12:59:23 -0400 Subject: [Python-Dev] Dynamic module namspaces In-Reply-To: <44BB90B2.5010907@async.com.br> Message-ID: <20060717165923.29014.1363902968.divmod.quotient.41335@ohm> On Mon, 17 Jul 2006 10:29:22 -0300, Johan Dahlin wrote: >I consider __getattribute__ a hack, being able to override __dict__ is less >hackish, IMHO. Why do you feel one is more "hackish" than the other? In my experience the opposite is true: certain C APIs expect __dict__ to be a "real" dictionary, and if you monkey with it they won't call the overridden functions you expect, whereas things accessing attributes will generally call through all the appropriate Python-level APIs. This makes sense to me for efficiency reasons and for clarity as well; if you're trawling around in a module's __dict__ then you'd better be ready for what you're going to get - *especially* if the module is actually a package. Even in "normal" python code, packages can have names which would be bound if they were imported, but aren't yet. From jcarlson at uci.edu Mon Jul 17 19:10:35 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 17 Jul 2006 10:10:35 -0700 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: References: Message-ID: <20060717092746.0CB0.JCARLSON@uci.edu> Boris Borcic wrote: > Guido van Rossum wrote: > > You must be misunderstanding. > > I don't think so. You appeared to say that the language changes too much because > everyone wants different changes - that accumulate. I suggested a mechanism > allowing people to see only the changes they want - or none at all - might be > devised. Let us imagine that such a system were devised and implemented. Invariably user X and Y would have different sets of changes that they want to use. Presumably, if the features were nontrivial, then they would no longer be able to exchange code because it would have been directed at a different 'version' of Python, whose syntax or semantics were different. What you are suggesting is essentially built-in language fragmentation, which is quite a bad idea. - Josiah From guido at python.org Mon Jul 17 19:25:09 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Jul 2006 10:25:09 -0700 Subject: [Python-Dev] Problem with super() usage In-Reply-To: <44B9F4DB.3030004@canterbury.ac.nz> References: <44B9F4DB.3030004@canterbury.ac.nz> Message-ID: On 7/16/06, Greg Ewing wrote: > For about the third time in my life, I thought I might > have found a use for cooperative super calls, but I've > run into another problem with the concept. > > Consider: > > class A(object): > def m(self): > print "A.m" > > class B(object): > def m(self): > print "B.m" > super(B, self).m() > > class C(B, A): > def m(self): > print "C.m" > super(C, self).m() > > >>> c = C() > >>> c.m() > C.m > B.m > A.m > > Okay so far, but... what if I want to use class B on > its own, or in combination with other classes that don't > have an m() method? > > >>> b = B() > >>> b.m() > B.m > Traceback (most recent call last): > File "", line 1, in ? > File "", line 4, in m > AttributeError: 'super' object has no attribute 'm' > > In general, how is one supposed to use super() in a > class which may or may not be at the end of the mro > with respect to a particular method? One isn't. In the world where cooperative multiple inheritance originated (C++), this would be a static error. You can only use super when you're overriding a method, not when you're defining a new method. To do what you want to do in such a world, you'd have to create a base class with a dummy implementation and inherit from that. You can do that in Python, too. > The only thing I can think of is to write all my > super()-using methods defensively like this: > > def m(self): > ... > s = super(C, self) > if hasattr(s, 'm'): > s.m() > > which seems awfully tedious. Definitely don't that. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From arigo at tunes.org Mon Jul 17 20:22:35 2006 From: arigo at tunes.org (Armin Rigo) Date: Mon, 17 Jul 2006 20:22:35 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <3e1553560607130052m66899441t50065786495928fe@mail.gmail.com> <44B60C0F.7060405@canterbury.ac.nz> <928173CD-DD5D-4B10-9CC4-ED63CC90933A@redivi.com> <3e1553560607130502o253a8359mf327964ee5d8ece8@mail.gmail.com> Message-ID: <20060717182235.GA20517@code0.codespeak.net> Hi Jeroen, On Thu, Jul 13, 2006 at 02:02:22PM +0200, Jeroen Ruigrok van der Werven wrote: > He doesn't specifically need the builtin types to be extendable. It's > just nice to be able to define a single class in multiple modules. There are various simple ways to do this; the one I'm using from time to time is the "extendabletype" metaclass from: http://codespeak.net/svn/pypy/dist/pypy/annotation/pairtype.py Example: class A: __metaclass__ = extendabletype def f(...): ... Somewhere else: class __extend__(A): def g(...): ... FWIW the above 30-lines file also contains a fast double-dispatch multimethod implementation :-) A bientot, Armin From arigo at tunes.org Mon Jul 17 20:25:21 2006 From: arigo at tunes.org (Armin Rigo) Date: Mon, 17 Jul 2006 20:25:21 +0200 Subject: [Python-Dev] User's complaints In-Reply-To: <6713F515-C684-486E-8FFF-EAC60C83E069@redivi.com> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <6713F515-C684-486E-8FFF-EAC60C83E069@redivi.com> Message-ID: <20060717182521.GB20517@code0.codespeak.net> Hi Bob, On Thu, Jul 13, 2006 at 12:58:08AM -0700, Bob Ippolito wrote: > > @main > > def whatever(): > > ... > > It would probably need to be called something else, because main is > often the name of the main function... Ah, but there is theoretically no name clash here :-) @main # <- from the built-ins def main(): # <- and only then set the global ... Just-making-a-stupid-point-and-not-endorsing-the-feature-ly yours, Armin From bob at redivi.com Mon Jul 17 21:39:08 2006 From: bob at redivi.com (Bob Ippolito) Date: Mon, 17 Jul 2006 12:39:08 -0700 Subject: [Python-Dev] User's complaints In-Reply-To: <20060717182521.GB20517@code0.codespeak.net> References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> <4c45c1530607130037r1c14e3f6u972232bdfa481534@mail.gmail.com> <6713F515-C684-486E-8FFF-EAC60C83E069@redivi.com> <20060717182521.GB20517@code0.codespeak.net> Message-ID: On Jul 17, 2006, at 11:25 AM, Armin Rigo wrote: > Hi Bob, > > On Thu, Jul 13, 2006 at 12:58:08AM -0700, Bob Ippolito wrote: >>> @main >>> def whatever(): >>> ... >> >> It would probably need to be called something else, because main is >> often the name of the main function... > > Ah, but there is theoretically no name clash here :-) > > @main # <- from the built-ins > def main(): # <- and only then set the global > ... > > > Just-making-a-stupid-point-and-not-endorsing-the-feature-ly yours, Of course it *works*, but it's still overriding a built-in... Who knows when assignment to main will become a SyntaxError like None ;) -bob From misa at redhat.com Mon Jul 17 21:39:55 2006 From: misa at redhat.com (Mihai Ibanescu) Date: Mon, 17 Jul 2006 15:39:55 -0400 Subject: [Python-Dev] logging module broken because of locale Message-ID: <20060717193955.GK3699@abulafia.devel.redhat.com> Hi, This is reported on sourceforge: http://sourceforge.net/tracker/index.php?func=detail&aid=1524081&group_id=5470&atid=105470 I am willing to try and patch the problem, but I'd like to discuss my ideas first. The basic problem is that, in some locale, "INFO".lower() != "info". So, initializing a dictionary with keys that appear to be lower-case in English is unsafe for other locale. Now, the other problem is, people can choose to initialize the locale any time they want, and can choose to change it too. My initial approach was to create the dictionary with strings to which lower() was explicitly called. But, since priority_names is class-scoped, it gets evaluated at module import time, which is way too early in the game (most likely the locale will be set afterwards). Any thoughts on the proper way to approach this? Thanks, Misa From bborcic at gmail.com Mon Jul 17 21:43:23 2006 From: bborcic at gmail.com (Boris Borcic) Date: Mon, 17 Jul 2006 21:43:23 +0200 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: <20060717092746.0CB0.JCARLSON@uci.edu> References: <20060717092746.0CB0.JCARLSON@uci.edu> Message-ID: Josiah Carlson wrote: > Boris Borcic wrote: >> Guido van Rossum wrote: >>> You must be misunderstanding. >> I don't think so. You appeared to say that the language changes too much because >> everyone wants different changes - that accumulate. I suggested a mechanism >> allowing people to see only the changes they want - or none at all - might be >> devised. > > Let us imagine that such a system were devised and implemented. A minimal requirement to such usage of the first person plural is some effort to make sure "we" talk about the same thing. > Invariably user X and Y would have different sets of changes that they > want to use. Presumably, if the features were nontrivial, then they > would no longer be able to exchange code because it would have been > directed at a different 'version' of Python, whose syntax or semantics > were different. Of course, and that's why in my initial post I was talking of transparent reversible transforms and central control of "styles" through the standard. Means not to fall into the trap you describe. Or else I would have asked for macros ! Are you implying that /no/ measure of language variability can be dealt with by such means as standards-controlled reversible transforms ? I guess not. I am just saying : (1) be aware that "reversible source to source transforms" means that language differences can *EFFECTIVELY* be made invisible - turned to purely private choices (in the sense of "because there exists perfect automatic two-way translation between equivalent languages", *NOT* "because we don't exchange code written in incompatible languages"). (2) of course that frame limits language differences, but nevertheless it relaxes that limit relative to the current situation of "no differences", and please don't write it off without any effort to evaluate its possibilities. Regards, BB -- 666 ? - 666 ~ .666 ~ 1 - 1/3 ~ 2/3 ~ tertium non datur ~ the excluded middle ~ "either with us, or against us" From fuzzyman at voidspace.org.uk Mon Jul 17 22:06:35 2006 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Mon, 17 Jul 2006 21:06:35 +0100 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: References: <20060717092746.0CB0.JCARLSON@uci.edu> Message-ID: <44BBEDCB.4070607@voidspace.org.uk> Boris Borcic wrote: >> Boris Borcic wrote: >> >> [snip..] > I am just saying : > > (1) be aware that "reversible source to source transforms" means that language > differences can *EFFECTIVELY* be made invisible - turned to purely private > choices (in the sense of "because there exists perfect automatic two-way > translation between equivalent languages", *NOT* "because we don't exchange code > written in incompatible languages"). > > (2) of course that frame limits language differences, but nevertheless it > relaxes that limit relative to the current situation of "no differences", and > please don't write it off without any effort to evaluate its possibilities. > > Something similar to your suggestions are already possible through third part extensions like EasyExtend or Logix. (Effectively implementing a separate compiler.) Readability (one way to do it...) are a central part of the Python philosophy. This philosophy means that it is important that code written by one developer should be readable by others. Your suggestions have little to no chance of making it into the core Python distribution. All the best, Michael Foord http://www.voidspace.org.uk/python/index.shtml > Regards, BB > -- > 666 ? - 666 ~ .666 ~ 1 - 1/3 ~ 2/3 ~ tertium non datur ~ the excluded middle > ~ "either with us, or against us" > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From jcarlson at uci.edu Mon Jul 17 22:10:48 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 17 Jul 2006 13:10:48 -0700 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: References: <20060717092746.0CB0.JCARLSON@uci.edu> Message-ID: <20060717125959.0CBF.JCARLSON@uci.edu> Boris Borcic wrote: > Josiah Carlson wrote: > > Invariably user X and Y would have different sets of changes that they > > want to use. Presumably, if the features were nontrivial, then they > > would no longer be able to exchange code because it would have been > > directed at a different 'version' of Python, whose syntax or semantics > > were different. > > Of course, and that's why in my initial post I was talking of transparent > reversible transforms and central control of "styles" through the standard. > Means not to fall into the trap you describe. Or else I would have asked for > macros ! Are you implying that /no/ measure of language variability can be dealt > with by such means as standards-controlled reversible transforms ? I guess not. Regardless of the existance of reversable transforms, a user's ability to understand and/or maintain code is dependant on the syntax and semantics of the language. In allowing different language variants, one is changing the user-understood meaning of a block of code, which necessarily increses the burden of programming and maintenance. - Josiah From t-bruch at microsoft.com Mon Jul 17 22:14:36 2006 From: t-bruch at microsoft.com (Bruce Christensen) Date: Mon, 17 Jul 2006 13:14:36 -0700 Subject: [Python-Dev] Pickling objects that return string from reduce Message-ID: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com> I'm still plugging away on IronPython's cPickle, and I've stumbled across another ambiguity in the pickle docs. From http://docs.python.org/lib/node69.html: "If a string is returned, it names a global variable whose contents are pickled as normal. The string returned by __reduce__ should be the object's local name relative to its module; the pickle module searches the module namespace to determine the object's module." What exactly does that last clause mean? Must the object have a __module__ attribute? What if it doesn't? What if the module exists but isn't currently imported? Is something like the following close? # Call copy_reg-registered func, obj.__reduce_ex__, or obj.__reduce__ result = call_reduce(obj) if type(result) == tuple: ... (do appropriate things here) elif isinstance(result, basestring): name = result module = "" try: module = obj.__module__ found_obj = getattr(sys.modules[module], name) except AttributeError, KeyError: raise PicklingError( "Can't pickle %r: it's not found as %s.%s" % (obj, module, name) ) if found_obj is not obj: raise PicklingError( "Can't pickle %r: it's not the same object as %s.%s" % (obj, module, name) ) emit("c%s\n%s\n" % module, name) Thanks, --Bruce From westymatt at gmail.com Mon Jul 17 22:25:29 2006 From: westymatt at gmail.com (matt westerburg) Date: Mon, 17 Jul 2006 16:25:29 -0400 Subject: [Python-Dev] new guy Message-ID: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com> Hi my name is Matt Westerburg, I am a student and have only recently gotten into Python. But have fallen in love with the language thus far. Fantastic language and thank you very much for making it what it is today. I am looking into getting into working on Python. Still need sometime working with the language, but I am interested in both the standard library and the interpreter. Can anyone recommend a good starting place? thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060717/9b4d9be5/attachment.htm From brett at python.org Mon Jul 17 22:53:34 2006 From: brett at python.org (Brett Cannon) Date: Mon, 17 Jul 2006 13:53:34 -0700 Subject: [Python-Dev] new guy In-Reply-To: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com> References: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com> Message-ID: On 7/17/06, matt westerburg wrote: > > Hi my name is Matt Westerburg, I am a student and have only recently > gotten into Python. But have fallen in love with the language thus far. > Fantastic language and thank you very much for making it what it is today. > I am looking into getting into working on Python. Still need sometime > working with the language, but I am interested in both the standard library > and the interpreter. Can anyone recommend a good starting place? > First, welcome! Second, in terms of a good starting point, you can read http://www.python.org/dev/intro for an introduction on how things work around here (roughly) and what it takes to get something in. Really the best way to get introduced is to read this mailing list and to dive into some source (reviewing a patch can be a good way to go about this). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060717/1b9467d2/attachment.html From martin at v.loewis.de Mon Jul 17 23:04:27 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 17 Jul 2006 23:04:27 +0200 Subject: [Python-Dev] Pickling objects that return string from reduce In-Reply-To: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com> References: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com> Message-ID: <44BBFB5B.2090806@v.loewis.de> Bruce Christensen wrote: > Is something like the following close? Close, yes. > if type(result) == tuple: > ... (do appropriate things here) > elif isinstance(result, basestring): > name = result > module = "" > try: > module = obj.__module__ > found_obj = getattr(sys.modules[module], name) > except AttributeError, KeyError: > raise PicklingError( > "Can't pickle %r: it's not found as %s.%s" > % (obj, module, name) > ) If obj has no __module__ attribute (or if it is None), pickle (didn't check cPickle) also does for n, module in sys.module.items(): if "module-ignored": continue if getattr(module, result, None) is obj: break # use n as module name If obj does have a __module__ attribute, it uses __import__ to import the module, just to make sure it gets into sys.modules. Otherwise, it looks correct. Regards, Martin From martin at v.loewis.de Mon Jul 17 23:10:00 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 17 Jul 2006 23:10:00 +0200 Subject: [Python-Dev] new guy In-Reply-To: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com> References: <4b490f2f0607171325l4aec5bf5v66e44682f6ba2773@mail.gmail.com> Message-ID: <44BBFCA8.4080503@v.loewis.de> matt westerburg wrote: > Hi my name is Matt Westerburg, I am a student and have only recently > gotten into Python. But have fallen in love with the language thus > far. Fantastic language and thank you very much for making it what it > is today. I am looking into getting into working on Python. Still need > sometime working with the language, but I am interested in both the > standard library and the interpreter. Can anyone recommend a good > starting place? As Brett says, reviewing bugs and patches would be most appreciated. Review a patch, and either tell the submitter how the patch should be improved, or tell us that you think the patch is ready to be applied (make sure the bug has documentation and/or test suite changes as necessary). Review a bug, and determine that it is a) reproducable b) constitutes a true problem (rather than a misunderstanding on the submitter's side); if both hold, try to come up with a patch. If it is not reproducable, tell the submitter and ask for more information. If it is not the true problem, tell us that you think it deserves to be closed. If you are uncertain which bug or patch to review, select one that a) is currently unassigned b) about a year old c) in an area of your personal interests Property a) and b) combined should guarantee that nobody beats you in fixing the problem while you are researching it. Regards, Martin From greg.ewing at canterbury.ac.nz Tue Jul 18 09:46:40 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Jul 2006 19:46:40 +1200 Subject: [Python-Dev] Problem with super() usage In-Reply-To: References: <44B9F4DB.3030004@canterbury.ac.nz> Message-ID: <44BC91E0.2040804@canterbury.ac.nz> Guido van Rossum wrote: > In the world where cooperative multiple inheritance > originated (C++), this would be a static error. I wasn't aware that C++ had anything resembling super(). Is it a recent addition to the language? -- Greg From bborcic at gmail.com Tue Jul 18 13:36:41 2006 From: bborcic at gmail.com (Boris Borcic) Date: Tue, 18 Jul 2006 13:36:41 +0200 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: <20060717125959.0CBF.JCARLSON@uci.edu> References: <20060717092746.0CB0.JCARLSON@uci.edu> <20060717125959.0CBF.JCARLSON@uci.edu> Message-ID: Josiah Carlson wrote: > Boris Borcic wrote: >> Of course, and that's why in my initial post I was talking of transparent >> reversible transforms and central control of "styles" through the standard. >> Means not to fall into the trap you describe. Or else I would have asked for >> macros ! Are you implying that /no/ measure of language variability can be dealt >> with by such means as standards-controlled reversible transforms ? I guess not. > > Regardless of the existance of reversable transforms, a user's ability > to understand and/or maintain code is dependant on the syntax and > semantics of the language. If you have an effective isomorphism, that's irrelevant. Everybody works with the language she understands. Ever tried a slide rule ? > In allowing different language variants, one > is changing the user-understood meaning of a block of code, which > necessarily increses the burden of programming and maintenance. Allowing different language variants connected by reversible transforms means one need not change any user's understood meaning of any block of code. The user stipulates the language variant she likes and the system translates back-and-forth from/to distinct variants other users might prefer. - BB -- 666 ? - 666 ~ .666 ~ 2/3 ~ 1-1/3 ~ tertium non datur ~ the excluded middle ~ "either with us, or against us" From misa at redhat.com Tue Jul 18 14:40:19 2006 From: misa at redhat.com (Mihai Ibanescu) Date: Tue, 18 Jul 2006 08:40:19 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <20060717193955.GK3699@abulafia.devel.redhat.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> Message-ID: <20060718124019.GB16294@abulafia.devel.redhat.com> On Mon, Jul 17, 2006 at 03:39:55PM -0400, Mihai Ibanescu wrote: > Hi, > > This is reported on sourceforge: > > http://sourceforge.net/tracker/index.php?func=detail&aid=1524081&group_id=5470&atid=105470 > > I am willing to try and patch the problem, but I'd like to discuss my ideas > first. > > The basic problem is that, in some locale, "INFO".lower() != "info". So, > initializing a dictionary with keys that appear to be lower-case in English > is unsafe for other locale. > > Now, the other problem is, people can choose to initialize the locale any time > they want, and can choose to change it too. My initial approach was to create > the dictionary with strings to which lower() was explicitly called. > > But, since priority_names is class-scoped, it gets evaluated at module import > time, which is way too early in the game (most likely the locale will be set > afterwards). > > Any thoughts on the proper way to approach this? To follow up on my own email: it looks like, even though in some locale "INFO".lower() != "info" u"INFO".lower() == "info" (at least in the Turkish locale). Is that guaranteed, at least for now (for the current versions of python)? Thanks, Misa From aahz at pythoncraft.com Tue Jul 18 14:50:45 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 18 Jul 2006 05:50:45 -0700 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: References: <20060717092746.0CB0.JCARLSON@uci.edu> <20060717125959.0CBF.JCARLSON@uci.edu> Message-ID: <20060718125045.GA17271@panix.com> Please. Just end this discussion. It ain't gonna happen. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian W. Kernighan From scott+python-dev at scottdial.com Tue Jul 18 15:10:11 2006 From: scott+python-dev at scottdial.com (Scott Dial) Date: Tue, 18 Jul 2006 09:10:11 -0400 Subject: [Python-Dev] Problem with super() usage In-Reply-To: <44BC91E0.2040804@canterbury.ac.nz> References: <44B9F4DB.3030004@canterbury.ac.nz> <44BC91E0.2040804@canterbury.ac.nz> Message-ID: <44BCDDB3.6090607@scottdial.com> Greg Ewing wrote: > Guido van Rossum wrote: >> In the world where cooperative multiple inheritance >> originated (C++), this would be a static error. > > I wasn't aware that C++ had anything resembling super(). > Is it a recent addition to the language? > It is much more explicit, but you call the function from the superclass's namespace: class B : public A { public: void m(void) { A::m(); // The call to my super }; }; C++ has no concept of MRO, so "super()" would be completely ambiguous. In fact, if you try to replicate your code in C++ using a generic M class (which defines a dummy m method), you'll get such an error from your compiler: "`M' is an ambiguous base of `C'" This is very much a dynamic language quirk that you can call out to a function that may or may not exist, and we should avoid comparing it to other languages which don't allow it. I agree with Guido that in python, the reasonable fix is to have a superclass which defines an empty method. -- Scott Dial scott at scottdial.com scodial at indiana.edu From exarkun at divmod.com Tue Jul 18 15:24:57 2006 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Tue, 18 Jul 2006 09:24:57 -0400 Subject: [Python-Dev] Problem with super() usage In-Reply-To: <44BCDDB3.6090607@scottdial.com> Message-ID: <20060718132457.29014.276758938.divmod.quotient.42664@ohm> On Tue, 18 Jul 2006 09:10:11 -0400, Scott Dial wrote: >Greg Ewing wrote: >> Guido van Rossum wrote: >>> In the world where cooperative multiple inheritance >>> originated (C++), this would be a static error. >> >> I wasn't aware that C++ had anything resembling super(). >> Is it a recent addition to the language? >> > >It is much more explicit, but you call the function from the >superclass's namespace: > > [snip - something totally unlike super] > >C++ has no concept of MRO, so "super()" would be completely ambiguous. I think this was Greg's point. Talking about C++ and super() is nonsensical. Jean-Paul From glyph at divmod.com Tue Jul 18 17:59:39 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Tue, 18 Jul 2006 11:59:39 -0400 Subject: [Python-Dev] Problem with super() usage In-Reply-To: <20060718132457.29014.276758938.divmod.quotient.42664@ohm> Message-ID: <20060718155939.29014.1269904967.divmod.quotient.42845@ohm> On Tue, 18 Jul 2006 09:24:57 -0400, Jean-Paul Calderone wrote: >On Tue, 18 Jul 2006 09:10:11 -0400, Scott Dial wrote: >>Greg Ewing wrote: >>> Guido van Rossum wrote: >>>> In the world where cooperative multiple inheritance >>>> originated (C++), this would be a static error. >>> I wasn't aware that C++ had anything resembling super(). >>> Is it a recent addition to the language? >>C++ has no concept of MRO, so "super()" would be completely ambiguous. >I think this was Greg's point. Talking about C++ and super() is >nonsensical. C++ originally specified multiple inheritance, but it wasn't "cooperative" in the sense that super is. In Lisp, though, where cooperative method dispatch originated, call-next-method does basically the same thing in the case where there's no next method: it calls "no-next-method" which signals a generic error. http://www.lisp.org/HyperSpec/Body/locfun_call-next-method.html However, you can write methods for no-next-method, so you can override that behavior as appropriate. In Python you might achieve a similar effect using a hack like the one Greg suggested, but in a slightly more systematic way; using Python's regular inheritance-based method ordering, of course, not bothering with multimethods. Stand-alone it looks like an awful hack, but with a bit of scaffolding I think it looks nice enough; at least, it looks like the Lisp solution, which while potentially ugly, is complete :). This is just implemented as a function for brevity; you could obviously use a proxy object with all the features of 'super', including optional self, method getters, etc. For cooperative classes you could implement noNextMethod to always be OK, or to provide an appropriate "null" value for a type map of a method's indicated return value ('' for str, 0 for int, None for object, etc). # ---cut here--- def callNextMethod(cls, self, methodName, *a, **k): sup = super(cls, self) method = getattr(sup, methodName, None) if method is not None: return method(*a, **k) else: return self.noNextMethod(methodName, *a, **k) class NextMethodHelper(object): def noNextMethod(self, methodName, *a, **k): return getattr(self, "noNext_"+methodName)(*a, **k) class A(object): def m(self): print "A.m" class B(NextMethodHelper): def m(self): print "B.m" return callNextMethod(B, self, "m") def noNext_m(self): # it's ok not to have an 'm'! print "No next M, but that's OK!" return None class C(B, A): def m(self): print "C.m" return callNextMethod(C, self, "m") #>>> c = C() #>>> c.m() #C.m #B.m #A.m #>>> b = B() #>>> b.m() #B.m #No next M, but that's OK! From tjreedy at udel.edu Tue Jul 18 18:19:08 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Jul 2006 12:19:08 -0400 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints References: <20060717092746.0CB0.JCARLSON@uci.edu> <20060717125959.0CBF.JCARLSON@uci.edu> Message-ID: "Boris Borcic" wrote in message news:e9ih56$qmu$1 at sea.gmane.org... > Allowing different language variants connected by reversible transforms > means > one need not change any user's understood meaning of any block of code. > The user > stipulates the language variant she likes and the system translates > back-and-forth from/to distinct variants other users might prefer. Editing systems like this are and should remain 3rd-party add-ons. This discussion belongs on the general python list (clp) or on the discussion list of current projects in this area. tjr From metawilm at gmail.com Tue Jul 18 18:22:11 2006 From: metawilm at gmail.com (Willem Broekema) Date: Tue, 18 Jul 2006 18:22:11 +0200 Subject: [Python-Dev] Problem with super() usage In-Reply-To: <20060718155939.29014.1269904967.divmod.quotient.42845@ohm> References: <20060718132457.29014.276758938.divmod.quotient.42664@ohm> <20060718155939.29014.1269904967.divmod.quotient.42845@ohm> Message-ID: On 7/18/06, glyph at divmod.com wrote: > C++ originally specified multiple inheritance, but it wasn't "cooperative" in > the sense that super is. In Lisp, though, where cooperative method dispatch > originated, call-next-method does basically the same thing in the case where > there's no next method: it calls "no-next-method" which signals a generic > error. Don't forget Lisp's "next-method-p", which tests ("-p" for "predicate") if there is any next method to call. Highly elegant, I'd say. > http://www.lisp.org/HyperSpec/Body/locfun_call-next-method.html http://www.lisp.org/HyperSpec/Body/locfun_next-method-p.html - Willem From guido at python.org Tue Jul 18 18:58:54 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Jul 2006 09:58:54 -0700 Subject: [Python-Dev] Problem with super() usage In-Reply-To: <44BC91E0.2040804@canterbury.ac.nz> References: <44B9F4DB.3030004@canterbury.ac.nz> <44BC91E0.2040804@canterbury.ac.nz> Message-ID: On 7/18/06, Greg Ewing wrote: > Guido van Rossum wrote: > > In the world where cooperative multiple inheritance > > originated (C++), this would be a static error. > > I wasn't aware that C++ had anything resembling super(). > Is it a recent addition to the language? I don't know about pure C++, but the C++ variant (or was it a library?) that originated the cooperative multitasking ideas does have a super call. And it is strongly typed as I explained. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Jul 18 19:02:20 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Jul 2006 10:02:20 -0700 Subject: [Python-Dev] Problem with super() usage In-Reply-To: <20060718132457.29014.276758938.divmod.quotient.42664@ohm> References: <44BCDDB3.6090607@scottdial.com> <20060718132457.29014.276758938.divmod.quotient.42664@ohm> Message-ID: On 7/18/06, Jean-Paul Calderone wrote: > I think this was Greg's point. Talking about C++ and super() is > nonsensical. If you're talking pure C++, yes. But I was talking about programming system built on top of C++ implementing cooperating multitasking. As I am fond of repeating, if the ideas originally came from Lisp, I don't know -- I got them from this book: Putting Metaclasses to Work: A New Dimension in Object-Oriented Programming, by Ira R. Forman and Scott H. Danforth. Addison-Wesley, 1999, ISBN 0-201-43305-2. It uses C++ (or perhaps a language very much like it), not Lisp. And it has super. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Jul 18 19:04:32 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Jul 2006 10:04:32 -0700 Subject: [Python-Dev] Python Style Sheets ? Re: User's complaints In-Reply-To: References: <3e1553560607122243q754dd71dq2e6d95cdf11e89dd@mail.gmail.com> Message-ID: On 7/17/06, Boris Borcic wrote: > Guido van Rossum wrote: > > You must be misunderstanding. > > I don't think so. You appeared to say that the language changes too much because > everyone wants different changes - that accumulate. I suggested a mechanism > allowing people to see only the changes they want - or none at all - might be > devised. Oh, but don't fool yourself into thinking that that's not a language change -- and an extremely drastic one at that. Nobody will be able to read other people's source code any more without first applying the conversion tool -- which is kind of painful when you're confronted with a code snippet in email or a book, for example. Let me rephrase that. I don't believe your proposal solves that problem, *and* I don't think it's a good idea for other reasons. Am I clear now? :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Jul 18 19:19:54 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Jul 2006 10:19:54 -0700 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <20060718124019.GB16294@abulafia.devel.redhat.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> Message-ID: Alternatively, does "info".upper() == "INFO" everywhere? On 7/18/06, Mihai Ibanescu wrote: > On Mon, Jul 17, 2006 at 03:39:55PM -0400, Mihai Ibanescu wrote: > > Hi, > > > > This is reported on sourceforge: > > > > http://sourceforge.net/tracker/index.php?func=detail&aid=1524081&group_id=5470&atid=105470 > > > > I am willing to try and patch the problem, but I'd like to discuss my ideas > > first. > > > > The basic problem is that, in some locale, "INFO".lower() != "info". So, > > initializing a dictionary with keys that appear to be lower-case in English > > is unsafe for other locale. > > > > Now, the other problem is, people can choose to initialize the locale any time > > they want, and can choose to change it too. My initial approach was to create > > the dictionary with strings to which lower() was explicitly called. > > > > But, since priority_names is class-scoped, it gets evaluated at module import > > time, which is way too early in the game (most likely the locale will be set > > afterwards). > > > > Any thoughts on the proper way to approach this? > > To follow up on my own email: it looks like, even though in some locale > "INFO".lower() != "info" > > u"INFO".lower() == "info" (at least in the Turkish locale). > > Is that guaranteed, at least for now (for the current versions of python)? > > Thanks, > Misa > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From misa at redhat.com Tue Jul 18 19:45:41 2006 From: misa at redhat.com (Mihai Ibanescu) Date: Tue, 18 Jul 2006 13:45:41 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> Message-ID: <20060718174541.GI27181@abulafia.devel.redhat.com> On Tue, Jul 18, 2006 at 10:19:54AM -0700, Guido van Rossum wrote: > Alternatively, does "info".upper() == "INFO" everywhere? Not in the Turkish locale :-( # begin /tmp/foo.py import locale locale.setlocale(locale.LC_ALL, '') print "info".upper() print "info".upper() == "INFO" # end /tmp/foo.py LANG=tr_TR.UTF-8 python /tmp/foo.py iNFO False Thanks, Misa From guido at python.org Tue Jul 18 19:53:23 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Jul 2006 10:53:23 -0700 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <20060718174541.GI27181@abulafia.devel.redhat.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <20060718174541.GI27181@abulafia.devel.redhat.com> Message-ID: And u"info".upper()? On 7/18/06, Mihai Ibanescu wrote: > On Tue, Jul 18, 2006 at 10:19:54AM -0700, Guido van Rossum wrote: > > Alternatively, does "info".upper() == "INFO" everywhere? > > Not in the Turkish locale :-( > > # begin /tmp/foo.py > import locale > > locale.setlocale(locale.LC_ALL, '') > > print "info".upper() > print "info".upper() == "INFO" > # end /tmp/foo.py > > LANG=tr_TR.UTF-8 python /tmp/foo.py > iNFO > False > > Thanks, > Misa > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Tue Jul 18 19:54:28 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 18 Jul 2006 19:54:28 +0200 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <20060718124019.GB16294@abulafia.devel.redhat.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> Message-ID: <44BD2054.4020705@v.loewis.de> Mihai Ibanescu wrote: > To follow up on my own email: it looks like, even though in some locale > "INFO".lower() != "info" > > u"INFO".lower() == "info" (at least in the Turkish locale). > > Is that guaranteed, at least for now (for the current versions of python)? It's guaranteed for now; unicode.lower is not locale-aware. Regards, Martin From foom at fuhm.net Tue Jul 18 20:01:09 2006 From: foom at fuhm.net (James Y Knight) Date: Tue, 18 Jul 2006 14:01:09 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <44BD2054.4020705@v.loewis.de> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> Message-ID: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> On Jul 18, 2006, at 1:54 PM, Martin v. L?wis wrote: > Mihai Ibanescu wrote: >> To follow up on my own email: it looks like, even though in some >> locale >> "INFO".lower() != "info" >> >> u"INFO".lower() == "info" (at least in the Turkish locale). >> >> Is that guaranteed, at least for now (for the current versions of >> python)? > > It's guaranteed for now; unicode.lower is not locale-aware. That seems backwards of how it should be ideally: the byte-string upper and lower should always do ascii uppering-and-lowering, and the unicode ones should do it according to locale. Perhaps that can be cleaned up in py3k? James From mal at egenix.com Tue Jul 18 20:38:16 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 18 Jul 2006 20:38:16 +0200 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> Message-ID: <44BD2A98.7000705@egenix.com> James Y Knight wrote: > On Jul 18, 2006, at 1:54 PM, Martin v. L?wis wrote: > >> Mihai Ibanescu wrote: >>> To follow up on my own email: it looks like, even though in some >>> locale >>> "INFO".lower() != "info" >>> >>> u"INFO".lower() == "info" (at least in the Turkish locale). >>> >>> Is that guaranteed, at least for now (for the current versions of >>> python)? >> It's guaranteed for now; unicode.lower is not locale-aware. > > That seems backwards of how it should be ideally: the byte-string > upper and lower should always do ascii uppering-and-lowering, and the > unicode ones should do it according to locale. Perhaps that can be > cleaned up in py3k? Actually, you've got that backwards ;-) ... There are no .lower()/.upper() methods for bytes. The reason these methods are locale aware for 8-bit strings lies in the fact that we're using the C lib functions, which are locale setting dependent - with all the drawbacks that go with it. The Unicode database OTOH *defines* the upper/lower case mapping in a locale independent way, so the mappings are guaranteed to always produce the same results on all platforms. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From guido at python.org Tue Jul 18 20:40:26 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Jul 2006 11:40:26 -0700 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> Message-ID: On 7/18/06, James Y Knight wrote: > On Jul 18, 2006, at 1:54 PM, Martin v. L?wis wrote: > > > Mihai Ibanescu wrote: > >> To follow up on my own email: it looks like, even though in some > >> locale > >> "INFO".lower() != "info" > >> > >> u"INFO".lower() == "info" (at least in the Turkish locale). > >> > >> Is that guaranteed, at least for now (for the current versions of > >> python)? > > > > It's guaranteed for now; unicode.lower is not locale-aware. > > That seems backwards of how it should be ideally: the byte-string > upper and lower should always do ascii uppering-and-lowering, and the > unicode ones should do it according to locale. Perhaps that can be > cleaned up in py3k? No, you've got it backwards. 8-bit strings are assumed to be encoded using the current locale's default encoding so upper and lower behave locale-dependent. Unicode strings don't need the locale as additional input for upper and lower; in a different locale you simply use a different code point. I believe the original issue was that in Turkish, there are i's with and without dots in both lower and upper case. I'm guessing that the ASCII code points are used for lowercase dotted i and uppercase undotted I; code points with the high bit set are used for the uppercase dotted i and lowercase undotted I (which I can't easily type here). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From misa at redhat.com Tue Jul 18 20:46:04 2006 From: misa at redhat.com (Mihai Ibanescu) Date: Tue, 18 Jul 2006 14:46:04 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <20060718174541.GI27181@abulafia.devel.redhat.com> Message-ID: <20060718184604.GA363@abulafia.devel.redhat.com> On Tue, Jul 18, 2006 at 10:53:23AM -0700, Guido van Rossum wrote: > And u"info".upper()? Yepp, that shows the right thing (at least in the several locales I tested, Turkish included). It's along the lines of u"INFO".lower() I was proposing in my second post :-) Misa From misa at redhat.com Tue Jul 18 20:52:38 2006 From: misa at redhat.com (Mihai Ibanescu) Date: Tue, 18 Jul 2006 14:52:38 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <44BD2054.4020705@v.loewis.de> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> Message-ID: <20060718185238.GB363@abulafia.devel.redhat.com> On Tue, Jul 18, 2006 at 07:54:28PM +0200, "Martin v. L?wis" wrote: > Mihai Ibanescu wrote: > > To follow up on my own email: it looks like, even though in some locale > > "INFO".lower() != "info" > > > > u"INFO".lower() == "info" (at least in the Turkish locale). > > > > Is that guaranteed, at least for now (for the current versions of python)? > > It's guaranteed for now; unicode.lower is not locale-aware. OK, should I write a patch for the logging module to convert the string to unicode before applying lower()? So far that seems like the way to go. Maybe this could also be explained in the documentation: http://docs.python.org/lib/node323.html I don't think I've seen it in the locale documentation that locale settings do not affect unicode strings, and that particular page says If, when coding a module for general use, you need a locale independent version of an operation that is affected by the locale (such as string.lower(), or certain formats used with time.strftime()), you will have to find a way to do it without using the standard library routine. Unicode might be a perfectly acceptable suggestion for others too. Misa From fdrake at acm.org Tue Jul 18 20:55:56 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue, 18 Jul 2006 14:55:56 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <20060718185238.GB363@abulafia.devel.redhat.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <20060718185238.GB363@abulafia.devel.redhat.com> Message-ID: <200607181455.56535.fdrake@acm.org> On Tuesday 18 July 2006 14:52, Mihai Ibanescu wrote: > Unicode might be a perfectly acceptable suggestion for others too. Are we still supporting builds that don't include Unicode? If so, that needs to be considered in a patch as well. -Fred -- Fred L. Drake, Jr. From martin at v.loewis.de Tue Jul 18 21:26:28 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 18 Jul 2006 21:26:28 +0200 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> Message-ID: <44BD35E4.3050402@v.loewis.de> James Y Knight wrote: > That seems backwards of how it should be ideally: the byte-string upper > and lower should always do ascii uppering-and-lowering, and the unicode > ones should do it according to locale. Perhaps that can be cleaned up in > py3k? Cleaned-up, yes. But it is currently not backwards. For a byte string, you need an encoding, which comes from the locale. So for byte strings, case-conversion *has* to be locale-aware (in principle, making it encoding-aware only would almost suffice, but there is no universal API for that). OTOH, for Unicode, due to the unification, case-conversion mostly does not need to be locale-aware. Nearly all case-conversions are only script-dependent, not language-dependent. So it is nearly possible to make case-conversion locale-independent, and that is what Python provides. The "nearly" above refers to *very* few exceptions, in *very* few languages. Most of the details are collected in UAX#21, some highlights are: - case conversions are not always reversible - sometimes, case conversion may convert a single character to multiple characters; the canonical example is German ? (considered lower-case) -> "SS" (historically, this is just typographical, since there is no upper case sharp s in our script) - sometimes, conversion depends on the position of the letter in the word, see Final_Sigma in SpecialCasing.txt, or on the subsequent combining accents, see Lithuanian More_Above I believe the unicode.lower behaviour is currently right for most applications, so it should continue to be the default. An additional locale-aware version should be added, but that probably means to incorporate ICU into Python, to get this and other locale properties right in a platform-independent fashion. Regards, Martin From martin at v.loewis.de Tue Jul 18 21:32:39 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 18 Jul 2006 21:32:39 +0200 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <44BD2A98.7000705@egenix.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> <44BD2A98.7000705@egenix.com> Message-ID: <44BD3757.7030101@v.loewis.de> M.-A. Lemburg wrote: > The Unicode database OTOH *defines* the upper/lower case mapping in > a locale independent way, so the mappings are guaranteed > to always produce the same results on all platforms. Actually, that isn't the full truth; see UAX#21, which is now official part of Unicode 4. It specifies two kinds of case conversion: simple case conversion, and full case conversion. Python only supports simple case conversion at the moment. Full case conversion is context (locale) dependent, and must take into account SpecialCasing.txt. Regards, Martin From rhettinger at ewtllc.com Tue Jul 18 22:37:36 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Tue, 18 Jul 2006 13:37:36 -0700 Subject: [Python-Dev] Strategy for converting the decimal module to C Message-ID: <44BD4690.3070402@ewtllc.com> I briefly had a chance to look at some of the work being done on a C implementation of decimal, and it looks like the approach is following the Python version too literally. Ideally, it should be written as if Python were not involved and afterwards add the appropriate method wrappers. Contexts should be small structures that include the traps and flags as bitfields. Likewise, the internal representation of a decimal should be in a simple structure using a byte array for the decimal digits -- it should not be a Python object. Internally, the implementation creates many temporary decimals for a scratchpad, it makes many copies of contexts, it frequently tests components of a context, and the functions frequently call each other. Those operations need to be as cheap as possible -- that means no internal tuple objects, no ref counting, and no passing variables through tuples of arguments, etc. I recommend writing most of the module to be independent of the Python C API. After a working implementation is built, grafting on the wrappers should be a trivial step. Unless we stay true to this course, the code will end-up being unnecessarily complex and the performance will be disappointing. my-two-cents, Raymond P.S. The dictionary approach to context objects should likely be abandoned for the C version. If the API has to change a bit, then so be it. From aahz at pythoncraft.com Tue Jul 18 23:00:22 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 18 Jul 2006 14:00:22 -0700 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <44BD4690.3070402@ewtllc.com> References: <44BD4690.3070402@ewtllc.com> Message-ID: <20060718210017.GA7850@panix.com> On Tue, Jul 18, 2006, Raymond Hettinger wrote: > > P.S. The dictionary approach to context objects should likely be > abandoned for the C version. If the API has to change a bit, then so > be it. Why do you say that? The rest I agree with; seems to me that making a small wrapper for dict access works well, too. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." --Brian W. Kernighan From mal at egenix.com Tue Jul 18 23:03:54 2006 From: mal at egenix.com (M.-A. Lemburg) Date: Tue, 18 Jul 2006 23:03:54 +0200 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <44BD3757.7030101@v.loewis.de> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> <44BD2A98.7000705@egenix.com> <44BD3757.7030101@v.loewis.de> Message-ID: <44BD4CBA.7040605@egenix.com> Martin v. L?wis wrote: > M.-A. Lemburg wrote: >> The Unicode database OTOH *defines* the upper/lower case mapping in >> a locale independent way, so the mappings are guaranteed >> to always produce the same results on all platforms. > > Actually, that isn't the full truth; see UAX#21, which is now official > part of Unicode 4. It specifies two kinds of case conversion: > simple case conversion, and full case conversion. Python only supports > simple case conversion at the moment. Full case conversion is context > (locale) dependent, and must take into account SpecialCasing.txt. Right. In fact, some case mappings are not available in the Unicode database, since that only contains mappings which don't increase or decrease the length of the Unicode string. A typical example is the German u'?'. u'?'.upper() would have to give u'SS', but instead returns u'?'. However, the point I wanted to make was that these mappings don't depend on the locale setting of the C lib - you have to explicitly access the mapping in the context of a locale and/or text. As an example, here's the definition for the dotted/dotless i's in Turkish taken from that file (http://www.unicode.org/Public/UNIDATA/SpecialCasing.txt): """ # The entries in this file are in the following machine-readable format: # # ; ; ; <upper> ; (<condition_list> ;)? # <comment> # ... # I and i-dotless; I-dot and i are case pairs in Turkish and Azeri # The following rules handle those cases. 0130; 0069; 0130; 0130; tr; # LATIN CAPITAL LETTER I WITH DOT ABOVE 0130; 0069; 0130; 0130; az; # LATIN CAPITAL LETTER I WITH DOT ABOVE # When lowercasing, remove dot_above in the sequence I + dot_above, which will turn into i. # This matches the behavior of the canonically equivalent I-dot_above 0307; ; 0307; 0307; tr After_I; # COMBINING DOT ABOVE 0307; ; 0307; 0307; az After_I; # COMBINING DOT ABOVE # When lowercasing, unless an I is before a dot_above, it turns into a dotless i. 0049; 0131; 0049; 0049; tr Not_Before_Dot; # LATIN CAPITAL LETTER I 0049; 0131; 0049; 0049; az Not_Before_Dot; # LATIN CAPITAL LETTER I # When uppercasing, i turns into a dotted capital I 0069; 0069; 0130; 0130; tr; # LATIN SMALL LETTER I 0069; 0069; 0130; 0130; az; # LATIN SMALL LETTER I # Note: the following case is already in the UnicodeData file. # 0131; 0131; 0049; 0049; tr; # LATIN SMALL LETTER DOTLESS I """ Note how the context of the usage of the code points matters when doing case-conversions. To make things even more complicated, there are so called language tags which can be embedded into the Unicode string, so the language can also change within a Unicode string. http://www.unicode.org/reports/tr7/ To get a feeling of what it takes to do locale aware handling of Unicode right, have a look at the Locale Data Markup Language (LDML): http://www.unicode.org/reports/tr35/ (hey, perhaps Google could contribute support for this to Python ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Jul 18 2006) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From martin at v.loewis.de Tue Jul 18 23:15:17 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 18 Jul 2006 23:15:17 +0200 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <44BD4CBA.7040605@egenix.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> <44BD2A98.7000705@egenix.com> <44BD3757.7030101@v.loewis.de> <44BD4CBA.7040605@egenix.com> Message-ID: <44BD4F65.9040307@v.loewis.de> M.-A. Lemburg wrote: > Right. In fact, some case mappings are not available in the Unicode > database, since that only contains mappings which don't increase or > decrease the length of the Unicode string. A typical example is the > German u'?'. u'?'.upper() would have to give u'SS', but instead > returns u'?'. Actually, that is in the Unicode database (SpecialCasing.txt): 00DF; 00DF; 0053 0073; 0053 0053; # LATIN SMALL LETTER SHARP S > However, the point I wanted to make was that these mappings don't depend > on the locale setting of the C lib - you have to explicitly > access the mapping in the context of a locale and/or text. I don't get that point. SpecialCasing.txt is clearly intended to take locale context into account. Whether this is the "C locale", or some other locale mechanism, is out of scope of the Unicode specification. It could be the C locale (and indeed, the C locale implementations often take the Unicode casing procedure into account these days). Regards, Martin From rhettinger at ewtllc.com Wed Jul 19 00:39:20 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Tue, 18 Jul 2006 15:39:20 -0700 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <20060718210017.GA7850@panix.com> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> Message-ID: <44BD6318.6010806@ewtllc.com> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060718/b66f8c5d/attachment.htm From tim.peters at gmail.com Wed Jul 19 01:01:37 2006 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 18 Jul 2006 19:01:37 -0400 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <44BD6318.6010806@ewtllc.com> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> <44BD6318.6010806@ewtllc.com> Message-ID: <1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com> [Raymond Hettinger] > ... > If the current approach gets in their way, the C implementers should feel free to > make an alternate design choice. I expect they will, eventually. Converting this to C is a big job, and at the NFS sprint we settled on an "incremental" strategy allowing most of the module to remain written in Python, converting methods to C one at a time. Changing the user-visible API is a hard egg to swallow, and it's unfortunate that the Python code used a dict to hold "flags" to begin with. The dict doesn't just record whether an exception has occurred, it also counts how many times the exception occurred. It's possible that someone, somewhere, has latched on to that as "a feature". From dalcinl at gmail.com Wed Jul 19 01:15:18 2006 From: dalcinl at gmail.com (Lisandro Dalcin) Date: Tue, 18 Jul 2006 20:15:18 -0300 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> <44BD6318.6010806@ewtllc.com> <1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com> Message-ID: <e7ba66e40607181615v72ac1b7eud8d5e3e094cc153a@mail.gmail.com> On 7/18/06, Tim Peters <tim.peters at gmail.com> wrote: > [Raymond Hettinger] > > ... > > If the current approach gets in their way, the C implementers should feel free to > > make an alternate design choice. > > I expect they will, eventually. Converting this to C is a big job, > and at the NFS sprint we settled on an "incremental" strategy allowing > most of the module to remain written in Python, converting methods to > C one at a time. Changing the user-visible API is a hard egg to > swallow, and it's unfortunate that the Python code used a dict to hold > "flags" to begin with. The dict doesn't just record whether an > exception has occurred, it also counts how many times the exception > occurred. It's possible that someone, somewhere, has latched on to > that as "a feature". Why not a 'cDecimal' module instead? -- Lisandro Dalc?n --------------- Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC) Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC) Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET) PTLC - G?emes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594 From rhettinger at ewtllc.com Wed Jul 19 02:06:06 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Tue, 18 Jul 2006 17:06:06 -0700 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <e7ba66e40607181615v72ac1b7eud8d5e3e094cc153a@mail.gmail.com> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> <44BD6318.6010806@ewtllc.com> <1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com> <e7ba66e40607181615v72ac1b7eud8d5e3e094cc153a@mail.gmail.com> Message-ID: <44BD776E.1040404@ewtllc.com> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060718/5d4775ac/attachment.html From rasky at develer.com Wed Jul 19 02:34:28 2006 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 19 Jul 2006 02:34:28 +0200 Subject: [Python-Dev] Strategy for converting the decimal module to C References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com><44BD6318.6010806@ewtllc.com> <1f7befae0607181601g65bc3e1ax5f598c82b9de71fb@mail.gmail.com> Message-ID: <089a01c6aacb$184cd7d0$d503030a@trilan> Tim Peters wrote: > Changing the user-visible API is a hard egg to > swallow, and it's unfortunate that the Python code used a dict to hold > "flags" to begin with. The dict doesn't just record whether an > exception has occurred, it also counts how many times the exception > occurred. It's possible that someone, somewhere, has latched on to > that as "a feature". Especially since it was a documented one: >>> import decimal >>> help(decimal.Context) Help on class Context in module decimal: class Context(__builtin__.object) | Contains the context for a Decimal instance. [...] | flags - When an exception is caused, flags[exception] is incremented. | (Whether or not the trap_enabler is set) | Should be reset by user of Decimal instance. [...] -- Giovanni Bajo From greg.ewing at canterbury.ac.nz Wed Jul 19 03:17:20 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Jul 2006 13:17:20 +1200 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <20060718124019.GB16294@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <9AC7F55A-4106-414F-9C20-64FFFD8C6131@fuhm.net> Message-ID: <44BD8820.9080602@canterbury.ac.nz> James Y Knight wrote: > That seems backwards of how it should be ideally: the byte-string > upper and lower should always do ascii uppering-and-lowering, and the > unicode ones should do it according to locale. Perhaps that can be > cleaned up in py3k? I would expect bytes objects not to have upper() and lower() methods at all in Py3k. -- Greg From g.brandl at gmx.net Wed Jul 19 09:46:59 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Jul 2006 09:46:59 +0200 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <44BD6318.6010806@ewtllc.com> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> <44BD6318.6010806@ewtllc.com> Message-ID: <e9ko1k$gv1$1@sea.gmane.org> Raymond Hettinger wrote: > Aahz wrote: >> On Tue, Jul 18, 2006, Raymond Hettinger wrote: >> >>> P.S. The dictionary approach to context objects should likely be >>> abandoned for the C version. If the API has to change a bit, then so >>> be it. >>> >> >> Why do you say that? The rest I agree with; seems to me that making a >> small wrapper for dict access works well, too. >> > I think it was tripping-up the folks working on the C implementation. > Georg can speak to it more directly. IIRC, the issue was that the > context object exposed a dictionary which a user could update directly > and there was no notification back to the surrounding object so it could > update an underlying bitfield representation. Yes, that's exactly what the problem was. Working with bitfields internally is fine as long as Python code doesn't want to change the dicts exposed as a wrapper. > If the current approach > gets in their way, the C implementers should feel free to make an > alternate design choice. +1. (cDecimal is an ugly name, but a sound concept) I don't know what progress Mateusz' work has made until now, but he wrote at one time that he would "start optimizing as soon as he's certain that it works". Georg From ncoghlan at gmail.com Wed Jul 19 12:31:16 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Jul 2006 20:31:16 +1000 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <e9ko1k$gv1$1@sea.gmane.org> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> <44BD6318.6010806@ewtllc.com> <e9ko1k$gv1$1@sea.gmane.org> Message-ID: <44BE09F4.6000009@gmail.com> Georg Brandl wrote: > Raymond Hettinger wrote: >> If the current approach >> gets in their way, the C implementers should feel free to make an >> alternate design choice. > > +1. (cDecimal is an ugly name, but a sound concept) > > I don't know what progress Mateusz' work has made until now, but he wrote > at one time that he would "start optimizing as soon as he's certain that > it works". dmath (decimal-math, modelled on cmath for complex-math) could work. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From mattjfleming at googlemail.com Wed Jul 19 12:45:05 2006 From: mattjfleming at googlemail.com (Matt Fleming) Date: Wed, 19 Jul 2006 11:45:05 +0100 Subject: [Python-Dev] [Python-checkins] r50708 - in python/trunk: Lib/test/test_sys.py Misc/NEWS Python/pystate.c In-Reply-To: <5ff4a1e50607190334x62ae8f56r64dd71bd1c6de261@mail.gmail.com> References: <20060719000321.36AF31E401E@bag.python.org> <20060719010840.GD2540@performancedrivers.com> <ee2a432c0607181936l6174016l6cd381678b722abe@mail.gmail.com> <e9kqrq$oto$1@sea.gmane.org> <5ff4a1e50607190334x62ae8f56r64dd71bd1c6de261@mail.gmail.com> Message-ID: <5ff4a1e50607190345w21f32050w55cec402f9845972@mail.gmail.com> On 19/07/06, Thomas Heller <theller at python.net> wrote: > Neal Norwitz schrieb: > > On 7/18/06, Jack Diederich <jackdied at jackdied.com> wrote: > >> > >> were pre-2003 and talking about mod_python. HURD and FreeBSD came up a > >> couple times. Do we need to add more *BSD buildbots? > > > > Yes. We only have OpenBSD now. It would be nice to have {Free,Net}BSD too.. > > Maybe some of the buildbots should (in addition to the normal build?) > configure Python with --without-threads? > I have an AMD64 NetBSD machine that isn't doing much at the moment, I can regurlary run tests (I submitted a patch not long back to make regrtest netbsd-3 aware). However, I can't turn it into a buildbot, sorry. Matt -- http://mattssanctuary.blogspot.com From rhettinger at ewtllc.com Wed Jul 19 21:24:18 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Wed, 19 Jul 2006 12:24:18 -0700 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <e9ko1k$gv1$1@sea.gmane.org> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> <44BD6318.6010806@ewtllc.com> <e9ko1k$gv1$1@sea.gmane.org> Message-ID: <44BE86E2.6070802@ewtllc.com> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060719/dc62ed02/attachment.html From g.brandl at gmx.net Wed Jul 19 21:42:26 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 19 Jul 2006 21:42:26 +0200 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <44BE86E2.6070802@ewtllc.com> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> <44BD6318.6010806@ewtllc.com> <e9ko1k$gv1$1@sea.gmane.org> <44BE86E2.6070802@ewtllc.com> Message-ID: <e9m1va$5pv$1@sea.gmane.org> Raymond Hettinger wrote: > >>> I think it was tripping-up the folks working on the C implementation. >>> Georg can speak to it more directly. IIRC, the issue was that the >>> context object exposed a dictionary which a user could update directly >>> and there was no notification back to the surrounding object so it could >>> update an underlying bitfield representation. >>> >> >> Yes, that's exactly what the problem was. Working with bitfields internally >> is fine as long as Python code doesn't want to change the dicts exposed >> as a wrapper. >> > > If you want to stick with dictionary-like access to traps and flags, > then use a dict subclass that overrides each of the mutating methods. That may be easiest to do in Python code. I'll leave it to Mateusz to decide. > Even then, we need to drop the concept of having the flags as counters > rather than booleans. Yes. Given that even Tim couldn't imagine a use case for counting the exceptions, I think it's sensible. Georg From nmm1 at cus.cam.ac.uk Wed Jul 19 22:45:00 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Wed, 19 Jul 2006 21:45:00 +0100 Subject: [Python-Dev] Strategy for converting the decimal module to C Message-ID: <E1G3Iui-0006Zk-CQ@draco.cus.cam.ac.uk> Georg Brandl <g.brandl at gmx.net> wrote: > > > Even then, we need to drop the concept of having the flags as counters > > rather than booleans. > > Yes. Given that even Tim couldn't imagine a use case for counting the > exceptions, I think it's sensible. Well, I can. There is a traditional, important use - tuning. When such arithmetic is implemented in hardware, it is normal for exceptional cases to be handled by interrupt, and that is VERY expensive - often 100-1,000 times the cost of a single operation, occasionally 10,000 times. It then becomes important to know how many of the things you got, to know whether it is worth putting code in to avoid them or even using a different algorithm. Now, it is perfectly correct to say that this does not apply to emulated arithmetic and that there is no justification for such ghastly implementations. But, regrettably, almost all exception handling on modern systems IS ghastly - at least by the standards of the 1960s. Whether you regard the use of Python for tuning code that is to be run using hardware, where the arithmetic will be performance- critical as important, is a matter of taste. I don't :-) Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From misa at redhat.com Wed Jul 19 23:29:19 2006 From: misa at redhat.com (Mihai Ibanescu) Date: Wed, 19 Jul 2006 17:29:19 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <200607181455.56535.fdrake@acm.org> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <20060718185238.GB363@abulafia.devel.redhat.com> <200607181455.56535.fdrake@acm.org> Message-ID: <20060719212919.GD15030@abulafia.devel.redhat.com> On Tue, Jul 18, 2006 at 02:55:56PM -0400, Fred L. Drake, Jr. wrote: > On Tuesday 18 July 2006 14:52, Mihai Ibanescu wrote: > > Unicode might be a perfectly acceptable suggestion for others too. > > Are we still supporting builds that don't include Unicode? If so, that needs > to be considered in a patch as well. Good point. Does the attached patch look reasonable? Thanks, Misa -------------- next part -------------- --- Python-2.4.3/Lib/logging/handlers.py.nolocale 2006-07-19 12:15:46.000000000 -0400 +++ Python-2.4.3/Lib/logging/handlers.py 2006-07-19 12:16:14.000000000 -0400 @@ -44,6 +44,12 @@ DEFAULT_SOAP_LOGGING_PORT = 9023 SYSLOG_UDP_PORT = 514 +# If python was not built with unicode support, use the str function instead +# of the unicode type, and hope locale doesn't break things. + +if not hasattr(__builtins__, 'unicode'): + unicode = str + class BaseRotatingHandler(logging.FileHandler): """ Base class for handlers that rotate log files at a certain point. @@ -162,7 +168,7 @@ """ def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None): BaseRotatingHandler.__init__(self, filename, 'a', encoding) - self.when = string.upper(when) + self.when = unicode(when).upper() self.backupCount = backupCount # Calculate the real rollover interval, which is just the number of # seconds between rollovers. Also set the filename suffix used when @@ -642,10 +648,12 @@ """ We need to convert record level to lowercase, maybe this will change in the future. + We convert it to unicode first, to avoid locale from changing the + meaning of lower() and upper() """ msg = self.log_format_string % ( self.encodePriority(self.facility, - string.lower(record.levelname)), + unicode(record.levelname).lower()), msg) try: if self.unixsocket: @@ -854,7 +862,7 @@ ("GET" or "POST") """ logging.Handler.__init__(self) - method = string.upper(method) + method = unicode(method).upper() if method not in ["GET", "POST"]: raise ValueError, "method must be GET or POST" self.host = host From jeremy at alum.mit.edu Thu Jul 20 00:03:17 2006 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Wed, 19 Jul 2006 18:03:17 -0400 Subject: [Python-Dev] Python sprint in NY and CA, Aug. 21-24 Message-ID: <e8bf7a530607191503p6038f538y6decf87bae1a808a@mail.gmail.com> I'd like to repeat my invitation to spend a week at Google in California or New York for a Python sprint. We are hosting sprints at our offices in Mountain View and New York City the week of Aug. 21, Monday through Thursday. We're planning to work broadly on Python 2.6 and Python 3000. If you're interested in the core implementation, the standard library, the build process, or documentation, they're all in scope. Working on a test process using large community projects like Twisted and Zope would be a great activity, too. There's a wiki page with some more details: http://wiki.python.org/moin/GoogleSprint Feel free to follow up with Neal or me if you have questions. Jeremy From brett at python.org Thu Jul 20 00:35:45 2006 From: brett at python.org (Brett Cannon) Date: Wed, 19 Jul 2006 15:35:45 -0700 Subject: [Python-Dev] new security doc using object-capabilities Message-ID: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> After various people suggesting object-capabilities, takling with Mark S. Miller of the E programming language, and the people Mark works with at HP Labs (who have been giving talks every week during this month here at Google on object-capabilities), I have decided to go with object-capabilities for securing interpreters. I have rewritten my design doc from scratch and deleted the old one. The new doc is named securing_python.txt and can be found through the svn web interface at http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log. I have pretty much ignored any concrete API and such and gone more with a conceptual doc to make sure the API does not get in the way of the core security model. Using object-capabilities should make the implementation much cleaner. There is much less work directly on the interpreter and more of it gets pushed up to extension modules. I also have the okay of my supervisor to use this approach in my dissertation so this will get done. Two things do fall out of all of this which will make development much more modular and easier. First, the memory cap work just becomes a special build on its own; no need to tie into the security work. So I will be cleaning up the bcannon-sandboxing branch code as it stands, and then either create a separate branch for the object-capabilities work, or create another branch for the memory cap stuff and shift the changes over there. I will most likely do the former so as to not lose the history on the checkins. I also plan to rewrite the import machinery in pure Python. This will make the code much more maintainable and make creating proxies for the import machinery much easier. I will be doing that in a directory in the sandbox initially since it needs to work from what Python has now (and possibly some new extension module code) before it can be integrated into the interpreter directly. Anyone who wants to help with that can. I already have some perliminary notes on the whole thing and I think it will be reasonably doable. Anyway, there you go. Here is to hoping I have thought this all through properly. =) -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060719/09a113fc/attachment.html From python-dev at zesty.ca Thu Jul 20 01:11:24 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Wed, 19 Jul 2006 18:11:24 -0500 (CDT) Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> Message-ID: <Pine.LNX.4.58.0607191810460.31087@server1.LFW.org> On Wed, 19 Jul 2006, Brett Cannon wrote: > I have decided to go with object-capabilities for > securing interpreters. I have rewritten my design doc from scratch and > deleted the old one. The new doc is named securing_python.txt and can be > found through the svn web interface at > http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log. This is amazing news!! I'm going off to read your document right now. -- ?!ng From fuzzyman at voidspace.org.uk Thu Jul 20 01:25:26 2006 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 20 Jul 2006 00:25:26 +0100 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> Message-ID: <44BEBF66.9020102@voidspace.org.uk> Brett Cannon wrote: > After various people suggesting object-capabilities, takling with Mark > S. Miller of the E programming language, and the people Mark works > with at HP Labs (who have been giving talks every week during this > month here at Google on object-capabilities), I have decided to go > with object-capabilities for securing interpreters. I have rewritten > my design doc from scratch and deleted the old one. The new doc is > named securing_python.txt and can be found through the svn web > interface at > http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log > <http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log> > . I have pretty much ignored any concrete API and such and gone more > with a conceptual doc to make sure the API does not get in the way of > the core security model. > This may not be relevant or possible, in which case I apologise, but the .NET model of creating application domains is extremely useful. It allows you to assign domains and run code within those domains. This means, for example, you can create a plugin system and run the plugins in a secure domain. I realise that this was the intent of the original rexec module, and your proposed new design (which is very exciting) overcomes the difficulties in that approach. The only approach using the new system would be interprocess communication (?) with a trusted interpreter communicating with an un-trusted one. Would the communication layer need to be implemented as a C extension, or will a standard Python API be possible ? Hmmm.... maybe I should read your doc. :-) Michael Foord http://www.voidspace.org.uk/python/index.shtml > Using object-capabilities should make the implementation much > cleaner. There is much less work directly on the interpreter and more > of it gets pushed up to extension modules. I also have the okay of my > supervisor to use this approach in my dissertation so this will get done. > > Two things do fall out of all of this which will make development much > more modular and easier. First, the memory cap work just becomes a > special build on its own; no need to tie into the security work. So I > will be cleaning up the bcannon-sandboxing branch code as it stands, > and then either create a separate branch for the object-capabilities > work, or create another branch for the memory cap stuff and shift the > changes over there. I will most likely do the former so as to not > lose the history on the checkins. > > I also plan to rewrite the import machinery in pure Python. This will > make the code much more maintainable and make creating proxies for the > import machinery much easier. I will be doing that in a directory in > the sandbox initially since it needs to work from what Python has now > (and possibly some new extension module code) before it can be > integrated into the interpreter directly. Anyone who wants to help > with that can. I already have some perliminary notes on the whole > thing and I think it will be reasonably doable. > > Anyway, there you go. Here is to hoping I have thought this all > through properly. =) > > -Brett > ------------------------------------------------------------------------ > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > From fuzzyman at voidspace.org.uk Thu Jul 20 01:50:33 2006 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 20 Jul 2006 00:50:33 +0100 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <44BEBF66.9020102@voidspace.org.uk> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <44BEBF66.9020102@voidspace.org.uk> Message-ID: <44BEC549.5060305@voidspace.org.uk> Michael Foord wrote: > Brett Cannon wrote: > >> After various people suggesting object-capabilities, takling with Mark >> S. Miller of the E programming language, and the people Mark works >> with at HP Labs (who have been giving talks every week during this >> month here at Google on object-capabilities), I have decided to go >> with object-capabilities for securing interpreters. I have rewritten >> my design doc from scratch and deleted the old one. The new doc is >> named securing_python.txt and can be found through the svn web >> interface at >> http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log >> <http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log> >> . I have pretty much ignored any concrete API and such and gone more >> with a conceptual doc to make sure the API does not get in the way of >> the core security model. >> >> > > This may not be relevant or possible, in which case I apologise, but > the .NET model of creating application domains is extremely useful. It > allows you to assign domains and run code within those domains. This > means, for example, you can create a plugin system and run the plugins > in a secure domain. > > I realise that this was the intent of the original rexec module, and > your proposed new design (which is very exciting) overcomes the > difficulties in that approach. The only approach using the new system > would be interprocess communication (?) with a trusted interpreter > communicating with an un-trusted one. Would the communication layer need > to be implemented as a C extension, or will a standard Python API be > possible ? Hmmm.... maybe I should read your doc. :-) > > Ok, started to read the doc - and realise it specifically addresses these issues. My apologies :-) Michael http://www.voidspace.org.uk/python/index.shtml > Michael Foord > http://www.voidspace.org.uk/python/index.shtml > > >> Using object-capabilities should make the implementation much >> cleaner. There is much less work directly on the interpreter and more >> of it gets pushed up to extension modules. I also have the okay of my >> supervisor to use this approach in my dissertation so this will get done. >> >> Two things do fall out of all of this which will make development much >> more modular and easier. First, the memory cap work just becomes a >> special build on its own; no need to tie into the security work. So I >> will be cleaning up the bcannon-sandboxing branch code as it stands, >> and then either create a separate branch for the object-capabilities >> work, or create another branch for the memory cap stuff and shift the >> changes over there. I will most likely do the former so as to not >> lose the history on the checkins. >> >> I also plan to rewrite the import machinery in pure Python. This will >> make the code much more maintainable and make creating proxies for the >> import machinery much easier. I will be doing that in a directory in >> the sandbox initially since it needs to work from what Python has now >> (and possibly some new extension module code) before it can be >> integrated into the interpreter directly. Anyone who wants to help >> with that can. I already have some perliminary notes on the whole >> thing and I think it will be reasonably doable. >> >> Anyway, there you go. Here is to hoping I have thought this all >> through properly. =) >> >> -Brett >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> http://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk >> >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From tim.peters at gmail.com Thu Jul 20 02:52:11 2006 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 19 Jul 2006 20:52:11 -0400 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <e9m1va$5pv$1@sea.gmane.org> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> <44BD6318.6010806@ewtllc.com> <e9ko1k$gv1$1@sea.gmane.org> <44BE86E2.6070802@ewtllc.com> <e9m1va$5pv$1@sea.gmane.org> Message-ID: <1f7befae0607191752j7b4a828eu99ea6cc28be40f28@mail.gmail.com> ... [Raymond] >> Even then, we need to drop the concept of having the flags as counters >> rather than booleans. [Georg Brandl] > Yes. Given that even Tim couldn't imagine a use case for counting the > exceptions, I think it's sensible. That's not it -- someone will "find a use" for anything. It's unfortunate that we used a dict with counts because the /standard/ we're trying to meet requires no such thing, and clearly had a "pile of on/off bits" model in mind. Extending a standard without strong need creates problems (for example, this one <0.5 wink>). From greg.ewing at canterbury.ac.nz Thu Jul 20 03:30:50 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Jul 2006 13:30:50 +1200 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <E1G3Iui-0006Zk-CQ@draco.cus.cam.ac.uk> References: <E1G3Iui-0006Zk-CQ@draco.cus.cam.ac.uk> Message-ID: <44BEDCCA.1070305@canterbury.ac.nz> Nick Maclaren wrote: > When such arithmetic is implemented in hardware, it is normal for > exceptional cases to be handled by interrupt, and that is VERY > expensive ... It then becomes important to know how > many of the things you got, to know whether it is worth putting > code in to avoid them or even using a different algorithm. But couldn't you just put in an interrupt handler that counts the interrupts, for the purpose of measurement? -- Greg From rhettinger at ewtllc.com Thu Jul 20 03:45:51 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Wed, 19 Jul 2006 18:45:51 -0700 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <1f7befae0607191752j7b4a828eu99ea6cc28be40f28@mail.gmail.com> References: <44BD4690.3070402@ewtllc.com> <20060718210017.GA7850@panix.com> <44BD6318.6010806@ewtllc.com> <e9ko1k$gv1$1@sea.gmane.org> <44BE86E2.6070802@ewtllc.com> <e9m1va$5pv$1@sea.gmane.org> <1f7befae0607191752j7b4a828eu99ea6cc28be40f28@mail.gmail.com> Message-ID: <44BEE04F.9000209@ewtllc.com> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060719/058ab3cb/attachment.html From t-bruch at microsoft.com Thu Jul 20 04:29:09 2006 From: t-bruch at microsoft.com (Bruce Christensen) Date: Wed, 19 Jul 2006 19:29:09 -0700 Subject: [Python-Dev] Pickling objects that return string from reduce In-Reply-To: <44BBFB5B.2090806@v.loewis.de> References: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com> <44BBFB5B.2090806@v.loewis.de> Message-ID: <3581AA168D87A2479D88EA319BDF7D32EF2FC7@RED-MSG-80.redmond.corp.microsoft.com> Martin v. L?wis" wrote: > If obj has no __module__ attribute (or if it is None), pickle > (didn't check cPickle) also does > > for n, module in sys.module.items(): > if "module-ignored": continue > if getattr(module, result, None) is obj: > break # use n as module name > > If obj does have a __module__ attribute, it uses __import__ > to import the module, just to make sure it gets into sys.modules. What is "module-ignored" above? It's obviously not a literal string... --Bruce From kbk at shore.net Thu Jul 20 06:55:00 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Thu, 20 Jul 2006 00:55:00 -0400 (EDT) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200607200455.k6K4t0Vw018820@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 398 open ( +5) / 3334 closed (+19) / 3732 total (+24) Bugs : 904 open ( -4) / 6011 closed (+36) / 6915 total (+32) RFE : 222 open ( -1) / 231 closed ( +2) / 453 total ( +1) New / Reopened Patches ______________________ Fix for #1513611 and #1511497; xml.sax imports (2006-07-10) http://python.org/sf/1519796 opened by ?iga Seilnacht telnetlib.py change to ease option handling. (2006-07-10) http://python.org/sf/1520081 opened by Ernest ter Kuile Support for PyGetSetDefs in pydoc, inspect, and types (2006-07-10) http://python.org/sf/1520294 opened by Barry A. Warsaw Distutils bugfix: Read $AR from the environment/Makefile. (2006-07-11) http://python.org/sf/1520877 opened by Douglas Greiman make install change: Allow $DESTDIR to be relative (2006-07-11) http://python.org/sf/1520879 opened by Douglas Greiman Fix tests that assume they can write to Lib/test (2006-07-12) http://python.org/sf/1520904 opened by Douglas Greiman Don't produce core file in test_subprocess.py (2006-07-12) http://python.org/sf/1520905 opened by Douglas Greiman Extra configurability for doctest TestCases (2006-07-12) http://python.org/sf/1521051 opened by Russell Keith-Magee --help and --version long options (2006-07-12) CLOSED http://python.org/sf/1521179 opened by Georg Brandl smtplib login fails with aol smtp server (2006-07-12) http://python.org/sf/1521196 opened by Peter Reintroduce index checking in 1-element ctypes arrays (2006-07-13) CLOSED http://python.org/sf/1521817 opened by Thomas Heller Fix grammatical errors in Doc/howto/doanddont.tex (2006-07-13) CLOSED http://python.org/sf/1521874 opened by Collin Winter Give Cookie.py its own _idmap (2006-07-13) http://python.org/sf/1521882 opened by Collin Winter Add some explication to PEP 3100 (2006-07-13) http://python.org/sf/1522038 opened by Collin Winter Remove operator.truth() and operator.abs() (2006-07-13) http://python.org/sf/1522059 opened by Collin Winter Improve docs for filter() (2006-07-13) CLOSED http://python.org/sf/1522211 opened by Collin Winter irda socket support (2006-07-14) http://python.org/sf/1522400 opened by ?????? Tix.Grid patch (2006-07-14) http://python.org/sf/1522587 opened by klappnase XML Test Runner for unittest module (2006-07-14) http://python.org/sf/1522704 opened by Sebastian Rittau (partial?) fix for Misc/python-config.in (2006-07-16) http://python.org/sf/1523356 opened by M. Levinson Var-default patch undid part of backtick to repr() patch (2006-07-18) CLOSED http://python.org/sf/1524429 opened by Graham Horler Fix Tkinter Tcl-commands memory-leaks (2006-07-18) http://python.org/sf/1524639 opened by Graham Horler Fix --without-threads build error (2006-07-18) CLOSED http://python.org/sf/1524724 opened by Matt Fleming ConfigParser: accept leading whitespace on options+comments (2006-07-18) http://python.org/sf/1524825 opened by Ken Lalonde Patches Closed ______________ Patch for bug 1441486: bad unary minus folding in compiler (2006-03-10) http://python.org/sf/1446922 closed by nascheme OpenVMS patches Modules directory (2006-07-04) http://python.org/sf/1516912 closed by nnorwitz Rough documentation for xml.etree.ElementTree (2006-06-10) http://python.org/sf/1504046 closed by nnorwitz turtle.py: correcting begin_fill (2006-07-09) http://python.org/sf/1519566 closed by loewis pdb: fix for #1472251('run/runeval' commands bug) (2006-04-18) http://python.org/sf/1472257 closed by jakamkon urllib2 redirection fix (2006-07-01) http://python.org/sf/1515745 closed by nnorwitz Python long option support (2006-05-03) http://python.org/sf/1481112 closed by gbrandl --help and --version long options (2006-07-12) http://python.org/sf/1521179 closed by gbrandl Reintroduce index checking in 1-element ctypes arrays (2006-07-13) http://python.org/sf/1521817 closed by theller Fix grammatical errors in Doc/howto/doanddont.tex (2006-07-13) http://python.org/sf/1521874 closed by gbrandl Improve docs for filter() (2006-07-13) http://python.org/sf/1522211 closed by rhettinger python / pythonw replacement in C (2004-10-29) http://python.org/sf/1056561 closed by etrepum OS X: Can't use #!/usr/bin/pythonw (2004-10-02) http://python.org/sf/1038911 closed by etrepum update the binhex module for Mach-O (2005-06-14) http://python.org/sf/1220874 closed by etrepum Remove dependencies on the sets module (2006-06-04) http://python.org/sf/1500609 closed by gbrandl ColorDelegator - Several bug fixes (2006-04-30) http://python.org/sf/1479219 closed by kbk Var-default patch undid part of backtick to repr() patch (2006-07-18) http://python.org/sf/1524429 closed by loewis zipfile: support for ZIP64 (2006-03-09) http://python.org/sf/1446489 closed by ronaldoussoren Fix --without-threads build error (2006-07-18) http://python.org/sf/1524724 closed by tim_one New / Reopened Bugs ___________________ Cannot use high-numbered sockets in 2.4.3 (2006-05-24) CLOSED http://python.org/sf/1494314 reopened by anthonybaxter Incorrect lineno's in code objects (2006-06-26) CLOSED http://python.org/sf/1512814 reopened by twouters AttributeError in the shelve module (2006-07-09) CLOSED http://python.org/sf/1519786 opened by Martin Kelly Proxy does not work in 2.4.3 (2006-07-10) http://python.org/sf/1519816 opened by Michal Niklas non-uniform behavior in 'startswith' / 'endswith' (2006-07-10) http://python.org/sf/1520176 opened by Milind __missing__ does not get called (2006-07-11) CLOSED http://python.org/sf/1520327 opened by Milind fcntl.ioctl fails to copy back exactly-1024 buffer (2006-07-11) http://python.org/sf/1520818 opened by Mark Eichin urrlib2 max_redirections=0 disables redirects (2006-07-11) http://python.org/sf/1520831 opened by rus_r_orange unpack list of singleton tuples not unpacking (2006-07-11) CLOSED http://python.org/sf/1520864 opened by Anthony Tuininga time.strftime breakage in 2.4/2.5 (2006-07-11) CLOSED http://python.org/sf/1520914 opened by Skip Montanaro ctypes test overwrites /dev/null (2006-07-12) CLOSED http://python.org/sf/1521375 opened by Scot Doyle file.seek() influelce write() when opened with a+ mode (2006-07-12) http://python.org/sf/1521491 opened by Lior isinstance failure in 2.6 Beta 2 (2006-07-13) CLOSED http://python.org/sf/1521726 opened by Nick Maclaren possible bug in mystrtol.c with recent gcc (2006-07-13) http://python.org/sf/1521947 opened by Marien Zwart shlex.split() does not tokenize like the shell (2006-07-13) http://python.org/sf/1521950 opened by Dan Christian filter() doesn't use __len__ of str/unicode/tuple subclasses (2006-07-13) CLOSED http://python.org/sf/1522016 opened by Collin Winter RPM build fails for Py2.5b2 (2006-07-13) http://python.org/sf/1522046 opened by Stefan Behnel _threading_local.py logic error in _localbase __new__ (2006-07-13) http://python.org/sf/1522237 opened by Tony Nelson Patch #1388073 is not mentioned in NEWS (2006-07-14) CLOSED http://python.org/sf/1522771 opened by Collin Winter incorrect tcp checksum (2006-07-15) CLOSED http://python.org/sf/1523136 opened by ecir hana threading.Thread Traceback (2006-07-16) CLOSED http://python.org/sf/1523465 opened by roee88 PyArg_ParseTupleAndKeywords potential core dump (2006-07-16) http://python.org/sf/1523610 opened by Eric Huss 2.4.2 file.read caches EOF state (2006-07-17) http://python.org/sf/1523853 opened by Jarkko Torppa logging using the SysLog handler fails if locale is set (2006-07-17) http://python.org/sf/1524081 opened by Mihai Ibanescu os.listdir doesn't check error code from FindNextFile (2006-07-18) http://python.org/sf/1524310 opened by Roger Upole configure --without-threads fails to build (2006-07-18) CLOSED http://python.org/sf/1524317 opened by Ram Bhamidipaty MemoryError with a lot of available memory - gc not called (2006-07-19) http://python.org/sf/1524938 opened by Mark Matusevich Webserver TypeError: expected read buffer, NoneType found (2006-07-19) http://python.org/sf/1525343 opened by jbet Build fails on OS X with case sensitive fs (2006-07-19) http://python.org/sf/1525447 opened by gideon may SimpleXMLRpcServer still uses sys.exc_value and sys.exc_type (2006-07-19) http://python.org/sf/1525469 opened by Russell Warren Document additions from PEP 302 (2006-07-19) http://python.org/sf/1525549 opened by Brett Cannon Malloc, memory error, failmalloc, low memory. (2006-07-19) http://python.org/sf/1525589 opened by Rene Dudfield ldap get_option(0) causes segfault (2006-07-19) CLOSED http://python.org/sf/1525590 opened by Sean Burford exec and eval allocate lots of memory and do not free it (2006-07-20) http://python.org/sf/1525678 opened by Connelly Bugs Closed ___________ yet another svn head compiler change (2006-03-02) http://python.org/sf/1441486 closed by nascheme Cannot use high-numbered sockets in 2.4.3 (2006-05-24) http://python.org/sf/1494314 closed by anthonybaxter Incorrect lineno's in code objects (2006-06-26) http://python.org/sf/1512814 closed by nnorwitz Incorrect lineno's in code objects (2006-06-26) http://python.org/sf/1512814 closed by nnorwitz c_void_pointer should accept a long pointer > 0x7fffffff (2006-07-06) http://python.org/sf/1518190 closed by theller ImportWarning should be removed (2006-06-30) http://python.org/sf/1515169 closed by nnorwitz test_ctypes fails on OSX 10.3 (2006-04-10) http://python.org/sf/1467450 closed by theller AttributeError in the shelve module (2006-07-10) http://python.org/sf/1519786 closed by gbrandl inspect.py: still infinite recursion inspecting frames (2006-07-03) http://python.org/sf/1516184 closed by pje AttributeError in upload_file on interrupted connection (2006-03-23) http://python.org/sf/1457312 closed by pje subprocess test cases fail with noexec-mounted /tmp (2006-03-06) http://python.org/sf/1444408 closed by astrand subprocess.CalledProcessError uses errno incorrectly (2005-12-08) http://python.org/sf/1376309 closed by astrand subprocess.py abuse of errno (2005-06-20) http://python.org/sf/1223937 closed by astrand __missing__ does not get called (2006-07-11) http://python.org/sf/1520327 closed by gbrandl unpack list of singleton tuples not unpacking (2006-07-11) http://python.org/sf/1520864 closed by nnorwitz time.strftime breakage in 2.4/2.5 (2006-07-11) http://python.org/sf/1520914 closed by bcannon ctypes test overwrites /dev/null (2006-07-12) http://python.org/sf/1521375 closed by theller isinstance failure in 2.5 Beta 2 (2006-07-13) http://python.org/sf/1521726 closed by gbrandl filter() doesn't use __len__ of str/unicode/tuple subclasses (2006-07-13) http://python.org/sf/1522016 closed by rhettinger msvccompiler.py using VC6 with Python 2.5a2 (2006-06-18) http://python.org/sf/1508010 closed by loewis incorrect locale.strcoll() return in Windows (2006-07-08) http://python.org/sf/1519069 closed by loewis Patch #1388073 is not mentioned in NEWS (2006-07-14) http://python.org/sf/1522771 closed by gbrandl IDLE fails to launch after 46272 (2.4 with 2.5 IDLE) (2006-06-27) http://python.org/sf/1513617 closed by kbk ext/win-cookbook.html has a broken link to distutils (2006-07-07) http://python.org/sf/1518772 closed by quiver PIL binary package missing jpeg support (pimp) (2003-07-23) http://python.org/sf/776600 closed by etrepum IDE needs easy access to builtin help() (2003-03-29) http://python.org/sf/711991 closed by etrepum Raise IDE output window over splash screen on early crash (2003-03-26) http://python.org/sf/710374 closed by etrepum IDE stdin doesn't have readlines (2003-03-26) http://python.org/sf/710373 closed by etrepum Mac IDE behaviour (output to console) (2002-06-24) http://python.org/sf/573174 closed by etrepum incorrect tcp checksum (2006-07-15) http://python.org/sf/1523136 closed by loewis threading.Thread Traceback (2006-07-16) http://python.org/sf/1523465 closed by tim_one "as" keyword sometimes highlighted in strings (2005-10-12) http://python.org/sf/1325071 closed by kbk email.Utils.py: "'" in RFC2231 header (2005-06-10) http://python.org/sf/1218081 closed by bwarsaw Incorrect RFC 2231 decoding (2005-01-15) http://python.org/sf/1102973 closed by bwarsaw link path probs on OSX re: Tcl, Tk & fink's /sw (07/13/05) http://python.org/sf/1237697 closed by sf-robot termios.c in qnx4.25 (09/19/05) http://python.org/sf/1295179 closed by sf-robot IDLE bug - changing shortcuts - Shift-Tab broken! (2005-04-08) http://python.org/sf/1179168 closed by kbk expat crash python (2005-09-20) http://python.org/sf/1296433 closed by bcannon configure --without-threads fails to build (2006-07-18) http://python.org/sf/1524317 closed by tim_one ldap get_option(0) causes segfault (2006-07-19) http://python.org/sf/1525590 closed by nnorwitz Getting an error message import site failed -v traceback. (07/05/06) http://python.org/sf/1517370 closed by sf-robot New / Reopened RFE __________________ support all of strftime(3) (2006-07-11) http://python.org/sf/1520662 opened by Toni Mueller RFE Closed __________ Add syntax coloring to Mac IDE (2002-08-07) http://python.org/sf/592047 closed by etrepum Enhance PackageManager functionality (2003-07-28) http://python.org/sf/779160 closed by etrepum From martin at v.loewis.de Thu Jul 20 08:47:40 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 20 Jul 2006 08:47:40 +0200 Subject: [Python-Dev] Pickling objects that return string from reduce In-Reply-To: <3581AA168D87A2479D88EA319BDF7D32EF2FC7@RED-MSG-80.redmond.corp.microsoft.com> References: <3581AA168D87A2479D88EA319BDF7D32E05276@RED-MSG-80.redmond.corp.microsoft.com> <44BBFB5B.2090806@v.loewis.de> <3581AA168D87A2479D88EA319BDF7D32EF2FC7@RED-MSG-80.redmond.corp.microsoft.com> Message-ID: <44BF270C.90202@v.loewis.de> Bruce Christensen wrote: >> If obj has no __module__ attribute (or if it is None), pickle >> (didn't check cPickle) also does >> >> for n, module in sys.module.items(): >> if "module-ignored": continue >> if getattr(module, result, None) is obj: >> break # use n as module name > > What is "module-ignored" above? It's obviously not a literal string... It's skipped if module is None (skip dummy package entries) or n=='__main__'. Regards, Martin From l.oluyede at gmail.com Thu Jul 20 10:17:32 2006 From: l.oluyede at gmail.com (Lawrence Oluyede) Date: Thu, 20 Jul 2006 10:17:32 +0200 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> Message-ID: <9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com> That's great. I just read your draft but I have little comments to do but before let me say that I liked the idea to borrow concepts from E. I've crossed the E's path in the beginning of this year and I found it a pot of really nice ideas (for promises and capabilities). Here are my comments about the draft: - it's not really clear to me what the "powerbox" is. I think I got the concept of "super process" but maybe it's to be clarified, isn't it? It become clear in the "threat model" paragraph - I hope no Rubystas will read the "Problem of No Private Namespace" section because they have private/protected keywords to enforce this stuff :-) Writing proxies in C will slow down the dev process (altough will speed up the performance maybe) but in a far future someone will come up with an alternative closer to the Python level - Can you write down a simple example of what you mean with "changing something of the built-in objects"? (in "Problem of mutable shared state") - What about the performance issues of the capabilities model overall? - I know what you meant to say but the paragraph about pythonicness and the security model seems a little "fuzzy" to me. Which are the boundaries of the allowed changes for the security stuff? - You don't say anything about networking and networked resources in the list of the standard sandboxed interpreter - Suppose we have a .py module. Based on your security model we can import it, right? When imported it generates a .pyc file. The second time we import it what happens? .pyc is ignored? import is not allowed at all? We can't rely on the name of the file.pyc because an attacker who knows the file.py is secure and the second import is done against file.pyc can replace the "secure" file.pyc with an implementation not secure and can do some kind of harm to the sandbox - About "Filesystem information". Does the sandboxed interpreter need to know all that information about file paths, files and so on? Can't we reset those attributes to something arbitrary? - About sys module: I think the best way is to have a purged fake sys module with only the stuff you need. pypy has the concept of faked modules too (altough for a different reason) - About networking: what do you think about the E's model of really safe networking, protected remotable objects and safe RPC? Is that model applicable to Python's in some way? We can't use the E's model as a whole (ask people to generate a safe key and send it by email is unfeasible) - is the protected memory model a some kind of memory monitor system? I think that's all for the draft. I wrote these comments during the reading of the document. Hope some of these help -- Lawrence http://www.oluyede.org/blog From fuzzyman at voidspace.org.uk Thu Jul 20 11:12:43 2006 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 20 Jul 2006 10:12:43 +0100 Subject: [Python-Dev] os.utime and os.chmod failures (etc) Python 2.5b2 Message-ID: <44BF490B.1090101@voidspace.org.uk> Hello all, There may be a reasonable cause for this (i.e. it is likely to be my fault) - but it is consistent across two different machines I have tried it on. With Python 2.5b2 (from the msi at Python.org), running on Windows XP Pro SP2, ``os.utime`` and ``os.chmod`` fail with WindowsError. The same code runs fine on Python 2.3 and Python 2.4. [err] shutil.copytree(thisentry, targetdir) [err] File "C:\Python25\lib\shutil.py", line 130, in copytree [err] copystat(src, dst) [err] File "C:\Python25\lib\shutil.py", line 67, in copystat [err] os.utime(dst, (st.st_atime, st.st_mtime)) [err] WindowsError: [Error 13] Access is denied: 'lib\\Pmw' [err] The script uses ``shutil.copytree`` to copy a directory (using relative paths). The source code snippet is : if os.path.isdir(targetdir): shutil.rmtree(targetdir) shutil.copytree(thisentry, targetdir) The code in shutil.py is : def copystat(src, dst): """Copy all stat info (mode bits, atime and mtime) from src to dst""" st = os.stat(src) mode = stat.S_IMODE(st.st_mode) if hasattr(os, 'utime'): os.utime(dst, (st.st_atime, st.st_mtime)) if hasattr(os, 'chmod'): os.chmod(dst, mode) Additionally, after running a couple of times I get core dumps when trying to run the interactive interpreter. Oddly IDLE and other programs still run. However this is after installing py2exe, wxPython and pywin32 (all of which have Python 2.5 builds), so there is possibly some weird interaction. My install directory has the following three dlls in it : MSVCIRT.dll MSVCP60.dll MSVCRT.dll It has none of the msvc7 dlls that I would expect. I'm now at work and so don't have time to experiment, but tonight I will try uninstalling all my Python 2.5 stuff and re-installing just 2.5b2. I will see what dlls are present, run the interpreter a few times, and test a simple script with shutil.copytree. All the best, Michael Foord http://www.voidspace.org.uk/python/index.shtml From nmm1 at cus.cam.ac.uk Thu Jul 20 12:03:41 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Thu, 20 Jul 2006 11:03:41 +0100 Subject: [Python-Dev] Strategy for converting the decimal module to C Message-ID: <E1G3VNd-0003SW-6E@draco.cus.cam.ac.uk> Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > > But couldn't you just put in an interrupt handler that > counts the interrupts, for the purpose of measurement? No, but the reasons are very arcane. The general reason is that taking an interrupt handler and returning is not transparent, and is often not possible on modern systems. If that problem is at the hardware level (as on the Alpha and 3086/7), you are stuffed. But, more often, it is due the the fact that the architecture means that such handling can only be done at maximally privileged level. Now, interrupting into that level has to be transparent, in order to support TLB misses, clock interrupts, device interrupts, machine-check interrupts and so on. But the kernels rarely support transparent callbacks from that state into user code (though they used to); it is actually VERY hard to do, and even the mainframes had problems. This very commonly means that such counting breaks other facilities, unless it is done IN the privileged code. Of course, a GOOD hardware architecture wouldn't leave the process state when it gets a floating-point interrupt, but would just invoke an asynchronous routine call. That used to be done. As I said, none of this is directly relevant to emulated implementations, such as the current Python ones, but it IS to the design of an arithmetic specification. It could become relevant if Python wants to start to use a hardware implementation, because your proposal would mean that it would have to try to ensure that such callbacks are transparent. As one of the few people still working who has extensive experience with doing that, I can assure you that it is an order of magnitude fouler than you can imagine. A decimal order of magnitude :-( But note that I wasn't saying that such things should be put into the API, merely that there is a very good reason to do so for hardware implementations and ones used to tune code for such implementations. Personally, I wouldn't bother. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From skip at pobox.com Thu Jul 20 12:09:38 2006 From: skip at pobox.com (skip at pobox.com) Date: Thu, 20 Jul 2006 05:09:38 -0500 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <20060719212919.GD15030@abulafia.devel.redhat.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <20060718185238.GB363@abulafia.devel.redhat.com> <200607181455.56535.fdrake@acm.org> <20060719212919.GD15030@abulafia.devel.redhat.com> Message-ID: <17599.22114.395451.103228@montanaro.dyndns.org> Misa> Good point. Does the attached patch look reasonable? ... Misa> - self.when = string.upper(when) Misa> + self.when = unicode(when).upper() ... The use of the string module instead of string methods suggests to me that the logging package attempts to work with older versions of Python. Looking at PEP 291 it looks like 1.5.2 compatibility is desired (no string methods, no unicode). I think a conscious decision by someone (probably Vinay Sajip) to give up that compatibility would be required. Skip From rasky at develer.com Thu Jul 20 12:36:35 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 20 Jul 2006 12:36:35 +0200 Subject: [Python-Dev] new security doc using object-capabilities References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> Message-ID: <012801c6abe8$5f9524d0$d503030a@trilan> Brett Cannon wrote: >> The new doc is named securing_python.txt and >> can be >> found through the svn web interface at >> http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log. How do you plan to handle CPU-hogs? Stuff like execution of a gigantic integer multiplication. This recipe for safe_eval: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496746 which is otherwise very cute, does not handle this case as well: it tries to catch and interrupt long-running operations through a secondary thread, but fails on a single long operation because the GIL is not released and the alarm thread does not get its chance to run. -- Giovanni Bajo From nmm1 at cus.cam.ac.uk Thu Jul 20 13:10:13 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Thu, 20 Jul 2006 12:10:13 +0100 Subject: [Python-Dev] new security doc using object-capabilities Message-ID: <E1G3WQ1-0007C7-Ov@libra.cus.cam.ac.uk> "Giovanni Bajo" <rasky at develer.com> wrote: > > This recipe for safe_eval: > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496746 > which is otherwise very cute, does not handle this case as well: it tries to > catch and interrupt long-running operations through a secondary thread, but > fails on a single long operation because the GIL is not released and the > alarm thread does not get its chance to run. Grin :-) You have put your finger on the Great Myth of such virtualisations, which applies to the system-level ones and even to the hardware-level ones. In practice, there is always some request that a sandbox can make to the hypervisor that can lock out or otherwise affect other sandboxes. The key is, of course, to admit that and to specify what is and is not properly virtualised, so that the consequences can at least be analysed. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From rasky at develer.com Thu Jul 20 13:40:06 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 20 Jul 2006 13:40:06 +0200 Subject: [Python-Dev] new security doc using object-capabilities References: <E1G3WQ1-0007C7-Ov@libra.cus.cam.ac.uk> Message-ID: <030001c6abf1$3f231690$d503030a@trilan> Nick Maclaren wrote: >> This recipe for safe_eval: >> http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/496746 >> which is otherwise very cute, does not handle this case as well: it >> tries to catch and interrupt long-running operations through a >> secondary thread, but fails on a single long operation because the >> GIL is not released and the alarm thread does not get its chance to >> run. > > Grin :-) > > You have put your finger on the Great Myth of such virtualisations, > which applies to the system-level ones and even to the hardware-level > ones. In practice, there is always some request that a sandbox can > make to the hypervisor that can lock out or otherwise affect other > sandboxes. > > The key is, of course, to admit that and to specify what is and is > not properly virtualised, so that the consequences can at least be > analysed. I agree, and in fact Brett's work on a proper security model is greatly welcome. It's just that us mere mortals need to use eval() *now*, and that recipe is good enough for many practice uses. If you can't win, you can at least lose with dignity :) -- Giovanni Bajo From ncoghlan at gmail.com Thu Jul 20 13:43:21 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Jul 2006 21:43:21 +1000 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> Message-ID: <44BF6C59.6010707@gmail.com> For code objects, their construction is already commonly written as "compile(source)". For type objects, the constructor doesn't let you do anything you can't already do with a class statement. It doesn't need securing. For rewriting import.c in Python, the PEP 302 compliant import system API in pkgutil would be a good starting point. Your doc also asks about the imp.get_suffixes() list, and wonder where to set it from Python. As far as I am aware, you can't. get_suffixes() is built from _PyImport_FileTab, which is a C array. A switch statement is used to get from the file table entries to the appropriate handler functions. Quoting from the suggestions I put to the Py3k list: Use smarter data structures --------------------------- Currently, the individual handlers to load a fully identified module are exposed to Python code in a way that reflects the C-style data structures used in the current implementation. Simply switching to more powerful data structures for the file type handlers (i.e. use a PyTuple for filedescr values, a PyList for _PyImport_FileTab, and a PyDict instead of a switch statement to go from filedescr values to module loading/initialisation functions) and manipulating them all as normal Python objects could make the code in import.c much easier to follow. Extensible file type handling ----------------------------- If the file type handlers are stored in normal Python data structures as described above, it becomes feasible to make the import system extensible to different file types as well as to different file locations. This could be handled on a per-package basis, e.g. via a __file_types__ special attribute in packages. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From theller at python.net Thu Jul 20 14:26:44 2006 From: theller at python.net (Thomas Heller) Date: Thu, 20 Jul 2006 14:26:44 +0200 Subject: [Python-Dev] os.utime and os.chmod failures (etc) Python 2.5b2 In-Reply-To: <44BF490B.1090101@voidspace.org.uk> References: <44BF490B.1090101@voidspace.org.uk> Message-ID: <e9nsq5$6m6$1@sea.gmane.org> Michael Foord schrieb: > Hello all, > > There may be a reasonable cause for this (i.e. it is likely to be my > fault) - but it is consistent across two different machines I have tried > it on. > > With Python 2.5b2 (from the msi at Python.org), running on Windows XP > Pro SP2, ``os.utime`` and ``os.chmod`` fail with WindowsError. The same > code runs fine on Python 2.3 and Python 2.4. > > [err] shutil.copytree(thisentry, targetdir) > [err] File "C:\Python25\lib\shutil.py", line 130, in copytree > [err] copystat(src, dst) > [err] File "C:\Python25\lib\shutil.py", line 67, in copystat > [err] os.utime(dst, (st.st_atime, st.st_mtime)) > [err] WindowsError: [Error 13] Access is denied: 'lib\\Pmw' > [err] > > The script uses ``shutil.copytree`` to copy a directory (using relative > paths). IMO this is a bug in Python 2.5, on Windows. The problem is that the call to 'copystat(src, dst)' was added to the shutil.copytree function, in svn r38363 probably. It will fail always on Windows, since os.utime does not work on directories (as the docs correctly explain). I guess that a patch similar to this one should fix it: Index: shutil.py =================================================================== --- shutil.py (Revision 50710) +++ shutil.py (Arbeitskopie) @@ -127,7 +127,12 @@ # continue with other files except Error, err: errors.extend(err.args[0]) - copystat(src, dst) + try: + copystat(src, dst) + except WindowsError: + pass + except OSError, err: + errors.extend(err.args[0]) if errors: raise Error, errors But you should report this to the bug tracker. Thomas From fuzzyman at voidspace.org.uk Thu Jul 20 14:43:48 2006 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 20 Jul 2006 13:43:48 +0100 Subject: [Python-Dev] os.utime and os.chmod failures (etc) Python 2.5b2 In-Reply-To: <e9nsq5$6m6$1@sea.gmane.org> References: <44BF490B.1090101@voidspace.org.uk> <e9nsq5$6m6$1@sea.gmane.org> Message-ID: <44BF7A84.2010908@voidspace.org.uk> Thomas Heller wrote: > Michael Foord schrieb: > >> Hello all, >> >> There may be a reasonable cause for this (i.e. it is likely to be my >> fault) - but it is consistent across two different machines I have tried >> it on. >> >> With Python 2.5b2 (from the msi at Python.org), running on Windows XP >> Pro SP2, ``os.utime`` and ``os.chmod`` fail with WindowsError. The same >> code runs fine on Python 2.3 and Python 2.4. >> >> [err] shutil.copytree(thisentry, targetdir) >> [err] File "C:\Python25\lib\shutil.py", line 130, in copytree >> [err] copystat(src, dst) >> [err] File "C:\Python25\lib\shutil.py", line 67, in copystat >> [err] os.utime(dst, (st.st_atime, st.st_mtime)) >> [err] WindowsError: [Error 13] Access is denied: 'lib\\Pmw' >> [err] >> >> The script uses ``shutil.copytree`` to copy a directory (using relative >> paths). >> > > IMO this is a bug in Python 2.5, on Windows. > The problem is that the call to 'copystat(src, dst)' was added to > the shutil.copytree function, in svn r38363 probably. It will fail > always on Windows, since os.utime does not work on directories (as the > docs correctly explain). > > I guess that a patch similar to this one should fix it: > > Index: shutil.py > =================================================================== > --- shutil.py (Revision 50710) > +++ shutil.py (Arbeitskopie) > @@ -127,7 +127,12 @@ > # continue with other files > except Error, err: > errors.extend(err.args[0]) > - copystat(src, dst) > + try: > + copystat(src, dst) > + except WindowsError: > + pass > + except OSError, err: > + errors.extend(err.args[0]) > if errors: > raise Error, errors > > But you should report this to the bug tracker. > Ok, thanks. Michael http://www.voidspace.org.uk/python/index.shtml > Thomas > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From mihaiibanescu at yahoo.com Thu Jul 20 15:49:05 2006 From: mihaiibanescu at yahoo.com (Mihai Ibanescu) Date: Thu, 20 Jul 2006 09:49:05 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <17599.22114.395451.103228@montanaro.dyndns.org> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <20060718185238.GB363@abulafia.devel.redhat.com> <200607181455.56535.fdrake@acm.org> <20060719212919.GD15030@abulafia.devel.redhat.com> <17599.22114.395451.103228@montanaro.dyndns.org> Message-ID: <20060720134905.GA25083@abulafia.devel.redhat.com> On Thu, Jul 20, 2006 at 05:09:38AM -0500, skip at pobox.com wrote: > > Misa> Good point. Does the attached patch look reasonable? > > ... > Misa> - self.when = string.upper(when) > Misa> + self.when = unicode(when).upper() > ... > > The use of the string module instead of string methods suggests to me that > the logging package attempts to work with older versions of Python. Looking > at PEP 291 it looks like 1.5.2 compatibility is desired (no string methods, > no unicode). I think a conscious decision by someone (probably Vinay Sajip) > to give up that compatibility would be required. Agreed. There is a note that the module should work with python >= 1.5.2 at the top of the module. It's up to Vinay to decide if we want to drop support for 1.5.2 in the module included in newer pythons, or the attached patch would make it work for 1.5.2 as well (as in "it's not more broken than before"). I would like to redo the patch once more to get rid of the try-except and use __builtins__ instead (but for some reason it kept jumping from being a module to being a dictionary and I just wanted the proof of concept). Misa -------------- next part -------------- --- Python-2.4.3/Lib/logging/handlers.py.nolocale 2006-07-19 12:15:46.000000000 -0400 +++ Python-2.4.3/Lib/logging/handlers.py 2006-07-20 09:45:57.000000000 -0400 @@ -162,7 +162,7 @@ """ def __init__(self, filename, when='h', interval=1, backupCount=0, encoding=None): BaseRotatingHandler.__init__(self, filename, 'a', encoding) - self.when = string.upper(when) + self.when = _upper(when) self.backupCount = backupCount # Calculate the real rollover interval, which is just the number of # seconds between rollovers. Also set the filename suffix used when @@ -645,7 +645,7 @@ """ msg = self.log_format_string % ( self.encodePriority(self.facility, - string.lower(record.levelname)), + _lower(record.levelname)), msg) try: if self.unixsocket: @@ -854,7 +854,7 @@ ("GET" or "POST") """ logging.Handler.__init__(self) - method = string.upper(method) + method = _upper(method) if method not in ["GET", "POST"]: raise ValueError, "method must be GET or POST" self.host = host @@ -1007,3 +1007,25 @@ self.flush() self.target = None BufferingHandler.close(self) + +def _upper(s): + """A version of upper() that tries to be locale-independent by converting + the string to unicode (which is not subject to case conversion being + locale specific) + """ + try: + ret = str(unicode(s).upper()) + except NameError: + ret = string.upper(s) + return ret + +def _lower(s): + """A version of lower() that tries to be locale-independent by converting + the string to unicode (which is not subject to case conversion being + locale specific) + """ + try: + ret = str(unicode(s).lower()) + except NameError: + ret = string.lower(s) + return ret From jimjjewett at gmail.com Thu Jul 20 17:39:27 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 20 Jul 2006 11:39:27 -0400 Subject: [Python-Dev] logging module broken because of locale Message-ID: <fb6fbf560607200839y75e6414eqe0f79c985999db60@mail.gmail.com> Mihai, It does make sense to document this limit for people writing subclasses, or using a Turkic codeset. I'm not sure that logging is the right place to document it, and I don't think changing the base classes is a good idea. TimedRotatingFileHandler and HTTPHandler restrict their input to a tiny subset of ASCII; anything that would be handled differently after these changes will already raise a ValueError unless you subclass. Even SysLogHandler.emit doesn't actually print the string; it is only used as a lookup key for a dictionary whose keys are all lower-case ASCII. In theory, you could monkey-patch that dictionary to add additional values, but then you might as well subclass to do the right thing with your new keys. (It *might* make sense to change the base function and dictionary to accept unicode, including undotted-i synonyms.) -jJ From martin at v.loewis.de Thu Jul 20 18:08:05 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 20 Jul 2006 18:08:05 +0200 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <20060720134905.GA25083@abulafia.devel.redhat.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <20060718185238.GB363@abulafia.devel.redhat.com> <200607181455.56535.fdrake@acm.org> <20060719212919.GD15030@abulafia.devel.redhat.com> <17599.22114.395451.103228@montanaro.dyndns.org> <20060720134905.GA25083@abulafia.devel.redhat.com> Message-ID: <44BFAA65.5040501@v.loewis.de> Mihai Ibanescu wrote: > It's up to Vinay to decide if we want to drop support for 1.5.2 in the module > included in newer pythons, or the attached patch would make it work for 1.5.2 > as well (as in "it's not more broken than before"). That still wouldn't work with Python 1.5.2, as that version did not support Unicode at all. Regards, Martin From mihaiibanescu at yahoo.com Thu Jul 20 18:13:14 2006 From: mihaiibanescu at yahoo.com (Mihai Ibanescu) Date: Thu, 20 Jul 2006 12:13:14 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <44BFAA65.5040501@v.loewis.de> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <20060718185238.GB363@abulafia.devel.redhat.com> <200607181455.56535.fdrake@acm.org> <20060719212919.GD15030@abulafia.devel.redhat.com> <17599.22114.395451.103228@montanaro.dyndns.org> <20060720134905.GA25083@abulafia.devel.redhat.com> <44BFAA65.5040501@v.loewis.de> Message-ID: <20060720161314.GI25083@abulafia.devel.redhat.com> On Thu, Jul 20, 2006 at 06:08:05PM +0200, "Martin v. L?wis" wrote: > Mihai Ibanescu wrote: > > It's up to Vinay to decide if we want to drop support for 1.5.2 in the module > > included in newer pythons, or the attached patch would make it work for 1.5.2 > > as well (as in "it's not more broken than before"). > > That still wouldn't work with Python 1.5.2, as that version did not > support Unicode at all. Yes, as I said, it won't be more broken than before applying the patch (my first patch was breaking 1.5.2 completely). If people choose to compile python 2.4.3 without unicode support, it won't work either. But if you choose to disable unicode you probably live in a very constrained environment and you may not be affected by the locale bug at all. I agree the patch is not perfect :-) but I think it solves the problem for most people. Misa From mihaiibanescu at yahoo.com Thu Jul 20 18:19:02 2006 From: mihaiibanescu at yahoo.com (mihaiibanescu at yahoo.com) Date: Thu, 20 Jul 2006 12:19:02 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <fb6fbf560607200839y75e6414eqe0f79c985999db60@mail.gmail.com> References: <fb6fbf560607200839y75e6414eqe0f79c985999db60@mail.gmail.com> Message-ID: <20060720161902.GJ25083@abulafia.devel.redhat.com> On Thu, Jul 20, 2006 at 11:39:27AM -0400, Jim Jewett wrote: > Mihai, > > It does make sense to document this limit for people writing > subclasses, or using a Turkic codeset. I'm not sure that logging is > the right place to document it, and I don't think changing the base > classes is a good idea. > > TimedRotatingFileHandler and HTTPHandler restrict their input to a > tiny subset of ASCII; anything that would be handled differently after > these changes will already raise a ValueError unless you subclass. > > Even SysLogHandler.emit doesn't actually print the string; it is only > used as a lookup key for a dictionary whose keys are all lower-case > ASCII. In theory, you could monkey-patch that dictionary to add > additional values, but then you might as well subclass to do the right > thing with your new keys. (It *might* make sense to change the base > function and dictionary to accept unicode, including undotted-i > synonyms.) Hi Jim, I am afraid you lost me. My initial approach was to compute the values of the dictionary based on the current locale - and that doesn't work. The issue is not that the dictionary doesn't accept unicode, it's that there's no reliable way to do lookups in it. If you could sketch an example of what you had in mind with the monkey-patching, please do so, since I don't think I grasped the idea. Misa From martin at v.loewis.de Thu Jul 20 18:41:04 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 20 Jul 2006 18:41:04 +0200 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <20060720161314.GI25083@abulafia.devel.redhat.com> References: <20060717193955.GK3699@abulafia.devel.redhat.com> <44BD2054.4020705@v.loewis.de> <20060718185238.GB363@abulafia.devel.redhat.com> <200607181455.56535.fdrake@acm.org> <20060719212919.GD15030@abulafia.devel.redhat.com> <17599.22114.395451.103228@montanaro.dyndns.org> <20060720134905.GA25083@abulafia.devel.redhat.com> <44BFAA65.5040501@v.loewis.de> <20060720161314.GI25083@abulafia.devel.redhat.com> Message-ID: <44BFB220.8060802@v.loewis.de> Mihai Ibanescu wrote: > Yes, as I said, it won't be more broken than before applying the patch (my > first patch was breaking 1.5.2 completely). Ah, I didn't notice that it deals with unicode() not being a builtin. That's fine then. Regards, Martin From brett at python.org Thu Jul 20 19:04:43 2006 From: brett at python.org (Brett Cannon) Date: Thu, 20 Jul 2006 10:04:43 -0700 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <012801c6abe8$5f9524d0$d503030a@trilan> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <012801c6abe8$5f9524d0$d503030a@trilan> Message-ID: <bbaeab100607201004j324b3856jb43b3cc3c3ec22c8@mail.gmail.com> On 7/20/06, Giovanni Bajo <rasky at develer.com> wrote: > > Brett Cannon wrote: > > >> The new doc is named securing_python.txt and > >> can be > >> found through the svn web interface at > >> > > http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log > . > > How do you plan to handle CPU-hogs? Stuff like execution of a gigantic > integer multiplication. I don't. =) Protecting the CPU is damn hard to do in any form of portable fashion. And even getting it to work on an OS you do know the details of leads to probably an interrupt implementation and that doesn't sound fun. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/4b9938f9/attachment.html From jimjjewett at gmail.com Thu Jul 20 19:06:32 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 20 Jul 2006 13:06:32 -0400 Subject: [Python-Dev] logging module broken because of locale In-Reply-To: <20060720161902.GJ25083@abulafia.devel.redhat.com> References: <fb6fbf560607200839y75e6414eqe0f79c985999db60@mail.gmail.com> <20060720161902.GJ25083@abulafia.devel.redhat.com> Message-ID: <fb6fbf560607201006k171f810am91b1425b02dfddfd@mail.gmail.com> On 7/20/06, mihaiibanescu at yahoo.com <mihaiibanescu at yahoo.com> wrote: > On Thu, Jul 20, 2006 at 11:39:27AM -0400, Jim Jewett wrote: > > Even SysLogHandler.emit doesn't actually print the string; it is only > > used as a lookup key for a dictionary whose keys are all lower-case > > ASCII. In theory, you could monkey-patch that dictionary to add > > additional values, but then you might as well subclass to do the right > > thing with your new keys. (It *might* make sense to change the base > > function and dictionary to accept unicode, including undotted-i > > synonyms.) > My initial approach was to compute the values of the dictionary based on the > current locale - and that doesn't work. Correct, it would need to be done by hand. As part of the current class construction, SysLogHandler.priority_names["info"] = SysLogHandler.LOG_INFO You could add other entries, so that SysLogHandler.priority_names["my_info"] = SysLogHandler.LOG_INFO If you also changed SysLogHandler.encode_priority to accept unicode keys, you could even add u"info" and the equivalent with an undotted-i. I'm not sure this is worth doing, though, since they supposedly represent symbolic constants. -jJ > The issue is not that the dictionary doesn't accept unicode, it's that there's > no reliable way to do lookups in it. If you could sketch an example of what > you had in mind with the monkey-patching, please do so, since I don't think I > grasped the idea. > > Misa > From brett at python.org Thu Jul 20 19:09:07 2006 From: brett at python.org (Brett Cannon) Date: Thu, 20 Jul 2006 10:09:07 -0700 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <44BF6C59.6010707@gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <44BF6C59.6010707@gmail.com> Message-ID: <bbaeab100607201009x28a7354elde374c6651b1a322@mail.gmail.com> On 7/20/06, Nick Coghlan <ncoghlan at gmail.com> wrote: > > For code objects, their construction is already commonly written as > "compile(source)". Right, but some people like to construct directly from bytecode. For type objects, the constructor doesn't let you do anything you can't > already do with a class statement. It doesn't need securing. I figured as much, but when I was making the list I was not sure and didn't want to stop my writing momentum to check. For rewriting import.c in Python, the PEP 302 compliant import system API in > pkgutil would be a good starting point. Yep. Plan on looking at all of the various modules in the stdlib that assist with importing, package PEP (I think there is one), and PEP 302. Your doc also asks about the imp.get_suffixes() list, and wonder where to > set > it from Python. > > As far as I am aware, you can't. get_suffixes() is built from > _PyImport_FileTab, which is a C array. A switch statement is used to get > from > the file table entries to the appropriate handler functions. Ah, OK. Quoting from the suggestions I put to the Py3k list: > > Use smarter data structures > --------------------------- > Currently, the individual handlers to load a fully identified module are > exposed to Python code in a way that reflects the C-style data structures > used > in the current implementation. > > Simply switching to more powerful data structures for the file type > handlers > (i.e. use a PyTuple for filedescr values, a PyList for _PyImport_FileTab, > and > a PyDict instead of a switch statement to go from filedescr values to > module > loading/initialisation functions) and manipulating them all as normal > Python > objects could make the code in import.c much easier to follow. Yep. I just kind of glanced at the rest of your suggestions, Nick, since I assumed a lot of it would change (or could be changed) if import was redone in as much Python as possible. Extensible file type handling > ----------------------------- > If the file type handlers are stored in normal Python data structures as > described above, it becomes feasible to make the import system extensible > to > different file types as well as to different file locations. Yep. Although I am more interested in restricting than broadening the file types. This could be handled on a per-package basis, e.g. via a __file_types__ > special attribute in packages. Maybe. I don't want to get into introducing new abilities to start, though. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/32151d26/attachment-0001.htm From brett at python.org Thu Jul 20 19:30:03 2006 From: brett at python.org (Brett Cannon) Date: Thu, 20 Jul 2006 10:30:03 -0700 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com> Message-ID: <bbaeab100607201030l2842b416wcfd8007c86c10e4f@mail.gmail.com> On 7/20/06, Lawrence Oluyede <l.oluyede at gmail.com> wrote: > > That's great. I just read your draft but I have little comments to do > but before let me say that I liked the idea to borrow concepts from E. > I've crossed the E's path in the beginning of this year and I found it > a pot of really nice ideas (for promises and capabilities). Here are > my comments about the draft: > > - it's not really clear to me what the "powerbox" is. I think I got > the concept of "super process" but maybe it's to be clarified, isn't > it? It become clear in the "threat model" paragraph The powerbox is the thing that gives your security domains their initial abilities. The OS gives the process its abilities, but it does not directly work with the interpreter. Since the process does, though, it is considered the powerbox and farms out abilities that it has been given by the OS. I have tried to clarify the definition at the start of the doc. - I hope no Rubystas will read the "Problem of No Private Namespace" > section because they have private/protected keywords to enforce this > stuff :-) Writing proxies in C will slow down the dev process (altough > will speed up the performance maybe) but in a far future someone will > come up with an alternative closer to the Python level Maybe. As I said in the doc, any changes must be Pythonic and adding private namespaces right now wouldn't be without much more thought and work. And if Ruby ends up with this security model but more thoroughly, more power to them. Their language is different in the right ways to support it. As for coding in C, thems the breaks. I plan in adding stuff to the stdlib for the common case. I might eventually think of a good, generic proxy object that could be used, but as of right now I am not worrying about that since it would be icing on the cake. - Can you write down a simple example of what you mean with "changing > something of the built-in objects"? (in "Problem of mutable shared > state") Done. - What about the performance issues of the capabilities model overall? Should be faster than an IBAC model since certain calls will not need to check the identity of the caller every time. But I am not worrying about performance, I am worrying about correctness, so I did not try to make any performance claims. - I know what you meant to say but the paragraph about pythonicness > and the security model seems a little "fuzzy" to me. Which are the > boundaries of the allowed changes for the security stuff? Being "pythonic" is a fuzzy term in itself and Guido is the only person who can make definitive claims over what is and is not Pythonic. As I have said, this doc was mostly written with python-dev in mind since they are the ones I have to convince to let this into the core and they all know the term. But I have tacked in a sentence on what the term means. - You don't say anything about networking and networked resources in > the list of the standard sandboxed interpreter Nope. Have not started worrying about that yet. Just trying to get the basic model laid out. - Suppose we have a .py module. Based on your security model we can > import it, right? When imported it generates a .pyc file. The second > time we import it what happens? .pyc is ignored? import is not allowed > at all? We can't rely on the name of the file.pyc because an attacker > who knows the file.py is secure and the second import is done against > file.pyc can replace the "secure" file.pyc with an implementation not > secure and can do some kind of harm to the sandbox It will be ignored. But I am hoping that through rewriting the import machinery more control over generating .pyc files can be had (see Skip Montanaro's PEP on this; forget the number). This is why exact details were left out of the implementation details. I just wanted people understand the approach to everything, not the concrete details of how it will be coded up. - About "Filesystem information". Does the sandboxed interpreter need > to know all that information about file paths, files and so on? Can't > we reset those attributes to something arbitrary? That is the point. It is not that the sandbox needs to know it, its that it needs to be hidden from the sandbox. - About sys module: I think the best way is to have a purged fake sys > module with only the stuff you need. pypy has the concept of faked > modules too (altough for a different reason) OK. - About networking: what do you think about the E's model of really > safe networking, protected remotable objects and safe RPC? Is that > model applicable to Python's in some way? We can't use the E's model > as a whole (ask people to generate a safe key and send it by email is > unfeasible) I have not looked at it. I am also not trying to build an RPC system *and* a security model for Python. That is just too much work right now. - is the protected memory model a some kind of memory monitor system? Basically. It just keeps a size_t on the memory cap and another on memory usage, and when memory is requested it makes sure that it won't go over the cap. And when memory is freed the usage goes down. It's very rough (hard to account for padding bits, etc. in C structs), but it should be good enough to prevent a program from hitting 800 MB when you really just wanted it to have 5 MB. I think that's all for the draft. I wrote these comments during the > reading of the document. > > Hope some of these help Thanks, Lawrence. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/77ae9261/attachment.html From rasky at develer.com Thu Jul 20 19:40:46 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 20 Jul 2006 19:40:46 +0200 Subject: [Python-Dev] new security doc using object-capabilities References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <012801c6abe8$5f9524d0$d503030a@trilan> <bbaeab100607201004j324b3856jb43b3cc3c3ec22c8@mail.gmail.com> Message-ID: <084e01c6ac23$a1930ec0$d503030a@trilan> Brett Cannon wrote: >> http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log >> . >> >> How do you plan to handle CPU-hogs? Stuff like execution of a >> gigantic integer multiplication. > > > I don't. =) Protecting the CPU is damn hard to do in any form of > portable fashion. And even getting it to work on an OS you do know > the details of leads to probably an interrupt implementation and > that doesn't sound fun. I think the trick used by the safe_eval recipe (a separate thread which interrupts the script through thread.interrupt_main()) shows that, in most cases, it's possible to make sure that an embedded script does not take too long to execute. Do you agree that this usage case ("allow me to timeout an embedded script") is something which would be a very good start in the right direction? Now, I wonder, in a restricted execution environment such as that depicted in your document, how many different ways are there to make the Python interpreter enter a long calcolation loop which does not release the GIL? I can think of bignum*bignum, bignum**bignum or similar mathematical operations, but there are really a few. If we could make those release the GIL (or poll some kind of watchdog used to abort them, pretty much like they normally poll CTRL+C), then the same trick used by the recipe could be used. -- Giovanni Bajo From brett at python.org Thu Jul 20 20:10:29 2006 From: brett at python.org (Brett Cannon) Date: Thu, 20 Jul 2006 11:10:29 -0700 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <084e01c6ac23$a1930ec0$d503030a@trilan> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <012801c6abe8$5f9524d0$d503030a@trilan> <bbaeab100607201004j324b3856jb43b3cc3c3ec22c8@mail.gmail.com> <084e01c6ac23$a1930ec0$d503030a@trilan> Message-ID: <bbaeab100607201110i4e4c924fvbc57425a7e4b3a36@mail.gmail.com> On 7/20/06, Giovanni Bajo <rasky at develer.com> wrote: > > Brett Cannon wrote: > > >> > > http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log > >> . > >> > >> How do you plan to handle CPU-hogs? Stuff like execution of a > >> gigantic integer multiplication. > > > > > > I don't. =) Protecting the CPU is damn hard to do in any form of > > portable fashion. And even getting it to work on an OS you do know > > the details of leads to probably an interrupt implementation and > > that doesn't sound fun. > > I think the trick used by the safe_eval recipe (a separate thread which > interrupts the script through thread.interrupt_main()) shows that, in most > cases, it's possible to make sure that an embedded script does not take > too > long to execute. Do you agree that this usage case ("allow me to timeout > an > embedded script") is something which would be a very good start in the > right > direction? Probably. I just don't feel like worrying about it right now. =) Now, I wonder, in a restricted execution environment such as that depicted > in your document, how many different ways are there to make the Python > interpreter enter a long calcolation loop which does not release the GIL? > I > can think of bignum*bignum, bignum**bignum or similar mathematical > operations, but there are really a few. If we could make those release the > GIL (or poll some kind of watchdog used to abort them, pretty much like > they > normally poll CTRL+C), then the same trick used by the recipe could be > used. Well, any work that does most of its calculation within C code and that does not touch base with the interpreter on a semi-regular basis would need to relesae the GIL. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/d60aea40/attachment.htm From pje at telecommunity.com Thu Jul 20 20:57:07 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 20 Jul 2006 14:57:07 -0400 Subject: [Python-Dev] Undocumented PEP 302 protocol change by need-for-speed sprint Message-ID: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com> While investigating the need to apply http://python.org/sf/1525766 I found that there was a modification to pkgutil during the need-for-speed sprint that affects the PEP 302 protocol in a backwards incompatible way. Specifically, PEP 302 documents that path_importer_cache always contains either importer objects or None. Any code written to obtain importer objects is therefore now broken, because import.c is slapping False in for non-existent filesystem paths. The pkgutil module was then hacked to work around this problem, thereby hiding the breakage from at least the standard library, but not any external libraries that follow the PEP 302 protocol to find importers. There are several options as to how to proceed: 1. Revert the change 2. Document the breakage, update PEP 302, and make everybody update their code 3. Make it not break existing code, by using a NonexistentPathImporter or NullImporter type in place of "False" in sys.path_importer_cache. Any thoughts? Personally, the only code I know of that implements the PEP 302 protocol besides the pkgutil module that would be affected is pkg_resources in setuptools, so it's not like I can't fix it for 2.5. However, I don't know if anybody else is using the protocol, and if so, how bad the breakage would be. This should really only affect code that is walking sys.path, because paths with "False" in sys.path_importer_cache by definition cannot have any importable modules associated with them. So, although I don't like option 2 on general principles, it may be an acceptable solution. From tjreedy at udel.edu Thu Jul 20 21:00:45 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 20 Jul 2006 15:00:45 -0400 Subject: [Python-Dev] new security doc using object-capabilities References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com> Message-ID: <e9ojsu$ui1$1@sea.gmane.org> "Lawrence Oluyede" <l.oluyede at gmail.com> wrote in message news:9eebf5740607200117r4d4613e2i91665ea211bab46 at mail.gmail.com... > - I know what you meant to say but the paragraph about pythonicness > and the security model seems a little "fuzzy" to me. I agree that this paragraph is weak and recommend that it be rewritten. In particular, I think the 'pythonic*' words should go, especially if you expect this document to be read by anyone other than dedicated pythonistas. I would start with something like "It is my goal that my thesis work be incorporated in some future version of the Python distribution. This has two constraints. First, changes to the core must not slow down normal operation. Second, visible changes must not violate the spirit and style of Python that make it a distinctive language." This alludes to the fact that your proposal discusses two highly overlapping yet separate projects: write a thesis that gains you a PhD degree; and produce an accepted patch set that give Python a useful security capability it does not now have. They have to be thought of as somewhat separate because you have two sets of 'overseers' and approvers: your thesis advisor and committee for the first; and Guido and other Python developers for the second. I think your thesis should currently be your first priority. Your current paragraph implied to me that you would not follow a promising line of research if you could not see how to make it 'pythonic'. If I were on your thesis committee, I think that would bother me ;-). In any case, I wish you the best with a double project that is obviously not a 'gimme'. Terry Jan Reedy (PhD, though not on any thesis committees) From l.oluyede at gmail.com Thu Jul 20 21:20:17 2006 From: l.oluyede at gmail.com (Lawrence Oluyede) Date: Thu, 20 Jul 2006 21:20:17 +0200 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607201030l2842b416wcfd8007c86c10e4f@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <9eebf5740607200117r4d4613e2i91665ea211bab46@mail.gmail.com> <bbaeab100607201030l2842b416wcfd8007c86c10e4f@mail.gmail.com> Message-ID: <9eebf5740607201220t3edcb31k4b08d6b27f1546d6@mail.gmail.com> > Should be faster than an IBAC model since certain calls will not need to > check the identity of the caller every time. > > But I am not worrying about performance, I am worrying about correctness, so > I did not try to make any performance claims. Got that. > Nope. Have not started worrying about that yet. Just trying to get the > basic model laid out. Ok sorry to have bothered > That is the point. It is not that the sandbox needs to know it, its that it > needs to be hidden from the sandbox. So I think that's a "simple" step during the importing step. > I have not looked at it. I am also not trying to build an RPC system *and* > a security model for Python. That is just too much work right now. Ok sorry :-) > Thanks, Lawrence. Thank you! -- Lawrence http://www.oluyede.org/blog From barry at python.org Thu Jul 20 21:27:39 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 20 Jul 2006 15:27:39 -0400 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc In-Reply-To: <7FE62713-24B0-47CE-BA94-41150DB683F4@python.org> References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org> <EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org> <44B64782.9010706@gmail.com> <7FE62713-24B0-47CE-BA94-41150DB683F4@python.org> Message-ID: <AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 13, 2006, at 12:12 PM, Barry Warsaw wrote: > I've updated SF patch #1520294 and assigned it back to Georg for > another quick review. Neal commented in the patch that it might help to explain the implementation a bit. I'd like to do that and also explain why I think this patch should go in 2.5. OTOH, I suspect most people just don't care, which is why I've gotten almost no comments on the patch (other than one or two mild nods of approval). As a reminder, this patch is really about providing useful help() at the interactive prompt for two specific types of C implemented descriptors, getset and member descriptors. Both objects can be given docstrings in the C code, but currently nothing in the tool chain pulls out and displays those docstrings, so help() on such objects is not very useful. While certainly a minor point, I think it improves the user experience to be able to display any existing docstrings for these types of objects. Of course, I'm scratching my own itch here because our app is deeply embedded and we have a lot of getset and member descriptors with docstrings. I'd like for our users to be able to type "help(someobj.member)" at our interactive prompt and get our docstring printed back at them. The implementation is fairly straightforward I add two new constants to types.py, a couple of is*() methods to inspect.py, and a bit of extra support to pydoc.py. Where it gets complicated is that I added a _types builtin module to provide a getset and member descriptor that could be turned into *Type constants in the types.py module. Why did I do this instead of trying to hunt down some existing getset or member descriptor? For one thing, there really aren't very good candidates for such objects in the built-in modules. You can't use objects like datetime.timedelta.days in types.py because datetime is not importable early enough for types.py to use it. Even if there were likely candidates, they would be accidents of implementation and it doesn't seem like a good idea to force force some future datetime maintainer to have to fix types.py when she decides that datetime.timedelta.days should be implemented in some other way. A 3rd party extension module doesn't work either because you really need the tie-in from types.py, inspect.py, and pydoc.py. You certainly don't want to go poking new values into types.py from the outside, and besides inspect.py and pydoc.py also need to learn about these fundamental built-in Python types. ISTM the most straightforward approach is to provide these types in a built-in C module, linked into the interpreter so types.py can get access to it as early as it needs to. Also, because the type that these descriptors are attached to cannot be instantiated from Python, they should be quite benign, existing only in order to give type() something to query. Neal also questions the reliance on the types.py module and whether we're moving away from it. My own feeling is that certainly for common types available as type constructors in the built-in namespace (e.g. list, set, dict, etc.), no, we certainly don't need types.py. But there are enough other tools in Pythonland that need to introspect about types that aren't readily available, and I see no reason why types.py shouldn't continue to fulfill those needs. How else would you find the type of frames or generators in a simple, documented, consistent way? Neal also asks about other implementations. I believe that I've patched types.py, inspect.py, and pydoc.py in such a way that if the implementation did not have these types available, those modules would not break. For example, GetSetDescriptorType and MemberDescriptorType are not defined if _types can't be imported. If those types aren't defined, the appropriate inspect.py is() methods will always return False. Thus, other implementations can do hasattr () tests on types.py or better yet, inspect.isgetsetdescriptor() and get an answer it can deal with. Further, by providing a _types built-in module, other implementations /could/ expose those types, if they were meaningful. I believe that an _types module provides a good place for those alternative implementations to hide types of their own should the need arise (e.g. if Jython had some implementation-specific JavaSpecialUnicornType, the object would live in Jython's _types.java and be exposed via a type(_type.javaspecialunicorn) in types.py). Could this wait until Python 2.6? Well, it /could/ but IMO it shouldn't. Improving the user experience by providing useful help() shouldn't be postponed when we have the opportunity to do so now, with low risk of breakage. Probably the biggest risk is whether I can modify the Windows build to DTRT with _types.c. I've been building enough Windows s/w lately that I think I can handle it, and if not, I'll just buy the Timbot lunch as payment for his help. I know what he likes to eat. So anyway, that's probably more text than the patch combined. :) I hope it helps. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRL/ZL3EjvBPtnXfVAQKEvgQAktmdjKnq5uf1+m5Wc5F1ZFY/LhdM1vQQ A2v6zgXTxdQteqTNds15BktGAUwRNKKB2pU/U2wTcfLx/4rfjvkFG3cRbhEY8jI0 wa8wzPrGs17VTp8Z/LI65F9DC8aMEa9RBr2fXeuhewXlAheiZT9doelds8b5y0T/ Ed9Yl4MJ0es= =Fh1P -----END PGP SIGNATURE----- From brett at python.org Thu Jul 20 21:28:46 2006 From: brett at python.org (Brett Cannon) Date: Thu, 20 Jul 2006 12:28:46 -0700 Subject: [Python-Dev] Undocumented PEP 302 protocol change by need-for-speed sprint In-Reply-To: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com> References: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com> Message-ID: <bbaeab100607201228y797b601alac230a13c00c9b59@mail.gmail.com> On 7/20/06, Phillip J. Eby <pje at telecommunity.com> wrote: > > While investigating the need to apply http://python.org/sf/1525766 I found > that there was a modification to pkgutil during the need-for-speed sprint > that affects the PEP 302 protocol in a backwards incompatible way. > > Specifically, PEP 302 documents that path_importer_cache always contains > either importer objects or None. Any code written to obtain importer > objects is therefore now broken, because import.c is slapping False in for > non-existent filesystem paths. > > The pkgutil module was then hacked to work around this problem, thereby > hiding the breakage from at least the standard library, but not any > external libraries that follow the PEP 302 protocol to find importers. > > There are several options as to how to proceed: > > 1. Revert the change > 2. Document the breakage, update PEP 302, and make everybody update their > code > 3. Make it not break existing code, by using a NonexistentPathImporter or > NullImporter type in place of "False" in sys.path_importer_cache. > > Any thoughts? Revert it. Is it really that much of a bonus to use False over None? Both evaluate to false and both are already singleton so you can use 'is' for testing. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/3e21b965/attachment.htm From g.brandl at gmx.net Thu Jul 20 21:32:58 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 20 Jul 2006 21:32:58 +0200 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc In-Reply-To: <AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org> References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org> <EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org> <44B64782.9010706@gmail.com> <7FE62713-24B0-47CE-BA94-41150DB683F4@python.org> <AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org> Message-ID: <e9olpa$4ql$1@sea.gmane.org> Barry Warsaw wrote: > Why did I do this instead of trying to hunt down some existing getset > or member descriptor? For one thing, there really aren't very good > candidates for such objects in the built-in modules. You can't use > objects like datetime.timedelta.days in types.py because datetime is > not importable early enough for types.py to use it. Even if there > were likely candidates, they would be accidents of implementation and > it doesn't seem like a good idea to force force some future datetime > maintainer to have to fix types.py when she decides that > datetime.timedelta.days should be implemented in some other way. > > A 3rd party extension module doesn't work either because you really > need the tie-in from types.py, inspect.py, and pydoc.py. You > certainly don't want to go poking new values into types.py from the > outside, and besides inspect.py and pydoc.py also need to learn about > these fundamental built-in Python types. > > ISTM the most straightforward approach is to provide these types in a > built-in C module, linked into the interpreter so types.py can get > access to it as early as it needs to. Also, because the type that > these descriptors are attached to cannot be instantiated from Python, > they should be quite benign, existing only in order to give type() > something to query. Perhaps you could put the objects into _testcapi. That way no new module has to be deployed (is _testcapi installed on every system?) Georg From pje at telecommunity.com Thu Jul 20 21:37:25 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 20 Jul 2006 15:37:25 -0400 Subject: [Python-Dev] Undocumented PEP 302 protocol change by need-for-speed sprint In-Reply-To: <bbaeab100607201228y797b601alac230a13c00c9b59@mail.gmail.co m> References: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com> <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060720153329.0202d570@sparrow.telecommunity.com> At 12:28 PM 7/20/2006 -0700, Brett Cannon wrote: >On 7/20/06, Phillip J. Eby ><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote: >>While investigating the need to apply >><http://python.org/sf/1525766>http://python.org/sf/1525766 I found >>that there was a modification to pkgutil during the need-for-speed sprint >>that affects the PEP 302 protocol in a backwards incompatible way. >> >>Specifically, PEP 302 documents that path_importer_cache always contains >>either importer objects or None. Any code written to obtain importer >>objects is therefore now broken, because import.c is slapping False in for >>non-existent filesystem paths. >> >>The pkgutil module was then hacked to work around this problem, thereby >>hiding the breakage from at least the standard library, but not any >>external libraries that follow the PEP 302 protocol to find importers. >> >>There are several options as to how to proceed: >> >>1. Revert the change >>2. Document the breakage, update PEP 302, and make everybody update their >>code >>3. Make it not break existing code, by using a NonexistentPathImporter or >>NullImporter type in place of "False" in sys.path_importer_cache. >> >>Any thoughts? > >Revert it. Is it really that much of a bonus to use False over >None? Both evaluate to false and both are already singleton so you can >use 'is' for testing. The changed code still uses None. PEP 302 defines None as meaning that a sys.path entry does not have an importer. It's just that the need-for-speed patch *adds* the use of True and False. None still means "no importer", but True now means "no importer, path exists" and False now means "no importer, path does not exist". The idea is that import.c can then skip checking the existence of the path when it sees True or False, but it then means that code that gets data from path_importer_cache needs to know about these new special values, or else it will get an attribute error when it tries to call True.find_module(). From brett at python.org Thu Jul 20 21:58:42 2006 From: brett at python.org (Brett Cannon) Date: Thu, 20 Jul 2006 12:58:42 -0700 Subject: [Python-Dev] Undocumented PEP 302 protocol change by need-for-speed sprint In-Reply-To: <5.1.1.6.0.20060720153329.0202d570@sparrow.telecommunity.com> References: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com> <5.1.1.6.0.20060720153329.0202d570@sparrow.telecommunity.com> Message-ID: <bbaeab100607201258sba22a52u4d713c07f92c4775@mail.gmail.com> On 7/20/06, Phillip J. Eby <pje at telecommunity.com> wrote: > > At 12:28 PM 7/20/2006 -0700, Brett Cannon wrote: > >On 7/20/06, Phillip J. Eby > ><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote: > >>While investigating the need to apply > >><http://python.org/sf/1525766>http://python.org/sf/1525766 I found > >>that there was a modification to pkgutil during the need-for-speed > sprint > >>that affects the PEP 302 protocol in a backwards incompatible way. > >> > >>Specifically, PEP 302 documents that path_importer_cache always contains > >>either importer objects or None. Any code written to obtain importer > >>objects is therefore now broken, because import.c is slapping False in > for > >>non-existent filesystem paths. > >> > >>The pkgutil module was then hacked to work around this problem, thereby > >>hiding the breakage from at least the standard library, but not any > >>external libraries that follow the PEP 302 protocol to find importers. > >> > >>There are several options as to how to proceed: > >> > >>1. Revert the change > >>2. Document the breakage, update PEP 302, and make everybody update > their > >>code > >>3. Make it not break existing code, by using a NonexistentPathImporter > or > >>NullImporter type in place of "False" in sys.path_importer_cache. > >> > >>Any thoughts? > > > >Revert it. Is it really that much of a bonus to use False over > >None? Both evaluate to false and both are already singleton so you can > >use 'is' for testing. > > The changed code still uses None. PEP 302 defines None as meaning that a > sys.path entry does not have an importer. It's just that the > need-for-speed patch *adds* the use of True and False. None still means > "no importer", but True now means "no importer, path exists" and False now > means "no importer, path does not exist". Ah. Sounds like None is not really even needed with the change (although I am not suggesting the removal of None). The idea is that import.c can then skip checking the existence of the path > when it sees True or False, but it then means that code that gets data > from > path_importer_cache needs to know about these new special values, or else > it will get an attribute error when it tries to call True.find_module(). > > Well, I have not played with the PEP 302 stuff so I don't know how helpful they are to have around. But it is definitely a semantic change that either needs to be reverted or documented. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/61ed8b42/attachment.html From glyph at divmod.com Thu Jul 20 22:17:46 2006 From: glyph at divmod.com (glyph at divmod.com) Date: Thu, 20 Jul 2006 16:17:46 -0400 Subject: [Python-Dev] Undocumented PEP 302 protocol change by need-for-speed sprint In-Reply-To: <5.1.1.6.0.20060720144252.02622318@sparrow.telecommunity.com> Message-ID: <20060720201746.29014.684853232.divmod.quotient.46224@ohm> On Thu, 20 Jul 2006 14:57:07 -0400, "Phillip J. Eby" <pje at telecommunity.com> wrote: >While investigating the need to apply http://python.org/sf/1525766 I found >that there was a modification to pkgutil during the need-for-speed sprint >that affects the PEP 302 protocol in a backwards incompatible way. It just so happens that the bug that is reported was probably reported because I'm working on some controversial new functionality in Twisted - controversial because it replicates the functionality that bug is about in pkgutil. This functionality does make some use of PEP 302 functionality :). See <http://twistedmatrix.com/trac/ticket/1940> >Specifically, PEP 302 documents that path_importer_cache always contains >either importer objects or None. Any code written to obtain importer >objects is therefore now broken, because import.c is slapping False in for >non-existent filesystem paths. Oddly, for once I'm going to say I don't care about this change. The code I've written so far doesn't depend on this, and I was pretty careful to be conservative about depending too much on the stuff described in PEP 302. It documents several features which don't exist (get_data, and methods in the "imp" module which don't exist in python2.3 or python2.4, where it was nominally accepted). >There are several options as to how to proceed: >2. Document the breakage, update PEP 302, and make everybody update their code Personally I'd prefer it if PEP 302 were updated for a variety of reasons. It's very hard to use as a reference for writing actual code because so many features are "optional" or "open issues", and there's no description in the PEP of what their status is. Better yet, this breakage (and other things) should be documented in the Python reference, and the PEP should link to the documentation for different versions, which can each describe the PEP's implementation status. The "importing modules" section of the library reference seems like a natural place to put it. From barry at python.org Thu Jul 20 22:45:47 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 20 Jul 2006 16:45:47 -0400 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc In-Reply-To: <e9olpa$4ql$1@sea.gmane.org> References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org> <EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org> <44B64782.9010706@gmail.com> <7FE62713-24B0-47CE-BA94-41150DB683F4@python.org> <AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org> <e9olpa$4ql$1@sea.gmane.org> Message-ID: <1A2C07C0-58C4-45F0-90C1-FD0C26B7D9C5@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 20, 2006, at 3:32 PM, Georg Brandl wrote: > Perhaps you could put the objects into _testcapi. That way no new > module > has to be deployed (is _testcapi installed on every system?) That doesn't seem importable in types.py either. You /could/ add that to Modules/config.c.in but I'm not sure it''s a good idea. - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRL/rgHEjvBPtnXfVAQLnHwP/Tzy1Iy8nbByuVaYaGrhel9zsdlbRXXlj 8wvLLlmzctmnzBrPBpZKV7aJMdiBd6261SKqPCrDZCj2LEplZWRa7za0zm3O2iZu F1kbWCKTwWeI2/kVNpP1lTkm53u9yYp8grjxe1Q6fniB5nqQH11j79JehAYpBwdj /RsVYoeQxng= =lIEO -----END PGP SIGNATURE----- From tdelaney at avaya.com Fri Jul 21 01:22:25 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Fri, 21 Jul 2006 09:22:25 +1000 Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from subprocess in 2.5 on Win32 Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com> Larry Hastings wrote: > I run the following script: > -- > from subprocess import * > Popen("ls -l") > -- > (yeah, I have ls.exe on Windows) > > Under Python 2.4.2, this simply dumped the results of ls.exe to the > terminal--sorry, to the "command shell". > > Under Python 2.5, both beta 1 and beta 2, it dumps the results to the > command shell, but *also* prints this: > > Exception exceptions.AttributeError: "'NoneType' object has no > attribute 'append'" in <bound method Popen.__del__ of > <subprocess.Popen object at 0x00C04EB0>> ignored > > Calling Popen() with a stdout = subprocess.PIPE does not throw this > exception. I've asked Larry to raise this on SourceForge, but with the SF email problems I thought I'd better forward it here. Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a position to have a look right now. Tim Delaney From bioinformed at gmail.com Fri Jul 21 01:37:49 2006 From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>) Date: Thu, 20 Jul 2006 19:37:49 -0400 Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from subprocess in 2.5 on Win32 In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com> Message-ID: <2e1434c10607201637l238635c7kc63cff228c5def82@mail.gmail.com> Reported to the list about a week ago, with analysis. Didn't get a response. Won't use sourceforge. Sorry about the top post. -Kevin On 7/20/06, Delaney, Timothy (Tim) <tdelaney at avaya.com> wrote: > > Larry Hastings wrote: > > > I run the following script: > > -- > > from subprocess import * > > Popen("ls -l") > > -- > > (yeah, I have ls.exe on Windows) > > > > Under Python 2.4.2, this simply dumped the results of ls.exe to the > > terminal--sorry, to the "command shell". > > > > Under Python 2.5, both beta 1 and beta 2, it dumps the results to the > > command shell, but *also* prints this: > > > > Exception exceptions.AttributeError: "'NoneType' object has no > > attribute 'append'" in <bound method Popen.__del__ of > > <subprocess.Popen object at 0x00C04EB0>> ignored > > > > Calling Popen() with a stdout = subprocess.PIPE does not throw this > > exception. > > I've asked Larry to raise this on SourceForge, but with the SF email > problems I thought I'd better forward it here. > > Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a position > to have a look right now. > > Tim Delaney > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jacobs%40bioinformed.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060720/a4ae87b3/attachment.htm From tdelaney at avaya.com Fri Jul 21 02:44:39 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Fri, 21 Jul 2006 10:44:39 +1000 Subject: [Python-Dev] Behavior change in subprocess.py Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1E847@au3010avexu1.global.avaya.com> Hah - just found it. I even remember reading it... I'll update the SF tracker (1526203) with your analysis. Tim Delaney -----Original Message----- From: python-dev-bounces+tdelaney=avaya.com at python.org [mailto:python-dev-bounces+tdelaney=avaya.com at python.org] On Behalf Of Kevin Jacobs <jacobs at bioinformed.com> Sent: Thursday, 13 July 2006 12:33 AM To: python-dev at python.org Subject: [Python-Dev] Behavior change in subprocess.py During my testing of Python 2.5b2, I've found something that may be worthy of discussion. I suspect that recent GC and finalization changes have altered the behavior of the Popen object in subprocess.py. I am now getting many many many finalization warnings in my code like: Exception exceptions.AttributeError: "'NoneType' object has no attribute 'append'" in <bound method Popen.__del__ of <subprocess.Popen object at 0x2aaaab910950>> ignored Is this a bug or a feature? Personally, I'd like to see these messages silenced, since it is being generated during interpreter shutdown. The following patch does the trick for me: --- /usr/local/lib/python2.5/subprocess.py 2006-07-11 14:11:59.000000000 -0400 +++ subprocess.py 2006-07-12 10:17:09.000000000 -0400 @@ -613,7 +613,7 @@ return # In case the child hasn't been waited on, check if it's done. self.poll(_deadstate=sys.maxint) - if self.returncode is None: + if self.returncode is None and _active is not None: # Child is still running, keep us alive until we can wait on it. _active.append(self) Note that popen.py does something similar, though I am not convinced that the test is right or if it is doing something more subtle: def __del__(self): # In case the child hasn't been waited on, check if it's done. self.poll(_deadstate=sys.maxint) if self.sts < 0: if _active: # Child is still running, keep us alive until we can wait on it. _active.append(self) Regards, -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/c21df170/attachment.html From greg.ewing at canterbury.ac.nz Fri Jul 21 03:17:43 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Jul 2006 13:17:43 +1200 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <E1G3VNd-0003SW-6E@draco.cus.cam.ac.uk> References: <E1G3VNd-0003SW-6E@draco.cus.cam.ac.uk> Message-ID: <44C02B37.5080106@canterbury.ac.nz> Nick Maclaren wrote: > Now, interrupting into that level has to be transparent, in order to > support TLB misses, clock interrupts, device interrupts, machine-check > interrupts and so on. I thought we were just talking about counting the number of floating point exceptions that a particular piece of code generates. Surely that's deterministic, and isn't affected by any of that stuff? -- Greg From brett at python.org Fri Jul 21 05:52:34 2006 From: brett at python.org (Brett Cannon) Date: Thu, 20 Jul 2006 20:52:34 -0700 Subject: [Python-Dev] first draft of bug guidelines for www.python.org/dev/ Message-ID: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> Here is a first stab at writing up guidelines for people to follow when reporting bug. If this goes well I will also do ones for patches, committing, and PEPs. -Brett --------------------------------------------------- These sets of guidelines are to help you file a bug report for the Python programming language on SourceForge_. If your bug is not for the language but for a third-party application, please report the bug to that third-party. *Please make sure to follow every step as it will make the lives of the Python developers much easier!!!* .. contents:: Get a SourceForge account ======================================= In order to file a bug report, you must have an account_ on SourceForge_. We realize some people would like to have anonymous bug reporting for various reasons (anonymity, ease or reporting, etc.). But SourceForge does not support anonymous reporting. Plus, by registering, you are notified by email when any action is been taken on your report. This can be very important if a Python developer needs more information from you about the bug. Start a new bug ======================================= You must be logged into SourceForge to file a bug! See `Get a SourceForge account`_ if you do not have one. Go to the `SourceForge bug page`_ to start a new bug report. There you will find a link called `Submit New`_. Click on it and it will allow you to fill out a new bug report. Once you click on the link, you are presented with a page that has several fields for you to fill in. Here is what to do for each field: * Category Set this to the area that the bug is related to (e.g., documentation, build, etc.). * Group Usually this is set the major.minor version of Python that you found the bug in. * Assigned To Leave this alone * Priority Leave this alone * Summary A one-line describing the problem so as to make it easy for developers to spot whether they have the expertise needed to work on the bug. * Detailed Description Following sections of this document discuss what should go in here. * Check to Upload and Attach a File If you are going to upload a file, you *must* check this box. * <File Location Field> Click the Browse button to upload any file to accompany your bug report (usually a succinct way to reproduce the bug). * File Description A one-line describing the file; no date info is needed since the upload is timestamped. Specify Python version ======================================= It is important that we have the most accurate version number of the interpreter you are using in order to best diagnose the issue. There are two ways to get us the version information. If you can run your Python interpreter, execute the following lines at an interpreter and paste the result into the ``Detailed Description`` field of the bug report:: >>> import sys >>> print sys.version If you are running a version of Python newer than 2.4 and are working from a source checkout of Python, the please also report the Subversion revision number for the root of your checkout:: python/trunk$ svnversion . If your bug is preventing you from running the interpreter, execute Python with teh ``-V`` command-line flag and paste the output:: python/trunk$ python -V Special settings for your Python interpreter ============================================ Sometimes your environment influences a bug and thus needs to be reported to help find the problem. This means we need to have reported: * Operating System * Environment Variables + PYTHONSTARTUP If this is set and might be causing the issue, please either upload the file or state what it does. + PYTHONCASEOK If your bug is on Windows and involves importing, please report if this environment variable is set or not. * site-packages If you have andy third-party packages installed that may be contributing to the bug, please report those. * Custom Patches Any differences between your code and the code the Python developers work off of needs to be reported. Sample code to reproduce bug ======================================== If you can, please upload a file the demonstrates the bug. The more succinct the better! And please do not forget to check the upload checkbox in the bug report. Submit! ======================================== At this point you should have a detailed bug report for developers to work off of. Click the ``Submit`` button and read on to see what you should do after the bug is reported. Respond to requests from developers ======================================== No matter how detailed the bug report, there is always the off-chance that a developer will need more information to fix a bug. Please be prompt in replying to requests for information by submitting a reply on the bug report. You may be asked to test out a patch. It is very important that you help with this, especially if the bug is not reproducible by the developer working on it. Write a patch! ======================================== If you are so inclined, patches for your own bug reports are always helpful! Please make sure to reference the tracker item # in the patch details. .. _SourceForge: http://www.sourceforge.net/ .. _SourceForge bug page: http://sourceforge.net/tracker/?group_id=5470&atid=105470 .. _account: http://sourceforge.net/account/newuser_emailverify.php .. _Submit New: http://sourceforge.net/tracker/?func=add&group_id=5470&atid=105470 From nyamatongwe at gmail.com Fri Jul 21 06:10:27 2006 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Fri, 21 Jul 2006 14:10:27 +1000 Subject: [Python-Dev] first draft of bug guidelines for www.python.org/dev/ In-Reply-To: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> Message-ID: <50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com> Brett Cannon: > But SourceForge does not support anonymous reporting. SourceForge does support anonymous reporting. A large proportion of the fault reports I receive for Scintilla are anonymous as indicated by "nobody" in the "Submitted By" column. https://sourceforge.net/tracker/?group_id=2439&atid=102439 Neil From fdrake at acm.org Fri Jul 21 06:25:17 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 21 Jul 2006 00:25:17 -0400 Subject: [Python-Dev] first draft of bug guidelines for www.python.org/dev/ In-Reply-To: <50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com> References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> <50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com> Message-ID: <200607210025.17429.fdrake@acm.org> On Friday 21 July 2006 00:10, Neil Hodgson wrote: > Brett Cannon: > > But SourceForge does not support anonymous reporting. > > SourceForge does support anonymous reporting. A large proportion of > the fault reports I receive for Scintilla are anonymous as indicated > by "nobody" in the "Submitted By" column. SourceForge supports anonymous reporting, but the Python project determined that the management cost of anonymous reports was higher than the value they provided. It might be time to reconsider that decision (though my position hasn't changed). -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From tjreedy at udel.edu Fri Jul 21 06:42:12 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 21 Jul 2006 00:42:12 -0400 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org><EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org><44B64782.9010706@gmail.com><7FE62713-24B0-47CE-BA94-41150DB683F4@python.org> <AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org> Message-ID: <e9plv5$rfe$1@sea.gmane.org> "Barry Warsaw" <barry at python.org> wrote in message news:AB1C2209-329E-4F3D-8B7A-2548A40F892A at python.org... > think this patch should go in 2.5. OTOH, I suspect most people just > don't care, which is why I've gotten almost no comments on the patch > (other than one or two mild nods of approval). I use help(ob) quite a bit, have noticed that somethings don't give anything useful, and definitely approve of making help more useful. tjr From oliphant.travis at ieee.org Fri Jul 21 09:42:51 2006 From: oliphant.travis at ieee.org (Travis E. Oliphant) Date: Fri, 21 Jul 2006 01:42:51 -0600 Subject: [Python-Dev] segfault when using PyGILState_Ensure/Release in Python2.3.4 Message-ID: <e9q0hs$jo4$1@sea.gmane.org> I'm hoping somebody here can help me with an error I'm getting in Python 2.3.4 but not in Python 2.4.2 when I use PyGILState_Ensure in NumPy on Linux. Perhaps somebody can point out what I'm doing wrong because while I've tried to understand the threading API it can be a bit confusing and maybe I'm doing it wrong. Right now, I've had to disable threading support in NumPy for Python 2.3 which is a bit annoying. The problem shows up when I've released the GIL using PyEval_SaveThread() in one section of code. Then the code calls functions that don't involve the Python C-API. Then another function sometimes requires use of the C-API to set a Python Error or issue a warning. So I call: _save = PyGILState_Ensure(); <Use Python C-API to issue an error or warning> Finally, before exiting this function PyGILState_Release(_save); is called. Later when control returns to the original caller that released the GIL, PyEval_RestoreThread() is called. But the segfault seems to be happening on the call to PyGILState_Release(_save); All of this works fine when it runs under Python 2.4.2, but under Python 2.3.4 I get a segfault. Does anybody have any ideas? Thanks very much. From ncoghlan at gmail.com Fri Jul 21 12:08:36 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jul 2006 20:08:36 +1000 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607201009x28a7354elde374c6651b1a322@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <44BF6C59.6010707@gmail.com> <bbaeab100607201009x28a7354elde374c6651b1a322@mail.gmail.com> Message-ID: <44C0A7A4.2030209@gmail.com> Brett Cannon wrote: > Extensible file type handling > ----------------------------- > If the file type handlers are stored in normal Python data > structures as > described above, it becomes feasible to make the import system > extensible to > different file types as well as to different file locations. > > > Yep. Although I am more interested in restricting than broadening the > file types. Either way you'd be mutating the list of recognised file types :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Fri Jul 21 12:14:40 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jul 2006 20:14:40 +1000 Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from subprocess in 2.5 on Win32 In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com> Message-ID: <44C0A910.4070002@gmail.com> Delaney, Timothy (Tim) wrote: > Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a position > to have a look right now. For those not watching python-checkins, a check for "is not None" has been added before the offending line in Popen.__del__. (by Georg, IIRC) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From nmm1 at cus.cam.ac.uk Fri Jul 21 12:18:04 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Fri, 21 Jul 2006 11:18:04 +0100 Subject: [Python-Dev] Strategy for converting the decimal module to C Message-ID: <E1G3s56-0000pe-M5@draco.cus.cam.ac.uk> Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > > > Now, interrupting into that level has to be transparent, in order to > > support TLB misses, clock interrupts, device interrupts, machine-check > > interrupts and so on. > > I thought we were just talking about counting the number > of floating point exceptions that a particular piece of > code generates. Surely that's deterministic, and isn't Er, no. Rather fundamentally, on two grounds. Please bear with me, as this IS relevant to Python. See the summary at the end if you like :-) The first is that such things are NOT deterministic, not even on simple CPUs - take a look at the Alpha architecture for an example, and then follow it up with the IA64 one if you have the stomach for it. But that wasn't my main point. It is that modern CPUs have a SINGLE interrupt mechanism (a mistake in itself, but they do), so a CPU may be interrupted when it is running a device driver, other kernel thread or within a system call as much as when running an application. In fact, to some extent, interrupt handlers can themselves be interrupted (let's skip the details). Now, in order to allow the application to run its handler, the state has to be saved, sanitised and converted back to application context; and conversely on return. That is hairy, and is why it is not possible to handle interrupts generated within system calls on many systems. But that is not directly Python's problem. What is, is that the code gets interrupted at an unpredictable place, and the registers and other state may not be consistent as the language run-time system and Python are concerned. It is critical (a) that a sane state is restored before calling the handler and (b) that calling the handler neither relies on nor disturbs any of the "in flight" actions in the interrupted code. To cut a long story short, it is impractical for a language run-time system to call user-defined handlers with any degree of reliability unless the compiled code and run-time interoperate carefully - I have been there and done that many times, but few people still working have. On architectures with out-of-order execution (and interrupts), you have to assume that an interrupt may occur anywhere, even when the code does not use the relevant facility. Floating-point overflow in the middle of a list insertion? That's to be expected. It becomes considerably easier if the (run-time system) interrupt handler merely needs to flag or count interrupts, as it can use a minimal handler which is defensive and non-intrusive. Even that is a pretty fair nightmare, as many systems temporarily corrupt critical registers when they think that it is safe. And few think of interrupts when deciding that .... So, in summary, please DON'T produce a design that relies on trapping floating-point exceptions and passing control to a Python function. This is several times harder than implementing fpectl. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From ndbecker2 at gmail.com Fri Jul 21 12:56:33 2006 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 21 Jul 2006 06:56:33 -0400 Subject: [Python-Dev] Document performance requirements? Message-ID: <e9qbtf$kk6$1@sea.gmane.org> For a recent project I needed to select a container. There are plenty of python data structures to choose from. It seems that information on performance is missing (or not easy to find). I think Python should include performance in the documentation of common data structures to help users select the appropriate types. Something in the style of c++ STL. From mwh at python.net Fri Jul 21 13:25:12 2006 From: mwh at python.net (Michael Hudson) Date: Fri, 21 Jul 2006 12:25:12 +0100 Subject: [Python-Dev] Ireland PyPy sprint 21th-27th August 2006 Message-ID: <2mirlryzc7.fsf@starship.python.net> The next PyPy sprint will happen in the nice city of Limerick in Ireland from 21st till 27th August. (Most people intend to arrive 20th August). The main focus of the sprint will be on JIT compiler works, various optimization works, porting extension modules, infrastructure works like a build tool for PyPy, or extended (distributed) testing. It's also open to new topics. If you are a student consider to participate in `Summer of PyPy`_ in order get funding for your travels and accomodation. The sprint is being hosted by University of Limerick (http://www.ul.ie/) - and is arranged in co-operation with people from our sister project Calibre (www.calibre.ie). Our contact at the University is P?r ?gerfalk and Eoin Oconchuir. .. _`Summer of PyPy`: http://codespeak.net/pypy/dist/pypy/doc/summer-of-pypy First day: introduction and workshop (possible to attend only this day) ------------------------------------------------------------------------ During the first day (21st of August) there will be talks on various subjects related to PyPy: * A tutorial and technical introduction to the PyPy codebase (suited for people interested in getting an overview of PyPy?s architecture and/or contributing to PyPy) * a workshop covering more in-depth technical aspects of PyPy and what PyPy can do for you. The workshop will also cover methodology, aiming at explaining the pros and cons of sprint-driven development. (suited for sprint attendants, students, staff and other interested parties from/around the University and the local area) The tutorial will be part of the sprint introduction - the workshop will take place if there is enough interest raised before the 21st of August from people planning to attend. You are of course welcome to attend just for this first day of the sprint. If you want to come ... ---------------------------- If you'd like to come, please subscribe to the `pypy-sprint mailing list`_ and drop a note about your interests and post any questions. More organisational information will be send to that list. We'll keep a list of `people`_ which we'll update (which you can do so yourself if you have codespeak commit rights). .. _`Calibre`: http://www.calibre.ie A small disclaimer: There might be people visiting the sprint in order to do research on how open source communities work, organize and communicate. This research might be done via filming, observing or interviewing. But of course you will be able to opt-out of being filmed at the sprint. Logistics -------------------------------------- NOTE: you need a UK style of power adapter (220V). The sprint will be held in the Computer Science Building, room CSG-025, University of Limerick (no 7 on http://www.ul.ie/main/places/campus.shtml). Bus 308 from Limerick city will take you to no 30 (approx.). See http://www.ul.ie/main/places/travel.shtml for more on how to get to UL. We will have access to the sprint facilities from 09:00-19:00 every day (it might be even later than 19:00). Monday-Wednesday, Friday-Sunday are sprint days, Thursday is likely a break day. Food on campus varies in price and quality ;-) : from ca 4 EUR to 7-8 EUR for a lunch. There are of course a lot more food alternatives in down town Limerick. Next Airports ------------------ Shannon Airport (SNN) is the nearest airport (Ryanair flies there) - you may check out more information about flights to/from the airport at http://www.shannonairport.com/index.html There are busses from there to downtown Limerick, and busses from Limerick to the UL campus. Taxis are about 35 EUR. Accomodation ----------------- There is a website address for campus accomodation at http://www.ul.ie/conference/accommodation.htm. The rate should be 49 euro for Bed and Breakfast. If you are interested in booking campus accommodation, please contact deborah.tudge at ul ie and make reference to the PyPy workshop and sprint. Please try to book as soon as possible. As an off-campus accommodation alternative you can also try: Castletroy Lodge and Castletroy Inn (Bed and Breakfast) Dublin Road (15 to 20 mins walk to UL) Tel: +353 61 338385 / +353 61 331167 .. _`pypy-sprint mailing list`: http://codespeak.net/mailman/listinfo/pypy-sprint .. _`people`: people.html -- surely, somewhere, somehow, in the history of computing, at least one manual has been written that you could at least remotely attempt to consider possibly glancing at. -- Adam Rixey From ncoghlan at gmail.com Fri Jul 21 13:49:32 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jul 2006 21:49:32 +1000 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <e9qbtf$kk6$1@sea.gmane.org> References: <e9qbtf$kk6$1@sea.gmane.org> Message-ID: <44C0BF4C.1010408@gmail.com> Neal Becker wrote: > For a recent project I needed to select a container. There are plenty of > python data structures to choose from. It seems that information on > performance is missing (or not easy to find). > > I think Python should include performance in the documentation of common > data structures to help users select the appropriate types. Something in > the style of c++ STL. Do you mean absolute performance, or do you mean algorithmic order guarantees? I thought the latter were already documented. . . Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Fri Jul 21 13:56:20 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jul 2006 21:56:20 +1000 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc In-Reply-To: <e9olpa$4ql$1@sea.gmane.org> References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org> <EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org> <44B64782.9010706@gmail.com> <7FE62713-24B0-47CE-BA94-41150DB683F4@python.org> <AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org> <e9olpa$4ql$1@sea.gmane.org> Message-ID: <44C0C0E4.6040403@gmail.com> Georg Brandl wrote: > Perhaps you could put the objects into _testcapi. That way no new module > has to be deployed (is _testcapi installed on every system?) Let's not corrupt _testcapi with something that has an entirely differently purpose. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ndbecker2 at gmail.com Fri Jul 21 14:03:26 2006 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 21 Jul 2006 08:03:26 -0400 Subject: [Python-Dev] Document performance requirements? References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com> Message-ID: <e9qfqt$1t3$1@sea.gmane.org> Nick Coghlan wrote: > Neal Becker wrote: >> For a recent project I needed to select a container. There are plenty of >> python data structures to choose from. It seems that information on >> performance is missing (or not easy to find). >> >> I think Python should include performance in the documentation of common >> data structures to help users select the appropriate types. Something in >> the style of c++ STL. > > Do you mean absolute performance, or do you mean algorithmic order > guarantees? I thought the latter were already documented. . . > The latter. Where is it documented? From ncoghlan at gmail.com Fri Jul 21 14:25:02 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Jul 2006 22:25:02 +1000 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <200607210759.56595.ndbecker2@gmail.com> References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com> <200607210759.56595.ndbecker2@gmail.com> Message-ID: <44C0C79E.3020309@gmail.com> Neal Becker wrote: > On Friday 21 July 2006 7:49 am, Nick Coghlan wrote: >> Neal Becker wrote: >>> For a recent project I needed to select a container. There are plenty of >>> python data structures to choose from. It seems that information on >>> performance is missing (or not easy to find). >>> >>> I think Python should include performance in the documentation of common >>> data structures to help users select the appropriate types. Something in >>> the style of c++ STL. >> Do you mean absolute performance, or do you mean algorithmic order >> guarantees? I thought the latter were already documented. . . >> > > The latter. Where are they documented? Just because I think something, it doesn't mean it's true :) The only reference I can actually find is the one in the collections module docs pointing out that collections.deque permits O(1) insertions and removals at the beginning of the sequence, as well as at the end (whereas lists are O(n) for operations at the beginning due to the resulting memory copying). However, I'm also struggling to think of a case other than list vs deque where the choice of a builtin or standard library data structure would be dictated by big-O() concerns. list vs array.array is based on memory efficiency list vs deque is based on whether or not you need O(1) push/pop at both ends list vs set is based on whether or not ordering matters set vs dict is based on whether or not you need to map keys to values Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From jason.orendorff at gmail.com Fri Jul 21 16:14:20 2006 From: jason.orendorff at gmail.com (Jason Orendorff) Date: Fri, 21 Jul 2006 10:14:20 -0400 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com> References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com> <200607210759.56595.ndbecker2@gmail.com> <44C0C79E.3020309@gmail.com> <bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com> Message-ID: <bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com> On 7/21/06, Nick Coghlan <ncoghlan at gmail.com> wrote: > However, I'm also struggling to think of a case other than list vs deque where > the choice of a builtin or standard library data structure would be dictated > by big-O() concerns. OK, but that doesn't mean the information is unimportant. +1 on making this something of a priority. People looking for this info should find it in the obvious place. Some are unobvious. (How fast is dict.__eq__ on average? Worst case?) -j From brett at python.org Fri Jul 21 16:40:07 2006 From: brett at python.org (Brett Cannon) Date: Fri, 21 Jul 2006 07:40:07 -0700 Subject: [Python-Dev] first draft of bug guidelines for www.python.org/dev/ In-Reply-To: <200607210025.17429.fdrake@acm.org> References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> <50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com> <200607210025.17429.fdrake@acm.org> Message-ID: <bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com> On 7/20/06, Fred L. Drake, Jr. <fdrake at acm.org> wrote: > > On Friday 21 July 2006 00:10, Neil Hodgson wrote: > > Brett Cannon: > > > But SourceForge does not support anonymous reporting. > > > > SourceForge does support anonymous reporting. A large proportion of > > the fault reports I receive for Scintilla are anonymous as indicated > > by "nobody" in the "Submitted By" column. > > SourceForge supports anonymous reporting, but the Python project > determined > that the management cost of anonymous reports was higher than the value > they > provided. OK. Wording has been changed in my copy. It might be time to reconsider that decision (though my position hasn't > changed). Sure. It can also wait until we begin discussing the transition to our next bug tracker. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/1f9d3087/attachment.html From foom at fuhm.net Fri Jul 21 17:05:51 2006 From: foom at fuhm.net (James Y Knight) Date: Fri, 21 Jul 2006 11:05:51 -0400 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <E1G3s56-0000pe-M5@draco.cus.cam.ac.uk> References: <E1G3s56-0000pe-M5@draco.cus.cam.ac.uk> Message-ID: <8611AEB2-5D94-4C28-96D6-8626BA5B6DFC@fuhm.net> On Jul 21, 2006, at 6:18 AM, Nick Maclaren wrote: > To cut a long story short, it is impractical for a language run-time > system to call user-defined handlers with any degree of reliability > unless the compiled code and run-time interoperate carefully - I have > been there and done that many times, but few people still working > have. > On architectures with out-of-order execution (and interrupts), you > have to assume that an interrupt may occur anywhere, even when the > code does not use the relevant facility. Floating-point overflow > in the middle of a list insertion? That's to be expected. While this _is_ a real problem, is it _not_ a general problem as you are describing it. Processors are perfectly capable of generating precise interrupts, and the inability to do so has nothing to do with the out-of-order execution, etc. Almost all interrupts are precise. The only interesting one which is not, on x86 processors, is the x87 floating point exception, which is basically for historical reasons. It has never been precise, ever since the actual 8087 coprocessor chip for the 8086. However, all is not lost: the exception cannot occur randomly. It can only occur on *some* floating point instruction, even if the instruction is not the one the error actually occurred in. So, unless your list insertion code uses floating point instructions, you should not get a floating point exception during your list insertion. Also, looking forward, the "simd" floating point instructions (ie mmx/ sse/sse2/sse3) _do_ generate precise interrupts. And on x86-64, x87 instructions are deprecated and everyone is recommended to use the simd ones, instead (so, for example, gcc defaults to using them). James From bioinformed at gmail.com Fri Jul 21 18:43:44 2006 From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>) Date: Fri, 21 Jul 2006 12:43:44 -0400 Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from subprocess in 2.5 on Win32 In-Reply-To: <44C0A910.4070002@gmail.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com> <44C0A910.4070002@gmail.com> Message-ID: <2e1434c10607210943p10171eb9n607c5a5821768c7c@mail.gmail.com> On 7/21/06, Nick Coghlan <ncoghlan at gmail.com> wrote: > > Delaney, Timothy (Tim) wrote: > > Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a position > > to have a look right now. > > For those not watching python-checkins, a check for "is not None" has been > added before the offending line in Popen.__del__. (by Georg, IIRC) > Many thanks for accepting my patch. There remains a potentially related problem in popen2.py, but it may be a lower priority, since most folks should be using subprocess. def __del__(self): # In case the child hasn't been waited on, check if it's done. self.poll(_deadstate=sys.maxint) if self.sts < 0: if _active: # Child is still running, keep us alive until we can wait on it. _active.append(self) The is _active check, unless it intendeds to check for either empty or None, should probably be revised to: def __del__(self): # In case the child hasn't been waited on, check if it's done. self.poll(_deadstate=sys.maxint) if self.sts < 0: if _active is None: # Child is still running, keep us alive until we can wait on it. _active.append(self) However, there may be a clever reason for doing what is doing that I do not see. Thanks again, -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/f57de211/attachment.html From rasky at develer.com Fri Jul 21 18:45:12 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 21 Jul 2006 18:45:12 +0200 Subject: [Python-Dev] Document performance requirements? References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com><200607210759.56595.ndbecker2@gmail.com> <44C0C79E.3020309@gmail.com><bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com> <bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com> Message-ID: <054801c6ace5$09478c50$d503030a@trilan> Jason Orendorff wrote: >> However, I'm also struggling to think of a case other than list vs >> deque where the choice of a builtin or standard library data >> structure would be dictated by big-O() concerns. > > OK, but that doesn't mean the information is unimportant. +1 on > making this something of a priority. People looking for this info > should find it in the obvious place. Some are unobvious. (How fast is > dict.__eq__ on average? Worst case?) I also found out that most people tend to think of Python's lists as a magical data structure optimized for many operations (like a "rope" or something complex like that). Documenting that it's just a bare vector (std::vector in C++) would be of great help. -- Giovanni Bajo From grig.gheorghiu at gmail.com Fri Jul 21 19:04:38 2006 From: grig.gheorghiu at gmail.com (Grig Gheorghiu) Date: Fri, 21 Jul 2006 10:04:38 -0700 Subject: [Python-Dev] Community buildbots -- reprise Message-ID: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com> Hi, This message is in response to Glyph's plea (<http://mail.python.org/pipermail/python-dev/2006-July/067366.html> <http://mail.python.org/pipermail/python-dev/2006-July/067366.html%3E>). Here's what Glyph said: "I would like to propose, although I certainly don't have time to implement, a program by which Python-using projects could contribute buildslaves which would run their projects' tests with the latest Python trunk. This would provide two useful incentives: Python code would gain a reputation as generally well-tested (since there is a direct incentive to write tests for your project: get notified when core python changes might break it), and the core developers would have instant feedback when a "small" change breaks more code than it was expected to." I'm volunteering to organize this effort, is there is enough interest on this list. In fact, I've done some prep work already: * got a domain name: pybots.org * got a $47/month Ubuntu-based VPS from JohnCompanies.com (root access and everything); it's available at master.pybots.org, and it's ready to be configured as a buildmaster for the pybots * got a mailing list: pybots at lists2.idyll.org <http://us.f545.mail.yahoo.com/ym/Compose?To=pybots at lists2.idyll.org&YY=8414&order=down&sort=date&pos=0&view=a&head=b> I can start configuring the Ubuntu machine as a buildmaster, and I can also add a buildslave on the same machine that will check out the latest Python trunk code, build it, then run the automated tests for a sample project -- let's say for Twisted, since Glyph was the one requesting this. This will also serve as a sample buildslave for other people who will be interested in running buildslaves for their own projects. Apart from the goals stated by Glyph, I see this as a very valuable effort in convincing people of the value of automated tests, Python-related or not. A secondary effect I'd like to see would be for these suites of tests to be invoked in a standard fashion -- maybe 'python setup.py test'. If PSF can contribute some $$$ towards the hosting of the master server, that would be appreciated, but not required. All that's required is enough interest from the community. Please let me know if you're interested. Grig <http://agiletesting.blogspot.com/> -- http://agiletesting.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/6ede65ff/attachment.htm From skip at pobox.com Fri Jul 21 19:22:09 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 21 Jul 2006 12:22:09 -0500 Subject: [Python-Dev] first draft of bug guidelines for www.python.org/dev/ In-Reply-To: <bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com> References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> <50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com> <200607210025.17429.fdrake@acm.org> <bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com> Message-ID: <17601.3393.544378.871413@montanaro.dyndns.org> Brett> Sure. It can also wait until we begin discussing the transition Brett> to our next bug tracker. Would be kinda nice if the new bug tracker allowed submitters to enter a followup email address without formally logging in. (Of course, email-based submissions would go a long way to minimizing the problem.) Skip From exarkun at divmod.com Fri Jul 21 19:27:41 2006 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Fri, 21 Jul 2006 13:27:41 -0400 Subject: [Python-Dev] Community buildbots -- reprise In-Reply-To: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com> Message-ID: <20060721172741.29014.1781472149.divmod.quotient.47688@ohm> On Fri, 21 Jul 2006 10:04:38 -0700, Grig Gheorghiu <grig.gheorghiu at gmail.com> wrote: >Hi, > >Apart from the goals stated by Glyph, I see this as a very valuable >effort in convincing people of the value of automated tests, >Python-related or not. A secondary effect I'd like to see would be for >these suites of tests to be invoked in a standard fashion -- maybe >'python setup.py test'. > >If PSF can contribute some $$$ towards the hosting of the master >server, that would be appreciated, but not required. All that's >required is enough interest from the community. > >Please let me know if you're interested. > This is certainly interesting to me. If you need any help setting up the Twisted buildslave, please let me know. Jean-Paul From g.brandl at gmx.net Fri Jul 21 19:39:40 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Fri, 21 Jul 2006 19:39:40 +0200 Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from subprocess in 2.5 on Win32 In-Reply-To: <2e1434c10607210943p10171eb9n607c5a5821768c7c@mail.gmail.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1E843@au3010avexu1.global.avaya.com> <44C0A910.4070002@gmail.com> <2e1434c10607210943p10171eb9n607c5a5821768c7c@mail.gmail.com> Message-ID: <e9r3ee$4on$1@sea.gmane.org> Kevin Jacobs <jacobs at bioinformed.com> wrote: > On 7/21/06, *Nick Coghlan* <ncoghlan at gmail.com > <mailto:ncoghlan at gmail.com>> wrote: > > Delaney, Timothy (Tim) wrote: > > Looks like there's a bug in Popen.__del__ in 2.5. I'm not in a > position > > to have a look right now. > > For those not watching python-checkins, a check for "is not None" > has been > added before the offending line in Popen.__del__. (by Georg, IIRC) > > > Many thanks for accepting my patch. There remains a potentially related > problem in popen2.py, but it may be a lower priority, since most folks > should be using subprocess. > > def __del__(self): > # In case the child hasn't been waited on, check if it's done. > self.poll(_deadstate=sys.maxint) > if self.sts < 0: > if _active: > # Child is still running, keep us alive until we can > wait on it. > _active.append(self) > > > The is _active check, unless it intendeds to check for either empty or > None, should probably be revised to: > > def __del__(self): > # In case the child hasn't been waited on, check if it's done. > self.poll(_deadstate=sys.maxint) > if self.sts < 0: > if _active is None: > # Child is still running, keep us alive until we can > wait on it. > _active.append(self) > > However, there may be a clever reason for doing what is doing that I do > not see. There's no reason (I know since I added the check myself ;). Thanks for pointing out that obvious bug. (fixed in rev 50759). Georg From john at ewtllc.com Fri Jul 21 19:44:02 2006 From: john at ewtllc.com (John Benediktsson) Date: Fri, 21 Jul 2006 10:44:02 -0700 Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from subprocess in 2.5 on Win32 Message-ID: <B6FAC926EFE7B348B12F29CF7E4A93D401328DC8@hammer.office.bhtrader.com> > The is _active check, unless it intendeds to check for either empty or > None, should probably be revised to: > > def __del__(self): > # In case the child hasn't been waited on, check if it's done. > self.poll(_deadstate=sys.maxint) > if self.sts < 0: > if _active is None: > # Child is still running, keep us alive until we can > wait on it. > _active.append(self) > It probably is obvious, but I think you mean: if _active is not None: ... Thanks, John. From foom at fuhm.net Fri Jul 21 19:48:28 2006 From: foom at fuhm.net (James Y Knight) Date: Fri, 21 Jul 2006 13:48:28 -0400 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <054801c6ace5$09478c50$d503030a@trilan> References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com><200607210759.56595.ndbecker2@gmail.com> <44C0C79E.3020309@gmail.com><bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com> <bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com> <054801c6ace5$09478c50$d503030a@trilan> Message-ID: <7458FE96-1A1D-40F3-B585-8E1DA55C4B14@fuhm.net> On Jul 21, 2006, at 12:45 PM, Giovanni Bajo wrote: > Jason Orendorff wrote: > >>> However, I'm also struggling to think of a case other than list vs >>> deque where the choice of a builtin or standard library data >>> structure would be dictated by big-O() concerns. >> >> OK, but that doesn't mean the information is unimportant. +1 on >> making this something of a priority. People looking for this info >> should find it in the obvious place. Some are unobvious. (How >> fast is >> dict.__eq__ on average? Worst case?) > > I also found out that most people tend to think of Python's lists as a > magical data structure optimized for many operations (like a "rope" or > something complex like that). Documenting that it's just a bare vector > (std::vector in C++) would be of great help. Indeed, I was talking to someone a while back who thought that lists were magically hashed, in that he did something like: dictionary = open("/usr/share/dict/words").readlines() and then expected: "word" in dictionary would be fast. And was very surprised when it turned out to be slow a linear search of the list. :) James From nnorwitz at gmail.com Fri Jul 21 20:31:08 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 21 Jul 2006 11:31:08 -0700 Subject: [Python-Dev] Community buildbots -- reprise In-Reply-To: <20060721172741.29014.1781472149.divmod.quotient.47688@ohm> References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com> <20060721172741.29014.1781472149.divmod.quotient.47688@ohm> Message-ID: <ee2a432c0607211131n67af50a5sa2518d5410a9c8e4@mail.gmail.com> I have a server up and running. I still need to polish some stuff off. I will mail more info when I get a chance. n -- On 7/21/06, Jean-Paul Calderone <exarkun at divmod.com> wrote: > On Fri, 21 Jul 2006 10:04:38 -0700, Grig Gheorghiu <grig.gheorghiu at gmail.com> wrote: > >Hi, > > > >Apart from the goals stated by Glyph, I see this as a very valuable > >effort in convincing people of the value of automated tests, > >Python-related or not. A secondary effect I'd like to see would be for > >these suites of tests to be invoked in a standard fashion -- maybe > >'python setup.py test'. > > > >If PSF can contribute some $$$ towards the hosting of the master > >server, that would be appreciated, but not required. All that's > >required is enough interest from the community. > > > >Please let me know if you're interested. > > > > This is certainly interesting to me. If you need any help setting up > the Twisted buildslave, please let me know. > > Jean-Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From bioinformed at gmail.com Fri Jul 21 20:53:56 2006 From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>) Date: Fri, 21 Jul 2006 14:53:56 -0400 Subject: [Python-Dev] FW: Bug? Certainly a new *behavior* from subprocess in 2.5 on Win32 In-Reply-To: <B6FAC926EFE7B348B12F29CF7E4A93D401328DC8@hammer.office.bhtrader.com> References: <B6FAC926EFE7B348B12F29CF7E4A93D401328DC8@hammer.office.bhtrader.com> Message-ID: <2e1434c10607211153g392d122bj83d705ae82f5fa49@mail.gmail.com> That'll teach me to fire off emails while running out the door. Thanks. -Kevin On 7/21/06, John Benediktsson <john at ewtllc.com> wrote: > > > > The is _active check, unless it intendeds to check for either empty or > > > None, should probably be revised to: > > > > def __del__(self): > > # In case the child hasn't been waited on, check if it's done. > > self.poll(_deadstate=sys.maxint) > > if self.sts < 0: > > if _active is None: > > # Child is still running, keep us alive until we can > > wait on it. > > _active.append(self) > > > > It probably is obvious, but I think you mean: > > if _active is not None: > ... > > Thanks, > John. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/jacobs%40bioinformed.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/73f4b83d/attachment.html From bioinformed at gmail.com Fri Jul 21 20:56:29 2006 From: bioinformed at gmail.com (Kevin Jacobs <jacobs@bioinformed.com>) Date: Fri, 21 Jul 2006 14:56:29 -0400 Subject: [Python-Dev] first draft of bug guidelines for www.python.org/dev/ In-Reply-To: <17601.3393.544378.871413@montanaro.dyndns.org> References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> <50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com> <200607210025.17429.fdrake@acm.org> <bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com> <17601.3393.544378.871413@montanaro.dyndns.org> Message-ID: <2e1434c10607211156s582441b1o753f25d2424e94b3@mail.gmail.com> On 7/21/06, skip at pobox.com <skip at pobox.com> wrote: > > > Brett> Sure. It can also wait until we begin discussing the > transition > Brett> to our next bug tracker. > > Would be kinda nice if the new bug tracker allowed submitters to enter a > followup email address without formally logging in. (Of course, > email-based > submissions would go a long way to minimizing the problem.) > It may just be bad karma, but SourceForge tends to lock or go off into lala land whenever I log in. Thus, I would file many bug reports, with a reply-to address, if non-login bug submissions where allowed. My long term hope is that you toss out SF and get something better. Thanks, -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060721/bd31a869/attachment.htm From ncoghlan at gmail.com Sat Jul 22 04:33:45 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Jul 2006 12:33:45 +1000 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com> References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com> <200607210759.56595.ndbecker2@gmail.com> <44C0C79E.3020309@gmail.com> <bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com> Message-ID: <44C18E89.5090603@gmail.com> Jason Orendorff wrote: > On 7/21/06, Nick Coghlan <ncoghlan at gmail.com> wrote: >> However, I'm also struggling to think of a case other than list vs >> deque where >> the choice of a builtin or standard library data structure would be >> dictated >> by big-O() concerns. > > OK, but that doesn't mean the information is unimportant. +1 on > making this something of a priority. People looking for this info > should find it in the obvious place. Some are unobvious. (How fast is > dict.__eq__ on average? Worst case?) Agreed, but there's more to doing that than just writing down the O() implied by the current CPython implementation - it's up to Guido to decide which of the constraints are part of the language definition, and which are implementation accidents (e.g. CPython's list.sort() operation was stable for at least one release before GvR made stability part of the definition of the method at the language level). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sat Jul 22 04:37:37 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Jul 2006 12:37:37 +1000 Subject: [Python-Dev] first draft of bug guidelines for www.python.org/dev/ In-Reply-To: <2e1434c10607211156s582441b1o753f25d2424e94b3@mail.gmail.com> References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> <50862ebd0607202110m661a3fc8sfd99b4a78b4db1b8@mail.gmail.com> <200607210025.17429.fdrake@acm.org> <bbaeab100607210740pf9e2db5r9f427c8f3184a9f7@mail.gmail.com> <17601.3393.544378.871413@montanaro.dyndns.org> <2e1434c10607211156s582441b1o753f25d2424e94b3@mail.gmail.com> Message-ID: <44C18F71.6020106@gmail.com> Kevin Jacobs <jacobs at bioinformed.com> wrote: > It may just be bad karma, but SourceForge tends to lock or go off into > lala land whenever I log in. Thus, I would file many bug reports, with > a reply-to address, if non-login bug submissions where allowed. My long > term hope is that you toss out SF and get something better. You're not the only one with that hope. With at least Trac, Jira and Roundup to choose from, the PSF's current tracker shootout should find us that replacement :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From arigo at tunes.org Sat Jul 22 09:07:10 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat, 22 Jul 2006 09:07:10 +0200 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> Message-ID: <20060722070710.GA22334@code0.codespeak.net> Hi Brett, On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote: > I also plan to rewrite the import machinery in pure Python. http://codespeak.net/svn/pypy/dist/pypy/module/__builtin__/importing.py A bientot, Armin From martin at v.loewis.de Sat Jul 22 10:34:08 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 22 Jul 2006 10:34:08 +0200 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com> References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com> <200607210759.56595.ndbecker2@gmail.com> <44C0C79E.3020309@gmail.com> <bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com> <bb8868b90607210714g2b7dd8e1he8fa00ba842fe3ee@mail.gmail.com> Message-ID: <44C1E300.8050803@v.loewis.de> Jason Orendorff wrote: > On 7/21/06, Nick Coghlan <ncoghlan at gmail.com> wrote: >> However, I'm also struggling to think of a case other than list vs deque where >> the choice of a builtin or standard library data structure would be dictated >> by big-O() concerns. > > OK, but that doesn't mean the information is unimportant. +1 on > making this something of a priority. People looking for this info > should find it in the obvious place. Some are unobvious. (How fast is > dict.__eq__ on average? Worst case?) Contributions are welcome. Regards, Martin From martin at v.loewis.de Sat Jul 22 10:39:35 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 22 Jul 2006 10:39:35 +0200 Subject: [Python-Dev] Community buildbots -- reprise In-Reply-To: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com> References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com> Message-ID: <44C1E447.2020404@v.loewis.de> Grig Gheorghiu wrote: > Please let me know if you're interested. As I said earlier: If you need some kind of post-commit trigger on the python repository to trigger a build, just let me know. We currently use a more-or-less plain svn_buildbot.py to trigger our own builds. Regards, Martin From arigo at tunes.org Sat Jul 22 13:19:07 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat, 22 Jul 2006 13:19:07 +0200 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <44C18E89.5090603@gmail.com> References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com> <200607210759.56595.ndbecker2@gmail.com> <44C0C79E.3020309@gmail.com> <bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com> <44C18E89.5090603@gmail.com> Message-ID: <20060722111907.GA13782@code0.codespeak.net> Hi, On Sat, Jul 22, 2006 at 12:33:45PM +1000, Nick Coghlan wrote: > Agreed, but there's more to doing that than just writing down the O() implied > by the current CPython implementation - it's up to Guido to decide which of > the constraints are part of the language definition, and which are > implementation accidents I think that O-wise the current CPython situation should be documented as a "minimal requirement" for implementations of the language, with just one exception: the well-documented "don't rely on this" hack in 2.4 to make repeated 'str += str' amortized linear, for which the 2.3 quadratic behavior is considered compliant enough. I suppose that allowing implementations to provide better algorithmic complexities than required is fine, although I can think of some problems with that (e.g. nice and efficient user code that would perform horribly badly on CPython). Armin From arigo at tunes.org Sat Jul 22 13:46:15 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat, 22 Jul 2006 13:46:15 +0200 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> Message-ID: <20060722114615.GB13782@code0.codespeak.net> Re-hi, On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote: > http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log. I'm not sure I understand what you propose to fix holes like constructors and __subclasses__: it seems that you want to remove them altogether (and e.g. make factory functions instead). That would completely break all programs, right? I mean, there is no way such changes would go into mainstream CPython. Or do you propose to maintain a CPython branch manually for the foreseeable future? (From experience this is a bad idea...) A bientot, Armin From grig.gheorghiu at gmail.com Sat Jul 22 17:32:07 2006 From: grig.gheorghiu at gmail.com (Grig Gheorghiu) Date: Sat, 22 Jul 2006 08:32:07 -0700 Subject: [Python-Dev] Community buildbots -- reprise In-Reply-To: <44C1E447.2020404@v.loewis.de> References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com> <44C1E447.2020404@v.loewis.de> Message-ID: <3f09d5a00607220832n57f0c5a6m8a193eb8e405846a@mail.gmail.com> On 7/22/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: > > Grig Gheorghiu wrote: > > Please let me know if you're interested. > > As I said earlier: If you need some kind of post-commit > trigger on the python repository to trigger a build, just > let me know. We currently use a more-or-less plain > svn_buildbot.py to trigger our own builds. Wouldn't that put too much of a burden on the python core build system? It would have to be aware of all the buildslaves running specific projects. I was thinking about having a dedicated buildmaster machine, such as the one Neal says he already has, and configure that machine to coordinate a small army of buildslaves which will be contributed for people interested in this effort. Grig -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060722/f68aec48/attachment.htm From martin at v.loewis.de Sat Jul 22 17:53:23 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 22 Jul 2006 17:53:23 +0200 Subject: [Python-Dev] Community buildbots -- reprise In-Reply-To: <3f09d5a00607220832n57f0c5a6m8a193eb8e405846a@mail.gmail.com> References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com> <44C1E447.2020404@v.loewis.de> <3f09d5a00607220832n57f0c5a6m8a193eb8e405846a@mail.gmail.com> Message-ID: <44C249F3.8000908@v.loewis.de> Grig Gheorghiu wrote: > As I said earlier: If you need some kind of post-commit > trigger on the python repository to trigger a build, just > let me know. We currently use a more-or-less plain > svn_buildbot.py to trigger our own builds. > > Wouldn't that put too much of a burden on the python core build system? > It would have to be aware of all the buildslaves running specific projects. If there is a single "community buildbot", then no. In any case, it's primarily administrative overhead, not so much cycles. python.org does so many things simultaneously, making it trigger an additional build remotely doesn't hurt. > I was thinking about having a dedicated buildmaster machine, such as the > one Neal says he already has, and configure that machine to coordinate a > small army of buildslaves which will be contributed for people > interested in this effort. Right. You still need to find out when to rebuild, and getting triggers from the source repositories is likely the easiest solution. Regards, Martin From brett at python.org Sat Jul 22 19:30:23 2006 From: brett at python.org (Brett Cannon) Date: Sat, 22 Jul 2006 10:30:23 -0700 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <20060722114615.GB13782@code0.codespeak.net> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> Message-ID: <bbaeab100607221030v5df16602ya72f58a43e82006b@mail.gmail.com> On 7/22/06, Armin Rigo <arigo at tunes.org> wrote: > Re-hi, > > On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote: > > http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log . > > I'm not sure I understand what you propose to fix holes like > constructors and __subclasses__: it seems that you want to remove them > altogether (and e.g. make factory functions instead). That would > completely break all programs, right? Not altogether, just constructors on select types who are considered dangerous from a security standpoint. The breakage won't be horrible, but it will be there for advanced Python code. I will try to make the wording more clear when I get back to work on Tuesday. > I mean, there is no way such > changes would go into mainstream CPython. If this has to wait until Py3k then so be it. > Or do you propose to maintain > a CPython branch manually for the foreseeable future? (From experience > this is a bad idea...) > Yeah, not my idea of fun either, but since this is a long term project, I will at least need to for the foreseeable future. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060722/3f1b39e2/attachment.htm From brett at python.org Sat Jul 22 19:33:19 2006 From: brett at python.org (Brett Cannon) Date: Sat, 22 Jul 2006 10:33:19 -0700 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <20060722070710.GA22334@code0.codespeak.net> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722070710.GA22334@code0.codespeak.net> Message-ID: <bbaeab100607221033t229d80c3h35b669009787cc87@mail.gmail.com> On 7/22/06, Armin Rigo <arigo at tunes.org> wrote: > > Hi Brett, > > On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote: > > I also plan to rewrite the import machinery in pure Python. > > http://codespeak.net/svn/pypy/dist/pypy/module/__builtin__/importing.py Thanks for the link, Armin. Since you guys don't have the import restrictions the CPython version would have and just have different coding needs for RPython obviously I can't just do a blind copy. But I will definitely take a look as I develop. Maybe you guys can even help to lower the duplication if it makes sense for you. BTW, do you guys happen to have extra tests from import? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060722/208e6338/attachment.html From grig.gheorghiu at gmail.com Sat Jul 22 20:16:00 2006 From: grig.gheorghiu at gmail.com (Grig Gheorghiu) Date: Sat, 22 Jul 2006 11:16:00 -0700 Subject: [Python-Dev] Community buildbots -- reprise In-Reply-To: <44C249F3.8000908@v.loewis.de> References: <3f09d5a00607211004kb45ffe2kcedcf418136e7102@mail.gmail.com> <44C1E447.2020404@v.loewis.de> <3f09d5a00607220832n57f0c5a6m8a193eb8e405846a@mail.gmail.com> <44C249F3.8000908@v.loewis.de> Message-ID: <3f09d5a00607221116m37854df2kafaa381c9e2f81dc@mail.gmail.com> On 7/22/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: > > Grig Gheorghiu wrote: > > As I said earlier: If you need some kind of post-commit > > trigger on the python repository to trigger a build, just > > let me know. We currently use a more-or-less plain > > svn_buildbot.py to trigger our own builds. > > > > Wouldn't that put too much of a burden on the python core build system? > > It would have to be aware of all the buildslaves running specific > projects. > > If there is a single "community buildbot", then no. In any case, it's > primarily administrative overhead, not so much cycles. python.org does > so many things simultaneously, making it trigger an additional build > remotely doesn't hurt. > > > I was thinking about having a dedicated buildmaster machine, such as the > > one Neal says he already has, and configure that machine to coordinate a > > small army of buildslaves which will be contributed for people > > interested in this effort. > > Right. You still need to find out when to rebuild, and getting triggers > from the source repositories is likely the easiest solution. I see....I guess I was thinking about building periodically (every X hours or at time Y) as opposed to getting svn triggers on each check-in. But if, as you're saying, the overhead on python.org is not too great, we can do what you suggested. Grig -- http://agiletesting.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060722/b7d1a491/attachment.htm From greg.ewing at canterbury.ac.nz Sun Jul 23 03:24:43 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 23 Jul 2006 13:24:43 +1200 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <20060722114615.GB13782@code0.codespeak.net> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> Message-ID: <44C2CFDB.4060700@canterbury.ac.nz> Armin Rigo wrote: > I'm not sure I understand what you propose to fix holes like > constructors and __subclasses__: it seems that you want to remove them > altogether (and e.g. make factory functions instead). That would > completely break all programs, right? I mean, there is no way such > changes would go into mainstream CPython. How much code is actually out there that uses __subclasses__? It seems like a fairly esoteric corner of the language to me. In any case, I think this approach should certainly be tried, and if it works out, considered for Py3k. -- Greg From david.nospam.hopwood at blueyonder.co.uk Sun Jul 23 03:18:48 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Sun, 23 Jul 2006 02:18:48 +0100 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <20060722114615.GB13782@code0.codespeak.net> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> Message-ID: <44C2CE78.9020802@blueyonder.co.uk> Armin Rigo wrote: > Re-hi, > > On Wed, Jul 19, 2006 at 03:35:45PM -0700, Brett Cannon wrote: > >>http://svn.python.org/view/python/branches/bcannon-sandboxing/securing_python.txt?rev=50717&view=log. > > I'm not sure I understand what you propose to fix holes like > constructors and __subclasses__: it seems that you want to remove them > altogether (and e.g. make factory functions instead). That would > completely break all programs, right? I mean, there is no way such > changes would go into mainstream CPython. If I understand correctly, the proposal is that any incompatible changes to the language would apply only in "sandboxed" interpreters. So there is no reason why support for these couldn't go into the main branch. Of course we want to minimize the changes that will need to be made to programs and libraries to make them work in a sandboxed interpreter, but not at the expense of security. Some incompatible changes will be necessary. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From arigo at tunes.org Sun Jul 23 08:54:10 2006 From: arigo at tunes.org (Armin Rigo) Date: Sun, 23 Jul 2006 08:54:10 +0200 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607221033t229d80c3h35b669009787cc87@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722070710.GA22334@code0.codespeak.net> <bbaeab100607221033t229d80c3h35b669009787cc87@mail.gmail.com> Message-ID: <20060723065409.GC13782@code0.codespeak.net> Hi Brett, On Sat, Jul 22, 2006 at 10:33:19AM -0700, Brett Cannon wrote: > Thanks for the link, Armin. Since you guys don't have the import > restrictions the CPython version would have and just have different coding > needs for RPython obviously I can't just do a blind copy. But I will > definitely take a look as I develop. Maybe you guys can even help to lower > the duplication if it makes sense for you. Yes, it should be possible to abstract the common logic in some way, using some kind of interface for all OS inspection and 'sys.modules' manipulations. > BTW, do you guys happen to have extra tests from import? Yes, there is http://codespeak.net/svn/pypy/dist/pypy/module/__builtin__/test/test_import.py which will also need a bit of rewriting, but that should be straightforward. A bientot, Armin From arigo at tunes.org Sun Jul 23 09:08:12 2006 From: arigo at tunes.org (Armin Rigo) Date: Sun, 23 Jul 2006 09:08:12 +0200 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <44C2CE78.9020802@blueyonder.co.uk> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> Message-ID: <20060723070811.GD13782@code0.codespeak.net> Hi David, hi Brett, On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote: > If I understand correctly, the proposal is that any incompatible changes > to the language would apply only in "sandboxed" interpreters. So there is > no reason why support for these couldn't go into the main branch. That's what I originally thought too, but Brett writes: Implementation Details ======================== An important point to keep in mind when reading about the implementation details for the security model is that these are general changes and are not special to any type of interpreter, sandboxed or otherwise. That means if a change to a built-in type is suggested and it does not involve a proxy, that change is meant Python-wide for *all* interpreters. So that's why I'm starting to worry that Brett is proposing to change the regular Python language too. However, Brett, you also say somewhere else that backward compatibility is not an issue. So I'm a bit confused actually... Also, I hate to sound self-centered, but I should point out somewhere that PyPy was started by people who no longer wanted to maintain a fork of CPython, and preferred to work on building CPython-like variants automatically. Many of the security features you list would be quite easier to implement and maintain in PyPy than CPython -- also from a security perspective: it is easier to be sure that some protection is complete, and remains complete over time, if it is systematically generated instead of hand-patched in a dozen places. A bientot, Armin From rasky at develer.com Sun Jul 23 15:30:50 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 23 Jul 2006 15:30:50 +0200 Subject: [Python-Dev] Document performance requirements? References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com><200607210759.56595.ndbecker2@gmail.com><44C0C79E.3020309@gmail.com><bb8868b90607210713p4599e8c3ka5fc8c61f8679486@mail.gmail.com><44C18E89.5090603@gmail.com> <20060722111907.GA13782@code0.codespeak.net> Message-ID: <008201c6ae5c$36b4ca40$d503030a@trilan> Armin Rigo wrote: > I think that O-wise the current CPython situation should be documented > as a "minimal requirement" for implementations of the language, with > just one exception: the well-documented "don't rely on this" hack in > 2.4 to make repeated 'str += str' amortized linear, for which the 2.3 > quadratic behavior is considered compliant enough. > > I suppose that allowing implementations to provide better algorithmic > complexities than required is fine, although I can think of some > problems with that (e.g. nice and efficient user code that would > perform horribly badly on CPython). I'm not sure big-O tells the whole truth. For instance, do we want to allow an implementation to use a hash table as underlying type for a list? It would match big-O requirements, but would still be slower than a plain array because of higher overhead of implementation (higher constant factor). And if this is allowed, I would like to find in CPython tutorials and documentations a simple statement like: "to implement the list and match its requirements, CPython choose a simple array as underlying data structure". -- Giovanni Bajo From tnelson at onresolve.com Sun Jul 23 19:42:59 2006 From: tnelson at onresolve.com (Trent Nelson) Date: Sun, 23 Jul 2006 19:42:59 +0200 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization Message-ID: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg> Hi, Has anyone else built Python with Visual Studio 2005 and played around with Profile Guided Optimization? I had to build Python from source w/ VS 2005 as I had a few .pyd's built with VS 2005 that I wanted to load; I ended up playing around with Profile Guided Optimization, running ``python.exe pystones.py'' to collect call-graph data after python.exe/Python24.dll had been instrumented, then recompiling with the optimizations fed back in. Results were interesting, an average speedup of around 33% was noticeable: ActiveState 2.4.3 python.exe: C:\Python24>python.exe Lib\test\pystone.py Pystone(1.1) time for 50000 passes = 0.980119 This machine benchmarks at 51014.2 pystones/second The python compiled from branches/release24-maint with VS 2005 + profile guided optimization: C:\Python24>python.exe Lib\test\pystone.py Pystone(1.1) time for 50000 passes = 0.73261 This machine benchmarks at 68249.2 pystones/second Is there any motivation in the Win32 Python dev camp to switch from VC6 to VS 2005? FWIW, although there were a shed-load of warnings when compiling python and pythoncore (and a lot more errors when compiling other modules), I only had to apply one patch to get it working well enough to run pystone.py. Without this patch, the VC8 CRT aborts at runtime as soon as an invalid signal is passed to signal(); which is inevitable given the current code in the initsignal() method: for (i = 1; i < NSIG; i++) { void (*t)(int); t = PyOS_getsig(i); Regards, Trent. -- http://www.onresolve.com Index: signalmodule.c =================================================================== --- signalmodule.c (revision 47196) +++ signalmodule.c (working copy) @@ -280,7 +280,21 @@ {NULL, NULL} /* sentinel */ }; +#define WIN32VS2005HACK +#ifdef WIN32VS2005HACK +#include <stdio.h> +#include <stdlib.h> +#include <crtdbg.h> +void dummy_handler(const wchar_t *exp, + const wchar_t *fn, + const wchar_t *file, + unsigned int line, + uintptr_t reserved) +{ +} +#endif + PyDoc_STRVAR(module_doc, "This module provides mechanisms to use signal handlers in Python.\n\ \n\ @@ -339,6 +353,12 @@ goto finally; Py_INCREF(IntHandler); +#ifdef WIN32VS2005HACK + (void)_set_invalid_parameter_handler(&dummy_handler); + _CrtSetReportMode(_CRT_ASSERT, 0); +#endif + + Handlers[0].tripped = 0; for (i = 1; i < NSIG; i++) { void (*t)(int); From tonynelson at georgeanelson.com Sun Jul 23 19:26:59 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Sun, 23 Jul 2006 13:26:59 -0400 Subject: [Python-Dev] Socket Timeouts patch 1519025 Message-ID: <v04020a00c0e95c95d9bc@[192.168.123.162]> I request a review of my patch (1519025) to get socket timeouts to work properly with errors and signals. I don't expect this patch would make it into 2.5, but perhaps it could be in 2.5.1, as it fixes a long-standing bug. I know that people are busy with getting 2.5 out the door, but it would be helpful for me to know if my current patch is OK before I start on another patch to make socket timeouts more useful. There is also a version of the patch for 2.4, which would make yum nicer in Fedora 4 and 5, and I think that passing a review would make the patch more acceptable to Fedora's maintainers. My next patch will, if it works, make socket timeouts easier to use per-thread, allow for the timing of entire operations rather than just timing transaction phases, allow for setting an acceptable rate for file transfers, and should be completely backward compatible, in that old code would be unaffected and new code would work as well as possible now on older unpatched versions. That's my plan, anyway. It would build on my current patch, at least in its principles. ____________________________________________________________________ TonyN.:' <mailto:tonynelson at georgeanelson.com> ' <http://www.georgeanelson.com/> From brett at python.org Sun Jul 23 22:00:06 2006 From: brett at python.org (Brett Cannon) Date: Sun, 23 Jul 2006 13:00:06 -0700 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <20060723070811.GD13782@code0.codespeak.net> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> Message-ID: <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com> On 7/23/06, Armin Rigo <arigo at tunes.org> wrote: > > Hi David, hi Brett, > > On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote: > > If I understand correctly, the proposal is that any incompatible changes > > to the language would apply only in "sandboxed" interpreters. So there > is > > no reason why support for these couldn't go into the main branch. > > That's what I originally thought too, but Brett writes: > > Implementation Details > ======================== > > An important point to keep in mind when reading about the > implementation details for the security model is that these are > general changes and are not special to any type of interpreter, > sandboxed or otherwise. That means if a change to a built-in type is > suggested and it does not involve a proxy, that change is meant > Python-wide for *all* interpreters. > > So that's why I'm starting to worry that Brett is proposing to change > the regular Python language too. Yes, I am proposing changing some constructors and methods on some built-in types for the regular languages. That's it. No new keywords or major semantic changes and such. If I make changes just for sandboxed interpreters it changes the general approach of the security model by then requiring an identity check to see if the interpreter is sandboxed or not. However, Brett, you also say somewhere > else that backward compatibility is not an issue. So I'm a bit confused > actually... Since this is my Ph.D. dissertation first and foremost, I am not going to tie my hands in such a way that I have to make too much of a compromise in order for this to work. I obviously don't want to change the feel of Python, but if I have to remove the constructor for code objects to prevent evil bytecode or __subclasses__() from object to prevent poking around stuff, then so be it. For this project, security is trumpeting backwards-compatibility when the latter is impossible in order to have the former. I will obviously try to minimize it, but something that works at such a basic level of the language is just going to require some changes for it to work. Also, I hate to sound self-centered, but I should point out somewhere > that PyPy was started by people who no longer wanted to maintain a fork > of CPython, and preferred to work on building CPython-like variants > automatically. Many of the security features you list would be quite > easier to implement and maintain in PyPy than CPython -- also from a > security perspective: it is easier to be sure that some protection is > complete, and remains complete over time, if it is systematically > generated instead of hand-patched in a dozen places. It doesn't sound self-centered. =) Problem is that my knowledge base is obviously all in CPython so my startup costs are much lower than if I tried this in PyPy. Plus there is the point of embedding this into Firefox (possibly) eventually. Does PyPy support embedding yet at the C level? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060723/229c8fb8/attachment.htm From rasky at develer.com Sun Jul 23 22:41:46 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 23 Jul 2006 22:41:46 +0200 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg> Message-ID: <034401c6ae98$6a0da920$d503030a@trilan> Trent Nelson wrote: > Has anyone else built Python with Visual Studio 2005 and played around > with Profile Guided Optimization? Yes, there was some work at the recent Need for Speed sprint. Python 2.5 has a PCBuild8 directory (for VS 2005) with a specific project for PGO. > Results were interesting, an average speedup of around 33% was > noticeable: Yes, they are. > Is there any motivation in the Win32 Python dev camp to switch from > VC6 to VS 2005? I think Martin decided to keep VC71 (Visual Studio .NET 2003) for another release cycle. Given the impressive results of VC8 with PGO, and the fact that Visual Studio Express 2005 is free forever, I would hope as well for the decision to be reconsidered. -- Giovanni Bajo From pje at telecommunity.com Sun Jul 23 22:45:54 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 23 Jul 2006 16:45:54 -0400 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.co m> References: <20060723070811.GD13782@code0.codespeak.net> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> Message-ID: <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote: >I obviously don't want to change the feel of Python, but if I have to >remove the constructor for code objects to prevent evil bytecode or >__subclasses__() from object to prevent poking around stuff, then so be >it. For this project, security is trumpeting backwards-compatibility when >the latter is impossible in order to have the former. I will obviously >try to minimize it, but something that works at such a basic level of the >language is just going to require some changes for it to work. Zope 3's sandboxing machinery manages to handle securing these things without any language changes. So, declaring it "impossible" to manage without backward compatibility seems inappropriate, or at least incorrect. But perhaps there is something I'm missing? From david.nospam.hopwood at blueyonder.co.uk Mon Jul 24 00:07:25 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Sun, 23 Jul 2006 23:07:25 +0100 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> References: <20060723070811.GD13782@code0.codespeak.net> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> Message-ID: <44C3F31D.3070201@blueyonder.co.uk> Phillip J. Eby wrote: > At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote: > >>I obviously don't want to change the feel of Python, but if I have to >>remove the constructor for code objects to prevent evil bytecode or >>__subclasses__() from object to prevent poking around stuff, then so be >>it. For this project, security is [trumping] backwards-compatibility when >>the latter is impossible in order to have the former. I will obviously >>try to minimize it, but something that works at such a basic level of the >>language is just going to require some changes for it to work. > > Zope 3's sandboxing machinery manages to handle securing these things > without any language changes. So, declaring it "impossible" to manage > without backward compatibility seems inappropriate, or at least > incorrect. ... if Zope's sandboxing is secure. I haven't done a security review of it, but your argument assumes that it is. In any case, Zope's sandboxing is not capability-based. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From foom at fuhm.net Mon Jul 24 01:59:08 2006 From: foom at fuhm.net (James Y Knight) Date: Sun, 23 Jul 2006 19:59:08 -0400 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization In-Reply-To: <034401c6ae98$6a0da920$d503030a@trilan> References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg> <034401c6ae98$6a0da920$d503030a@trilan> Message-ID: <646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net> On Jul 23, 2006, at 4:41 PM, Giovanni Bajo wrote: > I think Martin decided to keep VC71 (Visual Studio .NET 2003) for > another > release cycle. Given the impressive results of VC8 with PGO, and > the fact > that Visual Studio Express 2005 is free forever, I would hope as > well for > the decision to be reconsidered. Wasn't there a "Free Forever" 2003 edition too, which has since completely disappeared? Why do you think that MS won't stop distributing the Free Forever VS 2005 once VS 2005+1 comes out, the same way they did the 2003 one? James From david.nospam.hopwood at blueyonder.co.uk Mon Jul 24 01:32:47 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Mon, 24 Jul 2006 00:32:47 +0100 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com> Message-ID: <44C4071F.5050003@blueyonder.co.uk> Brett Cannon wrote: > On 7/23/06, Armin Rigo <arigo at tunes.org> wrote: > >> Hi David, hi Brett, >> >> On Sun, Jul 23, 2006 at 02:18:48AM +0100, David Hopwood wrote: >> > If I understand correctly, the proposal is that any incompatible >> > changes to the language would apply only in "sandboxed" interpreters. >> > So there is no reason why support for these couldn't go into the >> > main branch. >> >> That's what I originally thought too, but Brett writes: >> >> Implementation Details >> ======================== >> >> An important point to keep in mind when reading about the >> implementation details for the security model is that these are >> general changes and are not special to any type of interpreter, >> sandboxed or otherwise. That means if a change to a built-in type is >> suggested and it does not involve a proxy, that change is meant >> Python-wide for *all* interpreters. >> >> So that's why I'm starting to worry that Brett is proposing to change >> the regular Python language too. > > Yes, I am proposing changing some constructors and methods on some built-in > types for the regular languages. That's it. No new keywords or major > semantic changes and such. If I make changes just for sandboxed > interpreters it changes the general approach of the security model by then > requiring an identity check to see if the interpreter is sandboxed or not. I assume that the extent of incompatible changes would be limited as much as possible. So the only checks would be in operations that are directly affected by whatever incompatible changes are made. The performance and complexity costs of this are likely to be small -- or at least should not be assumed to be large before having hammered out a more detailed design. Suppose, for the sake of argument, that we introduced private methods and attributes. If an attribute in an existing standard library class was changed to be private, then code depending on it would break. But if there were a notion of a "compatibility private" attribute that acts as private only in a sandboxed interpreter, then no code running in an unprotected interpreter would break. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From tjreedy at udel.edu Mon Jul 24 04:45:27 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 23 Jul 2006 22:45:27 -0400 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg> <034401c6ae98$6a0da920$d503030a@trilan> Message-ID: <ea1c88$aa4$1@sea.gmane.org> "Giovanni Bajo" <rasky at develer.com> wrote in message news:034401c6ae98$6a0da920$d503030a at trilan... > that Visual Studio Express 2005 is free forever, I would hope as well for > the decision to be reconsidered. But is it freely redistributable forever? Or even now? I have the 2003 toolkit sitting on my disk, but I am apparently not supposed to send it to anyone else. tjr From nyamatongwe at gmail.com Mon Jul 24 05:03:22 2006 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Mon, 24 Jul 2006 13:03:22 +1000 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization In-Reply-To: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg> References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg> Message-ID: <50862ebd0607232003gc800a48r3853bd773d3a7f21@mail.gmail.com> Trent Nelson: > I ended up playing around with Profile Guided Optimization, running > ``python.exe pystones.py'' to collect call-graph data after > python.exe/Python24.dll had been instrumented, then recompiling with the > optimizations fed back in. It'd be an idea to build a larger body of Python code to run the profiling pass on so it doesn't just optimize the sort of code in pystone which is not very representative. Could run the test suite as it would have good coverage but would hit exceptional cases too heavily. Other compilers (Intel?) support profile directed optimization so would also benefit from such a body of code. Neil From pje at telecommunity.com Mon Jul 24 05:30:24 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 23 Jul 2006 23:30:24 -0400 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <44C3F31D.3070201@blueyonder.co.uk> References: <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <20060723070811.GD13782@code0.codespeak.net> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> At 11:07 PM 7/23/2006 +0100, David Hopwood wrote: >Phillip J. Eby wrote: > > At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote: > > > >>I obviously don't want to change the feel of Python, but if I have to > >>remove the constructor for code objects to prevent evil bytecode or > >>__subclasses__() from object to prevent poking around stuff, then so be > >>it. For this project, security is [trumping] backwards-compatibility when > >>the latter is impossible in order to have the former. I will obviously > >>try to minimize it, but something that works at such a basic level of the > >>language is just going to require some changes for it to work. > > > > Zope 3's sandboxing machinery manages to handle securing these things > > without any language changes. So, declaring it "impossible" to manage > > without backward compatibility seems inappropriate, or at least > > incorrect. > >... if Zope's sandboxing is secure. I haven't done a security review of it, >but your argument assumes that it is. What argument is that? I'm merely suggesting that coming up with a completely new way to secure Python without a serious consideration of existing practical prior art (with many years' deployment experience on the public internet!) seems ill-advised with respect to achieving practical goals. Brett's securing_python.txt don't refer to or cite Zope in any way, but rather relies on broad and unsupported assertions about what can or can't be done with Python. I hope he isn't doing the same in his thesis, as this is rather like writing about one's new theory of how to have a worldwide ball-kicking contest without making any reference as to how one's theory compares with the World Cup. I'm not saying Zope is better or worse. I'm simply saying that in a business context, a failure to compare and contrast a proposed "build" solution to show how it would be better than a well-established available "buy" solution would be called something like "lack of due diligence". I think in the academic context it might be called something like "failure to cite", but the general idea is the same, i.e., not doing your homework. :) In other words, if the solution being proposed is better than what Zope does, the appropriate thing in business is to show the reasons why, and the appropriate thing in science is to state a hypothesis regarding the differences, and then perform an experiment to either prove or disprove it. >In any case, Zope's sandboxing is not capability-based. You're right: you haven't done a review of it. :) If you had, you'd know that one proxy plus one namechecker equals one capability. In other words, you could take the restricted interpreter, the proxy mechanism, and the namechecker and leave most of the rest alone, and you'd have your capability system. Then you could focus more time and attention on the parts of the problem that Zope *doesn't* solve, instead of reinventing the ones that it already does. Now, if Brett believes that changing the Python language is a *better* way to implement capabilities than using proxies to implement them, then great. His paper should explain why, and (presumably) include experimental results to show that they're either better or worse than Zope's approach based on some criteria. The same information is relevant to Python-Dev as to what is an appropriate approach to support sandboxing in CPython. What are the advantages of a built-in approach versus an add-on approach? Are there interpreter facilities that could be added to shore up any awkward aspects of Zope's approach? (Whatever those might be.) For example, one part of Zope's approach uses a custom compiler and custom builtins in order to redefine how attribute access works in certain cases. Could these customizations be replaced with options built into the Python compiler and interpreter? What improvements would that result in? Simply handwaving all of these questions away, however, with broad assertions of superiority and without even attempting to compare the new work to Zope's existing work is really not acceptable for academia OR Python development. For the record: I have no personal interest in Zope's security system. I didn't develop it and haven't had the need to use it, myself. I once reviewed some of the code and offered some minor suggestions, mainly regarding performance improvement. My only axe to grind in this matter is what I've already stated: I think it would be crazy (in the "monumental waste of resources" sense) to consider putting *any* sandboxing system into CPython without tapping the Zope team's experiences. For example: having implemented such a system, what compiler or interpreter changes would've made the job easier? Meanwhile, what Brett does or doesn't put in his thesis is between him and his advisor, but what gets put into Python shouldn't be based on ignoring the existing field experience and state of the art. From unknown_kev_cat at hotmail.com Mon Jul 24 06:13:50 2006 From: unknown_kev_cat at hotmail.com (Joe Smith) Date: Mon, 24 Jul 2006 00:13:50 -0400 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg><034401c6ae98$6a0da920$d503030a@trilan> <646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net> Message-ID: <ea1hdl$k0v$1@sea.gmane.org> "James Y Knight" <foom at fuhm.net> wrote in message news:646666C6-A1DB-4D2C-8054-FA9F7E2129E4 at fuhm.net... > On Jul 23, 2006, at 4:41 PM, Giovanni Bajo wrote: >> I think Martin decided to keep VC71 (Visual Studio .NET 2003) for >> another >> release cycle. Given the impressive results of VC8 with PGO, and >> the fact >> that Visual Studio Express 2005 is free forever, I would hope as >> well for >> the decision to be reconsidered. > > Wasn't there a "Free Forever" 2003 edition too, which has since > completely disappeared? Why do you think that MS won't stop > distributing the Free Forever VS 2005 once VS 2005+1 comes out, the > same way they did the 2003 one? I am not aware of any full free ofeering based on 2003. There were ways to get the command line tools, but the GUI's were not available. Most of the freely available command line utilities are still available from microsoft if you find the correct page. I will note that according to the FAQ, Profile Guided Optimizations will not be available with Express. Because the compiler is the same, this means the tools to generate a profile are all that is missing. Express should still be able to build using the optimized profile. Microsoft as a general rule, does not go after people distributing products that Microsoft has labeled free, even after Microsoft no longer distributes that product. So the express editions will continue to be available long into the future if 2005+1 does not have a free version. (The logic behind this is that Microsoft would have a hard time explaining to a jury how somebody can "pirate" software that is available at no cost.) From unknown_kev_cat at hotmail.com Mon Jul 24 06:24:23 2006 From: unknown_kev_cat at hotmail.com (Joe Smith) Date: Mon, 24 Jul 2006 00:24:23 -0400 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg> <50862ebd0607232003gc800a48r3853bd773d3a7f21@mail.gmail.com> Message-ID: <ea1i1e$l5c$1@sea.gmane.org> "Neil Hodgson" <nyamatongwe at gmail.com> wrote in message news:50862ebd0607232003gc800a48r3853bd773d3a7f21 at mail.gmail.com... > Trent Nelson: > >> I ended up playing around with Profile Guided Optimization, running >> ``python.exe pystones.py'' to collect call-graph data after >> python.exe/Python24.dll had been instrumented, then recompiling with the >> optimizations fed back in. > > It'd be an idea to build a larger body of Python code to run the > profiling pass on so it doesn't just optimize the sort of code in > pystone which is not very representative. Could run the test suite as > it would have good coverage but would hit exceptional cases too > heavily. Other compilers (Intel?) support profile directed > optimization so would also benefit from such a body of code. > GCC suppost profiling optimized code. One caveat is that the profile-enabled builds ause a GPL'd library. Not a problem for Python right now as it is GPL-compatible, but a caveat non-the-less. That does not apply to the final optimized app. Anyway the flags used are: "-fprofile-generate" and "-fprofile-use". From martin at v.loewis.de Mon Jul 24 06:25:46 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 24 Jul 2006 06:25:46 +0200 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization In-Reply-To: <ea1hdl$k0v$1@sea.gmane.org> References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg><034401c6ae98$6a0da920$d503030a@trilan> <646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net> <ea1hdl$k0v$1@sea.gmane.org> Message-ID: <44C44BCA.30404@v.loewis.de> Joe Smith wrote: > Microsoft as a general rule, does not go after people distributing > products that Microsoft has labeled free, even after Microsoft no > longer distributes that product. So the express editions will > continue to be available long into the future if 2005+1 does not have > a free version. Interesting. So people can do the same with the free 2003 version. Regards, Martin From brett at python.org Mon Jul 24 06:27:50 2006 From: brett at python.org (Brett Cannon) Date: Sun, 23 Jul 2006 21:27:50 -0700 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <44C3F31D.3070201@blueyonder.co.uk> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> Message-ID: <bbaeab100607232127p1fdb401ct12d471925453cb9b@mail.gmail.com> On 7/23/06, Phillip J. Eby <pje at telecommunity.com> wrote: > > At 11:07 PM 7/23/2006 +0100, David Hopwood wrote: > >Phillip J. Eby wrote: [snip] Brett's securing_python.txt don't refer to or cite Zope in any way, but > rather relies on broad and unsupported assertions about what can or can't > be done with Python. I hope he isn't doing the same in his thesis, as > this > is rather like writing about one's new theory of how to have a worldwide > ball-kicking contest without making any reference as to how one's theory > compares with the World Cup. The design doc is not meant to be taken as any sort of draft of my thesis. I did read that link you sent me, Philip, but it was hard to follow. So I used Google to find another reference that explained it to me much more clearly. securing_python.txt is meant to explain what I am planning to python-dev so that if someone sees some fatal flaw they can speak up and let me know, not as a thorough comparison of why my approach is better than anyone other one. I'm not saying Zope is better or worse. I'm simply saying that in a > business context, a failure to compare and contrast a proposed "build" > solution to show how it would be better than a well-established available > "buy" solution would be called something like "lack of due diligence". I > think in the academic context it might be called something like "failure > to > cite", but the general idea is the same, i.e., not doing your > homework. :) > > In other words, if the solution being proposed is better than what Zope > does, the appropriate thing in business is to show the reasons why, and > the > appropriate thing in science is to state a hypothesis regarding the > differences, and then perform an experiment to either prove or disprove > it. I am not going to write out a blow-by-blow comparison right now. It will come with the thesis. And I am not expecting my approach or code to be checked in blindly anyway. >In any case, Zope's sandboxing is not capability-based. > > You're right: you haven't done a review of it. :) If you had, you'd know > that one proxy plus one namechecker equals one capability. In other > words, > you could take the restricted interpreter, the proxy mechanism, and the > namechecker and leave most of the rest alone, and you'd have your > capability system. Then you could focus more time and attention on the > parts of the problem that Zope *doesn't* solve, instead of reinventing the > ones that it already does. Right, but I am trying to remove the need for a namechecker which makes it an object-capabilities system. Now, if Brett believes that changing the Python language is a *better* way > to implement capabilities than using proxies to implement them, then > great. His paper should explain why, and (presumably) include > experimental > results to show that they're either better or worse than Zope's approach > based on some criteria. The same information is relevant to Python-Dev as > to what is an appropriate approach to support sandboxing in CPython. What > are the advantages of a built-in approach versus an add-on approach? Are > there interpreter facilities that could be added to shore up any awkward > aspects of Zope's approach? (Whatever those might be.) I think people are starting to lose sight of the purpose of the doc I wrote. It was to explain what I was doing for people to see if there was any fatal flaw and to keep people updated on what I am planning on doing. It is not meant to convince anyone that my way is the best way yet. I am not even going to attempt that until I have working code. For example, one part of Zope's approach uses a custom compiler and custom > builtins in order to redefine how attribute access works in certain > cases. Could these customizations be replaced with options built into the > Python compiler and interpreter? What improvements would that result in? Part of my point is to help alleviate the need for custom anything. Simply handwaving all of these questions away, however, with broad > assertions of superiority and without even attempting to compare the new > work to Zope's existing work is really not acceptable for academia OR > Python development. > > For the record: I have no personal interest in Zope's security system. I > didn't develop it and haven't had the need to use it, myself. I once > reviewed some of the code and offered some minor suggestions, mainly > regarding performance improvement. My only axe to grind in this matter is > what I've already stated: I think it would be crazy (in the "monumental > waste of resources" sense) to consider putting *any* sandboxing system > into > CPython without tapping the Zope team's experiences. For example: having > implemented such a system, what compiler or interpreter changes would've > made the job easier? > > Meanwhile, what Brett does or doesn't put in his thesis is between him and > his advisor, but what gets put into Python shouldn't be based on ignoring > the existing field experience and state of the art. There is no ignoring of anything. I understand their basic approach and I want to try another one. I like the fundemental design difference of object-capabilities and so I am going to give it a shot and see how it works out. If Zope's proxies are better, then fine, get them to contribute to the core so that we have security again. No one is proposing blind acceptance of my approach. But as it stands now, Guido likes my initial ideas (including the possible changes to the language), I like my approach, and my supervisor is willing to let me give it a shot, so that is good enough for me to move forward. There will be a thorough discussion I am sure once the code is finished and ready to be proposed for possible inclusion in CPython. Until then, though, just consider it my little experiment that I am just keeping people abreast of. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060723/bd0f74ea/attachment.htm From greg.ewing at canterbury.ac.nz Mon Jul 24 06:51:23 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 24 Jul 2006 16:51:23 +1200 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization In-Reply-To: <ea1hdl$k0v$1@sea.gmane.org> References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg> <034401c6ae98$6a0da920$d503030a@trilan> <646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net> <ea1hdl$k0v$1@sea.gmane.org> Message-ID: <44C451CB.3060607@canterbury.ac.nz> Joe Smith wrote: > Microsoft as a general rule, does not go after people distributing products > that Microsoft has labeled > free, even after Microsoft no longer distributes that product. But if the licence agreement technically forbids redistribution, it doesn't seem like a good idea to rely on Microsoft turning a blind eye to that. -- Greg From pje at telecommunity.com Mon Jul 24 08:10:44 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 24 Jul 2006 02:10:44 -0400 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607232127p1fdb401ct12d471925453cb9b@mail.gmail.co m> References: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <44C3F31D.3070201@blueyonder.co.uk> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> At 09:27 PM 7/23/2006 -0700, Brett Cannon wrote: >On 7/23/06, Phillip J. Eby ><<mailto:pje at telecommunity.com>pje at telecommunity.com> wrote: >>one proxy plus one namechecker equals one capability. In other words, >>you could take the restricted interpreter, the proxy mechanism, and the >>namechecker and leave most of the rest alone, and you'd have your >>capability system. Then you could focus more time and attention on the >>parts of the problem that Zope *doesn't* solve, instead of reinventing the >>ones that it already does. > >Right, but I am trying to remove the need for a namechecker which makes it >an object-capabilities system. As I said above: a namechecker plus a proxy *equals* an object capability. When I say "name checker" I mean the Zope type that allows you to specify a list of names that are allowed for a given object. This allowing is not based on identity or code signing or anything like that. It's just a list of attribute names: i.e. a capability mask over an existing object. When you create a proxy using this name mask, that proxy becomes a capability that allows access to the given names on the underlying object. >I like the fundemental design difference of object-capabilities It's not a difference at all, let alone a fundamental one. Zope just happens to allow other kinds of security checking *in addition to* capabilities, if you want them. However, most of its more basic encapsulation features are 100% capability based. Meanwhile, if you want to implement an object-capability system, you will need something that is basically a mask, to allow one piece of code to create capabilities that can be given to another. What you end up with for doing that is going to look almost exactly like a Zope proxy plus a Zope name checker. I hate to harp on this point, but there seems to be a trend that when people have capabilities on their mind, they tend to look at Zope and dismiss it as not being capability-based, when in fact Zope's approach is capabilities *plus* other things. (Of course, most of those "other things" have to do with closing holes like __subclasses__, while improving performance by still allowing lots of common objects not to be proxied.) From brett at python.org Mon Jul 24 09:50:09 2006 From: brett at python.org (Brett Cannon) Date: Mon, 24 Jul 2006 00:50:09 -0700 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <44C3F31D.3070201@blueyonder.co.uk> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> Message-ID: <bbaeab100607240050h142e6bbcle137ff393f454684@mail.gmail.com> On 7/23/06, Phillip J. Eby <pje at telecommunity.com> wrote: > > At 09:27 PM 7/23/2006 -0700, Brett Cannon wrote: > > When I say "name checker" I mean the Zope type that allows you to specify > a > list of names that are allowed for a given object. This allowing is not > based on identity or code signing or anything like that. It's just a list > of attribute names: i.e. a capability mask over an existing object. > > When you create a proxy using this name mask, that proxy becomes a > capability that allows access to the given names on the underlying object. OK, then using the term "namechecker" through me off. >I like the fundemental design difference of object-capabilities > > It's not a difference at all, let alone a fundamental one. Zope just > happens to allow other kinds of security checking *in addition to* > capabilities, if you want them. However, most of its more basic > encapsulation features are 100% capability based. > Meanwhile, if you want to implement an object-capability system, you will > need something that is basically a mask, to allow one piece of code to > create capabilities that can be given to another. What you end up with > for > doing that is going to look almost exactly like a Zope proxy plus a Zope > name checker. I hate to harp on this point, but there seems to be a trend that when > people have capabilities on their mind, they tend to look at Zope and > dismiss it as not being capability-based, when in fact Zope's approach is > capabilities *plus* other things. Well, Jim said that Zope proxies didn't conform to the strict definition of object-capabilities the last time this all came about: http://mail.python.org/pipermail/python-dev/2003-March/033884.html and http://mail.python.org/pipermail/python-dev/2003-March/033915.html . He said they *could* be made to be what object-capabilities is defined as, but they were not currently structured that way. Those comments are one of the reasons I never considered thinking of Zope proxies as a object-capabilities system. (Of course, most of those "other things" have to do with closing holes like > __subclasses__, while improving performance by still allowing lots of > common objects not to be proxied.) > > OK, then I need something clarified. If you read http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TransitionToSecurityProxies, it talks about creating the proxies. I get they restrict attribute access and wrap all returned objects in proxies themselves (unless they are considered safe). But to judge whether an attribute should be returned, it checks the security context. It also mentions how access to the security policy must be available so that proper security checks can be done to either grant or deny access. So what I want to know is if this security context is this global thing that proxies access every time to check whether something is allowed or not. Or is it a per-object specification? And what is the security domain for Zope proxies; objects, interpreter, running Python program, what? -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060724/133649ce/attachment-0001.html From nmm1 at cus.cam.ac.uk Mon Jul 24 11:46:52 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Mon, 24 Jul 2006 10:46:52 +0100 Subject: [Python-Dev] Strategy for converting the decimal module to C Message-ID: <E1G4x1Y-0005Vy-Js@virgo.cus.cam.ac.uk> James Y Knight <foom at fuhm.net> wrote: > > > To cut a long story short, it is impractical for a language run-time > > system to call user-defined handlers with any degree of reliability > > unless the compiled code and run-time interoperate carefully - I have > > been there and done that many times, but few people still working > > have. > > On architectures with out-of-order execution (and interrupts), you > > have to assume that an interrupt may occur anywhere, even when the > > code does not use the relevant facility. Floating-point overflow > > in the middle of a list insertion? That's to be expected. > > While this _is_ a real problem, is it _not_ a general problem as you > are describing it. Processors are perfectly capable of generating > precise interrupts, and the inability to do so has nothing to do with > the out-of-order execution, etc. Almost all interrupts are precise. I am sorry, but this is almost totally wrong, though I agree that you will get that impression upon reading the architecture books unless you are very deeply into that area. Let's skip the hardware issues, as they aren't what I am talking about (though see [*]). I am referring to the interaction between the compiled code, deep library functions and run-time interrupt handler. It is almost universal for some deep library functions and common for compiled code to leave data structures inconsistent in a short window that "cannot possibly fail" - indeed, most system interfaces do this around system calls. If an interrupt occurs then, the run-time system will receive control with those data structures in a state where they must not be accessed. And it is fairly common for such data structures to include ones critical to the functioning of the run-time system. Now, it IS possible to write run-time systems that are safe against this, and still allow asynchronous interrupts, but I am one of three people in the world that I know have done it in the past two decades. There may be as many as six, but I doubt more, and I know of no such implementation on any Unix or Microsoft system. It is even possible to do this for compiled code, but that is where the coordination between the compiler and run-time system comes in. > The only interesting one which is not, on x86 processors, is the x87 > floating point exception, ... Er, no. Try a machine-check in a TLB miss handler. But it is all pretty irrelevant, as the problem arises with asychronous exceptions (e.g. timer interrupts, signals from other processes), anyway. > Also, looking forward, the "simd" floating point instructions (ie mmx/ > sse/sse2/sse3) _do_ ... The critical problems with the x87 floating-point exception were resolved in the 80386. [*] Whether or not it is a fundamental problem, it is very much a general problem at present, and it will become more so as more CPUs implement micro-threading. For why it is tied up with out-of-order execution etc., consider a system with 100 operations flying, of which 10 are memory accesses, and then consider what happens when you have combinations of floating-point exceptions, TLB misses, machine-checks (e.g. ECC problems on memory) and device/timer interrupts. Once you add user-defined handlers into that mix, you either start exposing that mess to the program or have to implement them by stopping the CPU, unwinding the pipeline, and rerunning in very, very serial mode until the handler is called. Look at IA64 .... Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From ncoghlan at gmail.com Mon Jul 24 13:21:22 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Jul 2006 21:21:22 +1000 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com> References: <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <bbaeab100607231300w187dfa8dx87ef0a94935e56ce@mail.gmail.com> Message-ID: <44C4AD32.20309@gmail.com> Brett Cannon wrote: > On 7/23/06, *Armin Rigo* <arigo at tunes.org <mailto:arigo at tunes.org>> wrote: > Also, I hate to sound self-centered, but I should point out somewhere > that PyPy was started by people who no longer wanted to maintain a fork > of CPython, and preferred to work on building CPython-like variants > automatically. Many of the security features you list would be quite > easier to implement and maintain in PyPy than CPython -- also from a > security perspective: it is easier to be sure that some protection is > complete, and remains complete over time, if it is systematically > generated instead of hand-patched in a dozen places. > > > It doesn't sound self-centered. =) Problem is that my knowledge base > is obviously all in CPython so my startup costs are much lower than if I > tried this in PyPy. Plus there is the point of embedding this into > Firefox (possibly) eventually. Does PyPy support embedding yet at the C > level? Another rationale for basing the work on CPython is that it should be possible to implement the resulting security model regardless of the implementation language used for the interpreter core (C/Python, Java/Python, C#/Python, RPython/Python). If you can figure out how to do it in C, it should be feasible to do it in the others. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From arigo at tunes.org Mon Jul 24 14:56:40 2006 From: arigo at tunes.org (Armin Rigo) Date: Mon, 24 Jul 2006 14:56:40 +0200 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <008201c6ae5c$36b4ca40$d503030a@trilan> References: <e9qbtf$kk6$1@sea.gmane.org> <20060722111907.GA13782@code0.codespeak.net> <008201c6ae5c$36b4ca40$d503030a@trilan> Message-ID: <20060724125640.GA10183@code0.codespeak.net> Hi Giovanni, On Sun, Jul 23, 2006 at 03:30:50PM +0200, Giovanni Bajo wrote: > I'm not sure big-O tells the whole truth. For instance, do we want to allow > an implementation to use a hash table as underlying type for a list? It > would match big-O requirements, but would still be slower than a plain array > because of higher overhead of implementation (higher constant factor). A big-O difference can make the difference between a program that takes 0.5 seconds or 2 hours to run. This is more important than a constant factor difference, which different implementations are bound to exhibit anyway. > And if this is allowed, I would like to find in CPython tutorials and > documentations a simple statement like: "to implement the list and match its > requirements, CPython choose a simple array as underlying data structure". Yes, the big-O notes don't have to be too technical: the docs should tell people to think about Python lists as simple arrays, and the O requirements follow naturally. A bientot, Armin From amk at amk.ca Mon Jul 24 16:08:19 2006 From: amk at amk.ca (A.M. Kuchling) Date: Mon, 24 Jul 2006 10:08:19 -0400 Subject: [Python-Dev] Python sprint in Arlington July 29/30 Message-ID: <20060724140819.GA9507@rogue.amk.ca> The CanDo developers are sprinting for three days starting on this coming Friday, so there's space available for a Python sprint. I'll try to attend at least on Saturday (Sunday may not be possible for me). Does anyone want to come and work on Python stuff? If yes, please add your name to <http://wiki.python.org/moin/ArlingtonSprint>, which also has directions. Given that Python will still be in a code freeze, we'll have to be careful about code changes. I'm probably going to work on documentation issues. --amk From pje at telecommunity.com Mon Jul 24 18:32:32 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 24 Jul 2006 12:32:32 -0400 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <bbaeab100607240050h142e6bbcle137ff393f454684@mail.gmail.co m> References: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <44C3F31D.3070201@blueyonder.co.uk> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060724110644.02024180@sparrow.telecommunity.com> At 12:50 AM 7/24/2006 -0700, Brett Cannon wrote: >OK, then I need something clarified. If you read ><http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TransitionToSecurityProxies>http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TransitionToSecurityProxies >, it talks about creating the proxies. I get they restrict attribute >access and wrap all returned objects in proxies themselves (unless they >are considered safe). But to judge whether an attribute should be >returned, it checks the security context. That depends on the checker. The proxy implementation delegates all access decisions to a "checker" object. Some checkers check permissions, but a NamesChecker just checks a statically-defined list of names. > It also mentions how access to the security policy must be available so > that proper security checks can be done to either grant or deny access. > >So what I want to know is if this security context is this global thing >that proxies access every time to check whether something is allowed or not. Proxies don't do that; checkers do. The default Checker implementation doesn't even look at a security context if a name is declared public (i.e., it's a NamesChecker). Look at the zope.security.checker module for details. IOW, to make it a pure capabilities system, you would only *delete* code, not add any, as far as I can tell. > Or is it a per-object specification? Each proxy can have its own checker, but an individual checker instance can be shared between proxies. > And what is the security domain for Zope proxies; objects, interpreter, > running Python program, what? There are restricted eval and exec operations to run restricted code. The primary language limitations imposed are the lack of eval/exec by the restricted code, and lack of support for raise and try/except. Implementing these would require additional compiler hacking to add code to ensure that e.g. tracebacks get wrapped. From eswierk at arastra.com Mon Jul 24 19:20:34 2006 From: eswierk at arastra.com (Ed Swierk) Date: Mon, 24 Jul 2006 17:20:34 +0000 (UTC) Subject: [Python-Dev] setup.py and cross-compiling Message-ID: <loom.20060724T190947-141@post.gmane.org> I'm cross-compiling Python using uClibc buildroot, which installs target include files and libraries to a directory like ~/toolchain rather than /usr. I couldn't figure out any way to convincing the top-level python/setup.py to look in ~/toolchain instead of /usr when detecting what modules to build. I decided to hack up setup.py so that an optional root directory (passed via an environment variable) is prepended to all the hardcoded paths like "/usr/include", "/lib", "/lib64", and so on. Am I missing an easier solution to this problem? I'm happy to post my patch to setup.py if anyone is interested. --Ed From unknown_kev_cat at hotmail.com Mon Jul 24 19:35:36 2006 From: unknown_kev_cat at hotmail.com (Joe Smith) Date: Mon, 24 Jul 2006 13:35:36 -0400 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg><034401c6ae98$6a0da920$d503030a@trilan><646666C6-A1DB-4D2C-8054-FA9F7E2129E4@fuhm.net><ea1hdl$k0v$1@sea.gmane.org> <44C451CB.3060607@canterbury.ac.nz> Message-ID: <ea30d8$64o$1@sea.gmane.org> "Greg Ewing" <greg.ewing at canterbury.ac.nz> wrote in message news:44C451CB.3060607 at canterbury.ac.nz... > Joe Smith wrote: > >> Microsoft as a general rule, does not go after people distributing >> products >> that Microsoft has labeled >> free, even after Microsoft no longer distributes that product. > > But if the licence agreement technically forbids > redistribution, it doesn't seem like a good idea > to rely on Microsoft turning a blind eye to that. I agree, although it seems very unlikely Microsoft would bother as they are busy enough the the pirating of the for-cost software. I was a little surprised that they did not allow verbatim duplicates of that software, expecially as it has the feel of freeware, or even shareware (the commecial versions have additional features). I would actually not be surprised if somebody could convince Microsoft to allow them to distribute these versions even after Microsoft no longer supports them. That would be the ideal situation. From david.nospam.hopwood at blueyonder.co.uk Mon Jul 24 18:22:39 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Mon, 24 Jul 2006 17:22:39 +0100 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> References: <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <20060723070811.GD13782@code0.codespeak.net> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> Message-ID: <44C4F3CF.1060104@blueyonder.co.uk> Phillip J. Eby wrote: > At 11:07 PM 7/23/2006 +0100, David Hopwood wrote: >> Phillip J. Eby wrote: >> > At 01:00 PM 7/23/2006 -0700, Brett Cannon wrote: >> > >> >> I obviously don't want to change the feel of Python, but if I have to >> >> remove the constructor for code objects to prevent evil bytecode or >> >> __subclasses__() from object to prevent poking around stuff, then so be >> >> it. For this project, security is [trumping] backwards-compatibility when >> >> the latter is impossible in order to have the former. I will obviously >> >> try to minimize it, but something that works at such a basic level of the >> >> language is just going to require some changes for it to work. >> > >> > Zope 3's sandboxing machinery manages to handle securing these things >> > without any language changes. So, declaring it "impossible" to manage >> > without backward compatibility seems inappropriate, or at least >> > incorrect. >> >> ... if Zope's sandboxing is secure. I haven't done a security review >> of it, but your argument assumes that it is. > > What argument is that? You said "Zope 3's sandboxing machinery manages to handle securing these things without any language changes." This assertion assumes that Zope 3's sandboxing machinery is secure. > I'm merely suggesting that coming up with a > completely new way to secure Python without a serious consideration of > existing practical prior art (with many years' deployment experience on > the public internet!) seems ill-advised with respect to achieving > practical goals. > > Brett's securing_python.txt don't refer to or cite Zope in any way, This is indeed an omission that should be corrected, in order to explain why this project is not using Zope or following Zope's approach, and what the differences are. (I've explained some of them below.) > but rather relies on broad and unsupported assertions about what can or > can't be done with Python. I hope he isn't doing the same in his > thesis, as this is rather like writing about one's new theory of how to > have a worldwide ball-kicking contest without making any reference as to > how one's theory compares with the World Cup. > > I'm not saying Zope is better or worse. I'm simply saying that in a > business context, a failure to compare and contrast a proposed "build" > solution to show how it would be better than a well-established > available "buy" solution would be called something like "lack of due > diligence". I think in the academic context it might be called > something like "failure to cite", but the general idea is the same, > i.e., not doing your homework. :) > > In other words, if the solution being proposed is better than what Zope > does, the appropriate thing in business is to show the reasons why, and > the appropriate thing in science is to state a hypothesis regarding the > differences, and then perform an experiment to either prove or disprove it. I completely agree with this. >> In any case, Zope's sandboxing is not capability-based. > > You're right: you haven't done a review of it. :) I haven't done a detailed security review. However, I wouldn't have commented on it without knowing what its basic approach is. From <http://www.zope.org/Wikis/DevSite/Projects/ComponentArchitecture/TransitionToSecurityProxies>: # When an object is passed to untrusted code, it is wrapped in a security # proxy unless it is already wrapped. Security proxies mediate all accesses # to the wrapped object. Operations on security proxies return security # proxies as well. Security proxies passed from untrusted code to trusted # code remain wrapped, so untrusted code can't trick trusted code into # performing operations that the untrusted code could not perform. Restricting the actions of "trusted" code when called by "untrusted" code is precisely what a capability system does *not* do. Indeed, capability systems have no binary distinction between "trusted" and "untrusted" code (outside the system TCB, which is as small as possible); the only security distinction between protection domains is in what capabilities they hold. The main reason why capability systems do not have any such restriction is that it reduces the system's ability to support fine-grained delegation of authority. We want to be able to grant an object just the authority it needs for any particular task. For that to work, code that is otherwise untrusted must be able to use any capability it is given, even if it is a very powerful capability, and therefore must be able to call into more-trusted code without restriction. To characterise this as "tricking" the more-trusted code is a misconception: if it is not intended that a particular protection domain should have some authority, then it should not be given a capability for that authority in the first place. Capability-based security design emphasises preventing protection domains from gaining unintended capabilities; it is not about restricting their use once granted (except to the extent needed to support revocation, which is typically done via higher-level patterns rather than in the basic access control mechanisms). The kind of restrictions that Zope enforces would be a significant impediment to this approach: # With care, trusted code can explicitly unwrap security proxies and gain # additional access. In particular, security proxies cannot be stored in the # Zope object database. If an object wrapped by a security proxy is to be # stored in another object, the security proxy must be removed. This is a severe limitation to expressiveness; it effectively means that references to secure objects are not first-class. In a capability system, they must be first-class. > If you had, you'd know that one proxy plus one namechecker equals one > capability. No, because Zope enforces some restrictions that are not desired in a capability system. (It is absolutely mistaken to think that a system that enforces more restrictions must be more secure, in case anyone is under that impression.) I do not know whether the easiest way to get to a capability-secure version of Python would be by changing the restrictions enforced by Zope, or by adding restrictions to CPython (or another Python implementation). My point is simply that the security model of the Zope sandboxing system as it stands, is quite different from that of a capability system. [...] > Now, if Brett believes that changing the Python language is a *better* > way to implement capabilities than using proxies to implement them, then > great. His paper should explain why, and (presumably) include > experimental results to show that they're either better or worse than > Zope's approach based on some criteria. Requiring experimental results is not appropriate for a proposal that has not been implemented. It is also not a paper; it's an informal, early-stage design outline. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From unknown_kev_cat at hotmail.com Mon Jul 24 22:08:43 2006 From: unknown_kev_cat at hotmail.com (Joe Smith) Date: Mon, 24 Jul 2006 16:08:43 -0400 Subject: [Python-Dev] Python 2.4, VS 2005 & Profile Guided Optmization References: <AB1BB4CF81B9214682079EB55A7125EDEFA707@mapibe05.exchange.xchg><50862ebd0607232003gc800a48r3853bd773d3a7f21@mail.gmail.com> <ea1i1e$l5c$1@sea.gmane.org> Message-ID: <ea39ca$7af$1@sea.gmane.org> "Joe Smith" <unknown_kev_cat at hotmail.com> wrote in message news:ea1i1e$l5c$1 at sea.gmane.org... > > "Neil Hodgson" <nyamatongwe at gmail.com> wrote in message > news:50862ebd0607232003gc800a48r3853bd773d3a7f21 at mail.gmail.com... >> Trent Nelson: >> >>> I ended up playing around with Profile Guided Optimization, running >>> ``python.exe pystones.py'' to collect call-graph data after >>> python.exe/Python24.dll had been instrumented, then recompiling with the >>> optimizations fed back in. >> >> It'd be an idea to build a larger body of Python code to run the >> profiling pass on so it doesn't just optimize the sort of code in >> pystone which is not very representative. Could run the test suite as >> it would have good coverage but would hit exceptional cases too >> heavily. Other compilers (Intel?) support profile directed >> optimization so would also benefit from such a body of code. >> > > GCC suppost profiling optimized code. Err... That was supposed to say "supports profile-optimized code". The rest of the message was indeed about Profile Guided Optimization in gcc. From martin at v.loewis.de Mon Jul 24 23:25:58 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 24 Jul 2006 23:25:58 +0200 Subject: [Python-Dev] setup.py and cross-compiling In-Reply-To: <loom.20060724T190947-141@post.gmane.org> References: <loom.20060724T190947-141@post.gmane.org> Message-ID: <44C53AE6.7040105@v.loewis.de> Ed Swierk wrote: > I decided to hack up setup.py so that an optional root directory (passed via an > environment variable) is prepended to all the hardcoded paths like > "/usr/include", "/lib", "/lib64", and so on. I doubt this solves the problem. Distutils just doesn't support cross-compilation, period. The main problem is that it is the host python that runs setup.py, not the target python. Various parts of distutils assume that the information the interpreter provides is correct, yet it is not in a cross-compilation case. I'm uncertain how to solve this; using a target Makefile and pyconfig.h instead of the installed one might be a starting point. One should then review distutils to find out where it accesses host information when target information is required, and then find a way to come up with that information. If no information is available, then distutils should fail instead of returning incorrect information. Regards, Martin From scott+python-dev at scottdial.com Tue Jul 25 00:43:57 2006 From: scott+python-dev at scottdial.com (Scott Dial) Date: Mon, 24 Jul 2006 18:43:57 -0400 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <20060724125640.GA10183@code0.codespeak.net> References: <e9qbtf$kk6$1@sea.gmane.org> <20060722111907.GA13782@code0.codespeak.net> <008201c6ae5c$36b4ca40$d503030a@trilan> <20060724125640.GA10183@code0.codespeak.net> Message-ID: <44C54D2D.3070005@scottdial.com> Between the two of you, I think you have made the case that the language specification is better to not include such details. As you both note, it is difficult to capture the essence of what is desired from the performance of the implementation. To tag on other version, what about Big-O space concerns with things like list.sort. I'm sure there are other things to add as well. It seems reasonable to me that everyone has the same interests in mind when they write a program. Make it good, make it fast, make it small, etc. These sort of details should work themselves out if they are actually important. All of these algorithms should be treated as implementation accidents. Having the information about CPython's implementation in the docs would be good. And go most of the way towards having everyone on the same page. -- Scott Dial scott at scottdial.com scodial at indiana.edu From martin at v.loewis.de Tue Jul 25 01:19:25 2006 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Tue, 25 Jul 2006 01:19:25 +0200 Subject: [Python-Dev] setup.py and cross-compiling In-Reply-To: <c1bf1cf0607241519j157a7050l29998cecc4953485@mail.gmail.com> References: <loom.20060724T190947-141@post.gmane.org> <44C53AE6.7040105@v.loewis.de> <c1bf1cf0607241519j157a7050l29998cecc4953485@mail.gmail.com> Message-ID: <44C5557D.4010004@v.loewis.de> Ed Swierk wrote: > Well, it seems buildroot solves this main problem by building another > version of python and pygen that run on the build machine, and hacks > the Makefile to run setup.py with these instead of whatever happens to > be sitting in /usr/bin. If you think its useful, please submit a patch to sf.net/projects/python It would be good if you could get other people that attempt to cross-build Python to comment. A detailed howto might help to get them started with your code. Regards, Martin From greg.ewing at canterbury.ac.nz Tue Jul 25 02:04:44 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 25 Jul 2006 12:04:44 +1200 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> References: <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <44C3F31D.3070201@blueyonder.co.uk> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> Message-ID: <44C5601C.4070605@canterbury.ac.nz> Phillip J. Eby wrote: > When I say "name checker" I mean the Zope type that allows you to specify a > list of names that are allowed for a given object. This allowing is not > based on identity or code signing or anything like that. It's just a list > of attribute names: i.e. a capability mask over an existing object. But this is backwards from what a true object-capability system should be like if it's properly designed. Instead of starting with too-powerful objects and trying to hide some of their powers, the different powers should be separated into different objects in the first place. It sounds to me like Zope is using the approach it's using because it's having to work with Python as it currently is, not because its approach is the best one. -- Greg From greg.ewing at canterbury.ac.nz Tue Jul 25 02:14:34 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 25 Jul 2006 12:14:34 +1200 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <E1G4x1Y-0005Vy-Js@virgo.cus.cam.ac.uk> References: <E1G4x1Y-0005Vy-Js@virgo.cus.cam.ac.uk> Message-ID: <44C5626A.8080306@canterbury.ac.nz> Nick Maclaren wrote: > Er, no. Try a machine-check in a TLB miss handler. But it is all > pretty irrelevant, as the problem arises with asychronous exceptions > (e.g. timer interrupts, signals from other processes), anyway. But we weren't talking about asynchronous exceptions, we were talking about floating point exceptions. Unless your TLB miss handler uses floating point arithmethic, there's no way it can get interrupted by one. (And if it does use floating point arithmetic in a way that can cause an exception, you'd better write it to deal with that!) -- Greg From pje at telecommunity.com Tue Jul 25 03:19:33 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 24 Jul 2006 21:19:33 -0400 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <44C5601C.4070605@canterbury.ac.nz> References: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <44C3F31D.3070201@blueyonder.co.uk> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060724211442.0262c008@sparrow.telecommunity.com> At 12:04 PM 7/25/2006 +1200, Greg Ewing wrote: >Phillip J. Eby wrote: > > > When I say "name checker" I mean the Zope type that allows you to > specify a > > list of names that are allowed for a given object. This allowing is not > > based on identity or code signing or anything like that. It's just a list > > of attribute names: i.e. a capability mask over an existing object. > >But this is backwards from what a true object-capability >system should be like if it's properly designed. Instead >of starting with too-powerful objects and trying to >hide some of their powers, the different powers should >be separated into different objects in the first place. And what about code that needs to pass on a subset of a capability? You need the ability to create such capability-restricted subsets anyway, no matter how "pure" a system you start with. And being able to create capability masks for existing objects means you don't have to redesign every piece of code ever written for Python to make it secure. >It sounds to me like Zope is using the approach it's >using because it's having to work with Python as it >currently is, not because its approach is the best one. Well, that depends a lot on how you define "best". Practicality beats purity, doesn't it? ;) From david.nospam.hopwood at blueyonder.co.uk Tue Jul 25 02:36:48 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Tue, 25 Jul 2006 01:36:48 +0100 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <5.1.1.6.0.20060724211442.0262c008@sparrow.telecommunity.com> References: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <44C3F31D.3070201@blueyonder.co.uk> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> <5.1.1.6.0.20060724211442.0262c008@sparrow.telecommunity.com> Message-ID: <44C567A0.2080906@blueyonder.co.uk> Phillip J. Eby wrote: > At 12:04 PM 7/25/2006 +1200, Greg Ewing wrote: >>Phillip J. Eby wrote: >> >>>When I say "name checker" I mean the Zope type that allows you to specify >>>a list of names that are allowed for a given object. This allowing is not >>>based on identity or code signing or anything like that. It's just a list >>>of attribute names: i.e. a capability mask over an existing object. >> >>But this is backwards from what a true object-capability >>system should be like if it's properly designed. Instead >>of starting with too-powerful objects and trying to >>hide some of their powers, the different powers should >>be separated into different objects in the first place. > > And what about code that needs to pass on a subset of a capability? You > need the ability to create such capability-restricted subsets anyway, no > matter how "pure" a system you start with. That is true, but doing so for every secure object has significant costs. A simple wrapper is not sufficient, because it would not prevent a wrapped object from returning a reference to itself, bypassing the wrapper. To solve this problem you need to use the more complex Membrane pattern, which also wraps the results of method calls on a wrapped object, for example. In fact Zope's approach does implement a membrane, but this does not really dent the argument that Greg Ewing was making. A pure capability system incurs the complexity and performance costs of wrappers or membranes only in cases where they are needed, not for every object, and the complexity is only in user code, not in the system's security kernel. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From nnorwitz at gmail.com Tue Jul 25 06:20:40 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 24 Jul 2006 21:20:40 -0700 Subject: [Python-Dev] outstanding bugs to fix for 2.5 Message-ID: <ee2a432c0607242120o5365d6b4w2a8b200253686ea6@mail.gmail.com> There are still a bunch of outstanding bugs. rc1 is about a week away and it would be great to fix these. Many of these are also present in 2.4, but it would be nice to squash them in 2.5. Here's the list from PEP 356: http://python.org/sf/1526585 - SystemError concat long strings (2.4) http://python.org/sf/1523610 - PyArg_ParseTupleAndKeywords potential core dump (2.4) http://python.org/sf/1521947 - mystrtol.c fails with gcc 4.1 (2.4?) test_compile.test_unary_minus http://python.org/sf/1519025 - socket timeout crash when receive signal (2.4) http://python.org/sf/1517042 - Fix crashers/gc_inspection.py (2.4) http://python.org/sf/1515471 - stringobject (char buffers) http://python.org/sf/1513611 - XML: xml.sax.expatreader missing http://python.org/sf/1511497 - XML: xml.sax.ParseException issue http://python.org/sf/1475523 - gettext.py bug http://python.org/sf/1467929 - %-formatting and dicts (2.4) http://python.org/sf/1333982 - AST http://python.org/sf/1191458 - AST (test_trace issue mentioned below) It would be great to fix *all* of these. In this list, at least 3 (4?) can cause segfaults, and #1521947 can cause incorrect results. n From nnorwitz at gmail.com Tue Jul 25 07:50:46 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 24 Jul 2006 22:50:46 -0700 Subject: [Python-Dev] remaining issues from Klocwork static analysis Message-ID: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> I've fixed most of the problems (or determined they weren't problems) from all the warnings issued by Klocwork's static analysis tool. The following are outstanding issues. This first group looks like real problems to me: # 74 Object/funcobject.c:143 Suspicious deref of ptr before NULL check #169 Modules/threadmodule.c:497 Memory Leak # 28 Modules/_sre.c:987 Array Index Out of Bounds Buffer overflow, array index of 'mark' may be outside the bounds. Array 'mark' of size 200 declared at sre.h:77 may use index values 0..536870911. Also there are 3 similar errors on lines 1006, 1225, 1237. (Try limiting mark on line 589?) #174 Modules/unicodedata.c:432 Array Index Out of Bounds Buffer overflow, array index of 'decomp_prefix' may be outside the bounds. Array 'decomp_prefix' of size 18 declared at unicodedata_db.h:529 may use index values 18..255. Also there is one similar error on line 433. # 36 Modules/cPickle.c:3404 Memory Leak Memory leak. Dynamic memory stored in 's' allocated through function 'pystrndup' at line 3384 is lost at line 3404. s should not be freed on line 3407, but earlier. PDATA_PUSH can return on error and s will not be freed. # 61 Modules/_sqlite/cursor.c:599 Null pointer may be dereferenced Null pointer 'self->statement' that comes from line 674 may be dereferenced by passing argument 1 to function 'statement_mark_dirty' at line 599. Most of these seem suspect. I'm not so sure about them, but I haven't looked into some at all. Let me know if you want the details for any of these or if you can provide an analysis to demonstrate they are incorrect. Null pointer may be dereferenced Python/ast.c:641 Null pointer may be dereferenced Python/ast.c:656 Ptr will be derefed after it was positively checked for NULL Python/compile.c:3020 Null pointer may be passed to function that may dereference it Python/compile.c:4459 Array Index Out of Bounds Modules/_sre.c:987 Array Index Out of Bounds Object/longobject.c:1787 Array Index Out of Bounds Object/longobject.c:2475 Array Index Out of Bounds Python/sysmodule.c:1016 Array Index Out of Bounds Python/getpath.c:285 Buffer Overflow - Non-null Terminated String Python/getpath.c:432 Unvalidated User Input Buffer Overflow-Non-Null Terminated String Python/getpath.c:431 Unvalidated User Input Buffer Overflow-Non-Null Terminated String Python/getpath.c:496 Unvalidated User Input Buffer Overflow-Non-Null Terminated String Python/getpath.c:497 Let me know if you want more info about any particular report. It would be great to have some help and fix these. n From martin at v.loewis.de Tue Jul 25 07:54:24 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 25 Jul 2006 07:54:24 +0200 Subject: [Python-Dev] outstanding bugs to fix for 2.5 In-Reply-To: <ee2a432c0607242120o5365d6b4w2a8b200253686ea6@mail.gmail.com> References: <ee2a432c0607242120o5365d6b4w2a8b200253686ea6@mail.gmail.com> Message-ID: <44C5B210.6020902@v.loewis.de> Neal Norwitz wrote: > http://python.org/sf/1513611 - XML: xml.sax.expatreader missing > > It would be great to fix *all* of these. In this list, at least 3 > (4?) can cause segfaults, and #1521947 can cause incorrect results. IMO, 1513611 should block the release, since it's a regression from 2.4 that may break many SAX applications, and perhaps also DOM applications. Regards, Martin From martin at v.loewis.de Tue Jul 25 09:40:44 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 25 Jul 2006 09:40:44 +0200 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> Message-ID: <44C5CAFC.80004@v.loewis.de> Neal Norwitz wrote: > # 74 Object/funcobject.c:143 Suspicious deref of ptr before NULL check Not quite sure what it is complaining about, but else if (PyTuple_Check(closure)) { Py_XINCREF(closure); } looks indeed suspicious: Why do we check for NULL (XINCREF) when we know closure can't be NULL (Tuple_Check). Drop the X, and see if the warning goes away > #169 Modules/threadmodule.c:497 Memory Leak Does it say what memory is leaking? Perhaps it complains about boot not being released if ident is not -1, however, in that case, t_bootstrap will release the memory. > # 28 Modules/_sre.c:987 Array Index Out of Bounds > > Buffer overflow, array index of 'mark' may be outside the > bounds. Array 'mark' of size 200 declared at sre.h:77 may use > index values 0..536870911. Also there are 3 similar errors on > lines 1006, 1225, 1237. (Try limiting mark on line 589?) ISTM that SRE has a limit of 100 MARK opcodes, meaning a maximum of 100 groups per expression (so you need 200 mark pointers). This can't overrun as sre_compile refuses to compile expressions with more groups. Of course, a malicious application could craft the opcodes itself (bypassing sre_compile), in which case you could get a buffer overrun. The right solution is to have a dynamic marks array. > #174 Modules/unicodedata.c:432 Array Index Out of Bounds > > Buffer overflow, array index of 'decomp_prefix' may be outside the > bounds. Array 'decomp_prefix' of size 18 declared at > unicodedata_db.h:529 may use index values 18..255. Also there is one > similar error on line 433. This limit is enforced by Tools/unicode/makeunicodedata.py. There are only 18 decomposition prefixes at the moment, yet we use 8 bits for the decomposition prefix (makeunicodedata checks that prefix < 256) Looking at the code, I now wonder why decomp_data can't be "unsigned short", instead of "unsigned int" (the upper byte is the decomposition length, and it can't be larger than 256, either). > # 36 Modules/cPickle.c:3404 Memory Leak > > Memory leak. Dynamic memory stored in 's' allocated through > function 'pystrndup' at line 3384 is lost at line 3404. > > s should not be freed on line 3407, but earlier. > PDATA_PUSH can return on error and s will not be freed. Correct. We should not use macros with embedded return statements. > # 61 Modules/_sqlite/cursor.c:599 Null pointer may be dereferenced > > Null pointer 'self->statement' that comes from line 674 may be > dereferenced by passing argument 1 to function > 'statement_mark_dirty' at line 599. Looks like a problem. Maybe a break is missing after line 674? Regards, Martin From nmm1 at cus.cam.ac.uk Tue Jul 25 11:06:08 2006 From: nmm1 at cus.cam.ac.uk (Nick Maclaren) Date: Tue, 25 Jul 2006 10:06:08 +0100 Subject: [Python-Dev] Strategy for converting the decimal module to C Message-ID: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk> Greg Ewing <greg.ewing at canterbury.ac.nz> wrote: > > But we weren't talking about asynchronous exceptions, > we were talking about floating point exceptions. Unless > your TLB miss handler uses floating point arithmethic, > there's no way it can get interrupted by one. (And if > it does use floating point arithmetic in a way that > can cause an exception, you'd better write it to deal > with that!) I am really not getting my message across, am I? Yes, that is true - as far as it goes. The trouble is that designing systems based on assuming that IS true as far as it goes means that they don't work when it goes further. And it does. Here are a FEW of the many examples of where the simplistic model is likely to fail in an x86 context: The compiled code has made a data structure temporarily inconsistent because the operation is safe (say, list insertion), and then gets an asynchronous interrupt (e.g. SIGINT). The SIGINT handler does some operation (e.g. I/O) that implicitly uses floating-point, which then interrupts. The x86 architecture is extended to include out-of-order floating-point as it had in the past, many systems have today, and is very likely to happen in the future. It is one of the standard ways to get better performance, after all, and is on the increase. The x86 architecture is extended to support micro-threading. I have not been told by Intel or AMD that either have such plans, but I have very good reason to believe that both have such projects. IBM and Sun certainly do, though I don't know if IBM's is/are relevant. Regards, Nick Maclaren, University of Cambridge Computing Service, New Museums Site, Pembroke Street, Cambridge CB2 3QH, England. Email: nmm1 at cam.ac.uk Tel.: +44 1223 334761 Fax: +44 1223 334679 From g.brandl at gmx.net Tue Jul 25 11:57:23 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 25 Jul 2006 11:57:23 +0200 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <44C5CAFC.80004@v.loewis.de> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> Message-ID: <ea4pro$ere$1@sea.gmane.org> Martin v. L?wis wrote: > Neal Norwitz wrote: >> # 74 Object/funcobject.c:143 Suspicious deref of ptr before NULL check > > Not quite sure what it is complaining about, but > > else if (PyTuple_Check(closure)) { > Py_XINCREF(closure); > } > > looks indeed suspicious: Why do we check for NULL (XINCREF) when > we know closure can't be NULL (Tuple_Check). Drop the X, and see > if the warning goes away In comparison, the PyFunction_SetDefaults function does check for NULL, and raises an error in this case. However, since it is a C API function only, passing NULL is an error anyway. Georg From martin at v.loewis.de Tue Jul 25 21:44:12 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 25 Jul 2006 21:44:12 +0200 Subject: [Python-Dev] More tracker demos online Message-ID: <44C6748C.4060701@v.loewis.de> Currently, we have two running tracker demos online: Roundup: http://efod.se/python-tracker/ Jira: http://jira.python.atlassian.com/secure/Dashboard.jspa These installation are in various forms of demo mode and "pre-release" (meaning that the configuration is still not complete). They both use the sample data that Fredrik Lundh produced at some point, so don't be surprised that they are behind SF wrt. content. While these might not be in the final form of operation, I think users should already try to use them, to find out which one they like best. Discussions/Comments can be sent to infrastructure at python.org, however, for reports/reviews, please use the Wiki at http://wiki.python.org/moin/CallForTrackers You'll notice that it also lists Trac and Malone, however, it seems that there is no progress on importing SF data into these. Regards, Martin From kiko at async.com.br Tue Jul 25 22:38:36 2006 From: kiko at async.com.br (Christian Robottom Reis) Date: Tue, 25 Jul 2006 17:38:36 -0300 Subject: [Python-Dev] More tracker demos online In-Reply-To: <44C6748C.4060701@v.loewis.de> References: <44C6748C.4060701@v.loewis.de> Message-ID: <20060725203836.GA1542@anthem.async.com.br> On Tue, Jul 25, 2006 at 09:44:12PM +0200, "Martin v. L?wis" wrote: > You'll notice that it also lists Trac and Malone, however, > it seems that there is no progress on importing SF data > into these. Actually, James Henstridge has been working on an import into Launchpad (Malone is the codename for the bugtracker component of it) over last week. We have a demo site up at: https://demo.launchpad.net/products/python/+bugs Note that we're still ironing out some of the kinks with the import and the code running there, so there will be changes before the official announcement. -- Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3376 0125 From grig at gheorghiu.net Fri Jul 21 02:20:58 2006 From: grig at gheorghiu.net (Grig Gheorghiu) Date: Thu, 20 Jul 2006 17:20:58 -0700 (PDT) Subject: [Python-Dev] Community buildbots -- reprise Message-ID: <20060721002058.66596.qmail@web54515.mail.yahoo.com> Hi, This message is in response to Glyph's plea (<http://mail.python.org/pipermail/python-dev/2006-July/067366.html>). Here's what Glyph said: "I would like to propose, although I certainly don't have time to implement, a program by which Python-using projects could contribute buildslaves which would run their projects' tests with the latest Python trunk. This would provide two useful incentives: Python code would gain a reputation as generally well-tested (since there is a direct incentive to write tests for your project: get notified when core python changes might break it), and the core developers would have instant feedback when a "small" change breaks more code than it was expected to." I'm volunteering to organize this effort, is there is enough interest on this list. In fact, I've done some prep work already: * got a domain name: pybots.org * got a $47/month Ubuntu-based VPS from JohnCompanies.com (root access and everything); it's available at master.pybots.org, and it's ready to be configured as a buildmaster for the pybots * got a mailing list: pybots at lists2.idyll.org I can start configuring the Ubuntu machine as a buildmaster, and I can also add a buildslave on the same machine that will check out the latest Python trunk code, build it, then run the automated tests for a sample project -- let's say for Twisted, since Glyph was the one requesting this. This will also serve as a sample buildslave for other people who will be interested in running buildslaves for their own projects. Apart from the goals stated by Glyph, I see this as a very valuable effort in convincing people of the value of automated tests, Python-related or not. A secondary effect I'd like to see would be for these suites of tests to be invoked in a standard fashion -- maybe 'python setup.py test'. If PSF can contribute some $$$ towards the hosting of the master server, that would be appreciated, but not required. All that's required is enough interest from the community. Please let me know if you're interested. Grig ---- http://agiletesting.blogspot.com From ndbecker2 at gmail.com Fri Jul 21 13:59:56 2006 From: ndbecker2 at gmail.com (Neal Becker) Date: Fri, 21 Jul 2006 07:59:56 -0400 Subject: [Python-Dev] Document performance requirements? In-Reply-To: <44C0BF4C.1010408@gmail.com> References: <e9qbtf$kk6$1@sea.gmane.org> <44C0BF4C.1010408@gmail.com> Message-ID: <200607210759.56595.ndbecker2@gmail.com> On Friday 21 July 2006 7:49 am, Nick Coghlan wrote: > Neal Becker wrote: > > For a recent project I needed to select a container. There are plenty of > > python data structures to choose from. It seems that information on > > performance is missing (or not easy to find). > > > > I think Python should include performance in the documentation of common > > data structures to help users select the appropriate types. Something in > > the style of c++ STL. > > Do you mean absolute performance, or do you mean algorithmic order > guarantees? I thought the latter were already documented. . . > The latter. Where are they documented? From eswierk at arastra.com Tue Jul 25 00:19:46 2006 From: eswierk at arastra.com (Ed Swierk) Date: Mon, 24 Jul 2006 15:19:46 -0700 Subject: [Python-Dev] setup.py and cross-compiling In-Reply-To: <44C53AE6.7040105@v.loewis.de> References: <loom.20060724T190947-141@post.gmane.org> <44C53AE6.7040105@v.loewis.de> Message-ID: <c1bf1cf0607241519j157a7050l29998cecc4953485@mail.gmail.com> On 7/24/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: > The main problem is that it is the host python that runs setup.py, > not the target python. Various parts of distutils assume that the > information the interpreter provides is correct, yet it is not > in a cross-compilation case. Well, it seems buildroot solves this main problem by building another version of python and pygen that run on the build machine, and hacks the Makefile to run setup.py with these instead of whatever happens to be sitting in /usr/bin. Thus the modules that do get built seem to work just fine, but setup.py itself ignores all this careful hackery when determining what modules to build and configuring distutils to search for system libraries in various dynamically-discovered paths. --Ed From nyamatongwe at gmail.com Wed Jul 26 02:10:07 2006 From: nyamatongwe at gmail.com (Neil Hodgson) Date: Wed, 26 Jul 2006 10:10:07 +1000 Subject: [Python-Dev] More tracker demos online In-Reply-To: <44C6748C.4060701@v.loewis.de> References: <44C6748C.4060701@v.loewis.de> Message-ID: <50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com> Martin v. L?wis: > Currently, we have two running tracker demos online: After playing with them for 30 minutes, Jira seems to have too busy an interface and finicky behaviour: not liking the back button sometimes (similar to SF) and clicking on diffs wants to download them rather than view them. Its disappointing that Jira and Launchpad use different bug IDs as continuity should be maintained with the SF bug IDs which will be referred to in other areas such as commit messages. They do include the SF bug ID (as a field in Jira and a nickname in Launchpad) but this makes it harder to navigate between related bugs. I mostly looked at "os.startfile() still doesn't work with Unicode filenames" and I would have tagged the patch on SF with a "looks OK to me" if SF was working. The text in Launchpad was a bit sparsely formatted for me so would like to see if indvidual users can choose a different style. The others are OK although Roundup is clearer. Neil From brett at python.org Wed Jul 26 02:32:22 2006 From: brett at python.org (Brett Cannon) Date: Tue, 25 Jul 2006 17:32:22 -0700 Subject: [Python-Dev] More tracker demos online In-Reply-To: <50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com> References: <44C6748C.4060701@v.loewis.de> <50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com> Message-ID: <bbaeab100607251732u5771b6fet4b6e5a949297f8f9@mail.gmail.com> On 7/25/06, Neil Hodgson <nyamatongwe at gmail.com> wrote: > > Martin v. L?wis: > > > Currently, we have two running tracker demos online: > > After playing with them for 30 minutes, Jira seems to have too busy > an interface and finicky behaviour: not liking the back button > sometimes (similar to SF) and clicking on diffs wants to download them > rather than view them. Its disappointing that Jira and Launchpad use > different bug IDs as continuity should be maintained with the SF bug > IDs which will be referred to in other areas such as commit messages. Stuff like continuity in bug numbers and such can be fixed in the official tracker upon launch. More important is interface and general usage. And as Martin said, if you wish to discuss, please do so on the infrastructue mailing list. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060725/2af3211d/attachment.htm From nnorwitz at gmail.com Wed Jul 26 05:46:13 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 25 Jul 2006 20:46:13 -0700 Subject: [Python-Dev] Community buildbots -- reprise In-Reply-To: <20060721002058.66596.qmail@web54515.mail.yahoo.com> References: <20060721002058.66596.qmail@web54515.mail.yahoo.com> Message-ID: <ee2a432c0607252046p7a6c91a3gf1451fd1f2f3ed55@mail.gmail.com> If you want I can send you the build master cfg I setup on python.org and some simple instructions for how to connect to it. I don't have time to focus on this at the moment and probably won't until 2.5 is out. n -- On 7/20/06, Grig Gheorghiu <grig at gheorghiu.net> wrote: > Hi, > > This message is in response to Glyph's plea > (<http://mail.python.org/pipermail/python-dev/2006-July/067366.html>). > > Here's what Glyph said: > > "I would like to propose, although I certainly don't have time to > implement, a program by which Python-using projects could contribute > buildslaves which would run their projects' tests with the latest > Python trunk. This would provide two useful incentives: Python code > would gain a reputation as generally well-tested (since there is a > direct incentive to write tests for your project: get notified when > core python changes might break it), and the core developers would have > instant feedback when a "small" change breaks more code than it was > expected to." > > > I'm volunteering to organize this effort, is there is enough interest > on this list. In fact, I've done some prep work already: > > * got a domain name: pybots.org > * got a $47/month Ubuntu-based VPS from JohnCompanies.com (root access > and everything); it's available at master.pybots.org, and it's ready to > be configured as a buildmaster for the pybots > * got a mailing list: pybots at lists2.idyll.org > > I can start configuring the Ubuntu machine as a buildmaster, and I can > also add a buildslave on the same machine that will check out the > latest Python trunk code, build it, then run the automated tests for a > sample project -- let's say for Twisted, since Glyph was the one > requesting this. This will also serve as a sample buildslave for other > people who will be interested in running buildslaves for their own > projects. > > Apart from the goals stated by Glyph, I see this as a very valuable > effort in convincing people of the value of automated tests, > Python-related or not. A secondary effect I'd like to see would be for > these suites of tests to be invoked in a standard fashion -- maybe > 'python setup.py test'. > > If PSF can contribute some $$$ towards the hosting of the master > server, that would be appreciated, but not required. All that's > required is enough interest from the community. > > Please let me know if you're interested. > > Grig > > ---- > http://agiletesting.blogspot.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From nnorwitz at gmail.com Wed Jul 26 06:47:08 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 25 Jul 2006 21:47:08 -0700 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <44C5CAFC.80004@v.loewis.de> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> Message-ID: <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com> On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Neal Norwitz wrote: > > # 74 Object/funcobject.c:143 Suspicious deref of ptr before NULL check > > Not quite sure what it is complaining about, but > > else if (PyTuple_Check(closure)) { > Py_XINCREF(closure); > } > > looks indeed suspicious: Why do we check for NULL (XINCREF) when > we know closure can't be NULL (Tuple_Check). Drop the X, and see > if the warning goes away Yes, I definitely think dropping the X would make the warning go away. Do we want to check for a NULL pointer and raise an exception? The docs don't address the issue, so I think if we added a check, ie: if (closure && PyTuple_Check(closure)) and got rid of the X that would be fine as well. > > #169 Modules/threadmodule.c:497 Memory Leak > > Does it say what memory is leaking? Perhaps it complains about > boot not being released if ident is not -1, however, in that case, > t_bootstrap will release the memory. I believe you are right, I never traced through t_bootstrap. I think this is a false positive. There is some memory being leaked on thread creation as reported by valgrind IIRC. This doesn't seem to be it though. > > #174 Modules/unicodedata.c:432 Array Index Out of Bounds > > > > Buffer overflow, array index of 'decomp_prefix' may be outside the > > bounds. Array 'decomp_prefix' of size 18 declared at > > unicodedata_db.h:529 may use index values 18..255. Also there is one > > similar error on line 433. > > This limit is enforced by Tools/unicode/makeunicodedata.py. There are > only 18 decomposition prefixes at the moment, yet we use 8 bits for > the decomposition prefix (makeunicodedata checks that prefix < 256) Just to make sure I understand. The code in question is accessing decomp_prefix like this: decomp_prefix[decomp_data[index] & 255] So decomp_prefix will be accessed with the result of: decomp_data[index] & 255 The first line of data is (fro unicodedata_db.h) is: static unsigned int decomp_data[] = { 0, 257, 32, 514, 32, 776, 259, 97, 514, 32, 772, 259, 50, 259, 51, 514, If index == 2 (or 3-5, 7-10, etc), we have: decomp_prefix[decomp_data[2] & 255] decomp_prefix[32 & 255] decomp_prefix[32] which is larger than the max size of decomp_prefix (18). But from what I think you stated above, index can't equal those values and the code that prevents it is calculated a few lines above: index = decomp_index1[(code>>DECOMP_SHIFT)]; index = decomp_index2[(index<<DECOMP_SHIFT)+ (code&((1<<DECOMP_SHIFT)-1))]; Is that correct? If so, would it be correct to add: unsigned short prefix_index = decomp_data[index] & 255; assert(prefix_index < (sizeof(decomp_prefix)/sizeof(*decomp_prefix))); n From nnorwitz at gmail.com Wed Jul 26 06:49:56 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 25 Jul 2006 21:49:56 -0700 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <ea4pro$ere$1@sea.gmane.org> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> <ea4pro$ere$1@sea.gmane.org> Message-ID: <ee2a432c0607252149t6136b679w1630fc0fb3dcba63@mail.gmail.com> On 7/25/06, Georg Brandl <g.brandl at gmx.net> wrote: > Martin v. L?wis wrote: > > Neal Norwitz wrote: > >> # 74 Object/funcobject.c:143 Suspicious deref of ptr before NULL check > > > > Not quite sure what it is complaining about, but > > > > else if (PyTuple_Check(closure)) { > > Py_XINCREF(closure); > > } > > > > looks indeed suspicious: Why do we check for NULL (XINCREF) when > > we know closure can't be NULL (Tuple_Check). Drop the X, and see > > if the warning goes away > > In comparison, the PyFunction_SetDefaults function does check for > NULL, and raises an error in this case. However, since it is a C API function > only, passing NULL is an error anyway. Heh, that was me that added it 10 days ago. :-) Might as well do the same here. n From grig.gheorghiu at gmail.com Wed Jul 26 06:57:03 2006 From: grig.gheorghiu at gmail.com (Grig Gheorghiu) Date: Tue, 25 Jul 2006 21:57:03 -0700 Subject: [Python-Dev] Community buildbots -- reprise In-Reply-To: <ee2a432c0607252046p7a6c91a3gf1451fd1f2f3ed55@mail.gmail.com> References: <20060721002058.66596.qmail@web54515.mail.yahoo.com> <ee2a432c0607252046p7a6c91a3gf1451fd1f2f3ed55@mail.gmail.com> Message-ID: <3f09d5a00607252157q117c2e7dxd6c6f75ba7e4fd57@mail.gmail.com> On 7/25/06, Neal Norwitz <nnorwitz at gmail.com> wrote: > > If you want I can send you the build master cfg I setup on python.org > and some simple instructions for how to connect to it. I don't have > time to focus on this at the moment and probably won't until 2.5 is > out. > > n > -- Sure. I'm still a bit unclear on whether you want me to coordinate this by adding buildslaves to the build master cfg, adding build steps etc. I'll gladly do it if you need help. I'll need access to the server and proper permissions of course. Grig -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060725/dac00dfd/attachment.html From martin at v.loewis.de Wed Jul 26 07:05:44 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 26 Jul 2006 07:05:44 +0200 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com> Message-ID: <44C6F828.7020005@v.loewis.de> Neal Norwitz wrote: >> Not quite sure what it is complaining about, but >> >> else if (PyTuple_Check(closure)) { >> Py_XINCREF(closure); >> } >> >> looks indeed suspicious: Why do we check for NULL (XINCREF) when >> we know closure can't be NULL (Tuple_Check). Drop the X, and see >> if the warning goes away > > Yes, I definitely think dropping the X would make the warning go away. > Do we want to check for a NULL pointer and raise an exception? The > docs don't address the issue, so I think if we added a check, ie: if > (closure && PyTuple_Check(closure)) and got rid of the X that would be > fine as well. The docs do address the issue: \var{closure} must be \var{Py_None} or a tuple of cell objects. It doesn't allow for NULL, and None indicates that the closure should become NULL. The only caller of it in the core will never pass NULL. If you want to check that this is not NULL on the grounds that somebody may call it incorrectly, then you should also check that op is not NULL, because somebody may call it incorrectly. > The first line of data is (fro unicodedata_db.h) is: > > static unsigned int decomp_data[] = { > 0, 257, 32, 514, 32, 776, 259, 97, 514, 32, 772, 259, 50, 259, 51, 514, Read this as 0: sentinel 257 = 256 | 1: length 1, prefix 1 32: U+0020 514 = 512 | 2: length 2, prefix 2 32: U+0020 776: U+308 ... > Is that correct? If so, would it be correct to add: > > unsigned short prefix_index = decomp_data[index] & 255; > assert(prefix_index < (sizeof(decomp_prefix)/sizeof(*decomp_prefix))); Yes, that would be correct. Regards, Martin From martin at v.loewis.de Wed Jul 26 07:09:47 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 26 Jul 2006 07:09:47 +0200 Subject: [Python-Dev] More tracker demos online In-Reply-To: <50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com> References: <44C6748C.4060701@v.loewis.de> <50862ebd0607251710p454d3cc1s84622aee45774c77@mail.gmail.com> Message-ID: <44C6F91B.1010105@v.loewis.de> Neil Hodgson wrote: > Its disappointing that Jira and Launchpad use > different bug IDs as continuity should be maintained with the SF bug > IDs which will be referred to in other areas such as commit messages. My plan is to keep the SF redirector alive, so python.org/sf/<bugid> should continue to direct you to the right item in the new tracker. This can only be done when we actually make the switch, since currently the redirector still needs to direct to SF. Regards, Martin From nnorwitz at gmail.com Wed Jul 26 07:12:42 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 25 Jul 2006 22:12:42 -0700 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <44C6F828.7020005@v.loewis.de> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com> <44C6F828.7020005@v.loewis.de> Message-ID: <ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com> On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: > > > > Yes, I definitely think dropping the X would make the warning go away. > > Do we want to check for a NULL pointer and raise an exception? The > > docs don't address the issue, so I think if we added a check, ie: if > > (closure && PyTuple_Check(closure)) and got rid of the X that would be > > fine as well. > > The docs do address the issue: > > \var{closure} must be \var{Py_None} or a tuple of cell objects. > > It doesn't allow for NULL, and None indicates that the closure > should become NULL. The only caller of it in the core will never > pass NULL. > > If you want to check that this is not NULL on the grounds that > somebody may call it incorrectly, then you should also check that > op is not NULL, because somebody may call it incorrectly. We never really did address this issue did? A while back we talked about whether to assert vs check and do PyErr_BadInternalCall(). I don't remember a clear resolution (though my memory). I vaguely remember a preference towards asserting, but I don't know if that was in all cases or maybe it was just my preference. :-) I'm happy to assert here too. But it's really a broader question. I guess I'm even happy to just remove the X. It would be nice to handle this consistently going forward. n From martin at v.loewis.de Wed Jul 26 07:21:16 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 26 Jul 2006 07:21:16 +0200 Subject: [Python-Dev] More tracker demos online In-Reply-To: <ea6msv$hgf$1@sea.gmane.org> References: <44C6748C.4060701@v.loewis.de> <ea6msv$hgf$1@sea.gmane.org> Message-ID: <44C6FBCC.5080801@v.loewis.de> Terry Reedy wrote: > ""Martin v. L?wis"" <martin at v.loewis.de> wrote in message > news:44C6748C.4060701 at v.loewis.de... >> Currently, we have two running tracker demos online: >> >> Roundup: >> http://efod.se/python-tracker/ >> >> Jira: >> http://jira.python.atlassian.com/secure/Dashboard.jspa > > What user name and passwords will they accept, if any? The roundup installation accepts SF user names, no password; you can then set the password if you want to. The Jira installation accepts SF user names; you have make it send you a password reminder. The registered email address is the SF one. If you have problems getting in, please let me know. I (and the other infrastruture people) have admin privs on these installations, so I can learn how to administrate them if something goes wrong :-) Regards, Martin From martin at v.loewis.de Wed Jul 26 07:32:01 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 26 Jul 2006 07:32:01 +0200 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com> <44C6F828.7020005@v.loewis.de> <ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com> Message-ID: <44C6FE51.7060509@v.loewis.de> Neal Norwitz wrote: > We never really did address this issue did? A while back we talked > about whether to assert vs check and do PyErr_BadInternalCall(). I > don't remember a clear resolution (though my memory). I vaguely > remember a preference towards asserting, but I don't know if that was > in all cases or maybe it was just my preference. :-) > > I'm happy to assert here too. But it's really a broader question. I > guess I'm even happy to just remove the X. It would be nice to handle > this consistently going forward. I would just remove the X. If we want to handle it consistently, we would have to check all pointer parameters in all functions; this would be a huge task (and for little value, IMO). In any case, "closure && PyTuple_Check(closure)" would be wrong, since it then goes into PyErr_Format(PyExc_SystemError, "expected tuple for closure, got '%.100s'", closure->ob_type->tp_name); which crashes just the same. Regards, Martin From nnorwitz at gmail.com Wed Jul 26 07:41:14 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 25 Jul 2006 22:41:14 -0700 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <44C6FE51.7060509@v.loewis.de> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com> <44C6F828.7020005@v.loewis.de> <ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com> <44C6FE51.7060509@v.loewis.de> Message-ID: <ee2a432c0607252241i681435e2n7c846b2d46355936@mail.gmail.com> On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Neal Norwitz wrote: > > We never really did address this issue did? A while back we talked > > about whether to assert vs check and do PyErr_BadInternalCall(). I > > don't remember a clear resolution (though my memory). I vaguely > > remember a preference towards asserting, but I don't know if that was > > in all cases or maybe it was just my preference. :-) > > > > I'm happy to assert here too. But it's really a broader question. I > > guess I'm even happy to just remove the X. It would be nice to handle > > this consistently going forward. > > I would just remove the X. I'll do that here since it's the easiest. > If we want to handle it consistently, we would have to check all pointer > parameters in all functions; this would be a huge task (and for little > value, IMO). I'm not suggesting changing existing code, unless we find issues. I agree that it would be a huge task and of little value. I was thinking about for future code. I guess we aren't writing a lot of new C APIs in 2.x, so it really doesn't much there matter. Though for 3k, it would be nice to make it consistent as new APIs are written or old APIs are cleaned up. n From greg.ewing at canterbury.ac.nz Wed Jul 26 08:13:16 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 26 Jul 2006 18:13:16 +1200 Subject: [Python-Dev] new security doc using object-capabilities In-Reply-To: <44C567A0.2080906@blueyonder.co.uk> References: <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <bbaeab100607191535p543cb0ddj66410cf985dd9b77@mail.gmail.com> <20060722114615.GB13782@code0.codespeak.net> <44C2CE78.9020802@blueyonder.co.uk> <20060723070811.GD13782@code0.codespeak.net> <5.1.1.6.0.20060723164246.04144298@sparrow.telecommunity.com> <44C3F31D.3070201@blueyonder.co.uk> <5.1.1.6.0.20060723194448.0205fa50@sparrow.telecommunity.com> <5.1.1.6.0.20060724015521.0261d160@sparrow.telecommunity.com> <5.1.1.6.0.20060724211442.0262c008@sparrow.telecommunity.com> <44C567A0.2080906@blueyonder.co.uk> Message-ID: <44C707FC.6@canterbury.ac.nz> > Phillip J. Eby wrote: > > > And what about code that needs to pass on a subset of a capability? With one object == one capability, there is no such thing as a subset of a capability -- the capabilities are the atomic units at which you control access. So you need to make them fine-grained enough to begin with. -- Greg From g.brandl at gmx.net Wed Jul 26 09:16:25 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 26 Jul 2006 09:16:25 +0200 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com> Message-ID: <ea74pt$ign$1@sea.gmane.org> Neal Norwitz wrote: > On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> Neal Norwitz wrote: >> > # 74 Object/funcobject.c:143 Suspicious deref of ptr before NULL check >> >> Not quite sure what it is complaining about, but >> >> else if (PyTuple_Check(closure)) { >> Py_XINCREF(closure); >> } >> >> looks indeed suspicious: Why do we check for NULL (XINCREF) when >> we know closure can't be NULL (Tuple_Check). Drop the X, and see >> if the warning goes away > > Yes, I definitely think dropping the X would make the warning go away. > Do we want to check for a NULL pointer and raise an exception? The > docs don't address the issue, so I think if we added a check, ie: if > (closure && PyTuple_Check(closure)) and got rid of the X that would be > fine as well. You'll have to do something about the error message, then, since it uses closure->ob_type. Georg From greg.ewing at canterbury.ac.nz Wed Jul 26 09:39:40 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 26 Jul 2006 19:39:40 +1200 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk> References: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk> Message-ID: <44C71C3C.4030604@canterbury.ac.nz> Nick Maclaren wrote: > The compiled code has made a data structure temporarily inconsistent > because the operation is safe (say, list insertion), and then gets an > asynchronous interrupt (e.g. SIGINT). The SIGINT handler does some > operation (e.g. I/O) that implicitly uses floating-point, which then > interrupts. Well, of course anything can be made to happen asynchronously by calling it from something asynchronous, such as a SIGINT handler. That doesn't change the fact that the floating point operation itself is deterministic, including whether it causes an exception. Well-written programs don't do any more in a signal handler than is absolutely necessary, for reasons which apply equally well whether floating point is involved or not. I'd say the mistake was made right at the beginning by assuming that the data structure in question was safe while allowing a SIGINT to occur to a handler that's not careful enough about what it does. BTW, it seems to me you could get exactly the same problem if FP exceptions were handled entirely in user mode, as you suggest. Not that I don't agree that would be a good idea -- I do -- but it wouldn't prevent this particular kind of mistake. And all of this is getting rather far away from where we started, which was simply instrumenting a piece of code to count floating point exceptions. Such a program isn't going to be doing I/O in SIGINT handlers or installing FP exception handlers that mess with unrelated critical data structures. -- Greg From gh at ghaering.de Wed Jul 26 10:17:04 2006 From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=) Date: Wed, 26 Jul 2006 10:17:04 +0200 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <44C5CAFC.80004@v.loewis.de> References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> Message-ID: <44C72500.9080301@ghaering.de> Martin v. L?wis wrote: > Neal Norwitz wrote: >> # 61 Modules/_sqlite/cursor.c:599 Null pointer may be dereferenced >> >> Null pointer 'self->statement' that comes from line 674 may be >> dereferenced by passing argument 1 to function >> 'statement_mark_dirty' at line 599. > > Looks like a problem. Maybe a break is missing after line 674? The code is a bit complicated here, and admittedly not the nicest one, but I verified it and there is no problem here. _query_execute() is designed to handle both executemany() and execute(). multiple is a local variable that is 1 if there is a set of tuples of SQL parameters (executemany) instead of a single tuple of SQL parameters (execute), in which case it's 0. Before the while loop, the code makes sure that parameters_iter is an iterator that returns SQL parameter tuples. So if there is only a single parameter tuple, a temporary list is created an parameters_iter is an iterator over it. So, if !multiple (referenced code in line 674), the while-loop will only be executed once, and in the second loop, the while-loop will exit because then the following code will break: parameters = PyIter_Next(parameters_iter); if (!parameters) { break; } Code in line 599 is thus not executed with self->statement = 0. -- Gerhard From mwh at python.net Wed Jul 26 10:53:53 2006 From: mwh at python.net (Michael Hudson) Date: Wed, 26 Jul 2006 09:53:53 +0100 Subject: [Python-Dev] remaining issues from Klocwork static analysis In-Reply-To: <ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com> (Neal Norwitz's message of "Tue, 25 Jul 2006 22:12:42 -0700") References: <ee2a432c0607242250v5a7ac021p464b68c5c47ec538@mail.gmail.com> <44C5CAFC.80004@v.loewis.de> <ee2a432c0607252147u6e3e4b37i1a77eb8114ac7349@mail.gmail.com> <44C6F828.7020005@v.loewis.de> <ee2a432c0607252212k5276852dlacb0d73dbd984434@mail.gmail.com> Message-ID: <2mwta0ycf2.fsf@starship.python.net> "Neal Norwitz" <nnorwitz at gmail.com> writes: > On 7/25/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: >> > >> > Yes, I definitely think dropping the X would make the warning go away. >> > Do we want to check for a NULL pointer and raise an exception? The >> > docs don't address the issue, so I think if we added a check, ie: if >> > (closure && PyTuple_Check(closure)) and got rid of the X that would be >> > fine as well. >> >> The docs do address the issue: >> >> \var{closure} must be \var{Py_None} or a tuple of cell objects. >> >> It doesn't allow for NULL, and None indicates that the closure >> should become NULL. The only caller of it in the core will never >> pass NULL. >> >> If you want to check that this is not NULL on the grounds that >> somebody may call it incorrectly, then you should also check that >> op is not NULL, because somebody may call it incorrectly. > > We never really did address this issue did? A while back we talked > about whether to assert vs check and do PyErr_BadInternalCall(). I > don't remember a clear resolution (though my memory). I vaguely > remember a preference towards asserting, but I don't know if that was > in all cases or maybe it was just my preference. :-) > > I'm happy to assert here too. But it's really a broader question. I > guess I'm even happy to just remove the X. It would be nice to handle > this consistently going forward. I think I'm rather in favour of assert()ing this sort of thing. If you're programming in C, you can cause crashes any which way and removing one doesn't seem worth making correct usage pay any kind of (admittedly miniscule) performance penalty. It would be nice if API docs explicitly stated which pointer arguments could be NULL, and then it would be a programming error to pass a NULL pointer argument in any other place. I have no idea how far away from this we are already :-) Cheers, mwh -- Gullible editorial staff continues to post links to any and all articles that vaguely criticize Linux in any way. -- Reason #4 for quitting slashdot today, from http://www.cs.washington.edu/homes/klee/misc/slashdot.html From python at rcn.com Wed Jul 26 11:18:24 2006 From: python at rcn.com (Raymond Hettinger) Date: Wed, 26 Jul 2006 02:18:24 -0700 Subject: [Python-Dev] Strategy for converting the decimal module to C References: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk> <44C71C3C.4030604@canterbury.ac.nz> Message-ID: <003c01c6b094$72de56a0$0f10000a@RaymondLaptop1> Greg Ewing > And all of this is getting rather far away from where we > started, which was simply instrumenting a piece of code > to count floating point exceptions. I'm thinking of adding a note to the Py2.5 docs that the counting feature is not part of the standard and should not be expected to work on other implementations of the standard (including a planned CPython extension module). Raymond From facundobatista at gmail.com Wed Jul 26 17:52:48 2006 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 26 Jul 2006 12:52:48 -0300 Subject: [Python-Dev] Strategy for converting the decimal module to C In-Reply-To: <003c01c6b094$72de56a0$0f10000a@RaymondLaptop1> References: <E1G5Irg-0003Zi-9b@libra.cus.cam.ac.uk> <44C71C3C.4030604@canterbury.ac.nz> <003c01c6b094$72de56a0$0f10000a@RaymondLaptop1> Message-ID: <e04bdf310607260852qeee5239t97bad5a5f1424017@mail.gmail.com> 2006/7/26, Raymond Hettinger <python at rcn.com>: > Greg Ewing > > And all of this is getting rather far away from where we > > started, which was simply instrumenting a piece of code > > to count floating point exceptions. > > I'm thinking of adding a note to the Py2.5 docs that the counting feature is not > part of the standard and should not be expected to work on other implementations > of the standard (including a planned CPython extension module). +1 -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ From syfou at users.sourceforge.net Wed Jul 26 20:11:53 2006 From: syfou at users.sourceforge.net (Sylvain Fourmanoit) Date: Wed, 26 Jul 2006 14:11:53 -0400 (EDT) Subject: [Python-Dev] New miniconf module Message-ID: <Pine.LNX.4.64.0607261402410.5956@sylvain> I wrote a data persistence module called miniconf, aimed at making easy to create and safely retrieve configuration info from external, human-readable sources using Python syntax. I feel it would eventually make a nice addition to the standard library. The code was only newly refactored in this form, but it as been broadly distributed and used as a part of the adesklets project for over a year by a significant user base on multiple platforms. Here it is, as a patch against Python 2.5 SVN tree[1], or as a stand-alone module hosted on the Python Cheese Shop[2]; any feedback is welcomed. -- Sylvain <syfou at users.sourceforge.net> Hackers are just a migratory lifeform with a tropism for computers. [1]http://sourceforge.net/tracker/index.php?func=detail&aid=1527597&group_id=5470&atid=355470 [2]http://cheeseshop.python.org/pypi?:action=display&name=miniconf&version=1.0.1 From amk at amk.ca Wed Jul 26 19:58:24 2006 From: amk at amk.ca (A.M. Kuchling) Date: Wed, 26 Jul 2006 13:58:24 -0400 Subject: [Python-Dev] 2.5: uses of sys.exc_type, exc_value Message-ID: <20060726175824.GA17956@localhost.localdomain> http://www.python.org/sf/1525469 reports that SimpleXMLRPCServer.py still uses sys.exc_type and sys.exc_value when handling exceptions. These variables aren't thread-safe and sys.exc_info() is the better way. I have a patch attached to the bug that fixes the problem. Question 1: is this worth fixing for 2.5? (It's not really a bugfix, more of a style cleanup.) Question 2: I searched for uses of the old variables and found these: Lib/idlelib/WindowList.py: sys.exc_type, ":", sys.exc_value Lib/logging/__init__.py: return sys.exc_traceback.tb_frame.f_back Lib/lib-tk/Tkinter.py: exc, val, tb = sys.exc_type, sys.exc_value, sys.exc_traceback Lib/plat-mac/cfmfile.py: raise Res.Error, "no 'cfrg' resource found", sys.exc_traceback Lib/SocketServer.py: sys.exc_traceback = None # Help garbage collection Plus some references in the test suite, the demos, and faqwizard.py. SocketServer should use sys.exc_clear() instead. Tkinter.py could just call exc_info(), but I wonder if the usage of the variables is intentional here. sys.exc_info() was introduced in Python 1.5, so logging/__init__.py could be fixed without affecting 1.5.2 compatibility. Should the above uses be fixed, too? --amk From guido at python.org Wed Jul 26 20:21:50 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Jul 2006 11:21:50 -0700 Subject: [Python-Dev] 2.5: uses of sys.exc_type, exc_value In-Reply-To: <20060726175824.GA17956@localhost.localdomain> References: <20060726175824.GA17956@localhost.localdomain> Message-ID: <ca471dc20607261121i7bfe7115oa217c36170bddc92@mail.gmail.com> Clearly they should be fixed. Whether in 2.5 or 2.6 I'll leave up to Neal and Anthony. On 7/26/06, A.M. Kuchling <amk at amk.ca> wrote: > http://www.python.org/sf/1525469 reports that SimpleXMLRPCServer.py > still uses sys.exc_type and sys.exc_value when handling exceptions. > These variables aren't thread-safe and sys.exc_info() is the better > way. I have a patch attached to the bug that fixes the problem. > > Question 1: is this worth fixing for 2.5? (It's not really a bugfix, > more of a style cleanup.) > > Question 2: I searched for uses of the old variables and found these: > > Lib/idlelib/WindowList.py: sys.exc_type, ":", sys.exc_value > Lib/logging/__init__.py: return sys.exc_traceback.tb_frame.f_back > Lib/lib-tk/Tkinter.py: exc, val, tb = sys.exc_type, sys.exc_value, sys.exc_traceback > Lib/plat-mac/cfmfile.py: raise Res.Error, "no 'cfrg' resource found", sys.exc_traceback > Lib/SocketServer.py: sys.exc_traceback = None # Help garbage collection > > Plus some references in the test suite, the demos, and faqwizard.py. > > SocketServer should use sys.exc_clear() instead. Tkinter.py could > just call exc_info(), but I wonder if the usage of the variables is > intentional here. sys.exc_info() was introduced in Python 1.5, so > logging/__init__.py could be fixed without affecting 1.5.2 > compatibility. > > Should the above uses be fixed, too? > > --amk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Jul 26 20:40:27 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 26 Jul 2006 14:40:27 -0400 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix Message-ID: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> I posted last week about a need-for-speed patch that broke PEP 302 compliance, and asked if it should be fixed or reverted. I got exactly one response which said "yes, it should be fixed or reverted", which unfortunately didn't answer my question as to which one we should do. :) If we don't revert it, there are two ways to fix it. One is to just change PEP 302 so that the behavior is unbroken by definition. :) The other is to actually go ahead and fix it by adding PathImporter and NullImporter types to import.c, along with a factory function on sys.path_hooks to create them. (This would've been the PEP-compliant way to implement the need-for-speed patch.) So, "fix" by documentation, fix by fixing, or fix by reverting? Which should it be? From david.nospam.hopwood at blueyonder.co.uk Wed Jul 26 20:47:30 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Wed, 26 Jul 2006 19:47:30 +0100 Subject: [Python-Dev] New miniconf module In-Reply-To: <Pine.LNX.4.64.0607261402410.5956@sylvain> References: <Pine.LNX.4.64.0607261402410.5956@sylvain> Message-ID: <44C7B8C2.1060904@blueyonder.co.uk> Sylvain Fourmanoit wrote: > I wrote a data persistence module called miniconf, aimed at making > easy to create and safely retrieve configuration info from external, > human-readable sources using Python syntax. I feel it would eventually > make a nice addition to the standard library. >From a security point of view, this is a great improvement on the existing pickle, marshal, and shelve modules. Those modules could not be safely imported from restricted code. miniconf, OTOH, appears to have an interface compatible with capability security. (I have not checked that the compiler.ast module used in its implementation is safe.) However: +Limitations +=========== + +miniconf has a few limitations one should be aware of: [...] +- It is not preemptiple: concurrent calls to dump() or load() will + have unpredictable results and must be avoided. This limitation should be fixed before the module is added to the standard library, IMHO. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From pje at telecommunity.com Wed Jul 26 22:35:43 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 26 Jul 2006 16:35:43 -0400 Subject: [Python-Dev] New miniconf module In-Reply-To: <44C7B8C2.1060904@blueyonder.co.uk> References: <Pine.LNX.4.64.0607261402410.5956@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> Message-ID: <5.1.1.6.0.20060726162457.02d5a7c0@sparrow.telecommunity.com> At 07:47 PM 7/26/2006 +0100, David Hopwood wrote: >Sylvain Fourmanoit wrote: > > I wrote a data persistence module called miniconf, aimed at making > > easy to create and safely retrieve configuration info from external, > > human-readable sources using Python syntax. I feel it would eventually > > make a nice addition to the standard library. > > >From a security point of view, this is a great improvement on the existing >pickle, marshal, and shelve modules. Those modules could not be safely >imported from restricted code. > >miniconf, OTOH, appears to have an interface compatible with capability >security. (I have not checked that the compiler.ast module used in its >implementation is safe.) However: > >+Limitations >+=========== >+ >+miniconf has a few limitations one should be aware of: >[...] >+- It is not preemptiple: concurrent calls to dump() or load() will >+ have unpredictable results and must be avoided. > >This limitation should be fixed before the module is added to the standard >library, IMHO. It looks like it's trivial to fix; the code uses a strange and unnecessary complication of creating nested classes and nested singleton instances thereof. Getting rid of the singletons to create a new instance for each dump/load call would suffice to make the implementation re-entrant, although de-nesting the classes would also be a good idea. :) The loading code could also be made a lot faster by using a dictionary mapping AST node types to functions, instead of doing string manipulation for each node. Each function could take 'pedantic' as a parameter, which would eliminate the need to have an object at all, let alone a singleton. Finally, there is an interesting characteristic of the code's interpretation of names: any name other than 'True' is interpreted as 'False'! On the whole, though, I don't see a lot of difference between this format and say, JavaScript Object Notation (JSON), which can be parsed and generated by many other languages as well as multiple Python libraries already. From syfou at users.sourceforge.net Wed Jul 26 22:45:35 2006 From: syfou at users.sourceforge.net (Sylvain Fourmanoit) Date: Wed, 26 Jul 2006 16:45:35 -0400 (EDT) Subject: [Python-Dev] New miniconf module In-Reply-To: <44C7B8C2.1060904@blueyonder.co.uk> References: <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> Message-ID: <Pine.LNX.4.64.0607261613130.6037@sylvain> > miniconf, OTOH, appears to have an interface compatible with capability > security (I have not checked that the compiler.ast module used in its > implementation is safe.) I woudn't be 100% sure either (obviously, I didn't write this nice piece of code, let alone the underlying parser), but I read it and tried to abuse it without success (I haven't found obvious buffer overflow and such)... As far as I know, the abstract syntax tree generation exposed via compiler.ast is a safe operation, in the sense that it doesn't allow execution of code when feeded from arbitrary strings via compiler.parse(); in the worst case scenario, it raises a SyntaxError or similar exceptions, as documented... If anybody know more on this issue, I will be happy to hear about it. > miniconf has a few limitations one should be aware of: > > - It is not preemptiple: concurrent calls to dump() or load() will > have unpredictable results and must be avoided. > > This limitation should be fixed before the module is added to the > standard library, IMHO. If this is the general opinion, I will be glad to change this... The only reason miniconf is not thread-safe for the moment is that I chose to re-use over and over a single instance of each of my two processing classes to reduce resources usage, but this seems pretty pointless (and overly complicated) now that I look at it. Yours, -- Sylvain <syfou at users.sourceforge.net> Your files are now being encrypted and thrown into the bit bucket. EOF From syfou at users.sourceforge.net Wed Jul 26 23:43:21 2006 From: syfou at users.sourceforge.net (Sylvain Fourmanoit) Date: Wed, 26 Jul 2006 17:43:21 -0400 (EDT) Subject: [Python-Dev] New miniconf module In-Reply-To: <Pine.LNX.4.64.0607261613130.6037@sylvain> References: <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> Message-ID: <Pine.LNX.4.64.0607261742160.6037@sylvain> > It looks like it's trivial to fix; the code uses a strange and > unnecessary complication of creating nested classes and nested > singleton instances thereof. Getting rid of the singletons to create a > new instance for each dump/load call would suffice to make the > implementation re-entrant, although de-nesting the classes would also be > a good idea. :) OK then, I will change this. > The loading code could also be made a lot faster by using a dictionary > mapping AST node types to functions, instead of doing string > manipulation for each node. Each function could take 'pedantic' as a > parameter, which would eliminate the need to have an object at all, let > alone a singleton. > I am not convinced the current string manipulation for mapping the nodes types to the methods of the _Load class has such a significant impact on performance, but I will test your suggestion... The only difference with current code is that we use a dynamically computed string as the dictionary key to locate the function instead of the node type themselves as keys. > Finally, there is an interesting characteristic of the code's > interpretation of names: any name other than 'True' is interpreted as > 'False'! ;-) It will be corrected in the next release. > On the whole, though, I don't see a lot of difference between this format > and say, JavaScript Object Notation (JSON), which can be parsed and > generated by many other languages as well as multiple Python libraries > already. The difference is that this is Python code, already familiar to all Python coders... Besides, it sits directly on top of the real Python parser, mitigating the need of a new one, and keeping the added code complexity to a strict minimum. But I agree this looks a lot like JSON, since ecmascript syntax for literals looks a lot like the one of Python... For the same reasons there is a need for JSON, I think having something like miniconf in the standard lib would benefit the users. -- Sylvain <syfou at users.sourceforge.net> If you think the system is working, ask someone who's waiting for a prompt. From pje at telecommunity.com Thu Jul 27 00:03:50 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 26 Jul 2006 18:03:50 -0400 Subject: [Python-Dev] New miniconf module In-Reply-To: <Pine.LNX.4.64.0607261742160.6037@sylvain> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> Message-ID: <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> At 05:43 PM 7/26/2006 -0400, Sylvain Fourmanoit wrote: > > The loading code could also be made a lot faster by using a dictionary > > mapping AST node types to functions, instead of doing string > > manipulation for each node. Each function could take 'pedantic' as a > > parameter, which would eliminate the need to have an object at all, let > > alone a singleton. > > >I am not convinced the current string manipulation for mapping the nodes >types to the methods of the _Load class has such a significant impact on >performance, but I will test your suggestion... I haven't tested this with your code specifically, but I know that in the past I have nearly tripled the speed of AST-visiting code by doing this; string manipulation plus attribute lookup is a lot more expensive than direct dictionary lookups. > The only difference with >current code is that we use a dynamically computed string as the >dictionary key to locate the function instead of the node type themselves >as keys. Actually you're doing string manipulation plus an *attribute* lookup, and attribute lookups can involve multiple dictionary lookups. But anyway, test and see what you get. :) >But I agree this looks a lot like JSON, since ecmascript syntax for >literals looks a lot like the one of Python... For the same reasons there >is a need for JSON, I think having something like miniconf in the >standard lib would benefit the users. Actually, I would see more reason to include JSON in the standard library, since it's at least something approaching an internet protocol these days. From jjl at pobox.com Thu Jul 27 00:18:30 2006 From: jjl at pobox.com (John J Lee) Date: Wed, 26 Jul 2006 22:18:30 +0000 (UTC) Subject: [Python-Dev] New miniconf module In-Reply-To: <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> Message-ID: <Pine.LNX.4.64.0607262218060.8397@localhost> On Wed, 26 Jul 2006, Phillip J. Eby wrote: [...] > Actually, I would see more reason to include JSON in the standard library, > since it's at least something approaching an internet protocol these days. +1 John From bob at redivi.com Thu Jul 27 00:41:41 2006 From: bob at redivi.com (Bob Ippolito) Date: Wed, 26 Jul 2006 15:41:41 -0700 Subject: [Python-Dev] JSON implementation in Python 2.6 In-Reply-To: <Pine.LNX.4.64.0607262218060.8397@localhost> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> <Pine.LNX.4.64.0607262218060.8397@localhost> Message-ID: <661F2C7D-E446-4BB3-B5A7-8237F0850E91@redivi.com> On Jul 26, 2006, at 3:18 PM, John J Lee wrote: > On Wed, 26 Jul 2006, Phillip J. Eby wrote: > [...] >> Actually, I would see more reason to include JSON in the standard >> library, >> since it's at least something approaching an internet protocol >> these days. > > +1 If there's a consensus on that, my simplejson [1] implementation could migrate to the stdlib for 2.6. The API is modeled after marshal and pickle, the code should be PEP 8 compliant, its test suite has pretty good coverage, it's already used by (at least) TurboGears and Django, and it's the implementation currently "endorsed by" json.org. The work that would be required would be: - LaTeX docs (currently reST in docstrings) - Move the tests around and make them run from the suite rather than via nose - Possible module rename (jsonlib?) [1] http://undefined.org/python/#simplejson -bob From steven.bethard at gmail.com Thu Jul 27 01:42:58 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 26 Jul 2006 17:42:58 -0600 Subject: [Python-Dev] DRAFT: python-dev summary for 2006-06-16 to 2006-06-30 Message-ID: <d11dcfba0607261642t73095955lc807b4d9a29464a0@mail.gmail.com> Here's the draft for the second half of June. As always, comments and corrections are greatly appreciated. ============= Announcements ============= ------------------- Python 2.5 schedule ------------------- A number of bugs are being squashed as Python 2.5 moves towards its next release. See `PEP 356`_ for more details and the full schedule. .. _PEP 356: http://www.python.org/dev/peps/pep-0356/ Contributing threads: - `Beta 1 schedule ? (Bug in stringobject?) <http://mail.python.org/pipermail/python-dev/2006-June/066113.html>`__ - `Adding winerror module (Beta 1 schedule ?) <http://mail.python.org/pipermail/python-dev/2006-June/066159.html>`__ - `current 2.5 issues <http://mail.python.org/pipermail/python-dev/2006-June/066204.html>`__ - `TRUNK FREEZE IMMINENT FOR 2.5 BETA 1 - 00:00 UTC, 20-JUNE-2006 <http://mail.python.org/pipermail/python-dev/2006-June/066206.html>`__ - `beta1 coming real soon <http://mail.python.org/pipermail/python-dev/2006-June/066253.html>`__ - `RELEASED Python 2.5 (beta 1) <http://mail.python.org/pipermail/python-dev/2006-June/066318.html>`__ - `TRUNK is UNFROZEN, but in FEATURE FREEZE <http://mail.python.org/pipermail/python-dev/2006-June/066322.html>`__ - `2.5 and beyond <http://mail.python.org/pipermail/python-dev/2006-June/066807.html>`__ ----------------------------------------- Checkins for betas and release candidates ----------------------------------------- Anthony Baxter announced some guidelines for checkins for the beta and release candidate releases. For all beta releases: * All checkins must have an entry for Misc/NEWS, a test and docs * All checkins that add features must have approval from a release manager * All checkins must not break any of the buildbots For all release candidates: * All checkins must have approval from a release manager Approval from a release manager (Anthony or Neal) should preferably be obtained in public (e.g. the python-dev list) and should be noted in the commit message. Contributing threads: - `When to branch release25-maint? <http://mail.python.org/pipermail/python-dev/2006-June/066200.html>`__ - `RFC: trunk checkins between now and 2.5 final <http://mail.python.org/pipermail/python-dev/2006-June/066718.html>`__ ------------------------------------------- FishEye on the Python Subversion Repository ------------------------------------------- FishEye is once again `available for the Python repository`_. .. _available for the Python repository: http://fisheye3.cenqua.com/browse/python Contributing thread: - `FishEye on Python CVS Repository <http://mail.python.org/pipermail/python-dev/2006-June/066188.html>`__ ========= Summaries ========= --------------------------------- PEP 3103: A Switch/Case Statement --------------------------------- After Thomas Lee provided a `simple patch implementing a switch statement`_ for Python, there was a massive discussion about it and how `PEP 275`_ should best be implemented. After much discussion, basically three camps arose: * School I: The switch statement should just be syntactic sugar for the corresponding if/elif chain. * School II: The switch statement should dispatch on a precomputed dict of values. * School III: The switch statement should correspond to an if/elif chain but require all expressions to be hashable (to allow for better optimizations). School I was primarily concerned with the repetition of the ``x ==`` in something like:: if x == ...: ... elif x == ...: ... elif x == ...: ... else: ... School II seemed to feel that just aiding DRY was not enough to introduce a new construct, and that the switch statement should also be able to avoid the function definitions in dispatching code like:: def f(...): ... def g(...): ... def h(...): ... dispatch_dict = {x:f, y:g, z:h} dispatch_dict[value](*args) In order to optimize this kind of code, School II wanted to be able to compute the dispatch dict ahead of time, so that it wouldn't have be recomputed each time the switch statement was executed. There was a lot of discussion as to exactly when this freezing should occur, with some voting for module compilation time (allowing only constants in the cases), some voting for function definition time (allowing only constants and non-local names in the cases) and some voting for the first time the switch statement is executed (allowing only constants and both local and non-local names). Guido put together a thorough summary of the options in `PEP 3103`_. There was some discussion of introducing a ``static`` keyword which would cause an expression to be evaluated at function definition time, so that, for example, the following code would create a list of functions returning each of 0, 1, ... 9:: funcs = [lambda: (static i) for i in xrange(10)] The intention was that switch statement cases would then allow only constants or static expressions. Guido requested a separate PEP on the idea, and `Fredrik Lundh posted a proto-PEP`_, but at the time of this summary, no official PEP had been submitted. In the end, it looked like Guido was leaning towards the switch statement as syntactic sugar for a dispatching dict, with the dict frozen at function definition time (which would mean compile-time for module-level switch statements). However, the introduction of the statement seemed likely to be postponed at least until Python 3.0. .. _simple patch implementing a switch statement: http://bugs.python.org/1504199 .. _PEP 275: http://www.python.org/dev/peps/pep-0275/ .. _PEP 3103: http://www.python.org/dev/peps/pep-3103/ .. _Fredrik Lundh posted a proto-PEP: http://online.effbot.org/2006_06_01_archive.htm#pep-static Contributing threads: - `Switch statement <http://mail.python.org/pipermail/python-dev/2006-June/066086.html>`__ - `An obscene computed goto bytecode hack for "switch" :) <http://mail.python.org/pipermail/python-dev/2006-June/066115.html>`__ - `Simple Switch statement <http://mail.python.org/pipermail/python-dev/2006-June/066499.html>`__ - `Alternatives to switch? <http://mail.python.org/pipermail/python-dev/2006-June/066508.html>`__ - `Temporary Constantification <http://mail.python.org/pipermail/python-dev/2006-June/066531.html>`__ - `Simple Switch statementZ <http://mail.python.org/pipermail/python-dev/2006-June/066537.html>`__ - `PEP 3103: A Switch/Case Statement <http://mail.python.org/pipermail/python-dev/2006-June/066570.html>`__ - `School IIb? <http://mail.python.org/pipermail/python-dev/2006-June/066585.html>`__ - `Switch statement - handling errors <http://mail.python.org/pipermail/python-dev/2006-June/066650.html>`__ - `Split switch statement <http://mail.python.org/pipermail/python-dev/2006-June/066652.html>`__ - `once [was: Simple Switch statementZ] <http://mail.python.org/pipermail/python-dev/2006-June/066692.html>`__ ------------------------------ Restricted execution in Python ------------------------------ For his Ph.D. thesis, Brett Cannon is looking into adding facilities for restricted execution to Python, partly with the goal of getting Python into Firefox alongside Javascript. His restricted execution specifications aimed to take advantage of the C-to-Python language barrier to enforce security restrictions. Though there's no real way to get private attributes in pure Python, objects coded in C and exposed to Python can select which attributes are exposed, thus making the non-exposed attributes truly private to Python-level code. His initial draft aimed to hide as many "dangerous" objects as possible, and then cripple objects like ``file`` that would be difficult to hide. A number of people seemed to prefer a hiding-only approach, but comments from Armin Rigo seemed to suggest that plugging all the introspection holes that give access to file objects might be quite difficult. The discussion continued on into the next fortnight. Contributing threads: - `doc for new restricted execution design for Python <http://mail.python.org/pipermail/python-dev/2006-June/066344.html>`__ - `Is Lib/test/crashers/recursive_call.py really a crasher? <http://mail.python.org/pipermail/python-dev/2006-June/066627.html>`__ - `For sandboxing: alternative to crippling file() <http://mail.python.org/pipermail/python-dev/2006-June/066792.html>`__ ----------------------------------------------- NaN and infinities in Python float calculations ----------------------------------------------- Nick Maclaren asked about trying to get more platform-independent behavior in Python's floats, so that IEEE 754 values as in `PEP 754`_ would be produced more consistently. Currently, different OSes produce different results when these values are involved:: Python 2.4.2 (#1, May 2 2006, 08:28:01) [GCC 4.1.0 (SUSE Linux)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> a = "NaN" >>> b = float(a) >>> c = int(b) >>> d = (b == b) >>> print a, b, c, d NaN nan 0 False Python 2.3.3 (#1, Feb 18 2004, 11:58:04) [GCC 2.8.1] on sunos5 Type "help", "copyright", "credits" or "license" for more information. >>> a = "NaN" >>> b = float(a) >>> c = int(b) >>> d = (b == b) >>> print a, b, c, d NaN NaN 0 True Nick Maclaren suggested either raising an exception for all ambiguous or invalid operations, or returning NaN or infinity as appropriate and then raising exceptions whenever an operation that would lose the error indication was performed. Nick Coghlan explained that the decimal module already does most of this:: >>> from decimal import Decimal as d >>> nan = d('NaN') >>> int(nan) Traceback (most recent call last): ... decimal.InvalidOperation >>> >>> from decimal import getcontext, Overflow >>> ctx = getcontext() >>> ctx.traps[Overflow] = False >>> d('1e999999999') * 10 Decimal("Infinity") Nick Maclaren seemed to suggest that he would be working on a PEP and an implementation that would bring some of the decimal module consistencies to Python's floats as well. .. _PEP 754: http://www.python.org/dev/peps/pep-0754/ Contributing threads: - `Numerical robustness, IEEE etc. <http://mail.python.org/pipermail/python-dev/2006-June/066186.html>`__ - `Numerical robustness, IEEE etc. <http://mail.python.org/pipermail/python-dev/2006-June/066192.html>`__ - `Python memory model (low level) <http://mail.python.org/pipermail/python-dev/2006-June/066834.html>`__ ----------------------------------------------- ImportWarnings for directories without __init__ ----------------------------------------------- Ralf W. Grosse-Kunstleve complained that with Python 2.5 he started getting tons of "ImportWarning: Not importing directory" messages. James Y Knight pointed out that running Python in your home directory is quite likely to issue such warnings if you have *any* directories in your home directory that have the same name as a python module (e.g. ``readline``). A number of options for silencing the errors were discussed, including invoking Python like ``python -W'ignore:Not importing directory'`` and including ``warnings.filterwarnings('ignore', 'Not importing directory', ImportWarning)`` in site.py or .pythonrc.py. Two patches were provided that introduce the warning only if the import fails, `one by Shane Hathaway`_ and `one by Sergey A. Lipnevich`_. No final decision had been made at the time of this summary. .. _one by Shane Hathaway: http://bugs.python.org/1515361 .. _one by Sergey A. Lipnevich: http://bugs.python.org/1515609 Contributing threads: - `Dropping __init__.py requirement for subpackages <http://mail.python.org/pipermail/python-dev/2006-June/066280.html>`__ - `ImportWarning flood <http://mail.python.org/pipermail/python-dev/2006-June/066345.html>`__ ------------------ Updating turtle.py ------------------ Gregor Lingl proposed replacing turtle.py in the Python standard library with his `new xturtle.py module`_. The xturtle module is backwards compatible with the turtle module and adds a number of enhancements. However, Gregor's request came after Python 2.5's feature freeze, so he was told to propose it again in Python 2.6. There was some discussion about this -- as the stdlib turtle module is poorly tested, some contended that introducing the new APIs of xturtle would not make things any worse. A couple of compromises were offered: mentioning xturtle in the turtle module docs, and putting xturtle in the Tools directory. .. _new xturtle.py module: http://ada.rg16.asn-wien.ac.at/~python/xturtle/ Contributing threads: - `xturtle.py a replacement for turtle.py(!?) <http://mail.python.org/pipermail/python-dev/2006-June/066676.html>`__ - `xturtle.py - a replacement for turtle.py <http://mail.python.org/pipermail/python-dev/2006-June/066677.html>`__ - `xturtle.py a replacement for turtle.py(!?) ATTENTION PLEASE! <http://mail.python.org/pipermail/python-dev/2006-June/066734.html>`__ - `xturtle.py <http://mail.python.org/pipermail/python-dev/2006-June/066742.html>`__ ---------------------------------------------------------- Relative imports and PEP 338: Executing Modules as Scripts ---------------------------------------------------------- Relative imports, as described in `PEP 328`_, introduced problems for `PEP 338`_ which allows modules within packages and zipfiles to be run with the -m command-line switch. The -m switch sets the __name__ of the module to '__main__' so that ``if __name__ == '__main__'`` blocks will get executed. However, relative imports use __name__ to determine the parent package, so if a module that has a relative import is executed using the -m switch, the relative import will fail. Nick Coghlan suggested adding a __module_name__ attribute that would not be clobbered by the -m switch, but people generally seemed to think that it would be simpler to just require absolute imports in main modules. .. _PEP 328: http://www.python.org/dev/peps/pep-0328/ .. _PEP 338: http://www.python.org/dev/peps/pep-0338/ Contributing threads: - `PEP 338 vs PEP 328 - a limitation of the -m switch <http://mail.python.org/pipermail/python-dev/2006-June/066161.html>`__ - `PEP 328 and PEP 338, redux <http://mail.python.org/pipermail/python-dev/2006-June/066609.html>`__ - `[Python-checkins] r47142 - in python/trunk: Doc/lib/librunpy.tex Lib/runpy.py Lib/test/test_runpy.py <http://mail.python.org/pipermail/python-dev/2006-June/066690.html>`__ -------------------------------------------- Importing modules within unicode directories -------------------------------------------- Kristj?n V. J?nsson pointed out that currently, Python on Windows cannot import modules from directories with unicode names, even if the module names themselves are plain ASCII. Nick Coghlan suggested that this was likely because import.c was doing something like ``u'c:/tmp/\u814c'.encode('mbcs')``, getting back ``'c:/tmp/?'`` and being unable to do anything useful with that. Martin v. L?wis suggested using the 8.3 simplified filename used by DOS, at least until the import machinery gets reworked to better handle encodings, hopefully for Python 2.6. `Thomas Heller had provided a patch`_ for reworking import.c in this manner a while back, but it was large enough that no one had reviewed it. .. _Thomas Heller had provided a patch: http://bugs.python.org/1093253 Contributing thread: - `unicode imports <http://mail.python.org/pipermail/python-dev/2006-June/066103.html>`__ ---------------------------------------- MS VC++ 2003 toolkit no longer available ---------------------------------------- Bill Janssen pointed out that Python 2.4 on Windows expects to be compiled with the MS Visual C++ compiler version 7.1, and that the corresponding MS VC++ 2003 toolkit is no longer available. Fredrik Lundh explained that the compiler is still available in the .net SDK as well as being available to MSDN subscribers. There was again some discussion about moving to the VS 2005 toolkit for compiling Python. It would have made compiling for 64bit architectures somewhat easier, but would have meant that extension writers would have to install three different compilers just to compile extensions for Python 2.3, 2.4 and 2.5, and would also have given problems for MinGW users as MinGW does not yet easily support linking to the msvcr80 runtime library. Contributing threads: - `Python 2.4 extensions require VC 7.1? <http://mail.python.org/pipermail/python-dev/2006-June/066110.html>`__ - `Documentation enhancement: "MS free compiler"? <http://mail.python.org/pipermail/python-dev/2006-June/066182.html>`__ - `Documentation enhancement: "MS free compiler"? <http://mail.python.org/pipermail/python-dev/2006-June/066257.html>`__ --------------------------------- Keeping interned strings in a set --------------------------------- Alexander Belopolsky tried out the new set C API by `replacing the dict of interned strings with a set`_ instead. He had to make two changes to get this to work: there's currently no way to retrieve a single object from a set, and Py_Finalize() needed to be changed to finalize sets after strings (instead of the other way around as it used to be). There was some discussion about trying to get rid of PySet_Fini() so the latter problem wouldn't be an issue at all, but with all the other Py*Fini() functions already existing, it didn't seem worth it. The patch had no slowdown and reduced the memory consumption of the interning structure slightly. .. _replacing the dict of interned strings with a set: http://bugs.python.org/1507011 Contributing threads: - `Keeping interned strings in a set <http://mail.python.org/pipermail/python-dev/2006-June/066084.html>`__ - `Keeping interned strings in a set <http://mail.python.org/pipermail/python-dev/2006-June/066088.html>`__ - `setobject code <http://mail.python.org/pipermail/python-dev/2006-June/066116.html>`__ - `Proposal to eliminate PySet_Fini <http://mail.python.org/pipermail/python-dev/2006-June/066645.html>`__ ------------------------- Allowing empty subscripts ------------------------- Guido finally vetoed the proposal to allow ``x[()]`` to be written as ``x[]``. The use-cases were weak, and in most cases the functionality seemed better expressed as attribute access. Contributing threads: - `Pre-PEP: Allow Empty Subscript List Without Parentheses <http://mail.python.org/pipermail/python-dev/2006-June/066099.html>`__ - `Empty Subscript PEP on Wiki - keep or toss? <http://mail.python.org/pipermail/python-dev/2006-June/066848.html>`__ ------------------------------------- Creating range objects at the C level ------------------------------------- Ralf W. Grosse-Kunstleve asked about the removal of the C function ``PyRange_New()`` which had been deprecated in Python 2.4. The right way to create ranges is to call PyRange_Type with the appropriate parameters, e.g. something like ``PyObject_CallFunction((PyObject*) &PyRange_Type, "lll", start, stop, step)``. Ralf was nervous about this alternative because it also appeared to be undocumented, and requested that something like the above be at least put into the What's New document. Contributing threads: - `PyRange_New() alternative? <http://mail.python.org/pipermail/python-dev/2006-June/066343.html>`__ - `PyObject* aliasing (Was: PyRange_New() alternative?) <http://mail.python.org/pipermail/python-dev/2006-June/066477.html>`__ ---------------------------------- type(), __class__ and isinstance() ---------------------------------- Martin Maly pointed out that you can't fool isinstance() into thinking your object is not a subclass of its true base class:: >>> class C(object): ... pass ... >>> class D(object): ... __class__ = property(lambda self: C) ... >>> isinstance(D(), D) True >>> isinstance(D(), C) True Phillip J. Eby explained that isinstance() checks both the type() of the object and the __class__ attribute. In essence, you can lie about your __class__ to make isinstance() return True, but you can't lie to make it return False. Guido suggested that these issues, as well as lying about an object's __bases__, should be revisited for Python 3000. Contributing thread: - `Semantic of isinstance <http://mail.python.org/pipermail/python-dev/2006-June/066591.html>`__ ---------------------------------------------------------------- Requiring backward compatibility in the standard library modules ---------------------------------------------------------------- Ka-Ping Yee's uuid module, newly added for Python 2.5, contained a comment "This module works with Python 2.3 or higher". George Yoshida asked if that comment should be interpreted as requiring Python 2.3 compatibility. People generally felt like the list of backwards compatible modules in `PEP 291`_ should be as small as possible so as to keep maintenance as simple as possible. Ka-Ping removed the comment, and submitted the module to PyPI for Python 2.3 and 2.4 users. .. _PEP 291: http://www.python.org/dev/peps/pep-0291/ Contributing thread: - `uuid backward compatibility <http://mail.python.org/pipermail/python-dev/2006-June/066153.html>`__ --------------------- Figleaf code coverage --------------------- `Titus Brown offered some reports`_ from his `figleaf code coverage`_ utility. People seemed particularly interested in trying to get coverage across multiple platforms, perhaps using a BuildBot extension, and Titus said he'd try to look into it. Walter D?rwald also pointed to `his own code coverage module`_. .. _Titus Brown offered some reports: http://vallista.idyll.org/~t/temp/python2.4-svn/ .. _figleaf code coverage: http://darcs.idyll.org/~t/projects/figleaf-latest.tar.gz .. _his own code coverage module: http://styx.livinglogic.de/~walter/python/coverage/PythonCodeCoverage.py Contributing threads: - `Code coverage reporting. <http://mail.python.org/pipermail/python-dev/2006-June/066184.html>`__ - `Code coverage reporting. <http://mail.python.org/pipermail/python-dev/2006-June/066193.html>`__ ------------------------ Improving error messages ------------------------ Georg Brandl proposed going through abstract.c and modifying error messages like "object does not support item assignment" to also include the type of the object. He got little feedback, mainly because everyone seemed to think it was such an obviously good idea that there was no need for any. Python 2.5 now incorporates `Georg's better error messages`_. .. _Georg's better error messages: http://bugs.python.org/1507676 Contributing threads: - `Improve error msgs? <http://mail.python.org/pipermail/python-dev/2006-June/066048.html>`__ - `Improve error msgs? <http://mail.python.org/pipermail/python-dev/2006-June/066128.html>`__ ----------------------------------------- Allowing assignments in global statements ----------------------------------------- Talin proposed allowing a global statement to be combined with an assignment statement, e.g.:: global badger = 42 Guido suggested that such a desire was a sure indicator of overuse of ``global``. Contributing thread: - `Allow assignments in 'global' statements? <http://mail.python.org/pipermail/python-dev/2006-June/066347.html>`__ ----------------------------------------- Splitting Python tests from CPython tests ----------------------------------------- Frank Wierzbicki volunteered some time into splitting out CPython specific test from Python-the-language tests. Armin Rigo pointed him to PyPy's `tests modified to be more implementation independent`_. .. _tests modified to be more implementation independent: http://codespeak.net/svn/pypy/dist/lib-python/modified-2.4.1/test Contributing thread: - `Cleanup of test harness for Python <http://mail.python.org/pipermail/python-dev/2006-June/066817.html>`__ ----------------------------------------- A multi-dimensional array type for Python ----------------------------------------- For `Google's Summer of Code`_, Karol Langner will be working on `implementing a basic multi-dimensional array type`_ for Python core, based on the numpy_ array struct. He asked for any comments or suggestions that people had for the project. .. _Google's Summer of Code: http://code.google.com/summerofcode.html .. _numpy: http://www.numpy.org/ .. _implementing a basic multi-dimensional array type: http://scipy.org/BaseArray Contributing thread: - `basearray <http://mail.python.org/pipermail/python-dev/2006-June/066516.html>`__ ================== Previous Summaries ================== - `Source control tools <http://mail.python.org/pipermail/python-dev/2006-June/066187.html>`__ - `Dropping externally maintained packages (Was: Please stop changing wsgiref on the trunk) <http://mail.python.org/pipermail/python-dev/2006-June/066195.html>`__ =============== Skipped Threads =============== - `Last-minute curses patch <http://mail.python.org/pipermail/python-dev/2006-June/066095.html>`__ - `Bug in stringobject? <http://mail.python.org/pipermail/python-dev/2006-June/066100.html>`__ - `Fwd: subprocess.Popen(.... stdout=IGNORE, ...) <http://mail.python.org/pipermail/python-dev/2006-June/066111.html>`__ - `About dynamic module loading <http://mail.python.org/pipermail/python-dev/2006-June/066190.html>`__ - `PyString_FromFormat <http://mail.python.org/pipermail/python-dev/2006-June/066213.html>`__ - `Misleading error message from PyObject_GenericSetAttr <http://mail.python.org/pipermail/python-dev/2006-June/066227.html>`__ - `Bug: xml.dom.pulldom never gives you END_DOCUMENT events with an Expat parser <http://mail.python.org/pipermail/python-dev/2006-June/066228.html>`__ - `os.getmtime now returns a float? <http://mail.python.org/pipermail/python-dev/2006-June/066252.html>`__ - `XP build failing <http://mail.python.org/pipermail/python-dev/2006-June/066258.html>`__ - `ETree: xml vs xmlcore <http://mail.python.org/pipermail/python-dev/2006-June/066268.html>`__ - `test_ctypes failure on Mac OS X/PowerPC 10.3.9 (Panther) <http://mail.python.org/pipermail/python-dev/2006-June/066282.html>`__ - `Small sqlite3 test suite fix (Python 2.5b1 candidate) <http://mail.python.org/pipermail/python-dev/2006-June/066291.html>`__ - `Weekly Python Patch/Bug Summary <http://mail.python.org/pipermail/python-dev/2006-June/066350.html>`__ - `Things to remember when adding *packages* to stdlib <http://mail.python.org/pipermail/python-dev/2006-June/066353.html>`__ - `Moving the ctypes repository to python.org <http://mail.python.org/pipermail/python-dev/2006-June/066417.html>`__ - `PyObject_CallFunction and 'N' format char <http://mail.python.org/pipermail/python-dev/2006-June/066501.html>`__ - `pypy-0.9.0: stackless, new extension compiler <http://mail.python.org/pipermail/python-dev/2006-June/066512.html>`__ - `[Python-checkins] Things to remember when adding *packages* to stdlib <http://mail.python.org/pipermail/python-dev/2006-June/066515.html>`__ - `Import semantics <http://mail.python.org/pipermail/python-dev/2006-June/066523.html>`__ - `2.5b1 Windows install <http://mail.python.org/pipermail/python-dev/2006-June/066542.html>`__ - `Python-Dev Digest, Vol 35, Issue 143 <http://mail.python.org/pipermail/python-dev/2006-June/066577.html>`__ - `Problems building Python on OSX 10.4.6? <http://mail.python.org/pipermail/python-dev/2006-June/066579.html>`__ - `enhancements for uuid module <http://mail.python.org/pipermail/python-dev/2006-June/066583.html>`__ - `Do we need a bug triage day? <http://mail.python.org/pipermail/python-dev/2006-June/066647.html>`__ - `Oh-why that?? Please ignore one of the two <http://mail.python.org/pipermail/python-dev/2006-June/066678.html>`__ - `msvccompiler.py: some remarks <http://mail.python.org/pipermail/python-dev/2006-June/066768.html>`__ - `Joke: Rush Limbaugh (a joke in and of himself) <http://mail.python.org/pipermail/python-dev/2006-June/066780.html>`__ - `PyGIL_ and --without-threads <http://mail.python.org/pipermail/python-dev/2006-June/066784.html>`__ - `document @property? <http://mail.python.org/pipermail/python-dev/2006-June/066787.html>`__ - `Pickle implementation questions <http://mail.python.org/pipermail/python-dev/2006-June/066803.html>`__ - `sys.settrace() in Python 2.3 vs. 2.4 <http://mail.python.org/pipermail/python-dev/2006-June/066820.html>`__ - `how long to wait for expat to incorporate a fix to prevent a crasher? <http://mail.python.org/pipermail/python-dev/2006-June/066829.html>`__ - `LOAD_CONST POP_TOP <http://mail.python.org/pipermail/python-dev/2006-June/066832.html>`__ From tim.peters at gmail.com Thu Jul 27 02:02:37 2006 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 26 Jul 2006 20:02:37 -0400 Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery Message-ID: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com> Rarely I'll be running the Python tests in my sandbox from a DOS box, and the test run will just end. Like so: C:\Code\python\PCbuild>python -E -tt ../lib/test/regrtest.py -uall -rw test_softspace test_codecmaps_kr ... test_float test_userdict C:\Code\python\PCbuild> No indication of success or failure -- the process just vanishes mid-stream. Today I noticed this happened when the buildbot started to run tests, and I'm 100% sure it's due to this code in Tools/buildbot/kill_python.c (the buildbot log files showed that kill_python.c killed /some/ Python process, and the Python running release-build tests in my sandbox was the only plausible candidate): if ((strstr(path, "build\\pcbuild\\python_d.exe") != NULL) || (strstr(path, "build\\python.exe") != NULL)) { printf("Terminating %s (pid %d)\n", path, pids[i]); if (!TerminateProcess(hProcess, 1)) { The second clause in the first `if` looks for a substring match on: build\python.exe and that just happens to match a suffix of: C:\Code\python\PCbuild\python.exe which is the release-build Python I happen to be running in my sandbox. Why is the second clause there? That is, are we /trying/ to kill a release-build Python running from the user's sandbox, and if so why? Introducing the second clause was the sole change in rev 46817, and the checkin comment doesn't really explain it: Port cygwin kill_python changes from 2.4 branch. Since I don't know what it's trying to accomplish, I hesitate to change it. It's quite clear what the first clause is trying to accomplish, and that one hasn't caused any problems. From mattjfleming at googlemail.com Thu Jul 27 02:20:19 2006 From: mattjfleming at googlemail.com (Matt Fleming) Date: Thu, 27 Jul 2006 00:20:19 +0000 Subject: [Python-Dev] Improving unit tests for the standard library Message-ID: <5ff4a1e50607261720o2a98572ft4cab275ed91ad344@mail.gmail.com> Hi, after speaking with Neal off-list about writing tests for the pkgutil module, we agreed it would be a good idea to start a page on http://wiki.python.org/moin/ stating any tests for the standard library that either, a) need to be written b) can be improved I've started the page http://wiki.python.org/moin/ImprovingLibTests that lists all the test files I could think of that need to be written. Ive also included a table for improvements to existing tests, along with a column that allows you to specify exactly what needs improving. I hope this will be of use to people, and I hope people will find time to modify the page approriately. When I get some spare time from my SoC project, I'll be working my way through the list. Thanks, Matt -- http://mattssanctuary.blogspot.com From ocean at m2.ccsnet.ne.jp Thu Jul 27 03:15:23 2006 From: ocean at m2.ccsnet.ne.jp (H.Yamamoto) Date: Thu, 27 Jul 2006 10:15:23 +0900 Subject: [Python-Dev] patch for mbcs codec (again) Message-ID: <002601c6b11a$22e3fa70$0400a8c0@whiterabc2znlh> Hello. I noticed mbcs codec still has problem when calls StreamReader. Can anyone commit the patch "fix.patch version2" on http://python.org/sf/1455898 ? # Very sorry about this... I thought I checked this, but I definitely looked at something # wrong. From kbk at shore.net Thu Jul 27 04:18:12 2006 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed, 26 Jul 2006 22:18:12 -0400 (EDT) Subject: [Python-Dev] Weekly Python Patch/Bug Summary Message-ID: <200607270218.k6R2ICQ9021704@bayview.thirdcreek.com> Patch / Bug Summary ___________________ Patches : 401 open ( +3) / 3342 closed ( +8) / 3743 total (+11) Bugs : 896 open ( -8) / 6035 closed (+24) / 6931 total (+16) RFE : 224 open ( +2) / 233 closed ( +2) / 457 total ( +4) New / Reopened Patches ______________________ pkgutil.walk_packages ignores onerror arg (2006-07-20) CLOSED http://python.org/sf/1525766 opened by James Y Knight Tkdnd mouse cursor handling patch (2006-07-20) http://python.org/sf/1525806 opened by klappnase calltip awkwardly shortened (2006-07-20) CLOSED http://python.org/sf/1525817 opened by Lingl str.__iter__ and unicode.__iter__ (2006-07-21) http://python.org/sf/1526367 opened by Walter D??rwald Fix socketmodule compile on NetBSD (2006-07-21) http://python.org/sf/1526460 opened by Matt Fleming New module: miniconf (2006-07-24) http://python.org/sf/1527597 opened by S.Fourmanoit winsound - probably win9x port is not working (2006-07-24) CLOSED http://python.org/sf/1527744 opened by Hirokazu Yamamoto Expose case-insensitivity of string.Template (2006-07-25) http://python.org/sf/1528167 opened by Chad Whitacre PyShell.recall - fix indentation logic (2006-07-25) http://python.org/sf/1528468 opened by Tal Einat patch for mbcs codecs (2006-03-22) http://python.org/sf/1455898 reopened by ocean-city Patches Closed ______________ pkgutil.walk_packages ignores onerror arg (2006-07-20) http://python.org/sf/1525766 closed by gbrandl calltip awkwardly shortened (2006-07-20) http://python.org/sf/1525817 closed by loewis ParenMatch: workaround for misinterpreting of closing parens (2006-01-16) http://python.org/sf/1407280 closed by kbk Syntax-related improvements to IDLE (2004-02-28) http://python.org/sf/906702 closed by kbk (partial?) fix for Misc/python-config.in (2006-07-16) http://python.org/sf/1523356 closed by gbrandl ConnectRegistry blocks all threads (2006-03-12) http://python.org/sf/1448199 closed by loewis MS Windows - module search path fix (2005-07-04) http://python.org/sf/1232023 closed by loewis winsound - probably win9x port is not working (2006-07-24) http://python.org/sf/1527744 closed by gbrandl 1515163 fix - traceback and str exc (2006-06-30) http://python.org/sf/1515343 closed by gbrandl New / Reopened Bugs ___________________ inspect.py: still infinite recursion inspecting frames (2006-07-03) CLOSED http://python.org/sf/1516184 reopened by nnorwitz Bug in shutil.copytree on Windows (2006-07-20) http://python.org/sf/1525866 opened by Mike Foord email package quoted printable behaviour changed (2006-07-20) http://python.org/sf/1525919 opened by Thomas Arendsen Hein Win32: subprocess.Popen() w/o "pipe" throws an exception (2006-07-21) CLOSED http://python.org/sf/1526203 opened by Larry Hastings Concatenation on a long string breaks (2006-07-21) http://python.org/sf/1526585 opened by Jp Calderone current directory added to sys.path on win32 (2006-07-22) CLOSED http://python.org/sf/1526785 opened by John Ehresman unbalanced parentheses from command line crash pdb (2006-07-21) http://python.org/sf/1526834 opened by Ilya Sandler PythonLauncher uses incorrect working directory (2006-07-23) http://python.org/sf/1527397 opened by Bob Ippolito optparse should support arbitrary number of arguments (2006-07-24) http://python.org/sf/1527705 reopened by riteshsarraf optparse should support arbitrary number of arguments (2006-07-24) http://python.org/sf/1527705 opened by Ritesh Raj Sarraf tarfile chokes on ipython archive on Windows (2006-07-24) http://python.org/sf/1527974 opened by Arve Knudsen difflib.SequenceMatcher.find_longest_match() wrong result (2006-07-25) http://python.org/sf/1528074 opened by John Machin urllib2 data argument (2006-07-25) http://python.org/sf/1528258 opened by paul rubin forward in turtle module may cause incorrect display (2006-07-25) http://python.org/sf/1528363 opened by NatureMage IDLE: printing issue on OSX (2006-07-25) http://python.org/sf/1528593 opened by Ronald Oussoren Python 2.5b2 fails to build on Solaris 10 (2006-07-25) http://python.org/sf/1528620 opened by Guido Ostkamp Python 2.5b2 fails to build (GCC) on Solaris 10 (2006-07-26) http://python.org/sf/1529269 opened by Guido Ostkamp Bugs Closed ___________ inspect.py: still infinite recursion inspecting frames (2006-07-03) http://python.org/sf/1516184 closed by pje logging using the SysLog handler fails if locale is set (2006-07-17) http://python.org/sf/1524081 closed by vsajip Win32: subprocess.Popen() w/o "pipe" throws an exception (2006-07-21) http://python.org/sf/1526203 closed by gbrandl Malloc, memory error, failmalloc, low memory. (2006-07-19) http://python.org/sf/1525589 closed by nnorwitz current directory added to sys.path on win32 (2006-07-22) http://python.org/sf/1526785 closed by loewis IDLE (macosx): Class and Path browsers show Tk menu (2006-07-06) http://python.org/sf/1517996 closed by ronaldoussoren Column Number is not visible in MacOSX (2003-07-09) http://python.org/sf/768481 closed by kbk Tooltip-window doesn't vanish if... (2003-07-22) http://python.org/sf/775535 closed by kbk failure of test_ossaudiodev; elapsed time .1 sec faster (2006-06-05) http://python.org/sf/1501330 closed by gward Argument missing from calltip for new-style class init (2004-09-13) http://python.org/sf/1027566 closed by kbk length of unicode string changes print behaviour (2006-02-22) http://python.org/sf/1436532 closed by loewis exec and eval allocate lots of memory and do not free it (2006-07-20) http://python.org/sf/1525678 closed by loewis os.listdir doesn't check error code from FindNextFile (2006-07-18) http://python.org/sf/1524310 closed by loewis os.path.abspath() / os.chdir() buggy with unicode paths (2005-09-07) http://python.org/sf/1283895 closed by loewis traceback now masks some string exceptions (2006-06-30) http://python.org/sf/1515163 closed by gbrandl sys.ps1 not protected in EditorWindow.py (2004-08-16) http://python.org/sf/1010370 closed by kbk EditorWindow demo causes attr-error (2006-01-28) http://python.org/sf/1417598 closed by kbk Building 2.5a1 requires python (2006-04-06) http://python.org/sf/1465408 closed by anthonybaxter New / Reopened RFE __________________ New sequences for Unicode classes needed (2006-07-25) http://python.org/sf/1528154 opened by gmarketer RFE Closed __________ Add Windows 9x/ME (lack of) support information to README.TX (2006-06-22) http://python.org/sf/1510853 closed by loewis From collinw at gmail.com Thu Jul 27 04:24:37 2006 From: collinw at gmail.com (Collin Winter) Date: Wed, 26 Jul 2006 22:24:37 -0400 Subject: [Python-Dev] Which version of distutils to ship with Python 2.5? Message-ID: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com> Is it intentional that Python 2.5 is (currently) shipping with distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3) shipped with distutils 2.4.1? Judging from my own tests, distutils 2.4.1 fixed several bugs that some of my test suites depend on (the fixes, not the bugs ; ). Thanks, Collin Winter From david.nospam.hopwood at blueyonder.co.uk Thu Jul 27 04:19:40 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Thu, 27 Jul 2006 03:19:40 +0100 Subject: [Python-Dev] Internal namespace proposal In-Reply-To: <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org> <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> Message-ID: <44C822BC.1070204@blueyonder.co.uk> [This message is cc:d to the e-lang list, but please take any replies to python-dev at python.org.] Brett Cannon wrote: > On 7/19/06, Ka-Ping Yee <cap-talk at zesty.ca> wrote: > >> OMG!!! Is all i can say at the moment. Very excited. This is very encouraging. Thanks to ?!ng, Michael Chermside and others for making the case for capabilities. > Also realize that I am using object-capabilities to secure the interpreter, > not objects. That will be enough of a challenge to do for now. Who knows, > maybe some day Python can support object-capabilities at the object level, > but for now I am just trying to isolate and protect individual interpreters > in the same process. I think that the alternative of providing object-granularity protection domains straight away is more practical than you suggest, and I'd like to at least make sure that this possibility has been thoroughly explored. Below is a first-cut proposal for enforcing namespace restrictions, i.e. support for non-public attributes and methods, on Python objects and modules. It is not sufficient by itself to provide capability security, but it could be the basis for doing that at object granularity. (Note that this proposal would only affect sandboxed/restricted interpreters, at least for the time being. The encapsulation it provides is also useful for reasons other than security, and I think there is nothing about it that would be unreasonable to apply to an unrestricted interpreter, but for compatibility, that would have to be enabled by a __future__ option or similar.) Internal namespace proposal =========================== Existing Python code tends to use a convention where the names of attributes and methods intended only for internal use are prefixed by '_'. This convention comes from PEP 8 <http://www.python.org/dev/peps/pep-0008/>, which says: # In addition, the following special forms using leading or trailing # underscores are recognized (these can generally be combined with any case # convention): # # - _single_leading_underscore: weak "internal use" indicator. E.g. "from M # import *" does not import objects whose name starts with an underscore. # # - single_trailing_underscore_: used by convention to avoid conflicts with # Python keyword, e.g. # # Tkinter.Toplevel(master, class_='ClassName') # # - __double_leading_underscore: when naming a class attribute, invokes name # mangling (inside class FooBar, __boo becomes _FooBar__boo; see below). # # - __double_leading_and_trailing_underscore__: "magic" objects or # attributes that live in user-controlled namespaces. E.g. __init__, # __import__ or __file__. Never invent such names; only use them # as documented. I propose that the "internal" status of names beginning with _ (including those beginning with __) should be enforced in restricted interpreters. This is better than introducing a new annotation, because it will do the right thing for existing code that follows this part of PEP 8. More precisely: A restricted interpreter refuses access to any object attribute or method with a name beginning with '_' (by throwing a new exception type 'InternalAccessException'), unless the access is from a method and its static target is that method's first argument variable. Also, a restricted interpreter refuses access to any module-global variable or module-global function with a name beginning with '_' (by throwing 'InternalAccessException'), unless the access is statically from the same module. (A method's first argument is usually called 'self', but that's just a convention. By "static target", I mean that to access an internal attribute _foo in a method with first argument 'self', you must write "self._foo"; attempting to access "x._foo" will fail even if 'x' happens to be the same object as 'self'. This allows such accesses to be reported at compile-time, rather than only at run-time.) I am using the term "internal" rather than "private" or "protected", because these semantics are not the same as either "private" or "protected" in C++ or Java. In Python with this change, an object can only access its own internal methods and attributes. In C++ and Java, an object can access private and protected members of other objects of the same class. The rationale for this difference is explained below. The use of _single vs __double underscores encodes a useful distinction that would not change. Ignoring the point in the previous paragraph, a _single underscore is similar to "protected" in languages like C++ and Java, while a __double underscore is similar to "private". This is purely a consequence of the name mangling: if a class X and its subclass Y both name an attribute __foo, then we will end up with two attributes _X__foo and _Y__foo in instances of Y, which is the desired behaviour for private attributes. In the case of an attribute called _foo, OTOH, there can be only one such attribute per object, which is the desired behaviour for protected attributes. The name mangling also ensures that an object will not *accidentally* access a private attribute inherited from a superclass. However, in the same example, an instance of Y can still deliberately access the copy of the attribute inherited from X by specifying _X__foo. There is no security problem here, because Y cannot do anything as a result that it could not have done by copying X's code, rather than inheriting from it. Notice that this is only true because we restrict an object to only accessing its own internal attributes and methods; if we followed C++'s semantics where an object can access protected members of any superclass, this would break security. (Java solves this problem by applying a more complicated access rule for protected members, which I considered to be unintuitive. More details on request.) __dict__ is an internal attribute. This means that an object can only directly reflect on itself. I know that there are other means of reflection (e.g. using the 'inspect' module); blocking these or making them safe is a separate issue. If desired, it would be safe to add a 'publicdict' attribute to each object, or a 'publicdict(object)' built-in. This would return a *read-only* dict, probably created lazily if needed, giving access only to public (non-internal) attributes and methods. __init__ is an internal method. This is as it should be, because it should not be possible to call __init__ on an existing object; only to have __init__ implicitly called when a new object is constructed. __repr__ and __str__ are internal under these rules, and probably shouldn't be. Existing classes may expose private state in the strings returned by __repr__ or __str__, but in principle, there is nothing unsafe about being able to convert the public state of an object to a string. OTOH, this functionality is usually accessed via the built-ins 'repr' and 'str', which we could perhaps allow to access '__repr__' and '__str__' as a special case. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From nnorwitz at gmail.com Thu Jul 27 05:43:38 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 26 Jul 2006 20:43:38 -0700 Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery In-Reply-To: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com> References: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com> Message-ID: <ee2a432c0607262043i323ae8d5k8fdc81ca0b7e6b39@mail.gmail.com> On 7/26/06, Tim Peters <tim.peters at gmail.com> wrote: > > Today I noticed this happened when the buildbot started to run tests, > and I'm 100% sure it's due to this code in > Tools/buildbot/kill_python.c (the buildbot log files showed that > kill_python.c killed /some/ Python process, and the Python running > release-build tests in my sandbox was the only plausible candidate): > > if ((strstr(path, "build\\pcbuild\\python_d.exe") != NULL) || > (strstr(path, "build\\python.exe") != NULL)) { > printf("Terminating %s (pid %d)\n", path, pids[i]); > if (!TerminateProcess(hProcess, 1)) { > > The second clause in the first `if` looks for a substring match on: > > build\python.exe > > and that just happens to match a suffix of: > > C:\Code\python\PCbuild\python.exe > > which is the release-build Python I happen to be running in my sandbox. > > Why is the second clause there? That is, are we /trying/ to kill a > release-build Python running from the user's sandbox, and if so why? No, I don't believe that was the intent. The exe on cygwin uses the unix convention, not the Windows convention for the filename. ie, either a debug or release build on cygwin are both called python.exe. So the second clause is there to kill the process when it's running under cygwin. It's interesting that the process appears to be running as ./python.exe, but build shows up in filename. From that I deduce that it must contain the complete path. Assuming that is true, we can change the code to ensure that build is a directory since that's what buildbot does (add the leading \\): > (strstr(path, "\\build\\python.exe") != NULL)) { I tested this change with a different path, so I believe it will work fine and not catch PCbuild. I'll check in this change and add some comments. n From nnorwitz at gmail.com Thu Jul 27 06:08:40 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 26 Jul 2006 21:08:40 -0700 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> Message-ID: <ee2a432c0607262108n2599f9aere07bae83535179b1@mail.gmail.com> What is the behaviour that was added which broke compliance? What is the benefit of the behaviour? >From your description of fixing the problem, it seems there's some risk invovled as it's modiyfing import.c, plus adding new features. What is your recommendation? n -- On 7/26/06, Phillip J. Eby <pje at telecommunity.com> wrote: > I posted last week about a need-for-speed patch that broke PEP 302 > compliance, and asked if it should be fixed or reverted. I got exactly one > response which said "yes, it should be fixed or reverted", which > unfortunately didn't answer my question as to which one we should do. :) > > If we don't revert it, there are two ways to fix it. One is to just change > PEP 302 so that the behavior is unbroken by definition. :) The other is > to actually go ahead and fix it by adding PathImporter and NullImporter > types to import.c, along with a factory function on sys.path_hooks to > create them. (This would've been the PEP-compliant way to implement the > need-for-speed patch.) > > So, "fix" by documentation, fix by fixing, or fix by reverting? Which > should it be? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From martin at v.loewis.de Thu Jul 27 08:32:39 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 27 Jul 2006 08:32:39 +0200 Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery In-Reply-To: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com> References: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com> Message-ID: <44C85E07.3060707@v.loewis.de> Tim Peters wrote: > Today I noticed this happened when the buildbot started to run tests, > and I'm 100% sure it's due to this code in > Tools/buildbot/kill_python.c Didn't you know that you signed in to run arbitrary viruses, worms, and trojan horses when you added your machine to the buildbot infrastructure :-? You just haven't seen buildbot erasing your hard disk and filling your coffee machine with tea, yet. > (strstr(path, "build\\python.exe") != NULL)) { > Why is the second clause there? That's for Cygwin (i.e. Anthony Baxter's machine). As Neal suggests, preceding the executable path with another backslash should solve this problem. As a related note, this entire approach will also manage to kill python.exe from an unrelated buildbot installation, e.g. a 2.4 build job might kill python.exe from the trunk. This actually helped when I tried to get the Cygwin slave to get unstuck, and shouldn't do harm since we currently don't run to builds on the same slave simultaneously, but could be surprising when parallel builds are activated some day. Sorry for messing with your machine, Martin From martin at v.loewis.de Thu Jul 27 08:40:50 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 27 Jul 2006 08:40:50 +0200 Subject: [Python-Dev] Which version of distutils to ship with Python 2.5? In-Reply-To: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com> References: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com> Message-ID: <44C85FF2.3030701@v.loewis.de> Collin Winter wrote: > Is it intentional that Python 2.5 is (currently) shipping with > distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3) > shipped with distutils 2.4.1? Judging from my own tests, distutils > 2.4.1 fixed several bugs that some of my test suites depend on (the > fixes, not the bugs ; ). Are these bugs not fixed in the distutils that shipped with Python 2.5b2? In any case, I bumped the version number to 2.5, according to the policy discussed in http://mail.python.org/pipermail/distutils-sig/2005-January/004368.html Thanks for pointing this out. Regards, Martin From anthony at interlink.com.au Thu Jul 27 09:01:53 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu, 27 Jul 2006 17:01:53 +1000 Subject: [Python-Dev] Which version of distutils to ship with Python 2.5? In-Reply-To: <44C85FF2.3030701@v.loewis.de> References: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com> <44C85FF2.3030701@v.loewis.de> Message-ID: <200607271701.58223.anthony@interlink.com.au> On Thursday 27 July 2006 16:40, Martin v. L?wis wrote: > Collin Winter wrote: > > Is it intentional that Python 2.5 is (currently) shipping with > > distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and > > 2.4.3) shipped with distutils 2.4.1? Judging from my own tests, > > distutils 2.4.1 fixed several bugs that some of my test suites > > depend on (the fixes, not the bugs ; ). > > Are these bugs not fixed in the distutils that shipped with Python > 2.5b2? > > In any case, I bumped the version number to 2.5, according to the > policy discussed in > Could this not simply use the Python version number directly, instead? Separate version numbers only make sense if the package is separately distributed - and even then, something like Barry's setup for the email package could keep that version number out of the Python trunk. Fiddly little version numbers scattered throughout the standard library == pain. Anthony From greg.ewing at canterbury.ac.nz Thu Jul 27 09:14:27 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 27 Jul 2006 19:14:27 +1200 Subject: [Python-Dev] Internal namespace proposal In-Reply-To: <44C822BC.1070204@blueyonder.co.uk> References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org> <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> <44C822BC.1070204@blueyonder.co.uk> Message-ID: <44C867D3.90306@canterbury.ac.nz> David Hopwood wrote: > A restricted interpreter refuses access to any object attribute or method > with a name beginning with '_' (by throwing a new exception type > 'InternalAccessException'), unless the access is from a method and its > static target is that method's first argument variable. What's to stop def my_naughty_method(self): self = some_protected_object self._a_special_attribute = some_naughty_value > __init__ is an internal method. This is as it should be, because it should not > be possible to call __init__ on an existing object; only to have __init__ > implicitly called when a new object is constructed. What about calling an inherited __init__ method? Your proposed rule would seem to disallow BaseClass.__init__(self, ...) -- Greg From syfou at users.sourceforge.net Thu Jul 27 09:39:39 2006 From: syfou at users.sourceforge.net (Sylvain Fourmanoit) Date: Thu, 27 Jul 2006 03:39:39 -0400 (EDT) Subject: [Python-Dev] New miniconf module In-Reply-To: <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> Message-ID: <Pine.LNX.4.64.0607270200300.6000@sylvain> An updated version is now available, based to the feedback of Phillip J. Eby and David Hopwood (stand-alone module[1], patch[2]): - the module is now reentrant - the sloppy case with Name nodes is now covered properly - the node lookup procedure was optimized, leading to a 20% speed increase on the average case... Phillip, I was wrong to doubt you. ;-) There is undoubtedly still room from improvement, but that's a good start. >> But I agree this looks a lot like JSON, since ecmascript syntax for >> literals looks a lot like the one of Python... For the same reasons there >> is a need for JSON, I think having something like miniconf in the >> standard lib would benefit the users. > > Actually, I would see more reason to include JSON in the standard library, > since it's at least something approaching an internet protocol these days. Having JSON there would indeed be nice: In fact, I recall being initially surprised it was not supported by the standard library. But is there a need to choose? Why not have both? The miniconf approach has its advantages and differences: - The code is short and simple. Since all the real work is performed by the Python parser, very little has to be done on top of that: it should be easy to maintain, and will benefit of all the future work (patches, etc.) that will be integrated to it in the future. - The source it works on is valid Python source, which seems to be a plus for a dynamic, reflexive language such as Python... Things such as this will work: >>> from miniconf import dump >>> file('test.py','w').write(dump({'spam': 1})) >>> import test I know this in not the best example, but you get the idea... - Unlike JSON, miniconf is not introducing any new notation or syntax at all: it uses a strict, well defined subset of the Python grammar that every Python user is already familiar with; it is in no way a data-interchange format, but it feels pretty natural in a all-python environment... In that sense, it is well documented and standardized. - Am I missing something, or is JSON not supporting comments inside the parse tree? That's not really convenient for storage of configuration information. Anyway, if I had to choose between the two, I would definitively want simplejson part of the standard library well before miniconf, since it can be used in so many different situations, but I wouldn't choose JSON as a configuration format given the choice to use the Python notation employed by miniconf either. Yours, -- Sylvain <syfou at users.sourceforge.net> Nobody said computers were going to be polite. [1]http://cheeseshop.python.org/pypi?:action=display&name=miniconf&version=1.1.0 [2]http://sourceforge.net/tracker/index.php?func=detail&aid=1527597&group_id=5470&atid=355470 From theller at python.net Thu Jul 27 09:39:59 2006 From: theller at python.net (Thomas Heller) Date: Thu, 27 Jul 2006 09:39:59 +0200 Subject: [Python-Dev] Patch for building ctypes on more OpenBSD target platforms Message-ID: <ea9qke$nnu$1@sea.gmane.org> I've uploaded a patch sent to me in private email by Damien Miller, who is packaging ctypes for the OpenBSD port tree. I'm requesting permission to commit this for Python 2.5. http://python.org/sf/1529514 Thanks, Thomas From g.brandl at gmx.net Thu Jul 27 10:47:30 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Jul 2006 10:47:30 +0200 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <ee2a432c0607262108n2599f9aere07bae83535179b1@mail.gmail.com> References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <ee2a432c0607262108n2599f9aere07bae83535179b1@mail.gmail.com> Message-ID: <ea9ugp$3nu$1@sea.gmane.org> Neal Norwitz wrote: > What is the behaviour that was added which broke compliance? What is > the benefit of the behaviour? sys.path_importer_cache is now used to cache if a real directory exists on the filesystem. Previously, a value of None for a given sys.path entry told find_module that no import hook exist, so it should look for a filesystem directory. Now, the entry is set to True if that directory really exists and to False if it doesn't exist, thus saving quite a few open() calls to files in these not existing dirs. >>From your description of fixing the problem, it seems there's some > risk invovled as it's modiyfing import.c, plus adding new features. > What is your recommendation? I would prefer fixing the docs. Importing from filesystem directories is common enough to be special cased. Georg From arigo at tunes.org Thu Jul 27 12:33:31 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu, 27 Jul 2006 12:33:31 +0200 Subject: [Python-Dev] New miniconf module In-Reply-To: <Pine.LNX.4.64.0607270200300.6000@sylvain> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> <Pine.LNX.4.64.0607270200300.6000@sylvain> Message-ID: <20060727103331.GA31912@code0.codespeak.net> Hi, On Thu, Jul 27, 2006 at 03:39:39AM -0400, Sylvain Fourmanoit wrote: > Having JSON there would indeed be nice: In fact, I recall being initially > surprised it was not supported by the standard library. > > But is there a need to choose? Why not have both? The miniconf approach > has its advantages and differences: I support this point of view: miniconf fills the hole that the stdlib leaves for a safe and cross-version dumper/loader for simple objects using the Python syntax. In the same spirit, maybe it could be slightly re-oriented towards a dumper/loader for more than config files; for example, it could provide a safe inverse of repr() for common built-in types. Such a functionality has been discussed here a few times if I remember correctly, but the code in miniconf is very close to providing it. A bientot, Armin From arigo at tunes.org Thu Jul 27 12:39:20 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu, 27 Jul 2006 12:39:20 +0200 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> Message-ID: <20060727103920.GB31912@code0.codespeak.net> Hi Phillip, On Wed, Jul 26, 2006 at 02:40:27PM -0400, Phillip J. Eby wrote: > If we don't revert it, there are two ways to fix it. One is to just change > PEP 302 so that the behavior is unbroken by definition. :) The other is > to actually go ahead and fix it by adding PathImporter and NullImporter > types to import.c, along with a factory function on sys.path_hooks to > create them. (This would've been the PEP-compliant way to implement the > need-for-speed patch.) > > So, "fix" by documentation, fix by fixing, or fix by reverting? Which > should it be? "fix" by changing the definition looks like a bad idea to me. The import logic is already extremely complicated and delicate, any change to it is bound to break *some* code somewhere. So although import.c is already by far the longest piece of code around, I think that we need a patch doing the "right" thing, or else revert. A bientot, Armin From g.brandl at gmx.net Thu Jul 27 12:52:33 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Jul 2006 12:52:33 +0200 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <20060727103920.GB31912@code0.codespeak.net> References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> Message-ID: <eaa5rd$pe2$1@sea.gmane.org> Armin Rigo wrote: > Hi Phillip, > > On Wed, Jul 26, 2006 at 02:40:27PM -0400, Phillip J. Eby wrote: >> If we don't revert it, there are two ways to fix it. One is to just change >> PEP 302 so that the behavior is unbroken by definition. :) The other is >> to actually go ahead and fix it by adding PathImporter and NullImporter >> types to import.c, along with a factory function on sys.path_hooks to >> create them. (This would've been the PEP-compliant way to implement the >> need-for-speed patch.) >> >> So, "fix" by documentation, fix by fixing, or fix by reverting? Which >> should it be? > > "fix" by changing the definition looks like a bad idea to me. The > import logic is already extremely complicated and delicate, any change > to it is bound to break *some* code somewhere. Though beta1 and beta2 shipped with this change nobody reported any bug that could be linked to it. sys.path_importer_cache is quite an internal thing and most code, even import hooks, shouldn't have to deal with it. Georg From arigo at tunes.org Thu Jul 27 13:30:17 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu, 27 Jul 2006 13:30:17 +0200 Subject: [Python-Dev] Internal namespace proposal In-Reply-To: <44C822BC.1070204@blueyonder.co.uk> References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org> <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> <44C822BC.1070204@blueyonder.co.uk> Message-ID: <20060727113016.GC31912@code0.codespeak.net> Hi David, Your proposal is too vague to be useful. In Python I would not feel that any compiler-enforced restrictions are going to be too restrictive, and so I believe that your approach is not viable, but I cannot give you many concrete examples of why before you come up with a more concrete specification. More importantly, this is going to depend critically on a restricted interpreter in the first place, in the sense of Brett, but the safety of your proposal depends on the restricted interpreter forbidding many operations that it would not otherwise forbid for its original goal. For example, in Brett's use case there is no need to prevent reading the 'func_globals' attribute of function objects, but if that's allowed, then accessing any _attribute of any module is easy. About special methods: how do built-in functions like str(), int(), and so on, know in which context they are called? Surely you don't propose that '2+3' should be invalid because it accesses the hidden attribute '2 .__add__' ? How would you formulate a rule in term on Python's attribute look-up algorithm to prevent the following trivial attack? : x.py: # supposedly secure? _hidden = [1,2,3] class A: def __init__(self): self._authorized = ... def read(self): if not self._authorized: raise Forbidden return _hidden attack.py: import x class B(x.A): def __init__(self): self._authorized = True b = B() print b.read() # => [1,2,3] On any real-life example I'm sure that hacks like overriding selected methods on the instance itself would allow an attacker to confuse the remaining methods enough to leak hidden information. Here is a metaclass attack against the rule "self._attr is only allowed if syntactically inside the class definition of the exact class of self": class SupposedlySecure(object): _hidden = [1,2,3] class MetaAttack(type): def read(self): return self._hidden # seen as an instance attribute class Attack(SupposedlySecure): __metaclass__ = MetaAttack print Attack.read() A bientot, Armin. From amk at amk.ca Thu Jul 27 14:41:37 2006 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 27 Jul 2006 08:41:37 -0400 Subject: [Python-Dev] Release manager: pdb bugfix incompatibility Message-ID: <20060727124137.GA4071@rogue.amk.ca> Bug #1526834: if you do 'b f(' in pdb, the debugger crashes. This bug stems from pdb just sticking the string in a regex and compiling it. cre = re.compile(r'def\s+%s\s*[(]' % funcname) A side effect of this is that 'b f()' works to match the function 'f', because the empty parens are legal regex syntax declaring an empty group that matches the null string. It's easy to fix the crash by doing re.escape(funcname). But this means that 'b f()' will no longer find the function 'f' -- it will look for 'f(' followed by another paren, and won't find it. Should this be fixed by the re.escape(), or should the fix attempt to keep 'b f()' working? The pdb documentation doesn't claim that 'b f()' should work. --amk From fuzzyman at voidspace.org.uk Thu Jul 27 13:46:04 2006 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Thu, 27 Jul 2006 12:46:04 +0100 Subject: [Python-Dev] New miniconf module In-Reply-To: <20060727103331.GA31912@code0.codespeak.net> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> <Pine.LNX.4.64.0607270200300.6000@sylvain> <20060727103331.GA31912@code0.codespeak.net> Message-ID: <44C8A77C.6060407@voidspace.org.uk> Armin Rigo wrote: > Hi, > > On Thu, Jul 27, 2006 at 03:39:39AM -0400, Sylvain Fourmanoit wrote: > >> Having JSON there would indeed be nice: In fact, I recall being initially >> surprised it was not supported by the standard library. >> >> But is there a need to choose? Why not have both? The miniconf approach >> has its advantages and differences: >> > > I support this point of view: miniconf fills the hole that the stdlib > leaves for a safe and cross-version dumper/loader for simple objects > using the Python syntax. In the same spirit, maybe it could be slightly > re-oriented towards a dumper/loader for more than config files; for > example, it could provide a safe inverse of repr() for common built-in > types. Such a functionality has been discussed here a few times if I > remember correctly, but the code in miniconf is very close to providing > it. > ConfigObj [1] gained an 'unrepr' mode a while back. The code is simple, and originally came from CherryPy. It can (safely) unrepr basic datatypes. import compiler def getObj(s): s = "a=" + s p = compiler.parse(s) return p.getChildren()[1].getChildren()[0].getChildren()[1] class UnknownType(Exception): pass class Builder: def build(self, o): m = getattr(self, 'build_' + o.__class__.__name__, None) if m is None: raise UnknownType(o.__class__.__name__) return m(o) def build_List(self, o): return map(self.build, o.getChildren()) def build_Const(self, o): return o.value def build_Dict(self, o): d = {} i = iter(map(self.build, o.getChildren())) for el in i: d[el] = i.next() return d def build_Tuple(self, o): return tuple(self.build_List(o)) def build_Name(self, o): if o.name == 'None': return None if o.name == 'True': return True if o.name == 'False': return False # An undefinted Name raise UnknownType('Undefined Name') def build_Add(self, o): real, imag = map(self.build_Const, o.getChildren()) try: real = float(real) except TypeError: raise UnknownType('Add') if not isinstance(imag, complex) or imag.real != 0.0: raise UnknownType('Add') return real+imag def build_Getattr(self, o): parent = self.build(o.expr) return getattr(parent, o.attrname) def build_UnarySub(self, o): return -self.build_Const(o.getChildren()[0]) def build_UnaryAdd(self, o): return self.build_Const(o.getChildren()[0]) def unrepr(s): if not s: return s return Builder().build(getObj(s)) HTH Michael Foord [1] http://www.voidspace.org.uk/python/configobj.html > > A bientot, > > Armin > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk > > From arigo at tunes.org Thu Jul 27 15:54:17 2006 From: arigo at tunes.org (Armin Rigo) Date: Thu, 27 Jul 2006 15:54:17 +0200 Subject: [Python-Dev] New miniconf module In-Reply-To: <44C8A77C.6060407@voidspace.org.uk> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> <Pine.LNX.4.64.0607270200300.6000@sylvain> <20060727103331.GA31912@code0.codespeak.net> <44C8A77C.6060407@voidspace.org.uk> Message-ID: <20060727135417.GA21563@code0.codespeak.net> Hi Michael, On Thu, Jul 27, 2006 at 12:46:04PM +0100, Michael Foord wrote: > > leaves for a safe and cross-version dumper/loader for simple objects > > using the Python syntax. In the same spirit, maybe it could be slightly > > re-oriented towards a dumper/loader for more than config files; for > > example, it could provide a safe inverse of repr() for common built-in > > types. Such a functionality has been discussed here a few times if I > > remember correctly, but the code in miniconf is very close to providing > > it. > > > ConfigObj [1] gained an 'unrepr' mode a while back. The code is simple, > and originally came from CherryPy. I'm sure, but my point was that the discussed miniconf already contains mostly the same code already, so I suggested that it would be a worthwhile addition to it, in its stdlib-hole-filler role. If it goes in that direction, I'd suggest to rename the module to give it a name closer to existing persistence-related modules already in the stdlib. Armin From g.brandl at gmx.net Thu Jul 27 17:40:57 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Jul 2006 17:40:57 +0200 Subject: [Python-Dev] uuid test suite failing Message-ID: <eaamnu$mrq$1@sea.gmane.org> The UUID test suite, which wasn't run by regrtest.py until now, is now failing on some buildbots (and my machine). This should be fixed before releasing something. Georg From pje at telecommunity.com Thu Jul 27 17:58:15 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 27 Jul 2006 11:58:15 -0400 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <eaa5rd$pe2$1@sea.gmane.org> References: <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> Message-ID: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> At 12:52 PM 7/27/2006 +0200, Georg Brandl wrote: >Armin Rigo wrote: > > Hi Phillip, > > > > On Wed, Jul 26, 2006 at 02:40:27PM -0400, Phillip J. Eby wrote: > >> If we don't revert it, there are two ways to fix it. One is to just > change > >> PEP 302 so that the behavior is unbroken by definition. :) The other is > >> to actually go ahead and fix it by adding PathImporter and NullImporter > >> types to import.c, along with a factory function on sys.path_hooks to > >> create them. (This would've been the PEP-compliant way to implement the > >> need-for-speed patch.) > >> > >> So, "fix" by documentation, fix by fixing, or fix by reverting? Which > >> should it be? > > > > "fix" by changing the definition looks like a bad idea to me. The > > import logic is already extremely complicated and delicate, any change > > to it is bound to break *some* code somewhere. > >Though beta1 and beta2 shipped with this change nobody reported any bug that >could be linked to it. Because in at least setuptools' case, you have to be using unzipped namespace packages under the right set of circumstances to trigger a propblem. >sys.path_importer_cache is quite an internal thing Whose behavior is documented in a PEP. > and >most code, even import hooks, shouldn't have to deal with it. That doesn't make it unimportant. It's a visible change in specified behavior between Python versions -- precisely the sort of thing that makes people mad at us renegade cowboy Python-dev hackers changing their language for no apparent reason. The strftime thing that recently got hashed to death here was also an "internal thing" which "most code shouldn't have to deal with". This is precisely how these kinds of problems happen. So, this needs to either be documented in the What's New document and PEP 302 at a minimum, or it needs to be reverted, unless somebody wants to bless the feature addition to fix it. I'm willing to write code that makes it PEP 302 compliant, if the release manager will bless such an addition. But if that's not acceptable, then somebody needs to produce the necessary documentation updates or revert the patch. It absolutely should not be allowed to remain in *and* undocumented because it is a backwards-incompatible change to documented behavior of Python for two major releases (2.3 and 2.4). From g.brandl at gmx.net Thu Jul 27 18:20:39 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Jul 2006 18:20:39 +0200 Subject: [Python-Dev] uuid test suite failing In-Reply-To: <eaamnu$mrq$1@sea.gmane.org> References: <eaamnu$mrq$1@sea.gmane.org> Message-ID: <eaap2c$al$1@sea.gmane.org> Georg Brandl wrote: > The UUID test suite, which wasn't run by regrtest.py until now, is > now failing on some buildbots (and my machine). This should be fixed > before releasing something. Okay, after fixing the test on my machine (locale issue) it looks like some ifconfigs don't like to be called without arguments. "-a" seems to be supported everywhere though, so I guess it's reasonable to use that flag on every platform. Any objections? Georg From g.brandl at gmx.net Thu Jul 27 18:24:15 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Jul 2006 18:24:15 +0200 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> References: <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <eaa5rd$pe2$1@sea.gmane.org> <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> Message-ID: <eaap95$1bq$1@sea.gmane.org> Phillip J. Eby wrote: >>sys.path_importer_cache is quite an internal thing > > Whose behavior is documented in a PEP. Correct. >> and >>most code, even import hooks, shouldn't have to deal with it. > > That doesn't make it unimportant. It's a visible change in specified > behavior between Python versions -- precisely the sort of thing that makes > people mad at us renegade cowboy Python-dev hackers changing their language > for no apparent reason. The strftime thing that recently got hashed to > death here was also an "internal thing" which "most code shouldn't have to > deal with". > > This is precisely how these kinds of problems happen. > > So, this needs to either be documented in the What's New document and PEP > 302 at a minimum, or it needs to be reverted, unless somebody wants to > bless the feature addition to fix it. I agree with you (now). ;) > I'm willing to write code that makes it PEP 302 compliant, if the release > manager will bless such an addition. But if that's not acceptable, then > somebody needs to produce the necessary documentation updates or revert the > patch. A possible third option would be to store the information "this is an invalid path" somewhere else, that is, an internal dictionary only available to import.c. I will write up docs and update the PEP in any case, if the release manager agrees. Georg From amk at amk.ca Thu Jul 27 18:36:47 2006 From: amk at amk.ca (A.M. Kuchling) Date: Thu, 27 Jul 2006 12:36:47 -0400 Subject: [Python-Dev] uuid test suite failing In-Reply-To: <eaamnu$mrq$1@sea.gmane.org> References: <eaamnu$mrq$1@sea.gmane.org> Message-ID: <20060727163647.GA4175@rogue.amk.ca> On Thu, Jul 27, 2006 at 05:40:57PM +0200, Georg Brandl wrote: > The UUID test suite, which wasn't run by regrtest.py until now, is > now failing on some buildbots (and my machine). This should be fixed > before releasing something. Looking at the failures, there seem to be two problems on Unix variants: 1) on some, '/sbin/ifconfig' prints a help message; you need 'ifconfig -a' to print information about all interfaces. 2) on Solaris 9 (the only version in the SF compile farm), I can't figure out how to make ifconfig print MAC addresses at all. Searching online finds the incantation 'arp <hostname>' to print the MAC. The XP build fails because it seems to be getting different node IDs from different calls. The cygwin build is very unhappy, for reasons that don't look connected to the newly-enabled tests. --amk From david.nospam.hopwood at blueyonder.co.uk Thu Jul 27 17:18:01 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Thu, 27 Jul 2006 16:18:01 +0100 Subject: [Python-Dev] Internal namespace proposal In-Reply-To: <20060727113016.GC31912@code0.codespeak.net> References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org> <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> <44C822BC.1070204@blueyonder.co.uk> <20060727113016.GC31912@code0.codespeak.net> Message-ID: <44C8D929.2010508@blueyonder.co.uk> Armin Rigo wrote: > Hi David, > > Your proposal is too vague to be useful. In Python I would not feel > that any compiler-enforced restrictions are going to be too restrictive, > and so I believe that your approach is not viable, but I cannot give you > many concrete examples of why before you come up with a more concrete > specification. The intention was not to require the restrictions to be compiler-enforced; only to *allow* them to be compiler-enforced. Code like this, for example: def someMethod(self, x): if self == x: foo(x._internal) should not have to work. > More importantly, this is going to depend critically on a restricted > interpreter in the first place, in the sense of Brett, but the safety of > your proposal depends on the restricted interpreter forbidding many > operations that it would not otherwise forbid for its original goal. > For example, in Brett's use case there is no need to prevent reading the > 'func_globals' attribute of function objects, but if that's allowed, > then accessing any _attribute of any module is easy. I disagree that there is no need to prevent reading func_globals. func_globals is clearly incompatible with capability security (as are func_dict and func_closure; also the other function attributes should be read-only). Functions in a capability language should be opaque. I don't see that there is any problem with the proposal depending on a restricted interpreter to prevent access via loopholes such as func_globals, since that is the main intended context of its use. Remember that Brett's document stated that protection could only be obtained at interpreter granularity, rather than object granularity, primarily because objects have no way to prevent access to their private state. My intention in describing the basic idea of enforcing the PEP 8 convention for internal attributes/methods, was to get precisely this kind of feedback on potential problems. I have already obtained useful feedback (including yours), and will prepare a more concrete proposal based on it. > About special methods: how do built-in functions like str(), int(), and > so on, know in which context they are called? Surely you don't propose > that '2+3' should be invalid because it accesses the hidden attribute '2 > .__add__' ? This and other examples have convinced me that names starting and ending with double underscores should not automatically be considered internal. There are a few such names that should be internal (e.g. __dict__), but it is reasonable to treat those as special cases. > How would you formulate a rule in term on Python's attribute look-up > algorithm to prevent the following trivial attack? : > > x.py: > > # supposedly secure? > > _hidden = [1,2,3] > > class A: > def __init__(self): > self._authorized = ... > def read(self): > if not self._authorized: > raise Forbidden > return _hidden > > attack.py: > > import x > class B(x.A): > def __init__(self): > self._authorized = True > > b = B() > print b.read() # => [1,2,3] Inheritance should be defined as though the code of inherited methods and attributes were copied into the subclass (with global accesses updated to point to the original module). IOW, B acts as though it is defined like this: attack.py: class B(x.A): def __init__(self): self._authorized = True def read(self): if not self._authorized: raise Forbidden return x._hidden Since x._hidden is not accessible from attack.py, the attack fails. > On any real-life example I'm sure that hacks like overriding selected > methods on the instance itself would allow an attacker to confuse the > remaining methods enough to leak hidden information. Yes, Java was subject to many attacks of this type. However, a code-copying semantics for inheritance prevents all of them, by ensuring that a class cannot do anything by inheritance that it could not do without it. > Here is a metaclass attack against the rule "self._attr is only allowed > if syntactically inside the class definition of the exact class of > self": > > class SupposedlySecure(object): > _hidden = [1,2,3] > > class MetaAttack(type): > def read(self): > return self._hidden # seen as an instance attribute > > class Attack(SupposedlySecure): > __metaclass__ = MetaAttack > > print Attack.read() Metaclasses are a reflective feature; almost all such features would have to be limited in restricted interpreters. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From david.nospam.hopwood at blueyonder.co.uk Thu Jul 27 17:24:13 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Thu, 27 Jul 2006 16:24:13 +0100 Subject: [Python-Dev] Internal namespace proposal In-Reply-To: <44C8D929.2010508@blueyonder.co.uk> References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org> <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> <44C822BC.1070204@blueyonder.co.uk> <20060727113016.GC31912@code0.codespeak.net> <44C8D929.2010508@blueyonder.co.uk> Message-ID: <44C8DA9D.9060709@blueyonder.co.uk> David Hopwood wrote: > The intention was not to require the restrictions to be compiler-enforced; > only to *allow* them to be compiler-enforced. > > Code like this, for example: > > def someMethod(self, x): > if self == x: "if self is x:", I meant. > foo(x._internal) > > should not have to work. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From bob at redivi.com Thu Jul 27 19:04:35 2006 From: bob at redivi.com (Bob Ippolito) Date: Thu, 27 Jul 2006 10:04:35 -0700 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <eaa5rd$pe2$1@sea.gmane.org> References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <eaa5rd$pe2$1@sea.gmane.org> Message-ID: <1BFAA40C-5B6C-4376-B1C9-D2477EE689B9@redivi.com> On Jul 27, 2006, at 3:52 AM, Georg Brandl wrote: > Armin Rigo wrote: >> Hi Phillip, >> >> On Wed, Jul 26, 2006 at 02:40:27PM -0400, Phillip J. Eby wrote: >>> If we don't revert it, there are two ways to fix it. One is to >>> just change >>> PEP 302 so that the behavior is unbroken by definition. :) The >>> other is >>> to actually go ahead and fix it by adding PathImporter and >>> NullImporter >>> types to import.c, along with a factory function on >>> sys.path_hooks to >>> create them. (This would've been the PEP-compliant way to >>> implement the >>> need-for-speed patch.) >>> >>> So, "fix" by documentation, fix by fixing, or fix by reverting? >>> Which >>> should it be? >> >> "fix" by changing the definition looks like a bad idea to me. The >> import logic is already extremely complicated and delicate, any >> change >> to it is bound to break *some* code somewhere. > > Though beta1 and beta2 shipped with this change nobody reported any > bug that > could be linked to it. sys.path_importer_cache is quite an internal > thing and > most code, even import hooks, shouldn't have to deal with it. Anyone trying to emulate what imp.find_module does in a PEP 302 compliant way will need to introspect sys.path_importer_cache. I have some unreleased code based on the PEP 302 spec that does this and the way it was originally written would have broke in 2.5 if I had tested it there. Just because it's obscure doesn't mean we should go change how things work in a way that's not consistent with the documentation. The documentation should change to match the code or vice versa, though I really don't have any strong feelings one way or the other. -bob From g.brandl at gmx.net Thu Jul 27 19:13:43 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Jul 2006 19:13:43 +0200 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <1BFAA40C-5B6C-4376-B1C9-D2477EE689B9@redivi.com> References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <eaa5rd$pe2$1@sea.gmane.org> <1BFAA40C-5B6C-4376-B1C9-D2477EE689B9@redivi.com> Message-ID: <eaas5s$br4$1@sea.gmane.org> Bob Ippolito wrote: > Just because it's obscure doesn't mean we should go change how things > work in a way that's not consistent with the documentation. The > documentation should change to match the code or vice versa, though I > really don't have any strong feelings one way or the other. I never said it shouldn't be documented if the current code stays. Georg From ronaldoussoren at mac.com Thu Jul 27 19:14:13 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Thu, 27 Jul 2006 19:14:13 +0200 Subject: [Python-Dev] uuid test suite failing In-Reply-To: <eaap2c$al$1@sea.gmane.org> References: <eaamnu$mrq$1@sea.gmane.org> <eaap2c$al$1@sea.gmane.org> Message-ID: <D59E6AA9-C7D1-4439-9E45-12D11B6296ED@mac.com> On Jul 27, 2006, at 6:20 PM, Georg Brandl wrote: > Georg Brandl wrote: >> The UUID test suite, which wasn't run by regrtest.py until now, is >> now failing on some buildbots (and my machine). This should be fixed >> before releasing something. > > Okay, after fixing the test on my machine (locale issue) it looks like > some ifconfigs don't like to be called without arguments. "-a" > seems to > be supported everywhere though, so I guess it's reasonable to use that > flag on every platform. Any objections? IIRC at least some versions of HP-UX do not support the -a flag for ifconfig, I'll check this tomorrow. Ronald From collinw at gmail.com Thu Jul 27 19:51:51 2006 From: collinw at gmail.com (Collin Winter) Date: Thu, 27 Jul 2006 13:51:51 -0400 Subject: [Python-Dev] Which version of distutils to ship with Python 2.5? In-Reply-To: <44C85FF2.3030701@v.loewis.de> References: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com> <44C85FF2.3030701@v.loewis.de> Message-ID: <43aa6ff70607271051h3d87b807w5fad3195949d051e@mail.gmail.com> On 7/27/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: > Collin Winter wrote: > > Is it intentional that Python 2.5 is (currently) shipping with > > distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3) > > shipped with distutils 2.4.1? Judging from my own tests, distutils > > 2.4.1 fixed several bugs that some of my test suites depend on (the > > fixes, not the bugs ; ). > > Are these bugs not fixed in the distutils that shipped with Python 2.5b2? I now believe this to be a new regression that I had confused with an earlier bug report. I've filed a new report, http://python.org/sf/1529871. I'd appreciate it if anyone could shed some light on this. Thanks, Collin Winter From david.nospam.hopwood at blueyonder.co.uk Thu Jul 27 19:02:05 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Thu, 27 Jul 2006 18:02:05 +0100 Subject: [Python-Dev] Internal namespace proposal In-Reply-To: <44C867D3.90306@canterbury.ac.nz> References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org> <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> <44C822BC.1070204@blueyonder.co.uk> <44C867D3.90306@canterbury.ac.nz> Message-ID: <44C8F18D.5020503@blueyonder.co.uk> Greg Ewing wrote: > David Hopwood wrote: > >> A restricted interpreter refuses access to any object attribute or >> method with a name beginning with '_' (by throwing a new exception type >> 'InternalAccessException'), unless the access is from a method and its >> static target is that method's first argument variable. > > What's to stop > > def my_naughty_method(self): > self = some_protected_object > self._a_special_attribute = some_naughty_value That's a good point -- I didn't describe the intended restriction correctly. The reason for not just saying "... and its target is the object that was passed as that method's first argument", was that I want it to be possible to reject programs with internal accesses that cannot be statically recognized as legal. (This does not mean that a particular implementation would have to rely on static detection of non-internal accesses, or on a trusted compiler.) How about this: A restricted interpreter shall refuse access to any object attribute or method with a name beginning with '_', unless the access is from a method, and its target is the object that was passed as that method's first argument. If such an access is detected at run-time, then it shall be reported by throwing a new exception type 'InternalAccessException'. In addition, a program containing an access to an object attribute or method with a name beginning with '_', where the access is not from a method, or the target of the access is not the method's first argument variable, or there is an assignment to that variable in the method, is an illegal program. >> __init__ is an internal method. This is as it should be, because it >> should not be possible to call __init__ on an existing object ... from outside the object, that is ... >> ; only to have __init__ implicitly called when a new object is constructed. > > What about calling an inherited __init__ method? > Your proposed rule would seem to disallow > > BaseClass.__init__(self, ...) No, this call would be allowed because it is to self. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From mattjfleming at googlemail.com Thu Jul 27 23:21:52 2006 From: mattjfleming at googlemail.com (Matt Fleming) Date: Thu, 27 Jul 2006 21:21:52 +0000 Subject: [Python-Dev] uuid test suite failing In-Reply-To: <5ff4a1e50607271022x35175b0coda2148d9d0d1db4b@mail.gmail.com> References: <eaamnu$mrq$1@sea.gmane.org> <eaap2c$al$1@sea.gmane.org> <D59E6AA9-C7D1-4439-9E45-12D11B6296ED@mac.com> <5ff4a1e50607271022x35175b0coda2148d9d0d1db4b@mail.gmail.com> Message-ID: <5ff4a1e50607271421v71cdb8fbt569b9cc6a339ed0f@mail.gmail.com> > On 27/07/06, Ronald Oussoren <ronaldoussoren at mac.com> wrote: > > IIRC at least some versions of HP-UX do not support the -a flag for > > ifconfig, I'll check this tomorrow. > > > > Ronald > > td192> /usr/sbin/ifconfig usage: ifconfig interface [ af [ address [ dest_addr ] ] [ up ] [ down ][ netmask mask ] ] [ metric n ] [ arp | -arp ] [ plumb | unplumb ] td192> /usr/sbin/ifconfig -a ifconfig: no such interface td192> uname -a HP-UX td192 B.11.11 U 9000/800 1839940656 unlimited-user license Also fixed this test on my NetBSD machine by using 'ifconfig -a' and checking for 'address:' in the output. But as Ronald said, not all platforms support the '-a' flag. Not sure if this will fix the OpenBSD buildbot, I don't have access to an OpenBSD machine. Matt -- http://mattssanctuary.blogspot.com From pje at telecommunity.com Thu Jul 27 23:23:00 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 27 Jul 2006 17:23:00 -0400 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <eaa5rd$pe2$1@sea.gmane.org> References: <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> Message-ID: <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com> At 12:52 PM 7/27/2006 +0200, Georg Brandl wrote: >Though beta1 and beta2 shipped with this change nobody reported any bug that >could be linked to it Actually, that's no longer true. See: http://python.org/sf/1529871 This is a user-reported bug against 2.5b2, and I have just confirmed that it is caused by the need-for-speed PEP 302 breakage. Reverting r46372 fixes the reported problem. Of course, since I'm the author of the affected package, I can certainly fix the problem by updating my code. (And in fact, setuptools 0.7a1 works because it uses 2.5's pkgutil when available, and r46372 included a fix for pkgutil.) At this point, I'm going to wait another day for somebody to step up to fix the documentation, or the release manager to authorize inclusion of a proper fix (which is unlikely to be allowed, since it will mean adding two new types to import.c, and restructuring a bit of pkgutil to subclass from them). (It's a nice performance improvement you've got here, I would hate to see anything happen to it...) Personally, I would prefer to see it properly fixed in 2.5 rather than having to rip it out. It's more work for me to create the proper fix than it is to just work around it in my code, but it seems a more righteous labor, if you know what I mean. It also means that already-shipped and distributed versions of my code would work with the 2.5 release. From pje at telecommunity.com Thu Jul 27 23:28:23 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 27 Jul 2006 17:28:23 -0400 Subject: [Python-Dev] Which version of distutils to ship with Python 2.5? In-Reply-To: <43aa6ff70607271051h3d87b807w5fad3195949d051e@mail.gmail.co m> References: <44C85FF2.3030701@v.loewis.de> <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com> <44C85FF2.3030701@v.loewis.de> Message-ID: <5.1.1.6.0.20060727172329.028f1178@sparrow.telecommunity.com> At 01:51 PM 7/27/2006 -0400, Collin Winter wrote: >On 7/27/06, "Martin v. L?wis" <martin at v.loewis.de> wrote: > > Collin Winter wrote: > > > Is it intentional that Python 2.5 is (currently) shipping with > > > distutils 2.4.0, while Python 2.4 (at least 2.4.1, 2.4.2 and 2.4.3) > > > shipped with distutils 2.4.1? Judging from my own tests, distutils > > > 2.4.1 fixed several bugs that some of my test suites depend on (the > > > fixes, not the bugs ; ). > > > > Are these bugs not fixed in the distutils that shipped with Python 2.5b2? > >I now believe this to be a new regression that I had confused with an >earlier bug report. I've filed a new report, >http://python.org/sf/1529871. I'd appreciate it if anyone could shed >some light on this. As I noted on the bug itself, the problem is due to r46372, a patch introduced by the need-for-speed sprint and which broke CPython's compliance with PEP 302, by introducing non-None, non-importer values into sys.path_importer_cache. At the present time, I have not received an unequivocal response regarding how the problem should be corrected: 1. Update PEP 302 and the "What's New" documentation to reflect this backwards-incompatible change and require authors to update their code to work with Python 2.5 2. Fix the patch so it implements the speedup in a way that is conformant with PEP 302 as it currently stands (I have volunteered to do this, if the release manager(s) will authorize it) 3. Revert the patch. No one has volunteered for any of the work required for #1. I belive that #3 must be done if #2 is not approved and no volunteers surface for #1 in a timely manner. From barry at python.org Fri Jul 28 01:54:57 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 27 Jul 2006 19:54:57 -0400 Subject: [Python-Dev] Support for PyGetSetDefs in pydoc In-Reply-To: <e9plv5$rfe$1@sea.gmane.org> References: <A027CA35-EE77-4658-BA59-E849368C440B@python.org><EF6E6AF2-654F-4F53-91A8-A6C9B8A80745@python.org><44B64782.9010706@gmail.com><7FE62713-24B0-47CE-BA94-41150DB683F4@python.org> <AB1C2209-329E-4F3D-8B7A-2548A40F892A@python.org> <e9plv5$rfe$1@sea.gmane.org> Message-ID: <E666C3A3-9C45-41C7-B25A-C9CB244485B5@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Since Anthony didn't speak up, I took his silence as assent and went ahead and committed the changes. r50881 and r50885 for *nix and Windows, just in case the deafening silence turns into a howl of derision :). - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRMlSV3EjvBPtnXfVAQL3WQP9H2RBIDG3FCEkzHjzmwyRWl4HU467yWMQ bse0/XhUEAQHivwP2nLvAqn+Qrb8XaXIT3n5i9++saMFtxjTdfMJX2ZNBK+0JmVl N+XvhTIXIu9XJy47c4FsZ6tbfHVSKQ3KRaE81sfMYuKQsPCnB9cNskKEJEpaS0Cy F7GmpdE96sM= =T3Ia -----END PGP SIGNATURE----- From tim.peters at gmail.com Fri Jul 28 02:46:30 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 27 Jul 2006 20:46:30 -0400 Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery In-Reply-To: <44C85E07.3060707@v.loewis.de> References: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com> <44C85E07.3060707@v.loewis.de> Message-ID: <1f7befae0607271746y7bca18f2s439e8b5f996f132d@mail.gmail.com> [Martin v. L?wis] > Didn't you know that you signed in to run arbitrary viruses, worms, and > trojan horses when you added your machine to the buildbot infrastructure > :-? Hey, I signed up for that when I bought a Windows box :-) > You just haven't seen buildbot erasing your hard disk and filling > your coffee machine with tea, yet. Not the buildbot, no, but visiting random web pages does that routinely. >> (strstr(path, "build\\python.exe") != NULL)) { >> Why is the second clause there? > That's for Cygwin (i.e. Anthony Baxter's machine). As Neal suggests, > preceding the executable path with another backslash should solve > this problem. And he checked that in, and I haven't noticed another similar problem since (but then it was rare to begin with). > As a related note, this entire approach will also manage to kill > python.exe from an unrelated buildbot installation, e.g. a 2.4 > build job might kill python.exe from the trunk. This actually helped > when I tried to get the Cygwin slave to get unstuck, and shouldn't > do harm since we currently don't run to builds on the same slave > simultaneously, but could be surprising when parallel builds > are activated some day. I don't think we /can/ yet -- I believe some tests T exist that implicitly assume only one instance of T is running. I don't recall details, although I'm sure we'll bump into them sporadically the instant parallel builds are enabled. As a purely pragmatic matter, I expect my hard drive would quickly be reduced to dust if two instances of test_largefile ran simultaneously (Windows XP writes physical zeroes in the entire multi-GB file, and that takes finite time only if the disk head doesn't have to keep seeking). > Sorry for messing with your machine, No problem! That's what it's here for :-) From greg.ewing at canterbury.ac.nz Fri Jul 28 02:51:43 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 28 Jul 2006 12:51:43 +1200 Subject: [Python-Dev] Internal namespace proposal In-Reply-To: <44C8D929.2010508@blueyonder.co.uk> References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org> <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> <44C822BC.1070204@blueyonder.co.uk> <20060727113016.GC31912@code0.codespeak.net> <44C8D929.2010508@blueyonder.co.uk> Message-ID: <44C95F9F.2090307@canterbury.ac.nz> David Hopwood wrote: > Inheritance should be defined as though the code of inherited methods and > attributes were copied into the subclass (with global accesses updated to > point to the original module). You'll have to propose an implementation strategy for that which works without actually copying all the code, though. > Since x._hidden is not accessible from attack.py, the attack fails. But if _hidden were an attribute of the A instance that you were trying to protect, it would succeed. So you can't actually protect any direct attribute of a class that can be subclassed. Which means we're back to the situation of having to prevent access to class objects. -- Greg From david.nospam.hopwood at blueyonder.co.uk Fri Jul 28 03:40:53 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Fri, 28 Jul 2006 02:40:53 +0100 Subject: [Python-Dev] Internal namespace proposal In-Reply-To: <44C95F9F.2090307@canterbury.ac.nz> References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org> <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> <44C822BC.1070204@blueyonder.co.uk> <20060727113016.GC31912@code0.codespeak.net> <44C8D929.2010508@blueyonder.co.uk> <44C95F9F.2090307@canterbury.ac.nz> Message-ID: <44C96B25.5000709@blueyonder.co.uk> Greg Ewing wrote: > David Hopwood wrote: > >> Inheritance should be defined as though the code of inherited methods and >> attributes were copied into the subclass (with global accesses updated to >> point to the original module). > > You'll have to propose an implementation strategy for that > which works without actually copying all the code, though. The only difference between the copying semantics and the current semantics, is in the visibility of module-global internal variables and functions. It's sufficient to keep track of whether each class could access a variable or function that is internal to its module. If it does, then it cannot be subclassed from a different module. (It must not be possible to access internal variables/ functions reflectively.) The effect of this is that if a programmer intends a class to be subclassable from outside the module, they must make sure that all of the variables/functions it depends on are public. Anyone performing a security review of the module then does not have to consider inheritance from a different module when deciding which variables/functions might be accessible. There is a slightly more flexible version of this approach that is just as secure, provided that functions are restricted to be stateless. If it is possible to prove statically that a class does not access any internal *variables* of a module (regardless of whether it accesses internal functions), then it is safe to allow the class to be subclassed from another module. This is because the subclassing module could have copied all of the code of the original module (assuming it is written purely in Python); the only possible sources of authority in a capability language are from access to variables or primitives, not code. If a class is not written in Python, then we cannot analyse whether it accesses internal variables. In that case the class will be part of the TCB, and we have to trust the class writer to mark whether it can be safely subclassed from another module. >> Since x._hidden is not accessible from attack.py, the attack fails. > > But if _hidden were an attribute of the A instance that > you were trying to protect, it would succeed. No, attack.py could only access a _hidden attribute in an instance of B. This is harmless, because it could just as well define the _hidden attribute of B itself, rather than by subclassing. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From david.nospam.hopwood at blueyonder.co.uk Fri Jul 28 03:55:58 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Fri, 28 Jul 2006 02:55:58 +0100 Subject: [Python-Dev] Internal namespace proposal In-Reply-To: <4BE3C1EC-8DE0-4518-A8B6-68F98477433B@commonground.com.au> References: <Pine.LNX.4.58.0607191811400.31087@server1.LFW.org> <bbaeab100607191650j4584de2cge554274e100f125e@mail.gmail.com> <44C822BC.1070204@blueyonder.co.uk> <4BE3C1EC-8DE0-4518-A8B6-68F98477433B@commonground.com.au> Message-ID: <44C96EAE.6040907@blueyonder.co.uk> Richard Jones wrote: > On 27/07/2006, at 12:19 PM, David Hopwood wrote: > >> A restricted interpreter refuses access to any object attribute or >> method with a name beginning with '_' (by throwing a new exception type >> 'InternalAccessException'), unless the access is from a method and its >> static target is that method's first argument variable. >> >> Also, a restricted interpreter refuses access to any module-global >> variable or module-global function with a name beginning with '_' (by >> throwing 'InternalAccessException'), unless the access is statically >> from the same module. > > Note that this is a rule that Zope enforces in its restricted environment. Is that documented anywhere? -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From tim.peters at gmail.com Fri Jul 28 05:21:15 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 27 Jul 2006 23:21:15 -0400 Subject: [Python-Dev] how about adding ping's uuid module to the standard lib ? In-Reply-To: <dui1u8$ge1$1@sea.gmane.org> References: <dui1u8$ge1$1@sea.gmane.org> Message-ID: <1f7befae0607272021k17b65a88g356c781c34967cc7@mail.gmail.com> Georg discovered that test_uuid didn't run any tests, and fixed that on Thursday. A number of buildbots have failed that test since then. My XP box appears unique among the Windows buildbots in failing. It always fails like so: AssertionError: different sources disagree on node: from source 'getnode1', node was 00038a000015 from source 'getnode2', node was 00038a000015 from source 'ipconfig', node was 001111b2b7bf 0x00038a000015 /is/ the last 6 bytes returned by the Windows UuidCreateSequential() on my box. I confirmed that by writing a C program calling it directly. However, it doesn't appear to correspond to any MAC address of any HW on my box. All documented ways of determining the MAC address of my Ethernet card (using UuidCreateSequential for this appears to be folklore rather than documented behavior) agree that 0x001111b2b7bf is correct on this box; e.g., $ getmac /fo list /v Connection Name: Local Area Connection Network Adapter: Marvell Yukon 88E8050 PCI-E ASF Gigabit Ethernet Controller Physical Address: 00-11-11-B2-B7-BF Transport Name: \Device\Tcpip_... Connection Name: 1394 Connection Network Adapter: 1394 Net Adapter Physical Address: 62-A1-AC-6C-FD-BE Transport Name: \Device\Tcpip_... Connection Name: 1394 Connection 2 Network Adapter: 1394 Net Adapter Physical Address: E2-1F-01-C6-5D-88 Transport Name: \Device\Tcpip_... The last two are for firewire interfaces, and don't match the purported MAC address extracted from UuidCreateSequential's output anyway. So, at least on my box, this comment in uuid.py is incorrect (UuidCreateSequential does not behave as it says): # On Windows prior to 2000, UuidCreate gives a UUID containing the # hardware address. On Windows 2000 and later, UuidCreate makes a # random UUID and UuidCreateSequential gives a UUID containing the # hardware address. ... Unfortunately, uuid.getnode() tries things in this order on Windows: getters = [_windll_getnode, _netbios_getnode, _ipconfig_getnode] It's only the first one that returns the bogus 0x00038a000015; both of the latter return 0x001111B2B7BF. However, there's nothing I can do to that list to make test_uuid pass on this box. It wants to insist that all three ways of getting/guessing the MAC address return the same thing, and that's never going to happen here. Given that _windll_getnode's actual behavior appears to have nothing in common with what was expected for it here, best suggestion I can make is to throw its code away. From tim.peters at gmail.com Fri Jul 28 06:27:51 2006 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 28 Jul 2006 00:27:51 -0400 Subject: [Python-Dev] how about adding ping's uuid module to the standard lib ? In-Reply-To: <1f7befae0607272021k17b65a88g356c781c34967cc7@mail.gmail.com> References: <dui1u8$ge1$1@sea.gmane.org> <1f7befae0607272021k17b65a88g356c781c34967cc7@mail.gmail.com> Message-ID: <1f7befae0607272127u544fac8eq9608e95938ca2c55@mail.gmail.com> [Tim] > ... uuid.getnode() tries things in this order on Windows: > > getters = [_windll_getnode, _netbios_getnode, _ipconfig_getnode] > > It's only the first one that returns the bogus 0x00038a000015; both of > the latter return 0x001111B2B7BF [the correct MAC address for my > network card]. That was on my desktop XP Pro SP2 box. On my similar laptop box, it's quite different: _windll_getnode and _ipconfig_getnode return the MAC address of my wireless Ethernet adapter, while _netbios_getnode returns the MAC address of my LAN Ethernet card. > However, there's nothing I can do to that list to make test_uuid pass > on this box. It wants to insist that all three ways of > getting/guessing the MAC address return the same thing, and that's > never going to happen here. Or on my laptop, but for different reasons there. > Given that _windll_getnode's actual behavior appears to have nothing > in common with what was expected for it here, best suggestion I can > make is to throw its code away. Which wouldn't improve things on my laptop. Best next ;-) suggestion is to change test_uuid to stop believing that uuid.py knows multiple ways to find a well-defined MAC address. I'm going to make that change -- someone who hates that can revert it after they buy me two new computers that work they way they think computers should work ;-) From nnorwitz at gmail.com Fri Jul 28 06:49:48 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 27 Jul 2006 21:49:48 -0700 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com> References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <eaa5rd$pe2$1@sea.gmane.org> <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com> Message-ID: <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com> On 7/27/06, Phillip J. Eby <pje at telecommunity.com> wrote: > > Personally, I would prefer to see it properly fixed in 2.5 rather than > having to rip it out. It's more work for me to create the proper fix than > it is to just work around it in my code, but it seems a more righteous > labor, if you know what I mean. It also means that already-shipped and > distributed versions of my code would work with the 2.5 release. Based on this comment, is it really acceptable to just document a behaviour change? ISTM there should really only be 2 choices: fix 2.5 properly or revert the change. This seemed to be Armin's position. n From pje at telecommunity.com Fri Jul 28 06:56:41 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 28 Jul 2006 00:56:41 -0400 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com> References: <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com> <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <eaa5rd$pe2$1@sea.gmane.org> <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060728005338.02a25860@sparrow.telecommunity.com> At 09:49 PM 7/27/2006 -0700, Neal Norwitz wrote: >On 7/27/06, Phillip J. Eby <pje at telecommunity.com> wrote: >> >>Personally, I would prefer to see it properly fixed in 2.5 rather than >>having to rip it out. It's more work for me to create the proper fix than >>it is to just work around it in my code, but it seems a more righteous >>labor, if you know what I mean. It also means that already-shipped and >>distributed versions of my code would work with the 2.5 release. > >Based on this comment, is it really acceptable to just document a >behaviour change? ISTM there should really only be 2 choices: fix >2.5 properly or revert the change. This seemed to be Armin's >position. Well, it's a moot question since nobody has volunteered to update the docs. Fixing it and reverting it are the only options unless somebody steps up to do the doc work. I'll happily fix it or revert it, just tell me which one is the approved course of action and I'll get started. :) From rhettinger at ewtllc.com Fri Jul 28 07:32:52 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Thu, 27 Jul 2006 22:32:52 -0700 Subject: [Python-Dev] Py2.5 release schedule Message-ID: <44C9A184.7030701@ewtllc.com> I suggest that there be a third beta release and that we then wait just a bit before going final. The bugs that were found and fixed in the first two beta releases suggest that Py2.5 is not yet as stable as we would like. Over the next few days, I'll try to run it on as much third-party code as possible. That would have detected the recently surfaced grammar error a little bit earlier (the one where "for x, in listOfTuples" would not unpack). The release process itself is going well but I don't think the pervasive AST changes have been fully shaken-out yet. Raymond From martin at v.loewis.de Fri Jul 28 07:33:37 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 28 Jul 2006 07:33:37 +0200 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> References: <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> Message-ID: <44C9A1B1.3040102@v.loewis.de> Phillip J. Eby wrote: > I'm willing to write code that makes it PEP 302 compliant, if the release > manager will bless such an addition. But if that's not acceptable, then > somebody needs to produce the necessary documentation updates or revert the > patch. It absolutely should not be allowed to remain in *and* undocumented > because it is a backwards-incompatible change to documented behavior of > Python for two major releases (2.3 and 2.4). You don't need a release manager pronouncement for that. It's a bug, changing it is a bug fix, you don't need RM permission to fix a bug. Do you have a patch ready that restores path_importer_cache behavior, yet preserves the property that it caches existence of a directory? If not, I will have to produce one. Regards, Martin From anthony at interlink.com.au Fri Jul 28 07:39:48 2006 From: anthony at interlink.com.au (Anthony Baxter) Date: Fri, 28 Jul 2006 15:39:48 +1000 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <44C9A184.7030701@ewtllc.com> References: <44C9A184.7030701@ewtllc.com> Message-ID: <200607281539.50794.anthony@interlink.com.au> On Friday 28 July 2006 15:32, Raymond Hettinger wrote: > I suggest that there be a third beta release and that we then wait > just a bit before going final. > > The bugs that were found and fixed in the first two beta releases > suggest that Py2.5 is not yet as stable as we would like. Over the > next few days, I'll try to run it on as much third-party code as > possible. That would have detected the recently surfaced grammar > error a little bit earlier (the one where "for x, in listOfTuples" > would not unpack). > > The release process itself is going well but I don't think the > pervasive AST changes have been fully shaken-out yet. I've been thinking the same thing, too. A quick chat to Neal says that he also agrees. There's still a lot more bugs popping up than I'm really comfortable with. I guess this is inevitable - there's a lot of new stuff in 2.5. Does anyone disagree with making the next release beta3? Anthony -- Anthony Baxter <anthony at interlink.com.au> It's never too late to have a happy childhood. From greg at electricrain.com Fri Jul 28 07:54:39 2006 From: greg at electricrain.com (Gregory P. Smith) Date: Thu, 27 Jul 2006 22:54:39 -0700 Subject: [Python-Dev] [Windows, buildbot] kill_python.c mystery In-Reply-To: <44C85E07.3060707@v.loewis.de> References: <1f7befae0607261702o6d943f52x701be0d402345f65@mail.gmail.com> <44C85E07.3060707@v.loewis.de> Message-ID: <20060728055439.GD22921@zot.electricrain.com> > Didn't you know that you signed in to run arbitrary viruses, worms, and > trojan horses when you added your machine to the buildbot infrastructure > :-? You just haven't seen buildbot erasing your hard disk and filling > your coffee machine with tea, yet. VMware Server is free. Run buildbots in a VM. (but don't assume a VM protects you from trojans that are designed to break out of it) -g From nnorwitz at gmail.com Fri Jul 28 09:24:52 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 28 Jul 2006 00:24:52 -0700 Subject: [Python-Dev] uuid test suite failing In-Reply-To: <20060727163647.GA4175@rogue.amk.ca> References: <eaamnu$mrq$1@sea.gmane.org> <20060727163647.GA4175@rogue.amk.ca> Message-ID: <ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com> On 7/27/06, A.M. Kuchling <amk at amk.ca> wrote: > On Thu, Jul 27, 2006 at 05:40:57PM +0200, Georg Brandl wrote: > > The UUID test suite, which wasn't run by regrtest.py until now, is > > now failing on some buildbots (and my machine). This should be fixed > > before releasing something. > > Looking at the failures, there seem to be two problems on Unix variants: > 1) on some, '/sbin/ifconfig' prints a help message; you need 'ifconfig -a' > to print information about all interfaces. > 2) on Solaris 9 (the only version in the SF compile farm), I can't > figure out how to make ifconfig print MAC addresses at all. > Searching online finds the incantation 'arp <hostname>' to print the > MAC. This is such a mess. There are so many different ways of determining the MAC addr on each flavour of Unix it seems hopeless to try. I fixed _ifconfig_getnode so it should work on at least: Linux, Tru64, Solaris, and HP-UX. Who knows how many more variations there are. This only fixes 1 of the 2 failures in test_uuid. The other one is due to _unixdll_getnode() failing. This is because _uuid_generate_time is None because we couldn't find it in the uuid library. This is just broken, not sure if it's the code or the test though. We should handle the case if _uuid_generate_time and the others are None better. I don't know what to do in this case. Since getnode ignores exceptions, maybe it's the test that is broken? n From mwh at python.net Fri Jul 28 11:53:47 2006 From: mwh at python.net (Michael Hudson) Date: Fri, 28 Jul 2006 10:53:47 +0100 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <200607281539.50794.anthony@interlink.com.au> (Anthony Baxter's message of "Fri, 28 Jul 2006 15:39:48 +1000") References: <44C9A184.7030701@ewtllc.com> <200607281539.50794.anthony@interlink.com.au> Message-ID: <2mirlixdg4.fsf@starship.python.net> Anthony Baxter <anthony at interlink.com.au> writes: > On Friday 28 July 2006 15:32, Raymond Hettinger wrote: >> I suggest that there be a third beta release and that we then wait >> just a bit before going final. >> >> The bugs that were found and fixed in the first two beta releases >> suggest that Py2.5 is not yet as stable as we would like. Over the >> next few days, I'll try to run it on as much third-party code as >> possible. That would have detected the recently surfaced grammar >> error a little bit earlier (the one where "for x, in listOfTuples" >> would not unpack). >> >> The release process itself is going well but I don't think the >> pervasive AST changes have been fully shaken-out yet. > > I've been thinking the same thing, too. A quick chat to Neal says that > he also agrees. > > There's still a lot more bugs popping up than I'm really comfortable > with. I guess this is inevitable - there's a lot of new stuff in 2.5. > > Does anyone disagree with making the next release beta3? It seems like a good idea to me. I guess this will mean the final release will be pushed back a bit? Cheers, mwh -- Gevalia is undrinkable low-octane see-through only slightly roasted bilge water. Compared to .us coffee it is quite drinkable. -- M?ns Nilsson, asr From arigo at tunes.org Fri Jul 28 12:11:33 2006 From: arigo at tunes.org (Armin Rigo) Date: Fri, 28 Jul 2006 12:11:33 +0200 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat Message-ID: <20060728101133.GA339@code0.codespeak.net> Hi, There is an oversight in the design of __index__() that only just surfaced :-( It is responsible for the following behavior, on a 32-bit machine with >= 2GB of RAM: >>> s = 'x' * (2**100) # works! >>> len(s) 2147483647 This is because PySequence_Repeat(v, w) works by applying w.__index__ in order to call v->sq_repeat. However, __index__ is defined to clip the result to fit in a Py_ssize_t. This means that the above problem exists with all sequences, not just strings, given enough RAM to create such sequences with 2147483647 items. For reference, in 2.4 we correctly get an OverflowError. Argh! What should be done about it? A bientot, Armin. From barry at python.org Fri Jul 28 14:57:06 2006 From: barry at python.org (Barry Warsaw) Date: Fri, 28 Jul 2006 08:57:06 -0400 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <200607281539.50794.anthony@interlink.com.au> References: <44C9A184.7030701@ewtllc.com> <200607281539.50794.anthony@interlink.com.au> Message-ID: <BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 28, 2006, at 1:39 AM, Anthony Baxter wrote: > I've been thinking the same thing, too. A quick chat to Neal says that > he also agrees. > > There's still a lot more bugs popping up than I'm really comfortable > with. I guess this is inevitable - there's a lot of new stuff in 2.5. > > Does anyone disagree with making the next release beta3? +1. It would give me more type to port and test a few of my applications to the new version. FWIW, our commercial app went pretty smoothly, mostly dealing with Py_ssize_t conversions and adopting the new PySet C API. I'm still working on Mailman but the most painful thing so far has been the conversion of exceptions to new-style classes, and even that wasn't / too/ painful. I've only done limited testing of both, but I'm encouraged that the porting effort will be minor (probably no more than a day's worth of work for both apps combined). - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRMoJonEjvBPtnXfVAQJOjAP9HRbakdb39IOtFxGX/wP4QhiXAcNAbXXM hGFIJ6vC0Gp/SSlTVMYPF5oJMIzuDCIDzs4Nrbgysgfj6Ehyphei/ed8W94PHLat nh54Y0N5pvwLelHW6ChJBcIxulU8Fuj0Z9kIZCLiryTOAyXTh+t3+gZPEzWRY/tY v2hd9ERXDl8= =vK7s -----END PGP SIGNATURE----- From tomerfiliba at gmail.com Fri Jul 28 15:35:27 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Fri, 28 Jul 2006 15:35:27 +0200 Subject: [Python-Dev] patching pydoc? Message-ID: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com> i have a problem with pydoc in rpyc. i wanted help(obj), where obj is a NetProxy object, to work as if it were local. i followed the code starting from site.help to pydoc.doc, which is the ultimate function that generates and prints the text. i expected there would be some function in the middle that prepares the text, and another that writes it to the pager, but to my disappointment pydoc.doc does both. this means i can't transfer the document to my local machine (it's printed directly to the remote console). therefore, i would like to split this behavior into two parts: * render_doc - a function that returns the document text * doc - a function that calls render_doc and sends it to the pager this way no existing code breaks (no existing function signatures are changed) and i gain help on remote objects. i hope people would be in favor, as it's not such a big change anyway. is it possible to add to 2.5? -tomer this is the code of pydoc, starting at line 1457 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< def doc(thing, title='Python Library Documentation: %s', forceload=0): """Display text documentation, given an object or a path to an object.""" try: object, name = resolve(thing, forceload) desc = describe(object) module = inspect.getmodule(object) if name and '.' in name: desc += ' in ' + name[:name.rfind('.')] elif module and module is not object: desc += ' in module ' + module.__name__ if not (inspect.ismodule(object) or inspect.isclass(object) or inspect.isroutine(object) or isinstance(object, property)): # If the passed object is a piece of data or an instance, # document its available methods instead of its value. object = type(object) desc += ' object' pager(title % desc + '\n\n' + text.document(object, name)) except (ImportError, ErrorDuringImport), value: print value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is the suggested code <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< def render_doc(thing, title='Python Library Documentation: %s', forceload=0): """generate the text""" object, name = resolve(thing, forceload) desc = describe(object) module = inspect.getmodule(object) if name and '.' in name: desc += ' in ' + name[:name.rfind('.')] elif module and module is not object: desc += ' in module ' + module.__name__ if not (inspect.ismodule(object) or inspect.isclass(object) or inspect.isroutine(object) or isinstance(object, property)): # If the passed object is a piece of data or an instance, # document its available methods instead of its value. object = type(object) desc += ' object' return title % desc + '\n\n' + text.document(object, name) def doc(*args, **kwargs): """Display text documentation, given an object or a path to an object.""" try: text = render_doc(*args, **kwargs) pager(text) except (ImportError, ErrorDuringImport), value: print value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From skip at pobox.com Fri Jul 28 16:02:10 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 28 Jul 2006 09:02:10 -0500 Subject: [Python-Dev] Another uuid problem Message-ID: <17610.6370.422365.622901@montanaro.dyndns.org> I just tried building and testing 2.5 on a Solaris 10 box at work. The uuid test failed with sh: ifconfig: not found sh: ifconfig: not found sh: ifconfig: not found sh: arp: not found sh: ifconfig: not found sh: ifconfig: not found sh: ifconfig: not found sh: arp: not found In our environment at least it's uncommon for /usr/sbin to be in the PATH of non-privileged users: piggy:% type -a ifconfig -bash: type: ifconfig: not found piggy:% PATH=$PATH:/usr/sbin type -a ifconfig ifconfig is /usr/sbin/ifconfig Perhaps test_uuid needs to do a little investigation to find ifconfig and arp. Skip From david.nospam.hopwood at blueyonder.co.uk Fri Jul 28 14:50:51 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Fri, 28 Jul 2006 13:50:51 +0100 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <20060728101133.GA339@code0.codespeak.net> References: <20060728101133.GA339@code0.codespeak.net> Message-ID: <44CA082B.6030908@blueyonder.co.uk> Armin Rigo wrote: > Hi, > > There is an oversight in the design of __index__() that only just > surfaced :-( It is responsible for the following behavior, on a 32-bit > machine with >= 2GB of RAM: > > >>> s = 'x' * (2**100) # works! > >>> len(s) > 2147483647 > > This is because PySequence_Repeat(v, w) works by applying w.__index__ in > order to call v->sq_repeat. However, __index__ is defined to clip the > result to fit in a Py_ssize_t. Clipping the result sounds like it would *never* be a good idea. What was the rationale for that? It should throw an exception. -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From grig.gheorghiu at gmail.com Fri Jul 28 16:46:55 2006 From: grig.gheorghiu at gmail.com (Grig Gheorghiu) Date: Fri, 28 Jul 2006 07:46:55 -0700 Subject: [Python-Dev] uuid test suite failing In-Reply-To: <ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com> References: <eaamnu$mrq$1@sea.gmane.org> <20060727163647.GA4175@rogue.amk.ca> <ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com> Message-ID: <3f09d5a00607280746t57fc51b6m95cc8e1d1c927035@mail.gmail.com> On 7/28/06, Neal Norwitz <nnorwitz at gmail.com> wrote: > > On 7/27/06, A.M. Kuchling <amk at amk.ca> wrote: > > On Thu, Jul 27, 2006 at 05:40:57PM +0200, Georg Brandl wrote: > > > The UUID test suite, which wasn't run by regrtest.py until now, is > > > now failing on some buildbots (and my machine). This should be fixed > > > before releasing something. > > > > Looking at the failures, there seem to be two problems on Unix variants: > > 1) on some, '/sbin/ifconfig' prints a help message; you need 'ifconfig > -a' > > to print information about all interfaces. > > 2) on Solaris 9 (the only version in the SF compile farm), I can't > > figure out how to make ifconfig print MAC addresses at all. > > Searching online finds the incantation 'arp <hostname>' to print the > > MAC. > > This is such a mess. There are so many different ways of determining > the MAC addr on each flavour of Unix it seems hopeless to try. I > fixed _ifconfig_getnode so it should work on at least: Linux, Tru64, > Solaris, and HP-UX. Who knows how many more variations there are. > > This only fixes 1 of the 2 failures in test_uuid. The other one is > due to _unixdll_getnode() failing. This is because > _uuid_generate_time is None because we couldn't find it in the uuid > library. This is just broken, not sure if it's the code or the test > though. We should handle the case if _uuid_generate_time and the > others are None better. I don't know what to do in this case. > > Since getnode ignores exceptions, maybe it's the test that is broken? My 2 cents: since there is no POSIX standard for getting a list of network interfaces, trying to account for all the platform variations is one central location is hopeless. Instead, I think the onus should be on whomever is testing this on a particular platform -- in short, on the buildbot maintainer on that platform. There could be another regrtest.py-type suite, something like platform_regrtest.py for example, which could be composed of highly platform-dependent tests such as test_uuid.py. These tests would have empty methods such as _ifconfig_getnode, which would then be defined on a per-platform basis by the buildbot maintainer on that platform. The test would obviously fail by default, unless those methods are properly defined. Or these methods could account for just one platform, as an example of what to do on other platforms. Grig -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060728/aaf51821/attachment.html From mwh at python.net Fri Jul 28 17:26:37 2006 From: mwh at python.net (Michael Hudson) Date: Fri, 28 Jul 2006 16:26:37 +0100 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA082B.6030908@blueyonder.co.uk> (David Hopwood's message of "Fri, 28 Jul 2006 13:50:51 +0100") References: <20060728101133.GA339@code0.codespeak.net> <44CA082B.6030908@blueyonder.co.uk> Message-ID: <2mac6tyclu.fsf@starship.python.net> David Hopwood <david.nospam.hopwood at blueyonder.co.uk> writes: > Armin Rigo wrote: >> Hi, >> >> There is an oversight in the design of __index__() that only just >> surfaced :-( It is responsible for the following behavior, on a 32-bit >> machine with >= 2GB of RAM: >> >> >>> s = 'x' * (2**100) # works! >> >>> len(s) >> 2147483647 >> >> This is because PySequence_Repeat(v, w) works by applying w.__index__ in >> order to call v->sq_repeat. However, __index__ is defined to clip the >> result to fit in a Py_ssize_t. > > Clipping the result sounds like it would *never* be a good idea. What was > the rationale for that? It should throw an exception. Why would you expect range(10)[:2**32-1] and range(10)[:2**32] to do different things? Cheers, mwh -- This makes it possible to pass complex object hierarchies to a C coder who thinks computer science has made no worthwhile advancements since the invention of the pointer. -- Gordon McMillan, 30 Jul 1998 From ncoghlan at gmail.com Fri Jul 28 17:29:19 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Jul 2006 01:29:19 +1000 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA082B.6030908@blueyonder.co.uk> References: <20060728101133.GA339@code0.codespeak.net> <44CA082B.6030908@blueyonder.co.uk> Message-ID: <44CA2D4F.1020601@gmail.com> David Hopwood wrote: > Armin Rigo wrote: >> Hi, >> >> There is an oversight in the design of __index__() that only just >> surfaced :-( It is responsible for the following behavior, on a 32-bit >> machine with >= 2GB of RAM: >> >> >>> s = 'x' * (2**100) # works! >> >>> len(s) >> 2147483647 >> >> This is because PySequence_Repeat(v, w) works by applying w.__index__ in >> order to call v->sq_repeat. However, __index__ is defined to clip the >> result to fit in a Py_ssize_t. > > Clipping the result sounds like it would *never* be a good idea. What was > the rationale for that? It should throw an exception. A simple demonstration of the clipping behaviour that works on machines with limited memory: >>> (2**100).__index__() 2147483647 >>> (-2**100).__index__() -2147483648 PEP 357 doesn't even mention the issue, and the comment on long_index in the code doesn't give a rationale - it just notes that the function clips the result. Neither the PyNumber_AsIndex nor the __index__ documentation mention anything about the possibility of clipping, and there's no test case to verify this behaviour. I'm inclined to call it a bug, too, but I've cc'ed Travis to see if he can shed some light on the question - the implementation of long_index explicitly suppresses the overflow error generated by _long_as_ssize_t, so the current behaviour appears to be deliberate. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From pje at telecommunity.com Fri Jul 28 17:30:31 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 28 Jul 2006 11:30:31 -0400 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <44C9A1B1.3040102@v.loewis.de> References: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com> At 07:33 AM 7/28/2006 +0200, Martin v. L?wis wrote: >Phillip J. Eby wrote: > > I'm willing to write code that makes it PEP 302 compliant, if the release > > manager will bless such an addition. But if that's not acceptable, then > > somebody needs to produce the necessary documentation updates or revert > the > > patch. It absolutely should not be allowed to remain in *and* > undocumented > > because it is a backwards-incompatible change to documented behavior of > > Python for two major releases (2.3 and 2.4). > >You don't need a release manager pronouncement for that. It's a bug, >changing it is a bug fix, you don't need RM permission to fix a bug. > >Do you have a patch ready that restores path_importer_cache behavior, >yet preserves the property that it caches existence of a directory? >If not, I will have to produce one. The issue is that a proper fix that caches existence requires adding new types to import.c and thus might appear to be more of a feature. I was therefore reluctant to embark upon the work without some assurance that it wouldn't be rejected as adding a last-minute feature. From nnorwitz at gmail.com Fri Jul 28 17:53:07 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 28 Jul 2006 08:53:07 -0700 Subject: [Python-Dev] Another uuid problem In-Reply-To: <17610.6370.422365.622901@montanaro.dyndns.org> References: <17610.6370.422365.622901@montanaro.dyndns.org> Message-ID: <ee2a432c0607280853w1d5a26d7k47264bb1430be879@mail.gmail.com> It checks for ifconfig, /sbin/ifconfig, and /usr/sbin/ifconfig (same for arp). The problem is the os.pipe command doesn't hide these issues. It doesn't cause the test to fail, but is still broken. The test is presumably failing for the other reason I mentioned (unixdll_getnode). Let me know if you see that when running with -v. n -- On 7/28/06, skip at pobox.com <skip at pobox.com> wrote: > > I just tried building and testing 2.5 on a Solaris 10 box at work. The uuid > test failed with > > sh: ifconfig: not found > sh: ifconfig: not found > sh: ifconfig: not found > sh: arp: not found > sh: ifconfig: not found > sh: ifconfig: not found > sh: ifconfig: not found > sh: arp: not found > > In our environment at least it's uncommon for /usr/sbin to be in the PATH of > non-privileged users: > > piggy:% type -a ifconfig > -bash: type: ifconfig: not found > piggy:% PATH=$PATH:/usr/sbin type -a ifconfig > ifconfig is /usr/sbin/ifconfig > > Perhaps test_uuid needs to do a little investigation to find ifconfig and > arp. > > Skip > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com > From ncoghlan at gmail.com Fri Jul 28 17:54:30 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Jul 2006 01:54:30 +1000 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <2mac6tyclu.fsf@starship.python.net> References: <20060728101133.GA339@code0.codespeak.net> <44CA082B.6030908@blueyonder.co.uk> <2mac6tyclu.fsf@starship.python.net> Message-ID: <44CA3336.6030807@gmail.com> Michael Hudson wrote: > David Hopwood <david.nospam.hopwood at blueyonder.co.uk> writes: > >> Armin Rigo wrote: >>> Hi, >>> >>> There is an oversight in the design of __index__() that only just >>> surfaced :-( It is responsible for the following behavior, on a 32-bit >>> machine with >= 2GB of RAM: >>> >>> >>> s = 'x' * (2**100) # works! >>> >>> len(s) >>> 2147483647 >>> >>> This is because PySequence_Repeat(v, w) works by applying w.__index__ in >>> order to call v->sq_repeat. However, __index__ is defined to clip the >>> result to fit in a Py_ssize_t. >> Clipping the result sounds like it would *never* be a good idea. What was >> the rationale for that? It should throw an exception. > > Why would you expect range(10)[:2**32-1] and range(10)[:2**32] to do > different things? In that case, I believe it is the slice object that should be suppressing the overflow error (via PyErr_Occurred and PyErr_Matches) when calculating the indices for a given length, rather than having silent clipping be part of the basic implementation of long.__index__(). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From tim.peters at gmail.com Fri Jul 28 17:55:47 2006 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 28 Jul 2006 11:55:47 -0400 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <20060728101133.GA339@code0.codespeak.net> References: <20060728101133.GA339@code0.codespeak.net> Message-ID: <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com> [Armin Rigo] > There is an oversight in the design of __index__() that only just > surfaced :-( It is responsible for the following behavior, on a 32-bit > machine with >= 2GB of RAM: > > >>> s = 'x' * (2**100) # works! > >>> len(s) > 2147483647 > > This is because PySequence_Repeat(v, w) works by applying w.__index__ in > order to call v->sq_repeat. ? I don't see an invocation of __index__ or nb_index in PySequence_Repeat. To the contrary, its /incoming/ `count` argument is constrained to Py_ssize_t from the start: PyObject * PySequence_Repeat(PyObject *o, Py_ssize_t count) ... OK, I think you mean sequence_repeat() in abstract.c. That does invoke nb_index. But, as below, I don't think it should in this case. > However, __index__ is defined to clip the result to fit in a Py_ssize_t. > This means that the above problem exists > with all sequences, not just strings, given enough RAM to create such > sequences with 2147483647 items. > > For reference, in 2.4 we correctly get an OverflowError. > > Argh! What should be done about it? IMO, this is plain wrong. PEP 357 isn't entirely clear, but it is clear the author only had /slicing/ in mind (where clipping makes sense -- and which makes `__index__` a misleading name). Guido pointed out the ambiguity here: http://mail.python.org/pipermail/python-dev/2006-February/060624.html There's also an ambiguity when using simple indexing. When writing x[i] where x is a sequence and i an object that isn't int or long but implements __index__, I think i.__index__() should be used rather than bailing out. I suspect that you didn't think of this because you've already special-cased this in your code -- when a non-integer is passed, the mapping API is used (mp_subscript). This is done to suppose extended slicing. The built-in sequences (list, str, unicode, tuple for sure, probably more) that implement mp_subscript should probe for nb_index before giving up. The generic code in PyObject_GetItem should also check for nb_index before giving up. So, e.g., plain a[i] shouldn't use __index__ either if i is already int or long. I don't see any justification for invoking nb_index in sequence_repeat(), although if someone thinks it should, then as for plain indexing it certainly shouldn't invoke nb_index if the incoming count is an int or long to begin with. Ah, fudge. Contrary to Guido's advice above, I see that PyObject_GetItem() /also/ unconditionally invokes nb_index (even when the incoming key is already int or long). It shouldn't do that either (according to me). OTOH, in the long discussion about PEP 357, I'm not sure anyone except Travis was clear on whether nb_index was meant to apply only to sequence /slicing/ or was meant to apply "everywhere an object gets used in an index-like context". Clipping makes sense only for the former, but it looks like the implementation treats it more like the latter. This was probably exacerbated by: http://mail.python.org/pipermail/python-dev/2006-February/060663.html [Travis] There are other places in Python that check specifically for int objects and long integer objects and fail with anything else. Perhaps all of these should aslo call the __index__ slot. [Guido] Right, absolutely. This is a mess :-) From nnorwitz at gmail.com Fri Jul 28 17:59:51 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 28 Jul 2006 08:59:51 -0700 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <2mirlixdg4.fsf@starship.python.net> References: <44C9A184.7030701@ewtllc.com> <200607281539.50794.anthony@interlink.com.au> <2mirlixdg4.fsf@starship.python.net> Message-ID: <ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com> On 7/28/06, Michael Hudson <mwh at python.net> wrote: > Anthony Baxter <anthony at interlink.com.au> writes: > > > Does anyone disagree with making the next release beta3? > > It seems like a good idea to me. I guess this will mean the final > release will be pushed back a bit? Anthony and I talked about still having b3 on Aug 1. rc1 around Aug 17-18 (just before the Google sprint which Martin, Jeremy and I will be attending). Final around 24-29. We didn't discuss with Martin yet, so these dates are quite tentative. n From guido at python.org Fri Jul 28 18:05:57 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Jul 2006 09:05:57 -0700 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA2D4F.1020601@gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <44CA082B.6030908@blueyonder.co.uk> <44CA2D4F.1020601@gmail.com> Message-ID: <ca471dc20607280905y7447d005r2ab0ec664a69948@mail.gmail.com> Argh. I also find it a bug. I also feel responsible because I reviewed the patch. :-( In my recollection I tried to avoid this exact behavior. I wanted __index__() to just return the unclipped int or long value, but have a C API that clipped it for use in slice operations. It looks like I failed (the patch went through so many revisions that at some point I must've stopped caring). --Guido On 7/28/06, Nick Coghlan <ncoghlan at gmail.com> wrote: > David Hopwood wrote: > > Armin Rigo wrote: > >> Hi, > >> > >> There is an oversight in the design of __index__() that only just > >> surfaced :-( It is responsible for the following behavior, on a 32-bit > >> machine with >= 2GB of RAM: > >> > >> >>> s = 'x' * (2**100) # works! > >> >>> len(s) > >> 2147483647 > >> > >> This is because PySequence_Repeat(v, w) works by applying w.__index__ in > >> order to call v->sq_repeat. However, __index__ is defined to clip the > >> result to fit in a Py_ssize_t. > > > > Clipping the result sounds like it would *never* be a good idea. What was > > the rationale for that? It should throw an exception. > > A simple demonstration of the clipping behaviour that works on machines with > limited memory: > > >>> (2**100).__index__() > 2147483647 > >>> (-2**100).__index__() > -2147483648 > > PEP 357 doesn't even mention the issue, and the comment on long_index in the > code doesn't give a rationale - it just notes that the function clips the result. > > Neither the PyNumber_AsIndex nor the __index__ documentation mention anything > about the possibility of clipping, and there's no test case to verify this > behaviour. > > I'm inclined to call it a bug, too, but I've cc'ed Travis to see if he can > shed some light on the question - the implementation of long_index explicitly > suppresses the overflow error generated by _long_as_ssize_t, so the current > behaviour appears to be deliberate. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > http://www.boredomandlaziness.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tomerfiliba at gmail.com Fri Jul 28 18:07:30 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Fri, 28 Jul 2006 18:07:30 +0200 Subject: [Python-Dev] Fwd: patching pydoc? In-Reply-To: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com> References: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com> Message-ID: <1d85506f0607280907x22589f98od79e925baba24ac1@mail.gmail.com> submitted patch: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=1530482&group_id=5470 -tomer ---------- Forwarded message ---------- From: tomer filiba <tomerfiliba at gmail.com> Date: Jul 28, 2006 3:35 PM Subject: patching pydoc? To: python-dev at python.org i have a problem with pydoc in rpyc. i wanted help(obj), where obj is a NetProxy object, to work as if it were local. i followed the code starting from site.help to pydoc.doc, which is the ultimate function that generates and prints the text. i expected there would be some function in the middle that prepares the text, and another that writes it to the pager, but to my disappointment pydoc.doc does both. this means i can't transfer the document to my local machine (it's printed directly to the remote console). therefore, i would like to split this behavior into two parts: * render_doc - a function that returns the document text * doc - a function that calls render_doc and sends it to the pager this way no existing code breaks (no existing function signatures are changed) and i gain help on remote objects. i hope people would be in favor, as it's not such a big change anyway. is it possible to add to 2.5? -tomer this is the code of pydoc, starting at line 1457 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< def doc(thing, title='Python Library Documentation: %s', forceload=0): """Display text documentation, given an object or a path to an object.""" try: object, name = resolve(thing, forceload) desc = describe(object) module = inspect.getmodule(object) if name and '.' in name: desc += ' in ' + name[:name.rfind('.')] elif module and module is not object: desc += ' in module ' + module.__name__ if not (inspect.ismodule(object) or inspect.isclass(object) or inspect.isroutine(object) or isinstance(object, property)): # If the passed object is a piece of data or an instance, # document its available methods instead of its value. object = type(object) desc += ' object' pager(title % desc + '\n\n' + text.document(object, name)) except (ImportError, ErrorDuringImport), value: print value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> this is the suggested code <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< def render_doc(thing, title='Python Library Documentation: %s', forceload=0): """generate the text""" object, name = resolve(thing, forceload) desc = describe(object) module = inspect.getmodule(object) if name and '.' in name: desc += ' in ' + name[:name.rfind('.')] elif module and module is not object: desc += ' in module ' + module.__name__ if not (inspect.ismodule(object) or inspect.isclass(object) or inspect.isroutine(object) or isinstance(object, property)): # If the passed object is a piece of data or an instance, # document its available methods instead of its value. object = type(object) desc += ' object' return title % desc + '\n\n' + text.document(object, name) def doc(*args, **kwargs): """Display text documentation, given an object or a path to an object.""" try: text = render_doc(*args, **kwargs) pager(text) except (ImportError, ErrorDuringImport), value: print value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> From ncoghlan at gmail.com Fri Jul 28 18:33:01 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Jul 2006 02:33:01 +1000 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com> Message-ID: <44CA3C3D.90805@gmail.com> Tim Peters wrote: > OTOH, in the long discussion about PEP 357, I'm not sure anyone except > Travis was clear on whether nb_index was meant to apply only to > sequence /slicing/ or was meant to apply "everywhere an object gets > used in an index-like context". Clipping makes sense only for the > former, but it looks like the implementation treats it more like the > latter. This was probably exacerbated by: > > http://mail.python.org/pipermail/python-dev/2006-February/060663.html > > [Travis] > There are other places in Python that check specifically for int objects > and long integer objects and fail with anything else. Perhaps all of > these should aslo call the __index__ slot. > > [Guido] > Right, absolutely. > > This is a mess :-) I've been trawling through the code a bit, and I don't think it's as bad as all that. All I believe is really needed is to: - remove the PyErr_Occurred() check and its body from long_index in longobject.c - add a PyErr_Occurred() check to force a -1 return from PyNumber_Index in abstract.c - add a PyErr_Occurred() and PyErr_ExceptionMatches(PyOverflowError) check to invoke PyErr_Clear() in _PyEval_SliceIndex in ceval.c. Add test cases to test_index.py to check that: (2**100).__index__() == 2**100 (-2**100).__index__() == -2**100 slice(-2**100, 2**100).indices(sys.maxint) == (0, sys.maxint, 1) "a" * 2**100 raises OverflowError Add test cases to test_operator.py to check that: operator.index(2**100) == 2**100 operator.index(-2**100) == -2**100 Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Fri Jul 28 20:19:24 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Jul 2006 04:19:24 +1000 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA3C3D.90805@gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com> <44CA3C3D.90805@gmail.com> Message-ID: <44CA552C.9020805@gmail.com> Nick Coghlan wrote: > Tim Peters wrote: >> OTOH, in the long discussion about PEP 357, I'm not sure anyone except >> Travis was clear on whether nb_index was meant to apply only to >> sequence /slicing/ or was meant to apply "everywhere an object gets >> used in an index-like context". Clipping makes sense only for the >> former, but it looks like the implementation treats it more like the >> latter. This was probably exacerbated by: >> >> http://mail.python.org/pipermail/python-dev/2006-February/060663.html >> >> [Travis] >> There are other places in Python that check specifically for int objects >> and long integer objects and fail with anything else. Perhaps all of >> these should aslo call the __index__ slot. >> >> [Guido] >> Right, absolutely. >> >> This is a mess :-) > > I've been trawling through the code a bit, and I don't think it's as bad as > all that. Damn, it really is a mess. . . nb_index returns the Pyssize_t directly, and a whole heap of the code expects errors to be signalled via returning -1 before checking PyErr_Occurred(). To get it to work without clipping everywhere, wrap_lenfunc (typeobject.c), _PyEval_SliceIndex (ceval.c), PyNumber_Index (abstract.c) and sequence_repeat (abstract.c) all had to be modified to recognize PY_SSIZE_T_MIN and PY_SSIZE_T_MAX as potential error flags (in order to clear the overflow error for _PyEval_SliceIndex, and in order to propagate the exception for the other three). And using this approach still means that (2**100).__index__() raises an OverflowError. It would probably be cleaner to change the signature of nb_index to return a PyObject *, and let the code that uses it worry about how (or even whether!) to convert PyLong results to a Py_ssize_t. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From tjreedy at udel.edu Fri Jul 28 20:29:50 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 28 Jul 2006 14:29:50 -0400 Subject: [Python-Dev] patching pydoc? References: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com> Message-ID: <eadl2v$51q$1@sea.gmane.org> "tomer filiba" <tomerfiliba at gmail.com> wrote in message news:1d85506f0607280635q3a693682l230c7821dc6f408f at mail.gmail.com... ... > therefore, i would like to split this behavior into two parts: > * render_doc - a function that returns the document text > * doc - a function that calls render_doc and sends it to the pager > > this way no existing code breaks (no existing function signatures > are changed) and i gain help on remote objects. > i hope people would be in favor, as it's not such a big change anyway. > is it possible to add to 2.5? Giving the amount of hair-tearing over uuid and __index__, this seems like an especially bad day to ask for a new-feature variance in a time of feature freeze ;-). Some quick questions: * I presume you gain the new functionality by directly calling the factored-out render_doc and printing thru your own pager. Does everyone? * Would making pager() a parameter of doc() make sense? * Is pager() the only part of the original doc() that can generate ImportError, ErrorDuringImport? If not, the try/except should be in render_doc also or instead. * Why generalize the doc() signature? Bad calls will be traced as caught in render_doc instead of doc. Couldn't that potentially break a bad_call test? Terry Jan Reedy > this is the code of pydoc, starting at line 1457 > > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< > def doc(thing, title='Python Library Documentation: %s', forceload=0): > """Display text documentation, given an object or a path to an > object.""" > try: > object, name = resolve(thing, forceload) > desc = describe(object) > module = inspect.getmodule(object) > if name and '.' in name: > desc += ' in ' + name[:name.rfind('.')] > elif module and module is not object: > desc += ' in module ' + module.__name__ > if not (inspect.ismodule(object) or > inspect.isclass(object) or > inspect.isroutine(object) or > isinstance(object, property)): > # If the passed object is a piece of data or an instance, > # document its available methods instead of its value. > object = type(object) > desc += ' object' > pager(title % desc + '\n\n' + text.document(object, name)) > except (ImportError, ErrorDuringImport), value: > print value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > > this is the suggested code > > <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< > def render_doc(thing, title='Python Library Documentation: %s', > forceload=0): > """generate the text""" > object, name = resolve(thing, forceload) > desc = describe(object) > module = inspect.getmodule(object) > if name and '.' in name: > desc += ' in ' + name[:name.rfind('.')] > elif module and module is not object: > desc += ' in module ' + module.__name__ > if not (inspect.ismodule(object) or > inspect.isclass(object) or > inspect.isroutine(object) or > isinstance(object, property)): > # If the passed object is a piece of data or an instance, > # document its available methods instead of its value. > object = type(object) > desc += ' object' > return title % desc + '\n\n' + text.document(object, name) > > def doc(*args, **kwargs): > """Display text documentation, given an object or a path to an > object.""" > try: > text = render_doc(*args, **kwargs) > pager(text) > except (ImportError, ErrorDuringImport), value: > print value >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > From guido at python.org Fri Jul 28 20:31:09 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Jul 2006 11:31:09 -0700 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA552C.9020805@gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com> <44CA3C3D.90805@gmail.com> <44CA552C.9020805@gmail.com> Message-ID: <ca471dc20607281131m34f752b4pb1ff933f2d2262d9@mail.gmail.com> On 7/28/06, Nick Coghlan <ncoghlan at gmail.com> wrote: > Nick Coghlan wrote: > > Tim Peters wrote: > >> OTOH, in the long discussion about PEP 357, I'm not sure anyone except > >> Travis was clear on whether nb_index was meant to apply only to > >> sequence /slicing/ or was meant to apply "everywhere an object gets > >> used in an index-like context". Clipping makes sense only for the > >> former, but it looks like the implementation treats it more like the > >> latter. This was probably exacerbated by: > >> > >> http://mail.python.org/pipermail/python-dev/2006-February/060663.html > >> > >> [Travis] > >> There are other places in Python that check specifically for int objects > >> and long integer objects and fail with anything else. Perhaps all of > >> these should aslo call the __index__ slot. > >> > >> [Guido] > >> Right, absolutely. > >> > >> This is a mess :-) > > > > I've been trawling through the code a bit, and I don't think it's as bad as > > all that. > > Damn, it really is a mess. . . nb_index returns the Pyssize_t directly, and a > whole heap of the code expects errors to be signalled via returning -1 before > checking PyErr_Occurred(). > > To get it to work without clipping everywhere, wrap_lenfunc (typeobject.c), > _PyEval_SliceIndex (ceval.c), PyNumber_Index (abstract.c) and sequence_repeat > (abstract.c) all had to be modified to recognize PY_SSIZE_T_MIN and > PY_SSIZE_T_MAX as potential error flags (in order to clear the overflow error > for _PyEval_SliceIndex, and in order to propagate the exception for the other > three). > > And using this approach still means that (2**100).__index__() raises an > OverflowError. > > It would probably be cleaner to change the signature of nb_index to return a > PyObject *, and let the code that uses it worry about how (or even whether!) > to convert PyLong results to a Py_ssize_t. No time to look through the code here, but IMO it's acceptable (at least for 2.5) if (2**100).__index__() raises OverflowError, as long as x[:2**100] silently clips. __index__() is primarily meant to return a value useful for indexing concrete sequences, and 2**100 isn't. Certainly the exception is preferrable to the silently truncated result currently returned. Fortunately there's some extra time since we're now going to do a third beta. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Fri Jul 28 20:39:28 2006 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 28 Jul 2006 14:39:28 -0400 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA552C.9020805@gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com> <44CA3C3D.90805@gmail.com> <44CA552C.9020805@gmail.com> Message-ID: <1f7befae0607281139ycb89b13n3ac3d829499eeab1@mail.gmail.com> [Tim] >>> ... >>> This is a mess :-) [Nick Coghlan] >> I've been trawling through the code a bit, and I don't think it's as bad as >> all that. [also Nick, but older & wiser ;-)] > Damn, it really is a mess. . . nb_index returns the Pyssize_t directly, Bingo. It's a /conceptual/ mess. Best I can make out, Travis only cared about sequence slicing (not indexing), and then the machinery got hijacked to become a more general "can you faithfully act like an integer?" thing -- but kept a signature that made sense only for the original slicing use (where clipping is fine). > and a whole heap of the code expects errors to be signalled via returning -1 before > checking PyErr_Occurred(). > > To get it to work without clipping everywhere, wrap_lenfunc (typeobject.c), > _PyEval_SliceIndex (ceval.c), PyNumber_Index (abstract.c) and sequence_repeat > (abstract.c) all had to be modified to recognize PY_SSIZE_T_MIN and > PY_SSIZE_T_MAX as potential error flags (in order to clear the overflow error > for _PyEval_SliceIndex, and in order to propagate the exception for the other > three). > > And using this approach still means that (2**100).__index__() raises an > OverflowError. > > It would probably be cleaner to change the signature of nb_index to return a > PyObject *, Given that the more-general use is what everyone else either wanted, or simply /assumed/, in the original discussions, I expect it would be, although with the understanding that the PyObject * returned must be NULL (in case of error), or a Python int or long. > and let the code that uses it worry about how (or even whether!) > to convert PyLong results to a Py_ssize_t. A utility function or two could help, like one that converted to Py_ssize_t with clipping, and another that did the same but raised OverflowError if Py_ssize_t isn't big enough (and in the latter case a caller would do the usual business of checking for a -1 return and PyErr_Occurred()).. From syfou at users.sourceforge.net Fri Jul 28 21:24:19 2006 From: syfou at users.sourceforge.net (Sylvain Fourmanoit) Date: Fri, 28 Jul 2006 15:24:19 -0400 (EDT) Subject: [Python-Dev] New miniconf module In-Reply-To: <20060727103331.GA31912@code0.codespeak.net> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> <Pine.LNX.4.64.0607270200300.6000@sylvain> <20060727103331.GA31912@code0.codespeak.net> Message-ID: <Pine.LNX.4.64.0607281438120.10450@sylvain> Armin Rigo wrote: > In the same spirit, maybe it could be slightly re-oriented towards a > dumper/loader for more than config files; for example, it could provide > a safe inverse of repr() for common built-in types New version of miniconf (version 1.2.0) is out [1][2], including a unrepr() function; that's the only change this time. Michael Foord wrote: > ConfigObj [3] gained an 'unrepr' mode a while back. The code is simple, > and originally came from CherryPy. Thanks for the link! I completely missed ConfigObj. It indeed shares a lot with my code. At the core, it use many of the same ideas and implementation techniques... In many ways, it is also a lot more advanced, specialized than miniconf: for instance, it introduce a new, specialized Config File format, while my effort aimed at keeping things minimal. Armin Rigo wrote: > If it goes in that direction, I'd suggest to rename the module to give > it a name closer to existing persistence-related modules already in the > stdlib. I am not especially fond of the current miniconf name either; I didn't find something more suitable, yet evocative of what it does; I would be glad to hear any suggestion you or the rest of the developers would have. Yours, -- Sylvain <syfou at users.sourceforge.net> The only difference between a car salesman and a computer salesman is that the car salesman knows he's lying. [1]http://cheeseshop.python.org/pypi?:action=display&name=miniconf&version=1.2.0 [2]http://sourceforge.net/tracker/index.php?func=detail&aid=1527597&group_id=5470&atid=355470 [3]http://www.voidspace.org.uk/python/configobj.html P.-S. I am leaving the civilization (where I have some sort of network access) from July the 29th to August the 13th: I will be glad to address any comment, bug report or suggestion Python developers might want to discuss in the meantime as soon as I will be back. From fuzzyman at voidspace.org.uk Fri Jul 28 21:38:34 2006 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 28 Jul 2006 20:38:34 +0100 Subject: [Python-Dev] New miniconf module In-Reply-To: <Pine.LNX.4.64.0607281438120.10450@sylvain> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> <Pine.LNX.4.64.0607270200300.6000@sylvain> <20060727103331.GA31912@code0.codespeak.net> <Pine.LNX.4.64.0607281438120.10450@sylvain> Message-ID: <44CA67BA.9060803@voidspace.org.uk> Sylvain Fourmanoit wrote: > Armin Rigo wrote: > >> In the same spirit, maybe it could be slightly re-oriented towards a >> dumper/loader for more than config files; for example, it could provide >> a safe inverse of repr() for common built-in types >> > > New version of miniconf (version 1.2.0) is out [1][2], including a > unrepr() function; that's the only change this time. > > Michael Foord wrote: > >> ConfigObj [3] gained an 'unrepr' mode a while back. The code is simple, >> and originally came from CherryPy. >> > > Thanks for the link! I completely missed ConfigObj. It indeed shares a > lot with my code. At the core, it use many of the same ideas and > implementation techniques... In many ways, it is also a lot more > advanced, specialized than miniconf: for instance, it introduce a new, > specialized Config File format, while my effort aimed at keeping things > minimal. > > Cool - I made the post so you could reuse the unrepr code, but I'm glad you like ConfigObj. :-) If miniconf can store and restore instances of classes (which I guess it does if you see it as an improvement of pickle ?) then it does more than ConfigObj unrepr mode - which simply allows you to store basic data-types in config files (amongst all the other things it does...). Michael http://www.voidspace.org.uk/python/index.shtml From rhettinger at ewtllc.com Fri Jul 28 21:36:39 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Fri, 28 Jul 2006 12:36:39 -0700 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com> References: <44C9A184.7030701@ewtllc.com> <200607281539.50794.anthony@interlink.com.au> <2mirlixdg4.fsf@starship.python.net> <ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com> Message-ID: <44CA6747.2090603@ewtllc.com> Neal Norwitz wrote: >Anthony and I talked about still having b3 on Aug 1. rc1 around Aug >17-18 (just before the Google sprint which Martin, Jeremy and I will >be attending). Final around 24-29. We didn't discuss with Martin >yet, so these dates are quite tentative. > > If it doesn't muck-up your workflow, I would like to see all of these dates bumped back by about a week. In particular, it may be too aggressive to expect the __index__ issue be fully resolved by 8/1. Also, I have a few error-traps to add to setobject.c and need to review one of the implementation decisions for str.rpartition(). This weekend, I'll run the head on a few third-party packages to see if their test-suites still pass -- if not, it would be nice to have a bit of time to fix whatever arises. Raymond From fdrake at acm.org Fri Jul 28 22:07:46 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 28 Jul 2006 16:07:46 -0400 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com> References: <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com> <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com> Message-ID: <200607281607.46709.fdrake@acm.org> On Friday 28 July 2006 00:49, Neal Norwitz wrote: > Based on this comment, is it really acceptable to just document a > behaviour change? ISTM there should really only be 2 choices: fix > 2.5 properly or revert the change. This seemed to be Armin's > position. I agree those are the only reasonable solutions. I'd rather see things fixed, but I don't know how much time Phillip has to work on it. I'll be working on the straigtening out the xmlcore issue tonight/tomorrow. -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From bob at redivi.com Fri Jul 28 22:35:40 2006 From: bob at redivi.com (Bob Ippolito) Date: Fri, 28 Jul 2006 13:35:40 -0700 Subject: [Python-Dev] struct module and coercing floats to integers Message-ID: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com> It seems that the pre-2.5 struct module has some additional undocumented behavior[1] that didn't percolate into the new version: http://python.org/sf/1530559 Python 2.4 and previous will coerce floats to integers when necessary as such without any kind of complaint: $ python2.4 -c "import struct; print repr(struct.pack('>H', 0.9999999999999999))" '\x00\x00' Python 2.5 refuses to coerce float to int: $ python2.5 -c "import struct; print repr(struct.pack('>H', 0.9999999999999999))" Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/bob/src/python/Lib/struct.py", line 63, in pack return o.pack(*args) TypeError: unsupported operand type(s) for &: 'float' and 'long' The available options are to: 1. Reinstate the pre-2.5 weirdness 2. Reinstate the pre-2.5 weirdness with a DeprecationWarning 3. Break existing code that relies on undocumented behavior (seems more like a bug than lack of specification) Either 2 or 3 seems reasonable to me, with a preference for 3 because none of my code depends on old bugs in the struct module :) As far as precedent goes, the array module *used* to coerce floats silently, but it's had a DeprecationWarning since at least Python 2.3 (but perhaps even earlier). Maybe it's time to promote that warning to an exception for Python 2.5? [1] The pre-2.5 behavior should really be considered a bug, the documentation says "Return a string containing the values v1, v2, ... packed according to the given format. The arguments must match the values required by the format exactly." I wouldn't consider arbitrary floating point numbers to match the value required by an integer format exactly. Floats are not in general interchangeable with integers in Python anyway (e.g. list indexes, etc.). -bob From pje at telecommunity.com Fri Jul 28 22:42:00 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 28 Jul 2006 16:42:00 -0400 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <200607281607.46709.fdrake@acm.org> References: <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com> <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <5.1.1.6.0.20060727171324.0262c318@sparrow.telecommunity.com> <ee2a432c0607272149xc22baf4u2945da575237a83@mail.gmail.com> Message-ID: <5.1.1.6.0.20060728163317.0377cbf8@sparrow.telecommunity.com> At 04:07 PM 7/28/2006 -0400, Fred L. Drake, Jr. wrote: >On Friday 28 July 2006 00:49, Neal Norwitz wrote: > > Based on this comment, is it really acceptable to just document a > > behaviour change? ISTM there should really only be 2 choices: fix > > 2.5 properly or revert the change. This seemed to be Armin's > > position. > >I agree those are the only reasonable solutions. I'd rather see things >fixed, >but I don't know how much time Phillip has to work on it. I'll be working on >the straigtening out the xmlcore issue tonight/tomorrow. I'm testing a semi-final version of the fix now. It's minimally intrusive in that it only adds an "imp.NullImporter" type to replace the "False" value, and it keeps "None" to mean that the built-in import machinery should be used. From martin at v.loewis.de Fri Jul 28 22:47:34 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 28 Jul 2006 22:47:34 +0200 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com> References: <44C9A184.7030701@ewtllc.com> <200607281539.50794.anthony@interlink.com.au> <2mirlixdg4.fsf@starship.python.net> <ee2a432c0607280859r6c293fa5l78c66011829ff466@mail.gmail.com> Message-ID: <44CA77E6.7060200@v.loewis.de> Neal Norwitz wrote: > Anthony and I talked about still having b3 on Aug 1. rc1 around Aug > 17-18 (just before the Google sprint which Martin, Jeremy and I will > be attending). Final around 24-29. We didn't discuss with Martin > yet, so these dates are quite tentative. That doesn't work for me. The final release must either happen before Aug 19, or after Sep 9, or somebody else must roll the Windows binaries. Regards, Martin From martin at v.loewis.de Fri Jul 28 22:55:46 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 28 Jul 2006 22:55:46 +0200 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com> References: <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> <5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com> Message-ID: <44CA79D2.7000209@v.loewis.de> Phillip J. Eby wrote: > The issue is that a proper fix that caches existence requires adding new > types to import.c and thus might appear to be more of a feature. I was > therefore reluctant to embark upon the work without some assurance that it > wouldn't be rejected as adding a last-minute feature. So do you have a patch, or are going to write one? Regards, Martin From fuzzyman at voidspace.org.uk Fri Jul 28 23:20:31 2006 From: fuzzyman at voidspace.org.uk (Michael Foord) Date: Fri, 28 Jul 2006 22:20:31 +0100 Subject: [Python-Dev] Patch Against shutil.copytree Bug Message-ID: <44CA7F9F.3080402@voidspace.org.uk> Hello all, Sourceforge is being anal and won't let me log in. Attached is a patch for the shutil test, which would pickup bug "[1525866] Bug in shutil.copytree on Windows". It passes against a bug-fixed shutil [1] on my XP box, and fails on an un-fixed one. It only tests basic functionality of copytree. I did add tests that copystat had worked, but the mtime results were consistently off on Windows [2] so I removed that test. I would appreciate it if someone could post this patch to sourceforge. IMO this bug needs fixing before 2.5 final, the fix is trivial and shutil.copytree is broken on windows without it. All the best, Michael Foord [1] Bugfix supplied by Thomas Heller and is attached to the sourceforge bug report. [2] And consistently off in a weird way FWIW... -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: test_shutil.diff Url: http://mail.python.org/pipermail/python-dev/attachments/20060728/b83b9152/attachment.diff From gward-1337f07a94b43060ff5c1ea922ed93d6 at python.net Wed Jul 26 04:32:13 2006 From: gward-1337f07a94b43060ff5c1ea922ed93d6 at python.net (Greg Ward) Date: Tue, 25 Jul 2006 22:32:13 -0400 Subject: [Python-Dev] httplib and bad response chunking Message-ID: <20060726023213.GA3106@cthulhu.gerg.ca> So I accidentally discovered the other day that httplib does not handle a particular type of mangled HTTP response very well. In particular, it tends to blow up with an undocumented ValueError when the server screws up "chunked" encoding. I'm not the first to discover this, either: see http://www.python.org/sf/1486335 . <digression> HTTP 1.1 response chunking allows clients to know how many bytes of response to expect for dynamic content, i.e. when it's not possible to include a "Content-length" header. A chunked response might look like this: 0005\r\nabcd\n\r\n0004\r\nabc\n\r\n0\r\n\r\n which means: 0x0005 bytes in first chunk, which is "abcd\n" 0x0004 bytes in second chunk, which is "abc\n" Each chunk size is terminated with "\r\n"; each chunk is terminated with "\r\n"; end of response is indicated by a chunk of 0 bytes, hence the "\r\n\r\n" at the end. Details in RFC 2616: http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6.1 </digression> Anyways, what I discovered in the wild the other day was a response like this: 0005\r\nabcd\n\r\n0004\r\nabc\n\r\n\r\n i.e. the chunk-size for the terminating empty chunk was missing. This cause httplib.py to blow up with ValueError because it tried to call int(line, 16) assuming that 'line' contained a hex number, when in fact it was the empty string. Oops. IMHO the minimal fix is to turn ValueError into HTTPException (or a subclass thereof); httplib should not raise ValueError just because some server sends a bad response. (The server in question was Apache 2.0.52 running PHP 4.3.9 sending a big hairy error page because the database was down.) Where I'm getting hung up is how far to test this stuff. I have discovered other hypothetical cases of bad chunking that cause httplib to go into an infinite loop or block forever on socket.readline(). Should we worry about those cases as well, despite not having seen them happen in the wild? More annoying, I can reproduce the "block forever" case using a real socket, but not using the StringIO-based FakeSocket class in test_httplib. Anyways, I've cobbled together a crude hack to test_httplib.py that exposes the problem: http://sourceforge.net/tracker/download.php?group_id=5470&atid=105470&file_id=186245&aid=1486335 Feedback welcome. (Fixing the inadvertent ValueError is trivial, so I'm concentrating on getting the tests right first.) Oh yeah, my patch is relative to the 2.4 branch. Greg -- Greg Ward <gward at python.net> http://www.gerg.ca/ I don't believe there really IS a GAS SHORTAGE.. I think it's all just a BIG HOAX on the part of the plastic sign salesmen -- to sell more numbers!! From richardjones at optushome.com.au Wed Jul 26 06:26:29 2006 From: richardjones at optushome.com.au (Richard Jones) Date: Wed, 26 Jul 2006 14:26:29 +1000 Subject: [Python-Dev] More tracker demos online In-Reply-To: <ea6msv$hgf$1@sea.gmane.org> References: <44C6748C.4060701@v.loewis.de> <ea6msv$hgf$1@sea.gmane.org> Message-ID: <200607261426.29143.richardjones@optushome.com.au> On Wednesday 26 July 2006 13:17, Terry Reedy wrote: > ""Martin v. L?wis"" <martin at v.loewis.de> wrote in message > news:44C6748C.4060701 at v.loewis.de... > > > Currently, we have two running tracker demos online: > > > > Roundup: > > http://efod.se/python-tracker/ > > > > Jira: > > http://jira.python.atlassian.com/secure/Dashboard.jspa > > What user name and passwords will they accept, if any? For the Roundup tracker at least you should be able to recover your password using your sourceforge email address. Richard From oliphant.travis at ieee.org Fri Jul 28 18:15:51 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Fri, 28 Jul 2006 10:15:51 -0600 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA2D4F.1020601@gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <44CA082B.6030908@blueyonder.co.uk> <44CA2D4F.1020601@gmail.com> Message-ID: <44CA3837.3040506@ieee.org> Nick Coghlan wrote: > David Hopwood wrote: >> Armin Rigo wrote: >>> Hi, >>> >>> There is an oversight in the design of __index__() that only just >>> surfaced :-( It is responsible for the following behavior, on a 32-bit >>> machine with >= 2GB of RAM: >>> >>> >>> s = 'x' * (2**100) # works! >>> >>> len(s) >>> 2147483647 >>> >>> This is because PySequence_Repeat(v, w) works by applying >>> w.__index__ in >>> order to call v->sq_repeat. However, __index__ is defined to clip the >>> result to fit in a Py_ssize_t. >> >> Clipping the result sounds like it would *never* be a good idea. What >> was >> the rationale for that? It should throw an exception. > > A simple demonstration of the clipping behaviour that works on > machines with limited memory: > > >>> (2**100).__index__() > 2147483647 > >>> (-2**100).__index__() > -2147483648 > > PEP 357 doesn't even mention the issue, and the comment on long_index > in the code doesn't give a rationale - it just notes that the function > clips the result. I can't think of the rationale so it was probably an unclear one and should be thought of as a bug. The fact that it isn't discussed in the PEP means it wasn't thought about clearly. I think I had the vague idea that .__index_() should always succeed. But, this shows a problem with that notion. > > I'm inclined to call it a bug, too, but I've cc'ed Travis to see if he > can shed some light on the question - the implementation of long_index > explicitly suppresses the overflow error generated by > _long_as_ssize_t, so the current behaviour appears to be deliberate. If it was deliberate, it was a hurried decision and one that should be re-thought and probably changed. I think the idea came from the fact that out-of-bounds slicing returns empty lists and since __index__ was primarily developed to allow integer-like objects to be used in slicing it adopted that behavior. In fact it looks like the comment above _long_index contains words from the comment above _PyEval_SliceIndex showing the direct borrowing of the idea. But, _long_index is clearly the wrong place to handle the situation since it is used by more than just the slicing code. An error return is already handled by the _Eval_SliceIndex code anyway. I say it's a bug that should be fixed. Don't clear the error, raise it. -Travis From pje at telecommunity.com Sat Jul 29 00:00:36 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 28 Jul 2006 18:00:36 -0400 Subject: [Python-Dev] Release manager pronouncement needed: PEP 302 Fix In-Reply-To: <44CA79D2.7000209@v.loewis.de> References: <5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com> <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060726142456.036d9008@sparrow.telecommunity.com> <20060727103920.GB31912@code0.codespeak.net> <5.1.1.6.0.20060727114924.0262c980@sparrow.telecommunity.com> <5.1.1.6.0.20060728112800.03a7fb48@sparrow.telecommunity.com> Message-ID: <5.1.1.6.0.20060728173543.01b7c120@sparrow.telecommunity.com> At 10:55 PM 7/28/2006 +0200, Martin v. L?wis wrote: >Phillip J. Eby wrote: > > The issue is that a proper fix that caches existence requires adding new > > types to import.c and thus might appear to be more of a feature. I was > > therefore reluctant to embark upon the work without some assurance that it > > wouldn't be rejected as adding a last-minute feature. > >So do you have a patch, or are going to write one? Yes, it's checked in as r50916. It ultimately turned out to be simpler than I thought; only one new type (imp.NullImporter) was required. From martin at v.loewis.de Sat Jul 29 00:44:46 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 29 Jul 2006 00:44:46 +0200 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA3837.3040506@ieee.org> References: <20060728101133.GA339@code0.codespeak.net> <44CA082B.6030908@blueyonder.co.uk> <44CA2D4F.1020601@gmail.com> <44CA3837.3040506@ieee.org> Message-ID: <44CA935E.4010901@v.loewis.de> Travis Oliphant wrote: > I say it's a bug that should be fixed. Don't clear the error, raise it. Several people have said this, but I don't think it can work. If you raise an OverflowError in __index__, the slicing code cannot know whether this meant as overflow or underflow (in a signed sense). Regards, Martin From jjl at pobox.com Sat Jul 29 00:59:37 2006 From: jjl at pobox.com (John J Lee) Date: Fri, 28 Jul 2006 22:59:37 +0000 (UTC) Subject: [Python-Dev] httplib and bad response chunking In-Reply-To: <20060726023213.GA3106@cthulhu.gerg.ca> References: <20060726023213.GA3106@cthulhu.gerg.ca> Message-ID: <Pine.LNX.4.64.0607282251320.8644@localhost> On Tue, 25 Jul 2006, Greg Ward wrote: [...] > Where I'm getting hung up is how far to test this stuff. Stop when you run out of time ;-) > I have > discovered other hypothetical cases of bad chunking that cause httplib > to go into an infinite loop or block forever on socket.readline(). > Should we worry about those cases as well, despite not having seen them > happen in the wild? More annoying, I can reproduce the "block forever" > case using a real socket, but not using the StringIO-based FakeSocket > class in test_httplib. They have been seen in the wild :-) http://python.org/sf/1411097 The IP address referenced isn't under my control, I don't know if it still provokes the error, but the problem is clear. John From rrr at ronadam.com Sat Jul 29 01:06:02 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 28 Jul 2006 18:06:02 -0500 Subject: [Python-Dev] patching pydoc? In-Reply-To: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com> References: <1d85506f0607280635q3a693682l230c7821dc6f408f@mail.gmail.com> Message-ID: <44CA985A.3030408@ronadam.com> tomer filiba wrote: > i have a problem with pydoc in rpyc. i wanted help(obj), where obj > is a NetProxy object, to work as if it were local. > > i followed the code starting from site.help to pydoc.doc, which is the > ultimate function that generates and prints the text. i expected there > would be some function in the middle that prepares the text, and > another that writes it to the pager, but to my disappointment pydoc.doc > does both. > > this means i can't transfer the document to my local machine (it's > printed directly to the remote console). > > therefore, i would like to split this behavior into two parts: > * render_doc - a function that returns the document text > * doc - a function that calls render_doc and sends it to the pager > > this way no existing code breaks (no existing function signatures > are changed) and i gain help on remote objects. > i hope people would be in favor, as it's not such a big change anyway. > > > is it possible to add to 2.5? > > -tomer Hi Tomer, This doesn't address your small patch since the following described project is, not complete, will be either be a separate module or package, (if/when it's completed), and because it's more of a complete rewrite than not, would be ready and tested no sooner than 2.6. But you might be interested in taking a look at what I've attempted to do so far. (This is too premature to consider adding to python distribution but maybe after completing, and after it is used in the community it could be a candidate.) I am/was working on cleaning up pydoc by trying to separate the various formating, server, and output routines, because they are currently very much intertwined making enhancing and/or maintaining pydoc more difficult than it could be. This has resulted rewriting most of the top level functions, which after having some success, has been put on hold because I have other things to do that are more immediate at the moment. What I have done (so far) is separate the file pydoc.py into multiple files in order to begin to better organize the code, And rewriting each of those files so they are less interdependent on each other and more complete in them selves with the possibility of putting it back together in a single file or a package at some point. (I'm leaning on it being a package myself.) My attempt has focused on having pydoc create an intermediate xml-tree data structure that is then be used as the source to generate text, html, and other outputs. This separates the introspective and formatting functions of pydoc. The xml to html function mostly work. (some dependency link tags still need to be reimplemented.) Another improvement I've made is having the html server prepend an html search bar to each page which eliminates the need to have a separate TK search window. This makes interactively browsing with pydoc very easy and nice. :-) It also uses a style sheet to format the html pages, making it easier to change colors, indents, fonts, and borders, etc... If you are interested in working on this or would just like to take a look, let me know, I can put it in a zip file and send it to you. It's not as well organized as I would like at the moment, but the xml and html generation functions work. No attempt to create plain text output from the xml has been made yet. I intended to write the xml to text (and paging help) functions from scratch once the html output is complete and fully working. (mostly done) I think it will not be difficult to get the output very similar to the existing pydoc help text form. I'm thinking it may be better to share it at this time rather than let it sit on my hard drive and possibly get waylayed, or worse, forgotten indefinitely. If this is the "right approach", and is "desirable", maybe it could make a good community project. Possibly someone could post it and/or manage it someplace? It may need a bit more cleaning up before it's ready for that though. Some additional thoughts of why to do this are: I'm hoping there may be a way to run the html server on a web site to generate documentation directly (pydoc-live?) instead of pre-generating html files and placing them on the web site. (Possible?, Has that already been done?) I know pydoc html files are available on line, but I think they are all pregenerated by pydoc first then uploaded. That may be preferred in most cases because it would require less server overhead and be more secure, but live pydocs would give more immediate and dynamic feedback when a project is being updated and changed often. Another far off (and more ambitious 3.0 ?) goal may be to use the pydoc generated xml as a source to insert doc strings and other information directly into the python user manuals. That may improve both the python library manuals as well as the pydoc help information by having them more closely tied together. Inconsistencies would be found sooner. The doc strings could be the brief quick ref documents, which it currently isn't always brief and/or present, and the python manual could include the pydoc output (or selected parts of it) along with the longer explanation and discussion. Having pydoc produce xml as an intermediate format makes these types of things easier to do. Cheers, Ron Adam From tomerfiliba at gmail.com Sat Jul 29 02:11:19 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 29 Jul 2006 02:11:19 +0200 Subject: [Python-Dev] patching pydoc? Message-ID: <1d85506f0607281711q1152506bn4c3bc8a2bcadb0ef@mail.gmail.com> > Giving the amount of hair-tearing over uuid and __index__, this seems like > an especially bad day to ask for a new-feature variance in a time of > feature freeze ;-). yeah, i guess so. > * Would making pager() a parameter of doc() make sense? not at all. > * I presume you gain the new functionality by directly calling the > factored-out render_doc and printing thru your own pager. Does everyone? i don't quite get the question, but yes, i plan to call render_doc instead of doc, getting the doc text, and printing it on my own console using a pager/whatever. about the signature, okay, it can easily be fixed. i didn't think about that too much. about the other point -- ImportError etc -- i tried to go "dumb" as much as possible and not make any change that would break something. pydoc is poorly written anyway, and as ron adam noted, there is a need for at least a cleaned-up version of pydoc, or even a redesigned package. and it ought to be a package. it's a module of ~2000 lines. that's way too much for a module. i made the minimal changes needed to separate the "generate text" part from the "generate and display" part. and i kept all the semantics in tact (including who handles the errors) and therefore, i don't expect it to break any tests (after restoring the original doc signature, that is). i'd hate to wait for 2.6 for such a small change. today my solution is to replace pydoc.pager by a fake pager that hands over the result. this is of course not thread safe, as multiple threads calling help() would mix their outputs. not very likely, but i'd love to see this code go away asap. thanks for the comments, -tomer > From: Terry Reedy <tjreedy <at> udel.edu> > Subject: Re: patching pydoc? > Newsgroups: gmane.comp.python.devel > Date: 2006-07-28 18:29:50 GMT (5 hours and 27 minutes ago) > > "tomer filiba" <tomerfiliba <at> gmail.com> wrote in message > news:1d85506f0607280635q3a693682l230c7821dc6f408f <at> mail.gmail.com... > ... > > therefore, i would like to split this behavior into two parts: > > * render_doc - a function that returns the document text > > * doc - a function that calls render_doc and sends it to the pager > > > > this way no existing code breaks (no existing function signatures > > are changed) and i gain help on remote objects. > > i hope people would be in favor, as it's not such a big change anyway. > > is it possible to add to 2.5? > > Giving the amount of hair-tearing over uuid and __index__, this seems like > an especially bad day to ask for a new-feature variance in a time of > feature freeze ;-). > > Some quick questions: > * I presume you gain the new functionality by directly calling the > factored-out render_doc and printing thru your own pager. Does everyone? > * Would making pager() a parameter of doc() make sense? > * Is pager() the only part of the original doc() that can generate > ImportError, ErrorDuringImport? If not, the try/except should be in > render_doc also or instead. > * Why generalize the doc() signature? Bad calls will be traced as caught > in render_doc instead of doc. Couldn't that potentially break a bad_call > test? > > Terry Jan Reedy From chrism at plope.com Sat Jul 29 02:02:10 2006 From: chrism at plope.com (Chris McDonough) Date: Fri, 28 Jul 2006 20:02:10 -0400 Subject: [Python-Dev] cgi.FieldStorage DOS (sf bug #1112549) Message-ID: <D38E3187-946F-4649-9E6D-F366E1F7E52C@plope.com> From the initial bugreport (http://sourceforge.net/tracker/index.php? func=detail&aid=1112549&group_id=5470&atid=105470) """ Various parts of cgi.FieldStorage call its "read_lines_to_outerboundary", "read_lines" and "skip_lines" methods. These methods use the "readline" method of the file object that represents an input stream. The input stream is typically data supplied by an untrusted source (such as a user uploading a file from a web browser). The input data is not required by the RFC 822/1521/1522/1867 specifications to contain any newline characters. For example, it is within the bounds of the specification to supply a a multipart/form-data input stream with a "file-data" part that consists of a 2GB string composed entirely of "x" characters (which happens to be something I did that led me to noticing this bug). """ This bug has been around for about a year but I just worked up a patch yesterday that applies OK against current SVN. It's attached to the issue. Would someone be so kind as to check it in? Guido has already reviewed it, I believe. - C From david.nospam.hopwood at blueyonder.co.uk Sat Jul 29 02:14:21 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Sat, 29 Jul 2006 01:14:21 +0100 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA935E.4010901@v.loewis.de> References: <20060728101133.GA339@code0.codespeak.net> <44CA082B.6030908@blueyonder.co.uk> <44CA2D4F.1020601@gmail.com> <44CA3837.3040506@ieee.org> <44CA935E.4010901@v.loewis.de> Message-ID: <44CAA85D.5010907@blueyonder.co.uk> Martin v. L?wis wrote: > Travis Oliphant wrote: > >>I say it's a bug that should be fixed. Don't clear the error, raise it. > > Several people have said this, but I don't think it can work. > > If you raise an OverflowError in __index__, the slicing code cannot know > whether this meant as overflow or underflow (in a signed sense). Why not use IndexError for an underflow, and OverflowError for an overflow? -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From cvaughn at gmail.com Sat Jul 29 03:34:36 2006 From: cvaughn at gmail.com (Charles Vaughn) Date: Fri, 28 Jul 2006 20:34:36 -0500 Subject: [Python-Dev] Eliminating loops Message-ID: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com> I'm looking for a way of modifying the compiler to eliminate any loops and recursion from code. It's for a high speed data processing application. The alternative is a custom language that is little more than gloryfied assembly. I'd like to be able to use everything else around Python, but we can't allow the users to create more than O(1) complexity. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060728/0e0b1009/attachment.html From brett at python.org Sat Jul 29 03:40:43 2006 From: brett at python.org (Brett Cannon) Date: Fri, 28 Jul 2006 18:40:43 -0700 Subject: [Python-Dev] Eliminating loops In-Reply-To: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com> References: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com> Message-ID: <bbaeab100607281840g6ea3db63y5eb89b11bc1a424@mail.gmail.com> On 7/28/06, Charles Vaughn <cvaughn at gmail.com> wrote: > > I'm looking for a way of modifying the compiler to eliminate any loops and > recursion from code. It's for a high speed data processing application. > The alternative is a custom language that is little more than gloryfied > assembly. I'd like to be able to use everything else around Python, but we > can't allow the users to create more than O(1) complexity. > Well, the problem is how are you going to infer how many times to unroll the loop? range() and xrange() can change since they are only built-ins. You would need type inference on some locally defined variable that was set to a syntactically defined atomic type in order to know the proper number of times to unroll the loop and that might not work for your needs (on top of being a pain to write out a list with however many values you want to have the loop iterate over). -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060728/ed111499/attachment-0001.htm From greg.ewing at canterbury.ac.nz Sat Jul 29 04:50:21 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 29 Jul 2006 14:50:21 +1200 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <20060728101133.GA339@code0.codespeak.net> References: <20060728101133.GA339@code0.codespeak.net> Message-ID: <44CACCED.4090307@canterbury.ac.nz> Armin Rigo wrote: > This is because PySequence_Repeat(v, w) works by applying w.__index__ in > order to call v->sq_repeat. Why does it do that? Shouldn't __index__ only be used for numbers which are going to be used as an index? > However, __index__ is defined to clip the > result to fit in a Py_ssize_t. Why is it defined to do this instead of raising OverflowError? -- Greg From greg.ewing at canterbury.ac.nz Sat Jul 29 05:09:46 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 29 Jul 2006 15:09:46 +1200 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com> Message-ID: <44CAD17A.9090902@canterbury.ac.nz> Tim Peters wrote: > So, e.g., plain a[i] shouldn't use __index__ either if i is already > int or long. I don't see any justification for invoking nb_index in > sequence_repeat(), although if someone thinks it should, then as for > plain indexing it certainly shouldn't invoke nb_index if the incoming > count is an int or long to begin with. Hmmm. So that means anything accepting an integer index needs to do its own range checking anyway. So having __index__ do clipping is at best unnecessary and at worst counterproductive, since it could suppress an overflow or range exception that *should* be produced by the code using the index, and would be if it got the equivalent index value as an int or long directly. -- Greg From greg.ewing at canterbury.ac.nz Sat Jul 29 05:11:57 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 29 Jul 2006 15:11:57 +1200 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <ca471dc20607280905y7447d005r2ab0ec664a69948@mail.gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <44CA082B.6030908@blueyonder.co.uk> <44CA2D4F.1020601@gmail.com> <ca471dc20607280905y7447d005r2ab0ec664a69948@mail.gmail.com> Message-ID: <44CAD1FD.5090602@canterbury.ac.nz> Guido van Rossum wrote: > In my recollection I tried to avoid this exact behavior. I wanted > __index__() to just return the unclipped int or long value, but have a > C API that clipped it for use in slice operations. So is there still a chance to fix it? -- Greg From murman at gmail.com Sat Jul 29 05:14:58 2006 From: murman at gmail.com (Michael Urman) Date: Fri, 28 Jul 2006 22:14:58 -0500 Subject: [Python-Dev] struct module and coercing floats to integers In-Reply-To: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com> References: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com> Message-ID: <dcbbbb410607282014w30459351ifb09c56075997be1@mail.gmail.com> On 7/28/06, Bob Ippolito <bob at redivi.com> wrote: > http://python.org/sf/1530559 > > [1] The pre-2.5 behavior should really be considered a bug, the > documentation says "Return a string containing the values v1, v2, ... > packed according to the given format. The arguments must match the > values required by the format exactly." I wouldn't consider arbitrary > floating point numbers to match the value required by an integer > format exactly. Floats are not in general interchangeable with > integers in Python anyway (e.g. list indexes, etc.). While it may be a bug, it's not as hard to run into, nor as illogical as the presentation here makes it sound. The original code[1] took a float value between 0 and 2, and wanted to use pack('>H', round(value * 32768)) The workaround is a trivial change pack('>H', int(round(value * 32768))) but the timeframe is less than ideal, as working code will suddenly stop and recieve only mildly helpful error message. The fact that round returns a float rather than an int, while intentional, does not feature prominently in one's mine when the first version yielded the expected results. I would appreciate option 2 which retains compatibility but warns that the construct is bad. I will accept any of the options, as it's clear that floats don't make sense. It's just unfortunate that the previous implementation let them through in a way the new implementation does not. [1] http://www.sacredchao.net/quodlibet/changeset/3706 Michael -- Michael Urman http://www.tortall.net/mu/blog From greg.ewing at canterbury.ac.nz Sat Jul 29 05:44:33 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 29 Jul 2006 15:44:33 +1200 Subject: [Python-Dev] Rounding float to int directly (Re: struct module and coercing floats to integers) In-Reply-To: <dcbbbb410607282014w30459351ifb09c56075997be1@mail.gmail.com> References: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com> <dcbbbb410607282014w30459351ifb09c56075997be1@mail.gmail.com> Message-ID: <44CAD9A1.8080000@canterbury.ac.nz> Michael Urman wrote: > The fact that > round returns a float rather than an int, while intentional, does not > feature prominently in one's mine when the first version yielded the > expected results. As an aside, does anyone else think that it would be useful to have a builtin which rounds and converts to an int in one go? Whenever I use round(), I almost always want the result as an int, and making me do it in two steps seems unnecessarily bothersome. Since automatic float->int coercion is being increasingly disallowed, use cases for this are becoming more and more common. -- Greg From ncoghlan at gmail.com Sat Jul 29 05:49:51 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Jul 2006 13:49:51 +1000 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CA935E.4010901@v.loewis.de> References: <20060728101133.GA339@code0.codespeak.net> <44CA082B.6030908@blueyonder.co.uk> <44CA2D4F.1020601@gmail.com> <44CA3837.3040506@ieee.org> <44CA935E.4010901@v.loewis.de> Message-ID: <44CADADF.1040801@gmail.com> Martin v. L?wis wrote: > Travis Oliphant wrote: >> I say it's a bug that should be fixed. Don't clear the error, raise it. > > Several people have said this, but I don't think it can work. > > If you raise an OverflowError in __index__, the slicing code cannot know > whether this meant as overflow or underflow (in a signed sense). It can actually, but you have to allow 3 possible error return values from nb_index (-1, PY_SSIZE_T_MIN and PY_SSIZE_T_MAX). This is ugly as hell [1], so I'm going to try to work out a patch that changes the signature of nb_index to return PyObject * (leaving the client code in the core to decide whether to clip, overflow, or leave the result alone). Cheers, Nick. [1] http://www.python.org/sf/1530738 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sat Jul 29 05:53:21 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Jul 2006 13:53:21 +1000 Subject: [Python-Dev] New miniconf module In-Reply-To: <Pine.LNX.4.64.0607281438120.10450@sylvain> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> <Pine.LNX.4.64.0607270200300.6000@sylvain> <20060727103331.GA31912@code0.codespeak.net> <Pine.LNX.4.64.0607281438120.10450@sylvain> Message-ID: <44CADBB1.6090401@gmail.com> Sylvain Fourmanoit wrote: > Armin Rigo wrote: >> If it goes in that direction, I'd suggest to rename the module to give >> it a name closer to existing persistence-related modules already in the >> stdlib. > > I am not especially fond of the current miniconf name either; I didn't > find something more suitable, yet evocative of what it does; I would be > glad to hear any suggestion you or the rest of the developers would have. pyson :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From jcarlson at uci.edu Sat Jul 29 10:05:53 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 29 Jul 2006 01:05:53 -0700 Subject: [Python-Dev] Eliminating loops In-Reply-To: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com> References: <3bcbd0d70607281834g3b21146bi5dfd700cea47d4f2@mail.gmail.com> Message-ID: <20060729004957.E623.JCARLSON@uci.edu> "Charles Vaughn" <cvaughn at gmail.com> wrote: > I'm looking for a way of modifying the compiler to eliminate any loops and > recursion from code. It's for a high speed data processing application. > The alternative is a custom language that is little more than gloryfied > assembly. I'd like to be able to use everything else around Python, but we > can't allow the users to create more than O(1) complexity. One of the larger, if not largest advances in computer science in the last 50 years was the design and implementation of looping as a construct, not just an artifact of structured gotos, but as a method of implementing algorithms. With your proposed removal of loops and recursion, you are effectively saying that users should be given a turing machine because you are afraid of them doing foolish things with the language. Well, since the user is going to do foolish things with the language anyways, about all you can really do is to test, analyze, and verify. Oh, and educate. What do I mean? If your users have access to sequences of any type, and they can't perform 'x in y', then they are going to write their own contains/index function... def index(x,y): if x == y[0]: return 0 elif x == y[1]: return 1 elif x == y[2]: return 2 ... If they aren't given acess to sequences, then they will do what used to be done in QuakeC back in the day... def index(x,y): if x == y0: return 0 elif x == y1: return 1 elif x == y2: return 2 ... While algorithm design and analysis isn't something that everyone can do (some just can't handle the math), the users really should understand at least a bit about algorithms before they work on the "high speed data processing" application. - Josiah From g.brandl at gmx.net Sat Jul 29 10:07:42 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 29 Jul 2006 10:07:42 +0200 Subject: [Python-Dev] Using Python docs Message-ID: <eaf4u4$gfc$1@sea.gmane.org> Regarding bug 469773, I think it would be great to have such a document "Using Python", containing the manual page and platform- specific hints on how to invoke the interpreter and scripts (e.g. explaining the shebang for Unices). I'd be willing to help write up such a document. Another thing that could be helpful is a list of "frequently needed documentation sections", that is, a list of keywords and respective links to topics that are hard to find for newbies, such as the section "String formatting" or some locations in the reference manual. Georg From arigo at tunes.org Sat Jul 29 10:56:12 2006 From: arigo at tunes.org (Armin Rigo) Date: Sat, 29 Jul 2006 10:56:12 +0200 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <ca471dc20607281131m34f752b4pb1ff933f2d2262d9@mail.gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <1f7befae0607280855l3f71febbo95ff5751fe31d99c@mail.gmail.com> <44CA3C3D.90805@gmail.com> <44CA552C.9020805@gmail.com> <ca471dc20607281131m34f752b4pb1ff933f2d2262d9@mail.gmail.com> Message-ID: <20060729085612.GA30795@code0.codespeak.net> Hi Guido, On Fri, Jul 28, 2006 at 11:31:09AM -0700, Guido van Rossum wrote: > No time to look through the code here, but IMO it's acceptable (at > least for 2.5) if (2**100).__index__() raises OverflowError, as long > as x[:2**100] silently clips. __index__() is primarily meant to return > a value useful for indexing concrete sequences, and 2**100 isn't. If nb_index keeps returning a Py_ssize_t with clipping, it means that there is no way to write in pure Python an object that emulates a long -- only an int. Sounds inconsistent with the int/long unification trend for pure Python code. It would make it awkward to write, say, pure Python classes that pretend to be very large sequences, because using __index__ in such code wouldn't work. Another example of this is that if places like sequence_repeat are made to use the following pseudo-logic: if isinstance(w, long) and w > sys.maxint: raise OverflowError else: i = w.__index__() then if an object 'l' is an emulated pseudo-long, then "x"*l will still silently clip the pseudo-long to sys.maxint. I'm more in favor of changing nb_index to return a PyObject *, since now is our last chance to do so. A pair of API functions can be added to return a Py_ssize_t with either the proper clipping, or the proper OverflowError'ing. A bientot, Armin. From ncoghlan at gmail.com Sat Jul 29 16:06:53 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Jul 2006 00:06:53 +1000 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <20060728101133.GA339@code0.codespeak.net> References: <20060728101133.GA339@code0.codespeak.net> Message-ID: <44CB6B7D.1040001@gmail.com> Armin Rigo wrote: > Hi, > > There is an oversight in the design of __index__() that only just > surfaced :-( It is responsible for the following behavior, on a 32-bit > machine with >= 2GB of RAM: > > >>> s = 'x' * (2**100) # works! > >>> len(s) > 2147483647 > > This is because PySequence_Repeat(v, w) works by applying w.__index__ in > order to call v->sq_repeat. However, __index__ is defined to clip the > result to fit in a Py_ssize_t. This means that the above problem exists > with all sequences, not just strings, given enough RAM to create such > sequences with 2147483647 items. > > For reference, in 2.4 we correctly get an OverflowError. > > Argh! What should be done about it? I've now got a patch on SF that aims to fix this properly [1]. The gist of the patch: 1. Redesign the PyNumber_Index C API to serve the actual use cases in the interpreter core and the standard library. The PEP 357 abstract C API as written was bypassed by nearly all of the uses in the core and the standard library. The patch redesigns that API to reduce code duplication between the various parts of the code base that were previously calling nb_index directly. The principal change is to provide an "is_index" output variable that the various mp_subscript implementations can use to determine whether or not the passed in object was an index or not, rather than having to repeat the type check everywhere. The rationale for doing it this way: a. you may want to try something else (e.g. the mp_subscript implementations in the standard library try indexing before checking to see if the object is a slice object) b. a different error message may be wanted (e.g. the normal indexing related Type Error doesn't make sense for sequence repetition) c. a separate checking function would lead to repeating the check on common code paths (e.g. if an mp_subscript implementation did the type check first, and then PyNumber_Check did it again to see whether or not to raise an error) The output variable solves the problem nicely: either pass in NULL to get the default behaviour of raising a sequence indexing TypeError, or pass in a pointer to a C int in order to be told whether or not the typecheck succeeded without an exception actually being set if it fails (if the typecheck passes, but the actual call fails, the exception state is set as normal). Additionally, PyNumber_Index is redefined to raise an IndexError for values that cannot be represented as a Py_ssize_t. The choice of IndexError was made based on the dominant usage in the standard library (IndexError is the correct error to raise so that an mp_subscript implementation does the right thing). There are only a few places that need to override the IndexError to replace it with OverflowError (the length argument to slice.indices, sequence repetition, the mmap constructor), whereas all of the sequence objects (list, tuple, string, unicode, array), as well as PyObject_Get/Set/DelItem, need it to raise IndexError. Raising IndexError also benefits sequences implemented in Python, which can simply do: def __getitem__(self, idx): if isinstance(idx, slice): return self._get_slice(idx) idx = operator.index(idx) # Will raise IndexError on overflow A second API function PyNumber_SliceIndex is added so that the clipping semantics are still available where needed and _PyEval_SliceIndex is modified to use the new public API. This is exposed to Python code as operator.sliceindex(). With the redesigned C API, the *only* code that calls the nb_index slot directly is the two functions in abstract.c. Everything else uses one or the other of those interfaces. Code duplication was significantly reduced as a result, and it should be much easier for 3rd party C libraries to do what they need to do (i.e. implementing mp_subscript slots). 2. Redefine nb_index to return a PyObject * Returning the PyInt/PyLong objects directly from nb_index greatly simplified the implementation of the nb_index methods for the affected classes. For classic classes, instance_index could be modified to simply return the result of calling __index__, as could slot_nb_index for new-style classes. For the standard library classes, the existing int_int method, and the long_long method could be used instead of needing new functions. This convenience should hold true for extension classes - instead of needing to implement __index__ separately, they should be able to reuse their existing __int__ or __long__ implementations. The other benefit is that the logic to downconvert to Py_ssize_t that was formerly invoked by long's __index__ method is now instead invoked by PyNumber_Index and PyNumber_SliceIndex. This means that directly calling an __index__() method allows large long results to be passed through unaffected, but calling the indexing operator will raise IndexError if the long is outside the memory address space: (2 ** 100).__index__() == (2**100) # This works operator.index(2**100) # This raises IndexError The patch includes additions to test_index.py to cover these limit cases, as well as the necessary updates to the C API and operator module documentation. Cheers, Nick. [1] http://www.python.org/sf/1530738 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From david.nospam.hopwood at blueyonder.co.uk Sat Jul 29 16:09:03 2006 From: david.nospam.hopwood at blueyonder.co.uk (David Hopwood) Date: Sat, 29 Jul 2006 15:09:03 +0100 Subject: [Python-Dev] New miniconf module In-Reply-To: <44CADBB1.6090401@gmail.com> References: <Pine.LNX.4.64.0607261613130.6037@sylvain> <Pine.LNX.4.64.0607261402410.5956@sylvain> <44C7B8C2.1060904@blueyonder.co.uk> <Pine.LNX.4.64.0607261613130.6037@sylvain> <5.1.1.6.0.20060726180017.0262c990@sparrow.telecommunity.com> <Pine.LNX.4.64.0607270200300.6000@sylvain> <20060727103331.GA31912@code0.codespeak.net> <Pine.LNX.4.64.0607281438120.10450@sylvain> <44CADBB1.6090401@gmail.com> Message-ID: <44CB6BFF.4010700@blueyonder.co.uk> Nick Coghlan wrote: > Sylvain Fourmanoit wrote: >>Armin Rigo wrote: >> >>>If it goes in that direction, I'd suggest to rename the module to give >>>it a name closer to existing persistence-related modules already in the >>>stdlib. >> >>I am not especially fond of the current miniconf name either; I didn't >>find something more suitable, yet evocative of what it does; I would be >>glad to hear any suggestion you or the rest of the developers would have. > > pyson :) Following the pattern of JSON, it would be "PYON" (PYthon Object Notation). -- David Hopwood <david.nospam.hopwood at blueyonder.co.uk> From ncoghlan at iinet.net.au Sat Jul 29 16:19:27 2006 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun, 30 Jul 2006 00:19:27 +1000 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CB6B7D.1040001@gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <44CB6B7D.1040001@gmail.com> Message-ID: <44CB6E6F.5010507@iinet.net.au> Nick Coghlan wrote: > The other benefit is that the logic to downconvert to Py_ssize_t that > was formerly invoked by long's __index__ method is now instead invoked > by PyNumber_Index and PyNumber_SliceIndex. This means that directly > calling an __index__() method allows large long results to be passed > through unaffected, but calling the indexing operator will raise > IndexError if the long is outside the memory address space: > > (2 ** 100).__index__() == (2**100) # This works > operator.index(2**100) # This raises IndexError > > The patch includes additions to test_index.py to cover these limit > cases, as well as the necessary updates to the C API and operator module > documentation. I forgot to mention the main benefit of this: when working with a pseudo-sequence rather than a concrete one, __index__() can be used directly to ensure you are working with integral data types while still allowing access to the full range of representable integer values. operator.index is available for when what you have really is a concrete data set that is limited to the memory capacity of a single machine, and operator.sliceindex for when you want to clamp at the memory address space limits rather than raising an exception. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From tonynelson at georgeanelson.com Sat Jul 29 17:29:03 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Sat, 29 Jul 2006 11:29:03 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 Message-ID: <v04020a01c0f129ae3d64@[192.168.123.162]> I'm trying to write a test for my Socket Timeouts patch [1], which fixes signal handling (notably Ctl-C == SIGINT == KeyboarInterrupt) on socket operations using a timeout. I don't see a portable way to send a signal, and asking the test runner to press Ctl-C is a non-starter. A "real" signal is needed to interrupt the select() (or equivalent) call, because that's what wasn't being handled correctly. The bug should happen on the other platforms I don't know how to test on. Is there a portable way to send a signal? SIGINT would be best, but another signal (such as SIGALRM) would do, I think. If not, should I write the test to only work on systems implementing SIGALRM, the signal I'm using now, or implementing kill(), or what? [1] <http://www.python.org/sf/1519025> ____________________________________________________________________ TonyN.:' <mailto:tonynelson at georgeanelson.com> ' <http://www.georgeanelson.com/> From ncoghlan at iinet.net.au Sat Jul 29 18:55:13 2006 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun, 30 Jul 2006 02:55:13 +1000 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CB6B7D.1040001@gmail.com> References: <20060728101133.GA339@code0.codespeak.net> <44CB6B7D.1040001@gmail.com> Message-ID: <44CB92F1.9040304@iinet.net.au> Nick Coghlan wrote: > Armin Rigo wrote: >> Hi, >> >> There is an oversight in the design of __index__() that only just >> surfaced :-( It is responsible for the following behavior, on a 32-bit >> machine with >= 2GB of RAM: >> >> >>> s = 'x' * (2**100) # works! >> >>> len(s) >> 2147483647 >> >> This is because PySequence_Repeat(v, w) works by applying w.__index__ in >> order to call v->sq_repeat. However, __index__ is defined to clip the >> result to fit in a Py_ssize_t. This means that the above problem exists >> with all sequences, not just strings, given enough RAM to create such >> sequences with 2147483647 items. >> >> For reference, in 2.4 we correctly get an OverflowError. >> >> Argh! What should be done about it? > > I've now got a patch on SF that aims to fix this properly [1]. I revised this patch to further reduce the code duplication associated with the indexing code in the standard library. The patch now has three new functions in the abstract C API: PyNumber_Index (used in a dozen or so places) - raises IndexError on overflow PyNumber_AsSsize_t (used in 3 places) - raises OverflowError on overflow PyNumber_AsClippedSsize_t() (used once, by _PyEval_SliceIndex) - clips to PY_SSIZE_T_MIN/MAX on overflow All 3 have an int * output argument allowing type errors to be flagged directly to the caller rather than through PyErr_Occurred(). Of the 3, only PyNumber_Index is exposed through the operator module. Probably the most interesting thing now would be for Travis to review it, and see whether it makes things easier to handle for the Numeric scalar types (given the amount of code the patch deleted from the builtin and standard library data types, hopefully the benefits to Numeric will be comparable). Cheers, Nick. [1] http://www.python.org/sf/1530738 -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From nnorwitz at gmail.com Sat Jul 29 21:31:29 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sat, 29 Jul 2006 12:31:29 -0700 Subject: [Python-Dev] test_uuid Message-ID: <ee2a432c0607291231s6ad5a9bdt8a3cfa599a52acb3@mail.gmail.com> Ping, I just checked in a change to disable testing 2 uuid functions (_ifconfig_get_node and unixdll_getnode) that fail on many platforms. Here's the message: """ Disable these tests until they are reliable across platforms. These problems may mask more important, real problems. One or both methods are known to fail on: Solaris, OpenBSD, Debian, Ubuntu. They pass on Windows and some Linux boxes. """ Can you fix these issues or at least provide guidance how they should be fixed? n From python-dev at zesty.ca Sat Jul 29 22:00:50 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Sat, 29 Jul 2006 15:00:50 -0500 (CDT) Subject: [Python-Dev] uuid test suite failing In-Reply-To: <ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com> References: <eaamnu$mrq$1@sea.gmane.org> <20060727163647.GA4175@rogue.amk.ca> <ee2a432c0607280024j56bf7facmd9ebdd26aafc013f@mail.gmail.com> Message-ID: <Pine.LNX.4.58.0607291458310.23895@server1.LFW.org> On Fri, 28 Jul 2006, Neal Norwitz wrote: > This only fixes 1 of the 2 failures in test_uuid. The other one is > due to _unixdll_getnode() failing. This is because > _uuid_generate_time is None because we couldn't find it in the uuid > library. This is just broken, not sure if it's the code or the test > though. We should handle the case if _uuid_generate_time and the > others are None better. I don't know what to do in this case. The design intention is this: each of the various *_getnode() functions is supposed to work on the platform for which it was written. For example, _windll_getnode() is supposed to work on Windows, and will raise an exception on other platforms; if it raises an exception on Windows, something is wrong (the code's expectations of the OS are not met). When uuid_generate_time is unavailable, _unixdll_getnode() is supposed to fail. The getnode() function is just supposed to get an available MAC address; that's why it catches any exceptions raised by the *_getnode() functions. -- ?!ng From python-dev at zesty.ca Sat Jul 29 22:05:16 2006 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Sat, 29 Jul 2006 15:05:16 -0500 (CDT) Subject: [Python-Dev] test_uuid In-Reply-To: <ee2a432c0607291231s6ad5a9bdt8a3cfa599a52acb3@mail.gmail.com> References: <ee2a432c0607291231s6ad5a9bdt8a3cfa599a52acb3@mail.gmail.com> Message-ID: <Pine.LNX.4.58.0607291501220.23895@server1.LFW.org> On Sat, 29 Jul 2006, Neal Norwitz wrote: > I just checked in a change to disable testing 2 uuid functions > (_ifconfig_get_node and unixdll_getnode) that fail on many platforms. > Here's the message: > > """ > Disable these tests until they are reliable across platforms. These > problems may mask more important, real problems. > > One or both methods are known to fail on: Solaris, OpenBSD, Debian, Ubuntu. > They pass on Windows and some Linux boxes. > """ _ifconfig_get_node() should work on all Linuxes. (Thanks for fixing it to work on more types of Unix.) It's okay for unixdll_getnode to fail when the necessary shared library is not available. Ideally, test_uuid should serve as a record of which platforms we expect these routines to work on. The uuid module as a whole isn't broken if one of these routines fails; it just means that we don't have complete platform coverage and/or test_uuid has inaccurate expectations of which routines work on which platforms. -- ?!ng From jcarlson at uci.edu Sat Jul 29 23:38:38 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 29 Jul 2006 14:38:38 -0700 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <v04020a01c0f129ae3d64@[192.168.123.162]> References: <v04020a01c0f129ae3d64@[192.168.123.162]> Message-ID: <20060729143048.E630.JCARLSON@uci.edu> Tony Nelson <tonynelson at georgeanelson.com> wrote: > > I'm trying to write a test for my Socket Timeouts patch [1], which fixes > signal handling (notably Ctl-C == SIGINT == KeyboarInterrupt) on socket > operations using a timeout. I don't see a portable way to send a signal, > and asking the test runner to press Ctl-C is a non-starter. A "real" > signal is needed to interrupt the select() (or equivalent) call, because > that's what wasn't being handled correctly. The bug should happen on the > other platforms I don't know how to test on. > > Is there a portable way to send a signal? SIGINT would be best, but > another signal (such as SIGALRM) would do, I think. According to my (limited) research on signals, Windows signal support is horrible. I have not been able to have Python send signals of any kind other than SIGABRT, and then only to the currently running process, which kills it (regardless of whether you have a signal handler or not). > If not, should I write the test to only work on systems implementing > SIGALRM, the signal I'm using now, or implementing kill(), or what? I think that most non-Windows platforms should have non-braindead signal support, though the signal module seems to be severely lacking in sending any signal except for SIGALRM, and the os module has its fingers on SIGABRT. If someone is looking for a project for 2.6 that digs into all sorts of platform-specific nastiness, they could add actual signal sending to the signal module (at least for unix systems). - Josiah From exarkun at divmod.com Sat Jul 29 23:44:28 2006 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Sat, 29 Jul 2006 17:44:28 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <20060729143048.E630.JCARLSON@uci.edu> Message-ID: <20060729214428.1717.1630815704.divmod.quotient.6244@ohm> On Sat, 29 Jul 2006 14:38:38 -0700, Josiah Carlson <jcarlson at uci.edu> wrote: > >If someone is looking for a project for 2.6 that digs into all sorts of >platform-specific nastiness, they could add actual signal sending to the >signal module (at least for unix systems). > Maybe I am missing something obvious, but what is necessary beyond os.kill()? What /would/ be useful is a complete sigaction wrapper, but that's a completely separate topic. Jean-Paul From tonynelson at georgeanelson.com Sun Jul 30 00:18:41 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Sat, 29 Jul 2006 18:18:41 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <20060729143048.E630.JCARLSON@uci.edu> References: <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> Message-ID: <v04020a01c0f18c283570@[192.168.123.162]> At 2:38 PM -0700 7/29/06, Josiah Carlson wrote: >Tony Nelson <tonynelson at georgeanelson.com> wrote: >> >> I'm trying to write a test for my Socket Timeouts patch [1], which fixes >> signal handling (notably Ctl-C == SIGINT == KeyboarInterrupt) on socket >> operations using a timeout. I don't see a portable way to send a signal, >> and asking the test runner to press Ctl-C is a non-starter. A "real" >> signal is needed to interrupt the select() (or equivalent) call, because >> that's what wasn't being handled correctly. The bug should happen on the >> other platforms I don't know how to test on. >> >> Is there a portable way to send a signal? SIGINT would be best, but >> another signal (such as SIGALRM) would do, I think. > >According to my (limited) research on signals, Windows signal support is >horrible. I have not been able to have Python send signals of any kind >other than SIGABRT, and then only to the currently running process, >which kills it (regardless of whether you have a signal handler or not). Hmm, OK, darn, thanks. MSWindows does allow users to press Ctl-C to send a KeyboardInterrupt, so it's just too bad if I can't find a way to test it from a script. >> If not, should I write the test to only work on systems implementing >> SIGALRM, the signal I'm using now, or implementing kill(), or what? > >I think that most non-Windows platforms should have non-braindead signal >support, though the signal module seems to be severely lacking in >sending any signal except for SIGALRM, and the os module has its fingers >on SIGABRT. The test now checks "hasattr(signal, 'alarm')" before proceeding, so at least it won't die horribly. >If someone is looking for a project for 2.6 that digs into all sorts of >platform-specific nastiness, they could add actual signal sending to the >signal module (at least for unix systems). Isn't signal sending the province of kill (2) (or os.kill()) in python)? Not that I know much about it. BTW, I picked SIGALRM because I could do it all with one thread. Reading POSIX, ISTM that if I sent the signal from another thread, it would bounce off that thread to the main thread during the call to kill(), at which point I got the willies. OTOH, if kill() is more widely available than alarm(), I'll give it a try, but going by the docs, I'd say it isn't. ____________________________________________________________________ TonyN.:' <mailto:tonynelson at georgeanelson.com> ' <http://www.georgeanelson.com/> From martin at v.loewis.de Sun Jul 30 09:42:14 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 Jul 2006 09:42:14 +0200 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <v04020a01c0f18c283570@[192.168.123.162]> References: <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> Message-ID: <44CC62D6.9070703@v.loewis.de> Tony Nelson schrieb: > Hmm, OK, darn, thanks. MSWindows does allow users to press Ctl-C to send a > KeyboardInterrupt, so it's just too bad if I can't find a way to test it > from a script. You can use GenerateConsoleCtrlEvent to send Ctrl-C to all processes that share the console of the calling process. > BTW, I picked SIGALRM because I could do it all with one thread. Reading > POSIX, ISTM that if I sent the signal from another thread, it would bounce > off that thread to the main thread during the call to kill(), at which > point I got the willies. OTOH, if kill() is more widely available than > alarm(), I'll give it a try, but going by the docs, I'd say it isn't. Indeed, alarm should be available on any POSIX system. Regards, Martin From phd at mail2.phd.pp.ru Sun Jul 30 11:29:57 2006 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Sun, 30 Jul 2006 13:29:57 +0400 Subject: [Python-Dev] first draft of bug guidelines for www.python.org/dev/ In-Reply-To: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> References: <bbaeab100607202052t68ac220cs70b32a7c6a3190a2@mail.gmail.com> Message-ID: <20060730092957.GB29899@phd.pp.ru> On Thu, Jul 20, 2006 at 08:52:34PM -0700, Brett Cannon wrote: > * Summary > A one-line describing the problem so as to make it easy for > developers to spot whether they have the expertise needed to work on > the bug. Summary is also displayed as a title on index and search pages, so it is important to make the summary right - short and descriptive. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From martin at v.loewis.de Sun Jul 30 15:14:13 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 Jul 2006 15:14:13 +0200 Subject: [Python-Dev] Which version of distutils to ship with Python 2.5? In-Reply-To: <200607271701.58223.anthony@interlink.com.au> References: <43aa6ff70607261924w65e01325o854f87d4c109026a@mail.gmail.com> <44C85FF2.3030701@v.loewis.de> <200607271701.58223.anthony@interlink.com.au> Message-ID: <44CCB0A5.30706@v.loewis.de> Anthony Baxter schrieb: >> In any case, I bumped the version number to 2.5, according to the >> policy discussed in >> > > Could this not simply use the Python version number directly, instead? See the prior discussion at http://mail.python.org/pipermail/distutils-sig/2005-January/004366.html Some people still believe (at least, believed in January 2005), that distutils is developed independently of Python, and thus deserves its own version number. Of course, Andrew Kuchling officially declared in r1982 of pep 291 that there won't be any further stand-alone distutils releases, and therefore, backwards compatibility with 2.1 is not necessary anymore. So I changed distutils.__version__ again, to be derived from sys.version_info. I left the numerous comments still in distutils that compatibility with 2.1 is desired. We should remove these after 2.5 is released (or perhaps even before that). Regards, Martin From jcarlson at uci.edu Sun Jul 30 18:50:36 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 30 Jul 2006 09:50:36 -0700 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <20060729214428.1717.1630815704.divmod.quotient.6244@ohm> References: <20060729143048.E630.JCARLSON@uci.edu> <20060729214428.1717.1630815704.divmod.quotient.6244@ohm> Message-ID: <20060730093610.E634.JCARLSON@uci.edu> Jean-Paul Calderone <exarkun at divmod.com> wrote: > > On Sat, 29 Jul 2006 14:38:38 -0700, Josiah Carlson <jcarlson at uci.edu> wrote: > > > >If someone is looking for a project for 2.6 that digs into all sorts of > >platform-specific nastiness, they could add actual signal sending to the > >signal module (at least for unix systems). > > Maybe I am missing something obvious, but what is necessary beyond > os.kill()? I note that os.kill() does everything necessary for posix systems. I didn't notice that it took an argument for the kind of signal. A new project for someone: combine all of the methods available to Windows into a single function. > What /would/ be useful is a complete sigaction wrapper, but that's a > completely separate topic. Like atexit, only a stack per signal? - Josiah From tonynelson at georgeanelson.com Sun Jul 30 19:10:32 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Sun, 30 Jul 2006 13:10:32 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <44CC62D6.9070703@v.loewis.de> References: <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> Message-ID: <v04020a01c0f286dddce0@[192.168.123.162]> At 9:42 AM +0200 7/30/06, Martin v. L?wis wrote: >Tony Nelson schrieb: >> Hmm, OK, darn, thanks. MSWindows does allow users to press Ctl-C to send a >> KeyboardInterrupt, so it's just too bad if I can't find a way to test it >> from a script. > >You can use GenerateConsoleCtrlEvent to send Ctrl-C to all processes >that share the console of the calling process. That looks like it would work, but it seems prone to overkill. To avoid killing all the processes running from a console, the test would need to be run in a subprocess in a new process group. If the test simply sends the event to its own process, all the other processes in its process group would receive the event as well, and probably die. I would expect that all the processes sharing the console would die, but even if they didn't when I tried it, I couldn't be sure that it wouldn't happen elsewhere, say when run from a .bat file. Martin, your advice is usually spot-on, but I don't always understand it. Maybe using it here is just complicated. I expect that GenerateConsoleCtrlEvent() can be called through the ctypes module, though that would make backporting the test to 2.4 a bit more difficult. It looks like the subprocess module can be passed the needed creation flag to make a new process group. The subprocess can send the event to itself, and could return the test result in its result code, so that part isn't so bad. To avoid adding a new file to the distribution, test_socket.test_main() could be modified to look for a command line argument requesting the particular test action. >> BTW, I picked SIGALRM because I could do it all with one thread. Reading >> POSIX, ISTM that if I sent the signal from another thread, it would bounce >> off that thread to the main thread during the call to kill(), at which >> point I got the willies. OTOH, if kill() is more widely available than >> alarm(), I'll give it a try, but going by the docs, I'd say it isn't. > >Indeed, alarm should be available on any POSIX system. Well, if alarm() is available, then the test will work. If not, it will be silently skipped, as are some other tests already in test_socket.py. I can't offhand tell if MSWindows supports alarm(), but RiscOS and OS2 do not. ____________________________________________________________________ TonyN.:' <mailto:tonynelson at georgeanelson.com> ' <http://www.georgeanelson.com/> From exarkun at divmod.com Sun Jul 30 19:20:54 2006 From: exarkun at divmod.com (Jean-Paul Calderone) Date: Sun, 30 Jul 2006 13:20:54 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <20060730093610.E634.JCARLSON@uci.edu> Message-ID: <20060730172054.1717.102253317.divmod.quotient.7639@ohm> On Sun, 30 Jul 2006 09:50:36 -0700, Josiah Carlson <jcarlson at uci.edu> wrote: > >Jean-Paul Calderone <exarkun at divmod.com> wrote: >> >> On Sat, 29 Jul 2006 14:38:38 -0700, Josiah Carlson <jcarlson at uci.edu> wrote: >> > >> >If someone is looking for a project for 2.6 that digs into all sorts of >> >platform-specific nastiness, they could add actual signal sending to the >> >signal module (at least for unix systems). >> >> Maybe I am missing something obvious, but what is necessary beyond >> os.kill()? > >I note that os.kill() does everything necessary for posix systems. >I didn't notice that it took an argument for the kind of signal. > >A new project for someone: combine all of the methods available to >Windows into a single function. > >> What /would/ be useful is a complete sigaction wrapper, but that's a >> completely separate topic. > >Like atexit, only a stack per signal? I just mean a complete wrapping of sigaction(2). In particular, I need this functionality to properly install a SIGCHLD handler which does not interfer with various I/O functions by installing the handler with the SA_RESTART flag, which is not currently possible using the signal.signal function. Jean-Paul From barry at python.org Sun Jul 30 21:44:08 2006 From: barry at python.org (Barry Warsaw) Date: Sun, 30 Jul 2006 15:44:08 -0400 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org> References: <44C9A184.7030701@ewtllc.com> <200607281539.50794.anthony@interlink.com.au> <BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org> Message-ID: <3E70BA16-8DAA-487D-9901-B1597D912222@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Re: Python 2.5 compatibility On Jul 28, 2006, at 8:57 AM, Barry Warsaw wrote: > +1. It would give me more type to port and test a few of my > applications to the new version. > > I'm still working on Mailman but the most painful thing so far has > been the conversion of exceptions to new-style classes, and even > that wasn't /too/ painful. I believe I've finished porting Mailman to Python 2.5. None of the issues were insurmountable, but here they are FTR: 1) Exceptions are new-style classes but Mailman was doing one specific test against the type of an object to see if it needed to be instantiated. This test was written as: if isinstance(obj, ClassType) which fails in Python 2.5. I actually rewrote it like so: if isinstance(obj, ClassType) or isinstance(obj, type(type)) in MM2.1 because it has to be backward compatible to Python 2.1. 2) There was one place where I was commonly raising a string and that generates deprecation warnings, so I changed that to a class. There are a few other legacy string exceptions defined in the code, but they are rarely used and should all get rewritten as well. 3) Cookie.py's repr changed so that trailing semicolons are no longer output on the end of text lines. I understand this change was made so that Python cookies were more RFC compliant, but it broke some homegrown cookie text splitting we were doing. I changed this code to split on lines first. All in all, not too bad although the Cookie.py change took a while to track down! - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRM0MDnEjvBPtnXfVAQJ9ZAP/VdtM79SXgx7s/X0aEIu4HDZva7TkYyi6 dRzlgAtEV5BN1yYn+vzw8PBCtdy+9N3yYtv/zqdQP54mZDjsaGaNw6MiS0jsETRy 248hj3otL/00WTrKWh8/OvDlLW8KUNQI4MWBOMKJ/TqYW5Es4fJGEMtbO/xqGXXD /wgWmmLOOAE= =Mu8m -----END PGP SIGNATURE----- From fdrake at acm.org Sun Jul 30 21:55:40 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sun, 30 Jul 2006 15:55:40 -0400 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <3E70BA16-8DAA-487D-9901-B1597D912222@python.org> References: <44C9A184.7030701@ewtllc.com> <BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org> <3E70BA16-8DAA-487D-9901-B1597D912222@python.org> Message-ID: <200607301555.40541.fdrake@acm.org> On Sunday 30 July 2006 15:44, Barry Warsaw wrote: > if isinstance(obj, ClassType) or isinstance(obj, type(type)) Looks like you've got a possible name clash in the second isinstance. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From g.brandl at gmx.net Sun Jul 30 22:17:41 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 30 Jul 2006 22:17:41 +0200 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <3E70BA16-8DAA-487D-9901-B1597D912222@python.org> References: <44C9A184.7030701@ewtllc.com> <200607281539.50794.anthony@interlink.com.au> <BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org> <3E70BA16-8DAA-487D-9901-B1597D912222@python.org> Message-ID: <eaj42s$6n0$1@sea.gmane.org> Barry Warsaw wrote: > if isinstance(obj, ClassType) > > which fails in Python 2.5. I actually rewrote it like so: > > if isinstance(obj, ClassType) or isinstance(obj, type(type)) The second "type" seems to be superfluous. ;) Georg From fdrake at acm.org Sun Jul 30 22:27:16 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sun, 30 Jul 2006 16:27:16 -0400 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <eaj42s$6n0$1@sea.gmane.org> References: <44C9A184.7030701@ewtllc.com> <3E70BA16-8DAA-487D-9901-B1597D912222@python.org> <eaj42s$6n0$1@sea.gmane.org> Message-ID: <200607301627.17309.fdrake@acm.org> On Sunday 30 July 2006 16:17, Georg Brandl wrote: > The second "type" seems to be superfluous. ;) I was thinking it suggested there was a local named "type". But if not, yeah. I get the impression Barry's pretty new to this "Python thing." Wonder what he's been up to. ;-) -Fred -- Fred L. Drake, Jr. <fdrake at acm.org> From barry at python.org Sun Jul 30 23:07:22 2006 From: barry at python.org (Barry Warsaw) Date: Sun, 30 Jul 2006 17:07:22 -0400 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <200607301627.17309.fdrake@acm.org> References: <44C9A184.7030701@ewtllc.com> <3E70BA16-8DAA-487D-9901-B1597D912222@python.org> <eaj42s$6n0$1@sea.gmane.org> <200607301627.17309.fdrake@acm.org> Message-ID: <ADB1B68C-7C90-4466-BE43-921FDFA7ECEE@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Jul 30, 2006, at 4:27 PM, Fred L. Drake, Jr. wrote: > On Sunday 30 July 2006 16:17, Georg Brandl wrote: >> The second "type" seems to be superfluous. ;) > > I was thinking it suggested there was a local named "type". But if > not, yeah. > > I get the impression Barry's pretty new to this "Python thing." > Wonder what > he's been up to. ;-) As I mentioned, this has to be compatible with Python 2.1: Python 2.1.3+ (#1, Apr 25 2005, 22:52:02) [GCC 3.3.5-20050130 (Gentoo Linux 3.3.5.20050130-r1, ssp-3.3.5.20050130-1, pie- on linux2 Type "copyright", "credits" or "license" for more information. >>> isinstance(Exception, type) Traceback (most recent call last): File "<stdin>", line 1, in ? TypeError: isinstance() arg 2 must be a class or type >>> isinstance(Exception, type(type)) 0 Python 2.5b2 (trunk:50835, Jul 25 2006, 23:27:51) [GCC 3.4.6 (Gentoo 3.4.6-r1, ssp-3.4.5-1.0, pie-8.7.9)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> isinstance(Exception, type) True >>> isinstance(Exception, type(type)) True I thought that was rather clever actually. :) - -Barry -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.3 (Darwin) iQCVAwUBRM0fj3EjvBPtnXfVAQI1AQP+ID6BSbJ/4TL7cizvMjrxHD6JVjFKBzD6 7FCwXpvELQt7vlDWGrXWi+Lai/93nGqD42VYRSgtHqFP2gMYKEkM+TaLl91YFuSh B6jO7l5wW7SGlyLQQiibmPBp8uGDG30F1ylM9e9y9c69Hy1LJEa1sG8/AS/FiA+n fpQ/WlTNRvA= =uGDZ -----END PGP SIGNATURE----- From martin at v.loewis.de Sun Jul 30 23:34:30 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 Jul 2006 23:34:30 +0200 Subject: [Python-Dev] Py2.5 release schedule In-Reply-To: <200607301555.40541.fdrake@acm.org> References: <44C9A184.7030701@ewtllc.com> <BBBDED8E-ACDD-4176-985A-144188B52F1E@python.org> <3E70BA16-8DAA-487D-9901-B1597D912222@python.org> <200607301555.40541.fdrake@acm.org> Message-ID: <44CD25E6.5040608@v.loewis.de> Fred L. Drake, Jr. schrieb: > On Sunday 30 July 2006 15:44, Barry Warsaw wrote: > > if isinstance(obj, ClassType) or isinstance(obj, type(type)) > > Looks like you've got a possible name clash in the second isinstance. ;-) Nah, that's rather an entry to the obfuscated Python contest. The two occurrences of type really mean to refer to the same thing; this is the test whether obj _is a_ new-style class. Normally, you would write isinstance(obj, type), but that gives a TypeError in 2.1 (isinstance() arg 2 must be a class or type). In 2.1, type(type) is FunctionType, so the test should fail (in the context, as obj ought to be a string, an exception object, or an exception type). In 2.2 and later, we have >>> type(type) is type 1 # sometimes True instead I think I would have rewritten as try: # Instantiate it if possible and necessary exc = exc() except AttributeError: # no __call__; it's already an object pass (assuming that the mailman exceptions don't have __call__) or as if not isinstance(exc, Exception): exc = exc() (assuming that string exceptions are gone, but the code below already assumes that exc should be an object that supports exc.reason_notice()) Regards, Martin From martin at v.loewis.de Sun Jul 30 23:42:54 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 Jul 2006 23:42:54 +0200 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <v04020a01c0f286dddce0@[192.168.123.162]> References: <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> Message-ID: <44CD27DE.5060804@v.loewis.de> Tony Nelson schrieb: >> You can use GenerateConsoleCtrlEvent to send Ctrl-C to all processes >> that share the console of the calling process. [...] > Martin, your advice is usually spot-on, but I don't always understand it. > Maybe using it here is just complicated. This was really just in response to your remark that you couldn't find a way to send Ctrl-C programmatically. I researched (in the C library sources) how SIGINT was *generated* (through SetConsoleCtrlHandler), and that let me to a way to generate I didn't mean to suggest that you *should* use GenerateConsoleCtrlEvent, only that you could if you wanted to. > I expect that > GenerateConsoleCtrlEvent() can be called through the ctypes module, though > that would make backporting the test to 2.4 a bit more difficult. Well, if there was general utility to that API, I would prefer exposing it in the nt module. It doesn't quite fit into kill(2), as it doesn't allow to specify a pid of the target process, so perhaps it doesn't have general utility. In any case, that would have to wait for 2.6. Regards, Martin From tonynelson at georgeanelson.com Mon Jul 31 01:23:23 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Sun, 30 Jul 2006 19:23:23 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <44CD27DE.5060804@v.loewis.de> References: <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> Message-ID: <v04020a04c0f2da888957@[192.168.123.162]> At 11:42 PM +0200 7/30/06, Martin v. L?wis wrote: >Tony Nelson schrieb: >>> You can use GenerateConsoleCtrlEvent to send Ctrl-C to all processes >>> that share the console of the calling process. >[...] >> Martin, your advice is usually spot-on, but I don't always understand it. >> Maybe using it here is just complicated. > >This was really just in response to your remark that you couldn't >find a way to send Ctrl-C programmatically. I researched (in >the C library sources) how SIGINT was *generated* (through >SetConsoleCtrlHandler), and that let me to a way to generate [one.] Well, fine work there! >I didn't mean to suggest that you *should* use GenerateConsoleCtrlEvent, >only that you could if you wanted to. Hmm. Well, it would make the test possible on MSWindows as well as on OS's implementing alarm(2). If I figure out how to build Python on MSWindows, I might give it a try. I tried to get MSVC 7.1 via the .Net SDK, but it installed VS 8 instead, so I'm not quite sure how to proceed. >> I expect that >> GenerateConsoleCtrlEvent() can be called through the ctypes module, though >> that would make backporting the test to 2.4 a bit more difficult. > >Well, if there was general utility to that API, I would prefer exposing >it in the nt module. It doesn't quite fit into kill(2), as it doesn't >allow to specify a pid of the target process, so perhaps it doesn't >have general utility. In any case, that would have to wait for 2.6. A Process Group ID is the PID of the first process put in it, so it's sort of a PID. It just means a collection of processes, probably more than one. It seems to be mostly applicable to MSWindows, and isn't a suitable way to implement a form of kill(2). I hope that the Socket Timeouts patch 1519025 can make it into 2.5, or 2.5.1, as it is a bug fix. As such, it would probably be better to punt the test on MSWindows than to do a tricky fancy test that might have its own issues. ____________________________________________________________________ TonyN.:' <mailto:tonynelson at georgeanelson.com> ' <http://www.georgeanelson.com/> From tonynelson at georgeanelson.com Mon Jul 31 04:04:59 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Sun, 30 Jul 2006 22:04:59 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <v04020a04c0f2da888957@[192.168.123.162]> References: <44CD27DE.5060804@v.loewis.de> <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> Message-ID: <v04020a06c0f315064900@[192.168.123.162]> At 7:23 PM -0400 7/30/06, Tony Nelson wrote: ... >...I tried to get MSVC 7.1 via the .Net SDK, but it >installed VS 8 instead, so I'm not quite sure how to proceed. ... David Murmann suggested off-list that I'd probably installed the 2.0 .Net SDK, and that I should install the 1.1 .Net SDK, which is the correct one. Now I can try to build Python on MSWindows. ____________________________________________________________________ TonyN.:' <mailto:tonynelson at georgeanelson.com> ' <http://www.georgeanelson.com/> From martin at v.loewis.de Mon Jul 31 04:34:03 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 31 Jul 2006 04:34:03 +0200 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <v04020a04c0f2da888957@[192.168.123.162]> References: <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a04c0f2da888957@[192.168.123.162]> Message-ID: <44CD6C1B.6040401@v.loewis.de> Tony Nelson schrieb: > Hmm. Well, it would make the test possible on MSWindows as well as on OS's > implementing alarm(2). If I figure out how to build Python on MSWindows, I > might give it a try. I tried to get MSVC 7.1 via the .Net SDK, but it > installed VS 8 instead, so I'm not quite sure how to proceed. The .NET SDK (any version) is not suitable to build Python. You really need VS 2003; if you don't have it anymore, you might be able to find a copy of the free version of the VC Toolkit 2003 (VCToolkitSetup.exe) somewhere. Of course, just for testing, you can also install VS Express 2005, and use the PCbuild8 projects directory; these changes should work the same under both versions. Regards, Martin From tonynelson at georgeanelson.com Mon Jul 31 06:39:07 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Mon, 31 Jul 2006 00:39:07 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <44CD6C1B.6040401@v.loewis.de> References: <v04020a04c0f2da888957@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a04c0f2da888957@[192.168.123.162]> Message-ID: <v04020a07c0f320d50fb8@[192.168.123.162]> At 4:34 AM +0200 7/31/06, Martin v. L?wis wrote: >Tony Nelson schrieb: >>Hmm. Well, it would make the test possible on MSWindows as well as on >>OS's implementing alarm(2). If I figure out how to build Python on >>MSWindows, I might give it a try. I tried to get MSVC 7.1 via the .Net >>SDK, but it installed VS 8 instead, so I'm not quite sure how to proceed. > >The .NET SDK (any version) is not suitable to build Python. I do see the warning in the instructions about it not be an optimizing compiler. I've managed to build python.exe and the rt.bat tests mostly work -- 2 tests fail, test_popen, and test_cmd_line because of popen() failing. Hmm, actually, this might be a real problem with the MSWindows version of posix_popen() in Modules/posixmodule.c. The path to my built python.exe is: "E:\Documents and Settings\Tony Nelson\My Documents\Python\pydev\trunk\PCBuild\python.exe" (lots of spaces in it). It seems properly quoted in the test and when I do it by hand, but in a call to popen() it doesn't work: popen('"E:\Documents and Settings\Tony Nelson\My Documents\Python\pydev\trunk\PCBuild\python.exe" -c "import sys;sys.version_info"') The returned file object repr resembles one that does work. If I just use "python.exe" from within the PCBuild directory: popen('python.exe -c "import sys;sys.version_info"') I get the right version, and that's the only 2.5b2 python I've got, so the built python must be working, but the path, even quoted, isn't accepted by MSWindows XP SP2. Should I report a bug? It may well just be MSWindows weirdness, and not something that posixmodule.c can do anything about. OTOH, it does work from the command line. I'll bet I wouldn't have seen a thing if I'd checked out to "E:\pydev" instead. >You really need VS 2003; if you don't have it anymore, you might be able >to find a copy of the free version of the VC Toolkit 2003 >(VCToolkitSetup.exe) somewhere. I really never had VS 2003. It doesn't appear to be on microsoft.com anymore. I'm reluctant to try to steal a copy. >Of course, just for testing, you can also install VS Express 2005, and >use the PCbuild8 projects directory; these changes should work the >same under both versions. I'll try that if I have any real trouble with the non-optimized python or if you insist that it's necessary. ____________________________________________________________________ TonyN.:' <mailto:tonynelson at georgeanelson.com> ' <http://www.georgeanelson.com/> From greg at electricrain.com Mon Jul 31 06:48:55 2006 From: greg at electricrain.com (Gregory P. Smith) Date: Sun, 30 Jul 2006 21:48:55 -0700 Subject: [Python-Dev] httplib and bad response chunking In-Reply-To: <20060726023213.GA3106@cthulhu.gerg.ca> References: <20060726023213.GA3106@cthulhu.gerg.ca> Message-ID: <20060731044855.GK22921@zot.electricrain.com> On Tue, Jul 25, 2006 at 10:32:13PM -0400, Greg Ward wrote: > > what I discovered in the wild the other day was a response like this: > > 0005\r\nabcd\n\r\n0004\r\nabc\n\r\n\r\n > > i.e. the chunk-size for the terminating empty chunk was missing. > This cause httplib.py to blow up with ValueError because it tried to > call > > int(line, 16) > > assuming that 'line' contained a hex number, when in fact it was the > empty string. Oops. > > IMHO the minimal fix is to turn ValueError into HTTPException (or a > subclass thereof); httplib should not raise ValueError just because some > server sends a bad response. (The server in question was Apache 2.0.52 > running PHP 4.3.9 sending a big hairy error page because the database > was down.) IMNSHO httplib should be fixed and this shouldn't be an error at all as its in the wild and will only show up more and more in the future. Plus file a bug with the apache or php project as appropriate for having a non-RFC compliant response. This is part of the good old network programming addage of being lenient in what you accept. -g From tonynelson at georgeanelson.com Mon Jul 31 06:58:23 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Mon, 31 Jul 2006 00:58:23 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <v04020a07c0f320d50fb8@[192.168.123.162]> References: <44CD6C1B.6040401@v.loewis.de> <v04020a04c0f2da888957@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a04c0f2da888957@[192.168.123.162]> Message-ID: <v04020a08c0f33d81cce8@[192.168.123.162]> At 12:39 AM -0400 7/31/06, Tony Nelson wrote: > popen('"E:\Documents and Settings\Tony Nelson\My >Documents\Python\pydev\trunk\PCBuild\python.exe" -c "import >sys;sys.version_info"') Ehh, I must admit that I retyped that. Obviously what I typed would not work, but what I used was: python = '"' + sys.executable + '"' popen(python + ' -c "import sys;sys.version_info"' So there wasn't a problem with backslashes. I've also been using raw strings. And, as I said, the file objects looked OK, with backslashes where they should be. Sorry for the mistyping. ____________________________________________________________________ TonyN.:' <mailto:tonynelson at georgeanelson.com> ' <http://www.georgeanelson.com/> From tonynelson at georgeanelson.com Mon Jul 31 07:14:36 2006 From: tonynelson at georgeanelson.com (Tony Nelson) Date: Mon, 31 Jul 2006 01:14:36 -0400 Subject: [Python-Dev] Testing Socket Timeouts patch 1519025 In-Reply-To: <v04020a08c0f33d81cce8@[192.168.123.162]> References: <v04020a07c0f320d50fb8@[192.168.123.162]> <44CD6C1B.6040401@v.loewis.de> <v04020a04c0f2da888957@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f129ae3d64@[192.168.123.162]> <v04020a01c0f18c283570@[192.168.123.162]> <v04020a01c0f286dddce0@[192.168.123.162]> <v04020a04c0f2da888957@[192.168.123.162]> Message-ID: <v04020a09c0f34051763e@[192.168.123.162]> At 12:58 AM -0400 7/31/06, Tony Nelson wrote: >At 12:39 AM -0400 7/31/06, Tony Nelson wrote: > >> popen('"E:\Documents and Settings\Tony Nelson\My >>Documents\Python\pydev\trunk\PCBuild\python.exe" -c "import >>sys;sys.version_info"') > >Ehh, I must admit that I retyped that. Obviously what I typed would not >work, but what I used was: > > python = '"' + sys.executable + '"' > popen(python + ' -c "import sys;sys.version_info"' > >So there wasn't a problem with backslashes. I've also been using raw >strings. And, as I said, the file objects looked OK, with backslashes >where they should be. Sorry for the mistyping. OK, I recognize the bug now. It's that quote parsing bug in MSWindows (which I can find again if you want) which can be worked around by using an extra quote at the front (and maybe also the back): popen('""E:\Documents ... Not really a bug in Python at all. ____________________________________________________________________ TonyN.:' <mailto:tonynelson at georgeanelson.com> ' <http://www.georgeanelson.com/> From gward-1337f07a94b43060ff5c1ea922ed93d6 at python.net Mon Jul 31 04:09:02 2006 From: gward-1337f07a94b43060ff5c1ea922ed93d6 at python.net (Greg Ward) Date: Sun, 30 Jul 2006 22:09:02 -0400 Subject: [Python-Dev] httplib and bad response chunking In-Reply-To: <Pine.LNX.4.64.0607282251320.8644@localhost> References: <20060726023213.GA3106@cthulhu.gerg.ca> <Pine.LNX.4.64.0607282251320.8644@localhost> Message-ID: <20060731020902.GA2950@cthulhu.gerg.ca> [me, on 25 July] > I have > discovered other hypothetical cases of bad chunking that cause httplib > to go into an infinite loop or block forever on socket.readline(). > Should we worry about those cases as well, despite not having seen them > happen in the wild? More annoying, I can reproduce the "block forever" > case using a real socket, but not using the StringIO-based FakeSocket > class in test_httplib. [John J Lee] > They have been seen in the wild :-) > > http://python.org/sf/1411097 Thanks -- that was really all the encouragement I needed to keep banging away at this bug. Did you look at the crude attempt at testing for this bug that I hacked into test_httplib.py? I posted it to bug #1486335 here: http://sourceforge.net/tracker/download.php?group_id=5470&atid=105470&file_id=186245&aid=1486335 The idea is simple: put various chunked responses into strings and then feed those strings to HTTPConnection. The trouble is that StringIO does not behave the same as a real socket: where HTTPResponse fails one way reading from a real socket (eg. infinite loop), it fails differently (or not at all) reading from a StringIO. Makes testing with the FakeSocket class in test_httplib.py problematic. Maybe the right way to test httplib is to spawn a server process (thread?) to listen on some random port, feed various HTTP responses at HTTPConnection/HTTPResponse, and see what happens. I'm not sure how to do that portably, though. Well, I'll see if I can whip up a Unix-y solution and see if anyone knows how to make it portable. Greg -- Greg Ward <gward at python.net> http://www.gerg.ca/ Be careful: sometimes, you're only standing on the shoulders of idiots. From rhettinger at ewtllc.com Mon Jul 31 20:34:26 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Mon, 31 Jul 2006 11:34:26 -0700 Subject: [Python-Dev] Rounding float to int directly (Re: struct module and coercing floats to integers) In-Reply-To: <44CAD9A1.8080000@canterbury.ac.nz> References: <9E51A030-B610-414B-98E0-1923F1C3E862@redivi.com> <dcbbbb410607282014w30459351ifb09c56075997be1@mail.gmail.com> <44CAD9A1.8080000@canterbury.ac.nz> Message-ID: <44CE4D32.6040909@ewtllc.com> Greg Ewing wrote: >As an aside, does anyone else think that it would be >useful to have a builtin which rounds and converts to >an int in one go? Whenever I use round(), I almost >always want the result as an int, and making me do >it in two steps seems unnecessarily bothersome. > > I think this would harm more than it would help. It more confusing to have several rounding-thingies to choose from than it is have an explicit two-step. BTW, I thought the traditional idiom (for positive numbers) was: int(x+.5) >Since automatic float->int coercion is being increasingly >disallowed, use cases for this are becoming more and more >common. > >-- >Greg >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: http://mail.python.org/mailman/options/python-dev/rhettinger%40ewtllc.com > > From chad at zetaweb.com Mon Jul 31 20:38:49 2006 From: chad at zetaweb.com (Chad Whitacre) Date: Mon, 31 Jul 2006 14:38:49 -0400 Subject: [Python-Dev] Patch submitted, now what? Message-ID: <44CE4E39.5010006@zetaweb.com> Dear All, Last week I submitted a patch (my first), and now I'm wondering what my expectations should be. Do I sit around and wait? How long? Do I notify this list? Do I notify a specific person, say, an author or reviewer of the original code I modified? Do I use SF's assignment mechanism? Who do I assign it to? These are the questions I have, unanswered (afaict) by the patch documentation I've found: http://www.python.org/dev/patches/ http://www.python.org/dev/patches/style/ http://www.python.org/patches/ http://www.python.org/patches/style.html http://www.python.org/dev/tools/#patch-tracking Any insight? chad P.S. For the interested, here is my patch: "Expose case-insensitivity of string.Template" http://www.python.org/sf/1528167 From brett at python.org Mon Jul 31 20:53:14 2006 From: brett at python.org (Brett Cannon) Date: Mon, 31 Jul 2006 11:53:14 -0700 Subject: [Python-Dev] Patch submitted, now what? In-Reply-To: <44CE4E39.5010006@zetaweb.com> References: <44CE4E39.5010006@zetaweb.com> Message-ID: <bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com> On 7/31/06, Chad Whitacre <chad at zetaweb.com> wrote: > > Dear All, > > Last week I submitted a patch (my first), Thanks! and now I'm wondering what my > expectations should be. Do I sit around and wait? How long? Do I notify > this list? Do I notify a specific person, say, an author or reviewer of > the original code I modified? Do I use SF's assignment mechanism? Who do > I assign it to? Let it sit for now. We get email notifications when new patches come in. Since we are all volunteers it can take a little while before we get to it. And don't assign it to anyone. Let us handle that. These are the questions I have, unanswered (afaict) by the patch > documentation I've found: > > http://www.python.org/dev/patches/ > http://www.python.org/dev/patches/style/ > http://www.python.org/patches/ > http://www.python.org/patches/style.html > http://www.python.org/dev/tools/#patch-tracking Those docs needs to be overhauled. I am planning to consolidate into a single patch guidelines doc in a month or so. Any insight? > > > > > chad > > > P.S. For the interested, here is my patch: > > "Expose case-insensitivity of string.Template" > http://www.python.org/sf/1528167 > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060731/161ca6a3/attachment.htm From chad at zetaweb.com Mon Jul 31 21:43:20 2006 From: chad at zetaweb.com (Chad Whitacre) Date: Mon, 31 Jul 2006 15:43:20 -0400 Subject: [Python-Dev] Patch submitted, now what? In-Reply-To: <bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com> References: <44CE4E39.5010006@zetaweb.com> <bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com> Message-ID: <44CE5D58.6060407@zetaweb.com> Brett, Thanks for the helpful reply. > Let it sit for now. We get email notifications when new patches come in. Can I ask who "we" are? Is that the seven SF "Project Admins?" Is that the 68 SF "Developers?" And is every patch eventually responded to? Or do some simply fall by the wayside? > Those docs needs to be overhauled. I am planning to consolidate into a > single patch guidelines doc in a month or so. Great! I'd be happy to review it when the time comes, if that'd be helpful. Thanks again. chad From brett at python.org Mon Jul 31 21:56:58 2006 From: brett at python.org (Brett Cannon) Date: Mon, 31 Jul 2006 12:56:58 -0700 Subject: [Python-Dev] Patch submitted, now what? In-Reply-To: <44CE5D58.6060407@zetaweb.com> References: <44CE4E39.5010006@zetaweb.com> <bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com> <44CE5D58.6060407@zetaweb.com> Message-ID: <bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com> On 7/31/06, Chad Whitacre <chad at zetaweb.com> wrote: > > Brett, > > Thanks for the helpful reply. > > > > Let it sit for now. We get email notifications when new patches come > in. > > Can I ask who "we" are? Is that the seven SF "Project Admins?" Is that > the 68 SF "Developers?" "We" is most of the developers on python-dev. There is a Python patches mailing list that most developers subscribe to. And is every patch eventually responded to? Or do some simply fall by > the wayside? The latter unfortunatley. Since this all relies on people's volunteer efforts the patch usually has to matter to someone to lead to them putting the time and effort into getting it checked in. > Those docs needs to be overhauled. I am planning to consolidate into a > > single patch guidelines doc in a month or so. > > Great! I'd be happy to review it when the time comes, if that'd be > helpful. Yep, it would be. -Brett -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20060731/67f04a64/attachment.html From skip at pobox.com Mon Jul 31 22:06:47 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 31 Jul 2006 15:06:47 -0500 Subject: [Python-Dev] Patch submitted, now what? In-Reply-To: <bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com> References: <44CE4E39.5010006@zetaweb.com> <bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com> <44CE5D58.6060407@zetaweb.com> <bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com> Message-ID: <17614.25303.255526.960212@montanaro.dyndns.org> Chad> And is every patch eventually responded to? Or do some simply fall Chad> by the wayside? Brett> The latter unfortunatley. Since this all relies on people's Brett> volunteer efforts the patch usually has to matter to someone to Brett> lead to them putting the time and effort into getting it checked Brett> in. It's worth noting that a number of people will look at a patch after the submitter has reviewed five other patches or bug reports (does it look okay to you? does Python build with it applied? do all unit tests pass? if it needs documentation does it have it? etc). Do that, attach comments to each of the five, then send a note here listing the five you reviewed and the patch id of your patch and one of those patch angels will take a look at your patch (if they haven't already). Skip From chad at zetaweb.com Mon Jul 31 22:07:10 2006 From: chad at zetaweb.com (Chad Whitacre) Date: Mon, 31 Jul 2006 16:07:10 -0400 Subject: [Python-Dev] Patch submitted, now what? In-Reply-To: <bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com> References: <44CE4E39.5010006@zetaweb.com> <bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com> <44CE5D58.6060407@zetaweb.com> <bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com> Message-ID: <44CE62EE.2010801@zetaweb.com> Brett, > "We" is most of the developers on python-dev. There is a Python patches > mailing list that most developers subscribe to. Helpful, thanks. > The latter unfortunatley. Since this all relies on people's volunteer > efforts the patch usually has to matter to someone to lead to them putting > the time and effort into getting it checked in. Hrm, ok, also good to know. So it sounds like I wait another week or so and then maybe approach some folks individually. chad From chad at zetaweb.com Mon Jul 31 22:08:32 2006 From: chad at zetaweb.com (Chad Whitacre) Date: Mon, 31 Jul 2006 16:08:32 -0400 Subject: [Python-Dev] Patch submitted, now what? In-Reply-To: <17614.25303.255526.960212@montanaro.dyndns.org> References: <44CE4E39.5010006@zetaweb.com> <bbaeab100607311153q52753e27yf371e2ad19d61405@mail.gmail.com> <44CE5D58.6060407@zetaweb.com> <bbaeab100607311256u281af267wd5b4d79b7db7b203@mail.gmail.com> <17614.25303.255526.960212@montanaro.dyndns.org> Message-ID: <44CE6340.30505@zetaweb.com> Skip, > It's worth noting that a number of people will look at a patch after the > submitter has reviewed five other patches or bug reports Also helpful, thanks! chad From martin at v.loewis.de Mon Jul 31 22:48:36 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon, 31 Jul 2006 22:48:36 +0200 Subject: [Python-Dev] Patch submitted, now what? In-Reply-To: <44CE4E39.5010006@zetaweb.com> References: <44CE4E39.5010006@zetaweb.com> Message-ID: <44CE6CA4.1080908@v.loewis.de> Chad Whitacre schrieb: [I notice that my message comes across pretty negative. In a single sentence: We are all volunteers with limited time, and we contribute to Python because its fun and because it helps us solve our own problems.] > Last week I submitted a patch (my first), and now I'm wondering what my > expectations should be. As a starting point, it would be better if you had provided the patch number (1528167) in this message. That makes it easier to answer the questions. [I didn't read until the end of the message yet] > Do I sit around and wait? How long? It's a new feature, so it can't possibly get into Python 2.5. That means that no action is likely taken before October. It might take years until the patch gets considered. As others have mentioned, there are ways to expedite processing. > Do I notify this list? If you want discussion of the patch, you can do that. Depending on how many people care about that aspect of Python, you may or may not get a response. I personally don't see myself using $-interpolation in the next few years, so I doubt I will react in any way (the Python library would just be the same for me if that feature didn't exist, and your patch isn't a bug fix). Others are more interested in the feature (or else it wouldn't have gotten into the standard library), so they might respond. > Do I notify a specific person, say, an author or reviewer of > the original code I modified? You might do that, although some may consider this at the edge of spamming. > Do I use SF's assignment mechanism? Please don't. If your assignment is "incorrect", this is a guarantee that the patch will sit there forever: the person assigned won't react because he doesn't feel responsible if he isn't, and nobody else will feel responsible because the patch is assigned. As Skip explains, I have a 5-for-1-rule for people who really want to push their patches: If you review 5 patches, I will review yours (despite me normally ignoring patches to this part of the library). Regards, Martin From chad at zetaweb.com Mon Jul 31 23:07:23 2006 From: chad at zetaweb.com (Chad Whitacre) Date: Mon, 31 Jul 2006 17:07:23 -0400 Subject: [Python-Dev] Patch submitted, now what? In-Reply-To: <44CE6CA4.1080908@v.loewis.de> References: <44CE4E39.5010006@zetaweb.com> <44CE6CA4.1080908@v.loewis.de> Message-ID: <44CE710B.4010905@zetaweb.com> Martin, Thanks for the reply. > I notice that my message comes across pretty negative. No worries. I'm a volunteer too. :) > It's a new feature, so it can't possibly get into Python 2.5. That means > that no action is likely taken before October. It might take years until > the patch gets considered. Helpful, thanks. > As Skip explains, I have a 5-for-1-rule for people who really want > to push their patches: If you review 5 patches, I will review yours Yeah, this hit the bulls-eye for me. The 5-for-1 rule is a great way to specify how newcomers can influence patch workflow w/o becoming spammers. [Brett: +1 on mentioning this in the updated Patch Guidelines] Watch for another post once I've found the time to volunteer to review 5 other patches. :) Thanks again, all! Consider my original questions answered. chad From oliphant.travis at ieee.org Mon Jul 31 20:28:09 2006 From: oliphant.travis at ieee.org (Travis Oliphant) Date: Mon, 31 Jul 2006 12:28:09 -0600 Subject: [Python-Dev] Bad interaction of __index__ and sequence repeat In-Reply-To: <44CB92F1.9040304@iinet.net.au> References: <20060728101133.GA339@code0.codespeak.net> <44CB6B7D.1040001@gmail.com> <44CB92F1.9040304@iinet.net.au> Message-ID: <44CE4BB9.60903@ieee.org> Nick Coghlan wrote: > Nick Coghlan wrote: >> Armin Rigo wrote: >>> Hi, >>> >>> There is an oversight in the design of __index__() that only just >>> surfaced :-( It is responsible for the following behavior, on a 32-bit >>> machine with >= 2GB of RAM: >>> >>> >>> s = 'x' * (2**100) # works! >>> >>> len(s) >>> 2147483647 >>> >>> This is because PySequence_Repeat(v, w) works by applying >>> w.__index__ in >>> order to call v->sq_repeat. However, __index__ is defined to clip the >>> result to fit in a Py_ssize_t. This means that the above problem >>> exists >>> with all sequences, not just strings, given enough RAM to create such >>> sequences with 2147483647 items. >>> >>> For reference, in 2.4 we correctly get an OverflowError. >>> >>> Argh! What should be done about it? >> >> I've now got a patch on SF that aims to fix this properly [1]. > > I revised this patch to further reduce the code duplication associated > with the indexing code in the standard library. > > The patch now has three new functions in the abstract C API: > > PyNumber_Index (used in a dozen or so places) > - raises IndexError on overflow > PyNumber_AsSsize_t (used in 3 places) > - raises OverflowError on overflow > PyNumber_AsClippedSsize_t() (used once, by _PyEval_SliceIndex) > - clips to PY_SSIZE_T_MIN/MAX on overflow > > All 3 have an int * output argument allowing type errors to be flagged > directly to the caller rather than through PyErr_Occurred(). > > Of the 3, only PyNumber_Index is exposed through the operator module. > > Probably the most interesting thing now would be for Travis to review > it, and see whether it makes things easier to handle for the Numeric > scalar types (given the amount of code the patch deleted from the > builtin and standard library data types, hopefully the benefits to > Numeric will be comparable). I noticed most of the checks for PyInt where removed in the patch. If I remember correctly, I left these in for "optimization." Other than that, I think the patch is great. As far as helping with NumPy, I think it will help to be able to remove special-checks for all the different integer-types. But, this has not yet been done in the NumPy code. -Travis