From guido at python.org  Wed Oct  1 00:04:03 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 15:04:03 -0700
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <20080930184751.31635.1484325691.divmod.xquotient.520@weber.divmod.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
	<20080930184751.31635.1484325691.divmod.xquotient.520@weber.divmod.com>
Message-ID: <ca471dc20809301504h2fc99567o34eb6c947d882c1e@mail.gmail.com>

On Tue, Sep 30, 2008 at 11:47 AM,  <glyph at divmod.com> wrote:
>
> On 05:56 pm, guido at python.org wrote:
>>
>> On Tue, Sep 30, 2008 at 10:59 AM,  <glyph at divmod.com> wrote:
>>>
>>> On 02:32 pm, guido at python.org wrote:
>
>>> In the absence of a 2.6 getcwdb, perhaps the fixer could just drop the
>>> "benefit of the doubt" case?  It could always be added to 2.7, and the
>>> parity release of 2to3 could have a --2.7 switch that would modify the
>>> behavior of this and other fixers.
>>
>> I'm not sure what you're proposing. *My* proposal is that 2to3 changes
>> os.getcwdu() calls to os.getcwd() and leaves os.getcwd() calls alone
>> -- there's no way to tell whether os.getcwdb() would be a better
>> match, and for portable code, it won't be (since os.getcwdb() is a
>> Unix-only thing).
>
> My proposal is simply to change getcwd to getcwdb, and getcwdu to getcwd.
>  This preserves whatever bytes/text behavior you are expecting from 2.6 into
> 3.0.  Granted, the fact that unicode is really always the right thing to do
> on Windows complicates things.

Plus, even on Linux Unicode is *usually* what you should be doing,
unless you're writing a backup tool.

> I already tend to avoid os.getcwd() though, and this is just one more reason
> to avoid it.  In the rare cases where I really do need it, it looks like
> os.path.abspath(b".") / os.path.abspath(u".") will provide the clarity that
> I want.

Or os.path.expanduser('~') vs. os.path.expanduser(b'~'). :-)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From rhamph at gmail.com  Wed Oct  1 00:04:09 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Tue, 30 Sep 2008 16:04:09 -0600
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48E29D7E.6080406@gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<48E0B8FB.9070701@egenix.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com>
	<ca471dc20809300722x4a700e85v153732843f43c920@mail.gmail.com>
	<48E29D7E.6080406@gmail.com>
Message-ID: <aac2c7cb0809301504v1b287c62xc8e58d8c8deaa7a2@mail.gmail.com>

On Tue, Sep 30, 2008 at 3:43 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Guido van Rossum wrote:
>> The callback would either be an extra argument to all
>> system calls (bad, ugly etc., and why not go with the existing unicode
>> encoding and error flags if we're adding extra args?) or would be
>> global, where I'd be worried that it might interfere with the proper
>> operation of library code that is several abstractions away from
>> whoever installed the callback, not under their control, and not
>> expecting the callback.
>>
>> I suppose I may have totally misunderstood your proposal, but in
>> general I find callbacks unwieldy.
>
> Not really - later in the email, I actually pointed out that exposing
> the unicode errors flag for the implicit PyUnicode_Decode invocations
> would be enough to enable a callback mechanism.
>
> However, James's post pointing out that this is a problem that also
> affects environment variables and command line arguments, not just file
> paths completely kills any hope of purely callback based approach - that
> processing needs to "just work" without any additional intervention from
> the application.
>
> Of the suggestions I've seen so far, I like Marcin's Mono-inspired
> NULL-escape codec idea the best. Since these strings all come from parts
> of the environment where NULLs are not permitted, a simple "'\0' in
> text" check will immediately identify any strings where decoding failed
> (for applications which care about the difference and want to try to do
> better), while applications which don't care will receive perfectly
> valid Python strings that can be passed around and manipulated as if the
> decoding error never happened.

It avoids the technical problems, but it's still magical behaviour
that users have to learn, whereas bytes/unicode polymorphism uses the
distinctions you should already know about.

There's also a problem of how to turn it on.  I'm against
automatically Python changing the filesystem encoding, no matter how
well intentioned.  Better to let the app do that, which is easy and
could be done for all apps (not just python!) if someone defined a
libc encoding of "null-escaped UTF-8".

On the whole I'm only -0 on it (compared to -1 for UTF-8b).


-- 
Adam Olsen, aka Rhamphoryncus

From Jack.Jansen at cwi.nl  Wed Oct  1 00:05:22 2008
From: Jack.Jansen at cwi.nl (Jack Jansen)
Date: Wed, 1 Oct 2008 00:05:22 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E29D3B.5030900@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbtq8t$3dl$1@ger.gmane.org>	<ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>
	<gbtvd3$na4$1@ger.gmane.org> <48E29D3B.5030900@v.loewis.de>
Message-ID: <EF183EA0-B073-4883-9362-C8B6C8E470D3@cwi.nl>


On  30-Sep-2008, at 23:42 , Martin v. L?wis wrote:
> It's the other way 'round: On Windows, Unicode file names are the
> natural choice, and byte strings have limitations. In a sense, Windows
> got it right - but then, they started later. Unix missed the  
> opportunity
> of declaring that all file APIs are UTF-8 (except for Plan-9 and OS X,
> neither being "true" Unix).


How does windows (and Python on windows) handle NFC versus NFD issues?  
Can I have two files called "?mlaut.txt", one in NFD and one NFC form?  
And are both of those representable on the Python side (i.e. can they  
both be returned from listdir() and passed to open())? CIf I compare  
these two filenames, do they compare differently?
--
Jack Jansen, <Jack.Jansen at cwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma  
Goldman


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081001/83be838a/attachment.htm>

From guido at python.org  Wed Oct  1 00:05:36 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 15:05:36 -0700
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <fb73205e0809301207g4e212aa8lc5ae2981795d7744@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
	<fb73205e0809301207g4e212aa8lc5ae2981795d7744@mail.gmail.com>
Message-ID: <ca471dc20809301505i1151080brb8d34526960c7fc@mail.gmail.com>

On Tue, Sep 30, 2008 at 12:07 PM, Simon Cross
<hodgestar+pythondev at gmail.com> wrote:
> On Tue, Sep 30, 2008 at 7:56 PM, Guido van Rossum <guido at python.org> wrote:
>> (since os.getcwdb() is a Unix-only thing).
>
> I would be happier if all the Unix byte functions existed on Windows
> fell back to something like encoding the filenames to/from UTF-8. Then
> at least it would be possible for programs to support reading all
> files on both Unix and Windows without having to perform some sort of
> explicit check to see whether os.getcwdb() and friends are supported.

Actually on Windows the syscalls use the encoding that Microsoft uses
-- when using bytes we use the Windows bytes API and when using str we
use the Windows wide API. That's the most platform-compatible
approach.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From ncoghlan at gmail.com  Wed Oct  1 00:18:33 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 01 Oct 2008 08:18:33 +1000
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <aac2c7cb0809301504v1b287c62xc8e58d8c8deaa7a2@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>	
	<200809291250.03291.eckhardt@satorlaser.com>	
	<48E0B8FB.9070701@egenix.com>	
	<200809291359.06334.eckhardt@satorlaser.com>	
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>	
	<48E15B83.9040205@v.loewis.de>	
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>	
	<48E1F53B.7030901@gmail.com>	
	<ca471dc20809300722x4a700e85v153732843f43c920@mail.gmail.com>	
	<48E29D7E.6080406@gmail.com>
	<aac2c7cb0809301504v1b287c62xc8e58d8c8deaa7a2@mail.gmail.com>
Message-ID: <48E2A5B9.5090005@gmail.com>

Adam Olsen wrote:
> On Tue, Sep 30, 2008 at 3:43 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Of the suggestions I've seen so far, I like Marcin's Mono-inspired
>> NULL-escape codec idea the best. Since these strings all come from parts
>> of the environment where NULLs are not permitted, a simple "'\0' in
>> text" check will immediately identify any strings where decoding failed
>> (for applications which care about the difference and want to try to do
>> better), while applications which don't care will receive perfectly
>> valid Python strings that can be passed around and manipulated as if the
>> decoding error never happened.
> 
> It avoids the technical problems, but it's still magical behaviour
> that users have to learn, whereas bytes/unicode polymorphism uses the
> distinctions you should already know about.
> 
> There's also a problem of how to turn it on.  I'm against
> automatically Python changing the filesystem encoding, no matter how
> well intentioned.  Better to let the app do that, which is easy and
> could be done for all apps (not just python!) if someone defined a
> libc encoding of "null-escaped UTF-8".
> 
> On the whole I'm only -0 on it (compared to -1 for UTF-8b).

For the decoding side, you wouldn't need to do it as a codec - you could
do it as a 'nullescape' error handler (since NULLs can't be present in
the byte sequences being decoded, there is no need to worry about
escaping anything when decoding is successful).

Converting those NULL escaped strings back into something the filesystem
can understand would obviously need a custom codec though, but some kind
of application level handling of bad filenames is going to be needed no
matter how we deal with bad encoding on the input side.

That said, I don't think this is something we (or, more to the point,
Guido) need to make a decision on right now - for 3.0, having
bytes-level APIs that can see everything, and Unicode APIs that ignore
badly encoded filenames is worth trying. If it proves inadequate, then
we can revisit the idea of some kind of implicit escaping mechanism in
the Unicode APIs for 3.1 when there is more time for a proper PEP.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From guido at python.org  Wed Oct  1 00:20:57 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 15:20:57 -0700
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48E29D7E.6080406@gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<48E0B8FB.9070701@egenix.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com>
	<ca471dc20809300722x4a700e85v153732843f43c920@mail.gmail.com>
	<48E29D7E.6080406@gmail.com>
Message-ID: <ca471dc20809301520r69e6d102gc865b7ff0dacab60@mail.gmail.com>

On Tue, Sep 30, 2008 at 2:43 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Of the suggestions I've seen so far, I like Marcin's Mono-inspired
> NULL-escape codec idea the best. Since these strings all come from parts
> of the environment where NULLs are not permitted, a simple "'\0' in
> text" check will immediately identify any strings where decoding failed
> (for applications which care about the difference and want to try to do
> better), while applications which don't care will receive perfectly
> valid Python strings that can be passed around and manipulated as if the
> decoding error never happened.

I'm not so sure. While it maintains *internal* consistency, printing
and displaying those filenames isn't likely going to give useful
results. E.g. on the terminal emulator I happen to be using right now
null bytes are simply ignored. Another danger might be that the null
character may be seen as the end of a string by some other library.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Wed Oct  1 00:21:04 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 01 Oct 2008 00:21:04 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <ca471dc20809301434u6116391cje5778bcef5048cc9@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>	<gbsgk6$kc1$1@ger.gmane.org>	<ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>	<gbtnjo$quh$1@ger.gmane.org>	<ca471dc20809301045r59251402g3fe947dec3bc7f22@mail.gmail.com>	<48E28C31.6060606@v.loewis.de>
	<ca471dc20809301434u6116391cje5778bcef5048cc9@mail.gmail.com>
Message-ID: <48E2A650.4000108@v.loewis.de>

>> My concern still is that it brings the bytes type into the status of
>> another character string type, which is really bad, and will require
>> further modifications to Python for the lifetime of 3.x.
> 
> I'd like to understand why this is "really bad". I though it was by
> design that the str and bytes types behave pretty similarly. You can
> use both as dict keys.

If they have to behave pretty similarly, they have to be supported in
all APIs that deal with text. For example, people will demand that
printing bytes should just copy them onto the stream (rather than
invoking repr()), and writing them onto a text stream should work the
same way. GUI library should support them, the XML libraries, and so
on.

Where will you stop, and tell people that bytes are just not supposed
to do this or that?

>> This is because applications will then regularly use byte strings for
>> file names on Unix, and regular strings on Windows, and then expect
>> the program to work the same without further modifications.
> 
> It seems that bytes arguments actually *do* work on Windows -- somehow
> they get decoded. (Unless Terry's report was from 2.x.)

To a limited degree - see my other message. Don't try to listdir a
directory with characters outside CP_ACP (it will give you invalid
file names).

> Actually something like that may not be a bad idea. Ian Bicking's
> webob supports similar double APIs for getting the request parameters
> out of a request object; I believe request.GET['x'] is a text object
> and request.GET_str['x'] is the corresponding uninterpreted bytes
> sequence. I would prefer to have os.environb over os.environ[b"PATH"]
> though.

And would you keep them synchronized?

> I assume at some point we can stop and have sufficiently low-level
> interfaces that everyone can agree are in bytes only. Bytes aren't
> going away. How does Java deal with this? Its File class doesn't seem
> to deal in bytes at all. What would its listFiles() method do with
> undecodable filenames?

Apparently (JDK 1.5.0_16, on Linux), it decodes undecodable bytes/byte
sequences as U+FFFD (REPLACEMENT CHARACTER). Opening such a file will
fail with FileNotFoundException.

IOW, Java hasn't solved the problem in the last 10 years. Marcin
Kowalczyk did a more thorough analysis about a year ago in

http://mail.python.org/pipermail/python-3000/2007-September/010450.html

Regards,
Martin



From guido at python.org  Wed Oct  1 00:23:15 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 15:23:15 -0700
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48E2A5B9.5090005@gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com>
	<ca471dc20809300722x4a700e85v153732843f43c920@mail.gmail.com>
	<48E29D7E.6080406@gmail.com>
	<aac2c7cb0809301504v1b287c62xc8e58d8c8deaa7a2@mail.gmail.com>
	<48E2A5B9.5090005@gmail.com>
Message-ID: <ca471dc20809301523i28474685mab945a1d39000310@mail.gmail.com>

On Tue, Sep 30, 2008 at 3:18 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> That said, I don't think this is something we (or, more to the point,
> Guido) need to make a decision on right now - for 3.0, having
> bytes-level APIs that can see everything, and Unicode APIs that ignore
> badly encoded filenames is worth trying. If it proves inadequate, then
> we can revisit the idea of some kind of implicit escaping mechanism in
> the Unicode APIs for 3.1 when there is more time for a proper PEP.

Right. Given that most syscalls already support both bytes and
(unicode) str, the simplest thing to do is to take this a bit further,
along the lines of Victor's patches, which I'm reviewing in Rietveld
right now:

http://codereview.appspot.com/3055

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Wed Oct  1 00:28:22 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 01 Oct 2008 00:28:22 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <83758335-97EA-441B-A783-05F16EBE6D7A@fuhm.net>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbtq8t$3dl$1@ger.gmane.org>
	<48E29CB1.5010309@v.loewis.de>
	<83758335-97EA-441B-A783-05F16EBE6D7A@fuhm.net>
Message-ID: <48E2A806.6020607@v.loewis.de>

> Yes! If there is a byte-string access method for Windows, pretty please
> make it decode from UTF-8 internally and call the Unicode version of the
> Windows APIs. The non-unicode windows APIs are pretty much just broken
> -- Ideally, Python should never be calling those.

I don't think we will manage to release Python 3.0 this year if that
change is to be implemented. And then, I don't think the release manager
will agree to such a delay.

I disagree that the ANSI APIs are broken. For most users (and by that,
I mean much more than 99% of the world population with access to
Windows computers), they work just fine. You have to deliberately try
to break them, or work in an environment were you speak multiple
languages (with conflicting scripts) simultaneously. Practicality
beats purity, and I applaud Microsoft for such a foresighted design
(they are guilty for bad designs in other places, but this one really
gives a good tradeoff of all issues, all things considered).

Regards,
Martin

From martin at v.loewis.de  Wed Oct  1 00:32:03 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 01 Oct 2008 00:32:03 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <EF183EA0-B073-4883-9362-C8B6C8E470D3@cwi.nl>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbtq8t$3dl$1@ger.gmane.org>	<ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>	<gbtvd3$na4$1@ger.gmane.org>
	<48E29D3B.5030900@v.loewis.de>
	<EF183EA0-B073-4883-9362-C8B6C8E470D3@cwi.nl>
Message-ID: <48E2A8E3.3070805@v.loewis.de>


> How does windows (and Python on windows) handle NFC versus NFD issues?

That's left to the application.

> Can I have two files called "?mlaut.txt", one in NFD and one NFC form?

Yes, you can. It sounds confusing, but only in a theoretical way. You
never have combining characters on Windows (at least, I don't). The
keyboard input defaults to NFC, and users normally don't type file
names, anyways, except when creating the files - later, they just use
the mouse to indicate what file they want to act on.

> And are both of those representable on the Python side (i.e. can they
> both be returned from listdir() and passed to open())?

Certainly!

> CIf I compare
> these two filenames, do they compare differently? 

Certainly!

Regards,
Martin

From guido at python.org  Wed Oct  1 00:33:50 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 15:33:50 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E2A650.4000108@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<gbsgk6$kc1$1@ger.gmane.org>
	<ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>
	<gbtnjo$quh$1@ger.gmane.org>
	<ca471dc20809301045r59251402g3fe947dec3bc7f22@mail.gmail.com>
	<48E28C31.6060606@v.loewis.de>
	<ca471dc20809301434u6116391cje5778bcef5048cc9@mail.gmail.com>
	<48E2A650.4000108@v.loewis.de>
Message-ID: <ca471dc20809301533y477c19dx699255c14290ae97@mail.gmail.com>

On Tue, Sep 30, 2008 at 3:21 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>> My concern still is that it brings the bytes type into the status of
>>> another character string type, which is really bad, and will require
>>> further modifications to Python for the lifetime of 3.x.
>>
>> I'd like to understand why this is "really bad". I though it was by
>> design that the str and bytes types behave pretty similarly. You can
>> use both as dict keys.
>
> If they have to behave pretty similarly, they have to be supported in
> all APIs that deal with text.

I don't see how you get from "pretty similarly" to "all APIs". :-)

> For example, people will demand that
> printing bytes should just copy them onto the stream (rather than
> invoking repr()), and writing them onto a text stream should work the
> same way. GUI library should support them, the XML libraries, and so
> on.
>
> Where will you stop, and tell people that bytes are just not supposed
> to do this or that?

Printing a bytes object already works, and displays its repr(), which
is guaranteed to be pure ASCII (unlike the repr() of a unicode str
object in Py3k). All the others you mention will cause breakage as
they should -- these errors exist to force the programmer to think
about encodings or conversions. I don't see that as a big burden
because the only way there could be bytes here in the first place is
when the user explicitly requested bytes. A program that only ever
passes text strings to the os module is only ever going to get text
strings back.

>>> This is because applications will then regularly use byte strings for
>>> file names on Unix, and regular strings on Windows, and then expect
>>> the program to work the same without further modifications.
>>
>> It seems that bytes arguments actually *do* work on Windows -- somehow
>> they get decoded. (Unless Terry's report was from 2.x.)
>
> To a limited degree - see my other message. Don't try to listdir a
> directory with characters outside CP_ACP (it will give you invalid
> file names).

Understood.

>> Actually something like that may not be a bad idea. Ian Bicking's
>> webob supports similar double APIs for getting the request parameters
>> out of a request object; I believe request.GET['x'] is a text object
>> and request.GET_str['x'] is the corresponding uninterpreted bytes
>> sequence. I would prefer to have os.environb over os.environ[b"PATH"]
>> though.
>
> And would you keep them synchronized?

Yes, the bytes versions would be the canonical version and the str
version would wrap around that -- though updating the str version
would also update the bytes version. Some keys would be missing from
the str version (or perhaps they would raise exceptions or default to
some other error handler, like ignore or replace).

>> I assume at some point we can stop and have sufficiently low-level
>> interfaces that everyone can agree are in bytes only. Bytes aren't
>> going away. How does Java deal with this? Its File class doesn't seem
>> to deal in bytes at all. What would its listFiles() method do with
>> undecodable filenames?
>
> Apparently (JDK 1.5.0_16, on Linux), it decodes undecodable bytes/byte
> sequences as U+FFFD (REPLACEMENT CHARACTER). Opening such a file will
> fail with FileNotFoundException.
>
> IOW, Java hasn't solved the problem in the last 10 years. Marcin
> Kowalczyk did a more thorough analysis about a year ago in
>
> http://mail.python.org/pipermail/python-3000/2007-September/010450.html

I can't say I like the Java solution. I would like to be able to write
a robust backup tool in Python, even if the code needed to make it
work everywhere isn't going to win any prizes (due to the need to use
bytes on Unix, str on Windows).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From foom at fuhm.net  Wed Oct  1 00:36:23 2008
From: foom at fuhm.net (James Y Knight)
Date: Tue, 30 Sep 2008 18:36:23 -0400
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E2A650.4000108@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>	<gbsgk6$kc1$1@ger.gmane.org>	<ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>	<gbtnjo$quh$1@ger.gmane.org>	<ca471dc20809301045r59251402g3fe947dec3bc7f22@mail.gmail.com>	<48E28C31.6060606@v.loewis.de>
	<ca471dc20809301434u6116391cje5778bcef5048cc9@mail.gmail.com>
	<48E2A650.4000108@v.loewis.de>
Message-ID: <0DBCA888-43DA-4DE9-952F-A377E96B286D@fuhm.net>

On Sep 30, 2008, at 6:21 PM, Martin v. L?wis wrote:
> IOW, Java hasn't solved the problem in the last 10 years.

Java is already really bad at being a small little language to write  
cooperating tools in. I'd never even attempt to write a little  
pipeline filter in Java -- I've already pretty much learned to expect  
Java applications to be in their own world, so I'd hardly find it  
surprising if a Java app could only read files it wrote itself,  
nevermind files in odd encodings.

Python, on the other hand, is an awesome tool for writing small little  
scripts that interact well with the surrounding environment, Just The  
Way It Is, without trying to layer so much abstraction upon it so that  
you lose functionality. Moving away from that would be unfortunate.

James

From Jack.Jansen at cwi.nl  Wed Oct  1 00:49:57 2008
From: Jack.Jansen at cwi.nl (Jack Jansen)
Date: Wed, 1 Oct 2008 00:49:57 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E2A8E3.3070805@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbtq8t$3dl$1@ger.gmane.org>	<ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>	<gbtvd3$na4$1@ger.gmane.org>
	<48E29D3B.5030900@v.loewis.de>
	<EF183EA0-B073-4883-9362-C8B6C8E470D3@cwi.nl>
	<48E2A8E3.3070805@v.loewis.de>
Message-ID: <82D029DA-C218-4631-A68E-CE3DBB03494A@cwi.nl>


On  1-Oct-2008, at 00:32 , Martin v. L?wis wrote:

>
>> How does windows (and Python on windows) handle NFC versus NFD  
>> issues?
>
> That's left to the application.
>
>> Can I have two files called "?mlaut.txt", one in NFD and one NFC  
>> form?
>
> Yes, you can. It sounds confusing, but only in a theoretical way. You
> never have combining characters on Windows (at least, I don't). The
> keyboard input defaults to NFC, and users normally don't type file
> names, anyways, except when creating the files - later, they just use
> the mouse to indicate what file they want to act on.
>
>> And are both of those representable on the Python side (i.e. can they
>> both be returned from listdir() and passed to open())?
>
> Certainly!
>
>> CIf I compare
>> these two filenames, do they compare differently?
>
> Certainly!

Actually, that all sounds pretty non-confusing to me:-)

So, normal users will always have the one form, and if by chance they  
get the other form they can still use the file. Also from Python, even  
when doing listdir() and then open(), everything will work just as  
expected. That there are two files that have a similar visual  
representation is not too bad, the same happens with ellipses versus  
dot-dot-dot and many other cases.

Which means the only problem area left is unix filesystems (whether on  
Linux or mounted remotely on MacOS or whatever), where filenames are  
really byte strings with only / and nul illegal.



--
Jack Jansen, <Jack.Jansen at cwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma  
Goldman


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081001/10203f3f/attachment.htm>

From steve at pearwood.info  Wed Oct  1 01:08:39 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 1 Oct 2008 09:08:39 +1000
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E29CB1.5010309@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbtq8t$3dl$1@ger.gmane.org> <48E29CB1.5010309@v.loewis.de>
Message-ID: <200810010908.40274.steve@pearwood.info>

On Wed, 1 Oct 2008 07:40:01 am Martin v. L?wis wrote:
> >> On Windows, we might reject bytes filenames for all file
> >> operations: open(), unlink(), os.path.join(), etc. (raise a
> >> TypeError or UnicodeError)
> >
> > Since I've seen no objections to this yet: please no. If we offer a
> > "lower-level" bytes filename API, it should work for all platforms.
>
> Unfortunately, it can't. You cannot represent all possible file names
> in a byte string in Windows (just as you can't do so in a Unicode
> string on Unix).


Sorry, maybe I'm just being thick here, but I don't understand how that 
is possible. On the physical disk, each Windows file name must be 
represented by a byte string, yes? So how is it possible that there are 
Windows files with names that can't be represented as a byte string? 
What have I missed?



-- 
Steven

From guido at python.org  Wed Oct  1 01:21:37 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 16:21:37 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <200810010908.40274.steve@pearwood.info>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbtq8t$3dl$1@ger.gmane.org> <48E29CB1.5010309@v.loewis.de>
	<200810010908.40274.steve@pearwood.info>
Message-ID: <ca471dc20809301621o5d1d0ec2u6a296facdc3db043@mail.gmail.com>

On Tue, Sep 30, 2008 at 4:08 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Wed, 1 Oct 2008 07:40:01 am Martin v. L?wis wrote:
>> >> On Windows, we might reject bytes filenames for all file
>> >> operations: open(), unlink(), os.path.join(), etc. (raise a
>> >> TypeError or UnicodeError)
>> >
>> > Since I've seen no objections to this yet: please no. If we offer a
>> > "lower-level" bytes filename API, it should work for all platforms.
>>
>> Unfortunately, it can't. You cannot represent all possible file names
>> in a byte string in Windows (just as you can't do so in a Unicode
>> string on Unix).
>
> Sorry, maybe I'm just being thick here, but I don't understand how that
> is possible. On the physical disk, each Windows file name must be
> represented by a byte string, yes? So how is it possible that there are
> Windows files with names that can't be represented as a byte string?
> What have I missed?

I believe on disk it uses UTF-16.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From steve at pearwood.info  Wed Oct  1 02:04:39 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 1 Oct 2008 10:04:39 +1000
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <ca471dc20809301621o5d1d0ec2u6a296facdc3db043@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<200810010908.40274.steve@pearwood.info>
	<ca471dc20809301621o5d1d0ec2u6a296facdc3db043@mail.gmail.com>
Message-ID: <200810011004.39759.steve@pearwood.info>

On Wed, 1 Oct 2008 09:21:37 am you wrote:
> On Tue, Sep 30, 2008 at 4:08 PM, Steven D'Aprano <steve at pearwood.info> 
wrote:
> > On Wed, 1 Oct 2008 07:40:01 am Martin v. L?wis wrote:
> >> >> On Windows, we might reject bytes filenames for all file
> >> >> operations: open(), unlink(), os.path.join(), etc. (raise a
> >> >> TypeError or UnicodeError)
> >> >
> >> > Since I've seen no objections to this yet: please no. If we
> >> > offer a "lower-level" bytes filename API, it should work for all
> >> > platforms.
> >>
> >> Unfortunately, it can't. You cannot represent all possible file
> >> names in a byte string in Windows (just as you can't do so in a
> >> Unicode string on Unix).
> >
> > Sorry, maybe I'm just being thick here, but I don't understand how
> > that is possible. On the physical disk, each Windows file name must
> > be represented by a byte string, yes? So how is it possible that
> > there are Windows files with names that can't be represented as a
> > byte string? What have I missed?
>
> I believe on disk it uses UTF-16.

Which is made up of bytes. There may be byte sequences that are illegal 
UTF-16, but that's not what Martin said. I don't understand how there 
can be UTF-16 sequences which don't correspond to some sequence of 
bytes. How would they be represented in memory? Is this to do with the 
endianness of the UTF-16 sequence?


-- 
Steven

From murman at gmail.com  Wed Oct  1 02:16:19 2008
From: murman at gmail.com (Michael Urman)
Date: Tue, 30 Sep 2008 19:16:19 -0500
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <200810011004.39759.steve@pearwood.info>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<200810010908.40274.steve@pearwood.info>
	<ca471dc20809301621o5d1d0ec2u6a296facdc3db043@mail.gmail.com>
	<200810011004.39759.steve@pearwood.info>
Message-ID: <dcbbbb410809301716v3a1fc81dn39e3feb4fa293bec@mail.gmail.com>

On Tue, Sep 30, 2008 at 7:04 PM, Steven D'Aprano <steve at pearwood.info> wrote:
>> I believe on disk it uses UTF-16.
>
> Which is made up of bytes. There may be byte sequences that are illegal
> UTF-16, but that's not what Martin said. I don't understand how there
> can be UTF-16 sequences which don't correspond to some sequence of
> bytes. How would they be represented in memory? Is this to do with the
> endianness of the UTF-16 sequence?

It has to do with the internal mapping between the ANSI and Unicode
functions. On NT systems, CreateFileA will map the ANSI bytestring to
a Unicode filename via the active code page, and call CreateFileW
accordingly. The active code page cannot be set to something as useful
as UTF-8, so given any actual code page (1252, 932, etc.) there are
Unicode strings that cannot be represented with a bytestring provided
to the ANSI function.
-- 
Michael Urman

From victor.stinner at haypocalc.com  Wed Oct  1 02:17:33 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Wed, 1 Oct 2008 02:17:33 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E2A806.6020607@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<83758335-97EA-441B-A783-05F16EBE6D7A@fuhm.net>
	<48E2A806.6020607@v.loewis.de>
Message-ID: <200810010217.33570.victor.stinner@haypocalc.com>

Le Wednesday 01 October 2008 00:28:22 Martin v. L?wis, vous avez ?crit?:
> I don't think we will manage to release Python 3.0 this year if that
> change is to be implemented. And then, I don't think the release manager
> will agree to such a delay.

The minimum change is to disallow bytes/str mix:
 - os.listdir(unicode)->unicode and ignore invalid files
   (current behaviour is to return unicode and bytes)
 - os.readlink(unicode)->unicode or raise an error
   (current behaviour is to return unicode or bytes)
 - remove os.getcwdu() (use its code -which is better- for getcwd) 
   and fix the test_unicode_file.py

listdir() change (ignore invalid filenames) is important to avoid strange bugs 
in os.path.*(), glob.*() or on displaying a filename.

I can generate a specific patch for these issues. It's just a subset of my 
last patch.

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From greg.ewing at canterbury.ac.nz  Wed Oct  1 02:33:51 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 01 Oct 2008 12:33:51 +1200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <48E243CA.1090604@egenix.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de> <48E20017.3020405@egenix.com>
	<ca471dc20809300705x58aa87acn5e3760891e6b57b9@mail.gmail.com>
	<48E243CA.1090604@egenix.com>
Message-ID: <48E2C56F.50300@canterbury.ac.nz>

M.-A. Lemburg wrote:
> In the end, I think it's better not to be clever and just return
> the filenames that cannot be decoded as bytes objects in os.listdir().

But since it's a rare occurrence, most applications are
just going to ignore the issue, and then fail unexpectedly
one day on some unsuspecting user that doesn't have the
inclination to go diving into the code to fix it.

-- 
Greg


From glyph at divmod.com  Wed Oct  1 03:27:26 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Wed, 01 Oct 2008 01:27:26 -0000
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <ca471dc20809301437y36d389bbn932ab2cf810727f0@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com> <1222771976.2598.39.camel@localhost>
	<ca471dc20809300726g2d3b0d43qcaad50608d0283b3@mail.gmail.com>
	<1222785585.2598.45.camel@localhost>
	<20080930181231.31635.1557225685.divmod.xquotient.503@weber.divmod.com>
	<ca471dc20809301116y480bd028qbcdca44f05c62596@mail.gmail.com>
	<20080930184258.31635.76988507.divmod.xquotient.511@weber.divmod.com>
	<ca471dc20809301437y36d389bbn932ab2cf810727f0@mail.gmail.com>
Message-ID: <20081001012726.31635.1673198315.divmod.xquotient.605@weber.divmod.com>

On 30 Sep, 09:37 pm, guido at python.org wrote:
>On Tue, Sep 30, 2008 at 11:42 AM,  <glyph at divmod.com> wrote:
>>There are other ways to glean this knowledge; for example, looking at 
>>the
>>'iocharset' or 'nls' mount options supplied to mount various 
>>filesystems.

>I know we could do a better job, but absent anyone who knows what
>they're doing we've chosen a fairly conservative approach. I certainly
>hope that someone will contribute some mean encoding-guessing code to
>the stdlib that users can use. I'm not sure if I'll ever endorse doing
>this automatically in io.open(), though I'd be fine with a convention
>like passing encoding="guess".

I think the conservative approach is actually correct, or rather, as 
close to correct as it is possible to get in this mess.  Inspecting 
these fantastically obscure options is only likely to be helpful in a 
tool which tries to correct filesystem encoding errors on legacy data. 
I wouldn't even know about them if I hadn't written several such tools 
(well, just little scripts, really) in the past.  I was just verifying 
that I wasn't missing some "right way" which would let someone else do 
the guesswork for me.

In reality, you have two options for filesystem encoding on Linux:

  * UTF-8
  * fall in a well and die

The OS will happily let you create a completely nonsensical environment 
where no application can possibly do anything reasonable: set LC_ALL to 
KOI8R, mount your USB keychain as Shift_JIS and your windows partition 
as ISO-8859-8.  Of course nobody would actually _do_ this, because they 
want things to work, so everything is gradually evolving to a default of 
UTF-8 everywhere.  In practice, however, there are still problems with 
CIFS/SMB shares where other clients have different ideas about encoding. 
I've experienced this most commonly when sharing with Macs, which have 
very particular and different ideas about normalization, as has already 
been discussed in this thread.

From glyph at divmod.com  Wed Oct  1 04:06:25 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Wed, 01 Oct 2008 02:06:25 -0000
Subject: [Python-Dev] [Python-3000] New proposition for Python3
	bytes	filename issue
In-Reply-To: <ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de>
	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
	<48E2865A.3010404@v.loewis.de>
	<ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>
Message-ID: <20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>

On 30 Sep, 09:22 pm, guido at python.org wrote:
>On Tue, Sep 30, 2008 at 1:04 PM, "Martin v. L?wis" <martin at v.loewis.de> 
>wrote:
>>Guido van Rossum wrote:
>>>On Mon, Sep 29, 2008 at 11:00 PM, "Martin v. L?wis" 
>>><martin at v.loewis.de> wrote:

>>>Martin, I don't understand why you are in favor of storing raw bytes
>>>encoded as Latin-1 in Unicode string objects, which clearly gives 
>>>rise
>>>to mojibake.

This is my word of the day, by the way.  Reading this whole thread was 
_totally_ worth it to learn about "mojibake".  Obviously I'm familiar 
with the phenomenon but somehow I'd never heard this awesome term 
before.
>I am also encouraged by Glyph's support for (a). He has a lot of
>practical experience.

Thanks for the vote of confidence.  I hope for all our sakes that you're 
not over-valuing that experience ;-).

For what it's worth, I can see MvL's point in that I think there is some 
danger in generating confusion by adding _too many_ string-like 
functions to the bytes type.  I don't want my suggestion to contribute 
to the confusion between bytes and text.

However, Martin, I can promise you that I will _never_ ask for any 
convenience functions related to bytes as a result of this decision.  I 
want bytes to come back from filesystem APIs because I intend to have a 
wrapper layer which knows two things about the file: the bytes (which 
are needed to talk to POSIX filesystem APIs) and the characters (which 
are computed from those bytes, can be safely renormalized, displayed to 
users, etc).  On Windows this filesystem wrapper will necessarily behave 
differently, and will not use bytes for anything.  Any formatting beyond 
joining path segments together and possibly splitting extensions off 
will be done on character strings, not byte strings.

The proposal of using U+0000 seems like it would have been almost the 
same from such a wrapper's perspective, except (A) people using the 
filesystem APIs without the benefit of such a wrapper would have been 
even more screwed, and (B) there are a few nasty corner-cases when 
dealing with surrogate (i.e. invalid, in UTF-8) code points which I'm 
not quite sure what it would have done with.

Guido already mentioned "libraries" as a hypothetical issue, but here's 
a real-world problem that results from putting NULLs into filenames. 
Consider this program:

    import gtk
    w = gtk.Window()
    b = gtk.Button(u"\u0000/hello/world")
    w.add(b)
    w.show_all()
    gtk.main()

which emits this message:
    TypeError: OGtkButton.__init__() argument 1 must be string without 
null bytes or None, not unicode

SQLite has a similar problem with NULLs, and I'm definitely sticking 
paths in there, too.

Eventually I'd like to propose such a path type for inclusion in the 
stdlib, but that will have to wait for issues like 
<http://twistedmatrix.com/trac/ticket/2366> to be resolved.

From rhamph at gmail.com  Wed Oct  1 04:22:08 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Tue, 30 Sep 2008 20:22:08 -0600
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de>
	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
	<48E2865A.3010404@v.loewis.de>
	<ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>
	<20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
Message-ID: <aac2c7cb0809301922o60b4cb09n6132dee3587ddb08@mail.gmail.com>

On Tue, Sep 30, 2008 at 8:06 PM,  <glyph at divmod.com> wrote:
> The proposal of using U+0000 seems like it would have been almost the same
> from such a wrapper's perspective, except (A) people using the filesystem
> APIs without the benefit of such a wrapper would have been even more
> screwed, and (B) there are a few nasty corner-cases when dealing with
> surrogate (i.e. invalid, in UTF-8) code points which I'm not quite sure what
> it would have done with.

Surrogates in UTF-8 *should* be treated as errors, but current python
is far too lax.  That actually leads to another problem: improving
validating will change what gets escaped and what doesn't.

http://bugs.python.org/issue3297
http://bugs.python.org/issue3672



-- 
Adam Olsen, aka Rhamphoryncus

From foom at fuhm.net  Wed Oct  1 05:32:04 2008
From: foom at fuhm.net (James Y Knight)
Date: Tue, 30 Sep 2008 23:32:04 -0400
Subject: [Python-Dev] [Python-3000] New proposition for Python3
	bytes	filename issue
In-Reply-To: <20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de>
	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
	<48E2865A.3010404@v.loewis.de>
	<ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>
	<20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
Message-ID: <22920D6A-8B70-4E6D-BE99-D7447D831B41@fuhm.net>


On Sep 30, 2008, at 10:06 PM, glyph at divmod.com wrote:
> However, Martin, I can promise you that I will _never_ ask for any  
> convenience functions related to bytes as a result of this  
> decision.  I want bytes to come back from filesystem APIs because I  
> intend to have a wrapper layer which knows two things about the  
> file: the bytes (which are needed to talk to POSIX filesystem APIs)  
> and the characters (which are computed from those bytes, can be  
> safely renormalized, displayed to users, etc).  On Windows this  
> filesystem wrapper will necessarily behave differently, and will not  
> use bytes for anything.  Any formatting beyond joining path segments  
> together and possibly splitting extensions off will be done on  
> character strings, not byte strings.

Can you clarify what proposal you are supporting for Python:

1) Two sets of APIs, one returning unicode strings, and one returning  
bytestrings. (subpoints: what does the unicode-returning API do when  
it cannot decode the bytestring into unicode? raise exception, pretend  
argument/envvar/file didn't exist/?)

or

2) All APIs return bytestrings only. Converting to unicode is  
considered lossy, and would have to be done by applications for  
display purposes only.

I really don't understand the reasoning for (1). It seems to me that  
most software (probably including all of the Python stdlib) would  
continue to use the unicode string API. Switching all of the Python  
stdlib to use the bytestring APIs instead would certainly be a large  
undertaking, and would have all sorts of ripple-on API changes (e.g.  
__file__). So I can only imagine that if you're proposing (1), you're  
doing so without the intention of suggesting that Python be converted  
to use it.

And so, of course, that doesn't really fix things (such as getcwd  
failing if your cwd is a path that is undecodeable in the current  
locale, or well, currently, python refusing to even start).

If you're proposing (2), it's at least as large an undertaking as (1)  
+ converting Python to use the optional bytestring APIs. But at least  
it avoids exposing an API that people ought not use, and does make it  
obvious what still needs to be fixed: the unfixed code simply won't  
run at all.

> The proposal of using U+0000 seems like it would have been almost  
> the same from such a wrapper's perspective, except (A) people using  
> the filesystem APIs without the benefit of such a wrapper would have  
> been even more screwed

I'm not sure what your "more screwed" is comparing against: current  
py3k behavior? (aka: decoding to Unicode in locale's specified  
encoding)? I don't see how you can really be more screwed than that:  
not only can't you send your filename to display in a Gtk+ button, you  
can't access it at all, even staying within python.

> and (B) there are a few nasty corner-cases when dealing with  
> surrogate (i.e. invalid, in UTF-8) code points which I'm not quite  
> sure what it would have done with.

The lone-surrogate-pair proposal was a totally different proposal than  
the U+0000 one.

James

From tjreedy at udel.edu  Wed Oct  1 05:46:00 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 30 Sep 2008 23:46:00 -0400
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <ca471dc20809301406s286e2328u853dbc453f1faf13@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbtq8t$3dl$1@ger.gmane.org>	<ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>	<gbtveu$ocl$1@ger.gmane.org>
	<ca471dc20809301406s286e2328u853dbc453f1faf13@mail.gmail.com>
Message-ID: <gburpm$34j$1@ger.gmane.org>

Guido van Rossum wrote:

> No, that's because bytes is missing from the explicit list of
> allowable types in io.open. Victor has a one-line trivial patch for
> this. Could you try this though?
> 
>>>> import _fileio
>>>> _fileio._FileIO(b'tem')

 >>> import _fileio
 >>> _fileio._FileIO(b'tem')
_fileio._FileIO(3, 'r')
 >>>


From glyph at divmod.com  Wed Oct  1 07:19:47 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Wed, 01 Oct 2008 05:19:47 -0000
Subject: [Python-Dev] [Python-3000] New proposition for Python3
	bytes	filename issue
In-Reply-To: <22920D6A-8B70-4E6D-BE99-D7447D831B41@fuhm.net>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de>
	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
	<48E2865A.3010404@v.loewis.de>
	<ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>
	<20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
	<22920D6A-8B70-4E6D-BE99-D7447D831B41@fuhm.net>
Message-ID: <20081001051947.31635.1251804577.divmod.xquotient.807@weber.divmod.com>

On 03:32 am, foom at fuhm.net wrote:
>On Sep 30, 2008, at 10:06 PM, glyph at divmod.com wrote:

>Can you clarify what proposal you are supporting for Python:

Sure.  Neither of your descriptions is terribly accurate, but I'll try 
to explain.
>1) Two sets of APIs, one returning unicode strings, and one returning 
>bytestrings. (subpoints: what does the unicode-returning API do when 
>it cannot decode the bytestring into unicode? raise exception, pretend 
>argument/envvar/file didn't exist/?)

The only API discussed so far which would actually provide two variants 
is 'getcwd', which would have a 'getcwdb' that gives back bytes instead.

Pretty much every other API takes some kind of input.  listdir(bytes) 
would give back bytes, while listdir(text) would give back text. 
listdir(text) would skip undecodable filenames.

Similarly for all the other APIs in os and os.path that take pathnames 
for input.
>2) All APIs return bytestrings only. Converting to unicode is 
>considered lossy, and would have to be done by applications for 
>display purposes only.

This is a bad way to do things, because on Windows, filenames *really 
are* unicode.  Converting to bytes is what's lossy.  (See previous 
discussion of active codepages and CreateFileA/CreateFileW.)
>I really don't understand the reasoning for (1).

The reasoning is that a lot of software doesn't care if it's wrong for 
edge cases, it's really hard to come up with something that's correct 
with respect to all of those edge cases (absurdly difficult, if you need 
to stay in the straightjacket of string / bytes types, as well as 
provide a useful library interface - which is why we're having this 
discussion).  But, it should be _possible_ to write software that's 
correct in the face of those edge cases.

And - let's not forget this - the worlds of POSIX and Windows really are 
different and really do require subtly different inputs.  Python can try 
to paper over this like Java does and make it impossible to write 
certain classes of application, or it can just provide an ugly, slightly 
inconsistent API that exposes the ugly, slightly inconsistent reality. 
Modulo the issues you've raised which I don't think the proposal totally 
covers yet (abspath with a non-decodable cwd) I think it strikes a nice 
balance; allow people to live in the delusion of unicode-on-POSIX and 
have software that mostly works, most of the time, or allow them to face 
the unpleasantness and spend the effort to get something really solid.

I think the _right_ answer to all of this is to (A) make FilePath work 
completely correctly for every totally insane edge case ever, and (B) 
include it in the stdlib.  One day I think we'll do that.  But nobody 
has the time or energy to do even the first part of that *right now*, 
before 3.0 is released, so I'm just looking for something which it will 
be possible to build FilePath, or something like it, on top of, without 
breaking other people's applications who rely on the os module directly 
too badly.
>It seems to me that  most software (probably including all of the 
>Python stdlib) would  continue to use the unicode string API.

That's true.  And that software wouldn't handle these edge cases 
completely correctly.  As Guido put it, "it's a quality of 
implementation issue".
>Switching all of the Python  stdlib to use the bytestring APIs instead 
>would certainly be a large  undertaking, and would have all sorts of 
>ripple-on API changes (e.g.  __file__).

I am not quite sure what to do about __file__.  My preference would 
probably be to use unicode filename for consistency so it can always be 
displayed, but provide a second attribute (__open_file__?) that would be 
sometimes unicode, sometimes bytes, which would be guaranteed to work 
with open().  I suspect that most software which interacts with __file__ 
on a deep level would be of the variety which would deal with the edge 
cases.

But where the Python stdlib wants a pathname it should be accepting 
either bytes or unicode, as all of the os.path functions want.  This 
does kind of suck, but the alternatives are to encode crazy extra 
information in unicode path names that cannot be exchanged with other 
programs (or with users: NULL is potentially the worst bogus character 
from a UI perspective), or revert to bytes for everything (which is a 
non-solution, c.f. Windows above).
>So I can only imagine that if you're proposing (1), you're  doing so 
>without the intention of suggesting that Python be converted  to use 
>it.

Maybe updating the stdlib to be correct in the face of such changes is 
hard, but it doesn't seem intractible.  Taken together, it looks like 
there are only about 100 calls in the stdlib to both getcwd and abspath 
together, and I suspect many of them are for purely aesthetic purposes 
and could just be eliminated, and many of them are redefinitions of the 
functions and don't need any changes.

All the other path manipulation functions would continue to work as-is, 
although some of them might skip undecodable files.
>And so, of course, that doesn't really fix things (such as getcwd 
>failing if your cwd is a path that is undecodeable in the current 
>locale, or well, currently, python refusing to even start).

The proposal as I understand it so far doesn't address this 
specifically, so I'll try to.  os.getcwd, os.path.abspath, and 
os.path.realpath (when called with unicode) will probably need to do 
something gross if they're called on a non-decodable directory.  One 
thing that comes to mind is to create a temporary symbolic link and 
return u'/tmp/python-$YOURUID-undecodable/$GUID/something'.  I hope 
someone else has a better idea, especially since that sort of defeats 
the purpose of realpath.

On the other hand, even this strawman answer is correct for pretty much 
any sane purpose, and if you _really_ care, you need to learn that you 
have to use and ask for bytes, on POSIX, to deal with such corner cases.
>If you're proposing (2),  (...)

Luckily I'm not.
>>The proposal of using U+0000 seems like it would have been almost  the 
>>same from such a wrapper's perspective, except (A) people using  the 
>>filesystem APIs without the benefit of such a wrapper would have  been 
>>even more screwed
>
>I'm not sure what your "more screwed" is comparing against: current 
>py3k behavior? (aka: decoding to Unicode in locale's specified 
>encoding)? I don't see how you can really be more screwed than that: 
>not only can't you send your filename to display in a Gtk+ button, you 
>can't access it at all, even staying within python.

You're screwed if you're trying to access files in a portable way 
without worrying at all about encodings.  There are files you won't be 
able to access, there are conditions you won't be able to deal with. 
Sorry, but POSIX sucks and that's life.

You're _more_ screwed if you're trying to access those files in a 
portable way without worrying about encodings, and the API you're using 
is giving you back invalid, magic path names, with NULLs rather than 
being slightly lossy and dropping filenames you (obviously, by virtue of 
the way you requested those filenames) won't be able to deal with.

So I was talking here about the default behavior in the case of a naive 
program that wants to pretend all paths are unicode.
>>and (B) there are a few nasty corner-cases when dealing with 
>>surrogate (i.e. invalid, in UTF-8) code points which I'm not quite 
>>sure what it would have done with.
>
>The lone-surrogate-pair proposal was a totally different proposal than 
>the U+0000 one.

I wasn't referring to the lone-surrogate-pair encoding trick, I was 
referring to the fact that some people are going to want to treat 
surrogate pairs as encoding errors (i.e. include the NULL byte) and some 
will want to treat them as valid.  If you want them to be valid you have 
to normalize away the surrogates in order to talk to other software, but 
you can't do that because then you'll get different bytes when you re- 
encode them.

There's probably a way around that but it would be subtle and 
controversial no matter how you did it.

From martin at v.loewis.de  Wed Oct  1 07:21:44 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 01 Oct 2008 07:21:44 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <200810010908.40274.steve@pearwood.info>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbtq8t$3dl$1@ger.gmane.org>
	<48E29CB1.5010309@v.loewis.de>
	<200810010908.40274.steve@pearwood.info>
Message-ID: <48E308E8.5060302@v.loewis.de>

> Sorry, maybe I'm just being thick here, but I don't understand how that 
> is possible. On the physical disk, each Windows file name must be 
> represented by a byte string, yes? So how is it possible that there are 
> Windows files with names that can't be represented as a byte string? 
> What have I missed?

That we are not really free to choose the byte representation when
choosing byte strings. Microsoft has defined how char* (i.e. byte
strings) are to be interpreted when interpreting them as byte strings,
namely in the ANSI code page. That code page is not capable of
representing all file names.

We could, for example, use the same representation as is used on disk.
However,
a) there is no API to find out what that representation is, and
b) it is not null-byte free, a property often desired for file names,
   and
c) because it contains null bytes, it won't be easy to display such
   file names on stdout, or in a GUI window.

Regards,
Martin

From martin at v.loewis.de  Wed Oct  1 07:27:47 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 01 Oct 2008 07:27:47 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3	bytes
 filename issue
In-Reply-To: <20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>	<48E1C097.8030309@v.loewis.de>	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>	<48E2865A.3010404@v.loewis.de>	<ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>
	<20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
Message-ID: <48E30A53.5040708@v.loewis.de>

> However, Martin, I can promise you that I will _never_ ask for any
> convenience functions related to bytes as a result of this decision.

:-)

Regards,
Martin

From hodgestar+pythondev at gmail.com  Wed Oct  1 09:52:01 2008
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Wed, 1 Oct 2008 09:52:01 +0200
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <ca471dc20809301505i1151080brb8d34526960c7fc@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
	<fb73205e0809301207g4e212aa8lc5ae2981795d7744@mail.gmail.com>
	<ca471dc20809301505i1151080brb8d34526960c7fc@mail.gmail.com>
Message-ID: <fb73205e0810010052m10cbeab1wc2b54d49a40f36bb@mail.gmail.com>

On Wed, Oct 1, 2008 at 12:05 AM, Guido van Rossum <guido at python.org> wrote:
> Actually on Windows the syscalls use the encoding that Microsoft uses
> -- when using bytes we use the Windows bytes API and when using str we
> use the Windows wide API. That's the most platform-compatible
> approach.

Woot. As long as the Python file API is consistent across the two
platforms, I'm happy. :)

From eckhardt at satorlaser.com  Wed Oct  1 09:54:47 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Wed, 1 Oct 2008 09:54:47 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <48E20017.3020405@egenix.com>
References: <200809291407.55291.victor.stinner@haypocalc.com> 
	<48E1C097.8030309@v.loewis.de> <48E20017.3020405@egenix.com>
Message-ID: <200810010954.47564.eckhardt@satorlaser.com>

On Tuesday 30 September 2008, M.-A. Lemburg wrote:
> On 2008-09-30 08:00, Martin v. L?wis wrote:
> >> Change the default file system encoding to store bytes in Unicode is
> >> like introducing a new Python type: <fake Unicode for filename hacks>.
> >
> > Exactly. Seems like the best solution to me, despite your polemics.
>
> Not a bad idea... have os.listdir() return Unicode subclasses that work
> like file handles, ie. they have an extra buffer that holds the original
> bytes value received from the underlying C API.

Why does it have to be a Unicode subclass? In my eyes, a Unicode object 
promises a few things, in particular that it contains a Unicode string. If it 
now suddenly contains bytes without any further meaning, that would be bad.


What I wonder is what the requirements on path handling are. I'll try to list 
the ones I can see:

1. A path received from the system should be preserved, so it can be given to 
the system later on. IOW, the internal representation should not loose any 
information compared to the one used by the OS.

2. Typical operations like joining two path segments or moving to the parent 
dir should be defined.

3. There must be a way to display the path to the user. IOW, there should be a 
way to turn the path into a string that the user can recognise, according to 
some encoding. Note that this is not always possible, so this can fail.

4. There must be a way to receive a path from the user. That means that there 
must be a way from a user-entered string to a path. Note that this, too, 
isn't always possible and can fail.

5. The conversion between a string and a path should be configurable, defaults 
retrieved from the system. This is so that most operations will just work and 
do the thing that the user expects.

6. There should be a way to modify the path data itself. This of course 
requires knowledge about the internals but gives full power to the 
programmer.


For requirement 3, I would say a lossy conversion to a string would be enough, 
i.e. try to convert the path to a Unicode string and use a question mark or 
some escaping to mark parts that can't be decoded. It will allow users to 
recognise the decodeable parts of the path with hopefully just a few 
characters left without decoding.

For requirement 4, a failure to encode a string to a path must result in a 
loud failure, i.e. an exception. This is because the user entered a path that 
we can't use, any guessing what the user might have wanted is futile.


Are there any points to add?

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From hodgestar+pythondev at gmail.com  Wed Oct  1 10:05:38 2008
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Wed, 1 Oct 2008 10:05:38 +0200
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <ca471dc20809301504h2fc99567o34eb6c947d882c1e@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
	<20080930184751.31635.1484325691.divmod.xquotient.520@weber.divmod.com>
	<ca471dc20809301504h2fc99567o34eb6c947d882c1e@mail.gmail.com>
Message-ID: <fb73205e0810010105u154b4fa8l858bbaac1167861c@mail.gmail.com>

On Wed, Oct 1, 2008 at 12:04 AM, Guido van Rossum <guido at python.org> wrote:
> Plus, even on Linux Unicode is *usually* what you should be doing,
> unless you're writing a backup tool.

I still find this line of reasoning a bit worrying. Imagine an end
user application like a music player. The user discovers that he can't
see some .mp3 or .ogg file from the music player that is visibile is
the file manager. I would expect him to file a bug on the music
player. If the bug was closed with "fix the filename" I imagine the
user would respond with "but other programs can access it just fine".

I'm not unhappy with the solution Victor is proposing, but I imagine
that when I start coding projects in 3.0 I'll default to the bytes
versions of the filename methods and use
b"path".decode(sys.getfilesystemencoding(), "replace") if I need to
get Unicode.

From victor.stinner at haypocalc.com  Wed Oct  1 10:43:25 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Wed, 1 Oct 2008 10:43:25 +0200
Subject: [Python-Dev]
	=?utf-8?q?=5BPython-3000=5D_New_proposition_for_Pyth?=
	=?utf-8?q?on3_bytes=09filename_issue?=
In-Reply-To: <20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>
	<20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
Message-ID: <200810011043.25662.victor.stinner@haypocalc.com>

Le Wednesday 01 October 2008 04:06:25 glyph at divmod.com, vous avez ?crit?:
>     b = gtk.Button(u"\u0000/hello/world")
>
> which emits this message:
>     TypeError: OGtkButton.__init__() argument 1 must be string without
> null bytes or None, not unicode
>
> SQLite has a similar problem with NULLs, and I'm definitely sticking
> paths in there, too.

I think that you can say "all C libraries".

Would it possible to convert the encoded string to bytes just before call Gtk? 
(job done by some Python internals, not as an explicit conversion)

I don't know if it would help the discussion, but Java uses its own modified 
UTF-8 encoding:
 * NUL byte is encoded as 0xc0 0x80 instead of 0x00
 * Java doesn't support unicode > 0xFFFF (bouuuuh!)
http://java.sun.com/javase/6/docs/api/java/io/DataInput.html#modified-utf-8

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From mal at egenix.com  Wed Oct  1 11:32:30 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 01 Oct 2008 11:32:30 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <200810010954.47564.eckhardt@satorlaser.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de> <48E20017.3020405@egenix.com>
	<200810010954.47564.eckhardt@satorlaser.com>
Message-ID: <48E343AE.3080009@egenix.com>

On 2008-10-01 09:54, Ulrich Eckhardt wrote:
> On Tuesday 30 September 2008, M.-A. Lemburg wrote:
>> On 2008-09-30 08:00, Martin v. L?wis wrote:
>>>> Change the default file system encoding to store bytes in Unicode is
>>>> like introducing a new Python type: <fake Unicode for filename hacks>.
>>> Exactly. Seems like the best solution to me, despite your polemics.
>> Not a bad idea... have os.listdir() return Unicode subclasses that work
>> like file handles, ie. they have an extra buffer that holds the original
>> bytes value received from the underlying C API.
> 
> Why does it have to be a Unicode subclass? In my eyes, a Unicode object 
> promises a few things, in particular that it contains a Unicode string. If it 
> now suddenly contains bytes without any further meaning, that would be bad.

Please read my entire email. I was proposing to store the underlying
non-decodeable byte string value in such a subclass. The Unicode value
of the object would then be that underlying value decoded as e.g.
Latin-1 in order to be able to work on it as text.

Path operations would have to be made aware of such subclasses and
operate on the underlying bytes value.

However, like Guido mentioned, this only works if all components are
indeed aware of such subclasses... and that's likely to fail for
code outside the stdlib.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 01 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From stephen at xemacs.org  Wed Oct  1 12:25:36 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 01 Oct 2008 19:25:36 +0900
Subject: [Python-Dev] Patch for an initial support of bytes filename
	in	Python3
In-Reply-To: <fb73205e0810010105u154b4fa8l858bbaac1167861c@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
	<20080930184751.31635.1484325691.divmod.xquotient.520@weber.divmod.com>
	<ca471dc20809301504h2fc99567o34eb6c947d882c1e@mail.gmail.com>
	<fb73205e0810010105u154b4fa8l858bbaac1167861c@mail.gmail.com>
Message-ID: <873ajgpq73.fsf@xemacs.org>

Simon Cross writes:

 > I still find this line of reasoning a bit worrying. Imagine an end
 > user application like a music player. The user discovers that he can't
 > see some .mp3 or .ogg file from the music player that is visibile is
 > the file manager. I would expect him to file a bug on the music
 > player. If the bug was closed with "fix the filename" I imagine the
 > user would respond with "but other programs can access it just fine".

And the user would very likely be *wrong*.  The file manager is
displaying it, but in the nature of things file managers *don't access
files*, they access *directories*.  The files they pass to other apps
to access.

That's precisely the kind of situation that Georg Brandl was
describing with OpenOffice.

 > I'm not unhappy with the solution Victor is proposing, but I imagine
 > that when I start coding projects in 3.0 I'll default to the bytes
 > versions of the filename methods and use
 > b"path".decode(sys.getfilesystemencoding(), "replace") if I need to
 > get Unicode.

But now the user will file a bug because in the file opening dialog
they can't *read* their Chinese file names on their USB key because
they are appearing in (system encoding) Cyrillic.  Do you begin to see
the nature of the Catch-22 here?

I don't expect the user to be very sympathetic when you tell her to
fix the filenames, but it's not as easy as you would think to get this
right.


From hodgestar+pythondev at gmail.com  Wed Oct  1 12:48:35 2008
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Wed, 1 Oct 2008 12:48:35 +0200
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <873ajgpq73.fsf@xemacs.org>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
	<20080930184751.31635.1484325691.divmod.xquotient.520@weber.divmod.com>
	<ca471dc20809301504h2fc99567o34eb6c947d882c1e@mail.gmail.com>
	<fb73205e0810010105u154b4fa8l858bbaac1167861c@mail.gmail.com>
	<873ajgpq73.fsf@xemacs.org>
Message-ID: <fb73205e0810010348ra3a7346s8068e58bf4ca3c7a@mail.gmail.com>

On Wed, Oct 1, 2008 at 12:25 PM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Simon Cross writes:
>
>  > I still find this line of reasoning a bit worrying. Imagine an end
>  > user application like a music player. The user discovers that he can't
>  > see some .mp3 or .ogg file from the music player that is visibile is
>  > the file manager. I would expect him to file a bug on the music
>  > player. If the bug was closed with "fix the filename" I imagine the
>  > user would respond with "but other programs can access it just fine".
>
> And the user would very likely be *wrong*.  The file manager is
> displaying it, but in the nature of things file managers *don't access
> files*, they access *directories*.  The files they pass to other apps
> to access.

Exactly the same reasoning applies to files in a directory with an odd name.

>  > I'm not unhappy with the solution Victor is proposing, but I imagine
>  > that when I start coding projects in 3.0 I'll default to the bytes
>  > versions of the filename methods and use
>  > b"path".decode(sys.getfilesystemencoding(), "replace") if I need to
>  > get Unicode.
>
> But now the user will file a bug because in the file opening dialog
> they can't *read* their Chinese file names on their USB key because
> they are appearing in (system encoding) Cyrillic.  Do you begin to see
> the nature of the Catch-22 here?
>
> I don't expect the user to be very sympathetic when you tell her to
> fix the filenames, but it's not as easy as you would think to get this
> right.

a) There is some chance that at least ASCII characters will be
displayed correctly if getfilesystemencoding() is similar to the
encoding used and corrupted filenames will display correctly except
for corrupted characters.

b) The user will at least be able to access the file.

It's a more graceful degredation of functionality than not being able
to work with the file at all.

Schiavo
Simon

From barry at python.org  Wed Oct  1 14:23:19 2008
From: barry at python.org (Barry Warsaw)
Date: Wed, 1 Oct 2008 08:23:19 -0400
Subject: [Python-Dev] Python 2.6 final today
Message-ID: <1954BD33-02D8-4FC1-A027-97DCDD687A32@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I've been out of town since Friday, but I don't yet see anything in  
the 700 billion email messages I'm now catching up on that leads me to  
think we need to delay the release.  Yay!

I will be on irc later today and will be trolling through the tracker  
and buildbots soon.  Don't trust email to get an important issue in  
front of me today, please use irc or submit a showstopper bug against  
2.6 if something /must/ be addressed before today's release.

I'm going to make a test release at around 1600UTC today, just to see  
how building the docs and such go.  I'm still planning on doing the  
final final release at about 2200UTC.  If you need to coordinate with  
me (e.g. press releases, Windows builds, etc.) please meeting me on  
#python-dev on irc.freenode.net.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSONrt3EjvBPtnXfVAQJn7wP9HSieFM7daE5vbvsuJGZtHyC2NFmT5Rsm
Fd/ce6CvLzGEkUQ5GQs09TtiZZbIYiObUNkbVQBV8Zbu7A9S3fx7PBpHmPOnIIbr
Dfw39pphdKE76yoJmC7OkFTlDbOw6rbuD+JLAzCgcjxx1MqL1Cx08vl2/WEJf3Fl
izAVTI2Bwwc=
=BCvf
-----END PGP SIGNATURE-----

From ncoghlan at gmail.com  Wed Oct  1 14:43:23 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 01 Oct 2008 22:43:23 +1000
Subject: [Python-Dev] [Python-3000] New proposition for Python3	bytes
 filename issue
In-Reply-To: <20081001051947.31635.1251804577.divmod.xquotient.807@weber.divmod.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>	<48E1C097.8030309@v.loewis.de>	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>	<48E2865A.3010404@v.loewis.de>	<ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>	<20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>	<22920D6A-8B70-4E6D-BE99-D7447D831B41@fuhm.net>
	<20081001051947.31635.1251804577.divmod.xquotient.807@weber.divmod.com>
Message-ID: <48E3706B.9060308@gmail.com>

glyph at divmod.com wrote:
> The reasoning is that a lot of software doesn't care if it's wrong for
> edge cases, it's really hard to come up with something that's correct
> with respect to all of those edge cases (absurdly difficult, if you need
> to stay in the straightjacket of string / bytes types, as well as
> provide a useful library interface - which is why we're having this
> discussion).  But, it should be _possible_ to write software that's
> correct in the face of those edge cases.

I just wanted to highlight this as something to keep in mind during this
discussion: we want to keep the easy things easy and make the difficult
things possible.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From techtonik at gmail.com  Wed Oct  1 14:55:30 2008
From: techtonik at gmail.com (techtonik)
Date: Wed, 1 Oct 2008 15:55:30 +0300
Subject: [Python-Dev] Determine minimum required version for a script
Message-ID: <d34314100810010555i3012be78l5019aa1205d0f73e@mail.gmail.com>

Can somebody remind how to check script compatibility with old Python versions?

I can remember PHP_CompatInfo class for PHP that parses a script or directory to
find out the minimum version and extensions required for them to run,
and I wonder
if there was anything like this for Python?

-- 
--anatoly t.

From stephen at xemacs.org  Wed Oct  1 15:53:51 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Wed, 01 Oct 2008 22:53:51 +0900
Subject: [Python-Dev] Patch for an initial support of bytes filename
	in	Python3
In-Reply-To: <fb73205e0810010348ra3a7346s8068e58bf4ca3c7a@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
	<20080930184751.31635.1484325691.divmod.xquotient.520@weber.divmod.com>
	<ca471dc20809301504h2fc99567o34eb6c947d882c1e@mail.gmail.com>
	<fb73205e0810010105u154b4fa8l858bbaac1167861c@mail.gmail.com>
	<873ajgpq73.fsf@xemacs.org>
	<fb73205e0810010348ra3a7346s8068e58bf4ca3c7a@mail.gmail.com>
Message-ID: <87y718o1zk.fsf@xemacs.org>

Simon Cross writes:

 > a) There is some chance that at least ASCII characters will be
 > displayed correctly if getfilesystemencoding() is similar to the
 > encoding used and corrupted filenames will display correctly except
 > for corrupted characters.

All you're saying is that the cases *you* can imagine running into
work better.  All I'm saying is the opposite.  We're both right; the
point is that that means that Python can't be, not all of the time.

We know from experience (Emacs/Mule, Java) that trying to impose a
theoretical system on encoding just doesn't work by itself[1], and in
fact creates other problems by its very rigidity.  I'd like to see
Python not fall into that trap, too.


Footnotes: 
[1]  It needs system-level support as in Windows and Mac OS X.


From guido at python.org  Wed Oct  1 16:15:02 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 1 Oct 2008 07:15:02 -0700
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <fb73205e0810010105u154b4fa8l858bbaac1167861c@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
	<20080930184751.31635.1484325691.divmod.xquotient.520@weber.divmod.com>
	<ca471dc20809301504h2fc99567o34eb6c947d882c1e@mail.gmail.com>
	<fb73205e0810010105u154b4fa8l858bbaac1167861c@mail.gmail.com>
Message-ID: <ca471dc20810010715j6b4e1240gd84fe01004ddf01@mail.gmail.com>

On Wed, Oct 1, 2008 at 1:05 AM, Simon Cross
<hodgestar+pythondev at gmail.com> wrote:
> On Wed, Oct 1, 2008 at 12:04 AM, Guido van Rossum <guido at python.org> wrote:
>> Plus, even on Linux Unicode is *usually* what you should be doing,
>> unless you're writing a backup tool.
>
> I still find this line of reasoning a bit worrying. Imagine an end
> user application like a music player. The user discovers that he can't
> see some .mp3 or .ogg file from the music player that is visibile is
> the file manager. I would expect him to file a bug on the music
> player. If the bug was closed with "fix the filename" I imagine the
> user would respond with "but other programs can access it just fine".

I see nothing wrong with this scenario. If undecodable filenames are a
common thing then the authors of the music player should be using the
bytes variant of the API, and if they get enough bugs like this they
will fix their code to do so. OTOH if this is not common the response
"rename the file" is totally reasonable -- you have to prioritize your
bugs or else you'll never get any software released, and the
occasional work-around is a given.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From gh at ghaering.de  Wed Oct  1 16:50:18 2008
From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=)
Date: Wed, 01 Oct 2008 16:50:18 +0200
Subject: [Python-Dev] Determine minimum required version for a script
In-Reply-To: <d34314100810010555i3012be78l5019aa1205d0f73e@mail.gmail.com>
References: <d34314100810010555i3012be78l5019aa1205d0f73e@mail.gmail.com>
Message-ID: <48E38E2A.6090208@ghaering.de>

techtonik wrote:
> Can somebody remind how to check script compatibility with old Python versions?
> 
> I can remember PHP_CompatInfo class for PHP that parses a script or directory to
> find out the minimum version and extensions required for them to run,
> and I wonder
> if there was anything like this for Python?

You posted on the *python-dev* mailing list.

On this list the key *Python developers* discuss the future of the 
language and its implementation. Topics include Python design issues, 
release mechanics, and maintenance of existing releases.

Please, do not post general Python questions to this list! For help with 
Python please use the mailing list python-list at python.org or the 
newsgroup comp.lang.python. Messages are routed between the two. so 
they're basically the same thing.

-- Gerhard

From janssen at parc.com  Wed Oct  1 17:54:15 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 1 Oct 2008 08:54:15 PDT
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E343AE.3080009@egenix.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de> <48E20017.3020405@egenix.com>
	<200810010954.47564.eckhardt@satorlaser.com>
	<48E343AE.3080009@egenix.com>
Message-ID: <74342.1222876455@parc.com>

M.-A. Lemburg <mal at egenix.com> wrote:
> On 2008-10-01 09:54, Ulrich Eckhardt wrote:
> > On Tuesday 30 September 2008, M.-A. Lemburg wrote:
> >> On 2008-09-30 08:00, Martin v. L?wis wrote:
> >>>> Change the default file system encoding to store bytes in Unicode is
> >>>> like introducing a new Python type: <fake Unicode for filename hacks>.
> >>> Exactly. Seems like the best solution to me, despite your polemics.
> >> Not a bad idea... have os.listdir() return Unicode subclasses that work
> >> like file handles, ie. they have an extra buffer that holds the original
> >> bytes value received from the underlying C API.
> > 
> > Why does it have to be a Unicode subclass? In my eyes, a Unicode object 
> > promises a few things, in particular that it contains a Unicode string. If it 
> > now suddenly contains bytes without any further meaning, that would be bad.
> 
> Please read my entire email. I was proposing to store the underlying
> non-decodeable byte string value in such a subclass. The Unicode value
> of the object would then be that underlying value decoded as e.g.
> Latin-1 in order to be able to work on it as text.

I'm actually sort of liking this idea.  A Pathname class, for convenience
a subtype of String, but containing the underlying binary representation 
used by the OS.  Even non-unicode pathnames could be represented.

Bill

From barry at python.org  Wed Oct  1 18:01:33 2008
From: barry at python.org (Barry Warsaw)
Date: Wed, 1 Oct 2008 12:01:33 -0400
Subject: [Python-Dev] Python security team
In-Reply-To: <48E20D25.4090503@suse.cz>
References: <200809271754.29068.victor.stinner@haypocalc.com>	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>	<48E0C60B.5060006@novell.com>
	<ca471dc20809291016l516cae9ev8b9676ecef60f40@mail.gmail.com>
	<48E20D25.4090503@suse.cz>
Message-ID: <1CD8E216-42C8-454F-AE8B-ABC2D71FB5BC@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 30, 2008, at 7:27 AM, Jan Mate(jek <jmatejek at suse.cz> <Jan  
Mate> wrote:

> Thanks for your answer. I guess the process is the real problem then.
> - From what i could observe, the connection between vendor-sec and  
> PSRT is
> not really working as it should.
> (And then of course you need some kind of upstream flow too, because  
> not
> everyone reports to PSRT.)

Please remember that the proper way to contact the PSRT is via security at python.org 
.

FWIW, I am in favor of adding a few trusted people to the team, but  
only if they're willing to actually get stuff done :).  Clearly the  
current team is too swamped to act effectively, myself included.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOOe3XEjvBPtnXfVAQJ5JgP/dDg+SPLeQ4yBQ/CYxJEh3/Xm2B+2KV5U
9RUjp7W7z2iC/Bz7qwJlui0Z30KaaZ/whMqTuh+5ZYDlrmUDUh9Tl88OyngHOBxy
R/SYmluOlYUPdmjUHQYWXf5Bl9JVX9vtZ3LaFKPUo8KJf+dQDFSK3guxnIr5+Jjt
oJjX+52vilM=
=nJse
-----END PGP SIGNATURE-----

From glyph at divmod.com  Wed Oct  1 18:20:06 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Wed, 01 Oct 2008 16:20:06 -0000
Subject: [Python-Dev] [Python-3000] New proposition for Python3
	bytes	filename issue
In-Reply-To: <74342.1222876455@parc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de> <48E20017.3020405@egenix.com>
	<200810010954.47564.eckhardt@satorlaser.com>
	<48E343AE.3080009@egenix.com> <74342.1222876455@parc.com>
Message-ID: <20081001162006.31635.1753470290.divmod.xquotient.824@weber.divmod.com>


On 03:54 pm, janssen at parc.com wrote:
>I'm actually sort of liking this idea.  A Pathname class, for 
>convenience
>a subtype of String, but containing the underlying binary 
>representation
>used by the OS.  Even non-unicode pathnames could be represented.

On the one hand, I agree with you - except for the part where it's a 
subtype of String, that doesn't work.  In case I haven't mentioned it 
enough times already:

http://twistedmatrix.com/documents/8.1.0/api/twisted.python.filepath.FilePath.html

On the other hand, we've all been on this merry-go-round before:

    http://www.python.org/dev/peps/pep-0355/

Note especially the rejection notice: "Subclassing from str is a 
particularly bad idea".

Again, one day I'd really like to add one of these to Python.  Now is 
not the time.

From janssen at parc.com  Wed Oct  1 19:14:00 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 1 Oct 2008 10:14:00 PDT
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <20081001162006.31635.1753470290.divmod.xquotient.824@weber.divmod.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de> <48E20017.3020405@egenix.com>
	<200810010954.47564.eckhardt@satorlaser.com>
	<48E343AE.3080009@egenix.com> <74342.1222876455@parc.com>
	<20081001162006.31635.1753470290.divmod.xquotient.824@weber.divmod.com>
Message-ID: <75388.1222881240@parc.com>

glyph at divmod.com wrote:

> > I'm actually sort of liking this idea.  A Pathname class, for
> > convenience
> > a subtype of String, but containing the underlying binary
> > representation
> >used by the OS.  Even non-unicode pathnames could be represented.
> 
> On the one hand, I agree with you - except for the part where it's a
> subtype of String, that doesn't work.  In case I haven't mentioned it
> enough times already:
> 
> http://twistedmatrix.com/documents/8.1.0/api/twisted.python.filepath.FilePath.html
> 
> On the other hand, we've all been on this merry-go-round before:
> 
>    http://www.python.org/dev/peps/pep-0355/
> 
> Note especially the rejection notice: "Subclassing from str is a
> particularly bad idea".

Yes, the only real justification for it is to not break existing code
(otherwise, calling str() is not that much of an ordeal).

> On the other hand, we've all been on this merry-go-round before:
> 
>    http://www.python.org/dev/peps/pep-0355/

The very existence of os.path seems a good argument that something like
this is useful.  Perhaps PEP 355 just went too far.

Bill

From martin at v.loewis.de  Wed Oct  1 21:08:50 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 01 Oct 2008 21:08:50 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <200810011043.25662.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>	<20081001020625.31635.800517030.divmod.xquotient.681@weber.divmod.com>
	<200810011043.25662.victor.stinner@haypocalc.com>
Message-ID: <48E3CAC2.6010203@v.loewis.de>

>> SQLite has a similar problem with NULLs, and I'm definitely sticking
>> paths in there, too.
> 
> I think that you can say "all C libraries".

Just for the sake of nit-picking: the socket library, and the regular
POSIX stream IO library (as well as C standard "unformatted" IO) deal
just fine with embedded NULL characters.

>  * Java doesn't support unicode > 0xFFFF (bouuuuh!)

I don't think that is true anymore.

Regards,
Martin

From ncoghlan at gmail.com  Wed Oct  1 23:39:42 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 02 Oct 2008 07:39:42 +1000
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <75388.1222881240@parc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<48E1C097.8030309@v.loewis.de>
	<48E20017.3020405@egenix.com>	<200810010954.47564.eckhardt@satorlaser.com>	<48E343AE.3080009@egenix.com>
	<74342.1222876455@parc.com>	<20081001162006.31635.1753470290.divmod.xquotient.824@weber.divmod.com>
	<75388.1222881240@parc.com>
Message-ID: <48E3EE1E.5000300@gmail.com>

Bill Janssen wrote:
> Perhaps PEP 355 just went too far.

That was certainly one of the major objections to it. A filesystem path
object which didn't try to combine a half-dozen different modules into
methods on a single object, but instead focused on solving a few
specific problems with using raw strings as file paths would have a far
greater chance of acceptance.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From wgheath at gmail.com  Thu Oct  2 04:06:43 2008
From: wgheath at gmail.com (William Heath)
Date: Wed, 1 Oct 2008 19:06:43 -0700
Subject: [Python-Dev] self signing a py2exe winxp executable with signtool
Message-ID: <f08ddf990810011906n5df71276odbbe565984d2fe18@mail.gmail.com>

Hi All,
I am trying to figure out how to self sign a py2exe winxp executable with
signtool.  Anyone know?  I saw this which looked kind of promising:

http://markmail.org/message/zj5nzechzgmjuu7c#query:signtool%20python+page:1+mid:s4jrb2hter4zxvg3+state:results

-Tim

P.S.

Python rocks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081001/d5f27ce1/attachment.htm>

From barry at python.org  Thu Oct  2 05:46:45 2008
From: barry at python.org (Barry Warsaw)
Date: Wed, 1 Oct 2008 23:46:45 -0400
Subject: [Python-Dev] RELEASED Python 2.6 final
Message-ID: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On behalf of the Python development team and the Python community, I  
am happy to announce the release of Python 2.6 final.  This is the  
production-ready version of the latest in the Python 2 series.

There are many new features and modules, improvements, bug fixes, and  
other changes in Python 2.6.  Please see the "What's new" page for  
details

     http://docs.python.org/dev/whatsnew/2.6.html

as well as PEP 361

     http://www.python.org/dev/peps/pep-0361/

While Python 2.6 is backward compatible with earlier versions of  
Python, 2.6 has many tools and features that will help you migrate to  
Python 3.  Wherever possible, Python 3.0 features have been added  
without affecting existing code.  In other cases, the new features can  
be enabled through the use of __future__ imports and command line  
switches.

Python 3.0 is currently in release candidate and will be available  
later this year.  Both Python 2 and Python 3 will be supported for the  
foreseeable future.

Source tarballs, Windows installers, and Mac disk images can be  
downloaded from the Python 2.6 page:

     http://www.python.org/download/releases/2.6/

(Please note that due to quirks in the earth's time zones, the Windows  
installers will be available shortly.)

Bugs can be reported in the Python bug tracker:

     http://bugs.python.org

Enjoy,
- -Barry

Barry Warsaw
barry at python.org
Python 2.6/3.0 Release Manager
(on behalf of the entire python-dev team)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOREJ3EjvBPtnXfVAQLAigP/aEnrdvAqk7wbNQLFbmBonIr2YQbd1vEu
TyTr5imYXFWGNfv1/JMeMBjMfwpHi1bgPEDTLEZdhDRNj/G1h4NqqnpfJS0lfIaU
4JBKwnsO80se/RGyupcs5f09UdKxOljhbFKEw46CHDkd9lE+cqy2yhetEwyx3c3+
AVC11sjcO54=
=Oxo3
-----END PGP SIGNATURE-----

From aahz at pythoncraft.com  Thu Oct  2 05:55:40 2008
From: aahz at pythoncraft.com (Aahz)
Date: Wed, 1 Oct 2008 20:55:40 -0700
Subject: [Python-Dev] RELEASED Python 2.6 final
In-Reply-To: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
References: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
Message-ID: <20081002035540.GA3865@panix.com>

Huzzah!
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"...if I were on life-support, I'd rather have it run by a Gameboy than a
Windows box."  --Cliff Wells, comp.lang.python, 3/13/2002

From guido at python.org  Thu Oct  2 05:59:02 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 1 Oct 2008 20:59:02 -0700
Subject: [Python-Dev] RELEASED Python 2.6 final
In-Reply-To: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
References: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
Message-ID: <ca471dc20810012059j7125af12ub41be2a65a136e03@mail.gmail.com>

Congratulations, Barry!!!

On Wed, Oct 1, 2008 at 8:46 PM, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On behalf of the Python development team and the Python community, I am
> happy to announce the release of Python 2.6 final.  This is the
> production-ready version of the latest in the Python 2 series.
>
> There are many new features and modules, improvements, bug fixes, and other
> changes in Python 2.6.  Please see the "What's new" page for details
>
>    http://docs.python.org/dev/whatsnew/2.6.html
>
> as well as PEP 361
>
>    http://www.python.org/dev/peps/pep-0361/
>
> While Python 2.6 is backward compatible with earlier versions of Python, 2.6
> has many tools and features that will help you migrate to Python 3.
>  Wherever possible, Python 3.0 features have been added without affecting
> existing code.  In other cases, the new features can be enabled through the
> use of __future__ imports and command line switches.
>
> Python 3.0 is currently in release candidate and will be available later
> this year.  Both Python 2 and Python 3 will be supported for the foreseeable
> future.
>
> Source tarballs, Windows installers, and Mac disk images can be downloaded
> from the Python 2.6 page:
>
>    http://www.python.org/download/releases/2.6/
>
> (Please note that due to quirks in the earth's time zones, the Windows
> installers will be available shortly.)

Can someone who's still up fix add this note to the website? It looks
a little dodgy just linking to a 404 error... :-(

> Bugs can be reported in the Python bug tracker:
>
>    http://bugs.python.org
>
> Enjoy,
> - -Barry
>
> Barry Warsaw
> barry at python.org
> Python 2.6/3.0 Release Manager
> (on behalf of the entire python-dev team)
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (Darwin)
>
> iQCVAwUBSOREJ3EjvBPtnXfVAQLAigP/aEnrdvAqk7wbNQLFbmBonIr2YQbd1vEu
> TyTr5imYXFWGNfv1/JMeMBjMfwpHi1bgPEDTLEZdhDRNj/G1h4NqqnpfJS0lfIaU
> 4JBKwnsO80se/RGyupcs5f09UdKxOljhbFKEw46CHDkd9lE+cqy2yhetEwyx3c3+
> AVC11sjcO54=
> =Oxo3
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From steve at pearwood.info  Thu Oct  2 06:39:36 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 2 Oct 2008 14:39:36 +1000
Subject: [Python-Dev] RELEASED Python 2.6 final
In-Reply-To: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
References: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
Message-ID: <200810021439.36718.steve@pearwood.info>

On Thu, 2 Oct 2008 01:46:45 pm Barry Warsaw wrote:
> On behalf of the Python development team and the Python community, I
> am happy to announce the release of Python 2.6 final.  This is the
> production-ready version of the latest in the Python 2 series.

I'd like to thank you all very much for your hard work and for making 
such a great language. Cheers!



-- 
Steven

From divinekid at gmail.com  Thu Oct  2 06:19:32 2008
From: divinekid at gmail.com (Haoyu Bai)
Date: Thu, 2 Oct 2008 12:19:32 +0800
Subject: [Python-Dev] RELEASED Python 2.6 final
In-Reply-To: <ca471dc20810012059j7125af12ub41be2a65a136e03@mail.gmail.com>
References: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
	<ca471dc20810012059j7125af12ub41be2a65a136e03@mail.gmail.com>
Message-ID: <1d7983e80810012119u6f51b460lbdef685a035c0538@mail.gmail.com>

On Thu, Oct 2, 2008 at 11:59 AM, Guido van Rossum <guido at python.org> wrote:
> Congratulations, Barry!!!
>
> On Wed, Oct 1, 2008 at 8:46 PM, Barry Warsaw <barry at python.org> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On behalf of the Python development team and the Python community, I am
>> happy to announce the release of Python 2.6 final.  This is the
>> production-ready version of the latest in the Python 2 series.
>>
>> There are many new features and modules, improvements, bug fixes, and other
>> changes in Python 2.6.  Please see the "What's new" page for details
>>
>>    http://docs.python.org/dev/whatsnew/2.6.html
>>
>> as well as PEP 361
>>
>>    http://www.python.org/dev/peps/pep-0361/
>>
>> While Python 2.6 is backward compatible with earlier versions of Python, 2.6
>> has many tools and features that will help you migrate to Python 3.
>>  Wherever possible, Python 3.0 features have been added without affecting
>> existing code.  In other cases, the new features can be enabled through the
>> use of __future__ imports and command line switches.
>>
>> Python 3.0 is currently in release candidate and will be available later
>> this year.  Both Python 2 and Python 3 will be supported for the foreseeable
>> future.
>>
>> Source tarballs, Windows installers, and Mac disk images can be downloaded
>> from the Python 2.6 page:
>>
>>    http://www.python.org/download/releases/2.6/
>>
>> (Please note that due to quirks in the earth's time zones, the Windows
>> installers will be available shortly.)
>
> Can someone who's still up fix add this note to the website? It looks
> a little dodgy just linking to a 404 error... :-(
>
>> Bugs can be reported in the Python bug tracker:
>>
>>    http://bugs.python.org
>>
>> Enjoy,
>> - -Barry
>>
>> Barry Warsaw
>> barry at python.org
>> Python 2.6/3.0 Release Manager
>> (on behalf of the entire python-dev team)
>>
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG v1.4.9 (Darwin)
>>
>> iQCVAwUBSOREJ3EjvBPtnXfVAQLAigP/aEnrdvAqk7wbNQLFbmBonIr2YQbd1vEu
>> TyTr5imYXFWGNfv1/JMeMBjMfwpHi1bgPEDTLEZdhDRNj/G1h4NqqnpfJS0lfIaU
>> 4JBKwnsO80se/RGyupcs5f09UdKxOljhbFKEw46CHDkd9lE+cqy2yhetEwyx3c3+
>> AVC11sjcO54=
>> =Oxo3
>> -----END PGP SIGNATURE-----
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>>
>
>
>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)

Now almost all the pages on docs.python.org can't be accessed. For
example http://docs.python.org/lib/lib.html returns 403 forbidden.

Is the online docs under updating to 2.6, or there's something wrong?

-- 
Haoyu Bai

From lists at cheimes.de  Thu Oct  2 08:08:17 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 02 Oct 2008 08:08:17 +0200
Subject: [Python-Dev] RELEASED Python 2.6 final
In-Reply-To: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
References: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
Message-ID: <gc1ogi$ak6$1@ger.gmane.org>

Nice!

Python 2.7 is waiting, let's get started! :)

Christian


From martin at v.loewis.de  Thu Oct  2 08:28:27 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 02 Oct 2008 08:28:27 +0200
Subject: [Python-Dev] self signing a py2exe winxp executable with
	signtool
In-Reply-To: <f08ddf990810011906n5df71276odbbe565984d2fe18@mail.gmail.com>
References: <f08ddf990810011906n5df71276odbbe565984d2fe18@mail.gmail.com>
Message-ID: <48E46A0B.6070806@v.loewis.de>

> I am trying to figure out how to self sign a py2exe winxp executable
> with signtool.  Anyone know?  
Dear William,

This list (python-dev) is for the development of Python, not the
development with Python. I recommend to use either python-list, or
the py2exe-users list for this question.

Regards,
Martin

From itconsense at gmail.com  Thu Oct  2 09:57:43 2008
From: itconsense at gmail.com (itconsense at gmail.com)
Date: Thu, 02 Oct 2008 09:57:43 +0200
Subject: [Python-Dev] python docs website not accessible!
Message-ID: <gc1usj$qi3$1@ger.gmane.org>

Hi,

    I'm not sure, if this is the right place to post. The python-docs on www.python.org are not accessible.

The overview http://docs.python.org/lib/lib.html works fine, but no link on the page i have tried works.

http://docs.python.org/lib/doctest-unittest-api.html
http://docs.python.org/lib/profile-instant.html

Is python ceasing to be open source with 2.6? ;-)

Thanks for caring,
-Tom



From thomas at python.org  Thu Oct  2 11:59:11 2008
From: thomas at python.org (Thomas Wouters)
Date: Thu, 2 Oct 2008 11:59:11 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
Message-ID: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>

I hotfixed docs.python.org and www.python.org/doc with some cutesy improv --
the URLs changed from .../lib/ to ../library/, and any HTML pages inside
them are completely different. So, any http://docs.python.org/lib/... URL
now redirects to the toplevel http://docs.python.org/library/ (and similar
for www.python.org/doc/lib.) If anyone feels particularly frustrated by the
old URLs breaking, I wouldn't mind adding a redirection for each individual
URL as long as I don't have to build that mapping :-)

Georg is working on fixing the main www.python.org/doc page, I believe, as
well as providing downloadable docs.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081002/967eb064/attachment.htm>

From solipsis at pitrou.net  Thu Oct  2 12:44:49 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 2 Oct 2008 10:44:49 +0000 (UTC)
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
Message-ID: <loom.20081002T104308-982@post.gmane.org>

Thomas Wouters <thomas <at> python.org> writes:
> 
> If anyone feels particularly frustrated by the old URLs breaking, I wouldn't
mind adding a redirection for each individual URL as long as I don't have to
build that mapping

Well in general URLs aren't supposed to break (except the ones which are
deliberately temporary). Could a RewriteRule do the trick?



From thomas at python.org  Thu Oct  2 13:28:18 2008
From: thomas at python.org (Thomas Wouters)
Date: Thu, 2 Oct 2008 13:28:18 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <loom.20081002T104308-982@post.gmane.org>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
Message-ID: <9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>

On Thu, Oct 2, 2008 at 12:44, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Thomas Wouters <thomas <at> python.org> writes:
> >
> > If anyone feels particularly frustrated by the old URLs breaking, I
> wouldn't
> mind adding a redirection for each individual URL as long as I don't have
> to
> build that mapping
>
> Well in general URLs aren't supposed to break (except the ones which are
> deliberately temporary). Could a RewriteRule do the trick?
>

Not a single one, no. The URLs *all* changed. There is not a single one
that's the same. We may be able to do a single rewrite rule for most of the
module-*.html URLs, but everything else -- and there is quite a lot of
'else' in the 2.5-and-earlier docs -- needs a better mapping. Feel free to
send me that mapping :-)

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081002/5b84c4f5/attachment.htm>

From doug.hellmann at gmail.com  Thu Oct  2 13:52:16 2008
From: doug.hellmann at gmail.com (Doug Hellmann)
Date: Thu, 2 Oct 2008 07:52:16 -0400
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
Message-ID: <C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>


On Oct 2, 2008, at 7:28 AM, Thomas Wouters wrote:

>
>
> On Thu, Oct 2, 2008 at 12:44, Antoine Pitrou <solipsis at pitrou.net>  
> wrote:
> Thomas Wouters <thomas <at> python.org> writes:
> >
> > If anyone feels particularly frustrated by the old URLs breaking,  
> I wouldn't
> mind adding a redirection for each individual URL as long as I don't  
> have to
> build that mapping
>
> Well in general URLs aren't supposed to break (except the ones which  
> are
> deliberately temporary). Could a RewriteRule do the trick?
>
> Not a single one, no. The URLs *all* changed. There is not a single  
> one that's the same. We may be able to do a single rewrite rule for  
> most of the module-*.html URLs, but everything else -- and there is  
> quite a lot of 'else' in the 2.5-and-earlier docs -- needs a better  
> mapping. Feel free to send me that mapping :-)

Perhaps it has already been suggested and rejected for some reason,  
but we could include the major/minor version numbers in the URLs.   
That would make it easier to rewrite old URLs, and I assume there will  
be 2.x and 3.x documentation available online for some period of time?

docs.python.org/lib/* could redirect to docs.python.org/2.5/lib/*

docs.pyhton.org/ (note no *) could redirect to docs.python.org/2.6/  
and include a link to docs.python.org/3.0/

That way all of the old references (in Google and bookmarks) would  
still work.

Perhaps we should restore the old version of the files until this is  
resolved?  Being redirected to the top landing page is a little  
disconcerting if you come to the site through a search engine and  
aren't familiar with the organization of the manual.  For example, I  
went to look for the documentation on how slots work, and ended up at  
the top of the reference manual.  The local search didn't work (no  
results), "slots" isn't in the index, and google still has the old URL.

Doug

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081002/ebc39b17/attachment.htm>

From solipsis at pitrou.net  Thu Oct  2 13:35:11 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 02 Oct 2008 13:35:11 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
Message-ID: <1222947311.7054.3.camel@fsol>


> 
> Not a single one, no. The URLs *all* changed. There is not a single
> one that's the same. We may be able to do a single rewrite rule for
> most of the module-*.html URLs, but everything else -- and there is
> quite a lot of 'else' in the 2.5-and-earlier docs -- needs a better
> mapping. Feel free to send me that mapping :-)

My bad. I thought it was just a matter of doing a generic substitution.
Well, then we'll have to live with it I suppose :)

Regards

Antoine.



From barry at python.org  Thu Oct  2 14:11:47 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 2 Oct 2008 08:11:47 -0400
Subject: [Python-Dev] RELEASED Python 2.6 final
In-Reply-To: <1d7983e80810012119u6f51b460lbdef685a035c0538@mail.gmail.com>
References: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
	<ca471dc20810012059j7125af12ub41be2a65a136e03@mail.gmail.com>
	<1d7983e80810012119u6f51b460lbdef685a035c0538@mail.gmail.com>
Message-ID: <7E7417E8-8BA8-4974-94F7-ABC0A143F710@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 2, 2008, at 12:19 AM, Haoyu Bai wrote:

> Now almost all the pages on docs.python.org can't be accessed. For
> example http://docs.python.org/lib/lib.html returns 403 forbidden.

Thanks to Georg and Thomas, the docs should all be fixed now.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOS6g3EjvBPtnXfVAQKw5wP+I2L6qPZWp1qDs7qSRdlOE5xSAazhnuzE
h7gCUWah0tcewuJC38cE7kNAVkpmp9suBbGgI+FRYTeJJpoO109Io4cF4fRvYd2H
NpVfhIOo6VUchNTnsdtP4UzuaIKnkCKgWxMPPjMW9jEZlHPNdOC8stTsxOq1FWFt
hlJscML5yQA=
=wPLO
-----END PGP SIGNATURE-----

From g.brandl at gmx.net  Thu Oct  2 14:17:37 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Oct 2008 14:17:37 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
Message-ID: <gc2e55$bpb$1@ger.gmane.org>

Doug Hellmann schrieb:

>> Not a single one, no. The URLs *all* changed. There is not a single
>> one that's the same. We may be able to do a single rewrite rule for
>> most of the module-*.html URLs, but everything else -- and there is
>> quite a lot of 'else' in the 2.5-and-earlier docs -- needs a better
>> mapping. Feel free to send me that mapping :-)
> 
> Perhaps it has already been suggested and rejected for some reason, but
> we could include the major/minor version numbers in the URLs.  That
> would make it easier to rewrite old URLs, and I assume there will be 2.x
> and 3.x documentation available online for some period of time?
> 
> docs.python.org/lib/* could redirect to docs.python.org/2.5/lib/*

That would be possible, but not sensible IMO -- it doesn't make people
update their links, instead keeps links to outdated documentation.

> docs.pyhton.org/ (note no *) could redirect to docs.python.org/2.6/ and
> include a link to docs.python.org/3.0/

We already have archived versioned docs at http://www.python.org/doc/X.Y.

> That way all of the old references (in Google and bookmarks) would still
> work.
> 
> Perhaps we should restore the old version of the files until this is
> resolved?  Being redirected to the top landing page is a little
> disconcerting if you come to the site through a search engine and aren't
> familiar with the organization of the manual.  For example, I went to
> look for the documentation on how slots work, and ended up at the top of
> the reference manual.  The local search didn't work (no results),
> "slots" isn't in the index, and google still has the old URL.

__slots__ is in the index (with the underscores). The local search shows me
__slots__ as the first result when I search for "__slots__" or "slots".

As for Google, I can only assume it will soon update its index.

Nevertheless, I will come up with a mapping for the old module URLs,
which is relatively easy.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ncoghlan at gmail.com  Thu Oct  2 14:18:02 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 02 Oct 2008 22:18:02 +1000
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
Message-ID: <48E4BBFA.7010800@gmail.com>

Doug Hellmann wrote:
> Perhaps it has already been suggested and rejected for some reason, but
> we could include the major/minor version numbers in the URLs.  That
> would make it easier to rewrite old URLs, and I assume there will be 2.x
> and 3.x documentation available online for some period of time?

The old doc directories are already kept around (all the way back to 1.4
in fact: http://www.python.org/doc/1.4/)

As a quick fix for the old links, a rewrite rule to map such links to
the 2.5 docs seems like a very good idea to me. Since old URLs all use
abbreviations in the directory name (tut, lib, mac, ref, ext, api, doc,
inst, dist), it should be straightforward to redirect them without
affecting the links to the new docs (tutorial, using, reference, howto,
extending, c-api, install, distutils, documenting).


> Perhaps we should restore the old version of the files until this is
> resolved?  Being redirected to the top landing page is a little
> disconcerting if you come to the site through a search engine and aren't
> familiar with the organization of the manual.

A redirect rule to the 2.5.2 docs for the old naming scheme is probably
a better short-term solution.

>  For example, I went to
> look for the documentation on how slots work, and ended up at the top of
> the reference manual.  The local search didn't work (no results),
> "slots" isn't in the index, and google still has the old URL.

The quick search is actually working for me these days (it wasn't for a
while when the new docs were still in development). (e.g. the first hit
I get when searching for "slots" now is
http://www.python.org/doc/2.6/reference/datamodel.html?highlight=slots#__slots__)

I believe it's a Javascript based search though, so there may be issues
with browser compatibility (or user's with JS disabled).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From rhamph at gmail.com  Thu Oct  2 14:25:56 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Thu, 2 Oct 2008 06:25:56 -0600
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <gc2e55$bpb$1@ger.gmane.org>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
	<gc2e55$bpb$1@ger.gmane.org>
Message-ID: <aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>

On Thu, Oct 2, 2008 at 6:17 AM, Georg Brandl <g.brandl at gmx.net> wrote:
> Doug Hellmann schrieb:
>
>>> Not a single one, no. The URLs *all* changed. There is not a single
>>> one that's the same. We may be able to do a single rewrite rule for
>>> most of the module-*.html URLs, but everything else -- and there is
>>> quite a lot of 'else' in the 2.5-and-earlier docs -- needs a better
>>> mapping. Feel free to send me that mapping :-)
>>
>> Perhaps it has already been suggested and rejected for some reason, but
>> we could include the major/minor version numbers in the URLs.  That
>> would make it easier to rewrite old URLs, and I assume there will be 2.x
>> and 3.x documentation available online for some period of time?
>>
>> docs.python.org/lib/* could redirect to docs.python.org/2.5/lib/*
>
> That would be possible, but not sensible IMO -- it doesn't make people
> update their links, instead keeps links to outdated documentation.
>
>> docs.pyhton.org/ (note no *) could redirect to docs.python.org/2.6/ and
>> include a link to docs.python.org/3.0/
>
> We already have archived versioned docs at http://www.python.org/doc/X.Y.

Why not use versioned URLs, but with a  link at the top of old pages
saying they're outdated, linking to the new version.  Either way they
should update their links, but this way you don't shoot them in the
foot to do it.

Breaking old links should be avoided if at all possible.


-- 
Adam Olsen, aka Rhamphoryncus

From ncoghlan at gmail.com  Thu Oct  2 14:33:27 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 02 Oct 2008 22:33:27 +1000
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <gc2e55$bpb$1@ger.gmane.org>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
	<gc2e55$bpb$1@ger.gmane.org>
Message-ID: <48E4BF97.50706@gmail.com>

Georg Brandl wrote:
> Nevertheless, I will come up with a mapping for the old module URLs,
> which is relatively easy.

Best solution of all :)

I was actually only suggesting redirecting to the old docs until such a
mapping was available - but if that mapping will be available fairly
soon, then bumping old links up to the base URL for a day or two won't
be too bad.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From g.brandl at gmx.net  Thu Oct  2 14:34:19 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Oct 2008 14:34:19 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>	<gc2e55$bpb$1@ger.gmane.org>
	<aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>
Message-ID: <gc2f4f$ei8$1@ger.gmane.org>

Adam Olsen schrieb:
> On Thu, Oct 2, 2008 at 6:17 AM, Georg Brandl <g.brandl at gmx.net> wrote:
>> Doug Hellmann schrieb:
>>
>>>> Not a single one, no. The URLs *all* changed. There is not a single
>>>> one that's the same. We may be able to do a single rewrite rule for
>>>> most of the module-*.html URLs, but everything else -- and there is
>>>> quite a lot of 'else' in the 2.5-and-earlier docs -- needs a better
>>>> mapping. Feel free to send me that mapping :-)
>>>
>>> Perhaps it has already been suggested and rejected for some reason, but
>>> we could include the major/minor version numbers in the URLs.  That
>>> would make it easier to rewrite old URLs, and I assume there will be 2.x
>>> and 3.x documentation available online for some period of time?
>>>
>>> docs.python.org/lib/* could redirect to docs.python.org/2.5/lib/*
>>
>> That would be possible, but not sensible IMO -- it doesn't make people
>> update their links, instead keeps links to outdated documentation.
>>
>>> docs.pyhton.org/ (note no *) could redirect to docs.python.org/2.6/ and
>>> include a link to docs.python.org/3.0/
>>
>> We already have archived versioned docs at http://www.python.org/doc/X.Y.
> 
> Why not use versioned URLs, but with a  link at the top of old pages
> saying they're outdated, linking to the new version.  Either way they
> should update their links, but this way you don't shoot them in the
> foot to do it.

If linking to the new version could be done easily, we could as well directly
redirect. The problem is that having that mapping in the first place is hard.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From doug.hellmann at gmail.com  Thu Oct  2 14:35:00 2008
From: doug.hellmann at gmail.com (Doug Hellmann)
Date: Thu, 2 Oct 2008 08:35:00 -0400
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <gc2e55$bpb$1@ger.gmane.org>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
	<gc2e55$bpb$1@ger.gmane.org>
Message-ID: <9A6A51AD-6EA2-4CF3-8734-CAFD76CAF421@gmail.com>


On Oct 2, 2008, at 8:17 AM, Georg Brandl wrote:

> Doug Hellmann schrieb:
>
>>> Not a single one, no. The URLs *all* changed. There is not a single
>>> one that's the same. We may be able to do a single rewrite rule for
>>> most of the module-*.html URLs, but everything else -- and there is
>>> quite a lot of 'else' in the 2.5-and-earlier docs -- needs a better
>>> mapping. Feel free to send me that mapping :-)
>>
>> Perhaps it has already been suggested and rejected for some reason,  
>> but
>> we could include the major/minor version numbers in the URLs.  That
>> would make it easier to rewrite old URLs, and I assume there will  
>> be 2.x
>> and 3.x documentation available online for some period of time?
>>
>> docs.python.org/lib/* could redirect to docs.python.org/2.5/lib/*
>
> That would be possible, but not sensible IMO -- it doesn't make people
> update their links, instead keeps links to outdated documentation.

The documentation isn't outdated if you're still running Python 2.5,  
as a lot of people will be.  Not everyone gets to upgrade right away  
when there's a new release.  For example, the product we build at work  
depends on 2.5 and we don't have time in our schedule to upgrade right  
away.  It may be several months before we do.

>> docs.pyhton.org/ (note no *) could redirect to docs.python.org/2.6/  
>> and
>> include a link to docs.python.org/3.0/
>
> We already have archived versioned docs at http://www.python.org/doc/X.Y 
> .

Great, so we can just redirect the old links over there.  If you can  
make them point to the correct form of the new docs, that would be  
even better, but at least sending them to the old docs means they  
point to *something* useful.

>> That way all of the old references (in Google and bookmarks) would  
>> still
>> work.
>>
>> Perhaps we should restore the old version of the files until this is
>> resolved?  Being redirected to the top landing page is a little
>> disconcerting if you come to the site through a search engine and  
>> aren't
>> familiar with the organization of the manual.  For example, I went to
>> look for the documentation on how slots work, and ended up at the  
>> top of
>> the reference manual.  The local search didn't work (no results),
>> "slots" isn't in the index, and google still has the old URL.
>
> __slots__ is in the index (with the underscores). The local search  
> shows me
> __slots__ as the first result when I search for "__slots__" or  
> "slots".

OK, searching for "slots" at http://docs.python.org found several  
results this time.  I don't know why it would have given me no results  
the last time, but I found what I needed.

Doug


From doug.hellmann at gmail.com  Thu Oct  2 14:46:37 2008
From: doug.hellmann at gmail.com (Doug Hellmann)
Date: Thu, 2 Oct 2008 08:46:37 -0400
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <gc2f4f$ei8$1@ger.gmane.org>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>	<gc2e55$bpb$1@ger.gmane.org>
	<aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>
	<gc2f4f$ei8$1@ger.gmane.org>
Message-ID: <F477AE16-22BE-47FB-A1EA-84B09E37C6C7@gmail.com>


On Oct 2, 2008, at 8:34 AM, Georg Brandl wrote:

>
> If linking to the new version could be done easily, we could as well  
> directly
> redirect. The problem is that having that mapping in the first place  
> is hard.


I was looking for the easy route.  If the layout of the new docs  
changed completely, anything that starts with the old abbreviations (/ 
lib/, /tut/, /ref/, etc.) could just go over to the 2.5.2 docs,  
right?  You don't need to map every sub-section to its new URL unless  
you feel really strongly that links to pages in the old organization  
should point to the new location.

Doug


From jnoller at gmail.com  Thu Oct  2 15:08:34 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 2 Oct 2008 09:08:34 -0400
Subject: [Python-Dev] Doc nits question
Message-ID: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>

So, we just released and there are a few doc typo bugs being filed -
my question is if all doc-fixes have to wait for 2.6.1/2.7 or if we
can hotfix the 2.6 docs?

-jesse

From tseaver at palladion.com  Thu Oct  2 15:14:58 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 02 Oct 2008 09:14:58 -0400
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <gc2f4f$ei8$1@ger.gmane.org>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>	<gc2e55$bpb$1@ger.gmane.org>	<aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>
	<gc2f4f$ei8$1@ger.gmane.org>
Message-ID: <48E4C952.7020705@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Georg Brandl wrote:
> Adam Olsen schrieb:
>> On Thu, Oct 2, 2008 at 6:17 AM, Georg Brandl <g.brandl at gmx.net> wrote:
>>> Doug Hellmann schrieb:
>>>
>>>>> Not a single one, no. The URLs *all* changed. There is not a single
>>>>> one that's the same. We may be able to do a single rewrite rule for
>>>>> most of the module-*.html URLs, but everything else -- and there is
>>>>> quite a lot of 'else' in the 2.5-and-earlier docs -- needs a better
>>>>> mapping. Feel free to send me that mapping :-)
>>>> Perhaps it has already been suggested and rejected for some reason, but
>>>> we could include the major/minor version numbers in the URLs.  That
>>>> would make it easier to rewrite old URLs, and I assume there will be 2.x
>>>> and 3.x documentation available online for some period of time?
>>>>
>>>> docs.python.org/lib/* could redirect to docs.python.org/2.5/lib/*
>>> That would be possible, but not sensible IMO -- it doesn't make people
>>> update their links, instead keeps links to outdated documentation.
>>>
>>>> docs.pyhton.org/ (note no *) could redirect to docs.python.org/2.6/ and
>>>> include a link to docs.python.org/3.0/
>>> We already have archived versioned docs at http://www.python.org/doc/X.Y.
>> Why not use versioned URLs, but with a  link at the top of old pages
>> saying they're outdated, linking to the new version.  Either way they
>> should update their links, but this way you don't shoot them in the
>> foot to do it.
> 
> If linking to the new version could be done easily, we could as well directly
> redirect. The problem is that having that mapping in the first place is hard.

Why would you remove the old docs (ones with 2.5 in the URL)?  They
still provide value for folks who can't yet move to 2.6 / 3.0;  forcibly
redirecting a versioned URL to "current" can't possibley be sane.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFI5MlR+gerLs4ltQ4RAicWAKC6gxTtxq/CwZXH9SekRs7DD1fFTwCeMyb/
eJqkvkb4zdDGZG8oPvp1GjI=
=0Atv
-----END PGP SIGNATURE-----


From g.brandl at gmx.net  Thu Oct  2 15:19:43 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Oct 2008 15:19:43 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <48E4C952.7020705@palladion.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>	<gc2e55$bpb$1@ger.gmane.org>	<aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>	<gc2f4f$ei8$1@ger.gmane.org>
	<48E4C952.7020705@palladion.com>
Message-ID: <gc2hpj$p2j$1@ger.gmane.org>

Tres Seaver schrieb:
> Georg Brandl wrote:

>>>>> docs.pyhton.org/ (note no *) could redirect to docs.python.org/2.6/ and
>>>>> include a link to docs.python.org/3.0/
>>>> We already have archived versioned docs at http://www.python.org/doc/X.Y.
>>> Why not use versioned URLs, but with a  link at the top of old pages
>>> saying they're outdated, linking to the new version.  Either way they
>>> should update their links, but this way you don't shoot them in the
>>> foot to do it.
> 
>> If linking to the new version could be done easily, we could as well directly
>> redirect. The problem is that having that mapping in the first place is hard.
> 
> Why would you remove the old docs (ones with 2.5 in the URL)?  They
> still provide value for folks who can't yet move to 2.6 / 3.0;  forcibly
> redirecting a versioned URL to "current" can't possibley be sane.

That's true; it's also not what I meant. The versioned docs will of course
always stay there. The question is what to do for URLs that refer to
docs.python.org, but with old filenames.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From g.brandl at gmx.net  Thu Oct  2 15:21:10 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Oct 2008 15:21:10 +0200
Subject: [Python-Dev] Doc nits question
In-Reply-To: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>
Message-ID: <gc2hsa$p2j$2@ger.gmane.org>

Jesse Noller schrieb:
> So, we just released and there are a few doc typo bugs being filed -
> my question is if all doc-fixes have to wait for 2.6.1/2.7 or if we
> can hotfix the 2.6 docs?

I intend to set things up so that the docs at docs.python.org are continually
rebuilt, just like the /dev docs were until now.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From jnoller at gmail.com  Thu Oct  2 15:26:10 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 2 Oct 2008 09:26:10 -0400
Subject: [Python-Dev] Doc nits question
In-Reply-To: <gc2hsa$p2j$2@ger.gmane.org>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>
	<gc2hsa$p2j$2@ger.gmane.org>
Message-ID: <4222a8490810020626n3b3c75bo39f4b4552d6773d7@mail.gmail.com>

On Thu, Oct 2, 2008 at 9:21 AM, Georg Brandl <g.brandl at gmx.net> wrote:
> Jesse Noller schrieb:
>> So, we just released and there are a few doc typo bugs being filed -
>> my question is if all doc-fixes have to wait for 2.6.1/2.7 or if we
>> can hotfix the 2.6 docs?
>
> I intend to set things up so that the docs at docs.python.org are continually
> rebuilt, just like the /dev docs were until now.
>
> Georg

Fantastic, so the doc updates should go to the 2.6 branch, correct?
(Not that I'm suggesting checking in all willy nilly)

From ncoghlan at gmail.com  Thu Oct  2 15:35:45 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 02 Oct 2008 23:35:45 +1000
Subject: [Python-Dev] Doc nits question
In-Reply-To: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>
Message-ID: <48E4CE31.2010505@gmail.com>

Jesse Noller wrote:
> So, we just released and there are a few doc typo bugs being filed -
> my question is if all doc-fixes have to wait for 2.6.1/2.7 or if we
> can hotfix the 2.6 docs?

Well the fixes can definitely all go in to SVN on both the trunk and the
maintenance branch.

As to when we update docs.python.org from the maintenance branch... I
believe historically it has only been done at each new maintenance
release, but I don't see any fundamental problems with the idea of
updating it more frequently.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From g.brandl at gmx.net  Thu Oct  2 15:36:51 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Oct 2008 15:36:51 +0200
Subject: [Python-Dev] Doc nits question
In-Reply-To: <4222a8490810020626n3b3c75bo39f4b4552d6773d7@mail.gmail.com>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>	<gc2hsa$p2j$2@ger.gmane.org>
	<4222a8490810020626n3b3c75bo39f4b4552d6773d7@mail.gmail.com>
Message-ID: <gc2ipo$t6p$1@ger.gmane.org>

Jesse Noller schrieb:
> On Thu, Oct 2, 2008 at 9:21 AM, Georg Brandl <g.brandl at gmx.net> wrote:
>> Jesse Noller schrieb:
>>> So, we just released and there are a few doc typo bugs being filed -
>>> my question is if all doc-fixes have to wait for 2.6.1/2.7 or if we
>>> can hotfix the 2.6 docs?
>>
>> I intend to set things up so that the docs at docs.python.org are continually
>> rebuilt, just like the /dev docs were until now.
>>
>> Georg
> 
> Fantastic, so the doc updates should go to the 2.6 branch, correct?
> (Not that I'm suggesting checking in all willy nilly)

This is another thing that needs to be discussed: how to handle backports
between 2.6 and 2.7. Up to now, we backported changes from trunk to maint
manually, but after the experience we've had with svnmerge, I see several
possibilities:

1. Do bugfixes in maint, merge them to trunk via svnmerge. This has the
   drawback that you have to work in two different repos for fixes vs.
   new features. The advantage however is that normally all fixes that
   go into maint apply to trunk as well, so almost no blocks need to be done.
   However, since Py3k merges are done from trunk, the 3k merge will see
   merges as single commits, so they aren't easy to block if not applicable.
   This will mean more conflicts.

2. Do bugfixes in trunk, and merge them to maint via svnmerge.
   Arguments as for 1, but reversed: many blocks, but less problems with 3k.

3. Backport bugfixes manually, like for the previous maintenance branches.

cheers,
Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ncoghlan at gmail.com  Thu Oct  2 15:38:37 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 02 Oct 2008 23:38:37 +1000
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <gc2hpj$p2j$1@ger.gmane.org>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>	<gc2e55$bpb$1@ger.gmane.org>	<aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>	<gc2f4f$ei8$1@ger.gmane.org>	<48E4C952.7020705@palladion.com>
	<gc2hpj$p2j$1@ger.gmane.org>
Message-ID: <48E4CEDD.50109@gmail.com>

Georg Brandl wrote:
> That's true; it's also not what I meant. The versioned docs will of course
> always stay there. The question is what to do for URLs that refer to
> docs.python.org, but with old filenames.

I still like the idea of redirecting such URLs to the old 2.5.2 docs as
a short-term fix, with a more complex remapping to the appropriate 2.6
files when it is available.

(Whether or not the first part is worth doing obviously depends on how
much time you expect the second part to take).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From skip at pobox.com  Thu Oct  2 15:47:34 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 2 Oct 2008 08:47:34 -0500
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <1222947311.7054.3.camel@fsol>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<1222947311.7054.3.camel@fsol>
Message-ID: <18660.53494.563893.905668@montanaro-dyndns-org.local>

    >> Not a single one, no. The URLs *all* changed. There is not a single
    >> one that's the same. We may be able to do a single rewrite rule for
    >> most of the module-*.html URLs, but everything else -- and there is
    >> quite a lot of 'else' in the 2.5-and-earlier docs -- needs a better
    >> mapping. Feel free to send me that mapping :-)

    Antoine> My bad. I thought it was just a matter of doing a generic
    Antoine> substitution.  Well, then we'll have to live with it I suppose
    Antoine> :)

Unfortunately, without some mapping the search engines will toss everything
out.  While they will eventually get around to fetching
http://docs.python.org/ and traversing the tree of pages, but that might
take awhile.  I won't have time for the next day or two to scan the docs
error log, but if I can come up with a list of the ten most frequent
failures I suspect we can easily define RewriteRule directives for them.

Skip

From thomas at python.org  Thu Oct  2 15:51:08 2008
From: thomas at python.org (Thomas Wouters)
Date: Thu, 2 Oct 2008 15:51:08 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <18660.53494.563893.905668@montanaro-dyndns-org.local>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<1222947311.7054.3.camel@fsol>
	<18660.53494.563893.905668@montanaro-dyndns-org.local>
Message-ID: <9e804ac0810020651na3c0a8cv9c94954f0dea6401@mail.gmail.com>

On Thu, Oct 2, 2008 at 15:47, <skip at pobox.com> wrote:

>    >> Not a single one, no. The URLs *all* changed. There is not a single
>    >> one that's the same. We may be able to do a single rewrite rule for
>    >> most of the module-*.html URLs, but everything else -- and there is
>    >> quite a lot of 'else' in the 2.5-and-earlier docs -- needs a better
>    >> mapping. Feel free to send me that mapping :-)
>
>     Antoine> My bad. I thought it was just a matter of doing a generic
>    Antoine> substitution.  Well, then we'll have to live with it I suppose
>    Antoine> :)
>
> Unfortunately, without some mapping the search engines will toss everything
> out.  While they will eventually get around to fetching
> http://docs.python.org/ and traversing the tree of pages, but that might
> take awhile.  I won't have time for the next day or two to scan the docs
> error log, but if I can come up with a list of the ten most frequent
> failures I suspect we can easily define RewriteRule directives for them.


To be sure, the URLs *are* mapped. They're just mapped to something other
than they were mapped to before -- because those pages no longer exist for
the 'current version' of the documentation. Pages covering the same or
nearly the same thing may exist in some cases, but not in others. We can do
a best-effort to redirect the old URLs to something covering the same
information, or we can wait a few days to let search engines realize the
URLs changed, and let everyone else deal with searching a little further for
the information they had bookmarked.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081002/b14a01bd/attachment-0001.htm>

From skip at pobox.com  Thu Oct  2 15:51:51 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 2 Oct 2008 08:51:51 -0500
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <48E4BBFA.7010800@gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
	<48E4BBFA.7010800@gmail.com>
Message-ID: <18660.53751.895687.800610@montanaro-dyndns-org.local>


    Nick> The old doc directories are already kept around (all the way back
    Nick> to 1.4 in fact: http://www.python.org/doc/1.4/)

    Nick> As a quick fix for the old links, a rewrite rule to map such links
    Nick> to the 2.5 docs seems like a very good idea to me. Since old URLs
    Nick> all use abbreviations in the directory name (tut, lib, mac, ref,
    Nick> ext, api, doc, inst, dist), it should be straightforward to
    Nick> redirect them without affecting the links to the new docs
    Nick> (tutorial, using, reference, howto, extending, c-api, install,
    Nick> distutils, documenting).

Yes, we should probably still get the top-level links redirected to the new
docs though.  The 2.5 tutorial is probably going to get stale over time
while the 2.6 version will be updated at least until 2.7 is released.

Skip

From fdrake at acm.org  Thu Oct  2 15:56:25 2008
From: fdrake at acm.org (Fred Drake)
Date: Thu, 02 Oct 2008 09:56:25 -0400
Subject: [Python-Dev] Doc nits question
In-Reply-To: <gc2hsa$p2j$2@ger.gmane.org>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>
	<gc2hsa$p2j$2@ger.gmane.org>
Message-ID: <18122A2B-CCB0-4850-BDA4-10291780E9D2@acm.org>

On Oct 2, 2008, at 9:21 AM, Georg Brandl wrote:
> I intend to set things up so that the docs at docs.python.org are  
> continually
> rebuilt, just like the /dev docs were until now.

Wonderful!  This should help avoid repeat reports of simple typos.

At one point, we started to separate the documentation releases so  
that update releases could be easily pushed at times when there wasn't  
a corresponding Python release.  There are a couple of examples of  
these in the specific-versions list, IIRC.  These have version numbers  
ending with "p1" (for patch 1; no more than one patched version was  
ever released for any particular Python version).

It may be worth trying this or something like it again as well, if  
there's enough volunteer time available.  Such versions would need to  
be clearly marked on every page as to when they were updated, so that  
readers can tell if they have the latest update.


   -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>


From thomas at python.org  Thu Oct  2 16:41:39 2008
From: thomas at python.org (Thomas Wouters)
Date: Thu, 2 Oct 2008 16:41:39 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <18660.53751.895687.800610@montanaro-dyndns-org.local>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
	<48E4BBFA.7010800@gmail.com>
	<18660.53751.895687.800610@montanaro-dyndns-org.local>
Message-ID: <9e804ac0810020741g7d7cb193h647b512a0efec68b@mail.gmail.com>

On Thu, Oct 2, 2008 at 15:51, <skip at pobox.com> wrote:

>
>    Nick> The old doc directories are already kept around (all the way back
>    Nick> to 1.4 in fact: http://www.python.org/doc/1.4/)
>
>    Nick> As a quick fix for the old links, a rewrite rule to map such links
>    Nick> to the 2.5 docs seems like a very good idea to me. Since old URLs
>    Nick> all use abbreviations in the directory name (tut, lib, mac, ref,
>    Nick> ext, api, doc, inst, dist), it should be straightforward to
>    Nick> redirect them without affecting the links to the new docs
>    Nick> (tutorial, using, reference, howto, extending, c-api, install,
>    Nick> distutils, documenting).
>
> Yes, we should probably still get the top-level links redirected to the new
> docs though.  The 2.5 tutorial is probably going to get stale over time
> while the 2.6 version will be updated at least until 2.7 is released.
>

After discussing on #python-dev (briefly), I made the toplevel directories
refer to the new, 2.6 toplevel directories, but deeper URLs in the old
directories redirect to www.python.org/doc/2.5.2/. I still think this is the
wrong approach, especially in the long term: it means people who just follow
old documentation links will not see the new results, and search engines
will not realize the pages are effectively stale.

-- 
Thomas Wouters <thomas at python.org>

Hi! I'm a .signature virus! copy me into your .signature file to help me
spread!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081002/a240eb91/attachment.htm>

From aahz at pythoncraft.com  Thu Oct  2 16:47:13 2008
From: aahz at pythoncraft.com (Aahz)
Date: Thu, 2 Oct 2008 07:47:13 -0700
Subject: [Python-Dev] python docs website not accessible!
In-Reply-To: <gc1usj$qi3$1@ger.gmane.org>
References: <gc1usj$qi3$1@ger.gmane.org>
Message-ID: <20081002144713.GA5977@panix.com>

On Thu, Oct 02, 2008, itconsense at gmail.com wrote:
> 
>I'm not sure, if this is the right place to post. The python-docs on
>www.python.org are not accessible.

This is definitely the wrong place to post.  As usual for most sites,
webmaster at python.org is the right place.  But please don't bother, we've
already had about twenty other people reporting.  ;-)
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"...if I were on life-support, I'd rather have it run by a Gameboy than a
Windows box."  --Cliff Wells, comp.lang.python, 3/13/2002

From g.brandl at gmx.net  Thu Oct  2 19:17:40 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Oct 2008 19:17:40 +0200
Subject: [Python-Dev] Doc nits question
In-Reply-To: <18122A2B-CCB0-4850-BDA4-10291780E9D2@acm.org>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>	<gc2hsa$p2j$2@ger.gmane.org>
	<18122A2B-CCB0-4850-BDA4-10291780E9D2@acm.org>
Message-ID: <gc2vno$brt$1@ger.gmane.org>

Fred Drake schrieb:
> On Oct 2, 2008, at 9:21 AM, Georg Brandl wrote:
>> I intend to set things up so that the docs at docs.python.org are  
>> continually
>> rebuilt, just like the /dev docs were until now.
> 
> Wonderful!  This should help avoid repeat reports of simple typos.
> 
> At one point, we started to separate the documentation releases so  
> that update releases could be easily pushed at times when there wasn't  
> a corresponding Python release.  There are a couple of examples of  
> these in the specific-versions list, IIRC.  These have version numbers  
> ending with "p1" (for patch 1; no more than one patched version was  
> ever released for any particular Python version).
> 
> It may be worth trying this or something like it again as well, if  
> there's enough volunteer time available.  Such versions would need to  
> be clearly marked on every page as to when they were updated, so that  
> readers can tell if they have the latest update.

All Sphinx-generated pages currently have a "last update on:" in the footer.
Do you think that suffices for this purpose?

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From fdrake at acm.org  Thu Oct  2 19:22:11 2008
From: fdrake at acm.org (Fred Drake)
Date: Thu, 02 Oct 2008 13:22:11 -0400
Subject: [Python-Dev] Doc nits question
In-Reply-To: <gc2vno$brt$1@ger.gmane.org>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>
	<gc2hsa$p2j$2@ger.gmane.org>	<18122A2B-CCB0-4850-BDA4-10291780E9D2@acm.org>
	<gc2vno$brt$1@ger.gmane.org>
Message-ID: <16EFB37E-A1F8-438F-BBB6-9D3A30C50B73@acm.org>

On Oct 2, 2008, at 1:17 PM, Georg Brandl wrote:
> All Sphinx-generated pages currently have a "last update on:" in the  
> footer.
> Do you think that suffices for this purpose?

Yes, I do.


   -Fred

-- 
Fred L. Drake, Jr.  <fdrake at acm.org>


From theller at ctypes.org  Thu Oct  2 19:54:23 2008
From: theller at ctypes.org (Thomas Heller)
Date: Thu, 02 Oct 2008 19:54:23 +0200
Subject: [Python-Dev] Real segmentation fault handler
In-Reply-To: <200809300105.53473.victor.stinner@haypocalc.com>
References: <200809300105.53473.victor.stinner@haypocalc.com>
Message-ID: <gc31sg$k2v$1@ger.gmane.org>

Victor Stinner schrieb:
> Hi,
> 
> I would like to be able to catch SIGSEGV in my Python code! So I started to 
> hack Python trunk to support this feature. The idea is to use a signal 
> handler which call longjmp(), and add setjmp() at Py_EvalFrameEx() enter.

On windows, ctypes catches fatal errors (exception violations) in 
foreign function calls, thanks to windows structured exception handling.

On other platforms, there is the WAD module by David Beazley which
may do something similar:

http://www.dabeaz.com/papers/Python2001/python.html

I do not know whether the code itself is still available or not.

-- 
Thanks,
Thomas


From fredrik at pythonware.com  Thu Oct  2 19:50:30 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 02 Oct 2008 19:50:30 +0200
Subject: [Python-Dev] c99 comments in the 2.6 code base?
Message-ID: <gc31l9$ieu$2@ger.gmane.org>

http://drj11.wordpress.com/2008/10/02/python-and-bragging-about-c89/

mentions that Objects/frameobject.c contains a C99-style comment, which 
means that Python 2.6 won't build on AIX.

shouldn't we use a suitable gcc option for the buildbots to prevent that 
from happening?

</F>


From lists at cheimes.de  Thu Oct  2 20:23:18 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 02 Oct 2008 20:23:18 +0200
Subject: [Python-Dev] c99 comments in the 2.6 code base?
In-Reply-To: <gc31l9$ieu$2@ger.gmane.org>
References: <gc31l9$ieu$2@ger.gmane.org>
Message-ID: <gc33im$pu7$1@ger.gmane.org>

Fredrik Lundh wrote:
> http://drj11.wordpress.com/2008/10/02/python-and-bragging-about-c89/
> 
> mentions that Objects/frameobject.c contains a C99-style comment, which 
> means that Python 2.6 won't build on AIX.
> 
> shouldn't we use a suitable gcc option for the buildbots to prevent that 
> from happening?

Ouch! This shouldn't have happend. I'm going to discuss the matter on 
#python-dev. Perhaps --with-pydebug could add more restrict error 
checking to the Makefile like -std=c89 -pedantic -Werror

Christian


From solipsis at pitrou.net  Thu Oct  2 20:37:15 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 2 Oct 2008 18:37:15 +0000 (UTC)
Subject: [Python-Dev] c99 comments in the 2.6 code base?
References: <gc31l9$ieu$2@ger.gmane.org> <gc33im$pu7$1@ger.gmane.org>
Message-ID: <loom.20081002T183516-0@post.gmane.org>

Christian Heimes <lists <at> cheimes.de> writes:
> 
> Ouch! This shouldn't have happend. I'm going to discuss the matter on 
> #python-dev. Perhaps --with-pydebug could add more restrict error 
> checking to the Makefile like -std=c89 -pedantic -Werror

As discussed on python-dev, I think it should also be added in release mode.
Some developers probably never compile in debug mode (*), and compiling in
release mode is useful when you want to do performance tuning.

(*) not thinking of anyone in particular !

Regards

Antoine.



From g.brandl at gmx.net  Thu Oct  2 21:16:00 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Oct 2008 21:16:00 +0200
Subject: [Python-Dev] Bugfix porting policy (was Re: Doc nits question)
In-Reply-To: <gc2ipo$t6p$1@ger.gmane.org>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>	<gc2hsa$p2j$2@ger.gmane.org>	<4222a8490810020626n3b3c75bo39f4b4552d6773d7@mail.gmail.com>
	<gc2ipo$t6p$1@ger.gmane.org>
Message-ID: <gc36ll$40p$1@ger.gmane.org>

Just now, Christian decided for option 2...

Georg

> This is another thing that needs to be discussed: how to handle backports
> between 2.6 and 2.7. Up to now, we backported changes from trunk to maint
> manually, but after the experience we've had with svnmerge, I see several
> possibilities:
> 
> 1. Do bugfixes in maint, merge them to trunk via svnmerge. This has the
>    drawback that you have to work in two different repos for fixes vs.
>    new features. The advantage however is that normally all fixes that
>    go into maint apply to trunk as well, so almost no blocks need to be done.
>    However, since Py3k merges are done from trunk, the 3k merge will see
>    merges as single commits, so they aren't easy to block if not applicable.
>    This will mean more conflicts.
> 
> 2. Do bugfixes in trunk, and merge them to maint via svnmerge.
>    Arguments as for 1, but reversed: many blocks, but less problems with 3k.
> 
> 3. Backport bugfixes manually, like for the previous maintenance branches.


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From barry at python.org  Thu Oct  2 21:27:10 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 2 Oct 2008 15:27:10 -0400
Subject: [Python-Dev] RELEASED Python 2.6 final
In-Reply-To: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
References: <A790F490-45BE-433E-838E-DBF6D92CD575@python.org>
Message-ID: <E53151FD-F165-489B-AE4C-B3D531686F10@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 1, 2008, at 11:46 PM, Barry Warsaw wrote:

> On behalf of the Python development team and the Python community, I  
> am happy to announce the release of Python 2.6 final.  This is the  
> production-ready version of the latest in the Python 2 series.
>
> Source tarballs, Windows installers, and Mac disk images can be  
> downloaded from the Python 2.6 page:
>
>    http://www.python.org/download/releases/2.6/

Due to a minor snafu in our build scripts, the source tgz and tar.bz2  
files contained some extra cruft.  I have created and uploaded new  
tarballs but I have /not/ bumped the Python version number since they  
were made from exactly the same Subversion tag.  The new tarballs are  
identical to the originals except that they don't contain the cruft  
(.svn files and such).

If you have already downloaded the tarballs, you do not need to  
download the new ones. The new tarballs are about 2MB smaller though.

With apologies,
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCUAwUBSOUgjnEjvBPtnXfVAQJ3gQP4mxxW9kaaMlsg7yd1uNcgYa29pitYdF2+
DhFgrCajPZpskc3XlKbPcnPJWT8wtI/EIC5QcPEpAWCHECrTUHzPyGLNeMQz0kFF
/ZGCGbef7Mc/JaZvEyF6OATnKhYA5XyUOPdddygx6oar/Y6ZbK2JyLR4pvzh+gtQ
SA+u6OPIpQ==
=7uu8
-----END PGP SIGNATURE-----

From lists at cheimes.de  Thu Oct  2 21:47:15 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 02 Oct 2008 21:47:15 +0200
Subject: [Python-Dev] c99 comments in the 2.6 code base?
In-Reply-To: <gc31l9$ieu$2@ger.gmane.org>
References: <gc31l9$ieu$2@ger.gmane.org>
Message-ID: <gc38g3$b79$1@ger.gmane.org>

Fredrik Lundh wrote:
> http://drj11.wordpress.com/2008/10/02/python-and-bragging-about-c89/

I've found several more occasions of // comments and one usage of 
inline. We *really* should have some way to compile Python with C89 checks

Python doesn't compile with the -pedantic option but it compiles with 
-std=c89 -Werror after I've applied some patches. I've added a new make 
command to add extra checks. Maybe the build bots could use "make c89" 
instead of "make" to build Python?

c89:
         $(MAKE) CFLAGS="$(CFLAGS) -std=c89 -Werror"


Christian


From martin at v.loewis.de  Thu Oct  2 22:13:13 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 02 Oct 2008 22:13:13 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>	<gc2e55$bpb$1@ger.gmane.org>
	<aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>
Message-ID: <48E52B59.7060109@v.loewis.de>

> Why not use versioned URLs, but with a  link at the top of old pages
> saying they're outdated, linking to the new version.  Either way they
> should update their links, but this way you don't shoot them in the
> foot to do it.

Wouldn't that require changes to the old pages?

Regards,
Martin

From martin at v.loewis.de  Thu Oct  2 22:18:31 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 02 Oct 2008 22:18:31 +0200
Subject: [Python-Dev] Bugfix porting policy (was Re: Doc nits question)
In-Reply-To: <gc36ll$40p$1@ger.gmane.org>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>	<gc2hsa$p2j$2@ger.gmane.org>	<4222a8490810020626n3b3c75bo39f4b4552d6773d7@mail.gmail.com>	<gc2ipo$t6p$1@ger.gmane.org>
	<gc36ll$40p$1@ger.gmane.org>
Message-ID: <48E52C97.5030909@v.loewis.de>

>> 2. Do bugfixes in trunk, and merge them to maint via svnmerge.
>>    Arguments as for 1, but reversed: many blocks, but less problems with 3k.

I'm not so sure that we need to block all the changes that we don't
want, though: it would be sufficient to just not merge them, right?

(of course, somebody could go over it from time to time and block
everything older than a month that was still available, just to prevent
accidental merging)

Regards,
Martin

From martin at v.loewis.de  Thu Oct  2 22:19:17 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 02 Oct 2008 22:19:17 +0200
Subject: [Python-Dev] c99 comments in the 2.6 code base?
In-Reply-To: <gc31l9$ieu$2@ger.gmane.org>
References: <gc31l9$ieu$2@ger.gmane.org>
Message-ID: <48E52CC5.8080107@v.loewis.de>

> shouldn't we use a suitable gcc option for the buildbots to prevent that
> from happening?

Which one specifically?

Regards,
Martin

From musiccomposition at gmail.com  Thu Oct  2 22:26:20 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Thu, 2 Oct 2008 15:26:20 -0500
Subject: [Python-Dev] Bugfix porting policy (was Re: Doc nits question)
In-Reply-To: <48E52C97.5030909@v.loewis.de>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>
	<gc2hsa$p2j$2@ger.gmane.org>
	<4222a8490810020626n3b3c75bo39f4b4552d6773d7@mail.gmail.com>
	<gc2ipo$t6p$1@ger.gmane.org> <gc36ll$40p$1@ger.gmane.org>
	<48E52C97.5030909@v.loewis.de>
Message-ID: <1afaf6160810021326m5b54ad0dh2f890231be451b88@mail.gmail.com>

On Thu, Oct 2, 2008 at 3:18 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>> 2. Do bugfixes in trunk, and merge them to maint via svnmerge.
>>>    Arguments as for 1, but reversed: many blocks, but less problems with 3k.
>
> I'm not so sure that we need to block all the changes that we don't
> want, though: it would be sufficient to just not merge them, right?

A large merge queue would accumulate making hard for someone to pick
out the bugfixes. Of course, people could just merge fixes right after
they apply it to the trunk, though.

>
> (of course, somebody could go over it from time to time and block
> everything older than a month that was still available, just to prevent
> accidental merging)
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From martin at v.loewis.de  Thu Oct  2 22:31:38 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Thu, 02 Oct 2008 22:31:38 +0200
Subject: [Python-Dev] Bugfix porting policy (was Re: Doc nits question)
In-Reply-To: <1afaf6160810021326m5b54ad0dh2f890231be451b88@mail.gmail.com>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>	
	<gc2hsa$p2j$2@ger.gmane.org>	
	<4222a8490810020626n3b3c75bo39f4b4552d6773d7@mail.gmail.com>	
	<gc2ipo$t6p$1@ger.gmane.org> <gc36ll$40p$1@ger.gmane.org>	
	<48E52C97.5030909@v.loewis.de>
	<1afaf6160810021326m5b54ad0dh2f890231be451b88@mail.gmail.com>
Message-ID: <48E52FAA.6000104@v.loewis.de>

> A large merge queue would accumulate making hard for someone to pick
> out the bugfixes. Of course, people could just merge fixes right after
> they apply it to the trunk, though.

I think they should. To my knowledge, nobody goes through the changelog
anymore trying to find out what needs backporting. Tracking what still
needs merging should happen in the bug tracker, by leaving the report
open until merging has been done. Every change that isn't immediately
merged and doesn't have an open issue just won't get merged at all.

Regards,
Martin

From rhamph at gmail.com  Thu Oct  2 22:31:59 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Thu, 2 Oct 2008 14:31:59 -0600
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <48E52B59.7060109@v.loewis.de>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
	<gc2e55$bpb$1@ger.gmane.org>
	<aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>
	<48E52B59.7060109@v.loewis.de>
Message-ID: <aac2c7cb0810021331h20c8ce49kc4bede552acfe7b@mail.gmail.com>

On Thu, Oct 2, 2008 at 2:13 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> Why not use versioned URLs, but with a  link at the top of old pages
>> saying they're outdated, linking to the new version.  Either way they
>> should update their links, but this way you don't shoot them in the
>> foot to do it.
>
> Wouldn't that require changes to the old pages?

Hopefully just to whatever common templating they're using.  I'm not
familiar with how they're generated though.


-- 
Adam Olsen, aka Rhamphoryncus

From g.brandl at gmx.net  Thu Oct  2 22:33:49 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Oct 2008 22:33:49 +0200
Subject: [Python-Dev] Bugfix porting policy (was Re: Doc nits question)
In-Reply-To: <48E52FAA.6000104@v.loewis.de>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>		<gc2hsa$p2j$2@ger.gmane.org>		<4222a8490810020626n3b3c75bo39f4b4552d6773d7@mail.gmail.com>		<gc2ipo$t6p$1@ger.gmane.org>
	<gc36ll$40p$1@ger.gmane.org>		<48E52C97.5030909@v.loewis.de>	<1afaf6160810021326m5b54ad0dh2f890231be451b88@mail.gmail.com>
	<48E52FAA.6000104@v.loewis.de>
Message-ID: <gc3b7i$k01$1@ger.gmane.org>

Martin v. L?wis schrieb:
>> A large merge queue would accumulate making hard for someone to pick
>> out the bugfixes. Of course, people could just merge fixes right after
>> they apply it to the trunk, though.
> 
> I think they should. To my knowledge, nobody goes through the changelog
> anymore trying to find out what needs backporting. Tracking what still
> needs merging should happen in the bug tracker, by leaving the report
> open until merging has been done. Every change that isn't immediately
> merged and doesn't have an open issue just won't get merged at all.

This is why it's good to track what was merged and what not via svnmerge,
because it cannot miss commits. It also is easy for someone to select
which stuff to merge if the commit message on the trunk indicates
backportable fixes.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From martin at v.loewis.de  Thu Oct  2 22:36:35 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 02 Oct 2008 22:36:35 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <aac2c7cb0810021331h20c8ce49kc4bede552acfe7b@mail.gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	
	<loom.20081002T104308-982@post.gmane.org>	
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>	
	<gc2e55$bpb$1@ger.gmane.org>	
	<aac2c7cb0810020525o5722b2b5u5a10c98e8b38650b@mail.gmail.com>	
	<48E52B59.7060109@v.loewis.de>
	<aac2c7cb0810021331h20c8ce49kc4bede552acfe7b@mail.gmail.com>
Message-ID: <48E530D3.1070907@v.loewis.de>

>> Wouldn't that require changes to the old pages?
> 
> Hopefully just to whatever common templating they're using.  I'm not
> familiar with how they're generated though.

That's exactly the problem: they are generated. I don't think it's
feasible to regenerate them, and still expect the output to be the
same. Also, I don't think the generator supports templating in the
way you might expect it to. To be specific, it's latex2html.

Regards,
Martin


From g.brandl at gmx.net  Thu Oct  2 22:38:42 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 02 Oct 2008 22:38:42 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <9e804ac0810020741g7d7cb193h647b512a0efec68b@mail.gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	<loom.20081002T104308-982@post.gmane.org>	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>	<48E4BBFA.7010800@gmail.com>	<18660.53751.895687.800610@montanaro-dyndns-org.local>
	<9e804ac0810020741g7d7cb193h647b512a0efec68b@mail.gmail.com>
Message-ID: <gc3bgn$l4g$1@ger.gmane.org>

Thomas Wouters schrieb:

> After discussing on #python-dev (briefly), I made the toplevel
> directories refer to the new, 2.6 toplevel directories, but deeper URLs
> in the old directories redirect to www.python.org/doc/2.5.2/
> <http://www.python.org/doc/2.5.2/>. I still think this is the wrong
> approach, especially in the long term: it means people who just follow
> old documentation links will not see the new results, and search engines
> will not realize the pages are effectively stale.

I'll work on a more thorough redirection in about two weeks' time.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ncoghlan at gmail.com  Thu Oct  2 23:49:23 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 03 Oct 2008 07:49:23 +1000
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <9e804ac0810020741g7d7cb193h647b512a0efec68b@mail.gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>	
	<loom.20081002T104308-982@post.gmane.org>	
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>	
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>	
	<48E4BBFA.7010800@gmail.com>	
	<18660.53751.895687.800610@montanaro-dyndns-org.local>
	<9e804ac0810020741g7d7cb193h647b512a0efec68b@mail.gmail.com>
Message-ID: <48E541E3.4080709@gmail.com>

Thomas Wouters wrote:
> 
> 
> On Thu, Oct 2, 2008 at 15:51, <skip at pobox.com <mailto:skip at pobox.com>>
> wrote:
> 
> 
>        Nick> The old doc directories are already kept around (all the
>     way back
>        Nick> to 1.4 in fact: http://www.python.org/doc/1.4/)
> 
>        Nick> As a quick fix for the old links, a rewrite rule to map
>     such links
>        Nick> to the 2.5 docs seems like a very good idea to me. Since
>     old URLs
>        Nick> all use abbreviations in the directory name (tut, lib, mac,
>     ref,
>        Nick> ext, api, doc, inst, dist), it should be straightforward to
>        Nick> redirect them without affecting the links to the new docs
>        Nick> (tutorial, using, reference, howto, extending, c-api, install,
>        Nick> distutils, documenting).
> 
>     Yes, we should probably still get the top-level links redirected to
>     the new
>     docs though.  The 2.5 tutorial is probably going to get stale over time
>     while the 2.6 version will be updated at least until 2.7 is released.
> 
> 
> After discussing on #python-dev (briefly), I made the toplevel
> directories refer to the new, 2.6 toplevel directories, but deeper URLs
> in the old directories redirect to www.python.org/doc/2.5.2/
> <http://www.python.org/doc/2.5.2/>. I still think this is the wrong
> approach, especially in the long term: it means people who just follow
> old documentation links will not see the new results, and search engines
> will not realize the pages are effectively stale.

Agreed, but I think it's a better near-term solution than dumping deep
links back at the top of the relevant document (it always annoys me when
web sites do that).

Long term, remapping even the deep links to the appropriate part of the
new docs should hopefully be possible.

For the search engine issue, is there any way we can tell robots to
ignore the rewrite rules so they see the broken links? (although even
that may not be ideal, since what we really want is to tell the robot
the link is broken, and provide the new alternative)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From musiccomposition at gmail.com  Fri Oct  3 00:13:16 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Thu, 2 Oct 2008 17:13:16 -0500
Subject: [Python-Dev] 2to3 bug fixes
Message-ID: <1afaf6160810021513u78a76355m22637eb20b6e0205@mail.gmail.com>

What should the policy on 2to3 bug fixes be for the maintenance
branch? I'm asking because I remember vaguely someone suggesting that
new 2to3 fixers could fit into that category.

So, should I only merge "pure" bug fixes, or do I get to stretch the
definition a little?

-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From guido at python.org  Fri Oct  3 00:16:45 2008
From: guido at python.org (Guido van Rossum)
Date: Thu, 2 Oct 2008 15:16:45 -0700
Subject: [Python-Dev] 2to3 bug fixes
In-Reply-To: <1afaf6160810021513u78a76355m22637eb20b6e0205@mail.gmail.com>
References: <1afaf6160810021513u78a76355m22637eb20b6e0205@mail.gmail.com>
Message-ID: <ca471dc20810021516l4fa8de62w87d58e597d9873e1@mail.gmail.com>

On Thu, Oct 2, 2008 at 3:13 PM, Benjamin Peterson
<musiccomposition at gmail.com> wrote:
> What should the policy on 2to3 bug fixes be for the maintenance
> branch? I'm asking because I remember vaguely someone suggesting that
> new 2to3 fixers could fit into that category.
>
> So, should I only merge "pure" bug fixes, or do I get to stretch the
> definition a little?

We agreed that new fixers (and possibly other changes) to 2to3 are
fair game for bugfix releases.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From victor.stinner at haypocalc.com  Fri Oct  3 00:36:29 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Fri, 3 Oct 2008 00:36:29 +0200
Subject: [Python-Dev] 2to3 bug fixes
In-Reply-To: <1afaf6160810021513u78a76355m22637eb20b6e0205@mail.gmail.com>
References: <1afaf6160810021513u78a76355m22637eb20b6e0205@mail.gmail.com>
Message-ID: <200810030036.29727.victor.stinner@haypocalc.com>

Le Friday 03 October 2008 00:13:16 Benjamin Peterson, vous avez ?crit?:
> What should the policy on 2to3 bug fixes be for the maintenance
> branch? I'm asking because I remember vaguely someone suggesting that
> new 2to3 fixers could fit into that category.

Python3 removes os.getcwdu() and introduces os.getcwdb(). A fixer should 
replace "os.getcwdu()" to "os.getcwd()". See for example attached fixer 
(which also replaced "getcwdu()" to "getcwd()").

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: fix_getcwdu.py
Type: application/x-python
Size: 572 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081003/2435b3c2/attachment.bin>

From musiccomposition at gmail.com  Fri Oct  3 01:06:33 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Thu, 2 Oct 2008 18:06:33 -0500
Subject: [Python-Dev] 2to3 bug fixes
In-Reply-To: <200810030036.29727.victor.stinner@haypocalc.com>
References: <1afaf6160810021513u78a76355m22637eb20b6e0205@mail.gmail.com>
	<200810030036.29727.victor.stinner@haypocalc.com>
Message-ID: <1afaf6160810021606w2e5a9810gec94097ab5205ff6@mail.gmail.com>

On Thu, Oct 2, 2008 at 5:36 PM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Le Friday 03 October 2008 00:13:16 Benjamin Peterson, vous avez ?crit :
>> What should the policy on 2to3 bug fixes be for the maintenance
>> branch? I'm asking because I remember vaguely someone suggesting that
>> new 2to3 fixers could fit into that category.
>
> Python3 removes os.getcwdu() and introduces os.getcwdb(). A fixer should
> replace "os.getcwdu()" to "os.getcwd()". See for example attached fixer
> (which also replaced "getcwdu()" to "getcwd()").

Once again, please post it to the tracker and assign it to me.



-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From lists at cheimes.de  Fri Oct  3 01:30:59 2008
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 03 Oct 2008 01:30:59 +0200
Subject: [Python-Dev] c99 comments in the 2.6 code base?
In-Reply-To: <48E52CC5.8080107@v.loewis.de>
References: <gc31l9$ieu$2@ger.gmane.org> <48E52CC5.8080107@v.loewis.de>
Message-ID: <48E559B3.6020408@cheimes.de>

Martin v. L?wis wrote:
>> shouldn't we use a suitable gcc option for the buildbots to prevent that
>> from happening?
> 
> Which one specifically?

I suggest we add "-std=c89" to CFLAGS. We could also add a new target 
called buildbot to the Makefile that appends "-std=c89 -Werror" to 
CFLAGS. I don't think it's wise to add "-Werror" to the standard build 
target. However a new build target with extra checks should help to 
detect errors much sooner.

Christian

From martin at v.loewis.de  Fri Oct  3 02:11:11 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 03 Oct 2008 02:11:11 +0200
Subject: [Python-Dev] c99 comments in the 2.6 code base?
In-Reply-To: <48E559B3.6020408@cheimes.de>
References: <gc31l9$ieu$2@ger.gmane.org> <48E52CC5.8080107@v.loewis.de>
	<48E559B3.6020408@cheimes.de>
Message-ID: <48E5631F.3000104@v.loewis.de>

>>> shouldn't we use a suitable gcc option for the buildbots to prevent that
>>> from happening?
>>
>> Which one specifically?
> 
> I suggest we add "-std=c89" to CFLAGS.

That needs thorough testing, in particular across many old Linux
distributions. It might be that some sets of Linux header files
rely on GNU C extensions, without using the __extension__ keyword.

> We could also add a new target
> called buildbot to the Makefile that appends "-std=c89 -Werror" to
> CFLAGS. I don't think it's wise to add "-Werror" to the standard build
> target. However a new build target with extra checks should help to
> detect errors much sooner.

That would need to go along with a policy that Python must never cause
GCC to emit any warnings.

Regards,
Martin

From skip at pobox.com  Fri Oct  3 02:40:20 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 2 Oct 2008 19:40:20 -0500
Subject: [Python-Dev] c99 comments in the 2.6 code base?
In-Reply-To: <48E5631F.3000104@v.loewis.de>
References: <gc31l9$ieu$2@ger.gmane.org> <48E52CC5.8080107@v.loewis.de>
	<48E559B3.6020408@cheimes.de> <48E5631F.3000104@v.loewis.de>
Message-ID: <18661.27124.75089.716559@montanaro-dyndns-org.local>


    >>>> shouldn't we use a suitable gcc option for the buildbots to prevent
    >>>> that from happening?
    >>> 
    >>> Which one specifically?
    >> 
    >> I suggest we add "-std=c89" to CFLAGS.

    Martin> That needs thorough testing, in particular across many old Linux
    Martin> distributions. It might be that some sets of Linux header files
    Martin> rely on GNU C extensions, without using the __extension__
    Martin> keyword.

Surely we don't need to be that careful with the buildbots do we?  If
anything, I think it would be a good idea to be more strict for them than
the default.

Skip

From tjreedy at udel.edu  Fri Oct  3 02:54:48 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 02 Oct 2008 20:54:48 -0400
Subject: [Python-Dev] Doc nits question
In-Reply-To: <gc2vno$brt$1@ger.gmane.org>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>	<gc2hsa$p2j$2@ger.gmane.org>	<18122A2B-CCB0-4850-BDA4-10291780E9D2@acm.org>
	<gc2vno$brt$1@ger.gmane.org>
Message-ID: <gc3qgp$tnh$1@ger.gmane.org>

Georg Brandl wrote:
> Fred Drake schrieb:
>> On Oct 2, 2008, at 9:21 AM, Georg Brandl wrote:
>>> I intend to set things up so that the docs at docs.python.org are  
>>> continually
>>> rebuilt, just like the /dev docs were until now.

Will you do the same for the 3.0 version?
http://docs.python.org/dev/3.0/

The following page has no reference to the 3.0 version
http://www.python.org/doc/versions/
The 'unreleased' link at the top goes to a link that only referenced the 
SVN version and not the built version above.  Adding a link to the build 
version would have made it easier to find ;-).

> All Sphinx-generated pages currently have a "last update on:" in the footer.
> Do you think that suffices for this purpose?

It certainly would limit a search for closed issues not incorporated in 
the update (to avoiding duplication).


From nnorwitz at gmail.com  Fri Oct  3 06:04:32 2008
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Thu, 2 Oct 2008 21:04:32 -0700
Subject: [Python-Dev] Doc nits question
In-Reply-To: <gc2hsa$p2j$2@ger.gmane.org>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>
	<gc2hsa$p2j$2@ger.gmane.org>
Message-ID: <ee2a432c0810022104v72ce8622pfda23d9148c384c7@mail.gmail.com>

On Thu, Oct 2, 2008 at 6:21 AM, Georg Brandl <g.brandl at gmx.net> wrote:
> Jesse Noller schrieb:
>> So, we just released and there are a few doc typo bugs being filed -
>> my question is if all doc-fixes have to wait for 2.6.1/2.7 or if we
>> can hotfix the 2.6 docs?
>
> I intend to set things up so that the docs at docs.python.org are continually
> rebuilt, just like the /dev docs were until now.

The 2.6 docs are now updated similar to how 2.5 was (hourly).  2.5
docs are no longer updated. In case you can't guess the url, it's:

http://docs.python.org/dev/2.6/

3.0 should continue to work.  Let me know if you have any problems.

n

>
> Georg
>
> --
> Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
> Four shall be the number of spaces thou shalt indent, and the number of thy
> indenting shall be four. Eight shalt thou not indent, nor either indent thou
> two, excepting that thou then proceed to four. Tabs are right out.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/nnorwitz%40gmail.com
>

From adde at trialcode.com  Fri Oct  3 12:10:59 2008
From: adde at trialcode.com (Andreas Nilsson)
Date: Fri, 3 Oct 2008 12:10:59 +0200
Subject: [Python-Dev] if-syntax for regular for-loops
Message-ID: <652F78A2-2DD3-474E-9599-C7B6A3BA1097@trialcode.com>

Hi.
First post so here it goes.
My name is Adde, and I'm a Swedish software developer. I've been  
programming for about 23 years now since starting with Basic on the  
C64. I've been through most well known and a couple of lesser known  
languages in search of the perfect one. At the moment I'm writing a  
custom ctypes interface to the Firebird database (need access to  
advanced features, portability to Windows and I definitely don't enjoy  
setting up GCC on Windows).
I've programmed a lot of C/C++ in my days so I thought I'd at least  
join the list and see if anything piques my interest enough to dive in.

With that out of the way, on to todays subject:
I use list comprehensions and generator expressions a lot and lately  
I've found myself writing a lot of code like this:

for i in items if i.some_field == some_value: i.do_something()

Naturally it won't work but it seems like a pretty straight-forward  
extension to allow compressing simple loops to fit on one line. The  
alternative, in my eyes, suggests there's something more happening  
than a simple include-test which makes it harder to comprehend.

for i in items:
	if i.some_field == some_value: i.do_something()

One possibility of course is to use a generator-expression but that  
makes it look like there are two for loops and it feels like a waste  
setting up a generator just for filtering.

for i in (i for i in items if some_field == some_value):
	i.do_something()

Stupid idea? Am I missing some obviously better way of achieving the  
same result?

Thanks,
Adde

From 00ai99 at gmail.com  Fri Oct  3 12:35:12 2008
From: 00ai99 at gmail.com (David Gowers)
Date: Fri, 3 Oct 2008 20:05:12 +0930
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <652F78A2-2DD3-474E-9599-C7B6A3BA1097@trialcode.com>
References: <652F78A2-2DD3-474E-9599-C7B6A3BA1097@trialcode.com>
Message-ID: <23f4e3390810030335m69f53958v2d3bcf0617a8b65e@mail.gmail.com>

Hi Andreas,

On Fri, Oct 3, 2008 at 7:40 PM, Andreas Nilsson <adde at trialcode.com> wrote:
> Hi.
> First post so here it goes.
> My name is Adde, and I'm a Swedish software developer. I've been programming
> for about 23 years now since starting with Basic on the C64. I've been
> through most well known and a couple of lesser known languages in search of
> the perfect one. At the moment I'm writing a custom ctypes interface to the
> Firebird database (need access to advanced features, portability to Windows
> and I definitely don't enjoy setting up GCC on Windows).
> I've programmed a lot of C/C++ in my days so I thought I'd at least join the
> list and see if anything piques my interest enough to dive in.
>
> With that out of the way, on to todays subject:
> I use list comprehensions and generator expressions a lot and lately I've
> found myself writing a lot of code like this:
>
> for i in items if i.some_field == some_value: i.do_something()
>
> Naturally it won't work but it seems like a pretty straight-forward
> extension to allow compressing simple loops to fit on one line. The
> alternative, in my eyes, suggests there's something more happening than a
> simple include-test which makes it harder to comprehend.
>
> for i in items:
>        if i.some_field == some_value: i.do_something()
>
> One possibility of course is to use a generator-expression but that makes it
> look like there are two for loops and it feels like a waste setting up a
> generator just for filtering.
>
> for i in (i for i in items if some_field == some_value):
>        i.do_something()
>
> Stupid idea? Am I missing some obviously better way of achieving the same
> result?

List comprehension.


[i.do_something() for i in items if i.some_field == some_value]


With the restriction that the statement you use must seem to return an
expression..
For example


[print(i) for i in range(9) if i % 2]


Fails with SyntaxError, whereas


def f(v):
    print (v)
[f(i) for i in range(9) if i % 2]


correctly prints

1
3
5
7

HTH,
David

-- 
Everything has reasons. Nothing has justification.
?io havas kialojn; Nenia?o havas pravigeron.

From leif.walsh at gmail.com  Fri Oct  3 16:29:33 2008
From: leif.walsh at gmail.com (Leif Walsh)
Date: Fri, 3 Oct 2008 10:29:33 -0400
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <652F78A2-2DD3-474E-9599-C7B6A3BA1097@trialcode.com>
References: <652F78A2-2DD3-474E-9599-C7B6A3BA1097@trialcode.com>
Message-ID: <cc7430500810030729h76e0baa5m8481e5dc26e3a93a@mail.gmail.com>

On Fri, Oct 3, 2008 at 6:10 AM, Andreas Nilsson <adde at trialcode.com> wrote:
> With that out of the way, on to todays subject:
> I use list comprehensions and generator expressions a lot and lately I've
> found myself writing a lot of code like this:
>
> for i in items if i.some_field == some_value: i.do_something()
>
> Naturally it won't work but it seems like a pretty straight-forward
> extension to allow compressing simple loops to fit on one line. The
> alternative, in my eyes, suggests there's something more happening than a
> simple include-test which makes it harder to comprehend.
>
> for i in items:
>        if i.some_field == some_value: i.do_something()
>
> One possibility of course is to use a generator-expression but that makes it
> look like there are two for loops and it feels like a waste setting up a
> generator just for filtering.
>
> for i in (i for i in items if some_field == some_value):
>        i.do_something()
>
> Stupid idea? Am I missing some obviously better way of achieving the same
> result?

It's been discussed already.  Essentially, all that saves is a newline
or two, which, as I think has been generally accepted, tends to hurt
readability.

Here's the full discussion:
http://www.mail-archive.com/python-dev at python.org/msg29276.html

Other than that, welcome!

-- 
Cheers,
Leif

From status at bugs.python.org  Fri Oct  3 18:06:35 2008
From: status at bugs.python.org (Python tracker)
Date: Fri,  3 Oct 2008 18:06:35 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20081003160635.D8885785E0@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (09/26/08 - 10/03/08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2074 open (+39) / 13779 closed (+16) / 15853 total (+55)

Open issues with patches:   678

Average duration of open issues: 712 days.
Median duration of open issues: 1835 days.

Open Issues Breakdown
   open  2058 (+38)
pending    16 ( +1)

Issues Created Or Reopened (56)
_______________________________

pprint._safe_repr is not general enough in one instance          09/26/08
       http://bugs.python.org/issue3976    created  erickt                    
                                                                               

Check PyInt_AsSsize_t/PyLong_AsSsize_t error                     09/26/08
CLOSED http://bugs.python.org/issue3977    created  haypo                     
       patch                                                                   

ZipFileExt.read() can be incredibly slow                         09/26/08
       http://bugs.python.org/issue3978    created  lightstruk                
       patch                                                                   

Doctest failing when it should pass                              09/26/08
CLOSED http://bugs.python.org/issue3979    created  pupeno                    
                                                                               

win32file.GetCommState incorrect handling of DCB                 09/26/08
CLOSED http://bugs.python.org/issue3980    created  jiaailun                  
                                                                               

Python 3, IDLE does not start                                    09/27/08
CLOSED http://bugs.python.org/issue3981    created  dah                       
                                                                               

support .format for bytes                                        09/27/08
       http://bugs.python.org/issue3982    created  benjamin.peterson         
                                                                               

Typos in Documentation                                           09/28/08
CLOSED http://bugs.python.org/issue3983    created  Bk                        
                                                                               

python interpreter import dependency with disutils/util          09/28/08
       http://bugs.python.org/issue3984    created  tarek                     
       patch                                                                   

removed string module from distutils [patch]                     09/28/08
       http://bugs.python.org/issue3985    created  tarek                     
       patch                                                                   

removed string and type usage from distutils.cmd [patch]         09/28/08
       http://bugs.python.org/issue3986    created  tarek                     
       patch                                                                   

removed types from distutils.core [patch]                        09/28/08
       http://bugs.python.org/issue3987    created  tarek                     
       patch                                                                   

Byte warning mode and b'' != ''                                  09/28/08
       http://bugs.python.org/issue3988    created  christian.heimes          
       patch                                                                   

Tools\Scripts\2to3.py broken under 3.0 rc1 Windows               09/28/08
CLOSED http://bugs.python.org/issue3989    created  arnaud.faucher            
                                                                               

The Linux2 platform definition is incorrect for alpha, hppa, mip 09/28/08
       http://bugs.python.org/issue3990    created  ths                       
       patch                                                                   

urllib.request.urlopen does not handle non-ASCII characters      09/28/08
       http://bugs.python.org/issue3991    created  a.badger                  
                                                                               

removed custom log from distutils                                09/28/08
       http://bugs.python.org/issue3992    created  tarek                     
       patch                                                                   

Convert documentation to python 3.                               09/29/08
CLOSED http://bugs.python.org/issue3993    created  LambertDW                 
                                                                               

import fixer misses some symbols                                 09/29/08
       http://bugs.python.org/issue3994    created  mhammond                  
                                                                               

iso-xxx/cp1252 inconsistencies in Python 2.* not in 3.*          09/29/08
CLOSED http://bugs.python.org/issue3995    created  jmfauth                   
                                                                               

PyOS_CheckStack does not work                                    09/29/08
       http://bugs.python.org/issue3996    created  amaury.forgeotdarc        
       patch                                                                   

zipfile and winzip                                               09/29/08
       http://bugs.python.org/issue3997    created  vgeorge                   
                                                                               

List.sort docstring has obsolete cmp reference.                  09/29/08
CLOSED http://bugs.python.org/issue3998    created  tjreedy                   
       easy                                                                    

Real segmentation fault handler                                  09/30/08
       http://bugs.python.org/issue3999    created  haypo                     
       patch                                                                   

Additional 2to3 documentation updates                            09/30/08
       http://bugs.python.org/issue4000    created  LambertDW                 
                                                                               

2to3 does relative import for modules not in a package.          09/30/08
CLOSED http://bugs.python.org/issue4001    created  mhammond                  
       patch                                                                   

A Bug in the Documentation                                       09/30/08
CLOSED http://bugs.python.org/issue4002    created  fretai                    
                                                                               

Reference leak in test_cprofile                                  09/30/08
CLOSED http://bugs.python.org/issue4003    created  amaury.forgeotdarc        
       patch, needs review                                                     

missing newline in "Could not convert argument %s to string" err 09/30/08
       http://bugs.python.org/issue4004    created  haypo                     
       patch                                                                   

pydoc in web server mode tails at initial request                10/01/08
       http://bugs.python.org/issue4005    created  prologic                  
       patch, needs review                                                     

os.getenv silently discards env variables with non-UTF-8 values  10/01/08
       http://bugs.python.org/issue4006    created  a.badger                  
                                                                               

make clean fails to delete .a and .so.X.Y files                  10/01/08
       http://bugs.python.org/issue4007    created  skip.montanaro            
       patch, easy                                                             

IDLE: checksyntax() doesn't support Unicode?                     10/01/08
       http://bugs.python.org/issue4008    created  haypo                     
       patch                                                                   

A Minor Glitch in the Documentation                              10/01/08
CLOSED http://bugs.python.org/issue4009    created  fretai                    
                                                                               

configure options don't trickle down to distutils                10/01/08
       http://bugs.python.org/issue4010    created  skip.montanaro            
       patch, easy                                                             

Create DAG for PEP 101                                           10/01/08
       http://bugs.python.org/issue4011    created  brett.cannon              
                                                                               

Minor errors in multiprocessing docs                             10/02/08
       http://bugs.python.org/issue4012    created  kjohnson                  
                                                                               

Python 2.6 Doc/tools folder bigger than in 2.6rc2                10/02/08
       http://bugs.python.org/issue4013    created  koen                      
                                                                               

Python-2.6-py2.6.egg-info contains Alpha reference               10/02/08
       http://bugs.python.org/issue4014    created  koen                      
                                                                               

[patch] make installed scripts executable on windows             10/02/08
       http://bugs.python.org/issue4015    created  techtonik                 
       patch                                                                   

improve linecache: reuse tokenize.detect_encoding() and io.open( 10/02/08
       http://bugs.python.org/issue4016    created  haypo                     
       patch                                                                   

IDLE 2.6 broken on OSX (Leopard)                                 10/02/08
       http://bugs.python.org/issue4017    created  fzero                     
                                                                               

"for me" installer problem on x64 Vista                          10/02/08
       http://bugs.python.org/issue4018    created  jpe                       
                                                                               

2.6 (final) uses old icons in Start Menu                         10/02/08
CLOSED http://bugs.python.org/issue4019    created  craigneuro                
                                                                               

hasattr boundary case failure                                    10/02/08
CLOSED http://bugs.python.org/issue4020    created  Xore                      
                                                                               

tokenize.detect_encoding(): raise SyntaxError on codecs.lookup() 10/02/08
       http://bugs.python.org/issue4021    created  haypo                     
       patch, patch, needs review                                              

2.6 dependent on c:\python26\ on windows                         10/02/08
       http://bugs.python.org/issue4022    created  armandorowe               
                                                                               

convert os.getcwdu() to os.getcwd(), and getcwdu() to getcwd()   10/02/08
       http://bugs.python.org/issue4023    created  haypo                     
       patch                                                                   

float(0.0) singleton                                             10/03/08
       http://bugs.python.org/issue4024    created  ldeller                   
       patch                                                                   

C99 comments in Python 2.6 break build on AIX 6.1                10/03/08
       http://bugs.python.org/issue4025    created  drj                       
                                                                               

fcntl extension fails to build on AIX 6.1                        10/03/08
       http://bugs.python.org/issue4026    created  drj                       
                                                                               

wrong page index number in reference book of python documentatio 10/03/08
       http://bugs.python.org/issue4027    created  ray                       
                                                                               

Problem compiling the multiprocessing module on sunos5           10/03/08
       http://bugs.python.org/issue4028    created  jr244                     
                                                                               

Documentation displays incorrectly in iexplore.                  10/03/08
       http://bugs.python.org/issue4029    created  LambertDW                 
                                                                               

msi installer does not requires UAC permission on Vista          10/03/08
CLOSED http://bugs.python.org/issue4030    created  willwill                  
                                                                               

Fix for bugs relating to ntpath.expanduser()                     10/02/08
       http://bugs.python.org/issue957650  reopened christian.heimes          
       patch                                                                   



Issues Now Closed (38)
______________________

Line numbers reported by extract_stack are offset by the #-*- en  138 days
       http://bugs.python.org/issue2832    benjamin.peterson         
                                                                               

Release notes for 2.6b2 call it an alpha release                   61 days
       http://bugs.python.org/issue3457    akuchling                 
                                                                               

Site-specific configuration hook documentation incorrect           52 days
       http://bugs.python.org/issue3510    akuchling                 
                                                                               

Ctypes is confused by bitfields of varying integer types           48 days
       http://bugs.python.org/issue3547    theller                   
       patch, needs review                                                     

super is a built-in type                                           33 days
       http://bugs.python.org/issue3736    rhettinger                
                                                                               

"make check" suggest a testing target under GNU coding standards   31 days
       http://bugs.python.org/issue3758    ralph.corderoy            
       patch                                                                   

Integer overflow in _hashopenssl.c (CVE-2008-2316)                  9 days
       http://bugs.python.org/issue3886    benjamin.peterson         
       patch, patch, 64bit                                                     

imageop issue                                                      12 days
       http://bugs.python.org/issue3894    benjamin.peterson         
                                                                               

2.6 regression in socket.ssl method                                10 days
       http://bugs.python.org/issue3910    janssen                   
       patch                                                                   

ftplib.FTP.makeport() bug                                           8 days
       http://bugs.python.org/issue3911    benjamin.peterson         
       patch                                                                   

Patch to implement a real ftplib test suite                         6 days
       http://bugs.python.org/issue3939    benjamin.peterson         
       patch                                                                   

Help in IDLE should open the chm file                               6 days
       http://bugs.python.org/issue3941    georg.brandl              
                                                                               

PyObject_CheckReadBuffer crashes on memoryview object               3 days
       http://bugs.python.org/issue3946    benjamin.peterson         
       patch, needs review                                                     

Disable Py_USING_MEMORY_DEBUGGER!                                   7 days
       http://bugs.python.org/issue3951    haypo                     
       patch                                                                   

turtle.py - bug in Screen.__init__()                                5 days
       http://bugs.python.org/issue3956    loewis                    
       patch                                                                   

Section permalink html anchors are wrong                            3 days
       http://bugs.python.org/issue3960    georg.brandl              
                                                                               

bytearray().count()                                                 1 days
       http://bugs.python.org/issue3967    amaury.forgeotdarc        
       patch, needs review                                                     

s_push: parser stack overflow MemoryError                           1 days
       http://bugs.python.org/issue3971    benjamin.peterson         
                                                                               

collections.namedtuple uses exec to create new classes              5 days
       http://bugs.python.org/issue3974    rhettinger                
       patch                                                                   

Check PyInt_AsSsize_t/PyLong_AsSsize_t error                        3 days
       http://bugs.python.org/issue3977    benjamin.peterson         
       patch                                                                   

Doctest failing when it should pass                                 0 days
       http://bugs.python.org/issue3979    georg.brandl              
                                                                               

win32file.GetCommState incorrect handling of DCB                    0 days
       http://bugs.python.org/issue3980    loewis                    
                                                                               

Python 3, IDLE does not start                                       1 days
       http://bugs.python.org/issue3981    dah                       
                                                                               

Typos in Documentation                                              0 days
       http://bugs.python.org/issue3983    georg.brandl              
                                                                               

Tools\Scripts\2to3.py broken under 3.0 rc1 Windows                  3 days
       http://bugs.python.org/issue3989    loewis                    
                                                                               

Convert documentation to python 3.                                  0 days
       http://bugs.python.org/issue3993    georg.brandl              
                                                                               

iso-xxx/cp1252 inconsistencies in Python 2.* not in 3.*             2 days
       http://bugs.python.org/issue3995    rpetrov                   
                                                                               

List.sort docstring has obsolete cmp reference.                     0 days
       http://bugs.python.org/issue3998    benjamin.peterson         
       easy                                                                    

2to3 does relative import for modules not in a package.             0 days
       http://bugs.python.org/issue4001    benjamin.peterson         
       patch                                                                   

A Bug in the Documentation                                          0 days
       http://bugs.python.org/issue4002    georg.brandl              
                                                                               

Reference leak in test_cprofile                                     0 days
       http://bugs.python.org/issue4003    brett.cannon              
       patch, needs review                                                     

A Minor Glitch in the Documentation                                 0 days
       http://bugs.python.org/issue4009    georg.brandl              
                                                                               

2.6 (final) uses old icons in Start Menu                            1 days
       http://bugs.python.org/issue4019    loewis                    
                                                                               

hasattr boundary case failure                                       0 days
       http://bugs.python.org/issue4020    benjamin.peterson         
                                                                               

msi installer does not requires UAC permission on Vista             0 days
       http://bugs.python.org/issue4030    loewis                    
                                                                               

Traceback error when compiling Regex                              921 days
       http://bugs.python.org/issue1456280 timehorse                 
                                                                               

Use flush() before os.exevp()                                     711 days
       http://bugs.python.org/issue1579477 akuchling                 
                                                                               

asyncore/asynchat patches                                         476 days
       http://bugs.python.org/issue1736190 giampaolo.rodola          
       patch                                                                   



Top Issues Most Discussed (10)
______________________________

 27 os.listdir can return byte strings                               101 days
open    http://bugs.python.org/issue3187   

 18 test_multiprocessing fails on systems with HAVE_SEM_OPEN=0        30 days
open    http://bugs.python.org/issue3770   

 12 "for me" installer problem on x64 Vista                            1 days
open    http://bugs.python.org/issue4018   

 10 Regexp 2.7 (modifications to current re 2.2.2)                   171 days
open    http://bugs.python.org/issue2636   

  9 Byte warning mode and b'' != ''                                    5 days
open    http://bugs.python.org/issue3988   

  9 support .format for bytes                                          6 days
open    http://bugs.python.org/issue3982   

  8 configure --with-threads on cygwin => crash on thread related t   10 days
open    http://bugs.python.org/issue3947   

  8 Multi-process 2to3                                                70 days
open    http://bugs.python.org/issue3448   

  7 Minor errors in multiprocessing docs                               2 days
open    http://bugs.python.org/issue4012   

  7 configure options don't trickle down to distutils                  2 days
open    http://bugs.python.org/issue4010   




From algorias at yahoo.com  Fri Oct  3 18:03:20 2008
From: algorias at yahoo.com (Vitor Bosshard)
Date: Fri, 3 Oct 2008 09:03:20 -0700 (PDT)
Subject: [Python-Dev] if-syntax for regular for-loops
Message-ID: <239314.61318.qm@web54401.mail.yahoo.com>



----- Mensaje original ----
> De: Leif Walsh <leif.walsh at gmail.com>
> Para: Andreas Nilsson <adde at trialcode.com>
> CC: python-dev at python.org
> Enviado: viernes, 3 de octubre, 2008 10:29:33
> Asunto: Re: [Python-Dev] if-syntax for regular for-loops
> 
> On Fri, Oct 3, 2008 at 6:10 AM, Andreas Nilsson wrote:
> > With that out of the way, on to todays subject:
> > I use list comprehensions and generator expressions a lot and lately I've
> > found myself writing a lot of code like this:
> >
> > for i in items if i.some_field == some_value: i.do_something()
> >
> > Naturally it won't work but it seems like a pretty straight-forward
> > extension to allow compressing simple loops to fit on one line. The
> > alternative, in my eyes, suggests there's something more happening than a
> > simple include-test which makes it harder to comprehend.
> >
> > for i in items:
> >? ? ? ? if i.some_field == some_value: i.do_something()
> >
> > One possibility of course is to use a generator-expression but that makes it
> > look like there are two for loops and it feels like a waste setting up a
> > generator just for filtering.
> >
> > for i in (i for i in items if some_field == some_value):
> >? ? ? ? i.do_something()
> >
> > Stupid idea? Am I missing some obviously better way of achieving the same
> > result?
> 
> It's been discussed already.? Essentially, all that saves is a newline
> or two, which, as I think has been generally accepted, tends to hurt
> readability.

The exact same argument could be used for list comprehensions themselves.?They exist anyway,?creating inconsistency in the language?(being?almost?identical to for loops regarding syntax)

Vitor


      ____________________________________________________________________________________
Premios MTV 2008?En exclusiva! Fotos, nominados, videos, y mucho m?s! Mira aqu? http://mtvla.yahoo.com/

From musiccomposition at gmail.com  Fri Oct  3 23:26:19 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Fri, 3 Oct 2008 16:26:19 -0500
Subject: [Python-Dev] for __future__ import planning
Message-ID: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>

So now that we've released 2.6 and are working hard on shepherding 3.0
out the door, it's time to worry about the next set of releases. :)

I propose that we dramatically shorten our release cycle for 2.7/3.1
to roughly a year and put a strong focus stabilizing all the new
goodies we included in the last release(s). In the 3.x branch, we
should continue to solidify the new code and features that were
introduced. One 2.7's main objectives should be binding 3.x and 2.x
ever closer.

-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From lists at cheimes.de  Sat Oct  4 00:00:40 2008
From: lists at cheimes.de (Christian Heimes)
Date: Sat, 04 Oct 2008 00:00:40 +0200
Subject: [Python-Dev] for __future__ import planning
In-Reply-To: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
Message-ID: <48E69608.2040604@cheimes.de>

Benjamin Peterson wrote:
> I propose that we dramatically shorten our release cycle for 2.7/3.1
> to roughly a year and put a strong focus stabilizing all the new
> goodies we included in the last release(s). In the 3.x branch, we
> should continue to solidify the new code and features that were
> introduced. One 2.7's main objectives should be binding 3.x and 2.x
> ever closer.

Hey! That was my idea! I told you the very same idea on IRC a week ago. 
Shame on you!

:)

I'm +1 on the proposal. Let's focus on stability and performance for the 
next release. But before we start planning the next release we need to 
find a way to sync the development.
Soon we have to apply fixes to up to four (again FOUR) branches: 2.6, 
2.7, 3.0 and 3.1. We don't have to merge as many code as we did during 
the py3k phase. But it's still lots of work and we need all the 
(technical) help we can get.

Christian

From amauryfa at gmail.com  Sat Oct  4 00:14:43 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sat, 4 Oct 2008 00:14:43 +0200
Subject: [Python-Dev] python-checkins seems broken
Message-ID: <e27efe130810031514s483c883cg135503fefddad1e8@mail.gmail.com>

Hello,

I consult very regularly (100x a day) the python-checkins and
python-300-checkins mailing list archives:
http://mail.python.org/pipermail/python-checkins/2008-October/date.html#end
http://mail.python.org/pipermail/python-3000-checkins/2008-October/date.html#end

But they did not receive the svn checkins since Friday morning (CEST timezone).
They do display the buildbot failures however.

I miss these messages, they are for me the best way to keep in sync
with the developments.
(I think I have read all the commit diffs for three years at least)
They are specially important these days, where many people can work on
the same files.

Do other subscribed people receive these commit messages?
Is there a problem with the mailer, or some SVN trigger?

-- 
Amaury Forgeot d'Arc

From leif.walsh at gmail.com  Sat Oct  4 00:24:52 2008
From: leif.walsh at gmail.com (Leif Walsh)
Date: Fri, 3 Oct 2008 18:24:52 -0400
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <278E520B-A3D4-4F54-BCA1-49DEF6421B2D@trialcode.com>
References: <652F78A2-2DD3-474E-9599-C7B6A3BA1097@trialcode.com>
	<cc7430500810030729h76e0baa5m8481e5dc26e3a93a@mail.gmail.com>
	<278E520B-A3D4-4F54-BCA1-49DEF6421B2D@trialcode.com>
Message-ID: <cc7430500810031524q6ed4df4fm850ac0126cccbec0@mail.gmail.com>

On Fri, Oct 3, 2008 at 12:33 PM, Andreas Nilsson <adde at trialcode.com> wrote:
> Thanks for the pointer!
> I don't buy the argument that newlines automagically improves readability
> though. You also get increased nesting suggesting something interesting is
> happening where it isn't and that hurts readability.
> And as Vitor said, all other constructions of the form 'for i in items' can
> have if-conditions attached to them, it's really not that far-fetched to
> assume that the loop behaves the same way. Consistency good, surprises bad.

Yeah, I know what you mean, and I kind of liked the idea of adding the
if statement to the for loop (for consistency, if nothing else), but
it's been discussed before, and plenty of people have made the same
argument.  Probably not worth it.

-- 
Cheers,
Leif

From amauryfa at gmail.com  Sat Oct  4 00:56:27 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sat, 4 Oct 2008 00:56:27 +0200
Subject: [Python-Dev] Suspect intermittent failure in buildbots
Message-ID: <e27efe130810031556o266cc7d3hf28584ee7b0709ca@mail.gmail.com>

I've noticed an error that comes up from time to time in python 3.0 buildbots.
The error is always similar to this one:

Traceback (most recent call last):
  File "E:\cygwin\home\db3l\buildarea\3.0.bolen-windows\build\lib\test\test_io.py",
line 900, in testBasicIO
    self.assertEquals(f.write("abc"), 3)
  File "E:\cygwin\home\db3l\buildarea\3.0.bolen-windows\build\lib\io.py",
line 1486, in write
    b = encoder.encode(s)
  File "E:\cygwin\home\db3l\buildarea\3.0.bolen-windows\build\lib\encodings\ascii.py",
line 22, in encode
    return codecs.ascii_encode(input, self.errors)[0]
AttributeError: 'NoneType' object has no attribute 'ascii_encode'

The most recent is here:
http://www.python.org/dev/buildbot/3.0/AMD64%20W2k8%203.0/builds/843/step-test/0

but this already happened on various buildbots:
http://www.google.fr/search?q=%27NoneType%27+object+has+no+attribute+encode+site:mail.python.org/pipermail/python-checkins&filter=0

"x86 XP-4 3.0", "amd64 gentoo 3.0", "AMD64 W2k8 3.0", "x86 W2k8 3.0",
"g4 osx.4 3.0", "OS X x86 3.0"
"x86 XP-3 trunk"

yes, even on trunk!
Every time, a "codecs" global module variable has been reset to None,
either in a codec module (encoding/ascii.py, encoding/mac_roman.py) or
in test_io.py.
Every time, io.py is not far (which may be normal, it must have the
larger usage of encodings written in a .py)

I know that modules globals are reset to None on interpreter shutdown,
but it does not seem to be the case here: the unit test fail, and
fails again when run in verbose mode at the end.

I checked that the "codecs" name is a module global: the disassembler
shows a LOAD_GLOBAL opcode followed by LOAD_ATTR:
   0 LOAD_GLOBAL              0 (codecs)
   3 LOAD_ATTR                1 (ascii_decode)
   ...

I fail to imagine a reason, apart from a creeping memory error (in
dictionary lookup; chilling idea).
Thoughts?

-- 
Amaury Forgeot d'Arc

From barry at python.org  Sat Oct  4 00:56:29 2008
From: barry at python.org (Barry Warsaw)
Date: Fri, 3 Oct 2008 18:56:29 -0400
Subject: [Python-Dev] for __future__ import planning
In-Reply-To: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
Message-ID: <3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 3, 2008, at 5:26 PM, Benjamin Peterson wrote:

> So now that we've released 2.6 and are working hard on shepherding 3.0
> out the door, it's time to worry about the next set of releases. :)
>
> I propose that we dramatically shorten our release cycle for 2.7/3.1
> to roughly a year and put a strong focus stabilizing all the new
> goodies we included in the last release(s). In the 3.x branch, we
> should continue to solidify the new code and features that were
> introduced. One 2.7's main objectives should be binding 3.x and 2.x
> ever closer.

There are several things that I would like to see us concentrate on  
after the 3.0 release.  I agree that 3.1 should be primarily a  
stabilizing release.  I suspect that we will find a lot of things that  
need tweaking only after 3.0 final has been out there for a while.

I think 2.7 should continue along the path of convergence toward 3.x.   
The vision some of us talked about at Pycon was that at some point  
down the line, maybe there's no difference between "python2.9 -3" and  
"python3.3 -2".

I would really like to see us adopt a distributed version control  
system.

I want our maintenance branches to always be in a releasable state.  I  
want to be confident enough about the tree to be able to cut a point  
release at any time.  I want to release a new point release from the  
maint branches once a month.

Christian rightly points out that with four active trees, we're going  
to a pretty big challenge on our hands.  How do other large open  
source projects handle similar situations?

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOajHXEjvBPtnXfVAQJ5qgP+I6k+kHMG2zPTvMIstM2wRmhtAPd7kKz9
S6bXllUBzpxQGMYfqR3Ze5/SVUMEV2HvINPDfg816sGOoxs0fMeori398rU97bkH
tOFHOEi/KLKMdgGdjGWWnV+iPEGF6stPMX/6nGQDhM5NMzj81hBgF+7U+UNbS7iM
dT2wk3vSZHQ=
=q4kW
-----END PGP SIGNATURE-----

From guido at python.org  Sat Oct  4 01:02:52 2008
From: guido at python.org (Guido van Rossum)
Date: Fri, 3 Oct 2008 16:02:52 -0700
Subject: [Python-Dev] Suspect intermittent failure in buildbots
In-Reply-To: <e27efe130810031556o266cc7d3hf28584ee7b0709ca@mail.gmail.com>
References: <e27efe130810031556o266cc7d3hf28584ee7b0709ca@mail.gmail.com>
Message-ID: <ca471dc20810031602v7c3e12e1yae265ec9d2a67697@mail.gmail.com>

Module globals are also reset when the module *object* is
garbage-collected (e.g. it's removed from sys.modules and not
referenced elsewhere), but the module *dict* is still referenced. This
can happen if all uses of the module is of the form "from <module>
import <something>" where the <something> is a class or function, and
at least one of the users survives the garbage-collected module.

I suspect that something is resetting part of sys.modules content. It
is a known problem (in some parts) that encodings modules cannot be
reset that way. I suspect that there is code in the regrtest.py
framework that does this (resetting part of sys.modules) in order to
restore a clean environment in some cases. Or perhaps it's one of the
tests that does this.

On Fri, Oct 3, 2008 at 3:56 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
> I've noticed an error that comes up from time to time in python 3.0 buildbots.
> The error is always similar to this one:
>
> Traceback (most recent call last):
>  File "E:\cygwin\home\db3l\buildarea\3.0.bolen-windows\build\lib\test\test_io.py",
> line 900, in testBasicIO
>    self.assertEquals(f.write("abc"), 3)
>  File "E:\cygwin\home\db3l\buildarea\3.0.bolen-windows\build\lib\io.py",
> line 1486, in write
>    b = encoder.encode(s)
>  File "E:\cygwin\home\db3l\buildarea\3.0.bolen-windows\build\lib\encodings\ascii.py",
> line 22, in encode
>    return codecs.ascii_encode(input, self.errors)[0]
> AttributeError: 'NoneType' object has no attribute 'ascii_encode'
>
> The most recent is here:
> http://www.python.org/dev/buildbot/3.0/AMD64%20W2k8%203.0/builds/843/step-test/0
>
> but this already happened on various buildbots:
> http://www.google.fr/search?q=%27NoneType%27+object+has+no+attribute+encode+site:mail.python.org/pipermail/python-checkins&filter=0
>
> "x86 XP-4 3.0", "amd64 gentoo 3.0", "AMD64 W2k8 3.0", "x86 W2k8 3.0",
> "g4 osx.4 3.0", "OS X x86 3.0"
> "x86 XP-3 trunk"
>
> yes, even on trunk!
> Every time, a "codecs" global module variable has been reset to None,
> either in a codec module (encoding/ascii.py, encoding/mac_roman.py) or
> in test_io.py.
> Every time, io.py is not far (which may be normal, it must have the
> larger usage of encodings written in a .py)
>
> I know that modules globals are reset to None on interpreter shutdown,
> but it does not seem to be the case here: the unit test fail, and
> fails again when run in verbose mode at the end.
>
> I checked that the "codecs" name is a module global: the disassembler
> shows a LOAD_GLOBAL opcode followed by LOAD_ATTR:
>   0 LOAD_GLOBAL              0 (codecs)
>   3 LOAD_ATTR                1 (ascii_decode)
>   ...
>
> I fail to imagine a reason, apart from a creeping memory error (in
> dictionary lookup; chilling idea).
> Thoughts?
>
> --
> Amaury Forgeot d'Arc
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From brett at python.org  Sat Oct  4 01:34:15 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 3 Oct 2008 16:34:15 -0700
Subject: [Python-Dev] for __future__ import planning
In-Reply-To: <3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
Message-ID: <bbaeab100810031634n732b0966q882905d5b32e2ddb@mail.gmail.com>

On Fri, Oct 3, 2008 at 3:56 PM, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Oct 3, 2008, at 5:26 PM, Benjamin Peterson wrote:
>
>> So now that we've released 2.6 and are working hard on shepherding 3.0
>> out the door, it's time to worry about the next set of releases. :)
>>
>> I propose that we dramatically shorten our release cycle for 2.7/3.1
>> to roughly a year and put a strong focus stabilizing all the new
>> goodies we included in the last release(s). In the 3.x branch, we
>> should continue to solidify the new code and features that were
>> introduced. One 2.7's main objectives should be binding 3.x and 2.x
>> ever closer.
>
> There are several things that I would like to see us concentrate on after
> the 3.0 release.  I agree that 3.1 should be primarily a stabilizing
> release.  I suspect that we will find a lot of things that need tweaking
> only after 3.0 final has been out there for a while.
>
> I think 2.7 should continue along the path of convergence toward 3.x.  The
> vision some of us talked about at Pycon was that at some point down the
> line, maybe there's no difference between "python2.9 -3" and "python3.3 -2".
>

+1 from me. I think 2.7/3.1 should be used as a chance to get our
testing framework straightened out and have those releases be
extremely rock-solid (especially 2.7 as it might be the last in the
2.x series).

Oh, and getting import rewritten in pure Python for 3.1 of course. =)

> I would really like to see us adopt a distributed version control system.
>

Along the lines of making 2.7/3.1 very stable releases, I would love
to use the time to clean up our workflow. To me that means cleaning up
the workflow on the issue tracker and getting on to a DVCS to make it
as easy as possible for people to contribute patches and for us to do
reviews.

> I want our maintenance branches to always be in a releasable state.  I want
> to be confident enough about the tree to be able to cut a point release at
> any time.  I want to release a new point release from the maint branches
> once a month.
>

Wow! I guess release.py is going to get really automated then. =) That
or you are going to manage to con more of us to help out (and even cut
the release ourselves).

> Christian rightly points out that with four active trees, we're going to a
> pretty big challenge on our hands.  How do other large open source projects
> handle similar situations?
>

Beats me. Are that many projects crazy enough to have that many active branches?

-Brett

From greg.ewing at canterbury.ac.nz  Sat Oct  4 01:34:10 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 04 Oct 2008 11:34:10 +1200
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <239314.61318.qm@web54401.mail.yahoo.com>
References: <239314.61318.qm@web54401.mail.yahoo.com>
Message-ID: <48E6ABF2.7050205@canterbury.ac.nz>

Vitor Bosshard wrote:
> 
>>On Fri, Oct 3, 2008 at 6:10 AM, Andreas Nilsson wrote:
>>Essentially, all that saves is a newline
>>or two, which, as I think has been generally accepted, tends to hurt
>>readability.
> 
> The exact same argument could be used for list comprehensions themselves.

No, an LC saves more than newlines -- it saves the code
to set up and append to a list. This is a substantial
improvement when this code would otherwise swamp the
essentials of what's being done.

This doesn't apply to a plain for-loop that's not
building a list.

-- 
Greg

From eric at trueblade.com  Sat Oct  4 01:43:00 2008
From: eric at trueblade.com (Eric Smith)
Date: Fri, 03 Oct 2008 19:43:00 -0400
Subject: [Python-Dev] for __future__ import planning
In-Reply-To: <bbaeab100810031634n732b0966q882905d5b32e2ddb@mail.gmail.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<bbaeab100810031634n732b0966q882905d5b32e2ddb@mail.gmail.com>
Message-ID: <48E6AE04.3010302@trueblade.com>

Brett Cannon wrote:
>> Christian rightly points out that with four active trees, we're going to a
>> pretty big challenge on our hands.  How do other large open source projects
>> handle similar situations?
>>
> 
> Beats me. Are that many projects crazy enough to have that many active branches?

Is it really that bad? Once 3.0 is released, it's not like we're going 
to be patching 2.6 and 3.0 all that much. All the "real development" (by 
which I mean most of the checkins) will be on 2.7 and 3.1. The biggest 
challenge I see is the buildbots.

Eric.


From amk at amk.ca  Sat Oct  4 02:17:40 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 3 Oct 2008 20:17:40 -0400
Subject: [Python-Dev] [Python-checkins] python-checkins seems broken
In-Reply-To: <e27efe130810031514s483c883cg135503fefddad1e8@mail.gmail.com>
References: <e27efe130810031514s483c883cg135503fefddad1e8@mail.gmail.com>
Message-ID: <20081004001740.GA20814@amk.local>

On Sat, Oct 04, 2008 at 12:14:43AM +0200, Amaury Forgeot d'Arc wrote:
> Do other subscribed people receive these commit messages?
> Is there a problem with the mailer, or some SVN trigger?

It looks like mail from dinsdale.python.org to mail.python.org isn't
working due to a DNS issue:

rcpt to: amk at python.org
550 5.7.1 Client host rejected: cannot find your reverse hostname, [82.94.164.164]
data

I know there's a transition to new IP addresses going on for the
python.org machines, but Thomas or Sean probably needs to do something
with the DNS for this.

--amk


From solipsis at pitrou.net  Sat Oct  4 02:18:17 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 4 Oct 2008 00:18:17 +0000 (UTC)
Subject: [Python-Dev] =?utf-8?q?for_=5F=5Ffuture=5F=5F_import_planning?=
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<bbaeab100810031634n732b0966q882905d5b32e2ddb@mail.gmail.com>
Message-ID: <loom.20081004T001624-297@post.gmane.org>

Brett Cannon <brett <at> python.org> writes:
> 
> Beats me. Are that many projects crazy enough to have that many active
> branches?

Any project using branch-driven development has many active branches. Our
specificity is that we must maintain in sync two branches (trunk, py3k) which
have widely diverged from each other. Thus, merges are often non-trivial.

Regards

Antoine.



From ncoghlan at gmail.com  Sat Oct  4 04:26:30 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 04 Oct 2008 12:26:30 +1000
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <48E6ABF2.7050205@canterbury.ac.nz>
References: <239314.61318.qm@web54401.mail.yahoo.com>
	<48E6ABF2.7050205@canterbury.ac.nz>
Message-ID: <48E6D456.6080004@gmail.com>

Greg Ewing wrote:
> Vitor Bosshard wrote:
>> The exact same argument could be used for list comprehensions themselves.
> No, an LC saves more than newlines -- it saves the code
> to set up and append to a list. This is a substantial
> improvement when this code would otherwise swamp the
> essentials of what's being done.
> 
> This doesn't apply to a plain for-loop that's not
> building a list.

Not only do LCs make it obvious to the reader that "all this loop does
is build a list", but the speed increases from doing the iteration in
native code rather than pure Python are also non-trivial - every pass
through the main eval loop that can be safely avoided leads to a fairly
substantial time saving.

Generally speaking, syntactic sugar (or new builtins) need to take a
construct in idiomatic Python that is fairly obvious to an experienced
Python user and make it obvious to even new users, or else take an idiom
that is easy to get wrong when writing (or miss when reading) and make
it trivial to use correctly.

Providing significant performance improvements (usually in the form of
reduced memory usage or increased speed) also counts heavily in favour
of new constructs.

I strongly suggest browsing through past PEPs (both accepted and
rejected ones) before proposing syntax changes, but here are some
examples of syntactic sugar proposals that were accepted.


List/set/dict comprehensions
============================
(and the reduction builtins any(), all(), min(), max(), sum())

  target = [op(x) for x in source]

instead of:
  target = []
  for x in source:
    target.append(op(x))

The transformation ("op(x)") is far more prominent in the comprehension
version, as is the fact that all the loop does is produce a new list. I
include the various reduction builtins here, since they serve exactly
the same purpose of taking an idiomatic looping construct and turning it
into a single expression.


Generator expressions
=====================

  total = sum(x*x for x in source)

instead of:

  def _g(seq):
    for x in source:
      yield x*x
  total = sum(_g(x))

or:

  total = sum([x*x for x in source])

Here, the GE version has obvious readability gains over the generator
function version (as with comprehensions, it brings the operation being
applied to each element front and centre instead of burying it in the
middle of the code, as well as allowing reduction operations like sum()
to retain their prominence), but doesn't actually improve readability
significantly over the second LC-based version. The gain over the
latter, of course, is that the GE based version needs a lot less
*memory* than the LC version, and, as it consumes the source data
incrementally, can work on source iterators of arbitrary (even infinite)
length, and can also cope with source iterators with large time gaps
between items (e.g. reading from a socket) as each item will be returned
as it becomes available.


With statements
===============

  with lock:
    # perform synchronised operations

instead of:

  lock.aqcuire()
  try:
    # perform synchronised operations
  finally:
    lock.release()

This change was a gain for both readability and writability - there were
plenty of ways to get this kind of code wrong (e.g. leave out the
try-finally altogether, acquire the resource inside the try block
instead of before it, call the wrong method or spell the variable name
wrong when attempting to release the resource in the finally block), and
it wasn't easy to audit because the lock acquisition and release could
be separated by an arbitrary number of lines of code. By combining all
of that into a single line of code at the beginning of the block, the
with statement eliminated a lot of those issues, making the code much
easier to write correctly in the first place, and also easier to audit
for correctness later (just make sure the code is using the correct
context manager for the task at hand).


Function decorators
===================

  @classmethod
  def f(cls):
    # Method body

instead of:

  def f(cls):
    # Method body
  f = classmethod(f)

Easier to write (function name only written once instead of three
times), and easier to read (decorator names up top with the function
signature instead of buried after the function body). Some folks still
dislike the use of the @ symbol, but compared to the drawbacks of the
old approach, the dedicated function decorator syntax is a huge improvement.


Conditional expressions
=======================

  x = A if C else B

instead of:

  x = C and A or B

The addition of conditional expressions arguably wasn't a particularly
big win for readability, but it *was* a big win for correctness. The
and/or based workaround for lack of a true conditional expression was
not only hard to read if you weren't already familiar with the
construct, but using it was also a potential buggy if A could ever be
False while C was True (in such case, B would be returned from the
expression instead of A).

Except clause
=============

  except Exception as ex:

instead of:

  except Exception, ex:

Another example of changing the syntax to eliminate potential bugs (in
this case, except clauses like "except TypeError, AttributeError:", that
would actually never catch AttributeError, and would locally do
AttributeError=TypeError if a TypeError was caught).

Cheers,
Nick.

P.S. There's a fractionally better argument to be used in favour of
allowing an if condition on the for loop header line: it doesn't just
save a newline or improve consistency with comprehensions and generator
expressions, it saves an *indentation level*. And that gain is exactly
the rationale that was used to begin allowing:

  try:
    ...
  except:
    ...
  else:
    ...
  finally:
    ...

instead of requiring the extra indentation level:

  try:
    try:
      ...
    except:
      ...
    else:
      ...
  finally:
    ...

However, even that argument is greatly weakened in the for/if case by
the fact that the indentation level is being saved by moving the if
condition up and to the right after the for loop details, whereas in the
try-statement case there were absolutely no downsides (the redundant try
keyword was simply dropped entirely).

So I'm personally still -1 when it comes to incorporating an if clause
directly into the for loop syntax - it's only necessary in the GE/LC
case due to the fact that those don't support statement-based nesting.

(Tangent: the above two try/except examples are perfectly legal Py3k
code. Do we really need the "pass" statement anymore?)



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From steve at pearwood.info  Sat Oct  4 05:08:30 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Sat, 4 Oct 2008 13:08:30 +1000
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <48E6D456.6080004@gmail.com>
References: <239314.61318.qm@web54401.mail.yahoo.com>
	<48E6ABF2.7050205@canterbury.ac.nz> <48E6D456.6080004@gmail.com>
Message-ID: <200810041308.30800.steve@pearwood.info>

On Sat, 4 Oct 2008 12:26:30 pm Nick Coghlan wrote:

> (Tangent: the above two try/except examples are perfectly legal Py3k
> code. Do we really need the "pass" statement anymore?)

I can't imagine why you would think we don't need the pass statement. I 
often use it:

* For subclassing exceptions:

class MyTypeError(TypeError):
    pass

* As a placeholder for code I haven't written yet.
* As a no-op used in, e.g. the timeit module.

And probably a few other places as well.



-- 
Steven

From greg at krypto.org  Sat Oct  4 08:19:06 2008
From: greg at krypto.org (Gregory P. Smith)
Date: Fri, 3 Oct 2008 23:19:06 -0700
Subject: [Python-Dev] Real segmentation fault handler
In-Reply-To: <gc31sg$k2v$1@ger.gmane.org>
References: <200809300105.53473.victor.stinner@haypocalc.com>
	<gc31sg$k2v$1@ger.gmane.org>
Message-ID: <52dc1c820810032319k2c67a7d1o52604b3d772bca1@mail.gmail.com>

On Thu, Oct 2, 2008 at 10:54 AM, Thomas Heller <theller at ctypes.org> wrote:
> Victor Stinner schrieb:
>> Hi,
>>
>> I would like to be able to catch SIGSEGV in my Python code! So I started to
>> hack Python trunk to support this feature. The idea is to use a signal
>> handler which call longjmp(), and add setjmp() at Py_EvalFrameEx() enter.
>
> On windows, ctypes catches fatal errors (exception violations) in
> foreign function calls, thanks to windows structured exception handling.
>
> On other platforms, there is the WAD module by David Beazley which
> may do something similar:
>
> http://www.dabeaz.com/papers/Python2001/python.html
>
> I do not know whether the code itself is still available or not.

It appears to be here:

 http://web.archive.org/web/20030113032725/systems.cs.uchicago.edu/wad/

It may need a bit of attention to get it to work today, I haven't tried.

-gps

From martin at v.loewis.de  Sat Oct  4 09:22:37 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Oct 2008 09:22:37 +0200
Subject: [Python-Dev] for __future__ import planning
In-Reply-To: <48E6AE04.3010302@trueblade.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	<bbaeab100810031634n732b0966q882905d5b32e2ddb@mail.gmail.com>
	<48E6AE04.3010302@trueblade.com>
Message-ID: <48E719BD.9000701@v.loewis.de>

> Is it really that bad? Once 3.0 is released, it's not like we're going
> to be patching 2.6 and 3.0 all that much.

And unfortunately so. The 2.5 branch doesn't get the attention that it
should, let alone the 2.4 branch. We will continue to "have" them (even
if only for security patches).

Regards,
Martin

From martin at v.loewis.de  Sat Oct  4 09:40:26 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Oct 2008 09:40:26 +0200
Subject: [Python-Dev] [Python-checkins] python-checkins seems broken
In-Reply-To: <20081004001740.GA20814@amk.local>
References: <e27efe130810031514s483c883cg135503fefddad1e8@mail.gmail.com>
	<20081004001740.GA20814@amk.local>
Message-ID: <48E71DEA.5060205@v.loewis.de>

> I know there's a transition to new IP addresses going on for the
> python.org machines, but Thomas or Sean probably needs to do something
> with the DNS for this.

IIUC, it would be sufficient if these addresses get recognized as local.
Meanwhile, I have disabled the new interfaces.

Regards,
Martin

From g.brandl at gmx.net  Sat Oct  4 09:45:27 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 04 Oct 2008 09:45:27 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
Message-ID: <gc76uv$rr9$1@ger.gmane.org>

Barry Warsaw schrieb:
> On Oct 3, 2008, at 5:26 PM, Benjamin Peterson wrote:
> 
>> So now that we've released 2.6 and are working hard on shepherding 3.0
>> out the door, it's time to worry about the next set of releases. :)
> 
>> I propose that we dramatically shorten our release cycle for 2.7/3.1
>> to roughly a year and put a strong focus stabilizing all the new
>> goodies we included in the last release(s). In the 3.x branch, we
>> should continue to solidify the new code and features that were
>> introduced. One 2.7's main objectives should be binding 3.x and 2.x
>> ever closer.
> 
> There are several things that I would like to see us concentrate on  
> after the 3.0 release.  I agree that 3.1 should be primarily a  
> stabilizing release.  I suspect that we will find a lot of things that  
> need tweaking only after 3.0 final has been out there for a while.
> 
> I think 2.7 should continue along the path of convergence toward 3.x.   
> The vision some of us talked about at Pycon was that at some point  
> down the line, maybe there's no difference between "python2.9 -3" and  
> "python3.3 -2".

Especially 3.1 should also be a release where we focus as much on the
community as on the code. There are many people out there for whom
Python 3, as an incompatible language, is not an easy step to make,
especially those with huge 2.x codebases on their hands. They have
two problems: The libraries they depend on aren't ported, and the
KLOC of code they care about are hard and tedious work to port, not
to mention that it typically isn't viewed as productive work by those
who pay them.

We need to make 2to3 and related tools reliable and do more showcases
of porting, like Martin did with Django, so that people have real-world
examples at their disposal, by which they can estimate their own
porting needs. (Waiting for the extended community to deliver such
examples may be a mistake.)

We also need to commit to help people with porting. I propose a new
mailing list (e.g. python3-porting), parallel to python-list,
specifically for people going that way. I think it will help to
focus the community effort of getting Python 3 off the ground.

Last not least, there should be a *central* location on python.org where
specifically all resources on 2->3 transition are collected. Talks,
documents, links, and some crucial information many people seem to miss,
such as how long the 2.x series will at least be maintained. They depend
on this.

cheers,
Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From g.brandl at gmx.net  Sat Oct  4 09:47:05 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 04 Oct 2008 09:47:05 +0200
Subject: [Python-Dev] Doc nits question
In-Reply-To: <gc3qgp$tnh$1@ger.gmane.org>
References: <4222a8490810020608p7dffab32v53287577dc447edb@mail.gmail.com>	<gc2hsa$p2j$2@ger.gmane.org>	<18122A2B-CCB0-4850-BDA4-10291780E9D2@acm.org>	<gc2vno$brt$1@ger.gmane.org>
	<gc3qgp$tnh$1@ger.gmane.org>
Message-ID: <gc7720$rr9$2@ger.gmane.org>

Terry Reedy schrieb:
> Georg Brandl wrote:
>> Fred Drake schrieb:
>>> On Oct 2, 2008, at 9:21 AM, Georg Brandl wrote:
>>>> I intend to set things up so that the docs at docs.python.org are  
>>>> continually
>>>> rebuilt, just like the /dev docs were until now.
> 
> Will you do the same for the 3.0 version?
> http://docs.python.org/dev/3.0/

Yes. Nothing has changed there.

> The following page has no reference to the 3.0 version
> http://www.python.org/doc/versions/
> The 'unreleased' link at the top goes to a link that only referenced the 
> SVN version and not the built version above.  Adding a link to the build 
> version would have made it easier to find ;-).

http://www.python.org/dev/doc/, where the link leads to, references both
trunk and 3.0 here.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From eric at trueblade.com  Sat Oct  4 11:30:41 2008
From: eric at trueblade.com (Eric Smith)
Date: Sat, 04 Oct 2008 05:30:41 -0400
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <200810041308.30800.steve@pearwood.info>
References: <239314.61318.qm@web54401.mail.yahoo.com>	<48E6ABF2.7050205@canterbury.ac.nz>
	<48E6D456.6080004@gmail.com>
	<200810041308.30800.steve@pearwood.info>
Message-ID: <48E737C1.1040200@trueblade.com>

Steven D'Aprano wrote:
> On Sat, 4 Oct 2008 12:26:30 pm Nick Coghlan wrote:
> 
>> (Tangent: the above two try/except examples are perfectly legal Py3k
>> code. Do we really need the "pass" statement anymore?)
> 
> I can't imagine why you would think we don't need the pass statement. I 
> often use it:
> 
> * For subclassing exceptions:
> 
> class MyTypeError(TypeError):
>     pass
> 
> * As a placeholder for code I haven't written yet.
> * As a no-op used in, e.g. the timeit module.
> 
> And probably a few other places as well.

Nick is literally (pardon the pun) saying that '...' could take the 
place of 'pass' in 3.0. His original examples are valid syntax, as 
written with the ellipses. For example:

$ ./python
Python 3.0rc1+ (py3k:66789, Oct  4 2008, 05:26:45)
[GCC 4.1.2 20070626 (Red Hat 4.1.2-13)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> try:
...   ...
... except:
...   ...
... else:
...   ...
... finally:
...   ...
...
Ellipsis
Ellipsis
Ellipsis
 >>>

I think it's a little too cute, and 'pass' is preferable.

Eric.


From amk at amk.ca  Sat Oct  4 12:16:50 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Sat, 4 Oct 2008 06:16:50 -0400
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <gc76uv$rr9$1@ger.gmane.org>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<gc76uv$rr9$1@ger.gmane.org>
Message-ID: <20081004101650.GA22969@amk.local>

On Sat, Oct 04, 2008 at 09:45:27AM +0200, Georg Brandl wrote:
> Barry Warsaw schrieb:
> two problems: The libraries they depend on aren't ported, and the
> KLOC of code they care about are hard and tedious work to port, not
> to mention that it typically isn't viewed as productive work by those
> who pay them.

PyPI should also be enhanced to indicate which libraries are
3.x-compatible, to make it easier for 3.x users to find packages they
can use.

This may be more complicated than it sounds, because you'd probably
add a very general requirement-indicating feature to PyPI, not merely
a 'supports 3.0' Boolean on each record, and requirements are actually
pretty complicated: alternative packages, specific version numbers...

--amk


From martin at v.loewis.de  Sat Oct  4 12:24:43 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Oct 2008 12:24:43 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <20081004101650.GA22969@amk.local>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	<gc76uv$rr9$1@ger.gmane.org>
	<20081004101650.GA22969@amk.local>
Message-ID: <48E7446B.70501@v.loewis.de>

> This may be more complicated than it sounds, because you'd probably
> add a very general requirement-indicating feature to PyPI, not merely
> a 'supports 3.0' Boolean on each record, and requirements are actually
> pretty complicated: alternative packages, specific version numbers...

Can you propose a spec?

Regards,
Martin

From solipsis at pitrou.net  Sat Oct  4 12:36:41 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 4 Oct 2008 10:36:41 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?3=2E1_focus_=28was_Re=3A_for_=5F=5Ffuture?=
	=?utf-8?q?=5F=5F_import_planning=29?=
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	<gc76uv$rr9$1@ger.gmane.org>
	<20081004101650.GA22969@amk.local> <48E7446B.70501@v.loewis.de>
Message-ID: <loom.20081004T103221-543@post.gmane.org>

Martin v. L?wis <martin <at> v.loewis.de> writes:
> 
> > This may be more complicated than it sounds, because you'd probably
> > add a very general requirement-indicating feature to PyPI, not merely
> > a 'supports 3.0' Boolean on each record, and requirements are actually
> > pretty complicated: alternative packages, specific version numbers...
> 
> Can you propose a spec?

Setuptools already has a syntax for declaring requirements.
http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies




From habnabit at gmail.com  Sat Oct  4 12:47:25 2008
From: habnabit at gmail.com (Aaron Gallagher)
Date: Sat, 4 Oct 2008 03:47:25 -0700
Subject: [Python-Dev] Pickle, 3.0, and recursion
Message-ID: <85085447-A576-487C-B698-B15C847730B8@gmail.com>

Hi,

I posted a bug ticket on bugs.python.org, specifically #3119. Link: http://bugs.python.org/issue3119
The original issue is that the default Pickler can exceed the maximum  
recursion depth for structures that are fairly nested. There's a patch  
attached to the ticket that fixes this behavior by using a generator- 
based callstack. Unfortunately, this completely changes the API of  
Pickler.

As far as I know, such major API changes would not be accepted in to  
3.0 at this point. I tried to come up with an alternate version of the  
patch that just added a subclass, but I haven't been able to think of  
a way to do that that doesn't involve a lot of code duplication. It's  
easy to just copy-paste most of Pickler and give it a new name and  
change the API, but that's hardly an optimal solution.

Does anybody else have any thoughts about how this could be better  
implemented? I think this is a decent implementation, but at this  
point, it's too severe of a change. Maybe somebody else has a better  
idea of how this could fit cleanly into the existing Pickler API or  
write a subclass without a lot of messy duplication.

From martin at v.loewis.de  Sat Oct  4 14:22:31 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sat, 04 Oct 2008 14:22:31 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <loom.20081004T103221-543@post.gmane.org>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	<gc76uv$rr9$1@ger.gmane.org>	<20081004101650.GA22969@amk.local>
	<48E7446B.70501@v.loewis.de>
	<loom.20081004T103221-543@post.gmane.org>
Message-ID: <48E76007.2040408@v.loewis.de>

Antoine Pitrou wrote:
> Martin v. L?wis <martin <at> v.loewis.de> writes:
>>> This may be more complicated than it sounds, because you'd probably
>>> add a very general requirement-indicating feature to PyPI, not merely
>>> a 'supports 3.0' Boolean on each record, and requirements are actually
>>> pretty complicated: alternative packages, specific version numbers...
>> Can you propose a spec?
> 
> Setuptools already has a syntax for declaring requirements.
> http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies

That is underspecified for the issue at hand: What *specifically* should
PyPI look for to determine 3.0 support in a package, and how
*specifically* should it display that?

An alternative to specifying dependencies would be to specify a
classifier, e.g.

  Programming Language :: Python :: 3

or

  Programming Langauge :: Python3

Then, no changes to PyPI are needed (except for adding the classifier to
the database), and searching for Python-3-supporting packages could
go through http://pypi.python.org/pypi?%3Aaction=browse

Regards,
Martin

From barry at python.org  Sat Oct  4 19:08:00 2008
From: barry at python.org (Barry Warsaw)
Date: Sat, 4 Oct 2008 13:08:00 -0400
Subject: [Python-Dev] for __future__ import planning
In-Reply-To: <bbaeab100810031634n732b0966q882905d5b32e2ddb@mail.gmail.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<bbaeab100810031634n732b0966q882905d5b32e2ddb@mail.gmail.com>
Message-ID: <9ADABB92-8C0B-46CA-958B-6F6BEC3D3917@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 3, 2008, at 7:34 PM, Brett Cannon wrote:

> Wow! I guess release.py is going to get really automated then. =) That
> or you are going to manage to con more of us to help out (and even cut
> the release ourselves).

release.py is really coming along well.  I'm planning to spend some  
time next time around on updating large parts of the web site  
automatically.  Or maybe Benjamin will when he's RM :)

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOei8XEjvBPtnXfVAQLm+wQAgctR3FuylGaST4Y55JIpo8pPOZ8ouzXW
L8wymNAz70neXMI2rVWk/qKyRovkwIgSe2GfKX14Vi85WGOw0MFvo2IkCx9VqUML
GdpR2j0ixX512/bBknZ3+Jsz5yUrppXBqFOuMHdSkxg/3YZSWKYUjRAUVk/48ujL
zbiVD+VJKUQ=
=qjlH
-----END PGP SIGNATURE-----

From brett at python.org  Sat Oct  4 20:03:54 2008
From: brett at python.org (Brett Cannon)
Date: Sat, 4 Oct 2008 11:03:54 -0700
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <gc76uv$rr9$1@ger.gmane.org>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<gc76uv$rr9$1@ger.gmane.org>
Message-ID: <bbaeab100810041103j7502018fmdcd2b575f81371d3@mail.gmail.com>

On Sat, Oct 4, 2008 at 12:45 AM, Georg Brandl <g.brandl at gmx.net> wrote:
> Barry Warsaw schrieb:
>> On Oct 3, 2008, at 5:26 PM, Benjamin Peterson wrote:
>>
>>> So now that we've released 2.6 and are working hard on shepherding 3.0
>>> out the door, it's time to worry about the next set of releases. :)
>>
>>> I propose that we dramatically shorten our release cycle for 2.7/3.1
>>> to roughly a year and put a strong focus stabilizing all the new
>>> goodies we included in the last release(s). In the 3.x branch, we
>>> should continue to solidify the new code and features that were
>>> introduced. One 2.7's main objectives should be binding 3.x and 2.x
>>> ever closer.
>>
>> There are several things that I would like to see us concentrate on
>> after the 3.0 release.  I agree that 3.1 should be primarily a
>> stabilizing release.  I suspect that we will find a lot of things that
>> need tweaking only after 3.0 final has been out there for a while.
>>
>> I think 2.7 should continue along the path of convergence toward 3.x.
>> The vision some of us talked about at Pycon was that at some point
>> down the line, maybe there's no difference between "python2.9 -3" and
>> "python3.3 -2".
>
> Especially 3.1 should also be a release where we focus as much on the
> community as on the code. There are many people out there for whom
> Python 3, as an incompatible language, is not an easy step to make,
> especially those with huge 2.x codebases on their hands. They have
> two problems: The libraries they depend on aren't ported, and the
> KLOC of code they care about are hard and tedious work to port, not
> to mention that it typically isn't viewed as productive work by those
> who pay them.
>
> We need to make 2to3 and related tools reliable and do more showcases
> of porting, like Martin did with Django, so that people have real-world
> examples at their disposal, by which they can estimate their own
> porting needs. (Waiting for the extended community to deliver such
> examples may be a mistake.)
>
> We also need to commit to help people with porting. I propose a new
> mailing list (e.g. python3-porting), parallel to python-list,
> specifically for people going that way. I think it will help to
> focus the community effort of getting Python 3 off the ground.
>

This is a good idea; python-help for porting.

> Last not least, there should be a *central* location on python.org where
> specifically all resources on 2->3 transition are collected. Talks,
> documents, links, and some crucial information many people seem to miss,
> such as how long the 2.x series will at least be maintained. They depend
> on this.

That seems reasonable if someone gets around to doing it. =)

-Brett

From g.brandl at gmx.net  Sat Oct  4 20:11:10 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 04 Oct 2008 20:11:10 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <bbaeab100810041103j7502018fmdcd2b575f81371d3@mail.gmail.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	<gc76uv$rr9$1@ger.gmane.org>
	<bbaeab100810041103j7502018fmdcd2b575f81371d3@mail.gmail.com>
Message-ID: <gc8bk8$uqb$1@ger.gmane.org>

Brett Cannon schrieb:

>> Last not least, there should be a *central* location on python.org where
>> specifically all resources on 2->3 transition are collected. Talks,
>> documents, links, and some crucial information many people seem to miss,
>> such as how long the 2.x series will at least be maintained. They depend
>> on this.
> 
> That seems reasonable if someone gets around to doing it. =)

Well, since for >95% of the (potential) Py3k users it is more important than
e.g. the import rewrite in Python (no stab at you intended, Brett), it is
something someone will have to get around to doing.

I'm not excusing myself; in fact, I'd be happy to work on this, but overall
the team "Python 3 advocacy and support" should consist of more than one
person.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From martin at v.loewis.de  Sat Oct  4 21:17:21 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 04 Oct 2008 21:17:21 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <gc8bk8$uqb$1@ger.gmane.org>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	<gc76uv$rr9$1@ger.gmane.org>	<bbaeab100810041103j7502018fmdcd2b575f81371d3@mail.gmail.com>
	<gc8bk8$uqb$1@ger.gmane.org>
Message-ID: <48E7C141.8010903@v.loewis.de>

> Well, since for >95% of the (potential) Py3k users it is more important than
> e.g. the import rewrite in Python (no stab at you intended, Brett), it is
> something someone will have to get around to doing.
> 
> I'm not excusing myself; in fact, I'd be happy to work on this, but overall
> the team "Python 3 advocacy and support" should consist of more than one
> person.

I think this has time. I'm (now) confident that people will port to
Python 3 sooner rather than later, just because it's there. In fact,
we have to be careful not to talk too many people into porting, since
there will be some glitches which need to be resolved, and may not get
resolved before 3.2 or so. So people with a natural wariness are advised
to trust this wariness, or else all their concerns become
self-fulfilling prophecies.

Regards,
Martin

From brett at python.org  Sat Oct  4 21:36:17 2008
From: brett at python.org (Brett Cannon)
Date: Sat, 4 Oct 2008 12:36:17 -0700
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <48E7C141.8010903@v.loewis.de>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<gc76uv$rr9$1@ger.gmane.org>
	<bbaeab100810041103j7502018fmdcd2b575f81371d3@mail.gmail.com>
	<gc8bk8$uqb$1@ger.gmane.org> <48E7C141.8010903@v.loewis.de>
Message-ID: <bbaeab100810041236m63ddb1cbr2f053a238c52049d@mail.gmail.com>

[replying to both Georg and Martin]

On Sat, Oct 4, 2008 at 12:17 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> Well, since for >95% of the (potential) Py3k users it is more important than
>> e.g. the import rewrite in Python (no stab at you intended, Brett), it is
>> something someone will have to get around to doing.
>>

Don't worry, I realize my import work is approaching vaporware status
at this rate (still plugging away at it, though).

But you are right: helping people port to 3 will be the most important
thing we can help people with.

>> I'm not excusing myself; in fact, I'd be happy to work on this, but overall
>> the team "Python 3 advocacy and support" should consist of more than one
>> person.
>

I would definitely be willing to help.

So the mailing list is a good idea. Perhaps it should just be
python-porting so that it can also be used for people who have
problems with minor releases?

We could then have a /porting/ section to the site where we can
actually document after each release how to port to the newest
version.

And as for 2 -> 3 stuff, should probably provide the expected steps to
port, tips for pure Python code (and how to write 2.6/3.0 compatible
code), extension modules, and make it clear what our overall plan is
(e.g. 3.2 probably being the truly stable release semantically).

> I think this has time. I'm (now) confident that people will port to
> Python 3 sooner rather than later, just because it's there. In fact,
> we have to be careful not to talk too many people into porting, since
> there will be some glitches which need to be resolved, and may not get
> resolved before 3.2 or so. So people with a natural wariness are advised
> to trust this wariness, or else all their concerns become
> self-fulfilling prophecies.

Yes, people should be warned that if they are not ready to make
changes after each Python release that are probably more than they are
used to between minor releases, they might to hold off for 3.1 or 3.2.
But I don't want to be too discouraging as that might stifle any
forward momentum we might have and potentially leave 3 flat before it
even gets going.

-Brett

From facundobatista at gmail.com  Sun Oct  5 01:19:31 2008
From: facundobatista at gmail.com (Facundo Batista)
Date: Sat, 4 Oct 2008 20:19:31 -0300
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <bbaeab100810041236m63ddb1cbr2f053a238c52049d@mail.gmail.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<gc76uv$rr9$1@ger.gmane.org>
	<bbaeab100810041103j7502018fmdcd2b575f81371d3@mail.gmail.com>
	<gc8bk8$uqb$1@ger.gmane.org> <48E7C141.8010903@v.loewis.de>
	<bbaeab100810041236m63ddb1cbr2f053a238c52049d@mail.gmail.com>
Message-ID: <e04bdf310810041619i7d57f3ct646ea1e48e5c716b@mail.gmail.com>

2008/10/4 Brett Cannon <brett at python.org>:

> So the mailing list is a good idea. Perhaps it should just be
> python-porting so that it can also be used for people who have
> problems with minor releases?

+1. I'd try to help on that list, also.

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From ziade.tarek at gmail.com  Sun Oct  5 01:53:22 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Oct 2008 01:53:22 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <48E76007.2040408@v.loewis.de>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<gc76uv$rr9$1@ger.gmane.org> <20081004101650.GA22969@amk.local>
	<48E7446B.70501@v.loewis.de> <loom.20081004T103221-543@post.gmane.org>
	<48E76007.2040408@v.loewis.de>
Message-ID: <94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>

On Sat, Oct 4, 2008 at 2:22 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Antoine Pitrou wrote:
>> Martin v. L?wis <martin <at> v.loewis.de> writes:
>>>> This may be more complicated than it sounds, because you'd probably
>>>> add a very general requirement-indicating feature to PyPI, not merely
>>>> a 'supports 3.0' Boolean on each record, and requirements are actually
>>>> pretty complicated: alternative packages, specific version numbers...
>>> Can you propose a spec?
>>
>> Setuptools already has a syntax for declaring requirements.
>> http://peak.telecommunity.com/DevCenter/setuptools#declaring-dependencies
>
> That is underspecified for the issue at hand: What *specifically* should
> PyPI look for to determine 3.0 support in a package, and how
> *specifically* should it display that?
>
> An alternative to specifying dependencies would be to specify a
> classifier, e.g.
>
>  Programming Language :: Python :: 3
>
> or
>
>  Programming Langauge :: Python3
>
> Then, no changes to PyPI are needed (except for adding the classifier to
> the database), and searching for Python-3-supporting packages could
> go through http://pypi.python.org/pypi?%3Aaction=browse

Setuptools declares dependencies, but does not add a Python version requirement,
like what was proposed in PEP 345 (http://www.python.org/dev/peps/pep-0345/)
with a new metadata called 'Requires Python'

Even if the problem is fixed in short term with a Trove classifier, the metadata
of a package should provide this information imho.

>From there PyPI should be able to provide the Python version directly.

For a full list of dependencies this is another tough problem due to the dynamic
nature of the metadata: you have to run setup.py to get the list of dependencies
because they are determined by code. This is by design, and plans for
a refactoring are
being discussed in distutils-SIG since a few days.

In any case, extending metadata with supported Python version could
be a first simple, useful step to do in distutils and PyPI today. So
the Python version
comes as the same level than the Platform or the License.


Tarek

From martin at v.loewis.de  Sun Oct  5 06:59:36 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 05 Oct 2008 06:59:36 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	
	<gc76uv$rr9$1@ger.gmane.org> <20081004101650.GA22969@amk.local>	
	<48E7446B.70501@v.loewis.de>
	<loom.20081004T103221-543@post.gmane.org>	
	<48E76007.2040408@v.loewis.de>
	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>
Message-ID: <48E849B8.3010009@v.loewis.de>

> Setuptools declares dependencies, but does not add a Python version requirement,
> like what was proposed in PEP 345 (http://www.python.org/dev/peps/pep-0345/)
> with a new metadata called 'Requires Python'
> 
> Even if the problem is fixed in short term with a Trove classifier, 

Why would that be a short-term fix? It would work for the entire life
time of Python 3, and can be easily extended for Python 4, so it sounds
like a permanent solution to me.

As a permanent solution, you need to keep the problem statement in mind,
though: to allow people to determine whether a package works with Python 3.

> the metadata of a package should provide this information imho.

Well, the classifiers *are* part of the metadata of the package.

Regards,
Martin

From ziade.tarek at gmail.com  Sun Oct  5 10:25:49 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sun, 5 Oct 2008 10:25:49 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <48E849B8.3010009@v.loewis.de>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<gc76uv$rr9$1@ger.gmane.org> <20081004101650.GA22969@amk.local>
	<48E7446B.70501@v.loewis.de> <loom.20081004T103221-543@post.gmane.org>
	<48E76007.2040408@v.loewis.de>
	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>
	<48E849B8.3010009@v.loewis.de>
Message-ID: <94bdd2610810050125w95a5c12r3e9df88796c845f@mail.gmail.com>

On Sun, Oct 5, 2008 at 6:59 AM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> Setuptools declares dependencies, but does not add a Python version requirement,
>> like what was proposed in PEP 345 (http://www.python.org/dev/peps/pep-0345/)
>> with a new metadata called 'Requires Python'
>>
>> Even if the problem is fixed in short term with a Trove classifier,
>
> Why would that be a short-term fix? It would work for the entire life
> time of Python 3, and can be easily extended for Python 4, so it sounds
> like a permanent solution to me.
>
> As a permanent solution, you need to keep the problem statement in mind,
> though: to allow people to determine whether a package works with Python 3.
>
>> the metadata of a package should provide this information imho.
>
> Well, the classifiers *are* part of the metadata of the package.

Yes and they are redundant to other metadata, like the licence for instance..

So maybe the problem is the other way: some specific metadata should be removed
since they are expressed in the classifier medatata. (e.g. only in one place)

There should be one-- and preferably only one --obvious way to do it.

Regards
Tarek

From kevin at bud.ca  Sun Oct  5 10:46:24 2008
From: kevin at bud.ca (Kevin Teague)
Date: Sun, 5 Oct 2008 01:46:24 -0700
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <48E849B8.3010009@v.loewis.de>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	
	<gc76uv$rr9$1@ger.gmane.org> <20081004101650.GA22969@amk.local>	
	<48E7446B.70501@v.loewis.de>
	<loom.20081004T103221-543@post.gmane.org>	
	<48E76007.2040408@v.loewis.de>
	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>
	<48E849B8.3010009@v.loewis.de>
Message-ID: <0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>


On Oct 4, 2008, at 9:59 PM, Martin v. L?wis wrote:

>> Setuptools declares dependencies, but does not add a Python version  
>> requirement,
>> like what was proposed in PEP 345 (http://www.python.org/dev/peps/pep-0345/ 
>> )
>> with a new metadata called 'Requires Python'
>>
>> Even if the problem is fixed in short term with a Trove classifier,
>
> Why would that be a short-term fix? It would work for the entire life
> time of Python 3, and can be easily extended for Python 4, so it  
> sounds
> like a permanent solution to me.
>
> As a permanent solution, you need to keep the problem statement in  
> mind,
> though: to allow people to determine whether a package works with  
> Python 3.
>


The Setuptools dependencies fields are only for declaring  library  
dependencies, not dependencies such as the Python interpreter.

A problem with overloading the Categories field with Python version  
compatability information is that it makes for a poor user-interface.  
On the release page for a package, I'd rather see a Python Version  
field than having to look through a potentially large list of  
Categories.

There is a "Py Version" field for file uploads (which I'd really like  
to tweak the UI to read "Python Version"), maybe if you could specify  
for uploads of the "source" type the Python compatability of the  
source file? Even without  Python 3 it would be nice to upload a  
source file and indicate that you've used a Python 2.5 or 2.6ism in  
your code (or made use of newer standard library modules).

Although this raises the question, what is the recommended practice  
for version numbering for distributions attempting to support both  
Python 2 and 3? Say I have a distribution that I've made work with  
Python 2.6 and can run on Python 3 after running it through 2to3. The  
source code is different so that to me suggests different version  
numbers - but the API will be the same, so maybe the same version  
number should be used? That is should there be?

foobar-1.0-py2.6.tar.gz
foobar-1.0-py3.0.tar.gz

Or should it be something like?

foobar-1.0.tar.gz # Python 2
foobar-1.1.tar.gz # Python 3

Well, writing out that last example looks wrong to me, since 1.0 ->  
1.1 suggests bug fixes and API changes to the package. But someone  
could be thinking, "1.0 supports the Python 2 API and 1.1 supports the  
Python 3 language API". So at any rate it'd be good to make the way to  
handle this explicit in the (yet to be created) "2to3 central"  
documentation resource.



From martin at v.loewis.de  Sun Oct  5 11:05:29 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 05 Oct 2008 11:05:29 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	
	<gc76uv$rr9$1@ger.gmane.org> <20081004101650.GA22969@amk.local>	
	<48E7446B.70501@v.loewis.de>
	<loom.20081004T103221-543@post.gmane.org>	
	<48E76007.2040408@v.loewis.de>
	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>
	<48E849B8.3010009@v.loewis.de>
	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
Message-ID: <48E88359.90700@v.loewis.de>

> A problem with overloading the Categories field with Python version
> compatability information is that it makes for a poor user-interface. On
> the release page for a package, I'd rather see a Python Version field
> than having to look through a potentially large list of Categories.

That's an issue of how the page is rendered. Hence I asked for a full
specification.

> There is a "Py Version" field for file uploads (which I'd really like to
> tweak the UI to read "Python Version")

Feel free to submit a patch (or perhaps just a bug report).

> maybe if you could specify for
> uploads of the "source" type the Python compatability of the source
> file? Even without  Python 3 it would be nice to upload a source file
> and indicate that you've used a Python 2.5 or 2.6ism in your code (or
> made use of newer standard library modules).

I think that deviates from the subject, which is "3.1 focus", and
AMK's request to let people find out whether a package runs on Python 3.

> Although this raises the question, what is the recommended practice for
> version numbering for distributions attempting to support both Python 2
> and 3? Say I have a distribution that I've made work with Python 2.6 and
> can run on Python 3 after running it through 2to3. The source code is
> different so that to me suggests different version numbers - but the API
> will be the same, so maybe the same version number should be used? That
> is should there be?

It all depends on how you do the porting. If all it takes to run the
program through 2to3 to make it port correctly, I recommend to use
the build_py_2to3 build step of distutils in 3.0. Then the same source
can be installed for 2.x and 3.x, with no modifications.

Regards,
Martin


From ncoghlan at gmail.com  Sun Oct  5 11:53:35 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 05 Oct 2008 19:53:35 +1000
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <48E737C1.1040200@trueblade.com>
References: <239314.61318.qm@web54401.mail.yahoo.com>	<48E6ABF2.7050205@canterbury.ac.nz>	<48E6D456.6080004@gmail.com>	<200810041308.30800.steve@pearwood.info>
	<48E737C1.1040200@trueblade.com>
Message-ID: <48E88E9F.4050505@gmail.com>

Eric Smith wrote:
> I think it's a little too cute, and 'pass' is preferable.

Agreed - it was just a little surreal to put the "..." in as my usual
pseudo-code "stuff happens here" marker, only to realise what I had
written was still executable Py3k code.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Sun Oct  5 12:00:55 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 05 Oct 2008 20:00:55 +1000
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <48E7276D.8030106@plaut.com.au>
References: <48E6D456.6080004@gmail.com> <48E7276D.8030106@plaut.com.au>
Message-ID: <48E89057.1030609@gmail.com>

Matthew Hawkins wrote:
> If there's another way of doing this I'd like to hear it.

The pass statement is still the right way to denote an empty block (the
compiler is already able to detect that and optimise the code
appropriately).

My tangential comment was based on the fact that this keyword could, in
theory, be replaced with the ellipsis notation often used in pseudo-code
to denote "real code goes here".

I don't think pursuing such a switch is actually a good idea - it was
just odd to notice that the Py3k interpreter would quite happily execute
the example code in my postscript when I had really only intended to
write it as pseudo-code with sections missing.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From greg.ewing at canterbury.ac.nz  Sun Oct  5 12:05:24 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 05 Oct 2008 23:05:24 +1300
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <48E89057.1030609@gmail.com>
References: <48E6D456.6080004@gmail.com> <48E7276D.8030106@plaut.com.au>
	<48E89057.1030609@gmail.com>
Message-ID: <48E89164.8010205@canterbury.ac.nz>

Nick Coghlan wrote:
> it was
> just odd to notice that the Py3k interpreter would quite happily execute
> the example code in my postscript when I had really only intended to
> write it as pseudo-code with sections missing.

Well, they do say that Python is executable
pseudocode. :-)

-- 
Greg

From ncoghlan at gmail.com  Sun Oct  5 12:20:47 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 05 Oct 2008 20:20:47 +1000
Subject: [Python-Dev] if-syntax for regular for-loops
In-Reply-To: <48E89164.8010205@canterbury.ac.nz>
References: <48E6D456.6080004@gmail.com>
	<48E7276D.8030106@plaut.com.au>	<48E89057.1030609@gmail.com>
	<48E89164.8010205@canterbury.ac.nz>
Message-ID: <48E894FF.8060200@gmail.com>

Greg Ewing wrote:
> Nick Coghlan wrote:
>> it was
>> just odd to notice that the Py3k interpreter would quite happily execute
>> the example code in my postscript when I had really only intended to
>> write it as pseudo-code with sections missing.
> 
> Well, they do say that Python is executable
> pseudocode. :-)

Indeed :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From steve at holdenweb.com  Sun Oct  5 13:54:12 2008
From: steve at holdenweb.com (Steve Holden)
Date: Sun, 05 Oct 2008 07:54:12 -0400
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <48E88359.90700@v.loewis.de>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>		<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>		<gc76uv$rr9$1@ger.gmane.org>
	<20081004101650.GA22969@amk.local>		<48E7446B.70501@v.loewis.de>	<loom.20081004T103221-543@post.gmane.org>		<48E76007.2040408@v.loewis.de>	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>	<48E849B8.3010009@v.loewis.de>	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
	<48E88359.90700@v.loewis.de>
Message-ID: <48E8AAE4.6060104@holdenweb.com>

Martin v. L?wis wrote:
>> A problem with overloading the Categories field with Python version
>> compatability information is that it makes for a poor user-interface. On
>> the release page for a package, I'd rather see a Python Version field
>> than having to look through a potentially large list of Categories.
> 
> That's an issue of how the page is rendered. Hence I asked for a full
> specification.
> 
>> There is a "Py Version" field for file uploads (which I'd really like to
>> tweak the UI to read "Python Version")
> 
> Feel free to submit a patch (or perhaps just a bug report).
> 
>> maybe if you could specify for
>> uploads of the "source" type the Python compatability of the source
>> file? Even without  Python 3 it would be nice to upload a source file
>> and indicate that you've used a Python 2.5 or 2.6ism in your code (or
>> made use of newer standard library modules).
> 
> I think that deviates from the subject, which is "3.1 focus", and
> AMK's request to let people find out whether a package runs on Python 3.
> 
>> Although this raises the question, what is the recommended practice for
>> version numbering for distributions attempting to support both Python 2
>> and 3? Say I have a distribution that I've made work with Python 2.6 and
>> can run on Python 3 after running it through 2to3. The source code is
>> different so that to me suggests different version numbers - but the API
>> will be the same, so maybe the same version number should be used? That
>> is should there be?
> 
> It all depends on how you do the porting. If all it takes to run the
> program through 2to3 to make it port correctly, I recommend to use
> the build_py_2to3 build step of distutils in 3.0. Then the same source
> can be installed for 2.x and 3.x, with no modifications.
> 
Of course there is also the option of treating Python 3 as a different
language, and having a Py3Pi website as well. This might not be as
wasteful as it at first seems.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From steve at holdenweb.com  Sun Oct  5 13:54:12 2008
From: steve at holdenweb.com (Steve Holden)
Date: Sun, 05 Oct 2008 07:54:12 -0400
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <48E88359.90700@v.loewis.de>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>		<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>		<gc76uv$rr9$1@ger.gmane.org>
	<20081004101650.GA22969@amk.local>		<48E7446B.70501@v.loewis.de>	<loom.20081004T103221-543@post.gmane.org>		<48E76007.2040408@v.loewis.de>	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>	<48E849B8.3010009@v.loewis.de>	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
	<48E88359.90700@v.loewis.de>
Message-ID: <48E8AAE4.6060104@holdenweb.com>

Martin v. L?wis wrote:
>> A problem with overloading the Categories field with Python version
>> compatability information is that it makes for a poor user-interface. On
>> the release page for a package, I'd rather see a Python Version field
>> than having to look through a potentially large list of Categories.
> 
> That's an issue of how the page is rendered. Hence I asked for a full
> specification.
> 
>> There is a "Py Version" field for file uploads (which I'd really like to
>> tweak the UI to read "Python Version")
> 
> Feel free to submit a patch (or perhaps just a bug report).
> 
>> maybe if you could specify for
>> uploads of the "source" type the Python compatability of the source
>> file? Even without  Python 3 it would be nice to upload a source file
>> and indicate that you've used a Python 2.5 or 2.6ism in your code (or
>> made use of newer standard library modules).
> 
> I think that deviates from the subject, which is "3.1 focus", and
> AMK's request to let people find out whether a package runs on Python 3.
> 
>> Although this raises the question, what is the recommended practice for
>> version numbering for distributions attempting to support both Python 2
>> and 3? Say I have a distribution that I've made work with Python 2.6 and
>> can run on Python 3 after running it through 2to3. The source code is
>> different so that to me suggests different version numbers - but the API
>> will be the same, so maybe the same version number should be used? That
>> is should there be?
> 
> It all depends on how you do the porting. If all it takes to run the
> program through 2to3 to make it port correctly, I recommend to use
> the build_py_2to3 build step of distutils in 3.0. Then the same source
> can be installed for 2.x and 3.x, with no modifications.
> 
Of course there is also the option of treating Python 3 as a different
language, and having a Py3Pi website as well. This might not be as
wasteful as it at first seems.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From facundobatista at gmail.com  Sun Oct  5 15:32:36 2008
From: facundobatista at gmail.com (Facundo Batista)
Date: Sun, 5 Oct 2008 11:32:36 -0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
	<gc76uv$rr9$1@ger.gmane.org> <20081004101650.GA22969@amk.local>
	<48E7446B.70501@v.loewis.de> <loom.20081004T103221-543@post.gmane.org>
	<48E76007.2040408@v.loewis.de>
	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>
	<48E849B8.3010009@v.loewis.de>
	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
Message-ID: <e04bdf310810050632o46bc6651sf3e4522c3e01b1ff@mail.gmail.com>

2008/10/5 Kevin Teague <kevin at bud.ca>:

> on Python 3 after running it through 2to3. The source code is different so
> that to me suggests different version numbers - but the API will be the
> same, so maybe the same version number should be used? That is should there
> be?
>
> foobar-1.0-py2.6.tar.gz
> foobar-1.0-py3.0.tar.gz

More likely, in this way:

foobar-1.0-py2.tar.gz
foobar-1.0-py3.tar.gz


> Or should it be something like?
>
> foobar-1.0.tar.gz # Python 2
> foobar-1.1.tar.gz # Python 3

The problem here is that I can foresee libs or apps that have a mixed
path, starting to support Py 3 at some point, and deprecating the Py 2
support after a while:

foobar-1.0.tar.gz   # Py 2
foobar-1.1.tar.gz   # Py 2 & 3
foobar-1.2.tar.gz   # Py 2 & 3
foobar-1.3.tar.gz   # Py 2 & 3
foobar-1.4.tar.gz   # Py 2 & 3
foobar-1.5.tar.gz   # Py 3

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From python at rcn.com  Sun Oct  5 17:08:56 2008
From: python at rcn.com (Raymond Hettinger)
Date: Sun, 5 Oct 2008 08:08:56 -0700
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>		<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>		<gc76uv$rr9$1@ger.gmane.org><20081004101650.GA22969@amk.local>		<48E7446B.70501@v.loewis.de>	<loom.20081004T103221-543@post.gmane.org>		<48E76007.2040408@v.loewis.de>	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>	<48E849B8.3010009@v.loewis.de>	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca><48E88359.90700@v.loewis.de>
	<48E8AAE4.6060104@holdenweb.com>
Message-ID: <26455FED7E2B4F3A8C6D19892A993EB8@RaymondLaptop1>

[Steve Holden"]
>
> Of course there is also the option of treating Python 3 as a different
> language, and having a Py3Pi website as well. This might not be as
> wasteful as it at first seems.

It would be nice if we had a way of marking Py2.6 recipes that still
work when run through 2-to-3 and then automatically porting updates
to the Py3Pi website.


Raymond


From martin at v.loewis.de  Sun Oct  5 18:32:26 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 05 Oct 2008 18:32:26 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <48E8AAE4.6060104@holdenweb.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>		<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>		<gc76uv$rr9$1@ger.gmane.org>
	<20081004101650.GA22969@amk.local>		<48E7446B.70501@v.loewis.de>	<loom.20081004T103221-543@post.gmane.org>		<48E76007.2040408@v.loewis.de>	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>	<48E849B8.3010009@v.loewis.de>	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
	<48E88359.90700@v.loewis.de> <48E8AAE4.6060104@holdenweb.com>
Message-ID: <48E8EC1A.9000304@v.loewis.de>

> Of course there is also the option of treating Python 3 as a different
> language, and having a Py3Pi website as well. This might not be as
> wasteful as it at first seems.

Although it would be possible, I think it's not appropriate.

It would be fairly easy to implement, though: just create
pypi.python.org/python3 (say), along with /simple3 and /packages3.
I find it inappropriate for two reasons: over time, Python 2 will
disappear, and then we are left with the python3 URLs (similar
how Microsoft now needs to put 64-bit binaries into the system32
folder). In addition, packages that do support both versions need
to register twice, upload twice, etc.

In any case, if that route is chosen, we have to adjust distutils
before the 3.0 release is made.

Regards,
Martin

From martin at v.loewis.de  Sun Oct  5 18:38:56 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 05 Oct 2008 18:38:56 +0200
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <e04bdf310810050632o46bc6651sf3e4522c3e01b1ff@mail.gmail.com>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>	
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>	
	<gc76uv$rr9$1@ger.gmane.org> <20081004101650.GA22969@amk.local>	
	<48E7446B.70501@v.loewis.de>
	<loom.20081004T103221-543@post.gmane.org>	
	<48E76007.2040408@v.loewis.de>	
	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>	
	<48E849B8.3010009@v.loewis.de>	
	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
	<e04bdf310810050632o46bc6651sf3e4522c3e01b1ff@mail.gmail.com>
Message-ID: <48E8EDA0.5060501@v.loewis.de>

>> foobar-1.0-py2.6.tar.gz
>> foobar-1.0-py3.0.tar.gz
> 
> More likely, in this way:
> 
> foobar-1.0-py2.tar.gz
> foobar-1.0-py3.tar.gz

How do you implement this in distutils? People probably won't rename
the files from how sdist names them. So it's more likely that they
come up with things like

foobar_py3-1.0.tar.gz

and register a second project foobar_py3 in PyPI.

Regards,
Martin


From barry at barrys-emacs.org  Sun Oct  5 20:18:18 2008
From: barry at barrys-emacs.org (Barry Scott)
Date: Sun, 5 Oct 2008 19:18:18 +0100
Subject: [Python-Dev] porting pycxx and pysvn to python 3.0 hit a problem
Message-ID: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>

I have a version of PyCXX ported to python 3.0 rc1 and its passing  
its tests.
I'm porting pysvn to python 3.0 rc1 and hit an odd problem.

Given this dict:

	wc_status_kind_map = {
    	 	pysvn.wc_status_kind.added: 'A',
     		pysvn.wc_status_kind.replaced: 'R',
     		pysvn.wc_status_kind.unversioned: '?',
     	}

This fails:

	wc_status_kind_map[ pysvn.wc_status_kind.unversioned ]

With:

	KeyError: <wc_status_kind.unversioned>

Clearly I have a bug in the implementation of pysvn.wc_status_kind and
could do with some suggestions on what to check please.

I've assumed that if my objects support tp_hash and tp_compare
they can be used as keys to a dictionary. My test scripts shows
hash() and cmp() working.

Why does "key in wc_status_kind_wc" work when I use an object returned
by keys() by not when I use pysvn.wc_status_kind.unversioned?

Barry


--------- a.py --------
import pysvn

wc_status_kind_map = {
     pysvn.wc_status_kind.replaced: 'R',
     pysvn.wc_status_kind.unversioned: 'U',
     }

print( 'hash wc_status_kind', hash( "wc_status_kind" ) )
print( 'hash replace', hash( pysvn.wc_status_kind.replaced ) )
print( 'hash unversioned', hash( pysvn.wc_status_kind.unversioned ) )

print( 'cmp unversioned, unversioned', cmp 
( pysvn.wc_status_kind.unversioned, pysvn.wc_status_kind.unversioned ) )
print( 'cmp unversioned, replaced', cmp 
( pysvn.wc_status_kind.unversioned, pysvn.wc_status_kind.replaced ) )
print( 'cmp replaced, unversioned', cmp 
( pysvn.wc_status_kind.replaced, pysvn.wc_status_kind.unversioned ) )

for key in wc_status_kind_map.keys():
     print( '1 key', key, key in wc_status_kind_map, cmp( key,  
pysvn.wc_status_kind.unversioned ), hash( key ) )
     try:
         print( '1 lookup', wc_status_kind_map[ key ] )
     except:
         print( '1 failed' )

for key in [pysvn.wc_status_kind.added,
             pysvn.wc_status_kind.replaced,
             pysvn.wc_status_kind.unversioned]:
     print( '2 key', key, key in wc_status_kind_map, cmp( key,  
pysvn.wc_status_kind.unversioned ), hash( key ) )
     try:
         print( '2 lookup', wc_status_kind_map[ key ] )
     except:
         print( '2 failed' )

print( wc_status_kind_map[ pysvn.wc_status_kind.unversioned ] )

------------ Output -------
$ python3.0 a.py
hash wc_status_kind -586300918
hash replace -586300911
hash unversioned -586300916
cmp unversioned, unversioned 0
cmp unversioned, replaced -1
cmp replaced, unversioned 1
1 key replaced True 1 -586300911
1 lookup R
1 key unversioned True 0 -586300916
1 lookup U
2 key added False 1 -586300914
2 failed
2 key replaced False 1 -586300911
2 failed
2 key unversioned False 0 -586300916
2 failed
Traceback (most recent call last):
   File "a.py", line 32, in <module>
     print( wc_status_kind_map[ pysvn.wc_status_kind.unversioned ] )
KeyError: <wc_status_kind.unversioned>


From martin at v.loewis.de  Sun Oct  5 20:47:59 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 05 Oct 2008 20:47:59 +0200
Subject: [Python-Dev] porting pycxx and pysvn to python 3.0 hit a problem
In-Reply-To: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>
References: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>
Message-ID: <48E90BDF.9030501@v.loewis.de>

> Why does "key in wc_status_kind_wc" work when I use an object returned
> by keys() by not when I use pysvn.wc_status_kind.unversioned?

This is too little detail to come up with an explanation. Do your
objects support __eq__.

Regards,
Martin

From tjreedy at udel.edu  Sun Oct  5 23:49:04 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 05 Oct 2008 17:49:04 -0400
Subject: [Python-Dev] porting pycxx and pysvn to python 3.0 hit a problem
In-Reply-To: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>
References: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>
Message-ID: <gcbcoe$irn$1@ger.gmane.org>

Barry Scott wrote:

> for key in [pysvn.wc_status_kind.added,
>             pysvn.wc_status_kind.replaced,
>             pysvn.wc_status_kind.unversioned]:
>     print( '2 key', key, key in wc_status_kind_map, cmp( key, 
> pysvn.wc_status_kind.unversioned ), hash( key ) )
>     try:
>         print( '2 lookup', wc_status_kind_map[ key ] )
>     except:
>         print( '2 failed' )

> 2 key added False 1 -586300914
> 2 failed
> 2 key replaced False 1 -586300911
> 2 failed
> 2 key unversioned False 0 -586300916
> 2 failed

Given that p.we.x seems to always return the same object (since the 
hashes, which which appear to be ids, are the same), an __eq__ method 
(which gets called in preference to __cmp__), possibly inherited, that 
always return False is the only thing I can think of.  (Hence Martin's 
question, I presume).  I have no idea, however, how porting could make 
that happen.

tjr


From steve at holdenweb.com  Mon Oct  6 08:43:45 2008
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 06 Oct 2008 02:43:45 -0400
Subject: [Python-Dev] [Fwd: In Python 2.6, bytes is str]
Message-ID: <48E9B3A1.7090704@holdenweb.com>

This does make it look rather as though bytes == str was a decision
whose consequences weren't fully appreciated before implementation.

Was this horror anticipated?

regards
 Steve

-------- Original Message --------
Subject: In Python 2.6, bytes is str
Date: Sun, 05 Oct 2008 22:30:17 -0700
From: Bryan Olson <fakeaddress at nowhere.org>
Organization: at&t http://my.att.net/
To: python-list at python.org
Newsgroups: gmane.comp.python.general


Python 3 has the 'bytes' type, which the string type I've long wanted in
various languages. Among other advantages, it is immutable, and
therefore bytes objects can be dict keys. There's a mutable version too,
called "bytearray".

In Python 2.6, the name 'bytes' is defined, and bound to str. The 2.6
assignment presents some pitfalls. Be aware.

Consider constructing a bytes object as:

    b = bytes([68, 255, 0])

In Python 3.x, len(b) will be 3, which feels right.

In Python 2.6, len(b) will be  12, because b is the str, '[68, 255, 0]'.


I'm thinking I should just avoid using 'bytes' in Python 2.6. If there's
 another Python release between 2.6 and 3.gold, I'd advocate removing
the pre-defined 'bytes', or maybe defining it as something else.


-- 
--Bryan
--
http://mail.python.org/mailman/listinfo/python-list


-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From brett at python.org  Mon Oct  6 08:55:12 2008
From: brett at python.org (Brett Cannon)
Date: Sun, 5 Oct 2008 23:55:12 -0700
Subject: [Python-Dev] [Fwd: In Python 2.6, bytes is str]
In-Reply-To: <48E9B3A1.7090704@holdenweb.com>
References: <48E9B3A1.7090704@holdenweb.com>
Message-ID: <bbaeab100810052355q37f58924ue37cd6c7098afc66@mail.gmail.com>

On Sun, Oct 5, 2008 at 11:43 PM, Steve Holden <steve at holdenweb.com> wrote:
> This does make it look rather as though bytes == str was a decision
> whose consequences weren't fully appreciated before implementation.
>

Well, you could say that about almost any change.

> Was this horror anticipated?

I didn't anticipate it.

>
> regards
>  Steve
>
> -------- Original Message --------
> Subject: In Python 2.6, bytes is str
> Date: Sun, 05 Oct 2008 22:30:17 -0700
> From: Bryan Olson <fakeaddress at nowhere.org>
> Organization: at&t http://my.att.net/
> To: python-list at python.org
> Newsgroups: gmane.comp.python.general
>
>
> Python 3 has the 'bytes' type, which the string type I've long wanted in
> various languages. Among other advantages, it is immutable, and
> therefore bytes objects can be dict keys. There's a mutable version too,
> called "bytearray".
>
> In Python 2.6, the name 'bytes' is defined, and bound to str. The 2.6
> assignment presents some pitfalls. Be aware.
>
> Consider constructing a bytes object as:
>
>    b = bytes([68, 255, 0])
>
> In Python 3.x, len(b) will be 3, which feels right.
>
> In Python 2.6, len(b) will be  12, because b is the str, '[68, 255, 0]'.
>
>
> I'm thinking I should just avoid using 'bytes' in Python 2.6. If there's
>  another Python release between 2.6 and 3.gold, I'd advocate removing
> the pre-defined 'bytes', or maybe defining it as something else.

I think it needs to be made clear that bytes is there for type
compatibility (e.g., ``isinstance(ob, bytes)``). And the bytes
literals work how you would expect.

What I would like to know if there are any other gotchas beyond the
constructor as all of the other uses act as you would expect.

-Brett

From skip at pobox.com  Mon Oct  6 12:47:46 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 6 Oct 2008 05:47:46 -0500
Subject: [Python-Dev] In Python 2.6, bytes is str (fwd)
Message-ID: <18665.60626.52400.93567@montanaro-dyndns-org.local>


Saw this on python-list.  I suspect it's a known issue, but just in case, I
thought I'd pass it along.

Skip

-------------- next part --------------
An embedded message was scrubbed...
From: Bryan Olson <fakeaddress at nowhere.org>
Subject: In Python 2.6, bytes is str
Date: Sun, 05 Oct 2008 22:30:17 -0700
Size: 5473
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081006/55a23e7d/attachment.eml>

From ncoghlan at gmail.com  Mon Oct  6 13:17:34 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 06 Oct 2008 21:17:34 +1000
Subject: [Python-Dev] [Fwd: In Python 2.6, bytes is str]
In-Reply-To: <bbaeab100810052355q37f58924ue37cd6c7098afc66@mail.gmail.com>
References: <48E9B3A1.7090704@holdenweb.com>
	<bbaeab100810052355q37f58924ue37cd6c7098afc66@mail.gmail.com>
Message-ID: <48E9F3CE.1010100@gmail.com>

Brett Cannon wrote:
> What I would like to know if there are any other gotchas beyond the
> constructor as all of the other uses act as you would expect.

Mainly that indexing returns one character substrings instead of ints.

The bytes type is there as an indicator of what bytes literals represent
in 2.6 - yes, that's not the same thing as what they represent in 3.0,
and no, you don't magically get the benefits of byte/str separation in
the 2.x series.

Maybe that could be caveat'ed better somewhere in the docs, but it does
reflect the reality of the situation.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From facundobatista at gmail.com  Mon Oct  6 14:08:30 2008
From: facundobatista at gmail.com (Facundo Batista)
Date: Mon, 6 Oct 2008 09:08:30 -0300
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <48E8EDA0.5060501@v.loewis.de>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<20081004101650.GA22969@amk.local> <48E7446B.70501@v.loewis.de>
	<loom.20081004T103221-543@post.gmane.org>
	<48E76007.2040408@v.loewis.de>
	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>
	<48E849B8.3010009@v.loewis.de>
	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
	<e04bdf310810050632o46bc6651sf3e4522c3e01b1ff@mail.gmail.com>
	<48E8EDA0.5060501@v.loewis.de>
Message-ID: <e04bdf310810060508g5590aecaibed2853e6e717585@mail.gmail.com>

2008/10/5 "Martin v. L?wis" <martin at v.loewis.de>:

>>> foobar-1.0-py2.6.tar.gz
>>> foobar-1.0-py3.0.tar.gz
>>
>> More likely, in this way:
>>
>> foobar-1.0-py2.tar.gz
>> foobar-1.0-py3.tar.gz
>
> How do you implement this in distutils? People probably won't rename
> the files from how sdist names them. So it's more likely that they
> come up with things like

Sorry, I was not advising that structure, just mentioning that if
you'll follow it, don't include the minor version, just the major one.

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From solipsis at pitrou.net  Mon Oct  6 14:50:10 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 6 Oct 2008 12:50:10 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?3=2E1_focus_=28was_Re=3A_for_=5F=5Ffuture?=
	=?utf-8?q?=5F=5F_import_planning=29?=
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>		<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>		<gc76uv$rr9$1@ger.gmane.org>
	<20081004101650.GA22969@amk.local>		<48E7446B.70501@v.loewis.de>	<loom.20081004T103221-543@post.gmane.org>		<48E76007.2040408@v.loewis.de>	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>	<48E849B8.3010009@v.loewis.de>	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
	<48E88359.90700@v.loewis.de> <48E8AAE4.6060104@holdenweb.com>
	<48E8EC1A.9000304@v.loewis.de>
Message-ID: <loom.20081006T124905-343@post.gmane.org>

Martin v. L?wis <martin <at> v.loewis.de> writes:
> 
> Although it would be possible, I think it's not appropriate.

I also think it's inappropriate. We want people to know about the existence of
Python 3, and the best for that is to have Python 3-related information in the
standard PyPI site where people look for Python 2.x packages.

Regards

Antoine.



From p.f.moore at gmail.com  Mon Oct  6 17:28:09 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 6 Oct 2008 16:28:09 +0100
Subject: [Python-Dev] 3.1 focus (was Re: for __future__ import planning)
In-Reply-To: <loom.20081006T124905-343@post.gmane.org>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<loom.20081004T103221-543@post.gmane.org>
	<48E76007.2040408@v.loewis.de>
	<94bdd2610810041653i6a98eec2me51c236dae586265@mail.gmail.com>
	<48E849B8.3010009@v.loewis.de>
	<0B5B5078-5CE2-4F79-A018-BE004B8A9DB0@bud.ca>
	<48E88359.90700@v.loewis.de> <48E8AAE4.6060104@holdenweb.com>
	<48E8EC1A.9000304@v.loewis.de>
	<loom.20081006T124905-343@post.gmane.org>
Message-ID: <79990c6b0810060828v3443f553mdf94e5f2f770ee96@mail.gmail.com>

2008/10/6 Antoine Pitrou <solipsis at pitrou.net>:
> Martin v. L?wis <martin <at> v.loewis.de> writes:
>>
>> Although it would be possible, I think it's not appropriate.
>
> I also think it's inappropriate. We want people to know about the existence of
> Python 3, and the best for that is to have Python 3-related information in the
> standard PyPI site where people look for Python 2.x packages.

Agreed. Although there are significant differences to consider, I
think the last thing we want to do is to encourage the impression that
Python 3 is a different language from Python 2.

Paul.

From nnorwitz at gmail.com  Mon Oct  6 19:58:16 2008
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Mon, 6 Oct 2008 10:58:16 -0700
Subject: [Python-Dev] status of 2.5
Message-ID: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>

Should we plan to put out a final 2.5 release?  If so, should we
continue to backport fixes (like Martin's removal of Alpha in
setup.py)?  My preference is that we do put out a final 2.5 that has
all accumulated bug fixes.  Then close the branch.  That way if we put
out a security release for 2.5, it will be clean and easy.

n

From skip at pobox.com  Mon Oct  6 21:28:22 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 6 Oct 2008 14:28:22 -0500
Subject: [Python-Dev] status of 2.5
In-Reply-To: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>
References: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>
Message-ID: <18666.26326.600528.585944@montanaro-dyndns-org.local>


    Neal> Should we plan to put out a final 2.5 release?

I'm probably a bad person to ask.  At work we are still using 2.4. :-/

Skip

From python at rcn.com  Mon Oct  6 21:53:01 2008
From: python at rcn.com (Raymond Hettinger)
Date: Mon, 6 Oct 2008 12:53:01 -0700
Subject: [Python-Dev] status of 2.5
References: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>
Message-ID: <66FBEF6AD16E443AB7436E87485DEADD@RaymondLaptop1>

[Neal Norwitz]
> Should we plan to put out a final 2.5 release?  If so, should we
> continue to backport fixes (like Martin's removal of Alpha in
> setup.py)?  My preference is that we do put out a final 2.5 that has
> all accumulated bug fixes.  Then close the branch.  That way if we put
> out a security release for 2.5, it will be clean and easy.

The 2.6/3.0 development process was so disruptive that I doubt
that 2.5 received adequate attention for bug fixes.  Why not wait
two or three months for the dust to settle?


Raymond

From martin at v.loewis.de  Mon Oct  6 21:54:41 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 06 Oct 2008 21:54:41 +0200
Subject: [Python-Dev] status of 2.5
In-Reply-To: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>
References: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>
Message-ID: <48EA6D01.3090902@v.loewis.de>

Neal Norwitz wrote:
> Should we plan to put out a final 2.5 release?  If so, should we
> continue to backport fixes (like Martin's removal of Alpha in
> setup.py)?  My preference is that we do put out a final 2.5 that has
> all accumulated bug fixes.  Then close the branch.  That way if we put
> out a security release for 2.5, it will be clean and easy.

That is my plan also. I would like to release 2.5.2 two weeks after
Python 3.0, or on November 1st, whatever happens later (and it seems
that Python-3-plus-two-weeks happens later).

So one week after Python 3, there would be a release candidate, and
two weeks, the final release.

Simultaneously, I would also release 2.4.6.

If people think 2.5.2 should be released earlier than that, please
let me know.

Regards,
Martin

From martin at v.loewis.de  Mon Oct  6 22:38:39 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 06 Oct 2008 22:38:39 +0200
Subject: [Python-Dev] status of 2.5
In-Reply-To: <66FBEF6AD16E443AB7436E87485DEADD@RaymondLaptop1>
References: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>
	<66FBEF6AD16E443AB7436E87485DEADD@RaymondLaptop1>
Message-ID: <48EA774F.3090101@v.loewis.de>

> The 2.6/3.0 development process was so disruptive that I doubt
> that 2.5 received adequate attention for bug fixes.  Why not wait
> two or three months for the dust to settle?

Would it receive more attention in the next three months? Who
specifically would give it that attention?

Regards,
Martin

From amk at amk.ca  Mon Oct  6 23:28:50 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Mon, 6 Oct 2008 17:28:50 -0400
Subject: [Python-Dev] status of 2.5
In-Reply-To: <66FBEF6AD16E443AB7436E87485DEADD@RaymondLaptop1>
References: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>
	<66FBEF6AD16E443AB7436E87485DEADD@RaymondLaptop1>
Message-ID: <20081006212850.GA15996@amk-desktop.matrixgroup.net>

On Mon, Oct 06, 2008 at 12:53:01PM -0700, Raymond Hettinger wrote:
> The 2.6/3.0 development process was so disruptive that I doubt
> that 2.5 received adequate attention for bug fixes.  Why not wait
> two or three months for the dust to settle?

Can you please clarify your meaning? Do you mean that 

* we haven't been backporting fixes to 2.5?
* we should wait to see if any horrible problems are reported in 2.6?
* we need to look at the logs for commits that should be applied to 2.5?

--amk

From victor.stinner at haypocalc.com  Tue Oct  7 00:07:01 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 7 Oct 2008 00:07:01 +0200
Subject: [Python-Dev] I would like an Python account
In-Reply-To: <200809271732.19317.victor.stinner@haypocalc.com>
References: <200809271732.19317.victor.stinner@haypocalc.com>
Message-ID: <200810070007.01849.victor.stinner@haypocalc.com>

Hi,

Brett gave me permissions to edit the bug tracker, yeah!

> I also would like an SVN account

If I can't get an account, can anyone review my issues and/or commit the 
attached patches? Most contributions are recent but I'm waiting for some of 
these issues to be fixed before fixing other issues. Most issuses are 
specific to Python3.

=== Fix bugs ===

[Py3k] line number is wrong after encoding declaration (2008-03-18)
http://bugs.python.org/issue2384
 |--> one liner fix
 \--> patch + testcase

compile() cannot decode Latin-1 source encodings (2008-08-17)
http://bugs.python.org/issue3574
 \--> patch + multiple tests (eg. test_pep3120.py)

PyTraceBack_Print() doesn't respect # coding: xxx header (2008-09-26)
http://bugs.python.org/issue3975
 |--> complex patch: rewrite _Py_DisplaySourceLine() to support unicode 
 |--> patch
 \--> a testcase is included in the related issue: issue2384

IDLE: checksyntax() doesn't support Unicode? (2008-10-01)
http://bugs.python.org/issue4008
 |--> refactoring to reuse tokenizer.detect_encoding()
 \--> patch

missing newline in "Could not convert argument %s to string" 
error message (2008-09-30)
http://bugs.python.org/issue4004
 |--> trivial fix
 \--> patch


=== Improvments ===

Support bytes for os.exec*() (2008-10-03)
http://bugs.python.org/issue4035

Support bytes for subprocess.Popen() (2008-10-03)
http://bugs.python.org/issue4036


-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From python at rcn.com  Tue Oct  7 01:50:42 2008
From: python at rcn.com (Raymond Hettinger)
Date: Mon, 6 Oct 2008 16:50:42 -0700
Subject: [Python-Dev] status of 2.5
References: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com><66FBEF6AD16E443AB7436E87485DEADD@RaymondLaptop1>
	<20081006212850.GA15996@amk-desktop.matrixgroup.net>
Message-ID: <92CE8A4C2B9847E0A2E1B4B5AC36F1CE@RaymondLaptop1>

[A.M. Kuchling]
> Can you please clarify your meaning? Do you mean that 
> 
> * we haven't been backporting fixes to 2.5?

Unsure.  I surely have given zero attention to 2.5.

> * we should wait to see if any horrible problems are reported in 2.6?

Yes.  That would be a great idea.

> * we need to look at the logs for commits that should be applied to 2.5?

Yes.  That would be a great idea too.

ISTM, that 3.0 and 2.6 backports have consumed substantial developer attention over the last few months.

From tjreedy at udel.edu  Tue Oct  7 02:41:20 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 06 Oct 2008 20:41:20 -0400
Subject: [Python-Dev] status of 2.5
In-Reply-To: <48EA6D01.3090902@v.loewis.de>
References: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>
	<48EA6D01.3090902@v.loewis.de>
Message-ID: <gceb7f$8sh$1@ger.gmane.org>

Martin v. L?wis wrote:
> Neal Norwitz wrote:
>> Should we plan to put out a final 2.5 release?  If so, should we
>> continue to backport fixes (like Martin's removal of Alpha in
>> setup.py)?  My preference is that we do put out a final 2.5 that has
>> all accumulated bug fixes.  Then close the branch.  That way if we put
>> out a security release for 2.5, it will be clean and easy.

> That is my plan also. I would like to release 2.5.2 two weeks after
> Python 3.0, or on November 1st, whatever happens later (and it seems
> that Python-3-plus-two-weeks happens later).

You of course meant 2.5.3.


From barry at python.org  Tue Oct  7 02:47:57 2008
From: barry at python.org (Barry Warsaw)
Date: Mon, 6 Oct 2008 20:47:57 -0400
Subject: [Python-Dev] Proposed Python 3.0 schedule
Message-ID: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

So, we need to come up with a new release schedule for Python 3.0.  My  
suggestion:

15-Oct-2008 3.0 beta 4
05-Nov-2008 3.0 rc 2
19-Nov-2008 3.0 rc 3
03-Dec-2008 3.0 final

Given what still needs to be done, is this a reasonable schedule?  Do  
we need two more betas?

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOqxvnEjvBPtnXfVAQIR5QP/coSi2ltsZSpE2dyUg7Y35QcSk/+4ZbGK
zF0AgLaOkGs+DFnxRH9vy9kN3JaEkp1MhEpDjkomE7kNpnJB7bWotTrHI67HD9ma
ZDqqmaCc02IeUtLm7HuELvofjCgh+gryKWvRc71ErRHmn/YxMGr1OcEirPpx4nZ9
DeDV0OeUtTE=
=RchU
-----END PGP SIGNATURE-----

From musiccomposition at gmail.com  Tue Oct  7 02:52:54 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Mon, 6 Oct 2008 19:52:54 -0500
Subject: [Python-Dev] [Python-3000] Proposed Python 3.0 schedule
In-Reply-To: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
Message-ID: <1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>

On Mon, Oct 6, 2008 at 7:47 PM, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> So, we need to come up with a new release schedule for Python 3.0.  My
> suggestion:
>
> 15-Oct-2008 3.0 beta 4
> 05-Nov-2008 3.0 rc 2
> 19-Nov-2008 3.0 rc 3
> 03-Dec-2008 3.0 final
>
> Given what still needs to be done, is this a reasonable schedule?  Do we
> need two more betas?

I'm not sure we do. Correct me if I'm wrong, but the "big ticket",
issue bytes/unicode filepaths, has been resolved. And looking at the
tracker, I only see 18 release blockers.



-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From python at rcn.com  Tue Oct  7 03:48:18 2008
From: python at rcn.com (Raymond Hettinger)
Date: Mon, 6 Oct 2008 18:48:18 -0700
Subject: [Python-Dev] Proposed Python 3.0 schedule
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
Message-ID: <040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>

[Barry Warsaw]
> So, we need to come up with a new release schedule for Python 3.0.  My  
> suggestion:
> 
> 15-Oct-2008 3.0 beta 4
> 05-Nov-2008 3.0 rc 2
> 19-Nov-2008 3.0 rc 3
> 03-Dec-2008 3.0 final
> 
> Given what still needs to be done, is this a reasonable schedule?  Do  
> we need two more betas?

Yes to both questions.

I'm seeing that people are just starting to download and play with 3.0.
I expect that we'll start getting more feedback on conversion issues,
the C API, screwy interactions with operating systems, bytes/text issues,
unanticipated interactions with other tools, etc.  Each user will stress
it in new ways and perhaps reveal a bunch of little integration issues
and documentation issues.  Those little fixups way go a long way toward
establishing a good first impression and reputation for 3.0 from the outset.


Raymond



From barry at python.org  Tue Oct  7 04:13:06 2008
From: barry at python.org (Barry Warsaw)
Date: Mon, 6 Oct 2008 22:13:06 -0400
Subject: [Python-Dev] Proposed Python 3.0 schedule
In-Reply-To: <040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
Message-ID: <67693931-29C5-4FD7-8E83-D97F892F3BE3@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 6, 2008, at 9:48 PM, Raymond Hettinger wrote:

> [Barry Warsaw]
>> So, we need to come up with a new release schedule for Python 3.0.   
>> My  suggestion:
>> 15-Oct-2008 3.0 beta 4
>> 05-Nov-2008 3.0 rc 2
>> 19-Nov-2008 3.0 rc 3
>> 03-Dec-2008 3.0 final
>> Given what still needs to be done, is this a reasonable schedule?   
>> Do  we need two more betas?
>
> Yes to both questions.

I think that's contradictory :).  If we need two betas, then 05-Nov  
becomes beta 5, 19-Nov is rc 2.  If we don't need another rc then we  
can still do a final release on 03-Dec, otherwise we probably go 2  
weeks later.  I don't want to go much later than that though because  
then we get into the holiday season.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOrFs3EjvBPtnXfVAQJceQP/QJN7oLM4nG+iXmgdb0NmKzOzaE3J89sQ
UWZnc/hp618QNH4JWC8v2bYApFu+iVg3pcv1Lnmhuql6mOuDhSuKKJVA5jTdR7U2
2enhAEY2DXtmav/29nn2Fy6PYcWJy9pE2xBsbBW8qXc6tYww0iEBsz9SU68jPzPk
x5LFC5NqmXo=
=Kyr4
-----END PGP SIGNATURE-----

From foom at fuhm.net  Tue Oct  7 05:22:09 2008
From: foom at fuhm.net (James Y Knight)
Date: Mon, 6 Oct 2008 23:22:09 -0400
Subject: [Python-Dev] [Python-3000] Proposed Python 3.0 schedule
In-Reply-To: <1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
Message-ID: <9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>

On Oct 6, 2008, at 8:52 PM, Benjamin Peterson wrote:
> I'm not sure we do. Correct me if I'm wrong, but the "big ticket",
> issue bytes/unicode filepaths, has been resolved. And looking at the
> tracker, I only see 18 release blockers.


Well, if you mean that the resolution decided upon is to "simply"  
allow access to all system APIs using either byte or unicode strings,  
then it seems to me that there's a rather large amount of work left to  
do...

Here's some I found from a few minutes of futzing around with r66821  
of py3k on Linux.

  - Having os.getcwdb isn't much use when you can't even run python in  
the first place when the current directory has "bad" bytes in it.

Currently Python outputs:
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Fatal Python error: Py_Initialize: can't initialize sys standard streams
ImportError: No module named encodings.utf_8
Aborted

  - I'd think "find . -type f -print0 | xargs -0 python -c 'pass'"  
ought to work (with files with "bad" bytes being returned by find),  
which means that Python shouldn't blow up and refuse to start when  
there's a non-properly-encoding argv ("Could not convert argument 1 to  
string" and exiting isn't appropriate behavior).

  - Of course, just being able to start the interpreter isn't quite  
enough: you'll want to be able to access that argument list too,  
somehow (add sys.argvb?).

  - And then, getopt and optparse modules should work on bytestring  
vectors, so that you can use sys.argvb without writing your own  
argument parser. They don't currently.

  - There's no os.environb for bytewise access to the environment.  
Seems important.

  - Isn't it a potential security issue that " 'WHATEVER' in  
os.environ" can return False if WHATEVER had some "bad" bytes in it,  
but spawning a subprocess actually will include WHATEVER in the  
subprocess's environment? Actually, even better: the behavior depends  
on whether you use subprocess.call('foo') or subprocess.call('foo',  
os.environ). The first passes through the "bad" environment variables,  
while the second does not. A bit surprising, perhaps.

  - Shouldn't this work?
   subprocess.call(b'/bin/echo')
Currently raises an exception:
AttributeError: 'int' object has no attribute 'rfind'

  - I suppose sys.path should handle bytestrings on the path, and  
should be populated using the bytes-version of os.environ so that  
PYTHONPATH gets read in properly. Which of course implies that all the  
importers need to handle byte filenames.

  - zipfile.ZipFile(b'whatever.zip') doesn't work.

  - zipfile decodes/encodes the filenames inside the zip file to  
unicode, so thus can only handle correctly encoded filenames.

I'm sure there's even more APIs dealing with pathnames, command line  
arguments, or environment variables that ought to be able to handle  
both bytes and strings, that currently don't.

James

From martin at v.loewis.de  Tue Oct  7 09:16:20 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Oct 2008 09:16:20 +0200
Subject: [Python-Dev] status of 2.5
In-Reply-To: <92CE8A4C2B9847E0A2E1B4B5AC36F1CE@RaymondLaptop1>
References: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com><66FBEF6AD16E443AB7436E87485DEADD@RaymondLaptop1>	<20081006212850.GA15996@amk-desktop.matrixgroup.net>
	<92CE8A4C2B9847E0A2E1B4B5AC36F1CE@RaymondLaptop1>
Message-ID: <48EB0CC4.4050902@v.loewis.de>

> ISTM, that 3.0 and 2.6 backports have consumed substantial developer
> attention over the last few months.

Sure - but who is going to sit down and study the commit logs, to do
backporting? Unless someone specifically volunteers to do that (with
a specific timeline when he will start, and when he will complete),
I see little chance that that any additional backporting will happen.

I will send an announcement asking people to propose missing backports.

Regards,
Martin

From martin at v.loewis.de  Tue Oct  7 09:27:24 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Oct 2008 09:27:24 +0200
Subject: [Python-Dev] Python 2.5.3: call for patches
Message-ID: <48EB0F5C.602@v.loewis.de>

Within a few weeks, we will release Python 2.5.3. This will be the last
bug fix release of Python 2.5, afterwards, future releases of 2.5 will
only include security fixes, and no binaries (for Windows or OSX) will
be provided anymore (from python.org).

In principle, the release will include all changes that are already on
the release25-maint branch in subversion [1]. If you think that specific
changes should be considered, please create an issue in the bug tracker
[2], and label it with the 2.5.3 version. Backports of changes that
are already released in Python 2.6 but may apply to 2.5 are of
particular interest.

Regards,
Martin

[1] http://svn.python.org/projects/python/branches/release25-maint/
[2] http://bugs.python.org/

From martin at v.loewis.de  Tue Oct  7 09:47:20 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Oct 2008 09:47:20 +0200
Subject: [Python-Dev] [Python-3000] Proposed Python 3.0 schedule
In-Reply-To: <9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
Message-ID: <48EB1408.1030007@v.loewis.de>

> Here's some I found from a few minutes of futzing around with r66821 of
> py3k on Linux.
> 
>  - Having os.getcwdb isn't much use when you can't even run python in
> the first place when the current directory has "bad" bytes in it.

That's not true: it *is* of much use. Python will live in /usr/bin,
which has a nicely-decodable path.

> Currently Python outputs:
> Could not find platform independent libraries <prefix>
> Could not find platform dependent libraries <exec_prefix>
> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
> Fatal Python error: Py_Initialize: can't initialize sys standard streams
> ImportError: No module named encodings.utf_8
> Aborted

I can't reproduce that. This happens (for me) when Python lives in
a directory that has an undecodable path - not when the current
directory is undecodable.

>  - I'd think "find . -type f -print0 | xargs -0 python -c 'pass'" ought
> to work (with files with "bad" bytes being returned by find), which
> means that Python shouldn't blow up and refuse to start when there's a
> non-properly-encoding argv ("Could not convert argument 1 to string" and
> exiting isn't appropriate behavior).

Contributions are welcome. *Of course* can you access these files with
POSIX API. However, Python's path handling can't.

See above why I don't consider this as a serious bug, on Unix.

>  - Of course, just being able to start the interpreter isn't quite
> enough: you'll want to be able to access that argument list too, somehow
> (add sys.argvb?).

Perhaps. However, I don't see the need to be able to do so in Python
3.0.

>  - And then, getopt and optparse modules should work on bytestring
> vectors, so that you can use sys.argvb without writing your own argument
> parser. They don't currently.

And I hope they never will. Using bytes to represent this stuff will
just bring back the 2.x status, so some other solution must be found -
for 3.1 (or 3.2).

>  - There's no os.environb for bytewise access to the environment. Seems
> important.

Not to me. I don't have environment variables with non-ASCII characters
in them, and I think few other people do.

> I'm sure there's even more APIs dealing with pathnames, command line
> arguments, or environment variables that ought to be able to handle both
> bytes and strings, that currently don't.

Please, no.

Regards,
Martin

From victor.stinner at haypocalc.com  Tue Oct  7 11:30:35 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 7 Oct 2008 11:30:35 +0200
Subject: [Python-Dev] Proposed Python 3.0 schedule
In-Reply-To: <9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
Message-ID: <200810071130.35729.victor.stinner@haypocalc.com>

Hi,

First of all, please read my document:
http://wiki.python.org/moin/Python3UnicodeDecodeError

I moved the document to a public wiki to allow anyone to edit it!

Le Tuesday 07 October 2008 05:22:09 James Y Knight, vous avez ?crit?:
> On Oct 6, 2008, at 8:52 PM, Benjamin Peterson wrote:
> > I'm not sure we do. Correct me if I'm wrong, but the "big ticket",
> > issue bytes/unicode filepaths, has been resolved.

Python3 now accepts bytes for os.listdir(), open() (io.open()), os.unlink(), 
os.path.*(), etc. But it's not enough to say that Python3 can use bytes 
everywhere. It would take months or *years* to fix all issues related to 
bytes and unicode. Remember, this task started in 2000 with Python *2.0* 
(creation of the unicode type).

> Well, if you mean that the resolution decided upon is to "simply"
> allow access to all system APIs using either byte or unicode strings,
> then it seems to me that there's a rather large amount of work left to
> do...

If you know a problem, open a ticket and propose a solution. It's not possible 
to list all new problems since we don't know them yet :-)

>   - Having os.getcwdb isn't much use when you can't even run python in
> the first place when the current directory has "bad" bytes in it.

My python3.0 works correctly in a directory with an invalid name. What is your 
OS / locale / Python version? Please create a ticket if needed.

>   - I'd think "find . -type f -print0 | xargs -0 python -c 'pass'"
> ought to work (with files with "bad" bytes being returned by find),

First, fix your home directory :-) There are good tools (convmv?) to fix 
invalid filenames.

> which means that Python shouldn't blow up and refuse to start when
> there's a non-properly-encoding argv ("Could not convert argument 1 to
> string" and exiting isn't appropriate behavior)

Why not? It's a good idea to break compatibility to refuse invalid bytes 
sequences. You can still uses the command line, an input file or a GUI to 
read raw bytes sequences.

>   - Of course, just being able to start the interpreter isn't quite
> enough: you'll want to be able to access that argument list too,
> somehow (add sys.argvb?).

If we create sys.argvb, what shoul be done if sys.argv creation failed? 
sys.argv would be empty or unset? Or some values would be removed (and so 
argv[2] is argv[1])? I think that many (a lot of) programs suppose that 
sys.argv exists and "is valid". If you introduce a special case (sometimes, 
sys.argv doesn't exist or is truncated !?), it will introduce new issues.

>   - There's no os.environb for bytewise access to the environment.
> Seems important.

It would be strange if you can put a variable in bytes to os.environb whereas 
os.environ would not get the key. I know two major usages of the environment:
 (1) read a variable in Python
 (2) put a variable for a child process 

(1) can be done with os.getenv() and returns None if the variable (key or 
value) is an invalid bytes sequence.

(2) can be done with subprocess.Popen(). subprocess doesn't support bytes yet 
but I wrote patches: #4035 and #4036.

>   - Isn't it a potential security issue that " 'WHATEVER' in
> os.environ" can return False if WHATEVER had some "bad" bytes in it,
> but spawning a subprocess actually will include WHATEVER in the
> subprocess's environment?

Yes. Python should remove the key while creating os.environ.

> - Shouldn't this work? subprocess.call(b'/bin/echo')

Yes. Most programs (at least on Linux and Mac) supports bytes and so you 
should be able use bytes arguments in their command lines, see issues #4035 
and #4036.

>   - I suppose sys.path should handle bytestrings on the path, and
> should be populated using the bytes-version of os.environ so that
> PYTHONPATH gets read in properly. Which of course implies that all the
> importers need to handle byte filenames.

If your file system is broken, rename your directory but don't introduce a 
special case for sys.path. 

>   - zipfile.ZipFile(b'whatever.zip') doesn't work.

Since zipfile uses bytes in its file structure, zipfile should accept bytes. 
But the right question is: should this issue block Python3 or can we wait for 
Python 3.1 (maybe 3.0.1)?

--

People wants to try the new Python version! Python3 introduces new amazing 
features like "keyword only arguments". The bytes/unicode problem is old and 
only affects broken systems

Windows (90% of the computers in the world?) only uses characters for the 
filenames, environment and command line. Mac and Linux use UTF-8 most of the 
time, and slowly everything speaks UTF-8! Python3 should not be delayed 
because of this problem.

About the initial barry's question: why Python3 is delayed until december? 
There are too much open issues?

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From hrvoje.niksic at avl.com  Tue Oct  7 13:01:24 2008
From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=)
Date: Tue, 07 Oct 2008 13:01:24 +0200
Subject: [Python-Dev] Proposed Python 3.0 schedule
In-Reply-To: <200810071130.35729.victor.stinner@haypocalc.com>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
	<200810071130.35729.victor.stinner@haypocalc.com>
Message-ID: <1223377284.2598.96.camel@localhost>

On Tue, 2008-10-07 at 11:30 +0200, Victor Stinner wrote:
> >   - I'd think "find . -type f -print0 | xargs -0 python -c 'pass'"
> > ought to work (with files with "bad" bytes being returned by find),
> 
> First, fix your home directory :-) There are good tools (convmv?) to fix 
> invalid filenames.

Fixing the home directory doesn't help in the long run because files
with non-UTF-8 file names on a nominally UTF-8 system are not that
exceptional, they crop up all over the place in non-ASCII countries.
One can obtain them simply by copying stuff from a DVD someone else
burned, by downloading a Japanese-released torrent, or by copying files
from a shared hard drive.

> > which means that Python shouldn't blow up and refuse to start when
> > there's a non-properly-encoding argv ("Could not convert argument 1 to
> > string" and exiting isn't appropriate behavior)
> 
> Why not? It's a good idea to break compatibility to refuse invalid bytes 
> sequences. You can still uses the command line, an input file or a GUI to 
> read raw bytes sequences.

Maybe I am misunderstanding you, but if python blows up at startup when
unable to encode argv to Unicode, then how can you still use the command
line to access the actual file name?


From solipsis at pitrou.net  Tue Oct  7 13:31:31 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 7 Oct 2008 11:31:31 +0000 (UTC)
Subject: [Python-Dev] status of 2.5
References: <ee2a432c0810061058jf28bb13u464bf0e9dfd7e6b2@mail.gmail.com>
	<66FBEF6AD16E443AB7436E87485DEADD@RaymondLaptop1>
Message-ID: <loom.20081007T112206-26@post.gmane.org>

Raymond Hettinger <python <at> rcn.com> writes:
> 
> The 2.6/3.0 development process was so disruptive that I doubt
> that 2.5 received adequate attention for bug fixes.  Why not wait
> two or three months for the dust to settle?

I know that I, and a couple of others, have tried to backport "important" bug
fixes (by that I mean security fixes, crashers, memory leaks, as well as glaring
behaviour problems) to 2.5 when there was no risk to reduce stability or
compatibility.

It's also true that 2.6/3.0 were so disruptive that the reason why few things
were backported is simply that few things could be backported at all. Most
checkins were related to new functionality or modified behaviour, or new bugs
introduced by either of those.

Everyone can look at
http://code.python.org/hg/branches/release2.5-maint/shortlog to get an idea of
where 2.5 is. 2.5.2 was tagged ~7 months ago.

Regards

Antoine.



From facundobatista at gmail.com  Tue Oct  7 14:20:23 2008
From: facundobatista at gmail.com (Facundo Batista)
Date: Tue, 7 Oct 2008 09:20:23 -0300
Subject: [Python-Dev] [Python-3000]  Proposed Python 3.0 schedule
In-Reply-To: <040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
Message-ID: <e04bdf310810070520m7d3d3c49v6aae5ee81d61a59a@mail.gmail.com>

2008/10/6 Raymond Hettinger <python at rcn.com>:

>> 15-Oct-2008 3.0 beta 4
>> 05-Nov-2008 3.0 rc 2
>> 19-Nov-2008 3.0 rc 3
>> 03-Dec-2008 3.0 final
>>
>> Given what still needs to be done, is this a reasonable schedule?  Do  we
>> need two more betas?
>
> Yes to both questions.

I agree with you here.


> I'm seeing that people are just starting to download and play with 3.0.
> I expect that we'll start getting more feedback on conversion issues,
> the C API, screwy interactions with operating systems, bytes/text issues,
> unanticipated interactions with other tools, etc.  Each user will stress
> it in new ways and perhaps reveal a bunch of little integration issues
> and documentation issues.  Those little fixups way go a long way toward
> establishing a good first impression and reputation for 3.0 from the outset.

And maybe also here, but bounded.

I don't want to keep deferring 3.0 months and months, I prefer to have
a redesigned schedule now, and stick to it as much as possible, even
if the 3.0 version is not as robust as we would want.

Regards,

-- 
.    Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From mhammond at skippinet.com.au  Tue Oct  7 15:34:15 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Wed, 8 Oct 2008 00:34:15 +1100
Subject: [Python-Dev] [python-committers]  Proposed Python 3.0 schedule
In-Reply-To: <040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
Message-ID: <00e001c92881$68ba93c0$3a2fbb40$@com.au>

[when 2 mailing lists are not enough... :-]

> I'm seeing that people are just starting to download and play with 3.0.
> I expect that we'll start getting more feedback on conversion issues

+1 from this direction too.  pywin32 has recently started looking seriously
at py3k, and while things are in fairly good shape for us who are already
"on the bandwagon", cleaning up a few rough edges would help people's first
impressions - and as they say, you only get one chance at a good first
impression...

More specifically, I think 2to3 is shaping up well.  pywin32 is taking the
approach of "port where possible, but keep in py2x syntax and convert at
'setup.py' time" and this is working out fairly well (in fact, with just a
couple of helpers in pywintypes, I think we can support python 2.3 upwards).
I believe that many projects may well take a similar approach as it allows
them to defer a full commitment to py3k, so doing all we can to support this
might help with that first impression.  My experience is that this could
best be achieved by addressing the following issues before release:
 
* Almost all open 2to3 issues that aren't truly edge cases should be
resolved - if 2to3 doesn't work for people, they may be forced to (even
temporarily) "fork" their project, which will cause concern.  I'll note that
good recent progress is being made here, but its still worth mentioning...

* Better support for 2to3 in distutils (specifically, the support in
build_py is stale, plus 'build_scripts' and 'install_data' should convert
.py files to py3k syntax.)  An 'example' project that uses py2k syntax and
"just works" on py3k using this strategy might be useful here.

* A standard 'helper script' that allows people to use py3k to execute a
py2x syntax script by auto-converting the code.  I've a 10ish-line script
that uses lib2to3 plus exec() to achieve that result, but a helper in 2to3
for this would be nice.  For a concrete use-case, we want to keep our
distutils script in py2x syntax, but execute it via py3k.  Its very possible
this already exists and I've just missed it...

Either way, I'm fairly confident a pywin32 build for py3k will be available
in the next month or 2 (but as a result, I'm not really in a position to
help with the above for that period...)

Hopefully-helpfully,

Mark



From aahz at pythoncraft.com  Tue Oct  7 16:51:41 2008
From: aahz at pythoncraft.com (Aahz)
Date: Tue, 7 Oct 2008 07:51:41 -0700
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <48EB0F5C.602@v.loewis.de>
References: <48EB0F5C.602@v.loewis.de>
Message-ID: <20081007145140.GA11461@panix.com>

On Tue, Oct 07, 2008, "Martin v. L?wis" wrote:
> 
> In principle, the release will include all changes that are already on
> the release25-maint branch in subversion [1]. If you think that specific
> changes should be considered, please create an issue in the bug tracker
> [2], and label it with the 2.5.3 version. Backports of changes that
> are already released in Python 2.6 but may apply to 2.5 are of
> particular interest.

Just to emphasize this, "changes" means "bugfixes".  (I'm mentioning this
mainly because of the people who joined for 2.6/3.0.)  For more info,
see PEP6 about bugfix releases:
http://www.python.org/dev/peps/pep-0006/
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"...if I were on life-support, I'd rather have it run by a Gameboy than a
Windows box."  --Cliff Wells, comp.lang.python, 3/13/2002

From foom at fuhm.net  Tue Oct  7 17:51:19 2008
From: foom at fuhm.net (James Y Knight)
Date: Tue, 7 Oct 2008 11:51:19 -0400
Subject: [Python-Dev] [Python-3000] Proposed Python 3.0 schedule
In-Reply-To: <48EB1408.1030007@v.loewis.de>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
	<48EB1408.1030007@v.loewis.de>
Message-ID: <BDDCC54F-78D9-40DA-8F63-04DE4DCA0CEE@fuhm.net>

On Oct 7, 2008, at 3:47 AM, Martin v. L?wis wrote:
>> - Having os.getcwdb isn't much use when you can't even run python in
>> the first place when the current directory has "bad" bytes in it.
>
> That's not true: it *is* of much use. Python will live in /usr/bin,
> which has a nicely-decodable path.
>
>> Currently Python outputs:
>> Could not find platform independent libraries <prefix>
>> Could not find platform dependent libraries <exec_prefix>
>> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
>> Fatal Python error: Py_Initialize: can't initialize sys standard  
>> streams
>> ImportError: No module named encodings.utf_8
>> Aborted
>
> I can't reproduce that. This happens (for me) when Python lives in
> a directory that has an undecodable path - not when the current
> directory is undecodable.

Sorry about that: this test was indeed in error: I ran "../python"  
from an undecodeable current directory, rather than "/full/path/to/ 
python", or putting python on the PATH and running it as "python". The  
first does not work, but the other more common ways to start it do.

>>
>> I'm sure there's even more APIs dealing with pathnames, command line
>> arguments, or environment variables that ought to be able to handle  
>> both
>> bytes and strings, that currently don't.
>
> Please, no.

I completely and totally agree with your distate, it's rather gross to  
allow bytes-or-str for every API that touches anything like filenames/ 
argv/environ. That's why I was pushing for the reversible conversion  
to str...But if bytes-or-str is the solution that's been chosen for  
this issue, it ought to either be fully committed to and implemented,  
or at least fully recognized and documented as a half-baked solution.

Of course, if an reversible encoding into string solution is used  
instead, none of these things would need special treatment: they would  
all work already.

FWIW: Qt works fine with undecodeable filenames, and it too uses  
unicode strings everywhere in its API. I looked into what it does, and  
found that it uses your (Martin)'s original idea for solving this: it  
stores undecodeable bytes as characters from 0x10fe00 to 0x10feff  
(which is valid private-use codespace).  While that might not be  
ideally correct, since you lose those 256 PUA characters, even that is  
IMO better than pushing out bytes to every API, or worse, giving up  
and just having python unable to access files, as it is now.

See lines 3074: QString::toUtf8() and 3408: QString::fromUtf8()) of

http://www.google.com/codesearch?q=+show:o7fNK6SzOYs:NO-Bv-AR2rI:toIOngLf1V8&cs_p=http://ie.archive.ubuntu.com/trolltech/pub/qt/snapshots/qt-x11-opensource-src-4.4.0-snapshot-20070402.tar.bz2&cs_f=qt-x11-opensource-src-4.4.0-snapshot-20070402/src/corelib/tools/qstring.cpp

James

From martin at v.loewis.de  Tue Oct  7 21:40:21 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Oct 2008 21:40:21 +0200
Subject: [Python-Dev] [python-committers]  Proposed Python 3.0 schedule
In-Reply-To: <00e001c92881$68ba93c0$3a2fbb40$@com.au>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
	<00e001c92881$68ba93c0$3a2fbb40$@com.au>
Message-ID: <48EBBB25.70609@v.loewis.de>

> More specifically, I think 2to3 is shaping up well.  pywin32 is taking the
> approach of "port where possible, but keep in py2x syntax and convert at
> 'setup.py' time" and this is working out fairly well

I can't say how glad I am that you say that. It supports lib2to3 being a
proper library, despite the problems that this may cause in itself.

> * Better support for 2to3 in distutils (specifically, the support in
> build_py is stale, plus 'build_scripts' and 'install_data' should convert
> .py files to py3k syntax.)

Please do create a bug report for that. It sounds like it's easy to fix.

> An 'example' project that uses py2k syntax and
> "just works" on py3k using this strategy might be useful here.

Perhaps pywin32 :-?

I don't think a demo project would do much good, as it doesn't exercise
all the issues that may occur.

> * A standard 'helper script' that allows people to use py3k to execute a
> py2x syntax script by auto-converting the code.  I've a 10ish-line script
> that uses lib2to3 plus exec() to achieve that result, but a helper in 2to3
> for this would be nice.  For a concrete use-case, we want to keep our
> distutils script in py2x syntax, but execute it via py3k.  Its very possible
> this already exists and I've just missed it...

For the case of setup.py, I was hoping that it could be written in
compatible syntax even without needing conversion. That worked fine for
my Django port. Is that not the case for pywin32?

This specific issue might be out of scope for 3.x, IMO.

> Either way, I'm fairly confident a pywin32 build for py3k will be available
> in the next month or 2 (but as a result, I'm not really in a position to
> help with the above for that period...)

But please do file bug reports, preferably along with any patches to
distutils that you already have.

Regards,
Martin

From barry at barrys-emacs.org  Tue Oct  7 21:53:37 2008
From: barry at barrys-emacs.org (Barry Scott)
Date: Tue, 7 Oct 2008 20:53:37 +0100
Subject: [Python-Dev] porting pycxx and pysvn to python 3.0 hit a problem
In-Reply-To: <gcbcoe$irn$1@ger.gmane.org>
References: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>
	<gcbcoe$irn$1@ger.gmane.org>
Message-ID: <AB49706C-3378-43C7-A74E-F335276A7BCB@barrys-emacs.org>


On Oct 5, 2008, at 22:49, Terry Reedy wrote:

> Barry Scott wrote:
>
>> for key in [pysvn.wc_status_kind.added,
>>             pysvn.wc_status_kind.replaced,
>>             pysvn.wc_status_kind.unversioned]:
>>     print( '2 key', key, key in wc_status_kind_map, cmp( key,  
>> pysvn.wc_status_kind.unversioned ), hash( key ) )
>>     try:
>>         print( '2 lookup', wc_status_kind_map[ key ] )
>>     except:
>>         print( '2 failed' )
>
>> 2 key added False 1 -586300914
>> 2 failed
>> 2 key replaced False 1 -586300911
>> 2 failed
>> 2 key unversioned False 0 -586300916
>> 2 failed
>
> Given that p.we.x seems to always return the same object (since the  
> hashes, which which appear to be ids, are the same), an __eq__  
> method (which gets called in preference to __cmp__), possibly  
> inherited, that always return False is the only thing I can think  
> of.  (Hence Martin's question, I presume).  I have no idea,  
> however, how porting could make that happen.

The type is not derived so the __eq__ cannot be happening.

I guess I need to use gdb and figure out what is going on.

Barry


From martin at v.loewis.de  Tue Oct  7 21:56:13 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Oct 2008 21:56:13 +0200
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <20081007145140.GA11461@panix.com>
References: <48EB0F5C.602@v.loewis.de> <20081007145140.GA11461@panix.com>
Message-ID: <48EBBEDD.7010109@v.loewis.de>

> Just to emphasize this, "changes" means "bugfixes".  (I'm mentioning this
> mainly because of the people who joined for 2.6/3.0.)  For more info,
> see PEP6 about bugfix releases:
> http://www.python.org/dev/peps/pep-0006/

Thanks for clarifying this. For the last 2.5.x release in particular, we
will strictly enforce the "no new features" policy; users interested
in new features should switch to 2.6.

Regards,
Martin

From barry at barrys-emacs.org  Tue Oct  7 21:58:57 2008
From: barry at barrys-emacs.org (Barry Scott)
Date: Tue, 7 Oct 2008 20:58:57 +0100
Subject: [Python-Dev] porting pycxx and pysvn to python 3.0 hit a problem
In-Reply-To: <gcbcoe$irn$1@ger.gmane.org>
References: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>
	<gcbcoe$irn$1@ger.gmane.org>
Message-ID: <6B3E2AF6-B8E2-4310-86E4-1A3F9BA71D41@barrys-emacs.org>


On Oct 5, 2008, at 22:49, Terry Reedy wrote:

> Given that p.we.x seems to always return the same object (since the  
> hashes, which which appear to be ids, are the same), an __eq__  
> method (which gets called in preference to __cmp__), possibly  
> inherited, that always return False is the only thing I can think  
> of.  (Hence Martin's question, I presume).  I have no idea,  
> however, how porting could make that happen.

I see no reply from Martin. What was his question?

Barry

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081007/109aaa1a/attachment.htm>

From martin at v.loewis.de  Tue Oct  7 22:06:52 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Oct 2008 22:06:52 +0200
Subject: [Python-Dev] Python3UnicodeDecodeError (Was: Proposed Python 3.0
	schedule)
In-Reply-To: <200810071130.35729.victor.stinner@haypocalc.com>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
	<200810071130.35729.victor.stinner@haypocalc.com>
Message-ID: <48EBC15C.1050305@v.loewis.de>

> First of all, please read my document:
> http://wiki.python.org/moin/Python3UnicodeDecodeError

I have problems understanding that document. Is it supposed to
be a PEP (i.e. a proposal to enhance Python), or is it a description
of the status quo?

If it is a PEP, it should clearly separate status quo, specification,
and rationale (in any order that you find reasonable). It should also
have an "open issues" section, explicitly listing the questions that
haven't been resolved, and it should record objections to the proposal.

I think I would object to the specification (perhaps to the degree
of proposing a counter-PEP), but to do so, I first need a specification
to object to.

In terms of time-line, I think any such PEP is *clearly* out of scope
for Python 3.0. All the remaining issues should deferred to 3.1.

That the approach "we can use bytes in the file system API" was so
rushed into the code base is already unfortunate, but I can understand
the motivation - people want to write backup programs in Python.

If I take the text as if it was a specification, here are some of my
objections:

- Default encoding:
  a) seems irrelevant for the PEP. The default encoding doesn't nearly
     have the role anymore that it had in 2.x, and shouldn't have any
     effect on how file names are treated.
  b) I would propose that the notion of a default encoding is entirely
     eliminated from Python, along with sys.(get|set)defaultencoding
- argv and environ: are you suggesting that the behavior described
  in the PEP is desirable? I don't think it is (but I don't think it
  should change for 3.0, either, only for 3.1)

Regards,
Martin

From martin at v.loewis.de  Tue Oct  7 22:09:31 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Oct 2008 22:09:31 +0200
Subject: [Python-Dev] [Python-3000] Proposed Python 3.0 schedule
In-Reply-To: <BDDCC54F-78D9-40DA-8F63-04DE4DCA0CEE@fuhm.net>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>	<48EB1408.1030007@v.loewis.de>
	<BDDCC54F-78D9-40DA-8F63-04DE4DCA0CEE@fuhm.net>
Message-ID: <48EBC1FB.5090209@v.loewis.de>

James Y Knight wrote:
> or at least fully recognized and documented as a half-baked
> solution.

I would prefer that, leaving a full resolution to 3.1 (or perhaps 3.2).
If we wait long enough, the issue will disappear (a strategy that Sun
is apparently taking for Java :-)

Regards,
Martin

From martin at v.loewis.de  Tue Oct  7 22:11:19 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Oct 2008 22:11:19 +0200
Subject: [Python-Dev] porting pycxx and pysvn to python 3.0 hit a problem
In-Reply-To: <6B3E2AF6-B8E2-4310-86E4-1A3F9BA71D41@barrys-emacs.org>
References: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>	<gcbcoe$irn$1@ger.gmane.org>
	<6B3E2AF6-B8E2-4310-86E4-1A3F9BA71D41@barrys-emacs.org>
Message-ID: <48EBC267.6060303@v.loewis.de>

>> Given that p.we.x seems to always return the same object (since the
>> hashes, which which appear to be ids, are the same), an __eq__ method
>> (which gets called in preference to __cmp__), possibly inherited, that
>> always return False is the only thing I can think of.  (Hence Martin's
>> question, I presume).  I have no idea, however, how porting could make
>> that happen.
>>
> 
> I see no reply from Martin. What was his question?

Whether __eq__ was implemented for the specific type (and whether
that implementation was consistent with __hash__).

Regards,
Martin

From guido at python.org  Tue Oct  7 22:13:38 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 7 Oct 2008 13:13:38 -0700
Subject: [Python-Dev] [Fwd: In Python 2.6, bytes is str]
In-Reply-To: <48E9B3A1.7090704@holdenweb.com>
References: <48E9B3A1.7090704@holdenweb.com>
Message-ID: <ca471dc20810071313g5c102ff8w26a349e9aadd0aba@mail.gmail.com>

On Sun, Oct 5, 2008 at 11:43 PM, Steve Holden <steve at holdenweb.com> wrote:
> This does make it look rather as though bytes == str was a decision
> whose consequences weren't fully appreciated before implementation.
>
> Was this horror anticipated?

It was well understood that the bytes "type" in 2.6 and the bytes type
in 3.0 would behave quite different. Attempts to come up with a
separate bytes type that behaved more like its 3.0 counterpart were
doomed, because there just are too many places where the usage was
ambiguous. We should probably have written a PEP about this just to
prevent the discussion from being repeated all over again in this
thread.

The only two anticipated *reasonable* uses in 2.6 were the bytes
literal (b'abc') and tests for isinstance(x, bytes), which are flags
for 2to3 to keep these usages as bytes, not str.

I have no intention of rolling this back. It isn't compatible with
3.0, but then, 2.6 and 3.0 aren't in general meant to be compatible --
2.6 is a stepping stone, but that's not the same thing. It is
backwards compatible with prior versions because it is new in 2.6, so
it shouldn't break old code. I have only limited sympathy for people
who don't read documentation in this case.

--Guido

> -------- Original Message --------
> Subject: In Python 2.6, bytes is str
> Date: Sun, 05 Oct 2008 22:30:17 -0700
> From: Bryan Olson <fakeaddress at nowhere.org>
> Organization: at&t http://my.att.net/
> To: python-list at python.org
> Newsgroups: gmane.comp.python.general
>
>
> Python 3 has the 'bytes' type, which the string type I've long wanted in
> various languages. Among other advantages, it is immutable, and
> therefore bytes objects can be dict keys. There's a mutable version too,
> called "bytearray".
>
> In Python 2.6, the name 'bytes' is defined, and bound to str. The 2.6
> assignment presents some pitfalls. Be aware.
>
> Consider constructing a bytes object as:
>
>    b = bytes([68, 255, 0])
>
> In Python 3.x, len(b) will be 3, which feels right.
>
> In Python 2.6, len(b) will be  12, because b is the str, '[68, 255, 0]'.
>
>
> I'm thinking I should just avoid using 'bytes' in Python 2.6. If there's
>  another Python release between 2.6 and 3.gold, I'd advocate removing
> the pre-defined 'bytes', or maybe defining it as something else.
>
>
> --
> --Bryan
> --
> http://mail.python.org/mailman/listinfo/python-list
>
>
> --
> Steve Holden        +1 571 484 6266   +1 800 494 3119
> Holden Web LLC              http://www.holdenweb.com/
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From fdrake at acm.org  Tue Oct  7 22:18:09 2008
From: fdrake at acm.org (Fred Drake)
Date: Tue, 07 Oct 2008 16:18:09 -0400
Subject: [Python-Dev] Python3UnicodeDecodeError
In-Reply-To: <48EBC15C.1050305@v.loewis.de>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
	<200810071130.35729.victor.stinner@haypocalc.com>
	<48EBC15C.1050305@v.loewis.de>
Message-ID: <FCBFAB07-525D-4061-86C1-AE09BC426D11@acm.org>

On Oct 7, 2008, at 4:06 PM, Martin v. L?wis wrote:
>  b) I would propose that the notion of a default encoding is entirely
>     eliminated from Python, along with sys.(get|set)defaultencoding

+1


   -Fred

-- 
Fred Drake   <fdrake at acm.org>


From amk at amk.ca  Tue Oct  7 22:19:20 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Tue, 7 Oct 2008 16:19:20 -0400
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <48EB0F5C.602@v.loewis.de>
References: <48EB0F5C.602@v.loewis.de>
Message-ID: <20081007201920.GA14611@amk-desktop.matrixgroup.net>

On Tue, Oct 07, 2008 at 09:27:24AM +0200, "Martin v. L?wis" wrote:
> Within a few weeks, we will release Python 2.5.3. This will be the last
> bug fix release of Python 2.5, afterwards, future releases of 2.5 will
> only include security fixes, and no binaries (for Windows or OSX) will
> be provided anymore (from python.org).

I'm going to the library this evening, and can make my task looking
through the 2.5->2.6 log for candidates.  I won't do anything in
Roundup just yet, but I'll put the list in the wiki or post it here,
and then we can cut the list down further before creating any new
issues (or re-opening old ones) in Roundup.

--amk

From guido at python.org  Tue Oct  7 22:28:30 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 7 Oct 2008 13:28:30 -0700
Subject: [Python-Dev] [Python-3000] Proposed Python 3.0 schedule
In-Reply-To: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
Message-ID: <ca471dc20810071328t65ee07faw5e0672be8bdc3fea@mail.gmail.com>

On Mon, Oct 6, 2008 at 5:47 PM, Barry Warsaw <barry at python.org> wrote:
> So, we need to come up with a new release schedule for Python 3.0.  My
> suggestion:
>
> 15-Oct-2008 3.0 beta 4
> 05-Nov-2008 3.0 rc 2
> 19-Nov-2008 3.0 rc 3
> 03-Dec-2008 3.0 final
>
> Given what still needs to be done, is this a reasonable schedule?  Do we
> need two more betas?

I know I'm contradicting what I said earlier, but perhaps we should
just forget going back to beta and stick to ever-more-perfect release
candidates? In other worlds release candidates often contain tons of
imperfections (I believe I've seen this both for Java and Windows) and
the label "release candidate" more clearly encourages people to
download and play with it, which is what we need at this point! Then
the schedule would be something like

  15-Oct-2008 3.0 rc 2
  05-Nov-2008 3.0 rc 3
  19-Nov-2008 3.0 rc 4
  03-Dec-2008 3.0 final

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Oct  7 22:29:43 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 7 Oct 2008 13:29:43 -0700
Subject: [Python-Dev] [Python-3000]  Python3UnicodeDecodeError
In-Reply-To: <FCBFAB07-525D-4061-86C1-AE09BC426D11@acm.org>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
	<200810071130.35729.victor.stinner@haypocalc.com>
	<48EBC15C.1050305@v.loewis.de>
	<FCBFAB07-525D-4061-86C1-AE09BC426D11@acm.org>
Message-ID: <ca471dc20810071329i7ccd5651hcff7ebff344e1f30@mail.gmail.com>

On Tue, Oct 7, 2008 at 1:18 PM, Fred Drake <fdrake at acm.org> wrote:
> On Oct 7, 2008, at 4:06 PM, Martin v. L?wis wrote:
>>
>>  b) I would propose that the notion of a default encoding is entirely
>>    eliminated from Python, along with sys.(get|set)defaultencoding
>
> +1

I expect that the only effect of this change would be that the
filesystem encoding would become the de-facto default encoding for
other contexts as well.

Not that that is necessarily a bad thing...

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From rhamph at gmail.com  Tue Oct  7 22:45:11 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Tue, 7 Oct 2008 14:45:11 -0600
Subject: [Python-Dev] [Python-3000]  Proposed Python 3.0 schedule
In-Reply-To: <BDDCC54F-78D9-40DA-8F63-04DE4DCA0CEE@fuhm.net>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
	<48EB1408.1030007@v.loewis.de>
	<BDDCC54F-78D9-40DA-8F63-04DE4DCA0CEE@fuhm.net>
Message-ID: <aac2c7cb0810071345o2c20d3d1v42b45e79f3833c17@mail.gmail.com>

On Tue, Oct 7, 2008 at 9:51 AM, James Y Knight <foom at fuhm.net> wrote:
> On Oct 7, 2008, at 3:47 AM, Martin v. L?wis wrote:
>>>
>>> - Having os.getcwdb isn't much use when you can't even run python in
>>> the first place when the current directory has "bad" bytes in it.
>>
>> That's not true: it *is* of much use. Python will live in /usr/bin,
>> which has a nicely-decodable path.
>>
>>> Currently Python outputs:
>>> Could not find platform independent libraries <prefix>
>>> Could not find platform dependent libraries <exec_prefix>
>>> Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
>>> Fatal Python error: Py_Initialize: can't initialize sys standard streams
>>> ImportError: No module named encodings.utf_8
>>> Aborted
>>
>> I can't reproduce that. This happens (for me) when Python lives in
>> a directory that has an undecodable path - not when the current
>> directory is undecodable.
>
> Sorry about that: this test was indeed in error: I ran "../python" from an
> undecodeable current directory, rather than "/full/path/to/python", or
> putting python on the PATH and running it as "python". The first does not
> work, but the other more common ways to start it do.
>
>>>
>>> I'm sure there's even more APIs dealing with pathnames, command line
>>> arguments, or environment variables that ought to be able to handle both
>>> bytes and strings, that currently don't.
>>
>> Please, no.
>
> I completely and totally agree with your distate, it's rather gross to allow
> bytes-or-str for every API that touches anything like
> filenames/argv/environ. That's why I was pushing for the reversible
> conversion to str...But if bytes-or-str is the solution that's been chosen
> for this issue, it ought to either be fully committed to and implemented, or
> at least fully recognized and documented as a half-baked solution.
>
> Of course, if an reversible encoding into string solution is used instead,
> none of these things would need special treatment: they would all work
> already.
>
> FWIW: Qt works fine with undecodeable filenames, and it too uses unicode
> strings everywhere in its API. I looked into what it does, and found that it
> uses your (Martin)'s original idea for solving this: it stores undecodeable
> bytes as characters from 0x10fe00 to 0x10feff (which is valid private-use
> codespace).  While that might not be ideally correct, since you lose those
> 256 PUA characters, even that is IMO better than pushing out bytes to every
> API, or worse, giving up and just having python unable to access files, as
> it is now.
>
> See lines 3074: QString::toUtf8() and 3408: QString::fromUtf8()) of
>
> http://www.google.com/codesearch?q=+show:o7fNK6SzOYs:NO-Bv-AR2rI:toIOngLf1V8&cs_p=http://ie.archive.ubuntu.com/trolltech/pub/qt/snapshots/qt-x11-opensource-src-4.4.0-snapshot-20070402.tar.bz2&cs_f=qt-x11-opensource-src-4.4.0-snapshot-20070402/src/corelib/tools/qstring.cpp

So what does Qt do when given a file name already using those PUA?
Looks like they get passed through untouched when decoded, but will
get translated into invalid names upon encoding.  So you still have
file names you can't open, and you're incompatible with what other
libraries do.

The only thing going for Qt is that they seem specifically interested
in latin-1, rather than arbitrary bad names.  The latin-1 strings that
would correspond to the UTF-8 PUA used would include at least one
control character, as well as other unusual bits, so it's pretty
unlikely to encounter a real latin-1 file name like that.


-- 
Adam Olsen, aka Rhamphoryncus

From mal at egenix.com  Tue Oct  7 22:52:04 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 07 Oct 2008 22:52:04 +0200
Subject: [Python-Dev] Python3UnicodeDecodeError
In-Reply-To: <FCBFAB07-525D-4061-86C1-AE09BC426D11@acm.org>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>	<200810071130.35729.victor.stinner@haypocalc.com>	<48EBC15C.1050305@v.loewis.de>
	<FCBFAB07-525D-4061-86C1-AE09BC426D11@acm.org>
Message-ID: <48EBCBF4.7080200@egenix.com>

On 2008-10-07 22:18, Fred Drake wrote:
> On Oct 7, 2008, at 4:06 PM, Martin v. L?wis wrote:
>>  b) I would propose that the notion of a default encoding is entirely
>>     eliminated from Python, along with sys.(get|set)defaultencoding
> 
> +1

As already mentioned in my reply to Viktor: +1. It's not adjustable
anymore, so we might as well get rid off the sys module APIs.

The term "default encoding" itself still has some value in that it
is associated with the C API char* encoding used for PyUnicode
objects in Python 3.0.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 07 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From kristjan at ccpgames.com  Tue Oct  7 22:05:31 2008
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Tue, 7 Oct 2008 20:05:31 +0000
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <48EBBEDD.7010109@v.loewis.de>
References: <48EB0F5C.602@v.loewis.de> <20081007145140.GA11461@panix.com>
	<48EBBEDD.7010109@v.loewis.de>
Message-ID: <4E9372E6B2234D4F859320D896059A9510181248F0@exchis.ccp.ad.local>

Allow me to suggest that these get some attention:
http://bugs.python.org/issue3677
http://bugs.python.org/issue3367

Kristj?n

From ncoghlan at gmail.com  Tue Oct  7 23:47:30 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 08 Oct 2008 07:47:30 +1000
Subject: [Python-Dev] [Python-3000]  Proposed Python 3.0 schedule
In-Reply-To: <67693931-29C5-4FD7-8E83-D97F892F3BE3@python.org>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
	<67693931-29C5-4FD7-8E83-D97F892F3BE3@python.org>
Message-ID: <48EBD8F2.4090802@gmail.com>

Barry Warsaw wrote:
> On Oct 6, 2008, at 9:48 PM, Raymond Hettinger wrote:
> 
>> [Barry Warsaw]
>>> So, we need to come up with a new release schedule for Python 3.0. 
>>> My  suggestion:
>>> 15-Oct-2008 3.0 beta 4
>>> 05-Nov-2008 3.0 rc 2
>>> 19-Nov-2008 3.0 rc 3
>>> 03-Dec-2008 3.0 final
>>> Given what still needs to be done, is this a reasonable schedule? 
>>> Do  we need two more betas?
> 
>> Yes to both questions.
> 
> I think that's contradictory :).  If we need two betas, then 05-Nov
> becomes beta 5, 19-Nov is rc 2.  If we don't need another rc then we can
> still do a final release on 03-Dec, otherwise we probably go 2 weeks
> later.  I don't want to go much later than that though because then we
> get into the holiday season.

Do we need the full two weeks between rc's? Or is it too much of a pain
to cut releases 3 weeks in a row?

E.g. something like:

15-Oct-2008 3.0 beta 4
05-Nov-2008 3.0 beta 5
19-Nov-2008 3.0 rc 2
26-Nov-2008 3.0 rc 3 (if needed)
03-Dec-2008 3.0 final

Cheers,
Nick.

_______________________________________________
Python-3000 mailing list
Python-3000 at python.org
http://mail.python.org/mailman/listinfo/python-3000
Unsubscribe:
http://mail.python.org/mailman/options/python-3000/ncoghlan%40gmail.com

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From martin at v.loewis.de  Tue Oct  7 23:50:48 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 07 Oct 2008 23:50:48 +0200
Subject: [Python-Dev] [python-committers] [Python-3000] Proposed Python
 3.0 schedule
In-Reply-To: <48EBD8F2.4090802@gmail.com>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>	<67693931-29C5-4FD7-8E83-D97F892F3BE3@python.org>
	<48EBD8F2.4090802@gmail.com>
Message-ID: <48EBD9B8.4040102@v.loewis.de>

> Do we need the full two weeks between rc's?

If they are just other names for betas, yes. If they are true
release candidates (in the sense of "we really want to release this
as-is unless somebody tells us why this is a really bad idea"), then
no.

> Or is it too much of a pain
> to cut releases 3 weeks in a row?

It's a lot of effort, yes. Also for users, who will have barely
installed one release candidate when the next one comes out.

Regards,
Martin

From barry at barrys-emacs.org  Tue Oct  7 23:52:32 2008
From: barry at barrys-emacs.org (Barry Scott)
Date: Tue, 7 Oct 2008 22:52:32 +0100
Subject: [Python-Dev] porting pycxx and pysvn to python 3.0 hit a problem
In-Reply-To: <48E90BDF.9030501@v.loewis.de>
References: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>
	<48E90BDF.9030501@v.loewis.de>
Message-ID: <11E966A4-CC55-41A3-8D44-5D7099F54769@barrys-emacs.org>


On Oct 5, 2008, at 19:47, Martin v. L?wis wrote:

>> Why does "key in wc_status_kind_wc" work when I use an object  
>> returned
>> by keys() by not when I use pysvn.wc_status_kind.unversioned?
>
> This is too little detail to come up with an explanation. Do your
> objects support __eq__.
>
> Regards,
> Martin
>

I wrote a smaller version of the code that fails and had a session with
gdb.

My object implements tp_compare but lookdict() uses richcompare.
If richcompare is not implemented do_richcompare() falls back to
comparing PyObject * pointers - which is cause of the KeyError
as the objects cmp() eq but have different PyObect * values.

This is a change from V2 python where to be a key implementing
tp_hash and tp_compare is sufficient.

In V3 is it your intention that to be a key you must implement
tp_hash and tp_richcompare? If not I'll raise a bug against
3.0 on this issue.

Barry


From barry at python.org  Wed Oct  8 00:00:23 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 7 Oct 2008 18:00:23 -0400
Subject: [Python-Dev] [Python-3000] Proposed Python 3.0 schedule
In-Reply-To: <ca471dc20810071328t65ee07faw5e0672be8bdc3fea@mail.gmail.com>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<ca471dc20810071328t65ee07faw5e0672be8bdc3fea@mail.gmail.com>
Message-ID: <E16A1C28-F7C9-4CC1-836C-E254AFF5AE7E@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 7, 2008, at 4:28 PM, Guido van Rossum wrote:

> On Mon, Oct 6, 2008 at 5:47 PM, Barry Warsaw <barry at python.org> wrote:
>> So, we need to come up with a new release schedule for Python 3.0.   
>> My
>> suggestion:
>>
>> 15-Oct-2008 3.0 beta 4
>> 05-Nov-2008 3.0 rc 2
>> 19-Nov-2008 3.0 rc 3
>> 03-Dec-2008 3.0 final
>>
>> Given what still needs to be done, is this a reasonable schedule?   
>> Do we
>> need two more betas?
>
> I know I'm contradicting what I said earlier, but perhaps we should
> just forget going back to beta and stick to ever-more-perfect release
> candidates? In other worlds release candidates often contain tons of
> imperfections (I believe I've seen this both for Java and Windows) and
> the label "release candidate" more clearly encourages people to
> download and play with it, which is what we need at this point! Then
> the schedule would be something like
>
>  15-Oct-2008 3.0 rc 2
>  05-Nov-2008 3.0 rc 3
>  19-Nov-2008 3.0 rc 4
>  03-Dec-2008 3.0 final

I'm okay with that too.  It does seem odd to go back to beta then  
release another rc.  What's in a name, anyway? <wink>.  And it is good  
that more people are downloading it now that it's rc.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOvb93EjvBPtnXfVAQJTQAP/cmNdzd/SRymxXvW85EnW2NTHUkh1Auw9
bGlbSC0BF2p9ArgbDLPh/X4uatB3UaqoNeq5LTWHL2f9iCnsI7lFMPuexGr+3t4l
Xmld8qN77j4GpU6bXL8uUo3/vlhU4MiG5ETl0kMH30f47srOAAGEGZAqW9jAM92I
YSkQPSgBdYo=
=+s9t
-----END PGP SIGNATURE-----

From barry at python.org  Wed Oct  8 00:01:39 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 7 Oct 2008 18:01:39 -0400
Subject: [Python-Dev] [Python-3000]  Proposed Python 3.0 schedule
In-Reply-To: <48EBD8F2.4090802@gmail.com>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
	<67693931-29C5-4FD7-8E83-D97F892F3BE3@python.org>
	<48EBD8F2.4090802@gmail.com>
Message-ID: <3F6B0210-EE87-4CE4-B487-DF4AAF733637@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 7, 2008, at 5:47 PM, Nick Coghlan wrote:

> Barry Warsaw wrote:
>> On Oct 6, 2008, at 9:48 PM, Raymond Hettinger wrote:
>>
>>> [Barry Warsaw]
>>>> So, we need to come up with a new release schedule for Python 3.0.
>>>> My  suggestion:
>>>> 15-Oct-2008 3.0 beta 4
>>>> 05-Nov-2008 3.0 rc 2
>>>> 19-Nov-2008 3.0 rc 3
>>>> 03-Dec-2008 3.0 final
>>>> Given what still needs to be done, is this a reasonable schedule?
>>>> Do  we need two more betas?
>>
>>> Yes to both questions.
>>
>> I think that's contradictory :).  If we need two betas, then 05-Nov
>> becomes beta 5, 19-Nov is rc 2.  If we don't need another rc then  
>> we can
>> still do a final release on 03-Dec, otherwise we probably go 2 weeks
>> later.  I don't want to go much later than that though because then  
>> we
>> get into the holiday season.
>
> Do we need the full two weeks between rc's? Or is it too much of a  
> pain
> to cut releases 3 weeks in a row?
>
> E.g. something like:
>
> 15-Oct-2008 3.0 beta 4
> 05-Nov-2008 3.0 beta 5
> 19-Nov-2008 3.0 rc 2
> 26-Nov-2008 3.0 rc 3 (if needed)
> 03-Dec-2008 3.0 final

I won't be able to cut another release between the 15th and 5th, so at  
least that one should be 2 weeks.  If we don't need the additional rc,  
then we can release early, which would put us just before the US  
Thanksgiving holiday.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOvcQ3EjvBPtnXfVAQK5mwP9GQfw3zNvGhJWiSkZ2gQ1LNr0rnmfVmpF
WcDePkz3e5nsOjtkwiN0rlYHIQE9ySPfvtqqrInBW8y97y79mTjiM4S32XHLyAsd
WEWRb0ClcLuZs+JveAb8KF5pO0RlDgX9Dd6puuPr8kGa5aN/rosfsnXra1GrYpj3
JQghQ89JNkE=
=+Ymq
-----END PGP SIGNATURE-----

From barry at python.org  Wed Oct  8 00:15:31 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 7 Oct 2008 18:15:31 -0400
Subject: [Python-Dev] [Python-3000] Proposed Python 3.0 schedule
In-Reply-To: <ca471dc20810071328t65ee07faw5e0672be8bdc3fea@mail.gmail.com>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<ca471dc20810071328t65ee07faw5e0672be8bdc3fea@mail.gmail.com>
Message-ID: <E5E34408-545F-4D34-9663-15F9B65698F9@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 7, 2008, at 4:28 PM, Guido van Rossum wrote:

>  15-Oct-2008 3.0 rc 2
>  05-Nov-2008 3.0 rc 3
>  19-Nov-2008 3.0 rc 4
>  03-Dec-2008 3.0 final

I've updated PEP 361 and the Google calendar with this schedule,  
except that the PEP says that rc3 and rc4 are planned "if needed".

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOvfg3EjvBPtnXfVAQKDfwP/Sz9Ioe1tIrKtvD7JPG2cg2F+wfDJrc+9
vqfh6/eMWiUIOeSKJu6+gye7oXRcHwQXAPivNza3993HesOu0TjudnwXfkAlfsdE
m09Rh70AXQQiY7JX46etugRC4BwkuNeBo253cvmfo6hPK0ZhOHZSy3H1LkhvvLA6
Cq56CVqDUgs=
=i/Km
-----END PGP SIGNATURE-----

From barry at python.org  Wed Oct  8 00:16:56 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 7 Oct 2008 18:16:56 -0400
Subject: [Python-Dev] [Python-3000]  Proposed Python 3.0 schedule
In-Reply-To: <3F6B0210-EE87-4CE4-B487-DF4AAF733637@python.org>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
	<67693931-29C5-4FD7-8E83-D97F892F3BE3@python.org>
	<48EBD8F2.4090802@gmail.com>
	<3F6B0210-EE87-4CE4-B487-DF4AAF733637@python.org>
Message-ID: <AE0C94C8-10F3-4000-A36A-815C5A1CCB40@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 7, 2008, at 6:01 PM, Barry Warsaw wrote:

> I won't be able to cut another release between the 15th and 5th, so  
> at least that one should be 2 weeks.  If we don't need the  
> additional rc, then we can release early, which would put us just  
> before the US Thanksgiving holiday.

Er, /3/ weeks between rc2 and rc3.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSOvf2HEjvBPtnXfVAQJDsQP8DRL2gQDMf1eEvgmmijPtVdbfAypZ1XMY
huNzPu91v6dpvrogIP5MJbmJnSnka5yk78JIlkbTU4ZHS0ADsQX+IApU5y/SlO9Y
FDtIqb+NFoVRFj5xQaN/EEqO8kNpq3WPmaEQJ4HHeDUIzcrbsPxfCm+vbePgnGzI
AwhQqCzmX1I=
=aQnH
-----END PGP SIGNATURE-----

From foom at fuhm.net  Wed Oct  8 00:22:13 2008
From: foom at fuhm.net (James Y Knight)
Date: Tue, 7 Oct 2008 18:22:13 -0400
Subject: [Python-Dev] [Python-3000]  Proposed Python 3.0 schedule
In-Reply-To: <aac2c7cb0810071345o2c20d3d1v42b45e79f3833c17@mail.gmail.com>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
	<9212E57A-2292-43DC-9307-F05C0DD91CDA@fuhm.net>
	<48EB1408.1030007@v.loewis.de>
	<BDDCC54F-78D9-40DA-8F63-04DE4DCA0CEE@fuhm.net>
	<aac2c7cb0810071345o2c20d3d1v42b45e79f3833c17@mail.gmail.com>
Message-ID: <AA10B68C-B2D7-47B9-B38B-A8679A14CE8D@fuhm.net>

On Oct 7, 2008, at 4:45 PM, Adam Olsen wrote:
> So what does Qt do when given a file name already using those PUA?
> Looks like they get passed through untouched when decoded, but will
> get translated into invalid names upon encoding.

Well, I'd say that looks like a bug. It should probably decode those  
PUA characters as if they were undecodeable sequences so that they too  
roundtrip properly.

> So you still have
> file names you can't open

In practical terms, I suspect nobody has ever run into a file which  
has this problem. You certainly can't say that is the case for  
Python-3's current behavior; my suspicion is that anyone who uses any  
non-ascii filenames at all will run into issues with Python3's  
behavior at least once.

> , and you're incompatible with what other
> libraries do.

I'm sure there's a situation where that matters, but, at least I can  
run kpdf /any/arbitrary/file.pdf and have it work. And use the KDE  
file chooser, and have it able to browse my files, and choose any  
file, no matter what random characters it has in it. If there is an  
issue with interfacing to another library, the string can be converted  
to whatever the other library expects at the interface point...

People keep claiming that odd filenames are only going to be an issue  
for "backup tools", but I don't think that's true. I think it'll be an  
issue for most any program that reads user-specified files. Whether it  
be by running Python in an ASCII (e.g. "C") locale when there are  
files created with UTF-8 names, or by having copied/downloaded a file  
with an incorrectly encoded name, it's going to come up, and be an  
irritant when it does.

That Qt felt the need to make this change rather strengthens that  
point IMO...

> The only thing going for Qt is that they seem specifically interested
> in latin-1, rather than arbitrary bad names.  The latin-1 strings that
> would correspond to the UTF-8 PUA used would include at least one
> control character, as well as other unusual bits, so it's pretty
> unlikely to encounter a real latin-1 file name like that.


I'd say they're most concerned about files that their users are likely  
to run into, yes.

James

From martin at v.loewis.de  Wed Oct  8 00:31:40 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 08 Oct 2008 00:31:40 +0200
Subject: [Python-Dev] porting pycxx and pysvn to python 3.0 hit a problem
In-Reply-To: <11E966A4-CC55-41A3-8D44-5D7099F54769@barrys-emacs.org>
References: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>	<48E90BDF.9030501@v.loewis.de>
	<11E966A4-CC55-41A3-8D44-5D7099F54769@barrys-emacs.org>
Message-ID: <48EBE34C.1080102@v.loewis.de>

> In V3 is it your intention that to be a key you must implement
> tp_hash and tp_richcompare? If not I'll raise a bug against
> 3.0 on this issue.

I believe that cmp/tp_compare are being phased out, although
I think there was a heavy debate about this.

In any case, I think you really need to implement tp_richcompare.

Regards,
Martin

From barry at barrys-emacs.org  Wed Oct  8 01:20:55 2008
From: barry at barrys-emacs.org (Barry Scott)
Date: Wed, 8 Oct 2008 00:20:55 +0100
Subject: [Python-Dev] porting pycxx and pysvn to python 3.0 hit a problem
In-Reply-To: <48EBE34C.1080102@v.loewis.de>
References: <8847CE6D-1656-4BF1-A07A-D602453C3B80@barrys-emacs.org>	<48E90BDF.9030501@v.loewis.de>
	<11E966A4-CC55-41A3-8D44-5D7099F54769@barrys-emacs.org>
	<48EBE34C.1080102@v.loewis.de>
Message-ID: <52823E95-E653-4B9B-AFED-AA108E4018AC@barrys-emacs.org>


On Oct 7, 2008, at 23:31, Martin v. L?wis wrote:

>> In V3 is it your intention that to be a key you must implement
>> tp_hash and tp_richcompare? If not I'll raise a bug against
>> 3.0 on this issue.
>
> I believe that cmp/tp_compare are being phased out, although
> I think there was a heavy debate about this.

Given the problems I am seeing with tp_compare I'd advise
that you get rid of it for 3.0. The half hearted support is worst
then no support. At least if you remove tp_compare it forces
a porter to implement tp_richcompare.

>
> In any case, I think you really need to implement tp_richcompare.

I've added support already in PyCXX. I'll now implement
tp_richcompare for pysvn and document this in my PyCXX
porting guide.

Barry


From lists at cheimes.de  Wed Oct  8 01:25:25 2008
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 08 Oct 2008 01:25:25 +0200
Subject: [Python-Dev] [Fwd: In Python 2.6, bytes is str]
In-Reply-To: <ca471dc20810071313g5c102ff8w26a349e9aadd0aba@mail.gmail.com>
References: <48E9B3A1.7090704@holdenweb.com>
	<ca471dc20810071313g5c102ff8w26a349e9aadd0aba@mail.gmail.com>
Message-ID: <gcgr54$e15$1@ger.gmane.org>

Guido van Rossum wrote:
> It was well understood that the bytes "type" in 2.6 and the bytes type
> in 3.0 would behave quite different. Attempts to come up with a
> separate bytes type that behaved more like its 3.0 counterpart were
> doomed, because there just are too many places where the usage was
> ambiguous. We should probably have written a PEP about this just to
> prevent the discussion from being repeated all over again in this
> thread.
> 
> The only two anticipated *reasonable* uses in 2.6 were the bytes
> literal (b'abc') and tests for isinstance(x, bytes), which are flags
> for 2to3 to keep these usages as bytes, not str.

Full ack!

I replies this to the op on the general Python list:
---
I guess you got the intention of the bytes alias wrong. It's partly my 
fault because I didn't document the bytes alias. We are well aware that 
the bytes alias in 2.6 isn't fully compatible with the bytes type in 
3.0. The bytes alias isn't MEANT to be compatible, too.  :)

At first I wanted to backport the bytes type from 3.0 to 2.6. But it was 
too many work and the implications of yet another types were too 
complex. So I just added the alias in order to help people with writing 
forward compatible code like e.g. isinstance(egg, bytes).

Summa summarum the bytes alias was added for documentary purpose and to 
aid the 2to3 transition of code where 'str' is ambiguous.
--

> I have no intention of rolling this back. It isn't compatible with
> 3.0, but then, 2.6 and 3.0 aren't in general meant to be compatible --
> 2.6 is a stepping stone, but that's not the same thing. It is
> backwards compatible with prior versions because it is new in 2.6, so
> it shouldn't break old code. I have only limited sympathy for people
> who don't read documentation in this case.

Maybe the documentation isn't clear enough what purpose the bytes alias 
serves? I spent 5 minutes with the docs but I wasn't able to find a good 
explanation of the bytes alias

Christian


From steve at holdenweb.com  Wed Oct  8 02:37:36 2008
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 07 Oct 2008 20:37:36 -0400
Subject: [Python-Dev] [Fwd: In Python 2.6, bytes is str]
In-Reply-To: <gcgr54$e15$1@ger.gmane.org>
References: <48E9B3A1.7090704@holdenweb.com>	<ca471dc20810071313g5c102ff8w26a349e9aadd0aba@mail.gmail.com>
	<gcgr54$e15$1@ger.gmane.org>
Message-ID: <48EC00D0.7090407@holdenweb.com>

Christian Heimes wrote:
> Guido van Rossum wrote:
[...]
>> I have no intention of rolling this back. It isn't compatible with
>> 3.0, but then, 2.6 and 3.0 aren't in general meant to be compatible --
>> 2.6 is a stepping stone, but that's not the same thing. It is
>> backwards compatible with prior versions because it is new in 2.6, so
>> it shouldn't break old code. I have only limited sympathy for people
>> who don't read documentation in this case.
> 
> Maybe the documentation isn't clear enough what purpose the bytes alias
> serves? I spent 5 minutes with the docs but I wasn't able to find a good
> explanation of the bytes alias
> 
Yes, I think all that's really needed is a clarification in the
documentation. Just so people expect slightly kooky behavior of the kind
originally noted.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From steve at holdenweb.com  Wed Oct  8 02:37:36 2008
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 07 Oct 2008 20:37:36 -0400
Subject: [Python-Dev] [Fwd: In Python 2.6, bytes is str]
In-Reply-To: <gcgr54$e15$1@ger.gmane.org>
References: <48E9B3A1.7090704@holdenweb.com>	<ca471dc20810071313g5c102ff8w26a349e9aadd0aba@mail.gmail.com>
	<gcgr54$e15$1@ger.gmane.org>
Message-ID: <48EC00D0.7090407@holdenweb.com>

Christian Heimes wrote:
> Guido van Rossum wrote:
[...]
>> I have no intention of rolling this back. It isn't compatible with
>> 3.0, but then, 2.6 and 3.0 aren't in general meant to be compatible --
>> 2.6 is a stepping stone, but that's not the same thing. It is
>> backwards compatible with prior versions because it is new in 2.6, so
>> it shouldn't break old code. I have only limited sympathy for people
>> who don't read documentation in this case.
> 
> Maybe the documentation isn't clear enough what purpose the bytes alias
> serves? I spent 5 minutes with the docs but I wasn't able to find a good
> explanation of the bytes alias
> 
Yes, I think all that's really needed is a clarification in the
documentation. Just so people expect slightly kooky behavior of the kind
originally noted.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From amk at amk.ca  Wed Oct  8 02:44:09 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Tue, 7 Oct 2008 20:44:09 -0400
Subject: [Python-Dev] 2.5.3: assessing commits
Message-ID: <20081008004409.GA26215@amk.local>

I've begun the task of assessing the 2.6 commits, but the job is
unexpectedly large.

What I did:

* Took the output of 'svn log -r60999:66717'.  (2.5.2 was branched
  from rev. 60999, so I'm ignoring commits to the trunk before 2.5.2 was 
  branch, which may miss some things.)

* Wrote a little script to explode the log into a separate file for each 
  commit.  This resulted in about 2200 files.

* Did a long series of 'grep' and 'rm' commands to remove irrelevant files.
  For example, if the commit touches abc.py, bytesobject.c, bytearrayobject.c,
  etc. it was removed.

* Wrote a little script to find commits that only touch the docs, and moved them
  aside.

At this point I still have 1191 files left.  Many of these commits are
still irrelevant, but this is a lot for me to look through.  A tarball
with the remaining commits is at

      http://www.amk.ca/files/python/2.6-changes.tgz

Can we parallelize the job?  One person could take commits starting
with '01', another with '02', etc.  Or each person could assess the
commits they made.

--amk

From mhammond at skippinet.com.au  Wed Oct  8 03:04:36 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Wed, 8 Oct 2008 12:04:36 +1100
Subject: [Python-Dev] [python-committers]  Proposed Python 3.0 schedule
In-Reply-To: <48EBBB25.70609@v.loewis.de>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
	<00e001c92881$68ba93c0$3a2fbb40$@com.au>
	<48EBBB25.70609@v.loewis.de>
Message-ID: <014001c928e1$dc13af40$943b0dc0$@com.au>

> > * Better support for 2to3 in distutils (specifically, the support in
> > build_py is stale, plus 'build_scripts' and 'install_data' should
> > convert
> > .py files to py3k syntax.)
> 
> Please do create a bug report for that. It sounds like it's easy to
> fix.

Yeah, build_py is fairly easy to fix, but I also needed to extend the
support to build_scripts and install_data.  In addition, some already
reported bugs in 2to3 mean that some files fail to convert, and this breaks
the entire process - so as a result I ended up duplicating lib2to3's
'refactor_items()' but with exceptions being logged and ingored rather than
aborting the process.  Oh - and I deleted the .bak files (a copy of the
sources are converted, not the sources themselves)

Please see bugs 4072 and 4073  - but as mentioned below, the lack of a test
case means I didn't supply a tested patch.

> > An 'example' project that uses py2k syntax and
> > "just works" on py3k using this strategy might be useful here.
> 
> Perhaps pywin32 :-?
> 
> I don't think a demo project would do much good, as it doesn't exercise
> all the issues that may occur.

My idea was that the demo project would simply demonstrate the 2to3 concepts
that such a project could use.  pywin32 isn't a good example as it has a
very non-trivial setup.py and a large set of C extensions (the demo I had in
mind could avoid C extensions completely - C developers will already assume
#ifdef will be their friend, but .py code is the unknown...)

It would basically be a 'distutils demo', could have a single .py module and
a single .py script.  setup.py would support both 2.x and 3.x and would
demonstrate how the source is converted to py3k syntax before it is
installed into the py3k distribution.

It would also provide a useful test case - eg, for the distutils bug above,
I'm not sure how I can (a) demonstrate it is currently broken and (b)
demonstrate a patch corrects the problem.

> > * A standard 'helper script' that allows people to use py3k to
> > execute a py2x syntax script by auto-converting the code.  I've 
> > a 10ish-line script that uses lib2to3 plus exec() to achieve that 
> > result, but a helper in 2to3
> > for this would be nice.  For a concrete use-case, we want to keep our
> > distutils script in py2x syntax, but execute it via py3k.  Its very
> > possible this already exists and I've just missed it...
> 
> For the case of setup.py, I was hoping that it could be written in
> compatible syntax even without needing conversion. That worked fine for
> my Django port. Is that not the case for pywin32?

setup.py catches and examines some exceptions.  Consider the more general
case though - pywin32 has a number of tests all of which will also be
maintained in py2x syntax.  It is extremely convenient to be able to
execute:

% py3k run2.py my_test.py etc

And have 'my_test.py' (which is 2.x syntax) be executed directly by py3k
without doing a full 'setup.py install' or manually invoking 2to3 via a temp
file, etc.  As mentioned, 'run2.py' is quite short and just uses
lib2to3+exec, but I'm not sure everyone will work out how to roll their
own...

Specifically, I believe that a script with similar capabilities could be
installed with py3k in the "scripts" directory and it advertised as a
reasonable way to directly execute your *scripts* which, although py3x
compatible, are being maintained in py2x syntax.  Below is my quick attempt
at such a script, which I promptly stopped looking at as soon as it worked
(ie, I'm not sure if all those options are needed, etc), but it does let me
execute my tests using py3k directly from the source tree.
 
Cheers,

Mark

---
# This is a Python 3.x script to execute a python 2.x script by 2to3'ing it.
import sys
from lib2to3.refactor import RefactoringTool, get_fixers_from_package

fixers = get_fixers_from_package('lib2to3.fixes')
options = dict(doctests_only=False, fix=[], list_fixes=[], 
               print_function=False, verbose=False,
               write=True)
r = RefactoringTool(fixers, options)
script = sys.argv[1]
data = open(script).read()
print("Converting...")
got = r.refactor_string(data, script)
print("Executing...")
# nuke ourselves from argv
del sys.argv[1]
exec(str(got))
---


From mhammond at skippinet.com.au  Wed Oct  8 03:26:22 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Wed, 8 Oct 2008 12:26:22 +1100
Subject: [Python-Dev] [python-committers]  Proposed Python 3.0 schedule
In-Reply-To: <014001c928e1$dc13af40$943b0dc0$@com.au>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>	<00e001c92881$68ba93c0$3a2fbb40$@com.au>	<48EBBB25.70609@v.loewis.de>
	<014001c928e1$dc13af40$943b0dc0$@com.au>
Message-ID: <014201c928e4$e726bd20$b5743760$@com.au>

> at such a script, which I promptly stopped looking at as soon as it
> worked

Which is quite obvious really given that:

> # nuke ourselves from argv
> del sys.argv[1]

is removing the wrong value!

Mark


From amk at amk.ca  Wed Oct  8 14:06:09 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Wed, 8 Oct 2008 08:06:09 -0400
Subject: [Python-Dev] 2.5.3: assessing commits
In-Reply-To: <20081008004409.GA26215@amk.local>
References: <20081008004409.GA26215@amk.local>
Message-ID: <20081008120609.GA5995@amk-desktop.matrixgroup.net>

On Tue, Oct 07, 2008 at 08:44:09PM -0400, A.M. Kuchling wrote:
> At this point I still have 1191 files left.  Many of these commits are
> still irrelevant, but this is a lot for me to look through.  A tarball
> with the remaining commits is at
> 
>       http://www.amk.ca/files/python/2.6-changes.tgz

On the metro this morning, I worked on this some more and now have
only 858 files for consideration.  At this point I think we have to
look at commits individually.  I'll turn the lists of commits into a
set of wiki page that we can examine and edit down in parallel.

(I've updated the above tarball in the meantime.)

--amk

From sidnei at enfoldsystems.com  Wed Oct  8 19:15:15 2008
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Wed, 8 Oct 2008 14:15:15 -0300
Subject: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version of
	medusa
Message-ID: <a7a2b76b0810081015p55710cd1i589a45bf78fde0fb@mail.gmail.com>

I am working on getting Zope to run (or at least, start) with Python
2.6. It actually starts right now after applying some patches, which
is amazing on itself, but it dies right away due to changes in
asyncore that break Zope's internal version of medusa.

I've opened a bug against Zope on Launchpad, but someone suggested
that it might actually be a bug in python, in the sense that it
changed asyncore in a backwards-incompatible way. I wouldn't go that
far, since I think it's more likely that Zope's version of medusa is
poking into asyncore internals instead:

https://bugs.edge.launchpad.net/zope2/+bug/280020

I suspect a patch similar to this might be needed:

http://codereview.appspot.com/744/diff/1/23

Anyone cares to comment?

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From janssen at parc.com  Wed Oct  8 19:30:33 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 8 Oct 2008 10:30:33 PDT
Subject: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version
	of medusa
In-Reply-To: <a7a2b76b0810081015p55710cd1i589a45bf78fde0fb@mail.gmail.com>
References: <a7a2b76b0810081015p55710cd1i589a45bf78fde0fb@mail.gmail.com>
Message-ID: <5003.1223487033@parc.com>

Sidnei da Silva <sidnei at enfoldsystems.com> wrote:

> https://bugs.edge.launchpad.net/zope2/+bug/280020

I think there are real issues here with both asynchat and Medusa.
Asynchat has been heavily re-written, and the "ac_out_buffer" has
apparently disappeared.  But "ac_out_buffer_size" is still there.  That
strikes me as odd, and probably means that asynchat.py needs more
changes.  However, Medusa (basically just an application layer on top
of asyncore/asynchat) also needs to be re-written to take account of
the changes in asynchat.

Bill


From metawilm at gmail.com  Wed Oct  8 19:53:16 2008
From: metawilm at gmail.com (Willem Broekema)
Date: Wed, 8 Oct 2008 19:53:16 +0200
Subject: [Python-Dev] effect of "exec" on local scope
Message-ID: <f6bc9b490810081053g7368ca0bge379e8b39a8b3daf@mail.gmail.com>

The issue came up while trying to get some Sympy code running on CLPython.

class C:
 exec "a = 3"
 print locals()

1. Is it guaranteed that class C gets an attribute "a", i.e. that the
locals printed include {'a': 3}?
2. It it (also) guaranteed if it were in a function scope?

The complete syntax of exec is:
 exec CODE in Y, Z
where Y, Z are optional.

The documentation of "exec" says "if the optional parts are
omitted,the code is executed in the current scope." There are at least
two different interpretations:

 a. The code is executed in the current class scope, so the assignment
must have an effect on the class scope.

 b. The scope defaults to the local scope, by which is meant the
mapping returned by locals(), and of locals() the documentation says
that changes made to it may not influence the interpreter. (The
documentation of exec suggests using globals() and locals() as
arguments to exec, which seems hint at this interpretation.)

The relevant documentation:
 exec: http://docs.python.org/reference/simple_stmts.html#grammar-token-exec_stmt
 locals: http://docs.python.org/library/functions.html#locals

- Willem

From theller at ctypes.org  Wed Oct  8 20:16:09 2008
From: theller at ctypes.org (Thomas Heller)
Date: Wed, 08 Oct 2008 20:16:09 +0200
Subject: [Python-Dev] python dll no longer in system directory?
Message-ID: <gcitdb$nm5$1@ger.gmane.org>

Is it intended that python30.dll and python26.dll are not longer
installed in the \windows\system32 directory?

This (pythonxy.dll not on $PATH) causes problems for COM objects
implemented in Python.

Thanks,
Thomas


From lists at cheimes.de  Wed Oct  8 20:43:01 2008
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 08 Oct 2008 20:43:01 +0200
Subject: [Python-Dev] python dll no longer in system directory?
In-Reply-To: <gcitdb$nm5$1@ger.gmane.org>
References: <gcitdb$nm5$1@ger.gmane.org>
Message-ID: <gciuvl$t3o$1@ger.gmane.org>

Thomas Heller wrote:
> Is it intended that python30.dll and python26.dll are not longer
> installed in the \windows\system32 directory?
> 
> This (pythonxy.dll not on $PATH) causes problems for COM objects
> implemented in Python.

How did you install Python 2.6? Did you install it only for yourself or 
for all users?

Christian


From theller at ctypes.org  Wed Oct  8 20:51:39 2008
From: theller at ctypes.org (Thomas Heller)
Date: Wed, 08 Oct 2008 20:51:39 +0200
Subject: [Python-Dev] python dll no longer in system directory?
In-Reply-To: <gciuvl$t3o$1@ger.gmane.org>
References: <gcitdb$nm5$1@ger.gmane.org> <gciuvl$t3o$1@ger.gmane.org>
Message-ID: <gcivfr$v6k$1@ger.gmane.org>

Christian Heimes schrieb:
> Thomas Heller wrote:
>> Is it intended that python30.dll and python26.dll are not longer
>> installed in the \windows\system32 directory?
>> 
>> This (pythonxy.dll not on $PATH) causes problems for COM objects
>> implemented in Python.
> 
> How did you install Python 2.6? Did you install it only for yourself or 
> for all users?
> 

For all users.  I only deselected the 'register file extensions' option.
This is on XP SP3, with admin rights.

-- 
Thanks,
Thomas


From musiccomposition at gmail.com  Wed Oct  8 20:59:38 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Wed, 8 Oct 2008 12:59:38 -0600
Subject: [Python-Dev] [python-committers]  Proposed Python 3.0 schedule
In-Reply-To: <014001c928e1$dc13af40$943b0dc0$@com.au>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<040FDB9B68C549AE848AC35C0231DD70@RaymondLaptop1>
	<00e001c92881$68ba93c0$3a2fbb40$@com.au> <48EBBB25.70609@v.loewis.de>
	<014001c928e1$dc13af40$943b0dc0$@com.au>
Message-ID: <1afaf6160810081159o18e64e68te95ab94f1198472c@mail.gmail.com>

On 10/7/08, Mark Hammond <mhammond at skippinet.com.au> wrote:
> # This is a Python 3.x script to execute a python 2.x script by 2to3'ing it.
> import sys
> from lib2to3.refactor import RefactoringTool, get_fixers_from_package
>
> fixers = get_fixers_from_package('lib2to3.fixes')
> options = dict(doctests_only=False, fix=[], list_fixes=[],
>                print_function=False, verbose=False,
>                write=True)

Note that only the print_function option is used.

> r = RefactoringTool(fixers, options)
> script = sys.argv[1]
> data = open(script).read()
> print("Converting...")
> got = r.refactor_string(data, script)
> print("Executing...")
> # nuke ourselves from argv
> del sys.argv[1]
> exec(str(got))
> ---
>
> _______________________________________________
> python-committers mailing list
> python-committers at python.org
> http://mail.python.org/mailman/listinfo/python-committers
>


-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From tjreedy at udel.edu  Wed Oct  8 21:17:06 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 08 Oct 2008 15:17:06 -0400
Subject: [Python-Dev] effect of "exec" on local scope
In-Reply-To: <f6bc9b490810081053g7368ca0bge379e8b39a8b3daf@mail.gmail.com>
References: <f6bc9b490810081053g7368ca0bge379e8b39a8b3daf@mail.gmail.com>
Message-ID: <gcj0vh$62n$1@ger.gmane.org>

Willem Broekema wrote:
> The issue came up while trying to get some Sympy code running on CLPython.
> 
> class C:
>  exec "a = 3"
>  print locals()
> 
> 1. Is it guaranteed that class C gets an attribute "a", i.e. that the
> locals printed include {'a': 3}?
> 2. It it (also) guaranteed if it were in a function scope?
> 
> The complete syntax of exec is:
>  exec CODE in Y, Z
> where Y, Z are optional.
> 
> The documentation of "exec" says "if the optional parts are
> omitted,the code is executed in the current scope." There are at least
> two different interpretations:
> 
>  a. The code is executed in the current class scope, so the assignment
> must have an effect on the class scope.
> 
>  b. The scope defaults to the local scope, by which is meant the
> mapping returned by locals(), and of locals() the documentation says
> that changes made to it may not influence the interpreter. (The
> documentation of exec suggests using globals() and locals() as
> arguments to exec, which seems hint at this interpretation.)
> 
> The relevant documentation:
>  exec: http://docs.python.org/reference/simple_stmts.html#grammar-token-exec_stmt
>  locals: http://docs.python.org/library/functions.html#locals

The 3.0 doc for exec() has this warning:
"Warning
The default locals act as described for function locals() below: 
modifications to the default locals dictionary should not be attempted. 
Pass an explicit locals dictionary if you need to see effects of the 
code on locals after function exec() returns."

This implies interpretation b.

However, is spite of the warning, class locals is a dict and locals() is 
that dict, so a is available for further use in class code.

So the answer to question 1 for current CPython is yes.

Whether that is guaranteed for all implementations and versions is 
another story.

Functions are much trickier.  The local namespace is not a dict, and 
modifying the locals() dict does not modify the namespace.  The answer 
to 2. is No, not even now.

 >>> def f():
	exec('a=3')
	print(locals())
	return a

 >>> f()
{'a': 3}
Traceback (most recent call last):
   File "<pyshell#37>", line 1, in <module>
     f()
   File "<pyshell#36>", line 4, in f
     return a
NameError: global name 'a' is not defined

But why then is 'a' printed in the second call to locals (the implied 
one in exec being the first)?  It appears that a function or code object 
can have only only one repeatedly used shadow dict.  The 3.0 (and 2.5) 
doc says "locals()
Update and return a dictionary representing the current local symbol 
table."  Note "update"; I had missed that before.  To see this...

 >>> def g():
	a =  locals()
	b = locals()
	return id(a), id(b), a,b

 >>> g()
(20622048, 20622048, {'a': {...}}, {'a': {...}})

Inserting "print(a['a'])" between the locals calls raises KeyError.

Terry Jan Reedy



From kristjan at ccpgames.com  Wed Oct  8 21:27:06 2008
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Wed, 8 Oct 2008 19:27:06 +0000
Subject: [Python-Dev] __getattr__ and new style classes
Message-ID: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>

Hello there.
I've just noticed what I consider a performance problem:
Using new style classes to provide attribute-like access using __getattr__ is considerably slower than old style classes:  Observe:

s = """
class dude:
        def bar(self):pass
        def __getattr__(self, a): return a
class dude2(object):
        def bar(self):pass
        def __getattr__(self, a): return a
d = dude()
d2 = dude2()
d.a = d2.a = 1
"""
timeit.Timer(?d.foo?, s).timeit()
>0.32979211801421116
timeit.Timer(?d2.foo?, s).timeit()
> 1.1119853719342245

The overhead is almost 3 times as high.  I imagine that this is because new style classes must search further and harder before giving up and going to __getattr__.
For the bound method the difference is less:

timeit.Timer(?d.bar?, s).timeit()
> 0.11835480370018558
timeit.Timer(?d2.bar?, s).timeit()
> 0.17820851929263881

For fun, I also tested regular attributes, and see:

timeit.Timer(?d.a?, s).timeit()
> 0.069161394202183146
timeit.Timer(?d2.a?, s).timeit()
> 0.17966275972594303

I'm surprised that accessing instance attributes like this is twice as slow using new style classes.

Any thoughts on this?
We are using a lot of low-level attribute access magic in EVE and so it would appear that we are best served by sticking with old-style classes.  But these are going away eventually, so what to do?
Where is this extra overhead coming from?
And oh, it is no use using __getattribute__ instead, since it will always involve calls to object.__getattribute__ and become very slow.

Cheers,
Kristj?n


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081008/d4811484/attachment.htm>

From amk at amk.ca  Wed Oct  8 22:00:35 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Wed, 8 Oct 2008 16:00:35 -0400
Subject: [Python-Dev] 2.5.3: assessing commits
In-Reply-To: <20081008120609.GA5995@amk-desktop.matrixgroup.net>
References: <20081008004409.GA26215@amk.local>
	<20081008120609.GA5995@amk-desktop.matrixgroup.net>
Message-ID: <20081008200035.GA15065@amk-desktop.matrixgroup.net>

On Wed, Oct 08, 2008 at 08:06:09AM -0400, A.M. Kuchling wrote:
> look at commits individually.  I'll turn the lists of commits into a
> set of wiki pages that we can examine and edit down in parallel.

I decided to put them in SVN instead, in sandbox/py2.5.3/ .

How do we want to assess these commits for possible inclusion in
2.5.3?

--amk

From martin at v.loewis.de  Wed Oct  8 22:00:38 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 08 Oct 2008 22:00:38 +0200
Subject: [Python-Dev] python dll no longer in system directory?
In-Reply-To: <gcitdb$nm5$1@ger.gmane.org>
References: <gcitdb$nm5$1@ger.gmane.org>
Message-ID: <48ED1166.9030809@v.loewis.de>

Thomas Heller wrote:
> Is it intended that python30.dll and python26.dll are not longer
> installed in the \windows\system32 directory?

No, it's not. Please create a bug report (or, better, study the
msiexec logs, and msi.py, to find out why this happens).

I might not have time to look into this before the next release
candidate.

Regards,
Martin

From musiccomposition at gmail.com  Wed Oct  8 22:43:22 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Wed, 8 Oct 2008 15:43:22 -0500
Subject: [Python-Dev] [Python-3000] Proposed Python 3.0 schedule
In-Reply-To: <48EC4A57.8030608@hlabs.spb.ru>
References: <2A0F6C99-481A-473B-B1A5-7B9FB5A47D6F@python.org>
	<1afaf6160810061752w390ba174m66baf6646175105d@mail.gmail.com>
	<48EC4A57.8030608@hlabs.spb.ru>
Message-ID: <1afaf6160810081343h62bf5ab3pad21d32e68a48313@mail.gmail.com>

On Wed, Oct 8, 2008 at 12:51 AM, Dmitry Vasiliev <dima at hlabs.spb.ru> wrote:
>
> BTW, I think the following issues should be also marked as release blockers:

Agreed and done.

>
> - http://bugs.python.org/issue3714 (nntplib module broken by str to
> unicode conversion)
> - http://bugs.python.org/issue3725 (telnetlib module broken by str to
> unicode conversion)
> - http://bugs.python.org/issue3727 (poplib module broken by str to
> unicode conversion)
>
> --
> Dmitry Vasiliev <dima at hlabs.spb.ru>
> http://hlabs.spb.ru
>



-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From josiah.carlson at gmail.com  Wed Oct  8 23:26:00 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Wed, 8 Oct 2008 14:26:00 -0700
Subject: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version
	of medusa
In-Reply-To: <5003.1223487033@parc.com>
References: <a7a2b76b0810081015p55710cd1i589a45bf78fde0fb@mail.gmail.com>
	<5003.1223487033@parc.com>
Message-ID: <e6511dbf0810081426q4c576bbak8ab9a7e1d6936254@mail.gmail.com>

On Wed, Oct 8, 2008 at 10:30 AM, Bill Janssen <janssen at parc.com> wrote:
> Sidnei da Silva <sidnei at enfoldsystems.com> wrote:
>
>> https://bugs.edge.launchpad.net/zope2/+bug/280020
>
> I think there are real issues here with both asynchat and Medusa.
> Asynchat has been heavily re-written, and the "ac_out_buffer" has
> apparently disappeared.  But "ac_out_buffer_size" is still there.  That
> strikes me as odd, and probably means that asynchat.py needs more
> changes.  However, Medusa (basically just an application layer on top
> of asyncore/asynchat) also needs to be re-written to take account of
> the changes in asynchat.

ac_out_buffer was removed because it is unneeded (we have a deque; why
rely on an extra attribute?).
ac_out_buffer_size still remains as a blocksize in which to pre-split
outgoing data (if you have a 100k string you want to send, repeatedly
slicing it as you are able to send pieces is slow, but pre-slicing
chunks is fast, and generally results in being able to pass full chunk
to the underlying TCP/IP stack).

But yes, zope needs to be changed to reflect the updated
asyncore/asynchat semantics.  Trust me; it's faster, cleaner, and
easier to use now.

 - Josiah

From sidnei at enfoldsystems.com  Wed Oct  8 23:31:07 2008
From: sidnei at enfoldsystems.com (Sidnei da Silva)
Date: Wed, 8 Oct 2008 18:31:07 -0300
Subject: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version
	of medusa
In-Reply-To: <e6511dbf0810081426q4c576bbak8ab9a7e1d6936254@mail.gmail.com>
References: <a7a2b76b0810081015p55710cd1i589a45bf78fde0fb@mail.gmail.com>
	<5003.1223487033@parc.com>
	<e6511dbf0810081426q4c576bbak8ab9a7e1d6936254@mail.gmail.com>
Message-ID: <a7a2b76b0810081431g486f922bld4f33d67c762d6d3@mail.gmail.com>

On Wed, Oct 8, 2008 at 6:26 PM, Josiah Carlson <josiah.carlson at gmail.com> wrote:
> ac_out_buffer was removed because it is unneeded (we have a deque; why
> rely on an extra attribute?).
> ac_out_buffer_size still remains as a blocksize in which to pre-split
> outgoing data (if you have a 100k string you want to send, repeatedly
> slicing it as you are able to send pieces is slow, but pre-slicing
> chunks is fast, and generally results in being able to pass full chunk
> to the underlying TCP/IP stack).
>
> But yes, zope needs to be changed to reflect the updated
> asyncore/asynchat semantics.  Trust me; it's faster, cleaner, and
> easier to use now.

Great to know. So medusa should be changed similarly to the changes
made to asynchat? Your suggestion on the bugtracker is even better: to
subclass asynchat as much as possible.

-- 
Sidnei da Silva
Enfold Systems                http://enfoldsystems.com
Fax +1 832 201 8856     Office +1 713 942 2377 Ext 214

From ncoghlan at gmail.com  Wed Oct  8 23:37:19 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 09 Oct 2008 07:37:19 +1000
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
Message-ID: <48ED280F.6000302@gmail.com>

Kristj?n Valur J?nsson wrote:
> Hello there.
> 
> I?ve just noticed what I consider a performance problem:
> 
> Using new style classes to provide attribute-like access using
> __getattr__ is considerably slower than old style classes:  Observe:

I can't reproduce those relative numbers using SVN trunk. Using your
setup code (copied and pasted directly from your message to my
interpreter session) I got the following numbers:


>>> timeit.Timer('d.foo', s).timeit()
1.2362558841705322
>>> timeit.Timer('d2.foo', s).timeit()
1.1634600162506104
>>> timeit.Timer('d.foo', s).timeit()
1.1840031147003174
>>> timeit.Timer('d2.foo', s).timeit()
1.1554200649261475

(a slight speed advantage to the new-style class for __getattr__)

>>> timeit.Timer('d.bar', s).timeit()
0.17601609230041504
>>> timeit.Timer('d2.bar', s).timeit()
0.18697309494018555
>>> timeit.Timer('d.bar', s).timeit()
0.1711127758026123
>>> timeit.Timer('d2.bar', s).timeit()
0.1827549934387207

(very slight speed advantage to the old-style class for the unbound method)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From janssen at parc.com  Wed Oct  8 23:49:02 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 8 Oct 2008 14:49:02 PDT
Subject: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version
	of medusa
In-Reply-To: <a7a2b76b0810081431g486f922bld4f33d67c762d6d3@mail.gmail.com>
References: <a7a2b76b0810081015p55710cd1i589a45bf78fde0fb@mail.gmail.com>
	<5003.1223487033@parc.com>
	<e6511dbf0810081426q4c576bbak8ab9a7e1d6936254@mail.gmail.com>
	<a7a2b76b0810081431g486f922bld4f33d67c762d6d3@mail.gmail.com>
Message-ID: <12485.1223502542@parc.com>

Sidnei da Silva <sidnei at enfoldsystems.com> wrote:

> Great to know. So medusa should be changed similarly to the changes
> made to asynchat?

Hmmm.  Been a long time since a Medusa release, but I need a working
Medusa, too.

> Your suggestion on the bugtracker is even better: to
> subclass asynchat as much as possible.

That's basically what Medusa is, a set of subclasses.

Bill

From steve at pearwood.info  Thu Oct  9 00:35:46 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 9 Oct 2008 08:35:46 +1000
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
Message-ID: <200810090935.46724.steve@pearwood.info>

On Thu, 9 Oct 2008 06:27:06 am Kristj?n Valur J?nsson wrote:
> Hello there.
> I've just noticed what I consider a performance problem:
> Using new style classes to provide attribute-like access using
> __getattr__ is considerably slower than old style classes:  Observe:
>
> s = """
> class dude:
>         def bar(self):pass
>         def __getattr__(self, a): return a
> class dude2(object):
>         def bar(self):pass
>         def __getattr__(self, a): return a
> d = dude()
> d2 = dude2()
> d.a = d2.a = 1
> """
> timeit.Timer(?d.foo?, s).timeit()
> >0.32979211801421116
>
> timeit.Timer(?d2.foo?, s).timeit()
> > 1.1119853719342245

Not only don't I observe the same results as you, I'm afraid I can't 
even get your code to run. I get a SyntaxError from the funny quotes 
you're using: ?d.foo? instead of 'd.foo' or "d.foo".

Also, I trust you know enough not to pay too much attention to a single 
test result? Timing results are notoriously subject to interference 
from external processes.

Finally, when reporting performance problems, it will be VERY helpful to 
report the version and platform you are using. There's no point in 
having people looking for a slow-down in Python 2.6 if you're actually 
using Python 2.3 (say).

Anyway, for the record here are my results for Python 2.5. I don't 
believe the differences are significant.

>>> min(Timer('d.foo', s).repeat())
1.9228429794311523
>>> min(Timer('d2.foo', s).repeat())
2.1040639877319336

>>> min(Timer('d.bar', s).repeat())
0.68272304534912109
>>> min(Timer('d2.bar', s).repeat())
0.41949796676635742

>>> min(Timer('d.a', s).repeat())
0.45727396011352539
>>> min(Timer('d2.a', s).repeat())
0.59516501426696777


-- 
Steven

From lists at cheimes.de  Thu Oct  9 00:53:06 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 09 Oct 2008 00:53:06 +0200
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <200810090935.46724.steve@pearwood.info>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
	<200810090935.46724.steve@pearwood.info>
Message-ID: <gcjdkh$fmo$1@ger.gmane.org>

Steven D'Aprano wrote:
> Not only don't I observe the same results as you, I'm afraid I can't 
> even get your code to run. I get a SyntaxError from the funny quotes 
> you're using: ?d.foo? instead of 'd.foo' or "d.foo".

Kristj?n is using the old style and alternative syntax for repr(). 
Somehow the `` got screwed up by either his mailer or the mailing list. 
Don't be ashamed that you aren't aware of the alternative syntax. We 
keep it locked up in the cellar and it has been removed from the new, 
shiny Python 3 world.

 >>> `object`
"<type 'object'>"
 >>> `object` == repr(object)
True

Christian


From ondrej at certik.cz  Thu Oct  9 01:40:07 2008
From: ondrej at certik.cz (Ondrej Certik)
Date: Thu, 9 Oct 2008 01:40:07 +0200
Subject: [Python-Dev] effect of "exec" on local scope
In-Reply-To: <gcj0vh$62n$1@ger.gmane.org>
References: <f6bc9b490810081053g7368ca0bge379e8b39a8b3daf@mail.gmail.com>
	<gcj0vh$62n$1@ger.gmane.org>
Message-ID: <85b5c3130810081640r2557bffdl243ba2c2e6d6463f@mail.gmail.com>

Hi Terry,

On Wed, Oct 8, 2008 at 9:17 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> Willem Broekema wrote:
>>
>> The issue came up while trying to get some Sympy code running on CLPython.
>>
>> class C:
>>  exec "a = 3"
>>  print locals()
>>
>> 1. Is it guaranteed that class C gets an attribute "a", i.e. that the
>> locals printed include {'a': 3}?
>> 2. It it (also) guaranteed if it were in a function scope?
>>
>> The complete syntax of exec is:
>>  exec CODE in Y, Z
>> where Y, Z are optional.
>>
>> The documentation of "exec" says "if the optional parts are
>> omitted,the code is executed in the current scope." There are at least
>> two different interpretations:
>>
>>  a. The code is executed in the current class scope, so the assignment
>> must have an effect on the class scope.
>>
>>  b. The scope defaults to the local scope, by which is meant the
>> mapping returned by locals(), and of locals() the documentation says
>> that changes made to it may not influence the interpreter. (The
>> documentation of exec suggests using globals() and locals() as
>> arguments to exec, which seems hint at this interpretation.)
>>
>> The relevant documentation:
>>  exec:
>> http://docs.python.org/reference/simple_stmts.html#grammar-token-exec_stmt
>>  locals: http://docs.python.org/library/functions.html#locals
>
> The 3.0 doc for exec() has this warning:
> "Warning
> The default locals act as described for function locals() below:
> modifications to the default locals dictionary should not be attempted. Pass
> an explicit locals dictionary if you need to see effects of the code on
> locals after function exec() returns."
>
> This implies interpretation b.
>
> However, is spite of the warning, class locals is a dict and locals() is
> that dict, so a is available for further use in class code.
>
> So the answer to question 1 for current CPython is yes.
>
> Whether that is guaranteed for all implementations and versions is another
> story.
>
> Functions are much trickier.  The local namespace is not a dict, and
> modifying the locals() dict does not modify the namespace.  The answer to 2.
> is No, not even now.
>
>>>> def f():
>        exec('a=3')
>        print(locals())
>        return a
>
>>>> f()
> {'a': 3}
> Traceback (most recent call last):
>  File "<pyshell#37>", line 1, in <module>
>    f()
>  File "<pyshell#36>", line 4, in f
>    return a
> NameError: global name 'a' is not defined
>
> But why then is 'a' printed in the second call to locals (the implied one in
> exec being the first)?  It appears that a function or code object can have
> only only one repeatedly used shadow dict.  The 3.0 (and 2.5) doc says
> "locals()
> Update and return a dictionary representing the current local symbol table."
>  Note "update"; I had missed that before.  To see this...
>
>>>> def g():
>        a =  locals()
>        b = locals()
>        return id(a), id(b), a,b
>
>>>> g()
> (20622048, 20622048, {'a': {...}}, {'a': {...}})
>
> Inserting "print(a['a'])" between the locals calls raises KeyError.


Thanks very much for the thorough answer. The reason for Willem's
question is this code that we currently have in sympy (see [1] for the
whole thread):

class Basic(AssumeMeths):
  ...
  for k in AssumeMeths._assume_defined:
    exec "is_%s  = property(make__get_assumption('Basic', '%s'))" % (k,k)


Which works in CPython but fails in CLPython. From your answer it
seems to me that this code is not nice and we should not use it and
should rather use something like:

class Basic(AssumeMeths):
  ...

for k in AssumeMeths._assume_defined:
  setattr(Basic, 'is_%s' % k, property(make__get_assumption('Basic', '%s' % k)))


which should work on all platforms. What do you think?

Ondrej

[1] http://code.google.com/p/sympy/issues/detail?id=1134

From greg.ewing at canterbury.ac.nz  Thu Oct  9 01:55:23 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 09 Oct 2008 12:55:23 +1300
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
Message-ID: <48ED486B.60909@canterbury.ac.nz>

Kristj?n Valur J?nsson wrote:

> Using new style classes to provide attribute-like access using 
> __getattr__ is considerably slower than old style classes

Do you really need __getattr__, or could you use
properties instead?

-- 
Greg

From guido at python.org  Thu Oct  9 03:02:31 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 8 Oct 2008 18:02:31 -0700
Subject: [Python-Dev] effect of "exec" on local scope
In-Reply-To: <85b5c3130810081640r2557bffdl243ba2c2e6d6463f@mail.gmail.com>
References: <f6bc9b490810081053g7368ca0bge379e8b39a8b3daf@mail.gmail.com>
	<gcj0vh$62n$1@ger.gmane.org>
	<85b5c3130810081640r2557bffdl243ba2c2e6d6463f@mail.gmail.com>
Message-ID: <ca471dc20810081802m63930c34we7248524b86312a5@mail.gmail.com>

Well, I don't recall what CLPython is, but I believe it is broken and
that code should work -- there are (or used to be) examples of using
exec to populate classes in the standard library so while it may look
dodgy it really is exected to work...

On Wed, Oct 8, 2008 at 4:40 PM, Ondrej Certik <ondrej at certik.cz> wrote:
> Hi Terry,
>
> On Wed, Oct 8, 2008 at 9:17 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>> Willem Broekema wrote:
>>>
>>> The issue came up while trying to get some Sympy code running on CLPython.
>>>
>>> class C:
>>>  exec "a = 3"
>>>  print locals()
>>>
>>> 1. Is it guaranteed that class C gets an attribute "a", i.e. that the
>>> locals printed include {'a': 3}?
>>> 2. It it (also) guaranteed if it were in a function scope?
>>>
>>> The complete syntax of exec is:
>>>  exec CODE in Y, Z
>>> where Y, Z are optional.
>>>
>>> The documentation of "exec" says "if the optional parts are
>>> omitted,the code is executed in the current scope." There are at least
>>> two different interpretations:
>>>
>>>  a. The code is executed in the current class scope, so the assignment
>>> must have an effect on the class scope.
>>>
>>>  b. The scope defaults to the local scope, by which is meant the
>>> mapping returned by locals(), and of locals() the documentation says
>>> that changes made to it may not influence the interpreter. (The
>>> documentation of exec suggests using globals() and locals() as
>>> arguments to exec, which seems hint at this interpretation.)
>>>
>>> The relevant documentation:
>>>  exec:
>>> http://docs.python.org/reference/simple_stmts.html#grammar-token-exec_stmt
>>>  locals: http://docs.python.org/library/functions.html#locals
>>
>> The 3.0 doc for exec() has this warning:
>> "Warning
>> The default locals act as described for function locals() below:
>> modifications to the default locals dictionary should not be attempted. Pass
>> an explicit locals dictionary if you need to see effects of the code on
>> locals after function exec() returns."
>>
>> This implies interpretation b.
>>
>> However, is spite of the warning, class locals is a dict and locals() is
>> that dict, so a is available for further use in class code.
>>
>> So the answer to question 1 for current CPython is yes.
>>
>> Whether that is guaranteed for all implementations and versions is another
>> story.
>>
>> Functions are much trickier.  The local namespace is not a dict, and
>> modifying the locals() dict does not modify the namespace.  The answer to 2.
>> is No, not even now.
>>
>>>>> def f():
>>        exec('a=3')
>>        print(locals())
>>        return a
>>
>>>>> f()
>> {'a': 3}
>> Traceback (most recent call last):
>>  File "<pyshell#37>", line 1, in <module>
>>    f()
>>  File "<pyshell#36>", line 4, in f
>>    return a
>> NameError: global name 'a' is not defined
>>
>> But why then is 'a' printed in the second call to locals (the implied one in
>> exec being the first)?  It appears that a function or code object can have
>> only only one repeatedly used shadow dict.  The 3.0 (and 2.5) doc says
>> "locals()
>> Update and return a dictionary representing the current local symbol table."
>>  Note "update"; I had missed that before.  To see this...
>>
>>>>> def g():
>>        a =  locals()
>>        b = locals()
>>        return id(a), id(b), a,b
>>
>>>>> g()
>> (20622048, 20622048, {'a': {...}}, {'a': {...}})
>>
>> Inserting "print(a['a'])" between the locals calls raises KeyError.
>
>
> Thanks very much for the thorough answer. The reason for Willem's
> question is this code that we currently have in sympy (see [1] for the
> whole thread):
>
> class Basic(AssumeMeths):
>  ...
>  for k in AssumeMeths._assume_defined:
>    exec "is_%s  = property(make__get_assumption('Basic', '%s'))" % (k,k)
>
>
> Which works in CPython but fails in CLPython. From your answer it
> seems to me that this code is not nice and we should not use it and
> should rather use something like:
>
> class Basic(AssumeMeths):
>  ...
>
> for k in AssumeMeths._assume_defined:
>  setattr(Basic, 'is_%s' % k, property(make__get_assumption('Basic', '%s' % k)))
>
>
> which should work on all platforms. What do you think?
>
> Ondrej
>
> [1] http://code.google.com/p/sympy/issues/detail?id=1134
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From janssen at parc.com  Thu Oct  9 03:39:40 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 8 Oct 2008 18:39:40 PDT
Subject: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version
	of medusa
In-Reply-To: <e6511dbf0810081426q4c576bbak8ab9a7e1d6936254@mail.gmail.com>
References: <a7a2b76b0810081015p55710cd1i589a45bf78fde0fb@mail.gmail.com>
	<5003.1223487033@parc.com>
	<e6511dbf0810081426q4c576bbak8ab9a7e1d6936254@mail.gmail.com>
Message-ID: <29052.1223516380@parc.com>

Josiah Carlson <josiah.carlson at gmail.com> wrote:

> But yes, zope needs to be changed to reflect the updated
> asyncore/asynchat semantics.  Trust me; it's faster, cleaner, and
> easier to use now.

Just for completeness, I built a fresh 2.6, installed Medusa (from
http://www.amk.ca/python/code/medusa.html), and it runs just fine (well,
as well as it does on 2.5, anyway).  I think this is a Zope issue.

Bill

From guido at python.org  Thu Oct  9 04:18:11 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 8 Oct 2008 19:18:11 -0700
Subject: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version
	of medusa
In-Reply-To: <29052.1223516380@parc.com>
References: <a7a2b76b0810081015p55710cd1i589a45bf78fde0fb@mail.gmail.com>
	<5003.1223487033@parc.com>
	<e6511dbf0810081426q4c576bbak8ab9a7e1d6936254@mail.gmail.com>
	<29052.1223516380@parc.com>
Message-ID: <ca471dc20810081918q29e6bf1cx651827abe35ae3d7@mail.gmail.com>

On Wed, Oct 8, 2008 at 6:39 PM, Bill Janssen <janssen at parc.com> wrote:
> Josiah Carlson <josiah.carlson at gmail.com> wrote:
>
>> But yes, zope needs to be changed to reflect the updated
>> asyncore/asynchat semantics.  Trust me; it's faster, cleaner, and
>> easier to use now.
>
> Just for completeness, I built a fresh 2.6, installed Medusa (from
> http://www.amk.ca/python/code/medusa.html), and it runs just fine (well,
> as well as it does on 2.5, anyway).  I think this is a Zope issue.

Way back in the day, Zope monkeypatched whole parts of asyncore,
replacing them with some of its own code. If that's still the case,
this breakage should be no surprise.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tjreedy at udel.edu  Thu Oct  9 05:18:13 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 08 Oct 2008 23:18:13 -0400
Subject: [Python-Dev] effect of "exec" on local scope
In-Reply-To: <85b5c3130810081640r2557bffdl243ba2c2e6d6463f@mail.gmail.com>
References: <f6bc9b490810081053g7368ca0bge379e8b39a8b3daf@mail.gmail.com>	<gcj0vh$62n$1@ger.gmane.org>
	<85b5c3130810081640r2557bffdl243ba2c2e6d6463f@mail.gmail.com>
Message-ID: <gcjt5h$i3l$1@ger.gmane.org>

Ondrej Certik wrote:

> Which works in CPython but fails in CLPython. From your answer it
> seems to me that this code is not nice and we should not use it and
> should rather use something like:
> 
> class Basic(AssumeMeths):
>   ...
> 
> for k in AssumeMeths._assume_defined:
>   setattr(Basic, 'is_%s' % k, property(make__get_assumption('Basic', '%s' % k)))
> 
> which should work on all platforms. What do you think?

That is what setattr is for.  Many consider exec a last resort.  I think 
any further discussion should move to the general python list or c.l.p 
since this is not a develop-core-python issue.


From eckhardt at satorlaser.com  Thu Oct  9 10:50:24 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Thu, 9 Oct 2008 10:50:24 +0200
Subject: [Python-Dev] Subversion access down?
Message-ID: <200810091050.24723.eckhardt@satorlaser.com>

Hi!

Is it only me or does it fail for other people, too? I'm getting

| Server sent unexpected return value (503 Service
| Unavailable) in response to OPTIONS request 
| for 'http://svn.python.org/projects/python/trunk'  


Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From amauryfa at gmail.com  Thu Oct  9 11:40:30 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 9 Oct 2008 11:40:30 +0200
Subject: [Python-Dev] python dll no longer in system directory?
In-Reply-To: <48ED1166.9030809@v.loewis.de>
References: <gcitdb$nm5$1@ger.gmane.org> <48ED1166.9030809@v.loewis.de>
Message-ID: <e27efe130810090240ya86c5e1q9fe027f843659541@mail.gmail.com>

2008/10/8 "Martin v. L?wis" <martin at v.loewis.de>:
> Thomas Heller wrote:
>> Is it intended that python30.dll and python26.dll are not longer
>> installed in the \windows\system32 directory?
>
> No, it's not. Please create a bug report (or, better, study the
> msiexec logs, and msi.py, to find out why this happens).
>
> I might not have time to look into this before the next release
> candidate.

I confirm this. The following lines in msi.py seem to be the cause of
the change:

    #dlldir = PyDirectory(db, cab, root, srcdir, "DLLDIR", ".")
    #install python30.dll into root dir for now
    dlldir = root

They were added by r61109: " Bundle msvcr90.dll as a "private assembly". "
but I don't know if simply restoring the previous value will work in every case:
If the C Run-Time is installed "privately", then python26.dll must
stay in c:\python26.

-- 
Amaury Forgeot d'Arc

From ncoghlan at gmail.com  Thu Oct  9 13:16:39 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 09 Oct 2008 21:16:39 +1000
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <gcjdkh$fmo$1@ger.gmane.org>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>	<200810090935.46724.steve@pearwood.info>
	<gcjdkh$fmo$1@ger.gmane.org>
Message-ID: <48EDE817.6010907@gmail.com>

Christian Heimes wrote:
> Steven D'Aprano wrote:
>> Not only don't I observe the same results as you, I'm afraid I can't
>> even get your code to run. I get a SyntaxError from the funny quotes
>> you're using: ?d.foo? instead of 'd.foo' or "d.foo".
> 
> Kristj?n is using the old style and alternative syntax for repr().
> Somehow the `` got screwed up by either his mailer or the mailing list.
> Don't be ashamed that you aren't aware of the alternative syntax. We
> keep it locked up in the cellar and it has been removed from the new,
> shiny Python 3 world.

I think it's actually some single quotes that got mangled by the mailer.
Either way, something else is going on for Kristj?n to see such wildly
different results between old-style and new-style attribute access, when
the differences are in the noise for the other folks checking it.

I was able to get the old-style class to be consistently faster by going
back and trying the example on 2.4, but the change still wasn't even
close to the dramatic difference Kristj?n is seeing.

Kristj?n, is there a chance CCP optimised something in the old-style
class attribute lookup code and didn't get around to submitting the
patch back to python.org?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From kristjan at ccpgames.com  Thu Oct  9 13:21:59 2008
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Thu, 9 Oct 2008 11:21:59 +0000
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <48ED280F.6000302@gmail.com>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
	<48ED280F.6000302@gmail.com>
Message-ID: <4E9372E6B2234D4F859320D896059A95101822E1DC@exchis.ccp.ad.local>

Thanks for trying this out.
When I saw your comments, I realized that the difference is due to a speed patch that we have her at CCP.
The test was run using current 2.5 python, with a patch applied, as attached this message.
The patch was written by me a few years back when I noticed that python spends enormous time during regular
run creating exceptions that then get discarded internally.  Most of the cost comes from FormatException
when it is generating AttributeErrors.
So, when I thought I was complaining about slow new objects, I was really showing off my cool optimization.

The problem is, it is difficult to generalize this to python in general.  I spent some time last year to
try to improve new-style classes, by adding ts fields and telling the exception system to "expect a certain
type of exception, and if it is raised, use PyErr_SetNone() instead of the PyErr_Format because I will
clear it anyway".  I had macros such as
PyErr_Expect(AttributeError);
attr = PyObject_GetAttr(o, a);
PyErr_Expect(0);
if (!attr)
        PyErr_Clear()

The problem was, I wasn't able to come up with a simple patch that showed consistent speed improvements
in the performance testsuite without showing some slowdowns in other places.

Running regular python code through a profiler, and especially code that relies much on the use of
__getattr__() to emulate attribute access, will show hideous amounts of time spent formatting
attribute exceptions that get thrown away.

Any thoughts on how to do this better?

And, for completeness, a new test run of this, using python25, first using the neutered patch version
(where I have set softfail=0 in the relevant places) and then with the patch active:

import timeit
s = """
class dude:
        def bar(self):pass
        def __getattr__(self, a): return a
class dude2(object):
        def bar(self):pass
        def __getattr__(self, a): return a
d = dude()
d2 = dude2()
d.a = d2.a = 1
"""
print timeit.Timer("d.foo", s).timeit()
print timeit.Timer("d.bar", s).timeit()
print timeit.Timer("d.a", s).timeit()

print timeit.Timer("d2.foo", s).timeit()
print timeit.Timer("d2.bar", s).timeit()
print timeit.Timer("d2.a", s).timeit()

patch neutered
1.35734336404
0.157773452422
0.0937950608722
1.48494915604
0.240154539405
0.186524222345

patch active:
0.352850669641
0.147599760073
0.0910020300097
1.4453737036
0.212842069748
0.203442097864


Cheers,

Kristj?n
> -----Original Message-----
> From: Nick Coghlan [mailto:ncoghlan at gmail.com]
> Sent: Wednesday, October 08, 2008 21:37
> To: Kristj?n Valur J?nsson
> Cc: Python-Dev
> Subject: Re: [Python-Dev] __getattr__ and new style classes
>
> Kristj?n Valur J?nsson wrote:
> > Hello there.
> >
> > I?ve just noticed what I consider a performance problem:
> >
> > Using new style classes to provide attribute-like access using
> > __getattr__ is considerably slower than old style classes:  Observe:
>
> I can't reproduce those relative numbers using SVN trunk. Using your
> setup code (copied and pasted directly from your message to my
> interpreter session) I got the following numbers:
>
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: speedpatch.rar
Type: application/octet-stream
Size: 1639 bytes
Desc: speedpatch.rar
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081009/0834679a/attachment.obj>

From kristjan at ccpgames.com  Thu Oct  9 13:23:16 2008
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Thu, 9 Oct 2008 11:23:16 +0000
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <gcjdkh$fmo$1@ger.gmane.org>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
	<200810090935.46724.steve@pearwood.info> <gcjdkh$fmo$1@ger.gmane.org>
Message-ID: <4E9372E6B2234D4F859320D896059A95101822E1DE@exchis.ccp.ad.local>

No, it was really me being sloppy using outlook and fighting the editor trying to insert smart quotes :)
Sorry for the confusion.
K

> -----Original Message-----
> From: python-dev-bounces+kristjan=ccpgames.com at python.org
> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf
> Of Christian Heimes
> Kristj?n is using the old style and alternative syntax for repr().
> Somehow the `` got screwed up by either his mailer or the mailing list.
> Don't be ashamed that you aren't aware of the alternative syntax. We
> keep it locked up in the cellar and it has been removed from the new,
> shiny Python 3 world.
>


From lists at cheimes.de  Thu Oct  9 13:23:51 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 09 Oct 2008 13:23:51 +0200
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <48EDE817.6010907@gmail.com>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>	<200810090935.46724.steve@pearwood.info>
	<gcjdkh$fmo$1@ger.gmane.org> <48EDE817.6010907@gmail.com>
Message-ID: <48EDE9C7.1010202@cheimes.de>

Nick Coghlan wrote:
> I think it's actually some single quotes that got mangled by the mailer.
> Either way, something else is going on for Kristj?n to see such wildly
> different results between old-style and new-style attribute access, when
> the differences are in the noise for the other folks checking it.

I still think they are back ticks. Maybe the repr() function of new
style classes is slower than the old style repr(). It might explain the
difference.

Christian

From ncoghlan at gmail.com  Thu Oct  9 13:27:47 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 09 Oct 2008 21:27:47 +1000
Subject: [Python-Dev] effect of "exec" on local scope
In-Reply-To: <ca471dc20810081802m63930c34we7248524b86312a5@mail.gmail.com>
References: <f6bc9b490810081053g7368ca0bge379e8b39a8b3daf@mail.gmail.com>	<gcj0vh$62n$1@ger.gmane.org>	<85b5c3130810081640r2557bffdl243ba2c2e6d6463f@mail.gmail.com>
	<ca471dc20810081802m63930c34we7248524b86312a5@mail.gmail.com>
Message-ID: <48EDEAB3.2070208@gmail.com>

Guido van Rossum wrote:
> Well, I don't recall what CLPython is, but I believe it is broken and
> that code should work -- there are (or used to be) examples of using
> exec to populate classes in the standard library so while it may look
> dodgy it really is exected to work...

I think this behaviour (modifying locals() at class scope) was actually
implicitly blessed in PEP 3115, even though that PEP doesn't explicitly
state locals() in a class body is required to grant access to the
namespace returned by __prepare__().

Perhaps the time is right for the locals() documentation to be more
explicit regarding when modifying its contents will affect the current
namespace?
- yes at module scope (effectively the same as globals())
- yes at class scope
- maybe (but typically not) at function scope

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Thu Oct  9 13:34:55 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 09 Oct 2008 21:34:55 +1000
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <48EDE9C7.1010202@cheimes.de>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>	<200810090935.46724.steve@pearwood.info>
	<gcjdkh$fmo$1@ger.gmane.org> <48EDE817.6010907@gmail.com>
	<48EDE9C7.1010202@cheimes.de>
Message-ID: <48EDEC5F.90904@gmail.com>

Christian Heimes wrote:
> Nick Coghlan wrote:
>> I think it's actually some single quotes that got mangled by the mailer.
>> Either way, something else is going on for Kristj?n to see such wildly
>> different results between old-style and new-style attribute access, when
>> the differences are in the noise for the other folks checking it.
> 
> I still think they are back ticks. Maybe the repr() function of new
> style classes is slower than the old style repr(). It might explain the
> difference.

Nope - if they were backticks, you'd either get a NameError (if d and d2
aren't defined in the scope where the timeit calls are being made), or
you'd end up timing the evaluation of some string literals.

Mailer issues aside though, we still don't know where that initial set
of strange numbers is coming from.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Thu Oct  9 13:53:37 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 09 Oct 2008 21:53:37 +1000
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <4E9372E6B2234D4F859320D896059A95101822E1DC@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
	<48ED280F.6000302@gmail.com>
	<4E9372E6B2234D4F859320D896059A95101822E1DC@exchis.ccp.ad.local>
Message-ID: <48EDF0C1.5060705@gmail.com>

Kristj?n Valur J?nsson wrote:
> Running regular python code through a profiler, and especially code that relies much on the use of
> __getattr__() to emulate attribute access, will show hideous amounts of time spent formatting
> attribute exceptions that get thrown away.
> 
> Any thoughts on how to do this better?

If the time is being spent in PyErr_Format, how far could you get adding
a dedicated function for creating AttributeErrors? Something along the
lines of:

PyErr_AttributeError(PyObject *object, PyObject *attr_name)

It would then be possible to modify AttributeError to have obj_type and
attr_name attributes (holding the type name and the name of the missing
attribute), so that any string formatting could be deferred until repr()
was called on the AttributeError instance.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Thu Oct  9 13:56:50 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 09 Oct 2008 21:56:50 +1000
Subject: [Python-Dev] Subversion access down?
In-Reply-To: <200810091050.24723.eckhardt@satorlaser.com>
References: <200810091050.24723.eckhardt@satorlaser.com>
Message-ID: <48EDF182.8010302@gmail.com>

Ulrich Eckhardt wrote:
> Hi!
> 
> Is it only me or does it fail for other people, too? I'm getting
> 
> | Server sent unexpected return value (503 Service
> | Unavailable) in response to OPTIONS request 
> | for 'http://svn.python.org/projects/python/trunk'

svn+ssh access is working for me - it looks like the web gateway may
just need a kick to get it going again.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From amk at amk.ca  Thu Oct  9 14:07:42 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 9 Oct 2008 08:07:42 -0400
Subject: [Python-Dev] Subversion access down?
In-Reply-To: <200810091050.24723.eckhardt@satorlaser.com>
References: <200810091050.24723.eckhardt@satorlaser.com>
Message-ID: <20081009120742.GA6409@amk-desktop.matrixgroup.net>

On Thu, Oct 09, 2008 at 10:50:24AM +0200, Ulrich Eckhardt wrote:
> Is it only me or does it fail for other people, too? I'm getting
> 
> | Server sent unexpected return value (503 Service
> | Unavailable) in response to OPTIONS request 
> | for 'http://svn.python.org/projects/python/trunk'  

python.org seems to be having some network weirdness right now.
I'm looking into it.  (SVN access over SSH seems to be fine.)

--amk

From amk at amk.ca  Thu Oct  9 14:29:03 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 9 Oct 2008 08:29:03 -0400
Subject: [Python-Dev] Subversion access down?
In-Reply-To: <200810091050.24723.eckhardt@satorlaser.com>
References: <200810091050.24723.eckhardt@satorlaser.com>
Message-ID: <20081009122903.GA6127@amk-desktop.matrixgroup.net>

On Thu, Oct 09, 2008 at 10:50:24AM +0200, Ulrich Eckhardt wrote:
> Is it only me or does it fail for other people, too? I'm getting
> 
> | Server sent unexpected return value (503 Service
> | Unavailable) in response to OPTIONS request 
> | for 'http://svn.python.org/projects/python/trunk'  

svn.python.org is working again now.

--amk

From kristjan at ccpgames.com  Thu Oct  9 16:09:37 2008
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Thu, 9 Oct 2008 14:09:37 +0000
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <48EDF0C1.5060705@gmail.com>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
	<48ED280F.6000302@gmail.com>
	<4E9372E6B2234D4F859320D896059A95101822E1DC@exchis.ccp.ad.local>
	<48EDF0C1.5060705@gmail.com>
Message-ID: <4E9372E6B2234D4F859320D896059A95101822E418@exchis.ccp.ad.local>

It's an interesting idea...  But it seems hard to forge the magic that
does this (creating it in args[0] on demand) without some heavy work.

In my opinion, the problem is that we are raising all these errors
at all, with object creation and all that it entails.  We are using
exceptions very much as program logic to test for the existence of
attributes.

Python itself has the getattr(obj, attr, default) thing, encouraging
you to not use exceptions where the absence of an attribute is
fully excpected.  But the C layer doesn't have this.

In my opinion, the C api should have this capability, which is why
I bothered with implementing PyObject_GetAttrSoft() there.
Ideally, it should have a signature such as:
int PyObject_GetAttrSoft(PyObject **result, PyObject *obj, PyObject *attr, int raise);
where the return value indicates success (0) or an exception (-1) and the
"result" gets either NULL or something else in case of success.
"raise" would indicate whether we want the absence of an attibute to raise
an exception or not.

Now this is all well and good, but the big problem is that we invariably
end up calling the tp->tp_getattr slots that have no such functionality.

I'd actually be more inclined to try to go for a full solution using a
tp_gettattrc (c for conditional) with a default implementation....
Maybe I'll play with that over the weekend, see where it takes us.

K

> -----Original Message-----
> From: Nick Coghlan [mailto:ncoghlan at gmail.com]
> Sent: Thursday, October 09, 2008 11:54
> To: Kristj?n Valur J?nsson
> Cc: Python-Dev
> Subject: Re: [Python-Dev] __getattr__ and new style classes
>
> Kristj?n Valur J?nsson wrote:
> > Running regular python code through a profiler, and especially code
> that relies much on the use of
> > __getattr__() to emulate attribute access, will show hideous amounts
> of time spent formatting
> > attribute exceptions that get thrown away.
> >
> > Any thoughts on how to do this better?
>
> If the time is being spent in PyErr_Format, how far could you get
> adding
> a dedicated function for creating AttributeErrors? Something along the
> lines of:
>
> PyErr_AttributeError(PyObject *object, PyObject *attr_name)
>
> It would then be possible to modify AttributeError to have obj_type and
> attr_name attributes (holding the type name and the name of the missing
> attribute), so that any string formatting could be deferred until
> repr()
> was called on the AttributeError instance.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>             http://www.boredomandlaziness.org


From tseaver at palladion.com  Thu Oct  9 18:08:23 2008
From: tseaver at palladion.com (Tres Seaver)
Date: Thu, 09 Oct 2008 12:08:23 -0400
Subject: [Python-Dev] asyncore fixes in Python 2.6 broke Zope's version
	of medusa
In-Reply-To: <ca471dc20810081918q29e6bf1cx651827abe35ae3d7@mail.gmail.com>
References: <a7a2b76b0810081015p55710cd1i589a45bf78fde0fb@mail.gmail.com>	<5003.1223487033@parc.com>	<e6511dbf0810081426q4c576bbak8ab9a7e1d6936254@mail.gmail.com>	<29052.1223516380@parc.com>
	<ca471dc20810081918q29e6bf1cx651827abe35ae3d7@mail.gmail.com>
Message-ID: <48EE2C77.1050809@palladion.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Guido van Rossum wrote:
> On Wed, Oct 8, 2008 at 6:39 PM, Bill Janssen <janssen at parc.com> wrote:
>> Josiah Carlson <josiah.carlson at gmail.com> wrote:
>>
>>> But yes, zope needs to be changed to reflect the updated
>>> asyncore/asynchat semantics.  Trust me; it's faster, cleaner, and
>>> easier to use now.
>> Just for completeness, I built a fresh 2.6, installed Medusa (from
>> http://www.amk.ca/python/code/medusa.html), and it runs just fine (well,
>> as well as it does on 2.5, anyway).  I think this is a Zope issue.
> 
> Way back in the day, Zope monkeypatched whole parts of asyncore,
> replacing them with some of its own code. If that's still the case,
> this breakage should be no surprise.

Zope has been hooking the 'asyncore.loop' function (wrapping it in order
to add a "clean shutdown' flog) since 2001 at least (the 2.5 branch is
the earliest checkout I have, and it was branched in early January 2002).

Zope also began patched asyncore's logging functions in 2.7, and later
updated that patch to make the logger use the stdlib 'logging' module.

I think the change we need to make (in ZServer/medusa/http_server.py) is
to emulate the new 'writeable' implementation in asynchat, at least when
running under 2.6.

Zope's 'ZServer.medusa.http_server.http_request.writable()'
implementation is::

    def writable (self):
        # this is just the normal async_chat 'writable',
        # here for comparison
        return self.ac_out_buffer or len(self.producer_fifo)


The Python 2.5 asynchat.asynchat.writable does::

    def writable (self):
        "predicate for inclusion in the writable for select()"
        # return len(self.ac_out_buffer) or len(self.producer_fifo) or
        # (not self.connected)
        # this is about twice as fast, though not as clear.
        return not (
                (self.ac_out_buffer == '') and
                self.producer_fifo.is_empty() and
                self.connected
                )

The Python 2.6 asynchat.asynchat.writable now does::

    def writable (self):
        "predicate for inclusion in the writable for select()"
        return self.producer_fifo or (not self.connected)


medusa 0.5.4's medusa.http_server.http_request doesn't override
'writable', but it does have a broken 'writeable_for_proxy':


    def writable_for_proxy (self):
        # this version of writable supports the idea of a 'stalled'
        # producer
        # [i.e., it's not ready to produce any output yet] This is
        # needed by
        # the proxy, which will be waiting for the magic combination of
        # 1) hostname resolved
        # 2) connection made
        # 3) data available.
        if self.ac_out_buffer:
            return 1
        elif len(self.producer_fifo):
            p = self.producer_fifo.first()
            if hasattr (p, 'stalled'):
                return not p.stalled()
            else:
                return 1


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFI7ix3+gerLs4ltQ4RAldnAKC/QLJHmdE9dxInkWuIGja0gtSXYwCcCJcH
6XooEwW/AkJ1ntmGyxi8urM=
=1kps
-----END PGP SIGNATURE-----


From theller at ctypes.org  Thu Oct  9 20:21:43 2008
From: theller at ctypes.org (Thomas Heller)
Date: Thu, 09 Oct 2008 20:21:43 +0200
Subject: [Python-Dev] python dll no longer in system directory?
In-Reply-To: <e27efe130810090240ya86c5e1q9fe027f843659541@mail.gmail.com>
References: <gcitdb$nm5$1@ger.gmane.org> <48ED1166.9030809@v.loewis.de>
	<e27efe130810090240ya86c5e1q9fe027f843659541@mail.gmail.com>
Message-ID: <gcli3k$p3q$1@ger.gmane.org>

Amaury Forgeot d'Arc schrieb:
> 2008/10/8 "Martin v. L?wis" <martin at v.loewis.de>:
>> Thomas Heller wrote:
>>> Is it intended that python30.dll and python26.dll are not longer
>>> installed in the \windows\system32 directory?
>>
>> No, it's not. Please create a bug report (or, better, study the
>> msiexec logs, and msi.py, to find out why this happens).

Done.  http://bugs.python.org/issue4091

>> I might not have time to look into this before the next release
>> candidate.
> 
> I confirm this. The following lines in msi.py seem to be the cause of
> the change:
> 
>     #dlldir = PyDirectory(db, cab, root, srcdir, "DLLDIR", ".")
>     #install python30.dll into root dir for now
>     dlldir = root
> 
> They were added by r61109: " Bundle msvcr90.dll as a "private assembly". "
> but I don't know if simply restoring the previous value will work in every case:
> If the C Run-Time is installed "privately", then python26.dll must
> stay in c:\python26.
> 

Amaury, can you add your analysis to the tracker, please?

-- 
Thanks,
Thomas


From martin at v.loewis.de  Thu Oct  9 20:41:45 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 09 Oct 2008 20:41:45 +0200
Subject: [Python-Dev] python dll no longer in system directory?
In-Reply-To: <e27efe130810090240ya86c5e1q9fe027f843659541@mail.gmail.com>
References: <gcitdb$nm5$1@ger.gmane.org> <48ED1166.9030809@v.loewis.de>
	<e27efe130810090240ya86c5e1q9fe027f843659541@mail.gmail.com>
Message-ID: <48EE5069.2000205@v.loewis.de>

> I confirm this. The following lines in msi.py seem to be the cause of
> the change:
> 
>     #dlldir = PyDirectory(db, cab, root, srcdir, "DLLDIR", ".")
>     #install python30.dll into root dir for now
>     dlldir = root
> 
> They were added by r61109: " Bundle msvcr90.dll as a "private assembly". "
> but I don't know if simply restoring the previous value will work in every case:
> If the C Run-Time is installed "privately", then python26.dll must
> stay in c:\python26.

Ah, ok. Maybe I can find some time next week to look into this, but I
wouldn't mind if anybody else did.

In any case, thanks for this research so far (and too bad that nobody
noticed throughout all the beta releases).

Regards,
Martin


From amk at amk.ca  Thu Oct  9 22:24:37 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 9 Oct 2008 16:24:37 -0400
Subject: [Python-Dev] Distutils/packaging sprint this weekend
Message-ID: <20081009202437.GA15368@amk-desktop.matrixgroup.net>

Tarek Zidae' is organizing a sprint on general
distutils/setuptools/packaging this weekend.  Physically it's in
Arlington VA, but participants will be hanging out in #distutils on
freenode's IRC.

More information at
<http://www.openplans.org/projects/plone-conference-2008-dc/distribute>.

--am"If you're in Fairfax County and need a lift, let me know"k

From martin at v.loewis.de  Thu Oct  9 23:08:43 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 09 Oct 2008 23:08:43 +0200
Subject: [Python-Dev] [Python-checkins] r66863 -
	python/trunk/Modules/posixmodule.c
In-Reply-To: <gclkpu$93f$1@ger.gmane.org>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>	<48EE4C41.30905@v.loewis.de>
	<gclkpu$93f$1@ger.gmane.org>
Message-ID: <48EE72DB.1000002@v.loewis.de>

[switching to python-dev]

Georg Brandl wrote:
> Martin v. L?wis schrieb:
>> Raymond Hettinger wrote:
>>>> Merges should be handled by the original committer.
>>> Respectfully disagree.  Some people here having been taking
>>> responsibility for keeping the branches in-sync 
>> Which people specifically?
> 
> Specifically, Christian, Benjamin and myself have done larger merges
> to the 3k branch in the past, and if svnmerge is used, I suspect will
> do the same for 2.6.

That's different, though. Does any of you has actually taken
*responsibility* to do so, either unlimited, or with some limitation?
(e.g. for a year, or until 2.7 is released, or for all changes
but bsddb and Windows).
I would be (somewhat) happy to hear that, but I wouldn't really expect
it - we are all volunteers, and we typically consider taking
responsibility (e.g. as a release manager) very carefully.

Please don't get me wrong: I very much appreciate that you volunteer,
but I don't want to discharge any committer from merging on the
assumption that someone has formally taken responsibility.

I would be skeptical relying on such a commitment, knowing that RL
can get in the way too easily. E.g. Christian disappeared for some
time, and I completely sympathize with that - but it also tells
me that I can't trust on somebody doing something unless that someone
has explicitly said that he will do that, hoping that he will tell
me when the commitment is no longer valid (the same happened, e.g.,
in the Python job board, and happens all the time in other projects -
it took me about a year before I stepped back as the PyXML maintainer).

I can *also* sympathize with committers that say "we don't want to
backport, because we either don't have the time, or the expertise
(say, to install and run svnmerge on Windows)". I just accept that
not all patches that deserve backporting actually do get backported
(just as not all patches that deserve acceptance do get accepted,
in the first place).

Regards,
Martin



From amauryfa at gmail.com  Fri Oct 10 00:43:04 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 10 Oct 2008 00:43:04 +0200
Subject: [Python-Dev] [Python-checkins] r66863 -
	python/trunk/Modules/posixmodule.c
In-Reply-To: <48EE72DB.1000002@v.loewis.de>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>
	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>
	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>
	<48EE4C41.30905@v.loewis.de> <gclkpu$93f$1@ger.gmane.org>
	<48EE72DB.1000002@v.loewis.de>
Message-ID: <e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>

Hello,

Concerning the management of this particular change / development, I
see three additional issues:

- First, I think that the answer given here:
    http://mail.python.org/pipermail/python-checkins/2008-October/074659.html
does not match the question.
It seems to me that Skip was asking whether the "memory leak" impacted
the 2.6 branch, and the answer should have been "No": the change that
introduced the memory leak had just been committed 10 minutes before.

- Because of this misunderstanding, the changes to this
GetCurrentDirectoryW were backported to the release2.6 branch, despite
the fact that it's not a regression from a previous version, the NEWS
entry explicitly expresses doubts about the correction (which I happen
to share), there is no unit test and no item in the issue tracker.

- The backport to release26-maint http://svn.python.org/view?rev=66865&view=rev
also merged other changes (new unrelated unit tests). IMO unrelated
changes should be committed separately: different commit messages help
to understand the motivation of each backport.

I'm not blaming anyone; my feelings are certainly biased by the
somewhat strict procedures that we recently followed when the trunk
was in "release candidate" mode (submit a patch, ask for a review, add
the reviewer's name in the commit message).
I think that some of these rules should still apply to the maintenance branches.
On the other hand, I am all for a trunk branch with few restrictions:
it's here to share code with others and experiment with the next
python release. The need for stability is much lower.

-- 
Amaury Forgeot d'Arc

From python at rcn.com  Fri Oct 10 01:12:40 2008
From: python at rcn.com (Raymond Hettinger)
Date: Thu, 9 Oct 2008 16:12:40 -0700
Subject: [Python-Dev] Documentation idea
Message-ID: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>

Background
----------
In the itertools module docs, I included pure python equivalents for each of the C functions.  Necessarily, some of those 
equivalents are only approximate but they seem to have greatly enhanced the docs.  Something similar is in the builtin docs for 
any() and all().  The new collections.namedtuple() factory function also includes a verbose option that prints a pure python 
equivalent for the generated class. And in the decimal module, I took examples directly from the spec and included them in 
doc-testable docstrings.  This assured compliance with the spec while providing clear examples to anyone who bothers to look at the 
docstrings.

For itertools docs, I combined those best practices and included sample calls in the pure-python code (see the current docs for 
itertools to see what I mean -- perhaps look at the docs for a new tool like itertools.product() or itertools.izip_longest() to see 
how useful it is).

Bright idea
----------
Let's go one step further and do this just about everywhere and instead of putting it in the docs, attach an exec-able string as an 
attribute to our C functions.  Further, those pure python examples should include doctests so that the user can see a typical 
invocation and calling pattern.

Say we decide to call the attribute something like ".python", then you could write something like:

    >>> print(all.python)
   def all(iterable):
        '''Return True if all elements of the iterable are true.

        >>> all(isinstance(x, int) for x in [2, 4, 6.13, 8])
        False
        >>> all(isinstance(x, int) for x in [2, 4, 6, 8])
        True
        '''

        for element in iterable:
            if not element:
                 return False
        return True

There you have it, a docstring, doctestable examples, and pure python equivalent all in one place.  And since the attribute is 
distinguished from __doc__, we can insist that the string be exec-able (something we can't insist on for arbitrary docstrings).

Benefits
--------

* I think this will greatly improve the understanding of tools like str.split() which have proven to be difficult to document with 
straight prose.  Even with simple things like any() and all(), it makes it self-evident that the functions have early-out behavior 
upon hitting the first mismatch.

* The exec-able definitions and docstrings will be testable

* It will assist pypy style projects and other python implementations when they have to build equivalents to CPython.

* We've gotten good benefits from doctests for pure python functions, why not extend this best practice to our C functions.

* The whole language will become more self-explanatory and self-documenting.

* Will eliminate confusion about what functions were exactly intended to do.

* Will confer benefits similar to test driven development where the documentation and  pure python version are developed first and 
doctests gotten to pass, then the C version is created to match.

* For existing code, this is a perfect project for people who want to start contributing to the language but aren't ready to start 
writing C (the should be able to read C however so that the equivalent really does match the C code).


Limits
-----

* In some cases, there may be no pure python equivalent (i.e. sys.getsize()).

* Sometimes the equivalent can only be approximate because the actual C function is too complex (i.e. itertools.tee()).

* Some cases, like int(), are useful as a type, have multiple functions, and are hard to write as pure python equivalents.

* For starters, it probably only makes to do this for things that are more "algorithmic" like any() and all() or things that have a 
straight-forward equivalent like property() written using descriptors.


Premise
-------

Sometimes pure python is more expressive, precise, and easy to read than English prose. 


From martin at v.loewis.de  Fri Oct 10 01:12:47 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 10 Oct 2008 01:12:47 +0200
Subject: [Python-Dev] [Python-checkins] r66863
	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>	<48EE4C41.30905@v.loewis.de>
	<gclkpu$93f$1@ger.gmane.org>	<48EE72DB.1000002@v.loewis.de>
	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>
Message-ID: <48EE8FEF.20005@v.loewis.de>

> It seems to me that Skip was asking whether the "memory leak" impacted
> the 2.6 branch, and the answer should have been "No": the change that
> introduced the memory leak had just been committed 10 minutes before.

You are probably right (although it's not quite clear from Skip's question).

> - Because of this misunderstanding, the changes to this
> GetCurrentDirectoryW were backported to the release2.6 branch, despite
> the fact that it's not a regression from a previous version, the NEWS
> entry explicitly expresses doubts about the correction (which I happen
> to share), there is no unit test and no item in the issue tracker.

I think it is fine that this fix was backported (assuming, without
review, that the fix is actually correct).

It is a bugfix, and it shouldn't realistically break existing applications.

IOW, PEP 6 was followed (except that there is no Patch Czar).

> - The backport to release26-maint http://svn.python.org/view?rev=66865&view=rev
> also merged other changes (new unrelated unit tests). IMO unrelated
> changes should be committed separately: different commit messages help
> to understand the motivation of each backport.

Yes, that is unfortunate.

I'm skeptical that new tests actually need backporting at all. Python
doesn't really get better by new tests being added to an old branch.
Near-term, it might get worse because the new tests might cause false
positives, making users worried for no reason.

Regards,
Martin

From glyph at divmod.com  Fri Oct 10 01:35:37 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 09 Oct 2008 23:35:37 -0000
Subject: [Python-Dev] [Python-checkins] r66863
	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <48EE8FEF.20005@v.loewis.de>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>
	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>
	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>
	<48EE4C41.30905@v.loewis.de> <gclkpu$93f$1@ger.gmane.org>
	<48EE72DB.1000002@v.loewis.de>
	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>
	<48EE8FEF.20005@v.loewis.de>
Message-ID: <20081009233537.420.1440384488.divmod.xquotient.68@weber.divmod.com>

On 11:12 pm, martin at v.loewis.de wrote:
>>- The backport to release26-maint 
>>http://svn.python.org/view?rev=66865&view=rev
>>also merged other changes (new unrelated unit tests).

>>IMO unrelated
>>changes should be committed separately: different commit messages help
>>to understand the motivation of each backport.

This I agree with completely, but...
>I'm skeptical that new tests actually need backporting at all. Python
>doesn't really get better by new tests being added to an old branch.

Presumably if the new tests cover functionality that somebody cares 
about, it would be valuable to make sure that maintenance bugfixes don't 
break this functionality in maintenance branches either?  In fact, I 
would think that maintenance branches would want to be much _more_ 
paranoid about making sure that changes don't break anything.

(Again: not specific to these unrelated tests that are getting merged, 
it sounds like accidentally - it just seems strange, as a general 
principle or policy, to say that maintenance branches shouldn't have new 
tests added to them.)

From lists at cheimes.de  Fri Oct 10 01:33:00 2008
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 10 Oct 2008 01:33:00 +0200
Subject: [Python-Dev] Documentation idea
In-Reply-To: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
Message-ID: <gcm4bb$7p6$1@ger.gmane.org>

Raymond Hettinger wrote: lots of cool stuff!

The idea sounds great!

Are you planing to embed the pure python code in C code? That's going to 
increase the data segment of the executable. It should be possible to 
disable and remove the pure python example with a simple ./configure 
option and some macro magic. File size and in memory size is still 
critical for embedders.

Christian


From python at rcn.com  Fri Oct 10 01:50:13 2008
From: python at rcn.com (Raymond Hettinger)
Date: Thu, 9 Oct 2008 16:50:13 -0700
Subject: [Python-Dev] Documentation idea
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<gcm4bb$7p6$1@ger.gmane.org>
Message-ID: <70B3E5AD8A1443A3B979C4DECA916238@RaymondLaptop1>

[Christian Heimes]
> The idea sounds great!
>
> Are you planing to embed the pure python code in C code?

Am experimenting with a descriptor that fetches the attribute string from a separate text file.  This keeps the C build from getting 
fat.  More importantly, it let's us write the execable string in a more natural way (it bites to write C style docstrings using \n 
and trailing backslashes).  The best part is that people without C compilers can still submit patches to the text files.


Raymond 


From greg.ewing at canterbury.ac.nz  Fri Oct 10 02:15:48 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 10 Oct 2008 13:15:48 +1300
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <48EDF0C1.5060705@gmail.com>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
	<48ED280F.6000302@gmail.com>
	<4E9372E6B2234D4F859320D896059A95101822E1DC@exchis.ccp.ad.local>
	<48EDF0C1.5060705@gmail.com>
Message-ID: <48EE9EB4.5040004@canterbury.ac.nz>

Nick Coghlan wrote:

> If the time is being spent in PyErr_Format, how far could you get adding
> a dedicated function for creating AttributeErrors? Something along the
> lines of:
> 
> PyErr_AttributeError(PyObject *object, PyObject *attr_name)

More generally, it might be useful to have some mechanism for
deferred instantiation of exceptions, so you can do something
like indicate what type of exception you want to raise, and
specify a function and some arguments to call to instantiate
the exception, but the instantiation itself doesn't happen
unless the exception object is actually needed by Python
code.

-- 
Greg

From amauryfa at gmail.com  Fri Oct 10 02:28:49 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 10 Oct 2008 02:28:49 +0200
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <48EE9EB4.5040004@canterbury.ac.nz>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
	<48ED280F.6000302@gmail.com>
	<4E9372E6B2234D4F859320D896059A95101822E1DC@exchis.ccp.ad.local>
	<48EDF0C1.5060705@gmail.com> <48EE9EB4.5040004@canterbury.ac.nz>
Message-ID: <e27efe130810091728r57af8c3fo43690fb532b1a6e8@mail.gmail.com>

Hello,

On Fri, Oct 10, 2008 at 2:15 AM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nick Coghlan wrote:
>
>> If the time is being spent in PyErr_Format, how far could you get adding
>> a dedicated function for creating AttributeErrors? Something along the
>> lines of:
>>
>> PyErr_AttributeError(PyObject *object, PyObject *attr_name)
>
> More generally, it might be useful to have some mechanism for
> deferred instantiation of exceptions, so you can do something
> like indicate what type of exception you want to raise, and
> specify a function and some arguments to call to instantiate
> the exception, but the instantiation itself doesn't happen
> unless the exception object is actually needed by Python
> code.

But this is already the case, and the reason why there are three
variable to describe an exception: type, value and traceback.
At the C level, the value is often a string (when using PyErr_Format,
for example), or a tuple.
Normalization (=creation of the exception object) only occurs when
needed, e.g when entering an "except:" handler in python code, or when
the exception is printed.
However, the "value" string object must be created anyway, and this
could be avoided in most Getattr cases.

-- 
Amaury Forgeot d'Arc

From greg.ewing at canterbury.ac.nz  Fri Oct 10 02:37:32 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 10 Oct 2008 13:37:32 +1300
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <e27efe130810091728r57af8c3fo43690fb532b1a6e8@mail.gmail.com>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
	<48ED280F.6000302@gmail.com>
	<4E9372E6B2234D4F859320D896059A95101822E1DC@exchis.ccp.ad.local>
	<48EDF0C1.5060705@gmail.com> <48EE9EB4.5040004@canterbury.ac.nz>
	<e27efe130810091728r57af8c3fo43690fb532b1a6e8@mail.gmail.com>
Message-ID: <48EEA3CC.7090704@canterbury.ac.nz>

Amaury Forgeot d'Arc wrote:

> But this is already the case, and the reason why there are three
> variable to describe an exception: type, value and traceback.

Yes, but you only get one object for the value, which means
at least allocating a tuple if you want to be able to report
something like "AttributeError: 'fooblat' object has no
attribute 'asdf'".

I'm thinking about a C api that would let you supply multiple
pre-existing values without having to allocate anything.
Not sure exactly how it would work, though.

Also, what about Py3? Has the type/value separation gone away
completely, or is it still there at the C level?

-- 
Greg

From stephen at xemacs.org  Fri Oct 10 03:22:03 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 10 Oct 2008 10:22:03 +0900
Subject: [Python-Dev] [Python-checkins]
	r66863	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <48EE8FEF.20005@v.loewis.de>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>
	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>
	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>
	<48EE4C41.30905@v.loewis.de> <gclkpu$93f$1@ger.gmane.org>
	<48EE72DB.1000002@v.loewis.de>
	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>
	<48EE8FEF.20005@v.loewis.de>
Message-ID: <87skr5xn0k.fsf@xemacs.org>

"Martin v. L?wis" writes:

 > I'm skeptical that new tests actually need backporting at all. Python
 > doesn't really get better by new tests being added to an old branch.
 > Near-term, it might get worse because the new tests might cause false
 > positives, making users worried for no reason.

If they do fail, they're not "false" positives.  If they're "false",
then the test is broken, no?  So find a way to label them as tests
added ex-post, with the failures *not* being regressions but rather
latent bugs newly detected, and (presumably) as "wont-fix".

>From a QA point of view one would like to be able to assess how many
latent bugs are making it through to end-of-life.  The new tests will
help in that.

From brett at python.org  Fri Oct 10 03:41:26 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 9 Oct 2008 18:41:26 -0700
Subject: [Python-Dev] Documentation idea
In-Reply-To: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
Message-ID: <bbaeab100810091841x7ecfac59n835e4f0df483cabf@mail.gmail.com>

On Thu, Oct 9, 2008 at 4:12 PM, Raymond Hettinger <python at rcn.com> wrote:
[SNIP]
> Bright idea
> ----------
> Let's go one step further and do this just about everywhere and instead of
> putting it in the docs, attach an exec-able string as an attribute to our C
> functions.  Further, those pure python examples should include doctests so
> that the user can see a typical invocation and calling pattern.
>
> Say we decide to call the attribute something like ".python", then you could
> write something like:
>
>   >>> print(all.python)
>  def all(iterable):
>       '''Return True if all elements of the iterable are true.
>
>       >>> all(isinstance(x, int) for x in [2, 4, 6.13, 8])
>       False
>       >>> all(isinstance(x, int) for x in [2, 4, 6, 8])
>       True
>       '''
>
>       for element in iterable:
>           if not element:
>                return False
>       return True
>
> There you have it, a docstring, doctestable examples, and pure python
> equivalent all in one place.  And since the attribute is distinguished from
> __doc__, we can insist that the string be exec-able (something we can't
> insist on for arbitrary docstrings).
>

The idea is great. I assume the special file support is mostly for the
built-ins since extension modules can do what heapq does; have a pure
Python version people import and that code pulls in any supporting C
code.

As for an implementation, you could go as far as to have a flag in the
extension module that says, "look for Python equivalents" and during
module initialization find the file and pull it in. Although doing it
that way would not necessarily encourage people as much to start with
the pure Python version and then only do C equivalents when
performance or design requires it.

> Benefits
> --------
>
> * I think this will greatly improve the understanding of tools like
> str.split() which have proven to be difficult to document with straight
> prose.  Even with simple things like any() and all(), it makes it
> self-evident that the functions have early-out behavior upon hitting the
> first mismatch.
>
> * The exec-able definitions and docstrings will be testable
>

And have some way to test both the Python and C version with the same
tests (when possible)?

-Brett

From brett at python.org  Fri Oct 10 03:43:33 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 9 Oct 2008 18:43:33 -0700
Subject: [Python-Dev] __getattr__ and new style classes
In-Reply-To: <48EEA3CC.7090704@canterbury.ac.nz>
References: <4E9372E6B2234D4F859320D896059A95101822DFA4@exchis.ccp.ad.local>
	<48ED280F.6000302@gmail.com>
	<4E9372E6B2234D4F859320D896059A95101822E1DC@exchis.ccp.ad.local>
	<48EDF0C1.5060705@gmail.com> <48EE9EB4.5040004@canterbury.ac.nz>
	<e27efe130810091728r57af8c3fo43690fb532b1a6e8@mail.gmail.com>
	<48EEA3CC.7090704@canterbury.ac.nz>
Message-ID: <bbaeab100810091843v56e9ba48jbae271a4d920be94@mail.gmail.com>

On Thu, Oct 9, 2008 at 5:37 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Amaury Forgeot d'Arc wrote:
>
>> But this is already the case, and the reason why there are three
>> variable to describe an exception: type, value and traceback.
>
> Yes, but you only get one object for the value, which means
> at least allocating a tuple if you want to be able to report
> something like "AttributeError: 'fooblat' object has no
> attribute 'asdf'".
>
> I'm thinking about a C api that would let you supply multiple
> pre-existing values without having to allocate anything.
> Not sure exactly how it would work, though.
>
> Also, what about Py3? Has the type/value separation gone away
> completely, or is it still there at the C level?
>

It's still there. I think the major C changes for exceptions was
adding support for the three new attributes (__traceback__,
__context__, and __cause__).

-Brett

From dalcinl at gmail.com  Fri Oct 10 04:13:49 2008
From: dalcinl at gmail.com (Lisandro Dalcin)
Date: Thu, 9 Oct 2008 23:13:49 -0300
Subject: [Python-Dev] Documentation idea
In-Reply-To: <70B3E5AD8A1443A3B979C4DECA916238@RaymondLaptop1>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<gcm4bb$7p6$1@ger.gmane.org>
	<70B3E5AD8A1443A3B979C4DECA916238@RaymondLaptop1>
Message-ID: <e7ba66e40810091913y417812a2l9ec31a6f55f1731@mail.gmail.com>

On Thu, Oct 9, 2008 at 8:50 PM, Raymond Hettinger <python at rcn.com> wrote:
> [Christian Heimes]
>>
>> The idea sounds great!
>>
>> Are you planing to embed the pure python code in C code?
>
> Am experimenting with a descriptor that fetches the attribute string from a
> separate text file.

Have you ever considered the same approach for docstrings in C code?
As reference, NumPy already has some trickery for maintaining
docstrings outside C sources. Of course, descriptors would be a far
better for implementing and support this in core Python and other
projects...

>  This keeps the C build from getting fat.  More
> importantly, it let's us write the execable string in a more natural way (it
> bites to write C style docstrings using \n and trailing backslashes).  The
> best part is that people without C compilers can still submit patches to the
> text files.
>
>
> Raymond
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/dalcinl%40gmail.com
>



-- 
Lisandro Dalc?n
---------------
Centro Internacional de M?todos Computacionales en Ingenier?a (CIMEC)
Instituto de Desarrollo Tecnol?gico para la Industria Qu?mica (INTEC)
Consejo Nacional de Investigaciones Cient?ficas y T?cnicas (CONICET)
PTLC - G?emes 3450, (3000) Santa Fe, Argentina
Tel/Fax: +54-(0)342-451.1594

From python at rcn.com  Fri Oct 10 04:39:47 2008
From: python at rcn.com (Raymond Hettinger)
Date: Thu, 9 Oct 2008 19:39:47 -0700
Subject: [Python-Dev] Documentation idea
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<gcm4bb$7p6$1@ger.gmane.org>
	<70B3E5AD8A1443A3B979C4DECA916238@RaymondLaptop1>
	<e7ba66e40810091913y417812a2l9ec31a6f55f1731@mail.gmail.com>
Message-ID: <B8F9195FCB7848C59E84002982188364@RaymondLaptop1>

Yes, I'm looking a couple of different approaches to loading the strings.
For now though, I want to focus on the idea itself, not the implementation.
The important thing is to gather widespread support before getting into
the details of how the strings get loaded.


Raymond



----- Original Message ----- 
From: "Lisandro Dalcin" <dalcinl at gmail.com>

Have you ever considered the same approach for docstrings in C code?
As reference, NumPy already has some trickery for maintaining
docstrings outside C sources. Of course, descriptors would be a far
better for implementing and support this in core Python and other
projects...

>  This keeps the C build from getting fat.  More


From skip at pobox.com  Fri Oct 10 05:06:26 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 9 Oct 2008 22:06:26 -0500
Subject: [Python-Dev] test message - please ignore
Message-ID: <18670.50866.285700.681531@montanaro-dyndns-org.local>

(messing with the python.org spam filter - please ignore)

Skip

From skip at pobox.com  Fri Oct 10 05:10:43 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 9 Oct 2008 22:10:43 -0500
Subject: [Python-Dev] another test message - please ignore
Message-ID: <18670.51123.751853.19298@montanaro-dyndns-org.local>

still working on spam filter...

Skip

From glyph at divmod.com  Fri Oct 10 05:37:24 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 10 Oct 2008 03:37:24 -0000
Subject: [Python-Dev] Documentation idea
In-Reply-To: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
Message-ID: <20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>


On 9 Oct, 11:12 pm, python at rcn.com wrote:
>Background
>----------
>In the itertools module docs, I included pure python equivalents for 
>each of the C functions.  Necessarily, some of those equivalents are 
>only approximate but they seem to have greatly enhanced the docs.

Why not go the other direction?

Ostensibly the reason for writing a module like 'itertools' in C is 
purely for performance.  There's nothing that I'm aware of in that 
module which couldn't be in Python.

Similarly, cStringIO, cPickle, etc.  Everywhere these diverge, it is (if 
not a flat-out bug) not optimal.  External projects are encouraged by a 
wealth of documentation to solve performance problems in a similar way: 
implement in Python, once you've got the interface right, optimize into 
C.

So rather than have a C implementation, which points to Python, why not 
have a Python implementation that points at C?  'itertools' (and 
similar) can actually be Python modules, and use a decorator, let's call 
it "C", to do this:

    @C("_c_itertools.count")
    class count(object):
        """
        This is the documentation for both the C version of 
itertools.count
        and the Python version - since they should be the same, right?
        """

In Python itself, the Python module would mostly be for documentation, 
and therefore solve the problem that Raymond is talking about, but it 
could also be a handy fallback for sanity checking, testing, and use in 
other Python runtimes (ironpython, jython, pypy).  Many third-party 
projects already use ad-hoc mechanisms for doing this same thing, but an 
officially-supported way of saying "this works, but the optimized 
version is over here" would be a very useful convention.

In those modules which absolutely require some C stuff to work, the 
python module could still serve as documentation.

From jared.grubb at gmail.com  Fri Oct 10 05:54:30 2008
From: jared.grubb at gmail.com (Jared Grubb)
Date: Thu, 9 Oct 2008 20:54:30 -0700
Subject: [Python-Dev] Documentation idea
In-Reply-To: <20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>
Message-ID: <575C983D-AD00-4124-8F45-93A0AFCE6415@gmail.com>

This is a really interesting idea. If extra memory/lookup overhead is  
a concern, you could enable this new feature by default when the  
interactive interpreter is started (where it's more likely to be  
invoked), and turn it off by default when running scripts/modules.

Jared

On 9 Oct 2008, at 20:37, glyph at divmod.com wrote:

>
> On 9 Oct, 11:12 pm, python at rcn.com wrote:
>> Background
>> ----------
>> In the itertools module docs, I included pure python equivalents  
>> for each of the C functions.  Necessarily, some of those  
>> equivalents are only approximate but they seem to have greatly  
>> enhanced the docs.
>
> Why not go the other direction?
>
> Ostensibly the reason for writing a module like 'itertools' in C is  
> purely for performance.  There's nothing that I'm aware of in that  
> module which couldn't be in Python.
>
> Similarly, cStringIO, cPickle, etc.  Everywhere these diverge, it is  
> (if not a flat-out bug) not optimal.  External projects are  
> encouraged by a wealth of documentation to solve performance  
> problems in a similar way: implement in Python, once you've got the  
> interface right, optimize into C.
>
> So rather than have a C implementation, which points to Python, why  
> not have a Python implementation that points at C?  'itertools' (and  
> similar) can actually be Python modules, and use a decorator, let's  
> call it "C", to do this:
>
>   @C("_c_itertools.count")
>   class count(object):
>       """
>       This is the documentation for both the C version of  
> itertools.count
>       and the Python version - since they should be the same, right?
>       """
>
> In Python itself, the Python module would mostly be for  
> documentation, and therefore solve the problem that Raymond is  
> talking about, but it could also be a handy fallback for sanity  
> checking, testing, and use in other Python runtimes (ironpython,  
> jython, pypy).  Many third-party projects already use ad-hoc  
> mechanisms for doing this same thing, but an officially-supported  
> way of saying "this works, but the optimized version is over here"  
> would be a very useful convention.
>
> In those modules which absolutely require some C stuff to work, the  
> python module could still serve as documentation.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/jared.grubb%40gmail.com


From martin at v.loewis.de  Fri Oct 10 07:33:22 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 10 Oct 2008 07:33:22 +0200
Subject: [Python-Dev] [Python-checkins]
	r66863	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <87skr5xn0k.fsf@xemacs.org>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>	<48EE4C41.30905@v.loewis.de>	<gclkpu$93f$1@ger.gmane.org>	<48EE72DB.1000002@v.loewis.de>	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>	<48EE8FEF.20005@v.loewis.de>
	<87skr5xn0k.fsf@xemacs.org>
Message-ID: <48EEE922.9010705@v.loewis.de>

> If they do fail, they're not "false" positives.  If they're "false",
> then the test is broken, no?

Correct. But they might well be broken, no?

> So find a way to label them as tests
> added ex-post, with the failures *not* being regressions but rather
> latent bugs newly detected, and (presumably) as "wont-fix".

No such way exists, hence they can't be labeled that way.
No such labeling can be added, since that would be a new feature.

Regards,
Martin

From martin at v.loewis.de  Fri Oct 10 07:42:11 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 10 Oct 2008 07:42:11 +0200
Subject: [Python-Dev] [Python-checkins]
	r66863	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <20081009233537.420.1440384488.divmod.xquotient.68@weber.divmod.com>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>	<48EE4C41.30905@v.loewis.de>
	<gclkpu$93f$1@ger.gmane.org>	<48EE72DB.1000002@v.loewis.de>	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>	<48EE8FEF.20005@v.loewis.de>
	<20081009233537.420.1440384488.divmod.xquotient.68@weber.divmod.com>
Message-ID: <48EEEB33.3060505@v.loewis.de>

> Presumably if the new tests cover functionality that somebody cares
> about, it would be valuable to make sure that maintenance bugfixes don't
> break this functionality in maintenance branches either?

Yes. At the same time, there is also a risk that the new tests just fail
because they are not correctly formulated. Users don't report that
against the trunk, because they are not running the trunk. Instead,
they download the next maintenance release, run the tests, see that it
fails, get frightened, and return to the prior release (which didn't
have any failing tests).

There is a certain prevention already that later maintenance fixes don't
break the earlier ones: those fixes typically get checked into the trunk
also, where the tests do exist. So the committer would find out even
before the patch gets to the maintenance branch.

All that said, I don't want to state a policy about whether new tests
should or should not get added to maintenance branches. I'd leave this
up to the committers, and just wanted to caution about this risk.

Regards,
Martin

From andrew-pythondev at puzzling.org  Fri Oct 10 08:29:26 2008
From: andrew-pythondev at puzzling.org (Andrew Bennetts)
Date: Fri, 10 Oct 2008 17:29:26 +1100
Subject: [Python-Dev] [Python-checkins] r66863
	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <48EEEB33.3060505@v.loewis.de>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>
	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>
	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>
	<48EE4C41.30905@v.loewis.de> <gclkpu$93f$1@ger.gmane.org>
	<48EE72DB.1000002@v.loewis.de>
	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>
	<48EE8FEF.20005@v.loewis.de>
	<20081009233537.420.1440384488.divmod.xquotient.68@weber.divmod.com>
	<48EEEB33.3060505@v.loewis.de>
Message-ID: <20081010062926.GD32624@steerpike.home.puzzling.org>

"Martin v. L?wis" wrote:
[...]
> There is a certain prevention already that later maintenance fixes don't
> break the earlier ones: those fixes typically get checked into the trunk
> also, where the tests do exist. So the committer would find out even
> before the patch gets to the maintenance branch.

By this logic, the maintenance branch would need no tests at all,
because they are all in trunk already!

-Andrew.


From martin at v.loewis.de  Fri Oct 10 08:44:38 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 10 Oct 2008 08:44:38 +0200
Subject: [Python-Dev] [Python-checkins] r66863
	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <20081010062926.GD32624@steerpike.home.puzzling.org>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>
	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>
	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>
	<48EE4C41.30905@v.loewis.de> <gclkpu$93f$1@ger.gmane.org>
	<48EE72DB.1000002@v.loewis.de>
	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>
	<48EE8FEF.20005@v.loewis.de>
	<20081009233537.420.1440384488.divmod.xquotient.68@weber.divmod.com>
	<48EEEB33.3060505@v.loewis.de>
	<20081010062926.GD32624@steerpike.home.puzzling.org>
Message-ID: <48EEF9D6.5070403@v.loewis.de>

Andrew Bennetts wrote:
> "Martin v. L?wis" wrote:
> [...]
>> There is a certain prevention already that later maintenance fixes don't
>> break the earlier ones: those fixes typically get checked into the trunk
>> also, where the tests do exist. So the committer would find out even
>> before the patch gets to the maintenance branch.
> 
> By this logic, the maintenance branch would need no tests at all,
> because they are all in trunk already!

No, this is not the logic. The tests in the maintenance branch have gone
through alpha and beta releases, so end users have seen them already (at
least, some of them). Of course, it might still be possible that there
is an incorrect test in the released version; those can get fixed in
later maintenance releases.

So 2.6.0 will contain a lot of tests that have never been tested in
a wide variety of systems. Some are incorrect, and get fixed in 2.6.1,
and stay fixed afterwards. This is completely different from somebody
introducing a new test in 2.6.4. It means that there are more failures
in a maintenance release, not less as in the first case.

Of course, it might be that a developer deliberately wants to see
a certain test be run in the field, because there is a uncertainty
whether the fix actually works. However, it is questionable whether
such a fix would belong in the maintenance release.

Regards,
Martin

From stephen at xemacs.org  Fri Oct 10 09:02:45 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Fri, 10 Oct 2008 16:02:45 +0900
Subject: [Python-Dev]
	[Python-checkins]	r66863	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <48EEE922.9010705@v.loewis.de>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>
	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>
	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>
	<48EE4C41.30905@v.loewis.de> <gclkpu$93f$1@ger.gmane.org>
	<48EE72DB.1000002@v.loewis.de>
	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>
	<48EE8FEF.20005@v.loewis.de> <87skr5xn0k.fsf@xemacs.org>
	<48EEE922.9010705@v.loewis.de>
Message-ID: <87hc7lx78q.fsf@xemacs.org>

"Martin v. L?wis" writes:

 > > If they do fail, they're not "false" positives.  If they're "false",
 > > then the test is broken, no?
 > 
 > Correct. But they might well be broken, no?

I would hope some effort is made that they not be.  If they generate a
positive, I would expect that the contributor would try to fix that
before committing, no?  If they discover that it's "false", they fix
or remove the test; otherwise they document it.

 > > So find a way to label them as tests
 > > added ex-post, with the failures *not* being regressions but rather
 > > latent bugs newly detected, and (presumably) as "wont-fix".
 > 
 > No such way exists,

Add a documentation file called "README.expected-test-failures".
AFAIK documentation is always acceptable, right?

Whether that is an acceptable solution to the "latent bug" problem is
a different question.  I'd rather know that Python has unexpected
behavior, and have a sample program (== test) to demonstrate it, than
not.  YMMV.


From martin at v.loewis.de  Fri Oct 10 09:32:24 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 10 Oct 2008 09:32:24 +0200
Subject: [Python-Dev]
	[Python-checkins]	r66863	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <87hc7lx78q.fsf@xemacs.org>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>	<48EE4C41.30905@v.loewis.de>
	<gclkpu$93f$1@ger.gmane.org>	<48EE72DB.1000002@v.loewis.de>	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>	<48EE8FEF.20005@v.loewis.de>
	<87skr5xn0k.fsf@xemacs.org>	<48EEE922.9010705@v.loewis.de>
	<87hc7lx78q.fsf@xemacs.org>
Message-ID: <48EF0508.6030009@v.loewis.de>

>  > Correct. But they might well be broken, no?
> 
> I would hope some effort is made that they not be.  If they generate a
> positive, I would expect that the contributor would try to fix that
> before committing, no?  If they discover that it's "false", they fix
> or remove the test; otherwise they document it.

That assumes they know. We had recently a number of test cases that
fixed security problems, and the tests would only run correctly on
32-bit systems. On 64-bit systems, they would consume all memory,
and either bring the machine down, or complete eventually with a failure
(because the expected out-of-memory situation did not arise). The author
of the code was unaware of its dependency on the architecture, and the
test would run fine on his machine.

Likewise, we had test failures that only failed if a certain locale
was not available on a system, or had locale definitions that were
different from the ones on Linux. There is a lot of potential for
tests to only fail on systems which we don't have access to.

> Whether that is an acceptable solution to the "latent bug" problem is
> a different question.  I'd rather know that Python has unexpected
> behavior, and have a sample program (== test) to demonstrate it, than
> not.  YMMV.

And it does indeed for many people. We get a significant number of
reports from people who find that the test suite fails, and then are
unable to draw any conclusions from that other than Python is apparently
broken, and that they therefore shouldn't use it - even if the test
that fails is in a module that they likely never use.

Regards,
Martin

From jimjjewett at gmail.com  Fri Oct 10 17:37:03 2008
From: jimjjewett at gmail.com (Jim Jewett)
Date: Fri, 10 Oct 2008 11:37:03 -0400
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
Message-ID: <fb6fbf560810100837o7e7e256fxba8a7b6afd2efa08@mail.gmail.com>

> For the search engine issue, is there any way we can tell robots to
> ignore the rewrite rules so they see the broken links? (although even
> that may not be ideal, since what we really want is to tell the robot
> the link is broken, and provide the new alternative)

I may be missing something obvious, but isn't this the exact intent of

HTTP response code 301 Moved Permanently

http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.3.2

-jJ

From amk at amk.ca  Fri Oct 10 17:44:51 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Fri, 10 Oct 2008 11:44:51 -0400
Subject: [Python-Dev] [Python-checkins] r66863
	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <48EEF9D6.5070403@v.loewis.de>
References: <4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>
	<48EE4C41.30905@v.loewis.de> <gclkpu$93f$1@ger.gmane.org>
	<48EE72DB.1000002@v.loewis.de>
	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>
	<48EE8FEF.20005@v.loewis.de>
	<20081009233537.420.1440384488.divmod.xquotient.68@weber.divmod.com>
	<48EEEB33.3060505@v.loewis.de>
	<20081010062926.GD32624@steerpike.home.puzzling.org>
	<48EEF9D6.5070403@v.loewis.de>
Message-ID: <20081010154451.GA9433@amk-desktop.matrixgroup.net>

On Fri, Oct 10, 2008 at 08:44:38AM +0200, "Martin v. L?wis" wrote:
> So 2.6.0 will contain a lot of tests that have never been tested in
> a wide variety of systems. Some are incorrect, and get fixed in 2.6.1,
> and stay fixed afterwards. This is completely different from somebody
> introducing a new test in 2.6.4. It means that there are more failures
> in a maintenance release, not less as in the first case.

Indeed.  Looking at the commit logs, I had to split out all the
test-only commits into a separate list, and tests often took several
attempts to get working on all platforms.  I think backporting should
be prioritized into 1) bug fixes, 2) documentation improvements, 3)
tests.  For 2.5.3, we should just ignore 2) and 3); documentation
patches will require rewriting from reST into LaTeX, which is too much
effort for the return, and tests are even less useful to the end
users, being primarily useful for Python's developers.

(2.5.3 reminder: there are lists of commits in sandbox/py2.5.3 to be
considered.  I've seen no reactions on python-dev or modifications to
those files, so I don't think anyone else is looking at them.  Is
everyone waiting for the weekend, maybe?)

--amk

From jimjjewett at gmail.com  Fri Oct 10 17:51:59 2008
From: jimjjewett at gmail.com (Jim Jewett)
Date: Fri, 10 Oct 2008 11:51:59 -0400
Subject: [Python-Dev] syntax change justification
Message-ID: <fb6fbf560810100851g49c9204eqa12bc1927368aef8@mail.gmail.com>

Nick Coghlan's explanation of what justifies a syntax change (most of message
http://mail.python.org/pipermail/python-dev/2008-October/082831.html )
should probably be added to the standard docs/FAQs somewhere.

At the moment, I'm not sure exactly where, though.  At the moment, the
Developer FAQ (http://www.python.org/dev/faq/)  is mostly about using
specific tools (rather than design philosophy), and Nick's explanation
may be too detailed for the current Explanations section of
www.python.org/dev/

Possibly as a Meta-PEP?

-jJ

From status at bugs.python.org  Fri Oct 10 18:06:38 2008
From: status at bugs.python.org (Python tracker)
Date: Fri, 10 Oct 2008 18:06:38 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20081010160638.2122378547@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (10/03/08 - 10/10/08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2105 open (+49) / 13818 closed (+21) / 15923 total (+70)

Open issues with patches:   690

Average duration of open issues: 707 days.
Median duration of open issues: 1884 days.

Open Issues Breakdown
   open  2089 (+49)
pending    16 ( +0)

Issues Created Or Reopened (73)
_______________________________

zlib.crc32() and adler32() return value                          10/06/08
       http://bugs.python.org/issue1202    reopened facundobatista            
       64bit, easy                                                             

Fix test_cProfile                                                10/06/08
       http://bugs.python.org/issue2744    reopened benjamin.peterson         
                                                                               

"make check" suggest a testing target under GNU coding standards 10/03/08
CLOSED http://bugs.python.org/issue3758    reopened brett.cannon              
       patch                                                                   

08 value popups an stdin error, no date handle allowed           10/03/08
CLOSED http://bugs.python.org/issue4031    created  pgimelli                  
                                                                               

disutils cannot recognize ".dll.a" as library on cygwin          10/03/08
       http://bugs.python.org/issue4032    created  ocean-city                
                                                                               

python search path - .pth recursion                              10/03/08
       http://bugs.python.org/issue4033    created  jolleyjoe                 
                                                                               

traceback attribute error                                        10/03/08
       http://bugs.python.org/issue4034    created  ghazel                    
       patch, needs review                                                     

Support bytes for os.exec*()                                     10/03/08
       http://bugs.python.org/issue4035    created  haypo                     
       patch, patch                                                            

Support bytes for subprocess.Popen()                             10/03/08
       http://bugs.python.org/issue4036    created  haypo                     
       patch, patch                                                            

doctest.py should include method descriptors when looking inside 10/04/08
       http://bugs.python.org/issue4037    created  daaku                     
                                                                               

py3k error in distutils file_copy exception handlers             10/04/08
CLOSED http://bugs.python.org/issue4038    created  mhammond                  
       patch, patch                                                            

Add __enter__ and __exit__ methods to StringIO/cStringIO classes 10/04/08
CLOSED http://bugs.python.org/issue4039    created  peppergrower              
                                                                               

ignored exceptions in generators (regression?)                   10/04/08
       http://bugs.python.org/issue4040    created  benjamin.peterson         
                                                                               

reference to rexec in __import__                                 10/04/08
CLOSED http://bugs.python.org/issue4041    created  wplappert                 
                                                                               

IDLE won't start in 2.6 final. A known fix was overlooked?       10/04/08
CLOSED http://bugs.python.org/issue4042    created  rbtyod                    
                                                                               

Warnings in IDLE raise TypeError (such as attempting to import d 10/04/08
       http://bugs.python.org/issue4043    created  craigh                    
                                                                               

test_output_textcalendar fails on non-englisch locale            10/05/08
       http://bugs.python.org/issue4044    created  oefe                      
                                                                               

test_mboxmmdf_to_maildir fails on non-englisch locale            10/05/08
       http://bugs.python.org/issue4045    created  oefe                      
                                                                               

test_formatdate_usegmt fails on non-englisch locale              10/05/08
       http://bugs.python.org/issue4046    created  oefe                      
                                                                               

test_run_abort triggers CrashReporter on MacOS X                 10/05/08
       http://bugs.python.org/issue4047    created  oefe                      
                                                                               

parsermodule won't validate relative imports                     10/05/08
       http://bugs.python.org/issue4048    created  dbinger                   
       patch                                                                   

IDLE does not open at all                                        10/06/08
       http://bugs.python.org/issue4049    created  Chris_L                   
                                                                               

inspect.findsource() returns binary data for shared library modu 10/06/08
       http://bugs.python.org/issue4050    created  brodierao                 
       patch                                                                   

use of TCHAR under win32                                         10/06/08
       http://bugs.python.org/issue4051    created  eckhardt                  
       patch                                                                   

Build on 2.6 on AIX 5.3 fails (syntax error / undeclared identif 10/06/08
CLOSED http://bugs.python.org/issue4052    created  gerard                    
                                                                               

str.split unintentionally strips char 'I' from the string        10/06/08
CLOSED http://bugs.python.org/issue4053    created  Govind                    
                                                                               

str.split unintentionally strips char 'I' from the string        10/06/08
CLOSED http://bugs.python.org/issue4054    created  Govind                    
                                                                               

Documentation on website is missing section numbers              10/06/08
CLOSED http://bugs.python.org/issue4055    created  ivazquez                  
                                                                               

:Class: causes exception                                         10/06/08
CLOSED http://bugs.python.org/issue4056    created  ianb                      
                                                                               

Popen(..., cwd=...) does not set PWD environment variable        10/06/08
       http://bugs.python.org/issue4057    created  tebeka                    
                                                                               

markup in What's New in 2.6                                      10/06/08
CLOSED http://bugs.python.org/issue4058    created  lehmannro                 
       patch                                                                   

sqlite3 docs incomplete                                          10/06/08
CLOSED http://bugs.python.org/issue4059    created  lehmannro                 
       patch                                                                   

PyUnicode_DecodeUTF16(..., byteorder=0) gets it wrong on Mac OS  10/06/08
       http://bugs.python.org/issue4060    created  trentm                    
       patch                                                                   

summing two numbers-strange answer                               10/06/08
CLOSED http://bugs.python.org/issue4061    created  mhmtyozcu001              
                                                                               

Reference to non-existent __version__ in ast module              10/06/08
       http://bugs.python.org/issue4062    created  orestis                   
                                                                               

sphinx: make all-pdf does not exist.                             10/07/08
CLOSED http://bugs.python.org/issue4063    created  guettli                   
                                                                               

distutils.util.get_platform() is wrong for universal builds on m 10/07/08
       http://bugs.python.org/issue4064    created  ronaldoussoren            
       patch, needs review                                                     

_multiprocessing doesn't build on macosx 10.3                    10/07/08
       http://bugs.python.org/issue4065    created  ronaldoussoren            
                                                                               

smtplib SMTP_SSL._get_socket doesn't return a value              10/07/08
       http://bugs.python.org/issue4066    created  marcin.bachry             
       patch                                                                   

ast.fix_missing_locations() breaks if node doesn't have "_attrib 10/07/08
       http://bugs.python.org/issue4067    created  marcin.bachry             
       patch                                                                   

Backport fix for issue 3312                                      10/07/08
CLOSED http://bugs.python.org/issue4068    created  ghaering                  
       patch, patch                                                            

set.remove raises confusing KeyError                             10/07/08
CLOSED http://bugs.python.org/issue4069    created  cfbolz                    
       patch                                                                   

python tests failure if builddir <> sourcedir                    10/07/08
       http://bugs.python.org/issue4070    created  rpetrov                   
       patch                                                                   

ntpath.abspath fails for long str paths                          10/08/08
       http://bugs.python.org/issue4071    created  JDay                      
       patch                                                                   

build_py support for lib2to3 is stale                            10/08/08
       http://bugs.python.org/issue4072    created  mhammond                  
       patch                                                                   

distutils build_scripts and install_data commands need 2to3 supp 10/08/08
       http://bugs.python.org/issue4073    created  mhammond                  
                                                                               

Building a list of tuples has non-linear performance             10/08/08
       http://bugs.python.org/issue4074    created  gregory.p.smith           
                                                                               

Use WCHAR variant of OutputDebugString                           10/08/08
       http://bugs.python.org/issue4075    created  eckhardt                  
       patch                                                                   

Cannot build non-framework tkinter Python on Mac OS X.5          10/08/08
CLOSED http://bugs.python.org/issue4076    created  indiedan                  
                                                                               

Py_FatalError cleanup patch                                      10/08/08
       http://bugs.python.org/issue4077    created  eckhardt                  
       patch, easy                                                             

asyncore fixes are not backwards compatible                      10/08/08
       http://bugs.python.org/issue4078    created  sidnei                    
                                                                               

new urllib2.Request 'timeout' attribute needs to have a default  10/08/08
       http://bugs.python.org/issue4079    created  sidnei                    
                                                                               

pyunit - display time of each test case - patch                  10/08/08
       http://bugs.python.org/issue4080    created  pprokop                   
       patch                                                                   

Error copying directory to _static in Sphinx                     10/08/08
       http://bugs.python.org/issue4081    created  tcdelaney                 
                                                                               

python2.6 -m site doesn't run site._script() any more            10/08/08
       http://bugs.python.org/issue4082    created  christian.heimes          
                                                                               

try statement in language reference not updated for v2.6         10/09/08
       http://bugs.python.org/issue4083    created  davipo                    
                                                                               

Decimal.max(NaN, x) gives incorrect results when x is finite and 10/09/08
       http://bugs.python.org/issue4084    created  marketdickinson           
       patch                                                                   

2.5.2 whatsnew document corrupts names, by having broken HTML, a 10/09/08
CLOSED http://bugs.python.org/issue4085    created  drj                       
                                                                               

support %z format in time.strftime and _strptime?                10/09/08
       http://bugs.python.org/issue4086    created  skip.montanaro            
                                                                               

Document the effects of NotImplemented on == and !=              10/09/08
       http://bugs.python.org/issue4087    created  marketdickinson           
                                                                               

Patch to implement a real poplib test suite                      10/09/08
       http://bugs.python.org/issue4088    created  giampaolo.rodola          
       patch                                                                   

linking python2.6.dll crash on windows xp                        10/09/08
       http://bugs.python.org/issue4089    created  Manuel                    
                                                                               

Documenting set comparisons and operations                       10/09/08
       http://bugs.python.org/issue4090    created  tjreedy                   
                                                                               

python dll not installed in windows system directory             10/09/08
       http://bugs.python.org/issue4091    created  theller                   
                                                                               

inspect.getargvalues return type not ArgInfo                     10/09/08
       http://bugs.python.org/issue4092    created  castironpi                
                                                                               

add gc/memory management tests to pybench                        10/09/08
       http://bugs.python.org/issue4093    created  pitrou                    
                                                                               

"Future statements" Doc from 2.6 refers to 2.5                   10/09/08
CLOSED http://bugs.python.org/issue4094    created  martin.marcher            
                                                                               

Delivery Status                                                  10/10/08
CLOSED http://bugs.python.org/issue4095    created  nobody                    
                                                                               

Lib/lib2to3/*.pickle are shipped / modified in the build         10/10/08
       http://bugs.python.org/issue4096    created  doko                      
                                                                               

Traceback doesn't run back all the way                           10/10/08
       http://bugs.python.org/issue4097    created  pinkisntwell              
                                                                               

surprised by default list parameter                              10/10/08
CLOSED http://bugs.python.org/issue4098    created  trott                     
                                                                               

dir on a compiled re does not show pattern as a part of the list 10/10/08
       http://bugs.python.org/issue4099    created  bartoszr                  
                                                                               

xml.etree.ElementTree does not read xml-text over page bonderies 10/10/08
       http://bugs.python.org/issue4100    created  roland                    
                                                                               



Issues Now Closed (44)
______________________

test_socket.py fails                                              297 days
       http://bugs.python.org/issue1610    Tinctorius                
                                                                               

PythonLauncher not working correctly on OS X 10.5/Leopad          261 days
       http://bugs.python.org/issue1905    thelawnmowerman           
                                                                               

[Py3k] line number is wrong after encoding declaration            206 days
       http://bugs.python.org/issue2384    amaury.forgeotdarc        
       patch                                                                   

file that breaks 2to3 (despite being correct python)              185 days
       http://bugs.python.org/issue2532    collinwinter              
                                                                               

os.listdir can return byte strings                                105 days
       http://bugs.python.org/issue3187    loewis                    
       patch                                                                   

[proposal] alternative for re.sub                                  93 days
       http://bugs.python.org/issue3255    ocean-city                
                                                                               

float.as_integer_ratio method is not documented                    95 days
       http://bugs.python.org/issue3288    georg.brandl              
       patch                                                                   

2to3 Iterative Wildcard Matching                                   81 days
       http://bugs.python.org/issue3358    collinwinter              
       patch                                                                   

Fraction and Decimal in the Tutorial                               79 days
       http://bugs.python.org/issue3412    rhettinger                
                                                                               

PEP 3121 --- module state is not nul-initalized as claimed in th   38 days
       http://bugs.python.org/issue3740    loewis                    
       patch                                                                   

"make check" suggest a testing target under GNU coding standards    3 days
       http://bugs.python.org/issue3758    brett.cannon              
       patch                                                                   

_lsprof issue                                                      19 days
       http://bugs.python.org/issue3895    benjamin.peterson         
                                                                               

Building PDF documentation from tex files                          19 days
       http://bugs.python.org/issue3909    georg.brandl              
                                                                               

bisect insort C implementation ignores methods on list subclasse   16 days
       http://bugs.python.org/issue3935    georg.brandl              
       patch                                                                   

PyTraceBack_Print() doesn't respect # coding: xxx header           13 days
       http://bugs.python.org/issue3975    amaury.forgeotdarc        
       patch                                                                   

Additional 2to3 documentation updates                               5 days
       http://bugs.python.org/issue4000    georg.brandl              
                                                                               

2to3 does relative import for modules not in a package.             3 days
       http://bugs.python.org/issue4001    benjamin.peterson         
       patch                                                                   

missing newline in "Could not convert argument %s to string" err    7 days
       http://bugs.python.org/issue4004    amaury.forgeotdarc        
       patch                                                                   

Python-2.6-py2.6.egg-info contains Alpha reference                  4 days
       http://bugs.python.org/issue4014    loewis                    
                                                                               

convert os.getcwdu() to os.getcwd(), and getcwdu() to getcwd()      1 days
       http://bugs.python.org/issue4023    benjamin.peterson         
       patch                                                                   

Documentation displays incorrectly in iexplore.                     4 days
       http://bugs.python.org/issue4029    LambertDW                 
                                                                               

08 value popups an stdin error, no date handle allowed              0 days
       http://bugs.python.org/issue4031    amaury.forgeotdarc        
                                                                               

py3k error in distutils file_copy exception handlers                1 days
       http://bugs.python.org/issue4038    mhammond                  
       patch, patch                                                            

Add __enter__ and __exit__ methods to StringIO/cStringIO classes    0 days
       http://bugs.python.org/issue4039    benjamin.peterson         
                                                                               

reference to rexec in __import__                                    0 days
       http://bugs.python.org/issue4041    georg.brandl              
                                                                               

IDLE won't start in 2.6 final. A known fix was overlooked?          1 days
       http://bugs.python.org/issue4042    loewis                    
                                                                               

Build on 2.6 on AIX 5.3 fails (syntax error / undeclared identif    0 days
       http://bugs.python.org/issue4052    loewis                    
                                                                               

str.split unintentionally strips char 'I' from the string           0 days
       http://bugs.python.org/issue4053    haypo                     
                                                                               

str.split unintentionally strips char 'I' from the string           0 days
       http://bugs.python.org/issue4054    haypo                     
                                                                               

Documentation on website is missing section numbers                 2 days
       http://bugs.python.org/issue4055    georg.brandl              
                                                                               

:Class: causes exception                                            2 days
       http://bugs.python.org/issue4056    georg.brandl              
                                                                               

markup in What's New in 2.6                                         2 days
       http://bugs.python.org/issue4058    georg.brandl              
       patch                                                                   

sqlite3 docs incomplete                                             2 days
       http://bugs.python.org/issue4059    georg.brandl              
       patch                                                                   

summing two numbers-strange answer                                  0 days
       http://bugs.python.org/issue4061    benjamin.peterson         
                                                                               

sphinx: make all-pdf does not exist.                                1 days
       http://bugs.python.org/issue4063    georg.brandl              
                                                                               

Backport fix for issue 3312                                         1 days
       http://bugs.python.org/issue4068    ghaering                  
       patch, patch                                                            

set.remove raises confusing KeyError                                0 days
       http://bugs.python.org/issue4069    amaury.forgeotdarc        
       patch                                                                   

Cannot build non-framework tkinter Python on Mac OS X.5             1 days
       http://bugs.python.org/issue4076    amaury.forgeotdarc        
                                                                               

2.5.2 whatsnew document corrupts names, by having broken HTML, a    0 days
       http://bugs.python.org/issue4085    georg.brandl              
                                                                               

"Future statements" Doc from 2.6 refers to 2.5                      0 days
       http://bugs.python.org/issue4094    benjamin.peterson         
                                                                               

Delivery Status                                                     0 days
       http://bugs.python.org/issue4095    loewis                    
                                                                               

surprised by default list parameter                                 0 days
       http://bugs.python.org/issue4098    amaury.forgeotdarc        
                                                                               

optparse enable_interspersed_args disable_interspersed_args       982 days
       http://bugs.python.org/issue1415508 akuchling                 
       patch                                                                   

Failed to build Python 2.5.1 with sqlite3                         528 days
       http://bugs.python.org/issue1706863 ocean-city                
       patch                                                                   



Top Issues Most Discussed (10)
______________________________

 15 os.listdir can return byte strings                               105 days
closed  http://bugs.python.org/issue3187   

 14 python2.6 -m site doesn't run site._script() any more              2 days
open    http://bugs.python.org/issue4082   

 14 PyUnicode_DecodeUTF16(..., byteorder=0) gets it wrong on Mac OS    4 days
open    http://bugs.python.org/issue4060   

 13 Cross compiling patches for MINGW                                694 days
open    http://bugs.python.org/issue1597850

 12 PyTraceBack_Print() doesn't respect # coding: xxx header          13 days
closed  http://bugs.python.org/issue3975   

 12 compile() cannot decode Latin-1 source encodings                  55 days
open    http://bugs.python.org/issue3574   

 10 ntpath.abspath fails for long str paths                            3 days
open    http://bugs.python.org/issue4071   

  9 Use WCHAR variant of OutputDebugString                             2 days
open    http://bugs.python.org/issue4075   

  7 importing from UNC roots doesn't work                             46 days
open    http://bugs.python.org/issue3677   

  6 Document the effects of NotImplemented on == and !=                1 days
open    http://bugs.python.org/issue4087   




From jimjjewett at gmail.com  Fri Oct 10 18:11:03 2008
From: jimjjewett at gmail.com (Jim Jewett)
Date: Fri, 10 Oct 2008 12:11:03 -0400
Subject: [Python-Dev] backporting tests [was: [Python-checkins] r66863 -
	python/trunk/Modules/posixmodule.c]
Message-ID: <fb6fbf560810100911k1dd9f9a5u348359bfeb9efc05@mail.gmail.com>

In http://mail.python.org/pipermail/python-dev/2008-October/082994.html
Martin v. L?wis wrote:

> So 2.6.0 will contain a lot of tests that have never been tested in
> a wide variety of systems. Some are incorrect, and get fixed in 2.6.1,
> and stay fixed afterwards. This is completely different from somebody
> introducing a new test in 2.6.4. It means that there are more failures
> in a maintenance release, not less as in the first case.

If 2.6.1 has some (possibly accidental, but exposed to the users)
behavior that is not a clear bug, it should be kept through 2.6.x.
You may well want to change it in 2.7, but not in 2.6.4.  Adding a
test to 2.6.2 ensures that the behavior will not silently disappear
because of an unrelated bugfix in 2.6.3.

-jJ

From brett at python.org  Fri Oct 10 19:27:54 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 10 Oct 2008 10:27:54 -0700
Subject: [Python-Dev] Documentation idea
In-Reply-To: <20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>
Message-ID: <bbaeab100810101027x55652f5ak78707544196ddb41@mail.gmail.com>

On Thu, Oct 9, 2008 at 8:37 PM,  <glyph at divmod.com> wrote:
>
> On 9 Oct, 11:12 pm, python at rcn.com wrote:
>>
>> Background
>> ----------
>> In the itertools module docs, I included pure python equivalents for each
>> of the C functions.  Necessarily, some of those equivalents are only
>> approximate but they seem to have greatly enhanced the docs.
>
> Why not go the other direction?
>
> Ostensibly the reason for writing a module like 'itertools' in C is purely
> for performance.  There's nothing that I'm aware of in that module which
> couldn't be in Python.
>
> Similarly, cStringIO, cPickle, etc.  Everywhere these diverge, it is (if not
> a flat-out bug) not optimal.  External projects are encouraged by a wealth
> of documentation to solve performance problems in a similar way: implement
> in Python, once you've got the interface right, optimize into C.
>
> So rather than have a C implementation, which points to Python, why not have
> a Python implementation that points at C?  'itertools' (and similar) can
> actually be Python modules, and use a decorator, let's call it "C", to do
> this:
>
>   @C("_c_itertools.count")
>   class count(object):
>       """
>       This is the documentation for both the C version of itertools.count
>       and the Python version - since they should be the same, right?
>       """
>

And that decorator is generic enough to work for both classes and functions.

> In Python itself, the Python module would mostly be for documentation, and
> therefore solve the problem that Raymond is talking about, but it could also
> be a handy fallback for sanity checking, testing, and use in other Python
> runtimes (ironpython, jython, pypy).

Which is why I would love to make this almost a policy for new modules
that do not have any C dependency.

>  Many third-party projects already use
> ad-hoc mechanisms for doing this same thing, but an officially-supported way
> of saying "this works, but the optimized version is over here" would be a
> very useful convention.
>
> In those modules which absolutely require some C stuff to work, the python
> module could still serve as documentation.
>

Add to this some function to help test both the pure Python and C
implementation, like ``py_version, c_version =
import_versions('itertools', '_c_itertools')``, so you can run the
test suite against both versions, and you then have pretty much
everything covered for writing the code in Python to start and
optimizing as needed in C.

-Brett

From brett at python.org  Fri Oct 10 19:31:28 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 10 Oct 2008 10:31:28 -0700
Subject: [Python-Dev] syntax change justification
In-Reply-To: <fb6fbf560810100851g49c9204eqa12bc1927368aef8@mail.gmail.com>
References: <fb6fbf560810100851g49c9204eqa12bc1927368aef8@mail.gmail.com>
Message-ID: <bbaeab100810101031j49e6e0a8na1b88323556f96ec@mail.gmail.com>

On Fri, Oct 10, 2008 at 8:51 AM, Jim Jewett <jimjjewett at gmail.com> wrote:
> Nick Coghlan's explanation of what justifies a syntax change (most of message
> http://mail.python.org/pipermail/python-dev/2008-October/082831.html )
> should probably be added to the standard docs/FAQs somewhere.
>
> At the moment, I'm not sure exactly where, though.  At the moment, the
> Developer FAQ (http://www.python.org/dev/faq/)  is mostly about using
> specific tools (rather than design philosophy), and Nick's explanation
> may be too detailed for the current Explanations section of
> www.python.org/dev/
>
> Possibly as a Meta-PEP?
>

What we need (and hope through some miracle to have the time for) a
doc that explains the steps needed to report a bug/submit a patch
(basically the issue lifecycle) and how to propose a new language
feature. That would clean up the dev FAQ into basically being a svn
FAQ, and then give us docs to point people to when they ask how they
can contribute.

-Brett

From tjreedy at udel.edu  Fri Oct 10 22:45:02 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 10 Oct 2008 16:45:02 -0400
Subject: [Python-Dev] Documentation idea
In-Reply-To: <20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>
Message-ID: <gcoesc$r11$1@ger.gmane.org>

glyph at divmod.com wrote:
> 
> On 9 Oct, 11:12 pm, python at rcn.com wrote:
>> Background
>> ----------
>> In the itertools module docs, I included pure python equivalents for 
>> each of the C functions.  Necessarily, some of those equivalents are 
>> only approximate but they seem to have greatly enhanced the docs.
> 
> Why not go the other direction?
> 
> Ostensibly the reason for writing a module like 'itertools' in C is 
> purely for performance.  There's nothing that I'm aware of in that 
> module which couldn't be in Python.
> 
> Similarly, cStringIO, cPickle, etc.  Everywhere these diverge, it is (if 
> not a flat-out bug) not optimal.  External projects are encouraged by a 
> wealth of documentation to solve performance problems in a similar way: 
> implement in Python, once you've got the interface right, optimize into C.
> 
> So rather than have a C implementation, which points to Python, why not 
> have a Python implementation that points at C?  'itertools' (and 
> similar) can actually be Python modules, and use a decorator, let's call 
> it "C", to do this:
> 
>    @C("_c_itertools.count")
>    class count(object):
>        """
>        This is the documentation for both the C version of itertools.count
>        and the Python version - since they should be the same, right?
>        """

The ancient string module did something like this, except that the 
rebinding of function names was done at the end by 'from _string import 
*' where _string had C versions of some but not all of the functions in 
string.  (And the list of replacements could vary by version and 
platform and compiler switches.)  This was great for documenting the 
string module.  It was some of the first Python code I studied after the 
tutorial.

The problem with that and the above (with modification, see below) is 
the creation and discarding of unused function objects and the time 
required to do so.

The advantage of the decorator version is that the compiler or module 
loader could be special cased to recognize the 'C' decorator and try it 
first *before* using the Python version, which would serve as a backup. 
  There could be a standard version in builtins that people could 
replace to implement non-standard loading on a particular system.  To 
cater to other implementations, the name could be something other than 
'C', or we could define 'C' to be the initial of "Code" (in the 
implementation language).  Either way, other implementation could start 
with a do-nothing "C" decorator and run the file as is, then gradually 
replace with lower-level code.

Terry Jan Reedy


From tjreedy at udel.edu  Fri Oct 10 22:53:28 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 10 Oct 2008 16:53:28 -0400
Subject: [Python-Dev] syntax change justification
In-Reply-To: <fb6fbf560810100851g49c9204eqa12bc1927368aef8@mail.gmail.com>
References: <fb6fbf560810100851g49c9204eqa12bc1927368aef8@mail.gmail.com>
Message-ID: <gcofc5$r11$2@ger.gmane.org>

Jim Jewett wrote:
> Nick Coghlan's explanation of what justifies a syntax change (most of message
> http://mail.python.org/pipermail/python-dev/2008-October/082831.html )
> should probably be added to the standard docs/FAQs somewhere.

I agree that this was a helpful explanation.  A link to the original 
would be easy to add now.


From brett at python.org  Fri Oct 10 23:57:06 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 10 Oct 2008 14:57:06 -0700
Subject: [Python-Dev] Documentation idea
In-Reply-To: <gcoesc$r11$1@ger.gmane.org>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>
	<gcoesc$r11$1@ger.gmane.org>
Message-ID: <bbaeab100810101457p7848ac90r5dc59557a49ec58a@mail.gmail.com>

On Fri, Oct 10, 2008 at 1:45 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> glyph at divmod.com wrote:
>>
>> On 9 Oct, 11:12 pm, python at rcn.com wrote:
>>>
>>> Background
>>> ----------
>>> In the itertools module docs, I included pure python equivalents for each
>>> of the C functions.  Necessarily, some of those equivalents are only
>>> approximate but they seem to have greatly enhanced the docs.
>>
>> Why not go the other direction?
>>
>> Ostensibly the reason for writing a module like 'itertools' in C is purely
>> for performance.  There's nothing that I'm aware of in that module which
>> couldn't be in Python.
>>
>> Similarly, cStringIO, cPickle, etc.  Everywhere these diverge, it is (if
>> not a flat-out bug) not optimal.  External projects are encouraged by a
>> wealth of documentation to solve performance problems in a similar way:
>> implement in Python, once you've got the interface right, optimize into C.
>>
>> So rather than have a C implementation, which points to Python, why not
>> have a Python implementation that points at C?  'itertools' (and similar)
>> can actually be Python modules, and use a decorator, let's call it "C", to
>> do this:
>>
>>   @C("_c_itertools.count")
>>   class count(object):
>>       """
>>       This is the documentation for both the C version of itertools.count
>>       and the Python version - since they should be the same, right?
>>       """
>
> The ancient string module did something like this, except that the rebinding
> of function names was done at the end by 'from _string import *' where
> _string had C versions of some but not all of the functions in string.  (And
> the list of replacements could vary by version and platform and compiler
> switches.)  This was great for documenting the string module.  It was some
> of the first Python code I studied after the tutorial.
>
> The problem with that and the above (with modification, see below) is the
> creation and discarding of unused function objects and the time required to
> do so.
>
> The advantage of the decorator version is that the compiler or module loader
> could be special cased to recognize the 'C' decorator and try it first
> *before* using the Python version, which would serve as a backup.  There
> could be a standard version in builtins that people could replace to
> implement non-standard loading on a particular system.  To cater to other
> implementations, the name could be something other than 'C', or we could
> define 'C' to be the initial of "Code" (in the implementation language).
>  Either way, other implementation could start with a do-nothing "C"
> decorator and run the file as is, then gradually replace with lower-level
> code.
>

The decorator doesn't have to require any special casing at all
(changing the parameters to keep the code short)::

  def C(module_name, want):
     def choose_version(ob):
         try:
           module = __import__(module_name, fromlist=[want])
           return getattr(module, want)
          except (ImportError, AttributeError):
            return ob
      return choose_version

The cost is purely during importation of the module and does nothing
fancy at all and relies on stuff already available in all Python VMs.

-Brett

From ocean-city at m2.ccsnet.ne.jp  Sat Oct 11 00:07:56 2008
From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto)
Date: Sat, 11 Oct 2008 07:07:56 +0900
Subject: [Python-Dev] [Python-checkins]
	r66863	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <48EE8FEF.20005@v.loewis.de>
References: <002901c92a2e$e77f1f60$0200a8c0@whiterabc2znlh>	<bbaeab100810091025n2b154491u89ec966e500c086b@mail.gmail.com>	<4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>	<48EE4C41.30905@v.loewis.de>	<gclkpu$93f$1@ger.gmane.org>	<48EE72DB.1000002@v.loewis.de>	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>
	<48EE8FEF.20005@v.loewis.de>
Message-ID: <48EFD23C.6060104@m2.ccsnet.ne.jp>


>> It seems to me that Skip was asking whether the "memory leak" impacted
>> the 2.6 branch, and the answer should have been "No": the change that
>> introduced the memory leak had just been committed 10 minutes before.
>>     
>
> You are probably right (although it's not quite clear from Skip's question).
>   
Umm, sorry for misunderstandings.  I thought he indicated the set of two 
patches.
>> - Because of this misunderstanding, the changes to this
>> GetCurrentDirectoryW were backported to the release2.6 branch, despite
>> the fact that it's not a regression from a previous version, the NEWS
>> entry explicitly expresses doubts about the correction (which I happen
>> to share), there is no unit test and no item in the issue tracker.
>>     
>
> I think it is fine that this fix was backported (assuming, without
> review, that the fix is actually correct).
>
> It is a bugfix, and it shouldn't realistically break existing applications.
>
> IOW, PEP 6 was followed (except that there is no Patch Czar).
>   
Thanks, I'm a bit relaxed. :-)    
>> - The backport to release26-maint http://svn.python.org/view?rev=66865&view=rev
>> also merged other changes (new unrelated unit tests). IMO unrelated
>> changes should be committed separately: different commit messages help
>> to understand the motivation of each backport.
>>     
>
> Yes, that is unfortunate.
>
> I'm skeptical that new tests actually need backporting at all. Python
> doesn't really get better by new tests being added to an old branch.
> Near-term, it might get worse because the new tests might cause false
> positives, making users worried for no reason.
>   
OK, I'll do separate commit for release26-maint even via svnmerge.py  (I 
did same way as in py3k)
But I'm bit confused. This is difficult problem for me, so I 'll commit 
to only trunk until some consensus
will be established.

From ncoghlan at gmail.com  Sat Oct 11 03:05:29 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 11 Oct 2008 11:05:29 +1000
Subject: [Python-Dev] syntax change justification
In-Reply-To: <gcofc5$r11$2@ger.gmane.org>
References: <fb6fbf560810100851g49c9204eqa12bc1927368aef8@mail.gmail.com>
	<gcofc5$r11$2@ger.gmane.org>
Message-ID: <48EFFBD9.8040101@gmail.com>

Terry Reedy wrote:
> Jim Jewett wrote:
>> Nick Coghlan's explanation of what justifies a syntax change (most of
>> message
>> http://mail.python.org/pipermail/python-dev/2008-October/082831.html )
>> should probably be added to the standard docs/FAQs somewhere.
> 
> I agree that this was a helpful explanation.  A link to the original
> would be easy to add now.

A link from either the dev FAQ (like the one to Raymond's School of Hard
Knocks) or from PEP 1 (PEP Purpose and Guidelines) might work, but I
will leave it to others to decide whether it is worth linking to or not :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From tjreedy at udel.edu  Sat Oct 11 06:46:19 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 11 Oct 2008 00:46:19 -0400
Subject: [Python-Dev] Documentation idea
In-Reply-To: <bbaeab100810101457p7848ac90r5dc59557a49ec58a@mail.gmail.com>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>	<20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>	<gcoesc$r11$1@ger.gmane.org>
	<bbaeab100810101457p7848ac90r5dc59557a49ec58a@mail.gmail.com>
Message-ID: <gcpb2o$7i6$1@ger.gmane.org>

Brett Cannon wrote:
> On Fri, Oct 10, 2008 at 1:45 PM, Terry Reedy <tjreedy at udel.edu> wrote:

>> The advantage of the decorator version is that the compiler or module loader
>> could be special cased to recognize the 'C' decorator and try it first
>> *before* using the Python version, which would serve as a backup.  There
>> could be a standard version in builtins that people could replace to
>> implement non-standard loading on a particular system.  To cater to other
>> implementations, the name could be something other than 'C', or we could
>> define 'C' to be the initial of "Code" (in the implementation language).
>>  Either way, other implementation could start with a do-nothing "C"
>> decorator and run the file as is, then gradually replace with lower-level
>> code.
>>
> 
> The decorator doesn't have to require any special casing at all
> (changing the parameters to keep the code short)::
> 
>   def C(module_name, want):
>      def choose_version(ob):
>          try:
>            module = __import__(module_name, fromlist=[want])
>            return getattr(module, want)
>           except (ImportError, AttributeError):
>             return ob
>       return choose_version
> 
> The cost is purely during importation of the module and does nothing
> fancy at all and relies on stuff already available in all Python VMs.

If I understand correctly, this decorator would only be applied *after* 
the useless Python level function object was created.  I was proposing 
bypassing that step when not necessary, and I believe special casing 
*would* be required for that.


From brett at python.org  Sat Oct 11 08:53:48 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 10 Oct 2008 23:53:48 -0700
Subject: [Python-Dev] Documentation idea
In-Reply-To: <gcpb2o$7i6$1@ger.gmane.org>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<20081010033724.420.1883106988.divmod.xquotient.93@weber.divmod.com>
	<gcoesc$r11$1@ger.gmane.org>
	<bbaeab100810101457p7848ac90r5dc59557a49ec58a@mail.gmail.com>
	<gcpb2o$7i6$1@ger.gmane.org>
Message-ID: <bbaeab100810102353y549c6561ye7bef79dbe466d5c@mail.gmail.com>

On Fri, Oct 10, 2008 at 9:46 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> Brett Cannon wrote:
>>
>> On Fri, Oct 10, 2008 at 1:45 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>
>>> The advantage of the decorator version is that the compiler or module
>>> loader
>>> could be special cased to recognize the 'C' decorator and try it first
>>> *before* using the Python version, which would serve as a backup.  There
>>> could be a standard version in builtins that people could replace to
>>> implement non-standard loading on a particular system.  To cater to other
>>> implementations, the name could be something other than 'C', or we could
>>> define 'C' to be the initial of "Code" (in the implementation language).
>>>  Either way, other implementation could start with a do-nothing "C"
>>> decorator and run the file as is, then gradually replace with lower-level
>>> code.
>>>
>>
>> The decorator doesn't have to require any special casing at all
>> (changing the parameters to keep the code short)::
>>
>>  def C(module_name, want):
>>     def choose_version(ob):
>>         try:
>>           module = __import__(module_name, fromlist=[want])
>>           return getattr(module, want)
>>          except (ImportError, AttributeError):
>>            return ob
>>      return choose_version
>>
>> The cost is purely during importation of the module and does nothing
>> fancy at all and relies on stuff already available in all Python VMs.
>
> If I understand correctly, this decorator would only be applied *after* the
> useless Python level function object was created.

Yes.

>  I was proposing bypassing
> that step when not necessary, and I believe special casing *would* be
> required for that.

Yes, that would.

-Brett

From ziade.tarek at gmail.com  Sat Oct 11 17:08:19 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 11 Oct 2008 11:08:19 -0400
Subject: [Python-Dev] Distutils/packaging sprint this weekend
In-Reply-To: <20081009202437.GA15368@amk-desktop.matrixgroup.net>
References: <20081009202437.GA15368@amk-desktop.matrixgroup.net>
Message-ID: <94bdd2610810110808v532f7c1br6927c82d7cfcf1d4@mail.gmail.com>

Hi,

We have started the sprint in D.C. on packaging matters. Our tasks for
the moment are:

    * re-read all the threads in distutils-ML since mid-september.
    * write a short list of actions that can be done NOW in distutils,
setuptools and PyPI
    * write a short list of writing that can be done to define a "new" tool

Please, join at any time in #distutils in freenode, and ping us, if
you want to get involved
in this sprint.

Cheers
Tarek


On Thu, Oct 9, 2008 at 4:24 PM, A.M. Kuchling <amk at amk.ca> wrote:
> Tarek Zidae' is organizing a sprint on general
> distutils/setuptools/packaging this weekend.  Physically it's in
> Arlington VA, but participants will be hanging out in #distutils on
> freenode's IRC.
>
> More information at
> <http://www.openplans.org/projects/plone-conference-2008-dc/distribute>.
>
> --am"If you're in Fairfax County and need a lift, let me know"k
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com
>



-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/

From taskinoor.hasan at csebuet.org  Sun Oct 12 14:52:48 2008
From: taskinoor.hasan at csebuet.org (Taskinoor Hasan)
Date: Sun, 12 Oct 2008 18:52:48 +0600
Subject: [Python-Dev] 2 modifications in robotparser.py
Message-ID: <79153a2e0810120552j2a52b77fi26c2f56b97d7cdcd@mail.gmail.com>

im a novice python programmer. i have made two changes to robotparser.py. i
apologize if this is the wrong list to post this mail.

1. some sites /* specially wikipedia */ returns 403 when default User-Agent
is used. so i have changed the code to use urllib2 and added a
set_user_agent method. this is simple.

2. this problem is slight complicated. please check the robots.txt file from
mathworld.

   http://mathworld.wolfram.com/robots.txt

it contains 2 User-Agent: * lines.

from http://www.robotstxt.org/norobots-rfc.txt

These name tokens are used in User-agent lines in /robots.txt to
identify to which specific robots the record applies. The robot
must obey the first record in /robots.txt that contains a User-
Agent line whose value contains the name token of the robot as a
substring. The name comparisons are case-insensitive. If no such
record exists, it should obey the first record with a User-agent
line with a "*" value, if present. If no record satisfied either
condition, or no records are present at all, access is unlimited.


but it seems that our robotparser is obeying the 2nd one. the problem
occures because robotparser assumes that no robots.txt will contain two *
user-agent. it should not have two two such line, but in reality many site
may have two.

so i have changed the code as follow:

    def _add_entry(self, entry):
        if "*" in entry.useragents:
            # the default entry is considered last
            if self.default_entry == None:
                   self.default_entry = entry
        else:
            self.entries.append(entry)

and at the end of parse(self, lines) method

        if state==2:
#            self.entries.append(entry)
            self._add_entry(entry)

red marked lines are added by me.

as im a very novice python programmer, i really want some experts comment
about this matter.

i apologize again if im wasting ur times.

thanks in advance
Taskinoor Hasan Sajid
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081012/1ff8b563/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: customrobotparser.py
Type: application/octet-stream
Size: 11127 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081012/1ff8b563/attachment.obj>

From skip at pobox.com  Sun Oct 12 15:26:51 2008
From: skip at pobox.com (skip at pobox.com)
Date: Sun, 12 Oct 2008 08:26:51 -0500
Subject: [Python-Dev] 2 modifications in robotparser.py
In-Reply-To: <79153a2e0810120552j2a52b77fi26c2f56b97d7cdcd@mail.gmail.com>
References: <79153a2e0810120552j2a52b77fi26c2f56b97d7cdcd@mail.gmail.com>
Message-ID: <18673.64283.790364.177753@montanaro-dyndns-org.local>


    Taskinoor> im a novice python programmer. i have made two changes to
    Taskinoor> robotparser.py. i apologize if this is the wrong list to post
    Taskinoor> this mail.

Thanks for the message.  Can you please file a bug report at
http://bugs.python.org though?

Skip

From taskinoor.hasan at csebuet.org  Sun Oct 12 15:48:02 2008
From: taskinoor.hasan at csebuet.org (Taskinoor Hasan)
Date: Sun, 12 Oct 2008 19:48:02 +0600
Subject: [Python-Dev] 2 modifications in robotparser.py
In-Reply-To: <18673.64283.790364.177753@montanaro-dyndns-org.local>
References: <79153a2e0810120552j2a52b77fi26c2f56b97d7cdcd@mail.gmail.com>
	<18673.64283.790364.177753@montanaro-dyndns-org.local>
Message-ID: <79153a2e0810120648k6dd4fcadye81174f2354f160b@mail.gmail.com>

On Sun, Oct 12, 2008 at 7:26 PM, <skip at pobox.com> wrote:

>
>    Taskinoor> im a novice python programmer. i have made two changes to
>    Taskinoor> robotparser.py. i apologize if this is the wrong list to post
>    Taskinoor> this mail.
>
> Thanks for the message.  Can you please file a bug report at
> http://bugs.python.org though?


thanks. issue created. here is the link : http://bugs.python.org/issue4108
i have already fixed this problem but not 100% sure about the status.


>
>
> Skip
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081012/05e169b2/attachment.htm>

From regebro at gmail.com  Sun Oct 12 19:23:50 2008
From: regebro at gmail.com (Lennart Regebro)
Date: Sun, 12 Oct 2008 19:23:50 +0200
Subject: [Python-Dev] for __future__ import planning
In-Reply-To: <3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
References: <1afaf6160810031426n21514e81ma213b084aff20648@mail.gmail.com>
	<3DDCFDD1-52DB-487D-AEB4-758CF868945D@python.org>
Message-ID: <319e029f0810121023i2bff937ah35279592f316f8b0@mail.gmail.com>

On Sat, Oct 4, 2008 at 00:56, Barry Warsaw <barry at python.org> wrote:
> I think 2.7 should continue along the path of convergence toward 3.x.  The
> vision some of us talked about at Pycon was that at some point down the
> line, maybe there's no difference between "python2.9 -3" and "python3.3 -2".

I like that. Do we know what the next "hurdle" would be? The testing I
have done seems to indicate that one major area is handling of binary
file data that may or may not be binary. Is a real bytes type (and not
just an alternate spelling for str) a possibility? It may be that this
isn't a problem in practice, I don't know yet. :)

-- 
Lennart Regebro: Zope and Plone consulting.
http://www.colliberty.com/
+33 661 58 14 64

From martin at v.loewis.de  Sun Oct 12 20:15:35 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 12 Oct 2008 20:15:35 +0200
Subject: [Python-Dev] [Python-checkins]
	r66863	-	python/trunk/Modules/posixmodule.c
In-Reply-To: <20081010154451.GA9433@amk-desktop.matrixgroup.net>
References: <4DFFAAD175FD4ADAB215A292BC5BB095@RaymondLaptop1>	<48EE4C41.30905@v.loewis.de>
	<gclkpu$93f$1@ger.gmane.org>	<48EE72DB.1000002@v.loewis.de>	<e27efe130810091543t694d2efqffa89360772d3a89@mail.gmail.com>	<48EE8FEF.20005@v.loewis.de>	<20081009233537.420.1440384488.divmod.xquotient.68@weber.divmod.com>	<48EEEB33.3060505@v.loewis.de>	<20081010062926.GD32624@steerpike.home.puzzling.org>	<48EEF9D6.5070403@v.loewis.de>
	<20081010154451.GA9433@amk-desktop.matrixgroup.net>
Message-ID: <48F23EC7.9080802@v.loewis.de>

> (2.5.3 reminder: there are lists of commits in sandbox/py2.5.3 to be
> considered.  I've seen no reactions on python-dev or modifications to
> those files, so I don't think anyone else is looking at them.  Is
> everyone waiting for the weekend, maybe?)

I may have said that before: I don't have any plans to look through
change lists myself. If people want certain changes considered, they
will tell us; if nobody is interested in a certain backport, there is
no need to backport.

Regards,
Martin

From brett at python.org  Mon Oct 13 20:12:33 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 13 Oct 2008 11:12:33 -0700
Subject: [Python-Dev] Possible summits at PyCon (and question about
	sprinting implications)
Message-ID: <bbaeab100810131112o6d2e0aa2p420326abcd9ce86@mail.gmail.com>

 Over in PyConLand, there has been talk about trying to set up a
language summit the day before PyCon starts (the second day of
tutorials). The idea is to give the core developers and Python VM
implementers a day to sit around and talk about stuff without having
to eat into the sprints (I am not leading the organizing of it, so I
don't have any exact details beyond various ideas that have leaked
over to the program committee mailing list).

The idea was then floated about inviting the VM implementers of the
big three JavaScript VMs. But then this got re-targeted as possibly a
separate summit on Wednesday (first day of tutorials) where the
various dynamic language VM implementers could get together and talk.
PyCon would essentially act as the hosting site for this and as
motivation to maybe get some other VM folks to look at the language
and since no business or university would necessarily be interested
enough to make this happen.

But there is a worry that if people attended one or both of the
summits it would cut into people's sprint time. And if any sprint
group would be adversely affected by this, the core sprint would be
hit the hardest since the attendees of the sprint are the most likely
to attend either summit.

And so I have been tasked with asking people whether attending the
summits would put a crimp in their attendance of the sprints. Please
let me know if attending the summits (especially the VM one on
Wednesday) would cause you to skip out on the sprints in any way.

-Brett

From mborch at gmail.com  Mon Oct 13 21:15:10 2008
From: mborch at gmail.com (Malthe Borch)
Date: Mon, 13 Oct 2008 15:15:10 -0400
Subject: [Python-Dev] Transformation with ``parser`` ast
Message-ID: <gd06nu$rtr$1@ger.gmane.org>

The ``compiler.ast`` module makes parsing Python source-code and AST 
manipulation relatively painless and it's straight-forward to implement 
a transformer class.

However, I find that the ``compiler.pycodegen`` module imposes a hard 
limit on the length of functions since it calculates jump points in a 
recursion.

I'm using this module to compile an XML dynamic template into Python 
code (akin to "Mako") and functions may grow to a rather large size.

Now it seems that the ``parser`` module, which parses source code into 
``parser.st`` trees does not have the same limitations, however, I could 
not find a transformer class compatible with its tree structure.

What's the recommended way of working with the AST tree from the 
``parser`` module?

\malthe


From brett at python.org  Mon Oct 13 21:33:40 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 13 Oct 2008 12:33:40 -0700
Subject: [Python-Dev] Transformation with ``parser`` ast
In-Reply-To: <gd06nu$rtr$1@ger.gmane.org>
References: <gd06nu$rtr$1@ger.gmane.org>
Message-ID: <bbaeab100810131233r520bc157qe8ba46c0bc5ec4cd@mail.gmail.com>

This mailing list is for discussing the design of Python, not its use.
The best place to ask your question is comp.lang.python.

-Brett

On Mon, Oct 13, 2008 at 12:15 PM, Malthe Borch <mborch at gmail.com> wrote:
> The ``compiler.ast`` module makes parsing Python source-code and AST
> manipulation relatively painless and it's straight-forward to implement a
> transformer class.
>
> However, I find that the ``compiler.pycodegen`` module imposes a hard limit
> on the length of functions since it calculates jump points in a recursion.
>
> I'm using this module to compile an XML dynamic template into Python code
> (akin to "Mako") and functions may grow to a rather large size.
>
> Now it seems that the ``parser`` module, which parses source code into
> ``parser.st`` trees does not have the same limitations, however, I could not
> find a transformer class compatible with its tree structure.
>
> What's the recommended way of working with the AST tree from the ``parser``
> module?
>
> \malthe
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>

From martin at v.loewis.de  Tue Oct 14 20:35:55 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 14 Oct 2008 20:35:55 +0200
Subject: [Python-Dev] No manifest files on Windows?
Message-ID: <48F4E68B.2010701@v.loewis.de>

In http://bugs.python.org/issue4120, the author suggests that it might
be possible to completely stop using the manifest mechanism, for VS
2008. Given the many problems that this SxS stuff has caused, this
sounds like a very desirable route, although I haven't done any actual
testing yet.

Can all the Windows experts please comment? Could that work? Does it
have any downsides?

If it works, I would like to apply it to 3.0, although I probably
won't be able to apply it to tomorrow's rc. Would it also be possible
to change that in 2.6.1 (even though python26.dll in 2.6.0 already
includes a manifest, as do all the pyd files)?

Regards,
Martin

From greg.ewing at canterbury.ac.nz  Wed Oct 15 01:15:06 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 15 Oct 2008 12:15:06 +1300
Subject: [Python-Dev] [pygtk] Application name is '-c'
In-Reply-To: <WnXqa4tQ.1223975020.7564750.fma@alezan.org>
References: <WnXqa4tQ.1223975020.7564750.fma@alezan.org>
Message-ID: <48F527FA.7010308@canterbury.ac.nz>

Fr?d?ric wrote:
> In several places, instead of having my application name, I get '-c'.

This could be a result of the app having got launched
via 'python -c' somewhere along the way:

% python -c 'import sys; print sys.argv[0]'
-c

Not sure whether to regard this as a bug or not. It's
not clear what the app name *should* be when you use
-c. What do the python-dev folks think?

-- 
Greg

From guido at python.org  Wed Oct 15 01:36:30 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 14 Oct 2008 16:36:30 -0700
Subject: [Python-Dev] [pygtk] Application name is '-c'
In-Reply-To: <48F527FA.7010308@canterbury.ac.nz>
References: <WnXqa4tQ.1223975020.7564750.fma@alezan.org>
	<48F527FA.7010308@canterbury.ac.nz>
Message-ID: <ca471dc20810141636w5582c680x149b0208548d5435@mail.gmail.com>

On Tue, Oct 14, 2008 at 4:15 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Fr?d?ric wrote:
>>
>> In several places, instead of having my application name, I get '-c'.
>
> This could be a result of the app having got launched
> via 'python -c' somewhere along the way:
>
> % python -c 'import sys; print sys.argv[0]'
> -c
>
> Not sure whether to regard this as a bug or not. It's
> not clear what the app name *should* be when you use
> -c. What do the python-dev folks think?

It is a feature. I chose to set it to '-c' because there is no other
appropriate string to use.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From engelbert.gruber at gmail.com  Wed Oct 15 17:30:07 2008
From: engelbert.gruber at gmail.com (engelbert gruber)
Date: Wed, 15 Oct 2008 17:30:07 +0200
Subject: [Python-Dev] www.python.org/doc and docs.python.org hotfixed
In-Reply-To: <48E541E3.4080709@gmail.com>
References: <9e804ac0810020259w4b612cefk7f59f288c52e42d3@mail.gmail.com>
	<loom.20081002T104308-982@post.gmane.org>
	<9e804ac0810020428r51a31e5cha910842172db94c6@mail.gmail.com>
	<C220578E-DDAE-4D34-A18D-6D7103C6D348@gmail.com>
	<48E4BBFA.7010800@gmail.com>
	<18660.53751.895687.800610@montanaro-dyndns-org.local>
	<9e804ac0810020741g7d7cb193h647b512a0efec68b@mail.gmail.com>
	<48E541E3.4080709@gmail.com>
Message-ID: <ac12bf3a0810150830s3a1ad37dif752c49a2ed636e2@mail.gmail.com>

On Thu, Oct 2, 2008 at 11:49 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Long term, remapping even the deep links to the appropriate part of the
> new docs should hopefully be possible.
>
> For the search engine issue, is there any way we can tell robots to
> ignore the rewrite rules so they see the broken links? (although even
> that may not be ideal, since what we really want is to tell the robot
> the link is broken, and provide the new alternative)

wouldnt that be a situation to use hTTP 301/permanently moved headers

all the best

From fperez.net at gmail.com  Thu Oct 16 06:41:46 2008
From: fperez.net at gmail.com (Fernando Perez)
Date: Wed, 15 Oct 2008 21:41:46 -0700
Subject: [Python-Dev] Documentation idea
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
Message-ID: <gd6gmb$avv$1@ger.gmane.org>

Raymond Hettinger wrote:


> Bright idea
> ----------
> Let's go one step further and do this just about everywhere and instead of
> putting it in the docs, attach an exec-able string as an
> attribute to our C functions.  Further, those pure python examples should
> include doctests so that the user can see a typical invocation and calling
> pattern.
> 
> Say we decide to call the attribute something like ".python", then you
> could write something like:
> 
>     >>> print(all.python)
>    def all(iterable):
>         '''Return True if all elements of the iterable are true.
> 

[...]

+1 from the peanut gallery, with a note: since ipython is a common way for
many to use/learn python interactively, if this is adopted, we'd
*immediately* add to ipython's '?' introspection machinery the ability to
automatically find this information.  This way, when people type "all?"
or "all??" we'd fetch the doc and source code.

A minor question inspired by this: would it make sense to split the
docstring part from the code of this .python object?  I say this because in
principle, the docstring should be the same of the 'parent', and it would
simplify our implementation to eliminate the duplicate printout. 
The .python object could always be a special string-like object made from
combining the pure python code with a single docstring, common to the C and
the Python versions, that would remain exec-able.

In any case, details aside I think this is great and if it comes to pass,
we'll be happy to make it readily accessible to interactive users via
ipython.

Cheers,

f


From jnoller at gmail.com  Thu Oct 16 18:34:28 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 16 Oct 2008 12:34:28 -0400
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
Message-ID: <4222a8490810160934g53f4372aw582f864bb4db1230@mail.gmail.com>

Hi Skip,

I had been approached to do the exact same thing, are you trying to
back port the trunk version (2.6) or py3000?

On Thu, Oct 16, 2008 at 12:01 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> I'd like to try backporting the multiprocessing module to Python 2.4.  My first
> problem appears to be the reliance on a complete(?) rewrite of the buffer stuff.
>
> Any clues about transforming this code would be much appreciated.
>
> (Note: I'm backporting because the Python 2.6 version appears to be much more
> robust than the 0.52 third-party release.)
>
> Thanks,
>
> Skip
> _______________________________________________
> Python-3000 mailing list
> Python-3000 at python.org
> http://mail.python.org/mailman/listinfo/python-3000
> Unsubscribe: http://mail.python.org/mailman/options/python-3000/jnoller%40gmail.com
>

From jnoller at gmail.com  Thu Oct 16 18:37:26 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 16 Oct 2008 12:37:26 -0400
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <4222a8490810160934g53f4372aw582f864bb4db1230@mail.gmail.com>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<4222a8490810160934g53f4372aw582f864bb4db1230@mail.gmail.com>
Message-ID: <4222a8490810160937r35bf0b28m585f608930156553@mail.gmail.com>

Also note, for python 2.4/2.5 you are going to *need* the patch to bug
http://bugs.python.org/issue874900

On Thu, Oct 16, 2008 at 12:34 PM, Jesse Noller <jnoller at gmail.com> wrote:
> Hi Skip,
>
> I had been approached to do the exact same thing, are you trying to
> back port the trunk version (2.6) or py3000?
>
> On Thu, Oct 16, 2008 at 12:01 PM, Skip Montanaro
> <skip.montanaro at gmail.com> wrote:
>> I'd like to try backporting the multiprocessing module to Python 2.4.  My first
>> problem appears to be the reliance on a complete(?) rewrite of the buffer stuff.
>>
>> Any clues about transforming this code would be much appreciated.
>>
>> (Note: I'm backporting because the Python 2.6 version appears to be much more
>> robust than the 0.52 third-party release.)
>>
>> Thanks,
>>
>> Skip
>> _______________________________________________
>> Python-3000 mailing list
>> Python-3000 at python.org
>> http://mail.python.org/mailman/listinfo/python-3000
>> Unsubscribe: http://mail.python.org/mailman/options/python-3000/jnoller%40gmail.com
>>
>

From skip.montanaro at gmail.com  Thu Oct 16 18:01:08 2008
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 16 Oct 2008 11:01:08 -0500
Subject: [Python-Dev] Backporting multiprocessing?
Message-ID: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>

I'd like to try backporting the multiprocessing module to Python 2.4.  My first
problem appears to be the reliance on a complete(?) rewrite of the buffer stuff.

Any clues about transforming this code would be much appreciated.

(Note: I'm backporting because the Python 2.6 version appears to be much more
robust than the 0.52 third-party release.)

Thanks,

Skip

From skip.montanaro at gmail.com  Thu Oct 16 18:36:54 2008
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 16 Oct 2008 11:36:54 -0500
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <4222a8490810160934g53f4372aw582f864bb4db1230@mail.gmail.com>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<4222a8490810160934g53f4372aw582f864bb4db1230@mail.gmail.com>
Message-ID: <60bb7ceb0810160936s63c7bc08p1411fd83f7550439@mail.gmail.com>

>  I had been approached to do the exact same thing, are you trying to
>  back port the trunk version (2.6) or py3000?

I'm trying to backport from 2.6.  It appears that the buffer stuff is
completely
new though (backported from Python 3.0).

S

From scott+python-ideas at scottdial.com  Thu Oct 16 20:13:15 2008
From: scott+python-ideas at scottdial.com (Scott Dial)
Date: Thu, 16 Oct 2008 14:13:15 -0400
Subject: [Python-Dev] Documentation idea
In-Reply-To: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
Message-ID: <48F7843B.7070709@scottdial.com>

Raymond Hettinger wrote:
> * It will assist pypy style projects and other python implementations
> when they have to build equivalents to CPython.
> 
> * Will eliminate confusion about what functions were exactly intended to
> do.
> 
> * Will confer benefits similar to test driven development where the
> documentation and  pure python version are developed first and doctests
> gotten to pass, then the C version is created to match.

I haven't seen anyone comment about this assertion of "equivalence".
Doesn't it strike you as difficult to maintain *two* versions of every
function, and ensure they match *exactly*? The utility to PyPy-style
projects is minimized if the two version aren't identical. And while
it's possible to say, "the tests say they are equiavelent, so they are;"
history is quite clear about people depending on "features" that are
untested and were unintended side-effects of the manner in which
something was implemented. I think it would be a dilution of developer
man-hours to force them to maintain two versions in lock-step, and it
significantly adds to the burden of writing and reviewing potential
bugfixes.

While I applaud the idea of documenting C functions in this manner,
let's not confuse documentation with equivalence. If the standard
distribution of Python exports the C version, then all bets are off
whether the Python version is a drop-in replacement (even if the
buildbots regularly test them). I feel so strongly about this that I
think that the consideration of adding this should be frame /solely/ as
a documentation tool and nothing more.

-Scott

-- 
Scott Dial
scott at scottdial.com
scodial at cs.indiana.edu

From jnoller at gmail.com  Thu Oct 16 21:30:56 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 16 Oct 2008 15:30:56 -0400
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <gd84l0$v7$1@ger.gmane.org>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<gd84l0$v7$1@ger.gmane.org>
Message-ID: <4222a8490810161230xca234c6y15c8e0733fb62b2e@mail.gmail.com>

Do we want to start a google code project for this given all three of
us are interested in this? :)

On Thu, Oct 16, 2008 at 3:28 PM, Christian Heimes <lists at cheimes.de> wrote:
> Skip Montanaro wrote:
>>
>> I'd like to try backporting the multiprocessing module to Python 2.4.  My
>> first
>> problem appears to be the reliance on a complete(?) rewrite of the buffer
>> stuff.
>>
>> Any clues about transforming this code would be much appreciated.
>>
>> (Note: I'm backporting because the Python 2.6 version appears to be much
>> more
>> robust than the 0.52 third-party release.)
>
> Good timing, Skip! I was planing to do a backport to 2.5, too. I've some
> experience with both the old and the new buffer protocol. I might be of some
> assistance to you.
>
> I like to make as much code of the trunk version compatible with 2.5 and 2.4
> as possible. Let's see how far we can get with a bunch of macros and
> #ifdefs.
>
> Christian
>
> _______________________________________________
> Python-3000 mailing list
> Python-3000 at python.org
> http://mail.python.org/mailman/listinfo/python-3000
> Unsubscribe:
> http://mail.python.org/mailman/options/python-3000/jnoller%40gmail.com
>

From lists at cheimes.de  Thu Oct 16 21:38:27 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 16 Oct 2008 21:38:27 +0200
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <4222a8490810161230xca234c6y15c8e0733fb62b2e@mail.gmail.com>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>	
	<gd84l0$v7$1@ger.gmane.org>
	<4222a8490810161230xca234c6y15c8e0733fb62b2e@mail.gmail.com>
Message-ID: <48F79833.9000301@cheimes.de>

Jesse Noller wrote:
> Do we want to start a google code project for this given all three of
> us are interested in this? :)

Do we need (yet) another Google code project? Isn't svn.python.org 
sufficient for our needs? I'm -0 on a Google code project but I'll give 
you my gmail account if you insist on one.

Christian

From brett at python.org  Thu Oct 16 21:59:30 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 16 Oct 2008 12:59:30 -0700
Subject: [Python-Dev] Documentation idea
In-Reply-To: <48F7843B.7070709@scottdial.com>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<48F7843B.7070709@scottdial.com>
Message-ID: <bbaeab100810161259y4a56538end94698e25b67733a@mail.gmail.com>

On Thu, Oct 16, 2008 at 11:13 AM, Scott Dial
<scott+python-ideas at scottdial.com> wrote:
> Raymond Hettinger wrote:
>> * It will assist pypy style projects and other python implementations
>> when they have to build equivalents to CPython.
>>
>> * Will eliminate confusion about what functions were exactly intended to
>> do.
>>
>> * Will confer benefits similar to test driven development where the
>> documentation and  pure python version are developed first and doctests
>> gotten to pass, then the C version is created to match.
>
> I haven't seen anyone comment about this assertion of "equivalence".
> Doesn't it strike you as difficult to maintain *two* versions of every
> function, and ensure they match *exactly*?

More time-consuming than difficult. Raymond is currently talking about
things like built-ins and methods on types who do not exactly change
very often.

> The utility to PyPy-style
> projects is minimized if the two version aren't identical. And while
> it's possible to say, "the tests say they are equiavelent, so they are;"
> history is quite clear about people depending on "features" that are
> untested and were unintended side-effects of the manner in which
> something was implemented.

Right, and when we find out that there is a difference, we typically
standardize on a specific version and developers using the bogus
semantics switch.

> I think it would be a dilution of developer
> man-hours to force them to maintain two versions in lock-step, and it
> significantly adds to the burden of writing and reviewing potential
> bugfixes.
>

Well, I don't see this applying to every extension module in the
stdlib that does not already have a pure Python equivalent. This view
also assumes that if this position was taken people will continue to
write extension modules when they are not necessarily needed. If this
actually makes people to write more pure Python code over extension
modules I think that is a plus.

And Raymond, more than probably anyone, can address the overhead he
has faced in maintaining both the pure Python version of itertools in
the docs and the extension module.

> While I applaud the idea of documenting C functions in this manner,
> let's not confuse documentation with equivalence. If the standard
> distribution of Python exports the C version, then all bets are off
> whether the Python version is a drop-in replacement (even if the
> buildbots regularly test them).

Well, considering we have not even gotten far enough to actually do
this for the documentation case, I think worrying about equivalence
might be jumping the gun slightly as it is more work as you point out,
Scott.

But one thing about doing this is it might draw in the various
alternative VM folks to help maintain the Python code. If Jython,
IronPython, and/or PyPy actually use the Python code for themselves
then I suspect they would help with maintenance.

-Brett

From jnoller at gmail.com  Thu Oct 16 22:03:14 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 16 Oct 2008 16:03:14 -0400
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <48F79833.9000301@cheimes.de>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<gd84l0$v7$1@ger.gmane.org>
	<4222a8490810161230xca234c6y15c8e0733fb62b2e@mail.gmail.com>
	<48F79833.9000301@cheimes.de>
Message-ID: <4222a8490810161303t2a5edd14i934e0c7bc7f3f39e@mail.gmail.com>

On Thu, Oct 16, 2008 at 3:38 PM, Christian Heimes <lists at cheimes.de> wrote:
> Jesse Noller wrote:
>>
>> Do we want to start a google code project for this given all three of
>> us are interested in this? :)
>
> Do we need (yet) another Google code project? Isn't svn.python.org
> sufficient for our needs? I'm -0 on a Google code project but I'll give you
> my gmail account if you insist on one.
>
> Christian
>

I've not used svn.python.org for personal side/projects - also,
ideally the back port would be stand-alone and package-index
installable

From python at rcn.com  Thu Oct 16 23:11:40 2008
From: python at rcn.com (Raymond Hettinger)
Date: Thu, 16 Oct 2008 14:11:40 -0700
Subject: [Python-Dev] Documentation idea
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<48F7843B.7070709@scottdial.com>
Message-ID: <FEDCF14B54374F25A589CE5B749AC636@RaymondLaptop1>

> Raymond Hettinger wrote:
>> * It will assist pypy style projects and other python implementations
>> when they have to build equivalents to CPython.
>>
>> * Will eliminate confusion about what functions were exactly intended to
>> do.
>>
>> * Will confer benefits similar to test driven development where the
>> documentation and  pure python version are developed first and doctests
>> gotten to pass, then the C version is created to match.
>
> I haven't seen anyone comment about this assertion of "equivalence".
> Doesn't it strike you as difficult to maintain *two* versions of every
> function, and ensure they match *exactly*?

Glad you brought this up.  My idea is to present rough equivalence
in unoptimized python that is simple and clear.  The goal is to provide
better documentation where code is more precise than English prose.
That being said, some subset of the existing tests should be runnable
against the rough equivalent and the python code should incorporate doctests.
Running both sets of test should suffice to maintain the rough equivalence.

The notion of exact equivalence should be left to PyPy folks who can attest
that the code can get convoluted when you try to simulate exactly when
error checking is performed, read-only behavior for attributes, and making
the stacktraces look the same when there are errors.  In contrast, my
goal is an approximation that is executable but highly readable and expository.

My thought is to do this only with tools where it really does enhance the
documentation.  The exercise is worthwhile in and of itself.  For example,
I'm working on a pure python version of str.split() and quickly determined
that the docs are *still* in error even after many revisions over the years
(the whitespace version does not, in fact, start by stripping whitespace
from both ends).  Here's what I have so far:

def split(s, sep=None, maxsplit=-1):
    """split(S, [sep [,maxsplit]]) -> list of strings

    Return a list of the words in the string S, using sep as the
    delimiter string.  If maxsplit is given, at most maxsplit
    splits are done. If sep is not specified or is None, any
    whitespace string is a separator and empty strings are removed
    from the result.

    >>> from itertools import product
    >>> s = ' 11   2  333  4  '
    >>> split(s, None)
    ['11', '2', '333', '4']
    >>> n = 8
    >>> for s in product('ab ', repeat=n):
    ...     for maxsplit in range(-2, len(s)+2):
    ...         s = ''.join(s)
    ...         assert s.split(None, maxsplit) == split(s, None, maxsplit), namedtuple('Err', 'str maxsplit result target')(repr(s), 
maxsplit, split(s,None,maxsplit), s.split(None, maxsplit))

    """
    result = []
    spmode = True
    start = 0
    if maxsplit != 0:
        for i, c in enumerate(s):
            if spmode:
                if not c.isspace():
                    start = i
                    spmode = False
            elif c.isspace():
                result.append(s[start:i])
                start = i
                spmode = True
                if len(result) == maxsplit:
                    break
    rest = s[start:].lstrip()
    return (result + [rest]) if rest else result

Once I have the cleanest possible, self-explantory code that passes tests, I'll improve the variable names and make a more sensible 
docstring with readable examples.  Surprisingly, it hasn't been a trivial exercise to come-up with an equivalent that corresponds 
more closely to the way we think instead of corresponding the C code -- I want to show *what* is does more than *how* it does it.


Raymond


From doug.hellmann at gmail.com  Thu Oct 16 23:28:01 2008
From: doug.hellmann at gmail.com (Doug Hellmann)
Date: Thu, 16 Oct 2008 17:28:01 -0400
Subject: [Python-Dev] Documentation idea
In-Reply-To: <FEDCF14B54374F25A589CE5B749AC636@RaymondLaptop1>
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<48F7843B.7070709@scottdial.com>
	<FEDCF14B54374F25A589CE5B749AC636@RaymondLaptop1>
Message-ID: <23E84B70-49B4-4BD5-BDD5-1485B352C53E@gmail.com>


On Oct 16, 2008, at 5:11 PM, Raymond Hettinger wrote:

>> Raymond Hettinger wrote:
>>> * It will assist pypy style projects and other python  
>>> implementations
>>> when they have to build equivalents to CPython.
>>>
>>> * Will eliminate confusion about what functions were exactly  
>>> intended to
>>> do.
>>>
>>> * Will confer benefits similar to test driven development where the
>>> documentation and  pure python version are developed first and  
>>> doctests
>>> gotten to pass, then the C version is created to match.
>>
>> I haven't seen anyone comment about this assertion of "equivalence".
>> Doesn't it strike you as difficult to maintain *two* versions of  
>> every
>> function, and ensure they match *exactly*?
>
> Glad you brought this up.  My idea is to present rough equivalence
> in unoptimized python that is simple and clear.  The goal is to  
> provide
> better documentation where code is more precise than English prose.
> That being said, some subset of the existing tests should be runnable
> against the rough equivalent and the python code should incorporate  
> doctests.
> Running both sets of test should suffice to maintain the rough  
> equivalence.

This seems like a large undertaking.  I'm sure you're not  
underestimating the effort, but I have the sense that you may be  
overestimating the usefulness of the results (or maybe I'm  
underestimating them through some lack of understanding).  Would it be  
more optimal (in terms of both effort and results) to extend the  
existing documentation and/or docstrings with examples that use all of  
the functions so developers can see how to call them and what results  
to expect?

Doug


From python at rcn.com  Thu Oct 16 23:37:56 2008
From: python at rcn.com (Raymond Hettinger)
Date: Thu, 16 Oct 2008 14:37:56 -0700
Subject: [Python-Dev] Documentation idea
References: <D6DA0C73CDA04A60A441D12FF5BE6FD9@RaymondLaptop1>
	<48F7843B.7070709@scottdial.com>
	<FEDCF14B54374F25A589CE5B749AC636@RaymondLaptop1>
	<23E84B70-49B4-4BD5-BDD5-1485B352C53E@gmail.com>
Message-ID: <C57CCEB4706C40F1A24B1228AE759BD8@RaymondLaptop1>

From: "Doug Hellmann" <doug.hellmann at gmail.com
> This seems like a large undertaking.

Not necessarily.  It can be done incrementally, starting with things like str.split() that almost no one understands completely.  It 
should be put here and there where it adds some clarity.


> I'm sure you're not  underestimating the effort, but I have the sense that you may be  overestimating the usefulness of the 
> results (or maybe I'm  underestimating them through some lack of understanding).  Would it be  more optimal (in terms of both 
> effort and results) to extend the  existing documentation and/or docstrings with examples that use all of  the functions so 
> developers can see how to call them and what results  to expect?

The idea includes pure python code augmented by doctestable doctrings
with enough examples.  So, we're almost talking about the same thing.
There is one difference; since the new attribute is guaranteed to be
executable, it can be reliably run through doctest.  The same is *not* true
for arbitrary docstrings.


Raymond


From ncoghlan at gmail.com  Fri Oct 17 00:06:30 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 17 Oct 2008 08:06:30 +1000
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
Message-ID: <48F7BAE6.4070108@gmail.com>

Skip Montanaro wrote:
> (Note: I'm backporting because the Python 2.6 version appears to be much more
> robust than the 0.52 third-party release.)

As Jesse points out, some of that robustness comes from long-standing
bugs in the core getting fixed as a result of the addition of the
multiprocessing unit tests to the standard library test suite.

Not trying to discourage the project, just pointing out that it may not
be as effective as hoped without patching the older versions of the
interpreter.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From lists at cheimes.de  Fri Oct 17 00:21:39 2008
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 17 Oct 2008 00:21:39 +0200
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <48F7BAE6.4070108@gmail.com>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<48F7BAE6.4070108@gmail.com>
Message-ID: <48F7BE73.3010303@cheimes.de>

Nick Coghlan wrote:
> As Jesse points out, some of that robustness comes from long-standing
> bugs in the core getting fixed as a result of the addition of the
> multiprocessing unit tests to the standard library test suite.
> 
> Not trying to discourage the project, just pointing out that it may not
> be as effective as hoped without patching the older versions of the
> interpreter.

Oh h...
Are you able to recall a list of the most important bug fixes? Maybe we 
can get the bug fixes into 2.5.3 before it's too late.

Christian

From ncoghlan at gmail.com  Fri Oct 17 11:06:13 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 17 Oct 2008 19:06:13 +1000
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <48F7BE73.3010303@cheimes.de>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<48F7BAE6.4070108@gmail.com> <48F7BE73.3010303@cheimes.de>
Message-ID: <48F85585.8090201@gmail.com>

Christian Heimes wrote:
> Nick Coghlan wrote:
>> As Jesse points out, some of that robustness comes from long-standing
>> bugs in the core getting fixed as a result of the addition of the
>> multiprocessing unit tests to the standard library test suite.
>>
>> Not trying to discourage the project, just pointing out that it may not
>> be as effective as hoped without patching the older versions of the
>> interpreter.
> 
> Oh h...
> Are you able to recall a list of the most important bug fixes? Maybe we
> can get the bug fixes into 2.5.3 before it's too late.

The one Jesse linked in his python-dev post was the one that blocked it
the longest:
http://bugs.python.org/issue874900

However, if I'm reading the discussion in the tracker correctly, the fix
was applied to all 3 branches (2.5, trunk, 3k). So it is only people
using versions <= 2.5.2 that will suffer that particular problem.

I think there were a couple of others as well, but it would take a trawl
through the py3k mailing list archives to figure out what they were (I'm
pretty sure Jesse posted a list of the issues that needed to be fixed to
get the multiprocessing unit tests passing reliably).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From skip at pobox.com  Fri Oct 17 16:44:27 2008
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 17 Oct 2008 09:44:27 -0500
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <gd84l0$v7$1@ger.gmane.org>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<gd84l0$v7$1@ger.gmane.org>
Message-ID: <18680.42187.731857.59208@montanaro-dyndns-org.local>


    Christian> I like to make as much code of the trunk version compatible
    Christian> with 2.5 and 2.4 as possible. Let's see how far we can get
    Christian> with a bunch of macros and #ifdefs.

I'll follow your lead. ;-)

Skip

From skip at pobox.com  Fri Oct 17 16:45:37 2008
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 17 Oct 2008 09:45:37 -0500
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <4222a8490810161230xca234c6y15c8e0733fb62b2e@mail.gmail.com>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<gd84l0$v7$1@ger.gmane.org>
	<4222a8490810161230xca234c6y15c8e0733fb62b2e@mail.gmail.com>
Message-ID: <18680.42257.84013.537080@montanaro-dyndns-org.local>


    Jesse> Do we want to start a google code project for this given all
    Jesse> three of us are interested in this? :)

Maybe the svn repo could grow a backports sibling of sandbox.

Skip

From skip at pobox.com  Fri Oct 17 16:47:10 2008
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 17 Oct 2008 09:47:10 -0500
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <4222a8490810161303t2a5edd14i934e0c7bc7f3f39e@mail.gmail.com>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<gd84l0$v7$1@ger.gmane.org>
	<4222a8490810161230xca234c6y15c8e0733fb62b2e@mail.gmail.com>
	<48F79833.9000301@cheimes.de>
	<4222a8490810161303t2a5edd14i934e0c7bc7f3f39e@mail.gmail.com>
Message-ID: <18680.42350.962999.456461@montanaro-dyndns-org.local>


    Jesse> I've not used svn.python.org for personal side/projects - also,
    Jesse> ideally the back port would be stand-alone and package-index
    Jesse> installable

I wouldn't call this really a personal/side project.  OTOH, firing up a
Google Code project means you can admit project developers without giving
them the keys to the kingdom so-to-speak.

Skip

From skip at pobox.com  Fri Oct 17 16:50:34 2008
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 17 Oct 2008 09:50:34 -0500
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <48F7BE73.3010303@cheimes.de>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<48F7BAE6.4070108@gmail.com> <48F7BE73.3010303@cheimes.de>
Message-ID: <18680.42554.791704.321274@montanaro-dyndns-org.local>


    Christian> Oh h...  Are you able to recall a list of the most important
    Christian> bug fixes? Maybe we can get the bug fixes into 2.5.3 before
    Christian> it's too late.

Maybe doing the modest amount of translation required of the 2.6 unit tests
so they run under 0.52 would help.  See what fails and then see what fixes
correspond to fixing those failing tests.

Skip


From jnoller at gmail.com  Fri Oct 17 16:55:22 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 17 Oct 2008 10:55:22 -0400
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <18680.42350.962999.456461@montanaro-dyndns-org.local>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<gd84l0$v7$1@ger.gmane.org>
	<4222a8490810161230xca234c6y15c8e0733fb62b2e@mail.gmail.com>
	<48F79833.9000301@cheimes.de>
	<4222a8490810161303t2a5edd14i934e0c7bc7f3f39e@mail.gmail.com>
	<18680.42350.962999.456461@montanaro-dyndns-org.local>
Message-ID: <4222a8490810170755v15e18b4er8bfecefcc55975e1@mail.gmail.com>

On Fri, Oct 17, 2008 at 10:47 AM,  <skip at pobox.com> wrote:
>
>    Jesse> I've not used svn.python.org for personal side/projects - also,
>    Jesse> ideally the back port would be stand-alone and package-index
>    Jesse> installable
>
> I wouldn't call this really a personal/side project.  OTOH, firing up a
> Google Code project means you can admit project developers without giving
> them the keys to the kingdom so-to-speak.
>
> Skip
>

Fair enough :)

I fired up http://code.google.com/p/python-multiprocessing/ last
night, and added you and Christian - anyone else wanting in on this
can ping me.

Skip - I know you had some work you already had on the bench for
pulling out-repackaging the MP stuff, do you want to commit that and
then we can work from there?

We should also probably take this off the dev list

-jesse

From status at bugs.python.org  Fri Oct 17 18:06:35 2008
From: status at bugs.python.org (Python tracker)
Date: Fri, 17 Oct 2008 18:06:35 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20081017160635.A1E637829A@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (10/10/08 - 10/17/08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2104 open (+21) / 13859 closed (+19) / 15963 total (+40)

Open issues with patches:   689

Average duration of open issues: 714 days.
Median duration of open issues: 1891 days.

Open Issues Breakdown
   open  2088 (+21)
pending    16 ( +0)

Issues Created Or Reopened (41)
_______________________________

Get rid of more refercenes to __cmp__                            10/16/08
       http://bugs.python.org/issue1717    reopened georg.brandl              
       patch                                                                   

Missing type in "types" module                                   10/10/08
CLOSED http://bugs.python.org/issue4101    created  Antoine d'Otreppe         
                                                                               

sphinx: ansi color even on dumb terminal                         10/10/08
CLOSED http://bugs.python.org/issue4102    created  dottedmag                 
                                                                               

sphinx: latexwriter uses undefined 'excdescni' environment       10/10/08
CLOSED http://bugs.python.org/issue4103    created  dottedmag                 
                                                                               

Namespace inconsistency                                          10/10/08
CLOSED http://bugs.python.org/issue4104    created  Antoine d'Otreppe         
                                                                               

Renamed PyGILState_Acquire to PyGILState_Ensure in Docs/c-api/in 10/10/08
CLOSED http://bugs.python.org/issue4105    created  gpolo                     
       patch                                                                   

multiprocessing occasionally spits out exception during shutdown 10/11/08
       http://bugs.python.org/issue4106    created  skip.montanaro            
                                                                               

Backticks still mentioned in documentation                       10/12/08
CLOSED http://bugs.python.org/issue4107    created  jamesie                   
                                                                               

robotparser.py fail when more than one User-Agent: * is present  10/12/08
       http://bugs.python.org/issue4108    created  thsajid                   
                                                                               

Wrong UnboundLocalError with += operator                         10/12/08
CLOSED http://bugs.python.org/issue4109    created  atienza                   
                                                                               

A minor slip while typing                                        10/12/08
CLOSED http://bugs.python.org/issue4110    created  vinetou                   
                                                                               

Add DTrace probes                                                10/12/08
       http://bugs.python.org/issue4111    created  bretthoerner              
       patch                                                                   

Subprocess: Popen'ed children hang due to open pipes             10/13/08
       http://bugs.python.org/issue4112    created  boye                      
                                                                               

functools.partial(), no __name__; wrong __doc__                  10/13/08
       http://bugs.python.org/issue4113    created  stribb                    
                                                                               

struct returns incorrect 4 byte float                            10/13/08
CLOSED http://bugs.python.org/issue4114    created  TD22057                   
                                                                               

split() method                                                   10/13/08
CLOSED http://bugs.python.org/issue4115    created  terry.scott               
                                                                               

name conflict in ScrolledCanvas.__init__() in Lib/turtle.py      10/13/08
       http://bugs.python.org/issue4116    created  gregorlingl               
       patch                                                                   

missing update() in _Screen.setup() of Lib/turtle.py             10/13/08
       http://bugs.python.org/issue4117    created  gregorlingl               
       patch                                                                   

Built-in compile() and ast module doc issues                     10/13/08
       http://bugs.python.org/issue4118    created  tjreedy                   
                                                                               

ssl wrapper: add makefile() method                               10/13/08
CLOSED http://bugs.python.org/issue4119    created  haypo                     
                                                                               

Do not embed manifest files in *.pyd when compiling with MSVC    10/14/08
       http://bugs.python.org/issue4120    reopened loewis                    
       patch                                                                   

open(): use keyword only for arguments other than file and mode  10/14/08
       http://bugs.python.org/issue4121    created  haypo                     
       patch, patch, needs review                                              

undefined reference to _Py_ascii_whitespace                      10/14/08
CLOSED http://bugs.python.org/issue4122    created  schmir                    
       patch                                                                   

random.shuffle slow on deque                                     10/14/08
CLOSED http://bugs.python.org/issue4123    created  phr                       
       patch                                                                   

Patch for adding "default" to itemgetter and attrgetter          10/14/08
       http://bugs.python.org/issue4124    created  tebeka                    
       patch                                                                   

runtests.sh: use -bb flag of Python                              10/17/08
CLOSED http://bugs.python.org/issue4125    reopened amaury.forgeotdarc        
       patch                                                                   

remove not decodable environment variables                       10/15/08
       http://bugs.python.org/issue4126    created  haypo                     
       patch                                                                   

repr or reprlib?                                                 10/15/08
CLOSED http://bugs.python.org/issue4127    created  LambertDW                 
                                                                               

Performance regression in long division in 2.6                   10/15/08
       http://bugs.python.org/issue4128    created  fredrikj                  
                                                                               

C/API documentation: request for documentation of change to Py_s 10/15/08
       http://bugs.python.org/issue4129    created  schiotz                   
                                                                               

Intel icc 9.1 does not support __int128_t used by ctypes         10/15/08
       http://bugs.python.org/issue4130    created  jared.jennings            
                                                                               

Note that Firefox 3 does not write cookies.txt                   10/15/08
CLOSED http://bugs.python.org/issue4131    created  jjlee                     
       patch                                                                   

LaTeX Error: Environment cmemberdescni undefined.                10/16/08
CLOSED http://bugs.python.org/issue4132    created  wplappert                 
                                                                               

-R fix for setup.py                                              10/16/08
CLOSED http://bugs.python.org/issue4133    created  skip.montanaro            
       patch                                                                   

Peephole optimization: returning a temp                          10/16/08
CLOSED http://bugs.python.org/issue4134    created  novalis_dt                
       patch                                                                   

help("modules ftp") fails due to test modules                    10/16/08
       http://bugs.python.org/issue4135    created  throwaway                 
                                                                               

merge json library with simplejson 2.0.3                         10/16/08
       http://bugs.python.org/issue4136    created  bob.ippolito              
                                                                               

update SIG web pages                                             10/17/08
       http://bugs.python.org/issue4137    created  techtonik                 
                                                                               

IDLE crashes in my Windows Vista                                 10/17/08
       http://bugs.python.org/issue4138    created  xuejm1225                 
                                                                               

Major error in cmath routines                                    10/17/08
       http://bugs.python.org/issue4139    created  thor222                   
                                                                               

urllib2: request with digest auth through proxy fail             10/17/08
       http://bugs.python.org/issue4140    created  jan.kollhof               
                                                                               



Issues Now Closed (50)
______________________

cmath is numerically unsound                                      345 days
       http://bugs.python.org/issue1381    jcea                      
       patch                                                                   

class attribute cache failure (regression)                        267 days
       http://bugs.python.org/issue1878    jcea                      
                                                                               

patch to fix_import: UserDict -> collections                      250 days
       http://bugs.python.org/issue2046    benjamin.peterson         
       patch                                                                   

MSVCRT packing in Windows Installer (3.0a4)                       185 days
       http://bugs.python.org/issue2642    loewis                    
                                                                               

Regression for executing packages                                 162 days
       http://bugs.python.org/issue2751    ncoghlan                  
                                                                               

Add memory footprint query                                        149 days
       http://bugs.python.org/issue2898    jcea                      
       patch                                                                   

2to3 should have a way to disable some fixers                     142 days
       http://bugs.python.org/issue2956    benjamin.peterson         
       patch                                                                   

Idle uses old showwarning signature                                91 days
       http://bugs.python.org/issue3391    benjamin.peterson         
       patch                                                                   

sre "bytecode" verifier                                            73 days
       http://bugs.python.org/issue3487    jcea                      
       patch, patch                                                            

compile() cannot decode Latin-1 source encodings                   61 days
       http://bugs.python.org/issue3574    brett.cannon              
       patch                                                                   

_json: fix raise_errmsg(), py_encode_basestring_ascii() and line   57 days
       http://bugs.python.org/issue3623    benjamin.peterson         
       patch, needs review                                                     

sys.call_tracing segfaults                                         53 days
       http://bugs.python.org/issue3661    barry                     
       patch, needs review                                                     

Crash while compiling Python 3000 in OpenBSD 4.4                   52 days
       http://bugs.python.org/issue3685    barry                     
       patch, needs review                                                     

Py_InitModule* is still referenced in docs                         50 days
       http://bugs.python.org/issue3717    loewis                    
       patch, needs review                                                     

telnetlib module broken by str to unicode conversion               47 days
       http://bugs.python.org/issue3725    benjamin.peterson         
       patch                                                                   

warnings.catch_warnings fails gracelessly when recording warning   38 days
       http://bugs.python.org/issue3781    benjamin.peterson         
       patch, needs review                                                     

Make PyInstanceMethod_Type available or remove it                  41 days
       http://bugs.python.org/issue3787    benjamin.peterson         
       patch, patch, needs review                                              

Building PDF documentation from tex files                          27 days
       http://bugs.python.org/issue3909    georg.brandl              
       patch                                                                   

Byte warning mode and b'' != ''                                    19 days
       http://bugs.python.org/issue3988    barry                     
       patch                                                                   

"for me" installer problem on x64 Vista                            11 days
       http://bugs.python.org/issue4018    loewis                    
       patch, needs review                                                     

wrong page index number in reference book of python documentatio   14 days
       http://bugs.python.org/issue4027    wplappert                 
                                                                               

Warnings in IDLE raise TypeError (such as attempting to import d   11 days
       http://bugs.python.org/issue4043    benjamin.peterson         
                                                                               

build_py support for lib2to3 is stale                               7 days
       http://bugs.python.org/issue4072    loewis                    
       patch                                                                   

try statement in language reference not updated for v2.6            8 days
       http://bugs.python.org/issue4083    georg.brandl              
                                                                               

Patch to implement a real poplib test suite                         2 days
       http://bugs.python.org/issue4088    benjamin.peterson         
       patch                                                                   

linking python2.6.dll crash on windows xp                           1 days
       http://bugs.python.org/issue4089    theller                   
                                                                               

python dll not installed in windows system directory                8 days
       http://bugs.python.org/issue4091    loewis                    
       patch, needs review                                                     

Missing type in "types" module                                      0 days
       http://bugs.python.org/issue4101    benjamin.peterson         
                                                                               

sphinx: ansi color even on dumb terminal                            6 days
       http://bugs.python.org/issue4102    georg.brandl              
                                                                               

sphinx: latexwriter uses undefined 'excdescni' environment          6 days
       http://bugs.python.org/issue4103    georg.brandl              
                                                                               

Namespace inconsistency                                             2 days
       http://bugs.python.org/issue4104    loewis                    
                                                                               

Renamed PyGILState_Acquire to PyGILState_Ensure in Docs/c-api/in    0 days
       http://bugs.python.org/issue4105    benjamin.peterson         
       patch                                                                   

Backticks still mentioned in documentation                          0 days
       http://bugs.python.org/issue4107    benjamin.peterson         
                                                                               

Wrong UnboundLocalError with += operator                            0 days
       http://bugs.python.org/issue4109    benjamin.peterson         
                                                                               

A minor slip while typing                                           0 days
       http://bugs.python.org/issue4110    georg.brandl              
                                                                               

struct returns incorrect 4 byte float                               0 days
       http://bugs.python.org/issue4114    loewis                    
                                                                               

split() method                                                      0 days
       http://bugs.python.org/issue4115    terry.scott               
                                                                               

ssl wrapper: add makefile() method                                  0 days
       http://bugs.python.org/issue4119    haypo                     
                                                                               

undefined reference to _Py_ascii_whitespace                         0 days
       http://bugs.python.org/issue4122    amaury.forgeotdarc        
       patch                                                                   

random.shuffle slow on deque                                        2 days
       http://bugs.python.org/issue4123    benjamin.peterson         
       patch                                                                   

runtests.sh: use -bb flag of Python                                 0 days
       http://bugs.python.org/issue4125    barry                     
       patch                                                                   

repr or reprlib?                                                    0 days
       http://bugs.python.org/issue4127    benjamin.peterson         
                                                                               

Note that Firefox 3 does not write cookies.txt                      1 days
       http://bugs.python.org/issue4131    georg.brandl              
       patch                                                                   

LaTeX Error: Environment cmemberdescni undefined.                   0 days
       http://bugs.python.org/issue4132    georg.brandl              
                                                                               

-R fix for setup.py                                                 0 days
       http://bugs.python.org/issue4133    benjamin.peterson         
       patch                                                                   

Peephole optimization: returning a temp                             0 days
       http://bugs.python.org/issue4134    novalis_dt                
       patch                                                                   

Fix for bugs relating to ntpath.expanduser()                       13 days
       http://bugs.python.org/issue957650  loewis                    
       patch                                                                   

Traceback error when compiling Regex                              936 days
       http://bugs.python.org/issue1456280 timehorse                 
                                                                               

object.__init__ shouldn't allow args/kwds                         574 days
       http://bugs.python.org/issue1683368 jcea                      
                                                                               

Method cache                                                      571 days
       http://bugs.python.org/issue1685986 jcea                      
       patch                                                                   



Top Issues Most Discussed (10)
______________________________

 13 poplib module broken by str to unicode conversion                 49 days
open    http://bugs.python.org/issue3727   

 13 create a numbits() method for int and long types                  85 days
open    http://bugs.python.org/issue3439   

 11 struct returns incorrect 4 byte float                              0 days
closed  http://bugs.python.org/issue4114   

 11 wrong page index number in reference book of python documentati   14 days
closed  http://bugs.python.org/issue4027   

  8 imaplib does not run under Python 3                              386 days
open    http://bugs.python.org/issue1210   

  7 Tkinter cannot find Tcl/Tk on Mac OS X                            15 days
open    http://bugs.python.org/issue4017   

  7 compile() cannot decode Latin-1 source encodings                  61 days
closed  http://bugs.python.org/issue3574   

  6 remove not decodable environment variables                         3 days
open    http://bugs.python.org/issue4126   

  6 random.shuffle slow on deque                                       2 days
closed  http://bugs.python.org/issue4123   

  6 Do not embed manifest files in *.pyd when compiling with MSVC      3 days
open    http://bugs.python.org/issue4120   




From lists at cheimes.de  Fri Oct 17 20:56:58 2008
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 17 Oct 2008 20:56:58 +0200
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <18680.42554.791704.321274@montanaro-dyndns-org.local>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<48F7BAE6.4070108@gmail.com> <48F7BE73.3010303@cheimes.de>
	<18680.42554.791704.321274@montanaro-dyndns-org.local>
Message-ID: <48F8DFFA.9090707@cheimes.de>

skip at pobox.com wrote:
>     Christian> Oh h...  Are you able to recall a list of the most important
>     Christian> bug fixes? Maybe we can get the bug fixes into 2.5.3 before
>     Christian> it's too late.
> 
> Maybe doing the modest amount of translation required of the 2.6 unit tests
> so they run under 0.52 would help.  See what fails and then see what fixes
> correspond to fixing those failing tests.

Sounds like a good plan. Let's get started! Are you going to commit your 
work to the Google Code repository anytime soon?

Christian


From gregor.lingl at aon.at  Sat Oct 18 00:18:00 2008
From: gregor.lingl at aon.at (Gregor Lingl)
Date: Sat, 18 Oct 2008 00:18:00 +0200
Subject: [Python-Dev] Old green snake icon on Windows
Message-ID: <48F90F18.5010005@aon.at>

I've installed the new Python 2.6 for windows (Windows-installer)
on several machines among them a new one which has never seen
Python before and all these installations show the old fashioned green
snake logo (from Python 2.3 or before, I think) in the automatically
created menu-entries.

Has anybody else observed this?

What could be the reason for this?

Is this intentional, and if so what is the intention?

Or is there something wrong with the installer?
 
(I even cannot change  the icons for the menu-entries  because  these
links don't behave like ordinary links and the change-icon-button
in the property dialog is greyed out).

Should this be posted to the issue-tracker
Regards,
Gregor

From martin at v.loewis.de  Sat Oct 18 00:41:00 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 18 Oct 2008 00:41:00 +0200
Subject: [Python-Dev] Old green snake icon on Windows
In-Reply-To: <48F90F18.5010005@aon.at>
References: <48F90F18.5010005@aon.at>
Message-ID: <48F9147C.7050704@v.loewis.de>

> Has anybody else observed this?

Yes, see http://bugs.python.org/issue4019

> What could be the reason for this?

I forgot to invoke nmake in the appropriate directory.

> Is this intentional, and if so what is the intention?

It's not intentional.

> Should this be posted to the issue-tracker

No, see above.

Regards,
Martin

P.S. It's too bad that nobody noticed this in all these alpha and beta
releases.

From helmert at informatik.uni-freiburg.de  Sat Oct 18 22:05:26 2008
From: helmert at informatik.uni-freiburg.de (Malte Helmert)
Date: Sat, 18 Oct 2008 22:05:26 +0200
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <48EBBEDD.7010109@v.loewis.de>
References: <48EB0F5C.602@v.loewis.de> <20081007145140.GA11461@panix.com>
	<48EBBEDD.7010109@v.loewis.de>
Message-ID: <gddfb9$td3$1@ger.gmane.org>

Martin v. L?wis wrote:
>> Just to emphasize this, "changes" means "bugfixes".  (I'm mentioning this
>> mainly because of the people who joined for 2.6/3.0.)  For more info,
>> see PEP6 about bugfix releases:
>> http://www.python.org/dev/peps/pep-0006/
> 
> Thanks for clarifying this. For the last 2.5.x release in particular, we
> will strictly enforce the "no new features" policy; users interested
> in new features should switch to 2.6.

May I suggest http://bugs.python.org/issue1040026 ?

It has a fairly simple patch (posixmodule.diff), a new test
(test_posix5.PATCH), and it fixes a bug that makes os.times unusable on
common platforms.

Malte


From martin at v.loewis.de  Sun Oct 19 00:38:00 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 19 Oct 2008 00:38:00 +0200
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <gddfb9$td3$1@ger.gmane.org>
References: <48EB0F5C.602@v.loewis.de>
	<20081007145140.GA11461@panix.com>	<48EBBEDD.7010109@v.loewis.de>
	<gddfb9$td3$1@ger.gmane.org>
Message-ID: <48FA6548.20509@v.loewis.de>

> May I suggest http://bugs.python.org/issue1040026 ?
> 
> It has a fairly simple patch (posixmodule.diff), a new test
> (test_posix5.PATCH), and it fixes a bug that makes os.times unusable on
> common platforms.

In the current form, I'm skeptical about applying this patch to 2.5.2.

It has the possibility of breaking compilation; such patches are
unacceptable. See my comments for details.

Regards,
Martin

From helmert at informatik.uni-freiburg.de  Sun Oct 19 00:46:29 2008
From: helmert at informatik.uni-freiburg.de (Malte Helmert)
Date: Sun, 19 Oct 2008 00:46:29 +0200
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <48FA6548.20509@v.loewis.de>
References: <48EB0F5C.602@v.loewis.de>	<20081007145140.GA11461@panix.com>	<48EBBEDD.7010109@v.loewis.de>	<gddfb9$td3$1@ger.gmane.org>
	<48FA6548.20509@v.loewis.de>
Message-ID: <gddop6$pga$1@ger.gmane.org>

Martin v. L?wis wrote:
>> May I suggest http://bugs.python.org/issue1040026 ?
>>
>> It has a fairly simple patch (posixmodule.diff), a new test
>> (test_posix5.PATCH), and it fixes a bug that makes os.times unusable on
>> common platforms.
> 
> In the current form, I'm skeptical about applying this patch to 2.5.2.
> 
> It has the possibility of breaking compilation; such patches are
> unacceptable. See my comments for details.

OK, these should be easy to address. (Comments in the tracker.)

Malte


From skip at pobox.com  Sun Oct 19 02:39:35 2008
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 18 Oct 2008 19:39:35 -0500
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <48F8DFFA.9090707@cheimes.de>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<48F7BAE6.4070108@gmail.com> <48F7BE73.3010303@cheimes.de>
	<18680.42554.791704.321274@montanaro-dyndns-org.local>
	<48F8DFFA.9090707@cheimes.de>
Message-ID: <18682.33223.833779.771437@montanaro-dyndns-org.local>


    >> Maybe doing the modest amount of translation required of the 2.6 unit
    >> tests so they run under 0.52 would help.  See what fails and then see
    >> what fixes correspond to fixing those failing tests.

    Christian> Sounds like a good plan. Let's get started! Are you going to
    Christian> commit your work to the Google Code repository anytime soon?

Folks,

My apologies.  I have been essentially off-net for the past couple of days.
Reason one: we are in the midst of moving.  Reason two: our first grandchild
(Carmine Michael Montanaro) was born early Friday morning.  (yay!)  Between
visiting Carmine and moving/packing I haven't really been close to a
computer since Thursday mid-afternoon.  (I'm writing this reply off-net at
the moment.  Who knows when I'll get back within range of a wireless
signal.)

I will try to get close enough to the net for a small amount of time Sunday
and upload what I have to Google Code.  It ain't much, so if you're
impatient, you can pretty much replicate what I did:

    find . -name '*processing*' | egrep -v framework\|build\|PC | xargs tar --create --verbose --file=$HOME/tmp/multiprocessing.tar --exclude=.svn --exclude='*.pyc'

Skip


From tom.brown.code at gmail.com  Mon Oct 20 07:06:51 2008
From: tom.brown.code at gmail.com (Tom Brown)
Date: Sun, 19 Oct 2008 22:06:51 -0700
Subject: [Python-Dev] [Csv] skipfinalspace
In-Reply-To: <9789242b0810192154w557dedd8seba60c3deb168f12@mail.gmail.com>
References: <9789242b0810181615w5b1325c6h2149990854cff83d@mail.gmail.com>
	<20081019234604.B70DB59C001@longblack.object-craft.com.au>
	<9789242b0810192154w557dedd8seba60c3deb168f12@mail.gmail.com>
Message-ID: <9789242b0810192206n318fb7cao8bba9341695af053@mail.gmail.com>

(Continuing thread started at
http://mail.python.org/pipermail/csv/2008-October/000688.html)
On Sun, Oct 19, 2008 at 16:46, Andrew McNamara
<andrewm at object-craft.com.au>wrote:

> >I downloaded the 2.6 source tar ball, but is it too late for new features
> to
> >get into versions <3?
>
> Yep.
>
> >How would you feel about adding the following tests to
> Lib/test/test_csv.py
> >and getting them to pass?
> >
> >Also http://www.python.org/doc/2.5.2/lib/csv-fmt-params.html says
> >"*skipinitialspace *When True, whitespace immediately following the
> >delimiter is ignored."
> >but my tests show whitespace at the start of any field is ignored,
> including
> >the first field.
>
> I suspect (but I haven't checked) that it means "after the delimiter and
> before any quoted field (or some variation on that).


I agree that whitespace after the delimiter and before any quoted field is
skipped. Also whitespace after the start of the line and before any quoted
field is skipped.


>
>
> All of the "dialect" parameters are there to allow parsing of a specific
> common form of CSV file. Because there is no formal definition of the
> format, the module simply aims to parse (and produce the same result)
> as common applications such as Excel and Access. Changing the behaviour
> in any non-backwards compatible way is sure to get screams of anguish
> from many users. Even when the behaviour appears to be a bug, you can
> be sure people are counting on it working like that.


skipinitialspace defaults to false and by the same logic skipfinalspace
should default to false to preserve compatibility with the csv module in
2.6. On the other hand, the switch to version 3 is as good a time as any to
break backwards compatibility to adopt something that works better for new
users.

Based on my experience parsing several hundred csv generated by many
different people I think it would be nice to at least have a dialect that is
excel + skipinitialspace=True + skipfinalspace=True.


>
> BTW, this discussion probably should move to python-dev.
>
> --
> Andrew McNamara, Senior Developer, Object Craft
> http://www.object-craft.com.au/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081019/9f073f19/attachment.htm>

From andrewm at object-craft.com.au  Mon Oct 20 08:38:05 2008
From: andrewm at object-craft.com.au (Andrew McNamara)
Date: Mon, 20 Oct 2008 17:38:05 +1100
Subject: [Python-Dev] [Csv] skipfinalspace
In-Reply-To: <9789242b0810192206n318fb7cao8bba9341695af053@mail.gmail.com> 
References: <9789242b0810181615w5b1325c6h2149990854cff83d@mail.gmail.com>
	<20081019234604.B70DB59C001@longblack.object-craft.com.au>
	<9789242b0810192154w557dedd8seba60c3deb168f12@mail.gmail.com>
	<9789242b0810192206n318fb7cao8bba9341695af053@mail.gmail.com>
Message-ID: <20081020063805.6961759C001@longblack.object-craft.com.au>

>>>I downloaded the 2.6 source tar ball, but is it too late for new
>>>features to get into versions <3?
>>
>> Yep.

Sigh - I should slow down and actually read the e-mail I'm replying
to. It is not too late to get features into versions <3. It is, however,
too late to get features into 2.6, which was not what you asked, but
what I was answering "Yep" to.

>>>How would you feel about adding the following tests to
>>>Lib/test/test_csv.py and getting them to pass?

I have no real objection to someone adding a skipfinalspace parameter and
associated tests, although I have no time to do it myself at the moment.

>> >Also http://www.python.org/doc/2.5.2/lib/csv-fmt-params.html says
>> >"*skipinitialspace *When True, whitespace immediately following the
>> >delimiter is ignored."
>> >but my tests show whitespace at the start of any field is ignored,
>> >including the first field.
>>
>> I suspect (but I haven't checked) that it means "after the delimiter and
>> before any quoted field (or some variation on that).
>
>I agree that whitespace after the delimiter and before any quoted field is
>skipped. Also whitespace after the start of the line and before any quoted
>field is skipped.

I'm not sure if we're talking about the same thing - it seems to work as I
expect it to work:

    >>> list(csv.reader([' foo, bar']))
    [[' foo', ' bar']]
    >>> list(csv.reader([' foo, bar'], skipinitialspace=1))
    [['foo', 'bar']]

BTW, I think the reason "skipinitialspace" exists at all is to support
this:

    >>> list(csv.reader([' foo, " bar"']))
    [[' foo', ' " bar"']]
    >>> list(csv.reader([' foo, " bar"'], skipinitialspace=1))
    [['foo', ' bar']]

The quoting is only valid if the quote is the first character encountered
in the field (this is how Excel works). However, some other CSV generators
insert a space after the comma, and expect the parser to still treat it
as a quoted field - so skipinitialspace eats the space leading up the
quote, but does not eat any space after the quote (hence the "initial"
in the name).

For symmetry, a "skipfinalspace" option should do the same - only eat
space after the quote (if quotes are used) - however this will be rather
hard to implement as the parser state has already rolled on, and you
no longer know that whether the field was quoted. Eating spaces that
appeared within the quotes is the wrong thing to do.

>skipinitialspace defaults to false and by the same logic skipfinalspace
>should default to false to preserve compatibility with the csv module in
>2.6. On the other hand, the switch to version 3 is as good a time as any to
>break backwards compatibility to adopt something that works better for new
>users.

No, by default it needs to work like Excel, because this is the defacto
standard.

>Based on my experience parsing several hundred csv generated by many
>different people I think it would be nice to at least have a dialect that is
>excel + skipinitialspace=True + skipfinalspace=True.

Once the "skipfinalspace" parameter is implemented, there is nothing
stopping you creating such a dialect in your code, but I don't support
adding it to the standard library - the dialects in the std lib should
be well defined (in some way).

BTW, it's not necessary to create dialect objects: as I've done above,
users can pass keyword parameters to the parser if it's more convenient.

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/

From sjmachin at lexicon.net  Mon Oct 20 09:48:10 2008
From: sjmachin at lexicon.net (John Machin)
Date: Mon, 20 Oct 2008 18:48:10 +1100
Subject: [Python-Dev] [Csv] skipfinalspace
In-Reply-To: <9789242b0810192206n318fb7cao8bba9341695af053@mail.gmail.com>
References: <9789242b0810181615w5b1325c6h2149990854cff83d@mail.gmail.com>	<20081019234604.B70DB59C001@longblack.object-craft.com.au>	<9789242b0810192154w557dedd8seba60c3deb168f12@mail.gmail.com>
	<9789242b0810192206n318fb7cao8bba9341695af053@mail.gmail.com>
Message-ID: <48FC37BA.20303@lexicon.net>

Tom Brown wrote:
> (Continuing thread started at 
> http://mail.python.org/pipermail/csv/2008-October/000688.html)
> 
> On Sun, Oct 19, 2008 at 16:46, Andrew McNamara 
> <andrewm at object-craft.com.au <mailto:andrewm at object-craft.com.au>> wrote:
> 
>      >I downloaded the 2.6 source tar ball, but is it too late for new
>     features to
>      >get into versions <3?
> 
>     Yep.
> 
>      >How would you feel about adding the following tests to
>     Lib/test/test_csv.py
>      >and getting them to pass?
>      >
>      >Also http://www.python.org/doc/2.5.2/lib/csv-fmt-params.html says
>      >"*skipinitialspace *When True, whitespace immediately following the
>      >delimiter is ignored."
>      >but my tests show whitespace at the start of any field is ignored,
>     including
>      >the first field.
> 
>     I suspect (but I haven't checked) that it means "after the delimiter and
>     before any quoted field (or some variation on that).
> 
> I agree that whitespace after the delimiter and before any quoted field 
> is skipped. Also whitespace after the start of the line and before any 
> quoted field is skipped.

>     All of the "dialect" parameters are there to allow parsing of a specific
>     common form of CSV file. Because there is no formal definition of the
>     format, the module simply aims to parse (and produce the same result)
>     as common applications such as Excel and Access. Changing the behaviour
>     in any non-backwards compatible way is sure to get screams of anguish
>     from many users. Even when the behaviour appears to be a bug, you can
>     be sure people are counting on it working like that.
> 
> 
> skipinitialspace defaults to false and by the same logic skipfinalspace 
> should default to false to preserve compatibility with the csv module in 
> 2.6. On the other hand, the switch to version 3 is as good a time as any 
> to break backwards compatibility to adopt something that works better 
> for new users.

Read Andrew's lips: They don't want "better", they want "the same as MS".

> Based on my experience parsing several hundred csv generated by many 
> different people I think it would be nice to at least have a dialect 
> that is excel + skipinitialspace=True + skipfinalspace=True.

Based on my experience extracting data from innumerable csv files (and 
infinite varieties thereof), spreadsheet files, and database tables, in 
99.99% of cases one should automatically apply the following 
transformations to each text field:
    * strip leading whitespace
    * strip trailing whitespace
    * replace embedded runs of whitespace by a single space
and one needs to ensure that the definition of whitespace includes the 
no-break space (NBSP) character.

As this "space normalisation" is needed for all input sources, the csv 
module is IMHO the wrong place to put it. A string method would be a 
better idea.

Cheers,
John

From asmodai at in-nomine.org  Mon Oct 20 11:57:36 2008
From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven)
Date: Mon, 20 Oct 2008 11:57:36 +0200
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <48EB0F5C.602@v.loewis.de>
References: <48EB0F5C.602@v.loewis.de>
Message-ID: <20081020095736.GD98121@nexus.in-nomine.org>

Martin,

-On [20081007 09:27], "Martin v. L?wis" (martin at v.loewis.de) wrote:
>Within a few weeks, we will release Python 2.5.3. This will be the last
>bug fix release of Python 2.5, afterwards, future releases of 2.5 will
>only include security fixes, and no binaries (for Windows or OSX) will
>be provided anymore (from python.org).

Since we tripped over these with Trac/Genshi we would appreciate if the
following could be applied (if not already):

http://bugs.python.org/issue2231
http://bugs.python.org/issue2246

(http://bugs.python.org/issue2321 seems to be in 2.5 already based on the
last comment)

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
Ignorance is the opportunity to learn...

From skip at pobox.com  Mon Oct 20 18:01:30 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 20 Oct 2008 11:01:30 -0500
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <48F8DFFA.9090707@cheimes.de>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<48F7BAE6.4070108@gmail.com> <48F7BE73.3010303@cheimes.de>
	<18680.42554.791704.321274@montanaro-dyndns-org.local>
	<48F8DFFA.9090707@cheimes.de>
Message-ID: <18684.43866.322328.195968@montanaro-dyndns-org.local>


    >> Maybe doing the modest amount of translation required of the 2.6 unit
    >> tests so they run under 0.52 would help.  See what fails and then see
    >> what fixes correspond to fixing those failing tests.

    Christian> Sounds like a good plan. Let's get started! Are you going to
    Christian> commit your work to the Google Code repository anytime soon?

I checked in the contents of my multiprocessing.tar file and opened issues
#1 and #2.

Skip


From amk at amk.ca  Mon Oct 20 19:06:53 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Mon, 20 Oct 2008 13:06:53 -0400
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <20081020095736.GD98121@nexus.in-nomine.org>
References: <48EB0F5C.602@v.loewis.de>
	<20081020095736.GD98121@nexus.in-nomine.org>
Message-ID: <20081020170653.GA11672@amk-desktop.matrixgroup.net>

yOn Mon, Oct 20, 2008 at 11:57:36AM +0200, Jeroen Ruigrok van der Werven wrote:
> http://bugs.python.org/issue2231

This fixes a memory leak in itertools.chain(), which was greatly
changed between 2.5 and 2.6, and the patch was to code not present in
2.5.  Are you sure this bug affected 2.5 at all?

> http://bugs.python.org/issue2246

Already backported to 2.5 in rev. 61287.

> (http://bugs.python.org/issue2321 seems to be in 2.5 already 

Correct; rev. 61485.

--amk

From martin at v.loewis.de  Mon Oct 20 23:29:12 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Mon, 20 Oct 2008 23:29:12 +0200
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <20081020095736.GD98121@nexus.in-nomine.org>
References: <48EB0F5C.602@v.loewis.de>
	<20081020095736.GD98121@nexus.in-nomine.org>
Message-ID: <48FCF828.7040005@v.loewis.de>

> Since we tripped over these with Trac/Genshi we would appreciate if the
> following could be applied (if not already):

Ok, I've marked them as candidates for a backport.

Regards,
Martin

From tom.brown.code at gmail.com  Tue Oct 21 09:21:50 2008
From: tom.brown.code at gmail.com (Tom Brown)
Date: Tue, 21 Oct 2008 00:21:50 -0700
Subject: [Python-Dev] [Csv] skipfinalspace
In-Reply-To: <9789242b0810210021re3dd771o3a7a19f177d8be41@mail.gmail.com>
References: <9789242b0810181615w5b1325c6h2149990854cff83d@mail.gmail.com>
	<20081019234604.B70DB59C001@longblack.object-craft.com.au>
	<9789242b0810192154w557dedd8seba60c3deb168f12@mail.gmail.com>
	<9789242b0810192206n318fb7cao8bba9341695af053@mail.gmail.com>
	<48FC37BA.20303@lexicon.net>
	<9789242b0810210021re3dd771o3a7a19f177d8be41@mail.gmail.com>
Message-ID: <9789242b0810210021v344a6d86sec0859633a639724@mail.gmail.com>

On Mon, Oct 20, 2008 at 00:48, John Machin <sjmachin at lexicon.net> wrote:

> Tom Brown wrote:
>
>> (Continuing thread started at
>> http://mail.python.org/pipermail/csv/2008-October/000688.html)
>>
>> On Sun, Oct 19, 2008 at 16:46, Andrew McNamara <
>> andrewm at object-craft.com.au <mailto:andrewm at object-craft.com.au>> wrote:
>>
>>     >I downloaded the 2.6 source tar ball, but is it too late for new
>>    features to
>>     >get into versions <3?
>>
>>    Yep.
>>
>>     >How would you feel about adding the following tests to
>>    Lib/test/test_csv.py
>>     >and getting them to pass?
>>     >
>>     >Also http://www.python.org/doc/2.5.2/lib/csv-fmt-params.html says
>>     >"*skipinitialspace *When True, whitespace immediately following the
>>     >delimiter is ignored."
>>     >but my tests show whitespace at the start of any field is ignored,
>>    including
>>     >the first field.
>>
>>    I suspect (but I haven't checked) that it means "after the delimiter
>> and
>>    before any quoted field (or some variation on that).
>>
>> I agree that whitespace after the delimiter and before any quoted field is
>> skipped. Also whitespace after the start of the line and before any quoted
>> field is skipped.
>>
>
>     All of the "dialect" parameters are there to allow parsing of a
>> specific
>>    common form of CSV file. Because there is no formal definition of the
>>    format, the module simply aims to parse (and produce the same result)
>>    as common applications such as Excel and Access. Changing the behaviour
>>    in any non-backwards compatible way is sure to get screams of anguish
>>    from many users. Even when the behaviour appears to be a bug, you can
>>    be sure people are counting on it working like that.
>>
>>
>> skipinitialspace defaults to false and by the same logic skipfinalspace
>> should default to false to preserve compatibility with the csv module in
>> 2.6. On the other hand, the switch to version 3 is as good a time as any to
>> break backwards compatibility to adopt something that works better for new
>> users.
>>
>
> Read Andrew's lips: They don't want "better", they want "the same as MS".

okay.


>
>
>  Based on my experience parsing several hundred csv generated by many
>> different people I think it would be nice to at least have a dialect that is
>> excel + skipinitialspace=True + skipfinalspace=True.
>>
>
> Based on my experience extracting data from innumerable csv files (and
> infinite varieties thereof),

Wow, that is a _lot_ of files :-P

spreadsheet files, and database tables, in 99.99% of cases one should
> automatically apply the following transformations to each text field:
>   * strip leading whitespace
>   * strip trailing whitespace
>   * replace embedded runs of whitespace by a single space
> and one needs to ensure that the definition of whitespace includes the
> no-break space (NBSP) character.
>
> As this "space normalisation" is needed for all input sources, the csv
> module is IMHO the wrong place to put it. A string method would be a better
> idea.


I agree that strip() and something like re.sub(r"\s+", " " are handy. If
99.99% percent of csv readers should be applying these fixes to every field
perhaps there should be easy-to-enable option to apply it. Why force almost
everyone to discover they need the transformations and put a line of code
around csv reader?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081021/1293709e/attachment.htm>

From asmodai at in-nomine.org  Tue Oct 21 10:50:46 2008
From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven)
Date: Tue, 21 Oct 2008 10:50:46 +0200
Subject: [Python-Dev] Python 2.5.3: call for patches
In-Reply-To: <20081020170653.GA11672@amk-desktop.matrixgroup.net>
References: <48EB0F5C.602@v.loewis.de>
	<20081020095736.GD98121@nexus.in-nomine.org>
	<20081020170653.GA11672@amk-desktop.matrixgroup.net>
Message-ID: <20081021085045.GH98121@nexus.in-nomine.org>

-On [20081020 19:07], A.M. Kuchling (amk at amk.ca) wrote:
>This fixes a memory leak in itertools.chain(), which was greatly
>changed between 2.5 and 2.6, and the patch was to code not present in
>2.5.  Are you sure this bug affected 2.5 at all?

No, my mind was caught up between versions, so Raymond's closing the issue
is the logical thing to do. Apologies for wasting those few minutes.

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
Nothing is more honorable than enlightenment, nothing is more beautiful
than virtue...

From magnus at hetland.org  Tue Oct 21 13:03:41 2008
From: magnus at hetland.org (Magnus Lie Hetland)
Date: Tue, 21 Oct 2008 13:03:41 +0200
Subject: [Python-Dev] [Csv] skipfinalspace
In-Reply-To: <48FC37BA.20303@lexicon.net>
References: <9789242b0810181615w5b1325c6h2149990854cff83d@mail.gmail.com>	<20081019234604.B70DB59C001@longblack.object-craft.com.au>	<9789242b0810192154w557dedd8seba60c3deb168f12@mail.gmail.com>
	<9789242b0810192206n318fb7cao8bba9341695af053@mail.gmail.com>
	<48FC37BA.20303@lexicon.net>
Message-ID: <87772659-8D79-4CEF-BF7A-E633A38D4A25@hetland.org>

On Oct 20, 2008, at 09:48, John Machin wrote:

> Based on my experience extracting data from innumerable csv files  
> (and infinite varieties thereof), spreadsheet files, and database  
> tables, in 99.99% of cases one should automatically apply the  
> following transformations to each text field:
>   * strip leading whitespace
>   * strip trailing whitespace
>   * replace embedded runs of whitespace by a single space
> and one needs to ensure that the definition of whitespace includes  
> the no-break space (NBSP) character.
>
> As this "space normalisation" is needed for all input sources, the  
> csv module is IMHO the wrong place to put it. A string method would  
> be a better idea.

Hm. It seems quite familiar, somehow...

You could certainly do the following (for each field)...

   " ".join(field.split())

... but I seem to recall running across something that did this?  
(Maybe I'm confusing it with some other issue, with the  
string.capwords function versis str.title :)

-- 
Magnus Lie Hetland
http://hetland.org



From cadr4u at gmail.com  Wed Oct 22 14:16:55 2008
From: cadr4u at gmail.com (J. Sievers)
Date: Wed, 22 Oct 2008 14:16:55 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
Message-ID: <87hc74n7t4.fsf@svirfneblin.org>

Hi,

I implemented a variant of the CPython VM on top of Gforth's Vmgen; this made
it fairly straightforward to add direct threaded code and superinstructions for
the various permutations of LOAD_CONST, LOAD_FAST, and most of the two-argument
VM instructions.

Sources:
  http://svirfneblin.org/stuff/VPython-0.1.tar.gz

Pybench output:

Test                             minimum run-time        average  run-time
                                 this    other   diff    this    other   diff
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:   104ms   126ms  -17.3%   110ms   128ms  -14.0%
           BuiltinMethodLookup:    98ms   136ms  -27.9%   100ms   139ms  -28.1%
                 CompareFloats:    59ms   121ms  -51.1%    60ms   123ms  -50.7%
         CompareFloatsIntegers:    67ms   121ms  -45.0%    73ms   136ms  -46.7%
               CompareIntegers:    57ms   157ms  -63.7%    58ms   161ms  -64.1%
        CompareInternedStrings:    55ms   143ms  -61.6%    57ms   161ms  -64.5%
                  CompareLongs:    56ms   122ms  -54.0%    61ms   141ms  -56.7%
                CompareStrings:    71ms   126ms  -43.6%    72ms   131ms  -44.7%
                CompareUnicode:    82ms   123ms  -33.5%    83ms   131ms  -36.7%
                 ConcatStrings:   119ms   146ms  -18.4%   129ms   158ms  -18.4%
                 ConcatUnicode:    90ms   109ms  -17.2%    97ms   121ms  -19.8%
               CreateInstances:   116ms   124ms   -6.6%   118ms   127ms   -7.0%
            CreateNewInstances:   109ms   119ms   -7.9%   113ms   121ms   -6.6%
       CreateStringsWithConcat:    97ms   162ms  -40.3%    99ms   169ms  -41.5%
       CreateUnicodeWithConcat:    90ms   116ms  -22.8%    97ms   122ms  -20.8%
                  DictCreation:    87ms   122ms  -28.6%    91ms   127ms  -28.0%
             DictWithFloatKeys:    98ms   139ms  -29.5%   105ms   148ms  -29.3%
           DictWithIntegerKeys:    71ms   133ms  -46.7%    74ms   136ms  -46.0%
            DictWithStringKeys:    62ms   126ms  -51.0%    64ms   128ms  -50.3%
                      ForLoops:    68ms   135ms  -49.2%    69ms   136ms  -49.2%
                    IfThenElse:    63ms   130ms  -51.6%    64ms   134ms  -51.9%
                   ListSlicing:   122ms   123ms   -0.9%   126ms   125ms   +0.8%
                NestedForLoops:    89ms   149ms  -40.2%    93ms   152ms  -38.9%
          NormalClassAttribute:    88ms   132ms  -33.1%    95ms   134ms  -29.5%
       NormalInstanceAttribute:    72ms   116ms  -37.9%    77ms   118ms  -34.8%
           PythonFunctionCalls:    90ms   122ms  -26.1%    94ms   125ms  -24.7%
             PythonMethodCalls:   117ms   144ms  -18.8%   121ms   147ms  -17.8%
                     Recursion:   121ms   180ms  -32.6%   124ms   184ms  -32.4%
                  SecondImport:   144ms   139ms   +3.5%   150ms   143ms   +4.8%
           SecondPackageImport:   151ms   145ms   +3.9%   156ms   149ms   +4.3%
         SecondSubmoduleImport:   178ms   168ms   +5.8%   186ms   176ms   +5.4%
       SimpleComplexArithmetic:    71ms   112ms  -36.7%    76ms   123ms  -38.3%
        SimpleDictManipulation:    77ms   139ms  -44.3%    78ms   140ms  -44.3%
         SimpleFloatArithmetic:    61ms   124ms  -50.7%    63ms   126ms  -50.2%
      SimpleIntFloatArithmetic:    61ms   121ms  -49.4%    62ms   123ms  -49.5%
       SimpleIntegerArithmetic:    61ms   121ms  -49.5%    62ms   123ms  -49.8%
        SimpleListManipulation:    58ms   116ms  -50.0%    58ms   117ms  -50.2%
          SimpleLongArithmetic:    89ms   121ms  -26.3%    91ms   124ms  -27.0%
                    SmallLists:    79ms   116ms  -31.8%    82ms   122ms  -32.6%
                   SmallTuples:    91ms   117ms  -22.6%    93ms   122ms  -23.6%
         SpecialClassAttribute:    84ms   132ms  -36.4%    93ms   134ms  -30.4%
      SpecialInstanceAttribute:   111ms   153ms  -27.6%   114ms   155ms  -26.2%
                StringMappings:   102ms   115ms  -11.1%   104ms   117ms  -10.9%
              StringPredicates:   100ms   136ms  -26.7%   101ms   137ms  -26.1%
                 StringSlicing:    79ms   114ms  -30.2%    84ms   119ms  -29.9%
                     TryExcept:    68ms   145ms  -53.2%    69ms   148ms  -53.6%
                TryRaiseExcept:   106ms   104ms   +2.7%   109ms   106ms   +2.8%
                  TupleSlicing:   108ms   126ms  -14.4%   113ms   130ms  -13.0%
               UnicodeMappings:   150ms   150ms   -0.4%   152ms   154ms   -1.7%
             UnicodePredicates:   106ms   130ms  -18.3%   108ms   133ms  -18.3%
             UnicodeProperties:    94ms   111ms  -15.2%    97ms   115ms  -15.5%
                UnicodeSlicing:   101ms   130ms  -22.5%   105ms   136ms  -22.6%
-------------------------------------------------------------------------------
Totals:                          4750ms  6788ms  -30.0%  4929ms  7035ms  -29.9%

``other'' is vanilla Python 2.5.2.
For details, see README, TODO, and PYBENCH which come with the sources.
Feedback is, of course, very welcome and it'd be great to have some pybench
results from different machines.

Cheers,
-jakob

From p.f.moore at gmail.com  Wed Oct 22 14:34:00 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 22 Oct 2008 13:34:00 +0100
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <87hc74n7t4.fsf@svirfneblin.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
Message-ID: <79990c6b0810220534h196f4345n23598f9995cd4ab4@mail.gmail.com>

2008/10/22 J. Sievers <cadr4u at gmail.com>:
> I implemented a variant of the CPython VM on top of Gforth's Vmgen; this made
> it fairly straightforward to add direct threaded code and superinstructions for
> the various permutations of LOAD_CONST, LOAD_FAST, and most of the two-argument
> VM instructions.
[...]
> Totals:                          4750ms  6788ms  -30.0%  4929ms  7035ms  -29.9%
>
> ``other'' is vanilla Python 2.5.2.
> For details, see README, TODO, and PYBENCH which come with the sources.
> Feedback is, of course, very welcome and it'd be great to have some pybench
> results from different machines.

Am I reading this right? You got a 30% speed improvement over
CPython??? (I never read pybench results right, so if this is actually
30% slower than CPython, let me know and I'll shut up :-)) If so, what
(if any) limitations are there to the implementation?

Paul.

From mal at egenix.com  Wed Oct 22 14:57:17 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 22 Oct 2008 14:57:17 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <87hc74n7t4.fsf@svirfneblin.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
Message-ID: <48FF232D.4090807@egenix.com>

On 2008-10-22 14:16, J. Sievers wrote:
> Hi,
> 
> I implemented a variant of the CPython VM on top of Gforth's Vmgen; this made
> it fairly straightforward to add direct threaded code and superinstructions for
> the various permutations of LOAD_CONST, LOAD_FAST, and most of the two-argument
> VM instructions.

I suppose you get most of the speedup by using threaded code. Unfortunately,
that is only supported by gcc.

Do you get similar results for the switch based method that appears to be
available in VMgen ?

http://www.complang.tuwien.ac.at/anton/vmgen/html-docs/VM-engine.html

Thanks,
-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 22 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From lists at cheimes.de  Wed Oct 22 15:02:45 2008
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 22 Oct 2008 15:02:45 +0200
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <18684.43866.322328.195968@montanaro-dyndns-org.local>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<48F7BAE6.4070108@gmail.com> <48F7BE73.3010303@cheimes.de>
	<18680.42554.791704.321274@montanaro-dyndns-org.local>
	<48F8DFFA.9090707@cheimes.de>
	<18684.43866.322328.195968@montanaro-dyndns-org.local>
Message-ID: <48FF2475.8090006@cheimes.de>

skip at pobox.com wrote:
> I checked in the contents of my multiprocessing.tar file and opened issues
> #1 and #2.

I added a setup.py, disabled recv_bytes_into for now and fixed lots of
naming issues. The multiprocessing code is using the new names of the
threading module (current_thread, is_alive etc.) but Python 2.5 just
have the old names (currentThread, isAlive).

$ python2.5 setup.py build_ext -i
$ PYTHONPATH=Lib python2.5 Lib/test/test_multiprocessing.py

======================================================================
ERROR: test_connection (__main__.WithProcessesTestConnection)
----------------------------------------------------------------------
Traceback (most recent call last):
   File "Lib/test/test_multiprocessing.py", line 1220, in test_connection
     self.assertEqual(conn.recv_bytes_into(buffer),
AttributeError: '_multiprocessing.Connection' object has no attribute
'recv_bytes_into'

----------------------------------------------------------------------
Ran 123 tests in 12.309s

FAILED (errors=1)

:)

Christian


From jnoller at gmail.com  Wed Oct 22 15:05:16 2008
From: jnoller at gmail.com (jnoller at gmail.com)
Date: Wed, 22 Oct 2008 06:05:16 -0700
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
Message-ID: <00151757357c13daba0459d7331b@google.com>

Maybe we should backport those handy pep8 threading names ... ... Ok maybe  
not.

On Oct 22, 2008 9:02am, Christian Heimes <lists at cheimes.de> wrote:
> skip at pobox.com wrote:
>
>
> I checked in the contents of my multiprocessing.tar file and opened issues
>
> #1 and #2.
>
>
>
>
> I added a setup.py, disabled recv_bytes_into for now and fixed lots of
>
> naming issues. The multiprocessing code is using the new names of the
>
> threading module (current_thread, is_alive etc.) but Python 2.5 just
>
> have the old names (currentThread, isAlive).
>
>
>
> $ python2.5 setup.py build_ext -i
>
> $ PYTHONPATH=Lib python2.5 Lib/test/test_multiprocessing.py
>
>
>
> ======================================================================
>
> ERROR: test_connection (__main__.WithProcessesTestConnection)
>
> ----------------------------------------------------------------------
>
> Traceback (most recent call last):
>
> File "Lib/test/test_multiprocessing.py", line 1220, in test_connection
>
> self.assertEqual(conn.recv_bytes_into(buffer),
>
> AttributeError: '_multiprocessing.Connection' object has no attribute
>
> 'recv_bytes_into'
>
>
>
> ----------------------------------------------------------------------
>
> Ran 123 tests in 12.309s
>
>
>
> FAILED (errors=1)
>
>
>
> :)
>
>
>
> Christian
>
>
>
> _______________________________________________
>
> Python-3000 mailing list
>
> Python-3000 at python.org
>
> http://mail.python.org/mailman/listinfo/python-3000
>
> Unsubscribe:  
http://mail.python.org/mailman/options/python-3000/jnoller%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081022/67e55fe4/attachment.htm>

From santagada at gmail.com  Wed Oct 22 15:34:53 2008
From: santagada at gmail.com (Leonardo Santagada)
Date: Wed, 22 Oct 2008 11:34:53 -0200
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <87hc74n7t4.fsf@svirfneblin.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
Message-ID: <0967E5F1-DF9A-42D3-AC2C-140C6B55A6FD@gmail.com>


On Oct 22, 2008, at 10:16 AM, J. Sievers wrote:

> Hi,
>
> I implemented a variant of the CPython VM on top of Gforth's Vmgen;  
> this made
> it fairly straightforward to add direct threaded code and  
> superinstructions for
> the various permutations of LOAD_CONST, LOAD_FAST, and most of the  
> two-argument
> VM instructions.


Is it complete? I don't want to be rude or anything, but the idea I  
seem repeated a lot is that it is really easy to do a very fast VM  
that supports 95% of the language.

How much of the standard python tests your VM pass?

What is missing, and why?


But besides that, it is always very cool to see another vm project  
starting,
Good Luck

--
Leonardo Santagada
santagada at gmail.com




From kristjan at ccpgames.com  Wed Oct 22 15:49:52 2008
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Wed, 22 Oct 2008 13:49:52 +0000
Subject: [Python-Dev] heapq, min and max
Message-ID: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local>

Hello there.
I was surprised to find recently that the heapq module is still a pure python implementation.
A few years ago we wrote our own in C for use in Eve-online, and we usually do a
import blue.heapq as heapq.
Having a python implementation of it almost completely negates any benefit of using that in place of sort() unles the comparison is really expensive.
I'd be happy to donate an implementation if there is any interest.

I also recently came across the recommendation that one should use the "min" and "max"  builtins instead of using heapq or sort() when trying to find
a single smallest element.
Well, this is also not true, for simple comparisons, because currently "min" and "max" use the iterator protocol, whereas sort() (and blue.heapq.heapify)
use direct list access, thus halving the number of python method calls approximately.

 s="""
import random
l = [random.random() for i in xrange(10000)]
"""
timeit.Timer("(l.sort(), l[-1])", s).timeit(1000)
0.29406761513791935
timeit.Timer("max(l)", s).timeit(1000)
0.4760155766743992
>>>

Now, this is easy enough to rectify, by providing a list specialization for min_max().  This would also make sure that "min" is truly the fastest
way to find the minimum member of a list.

Would there be interest in this?  (I will be patching the CCP version of python for it anyway).

We are using 2.5.3, but these changes should be directly applicable to 2.6 too.

K
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081022/db27ea8f/attachment-0001.htm>

From solipsis at pitrou.net  Wed Oct 22 16:05:41 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 22 Oct 2008 14:05:41 +0000 (UTC)
Subject: [Python-Dev] heapq, min and max
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local>
Message-ID: <loom.20081022T135723-704@post.gmane.org>

Kristj?n Valur J?nsson <kristjan <at> ccpgames.com> writes:
> timeit.Timer("(l.sort(), l[-1])",
> s).timeit(1000)
> 
> 0.29406761513791935

This is clearly wrong. l.sort() will sort the list in place when it is first
invoked, and therefore will be very fast in subsequent calls.

Compare with:

    timeit.Timer("sorted(l)[-1]", s).timeit(1000)

which really does what you are looking for (sorting then taking the max *on
every iteration*), and is much slower.

As for heapq, its critical parts are already implemented in C as of Python 2.5.2:

>>> import heapq, _heapq
>>> _heapq
<module '_heapq' from '/usr/lib/python2.5/lib-dynload/_heapq.so'>
>>> heapq.heapify is _heapq.heapify
True




From kristjan at ccpgames.com  Wed Oct 22 16:37:29 2008
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Wed, 22 Oct 2008 14:37:29 +0000
Subject: [Python-Dev] heapq, min and max
In-Reply-To: <loom.20081022T135723-704@post.gmane.org>
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local>
	<loom.20081022T135723-704@post.gmane.org>
Message-ID: <4E9372E6B2234D4F859320D896059A951018565DC2@exchis.ccp.ad.local>

Ooops, you are right.
Silly error on my part.
Still, my comment about heapq still stands,

and
[hack, slash]

0.39713821814841893   (old)
0.35184029691278162   (hakced, for special list treatment)

So, there is a 12% performance boost to be had by specializing for lists.
How about it?

K

> -----Original Message-----
> From: python-dev-bounces+kristjan=ccpgames.com at python.org
> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf
> Of Antoine Pitrou
> Sent: Wednesday, October 22, 2008 14:06
> To: python-dev at python.org
> Subject: Re: [Python-Dev] heapq, min and max
>

> This is clearly wrong. l.sort() will sort the list in place when it is
> first
> invoked, and therefore will be very fast in subsequent calls.
>


From solipsis at pitrou.net  Wed Oct 22 16:40:55 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 22 Oct 2008 14:40:55 +0000 (UTC)
Subject: [Python-Dev] heapq, min and max
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local>
	<loom.20081022T135723-704@post.gmane.org>
	<4E9372E6B2234D4F859320D896059A951018565DC2@exchis.ccp.ad.local>
Message-ID: <loom.20081022T144003-886@post.gmane.org>

Kristj?n Valur J?nsson <kristjan <at> ccpgames.com> writes:
> 
> 0.39713821814841893   (old)
> 0.35184029691278162   (hakced, for special list treatment)
> 
> So, there is a 12% performance boost to be had by specializing for lists.
> How about it?

It depends on the added code complexity. In any case, you should open an entry
on the tracker and post your patch there.

Regards

Antoine.



From kristjan at ccpgames.com  Wed Oct 22 16:50:23 2008
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Wed, 22 Oct 2008 14:50:23 +0000
Subject: [Python-Dev] heapq, min and max
In-Reply-To: <loom.20081022T144003-886@post.gmane.org>
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local>
	<loom.20081022T135723-704@post.gmane.org>
	<4E9372E6B2234D4F859320D896059A951018565DC2@exchis.ccp.ad.local>
	<loom.20081022T144003-886@post.gmane.org>
Message-ID: <4E9372E6B2234D4F859320D896059A951018565DDB@exchis.ccp.ad.local>

Ok.  And sorry, I missed your part about heapq now having a c implementation.
This is indeed good, I was misled by the presence of heapq.py.

However, our own heapify() implementation is some 10% faster on a 10000 element
list of floats than the _heapq.heapify() version.
I'll investigate the difference.

K

> -----Original Message-----
> From: python-dev-bounces+kristjan=ccpgames.com at python.org
> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf
> Of Antoine Pitrou
> Sent: Wednesday, October 22, 2008 14:41
> To: python-dev at python.org
> Subject: Re: [Python-Dev] heapq, min and max
>
> Kristj?n Valur J?nsson <kristjan <at> ccpgames.com> writes:
> >
> > 0.39713821814841893   (old)
> > 0.35184029691278162   (hakced, for special list treatment)
> >
> > So, there is a 12% performance boost to be had by specializing for
> lists.
> > How about it?
>
> It depends on the added code complexity. In any case, you should open
> an entry
> on the tracker and post your patch there.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-
> dev/kristjan%40ccpgames.com

From skip at pobox.com  Wed Oct 22 16:52:27 2008
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 22 Oct 2008 09:52:27 -0500
Subject: [Python-Dev] [Python-3000] Backporting multiprocessing?
In-Reply-To: <gdncc5$hmt$1@ger.gmane.org>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>
	<48F7BAE6.4070108@gmail.com> <48F7BE73.3010303@cheimes.de>
	<18680.42554.791704.321274@montanaro-dyndns-org.local>
	<48F8DFFA.9090707@cheimes.de>
	<18684.43866.322328.195968@montanaro-dyndns-org.local>
	<48FF2475.8090006@cheimes.de> <gdncc5$hmt$1@ger.gmane.org>
Message-ID: <18687.15915.98565.882018@montanaro-dyndns-org.local>


    Christian> I just implemented the recv_bytes_into function with the old
    Christian> buffer protocol. All tests are passing on my Linux box
    Christian> (Ubuntu 8.04 with gcc 4.2, AMD64 processor).

Using Python v < 2.6?  So I don't need to horse around making
test_multiprocessing.py API compatible with processing 0.52?

Skip

From lists at cheimes.de  Wed Oct 22 17:07:02 2008
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 22 Oct 2008 17:07:02 +0200
Subject: [Python-Dev] Backporting multiprocessing?
In-Reply-To: <18687.15915.98565.882018@montanaro-dyndns-org.local>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>	<48F7BAE6.4070108@gmail.com>
	<48F7BE73.3010303@cheimes.de>	<18680.42554.791704.321274@montanaro-dyndns-org.local>	<48F8DFFA.9090707@cheimes.de>	<18684.43866.322328.195968@montanaro-dyndns-org.local>	<48FF2475.8090006@cheimes.de>
	<gdncc5$hmt$1@ger.gmane.org>
	<18687.15915.98565.882018@montanaro-dyndns-org.local>
Message-ID: <48FF4196.2020204@cheimes.de>

skip at pobox.com wrote:
> Using Python v < 2.6?  So I don't need to horse around making
> test_multiprocessing.py API compatible with processing 0.52?

With Python 2.5.2 and 2.6.0 all tests are passing with any error. With 
Python 2.4.5 seven tests are failing because 2.4 doesn't support mmap 
with a negative file number.

File ".../python-multiprocessing/Lib/multiprocessing/heap.py", line 56, 
in __init__
     self.buffer = mmap.mmap(-1, size)
EnvironmentError: [Errno 9] Bad file descriptor

Christian

From kristjan at ccpgames.com  Wed Oct 22 17:19:11 2008
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Wed, 22 Oct 2008 15:19:11 +0000
Subject: [Python-Dev] heapq, min and max
In-Reply-To: <4E9372E6B2234D4F859320D896059A951018565DDB@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local>
	<loom.20081022T135723-704@post.gmane.org>
	<4E9372E6B2234D4F859320D896059A951018565DC2@exchis.ccp.ad.local>
	<loom.20081022T144003-886@post.gmane.org>
	<4E9372E6B2234D4F859320D896059A951018565DDB@exchis.ccp.ad.local>
Message-ID: <4E9372E6B2234D4F859320D896059A951018565E20@exchis.ccp.ad.local>

I have submitted a patch for min() and max():

http://bugs.python.org/issue4174



> -----Original Message-----
> >
> > It depends on the added code complexity. In any case, you should open
> > an entry
> > on the tracker and post your patch there.


From skip at pobox.com  Wed Oct 22 18:05:36 2008
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 22 Oct 2008 11:05:36 -0500
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <87hc74n7t4.fsf@svirfneblin.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
Message-ID: <18687.20304.136117.227731@montanaro-dyndns-org.local>


    J> I implemented a variant of the CPython VM on top of Gforth's Vmgen; this made
    J> it fairly straightforward to add direct threaded code and superinstructions for
    J> the various permutations of LOAD_CONST, LOAD_FAST, and most of the two-argument
    J> VM instructions.

    J> Sources:
    J>   http://svirfneblin.org/stuff/VPython-0.1.tar.gz

Very interesting.  Trying to build with your changes on my Mac (OS X 10.5.5)
I get an error caused by a definition in code.h:

    ...
    gcc -c -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes  -I. -IInclude -I./Include -I/opt/local/include  -DPy_BUILD_CORE -o Python/mactoolboxglue.o Python/mactoolboxglue.c
    In file included from /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Headers/ControlDefinitions.h:36,
                     from /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Headers/HIToolbox.h:201,
                     from /System/Library/Frameworks/Carbon.framework/Headers/Carbon.h:29,
                     from Include/pymactoolbox.h:10,
                     from Python/mactoolboxglue.c:27:
    /System/Library/Frameworks/Carbon.framework/Frameworks/HIToolbox.framework/Headers/Lists.h:50: error: conflicting types for 'Cell'
    Include/code.h:15: error: previous declaration of 'Cell' was here
    make: *** [Python/mactoolboxglue.o] Error 1

Note that I'm not actually doing a Framework build.  Any way you can easily
rename your macros to avoid obvious name clashes like this?

Thanks,

-- 
Skip Montanaro - skip at pobox.com - http://www.webfast.com/~skip/

From tjreedy at udel.edu  Wed Oct 22 18:28:42 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 22 Oct 2008 12:28:42 -0400
Subject: [Python-Dev] heapq, min and max
In-Reply-To: <4E9372E6B2234D4F859320D896059A951018565DDB@exchis.ccp.ad.local>
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local>	<loom.20081022T135723-704@post.gmane.org>	<4E9372E6B2234D4F859320D896059A951018565DC2@exchis.ccp.ad.local>	<loom.20081022T144003-886@post.gmane.org>
	<4E9372E6B2234D4F859320D896059A951018565DDB@exchis.ccp.ad.local>
Message-ID: <gdnkbk$klu$1@ger.gmane.org>

Kristj?n Valur J?nsson wrote:
> Ok.  And sorry, I missed your part about heapq now having a c implementation.
> This is indeed good, I was misled by the presence of heapq.py.

Duplicate Python/C code will probably become more common.  Even if the 
Python is not used for prototyping (which I believe it was for heapq), 
it serves to document the intention of the C code and to be a ready to 
go version for non-C implementations.  And it can be used as a basis for 
modification by Pythoneers who want slightly different behavior.


From skip at pobox.com  Wed Oct 22 20:18:26 2008
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 22 Oct 2008 13:18:26 -0500
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <87hc74n7t4.fsf@svirfneblin.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
Message-ID: <18687.28274.807590.115918@montanaro-dyndns-org.local>

    J> I implemented a variant of the CPython VM on top of Gforth's Vmgen;
    J> this made it fairly straightforward to add direct threaded code and
    J> superinstructions for the various permutations of LOAD_CONST,
    J> LOAD_FAST, and most of the two-argument VM instructions.

    Skip> Trying to build with your changes on my Mac (OS X 10.5.5) I get an
    Skip> error caused by a definition in code.h:

I renamed Cell to _PyV_Cell and Inst to _PyV_Inst and got it to build.  I
get this pybench output on my Mac:

    -------------------------------------------------------------------------------
    PYBENCH 2.0
    -------------------------------------------------------------------------------
    * using Python 2.5.3a0
    * disabled garbage collection
    * system check interval set to maximum: 2147483647
    * using timer: time.time

    -------------------------------------------------------------------------------
    Benchmark: py25t.pybench
    -------------------------------------------------------------------------------

        Rounds: 10
        Warp:   10
        Timer:  time.time

        Machine Details:
           Platform ID:  Darwin-9.5.0-i386-32bit
           Processor:    i386

        Python:
           Executable:   /Users/skip/src/python/release25-maint/python.exe
           Version:      2.5.3a0
           Compiler:     GCC 4.0.1 (Apple Inc. build 5465)
           Bits:         32bit
           Build:        Oct 22 2008 13:12:03 (#release25-maint:66444M)
           Unicode:      UCS2


    -------------------------------------------------------------------------------
    Comparing with: py25.pybench
    -------------------------------------------------------------------------------

        Rounds: 10
        Warp:   10
        Timer:  time.time

        Machine Details:
           Platform ID:  Darwin-9.5.0-i386-32bit
           Processor:    i386

        Python:
           Executable:   /Users/skip/local/bin/python2.5
           Version:      2.5.3a0
           Compiler:     GCC 4.0.1 (Apple Inc. build 5465)
           Bits:         32bit
           Build:        Sep 13 2008 09:17:06 (#release25-maint:66444)
           Unicode:      UCS2


    Test                             minimum run-time        average  run-time
                                     this    other   diff    this    other   diff
    -------------------------------------------------------------------------------
              BuiltinFunctionCalls:   119ms   133ms  -10.6%   121ms   141ms  -14.1%
               BuiltinMethodLookup:   107ms   113ms   -5.2%   108ms   116ms   -6.5%
                     CompareFloats:    67ms    92ms  -27.7%    68ms    95ms  -28.4%
             CompareFloatsIntegers:    75ms    92ms  -17.9%    76ms    94ms  -19.1%
                   CompareIntegers:    53ms    89ms  -39.8%    54ms    91ms  -40.6%
            CompareInternedStrings:    78ms   102ms  -24.3%    78ms   109ms  -28.3%
                      CompareLongs:    64ms    85ms  -24.2%    65ms    85ms  -23.5%
                    CompareStrings:    60ms    77ms  -22.0%    62ms    80ms  -22.5%
                    CompareUnicode:    91ms   106ms  -13.8%    93ms   111ms  -16.0%
                     ConcatStrings:   140ms   141ms   -0.3%   144ms   144ms   +0.0%
                     ConcatUnicode:    83ms    82ms   +2.4%    91ms    85ms   +6.6%
                   CreateInstances:   138ms   145ms   -5.1%   140ms   148ms   -5.2%
                CreateNewInstances:   123ms   128ms   -3.7%   125ms   129ms   -3.5%
           CreateStringsWithConcat:   104ms   111ms   -7.0%   105ms   258ms  -59.1%
           CreateUnicodeWithConcat:    81ms    93ms  -13.2%    83ms   158ms  -47.7%
                      DictCreation:   104ms    96ms   +9.1%   106ms   102ms   +4.0%
                 DictWithFloatKeys:    88ms   104ms  -15.9%    92ms   110ms  -16.4%
               DictWithIntegerKeys:    79ms   111ms  -29.2%    81ms   113ms  -28.4%
                DictWithStringKeys:    77ms    90ms  -13.8%    84ms    93ms   -9.8%
                          ForLoops:    66ms    78ms  -15.2%    67ms    79ms  -14.8%
                        IfThenElse:    57ms    91ms  -37.4%    57ms    91ms  -37.4%
                       ListSlicing:   119ms   120ms   -0.4%   121ms   121ms   -0.3%
                    NestedForLoops:    86ms    98ms  -12.0%    87ms    99ms  -12.5%
              NormalClassAttribute:   113ms   111ms   +2.1%   118ms   113ms   +4.0%
           NormalInstanceAttribute:    89ms   102ms  -12.5%    91ms   104ms  -12.6%
               PythonFunctionCalls:    89ms   106ms  -15.8%    91ms   108ms  -15.0%
                 PythonMethodCalls:   141ms   158ms  -11.0%   149ms   161ms   -6.9%
                         Recursion:   122ms   137ms  -11.2%   123ms   140ms  -11.7%
                      SecondImport:    90ms    91ms   -0.3%    91ms    91ms   -0.3%
               SecondPackageImport:    95ms    95ms   +0.9%    97ms    96ms   +0.9%
             SecondSubmoduleImport:   124ms   124ms   +0.5%   126ms   125ms   +0.7%
           SimpleComplexArithmetic:    96ms   103ms   -7.2%    97ms   105ms   -6.8%
            SimpleDictManipulation:    85ms   101ms  -15.6%    90ms   103ms  -12.3%
             SimpleFloatArithmetic:    90ms    98ms   -8.1%    92ms   101ms   -8.3%
          SimpleIntFloatArithmetic:    64ms    78ms  -17.2%    65ms    79ms  -18.0%
           SimpleIntegerArithmetic:    60ms    78ms  -23.3%    60ms    79ms  -24.3%
            SimpleListManipulation:    71ms    85ms  -16.3%    73ms    87ms  -16.3%
              SimpleLongArithmetic:   103ms   114ms   -9.8%   106ms   116ms   -8.1%
                        SmallLists:   121ms   128ms   -5.5%   122ms   131ms   -6.9%
                       SmallTuples:   107ms   114ms   -6.2%   109ms   116ms   -6.0%
             SpecialClassAttribute:   100ms   110ms   -9.7%   102ms   111ms   -7.8%
          SpecialInstanceAttribute:   175ms   182ms   -3.7%   178ms   184ms   -3.5%
                    StringMappings:   172ms   177ms   -2.8%   175ms   178ms   -1.6%
                  StringPredicates:   147ms   146ms   +0.7%   149ms   147ms   +1.0%
                     StringSlicing:   112ms   112ms   +0.2%   116ms   120ms   -3.5%
                         TryExcept:    55ms    77ms  -28.3%    56ms    78ms  -27.7%
                    TryRaiseExcept:   105ms   104ms   +1.2%   106ms   105ms   +1.0%
                      TupleSlicing:   110ms   107ms   +3.2%   113ms   112ms   +0.7%
                   UnicodeMappings:   106ms   108ms   -1.5%   108ms   109ms   -1.6%
                 UnicodePredicates:   106ms   113ms   -5.4%   109ms   115ms   -5.6%
                 UnicodeProperties:   102ms   113ms   -9.7%   104ms   116ms  -10.2%
                    UnicodeSlicing:    93ms    98ms   -5.1%    95ms   102ms   -7.1%
    -------------------------------------------------------------------------------
    Totals:                          5106ms  5645ms   -9.5%  5221ms  5985ms  -12.8%

    (this=py25t.pybench, other=py25.pybench)

this == your threaded VM, other == the 2.5.3a0 VM.

Skip

From lists at cheimes.de  Wed Oct 22 20:33:00 2008
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 22 Oct 2008 20:33:00 +0200
Subject: [Python-Dev] Backporting multiprocessing?
In-Reply-To: <18687.15915.98565.882018@montanaro-dyndns-org.local>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>	<48F7BAE6.4070108@gmail.com>
	<48F7BE73.3010303@cheimes.de>	<18680.42554.791704.321274@montanaro-dyndns-org.local>	<48F8DFFA.9090707@cheimes.de>	<18684.43866.322328.195968@montanaro-dyndns-org.local>	<48FF2475.8090006@cheimes.de>
	<gdncc5$hmt$1@ger.gmane.org>
	<18687.15915.98565.882018@montanaro-dyndns-org.local>
Message-ID: <48FF71DC.8010904@cheimes.de>

skip at pobox.com wrote:
> Using Python v < 2.6?  So I don't need to horse around making
> test_multiprocessing.py API compatible with processing 0.52?

I've backported the Python 2.5 svn version of mmap to 2.4 and added it 
as multiprocessing._mmap25. The port is just a proof of concept and most 
like contains issues with ssize_t -> long transitions. But it's working.

With the latest svn checkout all tests are passing for 2.4.5, 2.5.2 and 
2.6.0 on my 64bit Ubuntu box. Somebody needs to test it on Windows, 
32bit Linux and BSD.

Christian

From kristjan at ccpgames.com  Wed Oct 22 22:37:07 2008
From: kristjan at ccpgames.com (=?utf-8?B?S3Jpc3Rqw6FuIFZhbHVyIErDs25zc29u?=)
Date: Wed, 22 Oct 2008 20:37:07 +0000
Subject: [Python-Dev] heapq, min and max
In-Reply-To: <loom.20081022T135723-704@post.gmane.org>
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local>
	<loom.20081022T135723-704@post.gmane.org>
Message-ID: <4E9372E6B2234D4F859320D896059A95101864C33A@exchis.ccp.ad.local>

I
Kt just occurred to me:
Even though l.sort() is sorting a presorted array, it still must be doing
10000-1 RichCompares minimum, just like max.  So how do we explain the
large difference?

> -----Original Message-----
> From: python-dev-bounces+kristjan=ccpgames.com at python.org
> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf
> Of Antoine Pitrou
> Sent: Wednesday, October 22, 2008 14:06
> To: python-dev at python.org
> Subject: Re: [Python-Dev] heapq, min and max
>
> Kristj?n Valur J?nsson <kristjan <at> ccpgames.com> writes:
> > timeit.Timer("(l.sort(), l[-1])",
> > s).timeit(1000)
> >
> > 0.29406761513791935
>
> This is clearly wrong. l.sort() will sort the list in place when it is
> first
> invoked, and therefore will be very fast in subsequent calls.
>


From python at rcn.com  Wed Oct 22 22:51:40 2008
From: python at rcn.com (Raymond Hettinger)
Date: Wed, 22 Oct 2008 13:51:40 -0700
Subject: [Python-Dev] heapq, min and max
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local><loom.20081022T135723-704@post.gmane.org>
	<4E9372E6B2234D4F859320D896059A95101864C33A@exchis.ccp.ad.local>
Message-ID: <399C79F341494A318BA032A0B295CDC2@RaymondLaptop1>

I think this should be taken off of python-dev until
you have some quality measurements,
know what's going on, and have an actionable idea.

Aside from list specialization versus a general iterator
protocol, there is no fat in the min/max implementation.
It loops, it compares, it returns.

If we wanted to go the distance and type specialize,
it is possible to use the same ideas that used in
Py2.6's builtin sum() to speed-up compares for particular types.
Personally, I think that would be a waste and would rather
keep the code simple.

Also, a primary use case for min/max is with just two inputs.
We don't want to slow that down in order to provide negligible
improvements to min/max for long sequences.


Raymond

----- Original Message ----- 
From: "Kristj?n Valur J?nsson" <kristjan at ccpgames.com>
To: "Antoine Pitrou" <solipsis at pitrou.net>; <python-dev at python.org>
Sent: Wednesday, October 22, 2008 1:37 PM
Subject: Re: [Python-Dev] heapq, min and max


> I
> Kt just occurred to me:
> Even though l.sort() is sorting a presorted array, it still must be doing
> 10000-1 RichCompares minimum, just like max.  So how do we explain the
> large difference?
>
>> -----Original Message-----
>> From: python-dev-bounces+kristjan=ccpgames.com at python.org
>> [mailto:python-dev-bounces+kristjan=ccpgames.com at python.org] On Behalf
>> Of Antoine Pitrou
>> Sent: Wednesday, October 22, 2008 14:06
>> To: python-dev at python.org
>> Subject: Re: [Python-Dev] heapq, min and max
>>
>> Kristj?n Valur J?nsson <kristjan <at> ccpgames.com> writes:
>> > timeit.Timer("(l.sort(), l[-1])",
>> > s).timeit(1000)
>> >
>> > 0.29406761513791935
>>
>> This is clearly wrong. l.sort() will sort the list in place when it is
>> first
>> invoked, and therefore will be very fast in subsequent calls.
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/python%40rcn.com
> 


From solipsis at pitrou.net  Wed Oct 22 23:23:26 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 22 Oct 2008 21:23:26 +0000 (UTC)
Subject: [Python-Dev] heapq, min and max
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local><loom.20081022T135723-704@post.gmane.org>
	<4E9372E6B2234D4F859320D896059A95101864C33A@exchis.ccp.ad.local>
	<399C79F341494A318BA032A0B295CDC2@RaymondLaptop1>
Message-ID: <loom.20081022T211857-844@post.gmane.org>

Raymond Hettinger <python <at> rcn.com> writes:
> 
> If we wanted to go the distance and type specialize,
> it is possible to use the same ideas that used in
> Py2.6's builtin sum() to speed-up compares for particular types.

Would it be useful to create a macro that packs some of the optimizations in
http://bugs.python.org/issue3106 , so that interested code can get the
optimizations for free simply by using the macro?




From daniel at stutzbachenterprises.com  Thu Oct 23 01:30:03 2008
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Wed, 22 Oct 2008 18:30:03 -0500
Subject: [Python-Dev] heapq, min and max
In-Reply-To: <399C79F341494A318BA032A0B295CDC2@RaymondLaptop1>
References: <4E9372E6B2234D4F859320D896059A951018565D33@exchis.ccp.ad.local>
	<loom.20081022T135723-704@post.gmane.org>
	<4E9372E6B2234D4F859320D896059A95101864C33A@exchis.ccp.ad.local>
	<399C79F341494A318BA032A0B295CDC2@RaymondLaptop1>
Message-ID: <eae285400810221630w1019100fn78540f0bfdc5c277@mail.gmail.com>

On Wed, Oct 22, 2008 at 3:51 PM, Raymond Hettinger <python at rcn.com> wrote:

> Aside from list specialization versus a general iterator
> protocol, there is no fat in the min/max implementation.
> It loops, it compares, it returns.
>
> Also, a primary use case for min/max is with just two inputs.
> We don't want to slow that down in order to provide negligible
> improvements to min/max for long sequences.
>

If min/max were specialized for tuples, it would speed up the common case
where a handful of arguments are passed in.  Since tuples are immutable,
evil comparison functions can't permute the input, either.

I threw together a quick modification with tuple specialization, here are
the results:

2.6-pristine   2.6-patched
0.38usec       0.243usec      -36%     two integers
284usec       198usec          -30%    5000 integers
0.455usec     0.308usec      -32%     two datetime objects
600usec        536usec         -11%    5000 datetime objects

Not bad :-)

Raw commands and results are below:

Cashew:~/src-other/Python-2.6-pristine$ ./python.exe Lib/timeit.py
'max(2,8)'
1000000 loops, best of 3: 0.38 usec per loop

Cashew:/tmp/Python-2.6$ ./python.exe Lib/timeit.py 'max(2,8)'
1000000 loops, best of 3: 0.243 usec per loop

Cashew:~/src-other/Python-2.6-pristine$ ./python.exe Lib/timeit.py -s
'x=tuple(xrange(5000))' 'max(x)'
1000 loops, best of 3: 284 usec per loop

Cashew:/tmp/Python-2.6$ ./python.exe Lib/timeit.py -s
'x=tuple(xrange(5000))' 'max(x)'
10000 loops, best of 3: 198 usec per loop

Cashew:~/src-other/Python-2.6-pristine$ ./python.exe Lib/timeit.py -s
'import datetime' -s 'x = datetime.datetime.now()' -s 'y =
datetime.datetime.now()' 'max(x,y)'
1000000 loops, best of 3: 0.455 usec per loop

Cashew:/tmp/Python-2.6$ ./python.exe Lib/timeit.py -s 'import datetime' -s
'x = datetime.datetime.now()' -s 'y = datetime.datetime.now()' 'max(x,y)'
1000000 loops, best of 3: 0.308 usec per loop

Cashew:~/src-other/Python-2.6-pristine$ ./python.exe Lib/timeit.py -s
'import datetime' -s 'x = tuple(datetime.datetime.now() for x in
range(5000))' 'max(x)'
1000 loops, best of 3: 600 usec per loop

Cashew:/tmp/Python-2.6$ ./python.exe Lib/timeit.py -s 'import datetime' -s
'x = tuple(datetime.datetime.now() for x in range(5000))' 'max(x)'
1000 loops, best of 3: 536 usec per loop

(the one in /tmp/ is patched)

--
Daniel Stutzbach, Ph.D.
http://stutzbachenterprises.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081022/56e71619/attachment.htm>

From lists at cheimes.de  Thu Oct 23 02:49:48 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 23 Oct 2008 02:49:48 +0200
Subject: [Python-Dev] Backporting multiprocessing?
In-Reply-To: <18687.15915.98565.882018@montanaro-dyndns-org.local>
References: <60bb7ceb0810160901n367ce5f6r11f384e4661a56dc@mail.gmail.com>	<48F7BAE6.4070108@gmail.com>
	<48F7BE73.3010303@cheimes.de>	<18680.42554.791704.321274@montanaro-dyndns-org.local>	<48F8DFFA.9090707@cheimes.de>	<18684.43866.322328.195968@montanaro-dyndns-org.local>	<48FF2475.8090006@cheimes.de>
	<gdncc5$hmt$1@ger.gmane.org>
	<18687.15915.98565.882018@montanaro-dyndns-org.local>
Message-ID: <48FFCA2C.3090001@cheimes.de>

The latest svn version is now working with Python 2.4.4, Python 2.5.2 
and Python 2.6.0 on Linux (Ubuntu AMD64, Debian i386) and Windows XP. On 
Windows the multiprocessing module requires ctypes and pywin32 under 
Python 2.4.4.

Some of the examples aren't working correctly under 2.4 and 2.5. Jesse 
is looking into it.

Christian

From dripton at ripton.net  Thu Oct 23 03:48:15 2008
From: dripton at ripton.net (David Ripton)
Date: Wed, 22 Oct 2008 18:48:15 -0700
Subject: [Python-Dev] [ANN] VPython 0.1
Message-ID: <20081023014814.GA14030@vidar.dreamhost.com>

> Feedback is, of course, very welcome and it'd be great to have some pybench 
> results from different machines.

My results are very similar to Jakob's.

Gentoo Linux, 32-bit x86, Athlon 6400+ underclocked to 3.0 GHz.

make test:
282 tests OK.
5 tests failed:
    test_doctest test_hotshot test_inspect test_subprocess test_trace

-------------------------------------------------------------------------------
PYBENCH 2.0
-------------------------------------------------------------------------------
* using Python 2.5.2 (r252:60911, Oct 22 2008, 13:47:58) [GCC 4.1.2 20070214 ( (gdc 0.24, using dmd 1.020)) (Gentoo 4.1.2 p1.0.2)]
* disabled garbage collection
* system check interval set to maximum: 2147483647
* using timer: time.time

Calibrating tests. Please wait... done.

Running 10 round(s) of the suite at warp factor 10:

* Round 1 done in 8.474 seconds.
* Round 2 done in 8.389 seconds.
* Round 3 done in 8.438 seconds.
* Round 4 done in 8.411 seconds.
* Round 5 done in 8.484 seconds.
* Round 6 done in 8.471 seconds.
* Round 7 done in 8.492 seconds.
* Round 8 done in 8.549 seconds.
* Round 9 done in 8.429 seconds.
* Round 10 done in 8.542 seconds.

-------------------------------------------------------------------------------
Benchmark: 2008-10-22 20:45:22
-------------------------------------------------------------------------------

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    Linux-2.6.26-gentoo-r1-i686-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_6400+-with-glibc2.3
       Processor:      AMD Athlon(tm) 64 X2 Dual Core Processor 6400+

    Python:
       Implementation: n/a
       Executable:     /var/tmp/VPython-2.5.2/python
       Version:        2.5.2
       Compiler:       GCC 4.1.2 20070214 (  (gdc 0.24, using dmd 1.020)) (Gentoo 4.1.2 p1.0.2)
       Bits:           32bit
       Build:          Oct 22 2008 13:47:58 (#r252:60911)
       Unicode:        UCS2


-------------------------------------------------------------------------------
Comparing with: /tmp/vanilla252.pybench
-------------------------------------------------------------------------------

    Rounds: 10
    Warp:   10
    Timer:  time.time

    Machine Details:
       Platform ID:    Linux-2.6.26-gentoo-r1-i686-AMD_Athlon-tm-_64_X2_Dual_Core_Processor_6400+-with-glibc2.3
       Processor:      AMD Athlon(tm) 64 X2 Dual Core Processor 6400+

    Python:
       Implementation: n/a
       Executable:     /usr/local/bin/python2.5
       Version:        2.5.2
       Compiler:       GCC 4.1.2 20070214 (  (gdc 0.24, using dmd 1.020)) (Gentoo 4.1.2 p1.0.2)
       Bits:           32bit
       Build:          Oct 22 2008 20:39:10 (#r252:60911)
       Unicode:        UCS2


Test                             minimum run-time        average  run-time
                                 this    other   diff    this    other   diff
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:   144ms   205ms  -30.1%   162ms   240ms  -32.5%
           BuiltinMethodLookup:   164ms   222ms  -26.2%   167ms   236ms  -29.2%
                 CompareFloats:    90ms   211ms  -57.5%   103ms   222ms  -53.7%
         CompareFloatsIntegers:    88ms   182ms  -51.4%   107ms   200ms  -46.6%
               CompareIntegers:    63ms   258ms  -75.5%    84ms   272ms  -69.1%
        CompareInternedStrings:    93ms   252ms  -63.0%   103ms   261ms  -60.5%
                  CompareLongs:    65ms   180ms  -63.9%    87ms   203ms  -57.1%
                CompareStrings:   113ms   211ms  -46.5%   120ms   218ms  -44.9%
                CompareUnicode:   187ms   273ms  -31.7%   228ms   290ms  -21.4%
    ComplexPythonFunctionCalls:   261ms   330ms  -20.9%   277ms   336ms  -17.5%
                 ConcatStrings:   204ms   255ms  -20.2%   209ms   297ms  -29.7%
                 ConcatUnicode:   143ms   118ms  +20.3%   159ms   228ms  -30.0%
               CreateInstances:   172ms   112ms  +53.0%   187ms   211ms  -11.5%
            CreateNewInstances:   165ms   100ms  +65.0%   171ms   196ms  -12.6%
       CreateStringsWithConcat:   141ms   133ms   +5.8%   160ms   256ms  -37.3%
       CreateUnicodeWithConcat:   145ms   126ms  +14.8%   167ms   242ms  -30.9%
                  DictCreation:   129ms    98ms  +31.6%   131ms   184ms  -28.8%
             DictWithFloatKeys:   185ms   143ms  +29.6%   216ms   268ms  -19.6%
           DictWithIntegerKeys:   122ms   115ms   +6.0%   126ms   227ms  -44.4%
            DictWithStringKeys:    92ms   112ms  -17.6%   104ms   216ms  -51.8%
                      ForLoops:    98ms   224ms  -56.2%   117ms   243ms  -52.0%
                    IfThenElse:    89ms   221ms  -59.9%    97ms   237ms  -59.1%
                   ListSlicing:   123ms   111ms  +10.8%   131ms   141ms   -6.8%
                NestedForLoops:   138ms   234ms  -41.1%   153ms   262ms  -41.6%
          NormalClassAttribute:   131ms   225ms  -41.5%   139ms   243ms  -42.9%
       NormalInstanceAttribute:   121ms   191ms  -36.9%   121ms   210ms  -42.5%
           PythonFunctionCalls:   134ms   200ms  -32.6%   144ms   219ms  -34.2%
             PythonMethodCalls:   173ms   228ms  -23.9%   185ms   251ms  -26.5%
                     Recursion:   177ms   298ms  -40.5%   187ms   316ms  -40.8%
                  SecondImport:   135ms   133ms   +1.5%   160ms   147ms   +8.9%
           SecondPackageImport:   148ms   141ms   +5.0%   166ms   162ms   +2.7%
         SecondSubmoduleImport:   209ms   188ms  +11.4%   221ms   203ms   +8.6%
       SimpleComplexArithmetic:   131ms   219ms  -40.0%   139ms   239ms  -41.7%
        SimpleDictManipulation:   105ms   210ms  -49.9%   123ms   233ms  -47.1%
         SimpleFloatArithmetic:    93ms   224ms  -58.6%   109ms   246ms  -55.8%
      SimpleIntFloatArithmetic:    84ms   190ms  -56.0%    89ms   213ms  -58.4%
       SimpleIntegerArithmetic:    82ms   191ms  -57.1%    84ms   218ms  -61.5%
        SimpleListManipulation:    85ms   188ms  -54.6%    90ms   207ms  -56.7%
          SimpleLongArithmetic:   111ms   198ms  -44.0%   134ms   215ms  -37.6%
                    SmallLists:   126ms   182ms  -30.7%   143ms   202ms  -28.9%
                   SmallTuples:   132ms   193ms  -31.3%   143ms   210ms  -31.7%
         SpecialClassAttribute:   110ms   221ms  -50.4%   144ms   241ms  -40.1%
      SpecialInstanceAttribute:   146ms   236ms  -38.2%   165ms   258ms  -36.1%
                StringMappings:   177ms   209ms  -15.2%   186ms   218ms  -14.5%
              StringPredicates:   169ms   219ms  -22.9%   178ms   238ms  -25.0%
                 StringSlicing:   130ms   206ms  -37.0%   151ms   223ms  -32.4%
                     TryExcept:    92ms   230ms  -59.9%    94ms   258ms  -63.5%
                    TryFinally:   139ms   183ms  -23.6%   160ms   204ms  -21.8%
                TryRaiseExcept:   139ms   147ms   -5.0%   151ms   162ms   -6.7%
                  TupleSlicing:   135ms   174ms  -22.0%   151ms   190ms  -20.7%
               UnicodeMappings:   222ms   244ms   -8.9%   241ms   257ms   -6.3%
             UnicodePredicates:   170ms   214ms  -20.6%   179ms   227ms  -21.2%
             UnicodeProperties:   136ms   159ms  -14.9%   154ms   206ms  -25.3%
                UnicodeSlicing:   142ms   215ms  -34.1%   171ms   248ms  -31.3%
                   WithFinally:   208ms   260ms  -20.1%   212ms   271ms  -21.9%
               WithRaiseExcept:   175ms   193ms   -9.0%   186ms   209ms  -11.0%
-------------------------------------------------------------------------------
Totals:                          7682ms 10935ms  -29.8%  8468ms 12832ms  -34.0%

(this=2008-10-22 20:45:22, other=/tmp/vanilla252.pybench)

-- 
David Ripton    dripton at ripton.net

From andrewm at object-craft.com.au  Thu Oct 23 04:24:32 2008
From: andrewm at object-craft.com.au (Andrew McNamara)
Date: Thu, 23 Oct 2008 13:24:32 +1100
Subject: [Python-Dev] Python2.5 _sre deepcopy regression?
Message-ID: <20081023022432.B262B59C019@longblack.object-craft.com.au>

In version of Python prior to 2.5, it would appear that deepcopying
compiled regular expressions worked by accident:

2.4:

    >>> copy.deepcopy(re.compile(''))
    <_sre.SRE_Pattern object at 0xb7d53ef0>

2.5:

    >>> copy.deepcopy(re.compile(''))
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/usr/lib/python2.5/copy.py", line 173, in deepcopy
        y = copier(memo)
    TypeError: cannot deepcopy this pattern object

I say "by accident", since the SRE_Pattern object in 2.4 has
a __deepcopy__ method which raises the "cannot deepcopy this pattern
object" TypeError, however this method isn't found by copy.deepcopy()
in 2.4, and copy.deepcopy() falls back to using the pickle logic.

The _sre source has #ifdef-out support for __deepcopy__, issue 416670
has the gory details:

    http://bugs.python.org/issue416670

Changeset 38430 on the release24-maint branch introduced the changes
that stopped SRE_Pattern.__deepcopy__ being found. r38430 was a patch
forward ported from 2.3, but never ported to the trunk (probably a good
thing, too).

Thoughts?

-- 
Andrew McNamara, Senior Developer, Object Craft
http://www.object-craft.com.au/

From tjreedy at udel.edu  Thu Oct 23 06:19:47 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 23 Oct 2008 00:19:47 -0400
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <20081023014814.GA14030@vidar.dreamhost.com>
References: <20081023014814.GA14030@vidar.dreamhost.com>
Message-ID: <gdou0s$lr3$1@ger.gmane.org>

David Ripton wrote:
>> Feedback is, of course, very welcome and it'd be great to have some pybench 
>> results from different machines.
> 
> My results are very similar to Jakob's.

 From looking thru the vmgen manual, there are two things it is doing 
that CPython is not.  1. gcc-specific threaded code; claim doubles 
speed. 2. caching top of stack in a register; claim increases speed 
0-40%, depending on system.


From cadr4u at gmail.com  Thu Oct 23 09:08:33 2008
From: cadr4u at gmail.com (J. Sievers)
Date: Thu, 23 Oct 2008 09:08:33 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
Message-ID: <873aindc0e.fsf@svirfneblin.org>

Hey,

I hope you don't mind my replying in digest form.

First off, I guess I should be a little clearer as to what VPthon is
and what it does.

VPython is essentially a set of patches for CPython (in touches only
three files, diff -b is about 800 lines IIRC plus the switch statement
in ceval.c's EvalFrameEx()).


The main change is moving the VM instruction implementations, in
CPython, blocks of code following a case label, into a separate file,
adding Vmgen stack comments, and removing the explicit stack
manipulation code (plus some minor modification like renaming variables
to work with Vmgen's type prefixes and labels to enable the generation
of superinstructions).
Vmgen parses the stack comments and prints some macros around the
provided instruction code (incidentally, this reduced the 1500 line
switch body to about 1000 lines).
Interested parties should consult ceval.vmg and ceval-vm.i.

The nice thing about this is that:

a) It's fairly easy to implement different types of dispatch, simply by
changing a few macros (and while I haven't done this, it shouldn't be a
problem to add some switch dispatch #ifdefs for non-GCC platforms).

In particular, direct threaded code leads to less horrible branch
prediction than switch dispatch on many machines (exactly how
pronounced this effect is depends heavily on the specific
architecture).

b) Vmgen can generate superinstructions.
A quick primer:
A sequence of code such as LOAD_CONST LOAD_FAST BINARY_ADD will, in
CPython, push some constant onto the stack, push some local onto the
stack, then pop both off the stack, add them and push the result back
onto the stack.
Turning this into a superinstruction means inlining LOAD_CONST and
LOAD_FAST, modifying them to store the values they'd otherwise push
onto the stack in local variables and adding a version of BINARY_ADD
which reads its arguments from those local variables rather than the
stack (this reduces dispatch time in addition to pops and pushes).

David Gregg (and friends) recently published a paper comparing stack
based and register based VMs for Java and found that register based VMs
were substantially faster. The main reason for this appears to be the
absence of the various LOAD_ instructions in a register VM. They looked
at mitigating this using superinstructions but Java would have required
(again, IIRC) about a 1000 (which leads to substantial code growth).

Since Python doesn't have multiple (typed) versions of every
instruction (Java has iadd and so on) much fewer superinstructions are
necessary.

On my system, superinstructions account for about 10% of the 30%
performance gain.


As for limitations, as the README points out, currently 2 cases in
test_doctest fail, as well as 1 case in test_hotshot, test_inspect, and
test_subprocess. And most of the cases in test_trace.
The reason for this is, I suspect, that I removed the line tracing code
from ceval.c (I didn't want to look at it detail, and it doesn't seem
to affect anything else). I expect this would be a bit of work to fix
but I don't see it as a huge problem (in fact, if you don't use
settrace(?) it shouldn't affect you?).


Stack caching: a previous version of VPython supported this, but the
performance gain was minimal (maybe 1-2%, though if done really well
(e.g. using x as the top of stack cache), who knows, more may be
possible). Also, it let to some problems with the garbage collector
seeing an out-of-date stack_pointer[-1].


``Cell'' is, unfortunately, hardcoded into Vmgen. I could either patch
that or run ceval-vm.i through sed or something.


Finally, to the people who pointed out that VPython (the name) is
already taken: Darn! I really should have checked that! Will call it
something else in the future.

Anyway, HTH,
-jakob

From rhamph at gmail.com  Thu Oct 23 09:31:48 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Thu, 23 Oct 2008 01:31:48 -0600
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <873aindc0e.fsf@svirfneblin.org>
References: <873aindc0e.fsf@svirfneblin.org>
Message-ID: <aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>

On Thu, Oct 23, 2008 at 1:08 AM, J. Sievers <cadr4u at gmail.com> wrote:
> In particular, direct threaded code leads to less horrible branch
> prediction than switch dispatch on many machines (exactly how
> pronounced this effect is depends heavily on the specific
> architecture).

To clarify: This is *NOT* actually a form of threading, is it?  It
"merely" breaks the giant dispatch table into a series of small ones,
while also grouping instructions into larger superinstructions?  OS
threads are not touched at any point?


-- 
Adam Olsen, aka Rhamphoryncus

From mal at egenix.com  Thu Oct 23 12:02:12 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 23 Oct 2008 12:02:12 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <873aindc0e.fsf@svirfneblin.org>
References: <873aindc0e.fsf@svirfneblin.org>
Message-ID: <49004BA4.5050601@egenix.com>

On 2008-10-23 09:08, J. Sievers wrote:
> a) It's fairly easy to implement different types of dispatch, simply by
> changing a few macros (and while I haven't done this, it shouldn't be a
> problem to add some switch dispatch #ifdefs for non-GCC platforms).
> 
> In particular, direct threaded code leads to less horrible branch
> prediction than switch dispatch on many machines (exactly how
> pronounced this effect is depends heavily on the specific
> architecture).

Since VPython is GCC-only, how about naming the patch PyGCCVM ?!

I doubt that you'll find the same kind of performance increase
when using switch-based dispatching, but using more profile based
optimizations, it should be possible to come up with a solution
that provides a few 10% performance increase while still remaining
portable and readable.

When working on the switch statement in ceval some 10 years ago
it was easy to get a 10-20% performance increase by just moving
the switch cases around, breaking the switch in two groups of
opcodes that are used a lot and one for less often used ones and
then introducing a few fast paths via goto.

However, at that time CPUs had much smaller internal caches and
the 1.5.2 ceval VM had obviously hit some cache size limit on
my machine, since breaking the switch in two provided the best
performance increase. With todays CPUs, this shouldn't be a
problem anymore.

BTW: I hope you did not use pybench to get profiles of the opcodes.
That would most certainly result in good results for pybench, but
less good ones for general applications such as Django or Zope/Plone.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 23 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From greg.ewing at canterbury.ac.nz  Thu Oct 23 11:56:09 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 23 Oct 2008 22:56:09 +1300
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>
References: <873aindc0e.fsf@svirfneblin.org>
	<aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>
Message-ID: <49004A39.6050601@canterbury.ac.nz>

Adam Olsen wrote:

> To clarify: This is *NOT* actually a form of threading, is it?

I think the term "threaded code" is being used here in
the sense of Forth, i.e. instead of a sequence of small
integers that are dispatched using a switch statement,
you use the actual machine addresses of the switched-to
code as the virtual instruction opcodes, so the dispatch
is just an indirect jump.

I'm having trouble seeing how that would help with
branch prediction, though -- seems to me it should make
it worse if anything, since the CPU has no idea where
an indirect jump is going to go.

-- 
Greg

From amk at amk.ca  Thu Oct 23 14:25:56 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 23 Oct 2008 08:25:56 -0400
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>
References: <873aindc0e.fsf@svirfneblin.org>
	<aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>
Message-ID: <20081023122556.GA6003@amk-desktop.matrixgroup.net>

On Thu, Oct 23, 2008 at 01:31:48AM -0600, Adam Olsen wrote:
> To clarify: This is *NOT* actually a form of threading, is it?  It
> "merely" breaks the giant dispatch table into a series of small ones,
> while also grouping instructions into larger superinstructions?  OS
> threads are not touched at any point?

I haven't looked at Vmgen or VPython at all, but that's probably correct.
From: http://foldoc.org/foldoc.cgi?query=threaded+code

    threaded code: A technique for implementing virtual machine
    interpreters, introduced by J.R. Bell in 1973, where each op-code in
    the virtual machine instruction set is the address of some (lower
    level) code to perform the required operation. This kind of virtual
    machine can be implemented efficiently in machine code on most
    processors by simply performing an indirect jump to the address which
    is the next instruction.

    Many Forth implementations use threaded code and nowadays some use
    the term "threading" for almost any technique used to implement
    Forth's virtual machine.

--amk

From solipsis at pitrou.net  Thu Oct 23 15:13:53 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 23 Oct 2008 13:13:53 +0000 (UTC)
Subject: [Python-Dev] [ANN] VPython 0.1
References: <873aindc0e.fsf@svirfneblin.org>
	<aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>
	<20081023122556.GA6003@amk-desktop.matrixgroup.net>
Message-ID: <loom.20081023T130759-184@post.gmane.org>

A.M. Kuchling <amk <at> amk.ca> writes:
> 
>     threaded code: A technique for implementing virtual machine
>     interpreters, introduced by J.R. Bell in 1973, where each op-code in
>     the virtual machine instruction set is the address of some (lower
>     level) code to perform the required operation. This kind of virtual
>     machine can be implemented efficiently in machine code on most
>     processors by simply performing an indirect jump to the address which
>     is the next instruction.

Is this kind of optimization that useful on modern CPUs? It helps remove a
memory access to the switch/case lookup table, which should shave off the 3 CPU
cycles of latency of a modern L1 data cache, but it won't remove the branch
misprediction penalty of the indirect jump itself, which is more in the order of
10-20 CPU cycles depending on pipeline depth.

In 1973, CPUs were not pipelined and did not suffer any penalty for indirect
jumps, while lookups could be slow especially if they couldn't run in parallel
with other processing in the pipeline.

Thanks

Antoine.



From dripton at ripton.net  Thu Oct 23 15:19:25 2008
From: dripton at ripton.net (David Ripton)
Date: Thu, 23 Oct 2008 06:19:25 -0700
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <49004BA4.5050601@egenix.com>
References: <873aindc0e.fsf@svirfneblin.org> <49004BA4.5050601@egenix.com>
Message-ID: <20081023131925.GA24868@vidar.dreamhost.com>

On 2008.10.23 12:02:12 +0200, M.-A. Lemburg wrote:
> BTW: I hope you did not use pybench to get profiles of the opcodes.
> That would most certainly result in good results for pybench, but
> less good ones for general applications such as Django or Zope/Plone.

I was wondering about Pybench-specific optimization too, so last night I
ran a few dozen of my projecteuler.net solver programs with VPython.
Excluding the ones that are so fast that startup overhead dominates
runtime, the least speedup I saw versus vanilla 2.5.2 was ~10%, the best
was ~50%, and average was ~30%.  Pretty consistent with Pybench.

-- 
David Ripton    dripton at ripton.net

From mal at egenix.com  Thu Oct 23 15:21:21 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Thu, 23 Oct 2008 15:21:21 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <20081023131925.GA24868@vidar.dreamhost.com>
References: <873aindc0e.fsf@svirfneblin.org> <49004BA4.5050601@egenix.com>
	<20081023131925.GA24868@vidar.dreamhost.com>
Message-ID: <49007A51.8080306@egenix.com>

On 2008-10-23 15:19, David Ripton wrote:
> On 2008.10.23 12:02:12 +0200, M.-A. Lemburg wrote:
>> BTW: I hope you did not use pybench to get profiles of the opcodes.
>> That would most certainly result in good results for pybench, but
>> less good ones for general applications such as Django or Zope/Plone.
> 
> I was wondering about Pybench-specific optimization too, so last night I
> ran a few dozen of my projecteuler.net solver programs with VPython.
> Excluding the ones that are so fast that startup overhead dominates
> runtime, the least speedup I saw versus vanilla 2.5.2 was ~10%, the best
> was ~50%, and average was ~30%.  Pretty consistent with Pybench.

Thanks. That's good to know.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 23 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From solipsis at pitrou.net  Thu Oct 23 15:28:15 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 23 Oct 2008 13:28:15 +0000 (UTC)
Subject: [Python-Dev] [ANN] superinstructions (VPython 0.1)
References: <873aindc0e.fsf@svirfneblin.org>
Message-ID: <loom.20081023T132238-905@post.gmane.org>


Hi,

J. Sievers <cadr4u <at> gmail.com> writes:
> 
> A sequence of code such as LOAD_CONST LOAD_FAST BINARY_ADD will, in
> CPython, push some constant onto the stack, push some local onto the
> stack, then pop both off the stack, add them and push the result back
> onto the stack.
> Turning this into a superinstruction means inlining LOAD_CONST and
> LOAD_FAST, modifying them to store the values they'd otherwise push
> onto the stack in local variables and adding a version of BINARY_ADD
> which reads its arguments from those local variables rather than the
> stack (this reduces dispatch time in addition to pops and pushes).

The problem is that this only optimizes code like "x + 1" but not "1 + x" or "x
+ y". To make this generic a first step would be to try to fuse LOAD_CONST and
LOAD_FAST into a single opcode (and check it doesn't slow down the VM). This
could be possible by copying the constants table into the start of the frame's
variables array when the frame is created, so that the LOAD_FAST code still does
a single indexed array dereference. Since constants are constants, they don't
need to be copied again when the frame is re-used by a subsequent call of the
same function (but this would slow done recursive functions a bit, since those
have to create new frames each time they are called).

Then fusing e.g. LOAD_FAST LOAD_FAST BINARY_ADD into ADD_FAST_FAST would cover
many more cases than the optimization you are writing about, without any
explosion in the number of opcodes.

Regards

Antoine.



From skip at pobox.com  Thu Oct 23 17:19:45 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 23 Oct 2008 10:19:45 -0500
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <873aindc0e.fsf@svirfneblin.org>
References: <873aindc0e.fsf@svirfneblin.org>
Message-ID: <18688.38417.564620.418970@montanaro-dyndns-org.local>


    Jakob> David Gregg (and friends) recently published a paper comparing
    Jakob> stack based and register based VMs for Java and found that
    Jakob> register based VMs were substantially faster. The main reason for
    Jakob> this appears to be the absence of the various LOAD_ instructions
    Jakob> in a register VM.

Back in the day (2001?) with the help of Tim Peters and others here I wrote
most of what was necessary to convert from a stack-based to a register-based
virtual machine for CPython.  I had trouble with a couple VM instructions.
If memory serves correctly, exec was a problem because it could push a
variable number of results on the stack.  I never completed the project
because the instruction set kept growing and I couldn't keep up.  Still, it
looked promising because it avoided lots of intermediate loads and stores.

    Jakob> ``Cell'' is, unfortunately, hardcoded into Vmgen. I could either
    Jakob> patch that or run ceval-vm.i through sed or something.

If the patch is to work and eventually be accepted it will have to be
changed, probably with a little downstream find/replace.  You might want to
also file a bug report to the Vmgen folks.  If Vmgen is generating these
names the authors should really know to try to avoid name clashes.

Skip

From steve at holdenweb.com  Thu Oct 23 17:44:03 2008
From: steve at holdenweb.com (Steve Holden)
Date: Thu, 23 Oct 2008 11:44:03 -0400
Subject: [Python-Dev] [ANN] superinstructions (VPython 0.1)
In-Reply-To: <loom.20081023T132238-905@post.gmane.org>
References: <873aindc0e.fsf@svirfneblin.org>
	<loom.20081023T132238-905@post.gmane.org>
Message-ID: <gdq646$3a4$1@ger.gmane.org>

Antoine Pitrou wrote:
> Hi,
> 
> J. Sievers <cadr4u <at> gmail.com> writes:
>> A sequence of code such as LOAD_CONST LOAD_FAST BINARY_ADD will, in
>> CPython, push some constant onto the stack, push some local onto the
>> stack, then pop both off the stack, add them and push the result back
>> onto the stack.
>> Turning this into a superinstruction means inlining LOAD_CONST and
>> LOAD_FAST, modifying them to store the values they'd otherwise push
>> onto the stack in local variables and adding a version of BINARY_ADD
>> which reads its arguments from those local variables rather than the
>> stack (this reduces dispatch time in addition to pops and pushes).
> 
> The problem is that this only optimizes code like "x + 1" but not "1 + x" or "x
> + y". To make this generic a first step would be to try to fuse LOAD_CONST and
> LOAD_FAST into a single opcode (and check it doesn't slow down the VM). This
> could be possible by copying the constants table into the start of the frame's
> variables array when the frame is created, so that the LOAD_FAST code still does
> a single indexed array dereference. Since constants are constants, they don't
> need to be copied again when the frame is re-used by a subsequent call of the
> same function (but this would slow done recursive functions a bit, since those
> have to create new frames each time they are called).
> 
Though it would seem  redundant to create multiple copies of constant
structures. Wouldn't there be some way to optimize this to allow each
call to access the data from the same place?

> Then fusing e.g. LOAD_FAST LOAD_FAST BINARY_ADD into ADD_FAST_FAST would cover
> many more cases than the optimization you are writing about, without any
> explosion in the number of opcodes.
> 
regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From daniel at stutzbachenterprises.com  Thu Oct 23 17:49:07 2008
From: daniel at stutzbachenterprises.com (Daniel Stutzbach)
Date: Thu, 23 Oct 2008 10:49:07 -0500
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <loom.20081023T130759-184@post.gmane.org>
References: <873aindc0e.fsf@svirfneblin.org>
	<aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>
	<20081023122556.GA6003@amk-desktop.matrixgroup.net>
	<loom.20081023T130759-184@post.gmane.org>
Message-ID: <eae285400810230849l2fa5367au595c82ef65b052c2@mail.gmail.com>

On Thu, Oct 23, 2008 at 8:13 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> Is this kind of optimization that useful on modern CPUs? It helps remove a
> memory access to the switch/case lookup table, which should shave off the 3
> CPU
> cycles of latency of a modern L1 data cache, but it won't remove the branch
> misprediction penalty of the indirect jump itself, which is more in the
> order of
> 10-20 CPU cycles depending on pipeline depth.


I searched around for information on how threaded code interacts with branch
prediction, and here's what I found.  The short answer is that threaded code
significantly improves branch prediction.

Any bytecode interpreter has some kind of dispatch mechanism that jumps to
the next opcode handler.  With a while(1) {switch() {} } format, there's one
dispatch location in the machine code.  Processor branch prediction has a
horrible time trying to predict where that dispatch location is going to
jump to.  Here's some rough psuedo-assembly:

main_loop:
    compute next_handler
    jmp next_handler                ; abysmal branch prediction

handler1:
    ; do stuff
    jmp main_loop

handler2:
    ; do stuff
    jmp main_loop

With threaded code, every handler ends with its own dispatcher, so the
processor can make fine-grained predictions.  Since each opcode has its own
indirect branch instruction, the processor can track them separately and
make better predictions (e.g., it can figure out that opcode X is often
followed by opcode Y).

compute next_handler
jmp next_handler  ; executed only once

handler1:
    ; do stuff
    compute next_handler
    jmp next_handler

handler2:
    ; do stuff
    compute next_handler
    jmp next_handler

--
Daniel Stutzbach, Ph.D.
President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081023/b388836f/attachment.htm>

From greg.ewing at canterbury.ac.nz  Fri Oct 24 00:29:59 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 24 Oct 2008 11:29:59 +1300
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <eae285400810230849l2fa5367au595c82ef65b052c2@mail.gmail.com>
References: <873aindc0e.fsf@svirfneblin.org>
	<aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>
	<20081023122556.GA6003@amk-desktop.matrixgroup.net>
	<loom.20081023T130759-184@post.gmane.org>
	<eae285400810230849l2fa5367au595c82ef65b052c2@mail.gmail.com>
Message-ID: <4900FAE7.8050603@canterbury.ac.nz>

Daniel Stutzbach wrote:

> With threaded code, every handler ends with its own dispatcher, so the 
> processor can make fine-grained predictions.

I'm still wondering whether all this stuff makes a
noticeable difference in real-life Python code, which
spends most of its time doing expensive things like
attribute lookups and function calls, rather than
fiddling with integers in local variables.

-- 
Greg

From guido at python.org  Fri Oct 24 00:42:50 2008
From: guido at python.org (Guido van Rossum)
Date: Thu, 23 Oct 2008 15:42:50 -0700
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <87hc74n7t4.fsf@svirfneblin.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
Message-ID: <ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>

On Wed, Oct 22, 2008 at 5:16 AM, J. Sievers <cadr4u at gmail.com> wrote:
> I implemented a variant of the CPython VM on top of Gforth's Vmgen; this made
> it fairly straightforward to add direct threaded code and superinstructions for
> the various permutations of LOAD_CONST, LOAD_FAST, and most of the
> two-argument VM instructions.
>
> Sources:
>  http://svirfneblin.org/stuff/VPython-0.1.tar.gz

Hey Jakob,

This is very interesting (at this point I'm just lurking), but has
anyone pointed out yet that there already is something else called
VPython, which has a long standing "right" to the name?
http://www.vpython.org/ -- "3D Programming for Ordinary Mortals".

I hope you can find a better name for your project, otherwise there
will be no end of confusion...

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From greg.ewing at canterbury.ac.nz  Fri Oct 24 00:42:38 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 24 Oct 2008 11:42:38 +1300
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
Message-ID: <4900FDDE.5000802@canterbury.ac.nz>

Guido van Rossum wrote:

> there already is something else called VPython

Perhaps it could be called Fython (Python with a Forth-like VM)
or Thython (threaded-code Python).

-- 
Greg

From mhammond at skippinet.com.au  Fri Oct 24 01:29:28 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Fri, 24 Oct 2008 10:29:28 +1100
Subject: [Python-Dev] No manifest files on Windows?
In-Reply-To: <48F4E68B.2010701@v.loewis.de>
References: <48F4E68B.2010701@v.loewis.de>
Message-ID: <034401c93567$33f18660$9bd49320$@com.au>

> In http://bugs.python.org/issue4120, the author suggests that it might
> be possible to completely stop using the manifest mechanism, for VS
> 2008. Given the many problems that this SxS stuff has caused, this
> sounds like a very desirable route, although I haven't done any actual
> testing yet.
> 
> Can all the Windows experts please comment? Could that work? Does it
> have any downsides?
> 
> If it works, I would like to apply it to 3.0, although I probably
> won't be able to apply it to tomorrow's rc. Would it also be possible
> to change that in 2.6.1 (even though python26.dll in 2.6.0 already
> includes a manifest, as do all the pyd files)?

My take is that the bug is suggesting the manifest be dropped only from .pyd
files.  Python's executable and DLL will still have the manifest reference,
so the CRT is still loaded from a manifest (FWIW, the CRT will abort() if
initialized other than via a manifest!).

I don't see a downside and can see how it would help with private
assemblies.

[I've also added a comment to this effect to the bug]

Mark.


From glyph at divmod.com  Fri Oct 24 02:14:29 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Fri, 24 Oct 2008 00:14:29 -0000
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <4900FDDE.5000802@canterbury.ac.nz>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
Message-ID: <20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>

On 23 Oct, 10:42 pm, greg.ewing at canterbury.ac.nz wrote:
>Guido van Rossum wrote:
>>there already is something else called VPython
>
>Perhaps it could be called Fython (Python with a Forth-like VM)
>or Thython (threaded-code Python).

I feel like I've missed something important, but, why not just call it 
"Python"? ;-)

It's a substantial patch, but from what I understand it's a huge 
performance improvement and completely compatible, both at the C API and 
Python source levels.

Is there any reason this should be a separate project rather than just 
be rolled in to the core?  Obviously once issues like the 'Cell' macro 
are dealt with.

From greg.ewing at canterbury.ac.nz  Fri Oct 24 06:32:39 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Fri, 24 Oct 2008 17:32:39 +1300
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
Message-ID: <49014FE7.4090606@canterbury.ac.nz>

glyph at divmod.com wrote:

> Is there any reason this should be a separate project rather than just 
> be rolled in to the core?

Always keep in mind that one of the important characteristics
of CPython is that its implementation is very straightforward
and easy to follow. Replacing the ceval loop with machine
generated code would be a step away from that.

-- 
Greg

From tjreedy at udel.edu  Fri Oct 24 07:18:40 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 24 Oct 2008 01:18:40 -0400
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
References: <87hc74n7t4.fsf@svirfneblin.org>	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
Message-ID: <gdrlr9$jvq$1@ger.gmane.org>

glyph at divmod.com wrote:

> It's a substantial patch, but from what I understand it's a huge 
> performance improvement and completely compatible, both at the C API and 
> Python source levels.

I have not seen any Windows test yet.  The direct threading is 
gcc-specific, so there might be degradation with MSVC.


From martin at v.loewis.de  Fri Oct 24 08:31:48 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 24 Oct 2008 08:31:48 +0200
Subject: [Python-Dev] No manifest files on Windows?
In-Reply-To: <034401c93567$33f18660$9bd49320$@com.au>
References: <48F4E68B.2010701@v.loewis.de>
	<034401c93567$33f18660$9bd49320$@com.au>
Message-ID: <49016BD4.5040009@v.loewis.de>

> I don't see a downside and can see how it would help with private
> assemblies.
> 
> [I've also added a comment to this effect to the bug]

Thanks! I'll go ahead and accept this, then.

Regards,
Martin

From cadr4u at gmail.com  Fri Oct 24 09:41:12 2008
From: cadr4u at gmail.com (J. Sievers)
Date: Fri, 24 Oct 2008 09:41:12 +0200
Subject: [Python-Dev] [ANN] superinstructions (VPython 0.1)
References: <873aindc0e.fsf@svirfneblin.org>
	<loom.20081023T132238-905@post.gmane.org>
Message-ID: <87k5by30fb.fsf@svirfneblin.org>

Antoine Pitrou <solipsis at pitrou.net> writes:

> Hi,
>
> J. Sievers <cadr4u <at> gmail.com> writes:
>> 
>> A sequence of code such as LOAD_CONST LOAD_FAST BINARY_ADD will, in
>> CPython, push some constant onto the stack, push some local onto the
>> stack, then pop both off the stack, add them and push the result back
>> onto the stack.
>> Turning this into a superinstruction means inlining LOAD_CONST and
>> LOAD_FAST, modifying them to store the values they'd otherwise push
>> onto the stack in local variables and adding a version of BINARY_ADD
>> which reads its arguments from those local variables rather than the
>> stack (this reduces dispatch time in addition to pops and pushes).
>
> The problem is that this only optimizes code like "x + 1" but not "1 + x" or "x
> + y". To make this generic a first step would be to try to fuse LOAD_CONST and
> LOAD_FAST into a single opcode (and check it doesn't slow down the VM). This
> could be possible by copying the constants table into the start of the frame's
> variables array when the frame is created, so that the LOAD_FAST code still does
> a single indexed array dereference. Since constants are constants, they don't
> need to be copied again when the frame is re-used by a subsequent call of the
> same function (but this would slow done recursive functions a bit, since those
> have to create new frames each time they are called).
>
> Then fusing e.g. LOAD_FAST LOAD_FAST BINARY_ADD into ADD_FAST_FAST would cover
> many more cases than the optimization you are writing about, without any
> explosion in the number of opcodes.
>
> Regards
>
> Antoine.
>

I don't know that I'd call it an explosion. Currently there are ~150
superinstructions in all (problematic when using bytecode but
inconsequential once one is committed to threaded code).

A superinstruction definition in Vmgen, btw, looks as follows:

 fcbinary_add = load_fast load_const binary_add

-jakob


From cadr4u at gmail.com  Fri Oct 24 09:53:12 2008
From: cadr4u at gmail.com (J. Sievers)
Date: Fri, 24 Oct 2008 09:53:12 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
References: <873aindc0e.fsf@svirfneblin.org> <49004BA4.5050601@egenix.com>
Message-ID: <87abcu2zvb.fsf@svirfneblin.org>

"M.-A. Lemburg" <mal at egenix.com> writes:

[snip]
> BTW: I hope you did not use pybench to get profiles of the opcodes.
> That would most certainly result in good results for pybench, but
> less good ones for general applications such as Django or Zope/Plone.

Algorithm used for superinstruction selection:

1) idea: LOAD_CONST/LOAD_FAST + some suffix
2) potential suffixes:
   $ grep '..*(..*--..*)$' ceval.vmg | grep 'a1 a2 --' > INSTRUCTIONS
3) delete any instruction that I felt wouldn't be particularly frequent
   from INSTRUCTIONS (e.g. raise_varargs)
4) use awk to generate superinstruction definitions

Not only is this relatively unbiased but also very low effort.

-jakob


From cadr4u at gmail.com  Fri Oct 24 10:00:01 2008
From: cadr4u at gmail.com (J. Sievers)
Date: Fri, 24 Oct 2008 10:00:01 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
References: <873aindc0e.fsf@svirfneblin.org>
	<aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>
	<20081023122556.GA6003@amk-desktop.matrixgroup.net>
	<loom.20081023T130759-184@post.gmane.org>
	<eae285400810230849l2fa5367au595c82ef65b052c2@mail.gmail.com>
Message-ID: <874p322zjy.fsf@svirfneblin.org>

"Daniel Stutzbach" <daniel at stutzbachenterprises.com> writes:

[snip]

>    I searched around for information on how threaded code interacts with
>    branch prediction, and here's what I found.  The short answer is that
>    threaded code significantly improves branch prediction.

See ``Optimizing indirect branch prediction accuracy in virtual machine
interpreters'' and ``The Structure and Performance of Efficient
Interpreters''.

Cheers,
-jakob


From schmir at gmail.com  Fri Oct 24 10:00:33 2008
From: schmir at gmail.com (Ralf Schmitt)
Date: Fri, 24 Oct 2008 10:00:33 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <gdrlr9$jvq$1@ger.gmane.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<gdrlr9$jvq$1@ger.gmane.org>
Message-ID: <932f8baf0810240100q54cf75afl9fc3c3e9a5fc883a@mail.gmail.com>

On Fri, Oct 24, 2008 at 7:18 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> I have not seen any Windows test yet.  The direct threading is gcc-specific,
> so there might be degradation with MSVC.
>

erlang uses gcc to compile a single source file on windows and uses MS
VC++ to compile all others. They also need the gcc labels-as-values
extension and the file in question seems to be their bytecode
interpreter (beam_emu.c).

- Ralf

From cadr4u at gmail.com  Fri Oct 24 10:18:59 2008
From: cadr4u at gmail.com (J. Sievers)
Date: Fri, 24 Oct 2008 10:18:59 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
References: <873aindc0e.fsf@svirfneblin.org>
	<aac2c7cb0810230031v2bf68841vf0585057f0ae41bc@mail.gmail.com>
	<20081023122556.GA6003@amk-desktop.matrixgroup.net>
	<loom.20081023T130759-184@post.gmane.org>
	<eae285400810230849l2fa5367au595c82ef65b052c2@mail.gmail.com>
	<4900FAE7.8050603@canterbury.ac.nz>
Message-ID: <87prlq1k3w.fsf@svirfneblin.org>

Greg Ewing <greg.ewing at canterbury.ac.nz> writes:

> Daniel Stutzbach wrote:
>
>> With threaded code, every handler ends with its own dispatcher, so
>> the processor can make fine-grained predictions.
>
> I'm still wondering whether all this stuff makes a
> noticeable difference in real-life Python code, which
> spends most of its time doing expensive things like
> attribute lookups and function calls, rather than
> fiddling with integers in local variables.

Does it? Typically, VM interpreters spend a large percentage of their
cycles in opcode dispatch rather than opcode execution.
Running WITH_TSC compiled Python on some computation heavy application
(Mercurial maybe?) processing a typical workload would be interesting.

Also, I'd estimate that operations such as string concat and list
append (opcode BINARY_ADD) are fairly common, even in real-life Python
code.

Cheers,
-jakob


From mal at egenix.com  Fri Oct 24 10:41:10 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Fri, 24 Oct 2008 10:41:10 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <87abcu2zvb.fsf@svirfneblin.org>
References: <873aindc0e.fsf@svirfneblin.org> <49004BA4.5050601@egenix.com>
	<87abcu2zvb.fsf@svirfneblin.org>
Message-ID: <49018A26.8060409@egenix.com>

On 2008-10-24 09:53, J. Sievers wrote:
> "M.-A. Lemburg" <mal at egenix.com> writes:
> 
> [snip]
>> BTW: I hope you did not use pybench to get profiles of the opcodes.
>> That would most certainly result in good results for pybench, but
>> less good ones for general applications such as Django or Zope/Plone.
> 
> Algorithm used for superinstruction selection:
> 
> 1) idea: LOAD_CONST/LOAD_FAST + some suffix
> 2) potential suffixes:
>    $ grep '..*(..*--..*)$' ceval.vmg | grep 'a1 a2 --' > INSTRUCTIONS
> 3) delete any instruction that I felt wouldn't be particularly frequent
>    from INSTRUCTIONS (e.g. raise_varargs)
> 4) use awk to generate superinstruction definitions
> 
> Not only is this relatively unbiased but also very low effort.

Well, the "I felt wouldn't be particularly frequent" part does sound
a bit biased, but you obviously made good choices ;-)

I thought you used the tracing functions that Vmgen provides for
determining which combinations occur more often. That's how I worked
back then - I instrumented the interpreter and then let it run for
a few days doing whatever I worked on or with at the time.

I then found that it makes sense to process LOAD_FAST completely
outside the switch statement and to move common opcodes such as
CALL_* to the switch with the most used opcodes. Inlining the code
for calling C functions/methods also made a difference, since most
calls in Python are to C functions/methods.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Oct 24 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From cadr4u at gmail.com  Fri Oct 24 10:47:49 2008
From: cadr4u at gmail.com (J. Sievers)
Date: Fri, 24 Oct 2008 10:47:49 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
Message-ID: <87hc721iru.fsf@svirfneblin.org>

glyph at divmod.com writes:

> On 23 Oct, 10:42 pm, greg.ewing at canterbury.ac.nz wrote:
>>Guido van Rossum wrote:
>>>there already is something else called VPython
>>
>>Perhaps it could be called Fython (Python with a Forth-like VM)
>>or Thython (threaded-code Python).
>
> I feel like I've missed something important, but, why not just call it
> "Python"? ;-)
>
> It's a substantial patch, but from what I understand it's a huge
> performance improvement and completely compatible, both at the C API
> and Python source levels.
>
> Is there any reason this should be a separate project rather than just
> be rolled in to the core?  Obviously once issues like the 'Cell' macro
> are dealt with.

While it seems to work reliably, I don't think the current
implementation is really ``core-ready'' as it stands.
I consider it more of a prototype to demonstrate the potential impact
on these kinds of low-level dispatch optimizations (and for me
personally an opportunity to familiarize myself with the CPython VM).

IMO the main issues are:
 - Right now, CPython's bytecode is translated to direct threaded code
 lazily (when a code object is first evaluated). This would have to
 be merged into compile.c in some way plus some assorted minor changes.
 - The various things mentioned in TODO.
 - Finally, the core developers probably won't want to depend on Gforth
 (required to run Vmgen), so one might have to re-implement Vmgen (not
 a huge deal; I know that there are a number of unpublished versions of
 Vmgen floating around and IIRC one of them is even written in Python;
 even if not Vmgen isn't really that difficult to implement).
 Once that's done, however, I'd consider readability to have
 _increased_ if anything (compare the switch statement in vanilla
 Python 2.5.2's ceval.c to the patchset's ceval.vmg).

Note that none of the above are really show stoppers from a user's
point of view. It's about conforming to CPython's standards of
neatness.

Cheers,
-jakob


From solipsis at pitrou.net  Fri Oct 24 13:53:49 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 24 Oct 2008 11:53:49 +0000 (UTC)
Subject: [Python-Dev] [ANN] superinstructions (VPython 0.1)
References: <873aindc0e.fsf@svirfneblin.org>
	<loom.20081023T132238-905@post.gmane.org>
	<gdq646$3a4$1@ger.gmane.org>
Message-ID: <loom.20081024T114121-35@post.gmane.org>

Steve Holden <steve <at> holdenweb.com> writes:
> Though it would seem  redundant to create multiple copies of constant
> structures. Wouldn't there be some way to optimize this to allow each
> call to access the data from the same place?

It's just copying a table of pointers, so a bare memcpy() or its hand-written
equivalent is enough. Most functions shouldn't have lots of constants. Here is a
small test on some stdlib modules (with 2.5.2):

>>> def consts_per_func(mod):
...  return [len(f.func_code.co_consts) for f in vars(mod).values() if
hasattr(f, "func_code")]
... 
>>> consts_per_func(logging)
[3, 3, 2, 2, 2, 2, 3, 2, 3, 1, 3, 2, 4, 2, 3, 5, 3, 3, 1, 10, 3]
>>> consts_per_func(os)
[3, 4, 2, 4, 2, 1, 2, 1, 3, 1, 2, 2, 4, 2, 1, 2, 3, 3, 3, 3, 2, 2, 1, 1, 3, 2,
1, 1, 2, 1, 1]
>>> consts_per_func(threading)
[1, 3, 1, 1, 1, 3, 13, 1, 1, 1, 1, 1, 1, 2, 1, 1]
>>> consts_per_func(threading.Thread)
[1, 2, 1, 1, 3, 2, 2, 1, 2, 8, 2, 6, 3, 7, 13]
>>> consts_per_func(heapq)
[2, 3, 2, 2]
>>> consts_per_func(unittest.TestCase)
[2, 3, 2, 4, 2, 3, 4, 4, 3, 3, 4, 2, 3, 1, 1, 2, 1, 3, 4, 2, 2, 3, 2, 4, 2, 4,
3, 4, 2, 2, 2, 2, 4]

Regards

Antoine.



From stefan_ml at behnel.de  Fri Oct 24 14:34:22 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Fri, 24 Oct 2008 14:34:22 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <49014FE7.4090606@canterbury.ac.nz>
References: <87hc74n7t4.fsf@svirfneblin.org>	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>	<4900FDDE.5000802@canterbury.ac.nz>	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<49014FE7.4090606@canterbury.ac.nz>
Message-ID: <gdsfcd$204$1@ger.gmane.org>

Greg Ewing wrote:
> glyph at divmod.com wrote:
> 
>> Is there any reason this should be a separate project rather than just
>> be rolled in to the core?
> 
> Always keep in mind that one of the important characteristics
> of CPython is that its implementation is very straightforward
> and easy to follow. Replacing the ceval loop with machine
> generated code would be a step away from that.

Funny to hear that from the author of a well-known code generator. ;-)

I haven't looked at the specific Vmgen code in question, but I tend to find a
short DSL description of repetitive functionality much more straightforward
than the same thing implemented in custom, hand-optimised code in a general
purpose language like C. Just think of the switch split that MAL described in
one of his comments. Having two switch statements and a couple of separate
special cases for a single eval loop might look pretty arbitrary and not
straight forward at all to a reader who doesn't have enough background
regarding the performance characteristics of Python's VM statements.

Stefan


From skip at pobox.com  Fri Oct 24 17:02:51 2008
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 24 Oct 2008 10:02:51 -0500
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
Message-ID: <18689.58267.854372.503628@montanaro-dyndns-org.local>


    Guido> This is very interesting (at this point I'm just lurking), but
    Guido> has anyone pointed out yet that there already is something else
    Guido> called VPython, which has a long standing "right" to the name?

I believe Jakob has already been notified about this.  How about TPython?  A
quick google-check suggests that while there is at least one instance of
that name in use as related to Python, it seems to be fairly obscure and is
perhaps only used internally at CERN.

Skip

From skip at pobox.com  Fri Oct 24 17:07:20 2008
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 24 Oct 2008 10:07:20 -0500
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <gdrlr9$jvq$1@ger.gmane.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<gdrlr9$jvq$1@ger.gmane.org>
Message-ID: <18689.58536.870985.267266@montanaro-dyndns-org.local>


    Terry> I have not seen any Windows test yet.  The direct threading is
    Terry> gcc-specific, so there might be degradation with MSVC.

Not if a compiler #ifdef selects between two independent choices:

    #ifdef __GCC__       /* or whatever the right incantation is */
       #include "ceval-threaded.c"
    #else
       #include "ceval-switched.c"
    #endif

and so on...

BTW, as to the implementation of individual VM instructions I don't believe
the Vmgen stuff affects that.  It's just the way the instructions are
assembled.

Skip

From theller at ctypes.org  Fri Oct 24 17:21:38 2008
From: theller at ctypes.org (Thomas Heller)
Date: Fri, 24 Oct 2008 17:21:38 +0200
Subject: [Python-Dev] No manifest files on Windows?
In-Reply-To: <034401c93567$33f18660$9bd49320$@com.au>
References: <48F4E68B.2010701@v.loewis.de>
	<034401c93567$33f18660$9bd49320$@com.au>
Message-ID: <gdsp4o$a0j$1@ger.gmane.org>

Mark Hammond schrieb:
>> In http://bugs.python.org/issue4120, the author suggests that it might
>> be possible to completely stop using the manifest mechanism, for VS
>> 2008. Given the many problems that this SxS stuff has caused, this
>> sounds like a very desirable route, although I haven't done any actual
>> testing yet.
>> 
>> Can all the Windows experts please comment? Could that work? Does it
>> have any downsides?
>> 
>> If it works, I would like to apply it to 3.0, although I probably
>> won't be able to apply it to tomorrow's rc. Would it also be possible
>> to change that in 2.6.1 (even though python26.dll in 2.6.0 already
>> includes a manifest, as do all the pyd files)?
> 
> My take is that the bug is suggesting the manifest be dropped only from .pyd
> files.  Python's executable and DLL will still have the manifest reference,
> so the CRT is still loaded from a manifest (FWIW, the CRT will abort() if
> initialized other than via a manifest!).

What about COM objects: isn't pythoncom26.dll or _ctypes.pyd the first executable
image that is loaded first for them?  And how would they load the crt?

-- 
Thanks,
Thomas


From pje at telecommunity.com  Fri Oct 24 17:43:06 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Fri, 24 Oct 2008 11:43:06 -0400
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <87hc721iru.fsf@svirfneblin.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<87hc721iru.fsf@svirfneblin.org>
Message-ID: <20081024154345.42E503A408C@sparrow.telecommunity.com>

At 10:47 AM 10/24/2008 +0200, J. Sievers wrote:
>  - Right now, CPython's bytecode is translated to direct threaded code
>  lazily (when a code object is first evaluated). This would have to
>  be merged into compile.c in some way plus some assorted minor changes.

Don't you mean codeobject.c?  I don't see how the compiler relates, 
as Python programs can generate or transform bytecode.  (For example, 
Zope's Python sandboxing works that way.)


From status at bugs.python.org  Fri Oct 24 18:07:40 2008
From: status at bugs.python.org (Python tracker)
Date: Fri, 24 Oct 2008 18:07:40 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20081024160740.53A897821D@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (10/17/08 - 10/24/08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2124 open (+32) / 13891 closed (+20) / 16015 total (+52)

Open issues with patches:   700

Average duration of open issues: 713 days.
Median duration of open issues: 1918 days.

Open Issues Breakdown
   open  2108 (+32)
pending    16 ( +0)

Issues Created Or Reopened (54)
_______________________________

Memory leak in itertools.chain()                                 10/20/08
CLOSED http://bugs.python.org/issue2231    reopened loewis                    
       patch                                                                   

itertools.groupby() leaks memory with circular reference         10/20/08
CLOSED http://bugs.python.org/issue2246    reopened loewis                    
       patch                                                                   

Dis docs on CALL_FUNCTION a bit unclear                          10/17/08
CLOSED http://bugs.python.org/issue4141    created  novalis_dt                
       patch                                                                   

smtplib doesn't clear helo/ehlo flags on quit                    10/17/08
       http://bugs.python.org/issue4142    created  dwig                      
                                                                               

ast.Node objects do not have column number information           10/18/08
CLOSED http://bugs.python.org/issue4143    created  kevinwatters              
                                                                               

3 tutorial documentation errors                                  10/18/08
       http://bugs.python.org/issue4144    created  LambertDW                 
                                                                               

tabulary entries in PDF documentation                            10/19/08
       http://bugs.python.org/issue4145    created  wplappert                 
                                                                               

compilation of Modules/python.c fails on OpenBSD                 10/19/08
CLOSED http://bugs.python.org/issue4146    created  djmdjm                    
       patch, needs review                                                     

xml.dom.minidom toprettyxml: omit whitespace for text-only eleme 10/19/08
       http://bugs.python.org/issue4147    created  thomas.lee                
       patch                                                                   

Using blender                                                    10/20/08
CLOSED http://bugs.python.org/issue4148    created  mshee                     
                                                                               

Py_BuildValue and "y" format unit                                10/20/08
CLOSED http://bugs.python.org/issue4149    created  exe                       
                                                                               

pdb "up" command fails in generator frames                       10/20/08
CLOSED http://bugs.python.org/issue4150    created  arigo                     
       patch, patch                                                            

Separate build dir broken                                        10/20/08
       http://bugs.python.org/issue4151    created  nas                       
       patch                                                                   

ihooks module cannot handle absolute imports                     10/20/08
       http://bugs.python.org/issue4152    created  nas                       
                                                                               

Unicode HOWTO up to date?                                        10/20/08
       http://bugs.python.org/issue4153    created  tjreedy                   
                                                                               

More doc trivia                                                  10/20/08
CLOSED http://bugs.python.org/issue4154    created  LambertDW                 
                                                                               

Wrong math calculation                                           10/20/08
CLOSED http://bugs.python.org/issue4155    created  bubersson                 
                                                                               

Docs for BaseHandler.protocol_xxx methods are unclear            10/21/08
       http://bugs.python.org/issue4156    created  kjohnson                  
                                                                               

Tuple not callable in platform.py                                10/21/08
CLOSED http://bugs.python.org/issue4157    created  Feite                     
                                                                               

compilation of sqlite3 fails                                     10/21/08
       http://bugs.python.org/issue4158    created  jlp                       
                                                                               

Table about Standard Encodings is cut off at the bottom - 35 ent 10/21/08
       http://bugs.python.org/issue4159    created  wplappert                 
                                                                               

library.pdf - Section 12.13.2 Connection Objects - example cut o 10/21/08
       http://bugs.python.org/issue4160    created  wplappert                 
                                                                               

test_urllib fails with ValueError                                10/21/08
CLOSED http://bugs.python.org/issue4161    created  csernazs                  
       patch                                                                   

library.pdf - Section 17.6.4 Examples - Multiprocessing - Format 10/21/08
       http://bugs.python.org/issue4162    created  wplappert                 
                                                                               

textwrap wordsep_re Unicode                                      10/21/08
       http://bugs.python.org/issue4163    created  skirsche                  
       patch                                                                   

String double quoted fatal problem                               10/21/08
CLOSED http://bugs.python.org/issue4164    created  cliffdover88              
                                                                               

Failure building 64-bit Python on Leopard                        10/21/08
       http://bugs.python.org/issue4165    created  anand                     
                                                                               

extra "\fi" in sphinx.sty, line 62                               10/21/08
       http://bugs.python.org/issue4166    created  efiring                   
                                                                               

Inline Markup :const: shows up in Documentation                  10/22/08
       http://bugs.python.org/issue4167    created  wplappert                 
                                                                               

core dump exiting python                                         10/22/08
CLOSED http://bugs.python.org/issue4168    created  LambertDW                 
                                                                               

library/turtle.rst does not format properly in PDF mode          10/22/08
       http://bugs.python.org/issue4169    created  wplappert                 
                                                                               

segfault with defaultdict and pickle                             10/22/08
       http://bugs.python.org/issue4170    created  erickt                    
       patch                                                                   

SSL handshake fails after TCP connection in getpeername()        10/22/08
       http://bugs.python.org/issue4171    created  ddvoinikov                
                                                                               

distutils too picky about cygwin ld's version number             10/22/08
CLOSED http://bugs.python.org/issue4172    created  zooko                     
                                                                               

PDF documentation: long verbatim lines are cut off at right hand 10/22/08
       http://bugs.python.org/issue4173    created  wplappert                 
                                                                               

Performance optimization for min() and max() over lists          10/22/08
       http://bugs.python.org/issue4174    created  krisvale                  
       patch, patch, needs review                                              

Should Unix-only tty be present in Windows /Lib?                 10/22/08
CLOSED http://bugs.python.org/issue4175    created  tjreedy                   
                                                                               

segfault with pickle if 4th or 5th item of tuple returned by __r 10/22/08
       http://bugs.python.org/issue4176    created  ocean-city                
       patch, needs review                                                     

Crash in MIMEText on FreeBSD                                     10/22/08
       http://bugs.python.org/issue4177    created  surkamp                   
                                                                               

codecs: Documentation Inconsistency                              10/22/08
CLOSED http://bugs.python.org/issue4178    created  mfwitten                  
                                                                               

pdb: Allow the "list" command to return to the currently debugge 10/22/08
       http://bugs.python.org/issue4179    created  dclemente                 
       patch                                                                   

warnings.simplefilter("always") does not make warnings always sh 10/22/08
       http://bugs.python.org/issue4180    created  exarkun                   
       patch                                                                   

Invalid Behavior When a Default Argument is a Dictionary.        10/22/08
CLOSED http://bugs.python.org/issue4181    created  Pasha2009                 
                                                                               

warnings.warn shows the wrong filename and line number for stack 10/22/08
       http://bugs.python.org/issue4182    created  exarkun                   
                                                                               

test with pickle.HIGHEST_PROTOCOL missing                        10/23/08
CLOSED http://bugs.python.org/issue4183    created  ocean-city                
       patch, easy                                                             

Remove use of private attributes in smtpd                        10/23/08
       http://bugs.python.org/issue4184    created  richard                   
                                                                               

re module treats raw strings as normal strings                   10/23/08
CLOSED http://bugs.python.org/issue4185    created  ezio.melotti              
                                                                               

type() doesn't accept bases and dict keyword arguments           10/23/08
       http://bugs.python.org/issue4186    created  barry                     
                                                                               

IDLE will not start.                                             10/23/08
CLOSED http://bugs.python.org/issue4187    created  ldehkire                  
                                                                               

Lib/threading.py causes infinite recursion when running as verbo 10/23/08
       http://bugs.python.org/issue4188    created  gpolo                     
       patch                                                                   

Tilde compression isn't applied in TOC                           10/23/08
       http://bugs.python.org/issue4189    created  jeffreyharris             
                                                                               

IDLE (2.6) doesn't stay open                                     10/23/08
CLOSED http://bugs.python.org/issue4190    created  andrew.t.shaw7            
                                                                               

urlparse normalize URL path                                      10/24/08
       http://bugs.python.org/issue4191    created  monk.e.boy                
                                                                               

Subprocess error with I/O redirection to Pipes                   10/24/08
       http://bugs.python.org/issue4192    created  pakal                     
                                                                               



Issues Now Closed (37)
______________________

Memory leak in itertools.chain()                                    0 days
       http://bugs.python.org/issue2231    rhettinger                
       patch                                                                   

itertools.groupby() leaks memory with circular reference            0 days
       http://bugs.python.org/issue2246    loewis                    
       patch                                                                   

2to3 translates "import foobar" to "import .foobar" rather than   213 days
       http://bugs.python.org/issue2446    benjamin.peterson         
                                                                               

Document PEP 3118                                                 192 days
       http://bugs.python.org/issue2619    benjamin.peterson         
                                                                               

backport python 3.0 language functionality to python 2.6 by addi  114 days
       http://bugs.python.org/issue3238    kaizhu                    
       patch                                                                   

Pickler.dump from a badly initialized Pickler segfaults            54 days
       http://bugs.python.org/issue3664    amaury.forgeotdarc        
       patch                                                                   

"for me" installer problem on x64 Vista                            16 days
       http://bugs.python.org/issue4018    loewis                    
       patch, needs review                                                     

wrong page index number in reference book of python documentatio   16 days
       http://bugs.python.org/issue4027    wplappert                 
                                                                               

Reference to non-existent __version__ in ast module                14 days
       http://bugs.python.org/issue4062    benjamin.peterson         
                                                                               

ast.fix_missing_locations() breaks if node doesn't have "_attrib   12 days
       http://bugs.python.org/issue4067    aronacher                 
       patch                                                                   

Error copying directory to _static in Sphinx                       10 days
       http://bugs.python.org/issue4081    georg.brandl              
                                                                               

inspect.getargvalues return type not ArgInfo                       12 days
       http://bugs.python.org/issue4092    benjamin.peterson         
                                                                               

Namespace inconsistency                                             7 days
       http://bugs.python.org/issue4104    tjreedy                   
                                                                               

Performance regression in long division in 2.6                      2 days
       http://bugs.python.org/issue4128    loewis                    
                                                                               

IDLE crashes in my Windows Vista                                    2 days
       http://bugs.python.org/issue4138    loewis                    
                                                                               

Major error in cmath routines                                       0 days
       http://bugs.python.org/issue4139    marketdickinson           
                                                                               

Dis docs on CALL_FUNCTION a bit unclear                             0 days
       http://bugs.python.org/issue4141    benjamin.peterson         
       patch                                                                   

ast.Node objects do not have column number information              0 days
       http://bugs.python.org/issue4143    benjamin.peterson         
                                                                               

compilation of Modules/python.c fails on OpenBSD                    0 days
       http://bugs.python.org/issue4146    benjamin.peterson         
       patch, needs review                                                     

Using blender                                                       0 days
       http://bugs.python.org/issue4148    benjamin.peterson         
                                                                               

Py_BuildValue and "y" format unit                                   2 days
       http://bugs.python.org/issue4149    benjamin.peterson         
                                                                               

pdb "up" command fails in generator frames                          2 days
       http://bugs.python.org/issue4150    benjamin.peterson         
       patch, patch                                                            

More doc trivia                                                     0 days
       http://bugs.python.org/issue4154    benjamin.peterson         
                                                                               

Wrong math calculation                                              0 days
       http://bugs.python.org/issue4155    rhettinger                
                                                                               

Tuple not callable in platform.py                                   0 days
       http://bugs.python.org/issue4157    amaury.forgeotdarc        
                                                                               

test_urllib fails with ValueError                                   0 days
       http://bugs.python.org/issue4161    benjamin.peterson         
       patch                                                                   

String double quoted fatal problem                                  0 days
       http://bugs.python.org/issue4164    amaury.forgeotdarc        
                                                                               

core dump exiting python                                            1 days
       http://bugs.python.org/issue4168    benjamin.peterson         
                                                                               

distutils too picky about cygwin ld's version number                0 days
       http://bugs.python.org/issue4172    amaury.forgeotdarc        
                                                                               

Should Unix-only tty be present in Windows /Lib?                    0 days
       http://bugs.python.org/issue4175    loewis                    
                                                                               

codecs: Documentation Inconsistency                                 1 days
       http://bugs.python.org/issue4178    doerwalter                
                                                                               

Invalid Behavior When a Default Argument is a Dictionary.           0 days
       http://bugs.python.org/issue4181    christian.heimes          
                                                                               

test with pickle.HIGHEST_PROTOCOL missing                           0 days
       http://bugs.python.org/issue4183    ocean-city                
       patch, easy                                                             

re module treats raw strings as normal strings                      0 days
       http://bugs.python.org/issue4185    ezio.melotti              
                                                                               

IDLE will not start.                                                0 days
       http://bugs.python.org/issue4187    gpolo                     
                                                                               

IDLE (2.6) doesn't stay open                                        0 days
       http://bugs.python.org/issue4190    benjamin.peterson         
                                                                               

cygwinccompiler.get_versions fails on `ld -v` output             1706 days
       http://bugs.python.org/issue900977  amaury.forgeotdarc        
                                                                               



Top Issues Most Discussed (10)
______________________________

 11 Do not embed manifest files in *.pyd when compiling with MSVC     10 days
open    http://bugs.python.org/issue4120   

 10 tabulary entries in PDF documentation                              5 days
open    http://bugs.python.org/issue4145   

  9 Crash in MIMEText on FreeBSD                                       2 days
open    http://bugs.python.org/issue4177   

  7 warnings.simplefilter("always") does not make warnings always s    2 days
open    http://bugs.python.org/issue4180   

  7 merge json library with simplejson 2.0.3                           8 days
open    http://bugs.python.org/issue4136   

  6 Pickle stream for unicode object may contain non-ASCII characte  150 days
open    http://bugs.python.org/issue2980   

  5 Performance optimization for min() and max() over lists            2 days
open    http://bugs.python.org/issue4174   

  5 segfault with defaultdict and pickle                               2 days
open    http://bugs.python.org/issue4170   

  5 zipfile and winzip                                                25 days
open    http://bugs.python.org/issue3997   

  5 backport python 3.0 language functionality to python 2.6 by add  114 days
closed  http://bugs.python.org/issue3238   




From cadr4u at gmail.com  Fri Oct 24 18:08:47 2008
From: cadr4u at gmail.com (Jakob Sievers)
Date: Fri, 24 Oct 2008 18:08:47 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<gdrlr9$jvq$1@ger.gmane.org>
	<18689.58536.870985.267266@montanaro-dyndns-org.local>
Message-ID: <87iqri0ycw.fsf@svirfneblin.org>

skip at pobox.com writes:

> BTW, as to the implementation of individual VM instructions I don't believe
> the Vmgen stuff affects that.  It's just the way the instructions are
> assembled.

Vmgen handles the pushing and popping as well. E.g. ROT_THREE becomes:

  rot_three ( a1 a2 a3 -- a3 a1 a2 )

BINARY_POWER is:

  binary_power ( a1 a2 -- a  dec:a1 dec:a2  next:a )
  a = PyNumber_Power(a1, a2, Py_None);

(Here I have abused Vmgen a bit by declaring, in addition to the actual
value stack, some dummy stacks with different stack prefixes and using
the ``push'' instructions generated for those to do reference
counting.)

I should mention that some of the more involved instructions have no
declared effect (i.e. ( -- ) ) with stack manipulation still being
done by hand. 

Cheers,
-jakob


From cadr4u at gmail.com  Fri Oct 24 18:09:20 2008
From: cadr4u at gmail.com (Jakob Sievers)
Date: Fri, 24 Oct 2008 18:09:20 +0200
Subject: [Python-Dev] [ANN] VPython 0.1
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<18689.58267.854372.503628@montanaro-dyndns-org.local>
Message-ID: <87bpxa0ybz.fsf@svirfneblin.org>

skip at pobox.com writes:

>     Guido> This is very interesting (at this point I'm just lurking), but
>     Guido> has anyone pointed out yet that there already is something else
>     Guido> called VPython, which has a long standing "right" to the name?
>
> I believe Jakob has already been notified about this.  How about TPython?  A
> quick google-check suggests that while there is at least one instance of
> that name in use as related to Python, it seems to be fairly obscure and is
> perhaps only used internally at CERN.
>

TPython it is!

Cheers,
-jakob


From mhammond at skippinet.com.au  Sat Oct 25 02:47:26 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Sat, 25 Oct 2008 11:47:26 +1100
Subject: [Python-Dev] No manifest files on Windows?
In-Reply-To: <gdsp4o$a0j$1@ger.gmane.org>
References: <48F4E68B.2010701@v.loewis.de>
	<034401c93567$33f18660$9bd49320$@com.au>
	<gdsp4o$a0j$1@ger.gmane.org>
Message-ID: <03e001c9363b$42e6eb90$c8b4c2b0$@com.au>

> Mark Hammond schrieb:
> >> In http://bugs.python.org/issue4120, the author suggests that it
> might
> >> be possible to completely stop using the manifest mechanism, for VS
> >> 2008. Given the many problems that this SxS stuff has caused, this
> >> sounds like a very desirable route, although I haven't done any
> actual
> >> testing yet.
> >>
> >> Can all the Windows experts please comment? Could that work? Does it
> >> have any downsides?
> >>
> >> If it works, I would like to apply it to 3.0, although I probably
> >> won't be able to apply it to tomorrow's rc. Would it also be
> possible
> >> to change that in 2.6.1 (even though python26.dll in 2.6.0 already
> >> includes a manifest, as do all the pyd files)?
> >
> > My take is that the bug is suggesting the manifest be dropped only
> from .pyd
> > files.  Python's executable and DLL will still have the manifest
> reference,
> > so the CRT is still loaded from a manifest (FWIW, the CRT will
> abort() if
> > initialized other than via a manifest!).
> 
> What about COM objects: isn't pythoncom26.dll or _ctypes.pyd the first
> executable
> image that is loaded first for them?  And how would they load the crt?

Yeah - I don't think the manifest could be dropped from these files.
pythoncom is already loaded magically, but it would make sense to ensure the
patch is setup such that an extension can still request a manifest as normal
for the special cases when it is needed.  I think the vast majority of .pyd
files will not need the manifest...

But I'm surprised it hurts!  I'm surprised that if a .pyd references an
assembly already loaded into the process as a private assembly from another
directory, the load will fail unless there is *another* copy of the private
assembly next to the .pyd (the manifest reference is always a "strong"
reference including versions and hashes, so there is no ambiguity), but at
this stage I'm taking it on faith that the bug as reported does actually
exist - I've only ever tested with shared assembles.

Cheers,

Mark


From greg.ewing at canterbury.ac.nz  Sat Oct 25 05:33:23 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 25 Oct 2008 16:33:23 +1300
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <gdsfcd$204$1@ger.gmane.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<49014FE7.4090606@canterbury.ac.nz> <gdsfcd$204$1@ger.gmane.org>
Message-ID: <49029383.5030203@canterbury.ac.nz>

Stefan Behnel wrote:

> Funny to hear that from the author of a well-known code generator. ;-)

I've never claimed that anything about the implementation
of Pyrex is easy to follow. :-)

> Having two switch statements and a couple of separate
> special cases for a single eval loop might look pretty arbitrary and not
> straight forward at all to a reader who doesn't have enough background
> regarding the performance characteristics of Python's VM statements.

Maybe not, but at least you can follow what it's doing
just by knowing C. Introducing vmgen would introduce another
layer for the reader to learn about.

I'm not saying this is a bad enough problem to stop it
being done, just that it's something to consider that isn't
necessarily on the positive side.

-- 
Greg


From amk at amk.ca  Sat Oct 25 13:50:29 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Sat, 25 Oct 2008 07:50:29 -0400
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <49029383.5030203@canterbury.ac.nz>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<49014FE7.4090606@canterbury.ac.nz> <gdsfcd$204$1@ger.gmane.org>
	<49029383.5030203@canterbury.ac.nz>
Message-ID: <20081025115029.GA4892@amk.local>

On Sat, Oct 25, 2008 at 04:33:23PM +1300, Greg Ewing wrote:
> Maybe not, but at least you can follow what it's doing
> just by knowing C. Introducing vmgen would introduce another
> layer for the reader to learn about.

A stray thought: does using a generator for the VM make life easier
for the Stackless Python developers in any way?  Does it make it
possible for stock CPython to become stackless?

--amk

From pje at telecommunity.com  Sat Oct 25 19:59:39 2008
From: pje at telecommunity.com (Phillip J. Eby)
Date: Sat, 25 Oct 2008 13:59:39 -0400
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <20081025115029.GA4892@amk.local>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<49014FE7.4090606@canterbury.ac.nz> <gdsfcd$204$1@ger.gmane.org>
	<49029383.5030203@canterbury.ac.nz>
	<20081025115029.GA4892@amk.local>
Message-ID: <20081025175822.666153A4068@sparrow.telecommunity.com>

At 07:50 AM 10/25/2008 -0400, A.M. Kuchling wrote:
>On Sat, Oct 25, 2008 at 04:33:23PM +1300, Greg Ewing wrote:
> > Maybe not, but at least you can follow what it's doing
> > just by knowing C. Introducing vmgen would introduce another
> > layer for the reader to learn about.
>
>A stray thought: does using a generator for the VM make life easier
>for the Stackless Python developers in any way?  Does it make it
>possible for stock CPython to become stackless?

Dunno about that, but I do know that having stack effect info for the 
bytecode could help with things like bytecode verification (without 
having to define a bunch of magic constants that duplicate 
information from the innards of ceval.c).


From greg.ewing at canterbury.ac.nz  Sun Oct 26 01:07:31 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sun, 26 Oct 2008 12:07:31 +1300
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <20081025115029.GA4892@amk.local>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<49014FE7.4090606@canterbury.ac.nz> <gdsfcd$204$1@ger.gmane.org>
	<49029383.5030203@canterbury.ac.nz> <20081025115029.GA4892@amk.local>
Message-ID: <4903A6B3.9040606@canterbury.ac.nz>

A.M. Kuchling wrote:

> A stray thought: does using a generator for the VM make life easier
> for the Stackless Python developers in any way?  Does it make it
> possible for stock CPython to become stackless?

I doubt it. A major barrier to stacklessness is that
a lot of extension modules would need to be rewritten
in a completely different way. Just changing the VM
isn't going to make a difference to that.

-- 
Greg

From stefan_ml at behnel.de  Sun Oct 26 07:47:08 2008
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 26 Oct 2008 07:47:08 +0100
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <4903A6B3.9040606@canterbury.ac.nz>
References: <87hc74n7t4.fsf@svirfneblin.org>	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>	<4900FDDE.5000802@canterbury.ac.nz>	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>	<49014FE7.4090606@canterbury.ac.nz>
	<gdsfcd$204$1@ger.gmane.org>	<49029383.5030203@canterbury.ac.nz>
	<20081025115029.GA4892@amk.local>
	<4903A6B3.9040606@canterbury.ac.nz>
Message-ID: <ge13pc$49f$1@ger.gmane.org>

Greg Ewing wrote:
> A.M. Kuchling wrote:
> 
>> A stray thought: does using a generator for the VM make life easier
>> for the Stackless Python developers in any way?  Does it make it
>> possible for stock CPython to become stackless?
> 
> I doubt it. A major barrier to stacklessness is that
> a lot of extension modules would need to be rewritten
> in a completely different way. Just changing the VM
> isn't going to make a difference to that.

That's obviously a problem, but it only answers the second question, not the
first one.

Stefan


From cadr4u at gmail.com  Sun Oct 26 08:08:18 2008
From: cadr4u at gmail.com (Jakob Sievers)
Date: Sun, 26 Oct 2008 08:08:18 +0100
Subject: [Python-Dev] [ANN] VPython 0.1
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<87hc721iru.fsf@svirfneblin.org>
	<20081024154345.42E503A408C@sparrow.telecommunity.com>
Message-ID: <87hc6zal5p.fsf@svirfneblin.org>

"Phillip J. Eby" <pje at telecommunity.com> writes:

> At 10:47 AM 10/24/2008 +0200, J. Sievers wrote:
>>  - Right now, CPython's bytecode is translated to direct threaded code
>>  lazily (when a code object is first evaluated). This would have to
>>  be merged into compile.c in some way plus some assorted minor changes.
>
> Don't you mean codeobject.c?  I don't see how the compiler relates, as
> Python programs can generate or transform bytecode.  (For example,
> Zope's Python sandboxing works that way.)
>

Also good :).
(I was thinking about the superinstruction selection code which should
perhaps go into optimize_code() since it's a kind of peephole
optimization. The bytecodes->addresses part might even stay in ceval.c
I guess).

-jakob


From greg.ewing at canterbury.ac.nz  Sun Oct 26 22:49:51 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 27 Oct 2008 10:49:51 +1300
Subject: [Python-Dev] [ANN] VPython 0.1
In-Reply-To: <ge13pc$49f$1@ger.gmane.org>
References: <87hc74n7t4.fsf@svirfneblin.org>
	<ca471dc20810231542v7c9e65a0ifa9575cb27203be8@mail.gmail.com>
	<4900FDDE.5000802@canterbury.ac.nz>
	<20081024001429.6400.655501457.divmod.xquotient.310@weber.divmod.com>
	<49014FE7.4090606@canterbury.ac.nz> <gdsfcd$204$1@ger.gmane.org>
	<49029383.5030203@canterbury.ac.nz> <20081025115029.GA4892@amk.local>
	<4903A6B3.9040606@canterbury.ac.nz> <ge13pc$49f$1@ger.gmane.org>
Message-ID: <4904E5FF.3070706@canterbury.ac.nz>

Stefan Behnel wrote:

> That's obviously a problem, but it only answers the second question, not the
> first one. [does using a generator for the VM make life easier
> for the Stackless Python developers in any way?]

The Stackless Python developers themselves would have to answer
that one, but my guess is that it would help about the same
amount as it would help non-Stackless Python, i.e. there's
not much about stacklessness per se that makes it particularly
beneficial to have a generated VM.

-- 
Greg

From brett at python.org  Mon Oct 27 04:13:49 2008
From: brett at python.org (Brett Cannon)
Date: Sun, 26 Oct 2008 20:13:49 -0700
Subject: [Python-Dev] is the 'path' argument to an importer's find_module()
	just a hint?
Message-ID: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>

I just discovered frozen packages set their __path__ simply to their
name and not to a list as expected (http://bugs.python.org/issue4211).
This made me think about the 'path' argument to find_module() and
whether it can be treated as simply a hint or should always be
seriously looked at.

Take frozen modules, for instance. If the 'path' argument is meant to
always be considered then if a frozen module is within a package a
check should be done to make sure that the parent package is in 'path'
somewhere. But if it is simply a hint, then 'path' should be ignored
and whether the module can be found should depend fully on
imp.is_frozen().

So, what do people think? Should 'path' for find_module() always be
taken into consideration, or only when it happens to be convenient?

-Brett

From ncoghlan at gmail.com  Mon Oct 27 11:50:28 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 27 Oct 2008 20:50:28 +1000
Subject: [Python-Dev] is the 'path' argument to an importer's
 find_module() just a hint?
In-Reply-To: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
Message-ID: <49059CF4.2060906@gmail.com>

Brett Cannon wrote:
> I just discovered frozen packages set their __path__ simply to their
> name and not to a list as expected (http://bugs.python.org/issue4211).
> This made me think about the 'path' argument to find_module() and
> whether it can be treated as simply a hint or should always be
> seriously looked at.
> 
> Take frozen modules, for instance. If the 'path' argument is meant to
> always be considered then if a frozen module is within a package a
> check should be done to make sure that the parent package is in 'path'
> somewhere. But if it is simply a hint, then 'path' should be ignored
> and whether the module can be found should depend fully on
> imp.is_frozen().
> 
> So, what do people think? Should 'path' for find_module() always be
> taken into consideration, or only when it happens to be convenient?

>From PEP 302:

importer.find_module(fullname, path=None)

    This method will be called with the fully qualified name of the
    module.  If the importer is installed on sys.meta_path, it will
    receive a second argument, which is None for a top-level module, or
    package.__path__ for submodules or subpackages[7].  It should return
    a loader object if the module was found, or None if it wasn't.  If
    find_module() raises an exception, it will be propagated to the
    caller, aborting the import.


[7] The path argument to importer.find_module() is there because the
    pkg.__path__ variable may be needed at this point.  It may either
    come from the actual parent module or be supplied by
    imp.find_module() or the proposed imp.get_loader() function.


Note the first "may" in the footnote. If an importer doesn't need
pkg.__path__ to find submodules then it isn't obliged to use it, but the
import machinery provides it as a convenience.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From brett at python.org  Mon Oct 27 20:24:04 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 27 Oct 2008 12:24:04 -0700
Subject: [Python-Dev] is the 'path' argument to an importer's
	find_module() just a hint?
In-Reply-To: <49059CF4.2060906@gmail.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
	<49059CF4.2060906@gmail.com>
Message-ID: <bbaeab100810271224l426bdd47g27c4febce104972b@mail.gmail.com>

On Mon, Oct 27, 2008 at 3:50 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Brett Cannon wrote:
>> I just discovered frozen packages set their __path__ simply to their
>> name and not to a list as expected (http://bugs.python.org/issue4211).
>> This made me think about the 'path' argument to find_module() and
>> whether it can be treated as simply a hint or should always be
>> seriously looked at.
>>
>> Take frozen modules, for instance. If the 'path' argument is meant to
>> always be considered then if a frozen module is within a package a
>> check should be done to make sure that the parent package is in 'path'
>> somewhere. But if it is simply a hint, then 'path' should be ignored
>> and whether the module can be found should depend fully on
>> imp.is_frozen().
>>
>> So, what do people think? Should 'path' for find_module() always be
>> taken into consideration, or only when it happens to be convenient?
>
> From PEP 302:
>
> importer.find_module(fullname, path=None)
>
>    This method will be called with the fully qualified name of the
>    module.  If the importer is installed on sys.meta_path, it will
>    receive a second argument, which is None for a top-level module, or
>    package.__path__ for submodules or subpackages[7].  It should return
>    a loader object if the module was found, or None if it wasn't.  If
>    find_module() raises an exception, it will be propagated to the
>    caller, aborting the import.
>
>
> [7] The path argument to importer.find_module() is there because the
>    pkg.__path__ variable may be needed at this point.  It may either
>    come from the actual parent module or be supplied by
>    imp.find_module() or the proposed imp.get_loader() function.
>
>
> Note the first "may" in the footnote. If an importer doesn't need
> pkg.__path__ to find submodules then it isn't obliged to use it, but the
> import machinery provides it as a convenience.
>

Good enough for me. Then I am just going to ignore the 'path' argument
for frozen modules but use it to short-circuit imports for built-in
modules.

-Brett

From glyph at divmod.com  Mon Oct 27 20:31:15 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Mon, 27 Oct 2008 19:31:15 -0000
Subject: [Python-Dev] is the 'path' argument to an
	importer's	find_module() just a hint?
In-Reply-To: <bbaeab100810271224l426bdd47g27c4febce104972b@mail.gmail.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
	<49059CF4.2060906@gmail.com>
	<bbaeab100810271224l426bdd47g27c4febce104972b@mail.gmail.com>
Message-ID: <20081027193115.6400.126590000.divmod.xquotient.409@weber.divmod.com>

On 07:24 pm, brett at python.org wrote:
>On Mon, Oct 27, 2008 at 3:50 AM, Nick Coghlan <ncoghlan at gmail.com> 
>wrote:
>>Brett Cannon wrote:

>Good enough for me. Then I am just going to ignore the 'path' argument
>for frozen modules but use it to short-circuit imports for built-in
>modules.

For what it's worth, Twisted uses the __path__ attribute to facilitate 
its plugin mechanism.  If this changes so that importing from frozen 
packages no longer honors __path__, then Twisted will no longer support 
plugins if the package to be plugged into is frozen.

What is the motivation to change this?

From theller at ctypes.org  Mon Oct 27 21:03:32 2008
From: theller at ctypes.org (Thomas Heller)
Date: Mon, 27 Oct 2008 21:03:32 +0100
Subject: [Python-Dev] is the 'path' argument to an importer's
 find_module() just a hint?
In-Reply-To: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
Message-ID: <ge56ot$ajh$2@ger.gmane.org>

Brett Cannon schrieb:
> I just discovered frozen packages set their __path__ simply to their
> name and not to a list as expected (http://bugs.python.org/issue4211).
> This made me think about the 'path' argument to find_module() and
> whether it can be treated as simply a hint or should always be
> seriously looked at.
> 
> Take frozen modules, for instance. If the 'path' argument is meant to
> always be considered then if a frozen module is within a package a
> check should be done to make sure that the parent package is in 'path'
> somewhere. But if it is simply a hint, then 'path' should be ignored
> and whether the module can be found should depend fully on
> imp.is_frozen().
> 
> So, what do people think? Should 'path' for find_module() always be
> taken into consideration, or only when it happens to be convenient?

At the moment I don't care about find_module for frozen modules/packages
(is someone still using these?), but all the code I remember that
manipulates a packages __path__ would most certainly break if it
finds a string instead of a list.

Thomas


From skip at pobox.com  Mon Oct 27 21:22:43 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 27 Oct 2008 15:22:43 -0500
Subject: [Python-Dev] Why do we care about -OPT:Olimit=0?
Message-ID: <18694.8979.527287.230395@montanaro-dyndns-org.local>

Python's configure.in contains this check:

    # disable check for icc since it seems to pass, but generates a warning
    if test "$CC" = icc
    then
      ac_cv_opt_olimit_ok=no
    fi

    AC_MSG_CHECKING(whether $CC accepts -OPT:Olimit=0)
    AC_CACHE_VAL(ac_cv_opt_olimit_ok,
    [ac_save_cc="$CC"
    CC="$CC -OPT:Olimit=0"
    AC_TRY_RUN([int main() { return 0; }],
      ac_cv_opt_olimit_ok=yes,
      ac_cv_opt_olimit_ok=no,
      ac_cv_opt_olimit_ok=no)
    CC="$ac_save_cc"])
    AC_MSG_RESULT($ac_cv_opt_olimit_ok)

Why do we care about that particular obscure compiler flag?  I'm trying to
build Python 2.4.x with the Sun Studio 12 compiler, which appears to operate
in the same broken fashion as icc.  It warns that it doesn't support the
option but still exits with a zero status which makes configure incorrectly
think it does support the flag.

This same chunk of code seems to be in configure.in dating back to 1997 with
this comment:

    r8948 | guido | 1997-10-09 15:24:13 -0500 (Thu, 09 Oct 1997) | 2 lines

    Don Beaudry's changes to support SGI_ABI on Irix 6.x.

Can this check be ripped out?

Skip

From brett at python.org  Mon Oct 27 21:24:01 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 27 Oct 2008 13:24:01 -0700
Subject: [Python-Dev] is the 'path' argument to an importer's
	find_module() just a hint?
In-Reply-To: <20081027193115.6400.126590000.divmod.xquotient.409@weber.divmod.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
	<49059CF4.2060906@gmail.com>
	<bbaeab100810271224l426bdd47g27c4febce104972b@mail.gmail.com>
	<20081027193115.6400.126590000.divmod.xquotient.409@weber.divmod.com>
Message-ID: <bbaeab100810271324u128e2748wf835aca6ce657c91@mail.gmail.com>

On Mon, Oct 27, 2008 at 12:31 PM,  <glyph at divmod.com> wrote:
> On 07:24 pm, brett at python.org wrote:
>>
>> On Mon, Oct 27, 2008 at 3:50 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>
>>> Brett Cannon wrote:
>
>> Good enough for me. Then I am just going to ignore the 'path' argument
>> for frozen modules but use it to short-circuit imports for built-in
>> modules.
>
> For what it's worth, Twisted uses the __path__ attribute to facilitate its
> plugin mechanism.
>  If this changes so that importing from frozen packages no
> longer honors __path__, then Twisted will no longer support plugins if the
> package to be plugged into is frozen.
>

But how do you manipulate it for frozen packages currently? Importing
a frozen package currently has a bad __path__ value::

  >>> import __phello__; __phello__.__path__
  '__phello__'

Notice how that is not a list. Do you special-case frozen packages?

> What is the motivation to change this?

Ease of implementation for me (this has not gone far enough to be an
official change; as of this moment this is just for my import rewrite
vaporware). imp.is_frozen() tells you whether a module is frozen or
not, and thus whether to use imp.init_frozen(). So implementing a
meta_path importer for frozen modules can be nothing more than calls
to imp.is_frozen() and imp.init_frozen() if you just ignore the 'path'
argument.

-Brett

From brett at python.org  Mon Oct 27 21:26:06 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 27 Oct 2008 13:26:06 -0700
Subject: [Python-Dev] is the 'path' argument to an importer's
	find_module() just a hint?
In-Reply-To: <ge56ot$ajh$2@ger.gmane.org>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
	<ge56ot$ajh$2@ger.gmane.org>
Message-ID: <bbaeab100810271326g2f12b75bh26817206b0fe40bf@mail.gmail.com>

On Mon, Oct 27, 2008 at 1:03 PM, Thomas Heller <theller at ctypes.org> wrote:
> Brett Cannon schrieb:
>> I just discovered frozen packages set their __path__ simply to their
>> name and not to a list as expected (http://bugs.python.org/issue4211).
>> This made me think about the 'path' argument to find_module() and
>> whether it can be treated as simply a hint or should always be
>> seriously looked at.
>>
>> Take frozen modules, for instance. If the 'path' argument is meant to
>> always be considered then if a frozen module is within a package a
>> check should be done to make sure that the parent package is in 'path'
>> somewhere. But if it is simply a hint, then 'path' should be ignored
>> and whether the module can be found should depend fully on
>> imp.is_frozen().
>>
>> So, what do people think? Should 'path' for find_module() always be
>> taken into consideration, or only when it happens to be convenient?
>
> At the moment I don't care about find_module for frozen modules/packages
> (is someone still using these?),

Beats me. At this point I am only implementing them for
backwards-compatibility, but they are exposing some inconsistencies in
how various aspects of import are implemented. If I had my way they
would be an optional thing that you would have to explicitly put on
sys.meta_path to avoid the overhead.

> but all the code I remember that
> manipulates a packages __path__ would most certainly break if it
> finds a string instead of a list.
>

Oh, definitely. That's why I filed the bug report on top of asking for
feedback on this.

-Brett

From brett at python.org  Mon Oct 27 21:27:44 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 27 Oct 2008 13:27:44 -0700
Subject: [Python-Dev] Why do we care about -OPT:Olimit=0?
In-Reply-To: <18694.8979.527287.230395@montanaro-dyndns-org.local>
References: <18694.8979.527287.230395@montanaro-dyndns-org.local>
Message-ID: <bbaeab100810271327x52d41454ke3b4dd55f1abec4b@mail.gmail.com>

On Mon, Oct 27, 2008 at 1:22 PM,  <skip at pobox.com> wrote:
> Python's configure.in contains this check:
>
>    # disable check for icc since it seems to pass, but generates a warning
>    if test "$CC" = icc
>    then
>      ac_cv_opt_olimit_ok=no
>    fi
>
>    AC_MSG_CHECKING(whether $CC accepts -OPT:Olimit=0)
>    AC_CACHE_VAL(ac_cv_opt_olimit_ok,
>    [ac_save_cc="$CC"
>    CC="$CC -OPT:Olimit=0"
>    AC_TRY_RUN([int main() { return 0; }],
>      ac_cv_opt_olimit_ok=yes,
>      ac_cv_opt_olimit_ok=no,
>      ac_cv_opt_olimit_ok=no)
>    CC="$ac_save_cc"])
>    AC_MSG_RESULT($ac_cv_opt_olimit_ok)
>
> Why do we care about that particular obscure compiler flag?  I'm trying to
> build Python 2.4.x with the Sun Studio 12 compiler, which appears to operate
> in the same broken fashion as icc.  It warns that it doesn't support the
> option but still exits with a zero status which makes configure incorrectly
> think it does support the flag.
>
> This same chunk of code seems to be in configure.in dating back to 1997 with
> this comment:
>
>    r8948 | guido | 1997-10-09 15:24:13 -0500 (Thu, 09 Oct 1997) | 2 lines
>
>    Don Beaudry's changes to support SGI_ABI on Irix 6.x.
>
> Can this check be ripped out?

Irix support has been removed in 3.0 so it can at least be tossed
there if the OS is the only reason to have the flag aroung.

-Brett

From glyph at divmod.com  Mon Oct 27 21:32:10 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Mon, 27 Oct 2008 20:32:10 -0000
Subject: [Python-Dev] is the 'path' argument to an
	importer's	find_module() just a hint?
In-Reply-To: <bbaeab100810271324u128e2748wf835aca6ce657c91@mail.gmail.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
	<49059CF4.2060906@gmail.com>
	<bbaeab100810271224l426bdd47g27c4febce104972b@mail.gmail.com>
	<20081027193115.6400.126590000.divmod.xquotient.409@weber.divmod.com>
	<bbaeab100810271324u128e2748wf835aca6ce657c91@mail.gmail.com>
Message-ID: <20081027203210.6400.1308806761.divmod.xquotient.417@weber.divmod.com>

On 08:24 pm, brett at python.org wrote:
>On Mon, Oct 27, 2008 at 12:31 PM,  <glyph at divmod.com> wrote:
>>On 07:24 pm, brett at python.org wrote:
>>>
>>>On Mon, Oct 27, 2008 at 3:50 AM, Nick Coghlan <ncoghlan at gmail.com> 
>>>wrote:

>>For what it's worth, Twisted uses the __path__ attribute to facilitate 
>>its
>>plugin mechanism.

>But how do you manipulate it for frozen packages currently? Importing
>a frozen package currently has a bad __path__ value::

>  >>> import __phello__; __phello__.__path__
>  '__phello__'
>
>Notice how that is not a list. Do you special-case frozen packages?

Oh.  Well, in that case, it's already broken, nevermind :).  I naively 
assumed that someone would have reported a bug in Twisted if it were 
different from i.e. zip importing.  I suppose it makes sense - plugins 
are used for "twistd" subcommands, and it's unlikely that you'd want to 
freeze "twistd"; few applications use Twisted plugins themselves yet.

I misunderstood your earlier description; I thought that this (the 
invalid __path__) was the desired change, not the current state of the 
world.

(Of course if you could *change* frozen packages to have a valid 
__path__ I'd appreciate that...)

From ncoghlan at gmail.com  Mon Oct 27 21:30:41 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 28 Oct 2008 06:30:41 +1000
Subject: [Python-Dev] is the 'path' argument to
 an	importer's	find_module() just a hint?
In-Reply-To: <20081027193115.6400.126590000.divmod.xquotient.409@weber.divmod.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>	<49059CF4.2060906@gmail.com>	<bbaeab100810271224l426bdd47g27c4febce104972b@mail.gmail.com>
	<20081027193115.6400.126590000.divmod.xquotient.409@weber.divmod.com>
Message-ID: <490624F1.7090202@gmail.com>

glyph at divmod.com wrote:
> On 07:24 pm, brett at python.org wrote:
>> On Mon, Oct 27, 2008 at 3:50 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> Brett Cannon wrote:
> 
>> Good enough for me. Then I am just going to ignore the 'path' argument
>> for frozen modules but use it to short-circuit imports for built-in
>> modules.
> 
> For what it's worth, Twisted uses the __path__ attribute to facilitate
> its plugin mechanism.  If this changes so that importing from frozen
> packages no longer honors __path__, then Twisted will no longer support
> plugins if the package to be plugged into is frozen.
> 
> What is the motivation to change this?

If I had to guess, I'd say Brett found some time to tinker with his
import_in_py implementation. That has been an interesting exercise in
blowing dust out of some of the dark corners of CPython's current import
*implementation* (particularly in areas that stray outside of the
current documentation and PEP 302), and it isn't always clear which
behaviour can be safely skipped and which needs to be faithfully
replicated to avoid breaking 3rd party utilities.

If the current system is setting __path__ to a string in frozen
packages, I'd have to wonder how well any existing __path__ manipulation
tools handle frozen packages without special-casing them.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From brett at python.org  Mon Oct 27 21:44:29 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 27 Oct 2008 13:44:29 -0700
Subject: [Python-Dev] is the 'path' argument to an importer's
	find_module() just a hint?
In-Reply-To: <490624F1.7090202@gmail.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
	<49059CF4.2060906@gmail.com>
	<bbaeab100810271224l426bdd47g27c4febce104972b@mail.gmail.com>
	<20081027193115.6400.126590000.divmod.xquotient.409@weber.divmod.com>
	<490624F1.7090202@gmail.com>
Message-ID: <bbaeab100810271344m5b1c9deci1d462c93f9da9963@mail.gmail.com>

On Mon, Oct 27, 2008 at 1:30 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> glyph at divmod.com wrote:
>> On 07:24 pm, brett at python.org wrote:
>>> On Mon, Oct 27, 2008 at 3:50 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>>> Brett Cannon wrote:
>>
>>> Good enough for me. Then I am just going to ignore the 'path' argument
>>> for frozen modules but use it to short-circuit imports for built-in
>>> modules.
>>
>> For what it's worth, Twisted uses the __path__ attribute to facilitate
>> its plugin mechanism.  If this changes so that importing from frozen
>> packages no longer honors __path__, then Twisted will no longer support
>> plugins if the package to be plugged into is frozen.
>>
>> What is the motivation to change this?
>
> If I had to guess, I'd say Brett found some time to tinker with his
> import_in_py implementation.

Yep, it's called trying to prevent it from becoming true vaporware
while procrastinating. =)

> That has been an interesting exercise in
> blowing dust out of some of the dark corners of CPython's current import
> *implementation* (particularly in areas that stray outside of the
> current documentation and PEP 302), and it isn't always clear which
> behaviour can be safely skipped and which needs to be faithfully
> replicated to avoid breaking 3rd party utilities.
>

You can say that again. Once I have a backwards-compatible version
ready to be merged into the core I will have a PEP or two to write
that will help make the semantics more uniform and easier to follow.

> If the current system is setting __path__ to a string in frozen
> packages, I'd have to wonder how well any existing __path__ manipulation
> tools handle frozen packages without special-casing them.
>

I doubt anyone does. As Thomas asked, do people really even still use
frozen modules?

-Brett

From ncoghlan at gmail.com  Mon Oct 27 21:48:05 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 28 Oct 2008 06:48:05 +1000
Subject: [Python-Dev] is the 'path' argument to an
 importer's	find_module() just a hint?
In-Reply-To: <bbaeab100810271326g2f12b75bh26817206b0fe40bf@mail.gmail.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>	<ge56ot$ajh$2@ger.gmane.org>
	<bbaeab100810271326g2f12b75bh26817206b0fe40bf@mail.gmail.com>
Message-ID: <49062905.4010003@gmail.com>

Brett Cannon wrote:
> Beats me. At this point I am only implementing them for
> backwards-compatibility, but they are exposing some inconsistencies in
> how various aspects of import are implemented. If I had my way they
> would be an optional thing that you would have to explicitly put on
> sys.meta_path to avoid the overhead.

Implementing it that way for your import_in_py project would probably be
a good thing :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From glyph at divmod.com  Mon Oct 27 21:53:36 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Mon, 27 Oct 2008 20:53:36 -0000
Subject: [Python-Dev] is the 'path' argument to an
	importer's	find_module() just a hint?
Message-ID: <20081027205336.6400.1837682473.divmod.xquotient.443@weber.divmod.com>

On 08:44 pm, brett at python.org wrote:
>On Mon, Oct 27, 2008 at 1:30 PM, Nick Coghlan <ncoghlan at gmail.com> 
>wrote:

>>If the current system is setting __path__ to a string in frozen
>>packages, I'd have to wonder how well any existing __path__ 
>>manipulation
>>tools handle frozen packages without special-casing them.

>I doubt anyone does. As Thomas asked, do people really even still use
>frozen modules?

Well, speaking of vaporware, I know that when Divmod is discussing 
proprietary Python software with potential clients, Freeze often comes 
up.  We tend to discourage its actual use, but the *possibility* of 
jamming all the distributed Python modules into a shared library or 
executable and never loading them directly from the filesystem has been 
an important selling point of Python in the past.

I do know that a few commercial games have shipped with a big pile of 
frozen Python inside them; I don't know when the most recent one was.

From torne-pythondev at wolfpuppy.org.uk  Tue Oct 28 12:04:48 2008
From: torne-pythondev at wolfpuppy.org.uk (Torne Wuff)
Date: Tue, 28 Oct 2008 11:04:48 +0000
Subject: [Python-Dev] is the 'path' argument to an
	importer's	find_module() just a hint?
In-Reply-To: <bbaeab100810271344m5b1c9deci1d462c93f9da9963@mail.gmail.com>
References: <bbaeab100810262013k39fa3f4am7d9e0d514e1938e4@mail.gmail.com>
	<49059CF4.2060906@gmail.com>
	<bbaeab100810271224l426bdd47g27c4febce104972b@mail.gmail.com>
	<20081027193115.6400.126590000.divmod.xquotient.409@weber.divmod.com>
	<490624F1.7090202@gmail.com>
	<bbaeab100810271344m5b1c9deci1d462c93f9da9963@mail.gmail.com>
Message-ID: <20081028110448.GA14823@wolfpuppy.org.uk>

On Mon, Oct 27 08 at  1:44:29PM -0700, Brett Cannon wrote:
> I doubt anyone does. As Thomas asked, do people really even still use
> frozen modules?

I use frozen modules to load a filesystem driver into my bare-metal
Python port (since there are no FS syscalls and thus I can't import
the first python code from anywhere else). This may be something of a
special case. :)

I don't use __path__, though..

-- 
Torne Wuff
torne at wolfpuppy.org.uk

From guido at python.org  Tue Oct 28 23:56:03 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 28 Oct 2008 15:56:03 -0700
Subject: [Python-Dev] Has python-dev collapsed?
Message-ID: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>

Or have I been removed from the list? I'm not getting any mail it
seems. Or at least very little. Or are we all just tired of trying to
fix the release blockers in 3.0?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From lists at cheimes.de  Wed Oct 29 00:00:19 2008
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 29 Oct 2008 00:00:19 +0100
Subject: [Python-Dev] Has python-dev collapsed?
In-Reply-To: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>
References: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>
Message-ID: <ge85i3$2lj$1@ger.gmane.org>

Guido van Rossum wrote:
> Or have I been removed from the list? I'm not getting any mail it
> seems. Or at least very little. Or are we all just tired of trying to
> fix the release blockers in 3.0?

Don't worry yourself. You are still on the list and we are still 
fighting release blockers. But development has slowed down a lot in the 
past days.

But I've good news for you! Most release blockers have reviewed patches 
and are waiting for Barry. As a bonus Skip, Jesse and I have finished 
the first release of multiprocessing for 2.4 and 2.5. Yesterday I 
released the first version including Windows binaries.

Christian


From guido at python.org  Wed Oct 29 00:17:54 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 28 Oct 2008 16:17:54 -0700
Subject: [Python-Dev] Has python-dev collapsed?
In-Reply-To: <ge85i3$2lj$1@ger.gmane.org>
References: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>
	<ge85i3$2lj$1@ger.gmane.org>
Message-ID: <ca471dc20810281617x2cbc87a1t4c76a6262a07bb1d@mail.gmail.com>

On Tue, Oct 28, 2008 at 4:00 PM, Christian Heimes <lists at cheimes.de> wrote:
> Guido van Rossum wrote:
>>
>> Or have I been removed from the list? I'm not getting any mail it
>> seems. Or at least very little. Or are we all just tired of trying to
>> fix the release blockers in 3.0?
>
> Don't worry yourself. You are still on the list and we are still fighting
> release blockers. But development has slowed down a lot in the past days.

Yes, that's what it feels like. I hope the pace picks up again and we
can release 3.0 final in early December still. I really don't want to
carry it over to 2009.

> But I've good news for you! Most release blockers have reviewed patches and
> are waiting for Barry. As a bonus Skip, Jesse and I have finished the first
> release of multiprocessing for 2.4 and 2.5. Yesterday I released the first
> version including Windows binaries.

Excellent!!

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From lists at cheimes.de  Wed Oct 29 00:26:48 2008
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 29 Oct 2008 00:26:48 +0100
Subject: [Python-Dev] Has python-dev collapsed?
In-Reply-To: <ca471dc20810281617x2cbc87a1t4c76a6262a07bb1d@mail.gmail.com>
References: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>	
	<ge85i3$2lj$1@ger.gmane.org>
	<ca471dc20810281617x2cbc87a1t4c76a6262a07bb1d@mail.gmail.com>
Message-ID: <49079FB8.2000608@cheimes.de>

Guido van Rossum wrote:
> Yes, that's what it feels like. I hope the pace picks up again and we
> can release 3.0 final in early December still. I really don't want to
> carry it over to 2009.

Me, too. I don't want to check in fixes without explicit approval from 
Barry. The others are waiting for Barry as well.

Maybe we should select an assistant release manager for the next 
releases. It's lots of work to handle two releases at the same time. A 
single RM is also a single point of failure. You never know when 
personal affairs are more important than voluntary work.

Christian

From steve at holdenweb.com  Wed Oct 29 01:50:28 2008
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 28 Oct 2008 20:50:28 -0400
Subject: [Python-Dev] Has python-dev collapsed?
In-Reply-To: <49079FB8.2000608@cheimes.de>
References: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>		<ge85i3$2lj$1@ger.gmane.org>	<ca471dc20810281617x2cbc87a1t4c76a6262a07bb1d@mail.gmail.com>
	<49079FB8.2000608@cheimes.de>
Message-ID: <ge8c11$kv2$1@ger.gmane.org>

Christian Heimes wrote:
> Guido van Rossum wrote:
>> Yes, that's what it feels like. I hope the pace picks up again and we
>> can release 3.0 final in early December still. I really don't want to
>> carry it over to 2009.
> 
> Me, too. I don't want to check in fixes without explicit approval from
> Barry. The others are waiting for Barry as well.
> 
> Maybe we should select an assistant release manager for the next
> releases. It's lots of work to handle two releases at the same time. A
> single RM is also a single point of failure. You never know when
> personal affairs are more important than voluntary work.
> 
Pretty much whenever they come up ...

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From barry at python.org  Wed Oct 29 10:39:24 2008
From: barry at python.org (Barry Warsaw)
Date: Wed, 29 Oct 2008 09:39:24 +0000
Subject: [Python-Dev] Has python-dev collapsed?
In-Reply-To: <ca471dc20810281617x2cbc87a1t4c76a6262a07bb1d@mail.gmail.com>
References: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>
	<ge85i3$2lj$1@ger.gmane.org>
	<ca471dc20810281617x2cbc87a1t4c76a6262a07bb1d@mail.gmail.com>
Message-ID: <20081029093924.79cfd3d9@resist.wooz.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 28, 2008, at 04:17 PM, Guido van Rossum wrote:

>Yes, that's what it feels like. I hope the pace picks up again and we
>can release 3.0 final in early December still. I really don't want to
>carry it over to 2009.

Don't worry, it won't.  As I mentioned before, I've basically been unable to
work on any Python stuff these last two weeks.  I'll be back home on Saturday
so I can start looking at the 3.0 release blockers when I get back.  The
schedule still has us releasing Python 3.0 final on 3-Dec, and I'm hopeful we
can still meet that date.

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkkIL0wACgkQ2YZpQepbvXHxOQCdFUA4Q3frOjG2GX1ME0hc6uoZ
niwAn3deTj4E9KO+K4dMHWlKhKbjbaga
=0Py5
-----END PGP SIGNATURE-----

From amk at amk.ca  Wed Oct 29 22:14:54 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Wed, 29 Oct 2008 17:14:54 -0400
Subject: [Python-Dev] Has python-dev collapsed?
In-Reply-To: <49079FB8.2000608@cheimes.de>
References: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>
	<ge85i3$2lj$1@ger.gmane.org>
	<ca471dc20810281617x2cbc87a1t4c76a6262a07bb1d@mail.gmail.com>
	<49079FB8.2000608@cheimes.de>
Message-ID: <20081029211454.GA22943@amk-desktop.matrixgroup.net>

On Wed, Oct 29, 2008 at 12:26:48AM +0100, Christian Heimes wrote:
> Maybe we should select an assistant release manager for the next  
> releases. It's lots of work to handle two releases at the same time. A  

Will 3.1 and 2.7 also be parallel releases?  (I ask, not having read
the 3xxx PEPS at all.)  

If yes, why?  While I can see a case for 2.6/3.0 being in sync -- new
features in 2.6 ease the transition to 3.0 -- I'd imagine that 3.1
would be better with a shorter cycle (6-9 months) because there are
more possible rough edges to clean up.  3.1 would likely include
bugfixes from the eventual 2.7, so 3.1 might also trigger 2.6.1, but I
don't think there's any harm if 3.1 contains features or incompatible
fixes that are unavailable to 2.x users.

--amk

From glyph at divmod.com  Thu Oct 30 03:22:50 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 30 Oct 2008 02:22:50 -0000
Subject: [Python-Dev] Has python-dev collapsed?
In-Reply-To: <20081029211454.GA22943@amk-desktop.matrixgroup.net>
References: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>
	<ge85i3$2lj$1@ger.gmane.org>
	<ca471dc20810281617x2cbc87a1t4c76a6262a07bb1d@mail.gmail.com>
	<49079FB8.2000608@cheimes.de>
	<20081029211454.GA22943@amk-desktop.matrixgroup.net>
Message-ID: <20081030022250.6400.1062036738.divmod.xquotient.531@weber.divmod.com>


On 29 Oct, 09:14 pm, amk at amk.ca wrote:
>Will 3.1 and 2.7 also be parallel releases?  (I ask, not having read
>the 3xxx PEPS at all.)
>
>If yes, why?  While I can see a case for 2.6/3.0 being in sync -- new
>features in 2.6 ease the transition to 3.0 -- I'd imagine that 3.1
>would be better with a shorter cycle (6-9 months) because there are
>more possible rough edges to clean up.  3.1 would likely include
>bugfixes from the eventual 2.7, so 3.1 might also trigger 2.6.1, but I
>don't think there's any harm if 3.1 contains features or incompatible
>fixes that are unavailable to 2.x users.

2.7 is where a new version of 2to3 will be distributed.  As people 
actually start trying to migrate real-world code, lots of issues are 
inevitably going to be discovered there.  From my perspective, 2.7 and 
3.1 need to be synced because 2.7 is the tool that will actually enable 
users of existing code to *use* 3.1.

And this isn't just about a release of the 2to3 tool itself, but fixes 
in the 2.x backports of various stdlib features to deal with the kinds 
of parity issues that users are only going to discover in large-scale, 
real-world testing.

From sonia_16 at live.com  Thu Oct 30 07:35:46 2008
From: sonia_16 at live.com (Sonia)
Date: Thu, 30 Oct 2008 11:35:46 +0500
Subject: [Python-Dev] Save file by using file() function and fileDialog()
Message-ID: <BLU114-W292590BAFF20C6405545BD81210@phx.gbl>


 
I m using fileDialog(self, message, path, filename, filter, aStyle(opotional)) to save the file. 
In my code, i used fileDialog to show save dialog box and when we click on save button, it should save that file which is of txt type (filter = '*.txt'). But when we click on save button it returns error: "coercing to unicode: need string or buffer, list found" rather than saving / creating a file. I also tried f = file(myfile, 'w', 1000) instead of f = file(myfile, 'w'), where 1000 is buffer size, but both have the same error. where as if we use 




file(
filename[, mode[, bufsize]])
, then it creates a file of specified type. But it is not working in fileDialog(). Why?Where i m wrong?
 
What should i do?
 
Following is my code:
def on_btn2_mouseClick(self, event):        aStyle = wx.SAVE|wx.HIDE_READONLY|wx.OVERWRITE_PROMPT        filter = '*.txt'
        result = dialog.fileDialog(self, 'ist def', '', 'n', wildcard, aStyle)        myfile= result.paths        if result.accepted == True :            f = file(myfile, 'w')            f.write('my file')            f.close()
_________________________________________________________________
Connect to the next generation of MSN Messenger?
http://imagine-msn.com/messenger/launch80/default.aspx?locale=en-us&source=wlmailtagline
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20081030/b503b75f/attachment.htm>

From amauryfa at gmail.com  Thu Oct 30 10:02:50 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 30 Oct 2008 10:02:50 +0100
Subject: [Python-Dev] Save file by using file() function and fileDialog()
In-Reply-To: <BLU114-W292590BAFF20C6405545BD81210@phx.gbl>
References: <BLU114-W292590BAFF20C6405545BD81210@phx.gbl>
Message-ID: <e27efe130810300202j566a656s8ebb426aaeecd3e0@mail.gmail.com>

Hello,

On Thu, Oct 30, 2008 at 07:35, Sonia <sonia_16 at live.com> wrote:
>
> I m using fileDialog(self, message, path, filename, filter,
> aStyle(opotional)) to save the file.
> In my code, i used fileDialog to show save dialog box and when we click on
> save button, it should save that file which is of txt type (filter =
> '*.txt'). But when we click on save button it returns error: "coercing to
> unicode: need string or buffer, list found" rather than saving / creating a
> file. I also tried f = file(myfile, 'w', 1000) instead of f = file(myfile,
> 'w'), where 1000 is buffer size, but both have the same error. where as if
> we use
> file( filename[, mode[, bufsize]])
> , then it creates a file of specified type. But it is not working in
> fileDialog(). Why?Where i m wrong?
>
> What should i do?
>
> Following is my code:
> def on_btn2_mouseClick(self, event):
>         aStyle = wx.SAVE|wx.HIDE_READONLY|wx.OVERWRITE_PROMPT
>         filter = '*.txt'
>         result = dialog.fileDialog(self, 'ist def', '', 'n', wildcard,
> aStyle)
>         myfile= result.paths
>         if result.accepted == True :
>             f = file(myfile, 'w')
>             f.write('my file')
>             f.close()

The python-dev mailing list is about the development of Python, not
the development with Python.

Please send this kind of questions to the comp.lang.python newsgroup,
or the wxPython-users mailing list.
There are many python users there willing to help.

Furthermore wxPython is not developed here (or is it PythonCard?), so
you'll get little expertise from us.

(a hint however: "result.paths" likely returns a list of paths; try
"result.paths[0]" to get a file name)
-- 
Amaury Forgeot d'Arc

From victor.stinner at haypocalc.com  Thu Oct 30 11:08:35 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Thu, 30 Oct 2008 11:08:35 +0100
Subject: [Python-Dev] My patches
Message-ID: <200810301108.35591.victor.stinner@haypocalc.com>

Hi,

Since some months, I'm trying to improve Python but it's difficult because I'm 
not allowed to push patches and I have to wait for some reviews and then for 
someone interrested by my patches. Sometimes I just get a good reaction 
like "nice patch" and then nothing. Should I stop sending new patches and 
work on another project, improve old patches and send an email everydays to 
the mailing list to get some reaction?

Barry doesn't want to release Python 3.0rc2 because of release blocker issues. 
Guido asked if the mailing list has collapsed.

I don't understand: you want help, but some of my patches are waiting since 
many weeks ago... The problem is not just about me, but about anyone trying 
to contribute to Python: if it's to hard to contribute, people would just 
move to another more reactive project.

---

Well, here is a short list of my patches waiting to be commited:

http://bugs.python.org/issue3727 (fix poplib for python3)
 + patch posted 13 days ago

http://bugs.python.org/issue1210 (fix imaplib for python3)
 + patch posted 16 days ago

http://bugs.python.org/issue4036 (support bytes for subprocess.Popen)
 + patch posted 22 days ago

http://bugs.python.org/issue4021 (improve tokenize.detect_encoding)
 + patch posted 28 days ago
 + reviewed by amaury

http://bugs.python.org/issue4008 (IDLE: unicode fix checksyntax())
 + patch posted 28 days ago

http://bugs.python.org/issue3954 (fix _hotshot.logreader)
 + patch posted 39 days ago
 + reviewed by amaury and georg.brandl

http://bugs.python.org/issue3880 (fix _tkinter._flatten)
 + patch posted 44 days ago
 + reviewed by gpolo

http://bugs.python.org/issue3632 (ensure GIL in _PyObject_Dump)
 + patch posted 70 days ago
 + reviewed by amaury

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From barry at python.org  Thu Oct 30 12:04:42 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 30 Oct 2008 11:04:42 +0000
Subject: [Python-Dev] My patches
In-Reply-To: <200810301108.35591.victor.stinner@haypocalc.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
Message-ID: <20081030110442.1b6ce78e@resist.wooz.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 30, 2008, at 11:08 AM, Victor Stinner wrote:

>Since some months, I'm trying to improve Python but it's difficult because
>I'm not allowed to push patches and I have to wait for some reviews and then
>for someone interrested by my patches. Sometimes I just get a good reaction
>like "nice patch" and then nothing. Should I stop sending new patches and
>work on another project, improve old patches and send an email everydays to
>the mailing list to get some reaction?
>
>Barry doesn't want to release Python 3.0rc2 because of release blocker
>issues. Guido asked if the mailing list has collapsed.
>
>I don't understand: you want help, but some of my patches are waiting since 
>many weeks ago... The problem is not just about me, but about anyone trying 
>to contribute to Python: if it's to hard to contribute, people would just 
>move to another more reactive project.

Victor, don't despair!  Your contributions are appreciated.

Let me remind you though that I've been mostly unavailable for the past two
weeks at a work conference.  I won't have time to look at anything until
Monday at the earliest.  That's why I set the 3.0 schedule the way I did.

One of the reasons why I'm very keen on us moving to a distributed version
control system is to help break the logjam on core developers.  True, your
code will still not be able to land in the "official" branch without core
developer intervention, but you will be able to share your code, fixes,
branches with everyone in a much more live way than patches in a tracker.  It
will be much easier for others to merge your changes, test them, review them,
etc. and it will be much easier for you to track changes in the official trunk
as other code lands.

In any case, I know it's frustrating not to get good timely feedback, but such
is the nature of open source projects.  Please be patient and don't worry.
I'll look at your patches when I'm back in the real world.

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkkJlMoACgkQ2YZpQepbvXG0DgCePlFiKrAH/UQkQhLb8Mr7dwtd
mmUAoJN2pOJN40fOQ0otMSgeVCt5sqLL
=Mnta
-----END PGP SIGNATURE-----

From victor.stinner at haypocalc.com  Thu Oct 30 13:02:02 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Thu, 30 Oct 2008 13:02:02 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <20081030110442.1b6ce78e@resist.wooz.org>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
Message-ID: <200810301302.02796.victor.stinner@haypocalc.com>

> Let me remind you though that I've been mostly unavailable for the past two
> weeks at a work conference.

Cool, you're back :-) But my email was not against you.

> That's why I set the 3.0 schedule the way I did.

Personnaly, I don't want to get python 3.0 final with some broken modules or 
some criticial problems. So it's a good thing to delay the release until bugs 
are fixed.

> One of the reasons why I'm very keen on us moving to a distributed version
> control system is to help break the logjam on core developers.

Yeah, exactly :-) Does anyone already maintain a distributed tree? Mercurial, 
GIT, anything else? I tried Mercurial which is nice (at least some small 
project). But I think that GIT is the fatest and most robust system.

> you will be able to share your code, fixes, branches with everyone 
> in a much more live way than patches in a tracker.

Right and it's very difficult to manage patches using the tracker. After 
writing the patch, I have to revert all patches to be able to write a new 
patch because it's easier to generate a patch in this way. But some patches 
depend on other patches :-/

> In any case, I know it's frustrating not to get good timely feedback

A friend told me that his patch took 6 months to be applied :-/ (don't know 
which one)

If Python would be more reactive, more developer will be attracted. The 
communication is very important in an open source project. I contributed to 
many many projects, and I can say that Python is already one of the most 
reactive project! But it can be better ;-)

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From eckhardt at satorlaser.com  Thu Oct 30 13:19:15 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Thu, 30 Oct 2008 13:19:15 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <200810301302.02796.victor.stinner@haypocalc.com>
References: <200810301108.35591.victor.stinner@haypocalc.com> 
	<20081030110442.1b6ce78e@resist.wooz.org> 
	<200810301302.02796.victor.stinner@haypocalc.com>
Message-ID: <200810301319.15760.eckhardt@satorlaser.com>

On Thursday 30 October 2008, Victor Stinner wrote:
> > One of the reasons why I'm very keen on us moving to a distributed
> > version control system is to help break the logjam on core developers.
>
> Yeah, exactly :-) Does anyone already maintain a distributed tree?
> Mercurial, GIT, anything else?

Bazaar. Take a look at the developers' pages on python.org, they mention that 
a BZR checkout is available. I know that it works (though the initial 
checkout is glacially slow) but I don't know what "official" support it has 
or what is planned with it.

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From barry at python.org  Thu Oct 30 13:50:15 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 30 Oct 2008 12:50:15 +0000
Subject: [Python-Dev] My patches
In-Reply-To: <200810301302.02796.victor.stinner@haypocalc.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<200810301302.02796.victor.stinner@haypocalc.com>
Message-ID: <20081030125015.35b6d86e@resist.wooz.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 30, 2008, at 01:02 PM, Victor Stinner wrote:

>> Let me remind you though that I've been mostly unavailable for the past two
>> weeks at a work conference.
>
>Cool, you're back :-) But my email was not against you.

Well, not quite. :)  Monday.

>> One of the reasons why I'm very keen on us moving to a distributed version
>> control system is to help break the logjam on core developers.
>
>Yeah, exactly :-) Does anyone already maintain a distributed tree? Mercurial, 
>GIT, anything else? I tried Mercurial which is nice (at least some small 
>project). But I think that GIT is the fatest and most robust system.

We have experimental Bazaar and Mercurial mirrors of the various branches.
I've just put up a loggerhead (think ViewCVS for Bazaar) instance on
code.python.org.  http://code.python.org.  I haven't installed pysqlite yet on
that box, but when I do, it will be much more responsive.  (I also haven't
installed the init.d script yet so if it dies, let me know and I'll restart
it.)

Please note that these are *expermental*.  We have not officially chosen a
dvcs, or even decided that we're moving to one.  Brett as the head of the
infrastructure committee will have more to say about that.

>Right and it's very difficult to manage patches using the tracker. After 
>writing the patch, I have to revert all patches to be able to write a new 
>patch because it's easier to generate a patch in this way. But some patches 
>depend on other patches :-/

I think you will like using Bazaar^H^H^H^H^H^Ha dvcs :).

>If Python would be more reactive, more developer will be attracted. The 
>communication is very important in an open source project. I contributed to 
>many many projects, and I can say that Python is already one of the most 
>reactive project! But it can be better ;-)

I agree!  How can we improve our development process, given that we're an all
volunteer organization?

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkkJrYcACgkQ2YZpQepbvXGrgQCeIQV+n81qpZw/CaG3kzdCVYyU
RzYAn2lxqzwZ5QwBw38RkyNsix++h9iI
=O9T4
-----END PGP SIGNATURE-----

From ncoghlan at gmail.com  Thu Oct 30 14:14:27 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 30 Oct 2008 23:14:27 +1000
Subject: [Python-Dev] My patches
In-Reply-To: <200810301319.15760.eckhardt@satorlaser.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<200810301302.02796.victor.stinner@haypocalc.com>
	<200810301319.15760.eckhardt@satorlaser.com>
Message-ID: <4909B333.1090907@gmail.com>

Ulrich Eckhardt wrote:
> On Thursday 30 October 2008, Victor Stinner wrote:
>>> One of the reasons why I'm very keen on us moving to a distributed
>>> version control system is to help break the logjam on core developers.
>> Yeah, exactly :-) Does anyone already maintain a distributed tree?
>> Mercurial, GIT, anything else?
> 
> Bazaar. Take a look at the developers' pages on python.org, they mention that 
> a BZR checkout is available. I know that it works (though the initial 
> checkout is glacially slow) but I don't know what "official" support it has 
> or what is planned with it.

It's kept up to date, and will eventually move to a more complete DVCS
experiment (there are also mercurial and git mirrors being maintained,
but they haven't been linked from python.org yet - a trawl through the
python-dev archives should turn up the links to the URLs).

The PSF's infrastructure committee isn't that big though (and all
volunteers), and switching version control systems isn't exactly easy
(even the migration from Sourceforge CVS to python.org SVN took quite a
bit of effort from key people). The migration of all our regular
workflows from the familiar centralised VCS style to a DVCS style of
development promises to be pretty disruptive in the short term, no
matter how beneficial it will be in the long run.

That said, with the tracker migration from Sourceforge to Roundup behind
us, and a hopefully successful 3.0 release not too far away, it's
probably time to start giving the idea more serious thought.

Ultimately, any complete plan for migration from SVN to a DVCS will
likely need to come in the form of a meta-PEP like the one MvL wrote to
justify and document the migration from CVS to SVN:
http://www.python.org/dev/peps/pep-0347/

Unlike PEP 347 (where SVN was purpose built as a "better CVS", thus
making the overall migration path and workflow updates reasonably
straightforward), a PEP for migrating to a DVCS would need to provide
justification not only for moving to a DVCS in general, but also for the
choice of a particular DVCS. Similar to the exercise in selecting
Roundup, part of that would not only be the features of the tool itself,
but also the available volunteer expertise to maintain an instance of it
on python.org.

One thing that such a PEP could probably also use is additional feedback
from folks outside the core dev team on how a DVCS would benefit them
(since the core devs are the ones least affected by the limitations of a
centralised VCS - although having something better than svnmerge to
handle maintenance of multiple branches would be a very good thing for
us too!).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Thu Oct 30 14:20:39 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 30 Oct 2008 23:20:39 +1000
Subject: [Python-Dev] My patches
In-Reply-To: <20081030125015.35b6d86e@resist.wooz.org>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<20081030110442.1b6ce78e@resist.wooz.org>	<200810301302.02796.victor.stinner@haypocalc.com>
	<20081030125015.35b6d86e@resist.wooz.org>
Message-ID: <4909B4A7.9020002@gmail.com>

Barry Warsaw wrote:
> or even decided that we're moving to one.  Brett as the head of the
> infrastructure committee will have more to say about that.

While it is indeed the infrastructure committee's call (since they'll
shoulder the bulk of the effort in organising further investigation into
the idea), I personally believe that moving to some kind of DVCS makes
too much sense for it not to happen eventually.

Anything we can do to make it easier to maintain 4 active branches
(2.x-1 and 3.y-1 maintenance, 2.x and 3.y development) over the next few
years is going to be worth a fair amount of up front effort.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From lists at cheimes.de  Thu Oct 30 15:20:20 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 30 Oct 2008 15:20:20 +0100
Subject: [Python-Dev] Proper initialization of structs
Message-ID: <gecfr4$5p1$1@ger.gmane.org>

I like to raise attention for a problem revealed by 
http://bugs.python.org/issue4237

---
The bug was caused by a design flaw -- which was partly my fault. Some
elements of the PyFileIOObject struct were initialized in __new__ while
other parts were initialized in __init__. I've moved the initialization
to __new__.

We should add a rule that all struct members must be properly
initialized in __new__. In the past Victor's fuzzying tool has revealed
several crashers related to similar design flaws.

I'm raising the severity of the bug to release blocker because I can't
predict if the problem can be abused to crash the interpreter. We should
also review all __new__ and __init__ methods of objects and extension
modules for similar issues.
---

The same design flaw was responsible for bugs like the pickle crasher 
http://bugs.python.org/issue3664. I like to establish a rule that *all* 
struct members must be initialized properly in the type's tp_new function.

Comments?

Christian


From eric at trueblade.com  Thu Oct 30 15:22:07 2008
From: eric at trueblade.com (Eric Smith)
Date: Thu, 30 Oct 2008 10:22:07 -0400
Subject: [Python-Dev] My patches
In-Reply-To: <200810301319.15760.eckhardt@satorlaser.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<200810301302.02796.victor.stinner@haypocalc.com>
	<200810301319.15760.eckhardt@satorlaser.com>
Message-ID: <4909C30F.1070100@trueblade.com>

Ulrich Eckhardt wrote:
> On Thursday 30 October 2008, Victor Stinner wrote:
>>> One of the reasons why I'm very keen on us moving to a distributed
>>> version control system is to help break the logjam on core developers.
>> Yeah, exactly :-) Does anyone already maintain a distributed tree?
>> Mercurial, GIT, anything else?
> 
> Bazaar. Take a look at the developers' pages on python.org, they mention that 
> a BZR checkout is available. I know that it works (though the initial 
> checkout is glacially slow) but I don't know what "official" support it has 
> or what is planned with it.

I'd like to try it out, but the instructions on 
http://www.python.org/dev/bazaar/ say to get wget 
http://code.python.org/snapshots/python-bzr-snapshot.tar.bz2, which is a 
404. Anyone know the actual URL?

From amk at amk.ca  Thu Oct 30 16:04:27 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 30 Oct 2008 11:04:27 -0400
Subject: [Python-Dev] My patches
In-Reply-To: <20081030110442.1b6ce78e@resist.wooz.org>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
Message-ID: <20081030150427.GA9631@amk-desktop.matrixgroup.net>

On Thu, Oct 30, 2008 at 11:04:42AM +0000, Barry Warsaw wrote:
> One of the reasons why I'm very keen on us moving to a distributed version
> control system is to help break the logjam on core developers.  True, your
> code will still not be able to land in the "official" branch without core
> developer intervention, but you will be able to share your code, fixes,
> branches with everyone in a much more live way than patches in a tracker. 

I don't see how a DVCS will fix anything.  The bottleneck is in
assessing patches for inclusion in the master tree; not enough people
are doing that.  We'd just end up with lots of proposed branches
waiting to be merged, instead of patches to be applied.

(What a DVCS might enable is making it easier to do larger
experiments, like the recent Vmgen work, and publish them in a form
that people can download.  We could create SVN branches now, but that
means people would then have commit access to all of the Python
source.)

--amk

From arfrever.fta at gmail.com  Thu Oct 30 16:11:22 2008
From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis)
Date: Thu, 30 Oct 2008 16:11:22 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <20081030150427.GA9631@amk-desktop.matrixgroup.net>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
Message-ID: <71ac8a310810300811g3a241a85s85fdc48ffd3c8282@mail.gmail.com>

2008-10-30 16:04 A.M. Kuchling <amk at amk.ca> napisa?(a):
> On Thu, Oct 30, 2008 at 11:04:42AM +0000, Barry Warsaw wrote:
>> One of the reasons why I'm very keen on us moving to a distributed version
>> control system is to help break the logjam on core developers.  True, your
>> code will still not be able to land in the "official" branch without core
>> developer intervention, but you will be able to share your code, fixes,
>> branches with everyone in a much more live way than patches in a tracker.
>
> I don't see how a DVCS will fix anything.  The bottleneck is in
> assessing patches for inclusion in the master tree; not enough people
> are doing that.  We'd just end up with lots of proposed branches
> waiting to be merged, instead of patches to be applied.
>
> (What a DVCS might enable is making it easier to do larger
> experiments, like the recent Vmgen work, and publish them in a form
> that people can download.  We could create SVN branches now, but that
> means people would then have commit access to all of the Python
> source.)

SVN supports path-based authorization.
http://svnbook.red-bean.com/nightly/en/svn.serverconfig.pathbasedauthz.html

From fdrake at acm.org  Thu Oct 30 16:16:10 2008
From: fdrake at acm.org (Fred Drake)
Date: Thu, 30 Oct 2008 11:16:10 -0400
Subject: [Python-Dev] My patches
In-Reply-To: <71ac8a310810300811g3a241a85s85fdc48ffd3c8282@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<71ac8a310810300811g3a241a85s85fdc48ffd3c8282@mail.gmail.com>
Message-ID: <AA2F2910-5F5A-4763-A7AB-E901D15AB92C@acm.org>

On Oct 30, 2008, at 11:11 AM, Arfrever Frehtes Taifersar Arahesis wrote:
> SVN supports path-based authorization.
> http://svnbook.red-bean.com/nightly/en/svn.serverconfig.pathbasedauthz.html


This has worked well for us with contractors and partners, and isn't  
problematic or tedious to maintain.


   -Fred

-- 
Fred Drake   <fdrake at acm.org>


From steve at holdenweb.com  Thu Oct 30 16:20:55 2008
From: steve at holdenweb.com (Steve Holden)
Date: Thu, 30 Oct 2008 11:20:55 -0400
Subject: [Python-Dev] My patches
In-Reply-To: <71ac8a310810300811g3a241a85s85fdc48ffd3c8282@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<20081030110442.1b6ce78e@resist.wooz.org>	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<71ac8a310810300811g3a241a85s85fdc48ffd3c8282@mail.gmail.com>
Message-ID: <gecjcr$kd2$1@ger.gmane.org>

Arfrever Frehtes Taifersar Arahesis wrote:
> 2008-10-30 16:04 A.M. Kuchling <amk at amk.ca> napisa?(a):
>> On Thu, Oct 30, 2008 at 11:04:42AM +0000, Barry Warsaw wrote:
>>> One of the reasons why I'm very keen on us moving to a distributed version
>>> control system is to help break the logjam on core developers.  True, your
>>> code will still not be able to land in the "official" branch without core
>>> developer intervention, but you will be able to share your code, fixes,
>>> branches with everyone in a much more live way than patches in a tracker.
>> I don't see how a DVCS will fix anything.  The bottleneck is in
>> assessing patches for inclusion in the master tree; not enough people
>> are doing that.  We'd just end up with lots of proposed branches
>> waiting to be merged, instead of patches to be applied.
>>
>> (What a DVCS might enable is making it easier to do larger
>> experiments, like the recent Vmgen work, and publish them in a form
>> that people can download.  We could create SVN branches now, but that
>> means people would then have commit access to all of the Python
>> source.)
> 
> SVN supports path-based authorization.
> http://svnbook.red-bean.com/nightly/en/svn.serverconfig.pathbasedauthz.html

good point, but then we'd have an authentication management task ...

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From ironfroggy at gmail.com  Thu Oct 30 16:23:47 2008
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Thu, 30 Oct 2008 11:23:47 -0400
Subject: [Python-Dev] My patches
In-Reply-To: <20081030125015.35b6d86e@resist.wooz.org>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<200810301302.02796.victor.stinner@haypocalc.com>
	<20081030125015.35b6d86e@resist.wooz.org>
Message-ID: <76fd5acf0810300823x612b869bhe59304cf01cce822@mail.gmail.com>

On Thu, Oct 30, 2008 at 8:50 AM, Barry Warsaw <barry at python.org> wrote:
>
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Oct 30, 2008, at 01:02 PM, Victor Stinner wrote:
> >If Python would be more reactive, more developer will be attracted. The
> >communication is very important in an open source project. I contributed to
> >many many projects, and I can say that Python is already one of the most
> >reactive project! But it can be better ;-)
>
> I agree!  How can we improve our development process, given that we're an all
> volunteer organization?

On that note, are more volunteers needed? Are there any capacities in
which extra sets of hands could make help these improvements or would
it just be more overhead and managing people and figuring out what to
do?

Moderately related note, I'm going to start the python-dev list
summaries again (I have been the worst summary maintainer *ever*) and
I'd like to set up the process as a community effort. That is, I've
got some code I'm going to launch (probably at appengine) where edits
can be made to the summaries by anyone, to be reviewed and taken in,
before submitting the final summaries every two weeks. This should
make keeping them going more likely, because no one person will need
to find the time. It will also make the summaries more accurate if
people actually involved in the discussions can fix any mistakes or
repair omissions. I think the summaries are important, and one of many
puzzle pieces to keeping people informed, which is all part of moving
the development process along. Of course, this is all assuming no one
objects to that plan.

From p.f.moore at gmail.com  Thu Oct 30 16:55:38 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 30 Oct 2008 15:55:38 +0000
Subject: [Python-Dev] My patches
In-Reply-To: <20081030150427.GA9631@amk-desktop.matrixgroup.net>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
Message-ID: <79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>

2008/10/30 A.M. Kuchling <amk at amk.ca>:
> On Thu, Oct 30, 2008 at 11:04:42AM +0000, Barry Warsaw wrote:
>> One of the reasons why I'm very keen on us moving to a distributed version
>> control system is to help break the logjam on core developers.  True, your
>> code will still not be able to land in the "official" branch without core
>> developer intervention, but you will be able to share your code, fixes,
>> branches with everyone in a much more live way than patches in a tracker.
>
> I don't see how a DVCS will fix anything.  The bottleneck is in
> assessing patches for inclusion in the master tree; not enough people
> are doing that.  We'd just end up with lots of proposed branches
> waiting to be merged, instead of patches to be applied.

Agreed. There are lots of patches around, but not enough core dev
man-hours to review and apply them. As just adding extra people as
core devs isn't going to work (I don't believe it's *hard* to become a
core dev at the moment, it just needs a level of commitment that many
people can't offer), and as adding hours to the day isn't possible
(hmm, Guido - about that time machine?) I think the best way of
helping is with patch triage.

More people trawling the trackers and reviewing existing patches might
free up core dev time for looking at the more subtle stuff, as long as
(1) core devs could be happy to accept "this is OK, commit it"
comments from non-core devs, and/or (2) it's easy to locate stuff to
review, and just as importantly stuff which has been reviewed and is
"ready to go".

Question - is there anything Roundup can do to help triage? Extra
status or keyword values ("has patch", "ready to go", ...)? More
canned searches ("Show Open" and "Show Unassigned" aren't a lot of
help)? Custom reports (summaries by type)? Or are such things there
and simply not publicised enough?

I just did a quick experiment, checking for trivial documentation
patches I could review, and some things became obvious:

1. There is no way of telling which issues have a patch.
2. Some patches marked as "documentation" are doc fixes, others seem
to be issues where it has been decided that the behaviour is correct
as is, but needs to be documented. Fair enough, but it's much harder
to assess the latter, and there's no way of just grabbing the former
(for example, to spend a spare 30 minutes reviewing simple stuff).
3. There's nothing obvious I can do to move an issue forward. Sure, I
can make a comment, but that's about it. I'd like something that stood
a bit more chance of getting noticed (like a status change, or maybe a
list of people who think this is good to apply, which I can add myself
to).

All of which boil down to (1) quickly finding stuff I can deal with,
and (2) feeling like what I do has an effect.

Hmm, I've spent more time on this than I should have, and it's gone
way off topic. Is there anywhere better to discuss it?

Paul.

From draghuram at gmail.com  Thu Oct 30 17:02:14 2008
From: draghuram at gmail.com (Raghuram Devarakonda)
Date: Thu, 30 Oct 2008 12:02:14 -0400
Subject: [Python-Dev] My patches
In-Reply-To: <79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
Message-ID: <2c51ecee0810300902x49ed0061rb89ad8a332c3c2d2@mail.gmail.com>

On Thu, Oct 30, 2008 at 11:55 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> I just did a quick experiment, checking for trivial documentation
> patches I could review, and some things became obvious:
>
> 1. There is no way of telling which issues have a patch.

There is a "patch" keyword that is usually set for issues with patches
and search can be done for given keywords.

Thanks,
Raghu

From ziade.tarek at gmail.com  Thu Oct 30 17:09:00 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Thu, 30 Oct 2008 17:09:00 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
Message-ID: <94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>

On Thu, Oct 30, 2008 at 4:55 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>> I don't see how a DVCS will fix anything.  The bottleneck is in
>> assessing patches for inclusion in the master tree; not enough people
>> are doing that.  We'd just end up with lots of proposed branches
>> waiting to be merged, instead of patches to be applied.
>
> Agreed. There are lots of patches around, but not enough core dev
> man-hours to review and apply them. As just adding extra people as
> core devs isn't going to work (I don't believe it's *hard* to become a
> core dev at the moment, it just needs a level of commitment that many
> people can't offer), and as adding hours to the day isn't possible
> (hmm, Guido - about that time machine?) I think the best way of
> helping is with patch triage.
>

Since it is a hard and long process  "to know it all"  in Python, and
to become a core developer

What about having two level of devs ?

+ core developers
+ standard library developers

I mean, the standard library could be open ihmo to a wider range of people,
or maybe even having people specialized in some packages, modules, even
if they don't know anything about the C apis of the core.

Those "standard library developers" could be blessed to work on
specific areas of the standard
library and "followed" by a core developer that can just make sure
everything goes in the right direction
without having too much extra work for that.

Regards,
Tarek

-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/

From jnoller at gmail.com  Thu Oct 30 17:25:16 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 30 Oct 2008 12:25:16 -0400
Subject: [Python-Dev] My patches
In-Reply-To: <94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
	<94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
Message-ID: <4222a8490810300925x6ff186bdv68712fd4a294073e@mail.gmail.com>

On Thu, Oct 30, 2008 at 12:09 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
> On Thu, Oct 30, 2008 at 4:55 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>> I don't see how a DVCS will fix anything.  The bottleneck is in
>>> assessing patches for inclusion in the master tree; not enough people
>>> are doing that.  We'd just end up with lots of proposed branches
>>> waiting to be merged, instead of patches to be applied.
>>
>> Agreed. There are lots of patches around, but not enough core dev
>> man-hours to review and apply them. As just adding extra people as
>> core devs isn't going to work (I don't believe it's *hard* to become a
>> core dev at the moment, it just needs a level of commitment that many
>> people can't offer), and as adding hours to the day isn't possible
>> (hmm, Guido - about that time machine?) I think the best way of
>> helping is with patch triage.
>>
>
> Since it is a hard and long process  "to know it all"  in Python, and
> to become a core developer
>
> What about having two level of devs ?
>
> + core developers
> + standard library developers
>
> I mean, the standard library could be open ihmo to a wider range of people,
> or maybe even having people specialized in some packages, modules, even
> if they don't know anything about the C apis of the core.
>
> Those "standard library developers" could be blessed to work on
> specific areas of the standard
> library and "followed" by a core developer that can just make sure
> everything goes in the right direction
> without having too much extra work for that.
>
> Regards,
> Tarek

Interestingly enough, I consider myself in the "standard library
developers" RE: the multiprocessing package. I just thought that's how
things broke down unofficially. I personally don't feel comfortable
doing much of anything outside of my sandbox, but am more than willing
to commit patches that have been reviewed by people my senior (in
skill!).

-jesse

From stm at daimi.au.dk  Thu Oct 30 17:13:38 2008
From: stm at daimi.au.dk (Sigurd Torkel Meldgaard)
Date: Thu, 30 Oct 2008 17:13:38 +0100
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
Message-ID: <392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>

Hi to all Python developers

For a student project in a course on virtual machines, we are
evaluating the possibility to
experiment with removing the GIL from CPython

We have read the arguments against doing this at
http://www.python.org/doc/faq/library/#can-t-we-get-rid-of-the-global-interpreter-lock.

But we think it might be possible to do this with a different approach
than what has been tried till now.

The main reason for the necessity of the GIL is reference counting.

We believe that most of the slowdown in the free threading
implementation of Greg Stein was due to the need of atomic
refcounting, as this mail seems to confirm:
http://mail.python.org/pipermail/python-ideas/2007-April/000414.html

So we want to change CPython into having a "real" garbage collector -
removing all reference counting, and then the need for locks (or
atomic inc/dec ops) should be
highly alleviated.

Preferably the GC should be a high-performance one for instance a
generational one.

We believe that it can run quite a lot faster than ref-counting.

Shared datastructures would get their lock obviously.
Immutable objects (especially shared global objects, like True, False, Null)
would not.

Most of the interpreter structure would be per-thread, at that point.

We do not know how Greg Stein did his locking in the free threads
patch, but as a part of the course we learned there exists much faster
ways of locking than using OS-locks (faster for the uncontented case)
that are used in e.g. the HOT-SPOT java-compiler. This might make
"free threading" in python more attractive than some pessimists think.
(http://blogs.sun.com/dave/entry/biased_locking_in_hotspot)
In particular, we are talking about making the uncontended case go fast,
not about the independent part of stack-allocating the mutex
structure, which can only be done and is only needed in Java.

These ideas are similar to the ones used by Linux fast mutexes
(futexes), the implementation of mutexes in NPTL.

We have read this mail thread - so it seems that our idea surfaced,
but Greg didn't completely love it (he wanted to optimize refcounting
instead):
http://mail.python.org/pipermail/python-ideas/2007-April/000436.html

He was not totally negative however. His main objections are about:
- cache locality (He is in our opinion partially right, as seen in some
other paper time ago - any GC, copying GC in particular, doubles the
amount of used memory, so it's less cache-friendly). But still GCs are
overall competitive or faster than explicit management, and surely
much faster of refcounting.

We know it is the plan for PyPy to work in this way, and also that
Jython and Ironpython works like that (using the host vm's GC), so it
seems to be somehow agreeable with the python semantics (perhaps not
really with __del__ but they are not really nice anyway).

Was this ever tried for CPython?

Any other comments, encouragements or warnings on the project-idea?

Best regards: Paolo, Sigurd

From guido at python.org  Thu Oct 30 18:00:03 2008
From: guido at python.org (Guido van Rossum)
Date: Thu, 30 Oct 2008 10:00:03 -0700
Subject: [Python-Dev] Proper initialization of structs
In-Reply-To: <gecfr4$5p1$1@ger.gmane.org>
References: <gecfr4$5p1$1@ger.gmane.org>
Message-ID: <ca471dc20810301000o20430547u5490f5b182bbab0f@mail.gmail.com>

Yes, that's what __new__ / tp_new is for! Thanks for finding this.

On Thu, Oct 30, 2008 at 7:20 AM, Christian Heimes <lists at cheimes.de> wrote:
> I like to raise attention for a problem revealed by
> http://bugs.python.org/issue4237
>
> ---
> The bug was caused by a design flaw -- which was partly my fault. Some
> elements of the PyFileIOObject struct were initialized in __new__ while
> other parts were initialized in __init__. I've moved the initialization
> to __new__.
>
> We should add a rule that all struct members must be properly
> initialized in __new__. In the past Victor's fuzzying tool has revealed
> several crashers related to similar design flaws.
>
> I'm raising the severity of the bug to release blocker because I can't
> predict if the problem can be abused to crash the interpreter. We should
> also review all __new__ and __init__ methods of objects and extension
> modules for similar issues.
> ---
>
> The same design flaw was responsible for bugs like the pickle crasher
> http://bugs.python.org/issue3664. I like to establish a rule that *all*
> struct members must be initialized properly in the type's tp_new function.
>
> Comments?
>
> Christian
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From jnoller at gmail.com  Thu Oct 30 18:02:18 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Thu, 30 Oct 2008 13:02:18 -0400
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
Message-ID: <4222a8490810301002v792b9aa9nf2ea235322877f4a@mail.gmail.com>

On Thu, Oct 30, 2008 at 12:13 PM, Sigurd Torkel Meldgaard
<stm at daimi.au.dk> wrote:
> Hi to all Python developers
>
> For a student project in a course on virtual machines, we are
> evaluating the possibility to
> experiment with removing the GIL from CPython
>
> We have read the arguments against doing this at
> http://www.python.org/doc/faq/library/#can-t-we-get-rid-of-the-global-interpreter-lock.
>
> But we think it might be possible to do this with a different approach
> than what has been tried till now.
>
> The main reason for the necessity of the GIL is reference counting.
>
> We believe that most of the slowdown in the free threading
> implementation of Greg Stein was due to the need of atomic
> refcounting, as this mail seems to confirm:
> http://mail.python.org/pipermail/python-ideas/2007-April/000414.html
>
> So we want to change CPython into having a "real" garbage collector -
> removing all reference counting, and then the need for locks (or
> atomic inc/dec ops) should be
> highly alleviated.
>
> Preferably the GC should be a high-performance one for instance a
> generational one.
>
> We believe that it can run quite a lot faster than ref-counting.
>
> Shared datastructures would get their lock obviously.
> Immutable objects (especially shared global objects, like True, False, Null)
> would not.
>
> Most of the interpreter structure would be per-thread, at that point.
>
> We do not know how Greg Stein did his locking in the free threads
> patch, but as a part of the course we learned there exists much faster
> ways of locking than using OS-locks (faster for the uncontented case)
> that are used in e.g. the HOT-SPOT java-compiler. This might make
> "free threading" in python more attractive than some pessimists think.
> (http://blogs.sun.com/dave/entry/biased_locking_in_hotspot)
> In particular, we are talking about making the uncontended case go fast,
> not about the independent part of stack-allocating the mutex
> structure, which can only be done and is only needed in Java.
>
> These ideas are similar to the ones used by Linux fast mutexes
> (futexes), the implementation of mutexes in NPTL.
>
> We have read this mail thread - so it seems that our idea surfaced,
> but Greg didn't completely love it (he wanted to optimize refcounting
> instead):
> http://mail.python.org/pipermail/python-ideas/2007-April/000436.html
>
> He was not totally negative however. His main objections are about:
> - cache locality (He is in our opinion partially right, as seen in some
> other paper time ago - any GC, copying GC in particular, doubles the
> amount of used memory, so it's less cache-friendly). But still GCs are
> overall competitive or faster than explicit management, and surely
> much faster of refcounting.
>
> We know it is the plan for PyPy to work in this way, and also that
> Jython and Ironpython works like that (using the host vm's GC), so it
> seems to be somehow agreeable with the python semantics (perhaps not
> really with __del__ but they are not really nice anyway).
>
> Was this ever tried for CPython?
>
> Any other comments, encouragements or warnings on the project-idea?
>
> Best regards: Paolo, Sigurd

See also:
http://code.google.com/p/python-safethread/
https://launchpad.net/python-safethread

From guido at python.org  Thu Oct 30 18:06:12 2008
From: guido at python.org (Guido van Rossum)
Date: Thu, 30 Oct 2008 10:06:12 -0700
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
Message-ID: <ca471dc20810301006y7ee803b9nc24e7f5a28d83545@mail.gmail.com>

It's not that I have any love for the GIL, it just is the best
compromise I could find. I expect that you won't be able to do better,
but I wish you luck anyway.

On Thu, Oct 30, 2008 at 9:13 AM, Sigurd Torkel Meldgaard
<stm at daimi.au.dk> wrote:
> Hi to all Python developers
>
> For a student project in a course on virtual machines, we are
> evaluating the possibility to
> experiment with removing the GIL from CPython
>
> We have read the arguments against doing this at
> http://www.python.org/doc/faq/library/#can-t-we-get-rid-of-the-global-interpreter-lock.
>
> But we think it might be possible to do this with a different approach
> than what has been tried till now.
>
> The main reason for the necessity of the GIL is reference counting.
>
> We believe that most of the slowdown in the free threading
> implementation of Greg Stein was due to the need of atomic
> refcounting, as this mail seems to confirm:
> http://mail.python.org/pipermail/python-ideas/2007-April/000414.html
>
> So we want to change CPython into having a "real" garbage collector -
> removing all reference counting, and then the need for locks (or
> atomic inc/dec ops) should be
> highly alleviated.
>
> Preferably the GC should be a high-performance one for instance a
> generational one.
>
> We believe that it can run quite a lot faster than ref-counting.
>
> Shared datastructures would get their lock obviously.
> Immutable objects (especially shared global objects, like True, False, Null)
> would not.
>
> Most of the interpreter structure would be per-thread, at that point.
>
> We do not know how Greg Stein did his locking in the free threads
> patch, but as a part of the course we learned there exists much faster
> ways of locking than using OS-locks (faster for the uncontented case)
> that are used in e.g. the HOT-SPOT java-compiler. This might make
> "free threading" in python more attractive than some pessimists think.
> (http://blogs.sun.com/dave/entry/biased_locking_in_hotspot)
> In particular, we are talking about making the uncontended case go fast,
> not about the independent part of stack-allocating the mutex
> structure, which can only be done and is only needed in Java.
>
> These ideas are similar to the ones used by Linux fast mutexes
> (futexes), the implementation of mutexes in NPTL.
>
> We have read this mail thread - so it seems that our idea surfaced,
> but Greg didn't completely love it (he wanted to optimize refcounting
> instead):
> http://mail.python.org/pipermail/python-ideas/2007-April/000436.html
>
> He was not totally negative however. His main objections are about:
> - cache locality (He is in our opinion partially right, as seen in some
> other paper time ago - any GC, copying GC in particular, doubles the
> amount of used memory, so it's less cache-friendly). But still GCs are
> overall competitive or faster than explicit management, and surely
> much faster of refcounting.
>
> We know it is the plan for PyPy to work in this way, and also that
> Jython and Ironpython works like that (using the host vm's GC), so it
> seems to be somehow agreeable with the python semantics (perhaps not
> really with __del__ but they are not really nice anyway).
>
> Was this ever tried for CPython?
>
> Any other comments, encouragements or warnings on the project-idea?
>
> Best regards: Paolo, Sigurd
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From van.lindberg at gmail.com  Thu Oct 30 18:07:50 2008
From: van.lindberg at gmail.com (VanL)
Date: Thu, 30 Oct 2008 12:07:50 -0500
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
Message-ID: <gecpks$cgc$1@ger.gmane.org>

Sigurd Torkel Meldgaard wrote:
> Hi to all Python developers
> 
> For a student project in a course on virtual machines, we are
> evaluating the possibility to
> experiment with removing the GIL from CPython...

Just an FYI, these two particular students already introduced themselves
on the PyPy list. Paolo is a masters student with experience in the
Linux kernel; Sigurd is a PhD candidate.

Their professor is Lars Bak, the lead architect of the Google V8
Javascript engine. They spent some time working on V8 in the last couple
months.


From fdrake at acm.org  Thu Oct 30 18:00:35 2008
From: fdrake at acm.org (Fred Drake)
Date: Thu, 30 Oct 2008 13:00:35 -0400
Subject: [Python-Dev] Proper initialization of structs
In-Reply-To: <gecfr4$5p1$1@ger.gmane.org>
References: <gecfr4$5p1$1@ger.gmane.org>
Message-ID: <05D72249-2D59-47F2-BE26-FF9DC3541958@acm.org>

On Oct 30, 2008, at 10:20 AM, Christian Heimes wrote:
> I like to establish a rule that *all* struct members must be  
> initialized properly in the type's tp_new function.


I think this has always been a requirement.  The result of the "new"  
operation must conform to all the requirements that the type's C code  
demands.

It's good to move work into __init__ where reasonable, so that it can  
be avoided if a subclass wants it done in a completely different way,  
but new can't work that way.


   -Fred

-- 
Fred Drake   <fdrake at acm.org>


From van.lindberg at gmail.com  Thu Oct 30 18:20:39 2008
From: van.lindberg at gmail.com (VanL)
Date: Thu, 30 Oct 2008 12:20:39 -0500
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <gecpks$cgc$1@ger.gmane.org>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gecpks$cgc$1@ger.gmane.org>
Message-ID: <gecqd4$g9g$1@ger.gmane.org>

VanL wrote:

> Just an FYI, 

...and the FYI was to no one in particular, just interesting information
from the PyPy list. It is just unfortunate timing that made it look like
a cheeky reply to Guido.


From leif.walsh at gmail.com  Thu Oct 30 18:24:49 2008
From: leif.walsh at gmail.com (Leif Walsh)
Date: Thu, 30 Oct 2008 13:24:49 -0400
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <gecpks$cgc$1@ger.gmane.org>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gecpks$cgc$1@ger.gmane.org>
Message-ID: <cc7430500810301024w2429c515h2428edca4b243fda@mail.gmail.com>

On Thu, Oct 30, 2008 at 1:07 PM, VanL <van.lindberg at gmail.com> wrote:
> Just an FYI, these two particular students already introduced themselves
> on the PyPy list. Paolo is a masters student with experience in the
> Linux kernel; Sigurd is a PhD candidate.
>
> Their professor is Lars Bak, the lead architect of the Google V8
> Javascript engine. They spent some time working on V8 in the last couple
> months.

This just gives me really high expectations.  Go get 'em!

-- 
Cheers,
Leif

From guido at python.org  Thu Oct 30 18:25:35 2008
From: guido at python.org (Guido van Rossum)
Date: Thu, 30 Oct 2008 10:25:35 -0700
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <gecqd4$g9g$1@ger.gmane.org>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gecpks$cgc$1@ger.gmane.org> <gecqd4$g9g$1@ger.gmane.org>
Message-ID: <ca471dc20810301025p47843333yff585b8507d9d521@mail.gmail.com>

On Thu, Oct 30, 2008 at 10:20 AM, VanL <van.lindberg at gmail.com> wrote:
> VanL wrote:
>
>> Just an FYI,
>
> ...and the FYI was to no one in particular, just interesting information
> from the PyPy list. It is just unfortunate timing that made it look like
> a cheeky reply to Guido.

No offense taken. The V8 experience makes me feel much more optimistic
that they might actually pull this off. (I'm still skeptical about
support for extension modules, withougt which CPython is pretty lame.)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From eric at trueblade.com  Thu Oct 30 18:31:19 2008
From: eric at trueblade.com (Eric Smith)
Date: Thu, 30 Oct 2008 13:31:19 -0400
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <ca471dc20810301025p47843333yff585b8507d9d521@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>	<gecpks$cgc$1@ger.gmane.org>
	<gecqd4$g9g$1@ger.gmane.org>
	<ca471dc20810301025p47843333yff585b8507d9d521@mail.gmail.com>
Message-ID: <4909EF67.5010104@trueblade.com>

Guido van Rossum wrote:
> No offense taken. The V8 experience makes me feel much more optimistic
> that they might actually pull this off. (I'm still skeptical about
> support for extension modules, withougt which CPython is pretty lame.)

The need to modify all extension modules is the usual non-starter I see 
mentioned when this topic comes up. The OP really needs to think about 
that issue.



From guido at python.org  Thu Oct 30 18:34:19 2008
From: guido at python.org (Guido van Rossum)
Date: Thu, 30 Oct 2008 10:34:19 -0700
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <4909EF67.5010104@trueblade.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gecpks$cgc$1@ger.gmane.org> <gecqd4$g9g$1@ger.gmane.org>
	<ca471dc20810301025p47843333yff585b8507d9d521@mail.gmail.com>
	<4909EF67.5010104@trueblade.com>
Message-ID: <ca471dc20810301034x7d6adb88wb875100e648d5c28@mail.gmail.com>

On Thu, Oct 30, 2008 at 10:31 AM, Eric Smith <eric at trueblade.com> wrote:
> Guido van Rossum wrote:
>>
>> No offense taken. The V8 experience makes me feel much more optimistic
>> that they might actually pull this off. (I'm still skeptical about
>> support for extension modules, withougt which CPython is pretty lame.)
>
> The need to modify all extension modules is the usual non-starter I see
> mentioned when this topic comes up. The OP really needs to think about that
> issue.

It's a non-starter for immediate world-domination. But if they get the
core to be significantly faster I expect there will be motivation to
port extensions. There's also the PyPy effort to replace extension
modules with ctypes-based wrappers. I could also imagine that
extensions could be run in a sandbox that *does* use the equivalent of
the GIL.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From arigo at tunes.org  Thu Oct 30 18:16:20 2008
From: arigo at tunes.org (Armin Rigo)
Date: Thu, 30 Oct 2008 18:16:20 +0100
Subject: [Python-Dev] Classifying language vs. impl-detail tests
Message-ID: <20081030171620.GA18469@code0.codespeak.net>

Hi all,

Here is a first step towards classifying the Python test suite into
"real language tests" and "implementation details":

    http://bugs.python.org/issue4242

If the general approach seems acceptable to people, I would be willing
to port more of PyPy's test suite patches.  The net result would be to
move the test suite towards being more directly useful for alternate
Python implementations.  (Right now, all of them have some custom tests
plus their own similarly patched copy of the stdlib test suite.)

Note that the patch above is against 2.7 trunk; the 2.x line is what all
non-CPython implementations currently target.  The general idea (and to
a large extend the patches) can easily be forward-ported to 3.x when it
makes sense.

Also, the actual decision of what is a "real" or a "detail" test is of
course up to discussion.  The patch above does the classification for
test_descr.  It was obtained by running it with PyPy, and for each
failure either fixing PyPy, or argumenting (by adding a comment) the
reason for which it is a detail -- usually referring to "well-known
agreement" expressed on python-dev about the matter.  (Of course,
"detail" tests still run normally on top of CPython.)


A bientot,

Armin.

From brett at python.org  Thu Oct 30 19:30:21 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 30 Oct 2008 11:30:21 -0700
Subject: [Python-Dev] My patches
In-Reply-To: <4909B333.1090907@gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<200810301302.02796.victor.stinner@haypocalc.com>
	<200810301319.15760.eckhardt@satorlaser.com>
	<4909B333.1090907@gmail.com>
Message-ID: <bbaeab100810301130j45ce94f0j705245327eabbfb@mail.gmail.com>

On Thu, Oct 30, 2008 at 06:14, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Ulrich Eckhardt wrote:
>> On Thursday 30 October 2008, Victor Stinner wrote:
>>>> One of the reasons why I'm very keen on us moving to a distributed
>>>> version control system is to help break the logjam on core developers.
>>> Yeah, exactly :-) Does anyone already maintain a distributed tree?
>>> Mercurial, GIT, anything else?
>>
>> Bazaar. Take a look at the developers' pages on python.org, they mention that
>> a BZR checkout is available. I know that it works (though the initial
>> checkout is glacially slow) but I don't know what "official" support it has
>> or what is planned with it.
>
> It's kept up to date, and will eventually move to a more complete DVCS
> experiment (there are also mercurial and git mirrors being maintained,
> but they haven't been linked from python.org yet - a trawl through the
> python-dev archives should turn up the links to the URLs).
>
> The PSF's infrastructure committee isn't that big though (and all
> volunteers), and switching version control systems isn't exactly easy
> (even the migration from Sourceforge CVS to python.org SVN took quite a
> bit of effort from key people). The migration of all our regular
> workflows from the familiar centralised VCS style to a DVCS style of
> development promises to be pretty disruptive in the short term, no
> matter how beneficial it will be in the long run.
>
> That said, with the tracker migration from Sourceforge to Roundup behind
> us, and a hopefully successful 3.0 release not too far away, it's
> probably time to start giving the idea more serious thought.
>
> Ultimately, any complete plan for migration from SVN to a DVCS will
> likely need to come in the form of a meta-PEP like the one MvL wrote to
> justify and document the migration from CVS to SVN:
> http://www.python.org/dev/peps/pep-0347/
>

I have actually started such a PEP, so this is being worked on.

-Brett

From fuzzyman at voidspace.org.uk  Thu Oct 30 19:37:03 2008
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 30 Oct 2008 18:37:03 +0000
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <ca471dc20810301034x7d6adb88wb875100e648d5c28@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>	<gecpks$cgc$1@ger.gmane.org>
	<gecqd4$g9g$1@ger.gmane.org>	<ca471dc20810301025p47843333yff585b8507d9d521@mail.gmail.com>	<4909EF67.5010104@trueblade.com>
	<ca471dc20810301034x7d6adb88wb875100e648d5c28@mail.gmail.com>
Message-ID: <4909FECF.7000703@voidspace.org.uk>

Guido van Rossum wrote:
> On Thu, Oct 30, 2008 at 10:31 AM, Eric Smith <eric at trueblade.com> wrote:
>   
>> Guido van Rossum wrote:
>>     
>>> No offense taken. The V8 experience makes me feel much more optimistic
>>> that they might actually pull this off. (I'm still skeptical about
>>> support for extension modules, withougt which CPython is pretty lame.)
>>>       
>> The need to modify all extension modules is the usual non-starter I see
>> mentioned when this topic comes up. The OP really needs to think about that
>> issue.
>>     
>
> It's a non-starter for immediate world-domination. But if they get the
> core to be significantly faster I expect there will be motivation to
> port extensions. There's also the PyPy effort to replace extension
> modules with ctypes-based wrappers. I could also imagine that
> extensions could be run in a sandbox that *does* use the equivalent of
> the GIL.
>
>   
For IronClad [1], an implementation of the Python C API in C# to allow 
you to use Python C extensions from IronPython, we take a hybrid 
approach to garbage collection.

We hold a strong reference to objects created by extensions (preventing 
.NET from garbage collecting them) and allow the extension to manipulate 
the reference count in the usual way. Periodically we scan our "managed 
objects" [2] to see if there are objects whose reference count has 
dropped to zero and we then delete our strong reference - allowing 
garbage collection to happen normally.

We have also implemented a GIL for the C extensions even though 
IronPython has no GIL.

Simple extension modules can already be used with IronPython through 
Ironclad, and for Numpy (our goal) arrays can be created and used and we 
are working on 'everything else'.

Moving more C extensions to an implementation based on ctypes would be 
enormously useful for PyPy, IronPython and Jython, but ctypes is not yet 
as portable as Python itself which could be an issue (although one worth 
resolving).

Michael Foord


[1] http://code.google.com/p/ironclad
[2] Strictly speaking they are managed objects of unmanaged types, but 
we really need a better way of talking about them.

-- 
http://www.ironpythoninaction.com/


From brett at python.org  Thu Oct 30 19:38:08 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 30 Oct 2008 11:38:08 -0700
Subject: [Python-Dev] My patches
In-Reply-To: <20081030125015.35b6d86e@resist.wooz.org>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<200810301302.02796.victor.stinner@haypocalc.com>
	<20081030125015.35b6d86e@resist.wooz.org>
Message-ID: <bbaeab100810301138r5552a2e2x856c472eaea817b6@mail.gmail.com>

On Thu, Oct 30, 2008 at 05:50, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Oct 30, 2008, at 01:02 PM, Victor Stinner wrote:
[SNIP]
>>If Python would be more reactive, more developer will be attracted. The
>>communication is very important in an open source project. I contributed to
>>many many projects, and I can say that Python is already one of the most
>>reactive project! But it can be better ;-)
>
> I agree!  How can we improve our development process, given that we're an all
> volunteer organization?

And this is a key point that people seem to forget. Guido is the ONLY
developer who gets paid to work on Python, and that is only 50% of his
time (when he doesn't have a Google-related workload), and honestly I
would rather he not deal with patches that do not affect what he is
working on.

The rest of us spend our evenings and weekends on this on top of
trying to balance a normal life. In my case I had to stop my Python
work recently because I have been working on my Ph.D. thesis proposal.
Everyone has their lives that take priority.

And no, I do not think that giving out more commit privileges will
necessarily solve everything. Python is known for its quality and part
of that reason is we are careful about handing out privileges. This is
not to say we can't try to help move people along towards getting
privileges faster, but I am not interested in doing what Pugs does and
give everyone who has submitted a patch commit privs.

-Brett

From martin at v.loewis.de  Thu Oct 30 20:43:02 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 30 Oct 2008 20:43:02 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<20081030110442.1b6ce78e@resist.wooz.org>	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
Message-ID: <490A0E46.3070003@v.loewis.de>

> Question - is there anything Roundup can do to help triage? Extra
> status or keyword values ("has patch",

There is "patch" keyword already, and a public query "Patches"
(as well as "My Patches")

> "ready to go", ...)?

We could give more people the right to set the resolution to "Accepted".
This is a matter of trust, though - if the committer then still needs
to review it, anyway, nothing is gained.

> More
> canned searches ("Show Open" and "Show Unassigned" aren't a lot of
> help)?

Please go to the "edit" label next to "Search". You can store your
own searches, but you can also share searches with others.

> Custom reports (summaries by type)? 

This I don't understand - how is it different from a search?

> Or are such things there and simply not publicised enough?

Perhaps. I really don't know what percentage of interested users
is aware of roundup capabilities.

> I just did a quick experiment, checking for trivial documentation
> patches I could review, and some things became obvious:
> 
> 1. There is no way of telling which issues have a patch.

There is the patch keyword, and you can query for it. There is a
canned query published already.

> 2. Some patches marked as "documentation" are doc fixes, others seem
> to be issues where it has been decided that the behaviour is correct
> as is, but needs to be documented. Fair enough, but it's much harder
> to assess the latter, and there's no way of just grabbing the former
> (for example, to spend a spare 30 minutes reviewing simple stuff).

There is the "easy" keyword. Of course, it might also be useful to
triage more issues as "easy".

> 3. There's nothing obvious I can do to move an issue forward. Sure, I
> can make a comment, but that's about it. I'd like something that stood
> a bit more chance of getting noticed (like a status change, or maybe a
> list of people who think this is good to apply, which I can add myself
> to).

The "developer" role has more user interface. I've just given it to you.

> Hmm, I've spent more time on this than I should have, and it's gone
> way off topic. Is there anywhere better to discuss it?

There is the tracker-discuss list for discussion, and the meta tracker
for actual problems/wishes for the tracker.

Regards,
Martin

From martin at v.loewis.de  Thu Oct 30 20:46:18 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 30 Oct 2008 20:46:18 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<20081030110442.1b6ce78e@resist.wooz.org>	<20081030150427.GA9631@amk-desktop.matrixgroup.net>	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
	<94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
Message-ID: <490A0F0A.9070100@v.loewis.de>

> What about having two level of devs ?
> 
> + core developers
> + standard library developers

We effectively have that already. Many of the committers will
only ever commit to a single module (+docs and tests), as they
volunteered to maintain that very module (e.g. Lars Gust?bel
for the tarfile module).

If a committer agrees to restrict himself to a certain area,
he will usually keep that promise, so there isn't really any
need for a finer-grained access control, IMO.

Regards,
Martin

From martin at v.loewis.de  Thu Oct 30 20:48:23 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 30 Oct 2008 20:48:23 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <4222a8490810300925x6ff186bdv68712fd4a294073e@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<20081030110442.1b6ce78e@resist.wooz.org>	<20081030150427.GA9631@amk-desktop.matrixgroup.net>	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>	<94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
	<4222a8490810300925x6ff186bdv68712fd4a294073e@mail.gmail.com>
Message-ID: <490A0F87.2010703@v.loewis.de>

> Interestingly enough, I consider myself in the "standard library
> developers" RE: the multiprocessing package. I just thought that's how
> things broke down unofficially.

It's actually fairly official (see my other message) - "you know who you
are". It has been working that way fine for the last few years. The
public record if it is in Misc/developers.txt (although that file might
be slightly incomplete/incorrect).

Regards,
Martin

From theller at ctypes.org  Thu Oct 30 20:54:49 2008
From: theller at ctypes.org (Thomas Heller)
Date: Thu, 30 Oct 2008 20:54:49 +0100
Subject: [Python-Dev] svnmerge init in the release26-maint branch?
Message-ID: <ged3e8$hsk$1@ger.gmane.org>

I have a fix for a modulefinder crash that I'm going to commit
into the trunk. Since the bug is also in release26-maint I will
also backport it.  While preparing this I noticed that in the
release26-maint branch 'svnmerge init' has not yet been done.

I assume that we will use svnmerge to merge from trunk to
release26-maint?  So, is it ok to do 'svnmerge init' in it?

-- 
Thanks,
Thomas


From lists at cheimes.de  Thu Oct 30 21:01:54 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 30 Oct 2008 21:01:54 +0100
Subject: [Python-Dev] svnmerge init in the release26-maint branch?
In-Reply-To: <ged3e8$hsk$1@ger.gmane.org>
References: <ged3e8$hsk$1@ger.gmane.org>
Message-ID: <490A12B2.9070107@cheimes.de>

Thomas Heller wrote:
> I have a fix for a modulefinder crash that I'm going to commit
> into the trunk. Since the bug is also in release26-maint I will
> also backport it.  While preparing this I noticed that in the
> release26-maint branch 'svnmerge init' has not yet been done.

Are you sure? I did a svnmerge init a couple of days after the 
release26-maint branch was created. I can see the properties on the 
branch, too.

.../python/release26-maint$ svn proplist -v . | grep svnmerge
   svnmerge-blocked : 
/python/trunk:66721-66722,66744-66745,66752,66756,66763-66765,66768,66791-66792,66857
   svnmerge-integrated : 
/python/trunk:1-66720,66723-66743,66746-66751,66753-66755,66757-66762,66766-66767,66769-66790,66793-66800,66802,66805-66812,66814-66821,66824-66831,66833-66835,66837-66851,66853,66858-66860,66862-66863,66881,66891,66906,66922,66928-66931,66933-66937,66939-66940,66958,66984,66995-66996,67000,67030-67031

Christian

From theller at ctypes.org  Thu Oct 30 21:16:45 2008
From: theller at ctypes.org (Thomas Heller)
Date: Thu, 30 Oct 2008 21:16:45 +0100
Subject: [Python-Dev] svnmerge init in the release26-maint branch?
In-Reply-To: <490A12B2.9070107@cheimes.de>
References: <ged3e8$hsk$1@ger.gmane.org> <490A12B2.9070107@cheimes.de>
Message-ID: <ged4nd$mfm$2@ger.gmane.org>

Christian Heimes schrieb:
> Thomas Heller wrote:
>> I have a fix for a modulefinder crash that I'm going to commit
>> into the trunk. Since the bug is also in release26-maint I will
>> also backport it.  While preparing this I noticed that in the
>> release26-maint branch 'svnmerge init' has not yet been done.
> 
> Are you sure? I did a svnmerge init a couple of days after the 
> release26-maint branch was created. I can see the properties on the 
> branch, too.
> 
> .../python/release26-maint$ svn proplist -v . | grep svnmerge
>    svnmerge-blocked : 
> /python/trunk:66721-66722,66744-66745,66752,66756,66763-66765,66768,66791-66792,66857
>    svnmerge-integrated : 
> /python/trunk:1-66720,66723-66743,66746-66751,66753-66755,66757-66762,66766-66767,66769-66790,66793-66800,66802,66805-66812,66814-66821,66824-66831,66833-66835,66837-66851,66853,66858-66860,66862-66863,66881,66891,66906,66922,66928-66931,66933-66937,66939-66940,66958,66984,66995-66996,67000,67030-67031

Oh, sorry - I forgot to 'svn up' first.

-- 
Thanks,
Thomas


From theller at ctypes.org  Thu Oct 30 21:29:23 2008
From: theller at ctypes.org (Thomas Heller)
Date: Thu, 30 Oct 2008 21:29:23 +0100
Subject: [Python-Dev] buildbots
Message-ID: <ged5f2$qlu$1@ger.gmane.org>

AFAICS there are no buildbots for the release26-maint branch.

Thomas


From martin at v.loewis.de  Thu Oct 30 21:38:12 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 30 Oct 2008 21:38:12 +0100
Subject: [Python-Dev] buildbots
In-Reply-To: <ged5f2$qlu$1@ger.gmane.org>
References: <ged5f2$qlu$1@ger.gmane.org>
Message-ID: <490A1B34.7080806@v.loewis.de>

> AFAICS there are no buildbots for the release26-maint branch.

That's correct. I'm waiting to create them until after we retire the
2.5 ones.

Regards,
Martin

From musiccomposition at gmail.com  Thu Oct 30 21:38:19 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Thu, 30 Oct 2008 15:38:19 -0500
Subject: [Python-Dev] Classifying language vs. impl-detail tests
In-Reply-To: <20081030171620.GA18469@code0.codespeak.net>
References: <20081030171620.GA18469@code0.codespeak.net>
Message-ID: <1afaf6160810301338k7459a6ccw9dc490543a1ac71c@mail.gmail.com>

On Thu, Oct 30, 2008 at 12:16 PM, Armin Rigo <arigo at tunes.org> wrote:
> Hi all,
>
> Here is a first step towards classifying the Python test suite into
> "real language tests" and "implementation details":
>
>    http://bugs.python.org/issue4242

I've actually implemented something like this along with more general
test skipping capabilities in my Bazaar testing branch. [1] It adds a
decorator @cpython_only that can be applied to tests.

>
> If the general approach seems acceptable to people, I would be willing
> to port more of PyPy's test suite patches.  The net result would be to
> move the test suite towards being more directly useful for alternate
> Python implementations.  (Right now, all of them have some custom tests
> plus their own similarly patched copy of the stdlib test suite.)

That would be great to see!

>
> Note that the patch above is against 2.7 trunk; the 2.x line is what all
> non-CPython implementations currently target.  The general idea (and to
> a large extend the patches) can easily be forward-ported to 3.x when it
> makes sense.
>
> Also, the actual decision of what is a "real" or a "detail" test is of
> course up to discussion.  The patch above does the classification for
> test_descr.  It was obtained by running it with PyPy, and for each
> failure either fixing PyPy, or argumenting (by adding a comment) the
> reason for which it is a detail -- usually referring to "well-known
> agreement" expressed on python-dev about the matter.  (Of course,
> "detail" tests still run normally on top of CPython.)

[1] http://code.python.org/python/users/benjamin.peterson/new_testing/main




-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From musiccomposition at gmail.com  Thu Oct 30 21:43:01 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Thu, 30 Oct 2008 15:43:01 -0500
Subject: [Python-Dev] buildbots
In-Reply-To: <490A1B34.7080806@v.loewis.de>
References: <ged5f2$qlu$1@ger.gmane.org> <490A1B34.7080806@v.loewis.de>
Message-ID: <1afaf6160810301343o29c4b26cs89859fc3da1e7dbc@mail.gmail.com>

On Thu, Oct 30, 2008 at 3:38 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> AFAICS there are no buildbots for the release26-maint branch.
>
> That's correct. I'm waiting to create them until after we retire the
> 2.5 ones.

That reminds me: Do we have a final release date for 2.5.3 set?



-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From martin at v.loewis.de  Thu Oct 30 21:46:50 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Thu, 30 Oct 2008 21:46:50 +0100
Subject: [Python-Dev] buildbots
In-Reply-To: <1afaf6160810301343o29c4b26cs89859fc3da1e7dbc@mail.gmail.com>
References: <ged5f2$qlu$1@ger.gmane.org> <490A1B34.7080806@v.loewis.de>
	<1afaf6160810301343o29c4b26cs89859fc3da1e7dbc@mail.gmail.com>
Message-ID: <490A1D3A.6010504@v.loewis.de>

Benjamin Peterson wrote:
> On Thu, Oct 30, 2008 at 3:38 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>> AFAICS there are no buildbots for the release26-maint branch.
>> That's correct. I'm waiting to create them until after we retire the
>> 2.5 ones.
> 
> That reminds me: Do we have a final release date for 2.5.3 set?

Not precisely. However, I want to make the release candidate for 2.5.3
a week after the 3.0 release, and the final release one week afterwards.
If the 3.0 release plan stands, that would give December 10 for the
release candidate, and December 17 for the final release.

Regards,
Martin

From amk at amk.ca  Thu Oct 30 22:17:02 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 30 Oct 2008 17:17:02 -0400
Subject: [Python-Dev] My patches
In-Reply-To: <79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
Message-ID: <20081030211702.GA11063@amk-desktop.matrixgroup.net>

On Thu, Oct 30, 2008 at 03:55:38PM +0000, Paul Moore wrote:
> 2. Some patches marked as "documentation" are doc fixes, others seem
> to be issues where it has been decided that the behaviour is correct
> as is, but needs to be documented. Fair enough, but it's much harder
> to assess the latter, and there's no way of just grabbing the former
> (for example, to spend a spare 30 minutes reviewing simple stuff).

Perhaps the documentation category could be split into 'Documentation'
and 'Documentation Needed'; the latter means the issue entails writing
new text as opposed to rewriting.  But I think on average
documentation issues get processed pretty quickly: Georg is responsive
to them, and patches are easy to apply because you don't need to worry
about breaking anything.

In general Python development is much less freewheeling and fun than
it used to be.  You could come up with new features and modules, add
lots of new capabilities to a module.  Today we're making much smaller
changes, discuss them at far great length, and have to worry about
breaking all the existing Python code out there, It's a sign of
Python's maturity, of course, and I'm not saying that the design
effort and the compatibility requirements should be dropped, but they
certainly act as a damper.

On some of my issues (esp. ones relating to curses and mailbox.py), I
feel paralyzed because problems are occurring on platforms I don't
have access to (e.g. FreeBSD).  The buildbots will report problems,
but then you have to debug them by committing changes, triggering a
build, and observing the results.  And all of these actions will send
e-mail to python-checkins.  (Imagine if every 'print "reached here!"'
you added while debugging was e-mailed to everyone...)

--amk

From g.brandl at gmx.net  Thu Oct 30 23:26:11 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 30 Oct 2008 22:26:11 +0000
Subject: [Python-Dev] My patches
In-Reply-To: <71ac8a310810300811g3a241a85s85fdc48ffd3c8282@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<20081030110442.1b6ce78e@resist.wooz.org>	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<71ac8a310810300811g3a241a85s85fdc48ffd3c8282@mail.gmail.com>
Message-ID: <ged8pr$65k$1@ger.gmane.org>

Arfrever Frehtes Taifersar Arahesis schrieb:
> 2008-10-30 16:04 A.M. Kuchling <amk at amk.ca> napisa?(a):
>> On Thu, Oct 30, 2008 at 11:04:42AM +0000, Barry Warsaw wrote:
>>> One of the reasons why I'm very keen on us moving to a distributed version
>>> control system is to help break the logjam on core developers.  True, your
>>> code will still not be able to land in the "official" branch without core
>>> developer intervention, but you will be able to share your code, fixes,
>>> branches with everyone in a much more live way than patches in a tracker.
>>
>> I don't see how a DVCS will fix anything.  The bottleneck is in
>> assessing patches for inclusion in the master tree; not enough people
>> are doing that.  We'd just end up with lots of proposed branches
>> waiting to be merged, instead of patches to be applied.
>>
>> (What a DVCS might enable is making it easier to do larger
>> experiments, like the recent Vmgen work, and publish them in a form
>> that people can download.  We could create SVN branches now, but that
>> means people would then have commit access to all of the Python
>> source.)
> 
> SVN supports path-based authorization.
> http://svnbook.red-bean.com/nightly/en/svn.serverconfig.pathbasedauthz.html

I don't see a problem with giving access to people wanting to work on
a branch, and telling them not to commit on the trunk.

We had and still have some such branches going on, e.g. for SoC students.
I also gave out access to several developers to work on the docs only,
or Sphinx which still is in Python's SVN repo.

After all, it's a VCS, and if someone plays against the rules, the commit is
reverted and the privs dropped in perhaps a minute.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From exarkun at divmod.com  Thu Oct 30 22:54:08 2008
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Thu, 30 Oct 2008 16:54:08 -0500
Subject: [Python-Dev] My patches
In-Reply-To: <20081030211702.GA11063@amk-desktop.matrixgroup.net>
Message-ID: <20081030215408.29191.796454703.divmod.quotient.47062@ohm>

On Thu, 30 Oct 2008 17:17:02 -0400, "A.M. Kuchling" <amk at amk.ca> wrote:
> [snip]
>
>On some of my issues (esp. ones relating to curses and mailbox.py), I
>feel paralyzed because problems are occurring on platforms I don't
>have access to (e.g. FreeBSD).  The buildbots will report problems,
>but then you have to debug them by committing changes, triggering a
>build, and observing the results.  And all of these actions will send
>e-mail to python-checkins.  (Imagine if every 'print "reached here!"'
>you added while debugging was e-mailed to everyone...)

I do that when I need to.  People whose lives would be ruined by the
receipt of such an email always have the option of leaving the checkin
list.

However, there is a buildbot feature named "try" which lets you submit
a patch (subject to authentication) and performs a build with the patch
applied.  This lets you try lots of little changes without getting your
VCS involved.  It needs to be enabled in the buildmaster configuration
and credentials created for any user who will be given access.

Jean-Paul

From amauryfa at gmail.com  Thu Oct 30 23:18:56 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 30 Oct 2008 23:18:56 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <20081030211702.GA11063@amk-desktop.matrixgroup.net>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
	<20081030211702.GA11063@amk-desktop.matrixgroup.net>
Message-ID: <e27efe130810301518k2cd13fe7idbc702713e9602bd@mail.gmail.com>

On Thu, Oct 30, 2008 at 22:17, A.M. Kuchling <amk at amk.ca> wrote:
> On some of my issues (esp. ones relating to curses and mailbox.py), I
> feel paralyzed because problems are occurring on platforms I don't
> have access to (e.g. FreeBSD).  The buildbots will report problems,
> but then you have to debug them by committing changes, triggering a
> build, and observing the results.  And all of these actions will send
> e-mail to python-checkins.  (Imagine if every 'print "reached here!"'
> you added while debugging was e-mailed to everyone...)

By the way, it seems that this python-checkins mailing list did not
archive the recent commits:
http://mail.python.org/pipermail/python-checkins/2008-October/date.html#end
I miss them... Can someone fix it?

-- 
Amaury Forgeot d'Arc

From alexandre at peadrop.com  Thu Oct 30 23:23:09 2008
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Thu, 30 Oct 2008 18:23:09 -0400
Subject: [Python-Dev] Proper initialization of structs
In-Reply-To: <05D72249-2D59-47F2-BE26-FF9DC3541958@acm.org>
References: <gecfr4$5p1$1@ger.gmane.org>
	<05D72249-2D59-47F2-BE26-FF9DC3541958@acm.org>
Message-ID: <acd65fa20810301523s4cbd8b6ex34a06162a59d23e4@mail.gmail.com>

On Thu, Oct 30, 2008 at 1:00 PM, Fred Drake <fdrake at acm.org> wrote:
> It's good to move work into __init__ where reasonable, so that it can be
> avoided if a subclass wants it done in a completely different way, but new
> can't work that way.
>

And that is exactly the reason why, the _pickle module doesn't use
__new__ for initialization.  Doing any kind of argument parsing in
__new__ prevents subclasses from customizing the arguments for their
__init__.

Although, I agree that __new__ should be used, whenever it is
possible, to initialize struct members.

-- Alexandre

From ncoghlan at gmail.com  Thu Oct 30 23:36:25 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 31 Oct 2008 08:36:25 +1000
Subject: [Python-Dev] Proper initialization of structs
In-Reply-To: <acd65fa20810301523s4cbd8b6ex34a06162a59d23e4@mail.gmail.com>
References: <gecfr4$5p1$1@ger.gmane.org>	<05D72249-2D59-47F2-BE26-FF9DC3541958@acm.org>
	<acd65fa20810301523s4cbd8b6ex34a06162a59d23e4@mail.gmail.com>
Message-ID: <490A36E9.4020207@gmail.com>

Alexandre Vassalotti wrote:
> On Thu, Oct 30, 2008 at 1:00 PM, Fred Drake <fdrake at acm.org> wrote:
>> It's good to move work into __init__ where reasonable, so that it can be
>> avoided if a subclass wants it done in a completely different way, but new
>> can't work that way.
>>
> 
> And that is exactly the reason why, the _pickle module doesn't use
> __new__ for initialization.  Doing any kind of argument parsing in
> __new__ prevents subclasses from customizing the arguments for their
> __init__.

No it doesn't - it just means the subclasses have to override __new__ as
well and then give the parent class the arguments it needs.

I've used this convention (*must* call parent class __new__ or the
instance will be broken, may call parent class __init__ if it is
helpful) many times, and it is far more robust than relying on
subclasses to remember to call the parent class __init__ when setting up
the class.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From foom at fuhm.net  Thu Oct 30 23:39:54 2008
From: foom at fuhm.net (James Y Knight)
Date: Thu, 30 Oct 2008 18:39:54 -0400
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <4909EF67.5010104@trueblade.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>	<gecpks$cgc$1@ger.gmane.org>
	<gecqd4$g9g$1@ger.gmane.org>
	<ca471dc20810301025p47843333yff585b8507d9d521@mail.gmail.com>
	<4909EF67.5010104@trueblade.com>
Message-ID: <2BAEE83C-E19A-4141-9435-CB1205B4C184@fuhm.net>


On Oct 30, 2008, at 1:31 PM, Eric Smith wrote:

> Guido van Rossum wrote:
>> No offense taken. The V8 experience makes me feel much more  
>> optimistic
>> that they might actually pull this off. (I'm still skeptical about
>> support for extension modules, withougt which CPython is pretty  
>> lame.)
>
> The need to modify all extension modules is the usual non-starter I  
> see mentioned when this topic comes up. The OP really needs to think  
> about that issue.


It's not a non-starter, it's just a non-finisher. :)

One could take an approach like Apple did for ObjC 2.0: libraries  
should be ported over time to be able to work with both refcounting  
and automatic-GC runtimes. When you link a program, you can choose to  
link it with the automatic GC objc runtime, as long as all the other  
frameworks you want to use are compatible with that.

What this would mean in python terms:
- Python would be able to be compiled in either refcounting or auto-gc  
mode.
- Extensions can be modified to be compatible with the auto-gc mode  
over the timespan of a few years.
- Then when most extensions have been adjusted, auto-gc would become  
the default mode for python to be compiled in.

It's seems theoretically entirely doable, but will surely be a lot of  
work...someone'd have to be ready to really do the hard work to push  
it through to completion.

James

From martin at v.loewis.de  Fri Oct 31 00:24:14 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 31 Oct 2008 00:24:14 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <20081030211702.GA11063@amk-desktop.matrixgroup.net>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<20081030110442.1b6ce78e@resist.wooz.org>	<20081030150427.GA9631@amk-desktop.matrixgroup.net>	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
	<20081030211702.GA11063@amk-desktop.matrixgroup.net>
Message-ID: <490A421E.8000202@v.loewis.de>

> On some of my issues (esp. ones relating to curses and mailbox.py), I
> feel paralyzed because problems are occurring on platforms I don't
> have access to (e.g. FreeBSD).  The buildbots will report problems,
> but then you have to debug them by committing changes, triggering a
> build, and observing the results.  And all of these actions will send
> e-mail to python-checkins.

I wouldn't worry about that. If the checkin message indicates that
this is testing out a certain buildbot issue, most python-checkin
readers probably will skip over it quickly.

If you are still worried: buildbot has a "try" feature, where developers
can try out changes on the slaves before committing them. This isn't
currently activated in the Python buildbot - but if you need it, we
could look into it (if you are curious, just experiment with it
yourself).

Regards,
Martin

From martin at v.loewis.de  Fri Oct 31 00:25:48 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 31 Oct 2008 00:25:48 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <e27efe130810301518k2cd13fe7idbc702713e9602bd@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<20081030110442.1b6ce78e@resist.wooz.org>	<20081030150427.GA9631@amk-desktop.matrixgroup.net>	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>	<20081030211702.GA11063@amk-desktop.matrixgroup.net>
	<e27efe130810301518k2cd13fe7idbc702713e9602bd@mail.gmail.com>
Message-ID: <490A427C.8050607@v.loewis.de>

> By the way, it seems that this python-checkins mailing list did not
> archive the recent commits:
> http://mail.python.org/pipermail/python-checkins/2008-October/date.html#end
> I miss them... Can someone fix it?

Which ones are you missing specifically?

Regards,
Martin


From musiccomposition at gmail.com  Fri Oct 31 00:27:49 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Thu, 30 Oct 2008 18:27:49 -0500
Subject: [Python-Dev] My patches
In-Reply-To: <490A427C.8050607@v.loewis.de>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
	<20081030211702.GA11063@amk-desktop.matrixgroup.net>
	<e27efe130810301518k2cd13fe7idbc702713e9602bd@mail.gmail.com>
	<490A427C.8050607@v.loewis.de>
Message-ID: <1afaf6160810301627x26a8b355u673daeb8e085f7b2@mail.gmail.com>

On Thu, Oct 30, 2008 at 6:25 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> By the way, it seems that this python-checkins mailing list did not
>> archive the recent commits:
>> http://mail.python.org/pipermail/python-checkins/2008-October/date.html#end
>> I miss them... Can someone fix it?
>
> Which ones are you missing specifically?

I haven't seen any of the ones today.


-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."

From p.f.moore at gmail.com  Fri Oct 31 00:34:32 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 30 Oct 2008 23:34:32 +0000
Subject: [Python-Dev] My patches
In-Reply-To: <490A0E46.3070003@v.loewis.de>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
	<490A0E46.3070003@v.loewis.de>
Message-ID: <79990c6b0810301634i49ca0d50l23d3601a8c5ceabb@mail.gmail.com>

2008/10/30 "Martin v. L?wis" <martin at v.loewis.de>:
>> Question - is there anything Roundup can do to help triage? Extra
>> status or keyword values ("has patch",
>
> There is "patch" keyword already, and a public query "Patches"
> (as well as "My Patches")

Sorry, I checked the keywords but missed it.

>> "ready to go", ...)?
>
> We could give more people the right to set the resolution to "Accepted".
> This is a matter of trust, though - if the committer then still needs
> to review it, anyway, nothing is gained.

Agreed. I was thinking vaguely in terms of a type of voting - rather
than a status or resolution, it might be more like the nosy list - a
list of people who have said they think the patch is OK. The more
people on the list, the stronger the assurance that it's acceptable.
It is still a matter of trust, of course - nothing can avoid that.

>> More
>> canned searches ("Show Open" and "Show Unassigned" aren't a lot of
>> help)?
>
> Please go to the "edit" label next to "Search". You can store your
> own searches, but you can also share searches with others.

I see it now. Thanks. It's not very obvious, but once you know it's
there, it looks fine.

>> Custom reports (summaries by type)?
>
> This I don't understand - how is it different from a search?

I was thinking in terms of summary reports - counts of numbers of
issues in various groups. The output layout is different from a
search. My idea was to make it easier to find areas which are worth
tackling (for example, if there are lots of documentation patches, it
might be worth a look through to see if any are trivial). It's graphs
and counts to help people to find areas they can help with that I was
thinking of.

It's not a very clear concept, even in my own mind though.

>> Or are such things there and simply not publicised enough?
>
> Perhaps. I really don't know what percentage of interested users
> is aware of roundup capabilities.

Fair point. My gut feeling is that more people would be interested if
we had ways of presenting the list of issues in better ways than the
current monolithic list. If people could see "hey, there's a lot of
documentation (or library, or C code, or whatever) patches which
haven't been looked at yet", they might be inclined to take a look.

Maybe even a simple graph of current issues on the python.org front
page, with a "Lend a hand!" type of heading, to suggest that people
could help.

There's still the trust issue you mentioned above, but my instinct is
that the average Python coder simply doesn't realise that they could
help - or they believe that taking a spare 15 minutes "wouldn't be
worth it".

>> 2. Some patches marked as "documentation" are doc fixes, others seem
>> to be issues where it has been decided that the behaviour is correct
>> as is, but needs to be documented. Fair enough, but it's much harder
>> to assess the latter, and there's no way of just grabbing the former
>> (for example, to spend a spare 30 minutes reviewing simple stuff).
>
> There is the "easy" keyword. Of course, it might also be useful to
> triage more issues as "easy".

That might help. But you can't search on combinations of keyword (e.g.
easy, with a patch). Maybe an extra property "Difficulty", with values
Easy, Moderate, Complex (and blank, for "not checked yet") would be
good. Interested parties could check for issues with blank difficulty,
and assign a difficulty level. That's useful triage, and not that hard
for anyone to do.

>> 3. There's nothing obvious I can do to move an issue forward. Sure, I
>> can make a comment, but that's about it. I'd like something that stood
>> a bit more chance of getting noticed (like a status change, or maybe a
>> list of people who think this is good to apply, which I can add myself
>> to).
>
> The "developer" role has more user interface. I've just given it to you.

Thanks. I'll try to justify it by doing a bit more on the tracker :-)

>> Hmm, I've spent more time on this than I should have, and it's gone
>> way off topic. Is there anywhere better to discuss it?
>
> There is the tracker-discuss list for discussion, and the meta tracker
> for actual problems/wishes for the tracker.

Thanks.
Paul.

From lists at cheimes.de  Fri Oct 31 00:43:46 2008
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 31 Oct 2008 00:43:46 +0100
Subject: [Python-Dev] Proper initialization of structs
In-Reply-To: <acd65fa20810301523s4cbd8b6ex34a06162a59d23e4@mail.gmail.com>
References: <gecfr4$5p1$1@ger.gmane.org>	
	<05D72249-2D59-47F2-BE26-FF9DC3541958@acm.org>
	<acd65fa20810301523s4cbd8b6ex34a06162a59d23e4@mail.gmail.com>
Message-ID: <490A46B2.6060207@cheimes.de>

Alexandre Vassalotti wrote:
> And that is exactly the reason why, the _pickle module doesn't use
> __new__ for initialization.  Doing any kind of argument parsing in
> __new__ prevents subclasses from customizing the arguments for their
> __init__.
> 
> Although, I agree that __new__ should be used, whenever it is
> possible, to initialize struct members.

You are missunderstanding me. I want everybody to set the struct members 
to *A* sensible default value, not *THE* value. Argument parsing can 
still happen in tp_init. tp_new should (or must?) set all struct members 
to sensible defaults like NULL for pointers, -1 or 0 for numbers etc.

Python uses malloc to allocate memory. Unless you are using debug builds 
the memory block is not initialized. In both cases the block of memory 
isn't zeroed. You all know the problems caused by uninitialized memory.

Christian

From alexandre at peadrop.com  Fri Oct 31 00:51:47 2008
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Thu, 30 Oct 2008 19:51:47 -0400
Subject: [Python-Dev] Proper initialization of structs
In-Reply-To: <490A36E9.4020207@gmail.com>
References: <gecfr4$5p1$1@ger.gmane.org>
	<05D72249-2D59-47F2-BE26-FF9DC3541958@acm.org>
	<acd65fa20810301523s4cbd8b6ex34a06162a59d23e4@mail.gmail.com>
	<490A36E9.4020207@gmail.com>
Message-ID: <acd65fa20810301651h166f7862j573c6fab9581e235@mail.gmail.com>

On Thu, Oct 30, 2008 at 6:36 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Alexandre Vassalotti wrote:
>> And that is exactly the reason why, the _pickle module doesn't use
>> __new__ for initialization.  Doing any kind of argument parsing in
>> __new__ prevents subclasses from customizing the arguments for their
>> __init__.
>
> No it doesn't - it just means the subclasses have to override __new__ as
> well and then give the parent class the arguments it needs.
>

Right, and that's what subclasses of built-in types need to do.
However for _pickle, I wanted to reduce interface differences between
the Python implementation. Perhaps, in that case, moving the content
of __init__ methods in both implementations would have been a better
choice. That would certainly make it easier to expose internal methods
easier to fix issue3385. Is it too late to change?

-- Alexandre

From martin at v.loewis.de  Fri Oct 31 00:54:51 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Fri, 31 Oct 2008 00:54:51 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <1afaf6160810301627x26a8b355u673daeb8e085f7b2@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>	
	<20081030110442.1b6ce78e@resist.wooz.org>	
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>	
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>	
	<20081030211702.GA11063@amk-desktop.matrixgroup.net>	
	<e27efe130810301518k2cd13fe7idbc702713e9602bd@mail.gmail.com>	
	<490A427C.8050607@v.loewis.de>
	<1afaf6160810301627x26a8b355u673daeb8e085f7b2@mail.gmail.com>
Message-ID: <490A494B.6090003@v.loewis.de>

> I haven't seen any of the ones today.

OK, I've respooled them.

Martin

From alexandre at peadrop.com  Fri Oct 31 01:10:38 2008
From: alexandre at peadrop.com (Alexandre Vassalotti)
Date: Thu, 30 Oct 2008 20:10:38 -0400
Subject: [Python-Dev] Proper initialization of structs
In-Reply-To: <490A46B2.6060207@cheimes.de>
References: <gecfr4$5p1$1@ger.gmane.org>
	<05D72249-2D59-47F2-BE26-FF9DC3541958@acm.org>
	<acd65fa20810301523s4cbd8b6ex34a06162a59d23e4@mail.gmail.com>
	<490A46B2.6060207@cheimes.de>
Message-ID: <acd65fa20810301710o282e1b64jdae15188868b1e93@mail.gmail.com>

[oops, I forgot to cc the list]

On Thu, Oct 30, 2008 at 7:43 PM, Christian Heimes <lists at cheimes.de> wrote:
> Alexandre Vassalotti wrote:
>>
>> And that is exactly the reason why, the _pickle module doesn't use
>> __new__ for initialization.  Doing any kind of argument parsing in
>> __new__ prevents subclasses from customizing the arguments for their
>> __init__.
>>
>> Although, I agree that __new__ should be used, whenever it is
>> possible, to initialize struct members.
>
> You are missunderstanding me. I want everybody to set the struct members to
> *A* sensible default value, not *THE* value. Argument parsing can still
> happen in tp_init. tp_new should (or must?) set all struct members to
> sensible defaults like NULL for pointers, -1 or 0 for numbers etc.
>
> Python uses malloc to allocate memory. Unless you are using debug builds the
> memory block is not initialized. In both cases the block of memory isn't
> zeroed. You all know the problems caused by uninitialized memory.
>

But what if  PyType_GenericAlloc is used for tp_alloc? As far as I
know, the memory block allocated with PyType_GenericAlloc is zeroed.

-- Alexandre

From dstanek at dstanek.com  Fri Oct 31 01:13:14 2008
From: dstanek at dstanek.com (David Stanek)
Date: Thu, 30 Oct 2008 20:13:14 -0400
Subject: [Python-Dev] My patches
In-Reply-To: <94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
	<94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
Message-ID: <de32cc030810301713l7adb0e24meb9a5dba11cee148@mail.gmail.com>

On Thu, Oct 30, 2008 at 12:09 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
> What about having two level of devs ?
>
> + core developers
> + standard library developers
>

I was also thinking about two levels of developers,  but structured a
little differently. We have the same core developers with permission
to commit anywhere in the repo. Then we have a set of sub-core (or
whatever) that only have permission to commit to experimental
branches.

These experimental branches would be the testing place for some
patches. Once a patch has had enough exposure in the experimental
branch and has been verified by several people the core team could
merge it into the main development branch. A set of buildbot slaves
can also check that branch on a variety of systems and architectures.

The structure could then look like:
  * trunk - the mainline of development
  * branches/release##maint - for each version
  * branches/experimental## - for the sub-core devs

This may start to be a bit messy if the experimental branch diverges
too much from trunk. trunk would also have to be periodically merged
into the experimental branches. Even so I think it could still be
manageable. And if not the no harm/no foul the experimental branches
could be abandoned and little core developer time would be wasted.

-- 
David
http://www.traceback.org

From victor.stinner at haypocalc.com  Fri Oct 31 01:27:35 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Fri, 31 Oct 2008 01:27:35 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <79990c6b0810301634i49ca0d50l23d3601a8c5ceabb@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<490A0E46.3070003@v.loewis.de>
	<79990c6b0810301634i49ca0d50l23d3601a8c5ceabb@mail.gmail.com>
Message-ID: <200810310127.35945.victor.stinner@haypocalc.com>

Le Friday 31 October 2008 00:34:32 Paul Moore, vous avez ?crit?:
> Agreed. I was thinking vaguely in terms of a type of voting - rather
> than a status or resolution, it might be more like the nosy list - a
> list of people who have said they think the patch is OK. The more
> people on the list, the stronger the assurance that it's acceptable.
> It is still a matter of trust, of course - nothing can avoid that.

I like this idea. But there are different things to review. Examples:
 - the bug report: is the bug reproductible? is the bug isolated?
 - a patch: the patch works? the patch looks correct? or invalid coding style, 
introduce a regression, or anything else

> I was thinking in terms of summary reports (...)

I think that you need an new information: the issue progress, eg.
 - initial state: 0% => need more information
 - bug isolated: 25% => need a patch
 - patch present: 50% => patch needs reviewers
 - patch reviewed: 75% => patch just have to be applied
 - issue closed: 100% (done)

Beginners can search for progress < 25%. They can try to reproduce a problem 
to check the Python version, the OS, etc. Or just help to give more 
informations about the issue.

Core developers just have to check for progress >= 75%.

> Fair point. My gut feeling is that more people would be interested if
> we had ways of presenting the list of issues in better ways than the
> current monolithic list. (...)

Why not using icons (at least on the HTML view)? It helps to see quickly many 
informations and generates smaller reports. We can have an icon for each 
keyword.

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From nas at arctrix.com  Fri Oct 31 02:27:12 2008
From: nas at arctrix.com (Neil Schemenauer)
Date: Fri, 31 Oct 2008 01:27:12 +0000 (UTC)
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
Message-ID: <gedmtg$eop$1@ger.gmane.org>

Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
> For a student project in a course on virtual machines, we are
> evaluating the possibility to experiment with removing the GIL
> from CPython

Hi,

It's great to hear of this kind of project.  I think what you want
to do is difficult but possible.  The major compilcation would be
that extension modules would have to re-written since they all
assume a reference counting GC.  A foreign function interface like
CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
does place different demands on extension module authors. 

As a student project, I think it would make sense to forget about
extensions and try to create a barebones interpreter based on
CPython's runtime, compiler, etc with the reference counting ripped
out and some other GC installed.

I once had the Boehm GC working with Python.  That's not too
interesting.  I'm not really up-to-date on current GC research but I
think your GC has to do some copying in order to get really good
performance.  Producing a barebones version of the Python
interpreter that utilizes a modern, copying GC would be a nice
achievement and could be a platform for future work.

Regards,

  Neil


From brett at python.org  Fri Oct 31 03:37:09 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 30 Oct 2008 19:37:09 -0700
Subject: [Python-Dev] My patches
In-Reply-To: <200810310127.35945.victor.stinner@haypocalc.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<490A0E46.3070003@v.loewis.de>
	<79990c6b0810301634i49ca0d50l23d3601a8c5ceabb@mail.gmail.com>
	<200810310127.35945.victor.stinner@haypocalc.com>
Message-ID: <bbaeab100810301937u3b066ff6q872671a626a970a0@mail.gmail.com>

On Thu, Oct 30, 2008 at 17:27, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Le Friday 31 October 2008 00:34:32 Paul Moore, vous avez ?crit :
>> Agreed. I was thinking vaguely in terms of a type of voting - rather
>> than a status or resolution, it might be more like the nosy list - a
>> list of people who have said they think the patch is OK. The more
>> people on the list, the stronger the assurance that it's acceptable.
>> It is still a matter of trust, of course - nothing can avoid that.
>
> I like this idea. But there are different things to review. Examples:
>  - the bug report: is the bug reproductible? is the bug isolated?
>  - a patch: the patch works? the patch looks correct? or invalid coding style,
> introduce a regression, or anything else
>
>> I was thinking in terms of summary reports (...)
>
> I think that you need an new information: the issue progress, eg.
>  - initial state: 0% => need more information
>  - bug isolated: 25% => need a patch
>  - patch present: 50% => patch needs reviewers
>  - patch reviewed: 75% => patch just have to be applied
>  - issue closed: 100% (done)
>
> Beginners can search for progress < 25%. They can try to reproduce a problem
> to check the Python version, the OS, etc. Or just help to give more
> informations about the issue.
>
> Core developers just have to check for progress >= 75%.
>

I have a similar list that I have been thinking about proposing. I did
a blog post about it at
http://sayspy.blogspot.com/2008/08/what-are-typical-steps-issue-goes.html
and received positive feedback:

* triage
* verify bug
* test needed
* needs patch
* patch review
* commit review
* committed/rejected

That way all the steps needed are obvious. I was going to start
working on proposing this after doing the first draft of the DVCS PEP.

-Brett

From brett at python.org  Fri Oct 31 04:55:19 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 30 Oct 2008 20:55:19 -0700
Subject: [Python-Dev] hg branch gone?
Message-ID: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>

I just tried to update my 3.0 branch in hg from
http://code.python.org/hg/branches/py3k/ and hg is telling me it's a
404. Anyone else having trouble?

-Brett

From ncoghlan at gmail.com  Fri Oct 31 07:24:57 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 31 Oct 2008 16:24:57 +1000
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <gedmtg$eop$1@ger.gmane.org>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org>
Message-ID: <490AA4B9.5060307@gmail.com>

Neil Schemenauer wrote:
> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>> For a student project in a course on virtual machines, we are
>> evaluating the possibility to experiment with removing the GIL
>> from CPython
> 
> Hi,
> 
> It's great to hear of this kind of project.  I think what you want
> to do is difficult but possible.  The major compilcation would be
> that extension modules would have to re-written since they all
> assume a reference counting GC.  A foreign function interface like
> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
> does place different demands on extension module authors. 

Michael Foord's comment about the way Ironclad does it suggests a
possible interim step that would allow existing extensions to be used at
the cost of some additional overhead: use free threading in the core,
but have an "extension module lock" that cuts things back to a single
thread whenever non-core code is invoked.

It wouldn't be as fast as a completely free-threaded setup (since the
EML would need to operate as a variant of a 'read/write' lock, with the
core code mostly leaving it in the multiple-threads-allowed 'read' mode,
promoting it to single-thread-only 'write' mode when potentially
dispatching to non-core code, and dropping back to the 'read' mode when
the dispatch landed back in core code again).

Over time, extension modules which were updated to use their own
internal locks could also drop back to free-threading mode. In essence,
the GIL would still exist, but instead of being a simple mutex the way
it is now, it would have an intermediate stage which allowed multiple
threads to continue running. Once most extension modules had been
converted, the function to promote the EML from multi-threaded mode to
single-thread only mode could be removed.

A mechanism like that would avoid one of the more subtle problems in
removing the GIL: currently, CPython-specific Python code can assume
that invocation of a C function from Python code is an atomic operation,
since the body of the function executes as a single bytecode (we're
talking about the case here where the C function is known not to release
the GIL).

I've written code which relied on the GIL in exactly that way, and I'm
sure I'm far from the only Python programmer to do so. If the GIL was
removed without adding an extension module lock, such applications would
require additional locking either in the C extension module or in the
Python code.

A possible starting point for such a project: move the locking
mechanisms that are implemented in Python in the threading module to
C-based implementations in _thread so that a richer low overhead locking
arrangement is available to extensions modules (and the rest of the core).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ncoghlan at gmail.com  Fri Oct 31 07:30:21 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 31 Oct 2008 16:30:21 +1000
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <4909FECF.7000703@voidspace.org.uk>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>	<gecpks$cgc$1@ger.gmane.org>	<gecqd4$g9g$1@ger.gmane.org>	<ca471dc20810301025p47843333yff585b8507d9d521@mail.gmail.com>	<4909EF67.5010104@trueblade.com>	<ca471dc20810301034x7d6adb88wb875100e648d5c28@mail.gmail.com>
	<4909FECF.7000703@voidspace.org.uk>
Message-ID: <490AA5FD.80105@gmail.com>

Michael Foord wrote:
> Moving more C extensions to an implementation based on ctypes would be
> enormously useful for PyPy, IronPython and Jython, but ctypes is not yet
> as portable as Python itself which could be an issue (although one worth
> resolving).

If I recall correctly, the main blocker to ctypes portability is libffi
portability. So if anyone would like to see ctypes on more platforms,
then that's the limitation they really need to attack.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------


From ncoghlan at gmail.com  Fri Oct 31 07:46:26 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 31 Oct 2008 16:46:26 +1000
Subject: [Python-Dev] My patches
In-Reply-To: <94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<20081030110442.1b6ce78e@resist.wooz.org>	<20081030150427.GA9631@amk-desktop.matrixgroup.net>	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
	<94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
Message-ID: <490AA9C2.403@gmail.com>

Paul Moore:
> 3. There's nothing obvious I can do to move an issue forward. Sure, I
>  can make a comment, but that's about it. I'd like something that
> stood a bit more chance of getting noticed (like a status change, or
> maybe a list of people who think this is good to apply, which I can
> add myself to).

Speaking for myself, I rarely spend any time going through the
tracker looking for items I can help handle*. So one of the things I
find *most* helpful is when someone adds me to the nosy list for a
tracker issue, since that will drop an email into my issue tracker
folder with a link straight to the issue.

Obviously I'd prefer it if people only did that when I really can help,
but I don't mind getting the occasional misdirected issue if it means
that more problems in my areas of expertise are brought directly to my
attention.

(*I only spend a fairly limited amount of time on python at the moment,
and keeping up with python-dev, python-3000, reviewing the two checkins
list and looking into items that are brought directly to my attention
fills that up pretty well)

Tarek Ziad? wrote:
> Since it is a hard and long process  "to know it all"  in Python, and
>  to become a core developer
> 
> What about having two level of devs ?
> 
> + core developers + standard library developers
> 
> I mean, the standard library could be open ihmo to a wider range of
> people, or maybe even having people specialized in some packages,
> modules, even if they don't know anything about the C apis of the
> core.
> 
> Those "standard library developers" could be blessed to work on 
> specific areas of the standard library and "followed" by a core
> developer that can just make sure everything goes in the right
> direction without having too much extra work for that.

I think you'll that most of the "core devs" have areas where we're
confident of our domain knowledge and are happy to accept and commit
patches, and other areas where we're willing to review things and
provide recommendations, and then areas where we have next to no
knowledge at all. The standard library is pretty huge, and outside a few
generally useful core modules, which sections of it are useful to you
depends greatly on what kind of application you're working on.

To use myself as an example, if a bug or patch relating to the compiler
, the import system, runpy, contextlib, functools or a few others areas
is brought to my attention, I'll generally consider myself competent to
both assess the problem and apply any fixes (I may not consider the
issue *urgent*, or else I'll want to spend some time just mulling it
over, so the tracker item may still hang around for a while, but I'll
get to them eventually).

Then there are a few other areas (such as threading and
multiprocessing), where I'll take an active interest in matters (because
it's code I either use, or am likely to use at some point), but usually
defer to others when it comes to making the actual commits. This is
particular so when areas have clear owners (e.g. Raymond for itertools,
Jesse for multiprocessing, Facundo/Raymond for decimal) or are fairly
complex and I know I haven't studied them enough to understand them
properly (e.g. the memory allocator).

And finally, there are large swathes of the standard library that I've
never even used, let alone looked at in a code editor (such as nearly
all of the network protocol and database libraries, as what I personally
use Python for tends to be much lower level than that). For those areas,
I don't really consider myself competent to even provide effective
review, so I tend to ignore those patches completely.

In the past, even Guido has explicitly deferred to the appropriate
subject matter experts when it came to areas of the standard library he
personally didn't use much.

So I'd suggest thinking about developer responsibilities more in terms
of areas of expertise rather than "levels" of developers. Those of us
that happen to understand the guts of the compiler or the VM aren't more
competent globally or more trusted than those maintaining the various
modules in the standard library - just interested in different things.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------


From g.brandl at gmx.net  Fri Oct 31 09:44:52 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 31 Oct 2008 08:44:52 +0000
Subject: [Python-Dev] hg branch gone?
In-Reply-To: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>
References: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>
Message-ID: <geed1t$uo3$1@ger.gmane.org>

Brett Cannon schrieb:
> I just tried to update my 3.0 branch in hg from
> http://code.python.org/hg/branches/py3k/ and hg is telling me it's a
> 404. Anyone else having trouble?

404 here too.

Since http://code.python.org/ serves the loggerhead Bazaar view, I suppose
the problem is that the loggerhead app greedily tries to serve all URLs
below code.python.org/.

Yet another hint at the evil force-Bazaar-upon-them conspiracy <wink>.

cheers,
Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From eric at trueblade.com  Fri Oct 31 08:50:58 2008
From: eric at trueblade.com (Eric Smith)
Date: Fri, 31 Oct 2008 03:50:58 -0400
Subject: [Python-Dev] hg branch gone?
In-Reply-To: <geed1t$uo3$1@ger.gmane.org>
References: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>
	<geed1t$uo3$1@ger.gmane.org>
Message-ID: <490AB8E2.9050004@trueblade.com>

Georg Brandl wrote:
> Brett Cannon schrieb:
>> I just tried to update my 3.0 branch in hg from
>> http://code.python.org/hg/branches/py3k/ and hg is telling me it's a
>> 404. Anyone else having trouble?
> 
> 404 here too.
> 
> Since http://code.python.org/ serves the loggerhead Bazaar view, I suppose
> the problem is that the loggerhead app greedily tries to serve all URLs
> below code.python.org/.
> 
> Yet another hint at the evil force-Bazaar-upon-them conspiracy <wink>.

I'm not so sure Bazaar isn't it's own victim :)

I posted this yesterday about using bzr:

 >I'd like to try it out, but the instructions on
 > http://www.python.org/dev/bazaar/ say to get wget
 > http://code.python.org/snapshots/python-bzr-snapshot.tar.bz2, which is
 > a 404. Anyone know the actual URL?


From rhamph at gmail.com  Fri Oct 31 08:54:14 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Fri, 31 Oct 2008 01:54:14 -0600
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <490AA4B9.5060307@gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
Message-ID: <aac2c7cb0810310054u48669de6rba56a093439f08f2@mail.gmail.com>

On Fri, Oct 31, 2008 at 12:24 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Neil Schemenauer wrote:
>> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>>> For a student project in a course on virtual machines, we are
>>> evaluating the possibility to experiment with removing the GIL
>>> from CPython
>>
>> Hi,
>>
>> It's great to hear of this kind of project.  I think what you want
>> to do is difficult but possible.  The major compilcation would be
>> that extension modules would have to re-written since they all
>> assume a reference counting GC.  A foreign function interface like
>> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
>> does place different demands on extension module authors.
>
> Michael Foord's comment about the way Ironclad does it suggests a
> possible interim step that would allow existing extensions to be used at
> the cost of some additional overhead: use free threading in the core,
> but have an "extension module lock" that cuts things back to a single
> thread whenever non-core code is invoked.

Ahh, or do it per object!  Convert the core to use a precise GC
references, but retain the refcount API as a sort of forced unknown
reference.  Needs bodging to retain our current handling of cycles in
refcounted objects, but it should be doable, and it potentially has a
lot less overhead (and is more scalable) than what I've got now with
safethread.


-- 
Adam Olsen, aka Rhamphoryncus

From ncoghlan at gmail.com  Fri Oct 31 09:11:10 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 31 Oct 2008 18:11:10 +1000
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <aac2c7cb0810310054u48669de6rba56a093439f08f2@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>	
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
	<aac2c7cb0810310054u48669de6rba56a093439f08f2@mail.gmail.com>
Message-ID: <490ABD9E.1040201@gmail.com>

Adam Olsen wrote:
> On Fri, Oct 31, 2008 at 12:24 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Neil Schemenauer wrote:
>>> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>>>> For a student project in a course on virtual machines, we are
>>>> evaluating the possibility to experiment with removing the GIL
>>>> from CPython
>>> Hi,
>>>
>>> It's great to hear of this kind of project.  I think what you want
>>> to do is difficult but possible.  The major compilcation would be
>>> that extension modules would have to re-written since they all
>>> assume a reference counting GC.  A foreign function interface like
>>> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
>>> does place different demands on extension module authors.
>> Michael Foord's comment about the way Ironclad does it suggests a
>> possible interim step that would allow existing extensions to be used at
>> the cost of some additional overhead: use free threading in the core,
>> but have an "extension module lock" that cuts things back to a single
>> thread whenever non-core code is invoked.
> 
> Ahh, or do it per object!  Convert the core to use a precise GC
> references, but retain the refcount API as a sort of forced unknown
> reference.  Needs bodging to retain our current handling of cycles in
> refcounted objects, but it should be doable, and it potentially has a
> lot less overhead (and is more scalable) than what I've got now with
> safethread.

The GIL protects more than just refcounting: it is used widely to
protect access to shared data structures both inside and outside the
core. Until all the extension modules implement their own locks instead
of relying on the GIL, some form of global lock needs to remain if you
want to preserve compatibility with existing extensions. And since being
able to use existing extensions is the main advantage of using CPython
rather than just writing a separate free-threaded Python VM, that seems
like a fairly important design goal.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------

From ziade.tarek at gmail.com  Fri Oct 31 09:18:07 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Fri, 31 Oct 2008 09:18:07 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <490AA9C2.403@gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<20081030110442.1b6ce78e@resist.wooz.org>
	<20081030150427.GA9631@amk-desktop.matrixgroup.net>
	<79990c6b0810300855r791fad95w940d5d2bc7d0abd6@mail.gmail.com>
	<94bdd2610810300909hf7c0337va779682982198b1d@mail.gmail.com>
	<490AA9C2.403@gmail.com>
Message-ID: <94bdd2610810310118y34b72d9ej6f40cbfc9216568f@mail.gmail.com>

On Fri, Oct 31, 2008 at 7:46 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> What about having two level of devs ?
>>
>> + core developers + standard library developers
>>
>> [cut]
>
>
> So I'd suggest thinking about developer responsibilities more in terms
> of areas of expertise rather than "levels" of developers. Those of us
> that happen to understand the guts of the compiler or the VM aren't more
> competent globally or more trusted than those maintaining the various
> modules in the standard library - just interested in different things.

Right,

I would like to share my experience about this, if it can be helpful.

I have focused so far in distutils, which, I believe
was not in the top priority of core developers during the last year.
(If this is not true
please forgive me).

Anyway, so I am starting to become quite specialized in this part of
Python, and
I pushed patches for it in the tracker.

The patches I wrote that made it so far took between 4 to 8 months to
be applied.

I have really simple patches for distutils that are just adding tests,
that are waiting for
review. I guess these patches will be reviewed in a few months, I am
failry confident
about that. I know core developers are drowned into more important topics.
And this area of Python is being highly discussed to be refactored, maybe
outside the stdlib at some point, but there are a *lot* of simple
things to do today in there.

So basically, if I want to be efficient in distutils I need to become
a core developer,
and this means (Guido said) I need to start providing patches for the
rest of the Python
code and eventually (after a few years I guess) maybe become a core developer.

Then I'll be able to work in distutils because at that point in the
future I'll be trusted.

I can't do that ! I don't have the time to become a Python core code expert.
But in my everyday work I became a packaging / deploying specialist.

So my point is : if I am "trusted" at some point in the work I am doing in
distutils, will I have a commit access there ?

If so, I 'll continue to focus on this package and to commit patches for it,
to try to gain that trust for sure. Otherwise I will need to work in a
third-party library
to be efficient, and stop working on patches for distutils.  And this
is, ihmo a bad thing
because this sdtlib package need some love.

Cheers
Tarek

From rhamph at gmail.com  Fri Oct 31 09:29:24 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Fri, 31 Oct 2008 02:29:24 -0600
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <490ABD9E.1040201@gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
	<aac2c7cb0810310054u48669de6rba56a093439f08f2@mail.gmail.com>
	<490ABD9E.1040201@gmail.com>
Message-ID: <aac2c7cb0810310129x5889400chf9cb106d4acd39d8@mail.gmail.com>

On Fri, Oct 31, 2008 at 2:11 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Adam Olsen wrote:
>> On Fri, Oct 31, 2008 at 12:24 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> Neil Schemenauer wrote:
>>>> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>>>>> For a student project in a course on virtual machines, we are
>>>>> evaluating the possibility to experiment with removing the GIL
>>>>> from CPython
>>>> Hi,
>>>>
>>>> It's great to hear of this kind of project.  I think what you want
>>>> to do is difficult but possible.  The major compilcation would be
>>>> that extension modules would have to re-written since they all
>>>> assume a reference counting GC.  A foreign function interface like
>>>> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
>>>> does place different demands on extension module authors.
>>> Michael Foord's comment about the way Ironclad does it suggests a
>>> possible interim step that would allow existing extensions to be used at
>>> the cost of some additional overhead: use free threading in the core,
>>> but have an "extension module lock" that cuts things back to a single
>>> thread whenever non-core code is invoked.
>>
>> Ahh, or do it per object!  Convert the core to use a precise GC
>> references, but retain the refcount API as a sort of forced unknown
>> reference.  Needs bodging to retain our current handling of cycles in
>> refcounted objects, but it should be doable, and it potentially has a
>> lot less overhead (and is more scalable) than what I've got now with
>> safethread.
>
> The GIL protects more than just refcounting: it is used widely to
> protect access to shared data structures both inside and outside the
> core. Until all the extension modules implement their own locks instead
> of relying on the GIL, some form of global lock needs to remain if you
> want to preserve compatibility with existing extensions. And since being
> able to use existing extensions is the main advantage of using CPython
> rather than just writing a separate free-threaded Python VM, that seems
> like a fairly important design goal.

safethread already handles that, via monitors.  The main thread is
given a monitor that all old, unshareable extension modules exist in.
Once they're updated and marked as shareable they can be used from any
monitor in any thread.


-- 
Adam Olsen, aka Rhamphoryncus

From martin at v.loewis.de  Fri Oct 31 09:45:40 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 31 Oct 2008 09:45:40 +0100
Subject: [Python-Dev] hg branch gone?
In-Reply-To: <490AB8E2.9050004@trueblade.com>
References: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>	<geed1t$uo3$1@ger.gmane.org>
	<490AB8E2.9050004@trueblade.com>
Message-ID: <490AC5B4.6080507@v.loewis.de>

>> Yet another hint at the evil force-Bazaar-upon-them conspiracy <wink>.
> 
> I'm not so sure Bazaar isn't it's own victim :)

I have now restored the original URL structure, and moved the loggerhead
installation to

http://code.python.org/loggerhead/

Regards,
Martin

From barry at python.org  Fri Oct 31 09:58:30 2008
From: barry at python.org (Barry Warsaw)
Date: Fri, 31 Oct 2008 08:58:30 +0000
Subject: [Python-Dev] hg branch gone?
In-Reply-To: <geed1t$uo3$1@ger.gmane.org>
References: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>
	<geed1t$uo3$1@ger.gmane.org>
Message-ID: <20081031085830.69353e6c@resist.wooz.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 31, 2008, at 08:44 AM, Georg Brandl wrote:

>Brett Cannon schrieb:
>> I just tried to update my 3.0 branch in hg from
>> http://code.python.org/hg/branches/py3k/ and hg is telling me it's a
>> 404. Anyone else having trouble?
>
>404 here too.
>
>Since http://code.python.org/ serves the loggerhead Bazaar view, I suppose
>the problem is that the loggerhead app greedily tries to serve all URLs
>below code.python.org/.
>
>Yet another hint at the evil force-Bazaar-upon-them conspiracy <wink>.

Naw, I think I just f'd up the Apache conf.  I'll try to fix that.

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkkKyLYACgkQ2YZpQepbvXG2pQCgh61b5hA7ECT5mNO+vbmJICYR
tRoAoJEa3XGzM06qNepsnhHldzLPF4kG
=8JK8
-----END PGP SIGNATURE-----

From barry at python.org  Fri Oct 31 10:10:43 2008
From: barry at python.org (Barry Warsaw)
Date: Fri, 31 Oct 2008 09:10:43 +0000
Subject: [Python-Dev] hg branch gone?
In-Reply-To: <20081031085830.69353e6c@resist.wooz.org>
References: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>
	<geed1t$uo3$1@ger.gmane.org>
	<20081031085830.69353e6c@resist.wooz.org>
Message-ID: <20081031091043.2fbc0ac0@resist.wooz.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 31, 2008, at 08:58 AM, Barry Warsaw wrote:

>Naw, I think I just f'd up the Apache conf.  I'll try to fix that.

And by "I'll" of course I meant "Martin". :)

Thanks Martin!
- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkkKy5MACgkQ2YZpQepbvXF8PgCdGou6DzXzBfOhZ4QvtJo3oQY4
QKMAmQECJQhxOxqLU1Og7K9OPVZ3VXvN
=BdOC
-----END PGP SIGNATURE-----

From barry at python.org  Fri Oct 31 11:25:24 2008
From: barry at python.org (Barry Warsaw)
Date: Fri, 31 Oct 2008 10:25:24 +0000
Subject: [Python-Dev] hg branch gone?
In-Reply-To: <490AB8E2.9050004@trueblade.com>
References: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>
	<geed1t$uo3$1@ger.gmane.org> <490AB8E2.9050004@trueblade.com>
Message-ID: <20081031102524.75dfd13f@resist.wooz.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Oct 31, 2008, at 03:50 AM, Eric Smith wrote:

>I posted this yesterday about using bzr:
>
> >I'd like to try it out, but the instructions on
> > http://www.python.org/dev/bazaar/ say to get wget
> > http://code.python.org/snapshots/python-bzr-snapshot.tar.bz2, which is
> > a 404. Anyone know the actual URL?

Looks like this is fixed now too.

Aside: Now that Python 2.6 isn't on the trunk, we need to add a bzr mirror for
python26-maint.

- -Barry
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)

iEYEARECAAYFAkkK3RQACgkQ2YZpQepbvXFDYQCgolodW5cMVjGPUzLRyK2vsaHE
2dUAoKJzwXNVmyNBzgFSSgnMS4dPx7Mp
=LD00
-----END PGP SIGNATURE-----

From skip at pobox.com  Fri Oct 31 11:28:00 2008
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 31 Oct 2008 05:28:00 -0500
Subject: [Python-Dev] hg branch gone?
In-Reply-To: <490AC5B4.6080507@v.loewis.de>
References: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>
	<geed1t$uo3$1@ger.gmane.org> <490AB8E2.9050004@trueblade.com>
	<490AC5B4.6080507@v.loewis.de>
Message-ID: <18698.56752.992826.599106@montanaro-dyndns-org.local>


    Martin> I have now restored the original URL structure, and moved the
    Martin> loggerhead installation to

    Martin> http://code.python.org/loggerhead/

A couple nits.  Leaving off the trailing / yields a 404.  (No big deal
though).  More importantly, there seem to be no images, e.g.:

    http://code.python.org/static/images/ico_folder.gif

Looks like it should be

    http://code.python.org/loggerhead/static/images/ico_folder.gif

Skip


From martin at v.loewis.de  Fri Oct 31 11:44:24 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 31 Oct 2008 11:44:24 +0100
Subject: [Python-Dev] hg branch gone?
In-Reply-To: <18698.56752.992826.599106@montanaro-dyndns-org.local>
References: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>
	<geed1t$uo3$1@ger.gmane.org> <490AB8E2.9050004@trueblade.com>
	<490AC5B4.6080507@v.loewis.de>
	<18698.56752.992826.599106@montanaro-dyndns-org.local>
Message-ID: <490AE188.1050402@v.loewis.de>

> A couple nits.  Leaving off the trailing / yields a 404.  (No big deal
> though).  More importantly, there seem to be no images, e.g.:
> 
>     http://code.python.org/static/images/ico_folder.gif
> 
> Looks like it should be
> 
>     http://code.python.org/loggerhead/static/images/ico_folder.gif

I'll leave these to Barry; the loggerhead installation is still not
complete (so it's not yet an official service).

Regards,
Martin

From martin at v.loewis.de  Fri Oct 31 12:29:18 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 31 Oct 2008 12:29:18 +0100
Subject: [Python-Dev] hg branch gone?
In-Reply-To: <18698.56752.992826.599106@montanaro-dyndns-org.local>
References: <bbaeab100810302055s654f7ee1k213b5e9acf04e877@mail.gmail.com>
	<geed1t$uo3$1@ger.gmane.org> <490AB8E2.9050004@trueblade.com>
	<490AC5B4.6080507@v.loewis.de>
	<18698.56752.992826.599106@montanaro-dyndns-org.local>
Message-ID: <490AEC0E.7030807@v.loewis.de>

> Looks like it should be
> 
>     http://code.python.org/loggerhead/static/images/ico_folder.gif

I did a quick hack for the static files, but many of the links still
don't work. loggerhead has specific support for reverse proxies
(through loggerhead.conf), but I couldn't figure out how to activate
that.

Regards,
Martin

From ironfroggy at gmail.com  Fri Oct 31 12:59:01 2008
From: ironfroggy at gmail.com (Calvin Spealman)
Date: Fri, 31 Oct 2008 07:59:01 -0400
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <490AA4B9.5060307@gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
Message-ID: <76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>

Has anyone made the argument for keeping the GIL to discourage
threading? I'm only throwing this out there and I'm sure we'd want to
improve things no matter what, but I would like to voice the concern
anyway. We all know there are people who think threading is the answer
to all things, and who don't understand the difficulties and problems
with threading. Often, when trying to persuade them of alternative
solutions, one selling point is "your python threads can't take
advantage of multiple cores, anyway, because of the GIL."

I worry, if it is ever actually removed, that I won't be able to
convince these people that you really don't need to spawn twenty
threads to write an IRC bot.

On Fri, Oct 31, 2008 at 2:24 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Neil Schemenauer wrote:
>> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>>> For a student project in a course on virtual machines, we are
>>> evaluating the possibility to experiment with removing the GIL
>>> from CPython
>>
>> Hi,
>>
>> It's great to hear of this kind of project.  I think what you want
>> to do is difficult but possible.  The major compilcation would be
>> that extension modules would have to re-written since they all
>> assume a reference counting GC.  A foreign function interface like
>> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
>> does place different demands on extension module authors.
>
> Michael Foord's comment about the way Ironclad does it suggests a
> possible interim step that would allow existing extensions to be used at
> the cost of some additional overhead: use free threading in the core,
> but have an "extension module lock" that cuts things back to a single
> thread whenever non-core code is invoked.
>
> It wouldn't be as fast as a completely free-threaded setup (since the
> EML would need to operate as a variant of a 'read/write' lock, with the
> core code mostly leaving it in the multiple-threads-allowed 'read' mode,
> promoting it to single-thread-only 'write' mode when potentially
> dispatching to non-core code, and dropping back to the 'read' mode when
> the dispatch landed back in core code again).
>
> Over time, extension modules which were updated to use their own
> internal locks could also drop back to free-threading mode. In essence,
> the GIL would still exist, but instead of being a simple mutex the way
> it is now, it would have an intermediate stage which allowed multiple
> threads to continue running. Once most extension modules had been
> converted, the function to promote the EML from multi-threaded mode to
> single-thread only mode could be removed.
>
> A mechanism like that would avoid one of the more subtle problems in
> removing the GIL: currently, CPython-specific Python code can assume
> that invocation of a C function from Python code is an atomic operation,
> since the body of the function executes as a single bytecode (we're
> talking about the case here where the C function is known not to release
> the GIL).
>
> I've written code which relied on the GIL in exactly that way, and I'm
> sure I'm far from the only Python programmer to do so. If the GIL was
> removed without adding an extension module lock, such applications would
> require additional locking either in the C extension module or in the
> Python code.
>
> A possible starting point for such a project: move the locking
> mechanisms that are implemented in Python in the threading module to
> C-based implementations in _thread so that a richer low overhead locking
> arrangement is available to extensions modules (and the rest of the core).
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
>



-- 
Read my blog! I depend on your acceptance of my opinion! I am interesting!
http://techblog.ironfroggy.com/
Follow me if you're into that sort of thing: http://www.twitter.com/ironfroggy

From phd at phd.pp.ru  Fri Oct 31 13:35:00 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Fri, 31 Oct 2008 15:35:00 +0300
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
	<76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
Message-ID: <20081031123500.GA14435@phd.pp.ru>

On Fri, Oct 31, 2008 at 07:59:01AM -0400, Calvin Spealman wrote:
> Has anyone made the argument for keeping the GIL to discourage
> threading?

   I haven't, but I would support such argument. In my humble opinion
multithreading is rarely a good answer, and for multicore CPUs it's a wrong
answer. For multicore CPUs the answer is multiprocessing + IPC (or a good
database; filesystem is not a good database because it requires locking).
   One big problem with multithreading - it violates "divide et conqueror"
principle. In programming almost everything is about separation of access
rights - namespaces, modules, etc. Multithreading model allows a number of
processes to share memory.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From lists at cheimes.de  Fri Oct 31 14:01:55 2008
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 31 Oct 2008 14:01:55 +0100
Subject: [Python-Dev] Proper initialization of structs
In-Reply-To: <acd65fa20810301710o282e1b64jdae15188868b1e93@mail.gmail.com>
References: <gecfr4$5p1$1@ger.gmane.org>	<05D72249-2D59-47F2-BE26-FF9DC3541958@acm.org>	<acd65fa20810301523s4cbd8b6ex34a06162a59d23e4@mail.gmail.com>	<490A46B2.6060207@cheimes.de>
	<acd65fa20810301710o282e1b64jdae15188868b1e93@mail.gmail.com>
Message-ID: <490B01C3.8030903@cheimes.de>

Alexandre Vassalotti wrote:
> But what if  PyType_GenericAlloc is used for tp_alloc? As far as I
> know, the memory block allocated with PyType_GenericAlloc is zeroed.

Oh, you are right. The generic allocator uses memset to initialize the
block with \0.

Christian


From lists at cheimes.de  Fri Oct 31 14:13:01 2008
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 31 Oct 2008 14:13:01 +0100
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <490AA5FD.80105@gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>	<gecpks$cgc$1@ger.gmane.org>	<gecqd4$g9g$1@ger.gmane.org>	<ca471dc20810301025p47843333yff585b8507d9d521@mail.gmail.com>	<4909EF67.5010104@trueblade.com>	<ca471dc20810301034x7d6adb88wb875100e648d5c28@mail.gmail.com>	<4909FECF.7000703@voidspace.org.uk>
	<490AA5FD.80105@gmail.com>
Message-ID: <gef08u$llj$1@ger.gmane.org>

Nick Coghlan wrote:
> If I recall correctly, the main blocker to ctypes portability is libffi
> portability. So if anyone would like to see ctypes on more platforms,
> then that's the limitation they really need to attack.

ctypes is also missing utilities to write code that works on 32 and 
64bit platforms. Without a tool like 
http://pypi.python.org/pypi/ctypes_configure it's very hard and tedious 
to avoid and fix segfaults.

Christian


From fuzzyman at voidspace.org.uk  Fri Oct 31 14:17:56 2008
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 31 Oct 2008 13:17:56 +0000
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>	<gedmtg$eop$1@ger.gmane.org>
	<490AA4B9.5060307@gmail.com>
	<76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
Message-ID: <490B0584.7080006@voidspace.org.uk>

Calvin Spealman wrote:
> Has anyone made the argument for keeping the GIL to discourage
> threading? I'm only throwing this out there and I'm sure we'd want to
> improve things no matter what, but I would like to voice the concern
> anyway. We all know there are people who think threading is the answer
> to all things, and who don't understand the difficulties and problems
> with threading. Often, when trying to persuade them of alternative
> solutions, one selling point is "your python threads can't take
> advantage of multiple cores, anyway, because of the GIL."
>
>   

There are many concurrency problems for which multiprocessing is *not* a 
good answer and for which careful use of threads is a relatively 
straightforward answer. For people with those sorts of problems CPython 
(and therefore the perception is Python) does not offer a usable 
concurrency story unless you are prepared to also write in C.

Unfortunately many in the Python community seem pathologically opposed 
to even good use of threading and refuse to believe that such situations 
even exist. In the meantime, people with these problems (or who believe 
they are likely to have such problems) simply turn to other languages.

All the best,

Michael Foord

> I worry, if it is ever actually removed, that I won't be able to
> convince these people that you really don't need to spawn twenty
> threads to write an IRC bot.
>
> On Fri, Oct 31, 2008 at 2:24 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>   
>> Neil Schemenauer wrote:
>>     
>>> Sigurd Torkel Meldgaard <stm at daimi.au.dk> wrote:
>>>       
>>>> For a student project in a course on virtual machines, we are
>>>> evaluating the possibility to experiment with removing the GIL
>>>> from CPython
>>>>         
>>> Hi,
>>>
>>> It's great to hear of this kind of project.  I think what you want
>>> to do is difficult but possible.  The major compilcation would be
>>> that extension modules would have to re-written since they all
>>> assume a reference counting GC.  A foreign function interface like
>>> CMU Lisp's "alien" or GHC's FFI is not necessarily any worse but it
>>> does place different demands on extension module authors.
>>>       
>> Michael Foord's comment about the way Ironclad does it suggests a
>> possible interim step that would allow existing extensions to be used at
>> the cost of some additional overhead: use free threading in the core,
>> but have an "extension module lock" that cuts things back to a single
>> thread whenever non-core code is invoked.
>>
>> It wouldn't be as fast as a completely free-threaded setup (since the
>> EML would need to operate as a variant of a 'read/write' lock, with the
>> core code mostly leaving it in the multiple-threads-allowed 'read' mode,
>> promoting it to single-thread-only 'write' mode when potentially
>> dispatching to non-core code, and dropping back to the 'read' mode when
>> the dispatch landed back in core code again).
>>
>> Over time, extension modules which were updated to use their own
>> internal locks could also drop back to free-threading mode. In essence,
>> the GIL would still exist, but instead of being a simple mutex the way
>> it is now, it would have an intermediate stage which allowed multiple
>> threads to continue running. Once most extension modules had been
>> converted, the function to promote the EML from multi-threaded mode to
>> single-thread only mode could be removed.
>>
>> A mechanism like that would avoid one of the more subtle problems in
>> removing the GIL: currently, CPython-specific Python code can assume
>> that invocation of a C function from Python code is an atomic operation,
>> since the body of the function executes as a single bytecode (we're
>> talking about the case here where the C function is known not to release
>> the GIL).
>>
>> I've written code which relied on the GIL in exactly that way, and I'm
>> sure I'm far from the only Python programmer to do so. If the GIL was
>> removed without adding an extension module lock, such applications would
>> require additional locking either in the C extension module or in the
>> Python code.
>>
>> A possible starting point for such a project: move the locking
>> mechanisms that are implemented in Python in the threading module to
>> C-based implementations in _thread so that a richer low overhead locking
>> arrangement is available to extensions modules (and the rest of the core).
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> ---------------------------------------------------------------
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ironfroggy%40gmail.com
>>
>>     
>
>
>
>   


-- 
http://www.ironpythoninaction.com/


From victor.stinner at haypocalc.com  Fri Oct 31 14:32:22 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Fri, 31 Oct 2008 14:32:22 +0100
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <gef08u$llj$1@ger.gmane.org>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<490AA5FD.80105@gmail.com> <gef08u$llj$1@ger.gmane.org>
Message-ID: <200810311432.22938.victor.stinner@haypocalc.com>

Le Friday 31 October 2008 14:13:01 Christian Heimes, vous avez ?crit?:
> ctypes is also missing utilities to write code that works on 32 and
> 64bit platforms. Without a tool like
> http://pypi.python.org/pypi/ctypes_configure it's very hard and tedious
> to avoid and fix segfaults.

I wrote some ctypes binding and some ctypes tools are missing. The most 
important lacks are:
 - get_errno()
 - size_t and <stdint.h> types

You may be interrested by my modules:

get errno value:
http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_errno.py
(ctypes_support has a get_errno function)

open the C library:
http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_libc.py
(ok, code is trivial, it's not critical to integrate it to ctypes)

*size_t* and <stdint.h> types!
http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_stdint.py
(my module is incomplete, some types are missing)

Some functions to ease ctypes development:
http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_tools.py
 - bytes2type() and bytes2array() are useful to convert a bytes array
   to a ctypes object

Should I open issues?

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From jnoller at gmail.com  Fri Oct 31 15:03:57 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Fri, 31 Oct 2008 10:03:57 -0400
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <490B0584.7080006@voidspace.org.uk>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
	<76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
	<490B0584.7080006@voidspace.org.uk>
Message-ID: <4222a8490810310703i6058fbf1j1b4f9dea511d3b77@mail.gmail.com>

On Fri, Oct 31, 2008 at 9:17 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
> Calvin Spealman wrote:
>>
>> Has anyone made the argument for keeping the GIL to discourage
>> threading? I'm only throwing this out there and I'm sure we'd want to
>> improve things no matter what, but I would like to voice the concern
>> anyway. We all know there are people who think threading is the answer
>> to all things, and who don't understand the difficulties and problems
>> with threading. Often, when trying to persuade them of alternative
>> solutions, one selling point is "your python threads can't take
>> advantage of multiple cores, anyway, because of the GIL."
>>
>>
>
> There are many concurrency problems for which multiprocessing is *not* a
> good answer and for which careful use of threads is a relatively
> straightforward answer. For people with those sorts of problems CPython (and
> therefore the perception is Python) does not offer a usable concurrency
> story unless you are prepared to also write in C.
>
> Unfortunately many in the Python community seem pathologically opposed to
> even good use of threading and refuse to believe that such situations even
> exist. In the meantime, people with these problems (or who believe they are
> likely to have such problems) simply turn to other languages.
>
> All the best,
>
> Michael Foord

There's room in python for both (all).

Threading, as given to us by Java and others is hard to get right and
is used by many because its "just there", but it is *still* useful for
many people. If python were to have free threading, courtesy of a lack
of the GIL, it would help those people quite a bit. Sometimes you just
need shared state. Myself? I used multiprocess *and* threads all the
time for various reasons.

Of course: Does shared state make more sense as implemented via the
Erlang green threads system - wherein threads are independent of one
another, but pass messages between one another via the parent? (Van
and I were talking about this for some time yesterday :))

But fundamentally - there's still an attraction to things like
IPC/Multiprocessing, Actors, message passing/etc as well. I don't see
free threading and other language improvements for additional styles
of concurrent/distributed programming as mutually exclusive.

Excluding the possibility of free-threading because "we don't like it"
is sorta silly. It would be equivalent of blocking aynsc libraries for
the same reason.

All of that being said: I think there's room to aim for a *better*
threading model, or even a series of higher level abstractions (on top
of the basic abstractions threading provides on top of thread for
example). For example: if you were building a Java application today,
would you use the basic java.lang.thread classes and all of the other
locking primitives, or would you go and use the abstractions and
utilities in the java.util.concurrent package?

Most people would reply with the latter: The language now provides to
you a proven set of tools which have been "proven" to work, are "safe"
- it doesn't make sense for them to go and build out their own thread
factory and what not.

Imagine a concurrent package for python: in it you have your basics -
threading, multiprocessing (maybe "acting" too, but that name sucks,
how about greenprocesses? :)) and you also have utilities like
multiprocessing.pool, multiprocessing.manager/etc,
threading.producer_consumer, etc. We build these out and make them "as
safe as possible".

Over time, we begin to hide more and more of the primitives and
encourage usage of safer, proven abstractions. Take a look at
something like Kamaelia - it's a dog pile of abstractors and metaphors
that make it dead simple to do concurrent things without exposing the
end-programmer to a lot of the underlying pain and basics. Sure: If
I'm and end user I have to decide "threads, processes or green
processes" but that's really an extension of the problem domain.
Threads aren't good for some problems, processes are bad for others,
etc.

Just some pie-in-the-sky thoughts. Fundamentally, the GIL is not an
end-user "feature" unless that end user is writing extension modules -
and in that case, it's a pretty nice feature. However end users
*benefit* from the GIL by having a rich standard library and a much
more rich ecosystem of 3rd party modules - and that benefit can't be
understated or sacrificed.

-jesse

From skip at pobox.com  Fri Oct 31 15:07:54 2008
From: skip at pobox.com (skip at pobox.com)
Date: Fri, 31 Oct 2008 09:07:54 -0500
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
	<76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
Message-ID: <18699.4410.316065.90256@montanaro-dyndns-org.local>


    Calvin> Has anyone made the argument for keeping the GIL to discourage
    Calvin> threading? I'm only throwing this out there and I'm sure we'd
    Calvin> want to improve things no matter what, but I would like to voice
    Calvin> the concern anyway. We all know there are people who think
    Calvin> threading is the answer to all things, and who don't understand
    Calvin> the difficulties and problems with threading. Often, when trying
    Calvin> to persuade them of alternative solutions, one selling point is
    Calvin> "your python threads can't take advantage of multiple cores,
    Calvin> anyway, because of the GIL."

I think the "we are all consenting adults here" rubric applies.  As Michael
Foord indicated there are tasks for which multithreading is appropriate.  If
people want to shoot themselves in the foot with threads that's their
prerogative.

Skip

From theller at ctypes.org  Fri Oct 31 15:54:05 2008
From: theller at ctypes.org (Thomas Heller)
Date: Fri, 31 Oct 2008 15:54:05 +0100
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <gef08u$llj$1@ger.gmane.org>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>	<gecpks$cgc$1@ger.gmane.org>	<gecqd4$g9g$1@ger.gmane.org>	<ca471dc20810301025p47843333yff585b8507d9d521@mail.gmail.com>	<4909EF67.5010104@trueblade.com>	<ca471dc20810301034x7d6adb88wb875100e648d5c28@mail.gmail.com>	<4909FECF.7000703@voidspace.org.uk>	<490AA5FD.80105@gmail.com>
	<gef08u$llj$1@ger.gmane.org>
Message-ID: <gef66b$apv$1@ger.gmane.org>

Christian Heimes schrieb:
> Nick Coghlan wrote:
>> If I recall correctly, the main blocker to ctypes portability is libffi
>> portability. So if anyone would like to see ctypes on more platforms,
>> then that's the limitation they really need to attack.
> 
> ctypes is also missing utilities to write code that works on 32 and 
> 64bit platforms. Without a tool like 
> http://pypi.python.org/pypi/ctypes_configure it's very hard and tedious 
> to avoid and fix segfaults.

There's a difference between 'ctypes does not compile or work on my platform'
and 'how to write portable code with ctypes'.

Of course I would very much welcome patches for the first issue,
plus tools or other helpers for the second.

-- 
Thanks,
Thomas


From theller at ctypes.org  Fri Oct 31 16:04:18 2008
From: theller at ctypes.org (Thomas Heller)
Date: Fri, 31 Oct 2008 16:04:18 +0100
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <200810311432.22938.victor.stinner@haypocalc.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>	<490AA5FD.80105@gmail.com>
	<gef08u$llj$1@ger.gmane.org>
	<200810311432.22938.victor.stinner@haypocalc.com>
Message-ID: <gef6pi$d59$1@ger.gmane.org>

Victor Stinner schrieb:
> Le Friday 31 October 2008 14:13:01 Christian Heimes, vous avez ?crit :
>> ctypes is also missing utilities to write code that works on 32 and
>> 64bit platforms. Without a tool like
>> http://pypi.python.org/pypi/ctypes_configure it's very hard and tedious
>> to avoid and fix segfaults.
> 
> I wrote some ctypes binding and some ctypes tools are missing. The most 
> important lacks are:
>  - get_errno()
>  - size_t and <stdint.h> types
> 
> You may be interrested by my modules:

Some of the problems you list here have already been fixed in Python 2.6:

> 
> get errno value:
> http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_errno.py
> (ctypes_support has a get_errno function)

A new calling convention has been added for *safe* access to errno.

> open the C library:
> http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_libc.py
> (ok, code is trivial, it's not critical to integrate it to ctypes)

This code is not in ctypes, but with Python2.6 it will also work on windows.

> *size_t* and <stdint.h> types!
> http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_stdint.py
> (my module is incomplete, some types are missing)

Well, the signed types are named c_int8, c_uint8, and so on.
size_t is named c_size_t, please look into Lib/ctypes/__init__.py.
IIRC, these types are present event longer.

> Some functions to ease ctypes development:
> http://python-ptrace.hachoir.org/trac/browser/trunk/ptrace/ctypes_tools.py
>  - bytes2type() and bytes2array() are useful to convert a bytes array
>    to a ctypes object

There are also new ctypes methods in Python 2.6 .from_buffer() and .from_buffer_copy()
which should cover this use case.

An example from your module:

> 124 	def bytes2type(bytes, type):
> 125 	    """
> 126 	    Cast a bytes string to an objet of the specified type.
> 127 	    """
> 128 	    return cast(bytes, POINTER(type))[0]
> 129 	

Personally I would prefer to spell out 'cast(bytes, POINTER(type))[0]' instead of
documenting and learning what bytes2type() does.

> Should I open issues?
> 

Please decide yourself.

Thanks,
Thomas


From guido at python.org  Fri Oct 31 16:07:17 2008
From: guido at python.org (Guido van Rossum)
Date: Fri, 31 Oct 2008 08:07:17 -0700
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
	<76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
Message-ID: <ca471dc20810310807t63d6563cxabd1577d886eca1@mail.gmail.com>

On Fri, Oct 31, 2008 at 4:59 AM, Calvin Spealman <ironfroggy at gmail.com> wrote:
> Has anyone made the argument for keeping the GIL to discourage
> threading?

Oooh, you are on to my secret plan! :-)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From victor.stinner at haypocalc.com  Fri Oct 31 17:46:21 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Fri, 31 Oct 2008 17:46:21 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <bbaeab100810301937u3b066ff6q872671a626a970a0@mail.gmail.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>
	<200810310127.35945.victor.stinner@haypocalc.com>
	<bbaeab100810301937u3b066ff6q872671a626a970a0@mail.gmail.com>
Message-ID: <200810311746.21794.victor.stinner@haypocalc.com>

> I have a similar list that I have been thinking about proposing. I did
> a blog post about it at
> http://sayspy.blogspot.com/2008/08/what-are-typical-steps-issue-goes.html
> and received positive feedback:
>
> * triage
> * verify bug
> * test needed
> * needs patch
> * patch review
> * commit review
> * committed/rejected

Would it be possible to add a new field (a combo box), let say "stage" 
(or "step")?

But we need also some hooks to automate stage changes:
 * attach a new patch => set stage to "patch review"
   (and maybe add the keyword "review")
 * remove "patch reviewed" keyword => set stage to "commit review"
 * set status to "closed" => set stage to "commited/rejected"

We can maybe use JavaScript for some operations (it's easy to catch a combobox 
change).

About the old issues, is it possible to write a script setting the stage using 
the same rules (using the keywords and status)?

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From martin at v.loewis.de  Fri Oct 31 17:54:59 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 31 Oct 2008 17:54:59 +0100
Subject: [Python-Dev] My patches
In-Reply-To: <200810311746.21794.victor.stinner@haypocalc.com>
References: <200810301108.35591.victor.stinner@haypocalc.com>	<200810310127.35945.victor.stinner@haypocalc.com>	<bbaeab100810301937u3b066ff6q872671a626a970a0@mail.gmail.com>
	<200810311746.21794.victor.stinner@haypocalc.com>
Message-ID: <490B3863.1040104@v.loewis.de>

> But we need also some hooks to automate stage changes:
>  * attach a new patch => set stage to "patch review"
>    (and maybe add the keyword "review")

The latter already happens automatically.

>  * remove "patch reviewed" keyword => set stage to "commit review"
>  * set status to "closed" => set stage to "commited/rejected"
> 
> We can maybe use JavaScript for some operations (it's easy to catch a combobox 
> change).

Automatic state changes are best done with roundup detectors - see the
detectors directory of the python-dev instance to get an idea of what
can be done (I used to think these things should be "react" actions,
but making them "audit" actions actually looks nicer in the history)

> About the old issues, is it possible to write a script setting the stage using 
> the same rules (using the keywords and status)?

Most certainly. See the scripts directory of the python-dev instance for
examples.

Regards,
Martin

P.S. These things might better belong to tracker-discuss.


From status at bugs.python.org  Fri Oct 31 18:06:37 2008
From: status at bugs.python.org (Python tracker)
Date: Fri, 31 Oct 2008 18:06:37 +0100 (CET)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20081031170637.089BC7829A@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (10/24/08 - 10/31/08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2153 open (+33) / 13915 closed (+19) / 16068 total (+52)

Open issues with patches:   705

Average duration of open issues: 710 days.
Median duration of open issues: 1951 days.

Open Issues Breakdown
   open  2137 (+33)
pending    16 ( +0)

Issues Created Or Reopened (52)
_______________________________

Miserable subprocess.Popen performance                           10/24/08
       http://bugs.python.org/issue4194    created  skip.montanaro            
                                                                               

Regression for executing packages                                10/24/08
       http://bugs.python.org/issue4195    created  vajda                     
       patch                                                                   

library documentation errors                                     10/24/08
CLOSED http://bugs.python.org/issue4196    created  LambertDW                 
                                                                               

Doctest module does not work with zipped packages                10/24/08
       http://bugs.python.org/issue4197    created  belopolsky                
       patch                                                                   

os.path.normcase gets fooled on windows with mapped linux networ 10/24/08
       http://bugs.python.org/issue4198    created  dawidjoubert              
                                                                               

add shorthand global and nonlocal statements                     10/24/08
       http://bugs.python.org/issue4199    created  benjamin.peterson         
       patch                                                                   

atexit module not safe in Python 3.0 with multiple interpreters  10/24/08
CLOSED http://bugs.python.org/issue4200    created  grahamd                   
       patch, needs review                                                     

Pdb cannot access source code in zipped packages.                10/24/08
       http://bugs.python.org/issue4201    created  belopolsky                
       patch                                                                   

Multiple interpreters and readline module hook functions.        10/25/08
       http://bugs.python.org/issue4202    created  grahamd                   
                                                                               

adapt sphinx-quickstart for windows                              10/25/08
       http://bugs.python.org/issue4203    created  timmie                    
                                                                               

Cannot build _multiprocessing, math, mmap and readline of Python 10/25/08
       http://bugs.python.org/issue4204    created  akitada                   
       patch                                                                   

unexpected str.__init__() behaviour on b''                       10/25/08
CLOSED http://bugs.python.org/issue4205    created  keldonin                  
                                                                               

multiprocessing docs                                             10/26/08
       http://bugs.python.org/issue4206    created  LambertDW                 
                                                                               

Remove backwards compatibility in _heapq for performance         10/26/08
CLOSED http://bugs.python.org/issue4207    created  krisvale                  
       patch, patch, easy                                                      

Make multiprocessing compatible with Python 2.4 and 2.5          10/26/08
       http://bugs.python.org/issue4208    created  christian.heimes          
       patch, patch                                                            

from __future__ import unicode_literals, print_function doesn't  10/26/08
CLOSED http://bugs.python.org/issue4209    created  benjamin.peterson         
       patch                                                                   

os.remove can be used to remove directories on SunOS             10/26/08
CLOSED http://bugs.python.org/issue4210    created  giampaolo.rodola          
                                                                               

frozen packages set an improper __path__ value                   10/27/08
       http://bugs.python.org/issue4211    created  brett.cannon              
                                                                               

email.LazyImporter does not use absolute imports                 10/27/08
       http://bugs.python.org/issue4212    created  brandonbloom              
                                                                               

Lower case file system encoding                                  10/27/08
CLOSED http://bugs.python.org/issue4213    created  christian.heimes          
       patch                                                                   

no extension debug info with msvc9compiler.py                    10/27/08
       http://bugs.python.org/issue4214    created  robind                    
                                                                               

Running Python 2.6 GUI on Windows Vista                          10/27/08
       http://bugs.python.org/issue4215    created  Elushin                   
                                                                               

subprocess.Popen hangs at communicate() when child exits         10/27/08
       http://bugs.python.org/issue4216    created  amy20_z                   
                                                                               

Add file comparisons to the unittest library                     10/28/08
       http://bugs.python.org/issue4217    created  wehart                    
                                                                               

tarfile module fails to correctly parse some .tar.gz archives?!  10/28/08
CLOSED http://bugs.python.org/issue4218    created  gerhard_xXx               
                                                                               

Problem with regular expression                                  10/28/08
CLOSED http://bugs.python.org/issue4219    created  carlosklock               
                                                                               

Builtins treated as free variables?                              10/28/08
       http://bugs.python.org/issue4220    created  dfugate                   
                                                                               

inconsistent exception from int is confusing                     10/28/08
       http://bugs.python.org/issue4221    created  exarkun                   
                                                                               

dis.findlinestarts is missing from dis.__all__ and from the onli 10/28/08
       http://bugs.python.org/issue4222    created  exarkun                   
                                                                               

inspect.getsource doesn't work on functions imported from a zipf 10/28/08
       http://bugs.python.org/issue4223    created  exarkun                   
                                                                               

** operating incorrectly for negative bases?                     10/28/08
CLOSED http://bugs.python.org/issue4224    created  martin.speleo             
                                                                               

unicode_literals doesn't work in exec                            10/28/08
CLOSED http://bugs.python.org/issue4225    created  benjamin.peterson         
       patch, needs review                                                     

Python core crashes with associated files.                       10/30/08
       http://bugs.python.org/issue4226    reopened christian.heimes          
                                                                               

unicode_literals and print_function do not work together.        10/29/08
CLOSED http://bugs.python.org/issue4227    created  wangchun                  
                                                                               

struct.pack('L', -1)                                             10/29/08
       http://bugs.python.org/issue4228    created  arigo                     
                                                                               

Can't Start Python                                               10/29/08
CLOSED http://bugs.python.org/issue4229    created  CliffB                    
                                                                               

"__getattr__" can't be a descriptor                              10/29/08
       http://bugs.python.org/issue4230    created  arigo                     
                                                                               

print "    should be   print("                                   10/29/08
CLOSED http://bugs.python.org/issue4231    created  LambertDW                 
                                                                               

tempfile.gettempdir() availability not documented                10/29/08
CLOSED http://bugs.python.org/issue4232    created  scop                      
                                                                               

open(0, closefd=False) prints 3 warnings                         10/30/08
       http://bugs.python.org/issue4233    created  amaury.forgeotdarc        
                                                                               

bin missing from documentation                                   10/30/08
CLOSED http://bugs.python.org/issue4234    created  v+python                  
       patch                                                                   

Crash when iImporting a builtin module crashes                   10/30/08
CLOSED http://bugs.python.org/issue4235    created  amaury.forgeotdarc        
                                                                               

Crash when importing builtin module during interpreter shutdown  10/30/08
       http://bugs.python.org/issue4236    created  amaury.forgeotdarc        
       patch                                                                   

io.FileIO('foo', 'rt') prints a RuntimeWarning                   10/30/08
CLOSED http://bugs.python.org/issue4237    created  amaury.forgeotdarc        
       patch, needs review                                                     

BSD support for multiprocessing.cpu_count                        10/30/08
       http://bugs.python.org/issue4238    created  epg                       
                                                                               

Email example should use SMTP.quit() rather than SMTP.close()    10/30/08
       http://bugs.python.org/issue4239    created  matt                      
                                                                               

parasite form feed character in email.mime.text                  10/30/08
CLOSED http://bugs.python.org/issue4240    created  tarek                     
                                                                               

zipfile.py -> is_zipfile leaves file open when error             10/30/08
       http://bugs.python.org/issue4241    created  eino                      
                                                                               

Classify language vs. impl-detail tests, step 1                  10/30/08
       http://bugs.python.org/issue4242    created  arigo                     
       patch, patch                                                            

has_key doc could be clearer                                     10/30/08
       http://bugs.python.org/issue4243    created  tjreedy                   
                                                                               

ihooks incompatible with absolute_import feature                 10/31/08
       http://bugs.python.org/issue4244    created  exarkun                   
                                                                               

threading documentation: reorder sections                        10/31/08
       http://bugs.python.org/issue4245    created  hoffman                   
                                                                               



Issues Now Closed (26)
______________________

ftplib needs a rewrite to use bytes/buffers                       388 days
       http://bugs.python.org/issue1248    haypo                     
                                                                               

backport python 3.0 language functionality to python 2.6 by addi  117 days
       http://bugs.python.org/issue3238    loewis                    
       patch                                                                   

Python turning off assertions (Windows)                            77 days
       http://bugs.python.org/issue3545    kevinwatters              
                                                                               

python3.0 interpreter on Cygwin ignores all arguments              71 days
       http://bugs.python.org/issue3626    amaury.forgeotdarc        
       patch                                                                   

Py_NewInterpreter does not work                                    62 days
       http://bugs.python.org/issue3723    christian.heimes          
       patch, needs review                                                     

segfault with defaultdict and pickle                                8 days
       http://bugs.python.org/issue4170    amaury.forgeotdarc        
       patch                                                                   

segfault with pickle if 4th or 5th item of tuple returned by __r    9 days
       http://bugs.python.org/issue4176    ocean-city                
       patch                                                                   

library documentation errors                                        0 days
       http://bugs.python.org/issue4196    benjamin.peterson         
                                                                               

atexit module not safe in Python 3.0 with multiple interpreters     6 days
       http://bugs.python.org/issue4200    christian.heimes          
       patch, needs review                                                     

unexpected str.__init__() behaviour on b''                          0 days
       http://bugs.python.org/issue4205    keldonin                  
                                                                               

Remove backwards compatibility in _heapq for performance            0 days
       http://bugs.python.org/issue4207    krisvale                  
       patch, patch, easy                                                      

from __future__ import unicode_literals, print_function doesn't     0 days
       http://bugs.python.org/issue4209    benjamin.peterson         
       patch                                                                   

os.remove can be used to remove directories on SunOS                0 days
       http://bugs.python.org/issue4210    giampaolo.rodola          
                                                                               

Lower case file system encoding                                     3 days
       http://bugs.python.org/issue4213    haypo                     
       patch                                                                   

tarfile module fails to correctly parse some .tar.gz archives?!     0 days
       http://bugs.python.org/issue4218    lars.gustaebel            
                                                                               

Problem with regular expression                                     0 days
       http://bugs.python.org/issue4219    georg.brandl              
                                                                               

** operating incorrectly for negative bases?                        0 days
       http://bugs.python.org/issue4224    benjamin.peterson         
                                                                               

unicode_literals doesn't work in exec                               2 days
       http://bugs.python.org/issue4225    benjamin.peterson         
       patch, needs review                                                     

unicode_literals and print_function do not work together.           0 days
       http://bugs.python.org/issue4227    christian.heimes          
                                                                               

Can't Start Python                                                  0 days
       http://bugs.python.org/issue4229    loewis                    
                                                                               

print "    should be   print("                                      0 days
       http://bugs.python.org/issue4231    benjamin.peterson         
                                                                               

tempfile.gettempdir() availability not documented                   0 days
       http://bugs.python.org/issue4232    benjamin.peterson         
                                                                               

bin missing from documentation                                      1 days
       http://bugs.python.org/issue4234    benjamin.peterson         
       patch                                                                   

Crash when iImporting a builtin module crashes                      0 days
       http://bugs.python.org/issue4235    amaury.forgeotdarc        
                                                                               

io.FileIO('foo', 'rt') prints a RuntimeWarning                      1 days
       http://bugs.python.org/issue4237    christian.heimes          
       patch, needs review                                                     

parasite form feed character in email.mime.text                     0 days
       http://bugs.python.org/issue4240    skip.montanaro            
                                                                               



Top Issues Most Discussed (10)
______________________________

 18 Cannot build _multiprocessing, math, mmap and readline of Pytho    6 days
open    http://bugs.python.org/issue4204   

 15 Miserable subprocess.Popen performance                             7 days
open    http://bugs.python.org/issue4194   

 13 atexit module not safe in Python 3.0 with multiple interpreters    6 days
closed  http://bugs.python.org/issue4200   

  9 Lower case file system encoding                                    3 days
closed  http://bugs.python.org/issue4213   

  8 io.FileIO('foo', 'rt') prints a RuntimeWarning                     1 days
closed  http://bugs.python.org/issue4237   

  8 inspect.getsource doesn't work on functions imported from a zip    3 days
open    http://bugs.python.org/issue4223   

  6 Classify language vs. impl-detail tests, step 1                    1 days
open    http://bugs.python.org/issue4242   

  6 Crash when importing builtin module during interpreter shutdown    2 days
open    http://bugs.python.org/issue4236   

  6 unicode_literals doesn't work in exec                              2 days
closed  http://bugs.python.org/issue4225   

  5 has_key doc could be clearer                                       1 days
open    http://bugs.python.org/issue4243   




From ellisonbg.net at gmail.com  Fri Oct 31 18:55:57 2008
From: ellisonbg.net at gmail.com (Brian Granger)
Date: Fri, 31 Oct 2008 10:55:57 -0700
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <ca471dc20810310807t63d6563cxabd1577d886eca1@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
	<76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
	<ca471dc20810310807t63d6563cxabd1577d886eca1@mail.gmail.com>
Message-ID: <6ce0ac130810311055t7c64ff57leb7bf6d106c94a87@mail.gmail.com>

>> Has anyone made the argument for keeping the GIL to discourage
>> threading?
>
> Oooh, you are on to my secret plan! :-)

I completely agree that there are other approaches to parallelism and
concurrency that are much better than threading.  However, I don't
think this is a good argument for having poor support for parallel
threads in Python (i.e. keeping the GIL).  The reason is that threads
are extremely useful (and often required) for implementing other
approaches to concurrency, such as message passing.  Two examples:

Take Erlang for example.  Erlang uses a shared nothing/message passing
approach to concurrency (Yes!).  However, Erlangs implementation of
this approach relies heavily upon threads in the low-level of the
interpreter.  Without this usage of threads Erlang would be able to
provide the multicore scalability that it does using message passing.

Same is true of MPI.  From the user's perspective MPI is just message
passing.  But, MPI implementations use threads internally extensively.

Bottom line: threads may be a bad end in themselves (I agree with
this), but they are a great means to better things.

Cheers,

Brian

From rhamph at gmail.com  Fri Oct 31 19:03:16 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Fri, 31 Oct 2008 12:03:16 -0600
Subject: [Python-Dev] Fwd: Removal of GIL through refcounting removal.
In-Reply-To: <6ce0ac130810311055t7c64ff57leb7bf6d106c94a87@mail.gmail.com>
References: <392528d30810300855g33b8130flc3098f81700bab08@mail.gmail.com>
	<392528d30810300913w723271d2i855b4d78fc2cd9b5@mail.gmail.com>
	<gedmtg$eop$1@ger.gmane.org> <490AA4B9.5060307@gmail.com>
	<76fd5acf0810310459v625e4b48u1e9a64a7057309e5@mail.gmail.com>
	<ca471dc20810310807t63d6563cxabd1577d886eca1@mail.gmail.com>
	<6ce0ac130810311055t7c64ff57leb7bf6d106c94a87@mail.gmail.com>
Message-ID: <aac2c7cb0810311103p5d0e2c34o410cc07bba2dc201@mail.gmail.com>

On Fri, Oct 31, 2008 at 11:55 AM, Brian Granger <ellisonbg.net at gmail.com> wrote:
>>> Has anyone made the argument for keeping the GIL to discourage
>>> threading?
>>
>> Oooh, you are on to my secret plan! :-)
>
> I completely agree that there are other approaches to parallelism and
> concurrency that are much better than threading.  However, I don't
> think this is a good argument for having poor support for parallel
> threads in Python (i.e. keeping the GIL).  The reason is that threads
> are extremely useful (and often required) for implementing other
> approaches to concurrency, such as message passing.  Two examples:
>
> Take Erlang for example.  Erlang uses a shared nothing/message passing
> approach to concurrency (Yes!).  However, Erlangs implementation of
> this approach relies heavily upon threads in the low-level of the
> interpreter.  Without this usage of threads Erlang would be able to
> provide the multicore scalability that it does using message passing.
>
> Same is true of MPI.  From the user's perspective MPI is just message
> passing.  But, MPI implementations use threads internally extensively.
>
> Bottom line: threads may be a bad end in themselves (I agree with
> this), but they are a great means to better things.

Despite the name, safethread has more to do with erlang than it does
traditional threading.  Alas,
greenmicropseudothreadactormonitortaskfibreropebobprocess is a little
too long for normal use, so I'm gonna keep calling it thread (and to
distinguish it from an OS process).


-- 
Adam Olsen, aka Rhamphoryncus

From musiccomposition at gmail.com  Fri Oct 31 21:52:31 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Fri, 31 Oct 2008 15:52:31 -0500
Subject: [Python-Dev] Has python-dev collapsed?
In-Reply-To: <20081030022250.6400.1062036738.divmod.xquotient.531@weber.divmod.com>
References: <ca471dc20810281556o86ed0bfweee4bb7b4b76d9d4@mail.gmail.com>
	<ge85i3$2lj$1@ger.gmane.org>
	<ca471dc20810281617x2cbc87a1t4c76a6262a07bb1d@mail.gmail.com>
	<49079FB8.2000608@cheimes.de>
	<20081029211454.GA22943@amk-desktop.matrixgroup.net>
	<20081030022250.6400.1062036738.divmod.xquotient.531@weber.divmod.com>
Message-ID: <1afaf6160810311352o22c1e7dekc5699e043ddc9349@mail.gmail.com>

On Wed, Oct 29, 2008 at 9:22 PM,  <glyph at divmod.com> wrote:
>
> On 29 Oct, 09:14 pm, amk at amk.ca wrote:
>>
>> Will 3.1 and 2.7 also be parallel releases?  (I ask, not having read
>> the 3xxx PEPS at all.)
>>
>> If yes, why?  While I can see a case for 2.6/3.0 being in sync -- new
>> features in 2.6 ease the transition to 3.0 -- I'd imagine that 3.1
>> would be better with a shorter cycle (6-9 months) because there are
>> more possible rough edges to clean up.  3.1 would likely include
>> bugfixes from the eventual 2.7, so 3.1 might also trigger 2.6.1, but I
>> don't think there's any harm if 3.1 contains features or incompatible
>> fixes that are unavailable to 2.x users.
>
> 2.7 is where a new version of 2to3 will be distributed.  As people actually
> start trying to migrate real-world code, lots of issues are inevitably going
> to be discovered there.  From my perspective, 2.7 and 3.1 need to be synced
> because 2.7 is the tool that will actually enable users of existing code to
> *use* 3.1.

FWIW, I'm also backporting new 2to3 features and bugfixes to the 2.6 series.



-- 
Cheers,
Benjamin Peterson
"There's nothing quite as beautiful as an oboe... except a chicken
stuck in a vacuum cleaner."