From anthon at mnt.org  Mon Sep  1 14:24:27 2008
From: anthon at mnt.org (Anthon van der Neut)
Date: Mon, 01 Sep 2008 14:24:27 +0200
Subject: [Python-Dev] Proposal: add odict to collections
In-Reply-To: <loom.20080615T093754-59@post.gmane.org>
References: <loom.20080614T231548-103@post.gmane.org>
	<op.ucr078vrhlrjc9@conan>	<BD99F89BF3164159904E42E5D328DC2A@RaymondLaptop1>	<200806151844.13685.steve@pearwood.info>
	<loom.20080615T093754-59@post.gmane.org>
Message-ID: <48BBDEFB.70004@mnt.org>

Sorry to pipe in so late, but this is actually the default behaviour of
my C implementation (which I call KIO (Key Insertion Order), there is an
option to change this to KVIO (Key (or) Value Insertion Order), which
moves the pair to the end.

Anthon

Armin Ronacher wrote:
> Steven D'Aprano <steve <at> pearwood.info> writes:
> 
>> Conceptually, I would expect the following behaviour:
>>
>>>>> od = odict()
>>>>> od[1] = 'spam'  # insert a new key
>>>>> od[2] = 'parrot'  # insert a new key
>>>>> od[1] = 'ham'  # modify existing key
>>>>> od.items()
>> [(1, 'ham'), (2, 'parrot')]
> That behavior is different to any ordered-dict implementation
> out there ;-)
> 
> Regards,
> Armin
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/anthon%40mnt.org
> 
> 

From ncoghlan at gmail.com  Mon Sep  1 15:24:02 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 01 Sep 2008 23:24:02 +1000
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
Message-ID: <48BBECF2.5020501@gmail.com>

I've been taking a close look at the API for multiprocessing and
threading, and have discovered a somewhat strange pattern that occurs
multiple times in both interfaces: factory functions with names that
start with a capital letter so they look like they're actually a class.

At first I thought it was a quirk of the multiprocessing implementation,
but then I discovered that the threading module also does it for all of
the synchronisation objects as well as threading.Timer, with examples like:

  def Event(*args, **kwds):
    return _Event(*args, **kwds)

Is this just intended to discourage subclassing? If so, why give the
misleading impression that these things can be subclassed by naming them
as if they were classes?

How should this be handled when it comes to the addition of PEP 8
compliant aliases? Add aliases for the factory functions that start with
a lower case letter, but otherwise leave the structure of the modules
unchanged? Or should the trivial functions like the one above be dropped
and made direct aliases for the classes they are currently wrapping?

Option 1 has the virtue of being perfectly backwards compatible in the
threading case, while option 2 is a little riskier, so I'm inclined to
go with option 1 (keeping the factory functions, but giving them
non-class like names in the case of multiprocessing, or aliases in the
case of threading).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From solipsis at pitrou.net  Mon Sep  1 16:36:00 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 1 Sep 2008 14:36:00 +0000 (UTC)
Subject: [Python-Dev]
	=?utf-8?q?Further_PEP_8_compliance_issues_in_threadi?=
	=?utf-8?q?ng_and=09multiprocessing?=
References: <48BBECF2.5020501@gmail.com>
Message-ID: <loom.20080901T143254-314@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> 
> Is this just intended to discourage subclassing? If so, why give the
> misleading impression that these things can be subclassed by naming them
> as if they were classes?
> 
> How should this be handled when it comes to the addition of PEP 8
> compliant aliases?

I don't see a problem for trivial functional wrappers to classes to be
capitalized like classes. So I'd suggest option 3: leave it as-is. Otherwise
option 2 (replace the wrappers with the actual classes) has my preference.




From musiccomposition at gmail.com  Mon Sep  1 16:42:06 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Mon, 1 Sep 2008 09:42:06 -0500
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
In-Reply-To: <loom.20080901T143254-314@post.gmane.org>
References: <48BBECF2.5020501@gmail.com>
	<loom.20080901T143254-314@post.gmane.org>
Message-ID: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>

On Mon, Sep 1, 2008 at 9:36 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>>
>> Is this just intended to discourage subclassing? If so, why give the
>> misleading impression that these things can be subclassed by naming them
>> as if they were classes?
>>
>> How should this be handled when it comes to the addition of PEP 8
>> compliant aliases?
>
> I don't see a problem for trivial functional wrappers to classes to be
> capitalized like classes. So I'd suggest option 3: leave it as-is. Otherwise
> option 2 (replace the wrappers with the actual classes) has my preference.

Yes, I believe that pretending that functions are classes is a fairly
common idiom in the stdlib and out, so I see no problem leaving them
alone. We haven't had any complaints about the threading Event
function yet either. :)

-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From exarkun at divmod.com  Mon Sep  1 17:00:14 2008
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Mon, 1 Sep 2008 11:00:14 -0400
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
 multiprocessing
In-Reply-To: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>
Message-ID: <20080901150014.29191.1765474940.divmod.quotient.20427@ohm>

On Mon, 1 Sep 2008 09:42:06 -0500, Benjamin Peterson <musiccomposition at gmail.com> wrote:
>On Mon, Sep 1, 2008 at 9:36 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>>>
>>> Is this just intended to discourage subclassing? If so, why give the
>>> misleading impression that these things can be subclassed by naming them
>>> as if they were classes?
>>>
>>> How should this be handled when it comes to the addition of PEP 8
>>> compliant aliases?
>>
>> I don't see a problem for trivial functional wrappers to classes to be
>> capitalized like classes. So I'd suggest option 3: leave it as-is. Otherwise
>> option 2 (replace the wrappers with the actual classes) has my preference.
>
>Yes, I believe that pretending that functions are classes is a fairly
>common idiom in the stdlib and out, so I see no problem leaving them
>alone. We haven't had any complaints about the threading Event
>function yet either. :)

Here's a complaint.  It's surprising that you can't use Event et al with
isinstance.  This is something I'm sure a lot of people run into (I did,
many years ago) when they start to use these APIs.  Once you do figure
out why it doesn't work, it's not clear how to do what you want, since
_Event is private.

Jean-Paul

From musiccomposition at gmail.com  Mon Sep  1 17:50:13 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Mon, 1 Sep 2008 10:50:13 -0500
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
In-Reply-To: <20080901150014.29191.1765474940.divmod.quotient.20427@ohm>
References: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>
	<20080901150014.29191.1765474940.divmod.quotient.20427@ohm>
Message-ID: <1afaf6160809010850n21f887f0jb9e4cb4be6241311@mail.gmail.com>

On Mon, Sep 1, 2008 at 10:00 AM, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> On Mon, 1 Sep 2008 09:42:06 -0500, Benjamin Peterson
>>
>> Yes, I believe that pretending that functions are classes is a fairly
>> common idiom in the stdlib and out, so I see no problem leaving them
>> alone. We haven't had any complaints about the threading Event
>> function yet either. :)
>
> Here's a complaint.  It's surprising that you can't use Event et al with
> isinstance.  This is something I'm sure a lot of people run into (I did,
> many years ago) when they start to use these APIs.  Once you do figure
> out why it doesn't work, it's not clear how to do what you want, since
> _Event is private.

Does anybody ever complain about not being able to use isinstance on
twisted.application.Application? (At least it's documented as a
function there.)

>
> Jean-Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From fredrik at pythonware.com  Mon Sep  1 17:54:27 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Mon, 01 Sep 2008 17:54:27 +0200
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
In-Reply-To: <1afaf6160809010850n21f887f0jb9e4cb4be6241311@mail.gmail.com>
References: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>	<20080901150014.29191.1765474940.divmod.quotient.20427@ohm>
	<1afaf6160809010850n21f887f0jb9e4cb4be6241311@mail.gmail.com>
Message-ID: <g9h37m$fav$2@ger.gmane.org>

Benjamin Peterson wrote:

> Does anybody ever complain about not being able to use isinstance on
> twisted.application.Application? (At least it's documented as a
> function there.)

the threading "non-classes" are documented to be factory functions on 
the module page.

</F>


From torne-pythondev at wolfpuppy.org.uk  Mon Sep  1 17:38:17 2008
From: torne-pythondev at wolfpuppy.org.uk (Torne Wuff)
Date: Mon, 1 Sep 2008 16:38:17 +0100
Subject: [Python-Dev] constness of interpreter data
Message-ID: <20080901153817.GA24171@wolfpuppy.org.uk>

libpython2.5.a contains quite a lot of .data that doesn't look like it
needs to be writable (my minimal interpreter binary has 105KB of
writable allocated data). A lot of these symbols look like they could
just be tagged const with no other changes to the interpreter; some of
them would require a proliferation of constness. I'm a bit new at this,
though, and it's possible that I'm wrong about some/most/all of these,
so I'm wondering what people think.

Attached is a patch which adds const to the easy ones:
  * Docstrings for extension functions (PyDoc_VAR in Python.h)
  * ascii->digit lookup table (_PyLong_DigitValue in longobject.c)
  * The copyright notice (cprt in getcopyright.c)

There are no errors or new warnings introduced in my build by this
patch, but I'm only compiling the interpreter itself. If anyone can
suggest a reason why any of those shouldn't be const, please do :)

Things that take up space that might be const-able with more effort, or
might not, I can't tell:
  * Token names (_PyParser_TokenNames in tokenizer.c)
  * Module function tables (and ensuing propagation of constness into
    PyModule_Init etc)
  * All the type and exception objects

Things that take up space but probably aren't const-able unless someone
who knows more than me has an idea:
  * Standard slot definitions (slotdefs in typeobject.c, but the
    interned string pointers get added at runtime)
  * The generated tables for the grammar (but the accelerator seems to
    want to change these at init time)

There's probably other things, but I didn't go through the entire symbol
table, just started with the big ones :)

So, er, is this a remotely sane thing to be doing, and does anyone have
suggestions? :)

-- 
Torne Wuff
torne at wolfpuppy.org.uk
-------------- next part --------------
A non-text attachment was scrubbed...
Name: constness.patch
Type: text/x-diff
Size: 2299 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080901/54a6b884/attachment-0001.patch>

From solipsis at pitrou.net  Mon Sep  1 17:39:18 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 1 Sep 2008 15:39:18 +0000 (UTC)
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
References: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>
	<20080901150014.29191.1765474940.divmod.quotient.20427@ohm>
Message-ID: <loom.20080901T153842-564@post.gmane.org>

Jean-Paul Calderone <exarkun <at> divmod.com> writes:
> 
> Here's a complaint.  It's surprising that you can't use Event et al with
> isinstance.  This is something I'm sure a lot of people run into (I did,
> many years ago) when they start to use these APIs.  Once you do figure
> out why it doesn't work, it's not clear how to do what you want, since
> _Event is private.

event_class = Event().__class__ ?

Not pretty I know :-)



From fredrik at pythonware.com  Mon Sep  1 18:28:19 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Mon, 01 Sep 2008 18:28:19 +0200
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
In-Reply-To: <loom.20080901T153842-564@post.gmane.org>
References: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>	<20080901150014.29191.1765474940.divmod.quotient.20427@ohm>
	<loom.20080901T153842-564@post.gmane.org>
Message-ID: <g9h575$m47$1@ger.gmane.org>

Antoine Pitrou wrote:

> event_class = Event().__class__ ?
> 
> Not pretty I know :-)

somewhat prettier, assuming 2.3 or newer:

 >>> import threading
 >>> e = threading.Event()
 >>> type(e)
<class 'threading._Event'>
 >>> isinstance(e, type(threading.Event()))
True

(but pretty OT)

</F>


From aahz at pythoncraft.com  Mon Sep  1 19:08:32 2008
From: aahz at pythoncraft.com (Aahz)
Date: Mon, 1 Sep 2008 10:08:32 -0700
Subject: [Python-Dev] constness of interpreter data
In-Reply-To: <20080901153817.GA24171@wolfpuppy.org.uk>
References: <20080901153817.GA24171@wolfpuppy.org.uk>
Message-ID: <20080901170832.GA19186@panix.com>

On Mon, Sep 01, 2008, Torne Wuff wrote:
> 
> Attached is a patch which adds const to the easy ones:
>   * Docstrings for extension functions (PyDoc_VAR in Python.h)
>   * ascii->digit lookup table (_PyLong_DigitValue in longobject.c)
>   * The copyright notice (cprt in getcopyright.c)

If you want this patch considered, please post it to bugs.python.org --
patches posted to the mailing list can't be tracked.  Thanks!
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

Adopt A Process -- stop killing all your children!

From ncoghlan at gmail.com  Mon Sep  1 23:25:24 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 02 Sep 2008 07:25:24 +1000
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
 multiprocessing
In-Reply-To: <g9h37m$fav$2@ger.gmane.org>
References: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>	<20080901150014.29191.1765474940.divmod.quotient.20427@ohm>	<1afaf6160809010850n21f887f0jb9e4cb4be6241311@mail.gmail.com>
	<g9h37m$fav$2@ger.gmane.org>
Message-ID: <48BC5DC4.8090804@gmail.com>

Fredrik Lundh wrote:
> Benjamin Peterson wrote:
> 
>> Does anybody ever complain about not being able to use isinstance on
>> twisted.application.Application? (At least it's documented as a
>> function there.)
> 
> the threading "non-classes" are documented to be factory functions on
> the module page.

Given the proximity of RC1, Antoine's option 3 (leaving the capitalised
factory functions in both multiprocessing and threading APIs) is
actually sounding pretty appealing to me at the moment.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From greg.ewing at canterbury.ac.nz  Tue Sep  2 03:04:18 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 02 Sep 2008 13:04:18 +1200
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
 multiprocessing
In-Reply-To: <loom.20080901T143254-314@post.gmane.org>
References: <48BBECF2.5020501@gmail.com>
	<loom.20080901T143254-314@post.gmane.org>
Message-ID: <48BC9112.3080809@canterbury.ac.nz>

Antoine Pitrou wrote:

> I don't see a problem for trivial functional wrappers to classes to be
> capitalized like classes.

The problem is that the capitalization makes you
think it's a class, suggesting you can do things
with it that you actually can't, e.g. subclassing.

I can't think of any reason to do this. If you
don't want to promise that something is a class,
what possible reason is there for naming it like
one?

I can see a reason for doing the opposite, though:
if something happens to be a class, but you don't
want to promise that, you could expose it under
a lower-case name, that can be replaced with a
factory function later.

In this case, the thing to decide is whether
Event will always be a direct class instantiation.
If so, rename _Event to Event and expose it
directly. If not, rename Event to event.

-- 
Greg

From tonynelson at georgeanelson.com  Tue Sep  2 03:36:17 2008
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Mon, 1 Sep 2008 21:36:17 -0400
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
 multiprocessing
In-Reply-To: <48BC9112.3080809@canterbury.ac.nz>
References: <48BBECF2.5020501@gmail.com>
	<loom.20080901T143254-314@post.gmane.org>
	<48BC9112.3080809@canterbury.ac.nz>
Message-ID: <p04330100c4e24795d576@[192.168.123.162]>

At 1:04 PM +1200 9/2/08, Greg Ewing wrote:
>Antoine Pitrou wrote:
>
>> I don't see a problem for trivial functional wrappers to classes to be
>> capitalized like classes.
>
>The problem is that the capitalization makes you
>think it's a class, suggesting you can do things
>with it that you actually can't, e.g. subclassing.

Or that it returns a new object of that kind.


>I can't think of any reason to do this. If you
>don't want to promise that something is a class,
>what possible reason is there for naming it like
>one?
 ...

Lower-case names return something about an object.  Capitalized names
return a new object of the named type (more or less), either via a Class
constructor or a Factory object.  That's /a/ reason, anyway.

I suppose the question is what a capitalized name promises.  If it means
only "Class", then how should "Returns a new object", either from a Class
or a Factory, be shown?  Perhaps a new convention is needed for Factories?
-- 
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From dreamingforward at gmail.com  Tue Sep  2 06:02:30 2008
From: dreamingforward at gmail.com (average)
Date: Mon, 1 Sep 2008 21:02:30 -0700
Subject: [Python-Dev] Things to Know About Super
Message-ID: <913f9f570809012102g609096bjac8a12891c01cabc@mail.gmail.com>

It seems that the frustration with super revolves around how Python
currently conflates (as well as many users) two very different types
of inheritance, both "is-a" and "has-a" (or compositional)
inheritance.  Unfortunately, Python assists this confusion because the
language doesn't provide a distinct enough way to differentiate
between them.

For the former (i.e. is-a inheritance), users should not have to
explicitly make a call to the super class (via super() or
otherwise)--this should be automatic for every method held in common
so as to guarantee the invariants of the simpler class.  For this type
of inheritance it makes more sense to be explicit about *bypassing* a
call to the parent class--IF necessary (perhaps to take advantage of
an optimization available only in the subclass)--rather than explicit
about *making* one.  This would remove many of the bugs, misuses, and
frustrations with super().  No more ambiguity and confusion because
the language now guarantees certain behavior.  Super() would then be
limited to cooperative mixin classes; for all others, explicit naming
of the class would be required.  But perhaps that there is some
language abstraction that has yet to be fully invented that would tie
it all together nicely.

By the way, regarding your trait implementation, it may be worthwhile
revisiting the discussion surrounding the Prothon language discussed a
few years ago on comp.lang.python which got rid of classes and
metaclasses and replaced them with prototypes.

Regards,

zipher

From ncoghlan at gmail.com  Tue Sep  2 11:52:05 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 02 Sep 2008 19:52:05 +1000
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
 multiprocessing
In-Reply-To: <p04330100c4e24795d576@[192.168.123.162]>
References: <48BBECF2.5020501@gmail.com>	<loom.20080901T143254-314@post.gmane.org>	<48BC9112.3080809@canterbury.ac.nz>
	<p04330100c4e24795d576@[192.168.123.162]>
Message-ID: <48BD0CC5.2050005@gmail.com>

Tony Nelson wrote:
> I suppose the question is what a capitalized name promises.  If it means
> only "Class", then how should "Returns a new object", either from a Class
> or a Factory, be shown?  Perhaps a new convention is needed for Factories?

Any function can always return a new object (e.g. operator.add(list1,
list2), so I don't think we need a special naming convention to
explicitly flag factory functions.

The question I am raising is whether or not aberrations in the other
direction (factory functions that are named like a class, incorrectly
implying they can be used as a base class or as part of isinstance() or
issubclass() checks) are enough of a concern to add additional aliases
to the threading API, and to further modify the multiprocessing API this
close to RC1.

(Issue 3352 actually provides a complete list of the names that are
potentially at issue for both multiprocessing and threading - note that
Ben, with my concurrence, has closed that issue on the assumption that
the current naming scheme for these factory functions is acceptable).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Tue Sep  2 11:53:39 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 02 Sep 2008 19:53:39 +1000
Subject: [Python-Dev] Things to Know About Super
In-Reply-To: <913f9f570809012102g609096bjac8a12891c01cabc@mail.gmail.com>
References: <913f9f570809012102g609096bjac8a12891c01cabc@mail.gmail.com>
Message-ID: <48BD0D23.6030806@gmail.com>

average wrote:
> It seems that the frustration with super revolves around how Python
> currently conflates (as well as many users) two very different types
> of inheritance, both "is-a" and "has-a" (or compositional)
> inheritance.  Unfortunately, Python assists this confusion because the
> language doesn't provide a distinct enough way to differentiate
> between them.

has-a should be modelling with attributes, not inheritance. The latter
relationship should always mean is-a (even for mixins).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From gnewsg at gmail.com  Tue Sep  2 13:30:27 2008
From: gnewsg at gmail.com (Giampaolo Rodola')
Date: Tue, 2 Sep 2008 04:30:27 -0700 (PDT)
Subject: [Python-Dev] Add python.exe to PATH environment variable
Message-ID: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>

Hi,
I've always found it strange that Python Windows installers never
managed to add the python executable to the PATH environment variable.
Are there plans for adding such a thing?

Thanks in advance

--- Giampaolo
http://code.google.com/p/pyftpdlib/

From amauryfa at gmail.com  Tue Sep  2 13:38:01 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 2 Sep 2008 13:38:01 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
Message-ID: <e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>

Hello,

Giampaolo Rodola' wrote:
> Hi,
> I've always found it strange that Python Windows installers never
> managed to add the python executable to the PATH environment variable.
> Are there plans for adding such a thing?

I don't think so.
See the discussion of http://bugs.python.org/issue3561

-- 
Amaury Forgeot d'Arc

From solipsis at pitrou.net  Tue Sep  2 13:46:52 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 2 Sep 2008 11:46:52 +0000 (UTC)
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
References: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>	<20080901150014.29191.1765474940.divmod.quotient.20427@ohm>	<1afaf6160809010850n21f887f0jb9e4cb4be6241311@mail.gmail.com>
	<g9h37m$fav$2@ger.gmane.org> <48BC5DC4.8090804@gmail.com>
Message-ID: <loom.20080902T114456-242@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> 
> Given the proximity of RC1, Antoine's option 3 (leaving the capitalised
> factory functions in both multiprocessing and threading APIs) is
> actually sounding pretty appealing to me at the moment.

I was actually suggesting this course for the threading module, whose API has
existed for a lot of time now, but I think it would be better to clean up
multiprocessing before its first stable relase. But the issue of time and
manpower starts being a bit critical :)




From mal at egenix.com  Tue Sep  2 14:09:09 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 02 Sep 2008 14:09:09 +0200
Subject: [Python-Dev] 2.6b3 Windows installers
Message-ID: <48BD2CE5.4020703@egenix.com>

The download page doesn't list any Windows installer for 2.6b3:

http://www.python.org/download/releases/2.6/

I suppose this is due to Martin building the installers and him not
be available at the moment.

Since Python on Windows will likely only get very few beta testers
without a Windows installer build, I'd suggest to postpone the
RC1 release that's planned for tomorrow to get more feedback for the
Windows builds.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 02 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From barry at python.org  Tue Sep  2 14:24:27 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 2 Sep 2008 08:24:27 -0400
Subject: [Python-Dev] 2.6b3 Windows installers
In-Reply-To: <48BD2CE5.4020703@egenix.com>
References: <48BD2CE5.4020703@egenix.com>
Message-ID: <8D7E55D5-E3BD-45C5-AED5-B6C438B754A1@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 2, 2008, at 8:09 AM, M.-A. Lemburg wrote:

> I suppose this is due to Martin building the installers and him not
> be available at the moment.

He should be back today.

> Since Python on Windows will likely only get very few beta testers
> without a Windows installer build, I'd suggest to postpone the
> RC1 release that's planned for tomorrow to get more feedback for the
> Windows builds.

I'd rather still release the rc tomorrow.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSL0we3EjvBPtnXfVAQKbRgQAhNM2q8M4Yd0IhbEd2+DWtZnfALdxXr8e
vfK0h3tH3+DZuEYyNLZHQOwLC7uUjqcKXc/y+v0EwY7Q13tYz0TCVcLY+0zbXsoi
on0TMh8kn/Wwlw/byKc8K3HFkKU4mhzxmLm4oq/8sbsQPABjkEtgNEXP4enmZFLU
U1R8E273MjY=
=e7/S
-----END PGP SIGNATURE-----

From ziade.tarek at gmail.com  Tue Sep  2 15:04:45 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 2 Sep 2008 15:04:45 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
Message-ID: <94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>

On Tue, Sep 2, 2008 at 1:38 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com>wrote:

> Hello,
>
> Giampaolo Rodola' wrote:
> > Hi,
> > I've always found it strange that Python Windows installers never
> > managed to add the python executable to the PATH environment variable.
>

+1

At this point, as far as I know, changing the PATH variable manually under
Windows is the most convenient
way to be able to work correctly with Python.


>
> > Are there plans for adding such a thing?
>
> I don't think so.
> See the discussion of http://bugs.python.org/issue3561
>

I don't understand why it is closed/wontfix though.

In the tracker Martin said:
"""

The very end-user related aspects of it need to be discussed
on python-dev (perhaps taking a poll whether this is desirable, whether
it should be optional, and if so, what the default should be)

"""

So I'd like to give my opinion here about this, because I've had this
problem while writing a book about Python.
(a book that is for people that use Python on *any* platform bien s?r, not
only Linux)

For me, the Windows installer does not work properly: I cannot open a
console and run Python right away.
I don't care about the technical complexity of setting it up automatically :
I simply think it is wrong not to be able to run
the interpreter the same way on Windows and Linux/Mac OS.

You may end up having to deal with it in your documentation. Look at all the
documentation out there. For instance Pylons:

http://wiki.pylonshq.com/display/pylonsdocs/Installing+Pylons

-> there's a small note on that page  "All windows users also should read
the section Post install tweaks for
Windows<http://wiki.pylonshq.com/display/pylonsdocs/Windows+Notes>after
installation"
-> windows users need to go there :
http://wiki.pylonshq.com/display/pylonsdocs/Windows+Notes

Why do we have to take care about that ? Python installers should.

A lot of package out there create console scripts (buildout, sphinx,
rst2html, etc..) that lives in /Script, and are not reachable
in Windows unless the PATH is changed.

So I don't see any good reason (besides the technical complexity) to add it
to the Windows installer.
Or at least do something that will let us run Python and third-party scripts
from the command line even if it is done with PATH.

So I would love to see this ticket open again; I personnaly would be in
favor of an automatic change of PATH by the installer.

My 2 cents
Tarek

-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080902/4a62d8df/attachment.htm>

From ziade.tarek at gmail.com  Tue Sep  2 15:10:57 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Tue, 2 Sep 2008 15:10:57 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
Message-ID: <94bdd2610809020610v36bae18fy5705bb702f954340@mail.gmail.com>

On Tue, Sep 2, 2008 at 3:04 PM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:

>
> So I don't see any good reason (besides the technical complexity) to add it
> to the Windows installer.
>

oups..  I don't see any good reason *not* to add it ... :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080902/1d165618/attachment.htm>

From lists at cheimes.de  Tue Sep  2 15:20:59 2008
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 02 Sep 2008 15:20:59 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
Message-ID: <g9jejr$8e9$1@ger.gmane.org>

Giampaolo Rodola' wrote:
> Hi,
> I've always found it strange that Python Windows installers never
> managed to add the python executable to the PATH environment variable.
> Are there plans for adding such a thing?

No, but I've added a little helper script several months ago. It adds
the Python and Script paths to %PATH%. See Tools/Scripts/win_add2path.py

Christian


From ncoghlan at gmail.com  Tue Sep  2 16:03:50 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 03 Sep 2008 00:03:50 +1000
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
 multiprocessing
In-Reply-To: <loom.20080902T114456-242@post.gmane.org>
References: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>	<20080901150014.29191.1765474940.divmod.quotient.20427@ohm>	<1afaf6160809010850n21f887f0jb9e4cb4be6241311@mail.gmail.com>	<g9h37m$fav$2@ger.gmane.org>
	<48BC5DC4.8090804@gmail.com>
	<loom.20080902T114456-242@post.gmane.org>
Message-ID: <48BD47C6.5040102@gmail.com>

Antoine Pitrou wrote:
> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>> Given the proximity of RC1, Antoine's option 3 (leaving the capitalised
>> factory functions in both multiprocessing and threading APIs) is
>> actually sounding pretty appealing to me at the moment.
> 
> I was actually suggesting this course for the threading module, whose API has
> existed for a lot of time now, but I think it would be better to clean up
> multiprocessing before its first stable relase. But the issue of time and
> manpower starts being a bit critical :)

Unfortunately, that's where the whole "close to a drop-in replacement
for threading" concept brings additions to the threading module API back
into play.

If I'd realised this a bit sooner I probably would have been pushing for
it to be dealt with for 2.6/3.0, but given that it's the kind of change
that we can easily do through the normal API deprecation process, I'm
really not comfortable messing with it this close to the release
(particularly after Jesse found a problem with the seemingly innocent
change to the multiprocessing implementation in issue 3589).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From jnoller at gmail.com  Tue Sep  2 16:58:59 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Tue, 2 Sep 2008 10:58:59 -0400
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
In-Reply-To: <48BD47C6.5040102@gmail.com>
References: <1afaf6160809010742j70f38817h98688319952f6d4e@mail.gmail.com>
	<20080901150014.29191.1765474940.divmod.quotient.20427@ohm>
	<1afaf6160809010850n21f887f0jb9e4cb4be6241311@mail.gmail.com>
	<g9h37m$fav$2@ger.gmane.org> <48BC5DC4.8090804@gmail.com>
	<loom.20080902T114456-242@post.gmane.org> <48BD47C6.5040102@gmail.com>
Message-ID: <4222a8490809020758o2cdae28fj4c43fbe73db6a1e@mail.gmail.com>

On Tue, Sep 2, 2008 at 10:03 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Antoine Pitrou wrote:
>> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>>> Given the proximity of RC1, Antoine's option 3 (leaving the capitalised
>>> factory functions in both multiprocessing and threading APIs) is
>>> actually sounding pretty appealing to me at the moment.
>>
>> I was actually suggesting this course for the threading module, whose API has
>> existed for a lot of time now, but I think it would be better to clean up
>> multiprocessing before its first stable relase. But the issue of time and
>> manpower starts being a bit critical :)
>
> Unfortunately, that's where the whole "close to a drop-in replacement
> for threading" concept brings additions to the threading module API back
> into play.
>
> If I'd realised this a bit sooner I probably would have been pushing for
> it to be dealt with for 2.6/3.0, but given that it's the kind of change
> that we can easily do through the normal API deprecation process, I'm
> really not comfortable messing with it this close to the release
> (particularly after Jesse found a problem with the seemingly innocent
> change to the multiprocessing implementation in issue 3589).
>
> Cheers,
> Nick.
>

Yes, the innocuous change in 3589 - which really made a lot of sense,
introduced a bug that didn't get caught until a complete make
distclean; rebuild - that actually scared me off of the idea of
addressing 3589 right now. I would move 3589 to 2.7/3.1 and file an
additional bug for any further pep8-ifying to both the threading and
mp APIs against 2.7 and 3.1

-jesse

From steve at pearwood.info  Tue Sep  2 18:36:41 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 3 Sep 2008 02:36:41 +1000
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
In-Reply-To: <48BBECF2.5020501@gmail.com>
References: <48BBECF2.5020501@gmail.com>
Message-ID: <200809030236.41944.steve@pearwood.info>

On Mon, 1 Sep 2008 11:24:02 pm Nick Coghlan wrote:
> I've been taking a close look at the API for multiprocessing and
> threading, and have discovered a somewhat strange pattern that occurs
> multiple times in both interfaces: factory functions with names that
> start with a capital letter so they look like they're actually a
> class.
>
> At first I thought it was a quirk of the multiprocessing
> implementation, but then I discovered that the threading module also
> does it for all of the synchronisation objects as well as
> threading.Timer, with examples like:
>
>   def Event(*args, **kwds):
>     return _Event(*args, **kwds)


Can anyone explain why those factory functions exist at all? They don't 
seem to do anything. Why not expose the class directly, instead of 
making it private and then exposing it via a factory function that does 
nothing else? Never mind whether or not the factory functions should 
start with an uppercase letter or not. Why are they even there?

Perhaps I'm missing some subtle (or even not-so-subtle) advantage, but 
it seems rather silly to me, as silly as this:

f = lambda x: function(x)
# better written as f = function

And yet threading.py has at least six examples just like that. What am I 
missing?


-- 
Steven

From guido at python.org  Tue Sep  2 18:51:04 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 2 Sep 2008 09:51:04 -0700
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
In-Reply-To: <200809030236.41944.steve@pearwood.info>
References: <48BBECF2.5020501@gmail.com>
	<200809030236.41944.steve@pearwood.info>
Message-ID: <ca471dc20809020951g17d168dhfcaf88f01631fcee@mail.gmail.com>

On Tue, Sep 2, 2008 at 9:36 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Mon, 1 Sep 2008 11:24:02 pm Nick Coghlan wrote:
>> I've been taking a close look at the API for multiprocessing and
>> threading, and have discovered a somewhat strange pattern that occurs
>> multiple times in both interfaces: factory functions with names that
>> start with a capital letter so they look like they're actually a
>> class.
>>
>> At first I thought it was a quirk of the multiprocessing
>> implementation, but then I discovered that the threading module also
>> does it for all of the synchronisation objects as well as
>> threading.Timer, with examples like:
>>
>>   def Event(*args, **kwds):
>>     return _Event(*args, **kwds)

> Can anyone explain why those factory functions exist at all? They don't
> seem to do anything. Why not expose the class directly, instead of
> making it private and then exposing it via a factory function that does
> nothing else? Never mind whether or not the factory functions should
> start with an uppercase letter or not. Why are they even there?
>
> Perhaps I'm missing some subtle (or even not-so-subtle) advantage, but
> it seems rather silly to me, as silly as this:
>
> f = lambda x: function(x)
> # better written as f = function
>
> And yet threading.py has at least six examples just like that. What am I
> missing?

I take full responsibility. This started out as an experiment in API
design, where I tried to make things look as much like the similar
Java API as possible (I didn't want to invent yet anotherwobbly
wheel). I specifically wanted these *not* to be classes so that people
wouldn't start subclassing them. At the time PEP-8 wasn't well
established (if at all) and I wanted the factory functions to look
like classes. I think in 2.7 / 3.1 we can change the factory functions
to conform to PEP-8 (leaving the old names in for a couple of
release). I still want the classes to be private, so that it's safe to
replace them with more efficient implementations on some platforms
without having to worry about subclasses.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From jcea at jcea.es  Tue Sep  2 19:05:46 2008
From: jcea at jcea.es (Jesus Cea)
Date: Tue, 02 Sep 2008 19:05:46 +0200
Subject: [Python-Dev] fileobj.read(float): warning or error?
In-Reply-To: <ca471dc20807291711m48d5837ap33c9b58f31eb82b5@mail.gmail.com>
References: <200807212117.14485.victor.stinner@haypocalc.com>	<20080722053744.GA11290@cskk.homeip.net>
	<ca471dc20807291711m48d5837ap33c9b58f31eb82b5@mail.gmail.com>
Message-ID: <48BD726A.9030200@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Guido van Rossum wrote:
> On Mon, Jul 21, 2008 at 10:37 PM, Cameron Simpson <cs at zip.com.au> wrote:
>> Leaving aside the 0.2 => 0 converstion, shouldn't read() raise an
>> exception if asked for < 1 bytes? Or is there a legitimate use for read(0)
>> with which I was not previously aware?
> 
> Indeed. read(0) is quite often generated as an edge case when one is
> computing buffer sizes, and returning an empty string is most
> definitely the right thing to do here (otherwise some application code
> becomes more complex by having to avoid calling read(0) at all).

How do you differenciate between that empty string (when doing
"read(0)"), from EOF (that is signaled by an empty string)?.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSL1yZplgi5GaxT1NAQJdTAP7B4GaeBRFg1A6PibmH+2cmJs3AIO2qWrx
xfgRO1QVF4OnxGWIKTTbKWX4whBY/zA3UUs35XMSRUROlxPR1dCNIvlaQb+rCuO6
AL0IkE5Fe6iN+VlS9UqarUla9vGhrqD9BxMZmDisIu4uKJi7c3ChlGKuatk16RBQ
BosUJe3VjNM=
=GkbX
-----END PGP SIGNATURE-----

From guido at python.org  Tue Sep  2 19:11:29 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 2 Sep 2008 10:11:29 -0700
Subject: [Python-Dev] fileobj.read(float): warning or error?
In-Reply-To: <48BD726A.9030200@jcea.es>
References: <200807212117.14485.victor.stinner@haypocalc.com>
	<20080722053744.GA11290@cskk.homeip.net>
	<ca471dc20807291711m48d5837ap33c9b58f31eb82b5@mail.gmail.com>
	<48BD726A.9030200@jcea.es>
Message-ID: <ca471dc20809021011x65703fd9la6caec0c839d3cb4@mail.gmail.com>

On Tue, Sep 2, 2008 at 10:05 AM, Jesus Cea <jcea at jcea.es> wrote:
> Guido van Rossum wrote:
>> On Mon, Jul 21, 2008 at 10:37 PM, Cameron Simpson <cs at zip.com.au> wrote:
>>> Leaving aside the 0.2 => 0 converstion, shouldn't read() raise an
>>> exception if asked for < 1 bytes? Or is there a legitimate use for read(0)
>>> with which I was not previously aware?
>>
>> Indeed. read(0) is quite often generated as an edge case when one is
>> computing buffer sizes, and returning an empty string is most
>> definitely the right thing to do here (otherwise some application code
>> becomes more complex by having to avoid calling read(0) at all).
>
> How do you differenciate between that empty string (when doing
> "read(0)"), from EOF (that is signaled by an empty string)?.

You don't. If you want to know whether you hit EOF you should try
reading a non-zero number of bytes. (Also note that getting fewer
bytes than you asked for is not enough to conclude that you have hit
EOF.)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From ijmorlan at cs.uwaterloo.ca  Tue Sep  2 19:17:02 2008
From: ijmorlan at cs.uwaterloo.ca (Isaac Morland)
Date: Tue, 2 Sep 2008 13:17:02 -0400 (EDT)
Subject: [Python-Dev] fileobj.read(float): warning or error?
In-Reply-To: <48BD726A.9030200@jcea.es>
References: <200807212117.14485.victor.stinner@haypocalc.com>
	<20080722053744.GA11290@cskk.homeip.net>
	<ca471dc20807291711m48d5837ap33c9b58f31eb82b5@mail.gmail.com>
	<48BD726A.9030200@jcea.es>
Message-ID: <Pine.GSO.4.64.0809021313050.19901@core.cs.uwaterloo.ca>

On Tue, 2 Sep 2008, Jesus Cea wrote:

>> Indeed. read(0) is quite often generated as an edge case when one is
>> computing buffer sizes, and returning an empty string is most
>> definitely the right thing to do here (otherwise some application code
>> becomes more complex by having to avoid calling read(0) at all).
>
> How do you differenciate between that empty string (when doing
> "read(0)"), from EOF (that is signaled by an empty string)?.

Why would you expect a difference between reading 0 bytes at EOF and 
reading 0 bytes anywhere else?  If you read(4) when at offset 996 in a 
1000-byte file I doubt you expect any special notification that you are 
now at EOF.

The Unix read() system call doesn't treat EOF as special other than it 
won't return bytes from "beyond" EOF and therefore even when reading a 
regular file could return fewer (including 0) bytes than asked for in the 
call.

Isaac Morland			CSCF Web Guru
DC 2554C, x36650		WWW Software Specialist

From jcea at jcea.es  Tue Sep  2 20:20:15 2008
From: jcea at jcea.es (Jesus Cea)
Date: Tue, 02 Sep 2008 20:20:15 +0200
Subject: [Python-Dev] Also bsddb (was: Re: Further PEP 8 compliance issues
 in threading and multiprocessing)
In-Reply-To: <20080901150014.29191.1765474940.divmod.quotient.20427@ohm>
References: <20080901150014.29191.1765474940.divmod.quotient.20427@ohm>
Message-ID: <48BD83DF.6080800@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jean-Paul Calderone wrote:
> Here's a complaint.  It's surprising that you can't use Event et al with
> isinstance.  This is something I'm sure a lot of people run into (I did,
> many years ago) when they start to use these APIs.  Once you do figure
> out why it doesn't work, it's not clear how to do what you want, since
> _Event is private.


I have similar issues with bsddb. Here, there are Factory functions like
"DBEnv" or "DB", creating "DBEnv" and "DB" class instances. Note the
name aliasing.

I don't understand why these functions exist. It causes a few problems,
like being unable to subclass the bsddb classes, for example.

For future bsddb4.6.4 (Late october, I think) I plan to remove the
Factory functions, exporting the classes directly, and allowing
subclassing. The code should be 100% compatible.

Any opinion?

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCUAwUBSL2D3Jlgi5GaxT1NAQLJugP4qynGMZI8nN06rDyPU8FQ2kHig6uReuSE
GW2fTuKXrYLlwRW5vA1GV/nA1y+6dUPuOF5erwCjVsXp28jMKNlk0BfIXmqe1wz9
+N6bIVYlFeChp5M05TDYaCNUNgRGuHURV44DvZ+vjr9GqxHuVWHHcl0EKTTSlpMi
K6FBYiZjbw==
=DGEq
-----END PGP SIGNATURE-----

From jcea at jcea.es  Tue Sep  2 20:31:44 2008
From: jcea at jcea.es (Jesus Cea)
Date: Tue, 02 Sep 2008 20:31:44 +0200
Subject: [Python-Dev] fileobj.read(float): warning or error?
In-Reply-To: <Pine.GSO.4.64.0809021313050.19901@core.cs.uwaterloo.ca>
References: <200807212117.14485.victor.stinner@haypocalc.com>	<20080722053744.GA11290@cskk.homeip.net>	<ca471dc20807291711m48d5837ap33c9b58f31eb82b5@mail.gmail.com>	<48BD726A.9030200@jcea.es>
	<Pine.GSO.4.64.0809021313050.19901@core.cs.uwaterloo.ca>
Message-ID: <48BD8690.6060405@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Isaac Morland wrote:
> On Tue, 2 Sep 2008, Jesus Cea wrote:
> 
>>> Indeed. read(0) is quite often generated as an edge case when one is
>>> computing buffer sizes, and returning an empty string is most
>>> definitely the right thing to do here (otherwise some application code
>>> becomes more complex by having to avoid calling read(0) at all).
>>
>> How do you differenciate between that empty string (when doing
>> "read(0)"), from EOF (that is signaled by an empty string)?.
> 
> Why would you expect a difference between reading 0 bytes at EOF and
> reading 0 bytes anywhere else?  If you read(4) when at offset 996 in a
> 1000-byte file I doubt you expect any special notification that you are
> now at EOF.

My message was an answer to Guido one, saying that some programs
calculate the read len substracting buffer lengths, so, then can try to
read 0 bytes. Guido argues that returning a empty string is the way to go.

My point is: we are simplifying the program considering "0" a valid len
counter, but we complicates it because now the code can't consider "" =
EOF if it actually asked for 0 bytes.

> The Unix read() system call doesn't treat EOF as special other than it
> won't return bytes from "beyond" EOF and therefore even when reading a
> regular file could return fewer (including 0) bytes than asked for in
> the call.

I always consider ""==EOF. I thought that was correct for non-blocking
sockets. Am I wrong?.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSL2GjZlgi5GaxT1NAQLniwP/SwdmA929j4oPplhtkVU82TYFoyevP/E2
QsHvCZ18CWYSa5LO00Vsd0Uo8ZQeqV8Gx6o2pG2ke66qI7c7pjTQcSO28Z3ztlVW
YZVbc46WGozjuiHh2tLVSckI4GyZJzs7+Btho2klE2dNygxWVEpT5Ueu+o2CK0Pl
Onf7jG4L+h0=
=YHQ/
-----END PGP SIGNATURE-----

From guido at python.org  Tue Sep  2 20:46:55 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 2 Sep 2008 11:46:55 -0700
Subject: [Python-Dev] fileobj.read(float): warning or error?
In-Reply-To: <48BD8690.6060405@jcea.es>
References: <200807212117.14485.victor.stinner@haypocalc.com>
	<20080722053744.GA11290@cskk.homeip.net>
	<ca471dc20807291711m48d5837ap33c9b58f31eb82b5@mail.gmail.com>
	<48BD726A.9030200@jcea.es>
	<Pine.GSO.4.64.0809021313050.19901@core.cs.uwaterloo.ca>
	<48BD8690.6060405@jcea.es>
Message-ID: <ca471dc20809021146u5411a461u3c97ba052cdb3434@mail.gmail.com>

On Tue, Sep 2, 2008 at 11:31 AM, Jesus Cea <jcea at jcea.es> wrote:
> Isaac Morland wrote:
>> On Tue, 2 Sep 2008, Jesus Cea wrote:
>>
>>>> Indeed. read(0) is quite often generated as an edge case when one is
>>>> computing buffer sizes, and returning an empty string is most
>>>> definitely the right thing to do here (otherwise some application code
>>>> becomes more complex by having to avoid calling read(0) at all).
>>>
>>> How do you differenciate between that empty string (when doing
>>> "read(0)"), from EOF (that is signaled by an empty string)?.
>>
>> Why would you expect a difference between reading 0 bytes at EOF and
>> reading 0 bytes anywhere else?  If you read(4) when at offset 996 in a
>> 1000-byte file I doubt you expect any special notification that you are
>> now at EOF.
>
> My message was an answer to Guido one, saying that some programs
> calculate the read len substracting buffer lengths, so, then can try to
> read 0 bytes. Guido argues that returning a empty string is the way to go.
>
> My point is: we are simplifying the program considering "0" a valid len
> counter, but we complicates it because now the code can't consider "" =
> EOF if it actually asked for 0 bytes.

Note that it has been like this for a very long time.

>> The Unix read() system call doesn't treat EOF as special other than it
>> won't return bytes from "beyond" EOF and therefore even when reading a
>> regular file could return fewer (including 0) bytes than asked for in
>> the call.
>
> I always consider ""==EOF. I thought that was correct for non-blocking
> sockets. Am I wrong?.

You can continue to assume this if you never pass 0 to read().

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From tjreedy at udel.edu  Tue Sep  2 23:14:08 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 02 Sep 2008 17:14:08 -0400
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
Message-ID: <g9kaav$hj2$1@ger.gmane.org>

Tarek Ziad? wrote:

> So I don't see any good reason (besides the technical complexity)

Unless and until someone is able and willing to deal with the technical 
complexity, that would seem to be a sufficient reason.

 >  to [not, I presume] add  it to the Windows installer.

> So I would love to see this ticket open again; I personnaly would be in 
> favor of an automatic change of PATH by the installer.

Martin said he would discuss a patch when there is a patch to discuss.
He feels strongly about there being a clean uninstall, including PATH 
restoration if it is changed.

The problem is that a) the Window's way to run user apps is via icons 
and menus and that b) the old DOS path/command way, based on Unix, has 
been neglected.

An alternative to manipulating PATH would be to make and add to the 
Start Menu a Command Prompt shortcut, call it Command Window or 
something, that starts in the Python directory.  Then one could enter 
 >python or >Scripts/goforit without chdir-ing to the Python directory 
first.  The background could even be set to green, for instance, to 
differentiate it from the standard Command Prompt window.

Terry Jan Reedy


From musiccomposition at gmail.com  Tue Sep  2 23:58:56 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Tue, 2 Sep 2008 16:58:56 -0500
Subject: [Python-Dev] 2.6b3 Windows installers
In-Reply-To: <8D7E55D5-E3BD-45C5-AED5-B6C438B754A1@python.org>
References: <48BD2CE5.4020703@egenix.com>
	<8D7E55D5-E3BD-45C5-AED5-B6C438B754A1@python.org>
Message-ID: <1afaf6160809021458ra11689g4995dca60b3f77f3@mail.gmail.com>

On Tue, Sep 2, 2008 at 7:24 AM, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Sep 2, 2008, at 8:09 AM, M.-A. Lemburg wrote:
>
>> I suppose this is due to Martin building the installers and him not
>> be available at the moment.
>
> He should be back today.
>
>> Since Python on Windows will likely only get very few beta testers
>> without a Windows installer build, I'd suggest to postpone the
>> RC1 release that's planned for tomorrow to get more feedback for the
>> Windows builds.
>
> I'd rather still release the rc tomorrow.

Even with 41 release blockers, are you still planning a release tomorrow?

I guess there's nothing wrong with that; the more testing the better!
2.6 and 3.0 are also pretty stable; we're just ironing out a few of
the rough spots. :)



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From barry at python.org  Wed Sep  3 00:03:52 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 2 Sep 2008 18:03:52 -0400
Subject: [Python-Dev] 2.6b3 Windows installers
In-Reply-To: <1afaf6160809021458ra11689g4995dca60b3f77f3@mail.gmail.com>
References: <48BD2CE5.4020703@egenix.com>
	<8D7E55D5-E3BD-45C5-AED5-B6C438B754A1@python.org>
	<1afaf6160809021458ra11689g4995dca60b3f77f3@mail.gmail.com>
Message-ID: <8FB68E96-C613-42CC-A39C-61305BDB6EEC@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 2, 2008, at 5:58 PM, Benjamin Peterson wrote:

> On Tue, Sep 2, 2008 at 7:24 AM, Barry Warsaw <barry at python.org> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On Sep 2, 2008, at 8:09 AM, M.-A. Lemburg wrote:
>>
>>> I suppose this is due to Martin building the installers and him not
>>> be available at the moment.
>>
>> He should be back today.
>>
>>> Since Python on Windows will likely only get very few beta testers
>>> without a Windows installer build, I'd suggest to postpone the
>>> RC1 release that's planned for tomorrow to get more feedback for the
>>> Windows builds.
>>
>> I'd rather still release the rc tomorrow.
>
> Even with 41 release blockers, are you still planning a release  
> tomorrow?
>
> I guess there's nothing wrong with that; the more testing the better!
> 2.6 and 3.0 are also pretty stable; we're just ironing out a few of
> the rough spots. :)

I hope to start going through the blockers tonight.  We'll see!

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSL24SHEjvBPtnXfVAQLQ0wP8D3zjs97No9dnR8WgOV/4IJHGQ4FK0e7k
XadqGmdEy0uLlpfjxCuBjWhdq5q1n/NPuJtX/wVtf0CQZtadb8gFde88+KyPkwYk
T8eAwz7OFv+kmclj2tnGh1Raps+0ocBrG5BWJG3D85ofGIZL4NZTLkDr9b6LDONA
lr85NvbpqMg=
=bj/j
-----END PGP SIGNATURE-----

From solipsis at pitrou.net  Wed Sep  3 00:46:08 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 2 Sep 2008 22:46:08 +0000 (UTC)
Subject: [Python-Dev] Add python.exe to PATH environment variable
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org>
Message-ID: <loom.20080902T224020-709@post.gmane.org>

Terry Reedy <tjreedy <at> udel.edu> writes:
> 
> An alternative to manipulating PATH would be to make and add to the 
> Start Menu a Command Prompt shortcut, call it Command Window or 
> something, that starts in the Python directory.

The reason for adding the directory to the PATH is for it to be recognized in
any command prompt, not only the Python-dedicated command prompt shortcut. 

Otherwise, the minute another project decides to do the same thing (provide a
custom command prompt shortcut rather than add its own directory to the PATH),
the user experience becomes very tedious.



From mal at egenix.com  Wed Sep  3 00:50:13 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 03 Sep 2008 00:50:13 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <g9kaav$hj2$1@ger.gmane.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org>
Message-ID: <48BDC325.1000901@egenix.com>

On 2008-09-02 23:14, Terry Reedy wrote:
> Tarek Ziad? wrote:
> 
>> So I don't see any good reason (besides the technical complexity)
> 
> Unless and until someone is able and willing to deal with the technical
> complexity, that would seem to be a sufficient reason.
> 
>>  to [not, I presume] add  it to the Windows installer.
> 
>> So I would love to see this ticket open again; I personnaly would be
>> in favor of an automatic change of PATH by the installer.
> 
> Martin said he would discuss a patch when there is a patch to discuss.
> He feels strongly about there being a clean uninstall, including PATH
> restoration if it is changed.
> 
> The problem is that a) the Window's way to run user apps is via icons
> and menus and that b) the old DOS path/command way, based on Unix, has
> been neglected.
> 
> An alternative to manipulating PATH would be to make and add to the
> Start Menu a Command Prompt shortcut, call it Command Window or
> something, that starts in the Python directory.  Then one could enter
>>python or >Scripts/goforit without chdir-ing to the Python directory
> first.  The background could even be set to green, for instance, to
> differentiate it from the standard Command Prompt window.

There already is a menu entry that starts the Python interpreter
on Windows, so why not use that ?

Also .py files are automatically associated with the last installed
Python interpreter, so the double-clicking on .py files works and is
probably the most common way of starting a Python file on Windows.

Adding paths to the PATH variable is not easy on Windows, esp. if
you want to support multiple Windows versions. The global PATH
settings are not read from autoexec.bat anymore (only once at boot
time). Instead those environment variables are managed via the
registry.

See e.g.

http://agiletesting.blogspot.com/2005/06/handling-path-windows-registry-value.html

for how to setup PATH to your liking using Python.

The problem is: how to undo those changes without accidentally
undoing an explicit change made by the user ?

BTW: Adding the Python dir to the PATH per default would cause
problems for users who regularly have multiple different
Python installations on a machine. If this is done, it should
be an install option and not forced.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 03 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From greg.ewing at canterbury.ac.nz  Wed Sep  3 01:42:29 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 03 Sep 2008 11:42:29 +1200
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
 multiprocessing
In-Reply-To: <p04330100c4e24795d576@[192.168.123.162]>
References: <48BBECF2.5020501@gmail.com>
	<loom.20080901T143254-314@post.gmane.org>
	<48BC9112.3080809@canterbury.ac.nz>
	<p04330100c4e24795d576@[192.168.123.162]>
Message-ID: <48BDCF65.5050902@canterbury.ac.nz>

Tony Nelson wrote:

> I suppose the question is what a capitalized name promises.  If it means
> only "Class", then how should "Returns a new object", either from a Class
> or a Factory, be shown?

Is there really a strong need to show that? There are
many ways in which functions could be categorized.
Is this particular one so important that it warrants
a naming convention?

-- 
Greg

From mhammond at skippinet.com.au  Wed Sep  3 02:04:55 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Wed, 3 Sep 2008 10:04:55 +1000
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <loom.20080902T224020-709@post.gmane.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org>
	<loom.20080902T224020-709@post.gmane.org>
Message-ID: <03b801c90d58$b5744970$205cdc50$@com.au>

> > An alternative to manipulating PATH would be to make and add to the
> > Start Menu a Command Prompt shortcut, call it Command Window or
> > something, that starts in the Python directory.
> 
> The reason for adding the directory to the PATH is for it to be
> recognized in any command prompt, not only the Python-dedicated 
> command prompt shortcut.

Actually, that is *your* reason for adding it to the global path.  But
adding it to the global path has many more side-effects than just allowing
'python' to work at the command-prompt - every running program of yours,
even those not related to command-prompts or Python, has the opportunity to
get confused about the files that might appear in those directories.

> Otherwise, the minute another project decides to do the same thing
> (provide a
> custom command prompt shortcut rather than add its own directory to the
> PATH), the user experience becomes very tedious.

The problem is that if every installer offers to update the path, it gets
quite tedious making corrections and tweaks (eg, I'm never offered the
option of *where* on the path to install it, and sometimes this is very
important!)

FWIW, my opinion is similar to how I read Martin's - that if a suitable,
safe patch that cleanly uninstalls can be found, it should be included, but
disabled by default.  Personally I'd never use it.

Cheers,

Mark


From solipsis at pitrou.net  Wed Sep  3 02:30:21 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Sep 2008 00:30:21 +0000 (UTC)
Subject: [Python-Dev] Add python.exe to PATH environment variable
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org>
	<loom.20080902T224020-709@post.gmane.org>
	<03b801c90d58$b5744970$205cdc50$@com.au>
Message-ID: <loom.20080903T002033-116@post.gmane.org>

Mark Hammond <mhammond <at> skippinet.com.au> writes:
> > 
> > The reason for adding the directory to the PATH is for it to be
> > recognized in any command prompt, not only the Python-dedicated 
> > command prompt shortcut.
> 
> Actually, that is *your* reason for adding it to the global path.

What do you mean? Are there other practical reasons than the above for adding
the directory to the PATH?

>  But
> adding it to the global path has many more side-effects than just allowing
> 'python' to work at the command-prompt - every running program of yours,
> even those not related to command-prompts or Python, has the opportunity to
> get confused about the files that might appear in those directories.

What do you mean by "get confused about the files that might appear in those
directories"?
Unless those running programs of mine deliberately crawl through the PATH (in
which case I assume they have a good reason of doing so), I don't understand in
what way they would "get confused". But I admit I'm not an expert Windows user.

Regards

Antoine.



From greg.ewing at canterbury.ac.nz  Wed Sep  3 02:38:40 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 03 Sep 2008 12:38:40 +1200
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
 multiprocessing
In-Reply-To: <200809030236.41944.steve@pearwood.info>
References: <48BBECF2.5020501@gmail.com>
	<200809030236.41944.steve@pearwood.info>
Message-ID: <48BDDC90.8060007@canterbury.ac.nz>

Steven D'Aprano wrote:
> Why not expose the class directly, instead of 
> making it private and then exposing it via a factory function that does 
> nothing else?

This option would also have the advantage of not
changing the API (unless there's code out there that
actually depends on them *not* being classes, and
that seems rather unlikely).

-- 
Greg

From greg.ewing at canterbury.ac.nz  Wed Sep  3 02:49:17 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 03 Sep 2008 12:49:17 +1200
Subject: [Python-Dev] fileobj.read(float): warning or error?
In-Reply-To: <48BD726A.9030200@jcea.es>
References: <200807212117.14485.victor.stinner@haypocalc.com>
	<20080722053744.GA11290@cskk.homeip.net>
	<ca471dc20807291711m48d5837ap33c9b58f31eb82b5@mail.gmail.com>
	<48BD726A.9030200@jcea.es>
Message-ID: <48BDDF0D.8030900@canterbury.ac.nz>

Jesus Cea wrote:

> How do you differenciate between that empty string (when doing
> "read(0)"), from EOF (that is signaled by an empty string)?.

If you need to be able to make that distinction, then
you have to be careful not to try to read 0 bytes.

Personally I've never come across a situation where
allowing read(0) to occur would have simplified the
code. In the usual keep-reading-until-we've-got-the-
required-number-of-bytes scenario, you're checking
for 0 bytes left to read in order to tell when to
stop.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Wed Sep  3 03:27:56 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 03 Sep 2008 13:27:56 +1200
Subject: [Python-Dev] fileobj.read(float): warning or error?
In-Reply-To: <48BD8690.6060405@jcea.es>
References: <200807212117.14485.victor.stinner@haypocalc.com>
	<20080722053744.GA11290@cskk.homeip.net>
	<ca471dc20807291711m48d5837ap33c9b58f31eb82b5@mail.gmail.com>
	<48BD726A.9030200@jcea.es>
	<Pine.GSO.4.64.0809021313050.19901@core.cs.uwaterloo.ca>
	<48BD8690.6060405@jcea.es>
Message-ID: <48BDE81C.6090706@canterbury.ac.nz>

Jesus Cea wrote:

> My point is: we are simplifying the program considering "0" a valid len
> counter, but we complicates it because now the code can't consider "" =
> EOF if it actually asked for 0 bytes.

What are you suggesting read(0) *should* do, then?
If it returns None or some other special value, or
raises an exception, then you need a special case
to handle that. So you've just substituted one special
case for another.

 > Isaac Morland wrote:
 >
> > The Unix read() system call doesn't treat EOF as special other than it
> > won't return bytes from "beyond" EOF and therefore even when reading a
> > regular file could return fewer (including 0) bytes than asked for in
> > the call.

No, that's not right -- a read of more than 0 bytes will
always block until at least 1 byte is available, or
something happens that counts as an EOF condition.

However, with some devices it's possible for what
counts as EOF to happen more than once, e.g. ttys.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Wed Sep  3 03:45:48 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 03 Sep 2008 13:45:48 +1200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <g9kaav$hj2$1@ger.gmane.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org>
Message-ID: <48BDEC4C.7050804@canterbury.ac.nz>

Terry Reedy wrote:

> An alternative to manipulating PATH would be to make and add to the 
> Start Menu a Command Prompt shortcut, call it Command Window or 
> something, that starts in the Python directory.

That doesn't seem very satisfactory, because the user
is going to want to work in the directory containing
his script, or the data he wants to run the script on.

A better way would be to start a command process with
the Python directory added to PATH just for that
process.

How easy would that be to do on Windows? Do environment
variables get inherited by child processes the way
they do on Unix?

-- 
Greg

From tjreedy at udel.edu  Wed Sep  3 04:18:32 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 02 Sep 2008 22:18:32 -0400
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BDC325.1000901@egenix.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>
	<48BDC325.1000901@egenix.com>
Message-ID: <g9ks5n$um4$1@ger.gmane.org>



M.-A. Lemburg wrote:
> On 2008-09-02 23:14, Terry Reedy wrote:

>> An alternative to manipulating PATH would be to make and add to the
>> Start Menu a Command Prompt shortcut, call it Command Window or
>> something, that starts in the Python directory.  Then one could enter
>>> python or >Scripts/goforit without chdir-ing to the Python directory
>> first.  The background could even be set to green, for instance, to
>> differentiate it from the standard Command Prompt window.
> 
> There already is a menu entry that starts the Python interpreter
> on Windows, so why not use that ?

I do and will*, but some people want to start a command window and then 
type 'python' just like they would on *nix (*without having to 
explicitly change to the Python directory*), or be able to run stuff in 
a subdirectory of the Python directory.  So I suggested an alternative 
to the request for PATH manipulation that could easily be done now.

* I recently *did* run Python from a command prompt so I could make sure 
it was running 'cmd.exe /u' and try changing the code page (but this is 
another story), but I simply moved to the directory first.  For me, not 
a big deal.

tjr


From greg.ewing at canterbury.ac.nz  Wed Sep  3 04:12:57 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 03 Sep 2008 14:12:57 +1200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BDC325.1000901@egenix.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
Message-ID: <48BDF2A9.5060203@canterbury.ac.nz>

M.-A. Lemburg wrote:

> The problem is: how to undo those changes without accidentally
> undoing an explicit change made by the user ?

Is that really much of an issue? If the PATH contains an
entry corresponding to the Python installation that's
being uninstalled, then it's not going to work once the
installation is gone, so removing it isn't going to do
much harm.

In any case, the danger could be reduced by picking
some distinctive name for a new environment variable that
a user isn't likely to come up with on their own, such
as __AUTOPYEXECDIR__, setting that to the Python directory,
and adding it to PATH. The uninstaller can then be fairly
selective about what it removes.

> BTW: Adding the Python dir to the PATH per default would cause
> problems for users who regularly have multiple different
> Python installations on a machine.

No more problem than having it set the file associations,
as far as I can see. If you have multiple Pythons, you're
going to have to be explicit about which one you want
from the command shell anyway, and not rely on a PATH
setting.

> If this is done, it should be an install option and not forced.

Certainly it should be an option. I'm not sure about
having it disabled by default, though, since naive users
are the ones that stand to benefit most from it, yet
they're least likely to know that they need to turn it
on.

-- 
Greg

From mhammond at skippinet.com.au  Wed Sep  3 04:22:55 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Wed, 3 Sep 2008 12:22:55 +1000
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <loom.20080903T002033-116@post.gmane.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org>
	<loom.20080902T224020-709@post.gmane.org>
	<03b801c90d58$b5744970$205cdc50$@com.au>
	<loom.20080903T002033-116@post.gmane.org>
Message-ID: <03d001c90d6b$fc7b2380$f5716a80$@com.au>

> Mark Hammond <mhammond <at> skippinet.com.au> writes:
> > >
> > > The reason for adding the directory to the PATH is for it to be
> > > recognized in any command prompt, not only the Python-dedicated
> > > command prompt shortcut.
> >
> > Actually, that is *your* reason for adding it to the global path.
> 
> What do you mean? Are there other practical reasons than the above for
> adding the directory to the PATH?

My point was that it applies to much more than command-prompts.  There are
pratical reasons for adding entries to your global PATH that don't include
the ability to run executables at the command-line.

> What do you mean by "get confused about the files that might appear in
> those directories"?

I mean that many Windows use the PATH, and as such, may fail if a new
directory is added to the PATH that contains a DLL they indirectly use.

I don't need any program *except* a command-prompt to be able to locate a
'python.exe', so I arrange for only command-prompts to have their PATH setup
that way.  If I *did* expect other programs to be able to find a
'python.exe', I'm not sure I'd want to risk that installing a later version
of Python will change what Python is found.

Cheers,

Mark


From curt at hagenlocher.org  Wed Sep  3 07:39:55 2008
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Tue, 2 Sep 2008 22:39:55 -0700
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <d2155e360809022238l60dd52f4ha8990487bfe04fc3@mail.gmail.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDEC4C.7050804@canterbury.ac.nz>
	<d2155e360809022238l60dd52f4ha8990487bfe04fc3@mail.gmail.com>
Message-ID: <d2155e360809022239n6d5b2886rf5823b780a360832@mail.gmail.com>

One of these days, I'm actually going to remember that I need to click
"Reply All" when posting to this list... .  Sorry for the duplicate,
Greg.

On Tue, Sep 2, 2008 at 6:45 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
>
> A better way would be to start a command process with
> the Python directory added to PATH just for that
> process.

This is similar to what Visual Studio or the Windows SDK do to give
you a command prompt with the PATH and other environmental variables
setup "correctly" -- they add a shortcut to a batch file that's set to
keep the command prompt open after the batch file runs.

> How easy would that be to do on Windows? Do environment
> variables get inherited by child processes the way
> they do on Unix?

Generally, yes.  I think there's a catch in that there are ways to
start processes that don't make them children of your process, but I
don't remember why I think that.

One other reason not to mess with the PATH -- at least by default --
is that the user may have multiple copies of Python installed.  I know
I have at least one machine with 2.4.5, 2.5.2, 2.6b2 and 3.0b2
installed -- and I don't want *any* of them in my path.

--
Curt Hagenlocher
curt at hagenlocher.org

From ziade.tarek at gmail.com  Wed Sep  3 09:57:12 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 3 Sep 2008 09:57:12 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <d2155e360809022239n6d5b2886rf5823b780a360832@mail.gmail.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDEC4C.7050804@canterbury.ac.nz>
	<d2155e360809022238l60dd52f4ha8990487bfe04fc3@mail.gmail.com>
	<d2155e360809022239n6d5b2886rf5823b780a360832@mail.gmail.com>
Message-ID: <94bdd2610809030057j7f899bbj8689df728ce0ca03@mail.gmail.com>

On Wed, Sep 3, 2008 at 7:39 AM, Curt Hagenlocher <curt at hagenlocher.org>wrote:

>
> One other reason not to mess with the PATH -- at least by default --
> is that the user may have multiple copies of Python installed.  I know
> I have at least one machine with 2.4.5, 2.5.2, 2.6b2 and 3.0b2
> installed -- and I don't want *any* of them in my path.


Why ? you have them under Linux, through different names (python2.4,
python2.5, etc)
maybe this should be the same under Windows, e.g. having a common directory
where
all Python installations add a binary with a specific name, and same thing
for binaries in Scripts,
(easy_install-2.4, easy_install-2.5)

a /usr/local/bin for Windows in a way... system32 ?

Tarek

>
>
> --
> Curt Hagenlocher
> curt at hagenlocher.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/ziade.tarek%40gmail.com
>



-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080903/d7643e39/attachment.htm>

From cesare.dimauro at a-tono.com  Wed Sep  3 10:15:37 2008
From: cesare.dimauro at a-tono.com (Cesare Di Mauro)
Date: Wed, 03 Sep 2008 10:15:37 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BDC325.1000901@egenix.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
Message-ID: <op.ugwbwbqy03jqhe@cesareprova.org>

On 03 sep 2008 at 00:50:13, M.-A. Lemburg <mal at egenix.com> wrote:

> There already is a menu entry that starts the Python interpreter
> on Windows, so why not use that ?

Because i need to start Python from folders which have
files that define a specific "environment".

I have several servers and applications that develop and test this way.

> Also .py files are automatically associated with the last installed
> Python interpreter, so the double-clicking on .py files works and is
> probably the most common way of starting a Python file on Windows.

99% of time I run Python from a command prompt (on specific
directories).

I use the default menu entry only when I have to play with Python to test some
pieces of code.

> Adding paths to the PATH variable is not easy on Windows, esp. if
> you want to support multiple Windows versions. The global PATH
> settings are not read from autoexec.bat anymore (only once at boot
> time). Instead those environment variables are managed via the
> registry.
>
> See e.g.
>
> http://agiletesting.blogspot.com/2005/06/handling-path-windows-registry-value.html
>
> for how to setup PATH to your liking using Python.
>
> The problem is: how to undo those changes without accidentally
> undoing an explicit change made by the user ?
>
> BTW: Adding the Python dir to the PATH per default would cause
> problems for users who regularly have multiple different
> Python installations on a machine. If this is done, it should
> be an install option and not forced.

Let the user to decide to update or not the PATH envar by marking a
chechbox in the setup process, displaying that doing that the
changes will NOT be reverted when uninstalling it.

Cheers,
Cesare

From ncoghlan at gmail.com  Wed Sep  3 12:25:11 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 03 Sep 2008 20:25:11 +1000
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
 multiprocessing
In-Reply-To: <48BDDC90.8060007@canterbury.ac.nz>
References: <48BBECF2.5020501@gmail.com>	<200809030236.41944.steve@pearwood.info>
	<48BDDC90.8060007@canterbury.ac.nz>
Message-ID: <48BE6607.4010203@gmail.com>

Greg Ewing wrote:
> Steven D'Aprano wrote:
>> Why not expose the class directly, instead of making it private and
>> then exposing it via a factory function that does nothing else?
> 
> This option would also have the advantage of not
> changing the API (unless there's code out there that
> actually depends on them *not* being classes, and
> that seems rather unlikely).

The multiprocessing implementation currently appears to have an annoying
circular import involving the multiprocessing package and the
_multiprocessing extension module - the factory functions in
multiprocessing.__init__ currently delay some of the relevant imports
long enough to keep any problems from actually occurring.

At some point that circular import should probably be eliminated (e.g.
via a _mp_bootstrap module), but until that happens at least some of the
factory functions in multiprocessing are actually required in order for
it to be possible to import the package at all.

And, as Guido already noted, leaving them as factory functions with
private implementations means users can't come to rely on the objects
being subclassable, and thus potentially allows more efficient
implementations (e.g. using near native locking implementations directly).

The only unfortunate aspect of the API is that the current names suggest
that these are factory functions rather than classes - if it hadn't been
for that, the question would have never even been raised. Since we
already have guidelines in place for dealing with ordinary API renames,
this is no longer something that strikes me as particularly urgent in
the 2.6/3.0 timeframe.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From mal at egenix.com  Wed Sep  3 13:25:47 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 03 Sep 2008 13:25:47 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BDF2A9.5060203@canterbury.ac.nz>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>
	<48BDC325.1000901@egenix.com> <48BDF2A9.5060203@canterbury.ac.nz>
Message-ID: <48BE743B.1000108@egenix.com>

On 2008-09-03 04:12, Greg Ewing wrote:
> M.-A. Lemburg wrote:
> 
>> The problem is: how to undo those changes without accidentally
>> undoing an explicit change made by the user ?
> 
> Is that really much of an issue? If the PATH contains an
> entry corresponding to the Python installation that's
> being uninstalled, then it's not going to work once the
> installation is gone, so removing it isn't going to do
> much harm.

You have a point there :-)

> In any case, the danger could be reduced by picking
> some distinctive name for a new environment variable that
> a user isn't likely to come up with on their own, such
> as __AUTOPYEXECDIR__, setting that to the Python directory,
> and adding it to PATH. The uninstaller can then be fairly
> selective about what it removes.
> 
>> BTW: Adding the Python dir to the PATH per default would cause
>> problems for users who regularly have multiple different
>> Python installations on a machine.
> 
> No more problem than having it set the file associations,
> as far as I can see. If you have multiple Pythons, you're
> going to have to be explicit about which one you want
> from the command shell anyway, and not rely on a PATH
> setting.

True, I have configured Windows to provide several
"Open with Python x.x" in order to have more flexibility.

However, always having the latest version on PATH is not
an option either, since e.g. I wouldn't want all .py scripts
to be run by Python 3.0 just because I installed it for
testing purposes.

>> If this is done, it should be an install option and not forced.
> 
> Certainly it should be an option. I'm not sure about
> having it disabled by default, though, since naive users
> are the ones that stand to benefit most from it, yet
> they're least likely to know that they need to turn it
> on.

In my experience, Windows apps seem to be moving aways from
cluttering up PATH and include each and every single app
dir to it.

Instead, if they bother at all, they simply place a .bat or
small .exe into the Window system dir (which already is
on PATH).

Perhaps we could have an option to place a "python.bat"
into C:\Windows\ or C:\Windows\System\.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 03 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From mal at egenix.com  Wed Sep  3 13:34:18 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Wed, 03 Sep 2008 13:34:18 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <op.ugwbwbqy03jqhe@cesareprova.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>
	<48BDC325.1000901@egenix.com> <op.ugwbwbqy03jqhe@cesareprova.org>
Message-ID: <48BE763A.1040408@egenix.com>

On 2008-09-03 10:15, Cesare Di Mauro wrote:
> On 03 sep 2008 at 00:50:13, M.-A. Lemburg <mal at egenix.com> wrote:
> 
>> There already is a menu entry that starts the Python interpreter
>> on Windows, so why not use that ?
> 
> Because i need to start Python from folders which have
> files that define a specific "environment".
> 
> I have several servers and applications that develop and test this way.

Same here, but I usually have a env.bat that sets up whatever
environment I need (including the required Python version) and
run that when opening a prompt to work on a particular project.

>> Also .py files are automatically associated with the last installed
>> Python interpreter, so the double-clicking on .py files works and is
>> probably the most common way of starting a Python file on Windows.
> 
> 99% of time I run Python from a command prompt (on specific
> directories).
> 
> I use the default menu entry only when I have to play with Python to test some
> pieces of code.

IMHO, the only point of having the installer do this for the user
is for the case where the user does not know how to manipulate
PATH on Windows, but still wants to use the command line to access it
directly.

How many users would fit that category ?

>> Adding paths to the PATH variable is not easy on Windows, esp. if
>> you want to support multiple Windows versions. The global PATH
>> settings are not read from autoexec.bat anymore (only once at boot
>> time). Instead those environment variables are managed via the
>> registry.
>>
>> See e.g.
>>
>> http://agiletesting.blogspot.com/2005/06/handling-path-windows-registry-value.html
>>
>> for how to setup PATH to your liking using Python.
>>
>> The problem is: how to undo those changes without accidentally
>> undoing an explicit change made by the user ?
>>
>> BTW: Adding the Python dir to the PATH per default would cause
>> problems for users who regularly have multiple different
>> Python installations on a machine. If this is done, it should
>> be an install option and not forced.
> 
> Let the user to decide to update or not the PATH envar by marking a
> chechbox in the setup process, displaying that doing that the
> changes will NOT be reverted when uninstalling it.

Hmm, I don't think that's a good way to go about this. The uninstall
should undo all changes.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 03 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From eric at trueblade.com  Wed Sep  3 13:59:10 2008
From: eric at trueblade.com (Eric Smith)
Date: Wed, 03 Sep 2008 07:59:10 -0400
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BE743B.1000108@egenix.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>	<48BDC325.1000901@egenix.com>
	<48BDF2A9.5060203@canterbury.ac.nz> <48BE743B.1000108@egenix.com>
Message-ID: <48BE7C0E.3080903@trueblade.com>

M.-A. Lemburg wrote:
> On 2008-09-03 04:12, Greg Ewing wrote:
>> M.-A. Lemburg wrote:
>>
>>> The problem is: how to undo those changes without accidentally
>>> undoing an explicit change made by the user ?
>> Is that really much of an issue? If the PATH contains an
>> entry corresponding to the Python installation that's
>> being uninstalled, then it's not going to work once the
>> installation is gone, so removing it isn't going to do
>> much harm.
> 
> You have a point there :-)

Unless you install it with the "add to path" option, deinstall it, and 
the reinstall without the "add to path" option. That is, assuming the 
install directory is the same every time.

Not that I think this is common, except in testing scenarios.

 > Perhaps we could have an option to place a "python.bat"
 > into C:\Windows\ or C:\Windows\System\.

Except you still have the "last in wins" issue, and you have to make a 
decision on whether or not to delete the file.  But I agree that 
managing a single batch file is easier than dealing with the PATH 
variable, and has fewer side effects (finding DLL's, etc.).

Eric.

From ijmorlan at cs.uwaterloo.ca  Wed Sep  3 14:55:33 2008
From: ijmorlan at cs.uwaterloo.ca (Isaac Morland)
Date: Wed, 3 Sep 2008 08:55:33 -0400 (EDT)
Subject: [Python-Dev] fileobj.read(float): warning or error?
In-Reply-To: <48BDE81C.6090706@canterbury.ac.nz>
References: <200807212117.14485.victor.stinner@haypocalc.com>
	<20080722053744.GA11290@cskk.homeip.net>
	<ca471dc20807291711m48d5837ap33c9b58f31eb82b5@mail.gmail.com>
	<48BD726A.9030200@jcea.es>
	<Pine.GSO.4.64.0809021313050.19901@core.cs.uwaterloo.ca>
	<48BD8690.6060405@jcea.es> <48BDE81C.6090706@canterbury.ac.nz>
Message-ID: <Pine.GSO.4.64.0809030853130.26304@core.cs.uwaterloo.ca>

On Wed, 3 Sep 2008, Greg Ewing wrote:

>> > The Unix read() system call doesn't treat EOF as special other than it
>> > won't return bytes from "beyond" EOF and therefore even when reading a
>> > regular file could return fewer (including 0) bytes than asked for in
>> > the call.
>
> No, that's not right -- a read of more than 0 bytes will
> always block until at least 1 byte is available, or
> something happens that counts as an EOF condition.
>
> However, with some devices it's possible for what
> counts as EOF to happen more than once, e.g. ttys.

Sorry, you're absolutely right.  I was thinking only of the fact that 
read() at EOF is not an error, rather than the blocking behaviour.  It 
sounds like Python read() really is very similar to Unix read() in 
behaviour.

Isaac Morland			CSCF Web Guru
DC 2554C, x36650		WWW Software Specialist

From theller at ctypes.org  Wed Sep  3 15:10:27 2008
From: theller at ctypes.org (Thomas Heller)
Date: Wed, 03 Sep 2008 15:10:27 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BE7C0E.3080903@trueblade.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>	<48BDC325.1000901@egenix.com>	<48BDF2A9.5060203@canterbury.ac.nz>
	<48BE743B.1000108@egenix.com> <48BE7C0E.3080903@trueblade.com>
Message-ID: <g9m2bt$66i$1@ger.gmane.org>

>  > Perhaps we could have an option to place a "python.bat"
>  > into C:\Windows\ or C:\Windows\System\.
> 
> Except you still have the "last in wins" issue, and you have to make a 
> decision on whether or not to delete the file.

If this is done the batch file should be named "python25.bat" or so
depending on the version.

-- 
Thanks,
Thomas


From khamidou at gmail.com  Wed Sep  3 15:17:26 2008
From: khamidou at gmail.com (karim hamidou)
Date: Wed, 3 Sep 2008 15:17:26 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <g9m2bt$66i$1@ger.gmane.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
	<48BDF2A9.5060203@canterbury.ac.nz> <48BE743B.1000108@egenix.com>
	<48BE7C0E.3080903@trueblade.com> <g9m2bt$66i$1@ger.gmane.org>
Message-ID: <9396537b0809030617r217fe59y6d52352f13dad720@mail.gmail.com>

On Wed, Sep 3, 2008 at 3:10 PM, Thomas Heller <theller at ctypes.org> wrote:
>>  > Perhaps we could have an option to place a "python.bat"
>>  > into C:\Windows\ or C:\Windows\System\.
>>
> If this is done the batch file should be named "python25.bat" or so
> depending on the version.
>
Instead of having a .bat file we could have a helper program that sets
the appropriate PATH and then runs cmd.exe.
It would be available from the start menu. This way, we wouldn't
clutter C:\Windows\System\ with batch files.

From p.f.moore at gmail.com  Wed Sep  3 15:46:29 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 3 Sep 2008 14:46:29 +0100
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <g9m2bt$66i$1@ger.gmane.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
	<48BDF2A9.5060203@canterbury.ac.nz> <48BE743B.1000108@egenix.com>
	<48BE7C0E.3080903@trueblade.com> <g9m2bt$66i$1@ger.gmane.org>
Message-ID: <79990c6b0809030646s15166e71hb910b57d4d15a870@mail.gmail.com>

On 03/09/2008, Thomas Heller <theller at ctypes.org> wrote:
> >  > Perhaps we could have an option to place a "python.bat"
> >  > into C:\Windows\ or C:\Windows\System\.
> >
> > Except you still have the "last in wins" issue, and you have to make a
> > decision on whether or not to delete the file.
>
> If this is done the batch file should be named "python25.bat" or so
> depending on the version.

Bat files don't work when called from another bat file. This hits me
regularly, when people supply wrapper bat files. Example:

myscript.bat:
@echo off
do some stuff
python my_py_script.py
do some more stuff

If "python" is a bat file, "do some more stuff" will never get run
(you need to say "call python" to allow bat files to nest).

-1 (or stronger!) on a bat file.

If it goes in anyway, make it optional and off by default. The
confusion and wasted time caused by bat files "stopping" part way
through is FAR worse than that caused by having to manage PATH for
yourself.

Paul.

PS If anyone knows a *good* way of writing wrapper scripts on Windows
which doesn't suffer from the bat file nesting problem above, please
let me (and the rest of the world!) know!

From mail at timgolden.me.uk  Wed Sep  3 15:54:35 2008
From: mail at timgolden.me.uk (Tim Golden)
Date: Wed, 03 Sep 2008 14:54:35 +0100
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <79990c6b0809030646s15166e71hb910b57d4d15a870@mail.gmail.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>
	<48BDC325.1000901@egenix.com>	<48BDF2A9.5060203@canterbury.ac.nz>
	<48BE743B.1000108@egenix.com>	<48BE7C0E.3080903@trueblade.com>
	<g9m2bt$66i$1@ger.gmane.org>
	<79990c6b0809030646s15166e71hb910b57d4d15a870@mail.gmail.com>
Message-ID: <48BE971B.6020403@timgolden.me.uk>

Paul Moore wrote:
> On 03/09/2008, Thomas Heller <theller at ctypes.org> wrote:
>>>  > Perhaps we could have an option to place a "python.bat"
>>>  > into C:\Windows\ or C:\Windows\System\.
>>>
>>> Except you still have the "last in wins" issue, and you have to make a
>>> decision on whether or not to delete the file.
>> If this is done the batch file should be named "python25.bat" or so
>> depending on the version.
> 
> Bat files don't work when called from another bat file. 

You can use "CALL" within one batch file to chain
another, returning afterwards to the first. But
this is obviously not the most transparent thing
on earth!

TJG

From theller at ctypes.org  Wed Sep  3 16:05:20 2008
From: theller at ctypes.org (Thomas Heller)
Date: Wed, 03 Sep 2008 16:05:20 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <79990c6b0809030646s15166e71hb910b57d4d15a870@mail.gmail.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>
	<48BDC325.1000901@egenix.com>	<48BDF2A9.5060203@canterbury.ac.nz>
	<48BE743B.1000108@egenix.com>	<48BE7C0E.3080903@trueblade.com>
	<g9m2bt$66i$1@ger.gmane.org>
	<79990c6b0809030646s15166e71hb910b57d4d15a870@mail.gmail.com>
Message-ID: <g9m5io$j4j$1@ger.gmane.org>

Paul Moore schrieb:
> Bat files don't work when called from another bat file. This hits me
> regularly, when people supply wrapper bat files. Example:
> 
> myscript.bat:
> @echo off
> do some stuff
> python my_py_script.py
> do some more stuff
> 
> If "python" is a bat file, "do some more stuff" will never get run
> (you need to say "call python" to allow bat files to nest).
> 
> -1 (or stronger!) on a bat file.
> 
> If it goes in anyway, make it optional and off by default. The
> confusion and wasted time caused by bat files "stopping" part way
> through is FAR worse than that caused by having to manage PATH for
> yourself.
> 
> Paul.
> 
> PS If anyone knows a *good* way of writing wrapper scripts on Windows
> which doesn't suffer from the bat file nesting problem above, please
> let me (and the rest of the world!) know!

/F's exemaker and Philip's setuptools script wrapper come to mind.

-- 
Thanks,
Thomas


From p.f.moore at gmail.com  Wed Sep  3 16:07:14 2008
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 3 Sep 2008 15:07:14 +0100
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BE971B.6020403@timgolden.me.uk>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
	<48BDF2A9.5060203@canterbury.ac.nz> <48BE743B.1000108@egenix.com>
	<48BE7C0E.3080903@trueblade.com> <g9m2bt$66i$1@ger.gmane.org>
	<79990c6b0809030646s15166e71hb910b57d4d15a870@mail.gmail.com>
	<48BE971B.6020403@timgolden.me.uk>
Message-ID: <79990c6b0809030707s281ea913m66ab565b075a9287@mail.gmail.com>

On 03/09/2008, Tim Golden <mail at timgolden.me.uk> wrote:
> You can use "CALL" within one batch file to chain
> another, returning afterwards to the first.

Correct. Sorry, I forgot to mention that.

> But this is obviously not the most transparent thing
> on earth!

Indeed - and it certainly isn't a "wrapper" in the sense that I would
use the word (which is more in the sense of "drop-in replacement").

Paul.

From ziade.tarek at gmail.com  Wed Sep  3 16:11:23 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Wed, 3 Sep 2008 16:11:23 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <79990c6b0809030646s15166e71hb910b57d4d15a870@mail.gmail.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
	<48BDF2A9.5060203@canterbury.ac.nz> <48BE743B.1000108@egenix.com>
	<48BE7C0E.3080903@trueblade.com> <g9m2bt$66i$1@ger.gmane.org>
	<79990c6b0809030646s15166e71hb910b57d4d15a870@mail.gmail.com>
Message-ID: <94bdd2610809030711x31f40ae2l8aa61221703944fc@mail.gmail.com>

On Wed, Sep 3, 2008 at 3:46 PM, Paul Moore <p.f.moore at gmail.com> wrote:

>
>
> PS If anyone knows a *good* way of writing wrapper scripts on Windows
> which doesn't suffer from the bat file nesting problem above, please
> let me (and the rest of the world!) know!


You can use setuptools console scripts, they are added in the Scripts
directory by default,
but this is optional

see :
http://peak.telecommunity.com/DevCenter/setuptools#automatic-script-creation

We write all our wrappers with it and... hum.. add /Scripts to PATH under
Windows :D

Tarek
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080903/b79e4a97/attachment.htm>

From cesare.dimauro at a-tono.com  Wed Sep  3 16:20:09 2008
From: cesare.dimauro at a-tono.com (Cesare Di Mauro)
Date: Wed, 03 Sep 2008 16:20:09 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BE763A.1040408@egenix.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
	<op.ugwbwbqy03jqhe@cesareprova.org> <48BE763A.1040408@egenix.com>
Message-ID: <op.ugwsrved03jqhe@cesareprova.org>

On 03 Sep 2008 at 13:34:18, M.-A. Lemburg <mal at egenix.com> wrote:

> Same here, but I usually have a env.bat that sets up whatever
> environment I need (including the required Python version) and
> run that when opening a prompt to work on a particular project.

> IMHO, the only point of having the installer do this for the user
> is for the case where the user does not know how to manipulate
> PATH on Windows, but still wants to use the command line to access it
> directly.
>
> How many users would fit that category ?

I don't know, but when you are working with just one python instance,
there's no need to use batch files neither to care about what PATH says:
it's always the right one instance that will be picked. :)

>> Let the user to decide to update or not the PATH envar by marking a
>> chechbox in the setup process, displaying that doing that the
>> changes will NOT be reverted when uninstalling it.
>
> Hmm, I don't think that's a good way to go about this. The uninstall
> should undo all changes.

The problem here is that uninstaller doesn't work this way.

May be an external program called by the uninstaller can take care
of this, removing what was added to PATH.

Cesare

From solipsis at pitrou.net  Wed Sep  3 16:23:27 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Sep 2008 14:23:27 +0000 (UTC)
Subject: [Python-Dev] Add python.exe to PATH environment variable
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org>
	<loom.20080902T224020-709@post.gmane.org>
	<03b801c90d58$b5744970$205cdc50$@com.au>
	<loom.20080903T002033-116@post.gmane.org>
	<03d001c90d6b$fc7b2380$f5716a80$@com.au>
Message-ID: <loom.20080903T141921-651@post.gmane.org>

Mark Hammond <mhammond <at> skippinet.com.au> writes:
> 
> I mean that many Windows use the PATH, and as such, may fail if a new
> directory is added to the PATH that contains a DLL they indirectly use.

Then it's just a matter of not putting any DLLs in those directories, isn't it?

> If I *did* expect other programs to be able to find a
> 'python.exe', I'm not sure I'd want to risk that installing a later version
> of Python will change what Python is found.

Most Linux distributions solve that by installing binaries named
/usr/bin/python2.4, /usr/bin/python2.5, etc., and making /usr/bin/python a
symlink to one of those. Thus if a program relies on particular Python version,
it can just use a specific executable rather than a generic one.




From skip at pobox.com  Wed Sep  3 16:39:33 2008
From: skip at pobox.com (skip at pobox.com)
Date: Wed, 3 Sep 2008 09:39:33 -0500
Subject: [Python-Dev] 2.6 doc searching issue?
Message-ID: <18622.41381.600223.440920@montanaro-dyndns-org.local>

>From this page:

    http://docs.python.org/dev/index.html

I searched for "csv" and got just one hit:

    http://docs.python.org/dev/contents.html?highlight=csv

Shouldn't it have at least matched the docs for the csv module itself, not
just the table of contents?

Thx,

Skip

From techtonik at gmail.com  Wed Sep  3 16:43:46 2008
From: techtonik at gmail.com (techtonik)
Date: Wed, 3 Sep 2008 17:43:46 +0300
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
Message-ID: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>

SVN checkout over HTTPS protocol requires password. Is it intentional
or just temporary server issue? I am behind a proxy that doesn't
support PROPFIND requests and I can't test SVN+SSH, because 22 port is
closed.

Site docs keep silence about that HTTPS is used at all. Shouldn't
authentication be asked only on write access in normal SVN setup?


BTW, link to "current svn tree" is broken here -
http://www.python.org/dev/patches/
-- 
--anatoly t.

From solipsis at pitrou.net  Wed Sep  3 18:08:07 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Sep 2008 16:08:07 +0000 (UTC)
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
Message-ID: <loom.20080903T160547-830@post.gmane.org>

techtonik <techtonik <at> gmail.com> writes:
> 
> SVN checkout over HTTPS protocol requires password. Is it intentional
> or just temporary server issue? I am behind a proxy that doesn't
> support PROPFIND requests and I can't test SVN+SSH, because 22 port is
> closed.

As a workaround, if you only need read-only access, you can use the Mercurial
mirrors which should work through your proxy (AFAIK Mercurial only uses GET and
POST).

Type "hg clone http://code.python.org/hg/trunk/" or "hg clone
http://code.python.org/hg/branches/py3k" depending on what you need exactly.

Regards

Antoine.



From guido at python.org  Wed Sep  3 18:13:58 2008
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 Sep 2008 09:13:58 -0700
Subject: [Python-Dev] Further PEP 8 compliance issues in threading and
	multiprocessing
In-Reply-To: <48BDDC90.8060007@canterbury.ac.nz>
References: <48BBECF2.5020501@gmail.com>
	<200809030236.41944.steve@pearwood.info>
	<48BDDC90.8060007@canterbury.ac.nz>
Message-ID: <ca471dc20809030913k387c5839jbcec1fe07bef541c@mail.gmail.com>

2008/9/2 Greg Ewing <greg.ewing at canterbury.ac.nz>:
> Steven D'Aprano wrote:
>>
>> Why not expose the class directly, instead of making it private and then
exposing it via a factory function that does nothing else?
>
> This option would also have the advantage of not
> changing the API (unless there's code out there that
> actually depends on them *not* being classes, and
> that seems rather unlikely).

No. Allowing them to be subclassed makes it harder to replace them on some
platforms with equivalent but faster implementations.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080903/8ae51b43/attachment.htm>

From techtonik at gmail.com  Thu Sep  4 00:39:07 2008
From: techtonik at gmail.com (techtonik)
Date: Thu, 4 Sep 2008 00:39:07 +0200
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <loom.20080903T160547-830@post.gmane.org>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
	<loom.20080903T160547-830@post.gmane.org>
Message-ID: <d34314100809031539m21fbfd00j555e6ec9c4bf15cd@mail.gmail.com>

On Wed, Sep 3, 2008 at 6:08 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>
> As a workaround, if you only need read-only access, you can use the Mercurial
> mirrors which should work through your proxy (AFAIK Mercurial only uses GET and
> POST).
>
> Type "hg clone http://code.python.org/hg/trunk/" or "hg clone
> http://code.python.org/hg/branches/py3k" depending on what you need exactly.

I do not need the whole branch - only a small subset of files related
to distutils. I know that bazaar can't do partial checkouts - it can
only fetch the whole branch. What about mercurial?

And why not to setup HTTPS for anonymous read and authorized write
access? It is not that hard to do and will solve many problems with
proxies.

-- 
--anatoly t.

From brett at python.org  Thu Sep  4 01:08:59 2008
From: brett at python.org (Brett Cannon)
Date: Wed, 3 Sep 2008 16:08:59 -0700
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <d34314100809031539m21fbfd00j555e6ec9c4bf15cd@mail.gmail.com>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
	<loom.20080903T160547-830@post.gmane.org>
	<d34314100809031539m21fbfd00j555e6ec9c4bf15cd@mail.gmail.com>
Message-ID: <bbaeab100809031608yfc70269y853ea0b1f1c2a2e5@mail.gmail.com>

On Wed, Sep 3, 2008 at 3:39 PM, techtonik <techtonik at gmail.com> wrote:
> On Wed, Sep 3, 2008 at 6:08 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>> As a workaround, if you only need read-only access, you can use the Mercurial
>> mirrors which should work through your proxy (AFAIK Mercurial only uses GET and
>> POST).
>>
>> Type "hg clone http://code.python.org/hg/trunk/" or "hg clone
>> http://code.python.org/hg/branches/py3k" depending on what you need exactly.
>
> I do not need the whole branch - only a small subset of files related
> to distutils. I know that bazaar can't do partial checkouts - it can
> only fetch the whole branch. What about mercurial?
>
> And why not to setup HTTPS for anonymous read and authorized write
> access? It is not that hard to do and will solve many problems with
> proxies.

Because it requires setting up a certificate. You can use HTTP to do a
read-only checkout.

-Brett

From solipsis at pitrou.net  Thu Sep  4 00:48:42 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 3 Sep 2008 22:48:42 +0000 (UTC)
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
	<loom.20080903T160547-830@post.gmane.org>
	<d34314100809031539m21fbfd00j555e6ec9c4bf15cd@mail.gmail.com>
Message-ID: <loom.20080903T224344-812@post.gmane.org>

techtonik <techtonik <at> gmail.com> writes:
> 
> I do not need the whole branch - only a small subset of files related
> to distutils. I know that bazaar can't do partial checkouts - it can
> only fetch the whole branch. What about mercurial?

Mercurial can't do it either. But I don't think it matters a lot; unless your
bandwidth is very low the full clone should be fast enough. Here it takes just
150s. for the trunk.

> And why not to setup HTTPS for anonymous read and authorized write
> access? It is not that hard to do and will solve many problems with
> proxies.

I don't know really, someone else should answer.




From musiccomposition at gmail.com  Thu Sep  4 01:18:33 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Wed, 3 Sep 2008 18:18:33 -0500
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <bbaeab100809031608yfc70269y853ea0b1f1c2a2e5@mail.gmail.com>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
	<loom.20080903T160547-830@post.gmane.org>
	<d34314100809031539m21fbfd00j555e6ec9c4bf15cd@mail.gmail.com>
	<bbaeab100809031608yfc70269y853ea0b1f1c2a2e5@mail.gmail.com>
Message-ID: <1afaf6160809031618q3ff6f068jbf80561e8774755@mail.gmail.com>

On Wed, Sep 3, 2008 at 6:08 PM, Brett Cannon <brett at python.org> wrote:
> On Wed, Sep 3, 2008 at 3:39 PM, techtonik <techtonik at gmail.com> wrote:
>> On Wed, Sep 3, 2008 at 6:08 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>
>>> As a workaround, if you only need read-only access, you can use the Mercurial
>>> mirrors which should work through your proxy (AFAIK Mercurial only uses GET and
>>> POST).
>>>
>>> Type "hg clone http://code.python.org/hg/trunk/" or "hg clone
>>> http://code.python.org/hg/branches/py3k" depending on what you need exactly.
>>
>> I do not need the whole branch - only a small subset of files related
>> to distutils. I know that bazaar can't do partial checkouts - it can
>> only fetch the whole branch. What about mercurial?
>>
>> And why not to setup HTTPS for anonymous read and authorized write
>> access? It is not that hard to do and will solve many problems with
>> proxies.
>
> Because it requires setting up a certificate. You can use HTTP to do a
> read-only checkout.

Or you could download a tarball snapshot: http://svn.python.org/snapshots/

>
> -Brett
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From python at rcn.com  Thu Sep  4 01:41:32 2008
From: python at rcn.com (Raymond Hettinger)
Date: Wed, 3 Sep 2008 16:41:32 -0700
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
Message-ID: <D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>

I think this should be deferred to Py3.1. 

This decision was not widely discussed and 
I think it likely that some users will
be surprised and dismayed.  The release
candidate seems to be the wrong time to
yank this out (in part because of the surprise
factor) and in part because I think the change
silently affects shelve performance so that the
impact may be significantly negative but not
readily apparent.

We don't have to take this out.  So why do
it hastily at the last minute and without some
discussion on comp.lang.python at least.

If it were any other release, we would have
disciplined ourselves to deprecate first and
remove a generation or two later.

Also, the reason for removal may yet disappear
if jcrea steps in an continues to make updates.

Also, the referenced note ( http://mail.python.org/pipermail/python-dev/2008-July/081379.html )
say to "start end-of-lifing it" which I took to mean deprecate rather than remove during a release candidate.


Raymond




----- Original Message ----- 
From: "Benjamin Peterson" <report at bugs.python.org>
To: <python-bugs-list at python.org>
Sent: Wednesday, September 03, 2008 4:32 PM
Subject: [issue3769] Deprecate bsddb for removal in 3.0


> 
> Benjamin Peterson <musiccomposition at gmail.com> added the comment:
> 
> Also see this:
> http://mail.python.org/pipermail/python-3000/2008-September/014712.html
> 
> _______________________________________
> Python tracker <report at bugs.python.org>
> <http://bugs.python.org/issue3769>
> _______________________________________
> _______________________________________________
> Python-bugs-list mailing list
> Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/python%40rcn.com
>

From brett at python.org  Thu Sep  4 02:12:41 2008
From: brett at python.org (Brett Cannon)
Date: Wed, 3 Sep 2008 17:12:41 -0700
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
In-Reply-To: <D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
Message-ID: <bbaeab100809031712y6bfc1e53md15f5c28d00f7ed3@mail.gmail.com>

On Wed, Sep 3, 2008 at 4:41 PM, Raymond Hettinger <python at rcn.com> wrote:
> I think this should be deferred to Py3.1.
> This decision was not widely discussed and I think it likely that some users
> will
> be surprised and dismayed.

Perhaps, but that could be said about almost any module that has been
removed through the stdlib reorg.

>  The release
> candidate seems to be the wrong time to
> yank this out (in part because of the surprise
> factor) and in part because I think the change
> silently affects shelve performance so that the
> impact may be significantly negative but not
> readily apparent.
>
> We don't have to take this out.

We don't have to remove anything that has gone through the stdlib
reorg, so that is not a solid argument.

>  So why do
> it hastily at the last minute and without some
> discussion on comp.lang.python at least.
>

It isn't being done "hastily"; this has been planned for a while.
People have just been too busy to get around to it. And we are not
changing any semantics or removing something from the language which
is what I view as what you don't change in an rc. So this might come
down to a different opinion of what one can do during an rc.

> If it were any other release, we would have
> disciplined ourselves to deprecate first and
> remove a generation or two later.
>

We are deprecating first in 2.6.

> Also, the reason for removal may yet disappear
> if jcrea steps in an continues to make updates.
>

OK, but none of his changes have received a code review, so if we are
going to go down the whole "disciplined" route about it being an rc
then we should back out all of Jesus' changes for both 2.6 and 3.0,
which puts us back to the same instability issues.

> Also, the referenced note (
> http://mail.python.org/pipermail/python-dev/2008-July/081379.html )
> say to "start end-of-lifing it" which I took to mean deprecate rather than
> remove during a release candidate.

Well, it was in the PEP before beta2 even went out the door.

-Brett

From greg.ewing at canterbury.ac.nz  Thu Sep  4 02:11:33 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 04 Sep 2008 12:11:33 +1200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BE7C0E.3080903@trueblade.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
	<48BDF2A9.5060203@canterbury.ac.nz> <48BE743B.1000108@egenix.com>
	<48BE7C0E.3080903@trueblade.com>
Message-ID: <48BF27B5.3000207@canterbury.ac.nz>

Eric Smith wrote:

> But I agree that 
> managing a single batch file is easier than dealing with the PATH 
> variable, and has fewer side effects (finding DLL's, etc.).

This would only be possible for an administrator
installation, though, not a per-user one.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Sep  4 02:15:21 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 04 Sep 2008 12:15:21 +1200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BE743B.1000108@egenix.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
	<48BDF2A9.5060203@canterbury.ac.nz> <48BE743B.1000108@egenix.com>
Message-ID: <48BF2899.3000705@canterbury.ac.nz>

M.-A. Lemburg wrote:

> However, always having the latest version on PATH is not
> an option either, since e.g. I wouldn't want all .py scripts
> to be run by Python 3.0 just because I installed it for
> testing purposes.

Keep in mind that the normal installation process on
unix *does* make "python" refer to the most recently
installed version, at least for 2.x, and it's not
considered a problem there.

In the case of 3.0, didn't we decide not to do that?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Thu Sep  4 02:27:28 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 04 Sep 2008 12:27:28 +1200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48BE971B.6020403@timgolden.me.uk>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <48BDC325.1000901@egenix.com>
	<48BDF2A9.5060203@canterbury.ac.nz> <48BE743B.1000108@egenix.com>
	<48BE7C0E.3080903@trueblade.com> <g9m2bt$66i$1@ger.gmane.org>
	<79990c6b0809030646s15166e71hb910b57d4d15a870@mail.gmail.com>
	<48BE971B.6020403@timgolden.me.uk>
Message-ID: <48BF2B70.3080407@canterbury.ac.nz>

Tim Golden wrote:

> You can use "CALL" within one batch file to chain
> another, returning afterwards to the first.

You need to know that what you're calling is a bat
file to have the foresight to do that, though. I
can imagine people not expecting "python" to be
a bat file.

Instead of a bat file, maybe generate a small exe
that does the job?

-- 
Greg

From barry at python.org  Thu Sep  4 03:25:26 2008
From: barry at python.org (Barry Warsaw)
Date: Wed, 3 Sep 2008 21:25:26 -0400
Subject: [Python-Dev] [Python-3000] bsddb finished for 2.6/3.0 (and
	"<class	'BytesWarning'>: str() on a bytes instance")
In-Reply-To: <48BF1749.7040803@jcea.es>
References: <48BED164.1050103@jcea.es>
	<g9mlg0$isv$1@ger.gmane.org>	<48BF0245.9000102@jcea.es>
	<EC821EDF-5198-4490-B9A8-AA691E7EF576@python.org>
	<48BF1749.7040803@jcea.es>
Message-ID: <DFB7E9A9-A814-4106-AB7C-24F3A2135300@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 3, 2008, at 7:01 PM, Jesus Cea wrote:

> Barry Warsaw wrote:
>> and I know Brett agrees, so that's it.  On IRC, I've just asked  
>> Benjamin
>> to do the honors for 3.0 and Brett will add the deprecations for 2.6.
>
> I just committed the fix for bsddb testsuite in Python 3.0 branch:
> http://www.python.org/dev/buildbot/3.0.stable/changes/2687
>
> Can I do anything to revert this decision?. If not, what can I do to  
> be
> reconsidered in 3.1?.

Start raising some pitchforks.  It looks like Raymond will join the  
march :).

Really, this is about what's best for Python and pybsddb.  In this  
article, Guido unambiguously states his opinion:

http://mail.python.org/pipermail/python-dev/2008-July/081362.html

"+1. In my recollection maintaining bsddb has been nothing but trouble
right from the start when we were all sitting together at "Zope Corp
North" in a rented office in McLean... We can remove it from 3.0. We
can't really remove it from 2.6, but we can certainly start
end-of-lifing it in 2.6."

Jesus, let me stress that IMO this is not a reflection on your work at  
all.  On the contrary, keeping it alive in 2.x and providing a really  
solid independent package for 3.0 is critical for its continued  
relevance to Python programmers.

I completely agree with Guido that bsddb (not pybsddb) has been a  
headache since forever.  For example, IIRC Sleepycat was notorious for  
changing the API in micro releases, though I don't know if that's  
still the case with the current maintainers.  I personally believe  
that Python and pybsddb are both better off with their own maintenance  
lifecycles so I stand by my decision that pulling it out of 3.0 is the  
right thing to do.  3.1 is far enough away that any decision we make  
in 3.0 can be re-evaluated.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSL85BnEjvBPtnXfVAQLfkwQAtoagOP37uAwL1r2H7w73erTsWBYHf4VH
KcTZsjeQ/mEvmaaJIG86ylAtpxmDmMF5x7OClR66bXXxf0oTnWV4KMC9rLdQW8R/
KpMIfuQw/501AQgFmcB0M6SQ6CYyJHU5K+K6X+ScOPHOJoG8usPK1pk8XFGOXBZK
UGXCEHVvlrk=
=7AOQ
-----END PGP SIGNATURE-----

From ctb at msu.edu  Thu Sep  4 04:56:09 2008
From: ctb at msu.edu (C. Titus Brown)
Date: Wed, 3 Sep 2008 19:56:09 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate bsddb
	for removal in 3.0)
In-Reply-To: <D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
Message-ID: <20080904025604.GD23640@idyll.org>

On Wed, Sep 03, 2008 at 04:41:32PM -0700, Raymond Hettinger wrote:
-> I think this should be deferred to Py3.1. 
-> 
-> This decision was not widely discussed and 
-> I think it likely that some users will
-> be surprised and dismayed.  The release
-> candidate seems to be the wrong time to
-> yank this out (in part because of the surprise
-> factor) and in part because I think the change
-> silently affects shelve performance so that the
-> impact may be significantly negative but not
-> readily apparent.

Related but tangential question that we were discussing on the pygr[0]
mailing list -- what is the "official" word on a scalable object store
in Python?  We've been using bsddb, but is there an alternative?  And
what if bsddb is removed?

It would be very nice to have a moderately scalable (thousands to
millions, if not billions) cross-platform object store backend
distributed with Python.

sqlite could be one choice, but I haven't used it much yet, so I don't
know.

thanks,
--titus

[0] Python graph database for bioinformatics, 

	http://code.google.com/p/pygr
-- 
C. Titus Brown, ctb at msu.edu

From reed at reedobrien.com  Thu Sep  4 05:16:31 2008
From: reed at reedobrien.com (Reed O'Brien)
Date: Wed, 3 Sep 2008 23:16:31 -0400
Subject: [Python-Dev] Python 3.0b3 documentation typo
Message-ID: <3C2319C9-31E6-41B3-8A3D-2159A77A1750@reedobrien.com>

I was reading through the Dictionary views section:

http://docs.python.org/dev/3.0/library/stdtypes.html#dictionary-view-objects

Unless I am mistaken; the intersection at the end of the example usage  
should be:

 >>> keys & {'eggs', 'bacon', 'salad'}
{'bacon'}


Cheers,
~ro

From barry at python.org  Thu Sep  4 05:41:27 2008
From: barry at python.org (Barry Warsaw)
Date: Wed, 3 Sep 2008 23:41:27 -0400
Subject: [Python-Dev] Not releasing rc1 tonight
Message-ID: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm not going to release rc1 tonight.  There are too many open release  
blockers that I don't want to defer, and I'd like the buildbots to  
churn through the bsddb removal on all platforms.  Let me first thank  
Benjamin, Brett, Mark and Antoine for their help on IRC tonight.

Here are the issues I'm not comfortable with deferring:

   3640 test_cpickle crash on AMD64 Windows build
874900 threading module can deadlock after fork
   3574 compile() cannot decode Latin-1 source encodings
   3657 pickle can pickle the wrong function
   3187 os.listdir can return byte strings
   3660 reference leaks in 3.0
   3594 PyTokenizer_FindEncoding() never succeeds
   3629 Py30b3 won't compile a regex that compiles with 2.5.2 and 30b2

In addition, Mark reported in IRC that there are some regressions in  
the logging module.

I appreciate any feedback or fixes you can provide on these issues.   
You might also want to look at the deferred blockers to see if there's  
anything that really should be blocking rc1.

I'd like to try again on Friday and stick to rc2 on the 17th.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSL9Y53EjvBPtnXfVAQJGXwP+JZUa5EWlQh7yzt7aFdEM3qgiFZnKqWhz
TN4Cen0/eK8c4+t8a5WC+OLvc/P3PhMPhLSnE+g6IqQUO+pt+2LANgpAvCUrUahc
Nk2pt3gCclcmWlzVvCBspVPZjFPkHsW0uVhgK6x1C/2Re90yjeBqPGgT4LGlmaR3
bz6A3iiUnk0=
=Y5aN
-----END PGP SIGNATURE-----

From brett at python.org  Thu Sep  4 05:45:20 2008
From: brett at python.org (Brett Cannon)
Date: Wed, 3 Sep 2008 20:45:20 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <20080904025604.GD23640@idyll.org>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
Message-ID: <bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>

On Wed, Sep 3, 2008 at 7:56 PM, C. Titus Brown <ctb at msu.edu> wrote:
> On Wed, Sep 03, 2008 at 04:41:32PM -0700, Raymond Hettinger wrote:
> -> I think this should be deferred to Py3.1.
> ->
> -> This decision was not widely discussed and
> -> I think it likely that some users will
> -> be surprised and dismayed.  The release
> -> candidate seems to be the wrong time to
> -> yank this out (in part because of the surprise
> -> factor) and in part because I think the change
> -> silently affects shelve performance so that the
> -> impact may be significantly negative but not
> -> readily apparent.
>
> Related but tangential question that we were discussing on the pygr[0]
> mailing list -- what is the "official" word on a scalable object store
> in Python?  We've been using bsddb, but is there an alternative?  And
> what if bsddb is removed?
>

Beyond shelve there are no official plans to add a specific object store.

-Brett

From python at rcn.com  Thu Sep  4 06:14:21 2008
From: python at rcn.com (Raymond Hettinger)
Date: Wed, 3 Sep 2008 21:14:21 -0700
Subject: [Python-Dev] Not releasing rc1 tonight
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
Message-ID: <7CA312873E964C5ABB4B4AA4F2793DF2@RaymondLaptop1>

[Barry]
> I'm not going to release rc1 tonight.  

Can I go ahead with some bug fixes and doc improvements
or should I wait until after Friday?


Raymond

From greg at krypto.org  Thu Sep  4 06:19:04 2008
From: greg at krypto.org (Gregory P. Smith)
Date: Wed, 3 Sep 2008 21:19:04 -0700
Subject: [Python-Dev] Python 3.0b3 documentation typo
In-Reply-To: <3C2319C9-31E6-41B3-8A3D-2159A77A1750@reedobrien.com>
References: <3C2319C9-31E6-41B3-8A3D-2159A77A1750@reedobrien.com>
Message-ID: <52dc1c820809032119u732cf250wf3e1625dc6fc0705@mail.gmail.com>

thanks.  in general report all issues even ones like this on
bugs.python.org rather than on a mailing list where they can get lost.

i've fixed this trivial one in py3k svn r66207.

On Wed, Sep 3, 2008 at 8:16 PM, Reed O'Brien <reed at reedobrien.com> wrote:
> I was reading through the Dictionary views section:
>
> http://docs.python.org/dev/3.0/library/stdtypes.html#dictionary-view-objects
>
> Unless I am mistaken; the intersection at the end of the example usage
> should be:
>
>>>> keys & {'eggs', 'bacon', 'salad'}
> {'bacon'}
>
>
> Cheers,
> ~ro
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>

From michele.simionato at gmail.com  Thu Sep  4 07:03:14 2008
From: michele.simionato at gmail.com (Michele Simionato)
Date: Thu, 4 Sep 2008 07:03:14 +0200
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
In-Reply-To: <D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
Message-ID: <4edc17eb0809032203y4125aa82jbabe8125d5bbc500@mail.gmail.com>

On Thu, Sep 4, 2008 at 1:41 AM, Raymond Hettinger <python at rcn.com> wrote:
> The release
> candidate seems to be the wrong time to
> yank this out (in part because of the surprise
> factor) and in part because I think the change
> silently affects shelve performance so that the
> impact may be significantly negative but not
> readily apparent.

I do not use bsddb directly, but I use shelve which on Linux usually
takes advantage of bsddb. Does removing bsddb mean that
I will not be able to read shelve files written with Python 2.5
with Python 3.0? That would be quite disturbing to me.

From azhmogin at Princeton.EDU  Thu Sep  4 05:41:03 2008
From: azhmogin at Princeton.EDU (Andrey Zhmoginov)
Date: Wed, 03 Sep 2008 23:41:03 -0400
Subject: [Python-Dev] C API for gc.enable() and gc.disable()
Message-ID: <48BF58CF.4050700@princeton.edu>

>
> Would anyone mind if I did add a public C API for gc.disable() and
> gc.enable()? I would like to use it as an optimization for the pickle
> module (I found out that I get a good 2x speedup just by disabling the
> GC while loading large pickles). Of course, I could simply import the
> gc module and call the functions there, but that seems overkill to me.
> I included the patch below for review.
>   
I don't know if the following question is relevant, but it seems that 
many people here are familiar with Python cyclic garbage collector.
I see Python [v2.5.2 (r252:60911, Jul 31 2008, 17:28:52)] crashing with 
Segment fault when I extend Python with a very simple module. This 
behavior is observed when I create a thousand of lists (it does not 
crash with 10-100) in the module with the garbage collector turned on. 
When I turn it off - everything is perfect. I suspect that it is my 
module, but if it is a Python bug ( _GC_Malloc? memory leak somewhere?), 
it may be worth reporting.

The gdb "where" reply is the following:

    #0  0x080d8de9 in PyErr_Occurred ()
    #1  0x080f508f in _PyObject_GC_Malloc ()
    #2  0x080f5155 in _PyObject_GC_New ()
    #3  0x08079c98 in PyList_New ()
    #4  0xb7f53519 in draw_d_simple () from ./rt/rt_core.so
    #5  0xb7cf7833 in ffi_call_SYSV () from 
/usr/lib/python2.5/lib-dynload/_ctypes.so
    #6  0xb7cf766a in ffi_call () from 
/usr/lib/python2.5/lib-dynload/_ctypes.so
    #7  0xb7cf2534 in _CallProc () from 
/usr/lib/python2.5/lib-dynload/_ctypes.so
    #8  0xb7cec02a in ?? () from /usr/lib/python2.5/lib-dynload/_ctypes.so
    #9  0x0805cb97 in PyObject_Call ()
    #10 0x080c7aa7 in PyEval_EvalFrameEx ()
    #11 0x080c96e5 in PyEval_EvalFrameEx ()
    #12 0x080cb1f7 in PyEval_EvalCodeEx ()
    #13 0x080cb347 in PyEval_EvalCode ()
    #14 0x080ea818 in PyRun_FileExFlags ()
    #15 0x080eaab9 in PyRun_SimpleFileExFlags ()
    #16 0x08059335 in Py_Main ()
    #17 0x080587f2 in main ()

The crashing python code is:

    from ctypes import *
    import gc
    core = CDLL( "./tst.so" )
    core.a.argtypes = []
    core.a.restype = py_object
    #gc.disable()
    core.a()
    #gc.enable()

And tst.cpp is:

    #include <Python.h>
    extern "C" { PyObject *a(); }
    PyObject *a()
    {
        int n = 1000;
        PyObject *item;
        for ( int i = 0; i < n; i++ ) item = PyList_New( 0 );      // 
Crashes here (somewhere in between).
        return item;
    }

tst.cpp is compiled with:

    g++ -shared -Wl,-soname,tst.tmp.so -o tst.so tst.o
    g++ -I /usr/include/python2.5 -Wall -fPIC -o tst.o -c tst.cpp

Thanks for your help!

- Andrew.

From mhammond at skippinet.com.au  Thu Sep  4 09:28:36 2008
From: mhammond at skippinet.com.au (Mark Hammond)
Date: Thu, 4 Sep 2008 17:28:36 +1000
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
Message-ID: <00cd01c90e5f$daf326a0$90d973e0$@com.au>

Barry writes:

> In addition, Mark reported in IRC that there are some regressions in
> the logging module.

3772 logging module fails with non-ascii data

Which according to the IRC discussion doesn't apply to py3k.  The fix for
2.6 is quite trivial...

Cheers,

Mark


From skip at pobox.com  Thu Sep  4 13:08:01 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 4 Sep 2008 06:08:01 -0500
Subject: [Python-Dev] [Python-3000] Not releasing rc1 tonight
In-Reply-To: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
Message-ID: <18623.49553.393129.601096@montanaro-dyndns-org.local>


    Barry> In addition, Mark reported in IRC that there are some regressions
    Barry> in the logging module.

Vinay apparently checked in some changes to the logging module with no
review.  In the absence of obvious bug fixes there that should probably be
reverted.

Skip

From skip at pobox.com  Thu Sep  4 13:10:07 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 4 Sep 2008 06:10:07 -0500
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
 bsddb for removal in 3.0)
In-Reply-To: <bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
Message-ID: <18623.49679.433662.690583@montanaro-dyndns-org.local>


    >> Related but tangential question that we were discussing on the
    >> pygr[0] mailing list -- what is the "official" word on a scalable
    >> object store in Python?  We've been using bsddb, but is there an
    >> alternative?  And what if bsddb is removed?

    Brett> Beyond shelve there are no official plans to add a specific
    Brett> object store.

Unless something has changed while I wasn't looking, shelve requires a
concrete module under the covers: bsddb, gdbm, ndbm, dumbdbm.  It's just a
thin layer over one of them that makes it appear as if you can have keys
which aren't strings.

Skip

From skip at pobox.com  Thu Sep  4 13:11:45 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 4 Sep 2008 06:11:45 -0500
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
In-Reply-To: <4edc17eb0809032203y4125aa82jbabe8125d5bbc500@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<4edc17eb0809032203y4125aa82jbabe8125d5bbc500@mail.gmail.com>
Message-ID: <18623.49777.544569.960357@montanaro-dyndns-org.local>


    Michele> I do not use bsddb directly, but I use shelve which on Linux
    Michele> usually takes advantage of bsddb. Does removing bsddb mean that
    Michele> I will not be able to read shelve files written with Python 2.5
    Michele> with Python 3.0? That would be quite disturbing to me.

Correctamundo.

Skip

From facundobatista at gmail.com  Thu Sep  4 13:31:18 2008
From: facundobatista at gmail.com (Facundo Batista)
Date: Thu, 4 Sep 2008 08:31:18 -0300
Subject: [Python-Dev] [Python-3000]  Not releasing rc1 tonight
In-Reply-To: <7CA312873E964C5ABB4B4AA4F2793DF2@RaymondLaptop1>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<7CA312873E964C5ABB4B4AA4F2793DF2@RaymondLaptop1>
Message-ID: <e04bdf310809040431x4db0bfb2y1e0ffaf3ef5725a5@mail.gmail.com>

2008/9/4 Raymond Hettinger <python at rcn.com>:

> Can I go ahead with some bug fixes and doc improvements
> or should I wait until after Friday?

Doc improvements: go ahead.

Bug fixes: the patchs should be revised by other developer.

(I'll be hanging around in #python-dev today and tomorrow, btw, ping
me if I can help you)

-- 
. Facundo

Blog: http://www.taniquetil.com.ar/plog/
PyAr: http://www.python.org/ar/

From amauryfa at gmail.com  Thu Sep  4 13:42:49 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 4 Sep 2008 13:42:49 +0200
Subject: [Python-Dev] C API for gc.enable() and gc.disable()
In-Reply-To: <48BF58CF.4050700@princeton.edu>
References: <48BF58CF.4050700@princeton.edu>
Message-ID: <e27efe130809040442r4eb0cf19y5ceed9f31b21820d@mail.gmail.com>

Hello,

Andrey Zhmoginov wrote:
> I don't know if the following question is relevant, but it seems that many
> people here are familiar with Python cyclic garbage collector.
> I see Python [v2.5.2 (r252:60911, Jul 31 2008, 17:28:52)] crashing with
> Segment fault when I extend Python with a very simple module. This behavior
> is observed when I create a thousand of lists (it does not crash with
> 10-100) in the module with the garbage collector turned on. When I turn it
> off - everything is perfect. I suspect that it is my module, but if it is a
> Python bug ( _GC_Malloc? memory leak somewhere?), it may be worth reporting.
>
> The gdb "where" reply is the following:
>
>   #0  0x080d8de9 in PyErr_Occurred ()
>   #1  0x080f508f in _PyObject_GC_Malloc ()
>   #2  0x080f5155 in _PyObject_GC_New ()
>   #3  0x08079c98 in PyList_New ()
>   #4  0xb7f53519 in draw_d_simple () from ./rt/rt_core.so
>   #5  0xb7cf7833 in ffi_call_SYSV () from
> /usr/lib/python2.5/lib-dynload/_ctypes.so
>   #6  0xb7cf766a in ffi_call () from
> /usr/lib/python2.5/lib-dynload/_ctypes.so
>   #7  0xb7cf2534 in _CallProc () from
> /usr/lib/python2.5/lib-dynload/_ctypes.so
>   #8  0xb7cec02a in ?? () from /usr/lib/python2.5/lib-dynload/_ctypes.so
>   #9  0x0805cb97 in PyObject_Call ()
>   #10 0x080c7aa7 in PyEval_EvalFrameEx ()
>   #11 0x080c96e5 in PyEval_EvalFrameEx ()
>   #12 0x080cb1f7 in PyEval_EvalCodeEx ()
>   #13 0x080cb347 in PyEval_EvalCode ()
>   #14 0x080ea818 in PyRun_FileExFlags ()
>   #15 0x080eaab9 in PyRun_SimpleFileExFlags ()
>   #16 0x08059335 in Py_Main ()
>   #17 0x080587f2 in main ()
>
> The crashing python code is:
>
>   from ctypes import *
>   import gc
>   core = CDLL( "./tst.so" )
>   core.a.argtypes = []
>   core.a.restype = py_object
>   #gc.disable()
>   core.a()
>   #gc.enable()
>
> And tst.cpp is:
>
>   #include <Python.h>
>   extern "C" { PyObject *a(); }
>   PyObject *a()
>   {
>       int n = 1000;
>       PyObject *item;
>       for ( int i = 0; i < n; i++ ) item = PyList_New( 0 );      // Crashes
> here (somewhere in between).
>       return item;
>   }
>
> tst.cpp is compiled with:
>
>   g++ -shared -Wl,-soname,tst.tmp.so -o tst.so tst.o
>   g++ -I /usr/include/python2.5 -Wall -fPIC -o tst.o -c tst.cpp

This has nothing to do with the garbage collection, but with the GIL
(the famous Global Interpreter Lock http://docs.python.org/api/threads.html )

When ctypes calls a CDLL function, it releases the GIL (to let
eventual other threads run)
Your example crashes because it calls python API functions when the
GIL is not held,
and is so invalid.

There are two solutions:
- re-acquire the GIL in your C functions with PyGILState_Ensure() & co
- use ctypes.PyDLL( "./tst.so" ), which does not release the GIL.

Hope this helps,

-- 
Amaury Forgeot d'Arc

From barry at python.org  Thu Sep  4 15:01:34 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Sep 2008 09:01:34 -0400
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <7CA312873E964C5ABB4B4AA4F2793DF2@RaymondLaptop1>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<7CA312873E964C5ABB4B4AA4F2793DF2@RaymondLaptop1>
Message-ID: <5FEA88D7-EEA9-40E1-AD03-FC2607565FEC@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 4, 2008, at 12:14 AM, Raymond Hettinger wrote:

> [Barry]
>> I'm not going to release rc1 tonight.
>
> Can I go ahead with some bug fixes and doc improvements
> or should I wait until after Friday?

Doc fixes are fine.  Please have bug fix patches reviewed by another  
python-dev member.  Bonus points for any bug fix that closes a release  
blocker! :)

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSL/cL3EjvBPtnXfVAQKnmgQAlx89LWeq0hEmTRvTGy/DHIYioARqAisG
wJAnZPqinbGI6pkyn0kiwgDOvNzstnFQSZsEFiAFU+iF+nbgkm8agcTf+eLXqCFK
y+o0xXTi7fLXKuaGioY54kz3BcwQH17Ul3X6vRxBdCWYesAe3rIXprnNgt/Euuyy
P5sZLKwfTls=
=b3n4
-----END PGP SIGNATURE-----

From barry at python.org  Thu Sep  4 15:02:44 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Sep 2008 09:02:44 -0400
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <00cd01c90e5f$daf326a0$90d973e0$@com.au>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<00cd01c90e5f$daf326a0$90d973e0$@com.au>
Message-ID: <50FD6660-44FE-433E-827C-B1ECD82FB55A@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 4, 2008, at 3:28 AM, Mark Hammond wrote:

> Barry writes:
>
>> In addition, Mark reported in IRC that there are some regressions in
>> the logging module.
>
> 3772 logging module fails with non-ascii data
>
> Which according to the IRC discussion doesn't apply to py3k.  The  
> fix for
> 2.6 is quite trivial...

Thanks.  Looks like Vinay committed this.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSL/cdHEjvBPtnXfVAQIb7gP9G2o8eSnWWfEmlanwoqiHGxgqUjQtx8Xz
es/Sjclk5KZ2X4I/jITJcOxGDfTT3h7FX9tDQiUaIzZAVB66qyzWc3957bUwqeqS
9HNqfB4OoIa1Ds2+lukXpEPci6eddl2xVFEkejgsfdyS4q2/K1/R6URTPCXnPNiH
zoiXNaEcBcM=
=Zk4M
-----END PGP SIGNATURE-----

From barry at python.org  Thu Sep  4 15:03:34 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Sep 2008 09:03:34 -0400
Subject: [Python-Dev] [Python-3000] Not releasing rc1 tonight
In-Reply-To: <18623.49553.393129.601096@montanaro-dyndns-org.local>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<18623.49553.393129.601096@montanaro-dyndns-org.local>
Message-ID: <4C4EC784-A7CA-4497-A0FF-1D9B5D0E8399@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 4, 2008, at 7:08 AM, skip at pobox.com wrote:

>
>    Barry> In addition, Mark reported in IRC that there are some  
> regressions
>    Barry> in the logging module.
>
> Vinay apparently checked in some changes to the logging module with no
> review.  In the absence of obvious bug fixes there that should  
> probably be
> reverted.

Or did he commit Mark's patch from bug 3772?  If so, that would count  
as a reviewed patch.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSL/cpnEjvBPtnXfVAQIkSwQApjBbIGgyV3X1oBhBLtRjTZrVDgFXPfRH
XyXtVd1r75PT+24UuqPHWNC9l+/sKnUaYqH3kJbHG2duMyr/duG7j6EIJLzOz+QC
SKwqtQr+WDBR0vpH3Q0wrUzQNXhtDyCjWx84IatRbKRVDUfbDlFQy/jj+SLvRBBR
WGJTAFP1x5g=
=mebg
-----END PGP SIGNATURE-----

From barry at python.org  Thu Sep  4 15:04:25 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Sep 2008 09:04:25 -0400
Subject: [Python-Dev] [Python-3000]  Not releasing rc1 tonight
In-Reply-To: <e04bdf310809040431x4db0bfb2y1e0ffaf3ef5725a5@mail.gmail.com>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<7CA312873E964C5ABB4B4AA4F2793DF2@RaymondLaptop1>
	<e04bdf310809040431x4db0bfb2y1e0ffaf3ef5725a5@mail.gmail.com>
Message-ID: <27C24817-848B-4173-B5F8-7240EB5B28D1@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 4, 2008, at 7:31 AM, Facundo Batista wrote:

> (I'll be hanging around in #python-dev today and tomorrow, btw, ping
> me if I can help you)

Me too, though I'm a bit busy at work.  Ping my nick 'barry' if you  
need any RM-level decision.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSL/c2XEjvBPtnXfVAQJ4EQP/SecaG0VRtsezedDRpX+zwmVo6W0n+9EP
rmKH5CWMSTWh53rXySCmE8IS2rrdhoyCZNSy0aERMTGz5JuEh/sw+O5EaxJQMFST
DdYx0aLRVwb62JaQHr7W7YyVWBG5+CQa3BehASFiwsw0dsAp0BpkwW1nIhybkLcW
hXNRzB2gwXI=
=9Mgt
-----END PGP SIGNATURE-----

From jcea at jcea.es  Thu Sep  4 15:23:22 2008
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 04 Sep 2008 15:23:22 +0200
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
 bsddb for removal in 3.0)
In-Reply-To: <bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
Message-ID: <48BFE14A.5020208@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brett Cannon wrote:
>> Related but tangential question that we were discussing on the pygr[0]
>> mailing list -- what is the "official" word on a scalable object store
>> in Python?  We've been using bsddb, but is there an alternative?  And
>> what if bsddb is removed?
> 
> Beyond shelve there are no official plans to add a specific object store.

If you are storing million of objects, you'd better use a transactional
storage, able to survive diskfulls or code/computer crashes.

I will maintain "bsddb" as a separate (downloadable via PYPI) package
whatever the fate of bsddb in Python stardard library be. So bsddb is a
pretty safe bet, even if you need to install it separately.

Compared to sqlite, you don't need to know SQL, you can finetuning (for
example, using ACI instead of ACID, deciding store by store), and you
can do replication and distributed transactions (useful, for example, if
your storage is bigger than a single machine capacity, like my case). If
you combine Berkeley DB with Durus, for example, all of this is
abstracted and you simply use "regular" python objects.

If you use bsddb, please consider to subscribe to pybsddb mailing list.
It has pretty low traffic and you can guide bsddb evolution (for
example, prioritize Berkeley DB binding additions).

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSL/hRZlgi5GaxT1NAQIeLgP/XPj32oLFS54QiHjTKrVKf4Bqc/JqFeJl
rasN/RM4hiqv3naueB90jPn2eMai3exCQXD85ew7YeMdWluNPEX/crBbhfN7n5M8
qP/GLWCqqDKWhPyvlInghQPoJUyv55TrPLsbUslCNyLAGFb79ETHs8MeaXn7Kx9o
+uAc01ifsoA=
=Or2m
-----END PGP SIGNATURE-----

From jcea at jcea.es  Thu Sep  4 15:31:17 2008
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 04 Sep 2008 15:31:17 +0200
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
In-Reply-To: <4edc17eb0809032203y4125aa82jbabe8125d5bbc500@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<4edc17eb0809032203y4125aa82jbabe8125d5bbc500@mail.gmail.com>
Message-ID: <48BFE325.5010905@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Michele Simionato wrote:
> I do not use bsddb directly, but I use shelve which on Linux usually
> takes advantage of bsddb. Does removing bsddb mean that
> I will not be able to read shelve files written with Python 2.5
> with Python 3.0? That would be quite disturbing to me.

Seems so.

If bsddb is actually unavailable in Python 3.0, you would need to
download/install it from PYPI. I'm committed to keep bsddb alive, in a
way or another :).

I'm thinking that if bsddb is discarded in Python 3.0, shelve probably
drop it also, so installing bsddb externally will not "magically" make
it available to 3.0 shelve. I can't comment about the python-dev plans here.

PS: Remember that if you are installing bsddb as a separate package, its
name will be "bsddb3", not "bsddb".


- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSL/jJZlgi5GaxT1NAQIzJAP8CiIxpoz553NDr+/1GkAfzs3W6fu1uBuo
XsCmbqkiOMe9fPOBNZnlfoBnGz4C4nlOlOzQV3RexRXBiWKOqUUg4DlJOJtrAMUO
ZNtcz9JyvjzmVNZMezgCsfjkEhzNABbCe9mXHQVCR5zlNZVUKpTW7A06/1eX/gMv
ECJqQta662o=
=6XBZ
-----END PGP SIGNATURE-----

From jcea at jcea.es  Thu Sep  4 15:35:57 2008
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 04 Sep 2008 15:35:57 +0200
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
In-Reply-To: <bbaeab100809031712y6bfc1e53md15f5c28d00f7ed3@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<bbaeab100809031712y6bfc1e53md15f5c28d00f7ed3@mail.gmail.com>
Message-ID: <48BFE43D.5060904@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brett Cannon wrote:
>> Also, the reason for removal may yet disappear
>> if jcrea steps in an continues to make updates.
> 
> OK, but none of his changes have received a code review, so if we are
> going to go down the whole "disciplined" route about it being an rc
> then we should back out all of Jesus' changes for both 2.6 and 3.0,
> which puts us back to the same instability issues.

I was wondering if somebody could write a "TO DO" list of things need to
keep bsddb in the stdlib. Then I could work to trying to comply :).

Yes, we are all very busy guys, but still...

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSL/kPZlgi5GaxT1NAQLu4AP/VSHPYOCQgQYFJsdi2MWXBpyY7TyC5XgT
Ks2uilXru/hsKQcegn8G6z/53Bt0Uu+oXZSQaZ51V8VnwDXEqaZ+GnKK+S2ky9m0
yomgMlKIZZJsOVd6X4HbLtrVYVKX8wQ224X/yCkw27OLzLIE9IDbUzEjC3+/A7mD
9IJu3B6IaLA=
=ZA8p
-----END PGP SIGNATURE-----

From skip at pobox.com  Thu Sep  4 15:45:43 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 4 Sep 2008 08:45:43 -0500
Subject: [Python-Dev] [Python-3000] Not releasing rc1 tonight
In-Reply-To: <4C4EC784-A7CA-4497-A0FF-1D9B5D0E8399@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<18623.49553.393129.601096@montanaro-dyndns-org.local>
	<4C4EC784-A7CA-4497-A0FF-1D9B5D0E8399@python.org>
Message-ID: <18623.59015.183377.941143@montanaro-dyndns-org.local>


    Barry> Or did he commit Mark's patch from bug 3772?  If so, that would
    Barry> count as a reviewed patch.

The checkin message says issue 3726:

    Author: vinay.sajip
    Date: Wed Sep  3 11:20:05 2008
    New Revision: 66180

    Log:
    Issue #3726: Allowed spaces in separators in logging configuration files.

    Modified:
       python/trunk/Lib/logging/config.py
       python/trunk/Lib/test/test_logging.py
       python/trunk/Misc/NEWS

I noticed because someone else (Brett?) questioned the apparent lack of
review.

Skip

From ncoghlan at gmail.com  Thu Sep  4 15:49:15 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 04 Sep 2008 23:49:15 +1000
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
In-Reply-To: <D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
Message-ID: <48BFE75B.1000502@gmail.com>

Raymond Hettinger wrote:
> I think this should be deferred to Py3.1.
> This decision was not widely discussed and I think it likely that some
> users will
> be surprised and dismayed.  The release
> candidate seems to be the wrong time to
> yank this out (in part because of the surprise
> factor) and in part because I think the change
> silently affects shelve performance so that the
> impact may be significantly negative but not
> readily apparent.

I don't use Python for database work myself, but something I am somewhat
disappointed to lose is the presence of a moderately complicated package
within the Python distribution itself which is making use of the various
2-to-3 migration tools to support both Python 2.x and 3.x with a single
mixed Python-and-C code base (along with automatic conversion via 2to3).

While that will still be visible to some degree due to the presence of
the 2.x version of the bsddb code in Python 2.6, I don't think it will
be quite the same as it would have been with the 3.x version also being
readily available as part of the standard 3.0 install.

Regardless, given that the removal of bsddb from the 3.0 branch is now a
done deal in svn, I suggest that all we can do is monitor how much
feedback we get indicating that the need to download bsddb as a separate
install is a significant hindrance to Py3k migrations. If the feedback
indicates it is necessary, restoring the module for 3.1 certainly isn't
an impossibility, but in the meantime there *are* real benefits to
decoupling the maintenance cycles (those wanting to get the most out of
Jesus's ongoing work in exposing more of the bsddb API are probably
going to want to be using the external pybsddb releases anyway rather
than waiting however many months it ends up being until we get to 2.7/3.1).

There's also a bit of a second shot at this for bsddb supporters, in
that some of the "omnibus" Python distribution providers like
ActiveState and Enthought may choose to include pybsddb once they start
releasing bundles for Python 3.0.

As far as the corporate scenarios go: if a corporate environment is *so*
screwed up as to simultaneously permit a migration from Python 2.x to
3.0 of an internal project that relies on bsddb, while at the same time
banning those performing the migration from installing the 3.0
compatible version of pybsddb, then they have much bigger problems than
which modules and packages we are choosing to include in the standard
library. In my experience, restrictive corporate environments are far
more likely to just disallow migrations to 3.0 altogether (and in many
cases, the decision makers disallowing such migrations probably won't be
wrong - the case for migrating an existing internal app to 3.0 is far,
far weaker than that for using 3.0 for new development).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From techtonik at gmail.com  Thu Sep  4 16:03:18 2008
From: techtonik at gmail.com (techtonik)
Date: Thu, 4 Sep 2008 17:03:18 +0300
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <bbaeab100809031608yfc70269y853ea0b1f1c2a2e5@mail.gmail.com>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
	<loom.20080903T160547-830@post.gmane.org>
	<d34314100809031539m21fbfd00j555e6ec9c4bf15cd@mail.gmail.com>
	<bbaeab100809031608yfc70269y853ea0b1f1c2a2e5@mail.gmail.com>
Message-ID: <d34314100809040703w3fdeb727sce1dff1cdef9b85c@mail.gmail.com>

On Thu, Sep 4, 2008 at 2:08 AM, Brett Cannon <brett at python.org> wrote:
>>
>> And why not to setup HTTPS for anonymous read and authorized write
>> access? It is not that hard to do and will solve many problems with
>> proxies.
>
> Because it requires setting up a certificate.

Certificate is already set.

$ svn co https://svn.python.org/projects/trunk
Error validating server certificate for 'https://svn.python.org:443':
 - The certificate is not issued by a trusted authority. Use the
   fingerprint to validate the certificate manually!
Certificate information:
 - Hostname: svn.python.org
 - Valid: from Apr 10 00:00:00 2007 GMT until Apr  9 23:59:59 2010 GMT
 - Issuer: http://www.usertrust.com, The USERTRUST Network, Salt Lake
City, UT, US
 - Fingerprint: fc:67:c1:59:f5:57:71:29:f5:13:50:bc:c9:16:f1:74:da:1f:12:98
(R)eject, accept (t)emporarily or accept (p)ermanently? t
Authentication realm: <https://svn.python.org:443> Subversion repository
Password for 'username':
Authentication realm: <https://svn.python.org:443> Subversion repository
Username: svn: PROPFIND request failed on '/projects/trunk'
svn: PROPFIND of '/projects/trunk': authorization failed
(https://svn.python.org)

> You can use HTTP to do a read-only checkout.

Well, the main problem is that I can't. As I already said my machine
is behind a proxy that doesn't support required WebDAV extensions for
HTTP protocol - http://subversion.tigris.org/faq.html#proxy

-- 
--anatoly t.

From techtonik at gmail.com  Thu Sep  4 16:21:31 2008
From: techtonik at gmail.com (techtonik)
Date: Thu, 4 Sep 2008 17:21:31 +0300
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <loom.20080903T224344-812@post.gmane.org>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
	<loom.20080903T160547-830@post.gmane.org>
	<d34314100809031539m21fbfd00j555e6ec9c4bf15cd@mail.gmail.com>
	<loom.20080903T224344-812@post.gmane.org>
Message-ID: <d34314100809040721x24a702b1tc156cda69ab53981@mail.gmail.com>

On Thu, Sep 4, 2008 at 1:48 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>
>> I do not need the whole branch - only a small subset of files related
>> to distutils. I know that bazaar can't do partial checkouts - it can
>> only fetch the whole branch. What about mercurial?
>
> Mercurial can't do it either. But I don't think it matters a lot; unless your
> bandwidth is very low the full clone should be fast enough. Here it takes just
> 150s. for the trunk.

Thanks for support. However, fetching whole branch is not an option, because
Internet is expensive here - even 10Mb tarball costs 20 cents. To make matters
worse I would say that it is more than 0.04% of an average salary.

-- 
--anatoly t.

From ctb at msu.edu  Thu Sep  4 16:22:18 2008
From: ctb at msu.edu (C. Titus Brown)
Date: Thu, 4 Sep 2008 07:22:18 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <48BFE14A.5020208@jcea.es>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es>
Message-ID: <20080904142218.GC7683@idyll.org>

On Thu, Sep 04, 2008 at 03:23:22PM +0200, Jesus Cea wrote:
-> -----BEGIN PGP SIGNED MESSAGE-----
-> Hash: SHA1
-> 
-> Brett Cannon wrote:
-> >> Related but tangential question that we were discussing on the pygr[0]
-> >> mailing list -- what is the "official" word on a scalable object store
-> >> in Python?  We've been using bsddb, but is there an alternative?  And
-> >> what if bsddb is removed?
-> > 
-> > Beyond shelve there are no official plans to add a specific object store.
-> 
-> If you are storing million of objects, you'd better use a transactional
-> storage, able to survive diskfulls or code/computer crashes.

We're using a write-once-read-many pattern of access, and it is simply a
cache of a separate file (that remains around), so no, we don't better
use a transactional storage :).

-> I will maintain "bsddb" as a separate (downloadable via PYPI) package
-> whatever the fate of bsddb in Python stardard library be. So bsddb is a
-> pretty safe bet, even if you need to install it separately.

Since I/we want to distribute pygr to end-users, this is really not a
pleasant prospect.  Also often the installation of Python itself goes
much more smoothly than the installation of separately compiled binary
packages, for all the obvious reasons (compiler/OS versions, lib
versions, etc. etc.)

-> Compared to sqlite, you don't need to know SQL, you can finetuning (for
-> example, using ACI instead of ACID, deciding store by store), and you
-> can do replication and distributed transactions (useful, for example, if
-> your storage is bigger than a single machine capacity, like my case). If
-> you combine Berkeley DB with Durus, for example, all of this is
-> abstracted and you simply use "regular" python objects.

I agree.  I like bsddb for just this reason and I'd like to continue
being able to use it!  I think that there are many reasons why having
such a thing in the stdlib is really useful and I also think it's worth
exploring the ramifications of taking it out...

--t
-- 
C. Titus Brown, ctb at msu.edu

From tonynelson at georgeanelson.com  Thu Sep  4 16:29:10 2008
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Thu, 4 Sep 2008 10:29:10 -0400
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
 bsddb for removal in 3.0)
In-Reply-To: <18623.49679.433662.690583@montanaro-dyndns-org.local>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
Message-ID: <p04330101c4e59ecad7d6@[192.168.123.162]>

At 6:10 AM -0500 9/4/08, skip at pobox.com wrote:
>    >> Related but tangential question that we were discussing on the
>    >> pygr[0] mailing list -- what is the "official" word on a scalable
>    >> object store in Python?  We've been using bsddb, but is there an
>    >> alternative?  And what if bsddb is removed?
>
>    Brett> Beyond shelve there are no official plans to add a specific
>    Brett> object store.
>
>Unless something has changed while I wasn't looking, shelve requires a
>concrete module under the covers: bsddb, gdbm, ndbm, dumbdbm.  It's just a
>thin layer over one of them that makes it appear as if you can have keys
>which aren't strings.

I thought that all that was happening was that BSDDB was becoming a
separate project.  If one needs BSDDB with Python2.6, one installs it.
Aren't there other parts of Python that require external modules, such as
Tk?  Using Tk requires installing it.  Such things are normally packaged by
each distro the same way as Python is packaged ("yum install tk bsddb").

Shipping an application to end users is a different problem.  Such packages
should include a private copy of Python as well as of any dependent
libraries, as tested.
-- 
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From phd at phd.pp.ru  Thu Sep  4 16:33:09 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Thu, 4 Sep 2008 18:33:09 +0400
Subject: [Python-Dev] bsddb
In-Reply-To: <48BFE14A.5020208@jcea.es>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es>
Message-ID: <20080904143309.GB23604@phd.pp.ru>

On Thu, Sep 04, 2008 at 03:23:22PM +0200, Jesus Cea wrote:
> Compared to sqlite, you don't need to know SQL, you can finetuning (for
> example, using ACI instead of ACID, deciding store by store), and you
> can do replication and distributed transactions (useful, for example, if
> your storage is bigger than a single machine capacity, like my case).

   Let me raise the glove. Compared to bsddb:

-- SQLite is public domain; the licensing terms of Berkeley DB[1] are not
   friendly to commercial applications: "Our open source license ...
   permits use of Berkeley DB in open source projects or in applications
   that are not distributed to third parties." I am not sure if using of
   PyBSDDB in commercial applications is considered "using of Berkeley DB
   in open source projects";
-- SQLite has a pretty stable API and a pretty stable on-disk format; for
   bsddb one needs to do dump/reload on every major release;
-- SQLite implements a subset of SQL - a powerful query language;
-- SQLite is extensible - one can write his/her own functions and
   aggregates, e.g.; PySQLite allows to write these functions in Python;
   PySQLite also allows to write data conversion functions that converts
   between Python and SQL data types;
-- a program can attach a few databases at once thus distributing loads
   between a number of disks, including network mounts.

[1] http://www.oracle.com/technology/software/products/berkeley-db/htdocs/licensing.html

> If
> you combine Berkeley DB with Durus, for example, all of this is
> abstracted and you simply use "regular" python objects.

   Durus (and ZODB) has an index of all objects, the index is stored in
memory AFAIK - a real problem if one has millions of objects. Does bsddb
help to mitigate the problem?

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From ctb at msu.edu  Thu Sep  4 16:37:51 2008
From: ctb at msu.edu (C. Titus Brown)
Date: Thu, 4 Sep 2008 07:37:51 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <p04330101c4e59ecad7d6@[192.168.123.162]>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
Message-ID: <20080904143751.GD7683@idyll.org>

On Thu, Sep 04, 2008 at 10:29:10AM -0400, Tony Nelson wrote:
-> At 6:10 AM -0500 9/4/08, skip at pobox.com wrote:
-> >    >> Related but tangential question that we were discussing on the
-> >    >> pygr[0] mailing list -- what is the "official" word on a scalable
-> >    >> object store in Python?  We've been using bsddb, but is there an
-> >    >> alternative?  And what if bsddb is removed?
-> >
-> >    Brett> Beyond shelve there are no official plans to add a specific
-> >    Brett> object store.
-> >
-> >Unless something has changed while I wasn't looking, shelve requires a
-> >concrete module under the covers: bsddb, gdbm, ndbm, dumbdbm.  It's just a
-> >thin layer over one of them that makes it appear as if you can have keys
-> >which aren't strings.
-> 
-> I thought that all that was happening was that BSDDB was becoming a
-> separate project.  If one needs BSDDB with Python2.6, one installs it.
-> Aren't there other parts of Python that require external modules, such as
-> Tk?  Using Tk requires installing it.  Such things are normally packaged by
-> each distro the same way as Python is packaged ("yum install tk bsddb").
-> 
-> Shipping an application to end users is a different problem.  Such packages
-> should include a private copy of Python as well as of any dependent
-> libraries, as tested.

Why?  On Mac OS X, for example, Python comes pre-installed -- not sure
if it comes with Tk yet, but the next version probably will.  On Windows
there's a handy few-click installer that installs Tk.  Is there some
reason why I shouldn't be relying on those distributions??

Requiring users to install anything at all imposes a barrier to use.
That barrier rises steeply in height the more packages (with versioning
issues, etc.) are needed.  This also increases the tech support burden
dramatically.

I'm happy to be told that bsddb is too much of a maintenance burden for
Python 2.6/3.0 to have -- especially since it's gone from 3.0 now ;) --
but I don't think the arguments that *it won't matter that it's not
there* have been very credible.  There's a BIG difference between things
that come with Python and things that are add-ons.

Right now, I'm teaching an intro programming course using Python.  It
doesn't seem like the students are going to need to install *anything*
other than base Python in order to play with full networking libraries &
sqlite databases, among other features.  And, for me and for them,
that's really great.

I don't think the convenience of "batteries *included*" should be
underestimated.

--t
-- 
C. Titus Brown, ctb at msu.edu

From barry at python.org  Thu Sep  4 16:39:46 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 4 Sep 2008 10:39:46 -0400
Subject: [Python-Dev] [Python-3000] Not releasing rc1 tonight
In-Reply-To: <18623.59015.183377.941143@montanaro-dyndns-org.local>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<18623.49553.393129.601096@montanaro-dyndns-org.local>
	<4C4EC784-A7CA-4497-A0FF-1D9B5D0E8399@python.org>
	<18623.59015.183377.941143@montanaro-dyndns-org.local>
Message-ID: <CDC12824-E981-4739-88EE-3C504EEB7D23@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 4, 2008, at 9:45 AM, skip at pobox.com wrote:

>
>    Barry> Or did he commit Mark's patch from bug 3772?  If so, that  
> would
>    Barry> count as a reviewed patch.
>
> The checkin message says issue 3726:
>
>    Author: vinay.sajip
>    Date: Wed Sep  3 11:20:05 2008
>    New Revision: 66180
>
>    Log:
>    Issue #3726: Allowed spaces in separators in logging  
> configuration files.
>
>    Modified:
>       python/trunk/Lib/logging/config.py
>       python/trunk/Lib/test/test_logging.py
>       python/trunk/Misc/NEWS
>
> I noticed because someone else (Brett?) questioned the apparent lack  
> of
> review.

Yep, that's a different issue.  Unless someone wants to vouch for the  
committed patch after the fact, could someone please revert the change  
and contact Vinay to get a proper fix reviewed?  I noticed that he  
says in the tracker issue that what was committed was modified from  
the posted patch.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSL/zMnEjvBPtnXfVAQJY3QP+LNXhx1YGuCHSw/D2n0yVBj1PLLUbgYnp
k/+zWWmvIRc8YSApV1YyYR4iXfqqYFoi1SH0eh7F1k9+2CZ51HHD0p6CZ0Eb1FQ2
405ocxT28R3UR/E0ozxFca3IuNhGPR2FI/BCfsLrdrA3UtHA4XvZMDvM3KxEMarl
9WdYgop/I8Y=
=b6Ry
-----END PGP SIGNATURE-----

From tonynelson at georgeanelson.com  Thu Sep  4 17:01:35 2008
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Thu, 4 Sep 2008 11:01:35 -0400
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
 bsddb for removal in 3.0)
In-Reply-To: <20080904143751.GD7683@idyll.org>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
	<20080904143751.GD7683@idyll.org>
Message-ID: <p04330102c4e5a718cbbe@[192.168.123.162]>

At 7:37 AM -0700 9/4/08, C. Titus Brown wrote:
>On Thu, Sep 04, 2008 at 10:29:10AM -0400, Tony Nelson wrote:
 ...
>-> Shipping an application to end users is a different problem.  Such packages
>-> should include a private copy of Python as well as of any dependent
>-> libraries, as tested.
>
>Why?  On Mac OS X, for example, Python comes pre-installed -- not sure
>if it comes with Tk yet, but the next version probably will.  On Windows
>there's a handy few-click installer that installs Tk.  Is there some
>reason why I shouldn't be relying on those distributions??

Yes.  An application is tested with one version of Python and one version
of its libraries.  When MOSX updates Python or some other library, you are
relying on their testing of your application.  Unless you are Adobe or
similarly large they didn't do that testing.  Perhaps you have noticed the
threads about installing a new Python release over the Python that came
with an OS, and how bad an idea that is?  This is the same issue, from the
other side.

>Requiring users to install anything at all imposes a barrier to use.
>That barrier rises steeply in height the more packages (with versioning
>issues, etc.) are needed.  This also increases the tech support burden
>dramatically.
 ...

Precisely why one needs to ship a single installer that installs the
complete application, including Python and any other libraries it needs.
-- 
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From python at rcn.com  Thu Sep  4 17:05:33 2008
From: python at rcn.com (Raymond Hettinger)
Date: Thu, 4 Sep 2008 08:05:33 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769]
	Deprecatebsddb for removal in 3.0)
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za><D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1><20080904025604.GD23640@idyll.org><bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com><18623.49679.433662.690583@montanaro-dyndns-org.local><p04330101c4e59ecad7d6@[192.168.123.162]>
	<20080904143751.GD7683@idyll.org>
Message-ID: <E8A2827BEE1F4E32B8B0E1C1DBEF1C6D@RaymondLaptop1>

[C. Titus Brown]
> I'm happy to be told that bsddb is too much of a maintenance burden for
> Python 2.6/3.0 to have -- especially since it's gone from 3.0 now ;) --
> but I don't think the arguments that *it won't matter that it's not
> there* have been very credible.  

Not credible, not widely discussed, not tested in a beta ...
No alternative provided, no deprecation period before it disappears ...  
The usual deliberative process has been completely bypassed.


Raymond

From amauryfa at gmail.com  Thu Sep  4 17:06:28 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Thu, 4 Sep 2008 17:06:28 +0200
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <d34314100809040721x24a702b1tc156cda69ab53981@mail.gmail.com>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
	<loom.20080903T160547-830@post.gmane.org>
	<d34314100809031539m21fbfd00j555e6ec9c4bf15cd@mail.gmail.com>
	<loom.20080903T224344-812@post.gmane.org>
	<d34314100809040721x24a702b1tc156cda69ab53981@mail.gmail.com>
Message-ID: <e27efe130809040806s3708ec4bk43f66435d8a66b09@mail.gmail.com>

Hello,

2008/9/4 techtonik <techtonik at gmail.com>:
> On Thu, Sep 4, 2008 at 1:48 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>>>
>>> I do not need the whole branch - only a small subset of files related
>>> to distutils. I know that bazaar can't do partial checkouts - it can
>>> only fetch the whole branch. What about mercurial?
>>
>> Mercurial can't do it either. But I don't think it matters a lot; unless your
>> bandwidth is very low the full clone should be fast enough. Here it takes just
>> 150s. for the trunk.
>
> Thanks for support. However, fetching whole branch is not an option, because
> Internet is expensive here - even 10Mb tarball costs 20 cents. To make matters
> worse I would say that it is more than 0.04% of an average salary.

Did you try to open your browser to (for example)
http://svn.python.org/projects/python/trunk/Lib/distutils/
and download the desired files from there?


-- 
Amaury Forgeot d'Arc

From ctb at msu.edu  Thu Sep  4 17:10:50 2008
From: ctb at msu.edu (C. Titus Brown)
Date: Thu, 4 Sep 2008 08:10:50 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <p04330102c4e5a718cbbe@[192.168.123.162]>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
	<20080904143751.GD7683@idyll.org>
	<p04330102c4e5a718cbbe@[192.168.123.162]>
Message-ID: <20080904151050.GA8632@idyll.org>

On Thu, Sep 04, 2008 at 11:01:35AM -0400, Tony Nelson wrote:
-> At 7:37 AM -0700 9/4/08, C. Titus Brown wrote:
-> >On Thu, Sep 04, 2008 at 10:29:10AM -0400, Tony Nelson wrote:
->  ...
-> >-> Shipping an application to end users is a different problem.  Such packages
-> >-> should include a private copy of Python as well as of any dependent
-> >-> libraries, as tested.
-> >
-> >Why?  On Mac OS X, for example, Python comes pre-installed -- not sure
-> >if it comes with Tk yet, but the next version probably will.  On Windows
-> >there's a handy few-click installer that installs Tk.  Is there some
-> >reason why I shouldn't be relying on those distributions??
-> 
-> Yes.  An application is tested with one version of Python and one version
-> of its libraries.  When MOSX updates Python or some other library, you are
-> relying on their testing of your application.  Unless you are Adobe or
-> similarly large they didn't do that testing.  Perhaps you have noticed the
-> threads about installing a new Python release over the Python that came
-> with an OS, and how bad an idea that is?  This is the same issue, from the
-> other side.

I have to say I've never had problems with a stock install of Python on
either Mac OS X or Windows (shockingly enough :).  I think this is good
advice for applications that rely on external libraries, but I just
don't see any problems with relying on Python 2.5 to contain all the
things that normally come with Python 2.5.  It seems like you're
pushing a pretty sharp dichotomy (trichotomy?) --

 - Python library/core developers should compile it all.

 - Python app developers can rely on what they install from binaries
   themselves, but not rely on it to be present on anyone else's machine
   or OS.

 - End users should be given a complete clean install of Python in a
   different location for each application they're using, even if those
   applications depend only on the stdlib.

This seems surprisingly complicated to me (and unnecessary, in my
limited experience) -- but it does validate my decade-old decision to
avoid writing end-user applications in Python, sadly enough.  It ends up
being less work to distribute and support a C/C++ app on Windows and Mac
OS X, for crikey's sake!

--t
-- 
C. Titus Brown, ctb at msu.edu

From jcea at jcea.es  Thu Sep  4 17:54:12 2008
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 04 Sep 2008 17:54:12 +0200
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
In-Reply-To: <48BFE75B.1000502@gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<48BFE75B.1000502@gmail.com>
Message-ID: <48C004A4.7030103@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Nick Coghlan wrote:
> While that will still be visible to some degree due to the presence of
> the 2.x version of the bsddb code in Python 2.6, I don't think it will
> be quite the same as it would have been with the 3.x version also being
> readily available as part of the standard 3.0 install.

Since 2.6 intention seems to mark this module as deprecated, I guess 2.x
bsddb presence in stock python will finish in 2.7. Moreover, I'm working
just now improving 2.x/3.x conversion code in pybsddb. I think this code
will be available in bsddb 4.7.4, and it will not be integrated in
Python 2.6 (that will include 4.7.3.minor releases, if we keep the
criteria of "only stability and security fixes in 2.6.x").

If the idea is to keep bsddb alive in 2.x, I don't see the point of not
keeping the 3.0 version, because the issues used to justify the removal
persist: I'm the only maintainer, little code review, buildbot issues, etc.

(I would like a comprehensive list, to be able to improve those
deficiencies).

In fact, if we keep bsddb in 2.x, the pressure to keep it in 3.x will be
higher.

> Regardless, given that the removal of bsddb from the 3.0 branch is now a
> done deal in svn, I suggest that all we can do is monitor how much

Any version control system can revert that with a single command :).

All I can say is that current bsddb code (in my personal repository)
passes all tests in current compiled python3.0 binary, called with the
"-bb" parameter flag (the "-bb" flag was something I learned yesterday).

> but in the meantime there *are* real benefits to
> decoupling the maintenance cycles (those wanting to get the most out of
> Jesus's ongoing work in exposing more of the bsddb API are probably
> going to want to be using the external pybsddb releases anyway rather
> than waiting however many months it ends up being until we get to 2.7/3.1).

The cycles are actually decoupled since I toke over the bsddb
maintenance (I've released a new version per month). So the release
cycles are not an issue.

The main issue here is 3.0 support, that I worked over the last couple
of months. It is done now. It couldn't be done faster because I was
learning 3.0 internals on-the-fly (there are NO docs about C module
migration; my experience there could be valuable) and 3.0 was a moving
target (still is). For example, when I left to summer holiday bsddb
worked flawless in Python 3.0b2. It failed in 3.0b3 because threading
renames done in python 3.0.

So, Python 3.0 is not waiting for bsddb to be ready, because it already
is (since yesterday). And future Python releases won't suffer because we
won't have any other major architectural reengineering of Python in a
long long time (I hope!).

That is, future Python releases would take whatever bsddb is available
at that time. No wait. No dependent release cycles. With my current
release schema of "a release per month", I can track python evolution
with little effort. For example, Python 2.5 to 2.6 was pretty painless,
even with the "PyBytes_*" ugliness.

> As far as the corporate scenarios go: if a corporate environment is *so*
> screwed up as to simultaneously permit a migration from Python 2.x to
> 3.0 of an internal project that relies on bsddb, while at the same time
> banning those performing the migration from installing the 3.0
> compatible version of pybsddb, then they have much bigger problems than
> which modules and packages we are choosing to include in the standard
> library.

Agreed. I was thinking about bsddb removal in 2.7.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSMAEnplgi5GaxT1NAQIrKgP/YAp45HUSG8Q+M355LTVqlcLMLkycpooc
fflW0MlQ3zZV307VBUbGo9urkS6h1pYhYByivApylhVqj8D4x8OEmMZk0lX8cegG
LYSBzs/sBeyxWWva6r5D9/4DsgJe9ZHqaBLMpy6ipPNVtUbMS61VTNovb3wP+f72
EnSIf9k/glM=
=QxRo
-----END PGP SIGNATURE-----

From janssen at parc.com  Thu Sep  4 18:25:43 2008
From: janssen at parc.com (Bill Janssen)
Date: Thu, 4 Sep 2008 09:25:43 PDT
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <20080904143751.GD7683@idyll.org> 
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
	<20080904143751.GD7683@idyll.org>
Message-ID: <08Sep4.092547pdt."58698"@synergy1.parc.xerox.com>

> I don't think the convenience of "batteries *included*" should be
> underestimated.

Yeah, but bsddb is one of those exploding batteries.  I've used it for
years, and have had lots and lots of problems with it.  Having SQLite
in there is great; now we need implementations of anydbm and shelve
which use it.

Bill

From janssen at parc.com  Thu Sep  4 18:39:19 2008
From: janssen at parc.com (Bill Janssen)
Date: Thu, 4 Sep 2008 09:39:19 PDT
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <20080904151050.GA8632@idyll.org> 
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
	<20080904143751.GD7683@idyll.org>
	<p04330102c4e5a718cbbe@[192.168.123.162]>
	<20080904151050.GA8632@idyll.org>
Message-ID: <08Sep4.093922pdt."58698"@synergy1.parc.xerox.com>

> I have to say I've never had problems with a stock install of Python on
> either Mac OS X or Windows (shockingly enough :).  I think this is good

I agree.  I just use the stock Python on OS X and Windows.  And it
seems to work well for my rather large and complicated (PIL, PyLucene,
Medusa, ReportLab, SSL, email-4) application.  Clearly Windows, with
its somewhat complicated PATH and DLL issues, might be problematic,
but I haven't seen that yet.

> advice for applications that rely on external libraries, but I just
> don't see any problems with relying on Python 2.5 to contain all the
> things that normally come with Python 2.5.  It seems like you're
> pushing a pretty sharp dichotomy (trichotomy?) --

Yeah, but this is just some random guy on the Python mailing list
(Tony, I apologize for not knowing who you are).  No need to take it
too seriously.

> but it does validate my decade-old decision to
> avoid writing end-user applications in Python, sadly enough.

Well, I don't do that either, but it's because of Python's lack of a
decent built-in GUI toolkit.  Sad.

Bill

From dbinger at mems-exchange.org  Thu Sep  4 18:55:01 2008
From: dbinger at mems-exchange.org (Binger David)
Date: Thu, 4 Sep 2008 12:55:01 -0400
Subject: [Python-Dev] Durus memory
In-Reply-To: <mailman.4138.1220543665.3486.python-dev@python.org>
References: <mailman.4138.1220543665.3486.python-dev@python.org>
Message-ID: <443F114E-D42A-464D-8E2B-7C3340807F52@mems-exchange.org>

I'm sorry this post is a bit off-topic, but I think I should correct  
this.

On Sep 4, 2008, at 11:54 AM, Oleg wrote:

> Durus (and ZODB) has an index of all objects, the index is stored in
> memory AFAIK - a real problem if one has millions of objects.


Durus now has an option to store the index in the data file in a form
that is usable directly from disk.  With this option, the in-memory
index is only for objects changed since the last pack.

Durus works fine with millions of objects, as long as you don't have
millions of rapidly-changing objects.

I haven't used the Berkeley DB storage for Durus, but I'm pretty
sure that the Durus/Python side of that would not use
any in-memory index.




From jcea at jcea.es  Thu Sep  4 19:01:47 2008
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 04 Sep 2008 19:01:47 +0200
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
 bsddb for removal in 3.0)
In-Reply-To: <20080904142218.GC7683@idyll.org>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>	<20080904025604.GD23640@idyll.org>	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>	<48BFE14A.5020208@jcea.es>
	<20080904142218.GC7683@idyll.org>
Message-ID: <48C0147B.6030107@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

C. Titus Brown wrote:
> On Thu, Sep 04, 2008 at 03:23:22PM +0200, Jesus Cea wrote:
> -> Brett Cannon wrote:
> -> >> Related but tangential question that we were discussing on the pygr[0]
> -> >> mailing list -- what is the "official" word on a scalable object store
> -> >> in Python?  We've been using bsddb, but is there an alternative?  And
> -> >> what if bsddb is removed?
> -> > 
> -> > Beyond shelve there are no official plans to add a specific object store.
> -> 
> -> If you are storing million of objects, you'd better use a transactional
> -> storage, able to survive diskfulls or code/computer crashes.
> 
> We're using a write-once-read-many pattern of access, and it is simply a
> cache of a separate file (that remains around), so no, we don't better
> use a transactional storage :).

If you can recreate the database in case of problems, and it is mostly
reads, then I would suggest you gdbm. I personally hate SQL and "SQL
fits all" mentality, and the mindset/impedance mismatch between python
and objects, and SQL world, but sure sqlite module could fill the bill
also... if you don't mind mixing two languages and two logics in your
code and your brain :).

> -> I will maintain "bsddb" as a separate (downloadable via PYPI) package
> -> whatever the fate of bsddb in Python stardard library be. So bsddb is a
> -> pretty safe bet, even if you need to install it separately.
> 
> Since I/we want to distribute pygr to end-users, this is really not a
> pleasant prospect.  Also often the installation of Python itself goes
> much more smoothly than the installation of separately compiled binary
> packages, for all the obvious reasons (compiler/OS versions, lib
> versions, etc. etc.)

I agree. I can check the library with Solaris 10 and several flavors of
Linux, but I'm particularly worried about Windows support. I'm unable to
provide precompiled libs, and 99.999% of windows users don't know what a
"compiler thing" is, let alone being able to compile anything by themselves.

> I agree.  I like bsddb for just this reason and I'd like to continue
> being able to use it!  I think that there are many reasons why having
> such a thing in the stdlib is really useful and I also think it's worth
> exploring the ramifications of taking it out...

+Inf

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSMAUeZlgi5GaxT1NAQKM5gQAhEO8OmVvVXr/jp1Hqj7DnxuPb0kabeGF
TjDyiiJutbDKNLZiqegV7nzNpfJBMMZXNXTG70Lbrh05GWdzYcLahHluPzmf3hu6
wPCjv63NijH4OCmhtGmN4vi/C6p5VX1NqktN0evX7FYNJrnoYVKBRSnFdF8aPSbI
wUKKSsihJTw=
=Zv+S
-----END PGP SIGNATURE-----

From skip at pobox.com  Thu Sep  4 19:03:38 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 4 Sep 2008 12:03:38 -0500
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
 bsddb for removal in 3.0)
In-Reply-To: <20080904142218.GC7683@idyll.org>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904142218.GC7683@idyll.org>
Message-ID: <18624.5354.48043.846565@montanaro-dyndns-org.local>


    > Compared to sqlite, you don't need to know SQL, you can finetuning
    > (for example, using ACI instead of ACID, deciding store by store), and
    > you can do replication and distributed transactions (useful, for
    > example, if your storage is bigger than a single machine capacity,
    > like my case). If you combine Berkeley DB with Durus, for example, all
    > of this is abstracted and you simply use "regular" python objects.

    Titus> I agree.  I like bsddb for just this reason and I'd like to
    Titus> continue being able to use it!  I think that there are many
    Titus> reasons why having such a thing in the stdlib is really useful
    Titus> and I also think it's worth exploring the ramifications of taking
    Titus> it out...

I suggested in another message (perhaps on another thread) that maybe a
dbm.sqlite module would be worth having.  It would have a dict-ish API like
the other dict-on-disk modules but use the sqlite module to read (SELECT)
and write (INSERT and UPDATE) key/value pairs from the underlying database.

Skip

From hall.jeff at gmail.com  Thu Sep  4 19:07:23 2008
From: hall.jeff at gmail.com (Jeff Hall)
Date: Thu, 4 Sep 2008 13:07:23 -0400
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <18624.5354.48043.846565@montanaro-dyndns-org.local>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904142218.GC7683@idyll.org>
	<18624.5354.48043.846565@montanaro-dyndns-org.local>
Message-ID: <1bc395c10809041007v63d13755k73f7586935f36449@mail.gmail.com>

Doesn't SQLlite still have a 4gb cap?

I'd personally prefer an open source solution (if that's Berkeley so be it
but there's plenty out there... MySQL for one)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080904/688a5333/attachment.htm>

From hall.jeff at gmail.com  Thu Sep  4 19:14:36 2008
From: hall.jeff at gmail.com (Jeff Hall)
Date: Thu, 4 Sep 2008 13:14:36 -0400
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <1bc395c10809041007v63d13755k73f7586935f36449@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904142218.GC7683@idyll.org>
	<18624.5354.48043.846565@montanaro-dyndns-org.local>
	<1bc395c10809041007v63d13755k73f7586935f36449@mail.gmail.com>
Message-ID: <1bc395c10809041014s1bba89cajef8e5cafb84f8639@mail.gmail.com>

never mind about the limit... I was thinking SQL Express

On Thu, Sep 4, 2008 at 1:07 PM, Jeff Hall <hall.jeff at gmail.com> wrote:

> Doesn't SQLlite still have a 4gb cap?
>
> I'd personally prefer an open source solution (if that's Berkeley so be it
> but there's plenty out there... MySQL for one)
>
>


-- 
Haikus are easy
Most make very little sense
Refrigerator
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080904/aef4a899/attachment.htm>

From phd at phd.pp.ru  Thu Sep  4 19:16:44 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Thu, 4 Sep 2008 21:16:44 +0400
Subject: [Python-Dev] SQLite (was: bsddb alternative)
In-Reply-To: <1bc395c10809041007v63d13755k73f7586935f36449@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904142218.GC7683@idyll.org>
	<18624.5354.48043.846565@montanaro-dyndns-org.local>
	<1bc395c10809041007v63d13755k73f7586935f36449@mail.gmail.com>
Message-ID: <20080904171644.GC27055@phd.pp.ru>

On Thu, Sep 04, 2008 at 01:07:23PM -0400, Jeff Hall wrote:
> Doesn't SQLlite still have a 4gb cap?

   http://sqlite.org/limits.html

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From jcea at jcea.es  Thu Sep  4 19:40:28 2008
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 04 Sep 2008 19:40:28 +0200
Subject: [Python-Dev] bsddb
In-Reply-To: <20080904143309.GB23604@phd.pp.ru>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>	<20080904025604.GD23640@idyll.org>	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>	<48BFE14A.5020208@jcea.es>
	<20080904143309.GB23604@phd.pp.ru>
Message-ID: <48C01D8C.6060801@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Oleg Broytmann wrote:
> -- SQLite is public domain; the licensing terms of Berkeley DB[1] are not
>    friendly to commercial applications: "Our open source license ...
>    permits use of Berkeley DB in open source projects or in applications
>    that are not distributed to third parties." I am not sure if using of
>    PyBSDDB in commercial applications is considered "using of Berkeley DB
>    in open source projects";

I can't comment on this. I'm not a lawyer.

> -- SQLite has a pretty stable API and a pretty stable on-disk format; for
>    bsddb one needs to do dump/reload on every major release;

Not at all. The worst thing you would need to do is a "db_upgrade", an
in-place operation. Lately it is pretty harmless and fast (like
upgrading the log format, not the database file format).

A stable fileformat is useful for long term support, but an evolving
format allows improvements. Following your reasoning, Python should be
keep in 1.0 era, for compatibility sake.

> -- SQLite implements a subset of SQL - a powerful query language;

Yes, a declarative language completely unrelated to Python.

> -- SQLite is extensible - one can write his/her own functions and
>    aggregates, e.g.; PySQLite allows to write these functions in Python;
>    PySQLite also allows to write data conversion functions that converts
>    between Python and SQL data types;

bsddb 4.7.4 (available next month) will allow to subclass DB/DBEnv, etc.
objects, so you can implement the logic you wish there. Until that, you
can do proxy/delegation (that is the way I'm doing 3.0 compatibility, BTW).

> -- a program can attach a few databases at once thus distributing loads
>    between a number of disks, including network mounts.

That is an OS issue. Any program get the benefice.

The problem is not disk capacity. Any modern machine can scale disk
without bound via iSCSI, for example (god bless ZFS!). The issue is
replication for redundancy, load sharing and high availability. These
things are available in bsddb 4.7.3 (that is, in Python 2.6).

How do you scale traffic demand in SQLite?. I can keep adding machines
to solve read requests, without sharing any disk between them. I can
launch 64 bsddb processes in a single 64 CPU machine to manage
(read/write) a single shared database. I don't know if SQLite can do
that. Berkeley DB can.

>    Durus (and ZODB) has an index of all objects, the index is stored in
> memory AFAIK - a real problem if one has millions of objects. Does bsddb
> help to mitigate the problem?

Latest Durus has not that issue, but you always can use another project
of mine:

Berkeley DB Backend Storage Engine for DURUS
http://www.jcea.es/programacion/durus-berkeleydbstorage.htm

This code (with some private enhancements for replication and
distributed transactions) manages a nearly 200 Terabytes Durus
repository without any sweat (~2^35 objects stored there).

In this particular instance, distributed transactions allows me to
partition data between several machines, with no sharing between them,
and replication allows redundancy and routing read requests to a less
loaded machine (the writes goes to the "master" machine, replication
from there is transparent).

SQLite is a good product. I just dislike "SQL fits all" model, as I
already said in another message. Using a SQL storage to save persistent
Python objects is ugly and SQL language is of no use there. You just
need "something" safe, scalable, configurable and being able to give you
opaque objects when an OID (Object ID) is presented. Compare a terabyte
Python "shelve" object, ease of use, transparence, etc., with keeping
the objects in a SQL database server.

Just my opinion, of course.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSMAdiJlgi5GaxT1NAQJxkwP/emM8dDKbnhxme76Nm3bXhA89NwCgQNQi
ojO0wkVZZ8ypUBNKwGM8PyDzGYoWGnh7VgylGb2bsPt67bCxrHjcXBNPaYrMN/fw
AETLlJUrhu9J17jPWKA+JU1FmC9oX34Ki580qMXI9nR51LVLU/1H6nM+KgA0slnn
uG3xvm5chfk=
=M46E
-----END PGP SIGNATURE-----

From phd at phd.pp.ru  Thu Sep  4 20:03:13 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Thu, 4 Sep 2008 22:03:13 +0400
Subject: [Python-Dev] bsddb
In-Reply-To: <48C01D8C.6060801@jcea.es>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904143309.GB23604@phd.pp.ru>
	<48C01D8C.6060801@jcea.es>
Message-ID: <20080904180313.GA30510@phd.pp.ru>

On Thu, Sep 04, 2008 at 07:40:28PM +0200, Jesus Cea wrote:
> A stable fileformat is useful for long term support, but an evolving
> format allows improvements.

   Once I upgraded Python on a Windows computer... I think it was 2.2 to
2.3 upgrade - and all my bsddb databases stopped working. I cannot call
this "improvement". I didn't have db_upgarde on that computer (or I didn't
know about it). Does PyBSDDB have db_upgrade in the distribution? Does
Python distribution have db_upgrade?

> Following your reasoning, Python should be
> keep in 1.0 era, for compatibility sake.

   Python? No. But persistent data structures? Yes! How many different
pickle data formats there were since Python 1.0? What is the oldest
pickle format modern Python can read? (Just using pickle as an example.)

> > -- SQLite implements a subset of SQL - a powerful query language;
> 
> Yes, a declarative language completely unrelated to Python.

   Sometimes being unrelated to Python is advantage. Written in C,
optimized for its tasks, the implementation of the query language
certainly can outperform Python.

> Using a SQL storage to save persistent
> Python objects is ugly

   No more ugly than any other storage. A matter of taste, I think.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From amk at amk.ca  Thu Sep  4 20:12:50 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Thu, 4 Sep 2008 14:12:50 -0400
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769]
	Deprecate	bsddb for removal in 3.0)
In-Reply-To: <08Sep4.092547pdt."58698"@synergy1.parc.xerox.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
	<20080904143751.GD7683@idyll.org>
	<08Sep4.092547pdt."58698"@synergy1.parc.xerox.com>
Message-ID: <20080904181249.GA20398@amk-desktop.matrixgroup.net>

On Thu, Sep 04, 2008 at 09:25:43AM -0700, Bill Janssen wrote:
> Yeah, but bsddb is one of those exploding batteries.  I've used it for
> years, and have had lots and lots of problems with it.  Having SQLite
> in there is great; now we need implementations of anydbm and shelve
> which use it.

What sort of problems?  When I've used BerkleyDB in the past, it
always took a fair bit of experimenting & searching to figure out the
right combination of flags to use (and the BerkeleyDB docs were very
low-level), but once that was done it seemed to work OK.
Incorporating Jesus's docs will help users with that issue; I'm
willing to work on that before 2.6final.

I think the primary annoyance is the instability of the bsddb tests,
and the resulting bad effect on buildbot's usefulness (as we all just
get accustomed to having a patchwork of red randomly mixed in).

So why not just strip down the test cases we run to avoid the
problematic tests?  That won't help Jesus debug on platforms he can't
access, but we could re-enable those tests after 2.7 or provide a
different buildbot target that runs the entire suite.

--amk

From ctb at msu.edu  Thu Sep  4 20:19:26 2008
From: ctb at msu.edu (C. Titus Brown)
Date: Thu, 4 Sep 2008 11:19:26 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <48C0147B.6030107@jcea.es>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904142218.GC7683@idyll.org>
	<48C0147B.6030107@jcea.es>
Message-ID: <20080904181925.GB13884@idyll.org>

On Thu, Sep 04, 2008 at 07:01:47PM +0200, Jesus Cea wrote:
-> -----BEGIN PGP SIGNED MESSAGE-----
-> Hash: SHA1
-> 
-> C. Titus Brown wrote:
-> > Since I/we want to distribute pygr to end-users, this is really not a
-> > pleasant prospect.  Also often the installation of Python itself goes
-> > much more smoothly than the installation of separately compiled binary
-> > packages, for all the obvious reasons (compiler/OS versions, lib
-> > versions, etc. etc.)
-> 
-> I agree. I can check the library with Solaris 10 and several flavors of
-> Linux, but I'm particularly worried about Windows support. I'm unable to
-> provide precompiled libs, and 99.999% of windows users don't know what a
-> "compiler thing" is, let alone being able to compile anything by themselves.

I believe I might be able to help you with this.  More off-list, in a
few weeks; if anyone else needs full Windoze access, Watch This Space, as
they say.

(Yes, I know access is not enough -- you really want someone to be
paying attention on Windows, too.  I'm working on a project or two
there; access to large quantities of talented students is opening up
some ideas :)

--titus
-- 
C. Titus Brown, ctb at msu.edu

From brett at python.org  Thu Sep  4 20:30:54 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Sep 2008 11:30:54 -0700
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
In-Reply-To: <48BFE43D.5060904@jcea.es>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<bbaeab100809031712y6bfc1e53md15f5c28d00f7ed3@mail.gmail.com>
	<48BFE43D.5060904@jcea.es>
Message-ID: <bbaeab100809041130g7ca19b3aw576418e42f83fb84@mail.gmail.com>

On Thu, Sep 4, 2008 at 6:35 AM, Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Brett Cannon wrote:
>>> Also, the reason for removal may yet disappear
>>> if jcrea steps in an continues to make updates.
>>
>> OK, but none of his changes have received a code review, so if we are
>> going to go down the whole "disciplined" route about it being an rc
>> then we should back out all of Jesus' changes for both 2.6 and 3.0,
>> which puts us back to the same instability issues.
>
> I was wondering if somebody could write a "TO DO" list of things need to
> keep bsddb in the stdlib. Then I could work to trying to comply :).
>

[Guido already made his public statement in support of removing
pybsddb from 3.0, so this is more just for general benefit of Jesus to
know why I think this all happened; I don't view these as points to
argue over]

* Follow python-dev practices. The biggest example of this was
checking in code during an rc release cycle without code review. That
was stated on python-committers which you should be subscribed to and
paying attention to.

* Maintain bsddb in Python and cut external releases separately. That
would help make bsddb feel more like a stdlib thing instead of
something that just gets dumped in our lap when we get close to a
release.

* Stay on top of the buildbots. test_bsddb has been such a consistent
failure on the buildbots that it has left a very sour taste in the
mouths of many core developers over the years (and I mean years;
Pythonlabs folks are saying how much they remember the bindings being
unstable back in the day).

* Convince some folks that Sleepycat is actually doing a decent job
now. As I believe Fred mentioned and you pointed out with the 4.7.0
release, Sleepycat does not always do solid releases.

* Get another committer to help you maintain the code. When Gregory
stepped down from maintaining bsddb, the code languished with its
traditionally flaky tests until you stepped forward. That suggests to
me that no one really wants to maintain that code but you. Sure,
people want the code to be there, but "want" does not translate to
man-hours to keep the code in good shape.

> Yes, we are all very busy guys, but still...

Yes, we are all busy, including you. And that is what makes bsddb the
largest maintenance headache in the stdlib; you are a single point of
failure for a chunk of code that has garnered a reputation over the
years as being flaky. And I realize the reputation is not your fault,
Jesus. And I understand people wanting bsddb to be there. But from a
core developer's POV, I want to keep the stdlib to code that at least
a couple of core developers would be willing to work on if a bug was
reported in the issue tracker; bsddb has not shown to be such code
base.

And just so people know, I hear the argument about keeping bsddb in
3.0 and then ripping it out in 3.1, but I'm cynical when it comes to
python-dev, so I see that as a potential ploy to keep the code in and
then have a year or so to argue about this all over again with no
change on either side.

Another thing to keep in mind with the whole shelve/dbm.any argument
is that for 3.1 there is nothing saying we can't change shelve and the
dbm package to allow 3rd-party code to register with the dbm package
such that bsddb can be used as needed behind the scenes.

-Brett

From curt at hagenlocher.org  Thu Sep  4 20:49:49 2008
From: curt at hagenlocher.org (Curt Hagenlocher)
Date: Thu, 4 Sep 2008 11:49:49 -0700
Subject: [Python-Dev] bsddb
In-Reply-To: <20080904143309.GB23604@phd.pp.ru>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904143309.GB23604@phd.pp.ru>
Message-ID: <d2155e360809041149p1d645a23ufc0d89e207207110@mail.gmail.com>

On Thu, Sep 4, 2008 at 7:33 AM, Oleg Broytmann <phd at phd.pp.ru> wrote:
>
> SQLite is public domain; the licensing terms of Berkeley DB[1] are not
> friendly to commercial applications: "Our open source license ...
> permits use of Berkeley DB in open source projects or in applications
> that are not distributed to third parties." I am not sure if using of
> PyBSDDB in commercial applications is considered "using of Berkeley DB
> in open source projects";

Wow, I hadn't realized that it was such a restrictive license.  When I
see "Berkeley" I think "BSD license".

--
Curt Hagenlocher
curt at hagenlocher.org

From josiah.carlson at gmail.com  Thu Sep  4 20:54:58 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 4 Sep 2008 11:54:58 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <18624.5354.48043.846565@montanaro-dyndns-org.local>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904142218.GC7683@idyll.org>
	<18624.5354.48043.846565@montanaro-dyndns-org.local>
Message-ID: <e6511dbf0809041154k621d24ddk5ccb608b656e78c3@mail.gmail.com>

On Thu, Sep 4, 2008 at 10:03 AM,  <skip at pobox.com> wrote:
>
>    > Compared to sqlite, you don't need to know SQL, you can finetuning
>    > (for example, using ACI instead of ACID, deciding store by store), and
>    > you can do replication and distributed transactions (useful, for
>    > example, if your storage is bigger than a single machine capacity,
>    > like my case). If you combine Berkeley DB with Durus, for example, all
>    > of this is abstracted and you simply use "regular" python objects.
>
>    Titus> I agree.  I like bsddb for just this reason and I'd like to
>    Titus> continue being able to use it!  I think that there are many
>    Titus> reasons why having such a thing in the stdlib is really useful
>    Titus> and I also think it's worth exploring the ramifications of taking
>    Titus> it out...
>
> I suggested in another message (perhaps on another thread) that maybe a
> dbm.sqlite module would be worth having.  It would have a dict-ish API like
> the other dict-on-disk modules but use the sqlite module to read (SELECT)
> and write (INSERT and UPDATE) key/value pairs from the underlying database.

I offered to write one of these a couple months ago as an option for
some users who would otherwise think to use bsddb, dbm, or anydbm.
Few people saw that anything like that would be useful, detractors
stating that the expansive options available in bsddb (most of which I
didn't realize existed) made it effectively irreplaceable to the vast
majority of people who actually use bsddb for anything nontrivial.

There wasn't even feigned interest in those use-cases involving
"trivial" disk-persistent dictionaries (of which 100% of my uses have
involved over the last 10 years).

 - Josiah

From phd at phd.pp.ru  Thu Sep  4 20:59:46 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Thu, 4 Sep 2008 22:59:46 +0400
Subject: [Python-Dev] 3rd-party dbms
Message-ID: <20080904185946.GA32410@phd.pp.ru>

Brett Cannon <brett at python.org> wrote:
> for 3.1 there is nothing saying we can't change shelve and the
> dbm package to allow 3rd-party code to register with the dbm package
> such that bsddb can be used as needed behind the scenes.

   Many years ago I wrote toy hashes based on ZODB and MetaKit. Registering
them with anydbm was easy:

import anydbm
anydbm._names.insert(len(anydbm._names)-1, ['ZODBhash', 'MKhash'])
            # Insert before dumbdbm

   More complex part was to make whichdb to recognize those hashes. I just
monkey-patched whichdb. If I were doing this now I'd do something similar
to atexit module - every hash module will register its own test function,
and whichdb will call them in turn until it finds which db it is.

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From jcea at jcea.es  Thu Sep  4 21:00:06 2008
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 04 Sep 2008 21:00:06 +0200
Subject: [Python-Dev] bsddb
In-Reply-To: <20080904180313.GA30510@phd.pp.ru>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>	<20080904025604.GD23640@idyll.org>	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>	<48BFE14A.5020208@jcea.es>
	<20080904143309.GB23604@phd.pp.ru>	<48C01D8C.6060801@jcea.es>
	<20080904180313.GA30510@phd.pp.ru>
Message-ID: <48C03036.90106@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Oleg Broytmann wrote:
>    Once I upgraded Python on a Windows computer... I think it was 2.2 to
> 2.3 upgrade - and all my bsddb databases stopped working. I cannot call
> this "improvement". I didn't have db_upgarde on that computer (or I didn't
> know about it). Does PyBSDDB have db_upgrade in the distribution? Does
> Python distribution have db_upgrade?

I can't comment about bsddb status before my maintenance era (since
March or so). But current release:

http://www.jcea.es/programacion/pybsddb_doc/db.html#upgrade

"""
[jcea at tesalia Modules]$ grep -i upgrade *
_bsddb.c:DB_upgrade(DBObject* self, PyObject* args)
_bsddb.c:    if (!PyArg_ParseTuple(args,"s|i:upgrade", &filename, &flags))
_bsddb.c:    err = self->db->upgrade(self->db, filename, flags);
_bsddb.c:    MAKE_ENTRY(nupgrade);
_bsddb.c:    {"upgrade",         (PyCFunction)DB_upgrade,
METH_VARARGS},
_bsddb.c:    ADD_INT(d, DB_UPGRADE);
_bsddb.c:    ADD_INT(d, DB_LOCK_UPGRADE);
_bsddb.c:    ADD_INT(d, DB_LOCK_UPGRADE_WRITE);
_bsddb.c:    ADD_INT(d, DB_LOCK_UPGRADE);
"""

>> Following your reasoning, Python should be
>> keep in 1.0 era, for compatibility sake.
> 
>    Python? No. But persistent data structures? Yes! How many different
> pickle data formats there were since Python 1.0? What is the oldest
> pickle format modern Python can read? (Just using pickle as an example.)

Modern bsddb can read relic bsddb data; just do a "db_upgrade".

>>> -- SQLite implements a subset of SQL - a powerful query language;
>> Yes, a declarative language completely unrelated to Python.
> 
>    Sometimes being unrelated to Python is advantage.

No argument here. My brain just work that way :).

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSMAwLJlgi5GaxT1NAQIKzgP5AeHeGF52DtGs/KMssduQczPnoH5ndgME
/265foN/qp/GM4kgunoOTPGd9kREVmxgduVaY9yNvVkQNH0WW+t+y41CIcwL36lG
EWXb+9eeAkBm7C0fFLwYZnqDva9/n9Ax7SkXHl+SOerL9Eq6rXzFXyHcTfyZtu8i
uI4q4n7nHQA=
=upd8
-----END PGP SIGNATURE-----

From brett at python.org  Thu Sep  4 21:03:07 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Sep 2008 12:03:07 -0700
Subject: [Python-Dev] 3rd-party dbms
In-Reply-To: <20080904185946.GA32410@phd.pp.ru>
References: <20080904185946.GA32410@phd.pp.ru>
Message-ID: <bbaeab100809041203h18e7587dv204306a3c5918b2d@mail.gmail.com>

On Thu, Sep 4, 2008 at 11:59 AM, Oleg Broytmann <phd at phd.pp.ru> wrote:
> Brett Cannon <brett at python.org> wrote:
>> for 3.1 there is nothing saying we can't change shelve and the
>> dbm package to allow 3rd-party code to register with the dbm package
>> such that bsddb can be used as needed behind the scenes.
>
>   Many years ago I wrote toy hashes based on ZODB and MetaKit. Registering
> them with anydbm was easy:
>
> import anydbm
> anydbm._names.insert(len(anydbm._names)-1, ['ZODBhash', 'MKhash'])
>            # Insert before dumbdbm
>
>   More complex part was to make whichdb to recognize those hashes. I just
> monkey-patched whichdb. If I were doing this now I'd do something similar
> to atexit module - every hash module will register its own test function,
> and whichdb will call them in turn until it finds which db it is.
>

That's what I figured could happen. And give dbm.any a public API to
register with it instead of setting a private attribute.

-Brett

From techtonik at gmail.com  Thu Sep  4 21:31:23 2008
From: techtonik at gmail.com (techtonik)
Date: Thu, 4 Sep 2008 21:31:23 +0200
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <e27efe130809040806s3708ec4bk43f66435d8a66b09@mail.gmail.com>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
	<loom.20080903T160547-830@post.gmane.org>
	<d34314100809031539m21fbfd00j555e6ec9c4bf15cd@mail.gmail.com>
	<loom.20080903T224344-812@post.gmane.org>
	<d34314100809040721x24a702b1tc156cda69ab53981@mail.gmail.com>
	<e27efe130809040806s3708ec4bk43f66435d8a66b09@mail.gmail.com>
Message-ID: <d34314100809041231w48197632qc6983d194f4f1b85@mail.gmail.com>

On Thu, Sep 4, 2008 at 5:06 PM, Amaury Forgeot d'Arc <amauryfa at gmail.com> >>
>
> Did you try to open your browser to (for example)
> http://svn.python.org/projects/python/trunk/Lib/distutils/
> and download the desired files from there?

Yes, but it's a waste of time. It is SVN that should be fixed unless
somebody name a good reason to disallow r/o anonymous access through
HTTPS .

-- 
--anatoly t.

From matt-python at theory.org  Thu Sep  4 21:31:27 2008
From: matt-python at theory.org (Matt Chisholm)
Date: Thu, 4 Sep 2008 12:31:27 -0700
Subject: [Python-Dev] patch for Cookie.py to add support for HttpOnly
Message-ID: <20080904193127.GF32176@tesla.theory.org>

Eighteen months ago, Arvin Schnell contributed a really
straightforward three-line patch to Cookie.py adding support for the
HttpOnly flag on cookies:

http://bugs.python.org/issue1638033

In the last eighteen months, HttpOnly has become a de-facto extension
to the cookie standard. It is now supported by IE 7, Firefox 3, and
Opera 9.5 (and there's a bug open against WebKit to support it):

http://www.owasp.org/index.php/HTTPOnly

Ruby, Perl, and PHP all support creating HttpOnly cookies now too. 

This article explains why HttpOnly is a good way to make cross-site
scripting (XSS) attacks significantly more difficult:

http://www.codinghorror.com/blog/archives/001167.htmllop

Unfortunately this patch appears to have been ignored for the last
year.

The last thing I want is a delay in the release of 2.6/3.0, but
Antoine Pitrou posted on the bug that it will have to wait for Python
2.7/3.1, because it is a feature request.  If I'm not mistaken, that
means no support for HttpOnly until sometime in 2010.

Do we really have to wait two more years to apply a three-line patch
which will bring Python in line with the industry state of the art and
improve security for Python web applications?  Is there a way that
this could go in to 2.6.1/3.0.1? 

-matt



From brett at python.org  Thu Sep  4 22:15:08 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Sep 2008 13:15:08 -0700
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
Message-ID: <bbaeab100809041315w46e5099o4e2da93b8d3b5725@mail.gmail.com>

On Wed, Sep 3, 2008 at 8:41 PM, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I'm not going to release rc1 tonight.  There are too many open release
> blockers that I don't want to defer, and I'd like the buildbots to churn
> through the bsddb removal on all platforms.  Let me first thank Benjamin,
> Brett, Mark and Antoine for their help on IRC tonight.
>
> Here are the issues I'm not comfortable with deferring:
>
>  3640 test_cpickle crash on AMD64 Windows build
> 874900 threading module can deadlock after fork
>  3574 compile() cannot decode Latin-1 source encodings
>  3657 pickle can pickle the wrong function
>  3187 os.listdir can return byte strings
>  3660 reference leaks in 3.0
>  3594 PyTokenizer_FindEncoding() never succeeds
>  3629 Py30b3 won't compile a regex that compiles with 2.5.2 and 30b2
>

I just added issue 3776 to this list: deprecate bsddb/dbhash in 2.6
for removal in 3.0 . There is a patch attached to the issue to be
reviewed.

-Brett

From musiccomposition at gmail.com  Thu Sep  4 23:24:36 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Thu, 4 Sep 2008 16:24:36 -0500
Subject: [Python-Dev] patch for Cookie.py to add support for HttpOnly
In-Reply-To: <20080904193127.GF32176@tesla.theory.org>
References: <20080904193127.GF32176@tesla.theory.org>
Message-ID: <1afaf6160809041424h54aca587s532c705d0bdba800@mail.gmail.com>

On Thu, Sep 4, 2008 at 2:31 PM, Matt Chisholm <matt-python at theory.org> wrote:
> Eighteen months ago, Arvin Schnell contributed a really
> straightforward three-line patch to Cookie.py adding support for the
> HttpOnly flag on cookies:
>
> http://bugs.python.org/issue1638033
>
> In the last eighteen months, HttpOnly has become a de-facto extension
> to the cookie standard. It is now supported by IE 7, Firefox 3, and
> Opera 9.5 (and there's a bug open against WebKit to support it):
>
> http://www.owasp.org/index.php/HTTPOnly
>
> Ruby, Perl, and PHP all support creating HttpOnly cookies now too.
>
> This article explains why HttpOnly is a good way to make cross-site
> scripting (XSS) attacks significantly more difficult:
>
> http://www.codinghorror.com/blog/archives/001167.htmllop
>
> Unfortunately this patch appears to have been ignored for the last
> year.
>
> The last thing I want is a delay in the release of 2.6/3.0, but
> Antoine Pitrou posted on the bug that it will have to wait for Python
> 2.7/3.1, because it is a feature request.  If I'm not mistaken, that
> means no support for HttpOnly until sometime in 2010.

I think we will try to shorten the release cycle for 2.7/3.1 so that
it is closer to a year.

>
> Do we really have to wait two more years to apply a three-line patch
> which will bring Python in line with the industry state of the art and
> improve security for Python web applications?  Is there a way that
> this could go in to 2.6.1/3.0.1?

Excepting it becoming a security issue, a BDFL or release manager
pronouncement, or the Spanish Inquisition, no, I'm afraid not.
>
> -matt
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From glyph at divmod.com  Fri Sep  5 00:11:53 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 04 Sep 2008 22:11:53 -0000
Subject: [Python-Dev] 'warnings' module changes still breaking Twisted,
	still looking for a fix
Message-ID: <20080904221153.25821.785590132.divmod.xquotient.15450@joule.divmod.com>


With the 2.6 final release impending, the Twisted community buildbot is 
still red, <http://bit.ly/zFymN>, but there only seems to be one real 
issue: the warn_explicit change.  This seems like it could be a pretty 
minor bit of maintenance to clear up on our end, if Python provided the 
appropriate hook.

The current solution we have in mind for this problem is to build our 
own warnings module <http://bit.ly/4F3Qr2>, but that is obviously 
inelegant.  While we do want to experiment with our own features for 
deprecation, users of Twisted will (rightfully) expect our "assertWarns" 
to continue covering warnings produced by the Python 2.6 warnings 
system.

The proposed solution to this problem seems to have been 
"warnings.catch_warnings()" <http://bugs.python.org/issue3602>, but it 
is insufficient for testing, since subsequent calls to 
"catch_warnings()" will not catch warnings suppressed by duplication 
suppression; without even getting into multiple tests testing the same 
code path, this breaks features like our 'trial' tool's "--until- 
failure" option, which runs a fixed group of tests repeatedly in an 
attempt to flush out buggy tests which have race conditions that cause 
them to fail intermittently.

We thought we'd filed a ticket for this before, but searching around in 
the issue tracker hasn't come up with anything.  JP Calderone has filed 
a new ticket for this problem: <http://bugs.python.org/issue3780> .

Any chance this could be made into a release blocker?  If this one issue 
were fixed, Python 2.6 would be the smallest disruption to Twisted since 
Python 2.0 :-).

From musiccomposition at gmail.com  Fri Sep  5 00:18:11 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Thu, 4 Sep 2008 17:18:11 -0500
Subject: [Python-Dev] 'warnings' module changes still breaking Twisted,
	still looking for a fix
In-Reply-To: <20080904221153.25821.785590132.divmod.xquotient.15450@joule.divmod.com>
References: <20080904221153.25821.785590132.divmod.xquotient.15450@joule.divmod.com>
Message-ID: <1afaf6160809041518rf45df74pbbb1c556826f5a08@mail.gmail.com>

On Thu, Sep 4, 2008 at 5:11 PM,  <glyph at divmod.com> wrote:
>
> With the 2.6 final release impending, the Twisted community buildbot is
> still red, <http://bit.ly/zFymN>, but there only seems to be one real issue:
> the warn_explicit change.  This seems like it could be a pretty minor bit of
> maintenance to clear up on our end, if Python provided the appropriate hook.
>
> The current solution we have in mind for this problem is to build our own
> warnings module <http://bit.ly/4F3Qr2>, but that is obviously inelegant.
>  While we do want to experiment with our own features for deprecation, users
> of Twisted will (rightfully) expect our "assertWarns" to continue covering
> warnings produced by the Python 2.6 warnings system.
>
> The proposed solution to this problem seems to have been
> "warnings.catch_warnings()" <http://bugs.python.org/issue3602>, but it is
> insufficient for testing, since subsequent calls to "catch_warnings()" will
> not catch warnings suppressed by duplication suppression; without even
> getting into multiple tests testing the same code path, this breaks features
> like our 'trial' tool's "--until- failure" option, which runs a fixed group
> of tests repeatedly in an attempt to flush out buggy tests which have race
> conditions that cause them to fail intermittently.

That's why catch_warning keeps track of the warnings filter too, so
you can call warnings.simplefilter("always") within the context
manager and the filter state will be restored.

>
> We thought we'd filed a ticket for this before, but searching around in the
> issue tracker hasn't come up with anything.  JP Calderone has filed a new
> ticket for this problem: <http://bugs.python.org/issue3780> .
>
> Any chance this could be made into a release blocker?  If this one issue
> were fixed, Python 2.6 would be the smallest disruption to Twisted since
> Python 2.0 :-).
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From brett at python.org  Fri Sep  5 00:35:33 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Sep 2008 15:35:33 -0700
Subject: [Python-Dev] 'warnings' module changes still breaking Twisted,
	still looking for a fix
In-Reply-To: <20080904221153.25821.785590132.divmod.xquotient.15450@joule.divmod.com>
References: <20080904221153.25821.785590132.divmod.xquotient.15450@joule.divmod.com>
Message-ID: <bbaeab100809041535i17d71a82gf289705db7767a6a@mail.gmail.com>

On Thu, Sep 4, 2008 at 3:11 PM,  <glyph at divmod.com> wrote:
>
> With the 2.6 final release impending, the Twisted community buildbot is
> still red, <http://bit.ly/zFymN>, but there only seems to be one real issue:
> the warn_explicit change.  This seems like it could be a pretty minor bit of
> maintenance to clear up on our end, if Python provided the appropriate hook.
>
> The current solution we have in mind for this problem is to build our own
> warnings module <http://bit.ly/4F3Qr2>, but that is obviously inelegant.

And possibly unnecessary if you just want the old pure Python
implementation for testing. It is not hard to force an import of
warnings to use the pure Python version and totally ignore the C
implementation. See test_warnings on how to pull that off. Then you
can do your hack of overriding warn_explicit().

>  While we do want to experiment with our own features for deprecation, users
> of Twisted will (rightfully) expect our "assertWarns" to continue covering
> warnings produced by the Python 2.6 warnings system.
>
> The proposed solution to this problem seems to have been
> "warnings.catch_warnings()" <http://bugs.python.org/issue3602>, but it is
> insufficient for testing, since subsequent calls to "catch_warnings()" will
> not catch warnings suppressed by duplication suppression; without even
> getting into multiple tests testing the same code path, this breaks features
> like our 'trial' tool's "--until- failure" option, which runs a fixed group
> of tests repeatedly in an attempt to flush out buggy tests which have race
> conditions that cause them to fail intermittently.
>
> We thought we'd filed a ticket for this before, but searching around in the
> issue tracker hasn't come up with anything.  JP Calderone has filed a new
> ticket for this problem: <http://bugs.python.org/issue3780> .

I am going to discuss technicalities on the issue.

-Brett

From glyph at divmod.com  Fri Sep  5 00:39:47 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 04 Sep 2008 22:39:47 -0000
Subject: [Python-Dev] 'warnings' module changes still breaking Twisted,
	still looking for a fix
Message-ID: <20080904223947.25821.1973582446.divmod.xquotient.15483@joule.divmod.com>

On 10:18 pm, musiccomposition at gmail.com wrote:
>That's why catch_warning keeps track of the warnings filter too, so
>you can call warnings.simplefilter("always") within the context
>manager and the filter state will be restored.

Thanks for the pointer - this is interesting.  I misunderstood the way 
the warnings filter worked.  It looks like this combination of features 
may in fact be all we need.  It would be nice if application code could 
otherwise "officially" keep track of the warnings filter (my 
understanding is that 'filters', despite the public-looking name, is 
private, since it is absent from the documentation?) but that doesn't 
sound like a release blocker.

From glyph at divmod.com  Fri Sep  5 00:46:41 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Thu, 04 Sep 2008 22:46:41 -0000
Subject: [Python-Dev] 'warnings' module changes still breaking Twisted,
	still looking for a fix
In-Reply-To: <bbaeab100809041535i17d71a82gf289705db7767a6a@mail.gmail.com>
References: <20080904221153.25821.785590132.divmod.xquotient.15450@joule.divmod.com>
	<bbaeab100809041535i17d71a82gf289705db7767a6a@mail.gmail.com>
Message-ID: <20080904224641.25821.1177384866.divmod.xquotient.15507@joule.divmod.com>


On 10:35 pm, brett at python.org wrote:
>It is not hard to force an import of
>warnings to use the pure Python version and totally ignore the C
>implementation. See test_warnings on how to pull that off. Then you
>can do your hack of overriding warn_explicit().

Benjamin Peterson's recommendation sounds like the winner (since we need 
to do *something* different in 2.6 anyway).  Forcing the import leads to 
a bunch of import-ordering problems, since the testing tool itself wants 
to emit warnings, and self-test, and then it has to import the module 
_and_ set an attribute on the SUT module...

Thanks for the tip, though.

From brett at python.org  Fri Sep  5 01:01:56 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Sep 2008 16:01:56 -0700
Subject: [Python-Dev] 'warnings' module changes still breaking Twisted,
	still looking for a fix
In-Reply-To: <20080904223947.25821.1973582446.divmod.xquotient.15483@joule.divmod.com>
References: <20080904223947.25821.1973582446.divmod.xquotient.15483@joule.divmod.com>
Message-ID: <bbaeab100809041601x6d875b0au2d15d4c50228b0ba@mail.gmail.com>

On Thu, Sep 4, 2008 at 3:39 PM,  <glyph at divmod.com> wrote:
> On 10:18 pm, musiccomposition at gmail.com wrote:
>>
>> That's why catch_warning keeps track of the warnings filter too, so
>> you can call warnings.simplefilter("always") within the context
>> manager and the filter state will be restored.
>
> Thanks for the pointer - this is interesting.  I misunderstood the way the
> warnings filter worked.  It looks like this combination of features may in
> fact be all we need.  It would be nice if application code could otherwise
> "officially" keep track of the warnings filter (my understanding is that
> 'filters', despite the public-looking name, is private, since it is absent
> from the documentation?) but that doesn't sound like a release blocker.

If you are after an API to return an opaque object that represents the
filter and be able to use that opaque object to reset the filter at a
later point (along with a matching reset function), then create an
issue for 2.7/3.1 and assign it to me as I was thinking about doing
that myself and would love to get Twisted's input on it.

_Brett

From tjreedy at udel.edu  Fri Sep  5 01:54:51 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 04 Sep 2008 19:54:51 -0400
Subject: [Python-Dev] Can/should built-in functions get __get__?
Message-ID: <g9psgb$ll4$1@ger.gmane.org>

One of the nice features of 3.0 is that differences between classes 
defined in C and Python (other than speed) are mostly erased or hidden 
from the view of a Python programmer.

However, there are still sometimes surprising and quite visible 
differences between 'functions' written in C and Python.  Can these be 
better unified also?

In particular, built-in functions, in spite of of being labeled 
'builtin_function_or_method', are not usable as methods because they 
lack the __get__ method needed to bind function to instance.

[Q. Any reason to not shorten that to 'built-in function'?]

So, as a c.l.p poster discovered, within a class statement,

__hash__ = lambda x: id(x) # works, while the apparently cleaner
__hash__ = id # raises TypeError: id() takes exactly one argument (0 given)

[Irony: trivial lambda wrapping, lambda x: f(x) has been described here 
as useless and diseased, but is seems that casting 'built-in function' 
to 'function' is not so completely useless. ;-]

In this case, __hash__ = object.__hash__ is available, as is 
object.__repr__, but this is not true in general for C functions.

The difference between a C function and a C method wrapper is 
tantalizingly small:

 >>> i = set(dir(id))
 >>> h = set(dir(object.__hash__))
 >>> i-h
{'__module__', '__self__'}
 >>> h-i
{'__objclass__', '__get__'}

Similarly, for
 >>> def f(): pass

 >>> ff = set(dir(f))
 >>> ff - i
{'__defaults__', '__annotations__', '__kwdefaults__', '__globals__', 
'__closure__', '__dict__', '__code__', '__get__'}

the only un-obvious difference is __get__,

So I cannot help but wonder: is this is essential? or could it be 
removed but has not yet?  Could the object wrapper for C functions get a 
__get__ method so they become methods when accessed via a class just 
like Python functions do?

Terry Jan Reedy


From skip at pobox.com  Fri Sep  5 02:02:39 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 4 Sep 2008 19:02:39 -0500
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
 bsddb for removal in 3.0)
In-Reply-To: <18624.5354.48043.846565@montanaro-dyndns-org.local>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904142218.GC7683@idyll.org>
	<18624.5354.48043.846565@montanaro-dyndns-org.local>
Message-ID: <18624.30495.869837.826547@montanaro-dyndns-org.local>


    me> I suggested in another message (perhaps on another thread) that
    me> maybe a dbm.sqlite module would be worth having.

http://bugs.python.org/issue3783

Skip

From lists at cheimes.de  Fri Sep  5 02:28:01 2008
From: lists at cheimes.de (Christian Heimes)
Date: Fri, 05 Sep 2008 02:28:01 +0200
Subject: [Python-Dev] Can/should built-in functions get __get__?
In-Reply-To: <g9psgb$ll4$1@ger.gmane.org>
References: <g9psgb$ll4$1@ger.gmane.org>
Message-ID: <g9pueg$r4q$1@ger.gmane.org>

Terry Reedy wrote:
> One of the nice features of 3.0 is that differences between classes 
> defined in C and Python (other than speed) are mostly erased or hidden 
> from the view of a Python programmer.
> 
> However, there are still sometimes surprising and quite visible 
> differences between 'functions' written in C and Python.  Can these be 
> better unified also?
> 
> In particular, built-in functions, in spite of of being labeled 
> 'builtin_function_or_method', are not usable as methods because they 
> lack the __get__ method needed to bind function to instance.

Python is far too late in the release cycle to introduce a drastic 
change. The issues has been discussed about half a year ago and we 
decided against changing PyCFunction.

But it was a good thing that you've raises your concern. I totally 
forgot about the new instancemethod type. My code is still in 
classobject.c but it's not exposed to Python level. IIRC the 
Pyrex/Cython guys are in need of such a feature.

Patch:

--- Python/bltinmodule.c        (revision 66222)
+++ Python/bltinmodule.c        (working copy)
@@ -2301,6 +2301,7 @@
         SETBUILTIN("frozenset",         &PyFrozenSet_Type);
         SETBUILTIN("property",          &PyProperty_Type);
         SETBUILTIN("int",               &PyLong_Type);
+       SETBUILTIN("instancemethod",    &PyInstanceMethod_Type);
         SETBUILTIN("list",              &PyList_Type);
         SETBUILTIN("map",               &PyMap_Type);
         SETBUILTIN("object",            &PyBaseObject_Type);

Result:

$ ./python
Python 3.0b3+ (py3k:66222M, Sep  5 2008, 02:24:22)
[GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
 >>> class Example:
...     id = instancemethod(id)
...
 >>> example = Example()
 >>> example.id() == id(example)
True

Christian


From brett at python.org  Fri Sep  5 06:13:30 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 4 Sep 2008 21:13:30 -0700
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
Message-ID: <bbaeab100809042113x6d2961a5k12226625693fc618@mail.gmail.com>

On Wed, Sep 3, 2008 at 8:41 PM, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I'm not going to release rc1 tonight.  There are too many open release
> blockers that I don't want to defer, and I'd like the buildbots to churn
> through the bsddb removal on all platforms.  Let me first thank Benjamin,
> Brett, Mark and Antoine for their help on IRC tonight.
>
> Here are the issues I'm not comfortable with deferring:
>
>  3640 test_cpickle crash on AMD64 Windows build
> 874900 threading module can deadlock after fork
>  3574 compile() cannot decode Latin-1 source encodings
>  3657 pickle can pickle the wrong function
>  3187 os.listdir can return byte strings
>  3660 reference leaks in 3.0
>  3594 PyTokenizer_FindEncoding() never succeeds
>  3629 Py30b3 won't compile a regex that compiles with 2.5.2 and 30b2
>

And because I can't stop causing trouble, I just uploaded a patch for
issue3781 which solidifies warnings.catch_warnings() and its API a
little bit more. Really simple patch.

-Brett

From josiah.carlson at gmail.com  Fri Sep  5 06:32:09 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 4 Sep 2008 21:32:09 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <18624.30495.869837.826547@montanaro-dyndns-org.local>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904142218.GC7683@idyll.org>
	<18624.5354.48043.846565@montanaro-dyndns-org.local>
	<18624.30495.869837.826547@montanaro-dyndns-org.local>
Message-ID: <e6511dbf0809042132i7389a14bq218f8ffd58ed9e6c@mail.gmail.com>

On Thu, Sep 4, 2008 at 5:02 PM,  <skip at pobox.com> wrote:
>
>    me> I suggested in another message (perhaps on another thread) that
>    me> maybe a dbm.sqlite module would be worth having.
>
> http://bugs.python.org/issue3783

I did a similar thing today.  I can post my version later today.

 - Josiah

From tjreedy at udel.edu  Fri Sep  5 08:43:17 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 05 Sep 2008 02:43:17 -0400
Subject: [Python-Dev] Can/should built-in functions get __get__?
In-Reply-To: <g9pueg$r4q$1@ger.gmane.org>
References: <g9psgb$ll4$1@ger.gmane.org> <g9pueg$r4q$1@ger.gmane.org>
Message-ID: <g9qke2$6tu$1@ger.gmane.org>



Christian Heimes wrote:
> Terry Reedy wrote:
>> One of the nice features of 3.0 is that differences between classes 
>> defined in C and Python (other than speed) are mostly erased or hidden 
>> from the view of a Python programmer.
>>
>> However, there are still sometimes surprising and quite visible 
>> differences between 'functions' written in C and Python.  Can these be 
>> better unified also?
>>
>> In particular, built-in functions, in spite of of being labeled 
>> 'builtin_function_or_method', are not usable as methods because they 
>> lack the __get__ method needed to bind function to instance.
> 
> Python is far too late in the release cycle to introduce a drastic 
> change.

Of course.  I should have been clear that I was asking for 3.1.

 > The issues has been discussed about half a year ago and we
> decided against changing PyCFunction.

I did't remember that.  Was it a permanent or provisional decision.

> But it was a good thing that you've raises your concern. I totally 
> forgot about the new instancemethod type. My code is still in 
> classobject.c but it's not exposed to Python level. IIRC the 
> Pyrex/Cython guys are in need of such a feature.
> 
> Patch:
> 
> --- Python/bltinmodule.c        (revision 66222)
> +++ Python/bltinmodule.c        (working copy)
> @@ -2301,6 +2301,7 @@
>         SETBUILTIN("frozenset",         &PyFrozenSet_Type);
>         SETBUILTIN("property",          &PyProperty_Type);
>         SETBUILTIN("int",               &PyLong_Type);
> +       SETBUILTIN("instancemethod",    &PyInstanceMethod_Type);
>         SETBUILTIN("list",              &PyList_Type);
>         SETBUILTIN("map",               &PyMap_Type);
>         SETBUILTIN("object",            &PyBaseObject_Type);
> 
> Result:
> 
> $ ./python
> Python 3.0b3+ (py3k:66222M, Sep  5 2008, 02:24:22)
> [GCC 4.2.3 (Ubuntu 4.2.3-2ubuntu7)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>  >>> class Example:
> ...     id = instancemethod(id)
> ...
>  >>> example = Example()
>  >>> example.id() == id(example)
> True

I consider that 2nd best but it will solve the problem once one is aware 
of it.  I am thinking of suggesting a new paragraph for the Built-in 
Functions sections warning of differences between built-in functions and 
function instances.

Terry


From asmodai at in-nomine.org  Fri Sep  5 09:34:28 2008
From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven)
Date: Fri, 5 Sep 2008 09:34:28 +0200
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769]
	Deprecate	bsddb for removal in 3.0)
In-Reply-To: <20080904142218.GC7683@idyll.org>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904142218.GC7683@idyll.org>
Message-ID: <20080905073428.GE34564@nexus.in-nomine.org>

-On [20080904 16:22], C. Titus Brown (ctb at msu.edu) wrote:
>I agree.  I like bsddb for just this reason and I'd like to continue
>being able to use it!  I think that there are many reasons why having
>such a thing in the stdlib is really useful and I also think it's worth
>exploring the ramifications of taking it out...

And having to install bsddb from an external source disables your use, how?

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
Infinite Dreams, I can't deny them, Infinity is hard to comprehend...

From fredrik at pythonware.com  Fri Sep  5 12:06:14 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Fri, 05 Sep 2008 12:06:14 +0200
Subject: [Python-Dev] Can/should built-in functions get __get__?
In-Reply-To: <g9psgb$ll4$1@ger.gmane.org>
References: <g9psgb$ll4$1@ger.gmane.org>
Message-ID: <g9r0ao$iek$1@ger.gmane.org>

Terry Reedy wrote:

> In particular, built-in functions, in spite of of being labeled 
> 'builtin_function_or_method', are not usable as methods because they 
> lack the __get__ method needed to bind function to instance.

They're not usable as Python-level instance methods, but they're 
definitely usable as methods, since they're used to implement methods 
for built-in types.

I'm probably missing something, but I don't think there's a convenient 
method to get the internal function from a built-in type.  However, you 
can find traces of them here and there:

 >>> "".upper.hello
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
AttributeError: 'builtin_function_or_method' object has no attribute 'hello'
 >>> "".upper.__call__
<method-wrapper '__call__' of builtin_function_or_method object at 
0x00C06260>

etc.

</F>


From kevin at bud.ca  Fri Sep  5 12:01:41 2008
From: kevin at bud.ca (Kevin Teague)
Date: Fri, 5 Sep 2008 03:01:41 -0700
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
	bsddb for removal in 3.0)
In-Reply-To: <20080904151050.GA8632@idyll.org>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
	<20080904143751.GD7683@idyll.org>
	<p04330102c4e5a718cbbe@[192.168.123.162]>
	<20080904151050.GA8632@idyll.org>
Message-ID: <30D8EF7E-14BE-488E-9853-6FD824F6CE4E@bud.ca>


On Sep 4, 2008, at 8:10 AM, C. Titus Brown wrote:
>
> I have to say I've never had problems with a stock install of Python  
> on
> either Mac OS X or Windows (shockingly enough :).  I think this is  
> good
> advice for applications that rely on external libraries, but I just
> don't see any problems with relying on Python 2.5 to contain all the
> things that normally come with Python 2.5.

There can be subtle differences between a "stock" Python and the  
system Python on Mac OS X 10.5. For example, Mac OS X compiles against  
EditLine instead of GNU Readline. From "man python" on Mac OS X:

"""
The  Python  inteterpreter supports editing of the current input line  
and history substitution, similar to facilities found in the Korn  
shell and the GNU Bash shell. However, rather than being implemented  
using the GNU Readline library, this Python interpreter uses the BSD  
EditLine library editline(3) with a GNU  Readline emulation layer.
...
For  example, the rlcompleter module, which defines a completion  
function for the readline modules, works correctly with the EditLine  
libraries, but needs to be initialized somewhat differently:
...
"""

Fairly rare that you'd trip over this minor difference though -  
EditLine is more a problem on Mac OS X when trying to compile your own  
Python, since you need to install and link against GNU Readline.

However, all does not seem to be right with the bsddb module on the  
system Python 2.5 on Mac OS X 10.5:

$ /usr/bin/python
Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
 >>> import bsddb
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
   File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/ 
python2.5/bsddb/__init__.py", line 51, in <module>
     import _bsddb
ImportError: No module named _bsddb
 >>>


From asmodai at in-nomine.org  Fri Sep  5 13:46:12 2008
From: asmodai at in-nomine.org (Jeroen Ruigrok van der Werven)
Date: Fri, 5 Sep 2008 13:46:12 +0200
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769]
	Deprecate	bsddb for removal in 3.0)
In-Reply-To: <30D8EF7E-14BE-488E-9853-6FD824F6CE4E@bud.ca>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
	<20080904143751.GD7683@idyll.org>
	<p04330102c4e5a718cbbe@[192.168.123.162]>
	<20080904151050.GA8632@idyll.org>
	<30D8EF7E-14BE-488E-9853-6FD824F6CE4E@bud.ca>
Message-ID: <20080905114612.GH34564@nexus.in-nomine.org>

-On [20080905 12:34], Kevin Teague (kevin at bud.ca) wrote:
>However, all does not seem to be right with the bsddb module on the  
>system Python 2.5 on Mac OS X 10.5:
>
> >>> import bsddb
[snip]
>ImportError: No module named _bsddb

The bsddb module is built separately from Python within FreeBSD's ports. I
think Apple did the same for Mac OS X.

ports/databases/py-bsddb
ports/databases/py-bsddb3

So for a fair number of systems the 'bsddb' module being an external
package/dependency is already true.

-- 
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
????? ?????? ??? ?? ??????
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
Honesty is the first chapter of the book of wisdom...

From kim.grasman at gmail.com  Fri Sep  5 13:56:02 2008
From: kim.grasman at gmail.com (=?ISO-8859-1?Q?Kim_Gr=E4sman?=)
Date: Fri, 5 Sep 2008 13:56:02 +0200
Subject: [Python-Dev] Bug in SimpleHTTPRequestHandler.send_head?
Message-ID: <beb8aa2f0809050456n38cea842mc76ffe94843f1021@mail.gmail.com>

Hi all,

I'm new to this group and the Python language as such. I stumbled on
it when I joined a project to build a rich network library for C++,
which in turn uses Python and its CGI HTTP server implementation as
part of its unit test suite.

We're having a little trouble when serving a text file containing
Windows line endings (CRLF) -- the resulting content contains Unix
line endings only (LF). This breaks our tests, because we can't verify
that the body, as parsed by our HTTP client, is the same as the source
file we're serving through the Python HTTP server.

I've isolated it to the SimpleHTTPRequestHandler.send_head method in
SimpleHTTPServer.py:

--
        ctype = self.guess_type(path)
        if ctype.startswith('text/'):
            mode = 'r'
        else:
            mode = 'rb'
        try:
            f = open(path, mode)
        except IOError:
            self.send_error(404, "File not found")
            return None
--

The f object is returned from this method, and used with
shutil.copyfileobj to copy the contents to the output stream.

This is easily fixed by omitting the content-type check entirely, and
blindly using mode 'rb', and I think that makes sense, because the
server should not be concerned with the contents of the body, so
treating it as a binary stream seems right.

This also fixes another issue, where the actual body size differs from
what's specified in the Content-Length header, because CR characters
are stripped when the body is served, but Content-Length contains the
source file's binary size.

I'm not sure which source control system you're using, so I won't try
to provide a patch, but I believe the code should read:

--
        if os.path.isdir(path):
            if not self.path.endswith('/'):
                # redirect browser - doing basically what apache does
                self.send_response(301)
                self.send_header("Location", self.path + "/")
                self.end_headers()
                return None
            for index in "index.html", "index.htm":
                index = os.path.join(path, index)
                if os.path.exists(index):
                    path = index
                    break
            else:
                return self.list_directory(path)
        #patch: removed content-type check
        try:
            f = open(path, 'rb')  #patch: always open in binary mode
        except IOError:
            self.send_error(404, "File not found")
            return None
        self.send_response(200)
        self.send_header("Content-type", self.guess_type(path))
#patch: content-type check here instead
        fs = os.fstat(f.fileno())
--

My changes marked with "#patch[...]".

Grateful for any comments!

Best wishes,
- Kim

From fuzzyman at voidspace.org.uk  Fri Sep  5 14:19:50 2008
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 05 Sep 2008 13:19:50 +0100
Subject: [Python-Dev] Bug in SimpleHTTPRequestHandler.send_head?
In-Reply-To: <beb8aa2f0809050456n38cea842mc76ffe94843f1021@mail.gmail.com>
References: <beb8aa2f0809050456n38cea842mc76ffe94843f1021@mail.gmail.com>
Message-ID: <48C123E6.80106@voidspace.org.uk>

Hello Kim,

Thanks for your post. The source code control used for Python is Subversion.

Patches submitted to this list will unfortunately get lost. Please post 
the bug report along with your comments and patch to the Python bug tracker:

http://bugs.python.org/

Michael Foord

Kim Gr?sman wrote:
> Hi all,
>
> I'm new to this group and the Python language as such. I stumbled on
> it when I joined a project to build a rich network library for C++,
> which in turn uses Python and its CGI HTTP server implementation as
> part of its unit test suite.
>
> We're having a little trouble when serving a text file containing
> Windows line endings (CRLF) -- the resulting content contains Unix
> line endings only (LF). This breaks our tests, because we can't verify
> that the body, as parsed by our HTTP client, is the same as the source
> file we're serving through the Python HTTP server.
>
> I've isolated it to the SimpleHTTPRequestHandler.send_head method in
> SimpleHTTPServer.py:
>
> --
>         ctype = self.guess_type(path)
>         if ctype.startswith('text/'):
>             mode = 'r'
>         else:
>             mode = 'rb'
>         try:
>             f = open(path, mode)
>         except IOError:
>             self.send_error(404, "File not found")
>             return None
> --
>
> The f object is returned from this method, and used with
> shutil.copyfileobj to copy the contents to the output stream.
>
> This is easily fixed by omitting the content-type check entirely, and
> blindly using mode 'rb', and I think that makes sense, because the
> server should not be concerned with the contents of the body, so
> treating it as a binary stream seems right.
>
> This also fixes another issue, where the actual body size differs from
> what's specified in the Content-Length header, because CR characters
> are stripped when the body is served, but Content-Length contains the
> source file's binary size.
>
> I'm not sure which source control system you're using, so I won't try
> to provide a patch, but I believe the code should read:
>
> --
>         if os.path.isdir(path):
>             if not self.path.endswith('/'):
>                 # redirect browser - doing basically what apache does
>                 self.send_response(301)
>                 self.send_header("Location", self.path + "/")
>                 self.end_headers()
>                 return None
>             for index in "index.html", "index.htm":
>                 index = os.path.join(path, index)
>                 if os.path.exists(index):
>                     path = index
>                     break
>             else:
>                 return self.list_directory(path)
>         #patch: removed content-type check
>         try:
>             f = open(path, 'rb')  #patch: always open in binary mode
>         except IOError:
>             self.send_error(404, "File not found")
>             return None
>         self.send_response(200)
>         self.send_header("Content-type", self.guess_type(path))
> #patch: content-type check here instead
>         fs = os.fstat(f.fileno())
> --
>
> My changes marked with "#patch[...]".
>
> Grateful for any comments!
>
> Best wishes,
> - Kim
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/
http://www.trypython.org/
http://www.ironpython.info/
http://www.theotherdelia.co.uk/
http://www.resolverhacks.net/


From solipsis at pitrou.net  Fri Sep  5 14:27:13 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 5 Sep 2008 12:27:13 +0000 (UTC)
Subject: [Python-Dev] Not releasing rc1 tonight
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
Message-ID: <loom.20080905T122512-354@post.gmane.org>

Barry Warsaw <barry <at> python.org> writes:
> 
> Here are the issues I'm not comfortable with deferring:
> 
>    3640 test_cpickle crash on AMD64 Windows build

There is a patch by Amaury which needs review.

> 874900 threading module can deadlock after fork

I've made a patch which needs review.

>    3660 reference leaks in 3.0

The last remaining patch (encode-leak2.patch) needs review as well :-)

Regards

Antoine.



From solipsis at pitrou.net  Fri Sep  5 14:56:39 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 5 Sep 2008 12:56:39 +0000 (UTC)
Subject: [Python-Dev] Python performance through times
Message-ID: <loom.20080905T125455-180@post.gmane.org>


Here's an interesting blog post comparing Python performance of various versions
from 2.2.3 upto the latest 3.0 beta.

http://www.ogre.com/node/147

The fact that only Mandelbrot calculation (a hardly representative benchmark for
a high-level dynamic language such as Python) has become significantly slower is
rather a good sign IMO.

cheers

Antoine.



From tonynelson at georgeanelson.com  Fri Sep  5 15:33:44 2008
From: tonynelson at georgeanelson.com (Tony Nelson)
Date: Fri, 5 Sep 2008 09:33:44 -0400
Subject: [Python-Dev] Bug in SimpleHTTPRequestHandler.send_head?
In-Reply-To: <48C123E6.80106@voidspace.org.uk>
References: <beb8aa2f0809050456n38cea842mc76ffe94843f1021@mail.gmail.com>
	<48C123E6.80106@voidspace.org.uk>
Message-ID: <p04330100c4e6e17ab336@[192.168.123.162]>

At 1:19 PM +0100 9/5/08, Michael Foord wrote:
>Hello Kim,
>
>Thanks for your post. The source code control used for Python is Subversion.
>
>Patches submitted to this list will unfortunately get lost. Please post
>the bug report along with your comments and patch to the Python bug tracker:
>
>http://bugs.python.org/

Patches are usually done with patch, using the output of diff -u.
bugs.python.org links to the Python wiki with Help : Tracker Documentation,
and searching the wiki can turn up some info on bug submission, but I don't
see any step-by-step instructions for newbies.

If you're not yet confident that this is really a bug or don't want to
wrestle with the bug tracker just now, you might get more disscussion on
the newsgroup comp.lang.python.  Probably the subject should not say "bug",
or you might only get suggestions to submit a bug, but rather something
like "Should SimpleHTTPRequestHandler.send_head() change text line
endings?", or whatever you think might provoke discussion.

FWIW, Python 2.6 and 3.0 are near release, so any accepted patch would at
the earliest go into the next after version of Python: 2.7 or 3.1.  Patches
often laguish and need a champion to push them through.  Helping review
other patches or bugs is one way to contribute.
-- 
____________________________________________________________________
TonyN.:'                       <mailto:tonynelson at georgeanelson.com>
      '                              <http://www.georgeanelson.com/>

From ocean-city at m2.ccsnet.ne.jp  Fri Sep  5 16:51:28 2008
From: ocean-city at m2.ccsnet.ne.jp (Hirokazu Yamamoto)
Date: Fri, 5 Sep 2008 23:51:28 +0900
Subject: [Python-Dev] Not releasing rc1 tonight
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
Message-ID: <004701c90f66$e0c3d0e0$0200a8c0@whiterabc2znlh>

"issue1040026 os.times() is bogus" won't be fixed?

From jcea at jcea.es  Fri Sep  5 17:55:13 2008
From: jcea at jcea.es (Jesus Cea)
Date: Fri, 05 Sep 2008 17:55:13 +0200
Subject: [Python-Dev] ?spurious? Timeout in BSDDB under MS Windows
Message-ID: <48C15661.9030901@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Trent, are you available to look at the ?spurious? timeout failures in
bsddb replication code in the Windows buildbot?.

Ten seconds timeout should be plenty enough. I can't debug any MS
Windows issue myself; this is a Microsoft-free environment.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSMFWXJlgi5GaxT1NAQK3BwQAlG532RIfCKRLRwQWYqUnL87F4ITxgu7W
6QgzIHoCvl/oA5G0MsaSJ6Qh3XdutH4RjovxH7Ky3rQ08lFa7iyg6mHuzkBTDbri
Pb7jgWHxQ/FImSD1rxqJHrLpKvRNoR0zjyDUVlEsgLElJ6wKGz4/eclJh0/0Wylv
QdTISCI4rR4=
=HBAA
-----END PGP SIGNATURE-----

From ncoghlan at gmail.com  Fri Sep  5 17:56:11 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 06 Sep 2008 01:56:11 +1000
Subject: [Python-Dev] Python performance through times
In-Reply-To: <loom.20080905T125455-180@post.gmane.org>
References: <loom.20080905T125455-180@post.gmane.org>
Message-ID: <48C1569B.8020901@gmail.com>

Antoine Pitrou wrote:
> Here's an interesting blog post comparing Python performance of various versions
> from 2.2.3 upto the latest 3.0 beta.
> 
> http://www.ogre.com/node/147
> 
> The fact that only Mandelbrot calculation (a hardly representative benchmark for
> a high-level dynamic language such as Python) has become significantly slower is
> rather a good sign IMO.

This comment makes the whole post fairly farcical though:

"I decided to not use pybench, but to take some of the benchmarks from
the Computer Language Benchmarks Game instead (hoping they are slightly
more "real use" realistic)."

>From what I can tell, the main graph is being dominated by the dramatic
mandelbrot slowdowns between 2.4 and 2.6, and that is heavy on the
complex arithmetic.

The numbers from pybench would have been far more enlightening.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From status at bugs.python.org  Fri Sep  5 18:06:33 2008
From: status at bugs.python.org (Python tracker)
Date: Fri,  5 Sep 2008 18:06:33 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20080905160633.7426D7856B@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (08/29/08 - 09/05/08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2023 open (+35) / 13586 closed (+21) / 15609 total (+56)

Open issues with patches:   642

Average duration of open issues: 710 days.
Median duration of open issues: 1765 days.

Open Issues Breakdown
   open  2009 (+35)
pending    14 ( +0)

Issues Created Or Reopened (57)
_______________________________

Misleading names for multiprocessing "convenience" functions     09/01/08
CLOSED http://bugs.python.org/issue3589    reopened jnoller                   
       needs review                                                            

import warning in multiprocessing                                08/29/08
CLOSED http://bugs.python.org/issue3731    created  pitrou                    
                                                                               

bdist_msi gives a deprecation warning when run with Python 2.6   08/29/08
CLOSED http://bugs.python.org/issue3732    created  lemburg                   
       patch, easy, needs review                                               

Adding bin and Scripts folder into PATH                          08/29/08
CLOSED http://bugs.python.org/issue3733    created  tarek                     
                                                                               

subclassing complex                                              08/29/08
       http://bugs.python.org/issue3734    created  gumtree                   
                                                                               

allow multiple threads to efficiently send the same requests to  08/29/08
       http://bugs.python.org/issue3735    created  DavidDecotigny            
                                                                               

super is a built-in type                                         08/29/08
       http://bugs.python.org/issue3736    created  tarek                     
                                                                               

An error in Python Library Reference document                    08/30/08
CLOSED http://bugs.python.org/issue3737    created  zouzhile                  
                                                                               

logging.Handler.close does something                             08/30/08
CLOSED http://bugs.python.org/issue3738    created  LambertDW                 
                                                                               

unicode-internal encoder reports wrong length                    08/30/08
       http://bugs.python.org/issue3739    created  doerwalter                
                                                                               

PEP 3121 --- module state is not nul-initalized as claimed in th 08/30/08
       http://bugs.python.org/issue3740    created  _doublep                  
                                                                               

DISTUTILS_USE_SDK set causes msvc9compiler.py to raise an except 08/31/08
       http://bugs.python.org/issue3741    created  TBBle                     
       patch                                                                   

Parsing XML file with Unicode characters causes problem          08/31/08
       http://bugs.python.org/issue3742    created  jaylogan                  
                                                                               

PY_FORMAT_SIZE_T is not for PyString_FromFormat                  08/31/08
       http://bugs.python.org/issue3743    created  amaury.forgeotdarc        
       patch, needs review                                                     

make altinstall installs pydoc instead of pydoc3.0               09/01/08
       http://bugs.python.org/issue3744    created  kune                      
                                                                               

_sha256 et al. encode to UTF-8 by default                        09/01/08
       http://bugs.python.org/issue3745    created  hagen                     
                                                                               

Sphinx producing duplicate id attributes, HTML fails validation. 09/01/08
       http://bugs.python.org/issue3746    created  gjhiggins                 
                                                                               

Fix caching in ABCMeta.__subclasscheck__                         09/01/08
CLOSED http://bugs.python.org/issue3747    created  ncoghlan                  
       patch, patch, needs review                                              

platform.architecture() prints bogus message on windows          09/01/08
CLOSED http://bugs.python.org/issue3748    created  ocean-city                
       patch                                                                   

incrementalencoder and incrementalencoder                        09/01/08
CLOSED http://bugs.python.org/issue3749    created  mft                       
                                                                               

test_bsddb3 skipped -- cannot import name test_support           09/01/08
CLOSED http://bugs.python.org/issue3750    created  pitrou                    
                                                                               

str.rpartition fails silently with unicode argument              09/01/08
CLOSED http://bugs.python.org/issue3751    created  forest_atq                
       patch                                                                   

test_bsddb broken                                                09/01/08
CLOSED http://bugs.python.org/issue3752    created  pitrou                    
                                                                               

bytearray incompatible with y#                                   09/01/08
CLOSED http://bugs.python.org/issue3753    created  Frostburn                 
                                                                               

minimal cross-compilation support for configure                  09/01/08
       http://bugs.python.org/issue3754    created  rpetrov                   
       patch                                                                   

Lists propagate through new instances of class if using append   09/02/08
CLOSED http://bugs.python.org/issue3755    created  supernova_hq              
                                                                               

re.escape() does not work with bytes()                           09/02/08
       http://bugs.python.org/issue3756    created  andrewmcnamara            
                                                                               

threading.local doesn't support cyclic garbage collecting        09/02/08
       http://bugs.python.org/issue3757    created  pitrou                    
                                                                               

"make check" suggest a testing target under GNU coding standards 09/02/08
       http://bugs.python.org/issue3758    created  ralph.corderoy            
                                                                               

test_asyncore.py leaks handle                                    09/02/08
CLOSED http://bugs.python.org/issue3759    created  ocean-city                
       patch, easy                                                             

PEP 3121 --- PyType_Copy is missing                              09/02/08
       http://bugs.python.org/issue3760    created  _doublep                  
                                                                               

urllib.request and urllib.response cannot handle HTTP1.1 chunked 09/02/08
       http://bugs.python.org/issue3761    created  chrisleow                 
                                                                               

platform.architecture() fails if python is lanched via its symbo 09/03/08
CLOSED http://bugs.python.org/issue3762    created  ocean-city                
       patch, easy                                                             

Python 3.0 beta 2 : json and urllib not working together?        09/03/08
       http://bugs.python.org/issue3763    created  swaroopch                 
                                                                               

asyncore differences between 2.x and 3.x                         09/03/08
       http://bugs.python.org/issue3764    created  giampaolo.rodola          
                                                                               

[patch] allow mmap take file offset as argument                  09/03/08
       http://bugs.python.org/issue3765    created  chrisl                    
                                                                               

socket.socket.recv broken (unbearably slow)                      09/03/08
       http://bugs.python.org/issue3766    created  thorben                   
                                                                               

tkColorChooser may fail if no color is selected                  09/03/08
       http://bugs.python.org/issue3767    created  gpolo                     
       patch                                                                   

Move test_py3kwarn over to new catch_warnings API                09/03/08
CLOSED http://bugs.python.org/issue3768    created  brett.cannon              
       patch, patch, needs review                                              

Deprecate bsddb for removal in 3.0                               09/03/08
CLOSED http://bugs.python.org/issue3769    created  brett.cannon              
       patch, patch, needs review                                              

test_multiprocessing fails on systems with HAVE_SEM_OPEN=0       09/04/08
       http://bugs.python.org/issue3770    created  djmdjm                    
                                                                               

test_httpservers intermittent failure                            09/04/08
       http://bugs.python.org/issue3771    created  djmdjm                    
                                                                               

logging module fails with non-ascii data                         09/04/08
CLOSED http://bugs.python.org/issue3772    created  mhammond                  
       patch                                                                   

Check for errors when using PyTokenizer_FindEncoding()           09/04/08
CLOSED http://bugs.python.org/issue3773    created  brett.cannon              
       patch                                                                   

tkinter Menu.delete bug                                          09/04/08
       http://bugs.python.org/issue3774    created  skomoroh                  
       patch, easy, needs review                                               

Update RELNOTES file                                             09/04/08
       http://bugs.python.org/issue3775    created  barry                     
                                                                               

deprecate bsddb/dbhash in 2.6 for removal in 3.0                 09/04/08
       http://bugs.python.org/issue3776    created  brett.cannon              
       patch, patch, needs review                                              

PyNumber_Long fails from Float                                   09/04/08
       http://bugs.python.org/issue3777    created  barry-scott               
                                                                               

python uninstaller leave registry entries                        09/04/08
       http://bugs.python.org/issue3778    created  donovaly                  
                                                                               

log into bugs.python.org requires cookies                        09/04/08
CLOSED http://bugs.python.org/issue3779    created  donovaly                  
                                                                               

No way to write unit tests for warnings                          09/04/08
CLOSED http://bugs.python.org/issue3780    created  exarkun                   
                                                                               

warnings.catch_warnings fails gracelessly when recording warning 09/04/08
       http://bugs.python.org/issue3781    created  exarkun                   
       patch, needs review                                                     

os.write accepts unicode strings                                 09/04/08
       http://bugs.python.org/issue3782    created  pitrou                    
                                                                               

dbm.sqlite proof of concept                                      09/04/08
       http://bugs.python.org/issue3783    created  skip.montanaro            
       patch                                                                   

Incorrect compiler options used for cc of Sun Studio 12          09/05/08
       http://bugs.python.org/issue3784    created  mschmarck                 
                                                                               

Can't build ctypes of Python 2.5.2 with Sun Studio 12            09/05/08
       http://bugs.python.org/issue3785    created  mschmarck                 
                                                                               

_curses, _curses_panel & _multiprocessing can't be build in 2.6b 09/05/08
       http://bugs.python.org/issue3786    created  mschmarck                 
                                                                               



Issues Now Closed (67)
______________________

LookupError: unknown encoding: X-MAC-JAPANESE                     322 days
       http://bugs.python.org/issue1276    josm                      
       patch                                                                   

New SSL module doesn't seem to verify hostname against commonNam  269 days
       http://bugs.python.org/issue1589    heikki                    
                                                                               

"RuntimeError: dictionary changed size during iteration" in Tkin  260 days
       http://bugs.python.org/issue1658    gpolo                     
       patch                                                                   

__eq__ / __hash__ check doesn't take inheritance into account       3 days
       http://bugs.python.org/issue2235    ncoghlan                  
       patch                                                                   

Update What's new in 2.6                                          172 days
       http://bugs.python.org/issue2305    facundobatista            
                                                                               

Clean up getargs.c and its formatting possibilities               170 days
       http://bugs.python.org/issue2322    barry                     
                                                                               

xml.sax.parser() doesn't terminate when given a filename          160 days
       http://bugs.python.org/issue2501    benjamin.peterson         
       patch                                                                   

Cannot use non-ascii letters in disutils if setuptools is used.   150 days
       http://bugs.python.org/issue2562    lemburg                   
       patch                                                                   

Update interface of weakref dictionaries                          102 days
       http://bugs.python.org/issue2965    barry                     
       patch                                                                   

VS8 include dirs grow without bound                                99 days
       http://bugs.python.org/issue2975    jkleckner                 
       patch                                                                   

Building a Win32 binary installer crashes                          75 days
       http://bugs.python.org/issue3160    pitrou                    
       patch, needs review                                                     

Deficiencies in multiprocessing/threading API                      50 days
       http://bugs.python.org/issue3352    ncoghlan                  
       patch, needs review                                                     

multiprocessing module is racy                                     45 days
       http://bugs.python.org/issue3419    jnoller                   
       patch, needs review                                                     

In function call, keyword arguments could follow *args             35 days
       http://bugs.python.org/issue3473    amaury.forgeotdarc        
       patch                                                                   

New Global Module Index glitch on WinXP                            25 days
       http://bugs.python.org/issue3520    georg.brandl              
                                                                               

Glitch in eval() doc                                               14 days
       http://bugs.python.org/issue3569    georg.brandl              
                                                                               

Interpreter name: python vs. python-3.0                            13 days
       http://bugs.python.org/issue3577    georg.brandl              
                                                                               

Misleading names for multiprocessing "convenience" functions        1 days
       http://bugs.python.org/issue3589    ncoghlan                  
       needs review                                                            

sax.parser considers XML as text rather than bytes                 16 days
       http://bugs.python.org/issue3590    benjamin.peterson         
                                                                               

PyTokenizer_FindEncoding() never succeeds                          16 days
       http://bugs.python.org/issue3594    brett.cannon              
       patch, needs review                                                     

Move test.test_suport.catch_warning() to the 'warnings' module     13 days
       http://bugs.python.org/issue3602    brett.cannon              
       patch                                                                   

test_multiprocessing failure (Unserializable message)              15 days
       http://bugs.python.org/issue3607    jnoller                   
                                                                               

2to3 refactoring                                                   12 days
       http://bugs.python.org/issue3637    benjamin.peterson         
       patch, needs review                                                     

segfaults calling warnings.warn() with non-string message          11 days
       http://bugs.python.org/issue3639    brett.cannon              
       patch                                                                   

segfault calling sys.excepthook with non-Exception argument         9 days
       http://bugs.python.org/issue3653    georg.brandl              
       patch, needs review                                                     

Reloading an extension module always leaks                         10 days
       http://bugs.python.org/issue3667    benjamin.peterson         
       patch, needs review                                                     

"s*" argument parser marker leaks memory                            5 days
       http://bugs.python.org/issue3668    pitrou                    
       patch, needs review                                                     

bsddb module leaks memory                                          10 days
       http://bugs.python.org/issue3673    jcea                      
       patch, patch                                                            

compilation --without-threads fails                                 7 days
       http://bugs.python.org/issue3683    benjamin.peterson         
       patch, needs review                                                     

Source tarball for Sphinx 0.4.3 missing                             3 days
       http://bugs.python.org/issue3695    georg.brandl              
                                                                               

Error parsing arguments on OpenBSD <= 4.4                           8 days
       http://bugs.python.org/issue3696    pitrou                    
       patch, needs review                                                     

"Fatal Python error: Cannot recover from stack overflow" on Wind    7 days
       http://bugs.python.org/issue3697    pitrou                    
       patch, needs review                                                     

open() on directory raises IOError with unhelpful message           5 days
       http://bugs.python.org/issue3703    benjamin.peterson         
       patch, needs review                                                     

help('finally') behaves bizarrely                                   3 days
       http://bugs.python.org/issue3707    georg.brandl              
       patch, easy, needs review                                               

os.urandom(1.1): infinite loop                                      5 days
       http://bugs.python.org/issue3708    gregory.p.smith           
       patch, easy                                                             

memoryview leaks references                                         4 days
       http://bugs.python.org/issue3712    pitrou                    
       patch                                                                   

hashlib's docstring throws exception in pydoc                       3 days
       http://bugs.python.org/issue3715    gregory.p.smith           
       easy                                                                    

mistake in 3.4.2 Customizing attribute access                       2 days
       http://bugs.python.org/issue3716    georg.brandl              
                                                                               

platform.py: _syscmd_file() can't handle target path with	space     5 days
       http://bugs.python.org/issue3719    jfdp                      
       patch                                                                   

invalid literal for int() with base 16: ''                          7 days
       http://bugs.python.org/issue3721    amaury.forgeotdarc        
                                                                               

Allow ',<whitespace>' delimiters in logging.config.fileConfig()     5 days
       http://bugs.python.org/issue3726    vsajip                    
       patch                                                                   

Update BaseHTTPServer docs                                          1 days
       http://bugs.python.org/issue3730    georg.brandl              
                                                                               

import warning in multiprocessing                                   3 days
       http://bugs.python.org/issue3731    pitrou                    
                                                                               

bdist_msi gives a deprecation warning when run with Python 2.6      3 days
       http://bugs.python.org/issue3732    ocean-city                
       patch, easy, needs review                                               

Adding bin and Scripts folder into PATH                             4 days
       http://bugs.python.org/issue3733    amaury.forgeotdarc        
                                                                               

An error in Python Library Reference document                       0 days
       http://bugs.python.org/issue3737    georg.brandl              
                                                                               

logging.Handler.close does something                                2 days
       http://bugs.python.org/issue3738    vsajip                    
                                                                               

Fix caching in ABCMeta.__subclasscheck__                            1 days
       http://bugs.python.org/issue3747    ncoghlan                  
       patch, patch, needs review                                              

platform.architecture() prints bogus message on windows             1 days
       http://bugs.python.org/issue3748    ocean-city                
       patch                                                                   

incrementalencoder and incrementalencoder                           0 days
       http://bugs.python.org/issue3749    georg.brandl              
                                                                               

test_bsddb3 skipped -- cannot import name test_support              0 days
       http://bugs.python.org/issue3750    jcea                      
                                                                               

str.rpartition fails silently with unicode argument                 0 days
       http://bugs.python.org/issue3751    amaury.forgeotdarc        
       patch                                                                   

test_bsddb broken                                                   2 days
       http://bugs.python.org/issue3752    jcea                      
                                                                               

bytearray incompatible with y#                                      0 days
       http://bugs.python.org/issue3753    georg.brandl              
                                                                               

Lists propagate through new instances of class if using append      0 days
       http://bugs.python.org/issue3755    benjamin.peterson         
                                                                               

test_asyncore.py leaks handle                                       2 days
       http://bugs.python.org/issue3759    ocean-city                
       patch, easy                                                             

platform.architecture() fails if python is lanched via its symbo    1 days
       http://bugs.python.org/issue3762    ocean-city                
       patch, easy                                                             

Move test_py3kwarn over to new catch_warnings API                   0 days
       http://bugs.python.org/issue3768    brett.cannon              
       patch, patch, needs review                                              

Deprecate bsddb for removal in 3.0                                  2 days
       http://bugs.python.org/issue3769    barry                     
       patch, patch, needs review                                              

logging module fails with non-ascii data                            0 days
       http://bugs.python.org/issue3772    vsajip                    
       patch                                                                   

Check for errors when using PyTokenizer_FindEncoding()              1 days
       http://bugs.python.org/issue3773    amaury.forgeotdarc        
       patch                                                                   

log into bugs.python.org requires cookies                           0 days
       http://bugs.python.org/issue3779    benjamin.peterson         
                                                                               

No way to write unit tests for warnings                             0 days
       http://bugs.python.org/issue3780    brett.cannon              
                                                                               

socket-module SSL is broken                                      2412 days
       http://bugs.python.org/issue508944  janssen                   
                                                                               

relocate cgi.parse_qs() into urlparse                            2200 days
       http://bugs.python.org/issue600362  facundobatista            
       patch                                                                   

New class special method lookup change                           2106 days
       http://bugs.python.org/issue643841  ncoghlan                  
       patch                                                                   

SSLObject breaks read semantics                                  1086 days
       http://bugs.python.org/issue1291446 ellisj                    
                                                                               



Top Issues Most Discussed (10)
______________________________

 17 Deprecate bsddb for removal in 3.0                                 2 days
closed  http://bugs.python.org/issue3769   

 17 segfault in for loop with evil iterator                            7 days
open    http://bugs.python.org/issue3720   

 13 Deficiencies in multiprocessing/threading API                     50 days
closed  http://bugs.python.org/issue3352   

 12 reference leaks in 3.0                                            12 days
open    http://bugs.python.org/issue3660   

 10 platform.py: _syscmd_file() can't handle target path with	space    5 days
closed  http://bugs.python.org/issue3719   

 10 Misleading names for multiprocessing "convenience" functions       1 days
closed  http://bugs.python.org/issue3589   

  9 Check for errors when using PyTokenizer_FindEncoding()             1 days
closed  http://bugs.python.org/issue3773   

  8 dbm.sqlite proof of concept                                        1 days
open    http://bugs.python.org/issue3783   

  8 Precompute range length                                          133 days
open    http://bugs.python.org/issue2690   

  7 No way to write unit tests for warnings                            0 days
closed  http://bugs.python.org/issue3780   




From techtonik at gmail.com  Fri Sep  5 20:13:02 2008
From: techtonik at gmail.com (techtonik)
Date: Fri, 5 Sep 2008 20:13:02 +0200
Subject: [Python-Dev] xmlrpc via proxy for bzr and bugzilla in 2.6
Message-ID: <d34314100809051113n3d6627b2w490756686bc5b1b6@mail.gmail.com>

With 2.6rc1 at the doors people are asking if xmlrpclib will be able
to communicate through a proxy? It causes bzr and bugzilla tools to
fail if used behind firewall, and there is no easy workaround for
users.

http://bugs.python.org/issue648658
-- 
--anatoly t.

From lrn1986 at gmail.com  Fri Sep  5 22:55:34 2008
From: lrn1986 at gmail.com (LRN)
Date: Sat, 06 Sep 2008 00:55:34 +0400
Subject: [Python-Dev] 2.6b3 and 3.0b3 Windows installers
Message-ID: <48C19CC6.4040607@gmail.com>

I was wondering, when new installers will be published? I made one on my 
own ( see http://lrn.1986.li/other/python-2.6.14127-x86.msi ), but i am 
not sure about it's correctness.

From musiccomposition at gmail.com  Fri Sep  5 23:38:41 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Fri, 5 Sep 2008 16:38:41 -0500
Subject: [Python-Dev] site.py and the checkout builddir
Message-ID: <1afaf6160809051438w35993a75i99c32e3a3f63fb56@mail.gmail.com>

In a checkout, site.py is currently responsible for adding the
distutils extension module builddir to sys.path. Usually, this is
unproblematic. However, when the initialization code needed by site
(in this case the standard io streams and the builtin open) relies on
C modules added from the builddir in site.py, it causes silent errors.
The calls to initstdio and initsite were moved to having the site
initialization first in r62778, so that _bytesio and _stringio could
be used when io was loaded. Unfortunately, site.py uses open, but this
error went undetected because stderr was not set up yet. (See issue
#3279)

That options as I see it are:
1. Switch the initialization order back to the original (io streams
first) and compile _bytesio and _stringio directly into the Python
binary. This is probably the easiest option.
2. Switch the initialization order back to the original and have
getpath.c put the builddir on sys.path. This might be difficult
because finding the distutils buildir seems to involve
disutils.utils.get_platform(). (I'm not a disutils expert; can the
location of the builddir be simplified?)

Anyway, note that this doesn't affect Python which has been installed
on a system because the dynlibs are already on sys.path.

-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From solipsis at pitrou.net  Fri Sep  5 23:51:32 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 5 Sep 2008 21:51:32 +0000 (UTC)
Subject: [Python-Dev] site.py and the checkout builddir
References: <1afaf6160809051438w35993a75i99c32e3a3f63fb56@mail.gmail.com>
Message-ID: <loom.20080905T214836-544@post.gmane.org>

Benjamin Peterson <musiccomposition <at> gmail.com> writes:
> 
> That options as I see it are:
> 1. Switch the initialization order back to the original (io streams
> first) and compile _bytesio and _stringio directly into the Python
> binary. This is probably the easiest option.

Since io.py imports _bytesio and _stringio, and io.py is always imported at
startup, this option sounds harmless.

Regards

Antoine.



From musiccomposition at gmail.com  Fri Sep  5 23:57:14 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Fri, 5 Sep 2008 16:57:14 -0500
Subject: [Python-Dev] site.py and the checkout builddir
In-Reply-To: <loom.20080905T214836-544@post.gmane.org>
References: <1afaf6160809051438w35993a75i99c32e3a3f63fb56@mail.gmail.com>
	<loom.20080905T214836-544@post.gmane.org>
Message-ID: <1afaf6160809051457y2a099acdt73b1784a7345d25d@mail.gmail.com>

On Fri, Sep 5, 2008 at 4:51 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Benjamin Peterson <musiccomposition <at> gmail.com> writes:
>>
>> That options as I see it are:
>> 1. Switch the initialization order back to the original (io streams
>> first) and compile _bytesio and _stringio directly into the Python
>> binary. This is probably the easiest option.
>
> Since io.py imports _bytesio and _stringio, and io.py is always imported at
> startup, this option sounds harmless.

Yeah, the only harm being that the python binary gets a little
heftier. (Not a big issue, since 3.0's libpython is half a megabyte
less than 2.6's)

>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From lists at cheimes.de  Sat Sep  6 00:53:16 2008
From: lists at cheimes.de (Christian Heimes)
Date: Sat, 06 Sep 2008 00:53:16 +0200
Subject: [Python-Dev] site.py and the checkout builddir
In-Reply-To: <1afaf6160809051438w35993a75i99c32e3a3f63fb56@mail.gmail.com>
References: <1afaf6160809051438w35993a75i99c32e3a3f63fb56@mail.gmail.com>
Message-ID: <g9sd8r$sul$1@ger.gmane.org>

Benjamin Peterson wrote:
> 1. Switch the initialization order back to the original (io streams
> first) and compile _bytesio and _stringio directly into the Python
> binary. This is probably the easiest option.

Oh, the modules are still shared libraries? That's clearly a mistake. It 
makes no sense to have both modules as shared libraries when they are 
loaded on every interpreter startup.

+1 for making them builtin modules.

Christian


From amauryfa at gmail.com  Sat Sep  6 00:59:50 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sat, 6 Sep 2008 00:59:50 +0200
Subject: [Python-Dev] xmlrpc via proxy for bzr and bugzilla in 2.6
In-Reply-To: <d34314100809051113n3d6627b2w490756686bc5b1b6@mail.gmail.com>
References: <d34314100809051113n3d6627b2w490756686bc5b1b6@mail.gmail.com>
Message-ID: <e27efe130809051559q63a2fdb5r4e6e3ebd25c2d33a@mail.gmail.com>

Hello,

techtonik wrote:
> With 2.6rc1 at the doors people are asking if xmlrpclib will be able
> to communicate through a proxy? It causes bzr and bugzilla tools to
> fail if used behind firewall, and there is no easy workaround for
> users.
>
> http://bugs.python.org/issue648658

It's very unlikely that such a change can be made at this stage.
There is not even a patch to review...

But the tracker seems to contains a workaround.
It seems to work without any modification of the standard library
Does it actually work? Could bzr use it?

-- 
Amaury Forgeot d'Arc

From greg.ewing at canterbury.ac.nz  Sat Sep  6 04:09:47 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Sat, 06 Sep 2008 14:09:47 +1200
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
 bsddb for removal in 3.0)
In-Reply-To: <30D8EF7E-14BE-488E-9853-6FD824F6CE4E@bud.ca>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
	<20080904143751.GD7683@idyll.org>
	<p04330102c4e5a718cbbe@[192.168.123.162]>
	<20080904151050.GA8632@idyll.org>
	<30D8EF7E-14BE-488E-9853-6FD824F6CE4E@bud.ca>
Message-ID: <48C1E66B.8070909@canterbury.ac.nz>

Kevin Teague wrote:

> There can be subtle differences between a "stock" Python and the  system 
> Python on Mac OS X 10.5.

Also there can be different versions of Python installed
in different versions of MacOSX. So if you distribute an app
that relies on the system Python, at the least you have
to test it against all the Python versions it's likely to
encounter, and possibly provide different versions of the
app for different versions of MacOSX.

Another thing to keep in mind is that if you use something
like py2app or py2exe, it doesn't include the whole Python
distribution, just the stdlib modules the app actually uses.
So it's not as bad as including a whole Python installation
for every app.

-- 
Greg

From steve at holdenweb.com  Sat Sep  6 05:42:09 2008
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 05 Sep 2008 23:42:09 -0400
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <g9ks5n$um4$1@ger.gmane.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>	<48BDC325.1000901@egenix.com>
	<g9ks5n$um4$1@ger.gmane.org>
Message-ID: <48C1FC11.9080702@holdenweb.com>

Terry Reedy wrote:
> 
> 
> M.-A. Lemburg wrote:
>> On 2008-09-02 23:14, Terry Reedy wrote:
> 
>>> An alternative to manipulating PATH would be to make and add to the
>>> Start Menu a Command Prompt shortcut, call it Command Window or
>>> something, that starts in the Python directory.  Then one could enter
>>>> python or >Scripts/goforit without chdir-ing to the Python directory
>>> first.  The background could even be set to green, for instance, to
>>> differentiate it from the standard Command Prompt window.
>>
>> There already is a menu entry that starts the Python interpreter
>> on Windows, so why not use that ?
> 
> I do and will*, but some people want to start a command window and then
> type 'python' just like they would on *nix (*without having to
> explicitly change to the Python directory*), or be able to run stuff in
> a subdirectory of the Python directory.  So I suggested an alternative
> to the request for PATH manipulation that could easily be done now.
> 
A cleaner (though still dirty) way to achieve this would be to add a
link to the appropriate "python.exe" in a directory already on the path
such as c:\Windows\system32

> * I recently *did* run Python from a command prompt so I could make sure
> it was running 'cmd.exe /u' and try changing the code page (but this is
> another story), but I simply moved to the directory first.  For me, not
> a big deal.
> 
My own solution, on systems where I haven't bothered to add \python25
and python25\Scripts to the PATH, is to simply use

  \python25\python

With tab expansion enabled by default it's easy enough.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From steve at holdenweb.com  Sat Sep  6 05:42:09 2008
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 05 Sep 2008 23:42:09 -0400
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <g9ks5n$um4$1@ger.gmane.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>	<48BDC325.1000901@egenix.com>
	<g9ks5n$um4$1@ger.gmane.org>
Message-ID: <48C1FC11.9080702@holdenweb.com>

Terry Reedy wrote:
> 
> 
> M.-A. Lemburg wrote:
>> On 2008-09-02 23:14, Terry Reedy wrote:
> 
>>> An alternative to manipulating PATH would be to make and add to the
>>> Start Menu a Command Prompt shortcut, call it Command Window or
>>> something, that starts in the Python directory.  Then one could enter
>>>> python or >Scripts/goforit without chdir-ing to the Python directory
>>> first.  The background could even be set to green, for instance, to
>>> differentiate it from the standard Command Prompt window.
>>
>> There already is a menu entry that starts the Python interpreter
>> on Windows, so why not use that ?
> 
> I do and will*, but some people want to start a command window and then
> type 'python' just like they would on *nix (*without having to
> explicitly change to the Python directory*), or be able to run stuff in
> a subdirectory of the Python directory.  So I suggested an alternative
> to the request for PATH manipulation that could easily be done now.
> 
A cleaner (though still dirty) way to achieve this would be to add a
link to the appropriate "python.exe" in a directory already on the path
such as c:\Windows\system32

> * I recently *did* run Python from a command prompt so I could make sure
> it was running 'cmd.exe /u' and try changing the code page (but this is
> another story), but I simply moved to the directory first.  For me, not
> a big deal.
> 
My own solution, on systems where I haven't bothered to add \python25
and python25\Scripts to the PATH, is to simply use

  \python25\python

With tab expansion enabled by default it's easy enough.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From steve at holdenweb.com  Sat Sep  6 13:26:36 2008
From: steve at holdenweb.com (Steve Holden)
Date: Sat, 06 Sep 2008 07:26:36 -0400
Subject: [Python-Dev] bsddb
In-Reply-To: <d2155e360809041149p1d645a23ufc0d89e207207110@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>	<20080904025604.GD23640@idyll.org>	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>	<48BFE14A.5020208@jcea.es>
	<20080904143309.GB23604@phd.pp.ru>
	<d2155e360809041149p1d645a23ufc0d89e207207110@mail.gmail.com>
Message-ID: <g9tpdo$j2s$1@ger.gmane.org>

Curt Hagenlocher wrote:
> On Thu, Sep 4, 2008 at 7:33 AM, Oleg Broytmann <phd at phd.pp.ru> wrote:
>> SQLite is public domain; the licensing terms of Berkeley DB[1] are not
>> friendly to commercial applications: "Our open source license ...
>> permits use of Berkeley DB in open source projects or in applications
>> that are not distributed to third parties." I am not sure if using of
>> PyBSDDB in commercial applications is considered "using of Berkeley DB
>> in open source projects";
> 
> Wow, I hadn't realized that it was such a restrictive license.  When I
> see "Berkeley" I think "BSD license".
> 
Well of course nowadays when you see "SleepyCat" you need to be thinking
"Oracle". They have dabbled in open source, but I didn't get the
impression that the support for Linux was particularly wholehearted, for
example.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From fredrik at pythonware.com  Sun Sep  7 16:51:33 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Sun, 07 Sep 2008 16:51:33 +0200
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
Message-ID: <ga0ppm$udq$1@ger.gmane.org>

Barry Warsaw wrote:

> I'm not going to release rc1 tonight.  There are too many open release 
> blockers that I don't want to defer, and I'd like the buildbots to churn 
> through the bsddb removal on all platforms.

> I'd like to try again on Friday and stick to rc2 on the 17th.

any news on this front?

(I have a few minor ET fixes, and possibly a Unicode 5.1 patch, but have 
had absolutely no time to spend on that.  is the window still open?)

</F>


From martin at v.loewis.de  Sun Sep  7 17:03:10 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 07 Sep 2008 17:03:10 +0200
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
Message-ID: <48C3ED2E.4060603@v.loewis.de>

> SVN checkout over HTTPS protocol requires password. Is it intentional
> or just temporary server issue? I am behind a proxy that doesn't
> support PROPFIND requests and I can't test SVN+SSH, because 22 port is
> closed.
> 
> Site docs keep silence about that HTTPS is used at all. Shouldn't
> authentication be asked only on write access in normal SVN setup?

Not necessarily - as you say, it's undocumented (and will remain so);
in any case, I have now granted anonymous read access to that
repository, through https.

That there was access at all was for the sake of a completely unrelated
repository for which even anonymous read access is denied (and remains
so).

Regards,
Martin

From barry at python.org  Sun Sep  7 18:02:06 2008
From: barry at python.org (Barry Warsaw)
Date: Sun, 7 Sep 2008 12:02:06 -0400
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <ga0ppm$udq$1@ger.gmane.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
Message-ID: <9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 7, 2008, at 10:51 AM, Fredrik Lundh wrote:

> Barry Warsaw wrote:
>
>> I'm not going to release rc1 tonight.  There are too many open  
>> release blockers that I don't want to defer, and I'd like the  
>> buildbots to churn through the bsddb removal on all platforms.
>
>> I'd like to try again on Friday and stick to rc2 on the 17th.
>
> any news on this front?
>
> (I have a few minor ET fixes, and possibly a Unicode 5.1 patch, but  
> have had absolutely no time to spend on that.  is the window still  
> open?)

There are 8 open release blockers, a few of which have patches that  
need review.  So I think we are still not ready to release rc1.  But  
it worries me because I think this is going to push the final release  
beyond our October 1st goal.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMP6/3EjvBPtnXfVAQIprQQAsWgxQPKyxM/rrG5TWL4UqI7xne6dLTjL
Nx3OBpi8hcNXEqyxzoosFXXZy4PpSWU+SwxuI1YQT9rUjv/ks6yxu3cBcEVhtEHV
KE34YS4D825tVGvbvpsOXF06fsfv5j5zZGB6hlSipZoiv1rhR3uEsO2zkWaI4eQ6
Ty2Cfuxu10A=
=8eP5
-----END PGP SIGNATURE-----

From g.brandl at gmx.net  Sun Sep  7 19:04:38 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 07 Sep 2008 19:04:38 +0200
Subject: [Python-Dev] 2.6 doc searching issue?
In-Reply-To: <18622.41381.600223.440920@montanaro-dyndns-org.local>
References: <18622.41381.600223.440920@montanaro-dyndns-org.local>
Message-ID: <ga11j9$kld$1@ger.gmane.org>

skip at pobox.com schrieb:
>>From this page:
> 
>     http://docs.python.org/dev/index.html
> 
> I searched for "csv" and got just one hit:
> 
>     http://docs.python.org/dev/contents.html?highlight=csv
> 
> Shouldn't it have at least matched the docs for the csv module itself, not
> just the table of contents?

It definitely should. We're currently working on improving the JS-based
search to search object (module, function, class) names before trying
full-text search.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From greg at krypto.org  Sun Sep  7 20:04:23 2008
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 7 Sep 2008 11:04:23 -0700
Subject: [Python-Dev] bsddb
In-Reply-To: <20080904143309.GB23604@phd.pp.ru>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904143309.GB23604@phd.pp.ru>
Message-ID: <52dc1c820809071104g29f69290o7cc7fc697b5a4c55@mail.gmail.com>

On Thu, Sep 4, 2008 at 7:33 AM, Oleg Broytmann <phd at phd.pp.ru> wrote:
> On Thu, Sep 04, 2008 at 03:23:22PM +0200, Jesus Cea wrote:
>> Compared to sqlite, you don't need to know SQL, you can finetuning (for
>> example, using ACI instead of ACID, deciding store by store), and you
>> can do replication and distributed transactions (useful, for example, if
>> your storage is bigger than a single machine capacity, like my case).
>
>   Let me raise the glove. Compared to bsddb:
>
> -- SQLite is public domain; the licensing terms of Berkeley DB[1] are not
>   friendly to commercial applications: "Our open source license ...
>   permits use of Berkeley DB in open source projects or in applications
>   that are not distributed to third parties." I am not sure if using of
>   PyBSDDB in commercial applications is considered "using of Berkeley DB
>   in open source projects";

FWIW, many years ago in the past when I asked sleepycat about this
(long before oracle bought them) they said that python was considered
to be the application.  Using berkeleydb via python for a commercial
application did not require a berkeleydb license.  But my legal advice
is worth as much as the paper its printed on.  Always ask your own
lawyer and oracle about such things.

-gps

From greg at krypto.org  Sun Sep  7 20:34:37 2008
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 7 Sep 2008 11:34:37 -0700
Subject: [Python-Dev] bsddb
In-Reply-To: <20080904180313.GA30510@phd.pp.ru>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904143309.GB23604@phd.pp.ru>
	<48C01D8C.6060801@jcea.es> <20080904180313.GA30510@phd.pp.ru>
Message-ID: <52dc1c820809071134j36c0297ase5f6cc71dfd830d0@mail.gmail.com>

On Thu, Sep 4, 2008 at 11:03 AM, Oleg Broytmann <phd at phd.pp.ru> wrote:
> On Thu, Sep 04, 2008 at 07:40:28PM +0200, Jesus Cea wrote:
>> A stable fileformat is useful for long term support, but an evolving
>> format allows improvements.
>
>   Once I upgraded Python on a Windows computer... I think it was 2.2 to
> 2.3 upgrade - and all my bsddb databases stopped working. I cannot call
> this "improvement". I didn't have db_upgarde on that computer (or I didn't
> know about it). Does PyBSDDB have db_upgrade in the distribution? Does
> Python distribution have db_upgrade?

Unfortunately that is a bad example and should be blamed on python and
not berkeleydb:

Going from python 2.2 -> 2.3 was when the bsddb module was renamed to
bsddb185 and the bsddb3 module was included as bsddb.  It was an
incompatible change because the underlying library was entirely
replaced with something much much different despite happening to share
the name and being made to support the same API.

You could probably have built the bsddb185 module and loaded your data
from that and rewritten it using the new bsddb module.  The db upgrade
API is available in the bsddb.db module.  I never got around to
writing the automatic on disk format upgrade code for the
bsddb/__init__.py API.  It wouldn't have solved the 2.2->2.3 problem
but it would remove the need to know about db_upgrade in all future
versions.

fwiw, a precompiled berkeleydb with db_upgrade.exe and friends has
always been available from sleepycat/oracle so a solution exists even
if it wasn't nicely documented in a "reading your data when upgrading
python HOWTO."

>> Following your reasoning, Python should be
>> keep in 1.0 era, for compatibility sake.
>
>   Python? No. But persistent data structures? Yes! How many different
> pickle data formats there were since Python 1.0? What is the oldest
> pickle format modern Python can read? (Just using pickle as an example.)

Python by itself won't solve your long term data warehousing needs.

The SQLite project's on disk format has already changed at least once
(sqlite2 -> sqlite3) and no doubt it could change again in the future.

The lesson for python: when that happens lets write the code to make
the transition between formats trivial.

-gps

From phd at phd.pp.ru  Sun Sep  7 21:01:21 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Sun, 7 Sep 2008 23:01:21 +0400
Subject: [Python-Dev] bsddb and sqlite
In-Reply-To: <52dc1c820809071134j36c0297ase5f6cc71dfd830d0@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904143309.GB23604@phd.pp.ru>
	<48C01D8C.6060801@jcea.es> <20080904180313.GA30510@phd.pp.ru>
	<52dc1c820809071134j36c0297ase5f6cc71dfd830d0@mail.gmail.com>
Message-ID: <20080907190121.GA20369@phd.pp.ru>

On Sun, Sep 07, 2008 at 11:34:37AM -0700, Gregory P. Smith wrote:
> You could probably have built the bsddb185 module and loaded your data
> from that and rewritten it using the new bsddb module.

   I built bsddb185, loaded old data, exported it to... I don't remember
now, but I clearly remember I stopped using bsddb.

> The lesson for python: when that happens lets write the code to make
> the transition between formats trivial.

   For me the lesson is different - do not include modules in the stdlib
that relies on unstable 3rd party libraries. I consider bsddb unstable.
sqlite is more stable, but PySQLite... there are many minor releases between
Python releases; my humble opinion is it'd be better to have one external
PySQLite module than two (PySQLite and sqlite3).

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From fredrik at pythonware.com  Sun Sep  7 22:12:54 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Sun, 07 Sep 2008 22:12:54 +0200
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
Message-ID: <ga1ck7$ldi$1@ger.gmane.org>

Barry Warsaw wrote:

>> (I have a few minor ET fixes, and possibly a Unicode 5.1 patch, but 
>> have had absolutely no time to spend on that.  is the window still open?)
> 
> There are 8 open release blockers, a few of which have patches that need 
> review.  So I think we are still not ready to release rc1.

So what's the new ETA?  Should I set aside some time to work on the 
patches, say, tomorrow, or is it too late?

</F>


From greg at krypto.org  Sun Sep  7 23:35:58 2008
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 7 Sep 2008 14:35:58 -0700
Subject: [Python-Dev] [issue3769] Deprecate bsddb for removal in 3.0
In-Reply-To: <bbaeab100809041130g7ca19b3aw576418e42f83fb84@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<bbaeab100809031712y6bfc1e53md15f5c28d00f7ed3@mail.gmail.com>
	<48BFE43D.5060904@jcea.es>
	<bbaeab100809041130g7ca19b3aw576418e42f83fb84@mail.gmail.com>
Message-ID: <52dc1c820809071435x39a14e4bo8704009878bb0112@mail.gmail.com>

On Thu, Sep 4, 2008 at 11:30 AM, Brett Cannon <brett at python.org> wrote:
> Another thing to keep in mind with the whole shelve/dbm.any argument
> is that for 3.1 there is nothing saying we can't change shelve and the
> dbm package to allow 3rd-party code to register with the dbm package
> such that bsddb can be used as needed behind the scenes.
>
> -Brett

Exactly.  That is what I think should really happen here.

From phd at phd.pp.ru  Sun Sep  7 23:55:29 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Mon, 8 Sep 2008 01:55:29 +0400
Subject: [Python-Dev] 3rd party dbm (was: [issue3769] Deprecate bsddb
	for removal in 3.0)
In-Reply-To: <52dc1c820809071435x39a14e4bo8704009878bb0112@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<bbaeab100809031712y6bfc1e53md15f5c28d00f7ed3@mail.gmail.com>
	<48BFE43D.5060904@jcea.es>
	<bbaeab100809041130g7ca19b3aw576418e42f83fb84@mail.gmail.com>
	<52dc1c820809071435x39a14e4bo8704009878bb0112@mail.gmail.com>
Message-ID: <20080907215529.GD20369@phd.pp.ru>

On Sun, Sep 07, 2008 at 02:35:58PM -0700, Gregory P. Smith wrote:
> On Thu, Sep 4, 2008 at 11:30 AM, Brett Cannon <brett at python.org> wrote:
> > Another thing to keep in mind with the whole shelve/dbm.any argument
> > is that for 3.1 there is nothing saying we can't change shelve and the
> > dbm package to allow 3rd-party code to register with the dbm package
> > such that bsddb can be used as needed behind the scenes.
> 
> Exactly.  That is what I think should really happen here.

   I will try to find a spare time to some job in the area. I am planning
API like this (in terms of Python 2.x with anydbm):

# dbm.something module
import anydbm
anydbm.register('something', whichdb_test_function)

   whichdb_test_function is not required - whichdb module can provide
a generic test function:

def generic_test(filename, module_name):
    module = __import__(module_name)
    try:
        module.open(filename)
    except:
        return False
    else:
        return True

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From steve at holdenweb.com  Mon Sep  8 00:43:30 2008
From: steve at holdenweb.com (Steve Holden)
Date: Sun, 07 Sep 2008 18:43:30 -0400
Subject: [Python-Dev] bsddb and sqlite
In-Reply-To: <20080907190121.GA20369@phd.pp.ru>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>	<20080904025604.GD23640@idyll.org>	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>	<48BFE14A.5020208@jcea.es>
	<20080904143309.GB23604@phd.pp.ru>	<48C01D8C.6060801@jcea.es>
	<20080904180313.GA30510@phd.pp.ru>	<52dc1c820809071134j36c0297ase5f6cc71dfd830d0@mail.gmail.com>
	<20080907190121.GA20369@phd.pp.ru>
Message-ID: <ga1let$bep$1@ger.gmane.org>

Oleg Broytmann wrote:
> On Sun, Sep 07, 2008 at 11:34:37AM -0700, Gregory P. Smith wrote:
>> You could probably have built the bsddb185 module and loaded your data
>> from that and rewritten it using the new bsddb module.
> 
>    I built bsddb185, loaded old data, exported it to... I don't remember
> now, but I clearly remember I stopped using bsddb.
> 
>> The lesson for python: when that happens lets write the code to make
>> the transition between formats trivial.
> 
>    For me the lesson is different - do not include modules in the stdlib
> that relies on unstable 3rd party libraries. I consider bsddb unstable.
> sqlite is more stable, but PySQLite... there are many minor releases between
> Python releases; my humble opinion is it'd be better to have one external
> PySQLite module than two (PySQLite and sqlite3).
> 
Unfortunately this advice should have been taken several years ago. The
fact is that there are almost certainly Python users who rely on the
presence of the bsddb module for production work, and simply removing it
without deprecation is bound to upset those users.

I'm particularly concerned that it appears that normal procedures have
been circumvented to enable its removal from 3.0. Since we have at least
one developer committed to ongoing support that seems both harsh and
unnecessary.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From guido at python.org  Mon Sep  8 01:01:49 2008
From: guido at python.org (Guido van Rossum)
Date: Sun, 7 Sep 2008 16:01:49 -0700
Subject: [Python-Dev] bsddb and sqlite
In-Reply-To: <ga1let$bep$1@ger.gmane.org>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904143309.GB23604@phd.pp.ru>
	<48C01D8C.6060801@jcea.es> <20080904180313.GA30510@phd.pp.ru>
	<52dc1c820809071134j36c0297ase5f6cc71dfd830d0@mail.gmail.com>
	<20080907190121.GA20369@phd.pp.ru> <ga1let$bep$1@ger.gmane.org>
Message-ID: <ca471dc20809071601j3d11e31bk80336b5e389d13dc@mail.gmail.com>

On Sun, Sep 7, 2008 at 3:43 PM, Steve Holden <steve at holdenweb.com> wrote:
> Oleg Broytmann wrote:
>> On Sun, Sep 07, 2008 at 11:34:37AM -0700, Gregory P. Smith wrote:
>>> You could probably have built the bsddb185 module and loaded your data
>>> from that and rewritten it using the new bsddb module.
>>
>>    I built bsddb185, loaded old data, exported it to... I don't remember
>> now, but I clearly remember I stopped using bsddb.
>>
>>> The lesson for python: when that happens lets write the code to make
>>> the transition between formats trivial.
>>
>>    For me the lesson is different - do not include modules in the stdlib
>> that relies on unstable 3rd party libraries. I consider bsddb unstable.
>> sqlite is more stable, but PySQLite... there are many minor releases between
>> Python releases; my humble opinion is it'd be better to have one external
>> PySQLite module than two (PySQLite and sqlite3).
>>
> Unfortunately this advice should have been taken several years ago. The
> fact is that there are almost certainly Python users who rely on the
> presence of the bsddb module for production work, and simply removing it
> without deprecation is bound to upset those users.

Those users would first have to port their code to Python 3.0. That
task is a lot larger than dealing with a separate download of bsddb.
It is not being removed from 2.6.

> I'm particularly concerned that it appears that normal procedures have
> been circumvented to enable its removal from 3.0. Since we have at least
> one developer committed to ongoing support that seems both harsh and
> unnecessary.

3.0 breaks a lot of things. Most of the library reorg may have been
discussed more than this particular removal, but that doesn't mean
that changes won't come as a surprise for most users. In this case, a
completely compatible module is available as a 3rd party download.
That's a lot less sever than the complete abandonment than the fat of
many other modules. It's just a matter of source code packaging.
Vendors can completely remove the difference in their packaging of the
binaries.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip at pobox.com  Mon Sep  8 02:34:03 2008
From: skip at pobox.com (skip at pobox.com)
Date: Sun, 7 Sep 2008 19:34:03 -0500
Subject: [Python-Dev] bsddb and sqlite
In-Reply-To: <ca471dc20809071601j3d11e31bk80336b5e389d13dc@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904143309.GB23604@phd.pp.ru>
	<48C01D8C.6060801@jcea.es> <20080904180313.GA30510@phd.pp.ru>
	<52dc1c820809071134j36c0297ase5f6cc71dfd830d0@mail.gmail.com>
	<20080907190121.GA20369@phd.pp.ru> <ga1let$bep$1@ger.gmane.org>
	<ca471dc20809071601j3d11e31bk80336b5e389d13dc@mail.gmail.com>
Message-ID: <18628.29435.3653.340003@montanaro-dyndns-org.local>

    >> Unfortunately this advice should have been taken several years
    >> ago. The fact is that there are almost certainly Python users who
    >> rely on the presence of the bsddb module for production work, and
    >> simply removing it without deprecation is bound to upset those users.

    Guido> Those users would first have to port their code to Python
    Guido> 3.0. That task is a lot larger than dealing with a separate
    Guido> download of bsddb.  It is not being removed from 2.6.

Perhaps 2.7 and 3.1 should have a conversion function at the dbm package
level which will convert a database from one format to another, e.g.:

    dbm.convert(srcdb, dstdb, dstfmt)

The format of srcdb should be discoverable from the file itself.

Skip


From martin at v.loewis.de  Mon Sep  8 06:03:30 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 08 Sep 2008 06:03:30 +0200
Subject: [Python-Dev] bsddb alternative (was Re: [issue3769] Deprecate
 bsddb for removal in 3.0)
In-Reply-To: <p04330101c4e59ecad7d6@[192.168.123.162]>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>	<20080904025604.GD23640@idyll.org>	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>	<18623.49679.433662.690583@montanaro-dyndns-org.local>
	<p04330101c4e59ecad7d6@[192.168.123.162]>
Message-ID: <48C4A412.2090801@v.loewis.de>

> I thought that all that was happening was that BSDDB was becoming a
> separate project.  If one needs BSDDB with Python2.6, one installs it.

No, not in the way you mean it.

> Aren't there other parts of Python that require external modules, such as
> Tk?

It's different. BSDDB (the Sleepycat-then-Oracle implementation) never
was part of Python; this hasn't changed. What *has* changed is that
the bsddb module (i.e. the Python wrapper) is now not part of Python
anymore, either, due to it being maintained separately. This is as if
Tkinter was removed from Python.

Regards,
Martin


From martin at v.loewis.de  Mon Sep  8 07:46:42 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 08 Sep 2008 07:46:42 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <03b801c90d58$b5744970$205cdc50$@com.au>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>	<loom.20080902T224020-709@post.gmane.org>
	<03b801c90d58$b5744970$205cdc50$@com.au>
Message-ID: <48C4BC42.1070402@v.loewis.de>

> FWIW, my opinion is similar to how I read Martin's - that if a suitable,
> safe patch that cleanly uninstalls can be found, it should be included, but
> disabled by default.  Personally I'd never use it.

That's my view also (less strict now; I previously would have rejected
it outright, as I didn't consider that it might be optional and off by
default).

Regards,
Martin

From martin at v.loewis.de  Mon Sep  8 07:53:40 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 08 Sep 2008 07:53:40 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <loom.20080903T141921-651@post.gmane.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>	<loom.20080902T224020-709@post.gmane.org>	<03b801c90d58$b5744970$205cdc50$@com.au>	<loom.20080903T002033-116@post.gmane.org>	<03d001c90d6b$fc7b2380$f5716a80$@com.au>
	<loom.20080903T141921-651@post.gmane.org>
Message-ID: <48C4BDE4.3020605@v.loewis.de>


>> I mean that many Windows use the PATH, and as such, may fail if a new
>> directory is added to the PATH that contains a DLL they indirectly use.
> 
> Then it's just a matter of not putting any DLLs in those directories, isn't it?

A. It's not just DLLs. Any program invoking CreateProcess might pick up
   the Python executable. This might be confusing if the program would
   previously pick up a different Python installation.
B. I don't think this can work: we *must* install DLLs into the Python
   directory - at least pythonxy.dll (at least the way Python is
   currently build - maybe SxS would allow to place it elsewhere).
   I *think* we also need to place the CRT manifest in the directory;
   not sure what consequences this has.

> Most Linux distributions solve that by installing binaries named
> /usr/bin/python2.4, /usr/bin/python2.5, etc., and making /usr/bin/python a
> symlink to one of those. Thus if a program relies on particular Python version,
> it can just use a specific executable rather than a generic one.

Unfortunately, symlinks are not available on Windows.

OTOH, other things *are* available, such as registered extensions.
For example, you don't need python on PATH to start a Python script;
just invoking the .py file will find the Python interpreter from the
registry.

I don't think it is wise to apply Unix solutions to Windows problems.

Regards,
Martin


From martin at v.loewis.de  Mon Sep  8 07:57:49 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 08 Sep 2008 07:57:49 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48C1FC11.9080702@holdenweb.com>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>	<48BDC325.1000901@egenix.com>	<g9ks5n$um4$1@ger.gmane.org>
	<48C1FC11.9080702@holdenweb.com>
Message-ID: <48C4BEDD.6010101@v.loewis.de>

> A cleaner (though still dirty) way to achieve this would be to add a
> link to the appropriate "python.exe" in a directory already on the path
> such as c:\Windows\system32

That would be fairly easy to implement. I suppose pythonw.exe wouldn't
need the same treatment, as people won't invoke it manually.

I guess that option wouldn't be available to non-admin users, though.

> My own solution, on systems where I haven't bothered to add \python25
> and python25\Scripts to the PATH, is to simply use
> 
>   \python25\python
> 
> With tab expansion enabled by default it's easy enough.

Same for me. That's why I like Python being installed in the root
directory (also a common complaint - why doesn't it install into
the Programs Files folder?)

Regards,
Martin

From martin at v.loewis.de  Mon Sep  8 08:03:11 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 08 Sep 2008 08:03:11 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <op.ugwsrved03jqhe@cesareprova.org>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>
	<48BDC325.1000901@egenix.com>	<op.ugwbwbqy03jqhe@cesareprova.org>
	<48BE763A.1040408@egenix.com> <op.ugwsrved03jqhe@cesareprova.org>
Message-ID: <48C4C01F.2090306@v.loewis.de>

> May be an external program called by the uninstaller can take care
> of this, removing what was added to PATH.

Or a custom action. There are ways to solve this problem - they just
take some effort to implement them.

Regards,
Martin

From martin at v.loewis.de  Mon Sep  8 08:35:18 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 08 Sep 2008 08:35:18 +0200
Subject: [Python-Dev] constness of interpreter data
In-Reply-To: <20080901153817.GA24171@wolfpuppy.org.uk>
References: <20080901153817.GA24171@wolfpuppy.org.uk>
Message-ID: <48C4C7A6.9080204@v.loewis.de>

> So, er, is this a remotely sane thing to be doing, and does anyone have
> suggestions? :)

Not for 2.6/3.0 - but after that: sounds good.

OTOH, I don't see a clear technical *need* for these data to be const.
In general, const-ness helps correctness and sharing across processes,
and the cases you provide are correct and provide sharing even without
being const.

Regards,
Martin

From python-dev-list at trentnelson.com  Mon Sep  8 12:24:03 2008
From: python-dev-list at trentnelson.com (Trent Nelson)
Date: Mon, 8 Sep 2008 10:24:03 +0000
Subject: [Python-Dev] ?spurious? Timeout in BSDDB under MS Windows
In-Reply-To: <48C15661.9030901@jcea.es>
References: <48C15661.9030901@jcea.es>
Message-ID: <20080908102403.GE63632@wind.teleri.net>

On Fri, Sep 05, 2008 at 05:55:13PM +0200, Jesus Cea wrote:
> Trent, are you available to look at the ?spurious? timeout failures in
> bsddb replication code in the Windows buildbot?.
> 
> Ten seconds timeout should be plenty enough. I can't debug any MS
> Windows issue myself; this is a Microsoft-free environment.

    I think I added in 10 seconds 'cause the tests kept failing when it
    was at 2 seconds ;-)
    
    I remember digging around the code a bit when I bumped bsddb to 4.7
    on Windows to try and figure out what was going on.  As far as I
    could tell it wasn't anything obvious caused by the Python code; is
    it possible this could be an issue with the underlying bsddb code?
    
    Side note: are all your recent bsddb changes that went into trunk
    also available in py3k?  I've had "bump py3k Windows buildbots to
    use bsddb 4.7.25" on my todo list for far too long!  

        Trent.


From solipsis at pitrou.net  Mon Sep  8 13:28:40 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 8 Sep 2008 11:28:40 +0000 (UTC)
Subject: [Python-Dev] Not releasing rc1 tonight
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
Message-ID: <loom.20080908T112613-297@post.gmane.org>

Fredrik Lundh <fredrik <at> pythonware.com> writes:
> 
> So what's the new ETA?  Should I set aside some time to work on the 
> patches, say, tomorrow, or is it too late?

Given the state of things in the tracker, I'd say it doesn't look too late.



From amk at amk.ca  Mon Sep  8 13:37:44 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Mon, 8 Sep 2008 07:37:44 -0400
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
Message-ID: <20080908113744.GA6051@amk-desktop.matrixgroup.net>

On Sun, Sep 07, 2008 at 12:02:06PM -0400, Barry Warsaw wrote:
> There are 8 open release blockers, a few of which have patches that need 
> review.  So I think we are still not ready to release rc1.  But it 
> worries me because I think this is going to push the final release  
> beyond our October 1st goal.

Should we try to schedule a bug evening some time this week?

--amk

From barry at python.org  Mon Sep  8 15:16:07 2008
From: barry at python.org (Barry Warsaw)
Date: Mon, 8 Sep 2008 09:16:07 -0400
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <ga1ck7$ldi$1@ger.gmane.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
Message-ID: <F1FCDA57-E3A8-44F6-AB40-83889C4CC3FA@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 7, 2008, at 4:12 PM, Fredrik Lundh wrote:

> Barry Warsaw wrote:
>
>>> (I have a few minor ET fixes, and possibly a Unicode 5.1 patch,  
>>> but have had absolutely no time to spend on that.  is the window  
>>> still open?)
>> There are 8 open release blockers, a few of which have patches that  
>> need review.  So I think we are still not ready to release rc1.
>
> So what's the new ETA?  Should I set aside some time to work on the  
> patches, say, tomorrow, or is it too late?

It's not too late.  If they fix bugs and the code gets reviewed then  
yes, you can check them in.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMUlmHEjvBPtnXfVAQJ51QP7BdUGcKN4+L9vD+g7y2TI0+TSw4Ms+eAc
yXprcbQnfGp1+uxzjiTCeAv0OSAodw4aakAaI4wzrAkKYNmsVaWOiGKiKrLvR7+Y
++qBxxxVwlKL606hlJCKgphD4hbZcW1w3wY94CXkmrTqyZe/XrStvBj7X10gWeYW
lwC3ATaQQ5Y=
=tyym
-----END PGP SIGNATURE-----

From barry at python.org  Mon Sep  8 15:19:39 2008
From: barry at python.org (Barry Warsaw)
Date: Mon, 8 Sep 2008 09:19:39 -0400
Subject: [Python-Dev] Not releasing rc1 tonight
In-Reply-To: <20080908113744.GA6051@amk-desktop.matrixgroup.net>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<20080908113744.GA6051@amk-desktop.matrixgroup.net>
Message-ID: <C09F4DE5-A7F9-4F0E-9573-338A6322FDC6@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 8, 2008, at 7:37 AM, A.M. Kuchling wrote:

> On Sun, Sep 07, 2008 at 12:02:06PM -0400, Barry Warsaw wrote:
>> There are 8 open release blockers, a few of which have patches that  
>> need
>> review.  So I think we are still not ready to release rc1.  But it
>> worries me because I think this is going to push the final release
>> beyond our October 1st goal.
>
> Should we try to schedule a bug evening some time this week?

Monday, Tuesday and Friday are good for me.  I might be around a bit  
on Wednesday evening.  (All times EDT/UTC-4).  I'm usually on irc.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMUma3EjvBPtnXfVAQIkEQP9HdcmjL3zRLO5yxBt3JEfxd2l924wFgAa
avi5VZMA3YFRCqfmfS/BBtng2qTSbzyL8UO9tWSVdtjLd62g2uLuS1UzcBJ+O8qE
I1veedtxxoSvjDOoVYmuYy3dS1ZFTEKvEs0PrE2ukoGzPkRpZTAQ1AeTxSXbuRc9
fZXgOWVYg6k=
=PQUF
-----END PGP SIGNATURE-----

From barry at python.org  Mon Sep  8 15:23:37 2008
From: barry at python.org (Barry Warsaw)
Date: Mon, 8 Sep 2008 09:23:37 -0400
Subject: [Python-Dev] Proposed revised schedule
In-Reply-To: <ga1ck7$ldi$1@ger.gmane.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
Message-ID: <D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I don't think there's any way we're going to make our October 1st  
goal.  We have 8 open release critical bugs, and 18 deferred  
blockers.  We do not have a beta3 Windows installer and I don't have  
high hopes for rectifying all of these problems in the next day or two.

I propose that we push the entire schedule back two weeks.  This means  
that the planned rc2 on 17-September becomes our rc1.  The planned  
final release for 01-October becomes our rc2, and we release the  
finals on 15-October.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMUnWXEjvBPtnXfVAQIEAQQAnut+CRyBAacC2zzptb5l9cphwke0sEjx
THJXHCBUfidaEV7SCtyfkh6i+IpqynvFRsKyOYSWsMojAa5rO/iM6ZJLkUav9c62
IzweJ6Nw3UnOJ/7xksCesDVxDRncFtvu0eRUZWDkOsrNawL+Z21DGKtAuau/pgiY
sFnKeyP7NX0=
=ZNPm
-----END PGP SIGNATURE-----

From guido at python.org  Mon Sep  8 19:13:08 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Sep 2008 10:13:08 -0700
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
Message-ID: <ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>

On Mon, Sep 8, 2008 at 6:23 AM, Barry Warsaw <barry at python.org> wrote:
> I don't think there's any way we're going to make our October 1st goal.  We
> have 8 open release critical bugs, and 18 deferred blockers.  We do not have
> a beta3 Windows installer and I don't have high hopes for rectifying all of
> these problems in the next day or two.
>
> I propose that we push the entire schedule back two weeks.  This means that
> the planned rc2 on 17-September becomes our rc1.  The planned final release
> for 01-October becomes our rc2, and we release the finals on 15-October.
>
> - -Barry

Perhaps it's time to separate the 2.6 and 3.0 release schedules? I
don't care if the next version of OSX contains 3.0 or not -- but I do
care about it having 2.6.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From zooko at zooko.com  Mon Sep  8 23:18:27 2008
From: zooko at zooko.com (zooko)
Date: Mon, 8 Sep 2008 15:18:27 -0600
Subject: [Python-Dev] bsddb
In-Reply-To: <52dc1c820809071104g29f69290o7cc7fc697b5a4c55@mail.gmail.com>
References: <1220484752.61.0.214853794698.issue3769@psf.upfronthosting.co.za>
	<D4C0145D0BCC4C00B9E9743D07D330C9@RaymondLaptop1>
	<20080904025604.GD23640@idyll.org>
	<bbaeab100809032045m37fdccb3i33274f001c6f3610@mail.gmail.com>
	<48BFE14A.5020208@jcea.es> <20080904143309.GB23604@phd.pp.ru>
	<52dc1c820809071104g29f69290o7cc7fc697b5a4c55@mail.gmail.com>
Message-ID: <379E1358-C2F5-4AE8-A615-F585FCE8F4C8@zooko.com>

On Sep 7, 2008, at 12:04 PM, Gregory P. Smith wrote:

> FWIW, many years ago in the past when I asked sleepycat about this
> (long before oracle bought them) they said that python was considered
> to be the application.  Using berkeleydb via python for a commercial
> application did not require a berkeleydb license.

They also posted a FAQ on their web site which included that  
statement, including specifically declaring that using BerkeleyDB via  
Python for a commercial product did not require a commercial licence.

Oh, look, it is still there:

http://www.oracle.com/technology/software/products/berkeley-db/htdocs/ 
licensing.html

"""
Q. Do I have to pay for a Berkeley DB license to use it in my Perl or  
Python scripts?

A. No, you may use the Berkeley DB open source license at no cost.  
The Berkeley DB open source license requires that software that uses  
Berkeley DB be freely redistributable. In the case of Perl or Python,  
that software is Perl or Python, and not your scripts. Any scripts  
you write are your property, including scripts that make use of  
Berkeley DB. None of the Perl, Python or Berkeley DB licenses place  
any restrictions on what you may do with them.
"""

Regards,

Zooko
---
http://allmydata.org -- Tahoe, the Least-Authority Filesystem
http://allmydata.com -- back up all your files for $5/month


From musiccomposition at gmail.com  Tue Sep  9 01:13:29 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Mon, 8 Sep 2008 18:13:29 -0500
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
Message-ID: <1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>

On Mon, Sep 8, 2008 at 12:13 PM, Guido van Rossum <guido at python.org> wrote:
> On Mon, Sep 8, 2008 at 6:23 AM, Barry Warsaw <barry at python.org> wrote:
>> I don't think there's any way we're going to make our October 1st goal.  We
>> have 8 open release critical bugs, and 18 deferred blockers.  We do not have
>> a beta3 Windows installer and I don't have high hopes for rectifying all of
>> these problems in the next day or two.
>>
>> I propose that we push the entire schedule back two weeks.  This means that
>> the planned rc2 on 17-September becomes our rc1.  The planned final release
>> for 01-October becomes our rc2, and we release the finals on 15-October.
>>
>> - -Barry
>
> Perhaps it's time to separate the 2.6 and 3.0 release schedules? I
> don't care if the next version of OSX contains 3.0 or not -- but I do
> care about it having 2.6.

I'm not really sure what good that would do us unless we wanted to
bring 3.0 back to the beta phase and continue to work on some larger
issues with it. I also suspect doing two separate, but close together
final releases would be more stressful than having them in lock and
step.

Just my pocket change, though.



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From guido at python.org  Tue Sep  9 01:25:10 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Sep 2008 16:25:10 -0700
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
Message-ID: <ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>

On Mon, Sep 8, 2008 at 4:13 PM, Benjamin Peterson
<musiccomposition at gmail.com> wrote:
> On Mon, Sep 8, 2008 at 12:13 PM, Guido van Rossum <guido at python.org> wrote:
>> On Mon, Sep 8, 2008 at 6:23 AM, Barry Warsaw <barry at python.org> wrote:
>>> I don't think there's any way we're going to make our October 1st goal.  We
>>> have 8 open release critical bugs, and 18 deferred blockers.  We do not have
>>> a beta3 Windows installer and I don't have high hopes for rectifying all of
>>> these problems in the next day or two.
>>>
>>> I propose that we push the entire schedule back two weeks.  This means that
>>> the planned rc2 on 17-September becomes our rc1.  The planned final release
>>> for 01-October becomes our rc2, and we release the finals on 15-October.
>>>
>>> - -Barry
>>
>> Perhaps it's time to separate the 2.6 and 3.0 release schedules? I
>> don't care if the next version of OSX contains 3.0 or not -- but I do
>> care about it having 2.6.
>
> I'm not really sure what good that would do us unless we wanted to
> bring 3.0 back to the beta phase and continue to work on some larger
> issues with it. I also suspect doing two separate, but close together
> final releases would be more stressful than having them in lock and
> step.

Well, from the number of release blockers it sounds like another 3.0
beta is the right thing. For 2.6 however I believe we're much closer
to the finish line -- there aren't all those bytes/str issues to clean
up, for example! And apparently the benefit of releasing on schedule
is that we will be included in OSX. That's a much bigger deal for 2.6
than for 3.0 (I doubt that Apple would add two versions anyway).

> Just my pocket change, though.
>
>
>
> --
> Cheers,
> Benjamin Peterson
> "There's no place like 127.0.0.1."
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From greg.ewing at canterbury.ac.nz  Tue Sep  9 02:02:32 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 09 Sep 2008 12:02:32 +1200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48C4BDE4.3020605@v.loewis.de>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>
	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>
	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>
	<g9kaav$hj2$1@ger.gmane.org> <loom.20080902T224020-709@post.gmane.org>
	<03b801c90d58$b5744970$205cdc50$@com.au>
	<loom.20080903T002033-116@post.gmane.org>
	<03d001c90d6b$fc7b2380$f5716a80$@com.au>
	<loom.20080903T141921-651@post.gmane.org>
	<48C4BDE4.3020605@v.loewis.de>
Message-ID: <48C5BD18.4030900@canterbury.ac.nz>

Martin v. L?wis wrote:

> OTOH, other things *are* available, such as registered extensions.
> For example, you don't need python on PATH to start a Python script;
> just invoking the .py file will find the Python interpreter from the
> registry.

But then you don't get to pass arguments to the program,
get to see the output before the window disappears, etc.

-- 
Greg

From lists at cheimes.de  Tue Sep  9 02:11:49 2008
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 09 Sep 2008 02:11:49 +0200
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>	<ga1ck7$ldi$1@ger.gmane.org>	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
Message-ID: <48C5BF45.9020807@cheimes.de>

Guido van Rossum wrote:
> Well, from the number of release blockers it sounds like another 3.0
> beta is the right thing. For 2.6 however I believe we're much closer
> to the finish line -- there aren't all those bytes/str issues to clean
> up, for example! And apparently the benefit of releasing on schedule
> is that we will be included in OSX. That's a much bigger deal for 2.6
> than for 3.0 (I doubt that Apple would add two versions anyway).

I'm on Guido's side.

Ok, from the marketing perspective it's a nice catch to release 2.6 and 
3.0 on the same day. "Python 2.6.0 and 3.0.0 released" makes a great 
headline.
But given the chance to get Python 2.6 into the next OSX version it's 
fine with me to release 3.0 a couple of weeks later. Python 3.0 is not 
ready for a release candidate. We just fixed a bunch of memory leaks and 
critical errors over the last week. And don't forget Windows! The 
Windows builds didn't get thorough testing because we didn't provide our 
tests with official builds.

I'm +1 for a 2.6rc and another beta of 3.0

Christian

From solipsis at pitrou.net  Tue Sep  9 02:22:16 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 9 Sep 2008 00:22:16 +0000 (UTC)
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>	<ga1ck7$ldi$1@ger.gmane.org>	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
	<48C5BF45.9020807@cheimes.de>
Message-ID: <loom.20080909T001934-453@post.gmane.org>

Christian Heimes <lists <at> cheimes.de> writes:
> 
> Ok, from the marketing perspective it's a nice catch to release 2.6 and 
> 3.0 on the same day. "Python 2.6.0 and 3.0.0 released" makes a great 
> headline.

It's not only the marketing. Having both releases in lock step means the
development process is synchronized between trunk and py3k, that there is no
loss of developer focus, and that merges/backports happen quite naturally.

But I don't think it's an overwhelming argument either. I would value it at
around 50 euro cents, not even the price of a good croissant ;-)

Regards

Antoine.



From guido at python.org  Tue Sep  9 02:50:52 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Sep 2008 17:50:52 -0700
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <loom.20080909T001934-453@post.gmane.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
	<48C5BF45.9020807@cheimes.de>
	<loom.20080909T001934-453@post.gmane.org>
Message-ID: <ca471dc20809081750n1bc10837r20ba917ef7221c41@mail.gmail.com>

On Mon, Sep 8, 2008 at 5:22 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> Christian Heimes <lists <at> cheimes.de> writes:
>>
>> Ok, from the marketing perspective it's a nice catch to release 2.6 and
>> 3.0 on the same day. "Python 2.6.0 and 3.0.0 released" makes a great
>> headline.
>
> It's not only the marketing. Having both releases in lock step means the
> development process is synchronized between trunk and py3k, that there is no
> loss of developer focus, and that merges/backports happen quite naturally.

I think that we've reached the point where very few things are merged
from 2.6 to 3.0 -- I see a lot more "block" commits than "merge"
commits lately. Also, the added activity in 3.0 doesn't involve merges
at all, because it's all 3.0-specific.

Sure, we lose the ability to add last-minute -3 warnings. But I think
that's a pretty minor issue (and those warnings have a tendency to
subtly break things occasionally, so we shouldn't do them last-minute
anyway).

> But I don't think it's an overwhelming argument either. I would value it at
> around 50 euro cents, not even the price of a good croissant ;-)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From stephen at xemacs.org  Tue Sep  9 03:31:44 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 09 Sep 2008 10:31:44 +0900
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <loom.20080909T001934-453@post.gmane.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
	<48C5BF45.9020807@cheimes.de>
	<loom.20080909T001934-453@post.gmane.org>
Message-ID: <87iqt614vz.fsf@xemacs.org>

Antoine Pitrou writes:

 > It's not only the marketing. Having both releases in lock step means the
 > development process is synchronized between trunk and py3k, that there is no
 > loss of developer focus, and that merges/backports happen quite naturally.

As usual, in theory precision is infinite, but in engineering practice
it's fuzzy.  "Lock step" doesn't mean "as fine as you can split a
second"; for 2.6/3.0 a couple of weeks separation is not going to
matter.  The important thing is to get right back on schedule for
releasing 2.7/3.1 together (if that's the plan).

Split-second precision does matter for marketing, though.<wink>

From steve at holdenweb.com  Tue Sep  9 03:47:57 2008
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 08 Sep 2008 21:47:57 -0400
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48C5BD18.4030900@canterbury.ac.nz>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>
	<loom.20080902T224020-709@post.gmane.org>	<03b801c90d58$b5744970$205cdc50$@com.au>	<loom.20080903T002033-116@post.gmane.org>	<03d001c90d6b$fc7b2380$f5716a80$@com.au>	<loom.20080903T141921-651@post.gmane.org>	<48C4BDE4.3020605@v.loewis.de>
	<48C5BD18.4030900@canterbury.ac.nz>
Message-ID: <ga4kll$ktj$1@ger.gmane.org>

Greg Ewing wrote:
> Martin v. L?wis wrote:
> 
>> OTOH, other things *are* available, such as registered extensions.
>> For example, you don't need python on PATH to start a Python script;
>> just invoking the .py file will find the Python interpreter from the
>> registry.
> 
> But then you don't get to pass arguments to the program,
> get to see the output before the window disappears, etc.
> 
I believe you are confusing the command-line PATHEXT mechanism with the
Explorer/registry execution mechanism:

C:\Users\sholden\Documents\dyjr>python
'python' is not recognized as an internal or external command,
operable program or batch file.

C:\Users\sholden\Documents\dyjr>manage.py
Type 'manage.py help' for usage.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From python at rcn.com  Tue Sep  9 04:07:48 2008
From: python at rcn.com (Raymond Hettinger)
Date: Mon, 8 Sep 2008 19:07:48 -0700
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org><ga0ppm$udq$1@ger.gmane.org><9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org><ga1ck7$ldi$1@ger.gmane.org><D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org><ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com><1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
Message-ID: <3B14584AAB7147C6892032FE3FAE3755@RaymondLaptop1>

[Guido van Rossum]
> Well, from the number of release blockers it sounds like another 3.0
> beta is the right thing. For 2.6 however I believe we're much closer
> to the finish line -- there aren't all those bytes/str issues to clean
> up, for example! And apparently the benefit of releasing on schedule
> is that we will be included in OSX. That's a much bigger deal for 2.6
> than for 3.0 (I doubt that Apple would add two versions anyway).

With the extra time, it would be worthwhile to add dbm.sqlite to 3.0
to compensate for the loss of bsddb so that shelves won't become
useless on Windows builds.


Raymond




From guido at python.org  Tue Sep  9 04:11:02 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 8 Sep 2008 19:11:02 -0700
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <3B14584AAB7147C6892032FE3FAE3755@RaymondLaptop1>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
	<3B14584AAB7147C6892032FE3FAE3755@RaymondLaptop1>
Message-ID: <ca471dc20809081911h6811accdx5065b1244d84c6ac@mail.gmail.com>

On Mon, Sep 8, 2008 at 7:07 PM, Raymond Hettinger <python at rcn.com> wrote:
> [Guido van Rossum]
>>
>> Well, from the number of release blockers it sounds like another 3.0
>> beta is the right thing. For 2.6 however I believe we're much closer
>> to the finish line -- there aren't all those bytes/str issues to clean
>> up, for example! And apparently the benefit of releasing on schedule
>> is that we will be included in OSX. That's a much bigger deal for 2.6
>> than for 3.0 (I doubt that Apple would add two versions anyway).
>
> With the extra time, it would be worthwhile to add dbm.sqlite to 3.0
> to compensate for the loss of bsddb so that shelves won't become
> useless on Windows builds.

So get started already! :-)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From skip at pobox.com  Tue Sep  9 04:12:23 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 8 Sep 2008 21:12:23 -0500
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <3B14584AAB7147C6892032FE3FAE3755@RaymondLaptop1>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
	<3B14584AAB7147C6892032FE3FAE3755@RaymondLaptop1>
Message-ID: <18629.56199.90786.234922@montanaro-dyndns-org.local>


    Raymond> With the extra time, it would be worthwhile to add dbm.sqlite
    Raymond> to 3.0 to compensate for the loss of bsddb so that shelves
    Raymond> won't become useless on Windows builds.

My vote is to separate 2.6 and 3.0 then come back together for 2.7 and 3.1.
I'm a bit less sure about adding dbm.sqlite.  Unless Josiah's version is
substantially faster and more robust I think my version needs to cook a bit
longer.  I'm just not comfortable enough with SQLite to pronounce my version
fit enough.  I only intended it as a proof-of-concept, and it's clear it has
some shortcomings.

Skip

From martin at v.loewis.de  Tue Sep  9 07:43:18 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Tue, 09 Sep 2008 07:43:18 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
In-Reply-To: <48C5BD18.4030900@canterbury.ac.nz>
References: <684883d8-4703-4a3a-8005-c2997d128c83@t54g2000hsg.googlegroups.com>	<e27efe130809020438x40021445g5fc0a34d06c3d5e0@mail.gmail.com>	<94bdd2610809020604s5d75c064ka2b8ba0628f0699d@mail.gmail.com>	<g9kaav$hj2$1@ger.gmane.org>
	<loom.20080902T224020-709@post.gmane.org>	<03b801c90d58$b5744970$205cdc50$@com.au>	<loom.20080903T002033-116@post.gmane.org>	<03d001c90d6b$fc7b2380$f5716a80$@com.au>	<loom.20080903T141921-651@post.gmane.org>	<48C4BDE4.3020605@v.loewis.de>
	<48C5BD18.4030900@canterbury.ac.nz>
Message-ID: <48C60CF6.4090908@v.loewis.de>

> But then you don't get to pass arguments to the program,
> get to see the output before the window disappears, etc.

Did you actually try before posting?

Regards,
Martin


From g.brandl at gmx.net  Tue Sep  9 09:22:34 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 09 Sep 2008 09:22:34 +0200
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>	<ga1ck7$ldi$1@ger.gmane.org>	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
Message-ID: <ga587v$nvj$1@ger.gmane.org>

Guido van Rossum schrieb:

>>> Perhaps it's time to separate the 2.6 and 3.0 release schedules? I
>>> don't care if the next version of OSX contains 3.0 or not -- but I do
>>> care about it having 2.6.
>>
>> I'm not really sure what good that would do us unless we wanted to
>> bring 3.0 back to the beta phase and continue to work on some larger
>> issues with it. I also suspect doing two separate, but close together
>> final releases would be more stressful than having them in lock and
>> step.
> 
> Well, from the number of release blockers it sounds like another 3.0
> beta is the right thing. For 2.6 however I believe we're much closer
> to the finish line -- there aren't all those bytes/str issues to clean
> up, for example! And apparently the benefit of releasing on schedule
> is that we will be included in OSX. That's a much bigger deal for 2.6
> than for 3.0 (I doubt that Apple would add two versions anyway).

Even if I can't contribute very much at the moment, I'm still +1 to that.
I doubt Python would get nice publicity if we released a 3.0 but had to
tell everyone, "but don't really use it yet, it may still contain any
number of showstoppers."

>> Just my pocket change, though.

I'm suspecting this is much more than $0.02 :D

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From mal at egenix.com  Tue Sep  9 10:39:15 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 09 Sep 2008 10:39:15 +0200
Subject: [Python-Dev] [Python-checkins] r66321 - in
 python/trunk:	Doc/library/warnings.rst
 Lib/asynchat.py	Lib/bsddb/test/test_early_close.py
 Lib/mimetools.py	Lib/test/test___all__.py
 Lib/test/test_exceptions.py	Lib/test/test_hmac.py Lib/test/test_import.py
 Lib/test/test_macostools.py
 Lib/test/test_pep352.py	Lib/test/test_py3kwarn.py
 Lib/test/test_random.py	Lib/test/test_re.py
 Lib/test/test_struct.py	Lib/test/test_structmembers.py
 Lib/test/test_sundry.py	Lib/test/test_support.py Lib/test/test_symtable.py
 Lib/test/test_urllib.py
 Lib/test/test_urllibnet.py	Lib/test/test_userstring.py
 Lib/test/test_warnings.py	Lib/warnings.py Misc/NEWS
In-Reply-To: <20080909004917.22E5F1E4006@bag.python.org>
References: <20080909004917.22E5F1E4006@bag.python.org>
Message-ID: <48C63633.7060504@egenix.com>

On 2008-09-09 02:49, brett.cannon wrote:
> Author: brett.cannon
> Date: Tue Sep  9 02:49:16 2008
> New Revision: 66321
> 
> Log:
> warnings.catch_warnings() now returns a list or None instead of the custom
> WarningsRecorder object. This makes the API simpler to use as no special object
> must be learned.
> 
> Closes issue 3781.
> Review by Benjamin Peterson.

This sounds a lot like a feature.

Why was it added just before RC1 ?

> Modified:
>    python/trunk/Doc/library/warnings.rst
>    python/trunk/Lib/asynchat.py
>    python/trunk/Lib/bsddb/test/test_early_close.py
>    python/trunk/Lib/mimetools.py
>    python/trunk/Lib/test/test___all__.py
>    python/trunk/Lib/test/test_exceptions.py
>    python/trunk/Lib/test/test_hmac.py
>    python/trunk/Lib/test/test_import.py
>    python/trunk/Lib/test/test_macostools.py
>    python/trunk/Lib/test/test_pep352.py
>    python/trunk/Lib/test/test_py3kwarn.py
>    python/trunk/Lib/test/test_random.py
>    python/trunk/Lib/test/test_re.py
>    python/trunk/Lib/test/test_struct.py
>    python/trunk/Lib/test/test_structmembers.py
>    python/trunk/Lib/test/test_sundry.py
>    python/trunk/Lib/test/test_support.py
>    python/trunk/Lib/test/test_symtable.py
>    python/trunk/Lib/test/test_urllib.py
>    python/trunk/Lib/test/test_urllibnet.py
>    python/trunk/Lib/test/test_userstring.py
>    python/trunk/Lib/test/test_warnings.py
>    python/trunk/Lib/warnings.py
>    python/trunk/Misc/NEWS

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 09 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From ncoghlan at gmail.com  Tue Sep  9 12:14:56 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 09 Sep 2008 20:14:56 +1000
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <ca471dc20809081750n1bc10837r20ba917ef7221c41@mail.gmail.com>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>	<ga1ck7$ldi$1@ger.gmane.org>	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>	<48C5BF45.9020807@cheimes.de>	<loom.20080909T001934-453@post.gmane.org>
	<ca471dc20809081750n1bc10837r20ba917ef7221c41@mail.gmail.com>
Message-ID: <48C64CA0.30404@gmail.com>

Guido van Rossum wrote:
> Sure, we lose the ability to add last-minute -3 warnings. But I think
> that's a pretty minor issue (and those warnings have a tendency to
> subtly break things occasionally, so we shouldn't do them last-minute
> anyway).

Hey, if we catch all the things that need -3 warnings now, what are we
meant to add in 2.7? :)

+1 for a 2.6 rc and another 3.0 beta.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Tue Sep  9 12:20:29 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 09 Sep 2008 20:20:29 +1000
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <18629.56199.90786.234922@montanaro-dyndns-org.local>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>	<ga1ck7$ldi$1@ger.gmane.org>	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>	<3B14584AAB7147C6892032FE3FAE3755@RaymondLaptop1>
	<18629.56199.90786.234922@montanaro-dyndns-org.local>
Message-ID: <48C64DED.3090103@gmail.com>

skip at pobox.com wrote:
>     Raymond> With the extra time, it would be worthwhile to add dbm.sqlite
>     Raymond> to 3.0 to compensate for the loss of bsddb so that shelves
>     Raymond> won't become useless on Windows builds.
> 
> My vote is to separate 2.6 and 3.0 then come back together for 2.7 and 3.1.
> I'm a bit less sure about adding dbm.sqlite.  Unless Josiah's version is
> substantially faster and more robust I think my version needs to cook a bit
> longer.  I'm just not comfortable enough with SQLite to pronounce my version
> fit enough.  I only intended it as a proof-of-concept, and it's clear it has
> some shortcomings.

Given that the *API* is fixed though, it is probably better to have the
module present in 3.0 and bring it back to the main line in 2.7.

If any absolute clangers from a performance/stability point of view get
past Raymond (and everyone else with an interest in this) then they can
be addressed in 3.0.1 in a few months time. Whereas if we leave the
module out entirely, then 3.0 users are completely out of luck until 3.1
(or have to download and possibly build pybsddb).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Tue Sep  9 12:31:11 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 09 Sep 2008 20:31:11 +1000
Subject: [Python-Dev] [Python-checkins] r66321 - in
 python/trunk:	Doc/library/warnings.rst
 Lib/asynchat.py	Lib/bsddb/test/test_early_close.py
 Lib/mimetools.py	Lib/test/test___all__.py
 Lib/test/test_exceptions.py	Lib/test/test_hmac.py Lib/test/test_import.py
 Lib/test/test_macostools.py
 Lib/test/test_pep352.py	Lib/test/test_py3kwarn.py
 Lib/test/test_random.py	Lib/test/test_re.py
 Lib/test/test_struct.py	Lib/test/test_structmembers.py
 Lib/test/test_sundry.py	Lib/test/test_support.py Lib/test/test_symtable.py
 Lib/test/test_urllib.py
 Lib/test/test_urllibnet.py	Lib/test/test_userstring.py
 Lib/test/test_warnings.py	Lib/warnings.py Misc/NEWS
In-Reply-To: <48C63633.7060504@egenix.com>
References: <20080909004917.22E5F1E4006@bag.python.org>
	<48C63633.7060504@egenix.com>
Message-ID: <48C6506F.8010906@gmail.com>

M.-A. Lemburg wrote:
> On 2008-09-09 02:49, brett.cannon wrote:
>> Author: brett.cannon
>> Date: Tue Sep  9 02:49:16 2008
>> New Revision: 66321
>>
>> Log:
>> warnings.catch_warnings() now returns a list or None instead of the custom
>> WarningsRecorder object. This makes the API simpler to use as no special object
>> must be learned.
>>
>> Closes issue 3781.
>> Review by Benjamin Peterson.
> 
> This sounds a lot like a feature.
> 
> Why was it added just before RC1 ?

It's also a bug that was introduced by the late API changes made to
WarningsRecorder in r66135 (when WarningsRecorder was moved from
test.test_support to warnings to make it officially supported for use
outside the Python test suite, the API was also changed).

Instead of changing the API *again* could we please get the API reverted
back to the way it was for the two months prior to r66135 and only
change the location from test_support to warnings? (obviously, the use
of @contextmanager should still be removed since we don't particularly
want warnings to depend on contextlib).

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From barry at python.org  Tue Sep  9 14:26:36 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 9 Sep 2008 08:26:36 -0400
Subject: [Python-Dev] [Python-checkins] r66321 - in
	python/trunk:	Doc/library/warnings.rst
	Lib/asynchat.py	Lib/bsddb/test/test_early_close.py
	Lib/mimetools.py	Lib/test/test___all__.py
	Lib/test/test_exceptions.py	Lib/test/test_hmac.py
	Lib/test/test_import.py Lib/test/test_macostools.py
	Lib/test/test_pep352.py	Lib/test/test_py3kwarn.py
	Lib/test/test_random.py	Lib/test/test_re.py
	Lib/test/test_struct.py	Lib/test/test_structmembers.py
	Lib/test/test_sundry.py	Lib/test/test_support.py
	Lib/test/test_symtable.py Lib/test/test_urllib.py
	Lib/test/test_urllibnet.py	Lib/test/test_userstring.py
	Lib/test/test_warnings.py	Lib/warnings.py Misc/NEWS
In-Reply-To: <48C6506F.8010906@gmail.com>
References: <20080909004917.22E5F1E4006@bag.python.org>
	<48C63633.7060504@egenix.com> <48C6506F.8010906@gmail.com>
Message-ID: <FD96209A-81BB-49C6-8F12-F925621CCA31@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 9, 2008, at 6:31 AM, Nick Coghlan wrote:

> It's also a bug that was introduced by the late API changes made to
> WarningsRecorder in r66135 (when WarningsRecorder was moved from
> test.test_support to warnings to make it officially supported for use
> outside the Python test suite, the API was also changed).
>
> Instead of changing the API *again* could we please get the API  
> reverted
> back to the way it was for the two months prior to r66135 and only
> change the location from test_support to warnings? (obviously, the use
> of @contextmanager should still be removed since we don't particularly
> want warnings to depend on contextlib).

As I commented in the bug, if this function is added to warnings I  
think it will be increasingly used as a way for code to suppress  
warnings.  If that's the behavior we want to allow, then I think  
making every warning but the last inaccessible is a bug.

If that's not a behavior we want to encourage then naming this  
function 'catch_warnings' and putting it in the warnings module sends  
the wrong message.  Better to stick it in unittest if test_support is  
unacceptable.  Or at least name it something that doesn't scream "use  
me in a way I don't intend!".

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMZrfHEjvBPtnXfVAQI6DgP8DyNKZyR846u1Qpgbr4X3wOM1J6EcqSlW
74WGUoU80MDWH5wvagAl7Er90oEh0QhX4ogX8UgF8iLSOC7jwa4CgpZsd9a5V8WB
8M0ia/G457vjGYg8XsaD5Dqvuv6NgMcW/2KJCJK3cuzEafan5C+I6n6lKQMAGOu0
j/9/PtIOKmI=
=btg/
-----END PGP SIGNATURE-----

From jnoller at gmail.com  Tue Sep  9 14:49:20 2008
From: jnoller at gmail.com (Jesse Noller)
Date: Tue, 9 Sep 2008 08:49:20 -0400
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
Message-ID: <DF2D22CB-BDF9-45BD-8928-0561CD4C5A3D@gmail.com>



On Sep 8, 2008, at 1:13 PM, "Guido van Rossum" <guido at python.org> wrote:

> On Mon, Sep 8, 2008 at 6:23 AM, Barry Warsaw <barry at python.org> wrote:
>> I don't think there's any way we're going to make our October 1st  
>> goal.  We
>> have 8 open release critical bugs, and 18 deferred blockers.  We do  
>> not have
>> a beta3 Windows installer and I don't have high hopes for  
>> rectifying all of
>> these problems in the next day or two.
>>
>> I propose that we push the entire schedule back two weeks.  This  
>> means that
>> the planned rc2 on 17-September becomes our rc1.  The planned final  
>> release
>> for 01-October becomes our rc2, and we release the finals on 15- 
>> October.
>>
>> - -Barry
>
> Perhaps it's time to separate the 2.6 and 3.0 release schedules? I
> don't care if the next version of OSX contains 3.0 or not -- but I do
> care about it having 2.6.
>

Given that 2.6 is going to be more widely adopted and used by both the  
community and OS distributors, I'm +1 on splitting the releases as well.

-Jesse 

From barry at python.org  Tue Sep  9 15:17:10 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 9 Sep 2008 09:17:10 -0400
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
Message-ID: <937C6D77-3168-4127-8D4F-59AA291F0A86@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 8, 2008, at 1:13 PM, Guido van Rossum wrote:

> Perhaps it's time to separate the 2.6 and 3.0 release schedules? I
> don't care if the next version of OSX contains 3.0 or not -- but I do
> care about it having 2.6.

I've talked with my contact at MajorOS Vendor (tm) and, as much as he  
can say, he would be fine with this.  They're having problems getting  
3rd party modules to build against 3.0 anyway, but if we can release a  
very solid 2.6 by the 1-Oct deadline, I would support splitting the  
releases.

I really don't like doing this, but if we can get 2.6 out on time, and  
3.0 doesn't lag too far behind, I'm okay with it.  We'll have to  
abbreviate the release schedule though, so everyone should concentrate  
on fixing the 2.6 showstoppers.  I think we need to get 2.6rc1 out  
this week, followed by 2.6rc2 next Wednesday as planned and 2.6final  
on 1-October.

I've shuffled the tracker to reduce all 3.0-only bugs to deferred  
blocker, and to increase all 2.6 deferred blockers to release  
blockers.  There are 11 open blocker issues for 2.6:

3629 Python won't compile a regex that compiles with 2.5.2 and 30b2
3640 test_cpickle crash on AMD64 Windows build
3777 long(4.2) now returns an int
3781 warnings.catch_warnings fails gracelessly when recording warnings  
but...
2876 Write UserDict fixer for 2to3
2350 'exceptions' import fixer
3642 Objects/obmalloc.c:529: warning: comparison is always false due...
3617 Add MS EULA to the list of third-party licenses in the Windows...
3657 pickle can pickle the wrong function
1868 threading.local doesn't free attrs when assigning thread exits
3809 test_logging leaving a 'test.blah' file behind

If we can close them by Wednesday or Thursday, and the 2.6 bots stay  
green, I will cut the 2.6rc1 release this week and the 2.6rc2 and  
final on schedule.

If you're on board with this, please do what you can to resolve these  
open issues.  As always, I'm on irc if you need to discuss anything.

Cheers,
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMZ3V3EjvBPtnXfVAQKLbAP6A9b0WBB0H/ONZbKie2TazK/qYLthYnZQ
iIpfJ2UboOA7dJ/ueXIsD413oI8GTbUOsUlJOWbSzAfJ6oBuPHrjr4IFRCZhchKG
lwViDaK/7aWgIusGFpt6y/SgwJBU531wb7o3Lx/P6rLx5Wh5Nr+tvhngt0WkSMSj
WtCsy3mmgmQ=
=3HdI
-----END PGP SIGNATURE-----

From barry at python.org  Tue Sep  9 15:21:53 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 9 Sep 2008 09:21:53 -0400
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
Message-ID: <914B35A3-C8C2-42B6-9A3B-11E1F0F03998@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 8, 2008, at 7:25 PM, Guido van Rossum wrote:

> Well, from the number of release blockers it sounds like another 3.0
> beta is the right thing. For 2.6 however I believe we're much closer
> to the finish line -- there aren't all those bytes/str issues to clean
> up, for example! And apparently the benefit of releasing on schedule
> is that we will be included in OSX. That's a much bigger deal for 2.6
> than for 3.0 (I doubt that Apple would add two versions anyway).

The MajorOS Vendor (tm) may be willing to ship a 3.0 beta if it's far  
enough along, though not as the primary Python version.  They clearly  
want 2.6 for that.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMZ4cXEjvBPtnXfVAQL4ygP/fLILvf3NhvmN3R2T7htGm08xt/bOBYGt
+BDrV4rapS4j3jo2Cx+McEdjJZCdq9x7BIaTN+4ITwq02LEY5fmhp6NkhzE1dlnq
qdgBq8x/Z4AnsxfydtqYrPhrzLWPpdEZElgll5FB6Dj6XIA7cB8tuds2cE7+OXJI
Guom1Y0k6Ao=
=u4FB
-----END PGP SIGNATURE-----

From barry at python.org  Tue Sep  9 15:23:28 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 9 Sep 2008 09:23:28 -0400
Subject: [Python-Dev] [Python-3000]   Proposed revised schedule
In-Reply-To: <3B14584AAB7147C6892032FE3FAE3755@RaymondLaptop1>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org><ga0ppm$udq$1@ger.gmane.org><9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org><ga1ck7$ldi$1@ger.gmane.org><D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org><ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com><1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
	<3B14584AAB7147C6892032FE3FAE3755@RaymondLaptop1>
Message-ID: <3DFD4AAC-D8EA-46E6-BC56-C713861C02B7@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 8, 2008, at 10:07 PM, Raymond Hettinger wrote:

> [Guido van Rossum]
>> Well, from the number of release blockers it sounds like another 3.0
>> beta is the right thing. For 2.6 however I believe we're much closer
>> to the finish line -- there aren't all those bytes/str issues to  
>> clean
>> up, for example! And apparently the benefit of releasing on schedule
>> is that we will be included in OSX. That's a much bigger deal for 2.6
>> than for 3.0 (I doubt that Apple would add two versions anyway).
>
> With the extra time, it would be worthwhile to add dbm.sqlite to 3.0
> to compensate for the loss of bsddb so that shelves won't become
> useless on Windows builds.

That seems risky to me.  First, it's a new feature.  Second, it will  
be largely untested code.  I would much rather see dbm.sqlite released  
as a separate package for possible integration into the core for 3.1.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMZ40XEjvBPtnXfVAQK2WQP/e3N2rYD2rbsoynEnXvAjzF8lPoPRFDvl
hbjERsbB93uSoBPHaTdjtXnW+InC0W4GC5ogHF9wARbzYTJaxx09WmjihX+PvgsW
JhXwLpG3gtyclfqSAF8MWZHc4UnKnyUt5UgYBlZrzT0z7FhWmelUPl8QhS8/2n9L
oT3qX8eLabI=
=Zu70
-----END PGP SIGNATURE-----

From barry at python.org  Tue Sep  9 15:25:03 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 9 Sep 2008 09:25:03 -0400
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <ga587v$nvj$1@ger.gmane.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>	<ga1ck7$ldi$1@ger.gmane.org>	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>
	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
	<ga587v$nvj$1@ger.gmane.org>
Message-ID: <BDEAED28-1FC6-4C58-B7BF-947326D49CBA@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 9, 2008, at 3:22 AM, Georg Brandl wrote:

> Even if I can't contribute very much at the moment, I'm still +1 to  
> that.
> I doubt Python would get nice publicity if we released a 3.0 but had  
> to
> tell everyone, "but don't really use it yet, it may still contain any
> number of showstoppers."

I completely agree.  We should not release anything that's not ready.   
Assuming that we all agree that 2.6 is much closer to being ready,  
that gives us two options: delay 2.6 to coincide with 3.0 or split the  
releases.  The latter seems like the wisest choice to meet our goals.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMZ5L3EjvBPtnXfVAQJwSQP/U7FFFI8ao5Xesf6F3QFIUMYFeISrlhof
9ynkQXAskUMelAfayGMSd2nD2+buXA7gyBWplAAEF2rtLhZ3N0+zeh/2HnqcY0b9
EtUM5shAIMlb2948IMoXlxSMplH5auBHMLYFnuPAIH9ERXsGVfyihLnUarAfzmT+
XrWfjrU62TA=
=CUR4
-----END PGP SIGNATURE-----

From skip at pobox.com  Tue Sep  9 15:53:11 2008
From: skip at pobox.com (skip at pobox.com)
Date: Tue, 9 Sep 2008 08:53:11 -0500
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <937C6D77-3168-4127-8D4F-59AA291F0A86@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>
	<ga0ppm$udq$1@ger.gmane.org>
	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>
	<ga1ck7$ldi$1@ger.gmane.org>
	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>
	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
	<937C6D77-3168-4127-8D4F-59AA291F0A86@python.org>
Message-ID: <18630.32711.828670.45997@montanaro-dyndns-org.local>

    Barry> 3777 long(4.2) now returns an int

Looks like Amaury has already taken care of this one.

Skip

From ncoghlan at gmail.com  Tue Sep  9 16:10:18 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 Sep 2008 00:10:18 +1000
Subject: [Python-Dev] [Python-checkins] r66321 - in
 python/trunk:	Doc/library/warnings.rst
 Lib/asynchat.py	Lib/bsddb/test/test_early_close.py
 Lib/mimetools.py	Lib/test/test___all__.py
 Lib/test/test_exceptions.py	Lib/test/test_hmac.py Lib/test/test_import.py
 Lib/test/test_macostools.py
 Lib/test/test_pep352.py	Lib/test/test_py3kwarn.py
 Lib/test/test_random.py	Lib/test/test_re.py
 Lib/test/test_struct.py	Lib/test/test_structmembers.py
 Lib/test/test_sundry.py	Lib/test/test_support.py Lib/test/test_symtable.py
 Lib/test/test_urllib.py
 Lib/test/test_urllibnet.py	Lib/test/test_userstring.py
 Lib/test/test_warnings.py	Lib/warnings.py Misc/NEWS
In-Reply-To: <FD96209A-81BB-49C6-8F12-F925621CCA31@python.org>
References: <20080909004917.22E5F1E4006@bag.python.org>
	<48C63633.7060504@egenix.com> <48C6506F.8010906@gmail.com>
	<FD96209A-81BB-49C6-8F12-F925621CCA31@python.org>
Message-ID: <48C683CA.6040302@gmail.com>

Barry Warsaw wrote:
> On Sep 9, 2008, at 6:31 AM, Nick Coghlan wrote:
> 
>> It's also a bug that was introduced by the late API changes made to
>> WarningsRecorder in r66135 (when WarningsRecorder was moved from
>> test.test_support to warnings to make it officially supported for use
>> outside the Python test suite, the API was also changed).
> 
>> Instead of changing the API *again* could we please get the API reverted
>> back to the way it was for the two months prior to r66135 and only
>> change the location from test_support to warnings? (obviously, the use
>> of @contextmanager should still be removed since we don't particularly
>> want warnings to depend on contextlib).
> 
> As I commented in the bug, if this function is added to warnings I think
> it will be increasingly used as a way for code to suppress warnings.  If
> that's the behavior we want to allow, then I think making every warning
> but the last inaccessible is a bug.
> 
> If that's not a behavior we want to encourage then naming this function
> 'catch_warnings' and putting it in the warnings module sends the wrong
> message.  Better to stick it in unittest if test_support is
> unacceptable.  Or at least name it something that doesn't scream "use me
> in a way I don't intend!".

The last version of test_support.catch_warnings() actually did make all
warnings caught available via the w.warnings attribute. The API and
implementation changes in the move from test_support to the warnings
module didn't actually add any new features - they just changed the
w.warnings[idx] spelling to w[idx], moved from caching an explicit list
of attributes on the WarningRecorder to a __getattr__ based lookup on
the last warning and from a generator-based context manager to a custom
class with __enter__ and __exit__ methods. The last was actually a good
idea due to interpreter startup implications, but the first two changes
were unnecessary (and the attribute handling change is  actually what
led to issue3781).

Regardless, my preference at this point is for us to revert back to only
providing this feature through the regression test suite, since that
implementation at least worked properly, and reconsider the matter of
proving a more public version through the warnings module for 2.7/3.1.

Yes, it will make life a little more difficult for the twisted folks if
they can't rely on our test support module being present for whatever
reason, but they do have the option of including their own version of
catch_warnings() as a fallback if the regression test suite can't be found.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Tue Sep  9 16:17:27 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 Sep 2008 00:17:27 +1000
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <914B35A3-C8C2-42B6-9A3B-11E1F0F03998@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>	<ga1ck7$ldi$1@ger.gmane.org>	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>	<1afaf6160809081613w266930efx75e9a6f5886b98e5@mail.gmail.com>	<ca471dc20809081625s16f5c6f4m243375298748b02d@mail.gmail.com>
	<914B35A3-C8C2-42B6-9A3B-11E1F0F03998@python.org>
Message-ID: <48C68577.6070705@gmail.com>

Barry Warsaw wrote:
> On Sep 8, 2008, at 7:25 PM, Guido van Rossum wrote:
> 
>> Well, from the number of release blockers it sounds like another 3.0
>> beta is the right thing. For 2.6 however I believe we're much closer
>> to the finish line -- there aren't all those bytes/str issues to clean
>> up, for example! And apparently the benefit of releasing on schedule
>> is that we will be included in OSX. That's a much bigger deal for 2.6
>> than for 3.0 (I doubt that Apple would add two versions anyway).
> 
> The MajorOS Vendor (tm) may be willing to ship a 3.0 beta if it's far
> enough along, though not as the primary Python version.  They clearly
> want 2.6 for that.

Given that the sum total of actual Python 3.0 programs is currently
pretty close to zero, I don't really see any reason for *any* OS vendor
(even Linux distros) to be including a 3.0 interpreter in their base
install at this point in time. I personally expect it to stay in the
"optional extras" category until some time next year.

Pessimists-have-more-opportunities-to-be-pleasantly-surprised'ly,
Nick.

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Tue Sep  9 16:21:14 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 Sep 2008 00:21:14 +1000
Subject: [Python-Dev] [Python-3000] Proposed revised schedule
In-Reply-To: <937C6D77-3168-4127-8D4F-59AA291F0A86@python.org>
References: <E4008E05-32A8-4D09-B56C-B324DF468434@python.org>	<ga0ppm$udq$1@ger.gmane.org>	<9ADBEA78-8FB6-4C92-8140-CF74643A76B3@python.org>	<ga1ck7$ldi$1@ger.gmane.org>	<D1F8B586-F45C-4B8C-9887-612A6A794A45@python.org>	<ca471dc20809081013s63ed2bc2ja10e0fa75059642f@mail.gmail.com>
	<937C6D77-3168-4127-8D4F-59AA291F0A86@python.org>
Message-ID: <48C6865A.5000703@gmail.com>

Barry Warsaw wrote:
> 3781 warnings.catch_warnings fails gracelessly when recording warnings

I just assigned this one to myself - I'll have a patch up for review
shortly (the patch will revert back to having this be a regression test
suite only feature).

Cheers,
Nick.

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From list8a.forest at tibit.com  Tue Sep  9 20:44:59 2008
From: list8a.forest at tibit.com (Forest)
Date: Tue, 9 Sep 2008 11:44:59 -0700 (PDT)
Subject: [Python-Dev] new ssl module is incompatible with servers that drop
	privileges
Message-ID: <12180.75.55.199.5.1220985899.squirrel@webmail.sonic.net>

I've been trying out the new ssl module, and I love it so far, except for
the way it accepts private keys and certificates.  It accept them only as
paths to their location on the file system, which I believe means that a
server can only support SSL if it has read permission to its private key
file when client connections arrive.  This is a problem for servers that
bind to their socket and drop privileges as soon as they start up, a
practice that is both common and recommended in the unix world.

IMHO, this severely limits the new ssl module's utility, and discourages
good security practices.

Wouldn't it be better if we could specify keys and certificates as bytes
or file-like objects?  This would solve the security issue, give
applications more flexibility in key management, and might also improve
performance slightly (by avoiding file system operations at accept()
time).

Perhaps there's a workaround that I haven't noticed yet?  A quick look at
the source code didn't reveal any obvious way to specify keys other than
as paths in the file system.

http://bugs.python.org/issue3823



From janssen at parc.com  Tue Sep  9 21:49:27 2008
From: janssen at parc.com (Bill Janssen)
Date: Tue, 9 Sep 2008 12:49:27 PDT
Subject: [Python-Dev] new ssl module is incompatible with servers that
	drop privileges
In-Reply-To: <12180.75.55.199.5.1220985899.squirrel@webmail.sonic.net> 
References: <12180.75.55.199.5.1220985899.squirrel@webmail.sonic.net>
Message-ID: <08Sep9.124937pdt."58698"@synergy1.parc.xerox.com>

> It accept them only as
> paths to their location on the file system, which I believe means that a
> server can only support SSL if it has read permission to its private key
> file when client connections arrive.  This is a problem for servers that
> bind to their socket and drop privileges as soon as they start up, a
> practice that is both common and recommended in the unix world.

Ah, excellent point.

> IMHO, this severely limits the new ssl module's utility, and discourages
> good security practices.

Please file a bug report.  A bug report with a patch and tests would
be even better :-).  Assign it to me.

> Wouldn't it be better if we could specify keys and certificates as bytes
> or file-like objects?  This would solve the security issue, give
> applications more flexibility in key management, and might also improve
> performance slightly (by avoiding file system operations at accept()
> time).

I like it!

Bill

From list8a.forest at tibit.com  Tue Sep  9 22:28:18 2008
From: list8a.forest at tibit.com (Forest)
Date: Tue, 9 Sep 2008 13:28:18 -0700 (PDT)
Subject: [Python-Dev] new ssl module is incompatible with servers that
 drop privileges
In-Reply-To: <08Sep9.124937pdt."58698"@synergy1.parc.xerox.com>
References: <12180.75.55.199.5.1220985899.squirrel@webmail.sonic.net>
	<08Sep9.124937pdt."58698"@synergy1.parc.xerox.com>
Message-ID: <14176.75.55.199.5.1220992098.squirrel@webmail.sonic.net>

On Tue, September 9, 2008 12:49 pm, Bill Janssen wrote:

>> IMHO, this severely limits the new ssl module's utility, and discourages
>> good security practices.
>
> Please file a bug report.  A bug report with a patch and tests would
> be even better :-).  Assign it to me.

I filed one, but the bug tracker doesn't seem to offer a way to assign it
to you.  I'll add you to the nosy list.

http://bugs.python.org/issue3823

I'm pretty swamped right now, so I don't think I can learn the code well
enough to make a patch in the few weeks before python 2.6 is released. 
(How nice it would be if the debut of this very useful module was free of
this problem!)  If I find some unexpected free time, I'll take a crack at
it.


From nnorwitz at gmail.com  Wed Sep 10 08:01:37 2008
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 9 Sep 2008 23:01:37 -0700
Subject: [Python-Dev] ?spurious? Timeout in BSDDB under MS Windows
In-Reply-To: <20080908102403.GE63632@wind.teleri.net>
References: <48C15661.9030901@jcea.es> <20080908102403.GE63632@wind.teleri.net>
Message-ID: <ee2a432c0809092301n54edfd56y745a66d60f17f9a9@mail.gmail.com>

On Mon, Sep 8, 2008 at 3:24 AM, Trent Nelson
<python-dev-list at trentnelson.com> wrote:
> On Fri, Sep 05, 2008 at 05:55:13PM +0200, Jesus Cea wrote:
>> Trent, are you available to look at the ?spurious? timeout failures in
>> bsddb replication code in the Windows buildbot?.
>>
>> Ten seconds timeout should be plenty enough. I can't debug any MS
>> Windows issue myself; this is a Microsoft-free environment.
>
>    I think I added in 10 seconds 'cause the tests kept failing when it
>    was at 2 seconds ;-)
>
>    I remember digging around the code a bit when I bumped bsddb to 4.7
>    on Windows to try and figure out what was going on.  As far as I
>    could tell it wasn't anything obvious caused by the Python code; is
>    it possible this could be an issue with the underlying bsddb code?

Are the buildbot(s) running under vmware?

n

From gh at ghaering.de  Thu Sep 11 11:40:59 2008
From: gh at ghaering.de (=?ISO-8859-1?Q?Gerhard_H=E4ring?=)
Date: Thu, 11 Sep 2008 11:40:59 +0200
Subject: [Python-Dev] Python 3.0: can we make dbm's .keys() return an
	iterator?
Message-ID: <48C8E7AB.6060207@ghaering.de>

As far as I can see, the specification of the dbm interface is the 
module docstring in dbm/__init__.py, which reads:

"""
[...]
It has the following interface (key and data are strings):

         d[key] = data   # store data at key (may override data at
                         # existing key)
         data = d[key]   # retrieve data at key (raise KeyError if no
                         # such key)
         del d[key]      # delete data stored at key (raises KeyError
                         # if no such key)
         flag = key in d # true if the key exists
         list = d.keys() # return a list of all existing keys (slow!)
"""

Now I thought that in Python 3.0, keys(), values() and friends should 
return iterators. Can we change at least the specification of the dbm 
module? We could then later in 3.1 change the implementations to return 
iterators instead of lists, too.

I stumbled upon it cos I'm trying to help Skip with the SQLite-based 
implementation.

-- Gerhard

From olemis at gmail.com  Thu Sep 11 17:34:41 2008
From: olemis at gmail.com (Olemis Lang)
Date: Thu, 11 Sep 2008 11:34:41 -0400
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <48C3ED2E.4060603@v.loewis.de>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>
	<48C3ED2E.4060603@v.loewis.de>
Message-ID: <24ea26600809110834t28ec1dfdh888390aa34c6a160@mail.gmail.com>

2008/9/7, "Martin v. L?wis" <martin at v.loewis.de>:
> Not necessarily - as you say, it's undocumented (and will remain so);
>  in any case, I have now granted anonymous read access to that
>  repository, through https.
>

Thnx a lot... Formerly  I could not access anything because of the
aforementioned authentication issue... Now public access is ok but...

I have recently tried to check out the latest versions of the PEPs and
I still can't do it. Something like this is what happens...

--------------------------------

$ svn co https://svn.python.org/projects/peps/trunk/ .

A    pep-0100.txt
A    pep-0102.txt

... # Many other files "added"

A    pep-0297.txt
A    pep-0299.txt
 U   .

Fetching external item into 'docutils'
svn: Can't connect to host 'svn.berlios.de': A connection attempt
failed because the connected party did not properly respond after a
period of time, or established connection failed because connected
host has failed to respond.

----------------------------------

Please I need some help so as to finally check out all the PEPs at once.

Is this a consequence of my systems settings (SVN, etc...)? Otherwise,
why is this "lovely" message shown? Please... how can I overcome this
situation?

Thnx.

-- 
Regards,

Olemis.

From fredrik at pythonware.com  Thu Sep 11 19:48:00 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Thu, 11 Sep 2008 19:48:00 +0200
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <24ea26600809110834t28ec1dfdh888390aa34c6a160@mail.gmail.com>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>	<48C3ED2E.4060603@v.loewis.de>
	<24ea26600809110834t28ec1dfdh888390aa34c6a160@mail.gmail.com>
Message-ID: <gablki$i9p$1@ger.gmane.org>

Olemis Lang wrote:

> Fetching external item into 'docutils'
> svn: Can't connect to host 'svn.berlios.de': A connection attempt
> failed because the connected party did not properly respond after a
> period of time, or established connection failed because connected
> host has failed to respond.

 > Please I need some help so as to finally check out all the PEPs at
 > once.

I suspect you already have all the PEP:s; the checkout command gets 
stuck when trying to fetch some additional software from an external
server.

try adding the --ignore-externals option to the checkout command, and 
see if this gets you any further.

then check if you can reach http://svn.berlios.de via your browser, or 
if some firewall rule gets in the way.

</F>


From martin at v.loewis.de  Thu Sep 11 23:39:46 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 11 Sep 2008 23:39:46 +0200
Subject: [Python-Dev] HTTPS read-only SVN access is denied?
In-Reply-To: <gablki$i9p$1@ger.gmane.org>
References: <d34314100809030743n3e4ddacambf7f612587634bef@mail.gmail.com>	<48C3ED2E.4060603@v.loewis.de>	<24ea26600809110834t28ec1dfdh888390aa34c6a160@mail.gmail.com>
	<gablki$i9p$1@ger.gmane.org>
Message-ID: <48C99022.1050605@v.loewis.de>


> then check if you can reach http://svn.berlios.de via your browser, or
> if some firewall rule gets in the way.

He probably can, but the firewall still gets in the way when he tries
to do the svn checkout - his firewall is incapable of forwarding OPTIONS
and other methods used by subversion. Hence he needs to use https, as
the firewall then puts through all traffic unmodified, thanks to the
CONNECT method.

So there will be no chance that he can checkout the berlios externals
directly, as their URL is defined in the Python repository (and will
remain at http, for the sake of most of us with sane networking
environments)

Regards,
Martin

From josiah.carlson at gmail.com  Fri Sep 12 07:44:57 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Thu, 11 Sep 2008 22:44:57 -0700
Subject: [Python-Dev] Python 3.0: can we make dbm's .keys() return an
	iterator?
In-Reply-To: <48C8E7AB.6060207@ghaering.de>
References: <48C8E7AB.6060207@ghaering.de>
Message-ID: <e6511dbf0809112244t2a521eaap8c38ce93c1a91acb@mail.gmail.com>

On Thu, Sep 11, 2008 at 2:40 AM, Gerhard H?ring <gh at ghaering.de> wrote:
> As far as I can see, the specification of the dbm interface is the module
> docstring in dbm/__init__.py, which reads:
>
> """
> [...]
> It has the following interface (key and data are strings):
>
>        d[key] = data   # store data at key (may override data at
>                        # existing key)
>        data = d[key]   # retrieve data at key (raise KeyError if no
>                        # such key)
>        del d[key]      # delete data stored at key (raises KeyError
>                        # if no such key)
>        flag = key in d # true if the key exists
>        list = d.keys() # return a list of all existing keys (slow!)
> """
>
> Now I thought that in Python 3.0, keys(), values() and friends should return
> iterators. Can we change at least the specification of the dbm module? We
> could then later in 3.1 change the implementations to return iterators
> instead of lists, too.
>
> I stumbled upon it cos I'm trying to help Skip with the SQLite-based
> implementation.

If it's to be following a variant of the dictionary API, the object
should return a view object:
http://docs.python.org/dev/3.0/library/stdtypes.html#dict .

I've updated my version of the library in the tracker, which includes
generic Keys, Values, and Items classes that implement the view
interface.

 - Josiah

From barry at python.org  Fri Sep 12 14:54:20 2008
From: barry at python.org (Barry Warsaw)
Date: Fri, 12 Sep 2008 08:54:20 -0400
Subject: [Python-Dev] Updated release schedule for 2.6 and 3.0
Message-ID: <93DF7138-3E99-44E3-AF7D-01D89D13E910@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

We had a lot of discussion recently about changing the release  
schedule and splitting Python 2.6 and 3.0.  There was general  
consensus that this was a good idea, in order to hit our October 1  
deadline for Python 2.6 final at least.

There is only one open blocker for 2.6, issue 3657.  Andrew, Fred, Tim  
and I (via IRC) will be getting together tonight to do some Python  
hacking, so we should resolve this issue and release 2.6rc1 tonight.

We'll have an abbreviated 2.6rc1, and I will release 2.6rc2 next  
Wednesday the 17th as planned. The final planned release of 2.6 will  
be Wednesday October 1st.

If 3.0 is looking better, I will release 3.0rc1 on Wednesday,  
otherwise we'll re-evaluate the release schedule for 3.0 as  
necessary.  This means currently the schedule looks like this:

Fri 12-Sep 2.6rc1
Wed 17-Sep 2.6rc2, 3.0rc1
Wed 01-Oct 2.6 final, 3.0rc2
Wed 15-Oct 3.0 final

I've updated the Python Release Schedule gcal and will update the PEP  
momentarily.  We'll close the tree later tonight (UTC-4).

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMpmfHEjvBPtnXfVAQJUiQP/eTXStyp9M0+Ja7iAFfYcpzfM19j9ddBr
ocMC+KTDSgci5/rmFw4KdMhqO9TBk2sXIdCd9GInnuMOKtndCKZ/PXaexnVvSVGb
P3CpkkMs/vG1itQIc/EXq6CUhzwuxEv9h8Wo8+zcmL05Cc1IrE5d2OYiO0+KQ8ei
lW+j/aNKMWY=
=w2oI
-----END PGP SIGNATURE-----

From barry at python.org  Fri Sep 12 15:20:46 2008
From: barry at python.org (Barry Warsaw)
Date: Fri, 12 Sep 2008 09:20:46 -0400
Subject: [Python-Dev] [Python-3000] Updated release schedule for 2.6 and
	3.0
In-Reply-To: <ffb592890809120619ne821addj108a22e7163afab6@mail.gmail.com>
References: <93DF7138-3E99-44E3-AF7D-01D89D13E910@python.org>
	<ffb592890809120619ne821addj108a22e7163afab6@mail.gmail.com>
Message-ID: <364A23D0-DC52-4C1D-B853-1D8A30C6C928@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 12, 2008, at 9:19 AM, Edward K. Ream wrote:

> On Fri, Sep 12, 2008 at 7:54 AM, Barry Warsaw <barry at python.org>  
> wrote:
>
>> We had a lot of discussion recently about changing the release  
>> schedule and
>> splitting Python 2.6 and 3.0.  There was general consensus that  
>> this was a
>> good idea, in order to hit our October 1 deadline for Python 2.6  
>> final at
>> least.
>
> Does it matter to anyone besides the you, the Python developers,
> whether the schedule slips by two weeks, or two months, for that
> matter?

For Python 3.0?  No.

> I am underwhelmed by 3.0 b3: sax and 2to3 are/were broken.  A b4
> release for 3.0 (at least) would seem more prudent.

We will release no Python before its time.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMpsr3EjvBPtnXfVAQKCUQP/WYfGeubbkWjmwI9mlQ4dMbVjGk15imAJ
ArIBs4sH9tbZTE12uNhjNgvXRbN+1QfejNeWOEJEAnPdErPAT0TKAmgA2Rj1MmjP
ook5+MbxkgkNnKbz8lozMPduclc7Djf22CYboAqiskK7G6LfD1fsCrIMEVSku/HX
dQpXGkG/C4g=
=FYgJ
-----END PGP SIGNATURE-----

From status at bugs.python.org  Fri Sep 12 18:06:35 2008
From: status at bugs.python.org (Python tracker)
Date: Fri, 12 Sep 2008 18:06:35 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20080912160635.7A272785CB@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (09/05/08 - 09/12/08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2019 open (+32) / 13652 closed (+30) / 15671 total (+62)

Open issues with patches:   637

Average duration of open issues: 716 days.
Median duration of open issues: 1772 days.

Open Issues Breakdown
   open  2005 (+31)
pending    14 ( +1)

Issues Created Or Reopened (67)
_______________________________

readline configuration for shared libs w/o curses dependencies   09/07/08
CLOSED http://bugs.python.org/issue1204    reopened gregory.p.smith           
       patch, 64bit                                                            

py3k fails under Windows if "-c" or "-m" is given a non-ascii va 09/09/08
       http://bugs.python.org/issue3705    reopened amaury.forgeotdarc        
       patch                                                                   

asyncore differences between 2.x and 3.x                         09/07/08
CLOSED http://bugs.python.org/issue3764    reopened gregory.p.smith           
       easy                                                                    

tkinter Menu.delete bug                                          09/12/08
       http://bugs.python.org/issue3774    reopened gpolo                     
       patch, easy, needs review                                               

warnings.catch_warnings fails gracelessly when recording warning 09/09/08
       http://bugs.python.org/issue3781    reopened ncoghlan                  
       patch                                                                   

Make PyInstanceMethod_Type available or remove it                09/05/08
       http://bugs.python.org/issue3787    created  christian.heimes          
       patch, patch, needs review                                              

test_cookie isn't comprehensive                                  09/05/08
       http://bugs.python.org/issue3788    created  benjamin.peterson         
       easy                                                                    

multiprocessing deadlocks when sending large data through Queue  09/05/08
CLOSED http://bugs.python.org/issue3789    created  DavidDecotigny            
                                                                               

in zlib decompressor objects, unused_data and unconsumed_tail mu 09/06/08
CLOSED http://bugs.python.org/issue3790    created  pitrou                    
       patch, patch, needs review                                              

bsddb not completely removed                                     09/06/08
       http://bugs.python.org/issue3791    created  amaury.forgeotdarc        
       patch, patch                                                            

Module variable overridden in child processes with multiprocessi 09/06/08
CLOSED http://bugs.python.org/issue3792    created  TarkaSteve                
                                                                               

Small RST fix in datamodel.rst                                   09/06/08
CLOSED http://bugs.python.org/issue3793    created  mgiuca                    
                                                                               

__div__ still documented in Python 3                             09/06/08
CLOSED http://bugs.python.org/issue3794    created  mgiuca                    
       patch                                                                   

wsgiref.simple_server fails to run demo_app                      09/06/08
CLOSED http://bugs.python.org/issue3795    created  Walling                   
                                                                               

some tests are not run in test_float                             09/06/08
CLOSED http://bugs.python.org/issue3796    created  amaury.forgeotdarc        
       patch, patch, needs review                                              

mmap, dbm, ossaudiodev, marshal & winreg return bytearray instea 09/06/08
CLOSED http://bugs.python.org/issue3797    created  gregory.p.smith           
       patch, patch, easy, needs review                                        

SystemExit incorrectly displays unicode message                  09/06/08
       http://bugs.python.org/issue3798    created  amaury.forgeotdarc        
                                                                               

Byte/string inconsistencies between different dbm modules        09/06/08
       http://bugs.python.org/issue3799    created  skip.montanaro            
       patch                                                                   

Fix for formatter.py                                             09/07/08
CLOSED http://bugs.python.org/issue3800    created  skomoroh                  
       patch                                                                   

cgi.parse_qsl does not return list                               09/07/08
CLOSED http://bugs.python.org/issue3801    created  acapnotic                 
       patch                                                                   

smtpd.py __getaddr insufficient handling                         09/08/08
       http://bugs.python.org/issue3802    created  marcus at internetnowasp.net 
                                                                               

Comparison operators - New rules undocumented in Python 3.0      09/08/08
CLOSED http://bugs.python.org/issue3803    created  mgiuca                    
       patch                                                                   

Test for issue2222 (r65745)                                      09/08/08
CLOSED http://bugs.python.org/issue3804    created  ocean-city                
       patch                                                                   

sslobj.read py3k takes odd arguments                             09/08/08
       http://bugs.python.org/issue3805    created  gregory.p.smith           
                                                                               

LockTests in test_imp should be skipped when thread is not avail 09/08/08
CLOSED http://bugs.python.org/issue3806    created  ocean-city                
       patch                                                                   

_multiprocessing build fails when configure --without-threads    09/08/08
       http://bugs.python.org/issue3807    created  ocean-city                
       patch, needs review                                                     

test_cgi is giving deprecation warnings                          09/09/08
CLOSED http://bugs.python.org/issue3808    created  benjamin.peterson         
                                                                               

test_logging leaving a 'test.blah' file behind                   09/09/08
CLOSED http://bugs.python.org/issue3809    created  brett.cannon              
                                                                               

os.chdir() et al: is the path str or bytes?                      09/09/08
       http://bugs.python.org/issue3810    created  ceder                     
                                                                               

Update Unicode database to 5.1.0                                 09/09/08
CLOSED http://bugs.python.org/issue3811    created  loewis                    
       patch, needs review                                                     

py3k build fails if configure --without-threads                  09/09/08
CLOSED http://bugs.python.org/issue3812    created  ocean-city                
       patch                                                                   

cannot lanch python.exe via symbolic link if HAVE_BROKEN_MBSTOWC 09/09/08
CLOSED http://bugs.python.org/issue3813    created  ocean-city                
       patch                                                                   

Add VCS support                                                  09/09/08
CLOSED http://bugs.python.org/issue3814    created  sdouche                   
       patch                                                                   

Python 3.0b3 - Idle doesn't start on win XPh                     09/09/08
CLOSED http://bugs.python.org/issue3815    created  vbr                       
                                                                               

__newobj__ pickle feature is not documented                      09/09/08
       http://bugs.python.org/issue3816    created  christian.heimes          
       easy                                                                    

ftplib: ABOR does not consider 225 response code                 09/09/08
       http://bugs.python.org/issue3817    created  giampaolo.rodola          
       patch                                                                   

ftplib.FTP.abort() should consider "225" a positive response cod 09/09/08
CLOSED http://bugs.python.org/issue3818    created  giampaolo.rodola          
       patch                                                                   

urllib2 sends Basic auth across redirects                        09/09/08
       http://bugs.python.org/issue3819    created  TFKyle                    
                                                                               

Python 3.0b3 doesn't start on German Win XP SP3/SP2              09/09/08
       http://bugs.python.org/issue3820    created  pietzcker                 
                                                                               

trace module bug when using --missing                            09/09/08
       http://bugs.python.org/issue3821    created  jjdominguezm              
       patch, needs review                                                     

zfill doc string uses inconsistent variable names                09/09/08
CLOSED http://bugs.python.org/issue3822    created  tjd                       
                                                                               

ssl.wrap_socket() is incompatible with servers that drop privile 09/09/08
       http://bugs.python.org/issue3823    created  forest                    
                                                                               

test_tarfile fails on cygwin (unicode decode error)              09/09/08
       http://bugs.python.org/issue3824    created  ocean-city                
       patch                                                                   

Major reworking of Python 2.5.2 re module                        09/09/08
       http://bugs.python.org/issue3825    created  mrabarnett                
       patch                                                                   

Self-reference in BaseHTTPRequestHandler descendants causes stuc 09/09/08
       http://bugs.python.org/issue3826    created  romkyns                   
                                                                               

memoryview.size is redundant                                     09/09/08
CLOSED http://bugs.python.org/issue3827    created  benjamin.peterson         
       patch, needs review                                                     

Bound methods compare 'successfully' with ints                   09/10/08
CLOSED http://bugs.python.org/issue3828    created  inducer                   
                                                                               

Tuple comparison masking exception                               09/10/08
       http://bugs.python.org/issue3829    created  fdrake                    
                                                                               

Tarfile has incorrect USTAR "VERSION" field (should be 00; is 0  09/10/08
CLOSED http://bugs.python.org/issue3830    created  SunriseProgrammer         
                                                                               

Multiprocessing: Expose underlying pipe in queues                09/11/08
       http://bugs.python.org/issue3831    created  TarkaSteve                
                                                                               

add force_shared Library option to create shared lib even with u 09/11/08
CLOSED http://bugs.python.org/issue3832    created  vajda                     
                                                                               

python-2.6b3.msi and python-2.6b3.amd64.msi can't both be instal 09/11/08
       http://bugs.python.org/issue3833    created  jretz                     
                                                                               

wsgiref.validator.InputWrapper readline method has wrong signatu 09/11/08
CLOSED http://bugs.python.org/issue3834    created  strogon14                 
                                                                               

tkinter goes into an infinite loop (pydoc.gui)                   09/11/08
       http://bugs.python.org/issue3835    created  HWJ                       
                                                                               

2to3 broken due to mixed 2.5 and 3.0 syntax                      09/11/08
CLOSED http://bugs.python.org/issue3836    created  bhy                       
       patch                                                                   

Comment for html_logo setting is out of date                     09/11/08
CLOSED http://bugs.python.org/issue3837    created  jmswisher                 
                                                                               

test_tarfile error on cygwin (Directory not empty)               09/11/08
       http://bugs.python.org/issue3838    created  ocean-city                
       patch, easy                                                             

wsgi.simple_server resets 'Content-Length' header on empty conte 09/11/08
       http://bugs.python.org/issue3839    created  nosklo                    
       patch                                                                   

if TESTFN == "/tmp/@test", some tests fail                       09/11/08
       http://bugs.python.org/issue3840    created  ocean-city                
       patch                                                                   

IDLE: quirky behavior when displaying strings longer than 4093 c 09/11/08
       http://bugs.python.org/issue3841    created  spmcinerney               
                                                                               

can't run close through itertools.chain on inner generator       09/11/08
       http://bugs.python.org/issue3842    created  dangyogi                  
                                                                               

hexadecimal, not decimal                                         09/11/08
CLOSED http://bugs.python.org/issue3843    created  maix                      
                                                                               

Script: find untested C functions                                09/12/08
       http://bugs.python.org/issue3844    created  ajaksu2                   
                                                                               

memory access before short string when checking suffix           09/12/08
       http://bugs.python.org/issue3845    created  doko                      
                                                                               

sqlite3 module: Improved concurrency                             09/12/08
       http://bugs.python.org/issue3846    created  ghaering                  
       needs review                                                            

Begging for review                                               09/12/08
       http://bugs.python.org/issue3847    created  ghaering                  
                                                                               

select.epoll calling register with the same fd fails             09/12/08
       http://bugs.python.org/issue3848    created  ionel.mc                  
                                                                               



Issues Now Closed (73)
______________________

readline configuration for shared libs w/o curses dependencies      0 days
       http://bugs.python.org/issue1204    gregory.p.smith           
       patch, 64bit                                                            

smtplib.SMTP docs                                                 319 days
       http://bugs.python.org/issue1317    akuchling                 
                                                                               

New SSL module doesn't seem to verify hostname against commonNam  275 days
       http://bugs.python.org/issue1589    janssen                   
                                                                               

Tools/i18n/msgfmt.py fixes for Python 3.0                         237 days
       http://bugs.python.org/issue1840    loewis                    
       patch                                                                   

ctypes.util.find_library(): posix .so without SONAME              202 days
       http://bugs.python.org/issue2145    theller                   
                                                                               

Faq 4.28 -- Trailing comas                                        172 days
       http://bugs.python.org/issue2420    akuchling                 
                                                                               

Cannot use non-ascii letters in disutils if setuptools is	used.   156 days
       http://bugs.python.org/issue2562    lemburg                   
       patch                                                                   

Documentation of new gobject types fails                          132 days
       http://bugs.python.org/issue2747    georg.brandl              
                                                                               

sphinx and virtualenv                                             118 days
       http://bugs.python.org/issue2859    georg.brandl              
                                                                               

optparse documentation: variable arguments example doesn't work    93 days
       http://bugs.python.org/issue3040    akuchling                 
                                                                               

ssl.SSLSocket implements methods that are overriden by socket.so   79 days
       http://bugs.python.org/issue3162    hodgestar                 
       patch                                                                   

import of site.py fails on startup                                 64 days
       http://bugs.python.org/issue3279    benjamin.peterson         
       patch, needs review                                                     

Documentation: timeit: "lower bound" should read "upper bound"     64 days
       http://bugs.python.org/issue3318    georg.brandl              
                                                                               

Use Python 3 lexer for 3.0 docs                                    55 days
       http://bugs.python.org/issue3376    georg.brandl              
       patch                                                                   

Updates to "Macintosh Library Modules" Section 1.1                 41 days
       http://bugs.python.org/issue3472    georg.brandl              
                                                                               

Zlib compress/decompress functions returning bytearray             37 days
       http://bugs.python.org/issue3492    pythonhacker              
       patch, easy, needs review                                               

zipfile has problem reading zip files over 2GB                     27 days
       http://bugs.python.org/issue3535    pitrou                    
       patch, needs review                                                     

test_unicode.test_raiseMemError fails in UCS4                      17 days
       http://bugs.python.org/issue3601    pitrou                    
       patch, needs review                                                     

broken link in curses module docs                                  18 days
       http://bugs.python.org/issue3604    akuchling                 
                                                                               

2to3: commands varible replaced by subprocess                      20 days
       http://bugs.python.org/issue3606    benjamin.peterson         
                                                                               

Python won't compile a regex that compiles with 2.5.2 and 30b2     20 days
       http://bugs.python.org/issue3629    gvanrossum                
       patch, needs review                                                     

invalid result value of _weakref.__init__()                        19 days
       http://bugs.python.org/issue3634    benjamin.peterson         
       patch                                                                   

test_cpickle crash on AMD64 Windows build                          21 days
       http://bugs.python.org/issue3640    amaury.forgeotdarc        
       patch, 64bit                                                            

Objects/obmalloc.c:529: warning: comparison is always false due    17 days
       http://bugs.python.org/issue3642    loewis                    
       patch, 64bit                                                            

readline module Crashs on OpenBSD/amd64                            16 days
       http://bugs.python.org/issue3645    gregory.p.smith           
       patch, 64bit, easy                                                      

fix for pychecker property complaints                              17 days
       http://bugs.python.org/issue3658    benjamin.peterson         
       patch, easy, needs review                                               

sqlite3.Connection.iterdump docs pythonicity                       13 days
       http://bugs.python.org/issue3669    akuchling                 
       patch                                                                   

os.urandom(1.1): infinite loop                                     12 days
       http://bugs.python.org/issue3708    benjamin.peterson         
       patch, easy                                                             

PY_FORMAT_SIZE_T is not for PyString_FromFormat                    10 days
       http://bugs.python.org/issue3743    benjamin.peterson         
       patch                                                                   

Sphinx producing duplicate id attributes, HTML fails validation.    9 days
       http://bugs.python.org/issue3746    georg.brandl              
                                                                               

re.escape() does not work with bytes()                              9 days
       http://bugs.python.org/issue3756    andrewmcnamara            
                                                                               

"make check" suggest a testing target under GNU coding standards    3 days
       http://bugs.python.org/issue3758    brett.cannon              
       patch                                                                   

asyncore differences between 2.x and 3.x                            0 days
       http://bugs.python.org/issue3764    giampaolo.rodola          
       easy                                                                    

Deprecate bsddb for removal in 3.0                                  2 days
       http://bugs.python.org/issue3769    brett.cannon              
       patch, patch, needs review                                              

deprecate bsddb/dbhash in 2.6 for removal in 3.0                    1 days
       http://bugs.python.org/issue3776    brett.cannon              
       patch, patch, needs review                                              

long(4.2) now returns an int                                        5 days
       http://bugs.python.org/issue3777    skip.montanaro            
       patch                                                                   

Incorrect compiler options used for cc of Sun Studio 12             3 days
       http://bugs.python.org/issue3784    loewis                    
                                                                               

Can't build ctypes of Python 2.5.2 with Sun Studio 12               0 days
       http://bugs.python.org/issue3785    theller                   
                                                                               

multiprocessing deadlocks when sending large data through Queue     0 days
       http://bugs.python.org/issue3789    jnoller                   
                                                                               

in zlib decompressor objects, unused_data and unconsumed_tail mu    1 days
       http://bugs.python.org/issue3790    gregory.p.smith           
       patch, patch, needs review                                              

Module variable overridden in child processes with multiprocessi    0 days
       http://bugs.python.org/issue3792    TarkaSteve                
                                                                               

Small RST fix in datamodel.rst                                      0 days
       http://bugs.python.org/issue3793    georg.brandl              
                                                                               

__div__ still documented in Python 3                                0 days
       http://bugs.python.org/issue3794    georg.brandl              
       patch                                                                   

wsgiref.simple_server fails to run demo_app                         0 days
       http://bugs.python.org/issue3795    pitrou                    
                                                                               

some tests are not run in test_float                                0 days
       http://bugs.python.org/issue3796    amaury.forgeotdarc        
       patch, patch, needs review                                              

mmap, dbm, ossaudiodev, marshal & winreg return bytearray instea    0 days
       http://bugs.python.org/issue3797    gregory.p.smith           
       patch, patch, easy, needs review                                        

Fix for formatter.py                                                2 days
       http://bugs.python.org/issue3800    georg.brandl              
       patch                                                                   

cgi.parse_qsl does not return list                                  0 days
       http://bugs.python.org/issue3801    facundobatista            
       patch                                                                   

Comparison operators - New rules undocumented in Python 3.0         2 days
       http://bugs.python.org/issue3803    georg.brandl              
       patch                                                                   

Test for issue2222 (r65745)                                         0 days
       http://bugs.python.org/issue3804    ocean-city                
       patch                                                                   

LockTests in test_imp should be skipped when thread is not avail    0 days
       http://bugs.python.org/issue3806    ocean-city                
       patch                                                                   

test_cgi is giving deprecation warnings                             0 days
       http://bugs.python.org/issue3808    facundobatista            
                                                                               

test_logging leaving a 'test.blah' file behind                      1 days
       http://bugs.python.org/issue3809    brett.cannon              
                                                                               

Update Unicode database to 5.1.0                                    2 days
       http://bugs.python.org/issue3811    loewis                    
       patch, needs review                                                     

py3k build fails if configure --without-threads                     0 days
       http://bugs.python.org/issue3812    ocean-city                
       patch                                                                   

cannot lanch python.exe via symbolic link if HAVE_BROKEN_MBSTOWC    0 days
       http://bugs.python.org/issue3813    ocean-city                
       patch                                                                   

Add VCS support                                                     0 days
       http://bugs.python.org/issue3814    pitrou                    
       patch                                                                   

Python 3.0b3 - Idle doesn't start on win XPh                        0 days
       http://bugs.python.org/issue3815    vbr                       
                                                                               

ftplib.FTP.abort() should consider "225" a positive response cod    0 days
       http://bugs.python.org/issue3818    georg.brandl              
       patch                                                                   

zfill doc string uses inconsistent variable names                   0 days
       http://bugs.python.org/issue3822    georg.brandl              
                                                                               

memoryview.size is redundant                                        1 days
       http://bugs.python.org/issue3827    amaury.forgeotdarc        
       patch, needs review                                                     

Bound methods compare 'successfully' with ints                      0 days
       http://bugs.python.org/issue3828    amaury.forgeotdarc        
                                                                               

Tarfile has incorrect USTAR "VERSION" field (should be 00; is 0     0 days
       http://bugs.python.org/issue3830    lars.gustaebel            
                                                                               

add force_shared Library option to create shared lib even with u    0 days
       http://bugs.python.org/issue3832    loewis                    
                                                                               

wsgiref.validator.InputWrapper readline method has wrong signatu    0 days
       http://bugs.python.org/issue3834    pje                       
                                                                               

2to3 broken due to mixed 2.5 and 3.0 syntax                         0 days
       http://bugs.python.org/issue3836    benjamin.peterson         
       patch                                                                   

Comment for html_logo setting is out of date                        0 days
       http://bugs.python.org/issue3837    georg.brandl              
                                                                               

hexadecimal, not decimal                                            0 days
       http://bugs.python.org/issue3843    benjamin.peterson         
                                                                               

-S hides standard dynamic modules                                2240 days
       http://bugs.python.org/issue586680  benjamin.peterson         
                                                                               

threading module can deadlock after fork                           20 days
       http://bugs.python.org/issue874900  gregory.p.smith           
       patch, needs review                                                     

SSLObject breaks read semantics                                  1087 days
       http://bugs.python.org/issue1291446 ellisj                    
                                                                               

Search is to long with regex like ^(.+|dontmatch)*$              1084 days
       http://bugs.python.org/issue1297193 pitrou                    
                                                                               

Add httponly to Cookie module                                     598 days
       http://bugs.python.org/issue1638033 benjamin.peterson         
       patch                                                                   



Top Issues Most Discussed (10)
______________________________

 31 warnings.catch_warnings fails gracelessly when recording warnin    3 days
open    http://bugs.python.org/issue3781   

 25 dbm.sqlite proof of concept                                        8 days
open    http://bugs.python.org/issue3783   

 11 long(4.2) now returns an int                                       5 days
closed  http://bugs.python.org/issue3777   

 10 Update Unicode database to 5.1.0                                   2 days
closed  http://bugs.python.org/issue3811   

 10 ssl.SSLSocket implements methods that are overriden by socket.s   79 days
closed  http://bugs.python.org/issue3162   

  9 tkinter Menu.delete bug                                            0 days
open    http://bugs.python.org/issue3774   

  9 socket.socket.recv broken (unbearably slow)                        9 days
open    http://bugs.python.org/issue3766   

  9 py3k fails under Windows if "-c" or "-m" is given a non-ascii v    3 days
open    http://bugs.python.org/issue3705   

  9 readline configuration for shared libs w/o curses dependencies     0 days
closed  http://bugs.python.org/issue1204   

  8 ssl.wrap_socket() is incompatible with servers that drop privil    3 days
open    http://bugs.python.org/issue3823   




From edreamleo at gmail.com  Fri Sep 12 15:19:27 2008
From: edreamleo at gmail.com (Edward K. Ream)
Date: Fri, 12 Sep 2008 08:19:27 -0500
Subject: [Python-Dev] [Python-3000] Updated release schedule for 2.6 and
	3.0
In-Reply-To: <93DF7138-3E99-44E3-AF7D-01D89D13E910@python.org>
References: <93DF7138-3E99-44E3-AF7D-01D89D13E910@python.org>
Message-ID: <ffb592890809120619ne821addj108a22e7163afab6@mail.gmail.com>

On Fri, Sep 12, 2008 at 7:54 AM, Barry Warsaw <barry at python.org> wrote:

> We had a lot of discussion recently about changing the release schedule and
> splitting Python 2.6 and 3.0.  There was general consensus that this was a
> good idea, in order to hit our October 1 deadline for Python 2.6 final at
> least.

Does it matter to anyone besides the you, the Python developers,
whether the schedule slips by two weeks, or two months, for that
matter?

I am underwhelmed by 3.0 b3: sax and 2to3 are/were broken.  A b4
release for 3.0 (at least) would seem more prudent.

Edward

From barry at python.org  Sat Sep 13 03:28:55 2008
From: barry at python.org (Barry Warsaw)
Date: Fri, 12 Sep 2008 21:28:55 -0400
Subject: [Python-Dev] RELEASED Python 2.6rc1
Message-ID: <49568982-472B-46BB-9001-12078706B238@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On behalf of the Python development team and the Python community, I  
am happy to announce the first release candidate for Python 2.6.

This is a release candidate, so while it is not suitable for  
production environments, we strongly encourage you to download the  
release and test it on your software.  We expect only critical bugs to  
be fixed between now and the final 2.6 release, still scheduled for  
October 1st, 2008.  There is one more release candidate planned for  
September 17th.

You might notice that unlike earlier releases, we are /not/ releasing  
Python 3.0rc1 at this time.  It was decided that 3.0 still needs time  
to resolve open issues and that we would not hold up the 2.6 release  
for this.  We feel that Python 2.6 is nearly ready for its final  
release.

If you find things broken or incorrect, please submit bug reports at

     http://bugs.python.org

For more information and downloadable distributions, see the Python  
2.6 website:

    http://www.python.org/download/releases/2.6/

(Note that the Windows installers will be uploaded shortly.)

See PEP 361 for release schedule details:

     http://www.python.org/dev/peps/pep-0361/

Enjoy,
- -Barry

Barry Warsaw
barry at python.org
Python 2.6/3.0 Release Manager
(on behalf of the entire python-dev team)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSMsXV3EjvBPtnXfVAQJFsgP9GxZYQocbDTd0Z/0yEjpHfZ/FTd8y83jV
5JouO07lB8XtLawnWB9hF8sUrCuBVog5He3mLVUPDmlyn30qvjYWMG2J6zW0yYMX
yZdjUyUmta0IMCsXe7YXj369xebh4nWuwG4tDygly4donA7GYPXAlxI48MmyDJxw
1v07LM4Dttw=
=Nd3s
-----END PGP SIGNATURE-----

From g.brandl at gmx.net  Sat Sep 13 11:49:51 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 13 Sep 2008 11:49:51 +0200
Subject: [Python-Dev] Do we still support MacOS < X?
Message-ID: <gag2cc$6jg$1@ger.gmane.org>

If not, I'll remove the traces from the docs, where they only serve to confuse
where MacOS X actually belongs under "Unix", not "Mac".

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From henning.vonbargen at arcor.de  Sat Sep 13 11:38:01 2008
From: henning.vonbargen at arcor.de (Henning von Bargen)
Date: Sat, 13 Sep 2008 11:38:01 +0200
Subject: [Python-Dev] Add python.exe to PATH environment variable
References: <mailman.5561.1220855729.3486.python-dev@python.org>
Message-ID: <2BA588E3B47347B2B5AEED70B79011F3@max>

One of the things I like about Python is that

* it doesn't actually need an installation. *

It is sufficient to set up PATH (and, if you like, PYTHONPATH)
accordingly to start python. However, you need the DLLs somewhere
in the Python directory, too.

That way it is possible to install Python applications in a minimal-invasive
fashion: Just copy one directory to the hard-disk (or a network drive) and
create a shortcut to some start-app.cmd script

For example, this is a great advantage if you need to install a Python 
application
on a client's application server,
since the client's IT departments in reponsibility for the server
absolutely don't like it when programs modify PATH or install DLLs in the
%WINDIR%\System32 directory - and they are right here!!!
Think of what happens if a program modifies PATH: The modification does not
affect running services, of course. But *after a reboot* (possibly a few 
months later),
the old services will find another version of some program than before, and 
then
someting is going to crash!
I have seen too much trouble of this kind after the installation of a 
different Perl
version or a another Oracle product. The reason is then quite hard to find: 
if the
installation of another software was 3 months ago, will anyone remember? And
will anyone even consider it could be the reason for the fault after the 
reboot?
Well - I will, but most others probably won't.

I don't want this kind of trouble too happen with Python!

So, please try keeping the Python installation as minimal invasive as 
possible.
Note: This could also be an option for the installation.

Henning


From amk at amk.ca  Sat Sep 13 14:03:50 2008
From: amk at amk.ca (A.M. Kuchling)
Date: Sat, 13 Sep 2008 08:03:50 -0400
Subject: [Python-Dev] 2.6 rc1 performance results
Message-ID: <20080913120350.GA19725@amk.local>

Three weeks ago, Antoine Pitrou posted the pybench results 
for 2.6 trunk:
http://mail.python.org/pipermail/python-dev/2008-August/081951.html

The big discovery in those results were TryExcept being 48% slower,
but there was a patch in the bug tracker to improve things.  I've
re-run the tests to check the results.

Disclaimer: these results are probably not directly comparable.
Antoine was using a "32-bit Linux installation on an Athlon 3600+ X2";
I'm on a Macbook.

Good news: TryExcept is now only 10% slower than 2.5, not 48%.

Bad news: the big slowdowns are:

                 CompareFloats:   117ms    98ms  +19.2%   118ms    99ms  +19.0%
               CompareIntegers:   110ms   104ms   +5.6%   110ms   105ms   +4.9%
            DictWithStringKeys:   118ms   105ms  +12.8%   133ms   108ms  +22.7%
                NestedForLoops:   125ms   116ms   +7.7%   127ms   118ms   +8.0%
                     Recursion:   193ms   159ms  +21.5%   197ms   163ms  +20.8%
                  SecondImport:   139ms   129ms   +8.4%   143ms   130ms   +9.9%
           SecondPackageImport:   150ms   139ms   +8.6%   152ms   140ms   +8.1%
         SecondSubmoduleImport:   211ms   191ms  +10.5%   214ms   195ms   +9.4%
       SimpleComplexArithmetic:   130ms   119ms   +9.4%   131ms   120ms   +9.2%

Antoine, your Recursion results were actually about the same (+2.2%)
from 2.5 to 2.6, so this big slowdown is novel.  I wonder if these
tests are simply slower on MacOS for some reason (compiler, CPU cache
size, etc.).  Does anyone see similar results?  Any idea what might
have made DictForStringKeys and Recursion slow down?

Complete results:

Test                             minimum run-time        average  run-time
                                 this    other   diff    this    other   diff
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:   140ms   148ms   -5.4%   142ms   153ms   -7.5%
           BuiltinMethodLookup:   120ms   135ms  -11.2%   122ms   137ms  -11.0%
                 CompareFloats:   117ms    98ms  +19.2%   118ms    99ms  +19.0%
         CompareFloatsIntegers:   109ms   119ms   -8.9%   109ms   121ms   -9.3%
               CompareIntegers:   110ms   104ms   +5.6%   110ms   105ms   +4.9%
        CompareInternedStrings:   128ms   153ms  -16.3%   131ms   158ms  -16.8%
                  CompareLongs:   102ms    99ms   +3.5%   105ms   101ms   +3.9%
                CompareStrings:   164ms   161ms   +2.0%   166ms   165ms   +0.7%
                CompareUnicode:   141ms   158ms  -10.5%   143ms   164ms  -12.6%
    ComplexPythonFunctionCalls:   159ms   272ms  -41.3%   164ms   277ms  -40.6%
                 ConcatStrings:   173ms   168ms   +3.2%   177ms   172ms   +3.1%
                 ConcatUnicode:   108ms   121ms  -10.8%   111ms   124ms  -10.4%
               CreateInstances:   168ms   180ms   -6.4%   176ms   182ms   -3.7%
            CreateNewInstances:   129ms   153ms  -15.6%   132ms   158ms  -16.0%
       CreateStringsWithConcat:   156ms   157ms   -0.7%   158ms   161ms   -1.9%
       CreateUnicodeWithConcat:   112ms   114ms   -1.8%   114ms   117ms   -2.2%
                  DictCreation:   104ms   112ms   -7.1%   106ms   114ms   -7.2%
             DictWithFloatKeys:   149ms   162ms   -7.7%   153ms   168ms   -8.7%
           DictWithIntegerKeys:   123ms   148ms  -16.8%   127ms   151ms  -15.9%
            DictWithStringKeys:   118ms   105ms  +12.8%   133ms   108ms  +22.7%
                      ForLoops:    91ms    88ms   +3.6%    91ms    88ms   +3.0%
                    IfThenElse:   108ms   102ms   +5.2%   109ms   103ms   +5.5%
                   ListSlicing:   155ms   239ms  -35.0%   157ms   241ms  -34.6%
                NestedForLoops:   125ms   116ms   +7.7%   127ms   118ms   +8.0%
          NormalClassAttribute:   135ms   140ms   -3.8%   139ms   146ms   -4.7%
       NormalInstanceAttribute:   123ms   126ms   -2.4%   125ms   130ms   -4.4%
           PythonFunctionCalls:   126ms   126ms   +0.0%   129ms   128ms   +0.9%
             PythonMethodCalls:   165ms   165ms   -0.1%   168ms   170ms   -1.1%
                     Recursion:   193ms   159ms  +21.5%   197ms   163ms  +20.8%
                  SecondImport:   139ms   129ms   +8.4%   143ms   130ms   +9.9%
           SecondPackageImport:   150ms   139ms   +8.6%   152ms   140ms   +8.1%
         SecondSubmoduleImport:   211ms   191ms  +10.5%   214ms   195ms   +9.4%
       SimpleComplexArithmetic:   130ms   119ms   +9.4%   131ms   120ms   +9.2%
        SimpleDictManipulation:   124ms   146ms  -14.6%   128ms   150ms  -14.8%
         SimpleFloatArithmetic:   127ms   132ms   -3.6%   131ms   144ms   -9.3%
      SimpleIntFloatArithmetic:    93ms   100ms   -6.5%    94ms   100ms   -5.6%
       SimpleIntegerArithmetic:    94ms    91ms   +2.8%    95ms    92ms   +3.1%
        SimpleListManipulation:   108ms   110ms   -1.1%   110ms   111ms   -1.2%
          SimpleLongArithmetic:   141ms   136ms   +3.8%   143ms   139ms   +2.8%
                    SmallLists:   157ms   151ms   +4.3%   160ms   156ms   +2.6%
                   SmallTuples:   123ms   123ms   -0.3%   125ms   127ms   -1.5%
         SpecialClassAttribute:   136ms   152ms  -10.5%   140ms   155ms  -10.2%
      SpecialInstanceAttribute:   225ms   256ms  -11.9%   227ms   258ms  -11.8%
                StringMappings:   197ms   206ms   -4.4%   201ms   209ms   -3.8%
              StringPredicates:   154ms   194ms  -20.5%   157ms   196ms  -20.2%
                 StringSlicing:   144ms   146ms   -1.1%   152ms   155ms   -1.9%
                     TryExcept:   111ms   101ms   +9.5%   114ms   102ms  +11.2%
                    TryFinally:   120ms   129ms   -6.9%   122ms   131ms   -6.8%
                TryRaiseExcept:   145ms   150ms   -4.0%   148ms   153ms   -3.2%
                  TupleSlicing:   141ms   140ms   +1.3%   152ms   159ms   -4.5%
               UnicodeMappings:   140ms   133ms   +5.9%   143ms   136ms   +5.5%
             UnicodePredicates:   123ms   137ms   -9.9%   125ms   138ms   -9.9%
             UnicodeProperties:   136ms   146ms   -6.7%   142ms   151ms   -6.3%
                UnicodeSlicing:   132ms   129ms   +2.0%   134ms   131ms   +2.2%
                   WithFinally:   167ms   168ms   -1.0%   172ms   173ms   -0.3%
               WithRaiseExcept:   134ms   150ms  -11.1%   140ms   156ms   -9.8%
-------------------------------------------------------------------------------
Totals:                          7659ms  8031ms   -4.6%  7834ms  8229ms   -4.8%

(this=p26.pybench, other=p25.pybench)


--amk

From solipsis at pitrou.net  Sat Sep 13 14:34:22 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 13 Sep 2008 12:34:22 +0000 (UTC)
Subject: [Python-Dev] 2.6 rc1 performance results
References: <20080913120350.GA19725@amk.local>
Message-ID: <loom.20080913T122417-601@post.gmane.org>

A.M. Kuchling <amk <at> amk.ca> writes:
> 
> Bad news: the big slowdowns are:
[snip]

I don't get the same results, but there can be significant variations between
two pybench runs. Did use the same compiler and the same flags for both Python
versions?



From exarkun at divmod.com  Sat Sep 13 15:39:43 2008
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Sat, 13 Sep 2008 09:39:43 -0400
Subject: [Python-Dev] 2.6 rc1 performance results
In-Reply-To: <20080913120350.GA19725@amk.local>
Message-ID: <20080913133943.29191.2050741059.divmod.quotient.25499@ohm>

On Sat, 13 Sep 2008 08:03:50 -0400, "A.M. Kuchling" <amk at amk.ca> wrote:
>Three weeks ago, Antoine Pitrou posted the pybench results
>for 2.6 trunk:
>http://mail.python.org/pipermail/python-dev/2008-August/081951.html
>
>The big discovery in those results were TryExcept being 48% slower,
>but there was a patch in the bug tracker to improve things.  I've
>re-run the tests to check the results.
>
>Disclaimer: these results are probably not directly comparable.
>Antoine was using a "32-bit Linux installation on an Athlon 3600+ X2";
>I'm on a Macbook.
>
>Good news: TryExcept is now only 10% slower than 2.5, not 48%.
>
>Bad news: the big slowdowns are:
>
>                 CompareFloats:   117ms    98ms  +19.2%   118ms    99ms  +19.0%
>               CompareIntegers:   110ms   104ms   +5.6%   110ms   105ms   +4.9%
>            DictWithStringKeys:   118ms   105ms  +12.8%   133ms   108ms  +22.7%
>                NestedForLoops:   125ms   116ms   +7.7%   127ms   118ms   +8.0%
>                     Recursion:   193ms   159ms  +21.5%   197ms   163ms  +20.8%
>                  SecondImport:   139ms   129ms   +8.4%   143ms   130ms   +9.9%
>           SecondPackageImport:   150ms   139ms   +8.6%   152ms   140ms   +8.1%
>         SecondSubmoduleImport:   211ms   191ms  +10.5%   214ms   195ms   +9.4%
>       SimpleComplexArithmetic:   130ms   119ms   +9.4%   131ms   120ms   +9.2%
>

I see similar results for some of these.  The complete results from a run
on an AMD Athlon(tm) 64 Processor 3200+ are attached.

Jean-Paul

-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: python-benchmark.txt
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080913/f3743f21/attachment.txt>

From musiccomposition at gmail.com  Sat Sep 13 17:24:38 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sat, 13 Sep 2008 10:24:38 -0500
Subject: [Python-Dev] Do we still support MacOS < X?
In-Reply-To: <gag2cc$6jg$1@ger.gmane.org>
References: <gag2cc$6jg$1@ger.gmane.org>
Message-ID: <1afaf6160809130824o7e86afe8g2ac80286d5b6de83@mail.gmail.com>

On Sat, Sep 13, 2008 at 4:49 AM, Georg Brandl <g.brandl at gmx.net> wrote:
> If not, I'll remove the traces from the docs, where they only serve to confuse
> where MacOS X actually belongs under "Unix", not "Mac".

Yes, according to PEP 11, support was removed in 2.4. I suppose this
also means we could killed macpath in py3k...



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From ncoghlan at gmail.com  Sat Sep 13 17:45:51 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Sep 2008 01:45:51 +1000
Subject: [Python-Dev] 2.6 rc1 performance results
In-Reply-To: <20080913120350.GA19725@amk.local>
References: <20080913120350.GA19725@amk.local>
Message-ID: <48CBE02F.9060207@gmail.com>

A.M. Kuchling wrote:
> Antoine, your Recursion results were actually about the same (+2.2%)
> from 2.5 to 2.6, so this big slowdown is novel.  I wonder if these
> tests are simply slower on MacOS for some reason (compiler, CPU cache
> size, etc.).  Does anyone see similar results?  Any idea what might
> have made DictForStringKeys and Recursion slow down?

The change to universal binaries, perhaps? Your results showed quite a
few slowdowns in number related code, while my local testing shows
primarily speed increases in those areas.

That said, I'm seeing big enough swings in the percentages between runs
that I'd like to get some tips on how to smooth out the variations -
e.g. will increasing the warp factor increasing the amount of time each
individual run takes?

Although on a Mac OS X specific front... could the conversion to
universal binaries have made a difference? Do you get the same
performance numbers for a local build as you do for the version from the
installer?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From martin at v.loewis.de  Sat Sep 13 17:59:27 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 13 Sep 2008 17:59:27 +0200
Subject: [Python-Dev] 2.6 rc1 performance results
In-Reply-To: <48CBE02F.9060207@gmail.com>
References: <20080913120350.GA19725@amk.local> <48CBE02F.9060207@gmail.com>
Message-ID: <48CBE35F.5010704@v.loewis.de>


> The change to universal binaries, perhaps?

That shouldn't really matter - the machine code should still be the
same, and it should all get loaded at program startup. IOW, startup
and imports may get slower, but otherwise, it should have no impact.

Regards,
Martin

From solipsis at pitrou.net  Sat Sep 13 18:03:47 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 13 Sep 2008 16:03:47 +0000 (UTC)
Subject: [Python-Dev] 2.6 rc1 performance results
References: <20080913120350.GA19725@amk.local> <48CBE02F.9060207@gmail.com>
Message-ID: <loom.20080913T160221-597@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> 
> That said, I'm seeing big enough swings in the percentages between runs
> that I'd like to get some tips on how to smooth out the variations -
> e.g. will increasing the warp factor increasing the amount of time each
> individual run takes?

Increasing the number of rounds (-n) is probably better.
Also, if you are on a laptop or a modern desktop machine, check that CPU
frequency scaling is disabled before running any benchmark (on Linux,
"cpufreq-set -g performance" does the trick).




From armin.ronacher at active-4.com  Sat Sep 13 21:50:13 2008
From: armin.ronacher at active-4.com (Armin Ronacher)
Date: Sat, 13 Sep 2008 19:50:13 +0000 (UTC)
Subject: [Python-Dev] Do we still support MacOS < X?
References: <gag2cc$6jg$1@ger.gmane.org>
	<1afaf6160809130824o7e86afe8g2ac80286d5b6de83@mail.gmail.com>
Message-ID: <loom.20080913T194843-752@post.gmane.org>

Hi,

Benjamin Peterson <musiccomposition <at> gmail.com> writes:

> On Sat, Sep 13, 2008 at 4:49 AM, Georg Brandl <g.brandl <at> gmx.net> wrote:
> > If not, I'll remove the traces from the docs, where they only serve to
> > confuse where MacOS X actually belongs under "Unix", not "Mac".
> 
> Yes, according to PEP 11, support was removed in 2.4. I suppose this
> also means we could killed macpath in py3k...
Pleaes don't do that.  The OS 9 path handling is still present in OS X GUI
applications.  While you don't see it that often because OS X doesn't know
the location bars but you can see it in iTunes for example.


Regards,
Armin



From armin.ronacher at active-4.com  Sat Sep 13 22:20:23 2008
From: armin.ronacher at active-4.com (Armin Ronacher)
Date: Sat, 13 Sep 2008 20:20:23 +0000 (UTC)
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
Message-ID: <loom.20080913T201341-491@post.gmane.org>

Hi everybody,

In Python 2.x when iterating over a weak key dictionary for example, the common
idom for doing that was calling dictionary.keys() to ensure that a list of all
objects is returned it was safe to iterate over as a weak reference could stop
existing during dict iteration which of course raises a runtime error by the
dict iterator.

This was documented behavior and worked pretty well, with the small problem that
suddenly all references in the dict wouldn't die until iteration is over because
the list holds references to the object.

This no longer works in Python 3 because .keys() on the weak key dictionary
returns a generator over the key view of the internal dict which of course has
the same problem as iterkeys in Python 2.x.

The following code shows the problem::

    from weakref import WeakKeyDictionary

    f1 = Foo()
    f2 = Foo()
    d = WeakKeyDictionary()
    d[f1] = 42
    d[f2] = 23

    i = iter(d.keys()) # or use d.keyrefs() here which has the same problem
    print(next(i))
    del f2
    print(next(i))

This example essentially dies with "RuntimeError: dictionary changed
size during iteration" as soon as f2 is deleted.

Iterating over weak key dictionaries might not be the most common task but I
know some situations where this is necessary.  Unfortunately I can't see a
way to achieve that in Python 3.

Regards,
Armin


From josiah.carlson at gmail.com  Sat Sep 13 22:29:15 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Sat, 13 Sep 2008 13:29:15 -0700
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <loom.20080913T201341-491@post.gmane.org>
References: <loom.20080913T201341-491@post.gmane.org>
Message-ID: <e6511dbf0809131329l13560b44ic877a929727490d4@mail.gmail.com>

On Sat, Sep 13, 2008 at 1:20 PM, Armin Ronacher
<armin.ronacher at active-4.com> wrote:
> Hi everybody,
>
> In Python 2.x when iterating over a weak key dictionary for example, the common
> idom for doing that was calling dictionary.keys() to ensure that a list of all
> objects is returned it was safe to iterate over as a weak reference could stop
> existing during dict iteration which of course raises a runtime error by the
> dict iterator.
>
> This was documented behavior and worked pretty well, with the small problem that
> suddenly all references in the dict wouldn't die until iteration is over because
> the list holds references to the object.
>
> This no longer works in Python 3 because .keys() on the weak key dictionary
> returns a generator over the key view of the internal dict which of course has
> the same problem as iterkeys in Python 2.x.
>
> The following code shows the problem::
>
>    from weakref import WeakKeyDictionary
>
>    f1 = Foo()
>    f2 = Foo()
>    d = WeakKeyDictionary()
>    d[f1] = 42
>    d[f2] = 23
>
>    i = iter(d.keys()) # or use d.keyrefs() here which has the same problem
>    print(next(i))
>    del f2
>    print(next(i))
>
> This example essentially dies with "RuntimeError: dictionary changed
> size during iteration" as soon as f2 is deleted.
>
> Iterating over weak key dictionaries might not be the most common task but I
> know some situations where this is necessary.  Unfortunately I can't see a
> way to achieve that in Python 3.

i = list(d.keys())

 - Josiah

From santagada at gmail.com  Sun Sep 14 01:05:39 2008
From: santagada at gmail.com (Leonardo Santagada)
Date: Sat, 13 Sep 2008 20:05:39 -0300
Subject: [Python-Dev] 2.6 rc1 performance results
In-Reply-To: <loom.20080913T160221-597@post.gmane.org>
References: <20080913120350.GA19725@amk.local> <48CBE02F.9060207@gmail.com>
	<loom.20080913T160221-597@post.gmane.org>
Message-ID: <EA92B090-E481-408F-8DA8-D13FAA1EF045@gmail.com>


On Sep 13, 2008, at 1:03 PM, Antoine Pitrou wrote:

> Nick Coghlan <ncoghlan <at> gmail.com> writes:
>>
>> That said, I'm seeing big enough swings in the percentages between  
>> runs
>> that I'd like to get some tips on how to smooth out the variations -
>> e.g. will increasing the warp factor increasing the amount of time  
>> each
>> individual run takes?
>
> Increasing the number of rounds (-n) is probably better.
> Also, if you are on a laptop or a modern desktop machine, check that  
> CPU
> frequency scaling is disabled before running any benchmark (on Linux,
> "cpufreq-set -g performance" does the trick).


I don't think there is any way to stop cpu frequency scalling on mac  
os x. Also comparing 2.6 rc1 to system python 2.5 is not fair either  
(does anyone really knows how apple compiled its python?). Also the  
performance of 2 diferent processor lines on different os insert a  
fair amount of variables to any comparison.

I would sugest compiling 2.5 and 2.6 from source, run the benchmark x  
times and take the smallest time of each test (so os and cpu scalling  
don't influence so much the benchmark) and then comparing the results.

--
Leonardo Santagada
santagada at gmail.com




From bretthoerner at gmail.com  Sun Sep 14 02:03:40 2008
From: bretthoerner at gmail.com (Brett Hoerner)
Date: Sat, 13 Sep 2008 19:03:40 -0500
Subject: [Python-Dev] 2.6 rc1 performance results
In-Reply-To: <EA92B090-E481-408F-8DA8-D13FAA1EF045@gmail.com>
References: <20080913120350.GA19725@amk.local> <48CBE02F.9060207@gmail.com>
	<loom.20080913T160221-597@post.gmane.org>
	<EA92B090-E481-408F-8DA8-D13FAA1EF045@gmail.com>
Message-ID: <e97e83a30809131703i3757fab5g249a7a5c42e5b62@mail.gmail.com>

On Sat, Sep 13, 2008 at 6:05 PM, Leonardo Santagada <santagada at gmail.com> wrote:
> I would sugest compiling 2.5 and 2.6 from source, run the benchmark x times
> and take the smallest time of each test (so os and cpu scalling don't
> influence so much the benchmark) and then comparing the results.

I didn't actually run them and pick the smallest, but I did just
compile both from source to keep the environments as close as
possible.

Both compiled from source with Apple's GCC 4.0.1 on a MacBook Pro
(Intel Core 2 Duo, 4GB RAM) running OS X 10.5.4.  Minimal apps
running, plugged in to avoid obvious CPU scaling (I'm sure it drops
when you're on battery).

this = Python 2.6.0rc1
other = Python 2.5.2

Brett

---

Test                             minimum run-time        average  run-time
                                 this    other   diff    this    other   diff
-------------------------------------------------------------------------------
          BuiltinFunctionCalls:   120ms   120ms   +0.2%   121ms   121ms   +0.1%
           BuiltinMethodLookup:    92ms   109ms  -15.7%    93ms   110ms  -15.4%
                 CompareFloats:    87ms    87ms   -0.8%    87ms    88ms   -0.9%
         CompareFloatsIntegers:    91ms    83ms   +9.6%    92ms    84ms   +9.6%
               CompareIntegers:    80ms    81ms   -1.4%    80ms    81ms   -1.5%
        CompareInternedStrings:    88ms    86ms   +2.9%    89ms    87ms   +1.3%
                  CompareLongs:    84ms    78ms   +8.1%    85ms    78ms   +8.0%
                CompareStrings:    69ms    71ms   -2.4%    72ms    74ms   -2.2%
                CompareUnicode:    96ms    96ms   -0.5%    99ms    99ms   -0.2%
    ComplexPythonFunctionCalls:   128ms     0ms     n/a   129ms     0ms     n/a
                 ConcatStrings:   125ms   119ms   +5.5%   129ms   122ms   +5.1%
                 ConcatUnicode:    78ms    70ms  +11.0%    79ms    71ms  +11.9%
               CreateInstances:   136ms   135ms   +1.1%   137ms   136ms   +0.8%
            CreateNewInstances:   102ms   118ms  -13.5%   103ms   118ms  -12.9%
       CreateStringsWithConcat:   111ms   117ms   -4.6%   112ms   118ms   -4.7%
       CreateUnicodeWithConcat:    82ms   122ms  -33.0%    84ms   125ms  -33.3%
                  DictCreation:    90ms    86ms   +4.9%    91ms    91ms   +0.5%
             DictWithFloatKeys:    89ms   107ms  -17.0%    91ms   110ms  -17.8%
           DictWithIntegerKeys:    86ms    85ms   +0.7%    87ms    86ms   +1.6%
            DictWithStringKeys:    79ms    80ms   -0.5%    80ms    81ms   -1.1%
                      ForLoops:    73ms    80ms   -8.6%    75ms    81ms   -7.5%
                    IfThenElse:    77ms    81ms   -4.7%    78ms    82ms   -4.8%
                   ListSlicing:   106ms   139ms  -23.9%   107ms   142ms  -24.2%
                NestedForLoops:   103ms   101ms   +1.5%   106ms   104ms   +1.8%
          NormalClassAttribute:    99ms   118ms  -16.3%   100ms   120ms  -16.7%
       NormalInstanceAttribute:    88ms   107ms  -17.9%    88ms   107ms  -17.6%
           PythonFunctionCalls:    89ms    94ms   -5.1%    90ms    95ms   -4.9%
             PythonMethodCalls:   131ms   135ms   -3.4%   132ms   137ms   -4.0%
                     Recursion:   124ms   128ms   -3.4%   125ms   130ms   -3.7%
                  SecondImport:    92ms    84ms   +9.1%    92ms    85ms   +8.9%
           SecondPackageImport:    97ms    88ms  +10.2%    98ms    89ms   +9.2%
         SecondSubmoduleImport:   125ms   112ms  +11.8%   126ms   113ms  +11.0%
       SimpleComplexArithmetic:   100ms    98ms   +2.4%   101ms    99ms   +1.8%
        SimpleDictManipulation:    88ms    92ms   -4.7%    89ms    94ms   -4.9%
         SimpleFloatArithmetic:    89ms   106ms  -16.2%    91ms   110ms  -16.5%
      SimpleIntFloatArithmetic:    73ms    87ms  -16.1%    73ms    87ms  -16.1%
       SimpleIntegerArithmetic:    73ms    88ms  -17.5%    73ms    89ms  -17.5%
        SimpleListManipulation:    84ms    78ms   +7.3%    85ms    84ms   +1.1%
          SimpleLongArithmetic:   108ms   106ms   +1.9%   109ms   107ms   +1.8%
                    SmallLists:   119ms   120ms   -0.9%   120ms   124ms   -3.1%
                   SmallTuples:   113ms   105ms   +7.7%   115ms   106ms   +7.9%
         SpecialClassAttribute:    96ms   116ms  -17.6%    97ms   118ms  -17.8%
      SpecialInstanceAttribute:   158ms   179ms  -11.9%   159ms   181ms  -12.1%
                StringMappings:   156ms   162ms   -3.8%   156ms   162ms   -3.6%
              StringPredicates:   128ms   153ms  -16.5%   129ms   154ms  -16.3%
                 StringSlicing:   113ms   100ms  +13.1%   121ms   103ms  +17.0%
                     TryExcept:    69ms    72ms   -3.6%    69ms    72ms   -3.8%
                    TryFinally:    96ms     0ms     n/a    97ms     0ms     n/a
                TryRaiseExcept:    98ms   101ms   -2.2%    99ms   101ms   -1.8%
                  TupleSlicing:   136ms   147ms   -7.2%   141ms   148ms   -4.9%
               UnicodeMappings:   112ms    99ms  +14.1%   113ms    99ms  +14.4%
             UnicodePredicates:   105ms   110ms   -5.0%   106ms   111ms   -5.0%
             UnicodeProperties:   106ms   100ms   +5.6%   107ms   101ms   +5.4%
                UnicodeSlicing:    94ms   113ms  -16.9%    99ms   115ms  -14.7%
                   WithFinally:   139ms     0ms     n/a   140ms     0ms     n/a
               WithRaiseExcept:   120ms     0ms     n/a   121ms     0ms     n/a
-------------------------------------------------------------------------------
Totals:                          5691ms  5452ms     n/a  5766ms  5532ms     n/a

(this=2008-09-13 18:57:13, other=../Python-2.5.2/pybench2.5)

From Scott.Daniels at Acm.Org  Sun Sep 14 02:21:36 2008
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Sat, 13 Sep 2008 17:21:36 -0700
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <e6511dbf0809131329l13560b44ic877a929727490d4@mail.gmail.com>
References: <loom.20080913T201341-491@post.gmane.org>
	<e6511dbf0809131329l13560b44ic877a929727490d4@mail.gmail.com>
Message-ID: <gahkmk$4n6$1@ger.gmane.org>

Josiah Carlson wrote:
> On Sat, Sep 13, 2008 at 1:20 PM, Armin Ronacher wrote:
>> Iterating over weak key dictionaries might not be the most common task but I
>> know some situations where this is necessary.  Unfortunately I can't see a
>> way to achieve that in Python 3.
> 
> i = list(d.keys())

Surely
     i = list(d)
is a more reasonable way to do this.  I seldom find a reason
to use .keys

--Scott David Daniels
Scott.Daniels at Acm.Org


From josiah.carlson at gmail.com  Sun Sep 14 02:45:10 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Sat, 13 Sep 2008 17:45:10 -0700
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <gahkmk$4n6$1@ger.gmane.org>
References: <loom.20080913T201341-491@post.gmane.org>
	<e6511dbf0809131329l13560b44ic877a929727490d4@mail.gmail.com>
	<gahkmk$4n6$1@ger.gmane.org>
Message-ID: <e6511dbf0809131745s3ad6fb76qf52a7478c3b8415a@mail.gmail.com>

On Sat, Sep 13, 2008 at 5:21 PM, Scott David Daniels
<Scott.Daniels at acm.org> wrote:
> Josiah Carlson wrote:
>>
>> On Sat, Sep 13, 2008 at 1:20 PM, Armin Ronacher wrote:
>>>
>>> Iterating over weak key dictionaries might not be the most common task
>>> but I
>>> know some situations where this is necessary.  Unfortunately I can't see
>>> a
>>> way to achieve that in Python 3.
>>
>> i = list(d.keys())
>
> Surely
>    i = list(d)
> is a more reasonable way to do this.  I seldom find a reason
> to use .keys

Definitely.  I was being lazy in my use of list(d.keys()) ;)

 - Josiah

From reed at reedobrien.com  Sun Sep 14 03:24:59 2008
From: reed at reedobrien.com (Reed O'Brien)
Date: Sat, 13 Sep 2008 21:24:59 -0400
Subject: [Python-Dev] Possible bug in array?
Message-ID: <B5DB46BC-83A2-4604-B2BB-52100B0C4AAD@reedobrien.com>

or in FreeBSD?

2.6rc1 and 3.0b3 both fail test_array on FreeBSD7 amd64

test_array passes in 2.5.2 on the same machine but fails test_list the  
same as test_array...

*** Signal 9

Not sure if I should file this as a bug or not. I didn't see anything  
on bugs.pyhton.org...

~ro



From musiccomposition at gmail.com  Sun Sep 14 03:26:36 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sat, 13 Sep 2008 20:26:36 -0500
Subject: [Python-Dev] Possible bug in array?
In-Reply-To: <B5DB46BC-83A2-4604-B2BB-52100B0C4AAD@reedobrien.com>
References: <B5DB46BC-83A2-4604-B2BB-52100B0C4AAD@reedobrien.com>
Message-ID: <1afaf6160809131826t13c4cbfes507b0fbc97d58459@mail.gmail.com>

On Sat, Sep 13, 2008 at 8:24 PM, Reed O'Brien <reed at reedobrien.com> wrote:
> or in FreeBSD?
>
> 2.6rc1 and 3.0b3 both fail test_array on FreeBSD7 amd64
>
> test_array passes in 2.5.2 on the same machine but fails test_list the same
> as test_array...
>
> *** Signal 9
>
> Not sure if I should file this as a bug or not. I didn't see anything on
> bugs.pyhton.org...

Please file a bug report. Bug reports (valid or not) tend to get lost
on the list.



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From rhamph at gmail.com  Sun Sep 14 03:52:59 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Sat, 13 Sep 2008 19:52:59 -0600
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <loom.20080913T201341-491@post.gmane.org>
References: <loom.20080913T201341-491@post.gmane.org>
Message-ID: <aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>

On Sat, Sep 13, 2008 at 2:20 PM, Armin Ronacher
<armin.ronacher at active-4.com> wrote:
> Hi everybody,
>
> In Python 2.x when iterating over a weak key dictionary for example, the common
> idom for doing that was calling dictionary.keys() to ensure that a list of all
> objects is returned it was safe to iterate over as a weak reference could stop
> existing during dict iteration which of course raises a runtime error by the
> dict iterator.
>
> This was documented behavior and worked pretty well, with the small problem that
> suddenly all references in the dict wouldn't die until iteration is over because
> the list holds references to the object.
>
> This no longer works in Python 3 because .keys() on the weak key dictionary
> returns a generator over the key view of the internal dict which of course has
> the same problem as iterkeys in Python 2.x.
>
> The following code shows the problem::
>
>    from weakref import WeakKeyDictionary
>
>    f1 = Foo()
>    f2 = Foo()
>    d = WeakKeyDictionary()
>    d[f1] = 42
>    d[f2] = 23
>
>    i = iter(d.keys()) # or use d.keyrefs() here which has the same problem
>    print(next(i))
>    del f2
>    print(next(i))
>
> This example essentially dies with "RuntimeError: dictionary changed
> size during iteration" as soon as f2 is deleted.
>
> Iterating over weak key dictionaries might not be the most common task but I
> know some situations where this is necessary.  Unfortunately I can't see a
> way to achieve that in Python 3.

IMO, this is a deeper problem than suggested.  As far as I know,
python does not (and should not) make promises as to when it'll
collect object.  We should expect weakrefs to be cleared at random
points, and code defensively.

One approach I've used before is to enqueue all the cleared weakrefs,
then process that queue when the WeakKeyDictionary is modified.


-- 
Adam Olsen, aka Rhamphoryncus

From hsoft at hardcoded.net  Sun Sep 14 08:00:45 2008
From: hsoft at hardcoded.net (Virgil Dupras)
Date: Sun, 14 Sep 2008 08:00:45 +0200
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <loom.20080913T201341-491@post.gmane.org>
References: <loom.20080913T201341-491@post.gmane.org>
Message-ID: <DC861C25-2903-4814-991F-BEA1E9804BAC@hardcoded.net>

I would also like to point out that I submitted a patch related to  
that a couple of months ago in:

http://bugs.python.org/issue839159

But it never got any attention :( I'm not sure if it is still relevant.

Virgil




On 13-Sep-08, at 10:20 PM, Armin Ronacher wrote:

> Hi everybody,
>
> In Python 2.x when iterating over a weak key dictionary for example,  
> the common
> idom for doing that was calling dictionary.keys() to ensure that a  
> list of all
> objects is returned it was safe to iterate over as a weak reference  
> could stop
> existing during dict iteration which of course raises a runtime  
> error by the
> dict iterator.
>
> This was documented behavior and worked pretty well, with the small  
> problem that
> suddenly all references in the dict wouldn't die until iteration is  
> over because
> the list holds references to the object.
>
> This no longer works in Python 3 because .keys() on the weak key  
> dictionary
> returns a generator over the key view of the internal dict which of  
> course has
> the same problem as iterkeys in Python 2.x.
>
> The following code shows the problem::
>
>    from weakref import WeakKeyDictionary
>
>    f1 = Foo()
>    f2 = Foo()
>    d = WeakKeyDictionary()
>    d[f1] = 42
>    d[f2] = 23
>
>    i = iter(d.keys()) # or use d.keyrefs() here which has the same  
> problem
>    print(next(i))
>    del f2
>    print(next(i))
>
> This example essentially dies with "RuntimeError: dictionary changed
> size during iteration" as soon as f2 is deleted.
>
> Iterating over weak key dictionaries might not be the most common  
> task but I
> know some situations where this is necessary.  Unfortunately I can't  
> see a
> way to achieve that in Python 3.
>
> Regards,
> Armin
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/hsoft%40hardcoded.net


From ncoghlan at gmail.com  Sun Sep 14 08:43:57 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Sep 2008 16:43:57 +1000
Subject: [Python-Dev] Do we still support MacOS < X?
In-Reply-To: <loom.20080913T194843-752@post.gmane.org>
References: <gag2cc$6jg$1@ger.gmane.org>	<1afaf6160809130824o7e86afe8g2ac80286d5b6de83@mail.gmail.com>
	<loom.20080913T194843-752@post.gmane.org>
Message-ID: <48CCB2AD.6050302@gmail.com>

Armin Ronacher wrote:
> Hi,
> 
> Benjamin Peterson <musiccomposition <at> gmail.com> writes:
>> Yes, according to PEP 11, support was removed in 2.4. I suppose this
>> also means we could killed macpath in py3k...
> Pleaes don't do that.  The OS 9 path handling is still present in OS X GUI
> applications.  While you don't see it that often because OS X doesn't know
> the location bars but you can see it in iTunes for example.

As far as I can recall, the fact that there are remnants of MacOS paths
lying around in OS X is actually the reason macpath was salvaged when
all of the other MacOS specific modules were removed from 3.0.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Sun Sep 14 11:07:48 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Sep 2008 19:07:48 +1000
Subject: [Python-Dev] [Python-checkins] Python Regression Test Failures
	basics (1)
In-Reply-To: <20080914083615.GA7037@python.psfb.org>
References: <20080914083615.GA7037@python.psfb.org>
Message-ID: <48CCD464.8080106@gmail.com>

Neal Norwitz wrote:
> test_epoll skipped -- kernel doesn't support epoll()
...
> test_ioctl skipped -- Unable to open /dev/tty
...
> test_multiprocessing skipped -- OSError raises on RLock creation, see
issue 3111!
...
> test test_normalization failed -- Traceback (most recent call last):
>   File "/tmp/python-test/local/lib/python2.6/test/test_normalization.py", line 90, in test_main
>     self.failUnless(X == NFC(X) == NFD(X) == NFKC(X) == NFKD(X), c)
> AssertionError: 6918
...
> 326 tests OK.
> 1 test failed:
>     test_normalization
...
> 3 skips unexpected on linux2:
>    test_epoll test_multiprocessing test_ioctl

Neal,

What environment are you using to run the debug-mode regression tests?
The above four tests run without any problems for me, but I'm just
running them in a normal Kubuntu desktop shell.

Cheers,
Nick.

P.S. I'm in the process of creating a --with-pydebug build to see if
that makes any difference.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From armin.ronacher at active-4.com  Sun Sep 14 11:35:52 2008
From: armin.ronacher at active-4.com (Armin Ronacher)
Date: Sun, 14 Sep 2008 09:35:52 +0000 (UTC)
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
References: <loom.20080913T201341-491@post.gmane.org>
	<e6511dbf0809131329l13560b44ic877a929727490d4@mail.gmail.com>
Message-ID: <loom.20080914T093423-487@post.gmane.org>

Hi,

Josiah Carlson <josiah.carlson <at> gmail.com> writes:

> i = list(d.keys())
Obviously that doesn't solve the problem.  list() consumes the generator
one after another, objects can still die when the list is created.  Imagine
the following example which uses threads::

    from time import sleep
    from weakref import WeakKeyDictionary
    from threading import Thread

    class Foo(object):
        pass

    d = WeakKeyDictionary()
    l = []

    for x in range(100000):
        obj = Foo()
        d[obj] = None
        l.append(obj)
    del obj


    def iterater():
        for item in list(d.keys()):
            pass

    Thread(target=iterater).start()

    while True:
        del l[0]

Regards,
Armin


From armin.ronacher at active-4.com  Sun Sep 14 11:39:31 2008
From: armin.ronacher at active-4.com (Armin Ronacher)
Date: Sun, 14 Sep 2008 09:39:31 +0000 (UTC)
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
References: <loom.20080913T201341-491@post.gmane.org>
	<aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>
Message-ID: <loom.20080914T093612-536@post.gmane.org>

Hi,

Adam Olsen <rhamph <at> gmail.com> writes:

> IMO, this is a deeper problem than suggested.  As far as I know,
> python does not (and should not) make promises as to when it'll
> collect object.  We should expect weakrefs to be cleared at random
> points, and code defensively.
It doesn't promise when objects are collected and that's not the problem
here.  The main problem is that the old solution for weakrefs relayed on
the fact that .keys() was considered atomic.  I don't say it has to
become again, but the weak dictionaries have to somehow counter that
problem.  They could for example only remove items from the internal dict
if no dict view to that dict is alive.

Speaking of atom keys() / values() / items() operations: I guess we will
see more of those problems in threaded situations when people start to
convert code over to Python.  I've seen quite a few situations where code
relays on keys() holding the interpreter lock.

Regards,
Armin


From ncoghlan at gmail.com  Sun Sep 14 13:59:39 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Sep 2008 21:59:39 +1000
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <loom.20080914T093612-536@post.gmane.org>
References: <loom.20080913T201341-491@post.gmane.org>	<aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>
	<loom.20080914T093612-536@post.gmane.org>
Message-ID: <48CCFCAB.4070809@gmail.com>

Armin Ronacher wrote:
> Speaking of atom keys() / values() / items() operations: I guess we will
> see more of those problems in threaded situations when people start to
> convert code over to Python.  I've seen quite a few situations where code
> relays on keys() holding the interpreter lock.

list(iter) doesn't re-enter the eval loop if the iterator is written in
C and hence won't let go of the GIL. As I believe the dict views in 3.0
are all C classes, I'd like to see a failing test case along these lines
that doesn't involve a dict subclass and code written in Python.

Note that I'm not saying that I know for certain that there aren't any
problems with list(d.keys()) that aren't seen with the direct list
conversion in 2.x - I'm just saying it may not be as easy to provoke
such problems as it initially appears due to the way the list
constructor interacts with the supplied iterable, so it is going to take
an actual failing test case to convince me.

Cheers,
Nick.



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From musiccomposition at gmail.com  Sun Sep 14 14:24:00 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sun, 14 Sep 2008 07:24:00 -0500
Subject: [Python-Dev] [Python-checkins] Python Regression Test Failures
	basics (1)
In-Reply-To: <48CCD464.8080106@gmail.com>
References: <20080914083615.GA7037@python.psfb.org>
	<48CCD464.8080106@gmail.com>
Message-ID: <1afaf6160809140524t3b5f91acufb0ca630a20e25fc@mail.gmail.com>

On Sun, Sep 14, 2008 at 4:07 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Neal Norwitz wrote:
>> test_epoll skipped -- kernel doesn't support epoll()
> ...
>> test_ioctl skipped -- Unable to open /dev/tty
> ...
>> test_multiprocessing skipped -- OSError raises on RLock creation, see
> issue 3111!
> ...
>> test test_normalization failed -- Traceback (most recent call last):
>>   File "/tmp/python-test/local/lib/python2.6/test/test_normalization.py", line 90, in test_main
>>     self.failUnless(X == NFC(X) == NFD(X) == NFKC(X) == NFKD(X), c)
>> AssertionError: 6918
> ...
>> 326 tests OK.
>> 1 test failed:
>>     test_normalization
> ...
>> 3 skips unexpected on linux2:
>>    test_epoll test_multiprocessing test_ioctl
>
> Neal,
>
> What environment are you using to run the debug-mode regression tests?
> The above four tests run without any problems for me, but I'm just
> running them in a normal Kubuntu desktop shell.

Something in Neal's build which has made a difference before is that
the tests are run after a "make install".

>
> Cheers,
> Nick.
>
> P.S. I'm in the process of creating a --with-pydebug build to see if
> that makes any difference.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> ---------------------------------------------------------------
>            http://www.boredomandlaziness.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From g.brandl at gmx.net  Sun Sep 14 14:28:00 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 14 Sep 2008 14:28:00 +0200
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <48CCFCAB.4070809@gmail.com>
References: <loom.20080913T201341-491@post.gmane.org>	<aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>	<loom.20080914T093612-536@post.gmane.org>
	<48CCFCAB.4070809@gmail.com>
Message-ID: <gaj00v$qa9$1@ger.gmane.org>

Nick Coghlan schrieb:
> Armin Ronacher wrote:
>> Speaking of atom keys() / values() / items() operations: I guess we will
>> see more of those problems in threaded situations when people start to
>> convert code over to Python.  I've seen quite a few situations where code
>> relays on keys() holding the interpreter lock.
> 
> list(iter) doesn't re-enter the eval loop if the iterator is written in
> C and hence won't let go of the GIL. As I believe the dict views in 3.0
> are all C classes, I'd like to see a failing test case along these lines
> that doesn't involve a dict subclass and code written in Python.

WeakKeyDictionary.keys() iterates over its .data dictionary's keys() instead
of list(data.keys()). Therefore, the user of WeakKeyDictionary has no chance
to do list(keys()) without running Python code.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From ncoghlan at gmail.com  Sun Sep 14 15:03:20 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 14 Sep 2008 23:03:20 +1000
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <gaj00v$qa9$1@ger.gmane.org>
References: <loom.20080913T201341-491@post.gmane.org>	<aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>	<loom.20080914T093612-536@post.gmane.org>	<48CCFCAB.4070809@gmail.com>
	<gaj00v$qa9$1@ger.gmane.org>
Message-ID: <48CD0B98.1070808@gmail.com>

Georg Brandl wrote:
> Nick Coghlan schrieb:
>> Armin Ronacher wrote:
>>> Speaking of atom keys() / values() / items() operations: I guess we will
>>> see more of those problems in threaded situations when people start to
>>> convert code over to Python.  I've seen quite a few situations where code
>>> relays on keys() holding the interpreter lock.
>> list(iter) doesn't re-enter the eval loop if the iterator is written in
>> C and hence won't let go of the GIL. As I believe the dict views in 3.0
>> are all C classes, I'd like to see a failing test case along these lines
>> that doesn't involve a dict subclass and code written in Python.
> 
> WeakKeyDictionary.keys() iterates over its .data dictionary's keys() instead
> of list(data.keys()). Therefore, the user of WeakKeyDictionary has no chance
> to do list(keys()) without running Python code.

Ouch, there's the dict subclass with Python code that I was asking for
before I got concerned (I forgot that the weakref module is only
partially implemented in C).

I agree this is a problem, but I'm struggling to think of possible
solutions that aren't either horrendously complicated or completely
contrary to the idea of view-based dicts.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From g.brandl at gmx.net  Sun Sep 14 15:46:32 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 14 Sep 2008 15:46:32 +0200
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <48CD0B98.1070808@gmail.com>
References: <loom.20080913T201341-491@post.gmane.org>	<aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>	<loom.20080914T093612-536@post.gmane.org>	<48CCFCAB.4070809@gmail.com>	<gaj00v$qa9$1@ger.gmane.org>
	<48CD0B98.1070808@gmail.com>
Message-ID: <gaj4k7$6qv$1@ger.gmane.org>

Nick Coghlan schrieb:
> Georg Brandl wrote:
>> Nick Coghlan schrieb:
>>> Armin Ronacher wrote:
>>>> Speaking of atom keys() / values() / items() operations: I guess we will
>>>> see more of those problems in threaded situations when people start to
>>>> convert code over to Python.  I've seen quite a few situations where code
>>>> relays on keys() holding the interpreter lock.
>>> list(iter) doesn't re-enter the eval loop if the iterator is written in
>>> C and hence won't let go of the GIL. As I believe the dict views in 3.0
>>> are all C classes, I'd like to see a failing test case along these lines
>>> that doesn't involve a dict subclass and code written in Python.
>> 
>> WeakKeyDictionary.keys() iterates over its .data dictionary's keys() instead
>> of list(data.keys()). Therefore, the user of WeakKeyDictionary has no chance
>> to do list(keys()) without running Python code.
> 
> Ouch, there's the dict subclass with Python code that I was asking for
> before I got concerned (I forgot that the weakref module is only
> partially implemented in C).
> 
> I agree this is a problem, but I'm struggling to think of possible
> solutions that aren't either horrendously complicated or completely
> contrary to the idea of view-based dicts.

In this case, what WeakKeyDictionary.keys() returns is not a view anyway,
so using list(self.data.keys()) in the method seems like a good compromise.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From solipsis at pitrou.net  Sun Sep 14 15:48:05 2008
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 14 Sep 2008 13:48:05 +0000 (UTC)
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
References: <loom.20080913T201341-491@post.gmane.org>	<aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>	<loom.20080914T093612-536@post.gmane.org>	<48CCFCAB.4070809@gmail.com>
	<gaj00v$qa9$1@ger.gmane.org> <48CD0B98.1070808@gmail.com>
Message-ID: <loom.20080914T134113-267@post.gmane.org>

Nick Coghlan <ncoghlan <at> gmail.com> writes:
> 
> I agree this is a problem, but I'm struggling to think of possible
> solutions that aren't either horrendously complicated or completely
> contrary to the idea of view-based dicts.

Keeping a count of the number of views currently alive for a given weak dict
doesn't seem "horrendously complicated" (*). Once this count falls to zero, you
can purge all the dead weakrefs from the dict (their keys would have been
previously queued, so that purging is O(dead weakrefs) and not O(dict length)).

(*) the most complicated is probably to write the view classes themselves



From nnorwitz at gmail.com  Sun Sep 14 19:53:20 2008
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Sun, 14 Sep 2008 10:53:20 -0700
Subject: [Python-Dev] [Python-checkins] Python Regression Test Failures
	basics (1)
In-Reply-To: <1afaf6160809140524t3b5f91acufb0ca630a20e25fc@mail.gmail.com>
References: <20080914083615.GA7037@python.psfb.org>
	<48CCD464.8080106@gmail.com>
	<1afaf6160809140524t3b5f91acufb0ca630a20e25fc@mail.gmail.com>
Message-ID: <ee2a432c0809141053o45809460j51a92bf3c8c0a16@mail.gmail.com>

On Sun, Sep 14, 2008 at 5:24 AM, Benjamin Peterson
<musiccomposition at gmail.com> wrote:
> On Sun, Sep 14, 2008 at 4:07 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Neal Norwitz wrote:
>>> test_epoll skipped -- kernel doesn't support epoll()
>> ...
>>> test_ioctl skipped -- Unable to open /dev/tty
>> ...
>>> test_multiprocessing skipped -- OSError raises on RLock creation, see
>> issue 3111!
>> ...
>>> test test_normalization failed -- Traceback (most recent call last):
>>>   File "/tmp/python-test/local/lib/python2.6/test/test_normalization.py", line 90, in test_main
>>>     self.failUnless(X == NFC(X) == NFD(X) == NFKC(X) == NFKD(X), c)
>>> AssertionError: 6918
>> ...
>>> 326 tests OK.
>>> 1 test failed:
>>>     test_normalization
>> ...
>>> 3 skips unexpected on linux2:
>>>    test_epoll test_multiprocessing test_ioctl
>>
>> Neal,
>>
>> What environment are you using to run the debug-mode regression tests?
>> The above four tests run without any problems for me, but I'm just
>> running them in a normal Kubuntu desktop shell.
>
> Something in Neal's build which has made a difference before is that
> the tests are run after a "make install".

Benjamin is correct.  See Misc/build.sh for the script that generates
this.  See http://docs.python.org/dev/results/ for details about the
most recent run.

n

From ctb at msu.edu  Sun Sep 14 21:31:01 2008
From: ctb at msu.edu (C. Titus Brown)
Date: Sun, 14 Sep 2008 12:31:01 -0700
Subject: [Python-Dev] ',' precedence in documentation
Message-ID: <20080914193101.GB30564@idyll.org>

Hi folks,

over on the pygr list, we were recently discussing a mildly confusing
edit I made:

   assert 'seq1' in self.db, self.db.keys() 

This was interpreted by someone as being

   assert 'seq1' in (self.db, self.db.keys())

which is very different from the actual meaning,

   assert ('seq1' in self.db), self.db.keys()

where 'self.db.keys()' serves as the message to be printed out when the
assertion fails.  Apart from questions of why I was apparently out to
confuse people with this edit, the question of ',' precedence came up.
Looking at

	http://docs.python.org/ref/summary.html

there's no mention of ',' binding and tuple creation vs the other
operators.  I'm not really sure how to think of it but it seems like
this is something that could be cleared up in the docs with a brief note
somewhere.  Thoughts?

cheers,
--titus
-- 
C. Titus Brown, ctb at msu.edu

From fredrik at pythonware.com  Sun Sep 14 21:47:19 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Sun, 14 Sep 2008 21:47:19 +0200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <20080914193101.GB30564@idyll.org>
References: <20080914193101.GB30564@idyll.org>
Message-ID: <gajpob$2n6$1@ger.gmane.org>

C. Titus Brown wrote:

> over on the pygr list, we were recently discussing a mildly confusing
> edit I made:
> 
>    assert 'seq1' in self.db, self.db.keys() 
> 
> This was interpreted by someone as being
> 
>    assert 'seq1' in (self.db, self.db.keys())
> 
> which is very different from the actual meaning,
> 
>    assert ('seq1' in self.db), self.db.keys()
> 
> where 'self.db.keys()' serves as the message to be printed out when the
> assertion fails.  Apart from questions of why I was apparently out to
> confuse people with this edit, the question of ',' precedence came up.
> Looking at
> 
> 	http://docs.python.org/ref/summary.html
> 
> there's no mention of ',' binding and tuple creation vs the other
> operators.

A bare "," is part of the "expression list" syntax; it's not an operator:

     http://docs.python.org/ref/exprlists.html

You have to look at the statement descriptions to find out whether a 
statement wants an expression or an expression list (e.g. "return" takes 
an expression list, while "assert" takes two expressions, not two 
expression lists).

</F>


From guido at python.org  Sun Sep 14 22:00:14 2008
From: guido at python.org (Guido van Rossum)
Date: Sun, 14 Sep 2008 13:00:14 -0700
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <gajpob$2n6$1@ger.gmane.org>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
Message-ID: <ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>

On Sun, Sep 14, 2008 at 12:47 PM, Fredrik Lundh <fredrik at pythonware.com> wrote:
> C. Titus Brown wrote:
>
>> over on the pygr list, we were recently discussing a mildly confusing
>> edit I made:
>>
>>   assert 'seq1' in self.db, self.db.keys()
>> This was interpreted by someone as being
>>
>>   assert 'seq1' in (self.db, self.db.keys())
>>
>> which is very different from the actual meaning,
>>
>>   assert ('seq1' in self.db), self.db.keys()
>>
>> where 'self.db.keys()' serves as the message to be printed out when the
>> assertion fails.  Apart from questions of why I was apparently out to
>> confuse people with this edit, the question of ',' precedence came up.
>> Looking at
>>
>>        http://docs.python.org/ref/summary.html
>>
>> there's no mention of ',' binding and tuple creation vs the other
>> operators.
>
> A bare "," is part of the "expression list" syntax; it's not an operator:
>
>    http://docs.python.org/ref/exprlists.html
>
> You have to look at the statement descriptions to find out whether a
> statement wants an expression or an expression list (e.g. "return" takes an
> expression list, while "assert" takes two expressions, not two expression
> lists).

Note that in any case, the 'in' operator binds more tightly than the
comma: e.g. f(x in y, z) means f((x in y), z). The only seeming
exception to this rule is the 'for' statement (and list comprehensions
and generator expressions) where 'in' is not an expression operator
but part of the statement syntax.

So if someone thought

  assert 'seq1' in self.db, self.db.keys()

meant

  assert 'seq1' in (self.db, self.db.keys())

they'd be in for a nasty surprise trying the same trick in an 'if'
statement, like this:

  if 'seq1' in self.db, self.db.keys(): ...

Fortunately that raises a syntax error. The really bad surprises come
from things like

  assert x in A, x in B

which I've seen assumed to mean

  assert (x in A) and (x in B)

in claimed analogy to the use of the comma in the print statement.

I think in general Python has erred on the side of having too many
different syntactical uses for commas. We killed a few in 3.0 with the
introduction of "except E as v" and the demotion of print to a
function call. Perhaps we should aim to kill "assert B, S" as well?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From musiccomposition at gmail.com  Sun Sep 14 22:06:09 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sun, 14 Sep 2008 15:06:09 -0500
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
Message-ID: <1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>

On Sun, Sep 14, 2008 at 3:00 PM, Guido van Rossum <guido at python.org> wrote:
>
> I think in general Python has erred on the side of having too many
> different syntactical uses for commas. We killed a few in 3.0 with the
> introduction of "except E as v" and the demotion of print to a
> function call. Perhaps we should aim to kill "assert B, S" as well?

And replace it with what?

>
> --
> --Guido van Rossum (home page: http://www.python.org/~guido/)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From gnewsg at gmail.com  Sun Sep 14 22:16:26 2008
From: gnewsg at gmail.com (Giampaolo Rodola')
Date: Sun, 14 Sep 2008 13:16:26 -0700 (PDT)
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
Message-ID: <0a7adf05-7811-4078-8a90-3830f0974dab@d77g2000hsb.googlegroups.com>

Hi,
I try to revamp a discussion I raised some time ago and which seems to
be not solved yet.
I'm interested in using the ssl module with asyncore but since there's
no real documentation about how using ssl module with non-blocking
sockets I've not been able to write something useful with it.
I took a look at Test/test_ssl.py which seems to include a test
consisting in an asyncore-based secure server but I noticed something
strange:

                def __init__(self, conn, certfile):
                    asyncore.dispatcher_with_send.__init__(self, conn)
                    self.socket = ssl.wrap_socket(conn,
server_side=True,
                                                  certfile=certfile,
 
do_handshake_on_connect=True)

I'm not sure about the real meaning of the do_handshake_on_connect
parameter but if it does what I think it should block the application
as long as the handshake isn't finished, which is not acceptable in an
asynchronous environment.
Am I misunderstanding something?
Does the ssl module can actually be used with non-blocking sockets? If
so which is the proper/recommended way to do that?


Thanks in advance


--- Giampaolo
http://code.google.com/p/pyftpdlib/

From brett at python.org  Sun Sep 14 22:36:26 2008
From: brett at python.org (Brett Cannon)
Date: Sun, 14 Sep 2008 13:36:26 -0700
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
Message-ID: <bbaeab100809141336o1cdb0d13he3e4d6cf8d35b119@mail.gmail.com>

On Sun, Sep 14, 2008 at 1:06 PM, Benjamin Peterson
<musiccomposition at gmail.com> wrote:
> On Sun, Sep 14, 2008 at 3:00 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> I think in general Python has erred on the side of having too many
>> different syntactical uses for commas. We killed a few in 3.0 with the
>> introduction of "except E as v" and the demotion of print to a
>> function call. Perhaps we should aim to kill "assert B, S" as well?
>
> And replace it with what?
>

That's the question being posed. =)

A quick browsing of the grammar shows us already using 'in', 'as',
'from', and 'else' as keywords that are keywords which are not
statements as their own ('in' is unique, as Guido pointed out because
of its use with 'for' and as an operator).

Something like ``assert _expr_ else _message_`` might work, but I am
not sure if people will misunderstand the meaning of 'else'. But if
does somewhat match the usage with 'if' expressions as the expression
following is not executed if the preceding expression is true.

-Brett

From martin at v.loewis.de  Sun Sep 14 22:45:39 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 14 Sep 2008 22:45:39 +0200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org>
	<gajpob$2n6$1@ger.gmane.org>	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
Message-ID: <48CD77F3.7020009@v.loewis.de>

> And replace it with what?

I'm sure this will evolve into a long discussion on syntax, so
I'll start: assert could become a function with one or two
arguments. The arguments would always get evaluated; the actual
exception-raising could still be conditional (perhaps through
a context manager).

Regards,
Martin

From ctb at msu.edu  Sun Sep 14 22:52:27 2008
From: ctb at msu.edu (C. Titus Brown)
Date: Sun, 14 Sep 2008 13:52:27 -0700
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CD77F3.7020009@v.loewis.de>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
	<48CD77F3.7020009@v.loewis.de>
Message-ID: <20080914205225.GA15493@idyll.org>

On Sun, Sep 14, 2008 at 10:45:39PM +0200, "Martin v. L?wis" wrote:
-> > And replace it with what?
-> 
-> I'm sure this will evolve into a long discussion on syntax, so
-> I'll start: assert could become a function with one or two
-> arguments. The arguments would always get evaluated; the actual
-> exception-raising could still be conditional (perhaps through
-> a context manager).

Doesn't this change the behavior in optimized code, for assert
statements with side effects?  If we don't have specialized syntax --
that is, if 'assert' just becomes another built-in function -- then it
can't be special-cased in the translation to byte-code, can it?

--titus
-- 
C. Titus Brown, ctb at msu.edu

From martin at v.loewis.de  Sun Sep 14 22:59:23 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 14 Sep 2008 22:59:23 +0200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <20080914205225.GA15493@idyll.org>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
	<48CD77F3.7020009@v.loewis.de> <20080914205225.GA15493@idyll.org>
Message-ID: <48CD7B2B.2090000@v.loewis.de>

> Doesn't this change the behavior in optimized code, for assert
> statements with side effects?

That's why I said it will be a long discussion :-)

I would also propose the elimination of the -O option, and the notion
of optimized code. People who want to save the time for execution of the
assert statements could use some kind of source transformation, which
might be easy to implement with lib2to3 (or a generalization thereof).

> If we don't have specialized syntax --
> that is, if 'assert' just becomes another built-in function -- then it
> can't be special-cased in the translation to byte-code, can it?

It still could. First, assert may continue to be a statement, even
though it is invoked as a function call, similar to None, True, False.
In that case, the compiler could still eliminate it, or (IMO preferred),
wrap it into a conditional statement that checks whether assertions are
disabled, and, if so, jumps over the assertion.

Even if it isn't a keyword, the compiler could still special-case it,
and add a run-time check to determine whether assert is still bound to
the built-in assert.

Regards,
Martin

From guido at python.org  Sun Sep 14 23:00:45 2008
From: guido at python.org (Guido van Rossum)
Date: Sun, 14 Sep 2008 14:00:45 -0700
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
Message-ID: <ca471dc20809141400i3b87e5fcucd5605c436397067@mail.gmail.com>

On Sun, Sep 14, 2008 at 1:06 PM, Benjamin Peterson
<musiccomposition at gmail.com> wrote:
> On Sun, Sep 14, 2008 at 3:00 PM, Guido van Rossum <guido at python.org> wrote:
>>
>> I think in general Python has erred on the side of having too many
>> different syntactical uses for commas. We killed a few in 3.0 with the
>> introduction of "except E as v" and the demotion of print to a
>> function call. Perhaps we should aim to kill "assert B, S" as well?
>
> And replace it with what?

I have no idea. Use your imagination. :-)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From fredrik at pythonware.com  Sun Sep 14 23:01:38 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Sun, 14 Sep 2008 23:01:38 +0200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CD7B2B.2090000@v.loewis.de>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
	<48CD77F3.7020009@v.loewis.de> <20080914205225.GA15493@idyll.org>
	<48CD7B2B.2090000@v.loewis.de>
Message-ID: <368a5cd50809141401h2f2607e5vee61848a2720756@mail.gmail.com>

> That's why I said it will be a long discussion :-)

if so, could you folks perhaps keep that discussion strictly on
python-dev at python.org?

</F>

From ctb at msu.edu  Sun Sep 14 23:04:39 2008
From: ctb at msu.edu (C. Titus Brown)
Date: Sun, 14 Sep 2008 14:04:39 -0700
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CD7B2B.2090000@v.loewis.de>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
	<48CD77F3.7020009@v.loewis.de> <20080914205225.GA15493@idyll.org>
	<48CD7B2B.2090000@v.loewis.de>
Message-ID: <20080914210439.GD15493@idyll.org>

On Sun, Sep 14, 2008 at 10:59:23PM +0200, "Martin v. L?wis" wrote:
-> > Doesn't this change the behavior in optimized code, for assert
-> > statements with side effects?
-> 
-> That's why I said it will be a long discussion :-)
-> 
-> I would also propose the elimination of the -O option, and the notion
-> of optimized code. People who want to save the time for execution of the
-> assert statements could use some kind of source transformation, which
-> might be easy to implement with lib2to3 (or a generalization thereof).

I would very much like this.  I think it solves some problems in the
unittest-refactoring area, too, where we have a proliferation of
'assert*' functions that *don't* get removed by optimization.  (Hey,
does anyone want to discuss what color THAT bikeshed roof should be some
more? ;)  In any case, I would still like to be able to test my
optimized code by using assert, which isn't possible currently.

cheers,
--titus
-- 
C. Titus Brown, ctb at msu.edu

From fuzzyman at voidspace.org.uk  Sun Sep 14 22:51:26 2008
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sun, 14 Sep 2008 21:51:26 +0100
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CD77F3.7020009@v.loewis.de>
References: <20080914193101.GB30564@idyll.org>	<gajpob$2n6$1@ger.gmane.org>	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
	<48CD77F3.7020009@v.loewis.de>
Message-ID: <48CD794E.4000701@voidspace.org.uk>

Martin v. L?wis wrote:
>> And replace it with what?
>>     
>
> I'm sure this will evolve into a long discussion on syntax, so
> I'll start: assert could become a function with one or two
> arguments. The arguments would always get evaluated; the actual
> exception-raising could still be conditional (perhaps through
> a context manager).
>   
+1 for a function
+0.5 for the else suggestion

Michael

> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/
http://www.trypython.org/
http://www.ironpython.info/
http://www.theotherdelia.co.uk/
http://www.resolverhacks.net/


From leec at chem.ucla.edu  Sun Sep 14 23:01:15 2008
From: leec at chem.ucla.edu (Christopher Lee)
Date: Sun, 14 Sep 2008 14:01:15 -0700
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <20080914200327.GE30564@idyll.org>
References: <20080914200327.GE30564@idyll.org>
Message-ID: <777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>


> ----- Forwarded message from Guido van Rossum <guido at python.org> -----
>
> Note that in any case, the 'in' operator binds more tightly than the
> comma: e.g. f(x in y, z) means f((x in y), z).

Exactly the point I was trying to make.  As far as the user's  
concerned, an expression list
x,y
can itself be part of an expression;
x in y
is also an expression.  Such expressions can be combined to form  
larger expressions, so it follows logically that you need to know  
which has precedence when *both* occur in an expression, e.g. whether
x in y,z
evaluates as
(x in y),z
or
x in (y,z)

The page that Fredrik sent you to (http://docs.python.org/ref/exprlists.html 
) doesn't address that question.  I still think the precedence table (http://docs.python.org/ref/summary.html 
) should show that "in" has higher precedence than comma in an  
expression.

Can anyone show us "where it is written" in the Python docs that "in"  
has higher precedence than comma, *and* why there is a good reason  
that this information should NOT be included in the precedence table?

-- Chris

Adding comma to the precedence table would also disambiguate  
expressions like:
x and y,z
x,y or z
not x,y



From greg.ewing at canterbury.ac.nz  Mon Sep 15 03:08:40 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 15 Sep 2008 13:08:40 +1200
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <loom.20080914T093612-536@post.gmane.org>
References: <loom.20080913T201341-491@post.gmane.org>
	<aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>
	<loom.20080914T093612-536@post.gmane.org>
Message-ID: <48CDB598.1050506@canterbury.ac.nz>

My thought on all this is to implement WeakKeyDictionary as
a C extension and have it atomically construct and return
a list for keys().

While this would not be exactly the same behaviour as an
ordinary dict, I don't expect that much code would notice
the difference.

-- 
Greg


From greg.ewing at canterbury.ac.nz  Mon Sep 15 03:17:45 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 15 Sep 2008 13:17:45 +1200
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <gaj4k7$6qv$1@ger.gmane.org>
References: <loom.20080913T201341-491@post.gmane.org>
	<aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>
	<loom.20080914T093612-536@post.gmane.org> <48CCFCAB.4070809@gmail.com>
	<gaj00v$qa9$1@ger.gmane.org> <48CD0B98.1070808@gmail.com>
	<gaj4k7$6qv$1@ger.gmane.org>
Message-ID: <48CDB7B9.3000005@canterbury.ac.nz>

Georg Brandl wrote:

> In this case, what WeakKeyDictionary.keys() returns is not a view anyway,
> so using list(self.data.keys()) in the method seems like a good compromise.

I don't think that helps, because the implementation has
to iterate over that to dereference the weak refs, and if
that's done in Python code there's a chance for a weak
ref to disappear in the process.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon Sep 15 03:27:30 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 15 Sep 2008 13:27:30 +1200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
Message-ID: <48CDBA02.9040802@canterbury.ac.nz>

Guido van Rossum wrote:

> Perhaps we should aim to kill "assert B, S" as well?

Maybe

   assert B else S

?
  --
Greg


From greg.ewing at canterbury.ac.nz  Mon Sep 15 03:38:58 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 15 Sep 2008 13:38:58 +1200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CD7B2B.2090000@v.loewis.de>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
	<48CD77F3.7020009@v.loewis.de> <20080914205225.GA15493@idyll.org>
	<48CD7B2B.2090000@v.loewis.de>
Message-ID: <48CDBCB2.8080506@canterbury.ac.nz>

Martin v. L?wis wrote:

> Even if it isn't a keyword, the compiler could still special-case it,
> and add a run-time check to determine whether assert is still bound to
> the built-in assert.

Then we would have something that looks exactly like
an ordinary function call but actually isn't. I'm
not sure that's a good idea.

Also there are issues such as what happens if you
shadow it.

-- 
Greg

From janssen at parc.com  Mon Sep 15 04:50:15 2008
From: janssen at parc.com (Bill Janssen)
Date: Sun, 14 Sep 2008 19:50:15 PDT
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <0a7adf05-7811-4078-8a90-3830f0974dab@d77g2000hsb.googlegroups.com>
References: <0a7adf05-7811-4078-8a90-3830f0974dab@d77g2000hsb.googlegroups.com>
Message-ID: <6896.1221447015@parc.com>

Hi, Giampaolo.

If you look a bit further in Lib/test/test_ssl.py, you'll see a
non-blocking use of the "do_handshake" method.  Basically, the flag
"do_handshake_on_connect" says whether you want this to happen
automatically and blockingly (True), or whether you want to do it
yourself (False).  In the test suite, the function
"testNonBlockingHandshake" does the async client-side handshake; the
server side logic is just the same, only it would happen in the server's
"handle new connection" code -- you'd have to add a state variable, and
bind handlers for "read_event" and "write_event", which would consult
the state variable to see whether they had finished the handshake yet.

I just made the server do it automatically to make life easier.

The hard part isn't really doing the non-blocking, it's trying to figure
out how to use asyncore correctly, IMO.

Giampaolo Rodola' <gnewsg at gmail.com> wrote:

> I'm interested in using the ssl module with asyncore but since there's
> no real documentation about how using ssl module with non-blocking

If you'd like to contribute a doc patch, that would be great.

Here's what it current says for do_handshake_on_connect:

  The parameter do_handshake_on_connect specifies whether to do the SSL
  handshake automatically after doing a socket.connect(), or whether the
  application program will call it explicitly, by invoking the
  SSLSocket.do_handshake() method. Calling SSLSocket.do_handshake()
  explicitly gives the program control over the blocking behavior of the
  socket I/O involved in the handshake.

and here's what the docs for do_handshake() says:

  SSLSocket.do_handshake()? Perform a TLS/SSL handshake. If this is used
  with a non-blocking socket, it may raise SSLError with an arg[0] of
  SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, in which case it must be
  called again until it completes successfully. For example, to simulate
  the behavior of a blocking socket, one might write:

    while True:
        try:
            s.do_handshake()
            break
        except ssl.SSLError, err:
            if err.args[0] == ssl.SSL_ERROR_WANT_READ:
                select.select([s], [], [])
            elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE:
                select.select([], [s], [])
            else:
                raise

Bill

From eric at trueblade.com  Mon Sep 15 05:36:57 2008
From: eric at trueblade.com (Eric Smith)
Date: Sun, 14 Sep 2008 23:36:57 -0400
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CDBA02.9040802@canterbury.ac.nz>
References: <20080914193101.GB30564@idyll.org>
	<gajpob$2n6$1@ger.gmane.org>	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<48CDBA02.9040802@canterbury.ac.nz>
Message-ID: <48CDD859.90407@trueblade.com>

Greg Ewing wrote:
> Guido van Rossum wrote:
> 
>> Perhaps we should aim to kill "assert B, S" as well?
> 
> Maybe
> 
>   assert B else S
> 
> ?

If we really want to change it, I think:
   assert B as S
is better because S is the string to report; that is, "if B is false, 
report the problem as the string S".

'else' implies to me what to do if you're not failing the assert, which 
is not the case.

Personally I'm not convinced the syntax is so unusual as it currently 
stands.

From leif.walsh at gmail.com  Mon Sep 15 05:55:00 2008
From: leif.walsh at gmail.com (Leif Walsh)
Date: Sun, 14 Sep 2008 23:55:00 -0400
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CDD859.90407@trueblade.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<48CDBA02.9040802@canterbury.ac.nz> <48CDD859.90407@trueblade.com>
Message-ID: <cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>

On Sun, Sep 14, 2008 at 11:36 PM, Eric Smith <eric at trueblade.com> wrote:
> If we really want to change it, I think:
>  assert B as S
> is better because S is the string to report; that is, "if B is false, report
> the problem as the string S".
>
> 'else' implies to me what to do if you're not failing the assert, which is
> not the case.

Doesn't imply that to me.  I read it as 'first you assert that B is
true; if not (else), you print S'.

Personally, I like 'else' better than 'as', because 'as' seems to
contain the notion of assignment.

-- 
Cheers,
Leif

From guido at python.org  Mon Sep 15 06:12:40 2008
From: guido at python.org (Guido van Rossum)
Date: Sun, 14 Sep 2008 21:12:40 -0700
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<48CDBA02.9040802@canterbury.ac.nz> <48CDD859.90407@trueblade.com>
	<cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>
Message-ID: <ca471dc20809142112j6a930a85o5e7bd621182e80b7@mail.gmail.com>

On Sun, Sep 14, 2008 at 8:55 PM, Leif Walsh <leif.walsh at gmail.com> wrote:
> On Sun, Sep 14, 2008 at 11:36 PM, Eric Smith <eric at trueblade.com> wrote:
>> If we really want to change it, I think:
>>  assert B as S
>> is better because S is the string to report; that is, "if B is false, report
>> the problem as the string S".
>>
>> 'else' implies to me what to do if you're not failing the assert, which is
>> not the case.
>
> Doesn't imply that to me.  I read it as 'first you assert that B is
> true; if not (else), you print S'.
>
> Personally, I like 'else' better than 'as', because 'as' seems to
> contain the notion of assignment.

That's my gut feeling too, but I don't like 'else' all that much
either (if would also make things like"assert x if t else b, msg" less
readable I think). Maybe "assert B with S"???

FWIW I don't like turning it into a function either, and I *really*
don't like keeping the keyword but changing the syntax to be
function-like.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Mon Sep 15 06:18:56 2008
From: guido at python.org (Guido van Rossum)
Date: Sun, 14 Sep 2008 21:18:56 -0700
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
References: <20080914200327.GE30564@idyll.org>
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
Message-ID: <ca471dc20809142118p7fd32d6ag9a75447e2e9049ff@mail.gmail.com>

On Sun, Sep 14, 2008 at 2:01 PM, Christopher Lee <leec at chem.ucla.edu> wrote:
>
>> ----- Forwarded message from Guido van Rossum <guido at python.org> -----
>>
>> Note that in any case, the 'in' operator binds more tightly than the
>> comma: e.g. f(x in y, z) means f((x in y), z).
>
> Exactly the point I was trying to make.  As far as the user's concerned, an
> expression list
> x,y
> can itself be part of an expression;
> x in y
> is also an expression.  Such expressions can be combined to form larger
> expressions, so it follows logically that you need to know which has
> precedence when *both* occur in an expression, e.g. whether
> x in y,z
> evaluates as
> (x in y),z
> or
> x in (y,z)
>
> The page that Fredrik sent you to
> (http://docs.python.org/ref/exprlists.html) doesn't address that question.
>  I still think the precedence table
> (http://docs.python.org/ref/summary.html) should show that "in" has higher
> precedence than comma in an expression.
>
> Can anyone show us "where it is written" in the Python docs that "in" has
> higher precedence than comma, *and* why there is a good reason that this
> information should NOT be included in the precedence table?
>
> -- Chris
>
> Adding comma to the precedence table would also disambiguate expressions
> like:
> x and y,z
> x,y or z
> not x,y

You don't seem to understand how to read syntax tables. The assert
syntax is given as "assert expression , expression" and the syntax for
"expression" doesn't allow a comma unless it's inside parentheses.
That is all you need here, and that is what Fredrik pointed out. The
precedence list only applies to expressions, not to other constructs
like statements. (You might note that it also doesn't list the
precedence of 'assert'.) The place to reference is
http://docs.python.org/ref/assert.html .

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From fredrik at pythonware.com  Mon Sep 15 07:35:07 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Mon, 15 Sep 2008 07:35:07 +0200
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
References: <20080914200327.GE30564@idyll.org>
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
Message-ID: <gaks6f$8l0$1@ger.gmane.org>

Christopher Lee wrote:

> precedence when *both* occur in an expression, e.g. whether

> x in y,z

what part of "comma is not an operator" is so hard to understand?  the 
above is not an expression.  it's two expressions, separated by commas. 
  the first expression stops at the comma.  the second expression 
follows after the comma.

commas can be a part of an expression only when they appear as part of a 
syntactic construct that allows commas (that is, parenthesized forms and 
displays, and the target list of a generator expression).

> evaluates as

> (x in y),z
> or
> x in (y,z)

> The page that Fredrik sent you to 
> (http://docs.python.org/ref/exprlists.html) doesn't address that 
> question.  I still think the precedence table 
> (http://docs.python.org/ref/summary.html) should show that "in" has 
> higher precedence than comma in an expression.

comma is not an operator.

> Can anyone show us "where it is written" in the Python docs that "in" 
> has higher precedence than comma, *and* why there is a good reason that 
> this information should NOT be included in the precedence table?

because comma is not an operator?  if we were to list everything that 
was not an operator in the precedence table, it would end up being very 
long and mostly pointless.

</F>


From g.brandl at gmx.net  Mon Sep 15 08:47:16 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 15 Sep 2008 08:47:16 +0200
Subject: [Python-Dev] Weak Dictionary Iteration Behavior in Python 3
In-Reply-To: <48CDB7B9.3000005@canterbury.ac.nz>
References: <loom.20080913T201341-491@post.gmane.org>	<aac2c7cb0809131852n6a829fa2t5259130d162a41dc@mail.gmail.com>	<loom.20080914T093612-536@post.gmane.org>
	<48CCFCAB.4070809@gmail.com>	<gaj00v$qa9$1@ger.gmane.org>
	<48CD0B98.1070808@gmail.com>	<gaj4k7$6qv$1@ger.gmane.org>
	<48CDB7B9.3000005@canterbury.ac.nz>
Message-ID: <gal0e5$gb1$1@ger.gmane.org>

Greg Ewing schrieb:
> Georg Brandl wrote:
> 
>> In this case, what WeakKeyDictionary.keys() returns is not a view anyway,
>> so using list(self.data.keys()) in the method seems like a good compromise.
> 
> I don't think that helps, because the implementation has
> to iterate over that to dereference the weak refs, and if
> that's done in Python code there's a chance for a weak
> ref to disappear in the process.

Yes, but in that case the ref will simply not be there anymore. It can't be
guaranteed anyway which objects will be alive at the precise moment you
call keys().  In contrast, iterating over the keys() view of the dictionary
one-by-one leads to an exception when one of the refs vanishes while still
iterating.

See Armin's test case for an example that demonstrates this.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From list-ener at strank.info  Mon Sep 15 09:42:11 2008
From: list-ener at strank.info (Stefan Rank)
Date: Mon, 15 Sep 2008 09:42:11 +0200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <ca471dc20809142112j6a930a85o5e7bd621182e80b7@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org>
	<gajpob$2n6$1@ger.gmane.org>	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>	<48CDBA02.9040802@canterbury.ac.nz>
	<48CDD859.90407@trueblade.com>	<cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>
	<ca471dc20809142112j6a930a85o5e7bd621182e80b7@mail.gmail.com>
Message-ID: <48CE11D3.4010202@strank.info>

on 15.09.2008 06:12 Guido van Rossum said the following:
> On Sun, Sep 14, 2008 at 8:55 PM, Leif Walsh <leif.walsh at gmail.com> wrote:
>> On Sun, Sep 14, 2008 at 11:36 PM, Eric Smith <eric at trueblade.com> wrote:

>>>  assert B as S

>> Personally, I like 'else' better than 'as', because 'as' seems to
>> contain the notion of assignment.

> That's my gut feeling too, but I don't like 'else' all that much
> either (if would also make things like"assert x if t else b, msg" less
> readable I think). Maybe "assert B with S"???
> 
> FWIW I don't like turning it into a function either, and I *really*
> don't like keeping the keyword but changing the syntax to be
> function-like.

FWIW, Java's assert syntax is::

   assert Expression1 : Expression2 ;

which I regularly tried to write in Python when I started with it...
(minus the semicolon of course)


From ncoghlan at gmail.com  Mon Sep 15 11:34:14 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Sep 2008 19:34:14 +1000
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <20080914193101.GB30564@idyll.org>
References: <20080914193101.GB30564@idyll.org>
Message-ID: <48CE2C16.3000903@gmail.com>

C. Titus Brown wrote:
> Hi folks,
> 
> over on the pygr list, we were recently discussing a mildly confusing
> edit I made:
> 
>    assert 'seq1' in self.db, self.db.keys() 

For those suggesting changing the syntax of the assert statement due to
this misinterpretation, what makes this any more ambiguous than doing
the same thing in a function call or tuple literal?

>>> '1' in '123', 2
(True, 2)
>>> 1 and 2, lambda: 1, 3 or 4
(2, <function <lambda> at 0xb7704a3c>, 3)

>>> x = '1' in '123', 2
>>> x
(True, 2)
>>> x = 1 and 2, lambda: 1, 3 or 4
>>> x
(2, <function <lambda> at 0xb770410c>, 3)

>>> def f(*args): print args
...
>>> f('1' in '123', 2)
(True, 2)
>>> f(1 and 2, lambda: 1, 3 or 4)
(2, <function <lambda> at 0xb770410c>, 3)

Defining ',' as an "operator" in the predecence table may be technically
incorrect (since there is no actual comma operator), but giving it an
entry indicating that it has even lower precedence than lambda would
make it clear how the above expressions are meant to be parsed (and when
parentheses are going to be needed to force the correct interpretation).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From ncoghlan at gmail.com  Mon Sep 15 11:37:54 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Sep 2008 19:37:54 +1000
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <gaks6f$8l0$1@ger.gmane.org>
References: <20080914200327.GE30564@idyll.org>	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
	<gaks6f$8l0$1@ger.gmane.org>
Message-ID: <48CE2CF2.1020109@gmail.com>

Fredrik Lundh wrote:
> Christopher Lee wrote:
>> Can anyone show us "where it is written" in the Python docs that "in"
>> has higher precedence than comma, *and* why there is a good reason
>> that this information should NOT be included in the precedence table?
> 
> because comma is not an operator?  if we were to list everything that
> was not an operator in the precedence table, it would end up being very
> long and mostly pointless.

While it may not make sense to list it in the table itself, having a
note on that page that comma is NOT an operator (unlike C), and hence
only permitted between expressions as part of another statement or
expressions that permits it may not be a bad idea.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From skip at pobox.com  Mon Sep 15 13:18:56 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 15 Sep 2008 06:18:56 -0500
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <ca471dc20809142112j6a930a85o5e7bd621182e80b7@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<48CDBA02.9040802@canterbury.ac.nz> <48CDD859.90407@trueblade.com>
	<cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>
	<ca471dc20809142112j6a930a85o5e7bd621182e80b7@mail.gmail.com>
Message-ID: <18638.17568.723894.644494@montanaro-dyndns-org.local>

How about

    assert B except S

?

Skip

From skip at pobox.com  Mon Sep 15 13:21:11 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 15 Sep 2008 06:21:11 -0500
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <gaks6f$8l0$1@ger.gmane.org>
References: <20080914200327.GE30564@idyll.org>
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
	<gaks6f$8l0$1@ger.gmane.org>
Message-ID: <18638.17703.868721.744929@montanaro-dyndns-org.local>


    Fredrik> if we were to list everything that was not an operator in the
    Fredrik> precedence table, it would end up being very long and mostly
    Fredrik> pointless.

OTOH, comma is an operator in (I believe) all C-derived languages.  That's
probably what throws people.  It might be worth a short note: on the
operator precedence page to point out that comma isn't an operator in
Python.

Skip

From steve at pearwood.info  Mon Sep 15 13:49:38 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 15 Sep 2008 21:49:38 +1000
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <48CDD859.90407@trueblade.com>
	<cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>
Message-ID: <200809152149.38989.steve@pearwood.info>

On Mon, 15 Sep 2008 01:55:00 pm Leif Walsh wrote:
> On Sun, Sep 14, 2008 at 11:36 PM, Eric Smith <eric at trueblade.com> 
wrote:
> > If we really want to change it, I think:
> >  assert B as S
> > is better because S is the string to report; that is, "if B is
> > false, report the problem as the string S".
> >
> > 'else' implies to me what to do if you're not failing the assert,
> > which is not the case.
>
> Doesn't imply that to me.  I read it as 'first you assert that B is
> true; if not (else), you print S'.
>
> Personally, I like 'else' better than 'as', because 'as' seems to
> contain the notion of assignment.

I agree with Leif on both his comments. I don't particularly see the 
need to change assert, but if it has to change, I'd be perfectly happy 
with one of "assert B else S" or "assert B except S".



-- 
Steven

From ncoghlan at gmail.com  Mon Sep 15 14:04:13 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 15 Sep 2008 22:04:13 +1000
Subject: [Python-Dev] [Python-checkins] Python Regression Test Failures
 basics (1)
In-Reply-To: <ee2a432c0809141053o45809460j51a92bf3c8c0a16@mail.gmail.com>
References: <20080914083615.GA7037@python.psfb.org>	
	<48CCD464.8080106@gmail.com>	
	<1afaf6160809140524t3b5f91acufb0ca630a20e25fc@mail.gmail.com>
	<ee2a432c0809141053o45809460j51a92bf3c8c0a16@mail.gmail.com>
Message-ID: <48CE4F3D.7050306@gmail.com>

Neal Norwitz wrote:
> On Sun, Sep 14, 2008 at 5:24 AM, Benjamin Peterson 
> <musiccomposition at gmail.com> wrote:
>> On Sun, Sep 14, 2008 at 4:07 AM, Nick Coghlan <ncoghlan at gmail.com>
>>  wrote:
>>> Neal Norwitz wrote:
>>>> test_epoll skipped -- kernel doesn't support epoll()
>>> ...
>>>> test_ioctl skipped -- Unable to open /dev/tty
>>> ...
>>>> test_multiprocessing skipped -- OSError raises on RLock 
>>>> creation, see
>>> issue 3111! ...
>>>> test test_normalization failed -- Traceback (most recent call 
>>>> last): File 
>>>> "/tmp/python-test/local/lib/python2.6/test/test_normalization.py",
>>>>  line 90, in test_main self.failUnless(X == NFC(X) == NFD(X) ==
>>>>  NFKC(X) == NFKD(X), c) AssertionError: 6918
>>> ...
>>>> 326 tests OK. 1 test failed: test_normalization
>>> ...
>>>> 3 skips unexpected on linux2: test_epoll test_multiprocessing 
>>>> test_ioctl
>>> Neal,
>>> 
>>> What environment are you using to run the debug-mode regression 
>>> tests? The above four tests run without any problems for me, but
>>>  I'm just running them in a normal Kubuntu desktop shell.
>> Something in Neal's build which has made a difference before is 
>> that the tests are run after a "make install".
> 
> Benjamin is correct.  See Misc/build.sh for the script that generates
>  this.  See http://docs.python.org/dev/results/ for details about the
>  most recent run.

Hmm, even after doing a make altinstall (don't want to mess with the
system Python!), I still can't reproduce those failures. I had to give
the test suite elevated privileges to get test_distutils and test_tcl to
work properly, and this machine has some audio issues that upset
test_linuxaudiodev and test_ossaudiodev, but the four tests that are
failing in Neal's tests all run fine.

Are these being run in a sub-environment of some kind to avoid messing
up the machine, and that environment is missing the /dev/* tree from the
filesystem? (the particular error being triggered by the
test_multiprocessing failure indicates that as a possible cause, and the
test_ioctl problem relates to not being able to find /dev/tty).

Since these tests are an additional kind of buildbot, I'd really like to
see them all passing before rc2 goes out, but I think I've reached the
limits of what I can figure out from this side of the Pacific.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From guido at python.org  Mon Sep 15 16:10:25 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Sep 2008 07:10:25 -0700
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <18638.17703.868721.744929@montanaro-dyndns-org.local>
References: <20080914200327.GE30564@idyll.org>
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
	<gaks6f$8l0$1@ger.gmane.org>
	<18638.17703.868721.744929@montanaro-dyndns-org.local>
Message-ID: <ca471dc20809150710s4bcdc0d3wd0500dcbfd9ade9b@mail.gmail.com>

On Mon, Sep 15, 2008 at 4:21 AM,  <skip at pobox.com> wrote:
> OTOH, comma is an operator in (I believe) all C-derived languages.

Even in C, the comma in argument lists is not the same as the comma operator!

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Mon Sep 15 16:15:58 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Sep 2008 07:15:58 -0700
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <200809152149.38989.steve@pearwood.info>
References: <20080914193101.GB30564@idyll.org> <48CDD859.90407@trueblade.com>
	<cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>
	<200809152149.38989.steve@pearwood.info>
Message-ID: <ca471dc20809150715t4836dde6g76d7734e5a039057@mail.gmail.com>

 On Mon, Sep 15, 2008 at 4:49 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Mon, 15 Sep 2008 01:55:00 pm Leif Walsh wrote:
>> On Sun, Sep 14, 2008 at 11:36 PM, Eric Smith <eric at trueblade.com>
> wrote:
>> > If we really want to change it, I think:
>> >  assert B as S
>> > is better because S is the string to report; that is, "if B is
>> > false, report the problem as the string S".
>> >
>> > 'else' implies to me what to do if you're not failing the assert,
>> > which is not the case.
>>
>> Doesn't imply that to me.  I read it as 'first you assert that B is
>> true; if not (else), you print S'.
>>
>> Personally, I like 'else' better than 'as', because 'as' seems to
>> contain the notion of assignment.
>
> I agree with Leif on both his comments. I don't particularly see the
> need to change assert, but if it has to change, I'd be perfectly happy
> with one of "assert B else S" or "assert B except S".

In private mail someone suggested the Java assert syntax:

  assert expression : expression

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From andymac at bullseye.apana.org.au  Mon Sep 15 16:18:40 2008
From: andymac at bullseye.apana.org.au (Andrew MacIntyre)
Date: Tue, 16 Sep 2008 00:18:40 +1000
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
Message-ID: <48CE6EC0.8050309@bullseye.andymac.org>

I noticed issue 3862 (http://bugs.python.org/issue3862), which appears 
to be exposed only on FreeBSD 7.0 amd64 (perhaps 64 bit FreeBSD 7, but 
amd64 is the only 64bit hardware I can check).

As far as I can make out from a couple of quick checks, this appears to
be happening because the process is killed before the malloc() that
exhausts the available swap space is allowed to report an out of memory
condition.

FreeBSD's default process virtual memory limit (ulimit -v) on both i386
and amd64 is "unlimited" for 6.3 and 7.0.  Setting ulimit -v to a finite
value on 7.0 amd64 seems to give rise to "normal" malloc() behaviour so
that test_array passes.

FreeBSD 7 uses a different malloc() implementation than FreeBSD 6 and
earlier, and the new implementation uses mmap() in preference to sbrk().
So I suspect that behaviour was "normal" on FreeBSD 6 amd64 (can't test
at the moment as I don't have that installed anywhere).  Behaviour
appears "normal" on FreeBSD 7 i386 as well.

I haven't yet tried posting a query to a FreeBSD list, as it could simply
be a bug on amd64, but I was wondering whether there was anything (other
than deactivating tests and documenting use of ulimit -v on this
platform) that could be done to work around this behaviour.

Andrew.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia

From fredrik at pythonware.com  Mon Sep 15 16:39:52 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Mon, 15 Sep 2008 16:39:52 +0200
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <48CE2CF2.1020109@gmail.com>
References: <20080914200327.GE30564@idyll.org>	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>	<gaks6f$8l0$1@ger.gmane.org>
	<48CE2CF2.1020109@gmail.com>
Message-ID: <gals3s$d6s$1@ger.gmane.org>

Nick Coghlan wrote:

> While it may not make sense to list it in the table itself, having a
> note on that page that comma is NOT an operator (unlike C), and hence
> only permitted between expressions as part of another statement or
> expressions that permits it may not be a bad idea.

better include "=" and augmented assignments as well.

on the other hand, we're talking about a single instance of a "someone" 
here -- it's not like there's any sign that the comma issue is something 
that lots of someones are confused about.

(...and given that the precedence table has been broken for over a 
decade (see http://bugs.python.org/issue3558) due to a bug in the 
Latex-to-HTML renderer, one might wonder if anyone gets that far into 
the language reference at all...)

</F>


From fredrik at pythonware.com  Mon Sep 15 17:04:01 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Mon, 15 Sep 2008 17:04:01 +0200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <ca471dc20809150715t4836dde6g76d7734e5a039057@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org>
	<48CDD859.90407@trueblade.com>	<cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>	<200809152149.38989.steve@pearwood.info>
	<ca471dc20809150715t4836dde6g76d7734e5a039057@mail.gmail.com>
Message-ID: <galth5$imr$1@ger.gmane.org>

Guido van Rossum wrote:

> In private mail someone suggested the Java assert syntax:
> 
>   assert expression : expression

and in 2015, we'll get a message from you where you say:

   "I think in general Python has erred on the side of having too many
   different syntactical uses for colons."

(yes, I've checked.  you shouldn't leave the time machine connected to 
the net when you're not there.)

</F>


From jjb5 at cornell.edu  Mon Sep 15 17:14:10 2008
From: jjb5 at cornell.edu (Joel Bender)
Date: Mon, 15 Sep 2008 11:14:10 -0400
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <ca471dc20809142118p7fd32d6ag9a75447e2e9049ff@mail.gmail.com>
References: <20080914200327.GE30564@idyll.org>	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
	<ca471dc20809142118p7fd32d6ag9a75447e2e9049ff@mail.gmail.com>
Message-ID: <48CE7BC2.20102@cornell.edu>

Guido van Rossum wrote:

> ...the syntax for "expression" doesn't allow a comma unless
> it's inside parentheses.

Perhaps a source of confusion might be that comma seems to act like a 
'tuple join operator' when it is not inside parentheses.

     >>> x = 1, 2
     >>> x
     (1, 2)

And there is at least one point in the documentation where the comma is 
described as an operator:

     <http://docs.python.org/ref/parenthesized.html>

     "Note that tuples are not formed by the parentheses, but rather
     by use of the comma operator."

As for the assert syntax, I would reuse the 'raise' keyword rather than 
'else':

     assert_stmt ::= "assert" <expression> [ "raise" <expression> ]

Which emphasizes that the expression is raised as an exception.


Joel


From skip at pobox.com  Mon Sep 15 17:41:10 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 15 Sep 2008 10:41:10 -0500
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <ca471dc20809150710s4bcdc0d3wd0500dcbfd9ade9b@mail.gmail.com>
References: <20080914200327.GE30564@idyll.org>
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
	<gaks6f$8l0$1@ger.gmane.org>
	<18638.17703.868721.744929@montanaro-dyndns-org.local>
	<ca471dc20809150710s4bcdc0d3wd0500dcbfd9ade9b@mail.gmail.com>
Message-ID: <18638.33302.32659.346495@montanaro-dyndns-org.local>

>>>>> "Guido" == Guido van Rossum <guido at python.org> writes:

    >> OTOH, comma is an operator in (I believe) all C-derived languages.

    Guido> Even in C, the comma in argument lists is not the same as the
    Guido> comma operator!

True enough, further adding to peoples' confusion.

If no change to the docs (in the form of a footnote?) is forthcoming,
perhaps we should add a question to the programming FAQ.

Skip

From guido at python.org  Mon Sep 15 18:19:53 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 15 Sep 2008 09:19:53 -0700
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <48CE7BC2.20102@cornell.edu>
References: <20080914200327.GE30564@idyll.org>
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
	<ca471dc20809142118p7fd32d6ag9a75447e2e9049ff@mail.gmail.com>
	<48CE7BC2.20102@cornell.edu>
Message-ID: <ca471dc20809150919t66b08c84k17ab69cafb83fae6@mail.gmail.com>

On Mon, Sep 15, 2008 at 8:14 AM, Joel Bender <jjb5 at cornell.edu> wrote:
> Guido van Rossum wrote:
>
>> ...the syntax for "expression" doesn't allow a comma unless
>> it's inside parentheses.
>
> Perhaps a source of confusion might be that comma seems to act like a 'tuple
> join operator' when it is not inside parentheses.

Um, the question I was answering specifically asked *where is this in the docs*.

> And there is at least one point in the documentation where the comma is
> described as an operator:
>
>    <http://docs.python.org/ref/parenthesized.html>
>
>    "Note that tuples are not formed by the parentheses, but rather
>    by use of the comma operator."

Good sleuthing. Since you have found an inconsistency, now all the
docs are useless?

> As for the assert syntax, I would reuse the 'raise' keyword rather than
> 'else':
>
>    assert_stmt ::= "assert" <expression> [ "raise" <expression> ]
>
> Which emphasizes that the expression is raised as an exception.

But it is not -- it is the message passed to the exception constructor!

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From leec at chem.ucla.edu  Mon Sep 15 20:50:21 2008
From: leec at chem.ucla.edu (Christopher Lee)
Date: Mon, 15 Sep 2008 11:50:21 -0700
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <ca471dc20809142118p7fd32d6ag9a75447e2e9049ff@mail.gmail.com>
References: <20080914200327.GE30564@idyll.org>
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
	<ca471dc20809142118p7fd32d6ag9a75447e2e9049ff@mail.gmail.com>
Message-ID: <C266DF95-44C2-4659-B202-08122756E41D@chem.ucla.edu>


On Sep 14, 2008, at 9:18 PM, Guido van Rossum wrote:

> You don't seem to understand how to read syntax tables. The assert
> syntax is given as "assert expression , expression" and the syntax for
> "expression" doesn't allow a comma unless it's inside parentheses.
> That is all you need here, and that is what Fredrik pointed out. The
> precedence list only applies to expressions, not to other constructs
> like statements. (You might note that it also doesn't list the
> precedence of 'assert'.) The place to reference is
> http://docs.python.org/ref/assert.html .

Thanks.  This is such a striking difference from C, where many of us  
draw our default language assumptions, that it might be worth  
highlighting in the Python docs, e.g. on the expression list page.   
Maybe something like <<Note: Python does not define comma as an  
operator, but instead just uses it as a separator in expression lists,  
function argument lists, and many other contexts.  Note also that this  
disambiguates precedence: expression operators have higher precedence  
than the commas of an expression list.  Thus the expression list "x  
and y, z" evaluates as "(x and y), z", not "x and (y, z)".>>   
Highlighting this would stop people with C backgrounds from assuming  
that comma is an operator, like I did.  Analyzing the full syntax  
table would obviously sort this out, but a lot of people just read the  
docs (like I did :-)...


From skip at pobox.com  Mon Sep 15 21:49:33 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 15 Sep 2008 14:49:33 -0500
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <C266DF95-44C2-4659-B202-08122756E41D@chem.ucla.edu>
References: <20080914200327.GE30564@idyll.org>
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
	<ca471dc20809142118p7fd32d6ag9a75447e2e9049ff@mail.gmail.com>
	<C266DF95-44C2-4659-B202-08122756E41D@chem.ucla.edu>
Message-ID: <18638.48205.774949.988574@montanaro-dyndns-org.local>


    Christopher> This is such a striking difference from C, where many of us
    Christopher> draw our default language assumptions, that it might be
    Christopher> worth highlighting in the Python docs, e.g. on the
    Christopher> expression list page.

I added a question and answer to the programming FAQ.  I agree it might be
worthwhile to add a footnote to the operator precedence page though.

Skip

From jjb5 at cornell.edu  Mon Sep 15 23:21:27 2008
From: jjb5 at cornell.edu (Joel Bender)
Date: Mon, 15 Sep 2008 17:21:27 -0400
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <ca471dc20809150919t66b08c84k17ab69cafb83fae6@mail.gmail.com>
References: <20080914200327.GE30564@idyll.org>	
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>	
	<ca471dc20809142118p7fd32d6ag9a75447e2e9049ff@mail.gmail.com>	
	<48CE7BC2.20102@cornell.edu>
	<ca471dc20809150919t66b08c84k17ab69cafb83fae6@mail.gmail.com>
Message-ID: <48CED1D7.9000308@cornell.edu>

> Good sleuthing. Since you have found an inconsistency, now all the
> docs are useless?

LOL!  Hardly!  I only found one inconsistent page, and it happens to be 
one that I found a little hard to understand while I was learning the 
language.

In ALGOL it is used as part of the statement, like:

     FOR I := 1, 5 STEP 2 UNTIL 9, 12 WHILE (X < 5) DO ...

So I'm quite happy with it being an expression separator.  I would 
re-write that part of the grammar to have a tuple_display like 
list_display, but it's not worth the effort.

> But it is not -- it is the message passed to the exception constructor!

I apologize, oversimplified the statement.  I wanted to express that 
it's not just passed as a parameter to the exception constructor, but 
the fact that the exception that that is constructed is also raised in 
some circumstances.  I didn't mean to imply that the expression itself 
was raised.  IMHO, the keyword "else" wasn't strong enough.


Joel


From musiccomposition at gmail.com  Tue Sep 16 00:13:21 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Mon, 15 Sep 2008 17:13:21 -0500
Subject: [Python-Dev] [Python-checkins] Python Regression Test Failures
	basics (1)
In-Reply-To: <48CE4F3D.7050306@gmail.com>
References: <20080914083615.GA7037@python.psfb.org>
	<48CCD464.8080106@gmail.com>
	<1afaf6160809140524t3b5f91acufb0ca630a20e25fc@mail.gmail.com>
	<ee2a432c0809141053o45809460j51a92bf3c8c0a16@mail.gmail.com>
	<48CE4F3D.7050306@gmail.com>
Message-ID: <1afaf6160809151513x65d4605en324733cd2611222f@mail.gmail.com>

On Mon, Sep 15, 2008 at 7:04 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Neal Norwitz wrote:
>> On Sun, Sep 14, 2008 at 5:24 AM, Benjamin Peterson
>> <musiccomposition at gmail.com> wrote:
>>> On Sun, Sep 14, 2008 at 4:07 AM, Nick Coghlan <ncoghlan at gmail.com>
>>>> What environment are you using to run the debug-mode regression
>>>> tests? The above four tests run without any problems for me, but
>>>>  I'm just running them in a normal Kubuntu desktop shell.
>>> Something in Neal's build which has made a difference before is
>>> that the tests are run after a "make install".
>>
>> Benjamin is correct.  See Misc/build.sh for the script that generates
>>  this.  See http://docs.python.org/dev/results/ for details about the
>>  most recent run.
>
> Hmm, even after doing a make altinstall (don't want to mess with the
> system Python!), I still can't reproduce those failures. I had to give
> the test suite elevated privileges to get test_distutils and test_tcl to
> work properly, and this machine has some audio issues that upset
> test_linuxaudiodev and test_ossaudiodev, but the four tests that are
> failing in Neal's tests all run fine.
>
> Are these being run in a sub-environment of some kind to avoid messing
> up the machine, and that environment is missing the /dev/* tree from the
> filesystem? (the particular error being triggered by the
> test_multiprocessing failure indicates that as a possible cause, and the
> test_ioctl problem relates to not being able to find /dev/tty).

I saw the error that Neal is getting once, and that was when I
interrupted test_normalization while it was fetching
NormalizationTest.txt. So maybe he just has a corrupt copy of that?
(Neal, will you try removing it?)

>
> Since these tests are an additional kind of buildbot, I'd really like to
> see them all passing before rc2 goes out, but I think I've reached the
> limits of what I can figure out from this side of the Pacific.




-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From steve at pearwood.info  Tue Sep 16 00:39:42 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 16 Sep 2008 08:39:42 +1000
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <gals3s$d6s$1@ger.gmane.org>
References: <20080914200327.GE30564@idyll.org> <48CE2CF2.1020109@gmail.com>
	<gals3s$d6s$1@ger.gmane.org>
Message-ID: <200809160839.42379.steve@pearwood.info>

On Tue, 16 Sep 2008 12:39:52 am Fredrik Lundh wrote:
> Nick Coghlan wrote:
> > While it may not make sense to list it in the table itself, having
> > a note on that page that comma is NOT an operator (unlike C), and
> > hence only permitted between expressions as part of another
> > statement or expressions that permits it may not be a bad idea.
>
> better include "=" and augmented assignments as well.
>
> on the other hand, we're talking about a single instance of a
> "someone" here -- it's not like there's any sign that the comma issue
> is something that lots of someones are confused about.

*puts up hand*

I was confused about the role of commas, and totally gobsmacked when I 
discovered that commas make tuples everywhere except when following an 
except statement. (At least that's how I interpreted what I was seeing 
with "try...except A, B, C, e".)

Right or wrong, my mental model is that commas are the tuple operator, 
except in special places where it isn't and you have to use 
parentheses. Well, actually I know the model is wrong *now*, because 
I've read this thread, but until now I would have cheerfully described 
the comma as an operator. Judging by the number of hits on Goggle 
for 'Python "comma operator"', I'm not alone: over 4700 hits for comma 
operator versus 800 for comma separator.



-- 
Steven

From steve at pearwood.info  Tue Sep 16 00:46:53 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 16 Sep 2008 08:46:53 +1000
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <200809160839.42379.steve@pearwood.info>
References: <20080914200327.GE30564@idyll.org> <gals3s$d6s$1@ger.gmane.org>
	<200809160839.42379.steve@pearwood.info>
Message-ID: <200809160846.53599.steve@pearwood.info>

On Tue, 16 Sep 2008 08:39:42 am Steven D'Aprano wrote:

> I was confused about the role of commas, and totally gobsmacked when
> I discovered that commas make tuples everywhere except when following
> an except statement. (At least that's how I interpreted what I was
> seeing with "try...except A, B, C, e".)

Oops. Except of course the above is a syntax error. I meant:

try...except (A, B, C), e


-- 
Steven

From greg.ewing at canterbury.ac.nz  Tue Sep 16 01:50:43 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 16 Sep 2008 11:50:43 +1200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CDD859.90407@trueblade.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<48CDBA02.9040802@canterbury.ac.nz> <48CDD859.90407@trueblade.com>
Message-ID: <48CEF4D3.5070607@canterbury.ac.nz>

Eric Smith wrote:

> 'else' implies to me what to do if you're not failing the assert, which 
> is not the case.

Strange, because I see it exactly the opposite way:
it's saying "assert that B is true, but if not,
report this error message."

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Sep 16 02:00:54 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 16 Sep 2008 12:00:54 +1200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <ca471dc20809142112j6a930a85o5e7bd621182e80b7@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<48CDBA02.9040802@canterbury.ac.nz> <48CDD859.90407@trueblade.com>
	<cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>
	<ca471dc20809142112j6a930a85o5e7bd621182e80b7@mail.gmail.com>
Message-ID: <48CEF736.6060707@canterbury.ac.nz>

Guido van Rossum wrote:

> if would also make things like"assert x if t else b, msg" less
> readable I think

Is that likely to turn up in real code much?

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Sep 16 02:33:20 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 16 Sep 2008 12:33:20 +1200
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <48CE7BC2.20102@cornell.edu>
References: <20080914200327.GE30564@idyll.org>
	<777CD866-5CD6-4546-9276-942EFB5C97EE@chem.ucla.edu>
	<ca471dc20809142118p7fd32d6ag9a75447e2e9049ff@mail.gmail.com>
	<48CE7BC2.20102@cornell.edu>
Message-ID: <48CEFED0.4060500@canterbury.ac.nz>

Joel Bender wrote:

> As for the assert syntax, I would reuse the 'raise' keyword rather than 
> 'else':
> 
>     assert_stmt ::= "assert" <expression> [ "raise" <expression> ]
> 
> Which emphasizes that the expression is raised as an exception.

However, it doesn't take the same kind of argument
as the "raise" statement, whereas this syntax might
suggest that it does.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue Sep 16 02:50:45 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 16 Sep 2008 12:50:45 +1200
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <200809160839.42379.steve@pearwood.info>
References: <20080914200327.GE30564@idyll.org> <48CE2CF2.1020109@gmail.com>
	<gals3s$d6s$1@ger.gmane.org> <200809160839.42379.steve@pearwood.info>
Message-ID: <48CF02E5.6070206@canterbury.ac.nz>

Steven D'Aprano wrote:

> I was confused about the role of commas, and totally gobsmacked when I 
> discovered that commas make tuples everywhere except when following an 
> except statement.

Um... you've never written a function call with more than
one argument before? :-)

-- 
Greg

From fredrik at pythonware.com  Tue Sep 16 09:28:28 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Tue, 16 Sep 2008 09:28:28 +0200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CEF4D3.5070607@canterbury.ac.nz>
References: <20080914193101.GB30564@idyll.org>
	<gajpob$2n6$1@ger.gmane.org>	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>	<48CDBA02.9040802@canterbury.ac.nz>
	<48CDD859.90407@trueblade.com> <48CEF4D3.5070607@canterbury.ac.nz>
Message-ID: <gann70$2r8$1@ger.gmane.org>

Greg Ewing wrote:

>> 'else' implies to me what to do if you're not failing the assert, 
>> which is not the case.
> 
> Strange, because I see it exactly the opposite way:
> it's saying "assert that B is true, but if not,
> report this error message."

Strange, because this whole thing is saying "AppleScript" to me.

</F>


From ncoghlan at gmail.com  Tue Sep 16 11:33:35 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 16 Sep 2008 19:33:35 +1000
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <48CF02E5.6070206@canterbury.ac.nz>
References: <20080914200327.GE30564@idyll.org>
	<48CE2CF2.1020109@gmail.com>	<gals3s$d6s$1@ger.gmane.org>
	<200809160839.42379.steve@pearwood.info>
	<48CF02E5.6070206@canterbury.ac.nz>
Message-ID: <48CF7D6F.2090409@gmail.com>

Greg Ewing wrote:
> Steven D'Aprano wrote:
> 
>> I was confused about the role of commas, and totally gobsmacked when I
>> discovered that commas make tuples everywhere except when following an
>> except statement.
> 
> Um... you've never written a function call with more than
> one argument before? :-)
> 
The existence of *args syntax makes that one a little blurry - there's
often a hidden tuple of arguments involved, and it is still a case of
"multiple expressions separated by commas". The special handling for
varargs and varkwds expansion and keyword arguments are marked within
the expressions with their own special syntax. Similarly, even though
there is no actual tuple created in list, dict and set displays,
function parameter lists and unpacking to multiple targets in an
assignment statement, the usages share the characteristic that the
things being separated are by and large equivalent.

So to answer my own question from earlier in the thread, except clauses
and assert statements are special because they are the only places where
the expressions to the left and right of the comma relate to completely
different concepts. We're fixing except clauses by replacing that comma
with an 'as', but assert statements are still a bit odd.

And to suggest another colour for the bikeshed, what if we just made
"assert" a normal builtin, and required people to compile them out
explicitly in optimised code via:

  if __debug__: assert(whatever, msg="whatever broke!")

A source code transform (ala 2to3) could easily translate the old assert
statements to the above.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From avanderneut at avid.com  Tue Sep 16 17:32:21 2008
From: avanderneut at avid.com (Anthon van der Neut)
Date: Tue, 16 Sep 2008 17:32:21 +0200
Subject: [Python-Dev] switching on -3 from within a program?
Message-ID: <48CFD185.8070003@avid.com>

With a minor modification to the Makefile I was able to get mod_wsgi
v2.3 to work with python2.6rc1.
I promptly got a warning in my apache log files on the depcrecated use
of 'sha' in the paste's cookie module, good! And easily fixed.

After that I was looking for a way to switch on the -3 warnings from
within my code to have extra warnings show up in my apache log file.

After reading some documentation I tried from the python2.6 prompt:

import sys
sys.py3kwarning = True
x = { 'abc': 1 }; x.has_key('abc')

which does not give a warning (starting python2.6 with the -3 option of
course does).

Is there anyway to switch this on from within a program with a Python
statement?
If not, would I need to make a small C-extension module (to be imported
as the first module) that sets Py_Py3WarningFlag or something else at
the C level, or would that better be done by mod_wsgi's C code.

Regards
Anthon

From lists at cheimes.de  Tue Sep 16 18:07:33 2008
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 16 Sep 2008 18:07:33 +0200
Subject: [Python-Dev] switching on -3 from within a program?
In-Reply-To: <48CFD185.8070003@avid.com>
References: <48CFD185.8070003@avid.com>
Message-ID: <gaolk5$cna$1@ger.gmane.org>

Anthon van der Neut wrote:
> import sys
> sys.py3kwarning = True
> x = { 'abc': 1 }; x.has_key('abc')
> 
> which does not give a warning (starting python2.6 with the -3 option of
> course does).
 > Is there anyway to switch this on from within a program with a Python
 > statement?

That doesn't work because "sys.py3kwarning = True" doesn't change the 
global C variable Py_Py3kWarningFlag. IIRC there is no API to change the 
flag from Python code.


> If not, would I need to make a small C-extension module (to be imported
> as the first module) that sets Py_Py3WarningFlag or something else at
> the C level, or would that better be done by mod_wsgi's C code.

I suggest that you introduce a new config option that sets 
Py_Py3kWarningFlag to 1.

Christian


From musiccomposition at gmail.com  Tue Sep 16 22:45:45 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Tue, 16 Sep 2008 15:45:45 -0500
Subject: [Python-Dev] switching on -3 from within a program?
In-Reply-To: <48CFD185.8070003@avid.com>
References: <48CFD185.8070003@avid.com>
Message-ID: <1afaf6160809161345j213f21a5hfd4f16e0f65d4239@mail.gmail.com>

On Tue, Sep 16, 2008 at 10:32 AM, Anthon van der Neut
<avanderneut at avid.com> wrote:
> With a minor modification to the Makefile I was able to get mod_wsgi
> v2.3 to work with python2.6rc1.
> I promptly got a warning in my apache log files on the depcrecated use
> of 'sha' in the paste's cookie module, good! And easily fixed.
>
> After that I was looking for a way to switch on the -3 warnings from
> within my code to have extra warnings show up in my apache log file.
>
> After reading some documentation I tried from the python2.6 prompt:
>
> import sys
> sys.py3kwarning = True
> x = { 'abc': 1 }; x.has_key('abc')
>
> which does not give a warning (starting python2.6 with the -3 option of
> course does).
>
> Is there anyway to switch this on from within a program with a Python
> statement?
> If not, would I need to make a small C-extension module (to be imported
> as the first module) that sets Py_Py3WarningFlag or something else at
> the C level, or would that better be done by mod_wsgi's C code.

You could also utilize a nifty ctypes trick:

import ctypes

def engage_py3k_warning():
     flag = ctypes.c_int.in_dll(ctypes.pythonapi, "Py_Py3kWarningFlag")
     flag.value = 1



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From steve at pearwood.info  Wed Sep 17 01:26:11 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 17 Sep 2008 09:26:11 +1000
Subject: [Python-Dev] ',' precedence in documentation]
In-Reply-To: <48CF02E5.6070206@canterbury.ac.nz>
References: <20080914200327.GE30564@idyll.org>
	<200809160839.42379.steve@pearwood.info>
	<48CF02E5.6070206@canterbury.ac.nz>
Message-ID: <200809170926.12104.steve@pearwood.info>

On Tue, 16 Sep 2008 10:50:45 am Greg Ewing wrote:
> Steven D'Aprano wrote:
> > I was confused about the role of commas, and totally gobsmacked
> > when I discovered that commas make tuples everywhere except when
> > following an except statement.
>
> Um... you've never written a function call with more than
> one argument before? :-)

Of course I had. If somebody had asked me, I probably would have guessed 
that there was some sort of special case in the parser for function 
calls. I didn't say my mental model was a good model, but it was the 
only one I had.

I have no doubt that there are others out there who are confused about 
the role of commas.


-- 
Steven

From gnewsg at gmail.com  Wed Sep 17 03:24:06 2008
From: gnewsg at gmail.com (Giampaolo Rodola')
Date: Tue, 16 Sep 2008 18:24:06 -0700 (PDT)
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <6896.1221447015@parc.com>
References: <0a7adf05-7811-4078-8a90-3830f0974dab@d77g2000hsb.googlegroups.com>
	<6896.1221447015@parc.com>
Message-ID: <e39d9e1e-af35-4bda-a1a8-fdacb262c9ac@e53g2000hsa.googlegroups.com>

I've tried to modify my existing asyncore-based code but I'm
encountering a lot of different problems I didn't manage to fix.
It seems that playing with the do_handshake_on_connect flag doesn't
make any difference.
I guess that without some kind of documentation describing how to deal
with non-blocking "ssl-wrapped" sockets I won't get too far.
I try to ask two questions in case the answers may help me in some
way:

1 - What pending() method is supposed to do (it's not documented)?
2 - By reading ssl.py code I noticed that when do_handshake_on_connect
flag is False the do_handshake() method is never called. Is it
supposed to be manually called when dealing with non-blocking sockets?

In the meanwhile I noticed something in the ssl.py code which seems to
be wrong:

    def recv (self, buflen=1024, flags=0):
        if self._sslobj:
            if flags != 0:
                raise ValueError(
                    "non-zero flags not allowed in calls to sendall()
on %s" %
                    self.__class__)
            while True:
                try:
                    return self.read(buflen)
                except SSLError, x:
                    if x.args[0] == SSL_ERROR_WANT_READ:
                        continue
                    else:
                        raise x
        else:
            return socket.recv(self, buflen, flags)

I don't know the low levels but that while statement which continues
in case of SSL_ERROR_WANT_READ seems to be wrong (blocking), at least
when dealing with non-blocking sockets. I think the proper way of
doing recv() here is letting SSL_ERROR_WANT_READ propagate and let the
upper application (e.g. asyncore) deal with it.


Hope this helps,

--- Giampaolo
http://code.google.com/p/pyftpdlib/downloads/list


On 15 Set, 04:50, Bill Janssen <jans... at parc.com> wrote:
> Hi, Giampaolo.
>
> If you look a bit further in Lib/test/test_ssl.py, you'll see a
> non-blocking use of the "do_handshake" method.  Basically, the flag
> "do_handshake_on_connect" says whether you want this to happen
> automatically and blockingly (True), or whether you want to do it
> yourself (False).  In the test suite, the function
> "testNonBlockingHandshake" does the async client-side handshake; the
> server side logic is just the same, only it would happen in the server's
> "handle new connection" code -- you'd have to add a state variable, and
> bind handlers for "read_event" and "write_event", which would consult
> the state variable to see whether they had finished the handshake yet.
>
> I just made the server do it automatically to make life easier.
>
> The hard part isn't really doing the non-blocking, it's trying to figure
> out how to use asyncore correctly, IMO.
>
> Giampaolo Rodola' <gne... at gmail.com> wrote:
> > I'm interested in using the ssl module with asyncore but since there's
> > no real documentation about how using ssl module with non-blocking
>
> If you'd like to contribute a doc patch, that would be great.
>
> Here's what it current says for do_handshake_on_connect:
>
>   The parameter do_handshake_on_connect specifies whether to do the SSL
>   handshake automatically after doing a socket.connect(), or whether the
>   application program will call it explicitly, by invoking the
>   SSLSocket.do_handshake() method. Calling SSLSocket.do_handshake()
>   explicitly gives the program control over the blocking behavior of the
>   socket I/O involved in the handshake.
>
> and here's what the docs for do_handshake() says:
>
>   SSLSocket.do_handshake()? Perform a TLS/SSL handshake. If this is used
>   with a non-blocking socket, it may raise SSLError with an arg[0] of
>   SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, in which case it must be
>   called again until it completes successfully. For example, to simulate
>   the behavior of a blocking socket, one might write:
>
>     while True:
>         try:
>             s.do_handshake()
>             break
>         except ssl.SSLError, err:
>             if err.args[0] == ssl.SSL_ERROR_WANT_READ:
>                 select.select([s], [], [])
>             elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE:
>                 select.select([], [s], [])
>             else:
>                 raise
>
> Bill
> _______________________________________________
> Python-Dev mailing list
> Python-... at python.orghttp://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

From gnewsg at gmail.com  Wed Sep 17 03:34:01 2008
From: gnewsg at gmail.com (Giampaolo Rodola')
Date: Tue, 16 Sep 2008 18:34:01 -0700 (PDT)
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <e39d9e1e-af35-4bda-a1a8-fdacb262c9ac@e53g2000hsa.googlegroups.com>
References: <0a7adf05-7811-4078-8a90-3830f0974dab@d77g2000hsb.googlegroups.com>
	<6896.1221447015@parc.com>
	<e39d9e1e-af35-4bda-a1a8-fdacb262c9ac@e53g2000hsa.googlegroups.com>
Message-ID: <e110ef23-508d-4561-b47b-c1aaaec011b7@s50g2000hsb.googlegroups.com>

Sorry, ignore my 2nd question, I see now that you already gave a very
clear answer in your first message.
I change my question: how am I supposed to know when the SSL hanshake
is completed? When pending() returns False?
If so I'd recommend to document the method.


--- Giampaolo
http://code.google.com/p/pyftpdlib/

On 17 Set, 03:24, "Giampaolo Rodola'" <gne... at gmail.com> wrote:
> I've tried to modify my existing asyncore-based code but I'm
> encountering a lot of different problems I didn't manage to fix.
> It seems that playing with the do_handshake_on_connect flag doesn't
> make any difference.
> I guess that without some kind of documentation describing how to deal
> with non-blocking "ssl-wrapped" sockets I won't get too far.
> I try to ask two questions in case the answers may help me in some
> way:
>
> 1 - What pending() method is supposed to do (it's not documented)?
> 2 - By reading ssl.py code I noticed that when do_handshake_on_connect
> flag is False the do_handshake() method is never called. Is it
> supposed to be manually called when dealing with non-blocking sockets?
>
> In the meanwhile I noticed something in the ssl.py code which seems to
> be wrong:
>
> ? ? def recv (self, buflen=1024, flags=0):
> ? ? ? ? if self._sslobj:
> ? ? ? ? ? ? if flags != 0:
> ? ? ? ? ? ? ? ? raise ValueError(
> ? ? ? ? ? ? ? ? ? ? "non-zero flags not allowed in calls to sendall()
> on %s" %
> ? ? ? ? ? ? ? ? ? ? self.__class__)
> ? ? ? ? ? ? while True:
> ? ? ? ? ? ? ? ? try:
> ? ? ? ? ? ? ? ? ? ? return self.read(buflen)
> ? ? ? ? ? ? ? ? except SSLError, x:
> ? ? ? ? ? ? ? ? ? ? if x.args[0] == SSL_ERROR_WANT_READ:
> ? ? ? ? ? ? ? ? ? ? ? ? continue
> ? ? ? ? ? ? ? ? ? ? else:
> ? ? ? ? ? ? ? ? ? ? ? ? raise x
> ? ? ? ? else:
> ? ? ? ? ? ? return socket.recv(self, buflen, flags)
>
> I don't know the low levels but that while statement which continues
> in case of SSL_ERROR_WANT_READ seems to be wrong (blocking), at least
> when dealing with non-blocking sockets. I think the proper way of
> doing recv() here is letting SSL_ERROR_WANT_READ propagate and let the
> upper application (e.g. asyncore) deal with it.
>
> Hope this helps,
>
> --- Giampaolohttp://code.google.com/p/pyftpdlib/downloads/list
>
> On 15 Set, 04:50, Bill Janssen <jans... at parc.com> wrote:
>
>
>
> > Hi, Giampaolo.
>
> > If you look a bit further in Lib/test/test_ssl.py, you'll see a
> > non-blocking use of the "do_handshake" method. ?Basically, the flag
> > "do_handshake_on_connect" says whether you want this to happen
> > automatically and blockingly (True), or whether you want to do it
> > yourself (False). ?In the test suite, the function
> > "testNonBlockingHandshake" does the async client-side handshake; the
> > server side logic is just the same, only it would happen in the server's
> > "handle new connection" code -- you'd have to add a state variable, and
> > bind handlers for "read_event" and "write_event", which would consult
> > the state variable to see whether they had finished the handshake yet.
>
> > I just made the server do it automatically to make life easier.
>
> > The hard part isn't really doing the non-blocking, it's trying to figure
> > out how to use asyncore correctly, IMO.
>
> > Giampaolo Rodola' <gne... at gmail.com> wrote:
> > > I'm interested in using the ssl module with asyncore but since there's
> > > no real documentation about how using ssl module with non-blocking
>
> > If you'd like to contribute a doc patch, that would be great.
>
> > Here's what it current says for do_handshake_on_connect:
>
> > ? The parameter do_handshake_on_connect specifies whether to do the SSL
> > ? handshake automatically after doing a socket.connect(), or whether the
> > ? application program will call it explicitly, by invoking the
> > ? SSLSocket.do_handshake() method. Calling SSLSocket.do_handshake()
> > ? explicitly gives the program control over the blocking behavior of the
> > ? socket I/O involved in the handshake.
>
> > and here's what the docs for do_handshake() says:
>
> > ? SSLSocket.do_handshake()? Perform a TLS/SSL handshake. If this is used
> > ? with a non-blocking socket, it may raise SSLError with an arg[0] of
> > ? SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE, in which case it must be
> > ? called again until it completes successfully. For example, to simulate
> > ? the behavior of a blocking socket, one might write:
>
> > ? ? while True:
> > ? ? ? ? try:
> > ? ? ? ? ? ? s.do_handshake()
> > ? ? ? ? ? ? break
> > ? ? ? ? except ssl.SSLError, err:
> > ? ? ? ? ? ? if err.args[0] == ssl.SSL_ERROR_WANT_READ:
> > ? ? ? ? ? ? ? ? select.select([s], [], [])
> > ? ? ? ? ? ? elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE:
> > ? ? ? ? ? ? ? ? select.select([], [s], [])
> > ? ? ? ? ? ? else:
> > ? ? ? ? ? ? ? ? raise
>
> > Bill
> > _______________________________________________
> > Python-Dev mailing list
> > Python-... at python.orghttp://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...
>
> _______________________________________________
> Python-Dev mailing list
> Python-... at python.orghttp://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...- Nascondi testo citato
>
> - Mostra testo citato -

From martin at v.loewis.de  Wed Sep 17 09:58:31 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 17 Sep 2008 09:58:31 +0200
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <48CE6EC0.8050309@bullseye.andymac.org>
References: <48CE6EC0.8050309@bullseye.andymac.org>
Message-ID: <48D0B8A7.1090402@v.loewis.de>

> I haven't yet tried posting a query to a FreeBSD list, as it could simply
> be a bug on amd64, but I was wondering whether there was anything (other
> than deactivating tests and documenting use of ulimit -v on this
> platform) that could be done to work around this behaviour.

I think it should be possible to debug this (for people with access to
such a system, and appropriate skills).

I find it hard to believe that a large malloc will simply crash the
process, rather than returning NULL. More likely, there is a NULL
returned somewhere, and Python (or libc) fails to check for it.

Regards,
Martni

From jcea at jcea.es  Wed Sep 17 11:59:15 2008
From: jcea at jcea.es (Jesus Cea)
Date: Wed, 17 Sep 2008 11:59:15 +0200
Subject: [Python-Dev] RELEASED Python 2.6rc1
In-Reply-To: <49568982-472B-46BB-9001-12078706B238@python.org>
References: <49568982-472B-46BB-9001-12078706B238@python.org>
Message-ID: <48D0D4F3.6030009@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Barry Warsaw wrote:
> On behalf of the Python development team and the Python community, I am
> happy to announce the first release candidate for Python 2.6.

In http://www.python.org/download/releases/2.6/ , release date for
2.6rc1 is 20-Aug-2008. That is not right.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSNDU6plgi5GaxT1NAQI0bgP8C8PcUBJA8HpSiWuE8W9Fd6p4HwzXysUQ
aA4szns+9FVKbXh0t63M0BGUM0pG4qUgy8lSQevPCxHT0DS2mXWoxAmxDWKthSUa
lU3n515KcXJ0nXr37tCsXay0YlH9qQfKU4KReG3KFQoegMzmB3kW7lwYlkk/bfsM
YmhHjM57WgA=
=sA1F
-----END PGP SIGNATURE-----

From ncoghlan at gmail.com  Wed Sep 17 12:06:18 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 17 Sep 2008 20:06:18 +1000
Subject: [Python-Dev] switching on -3 from within a program?
In-Reply-To: <1afaf6160809161345j213f21a5hfd4f16e0f65d4239@mail.gmail.com>
References: <48CFD185.8070003@avid.com>
	<1afaf6160809161345j213f21a5hfd4f16e0f65d4239@mail.gmail.com>
Message-ID: <48D0D69A.9090906@gmail.com>

Benjamin Peterson wrote:
> def engage_py3k_warning():
>      flag = ctypes.c_int.in_dll(ctypes.pythonapi, "Py_Py3kWarningFlag")
>      flag.value = 1

Note that tricks like this won't necessarily enable all python 3
warnings for modules that have been imported before the flag gets set.
To avoid unnecessary performance penalties as a result of Py3k warnings,
modules are permitted to do things like:

def func():
  # Do whatever

if sys.py3kwarning:
  _orig_func = func
  def func():
    warnings.py3kwarn("Goes away in 3.0")
    _orig_func()

Modules that are first imported after the flag has been set will
obviously have all of their warnings properly enabled.

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From anthon at mnt.org  Wed Sep 17 12:19:03 2008
From: anthon at mnt.org (Anthon van der Neut)
Date: Wed, 17 Sep 2008 12:19:03 +0200
Subject: [Python-Dev] switching on -3 from within a program?
In-Reply-To: <48D0D69A.9090906@gmail.com>
References: <48CFD185.8070003@avid.com>	<1afaf6160809161345j213f21a5hfd4f16e0f65d4239@mail.gmail.com>
	<48D0D69A.9090906@gmail.com>
Message-ID: <48D0D997.1080603@mnt.org>

Hi Nick,

I am aware of that (but others might not, so you are right to point this
out).

I did follow both Christian's and Benjamin's suggestions. The
implementation at the mod_wsgi C level, which is before any module
loading is more permanent, but the ctypes trick doesn't require an
apache2 restart, which is somewhat more comfortable.

Anthon

Nick Coghlan wrote:
> Benjamin Peterson wrote:
>> def engage_py3k_warning():
>>      flag = ctypes.c_int.in_dll(ctypes.pythonapi, "Py_Py3kWarningFlag")
>>      flag.value = 1
> 
> Note that tricks like this won't necessarily enable all python 3
> warnings for modules that have been imported before the flag gets set.
> To avoid unnecessary performance penalties as a result of Py3k warnings,
> modules are permitted to do things like:
> 
> def func():
>   # Do whatever
> 
> if sys.py3kwarning:
>   _orig_func = func
>   def func():
>     warnings.py3kwarn("Goes away in 3.0")
>     _orig_func()
> 
> Modules that are first imported after the flag has been set will
> obviously have all of their warnings properly enabled.
> 
> Cheers,
> Nick.
> 
> 

From barry at python.org  Wed Sep 17 15:46:19 2008
From: barry at python.org (Barry Warsaw)
Date: Wed, 17 Sep 2008 09:46:19 -0400
Subject: [Python-Dev] RELEASED Python 2.6rc1
In-Reply-To: <48D0D4F3.6030009@jcea.es>
References: <49568982-472B-46BB-9001-12078706B238@python.org>
	<48D0D4F3.6030009@jcea.es>
Message-ID: <8628A3D1-D09F-4B69-BB9E-A7B8B7F481B3@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 17, 2008, at 5:59 AM, Jesus Cea wrote:
>
> Barry Warsaw wrote:
>> On behalf of the Python development team and the Python community,  
>> I am
>> happy to announce the first release candidate for Python 2.6.
>
> In http://www.python.org/download/releases/2.6/ , release date for
> 2.6rc1 is 20-Aug-2008. That is not right.

Fixed, thanks.
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSNEKK3EjvBPtnXfVAQKscgP+LXF7eWCvQxdgQ1dREvA7OeSyh60FGvBW
SHMv0aHKyZgDZlgpjOB4kttVzYNWB0mm0jHfhuYvJ60CygNVCUCjxgZx8f6SK4B/
IaMjWoFH9e4P616jwknfsvxsmZhqzRF3eyIAz3vl7U9fQWv/jsjtaQl5FcIKuH10
33xxz4TH9hU=
=S0s8
-----END PGP SIGNATURE-----

From andymac at bullseye.apana.org.au  Wed Sep 17 16:53:07 2008
From: andymac at bullseye.apana.org.au (Andrew MacIntyre)
Date: Thu, 18 Sep 2008 00:53:07 +1000
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <48D0B8A7.1090402@v.loewis.de>
References: <48CE6EC0.8050309@bullseye.andymac.org>
	<48D0B8A7.1090402@v.loewis.de>
Message-ID: <48D119D3.5030708@bullseye.andymac.org>

Martin v. L?wis wrote:
>> I haven't yet tried posting a query to a FreeBSD list, as it could simply
>> be a bug on amd64, but I was wondering whether there was anything (other
>> than deactivating tests and documenting use of ulimit -v on this
>> platform) that could be done to work around this behaviour.
> 
> I think it should be possible to debug this (for people with access to
> such a system, and appropriate skills).
> 
> I find it hard to believe that a large malloc will simply crash the
> process, rather than returning NULL. More likely, there is a NULL
> returned somewhere, and Python (or libc) fails to check for it.

A simple C program doing a repetitive malloc(), much as pymalloc would
with continuous demand, does indeed not see any NULL from malloc() when
swap is exhausted but the process gets KILLed (the allocated memory does
have to be written to to force the situation...)

I'll take this up with FreeBSD folk, but I'm open to ideas as to how
best to deal with the problem in the context of the test suite pending
resolution by FreeBSD.

Regards,
Andrew.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia


From foom at fuhm.net  Wed Sep 17 17:00:41 2008
From: foom at fuhm.net (James Y Knight)
Date: Wed, 17 Sep 2008 11:00:41 -0400
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <48D119D3.5030708@bullseye.andymac.org>
References: <48CE6EC0.8050309@bullseye.andymac.org>
	<48D0B8A7.1090402@v.loewis.de>
	<48D119D3.5030708@bullseye.andymac.org>
Message-ID: <D29998F3-140A-49B2-8637-28FB1792DCA3@fuhm.net>


On Sep 17, 2008, at 10:53 AM, Andrew MacIntyre wrote:

> Martin v. L?wis wrote:
>>> I haven't yet tried posting a query to a FreeBSD list, as it could  
>>> simply
>>> be a bug on amd64, but I was wondering whether there was anything  
>>> (other
>>> than deactivating tests and documenting use of ulimit -v on this
>>> platform) that could be done to work around this behaviour.
>> I think it should be possible to debug this (for people with access  
>> to
>> such a system, and appropriate skills).
>> I find it hard to believe that a large malloc will simply crash the
>> process, rather than returning NULL. More likely, there is a NULL
>> returned somewhere, and Python (or libc) fails to check for it.
>
> A simple C program doing a repetitive malloc(), much as pymalloc would
> with continuous demand, does indeed not see any NULL from malloc()  
> when
> swap is exhausted but the process gets KILLed (the allocated memory  
> does
> have to be written to to force the situation...)
>
> I'll take this up with FreeBSD folk, but I'm open to ideas as to how
> best to deal with the problem in the context of the test suite pending
> resolution by FreeBSD.

Linux does the same thing, unless the user has explicitly configured  
that behavior off. Search the web for linux overcommit. It's  
controlled by the vm.overcommit_memory sysctl. Although linux's  
default is some heuristic that might make Python's test case work  
right in most cases, depending on malloc returning NULL is not  
something you can actually depend on.

James

From aleaxit at gmail.com  Wed Sep 17 17:21:55 2008
From: aleaxit at gmail.com (Alex Martelli)
Date: Wed, 17 Sep 2008 08:21:55 -0700
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <D29998F3-140A-49B2-8637-28FB1792DCA3@fuhm.net>
References: <48CE6EC0.8050309@bullseye.andymac.org>
	<48D0B8A7.1090402@v.loewis.de> <48D119D3.5030708@bullseye.andymac.org>
	<D29998F3-140A-49B2-8637-28FB1792DCA3@fuhm.net>
Message-ID: <e8a0972d0809170821k124aff81p3d19cbacae901398@mail.gmail.com>

Unbelievable as this may seem, this crazy over-committing malloc
behavior is by now "a classic" -- I first fought against it in 1990,
when IBM released AIX 3 for its then-new RS/6000 line of workstations;
in a later minor release they did provide a way to optionally switch
this off, but, like on Linux, it's a system-wide switch, NOT
per-process:-(.

I concur with http://www.win.tue.nl/~aeb/linux/lk/lk-9.html (the best
explanation I know of the subject, and recommended reading) which, on
this subject, says "Linux on the other hand is seriously broken."
(just like AIX 3 was).  Sad to learn that BSD is now also broken in
the same way:-(.


Alex
On Wed, Sep 17, 2008 at 8:00 AM, James Y Knight <foom at fuhm.net> wrote:
>
> On Sep 17, 2008, at 10:53 AM, Andrew MacIntyre wrote:
>
>> Martin v. L?wis wrote:
>>>>
>>>> I haven't yet tried posting a query to a FreeBSD list, as it could
>>>> simply
>>>> be a bug on amd64, but I was wondering whether there was anything (other
>>>> than deactivating tests and documenting use of ulimit -v on this
>>>> platform) that could be done to work around this behaviour.
>>>
>>> I think it should be possible to debug this (for people with access to
>>> such a system, and appropriate skills).
>>> I find it hard to believe that a large malloc will simply crash the
>>> process, rather than returning NULL. More likely, there is a NULL
>>> returned somewhere, and Python (or libc) fails to check for it.
>>
>> A simple C program doing a repetitive malloc(), much as pymalloc would
>> with continuous demand, does indeed not see any NULL from malloc() when
>> swap is exhausted but the process gets KILLed (the allocated memory does
>> have to be written to to force the situation...)
>>
>> I'll take this up with FreeBSD folk, but I'm open to ideas as to how
>> best to deal with the problem in the context of the test suite pending
>> resolution by FreeBSD.
>
> Linux does the same thing, unless the user has explicitly configured that
> behavior off. Search the web for linux overcommit. It's controlled by the
> vm.overcommit_memory sysctl. Although linux's default is some heuristic that
> might make Python's test case work right in most cases, depending on malloc
> returning NULL is not something you can actually depend on.
>
> James
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com
>

From martin at v.loewis.de  Wed Sep 17 17:45:05 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Wed, 17 Sep 2008 17:45:05 +0200
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <48D119D3.5030708@bullseye.andymac.org>
References: <48CE6EC0.8050309@bullseye.andymac.org>
	<48D0B8A7.1090402@v.loewis.de>
	<48D119D3.5030708@bullseye.andymac.org>
Message-ID: <48D12601.8020102@v.loewis.de>

> I'll take this up with FreeBSD folk, but I'm open to ideas as to how
> best to deal with the problem in the context of the test suite pending
> resolution by FreeBSD.

Not sure what the test purpose is: if it is to test that you get a
MemoryError in cases where you ask for more than Python could represent,
and the test errs on system where the requested size is actually
representable, the solution then is to fix the test case.

If the test purpose is to trigger a memory error for cases when the
system runs out of memory, the test case should set a ulimit to less
than the physical memory.

It might be useful to have an interpreter-maintained limit on the amount
of memory Python can consume, but such a feature is clearly out of scope
for the current state of the code.

Regards,
Martin

From jon+python-dev at unequivocal.co.uk  Wed Sep 17 18:34:13 2008
From: jon+python-dev at unequivocal.co.uk (Jon Ribbens)
Date: Wed, 17 Sep 2008 17:34:13 +0100
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <e8a0972d0809170821k124aff81p3d19cbacae901398@mail.gmail.com>
References: <48CE6EC0.8050309@bullseye.andymac.org>
	<48D0B8A7.1090402@v.loewis.de>
	<48D119D3.5030708@bullseye.andymac.org>
	<D29998F3-140A-49B2-8637-28FB1792DCA3@fuhm.net>
	<e8a0972d0809170821k124aff81p3d19cbacae901398@mail.gmail.com>
Message-ID: <20080917163413.GU30517@snowy.squish.net>

On Wed, Sep 17, 2008 at 08:21:55AM -0700, Alex Martelli wrote:
> Unbelievable as this may seem, this crazy over-committing malloc
> behavior is by now "a classic" -- I first fought against it in 1990,
> when IBM released AIX 3 for its then-new RS/6000 line of workstations;
> in a later minor release they did provide a way to optionally switch
> this off, but, like on Linux, it's a system-wide switch, NOT
> per-process:-(.
> 
> I concur with http://www.win.tue.nl/~aeb/linux/lk/lk-9.html (the best
> explanation I know of the subject, and recommended reading) which, on
> this subject, says "Linux on the other hand is seriously broken."
> (just like AIX 3 was).  Sad to learn that BSD is now also broken in
> the same way:-(.

It's now "now" also broken, it has been that way for a very long time.
For example, see this message I wrote back in July 1999 complaining
about FreeBSD overcommit:
http://www.mail-archive.com/freebsd-hackers at freebsd.org/msg01056.html

From janssen at parc.com  Wed Sep 17 18:46:25 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 17 Sep 2008 09:46:25 PDT
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <e39d9e1e-af35-4bda-a1a8-fdacb262c9ac@e53g2000hsa.googlegroups.com>
References: <0a7adf05-7811-4078-8a90-3830f0974dab@d77g2000hsb.googlegroups.com>
	<6896.1221447015@parc.com>
	<e39d9e1e-af35-4bda-a1a8-fdacb262c9ac@e53g2000hsa.googlegroups.com>
Message-ID: <44166.1221669985@parc.com>

Giampaolo Rodola' <gnewsg at gmail.com> wrote:

> 2 - By reading ssl.py code I noticed that when do_handshake_on_connect
> flag is False the do_handshake() method is never called. Is it
> supposed to be manually called when dealing with non-blocking sockets?

Yes.  Look at the example client in Lib/test/test_ssl.py.  The server
code should work the same way.

Bill

From janssen at parc.com  Wed Sep 17 18:46:59 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 17 Sep 2008 09:46:59 PDT
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <e110ef23-508d-4561-b47b-c1aaaec011b7@s50g2000hsb.googlegroups.com>
References: <0a7adf05-7811-4078-8a90-3830f0974dab@d77g2000hsb.googlegroups.com>
	<6896.1221447015@parc.com>
	<e39d9e1e-af35-4bda-a1a8-fdacb262c9ac@e53g2000hsa.googlegroups.com>
	<e110ef23-508d-4561-b47b-c1aaaec011b7@s50g2000hsb.googlegroups.com>
Message-ID: <44174.1221670019@parc.com>

Giampaolo Rodola' <gnewsg at gmail.com> wrote:

> I change my question: how am I supposed to know when the SSL hanshake
> is completed? When pending() returns False?

When do_handshake() doesn't raise an exception.

Bill

From janssen at parc.com  Wed Sep 17 19:07:03 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 17 Sep 2008 10:07:03 PDT
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <e39d9e1e-af35-4bda-a1a8-fdacb262c9ac@e53g2000hsa.googlegroups.com>
References: <0a7adf05-7811-4078-8a90-3830f0974dab@d77g2000hsb.googlegroups.com>
	<6896.1221447015@parc.com>
	<e39d9e1e-af35-4bda-a1a8-fdacb262c9ac@e53g2000hsa.googlegroups.com>
Message-ID: <44889.1221671223@parc.com>

Giampaolo Rodola' <gnewsg at gmail.com> wrote:

> In the meanwhile I noticed something in the ssl.py code which seems to
> be wrong:
> 
>     def recv (self, buflen=1024, flags=0):
>         if self._sslobj:
>             if flags != 0:
>                 raise ValueError(
>                     "non-zero flags not allowed in calls to sendall()
> on %s" %
>                     self.__class__)
>             while True:
>                 try:
>                     return self.read(buflen)
>                 except SSLError, x:
>                     if x.args[0] == SSL_ERROR_WANT_READ:
>                         continue
>                     else:
>                         raise x
>         else:
>             return socket.recv(self, buflen, flags)
> 
> I don't know the low levels but that while statement which continues
> in case of SSL_ERROR_WANT_READ seems to be wrong (blocking), at least
> when dealing with non-blocking sockets. I think the proper way of
> doing recv() here is letting SSL_ERROR_WANT_READ propagate and let the
> upper application (e.g. asyncore) deal with it.

It's an interesting point.  I'm not sure the underlying code will ever
raise this exception.  Please file a bug report to help us track this.

Thanks.

Bill

From arfrever.fta at gmail.com  Wed Sep 17 19:27:54 2008
From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis)
Date: Wed, 17 Sep 2008 19:27:54 +0200
Subject: [Python-Dev] RELEASED Python 2.6rc1
In-Reply-To: <8628A3D1-D09F-4B69-BB9E-A7B8B7F481B3@python.org>
References: <49568982-472B-46BB-9001-12078706B238@python.org>
	<48D0D4F3.6030009@jcea.es>
	<8628A3D1-D09F-4B69-BB9E-A7B8B7F481B3@python.org>
Message-ID: <200809171928.38884.Arfrever.FTA@gmail.com>

2008-09-17 15:46:19 Barry Warsaw napisa?(a):
> On Sep 17, 2008, at 5:59 AM, Jesus Cea wrote:
> >
> > Barry Warsaw wrote:
> >> On behalf of the Python development team and the Python community,
> >> I am
> >> happy to announce the first release candidate for Python 2.6.
> >
> > In http://www.python.org/download/releases/2.6/ , release date for
> > 2.6rc1 is 20-Aug-2008. That is not right.
> 
> Fixed, thanks.

Release date for 2.6rc1 should be 12-Sep-2008 instead of 17-Sep-2008.

-- 
Arfrever Frehtes Taifersar Arahesis
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: This is a digitally signed message part.
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080917/dff30cc9/attachment.pgp>

From janssen at parc.com  Wed Sep 17 19:40:01 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 17 Sep 2008 10:40:01 PDT
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <44889.1221671223@parc.com>
References: <0a7adf05-7811-4078-8a90-3830f0974dab@d77g2000hsb.googlegroups.com>
	<6896.1221447015@parc.com>
	<e39d9e1e-af35-4bda-a1a8-fdacb262c9ac@e53g2000hsa.googlegroups.com>
	<44889.1221671223@parc.com>
Message-ID: <56365.1221673201@parc.com>

Ah, now I remember.  It seems that sometimes when SSL_ERROR_WANT_READ
was returned, things would block; that is, the "handle_read" method on
asyncore.dispatcher was never called again, so the SSLSocket.recv()
method was never re-called.  There are several levels of buffering going
on, and I never figured out just why that was.  This (very rare) re-call
of "read" is to handle that.

Bill

Bill Janssen <janssen at parc.com> wrote:

> Giampaolo Rodola' <gnewsg at gmail.com> wrote:
> 
> > In the meanwhile I noticed something in the ssl.py code which seems to
> > be wrong:
> > 
> >     def recv (self, buflen=1024, flags=0):
> >         if self._sslobj:
> >             if flags != 0:
> >                 raise ValueError(
> >                     "non-zero flags not allowed in calls to sendall()
> > on %s" %
> >                     self.__class__)
> >             while True:
> >                 try:
> >                     return self.read(buflen)
> >                 except SSLError, x:
> >                     if x.args[0] == SSL_ERROR_WANT_READ:
> >                         continue
> >                     else:
> >                         raise x
> >         else:
> >             return socket.recv(self, buflen, flags)
> > 
> > I don't know the low levels but that while statement which continues
> > in case of SSL_ERROR_WANT_READ seems to be wrong (blocking), at least
> > when dealing with non-blocking sockets. I think the proper way of
> > doing recv() here is letting SSL_ERROR_WANT_READ propagate and let the
> > upper application (e.g. asyncore) deal with it.
> 
> It's an interesting point.  I'm not sure the underlying code will ever
> raise this exception.  Please file a bug report to help us track this.
> 
> Thanks.
> 
> Bill
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/janssen%40parc.com

From exarkun at divmod.com  Wed Sep 17 20:37:26 2008
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Wed, 17 Sep 2008 14:37:26 -0400
Subject: [Python-Dev] ssl module,
 non-blocking sockets and asyncore integration
In-Reply-To: <56365.1221673201@parc.com>
Message-ID: <20080917183726.29191.1713347132.divmod.quotient.27416@ohm>

On Wed, 17 Sep 2008 10:40:01 PDT, Bill Janssen <janssen at parc.com> wrote:
>Ah, now I remember.  It seems that sometimes when SSL_ERROR_WANT_READ
>was returned, things would block; that is, the "handle_read" method on
>asyncore.dispatcher was never called again, so the SSLSocket.recv()
>method was never re-called.  There are several levels of buffering going
>on, and I never figured out just why that was.  This (very rare) re-call
>of "read" is to handle that.
>

You certainly do need to call read again if OpenSSL fails an SSL_read with
a want-read error, but in asyncore, you don't want to do it right away,
you want to wait until the socket becomes readable again, otherwise you *do*
block waiting for bytes from the network.  See the SSL support in Twisted
for an example of the correct way to handle this.

Jean-Paul

From janssen at parc.com  Wed Sep 17 20:42:38 2008
From: janssen at parc.com (Bill Janssen)
Date: Wed, 17 Sep 2008 11:42:38 PDT
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <20080917183726.29191.1713347132.divmod.quotient.27416@ohm>
References: <20080917183726.29191.1713347132.divmod.quotient.27416@ohm>
Message-ID: <57704.1221676958@parc.com>

Jean-Paul Calderone <exarkun at divmod.com> wrote:

> On Wed, 17 Sep 2008 10:40:01 PDT, Bill Janssen <janssen at parc.com> wrote:
> >Ah, now I remember.  It seems that sometimes when SSL_ERROR_WANT_READ
> >was returned, things would block; that is, the "handle_read" method on
> >asyncore.dispatcher was never called again, so the SSLSocket.recv()
> >method was never re-called.  There are several levels of buffering going
> >on, and I never figured out just why that was.  This (very rare) re-call
> >of "read" is to handle that.
> >
> 
> You certainly do need to call read again if OpenSSL fails an SSL_read with
> a want-read error, but in asyncore, you don't want to do it right away,
> you want to wait until the socket becomes readable again, otherwise you *do*
> block waiting for bytes from the network.  See the SSL support in Twisted
> for an example of the correct way to handle this.
> 
> Jean-Paul

Yes, I understand, and that's how I started out.  The bug we were seeing
was that "handle_read" wasn't being called again by asyncore.

Bill

From santagada at gmail.com  Wed Sep 17 21:33:32 2008
From: santagada at gmail.com (Leonardo Santagada)
Date: Wed, 17 Sep 2008 16:33:32 -0300
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <48D12601.8020102@v.loewis.de>
References: <48CE6EC0.8050309@bullseye.andymac.org>
	<48D0B8A7.1090402@v.loewis.de>
	<48D119D3.5030708@bullseye.andymac.org>
	<48D12601.8020102@v.loewis.de>
Message-ID: <4DD82BD6-2D30-44BE-B165-60C8348EC19D@gmail.com>


On Sep 17, 2008, at 12:45 PM, Martin v. L?wis wrote:

>> I'll take this up with FreeBSD folk, but I'm open to ideas as to how
>> best to deal with the problem in the context of the test suite  
>> pending
>> resolution by FreeBSD.
>
> Not sure what the test purpose is: if it is to test that you get a
> MemoryError in cases where you ask for more than Python could  
> represent,
> and the test errs on system where the requested size is actually
> representable, the solution then is to fix the test case.
>
> If the test purpose is to trigger a memory error for cases when the
> system runs out of memory, the test case should set a ulimit to less
> than the physical memory.
>
> It might be useful to have an interpreter-maintained limit on the  
> amount
> of memory Python can consume, but such a feature is clearly out of  
> scope
> for the current state of the code.


There is an option at least on linux to hack using ld preload to use  
another memory manager that respond the way needed... at least that  
was what I was told today at lunch. (if ulimit is not enough for any  
reason).


--
Leonardo Santagada
santagada at gmail.com




From martin at v.loewis.de  Wed Sep 17 21:52:55 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Wed, 17 Sep 2008 21:52:55 +0200
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <4DD82BD6-2D30-44BE-B165-60C8348EC19D@gmail.com>
References: <48CE6EC0.8050309@bullseye.andymac.org>
	<48D0B8A7.1090402@v.loewis.de>
	<48D119D3.5030708@bullseye.andymac.org>
	<48D12601.8020102@v.loewis.de>
	<4DD82BD6-2D30-44BE-B165-60C8348EC19D@gmail.com>
Message-ID: <48D16017.8030708@v.loewis.de>

> There is an option at least on linux to hack using ld preload to use
> another memory manager that respond the way needed... at least that was
> what I was told today at lunch. (if ulimit is not enough for any reason).

For Python, there would be much less hackish ways. Most if not all calls
to memory allocators are channeled through Python API, which could be
diverted on source level also.

Regards,
Martin

From ncoghlan at gmail.com  Wed Sep 17 23:47:03 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 18 Sep 2008 07:47:03 +1000
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <57704.1221676958@parc.com>
References: <20080917183726.29191.1713347132.divmod.quotient.27416@ohm>
	<57704.1221676958@parc.com>
Message-ID: <48D17AD7.5030901@gmail.com>

Bill Janssen wrote:
> Jean-Paul Calderone <exarkun at divmod.com> wrote:
> 
>> On Wed, 17 Sep 2008 10:40:01 PDT, Bill Janssen <janssen at parc.com> wrote:
>>> Ah, now I remember.  It seems that sometimes when SSL_ERROR_WANT_READ
>>> was returned, things would block; that is, the "handle_read" method on
>>> asyncore.dispatcher was never called again, so the SSLSocket.recv()
>>> method was never re-called.  There are several levels of buffering going
>>> on, and I never figured out just why that was.  This (very rare) re-call
>>> of "read" is to handle that.
>>>
>> You certainly do need to call read again if OpenSSL fails an SSL_read with
>> a want-read error, but in asyncore, you don't want to do it right away,
>> you want to wait until the socket becomes readable again, otherwise you *do*
>> block waiting for bytes from the network.  See the SSL support in Twisted
>> for an example of the correct way to handle this.
>>
>> Jean-Paul
> 
> Yes, I understand, and that's how I started out.  The bug we were seeing
> was that "handle_read" wasn't being called again by asyncore.

It's probably worth sticking a comment in the code explaining why we're
taking the risk of temporarily blocking on a non-blocking socket (i.e.
until someone figures out how to reproduce that bug reliably so that a
more correct fix can be devised).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From gnewsg at gmail.com  Thu Sep 18 00:49:37 2008
From: gnewsg at gmail.com (Giampaolo Rodola')
Date: Wed, 17 Sep 2008 15:49:37 -0700 (PDT)
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <48D17AD7.5030901@gmail.com>
References: <20080917183726.29191.1713347132.divmod.quotient.27416@ohm> 
	<57704.1221676958@parc.com> <48D17AD7.5030901@gmail.com>
Message-ID: <240ddaf5-a361-4f8f-981c-c11855b910eb@c58g2000hsc.googlegroups.com>

Ok, here's some news, in case they can be of some interest.
I managed to write an asyncore disptacher which seems to work.
I used my test suite against it and 70 tests passed and 2 failed.
The tests failed because at a certain point a call to do_handhsake
results in an EOF exception, which is very strange since it is
supposed to raise SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE only.
I'll keep you updated in case I have some news.



--- Exception ---

  File "C:\python26\lib\ssl.py", line 293, in do_handshake
    self._sslobj.do_handshake()
SSLError: [Errno 8] _ssl.c:480: EOF occurred in violation of protocol



--- SSL dispatcher ----

class SSLConnection(asyncore.dispatcher):

    def __init__(self):
        self.ssl_handshake_pending = False

    def do_ssl_handshake(self):
        try:
            self.socket.do_handshake()
        except ssl.SSLError, err:
            if err.args[0] == ssl.SSL_ERROR_WANT_READ:
                self.ssl_handshake_pending = 'read'
            elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE:
                self.ssl_handshake_pending = 'write'
            else:
                raise
        else:
            self.ssl_handshake_pending = False

    def handle_read_event(self):
        if self.ssl_handshake_pending == 'read':
            self.do_ssl_handshake()
##            if not self.ssl_handshake_pending:
##                asyncore.dispatcher.handle_read_event(self)
        else:
            asyncore.dispatcher.handle_read_event(self)

    def handle_write_event(self):
        if self.ssl_handshake_pending == 'write':
            self.do_ssl_handshake()
##            if not self.ssl_handshake_pending:
##                asyncore.dispatcher.handle_write_event(self)
        else:
            asyncore.dispatcher.handle_write_event(self)



--- Giampaolo
http://code.google.com/p/pyftpdlib/

From Graham.Dumpleton at gmail.com  Thu Sep 18 01:39:45 2008
From: Graham.Dumpleton at gmail.com (Graham Dumpleton)
Date: Wed, 17 Sep 2008 16:39:45 -0700 (PDT)
Subject: [Python-Dev] switching on -3 from within a program?
In-Reply-To: <48CFD185.8070003@avid.com>
References: <48CFD185.8070003@avid.com>
Message-ID: <6ed8fd33-1a84-4423-bb09-cd3f42ea2411@25g2000prz.googlegroups.com>

On Sep 17, 1:32?am, Anthon van der Neut <avandern... at avid.com> wrote:
> With a minor modification to the Makefile I was able to getmod_wsgi
> v2.3 to work with python2.6rc1.

Which I believe was still required only because you have only
installed static Python library. :-)

> I promptly got a warning in my apache log files on the depcrecated use
> of 'sha' in the paste's cookie module, good! And easily fixed.
>
> After that I was looking for a way to switch on the -3 warnings from
> within my code to have extra warnings show up in my apache log file.

Check out mod_wsgi from subversion repository:

 svn co http://modwsgi.googlecode.com/svn/trunk/mod_wsgi

and build that.

When Pyhton 2.6 is being used, you should be able to have in Apache
configuration file at server scope.

  WSGIPy3kWarningFlag On

You must not be loading mod_python at same time else this will not be
honoured.

Graham

> After reading some documentation I tried from the python2.6 prompt:
>
> import sys
> sys.py3kwarning = True
> x = { 'abc': 1 }; x.has_key('abc')
>
> which does not give a warning (starting python2.6 with the -3 option of
> course does).
>
> Is there anyway to switch this on from within a program with a Python
> statement?
> If not, would I need to make a small C-extension module (to be imported
> as the first module) that sets Py_Py3WarningFlag or something else at
> the C level, or would that better be done bymod_wsgi'sC code.
>
> Regards
> Anthon
> _______________________________________________
> Python-Dev mailing list
> Python-... at python.orghttp://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

From barry at python.org  Thu Sep 18 07:40:18 2008
From: barry at python.org (Barry Warsaw)
Date: Thu, 18 Sep 2008 01:40:18 -0400
Subject: [Python-Dev] RELEASED Python 2.6rc2 and 3.0rc1
Message-ID: <D1B63EBE-3BA3-46EC-B8A3-F4AC81F0C50D@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On behalf of the Python development team and the Python community, I  
am happy to announce the second and final planned release candidate  
for Python 2.6, as well as the first release candidate for Python 3.0.

These are release candidates, so while they are not suitable for  
production environments, we strongly encourage you to download and  
test them on your software.  We expect only critical bugs to be fixed  
between now and the final releases.  Currently Python 2.6 is scheduled  
for October 1st, 2008.  Python 3.0 release candidate 2 is planned for  
October 1st, with the final release planned for October 15, 2008.

If you find things broken or incorrect, please submit bug reports at

     http://bugs.python.org

For more information and downloadable distributions, see the Python  
2.6 website:

     http://www.python.org/download/releases/2.6/

and the Python 3.0 web site:

     http://www.python.org/download/releases/3.0/

See PEP 361 for release schedule details:

     http://www.python.org/dev/peps/pep-0361/

Enjoy,
- -Barry

Barry Warsaw
barry at python.org
Python 2.6/3.0 Release Manager
(on behalf of the entire python-dev team)

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSNHpw3EjvBPtnXfVAQLW9wP/RBCaUvhuheIh+BjLLIHQFBQi7D3uVgqi
l0+4fhhoKGJvtWklLfSM9I1prcjH/d6tzUu4fIOjX7aM+wZSG++vkfmBoehnhyZW
AvU9Lax4mqDwhOJA2QA0WMx0obpYYVHeUl7D1g9kWzbRUkZDX9NZGMWThhEOC1qA
UA3bBYbvWiQ=
=BFNH
-----END PGP SIGNATURE-----

From andymac at bullseye.apana.org.au  Thu Sep 18 13:20:14 2008
From: andymac at bullseye.apana.org.au (Andrew MacIntyre)
Date: Thu, 18 Sep 2008 21:20:14 +1000
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <48D119D3.5030708@bullseye.andymac.org>
References: <48CE6EC0.8050309@bullseye.andymac.org>	<48D0B8A7.1090402@v.loewis.de>
	<48D119D3.5030708@bullseye.andymac.org>
Message-ID: <48D2396E.9010200@bullseye.andymac.org>

Andrew MacIntyre wrote:

> I'll take this up with FreeBSD folk, but I'm open to ideas as to how
> best to deal with the problem in the context of the test suite pending
> resolution by FreeBSD.

The response I got from Jason Evans (author of the new malloc()
implementation), along with that of another respondent, indicates that
the behaviour on FreeBSD 7.1 and later will (mostly) be restored to that
similar to 6.x and earlier through the default use of sbrk() and
consequent obedience to the data segment size limit (ulimit -d) - which
defaults to 512MB in a standard FreeBSD install in recent times.

The residual problem (as of 7.1) is that malloc() defaults to falling
back to the mmap() strategy when it can't get more address space via
sbrk().  As noted in the tracker item for issue 3862, the only way to
control this is the virtual memory size limit (ulimit -v), which
unfortunately defaults to "unlimited"...

FreeBSD's malloc() can be tuned in several ways, so it is possible to
force use of the sbrk() only strategy (as of 7.1) which would exactly
match behaviour of the old malloc().

It seems to me that the most practical way forward is to just institute a
policy that tests that want to try and test out of memory behaviour must
ensure that appropriate resource limits are in place; if they can't (such
as because the platform running the tests doesn't support
getrlimit()/setrlimit()) the test should be skipped.

As Mark Dickinson has suggested a patch for issue 3862 which should worm
around the issue with test_array on 64 bit platforms, I think we can move
forward for the time being.

Cheers,
Andrew.

-- 
-------------------------------------------------------------------------
Andrew I MacIntyre                     "These thoughts are mine alone..."
E-mail: andymac at bullseye.apana.org.au  (pref) | Snail: PO Box 370
        andymac at pcug.org.au             (alt) |        Belconnen ACT 2616
Web:    http://www.andymac.org/               |        Australia

From martin at v.loewis.de  Thu Sep 18 15:10:49 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 18 Sep 2008 15:10:49 +0200
Subject: [Python-Dev] FreeBSD 7 amd64 and large memory tests
In-Reply-To: <48D2396E.9010200@bullseye.andymac.org>
References: <48CE6EC0.8050309@bullseye.andymac.org>	<48D0B8A7.1090402@v.loewis.de>	<48D119D3.5030708@bullseye.andymac.org>
	<48D2396E.9010200@bullseye.andymac.org>
Message-ID: <48D25359.8050004@v.loewis.de>

> It seems to me that the most practical way forward is to just institute a
> policy that tests that want to try and test out of memory behaviour must
> ensure that appropriate resource limits are in place

IMO, there shouldn't be any tests in the test suite that rely on
exhaustion of all available memory. The MemoryError tests should all
deal with overflow situations only.

If stress-testing is desired, it should be done with platform support,
i.e. with a malloc implementation that randomly fails. OTOH, I would
hope that the static-analysis tools that Python gets run through find
failures to properly check for NULL results much better than
stress-testing.

Regards,
Martin

From gnewsg at gmail.com  Thu Sep 18 15:38:59 2008
From: gnewsg at gmail.com (Giampaolo Rodola')
Date: Thu, 18 Sep 2008 06:38:59 -0700 (PDT)
Subject: [Python-Dev] ssl module,
	non-blocking sockets and asyncore integration
In-Reply-To: <240ddaf5-a361-4f8f-981c-c11855b910eb@c58g2000hsc.googlegroups.com>
References: <20080917183726.29191.1713347132.divmod.quotient.27416@ohm> 
	<57704.1221676958@parc.com> <48D17AD7.5030901@gmail.com>
	<240ddaf5-a361-4f8f-981c-c11855b910eb@c58g2000hsc.googlegroups.com>
Message-ID: <53f7a9ad-897d-41ed-a482-2c4bb88335cc@8g2000hse.googlegroups.com>

Some good news: I finally figured out how to modify asyncore to make
it properly handle the non-blocking ssl-handshake.
I provided a patch for test_ssl.py in issue 3899.
Bill, could you please review it?


--- Giampaolo
http://code.google.com/p/pyftpdlib/

On 18 Set, 00:49, "Giampaolo Rodola'" <gne... at gmail.com> wrote:
> Ok, here's some news, in case they can be of some interest.
> I managed to write an asyncore disptacher which seems to work.
> I used my test suite against it and 70 tests passed and 2 failed.
> The tests failed because at a certain point a call to do_handhsake
> results in an EOF exception, which is very strange since it is
> supposed to raise SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE only.
> I'll keep you updated in case I have some news.
>
> --- Exception ---
>
> ? File "C:\python26\lib\ssl.py", line 293, in do_handshake
> ? ? self._sslobj.do_handshake()
> SSLError: [Errno 8] _ssl.c:480: EOF occurred in violation of protocol
>
> --- SSL dispatcher ----
>
> class SSLConnection(asyncore.dispatcher):
>
> ? ? def __init__(self):
> ? ? ? ? self.ssl_handshake_pending = False
>
> ? ? def do_ssl_handshake(self):
> ? ? ? ? try:
> ? ? ? ? ? ? self.socket.do_handshake()
> ? ? ? ? except ssl.SSLError, err:
> ? ? ? ? ? ? if err.args[0] == ssl.SSL_ERROR_WANT_READ:
> ? ? ? ? ? ? ? ? self.ssl_handshake_pending = 'read'
> ? ? ? ? ? ? elif err.args[0] == ssl.SSL_ERROR_WANT_WRITE:
> ? ? ? ? ? ? ? ? self.ssl_handshake_pending = 'write'
> ? ? ? ? ? ? else:
> ? ? ? ? ? ? ? ? raise
> ? ? ? ? else:
> ? ? ? ? ? ? self.ssl_handshake_pending = False
>
> ? ? def handle_read_event(self):
> ? ? ? ? if self.ssl_handshake_pending == 'read':
> ? ? ? ? ? ? self.do_ssl_handshake()
> ## ? ? ? ? ? ?if not self.ssl_handshake_pending:
> ## ? ? ? ? ? ? ? ?asyncore.dispatcher.handle_read_event(self)
> ? ? ? ? else:
> ? ? ? ? ? ? asyncore.dispatcher.handle_read_event(self)
>
> ? ? def handle_write_event(self):
> ? ? ? ? if self.ssl_handshake_pending == 'write':
> ? ? ? ? ? ? self.do_ssl_handshake()
> ## ? ? ? ? ? ?if not self.ssl_handshake_pending:
> ## ? ? ? ? ? ? ? ?asyncore.dispatcher.handle_write_event(self)
> ? ? ? ? else:
> ? ? ? ? ? ? asyncore.dispatcher.handle_write_event(self)
>
> --- Giampaolohttp://code.google.com/p/pyftpdlib/
> _______________________________________________
> Python-Dev mailing list
> Python-... at python.orghttp://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...

From jcea at jcea.es  Thu Sep 18 16:24:58 2008
From: jcea at jcea.es (Jesus Cea)
Date: Thu, 18 Sep 2008 16:24:58 +0200
Subject: [Python-Dev] bsddb tests disabled by default
Message-ID: <48D264BA.8040202@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Just installing 2.6rc2, I see that bsddb3 testsuite is disabled by default.

Current testsuite is far more fast and stable that the old one (entire
test: 17 seconds in my machine). I was wondering if it is time to enable
bsddb3 testsuite by default.

BTW: How is a "resource" enabled by an user, without touching sourcecode?.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSNJktZlgi5GaxT1NAQI7YQP+PwnNpfnJCsd3u/bAgjFQfHaRXMYlS1PN
dZPb8lkzMbyanNituTC9VLxI97BXsSPSM7VNnbyO3lBVSvJxvsDaRDmoUynno+VX
rw7+dD/mqKdyPujBjLzqYhbvQAoUOxLNac44/pTjvqoGiDa5CeR0AunUDnqnVVJa
4by7SBBxYrs=
=sllZ
-----END PGP SIGNATURE-----

From fdrake at acm.org  Thu Sep 18 16:33:10 2008
From: fdrake at acm.org (Fred Drake)
Date: Thu, 18 Sep 2008 10:33:10 -0400
Subject: [Python-Dev] bsddb tests disabled by default
In-Reply-To: <48D264BA.8040202@jcea.es>
References: <48D264BA.8040202@jcea.es>
Message-ID: <47468C0A-567F-405F-BC05-F3390F5B2773@acm.org>

On Sep 18, 2008, at 10:24 AM, Jesus Cea wrote:
> Current testsuite is far more fast and stable that the old one (entire
> test: 17 seconds in my machine). I was wondering if it is time to  
> enable
> bsddb3 testsuite by default.

Perhaps so.  That certainly improves the chances of finding problems  
early.

> BTW: How is a "resource" enabled by an user, without touching  
> sourcecode?.


regrtest.py takes a "-u" (for "use") option; take a look at how the  
value passed to that gets interpreted.  (Could use better  
documentation, if no one's improved it since I added the option  
umpteen gazillion years ago.)


   -Fred

-- 
Fred Drake   <fdrake at acm.org>


From mrs at mythic-beasts.com  Thu Sep 18 19:09:52 2008
From: mrs at mythic-beasts.com (Mark Seaborn)
Date: Thu, 18 Sep 2008 18:09:52 +0100 (BST)
Subject: [Python-Dev] ANNOUNCE: CapPython,
	an object-capability subset of Python
Message-ID: <20080918.180952.343191830.mrs@localhost.localdomain>

During the past couple of months I have been working on an
object-capability subset of Python - in other words, a restricted
execution scheme for sandboxing Python code.  It has been influenced
by other object-capability subset languages, such as Joe-E (a subset
of Java [1]), Caja/Cajita (subsets of Javascript [2]) and Caperl
(based on Perl [3]).  I'm calling it CapPython because the name
doesn't seem to have been taken yet. :-)

I believe it is now secure, so it seems like a good time to announce
it here!

The basic idea behind CapPython is to enforce encapsulation by
restricting access to private attributes of objects.  This is achieved
through a combination of static checking and limiting access to unsafe
builtins and modules.

Private attributes may only be accessed through "self" variables.
"Self" variables are defined as being the first arguments of functions
defined inside class definitions, with a few restrictions intended to
prevent these functions from escaping without being safely wrapped.
Private attribute names are those starting with "_".  Additionally,
"im_self", "im_func" and some other special cases are treated as
private attributes.  Assignments to attributes are only allowed via
"self" variables.

For example, the following code is accepted by the static verifier:

class Counter(object):
    def __init__(self):
        self._count = 0
    def get_next(self):
        self._count += 1
        return self._count

But the following code reads a private attribute and so it is rejected
as violating encapsulation:

counter._count -= 1

CapPython consists of three parts:
 - a static verifier;
 - a "safe exec" function, which will check code before executing it and
   can run code in a safe scope;
 - a module loader which implements a safe __import__ function.  Eventually
   this will be runnable as untrusted CapPython code.

I am documenting CapPython via my blog at the moment, with the
following posts so far:
http://lackingrhoticity.blogspot.com/2008/08/introducing-cappython.html
http://lackingrhoticity.blogspot.com/2008/09/dealing-with-modules-and-builtins-in.html
http://lackingrhoticity.blogspot.com/2008/09/cappython-unbound-methods-and-python-30.html

The code is available from a Bazaar repository on Launchpad:
https://code.launchpad.net/cappython

I am currently working on creating a simple example program, which
will be a wsgiref-based web server with a form for executing CapPython
code.  This involves taming some of the standard libraries to pass the
verifier.

There are some design notes here -
http://plash.beasts.org/wiki/CapPython
- although these notes are more a list of references and problems
CapPython needs to address than an explanation of the current design.
There was also a thread about CapPython on the e-lang mailing list:
http://www.eros-os.org/pipermail/e-lang/2008-August/012828.html

Mark

[1] http://code.google.com/p/joe-e/
[2] http://code.google.com/p/google-caja/
[3] http://caperl.links.org/

From brett at python.org  Thu Sep 18 19:43:26 2008
From: brett at python.org (Brett Cannon)
Date: Thu, 18 Sep 2008 10:43:26 -0700
Subject: [Python-Dev] bsddb tests disabled by default
In-Reply-To: <48D264BA.8040202@jcea.es>
References: <48D264BA.8040202@jcea.es>
Message-ID: <bbaeab100809181043r54245982x48eea94d56b1fa0d@mail.gmail.com>

On Thu, Sep 18, 2008 at 7:24 AM, Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Just installing 2.6rc2, I see that bsddb3 testsuite is disabled by default.
>
> Current testsuite is far more fast and stable that the old one (entire
> test: 17 seconds in my machine). I was wondering if it is time to enable
> bsddb3 testsuite by default.

Well, 'time' says the test takes 16.09 sec user and 16.09 sec system
on my MacBook, but a total execution time of almost 8 *minutes*. That
is too long to be on by default.

-Brett

From skip at pobox.com  Thu Sep 18 20:48:25 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 18 Sep 2008 13:48:25 -0500
Subject: [Python-Dev] What's New in 2.6 link wasn't what I expected
Message-ID: <18642.41593.298976.315811@montanaro-dyndns-org.local>

>From this page:

    http://www.python.org/download/releases/2.6/

I clicked the link labelled "What's new in Python 2.6rc2".  This didn't take
me to the What's New page.  Instead it took me to:

    http://www.python.org/download/releases/2.6/NEWS.txt

Seems a bit mislabelled if nothing else.  Poking around I see that the 2.5
What's New document is at:

    http://docs.python.org/dev/whatsnew/whatsnew25.html

but 

    http://docs.python.org/dev/whatsnew/whatsnew26.html

doesn't exist.

Am I missing something?

Skip

From tjreedy at udel.edu  Thu Sep 18 22:33:23 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 18 Sep 2008 16:33:23 -0400
Subject: [Python-Dev] ANNOUNCE: CapPython,
	an object-capability subset of Python
In-Reply-To: <20080918.180952.343191830.mrs@localhost.localdomain>
References: <20080918.180952.343191830.mrs@localhost.localdomain>
Message-ID: <gaudu8$9q5$1@ger.gmane.org>

Mark Seaborn wrote:
> During the past couple of months I have been working on an
> object-capability subset of Python - in other words, a restricted
> execution scheme for sandboxing Python code.  It has been influenced
> by other object-capability subset languages, such as Joe-E (a subset
> of Java [1]), Caja/Cajita (subsets of Javascript [2]) and Caperl
> (based on Perl [3]).  I'm calling it CapPython because the name
> doesn't seem to have been taken yet. :-)

No wonder ;-).  I like CapPy better, though there is a shareware screen 
capture program by that name. PyCap is taken.  CapThon is not.
> 
> I believe it is now secure, so it seems like a good time to announce
> it here!
> 
> The basic idea behind CapPython is to enforce encapsulation by
> restricting access to private attributes of objects.  This is achieved
> through a combination of static checking and limiting access to unsafe
> builtins and modules.
> 
> Private attributes may only be accessed through "self" variables.
> "Self" variables are defined as being the first arguments of functions
> defined inside class definitions, with a few restrictions intended to
> prevent these functions from escaping without being safely wrapped.

What about functions defined outside class definitions and then attached 
as an attribute.  Prevented?

> Private attribute names are those starting with "_".   Additionally,
> "im_self", "im_func" and some other special cases are treated as
> private attributes.

In 3.0, unbound methods are gone and im_self and im_func are __self__ 
and __func__ attributes of method objects.


From guido at python.org  Thu Sep 18 22:49:20 2008
From: guido at python.org (Guido van Rossum)
Date: Thu, 18 Sep 2008 13:49:20 -0700
Subject: [Python-Dev] ANNOUNCE: CapPython,
	an object-capability subset of Python
In-Reply-To: <gaudu8$9q5$1@ger.gmane.org>
References: <20080918.180952.343191830.mrs@localhost.localdomain>
	<gaudu8$9q5$1@ger.gmane.org>
Message-ID: <ca471dc20809181349r68cb302au6897c5db324c99a1@mail.gmail.com>

How about Capt'n Python? :-)

Anyway, this is way cool. Looking forward to kicking the tires!

On Thu, Sep 18, 2008 at 1:33 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> Mark Seaborn wrote:
>>
>> During the past couple of months I have been working on an
>> object-capability subset of Python - in other words, a restricted
>> execution scheme for sandboxing Python code.  It has been influenced
>> by other object-capability subset languages, such as Joe-E (a subset
>> of Java [1]), Caja/Cajita (subsets of Javascript [2]) and Caperl
>> (based on Perl [3]).  I'm calling it CapPython because the name
>> doesn't seem to have been taken yet. :-)
>
> No wonder ;-).  I like CapPy better, though there is a shareware screen
> capture program by that name. PyCap is taken.  CapThon is not.
>>
>> I believe it is now secure, so it seems like a good time to announce
>> it here!
>>
>> The basic idea behind CapPython is to enforce encapsulation by
>> restricting access to private attributes of objects.  This is achieved
>> through a combination of static checking and limiting access to unsafe
>> builtins and modules.
>>
>> Private attributes may only be accessed through "self" variables.
>> "Self" variables are defined as being the first arguments of functions
>> defined inside class definitions, with a few restrictions intended to
>> prevent these functions from escaping without being safely wrapped.
>
> What about functions defined outside class definitions and then attached as
> an attribute.  Prevented?
>
>> Private attribute names are those starting with "_".   Additionally,
>> "im_self", "im_func" and some other special cases are treated as
>> private attributes.
>
> In 3.0, unbound methods are gone and im_self and im_func are __self__ and
> __func__ attributes of method objects.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From phd at phd.pp.ru  Thu Sep 18 22:52:57 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Fri, 19 Sep 2008 00:52:57 +0400
Subject: [Python-Dev] ANNOUNCE: CapPython,
	an object-capability subset of Python
In-Reply-To: <gaudu8$9q5$1@ger.gmane.org>
References: <20080918.180952.343191830.mrs@localhost.localdomain>
	<gaudu8$9q5$1@ger.gmane.org>
Message-ID: <20080918205257.GA30485@phd.pp.ru>

On Thu, Sep 18, 2008 at 04:33:23PM -0400, Terry Reedy wrote:
> Mark Seaborn wrote:
> I'm calling it CapPython
> 
> No wonder ;-).  I like CapPy better, though there is a shareware screen 
> capture program by that name. PyCap is taken.  CapThon is not.

   CaPy, and make capybara its mascot. ;)

   Or may be "captyve" because the goal  of the project is to make some
code captive. :)

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From lists at cheimes.de  Thu Sep 18 23:01:10 2008
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 18 Sep 2008 23:01:10 +0200
Subject: [Python-Dev] ANNOUNCE: CapPython,
	an object-capability subset of Python
In-Reply-To: <ca471dc20809181349r68cb302au6897c5db324c99a1@mail.gmail.com>
References: <20080918.180952.343191830.mrs@localhost.localdomain>	<gaudu8$9q5$1@ger.gmane.org>
	<ca471dc20809181349r68cb302au6897c5db324c99a1@mail.gmail.com>
Message-ID: <48D2C196.2010302@cheimes.de>

Guido van Rossum wrote:
> How about Capt'n Python? :-)

Harr, harr! Grrrreat name :)

Christian

From mrs at mythic-beasts.com  Thu Sep 18 23:15:49 2008
From: mrs at mythic-beasts.com (Mark Seaborn)
Date: Thu, 18 Sep 2008 22:15:49 +0100 (BST)
Subject: [Python-Dev] ANNOUNCE: CapPython,
 an object-capability subset of Python
In-Reply-To: <gaudu8$9q5$1@ger.gmane.org>
References: <20080918.180952.343191830.mrs@localhost.localdomain>
	<gaudu8$9q5$1@ger.gmane.org>
Message-ID: <20080918.221549.424243434.mrs@localhost.localdomain>

Terry Reedy <tjreedy at udel.edu> wrote:

> Mark Seaborn wrote:

> > Private attributes may only be accessed through "self" variables.
> > "Self" variables are defined as being the first arguments of functions
> > defined inside class definitions, with a few restrictions intended to
> > prevent these functions from escaping without being safely wrapped.
> 
> What about functions defined outside class definitions and then attached 
> as an attribute.  Prevented?

Yes, that is prevented: attribute assignment is only allowed on "self"
variables, so you can't assign to class attributes.  Classes can't be
extended that way.  That should not be a big problem for
expressiveness; defining __getattr__ will still be possible.

CapPython has to prevent attribute assignment by default because
Python allows it on objects by default.

It would be possible to allow attribute assignment by having CapPython
rewrite it to a normal method call whose behaviour classes have to opt
into, rather than opt out of.  Currently CapPython does not do any
rewriting.

> > Private attribute names are those starting with "_".   Additionally,
> > "im_self", "im_func" and some other special cases are treated as
> > private attributes.
> 
> In 3.0, unbound methods are gone and im_self and im_func are __self__ 
> and __func__ attributes of method objects.

Yes.  The renaming of "im_self" and "im_func" is good.  The removal of
unbound methods is a *big* problem [1].

Regards,
Mark

[1] http://lackingrhoticity.blogspot.com/2008/09/cappython-unbound-methods-and-python-30.html

From guido at python.org  Thu Sep 18 23:53:15 2008
From: guido at python.org (Guido van Rossum)
Date: Thu, 18 Sep 2008 14:53:15 -0700
Subject: [Python-Dev] ANNOUNCE: CapPython,
	an object-capability subset of Python
In-Reply-To: <20080918.221549.424243434.mrs@localhost.localdomain>
References: <20080918.180952.343191830.mrs@localhost.localdomain>
	<gaudu8$9q5$1@ger.gmane.org>
	<20080918.221549.424243434.mrs@localhost.localdomain>
Message-ID: <ca471dc20809181453p36adeec3wd7c83099d32b0719@mail.gmail.com>

On Thu, Sep 18, 2008 at 2:15 PM, Mark Seaborn <mrs at mythic-beasts.com> wrote:
> Terry Reedy <tjreedy at udel.edu> wrote:
>
>> Mark Seaborn wrote:
>
>> > Private attributes may only be accessed through "self" variables.
>> > "Self" variables are defined as being the first arguments of functions
>> > defined inside class definitions, with a few restrictions intended to
>> > prevent these functions from escaping without being safely wrapped.
>>
>> What about functions defined outside class definitions and then attached
>> as an attribute.  Prevented?
>
> Yes, that is prevented: attribute assignment is only allowed on "self"
> variables, so you can't assign to class attributes.  Classes can't be
> extended that way.  That should not be a big problem for
> expressiveness; defining __getattr__ will still be possible.
>
> CapPython has to prevent attribute assignment by default because
> Python allows it on objects by default.
>
> It would be possible to allow attribute assignment by having CapPython
> rewrite it to a normal method call whose behaviour classes have to opt
> into, rather than opt out of.  Currently CapPython does not do any
> rewriting.
>
>> > Private attribute names are those starting with "_".   Additionally,
>> > "im_self", "im_func" and some other special cases are treated as
>> > private attributes.
>>
>> In 3.0, unbound methods are gone and im_self and im_func are __self__
>> and __func__ attributes of method objects.
>
> Yes.  The renaming of "im_self" and "im_func" is good.  The removal of
> unbound methods is a *big* problem [1].
>
> Regards,
> Mark
>
> [1] http://lackingrhoticity.blogspot.com/2008/09/cappython-unbound-methods-and-python-30.html

I don't know to what extent you want to modify Python fundamentals,
but I think this could be solved simply by adding a metaclass that
returns an unbound method object for C.f, couldn't it?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Fri Sep 19 00:49:28 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 19 Sep 2008 00:49:28 +0200
Subject: [Python-Dev] What's New in 2.6 link wasn't what I expected
In-Reply-To: <18642.41593.298976.315811@montanaro-dyndns-org.local>
References: <18642.41593.298976.315811@montanaro-dyndns-org.local>
Message-ID: <48D2DAF8.3050807@v.loewis.de>

> I clicked the link labelled "What's new in Python 2.6rc2".  This didn't take
> me to the What's New page.  Instead it took me to:
> 
>     http://www.python.org/download/releases/2.6/NEWS.txt
> 
> Seems a bit mislabelled if nothing else.

How so? The first major heading in that file reads

"What's New in Python 2.6 release candidate 2?"

> Am I missing something?

Perhaps reading the file that was linked might have helped.

Regards,
Martin

From amcnabb at mcnabbs.org  Fri Sep 19 01:44:59 2008
From: amcnabb at mcnabbs.org (Andrew McNabb)
Date: Thu, 18 Sep 2008 17:44:59 -0600
Subject: [Python-Dev] What's New in 2.6 link wasn't what I expected
In-Reply-To: <48D2DAF8.3050807@v.loewis.de>
References: <18642.41593.298976.315811@montanaro-dyndns-org.local>
	<48D2DAF8.3050807@v.loewis.de>
Message-ID: <20080918234459.GC2815@mcnabbs.org>

On Fri, Sep 19, 2008 at 12:49:28AM +0200, "Martin v. L?wis" wrote:
> > I clicked the link labelled "What's new in Python 2.6rc2".  This didn't take
> > me to the What's New page.  Instead it took me to:
> > 
> >     http://www.python.org/download/releases/2.6/NEWS.txt
> > 
> > Seems a bit mislabelled if nothing else.
> 
> How so? The first major heading in that file reads
> 
> "What's New in Python 2.6 release candidate 2?"
> 
> > Am I missing something?
> 
> Perhaps reading the file that was linked might have helped.

I ran into this, too (and I did read the file that was linked to).  I
had expected to find the "What's New" page describing what was new in
Python 2.6, and I would have expected a "Release Notes" link to point to
the NEWS.txt page.  That's how things were in the past, anyway:

http://www.python.org/download/releases/2.5.2/#what-s-new

It's a minor thing, but I did have to go hunt down the What's New file.

-- 
Andrew McNabb
http://www.mcnabbs.org/andrew/
PGP Fingerprint: 8A17 B57C 6879 1863 DE55  8012 AB4D 6098 8826 6868
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080918/491347d2/attachment.pgp>

From ncoghlan at gmail.com  Fri Sep 19 03:36:11 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 19 Sep 2008 11:36:11 +1000
Subject: [Python-Dev] What's New in 2.6 link wasn't what I expected
In-Reply-To: <18642.41593.298976.315811@montanaro-dyndns-org.local>
References: <18642.41593.298976.315811@montanaro-dyndns-org.local>
Message-ID: <48D3020B.7040800@gmail.com>

skip at pobox.com wrote:
>>From this page:
> 
>     http://www.python.org/download/releases/2.6/
> 
> I clicked the link labelled "What's new in Python 2.6rc2".  This didn't take
> me to the What's New page.  Instead it took me to:
> 
>     http://www.python.org/download/releases/2.6/NEWS.txt
> 
> Seems a bit mislabelled if nothing else.

I agree - as Andrew suggested, it would probably be better if the What's
New? list for 2.6 was a bit more like the 2.5 list:
http://www.python.org/download/releases/2.5/

>  Poking around I see that the 2.5
> What's New document is at:
> 
>     http://docs.python.org/dev/whatsnew/whatsnew25.html

That actually rewrites the URL to the URL for the current docs (removing
the 'dev' part):
http://docs.python.org/whatsnew/whatsnew25.html

> but 
> 
>     http://docs.python.org/dev/whatsnew/whatsnew26.html

The name was changed in the move to Sphinx for the docs - it's at:
http://docs.python.org/dev/whatsnew/2.6.html

(Unlike the old What's New directory, or the current library directory
and so forth, that directory doesn't have an index entry defined, so
simply leaving out the file name detail doesn't currently do the right
thing - you get the directory listing instead of the what's new doco)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From skip at pobox.com  Fri Sep 19 04:29:22 2008
From: skip at pobox.com (skip at pobox.com)
Date: Thu, 18 Sep 2008 21:29:22 -0500
Subject: [Python-Dev] What's New in 2.6 link wasn't what I expected
In-Reply-To: <48D2DAF8.3050807@v.loewis.de>
References: <18642.41593.298976.315811@montanaro-dyndns-org.local>
	<48D2DAF8.3050807@v.loewis.de>
Message-ID: <18643.3714.552085.142928@montanaro-dyndns-org.local>


    >> http://www.python.org/download/releases/2.6/NEWS.txt
    >> 
    >> Seems a bit mislabelled if nothing else.

    Martin> How so? The first major heading in that file reads

Seems more like the Misc/NEWS file to me.  I was expecting Andrew's What's
New document, which I eventually found at

    http://docs.python.org/dev/whatsnew/2.6.html

If you Google for "site:python.org whatsnew" seven of the first eight hits
are for various versions of Andrew's 2.x What's New doc.  That's what people
expect to find.  We shouldn't be changing that now.

Skip

From steve at holdenweb.com  Fri Sep 19 07:02:25 2008
From: steve at holdenweb.com (Steve Holden)
Date: Fri, 19 Sep 2008 01:02:25 -0400
Subject: [Python-Dev] What's New in 2.6 link wasn't what I expected
In-Reply-To: <18643.3714.552085.142928@montanaro-dyndns-org.local>
References: <18642.41593.298976.315811@montanaro-dyndns-org.local>	<48D2DAF8.3050807@v.loewis.de>
	<18643.3714.552085.142928@montanaro-dyndns-org.local>
Message-ID: <gavbq8$ned$1@ger.gmane.org>

skip at pobox.com wrote:
>     >> http://www.python.org/download/releases/2.6/NEWS.txt
>     >> 
>     >> Seems a bit mislabelled if nothing else.
> 
>     Martin> How so? The first major heading in that file reads
> 
> Seems more like the Misc/NEWS file to me.  I was expecting Andrew's What's
> New document, which I eventually found at
> 
>     http://docs.python.org/dev/whatsnew/2.6.html
> 
> If you Google for "site:python.org whatsnew" seven of the first eight hits
> are for various versions of Andrew's 2.x What's New doc.  That's what people
> expect to find.  We shouldn't be changing that now.
> 
+1
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From martin at v.loewis.de  Fri Sep 19 08:14:33 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 19 Sep 2008 08:14:33 +0200
Subject: [Python-Dev] What's New in 2.6 link wasn't what I expected
In-Reply-To: <18643.3714.552085.142928@montanaro-dyndns-org.local>
References: <18642.41593.298976.315811@montanaro-dyndns-org.local>
	<48D2DAF8.3050807@v.loewis.de>
	<18643.3714.552085.142928@montanaro-dyndns-org.local>
Message-ID: <48D34349.3050106@v.loewis.de>

>     >> http://www.python.org/download/releases/2.6/NEWS.txt
>     >> 
>     >> Seems a bit mislabelled if nothing else.
> 
>     Martin> How so? The first major heading in that file reads
> 
> Seems more like the Misc/NEWS file to me.

Correct.

> I was expecting Andrew's What's New document

Why that? The link deliberately says "What's new in Python 2.6rc2",
not "What's new in Python 2.6".

> If you Google for "site:python.org whatsnew" seven of the first eight hits
> are for various versions of Andrew's 2.x What's New doc.  That's what people
> expect to find.  We shouldn't be changing that now.

I admit that the traditional description for that link is "release
notes" or "detailed release notes".

Regards,
Martin

From barry at python.org  Fri Sep 19 13:20:37 2008
From: barry at python.org (Barry Warsaw)
Date: Fri, 19 Sep 2008 07:20:37 -0400
Subject: [Python-Dev] Python documentation
Message-ID: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Martin points out that in the past, as part of the release process,  
we've built separate downloadable documentation.

Do we still want to do that for Python 2.6 and 3.0, and if so, how do  
we go about doing that?  I have this feeling that building the  
documentation is much different now than in the past, and I don't  
really have a good feel for how it's done now.

If you think we should release separate downloadable documentation and  
can help integrate that into the release project, you just might be a  
Documentation Expert <wink>.  Please let me know if you can help.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSNOLBnEjvBPtnXfVAQIQnAQAm6thEThGufep6hzHxBwAN8MTsLb9jxsu
Z8GAtX1bdMNOrJczYpU6by0oXPLR2pupnGV1YrAyQyoqpk+K7W8by5Qtg8+ZZcYH
GerkqMVtNYn2zY1HhKigivp2JvlqIidRc5D36XS2EJixhZEPcOQDVm34THNQyRJT
QasCQwdSAHI=
=MbMY
-----END PGP SIGNATURE-----

From fijall at gmail.com  Fri Sep 19 13:31:48 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 19 Sep 2008 13:31:48 +0200
Subject: [Python-Dev] ANNOUNCE: CapPython,
	an object-capability subset of Python
In-Reply-To: <ca471dc20809181453p36adeec3wd7c83099d32b0719@mail.gmail.com>
References: <20080918.180952.343191830.mrs@localhost.localdomain>
	<gaudu8$9q5$1@ger.gmane.org>
	<20080918.221549.424243434.mrs@localhost.localdomain>
	<ca471dc20809181453p36adeec3wd7c83099d32b0719@mail.gmail.com>
Message-ID: <693bc9ab0809190431k7e1dedd6k5d06300376fd6e90@mail.gmail.com>

PyPy offers sandboxing interpreter without compromising language
features itself. Here are docs:
http://codespeak.net/pypy/dist/pypy/doc/sandbox.html

Also, are you aware of directory Lib/test/crashers (in python's svn)
which contains some possible ways to segfault cpython? (which can lead
to compromise later)

Cheers,
fijal

From fdrake at acm.org  Fri Sep 19 14:47:27 2008
From: fdrake at acm.org (Fred Drake)
Date: Fri, 19 Sep 2008 08:47:27 -0400
Subject: [Python-Dev] Python documentation
In-Reply-To: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>
Message-ID: <33550B11-37BB-4EA0-9406-0DBEC96DE46F@acm.org>

On Sep 19, 2008, at 7:20 AM, Barry Warsaw wrote:
> Martin points out that in the past, as part of the release process,  
> we've built separate downloadable documentation.
>
> Do we still want to do that for Python 2.6 and 3.0, ...?

Yes, I think so.  The downloads are very useful for people who  
regularly work disconnected from the public internet, or who are  
constrained by very small pipes.  The PDF downlaods are also pretty  
important for the smaller documents, especially the tutorial.  Many  
people want to walk away from the computer to read through that the  
first time.

> If you think we should release separate downloadable documentation  
> and can help integrate that into the release project, you just might  
> be a Documentation Expert <wink>.  Please let me know if you can help.


I think we should, but I'm hoping Georg has some ideas on how best to  
get the processes integrated.  :-)


   -Fred

-- 
Fred Drake   <fdrake at acm.org>


From python at rcn.com  Fri Sep 19 15:38:45 2008
From: python at rcn.com (Raymond)
Date: Fri, 19 Sep 2008 14:38:45 +0100
Subject: [Python-Dev] Python documentation
In-Reply-To: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>
Message-ID: <FB176030-1D37-4C56-9812-72FA4F114982@rcn.com>

+1.

I find the offline versions to be vital.

Sent from my iPhone

On Sep 19, 2008, at 12:20 PM, Barry Warsaw <barry at python.org> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Martin points out that in the past, as part of the release process,  
> we've built separate downloadable documentation.
>
> Do we still want to do that for Python 2.6 and 3.0, and if so, how  
> do we go about doing that?  I have this feeling that building the  
> documentation is much different now than in the past, and I don't  
> really have a good feel for how it's done now.
>
> If you think we should release separate downloadable documentation  
> and can help integrate that into the release project, you just might  
> be a Documentation Expert <wink>.  Please let me know if you can help.
>
> - -Barry
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (Darwin)
>
> iQCVAwUBSNOLBnEjvBPtnXfVAQIQnAQAm6thEThGufep6hzHxBwAN8MTsLb9jxsu
> Z8GAtX1bdMNOrJczYpU6by0oXPLR2pupnGV1YrAyQyoqpk+K7W8by5Qtg8+ZZcYH
> GerkqMVtNYn2zY1HhKigivp2JvlqIidRc5D36XS2EJixhZEPcOQDVm34THNQyRJT
> QasCQwdSAHI=
> =MbMY
> -----END PGP SIGNATURE-----
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/python%40rcn.com

From fijall at gmail.com  Fri Sep 19 17:43:42 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 19 Sep 2008 17:43:42 +0200
Subject: [Python-Dev] What this code should do?
Message-ID: <693bc9ab0809190843g38a09a69q39e0b23b04387759@mail.gmail.com>

Hello,

I'm a little clueless about exact semantics of following snippets:

http://paste.pocoo.org/show/85698/

is this fine?
or shall I fill the bug?
(the reason to ask is because a) django is relying on this b) pypy
implements it differently)

cheers,
fijal

From fuzzyman at voidspace.org.uk  Fri Sep 19 18:05:01 2008
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 19 Sep 2008 17:05:01 +0100
Subject: [Python-Dev] What this code should do?
In-Reply-To: <693bc9ab0809190843g38a09a69q39e0b23b04387759@mail.gmail.com>
References: <693bc9ab0809190843g38a09a69q39e0b23b04387759@mail.gmail.com>
Message-ID: <48D3CDAD.6020609@voidspace.org.uk>

Maciej Fijalkowski wrote:
> Hello,
>
> I'm a little clueless about exact semantics of following snippets:
>
> http://paste.pocoo.org/show/85698/
>
> is this fine?
>   

It looks right to me. :-)

In the first case the NameError is caught by the except and not 
re-raised (but still enters the finally after the except block) and in 
the second the NameError is caught by the finally that does re-raise.

What do you think should happen?

Michael

> or shall I fill the bug?
> (the reason to ask is because a) django is relying on this b) pypy
> implements it differently)
>
> cheers,
> fijal
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/
http://www.trypython.org/
http://www.ironpython.info/
http://www.resolverhacks.net/
http://www.theotherdelia.co.uk/


From status at bugs.python.org  Fri Sep 19 18:07:06 2008
From: status at bugs.python.org (Python tracker)
Date: Fri, 19 Sep 2008 18:07:06 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20080919160706.41F3678519@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (09/12/08 - 09/19/08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2035 open (+36) / 13696 closed (+24) / 15731 total (+60)

Open issues with patches:   652

Average duration of open issues: 715 days.
Median duration of open issues: 1793 days.

Open Issues Breakdown
   open  2022 (+36)
pending    13 ( +0)

Issues Created Or Reopened (60)
_______________________________

FUD in documentation for urllib.urlopen()                        09/12/08
       http://bugs.python.org/issue3849    created  raz                       
                                                                               

find_recursion_limit.py is broken                                09/12/08
CLOSED http://bugs.python.org/issue3850    created  pitrou                    
       patch, easy, needs review                                               

IDLE: Pressing "Home" on Windows places cursor before ">>>" inst 09/12/08
       http://bugs.python.org/issue3851    created  serwy                     
       patch                                                                   

kqueue.control requires 2 params while docs say max_events (the  09/12/08
       http://bugs.python.org/issue3852    created  ionel.mc                  
                                                                               

Windows SQLite DLL should be built with multithreading enabled   09/12/08
CLOSED http://bugs.python.org/issue3853    created  ghaering                  
       easy                                                                    

Document sqlite3 vs. threads                                     09/12/08
       http://bugs.python.org/issue3854    created  ghaering                  
                                                                               

Windows installation did not work; tried on two machines         09/13/08
CLOSED http://bugs.python.org/issue3855    created  MLModel                   
                                                                               

IDLE fails on startup on Mac 10.5 for 2.6b3 and 3.0b3            09/13/08
CLOSED http://bugs.python.org/issue3856    created  MLModel                   
                                                                               

ImportError: No module named test.test_support                   09/13/08
CLOSED http://bugs.python.org/issue3857    created  wplappert                 
                                                                               

make install tries to install files outside of --prefix          09/13/08
CLOSED http://bugs.python.org/issue3858    created  jjlee                     
                                                                               

test_sys.Sizeof fails on win64                                   09/13/08
CLOSED http://bugs.python.org/issue3859    created  amaury.forgeotdarc        
       patch                                                                   

GzipFile and BZ2File should support context manager protocol     09/13/08
       http://bugs.python.org/issue3860    created  hagen                     
                                                                               

distutils CCompiler._compile doesn't require lang keyword argume 09/14/08
CLOSED http://bugs.python.org/issue3861    created  ikelos                    
                                                                               

test_array fails on FreeBSD7 amd64                               09/14/08
       http://bugs.python.org/issue3862    created  robrien                   
       patch                                                                   

2.6rc1: test_threading hangs on FreeBSD 6.3 i386                 09/14/08
       http://bugs.python.org/issue3863    created  aimacintyre               
       patch                                                                   

26.rc1: test_signal issue on FreeBSD 6.3                         09/14/08
       http://bugs.python.org/issue3864    created  aimacintyre               
                                                                               

explain that profilers should be used for profiling, not benchma 09/14/08
       http://bugs.python.org/issue3865    created  effbot                    
                                                                               

int() doesn't 'guess'                                            09/14/08
CLOSED http://bugs.python.org/issue3866    created  kjohnson                  
                                                                               

What's New in 2.6 doesn't mention stricter object.__init__       09/14/08
CLOSED http://bugs.python.org/issue3867    created  spiv                      
                                                                               

patch for review: OS/2 EMX port fixes for 2.6                    09/14/08
       http://bugs.python.org/issue3868    created  aimacintyre               
       patch, patch                                                            

Arrow keys not working with Python 2.6rc1                        09/14/08
CLOSED http://bugs.python.org/issue3869    created  Chewie                    
                                                                               

Parser/asdl_c.py requires python in order to build python        09/14/08
CLOSED http://bugs.python.org/issue3870    created  gregory.p.smith           
                                                                               

cross building python for mingw32 with distutils                 09/14/08
       http://bugs.python.org/issue3871    created  rpetrov                   
                                                                               

Tix ComboBox error                                               09/15/08
       http://bugs.python.org/issue3872    created  dwahli                    
                                                                               

Unpickling is really slow                                        09/15/08
       http://bugs.python.org/issue3873    created  hagen                     
                                                                               

documentation bug: HTMLParser needs to document unknown_decl     09/15/08
       http://bugs.python.org/issue3874    created  freyley                   
                                                                               

provide a "cmem" role                                            09/15/08
       http://bugs.python.org/issue3875    created  benjamin.peterson         
                                                                               

multiprocessing does not compile on *BSD and potentialy other sy 09/16/08
       http://bugs.python.org/issue3876    created  henry.precheur            
                                                                               

test_fileio fails on OpenBSD 4.4                                 09/16/08
       http://bugs.python.org/issue3877    created  henry.precheur            
       patch                                                                   

urllib2 is not working with proxy for HTTPS                      09/16/08
CLOSED http://bugs.python.org/issue3878    created  anupam                    
                                                                               

2.6 regression in urllib.getproxies_environment                  09/16/08
       http://bugs.python.org/issue3879    created  vila                      
       patch, needs review                                                     

_tkinter._flatten() doesn't check PySequence_Size() error code   09/16/08
       http://bugs.python.org/issue3880    created  haypo                     
       patch                                                                   

IDLE won't start in custom directory.                            09/16/08
       http://bugs.python.org/issue3881    created  kimsey0                   
                                                                               

Bottom buttons of IDLE Preferences Pane on Mac not wide enough f 09/16/08
CLOSED http://bugs.python.org/issue3882    created  MLModel                   
                                                                               

Bottom buttons of IDLE Preferences Pane on Mac not wide enough f 09/16/08
       http://bugs.python.org/issue3883    created  MLModel                   
                                                                               

turtle in the tkinter package?                                   09/16/08
CLOSED http://bugs.python.org/issue3884    created  kirill_simonov            
                                                                               

errors on _bsddb creation and dealloc                            09/16/08
       http://bugs.python.org/issue3885    created  haypo                     
       patch, needs review                                                     

Integer overflow in _hashopenssl.c (CVE-2008-2316)               09/18/08
       http://bugs.python.org/issue3886    reopened benjamin.peterson         
       patch, patch, 64bit                                                     

Python 2.6 doesn't run after installation on amd64               09/17/08
CLOSED http://bugs.python.org/issue3887    created  jpe                       
                                                                               

[PATCH] Document more deprecated modules in What's New in Python 09/17/08
CLOSED http://bugs.python.org/issue3888    created  pjenvey                   
       patch                                                                   

Demo/parser/unparse.py                                           09/17/08
       http://bugs.python.org/issue3889    created  gregdarke                 
       patch                                                                   

ssl.SSLSocket.recv() implementation may not work with non-blocki 09/17/08
       http://bugs.python.org/issue3890    created  giampaolo.rodola          
                                                                               

collections.deque should have empty() method                     09/17/08
       http://bugs.python.org/issue3891    created  roysmith                  
                                                                               

bsddb: test01_basic_replication fails on Windows sometimes       09/18/08
       http://bugs.python.org/issue3892    created  benjamin.peterson         
                                                                               

timedelta overflows in a surprising way                          09/18/08
CLOSED http://bugs.python.org/issue3893    created  jjinux                    
                                                                               

imageop issue                                                    09/18/08
       http://bugs.python.org/issue3894    created  brett.cannon              
                                                                               

_lsprof issue                                                    09/18/08
       http://bugs.python.org/issue3895    created  brett.cannon              
                                                                               

idle should be installed as idle3.0                              09/18/08
       http://bugs.python.org/issue3896    created  HWJ                       
                                                                               

collections                                                      09/18/08
       http://bugs.python.org/issue3897    created  kadelka                   
                                                                               

3.0 documentation fails to build                                 09/18/08
CLOSED http://bugs.python.org/issue3898    created  loewis                    
                                                                               

test_ssl.py doesn't properly test ssl integration with asyncore  09/18/08
       http://bugs.python.org/issue3899    created  giampaolo.rodola          
       patch                                                                   

ctypes: wrong calling convention for _string_at                  09/18/08
CLOSED http://bugs.python.org/issue3900    created  haypo                     
       patch                                                                   

Slight readme.txt fix (VC9)                                      09/18/08
       http://bugs.python.org/issue3901    created  ocean-city                
       patch                                                                   

distutils does not correctly create packages for compiled extens 09/18/08
       http://bugs.python.org/issue3902    created  Peaker                    
                                                                               

pickle error in python3.0rc1                                     09/18/08
CLOSED http://bugs.python.org/issue3903    created  Georg                     
                                                                               

asynchat async_chat __init__() arguments changed in python 2.6   09/18/08
CLOSED http://bugs.python.org/issue3904    created  forest                    
                                                                               

subprocess failing in GUI applications on Windows                09/18/08
       http://bugs.python.org/issue3905    created  twhitema                  
                                                                               

lib2to3\main.py will not run                                     09/19/08
CLOSED http://bugs.python.org/issue3906    created  rupole                    
                                                                               

"for line in file" doesn't work for pipes                        09/19/08
CLOSED http://bugs.python.org/issue3907    created  endolith                  
                                                                               

Strange heapq behavior on Python 3.0 when overriding __le__      09/19/08
       http://bugs.python.org/issue3908    created  giampaolo.rodola          
                                                                               



Issues Now Closed (51)
______________________

Add a factorial function                                          182 days
       http://bugs.python.org/issue2138    maix                      
                                                                               

Nested module import clutters package namespace                   202 days
       http://bugs.python.org/issue2210    brett.cannon              
                                                                               

Fixer for new metaclass syntax is needed                          179 days
       http://bugs.python.org/issue2366    benjamin.peterson         
       patch, 26backport                                                       

Fixer for new integer literals are needed                         180 days
       http://bugs.python.org/issue2369    benjamin.peterson         
       26backport                                                              

Multiple buffer overflows in unicode processing                   158 days
       http://bugs.python.org/issue2620    brett.cannon              
       patch                                                                   

bsddb3 needs to be ported to Python 3.0                           124 days
       http://bugs.python.org/issue2887    jcea                      
                                                                               

2to3 doesn't handle print(whatever); print nor string.* function  106 days
       http://bugs.python.org/issue3000    benjamin.peterson         
                                                                               

integer overflow in hashlib causes wrong results for cryptograph  108 days
       http://bugs.python.org/issue3026    loewis                    
       patch                                                                   

sqlite defines a few global symbols.                               92 days
       http://bugs.python.org/issue3103    ghaering                  
                                                                               

Documentation: timeit: "lower bound" should read "upper bound"     68 days
       http://bugs.python.org/issue3318    unutbu                    
                                                                               

file read preallocs 'size' bytes which can cause memory	problems   36 days
       http://bugs.python.org/issue3531    pitrou                    
                                                                               

Add MS EULA to the list of third-party licenses in the	Windows i   25 days
       http://bugs.python.org/issue3617    loewis                    
       patch                                                                   

test_cpickle crash on AMD64 Windows build                          22 days
       http://bugs.python.org/issue3640    pitrou                    
       patch, 64bit                                                            

latexwriter: \footnotemark can only take numbers as arguments      20 days
       http://bugs.python.org/issue3655    georg.brandl              
                                                                               

incompatible arguments in warning formatting for idle              17 days
       http://bugs.python.org/issue3698    gpolo                     
                                                                               

os.urandom(1.1): infinite loop                                     22 days
       http://bugs.python.org/issue3708    rpetrov                   
       patch, easy                                                             

socket.socket.recv broken (unbearably slow)                        10 days
       http://bugs.python.org/issue3766    pitrou                    
                                                                               

os.write accepts unicode strings                                   11 days
       http://bugs.python.org/issue3782    pitrou                    
       patch                                                                   

python-2.6b3.msi and python-2.6b3.amd64.msi can't both be instal    2 days
       http://bugs.python.org/issue3833    jretz                     
                                                                               

can't run close through itertools.chain on inner generator          1 days
       http://bugs.python.org/issue3842    rhettinger                
                                                                               

hexadecimal, not decimal                                            2 days
       http://bugs.python.org/issue3843    benjamin.peterson         
                                                                               

sqlite3 module: Improved concurrency                                0 days
       http://bugs.python.org/issue3846    ghaering                  
       needs review                                                            

Begging for review                                                  0 days
       http://bugs.python.org/issue3847    loewis                    
                                                                               

find_recursion_limit.py is broken                                   1 days
       http://bugs.python.org/issue3850    pitrou                    
       patch, easy, needs review                                               

Windows SQLite DLL should be built with multithreading enabled      6 days
       http://bugs.python.org/issue3853    loewis                    
       easy                                                                    

Windows installation did not work; tried on two machines            0 days
       http://bugs.python.org/issue3855    loewis                    
                                                                               

IDLE fails on startup on Mac 10.5 for 2.6b3 and 3.0b3               2 days
       http://bugs.python.org/issue3856    gpolo                     
                                                                               

ImportError: No module named test.test_support                      0 days
       http://bugs.python.org/issue3857    benjamin.peterson         
                                                                               

make install tries to install files outside of --prefix             1 days
       http://bugs.python.org/issue3858    pitrou                    
                                                                               

test_sys.Sizeof fails on win64                                      3 days
       http://bugs.python.org/issue3859    schuppenies               
       patch                                                                   

distutils CCompiler._compile doesn't require lang keyword argume    0 days
       http://bugs.python.org/issue3861    ikelos                    
                                                                               

int() doesn't 'guess'                                               0 days
       http://bugs.python.org/issue3866    benjamin.peterson         
                                                                               

What's New in 2.6 doesn't mention stricter object.__init__          1 days
       http://bugs.python.org/issue3867    benjamin.peterson         
                                                                               

Arrow keys not working with Python 2.6rc1                           0 days
       http://bugs.python.org/issue3869    benjamin.peterson         
                                                                               

Parser/asdl_c.py requires python in order to build python           3 days
       http://bugs.python.org/issue3870    theller                   
                                                                               

urllib2 is not working with proxy for HTTPS                         0 days
       http://bugs.python.org/issue3878    amaury.forgeotdarc        
                                                                               

Bottom buttons of IDLE Preferences Pane on Mac not wide enough f    0 days
       http://bugs.python.org/issue3882    benjamin.peterson         
                                                                               

turtle in the tkinter package?                                      0 days
       http://bugs.python.org/issue3884    brett.cannon              
                                                                               

Python 2.6 doesn't run after installation on amd64                  3 days
       http://bugs.python.org/issue3887    loewis                    
                                                                               

[PATCH] Document more deprecated modules in What's New in Python    0 days
       http://bugs.python.org/issue3888    georg.brandl              
       patch                                                                   

timedelta overflows in a surprising way                             1 days
       http://bugs.python.org/issue3893    benjamin.peterson         
                                                                               

3.0 documentation fails to build                                    0 days
       http://bugs.python.org/issue3898    georg.brandl              
                                                                               

ctypes: wrong calling convention for _string_at                     0 days
       http://bugs.python.org/issue3900    haypo                     
       patch                                                                   

pickle error in python3.0rc1                                        0 days
       http://bugs.python.org/issue3903    benjamin.peterson         
                                                                               

asynchat async_chat __init__() arguments changed in python 2.6      0 days
       http://bugs.python.org/issue3904    josiahcarlson             
                                                                               

lib2to3\main.py will not run                                        0 days
       http://bugs.python.org/issue3906    benjamin.peterson         
                                                                               

"for line in file" doesn't work for pipes                           0 days
       http://bugs.python.org/issue3907    amaury.forgeotdarc        
                                                                               

-S hides standard dynamic modules                                2247 days
       http://bugs.python.org/issue586680  brett.cannon              
                                                                               

classic division in demos/ directory                             2035 days
       http://bugs.python.org/issue687648  akuchling                 
       patch, easy                                                             

digit-only tag values are mishandled in Tkinter                  1286 days
       http://bugs.python.org/issue1160383 gpolo                     
                                                                               

tkapp read-only attributes                                       1130 days
       http://bugs.python.org/issue1257772 gpolo                     
                                                                               



Top Issues Most Discussed (10)
______________________________

 11 errors on _bsddb creation and dealloc                              3 days
open    http://bugs.python.org/issue3885   

 11 Major reworking of Python 2.5.2 re module                         10 days
open    http://bugs.python.org/issue3825   

  9 2.6rc1: test_threading hangs on FreeBSD 6.3 i386                   5 days
open    http://bugs.python.org/issue3863   

  9 test_array fails on FreeBSD7 amd64                                 6 days
open    http://bugs.python.org/issue3862   

  8 bsddb: test01_basic_replication fails on Windows sometimes         2 days
open    http://bugs.python.org/issue3892   

  8 Integer overflow in _hashopenssl.c (CVE-2008-2316)                 2 days
open    http://bugs.python.org/issue3886   

  8 tkinter goes into an infinite loop (pydoc.gui)                     8 days
open    http://bugs.python.org/issue3835   

  7 Add MS EULA to the list of third-party licenses in the	Windows    25 days
closed  http://bugs.python.org/issue3617   

  6 find_recursion_limit.py is broken                                  1 days
closed  http://bugs.python.org/issue3850   

  6 sqlite3 module: Improved concurrency                               0 days
closed  http://bugs.python.org/issue3846   




From shigin at rambler-co.ru  Fri Sep 19 18:02:46 2008
From: shigin at rambler-co.ru (Alexander Shigin)
Date: Fri, 19 Sep 2008 20:02:46 +0400
Subject: [Python-Dev] What this code should do?
In-Reply-To: <693bc9ab0809190843g38a09a69q39e0b23b04387759@mail.gmail.com>
References: <693bc9ab0809190843g38a09a69q39e0b23b04387759@mail.gmail.com>
Message-ID: <1221840166.6131.3.camel@atlas>

? ???, 19/09/2008 ? 17:43 +0200, Maciej Fijalkowski ?????:
> Hello,
> 
> I'm a little clueless about exact semantics of following snippets:
> 
> http://paste.pocoo.org/show/85698/
> 
> is this fine?
> or shall I fill the bug?
> (the reason to ask is because a) django is relying on this b) pypy
> implements it differently)

It seems ok. The exception is raised in except clause and it doesn't
handle by any except.


From amauryfa at gmail.com  Fri Sep 19 18:26:05 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Fri, 19 Sep 2008 18:26:05 +0200
Subject: [Python-Dev] What this code should do?
In-Reply-To: <693bc9ab0809190843g38a09a69q39e0b23b04387759@mail.gmail.com>
References: <693bc9ab0809190843g38a09a69q39e0b23b04387759@mail.gmail.com>
Message-ID: <e27efe130809190926p4d37a588se6d02e8abf09927f@mail.gmail.com>

Hello Maciej,

Maciej Fijalkowski wrote:
> Hello,
>
> I'm a little clueless about exact semantics of following snippets:
>
> http://paste.pocoo.org/show/85698/
>
> is this fine?
> or shall I fill the bug?
> (the reason to ask is because a) django is relying on this b) pypy
> implements it differently)

Note that python 3.0 has a different behaviour; in the first sample, it prints:
A (<class 'NameError'> ...
B (<class 'ZeroDivisionError'>, ...

See the subtle differences between
http://docs.python.org/dev/library/sys.html#sys.exc_info
http://docs.python.org/dev/3.0/library/sys.html#sys.exc_info

-- 
Amaury Forgeot d'Arc

From mrs at mythic-beasts.com  Fri Sep 19 20:06:20 2008
From: mrs at mythic-beasts.com (Mark Seaborn)
Date: Fri, 19 Sep 2008 19:06:20 +0100 (BST)
Subject: [Python-Dev] Unbound methods (was:  ANNOUNCE: CapPython...)
In-Reply-To: <ca471dc20809181453p36adeec3wd7c83099d32b0719@mail.gmail.com>
References: <gaudu8$9q5$1@ger.gmane.org>
	<20080918.221549.424243434.mrs@localhost.localdomain>
	<ca471dc20809181453p36adeec3wd7c83099d32b0719@mail.gmail.com>
Message-ID: <20080919.190620.343190642.mrs@localhost.localdomain>

"Guido van Rossum" <guido at python.org> wrote:

> On Thu, Sep 18, 2008 at 2:15 PM, Mark Seaborn <mrs at mythic-beasts.com> wrote:

> > Yes.  The renaming of "im_self" and "im_func" is good.  The removal of
> > unbound methods is a *big* problem [1].
> >
> > Regards,
> > Mark
> >
> > [1] http://lackingrhoticity.blogspot.com/2008/09/cappython-unbound-methods-and-python-30.html
> 
> I don't know to what extent you want to modify Python fundamentals,
> but I think this could be solved simply by adding a metaclass that
> returns an unbound method object for C.f, couldn't it?

I have considered that, and it does appear to be possible to use
metaclasses for that.  It looks like CapPython could set the new
__build_class__ builtin (on a per-module basis), which means that the
verifier would not need to require that every class has a
"metaclass=safemetaclass" declaration.

However, there is a problem which occurs when CapPython code interacts
with normal Python code.

In Python 2.x, CapPython has the very nice property that it is usually
safe to pass normal objects and classes into CapPython code without
allowing the CapPython code to break encapsulation:

 * CapPython code can only use instance objects via their public
   interfaces.
 * If CapPython code receives a class object C, it can create a derived
   class D, but it cannot access private attributes of instances of C
   unless they are also instances of D.  Holding C gives you only limited
   authority: you can only look inside objects whose classes you have
   defined.

There are some builtin objects that are unsafe - e.g. open, getattr,
type - but it is rare for these to be passed around as first class
values.  In constrast, class objects are often passed around to be
used as constructors.

Without unbound methods, normal Python class objects become dangerous
objects.  It becomes much more likely that normal Python code could
accidentally pass a class object in to CapPython code.  So if Python
code defines

class C(object):
    def f(self):
        return self._foo

- then if CapPython code gets hold of C, it can apply C.f(x) to get
x._foo of any object.


I don't really understand the rationale for removing unbound methods.

OK, it simplifies the language slightly.  Sometimes that is good,
sometimes that is bad.

OK, there might occasionally be use cases where you want to define a
function in a class scope and get back the unwrapped function.  But
you can easily get it via the im_func attribute (now __func__).

One of the posts in the original discussion [1] said that removing
unbound methods brings class attributes into line with builtin methods
such as list.append on the grounds that

  list.append is list.__dict__["append"]

is True.  I don't agree: list.append already applies type check:

>>> class C(object): pass
>>> list.append(C(), 1)
TypeError: descriptor 'append' requires a 'list' object but received a 'C'

It has to do so otherwise the interpreter could crash.  The check
added by unbound methods makes class attributes consistent with these
builtins.  Removing unbound methods introduces an inconsistency.

Also, what about the "only one way to do it" principle?  If you want
to define a function that can be applied to any type, there's already
a way to do that: define it outside of a class.

Regards,
Mark

[1] http://mail.python.org/pipermail/python-dev/2005-January/050685.html

From walter at livinglogic.de  Fri Sep 19 20:57:45 2008
From: walter at livinglogic.de (=?UTF-8?B?V2FsdGVyIETDtnJ3YWxk?=)
Date: Fri, 19 Sep 2008 20:57:45 +0200
Subject: [Python-Dev] Code coverage
Message-ID: <48D3F629.1000804@livinglogic.de>

Hello all!

The code coverage site at http://coverage.livinglogic.de/ was broken for
the last few months. It's fixed again now and runs the test suite once
per day with

   regrtest.py -T -N -uurlfetch,largefile,network,decimal

Servus,
   Walter

From exarkun at divmod.com  Fri Sep 19 21:30:36 2008
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Fri, 19 Sep 2008 15:30:36 -0400
Subject: [Python-Dev] What this code should do?
In-Reply-To: <e27efe130809190926p4d37a588se6d02e8abf09927f@mail.gmail.com>
Message-ID: <20080919193036.29191.1190439591.divmod.quotient.28388@ohm>

On Fri, 19 Sep 2008 18:26:05 +0200, Amaury Forgeot d'Arc <amauryfa at gmail.com> wrote:
>Hello Maciej,
>
>Maciej Fijalkowski wrote:
>> Hello,
>>
>> I'm a little clueless about exact semantics of following snippets:
>>
>> http://paste.pocoo.org/show/85698/
>>
>> is this fine?
>> or shall I fill the bug?
>> (the reason to ask is because a) django is relying on this b) pypy
>> implements it differently)
>
>Note that python 3.0 has a different behaviour; in the first sample, it prints:
>A (<class 'NameError'> ...
>B (<class 'ZeroDivisionError'>, ...
>
>See the subtle differences between
>http://docs.python.org/dev/library/sys.html#sys.exc_info
>http://docs.python.org/dev/3.0/library/sys.html#sys.exc_info
>

The second example changes its behavior, too.  It gives back the NameError
from the exc_info call.  I'm having a hard time reconciling this with the
Python 3.0 documentation.  Can you shed some light?

Jean-Paul

From fijall at gmail.com  Fri Sep 19 21:34:48 2008
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 19 Sep 2008 21:34:48 +0200
Subject: [Python-Dev] What this code should do?
In-Reply-To: <20080919193036.29191.1190439591.divmod.quotient.28388@ohm>
References: <e27efe130809190926p4d37a588se6d02e8abf09927f@mail.gmail.com>
	<20080919193036.29191.1190439591.divmod.quotient.28388@ohm>
Message-ID: <693bc9ab0809191234q73d35d7bs532baeb183359369@mail.gmail.com>

On Fri, Sep 19, 2008 at 9:30 PM, Jean-Paul Calderone <exarkun at divmod.com> wrote:
> On Fri, 19 Sep 2008 18:26:05 +0200, Amaury Forgeot d'Arc
> <amauryfa at gmail.com> wrote:
>>
>> Hello Maciej,
>>
>> Maciej Fijalkowski wrote:
>>>
>>> Hello,
>>>
>>> I'm a little clueless about exact semantics of following snippets:
>>>
>>> http://paste.pocoo.org/show/85698/
>>>
>>> is this fine?
>>> or shall I fill the bug?
>>> (the reason to ask is because a) django is relying on this b) pypy
>>> implements it differently)
>>
>> Note that python 3.0 has a different behaviour; in the first sample, it
>> prints:
>> A (<class 'NameError'> ...
>> B (<class 'ZeroDivisionError'>, ...
>>
>> See the subtle differences between
>> http://docs.python.org/dev/library/sys.html#sys.exc_info
>> http://docs.python.org/dev/3.0/library/sys.html#sys.exc_info
>>
>
> The second example changes its behavior, too.  It gives back the NameError
> from the exc_info call.  I'm having a hard time reconciling this with the
> Python 3.0 documentation.  Can you shed some light?
>
> Jean-Paul
>

I think in python 2.x it's at least against the principle of least
surprise. It should not behave differently. The behavior of python 3
though it's even against docs :-/

From brett at python.org  Fri Sep 19 22:13:48 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 19 Sep 2008 13:13:48 -0700
Subject: [Python-Dev] Code coverage
In-Reply-To: <48D3F629.1000804@livinglogic.de>
References: <48D3F629.1000804@livinglogic.de>
Message-ID: <bbaeab100809191313g49fc429lfbe4b5f723dc675c@mail.gmail.com>

On Fri, Sep 19, 2008 at 11:57 AM, Walter D?rwald <walter at livinglogic.de> wrote:
> Hello all!
>
> The code coverage site at http://coverage.livinglogic.de/ was broken for
> the last few months. It's fixed again now and runs the test suite once
> per day with
>
>   regrtest.py -T -N -uurlfetch,largefile,network,decimal
>

Thanks, Walter! Hopefully once Benjamin's testing cleanup lands we
work on improving our test coverage.

-Brett

From jcea at jcea.es  Sat Sep 20 05:36:35 2008
From: jcea at jcea.es (Jesus Cea)
Date: Sat, 20 Sep 2008 05:36:35 +0200
Subject: [Python-Dev] bsddb tests disabled by default
In-Reply-To: <bbaeab100809181043r54245982x48eea94d56b1fa0d@mail.gmail.com>
References: <48D264BA.8040202@jcea.es>
	<bbaeab100809181043r54245982x48eea94d56b1fa0d@mail.gmail.com>
Message-ID: <48D46FC3.7030802@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brett Cannon wrote:
> Well, 'time' says the test takes 16.09 sec user and 16.09 sec system
> on my MacBook, but a total execution time of almost 8 *minutes*. That
> is too long to be on by default.

Uhmmmm... That is very strange.

Under Solaris 10:

"""
[jcea at tesalia trunk]$ time python2.6 test.py -bv
Found Berkeley DB 4.7 installation.
  include files in /usr/local/BerkeleyDB.4.7/include
  library files in /usr/local/BerkeleyDB.4.7/lib
  library name is libdb-4.7
running build
running build_py
running build_ext
Running tests from /export/home/pybsddb/trunk/build

-
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Berkeley DB 4.7.25: (May 15, 2008)
bsddb.db.version():   (4, 7, 25)
bsddb.db.__version__: 4.7.3pre9
bsddb.db.cvsid:       $Id: _bsddb.c 620 2008-09-18 14:59:59Z jcea $
py module:
/export/home/pybsddb/trunk/build/lib.solaris-2.10-i86pc-2.6/bsddb3/__init__.pyc
extension module:
/export/home/pybsddb/trunk/build/lib.solaris-2.10-i86pc-2.6/bsddb3/_pybsddb.so
python version:       2.6rc2 (r26rc2:66504, Sep 18 2008, 15:51:56)
[GCC 4.2.3]
My pid:               17223
-
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
.................................................................................................................................................................................................................................................................................................................................
- ----------------------------------------------------------------------
Ran 321 tests in 13.510s

OK

real    0m13.786s
user    0m8.544s
sys     0m1.636s

"""

A lot of the disk traffic generated by the testsuite is "syncronous". By
default, under unix, the testsuite should be stored in "/tmp", that is
usually a ramdisk or something similar. That is the case under Solaris 10.

I don't know about MacOS.

I'm executing the testsuite under linux, with a /tmp backed by a proper
persistent FS (ReiserFS3). This machine is fairly busy, so the testsuite
actual time should be better:

"""
jcea at castor:~/mnt/particion_1/python/pybsddb/trunk> time python test.py -bv
Found Berkeley DB 4.3 installation.
  include files in /usr/include/db4
  library files in /usr/lib
  library name is libdb-4.3
running build
running build_py
running build_ext
Running tests from /home/jcea/mnt/particion_1/python/pybsddb/trunk/build

-
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Sleepycat Software: Berkeley DB 4.3.27: (September  9, 2005)
bsddb.db.version():   (4, 3, 27)
bsddb.db.__version__: 4.7.3pre9
bsddb.db.cvsid:       $Id: _bsddb.c 620 2008-09-18 14:59:59Z jcea $
py module:
/home/jcea/mnt/particion_1/python/pybsddb/trunk/build/lib.linux-i686-2.5/bsddb3/__init__.pyc
extension module:
/home/jcea/mnt/particion_1/python/pybsddb/trunk/build/lib.linux-i686-2.5/bsddb3/_pybsddb.so
python version:       2.5.2 (r252:60911, Mar 13 2008, 12:51:08)
[GCC 4.0.2 20050901 (prerelease) (SUSE Linux)]
My pid:               19105
-
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
........................................................................................................................................................................................................................................................................................................................
- ----------------------------------------------------------------------
Ran 312 tests in 37.984s

OK

real    0m38.718s
user    0m17.905s
sys     0m3.252s
"""

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSNRvtplgi5GaxT1NAQKf6QP+P1pYzY02dlgJCKoLLlSjlFwOKa+uWrjK
pqbJJFKIf8RTbMWGIutYPr03pdI1T0Y3JadVfHDC/lAc/59BcbOtMhKYFlAFPlik
ZEC9oW02zzve0+thwpmxMPeKA6CeLboYW+cGkoUhtGayffQObrrTh0Zi47BcTUL6
e46liag7/ZA=
=TCJf
-----END PGP SIGNATURE-----

From brett at python.org  Sat Sep 20 06:46:31 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 19 Sep 2008 21:46:31 -0700
Subject: [Python-Dev] bsddb tests disabled by default
In-Reply-To: <48D46FC3.7030802@jcea.es>
References: <48D264BA.8040202@jcea.es>
	<bbaeab100809181043r54245982x48eea94d56b1fa0d@mail.gmail.com>
	<48D46FC3.7030802@jcea.es>
Message-ID: <bbaeab100809192146t2a0a8f68va2e1cf149384378e@mail.gmail.com>

On Fri, Sep 19, 2008 at 8:36 PM, Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Brett Cannon wrote:
>> Well, 'time' says the test takes 16.09 sec user and 16.09 sec system
>> on my MacBook, but a total execution time of almost 8 *minutes*. That
>> is too long to be on by default.
>
> Uhmmmm... That is very strange.

Well, it has always been that way for me, so I always assumed
test_bsddb3 was just a *really* long test.

> A lot of the disk traffic generated by the testsuite is "syncronous". By
> default, under unix, the testsuite should be stored in "/tmp", that is
> usually a ramdisk or something similar. That is the case under Solaris 10.
>
> I don't know about MacOS.
>

Don't think it is: drwxrwxrwt  11 root  wheel   374B 19 Sep 20:44 tmp/

> I'm executing the testsuite under linux, with a /tmp backed by a proper
> persistent FS (ReiserFS3). This machine is fairly busy, so the testsuite
> actual time should be better:
>

But you could have a faster CPU, more RAM, Reiser could easily be
faster than HFS+, etc. There is no way any of these comparisons are
going to work. OS X might just plain suck at running test_bsddb3.

Only thing I can think of is that Berkeley DB 4.7 is a ton faster than
4.6 or I am running something differently than you:

time ./python.exe Lib/test/regrtest.py -uall test_bsddb3
~/Dev/python/2.x/pristine
test_bsddb3
Berkeley DB 4.6.21: (September 27, 2007)
Test path prefix:
/var/folders/MN/MN-E3HgoFXSKDXb9le7FQ++++TI/-Tmp-/z-test_bsddb3-527
  test_bsddb3 still working, be patient...
1 test OK.
[48048 refs]
./python.exe Lib/test/regrtest.py -uall test_bsddb3  15.81s user
15.54s system 6% cpu 8:41.56 total


-Brett

From martin at v.loewis.de  Sat Sep 20 14:55:37 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 20 Sep 2008 14:55:37 +0200
Subject: [Python-Dev] bsddb tests disabled by default
In-Reply-To: <48D46FC3.7030802@jcea.es>
References: <48D264BA.8040202@jcea.es>	<bbaeab100809181043r54245982x48eea94d56b1fa0d@mail.gmail.com>
	<48D46FC3.7030802@jcea.es>
Message-ID: <48D4F2C9.7000502@v.loewis.de>

> real    0m13.786s

test_bsddb3 takes about 30s real time on my system (Linux, with
Berkeley DB 4.6.21).

I don't think the default (of requiring the bsddb resource) can
change for 2.6; we already have released rc2, so there won't be
any further release candidates.

For 2.7, I would still be hesitant to run this test by default - 30s
is too long, IMO. It is unlikely that regular changes to Python break
any of these tests, and if they do, the buildbots will tell. Anybody
changing the bsddb library itself will know to run the full test suite.

Regards,
Martin

From martin at v.loewis.de  Sat Sep 20 14:58:37 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 20 Sep 2008 14:58:37 +0200
Subject: [Python-Dev] bsddb tests disabled by default
In-Reply-To: <bbaeab100809192146t2a0a8f68va2e1cf149384378e@mail.gmail.com>
References: <48D264BA.8040202@jcea.es>	<bbaeab100809181043r54245982x48eea94d56b1fa0d@mail.gmail.com>	<48D46FC3.7030802@jcea.es>
	<bbaeab100809192146t2a0a8f68va2e1cf149384378e@mail.gmail.com>
Message-ID: <48D4F37D.2090106@v.loewis.de>

> Only thing I can think of is that Berkeley DB 4.7 is a ton faster than
> 4.6 or I am running something differently than you:

My suspicion is that there is a bug somewhere, probably in Berkeley DB.
For example, it might acquire some lock with a timeout, hoping that
normally, the lock gets released elsewhere quickly. On OSX, for whatever
reason, that assumption might be false, so the timeout eventually
occurs, along with retries and whatnot.

Of course, that's just a theory - one would have to debug the test suite
to find out what's really going on.

Regards,
Martin

From skip at pobox.com  Sat Sep 20 17:53:10 2008
From: skip at pobox.com (skip at pobox.com)
Date: Sat, 20 Sep 2008 10:53:10 -0500
Subject: [Python-Dev] bsddb tests disabled by default
In-Reply-To: <bbaeab100809192146t2a0a8f68va2e1cf149384378e@mail.gmail.com>
References: <48D264BA.8040202@jcea.es>
	<bbaeab100809181043r54245982x48eea94d56b1fa0d@mail.gmail.com>
	<48D46FC3.7030802@jcea.es>
	<bbaeab100809192146t2a0a8f68va2e1cf149384378e@mail.gmail.com>
Message-ID: <18645.7270.327912.397448@montanaro-dyndns-org.local>


    Brett> Well, it has always been that way for me, so I always assumed
    Brett> test_bsddb3 was just a *really* long test.

Slow for me, but not nearly as bad as for you:

    % time ./python.exe ../Lib/bsddb/test/test_all.py 

    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Sleepycat Software: Berkeley DB 4.4.20: (January 10, 2006)
    bsddb.db.version():   (4, 4, 20)
    bsddb.db.__version__: 4.7.3pre5
    bsddb.db.cvsid:       $Id: _bsddb.c 66182 2008-09-03 17:50:32Z jesus.cea $
    py module:            /Users/skip/src/python/trunk/Lib/bsddb/__init__.pyc
    extension module:     /Users/skip/src/python/trunk/regular/build/lib.macosx-10.3-i386-2.6/_bsddb.so
    python version:       2.6rc2+ (trunk:66519M, Sep 20 2008, 08:36:03) 
    [GCC 4.0.1 (Apple Inc. build 5465)]
    My pid:               82520
    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    ............................................................. ....
    ----------------------------------------------------------------------
    Ran 294 tests in 156.791s

    OK

    real    2m37.188s
    user    0m9.907s
    sys     0m6.196s

One thing I noticed was that you and I are both using BerkDB 4.4.20 while
Jesus is running 4.7.25.  I can't get to 4.7.x with MacPorts, but I can get
to 4.6.21.  I installed that, rebuild bsddb with it and reran the tests:

    % time ./python.exe ../Lib/bsddb/test/test_all.py 

    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    Berkeley DB 4.6.21: (September 27, 2007)
    bsddb.db.version():   (4, 6, 21)
    bsddb.db.__version__: 4.7.3pre5
    bsddb.db.cvsid:       $Id: _bsddb.c 66182 2008-09-03 17:50:32Z jesus.cea $
    py module:            /Users/skip/src/python/trunk/Lib/bsddb/__init__.pyc
    extension module:     /Users/skip/src/python/trunk/regular/build/lib.macosx-10.3-i386-2.6/_bsddb.so
    python version:       2.6rc2+ (trunk:66519M, Sep 20 2008, 08:36:03) 
    [GCC 4.0.1 (Apple Inc. build 5465)]
    My pid:               21203
    -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
    ............................................................................................................................................................................................................................................................................................................
    ----------------------------------------------------------------------
    Ran 300 tests in 557.679s

    OK

    real    9m18.093s
    user    0m10.499s
    sys     0m16.709s

Hmmm...  Those extra six tests are expensive!  I noticed there was a fair
chunk of time where the CPU meter showed the CPU essentially idle and the
dots were not moving.  I trust it was waiting for the rather modest disk on
my laptop.

Next stop, in-memory disk (all commands as root):

    hdid -nomount ram://1024
    newfs /dev/rdisk1
    mkdir /tmp/mem
    mount /dev/disk1 /tmp/mem
    chmod 1777 /tmp/mem

and rerun the tests with TMPDIR pointing at /tmp/mem.  Whoops, it looks like
test_support creates temp files in the current directory, ignoring TMPDIR or
tempfile.gettempdir().  (Why is that???)  So, cd to /tmp/mem first.  Whoops!
Again, the bsddb tests force the test database into /tmp.  Fix test_all.py
to use TMPDIR if set.  Damn!  1gb isn't enough.  I tried boosting it to 2gb.
Still no go.  Jesus, how big is your ramdisk?

Here's a couple line patch for bsddb/test/test_all.py that uses TMPDIR if
it's set.

Index: Lib/bsddb/test/test_all.py
===================================================================
--- Lib/bsddb/test/test_all.py  (revision 66520)
+++ Lib/bsddb/test/test_all.py  (working copy)
@@ -443,6 +443,9 @@
 def set_test_path_prefix(path) :
     get_new_path.prefix=path
 
+if "TMPDIR" in os.environ:
+    set_test_path_prefix(os.path.join(os.environ["TMPDIR"], "z-Berkeley_DB"))
+
 def remove_test_path_directory() :
     test_support.rmtree(get_new_path.prefix)

Skip

From fredrik at pythonware.com  Sun Sep 21 11:20:10 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Sun, 21 Sep 2008 11:20:10 +0200
Subject: [Python-Dev] how about updating PEP 290?
Message-ID: <gb53ka$6j0$1@ger.gmane.org>

the "Code Migration and Modernization" PEP hasn't been updated for 2.5 
and 2.6.

     http://www.python.org/dev/peps/pep-0290/

surely there's something new in 2.5 and 2.6 that might be worth mentioning?

</F>


From g.brandl at gmx.net  Sun Sep 21 12:03:43 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 21 Sep 2008 10:03:43 +0000
Subject: [Python-Dev] Python documentation
In-Reply-To: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>
Message-ID: <gb5663$iqt$1@ger.gmane.org>

Barry Warsaw schrieb:
> Martin points out that in the past, as part of the release process,
> we've built separate downloadable documentation.
> 
> Do we still want to do that for Python 2.6 and 3.0, and if so, how do
> we go about doing that?  I have this feeling that building the
> documentation is much different now than in the past, and I don't
> really have a good feel for how it's done now.
> 
> If you think we should release separate downloadable documentation and
> can help integrate that into the release project, you just might be a
> Documentation Expert <wink>.  Please let me know if you can help.

There is almost everything ready for doing this. There is a "download"
page in the HTML docs (that at the moment contains nothing).

If you tell me where the downloadable files will be on python.org, I can
add them to the "download" page and you only need to build the docs and
put them in that location. I've just added a "dist" target to the Doc/
Makefile, so a "make dist" should place all needed files in the Doc/dist
directory, from where you can copy them to the desired location.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From barry at python.org  Sun Sep 21 13:24:49 2008
From: barry at python.org (Barry Warsaw)
Date: Sun, 21 Sep 2008 07:24:49 -0400
Subject: [Python-Dev] Python documentation
In-Reply-To: <gb5663$iqt$1@ger.gmane.org>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>
	<gb5663$iqt$1@ger.gmane.org>
Message-ID: <E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 21, 2008, at 6:03 AM, Georg Brandl wrote:

> Barry Warsaw schrieb:
>> Martin points out that in the past, as part of the release process,
>> we've built separate downloadable documentation.
>>
>> Do we still want to do that for Python 2.6 and 3.0, and if so, how do
>> we go about doing that?  I have this feeling that building the
>> documentation is much different now than in the past, and I don't
>> really have a good feel for how it's done now.
>>
>> If you think we should release separate downloadable documentation  
>> and
>> can help integrate that into the release project, you just might be a
>> Documentation Expert <wink>.  Please let me know if you can help.
>
> There is almost everything ready for doing this. There is a "download"
> page in the HTML docs (that at the moment contains nothing).
>
> If you tell me where the downloadable files will be on python.org, I  
> can
> add them to the "download" page and you only need to build the docs  
> and
> put them in that location. I've just added a "dist" target to the Doc/
> Makefile, so a "make dist" should place all needed files in the Doc/ 
> dist
> directory, from where you can copy them to the desired location.

Benjamin has hacked on the release.py script to build and sign the  
documentation.  I haven't tried it yet but it looks like it does a  
'make html' and exports that.

Given the above, we should change that to 'make dist' and update the  
PEP to describe scp'ing them to dinsdale:/ftp/python/doc/X.Y using the  
templates on this page: http://docs.python.org/download.html

I notice that for 2.5, we zip'd and tar-bz2'd them.  Do we want to  
also support tgz?  We'll have to hack the release script to build the  
doc zips.  I'll try to test drive this whole process soon.

Thanks!
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSNYvAXEjvBPtnXfVAQKj/QQApu2YzQLZfpazIb6jPxtWDMnpW99+TrRP
miMEwURQmncYWIK1kt9RuBpjszjKKw5x/pP9DEy7Slx+AQq13q1U2Ddi8yQmvWGk
Sf3rRxBbgG8QM5H67toB/T6kDtti8C0F0OZZFZpG83nAVZuwtomw7ZYZS2P5Qzq+
eZnW5aANX4g=
=HF5j
-----END PGP SIGNATURE-----

From heshan.suri at gmail.com  Sun Sep 21 16:24:55 2008
From: heshan.suri at gmail.com (Heshan Suriyaarachchi)
Date: Sun, 21 Sep 2008 09:24:55 -0500
Subject: [Python-Dev] Deploying a Python Service on Apache Axis2
Message-ID: <12e5d0d90809210724i12435ce6r1c80234a43dea4a3@mail.gmail.com>

Hi guys,
     Apache Axis2/Java, is a popular open source Web service engine. It
currently supports exposing services written in Java, Javascript as Web
services. This article [1] discusses the Python data Binding that enable
exposing Web services written in Python.

[1] - http://wso2.org/library/articles/deploying-python-service-axis2
[2] - http://heshans.blogspot.com/2008/09/wso2-wsfjython-10-alpha.html
[3] - http://wso2.org/library/invoking-enterprise-web-services-using-jython

-- 
Regards,
Heshan Suriyaarachchi

http://heshans.blogspot.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080921/1b38b4db/attachment.htm>

From fdrake at acm.org  Sun Sep 21 17:35:45 2008
From: fdrake at acm.org (Fred Drake)
Date: Sun, 21 Sep 2008 11:35:45 -0400
Subject: [Python-Dev] Python documentation
In-Reply-To: <E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>
	<gb5663$iqt$1@ger.gmane.org>
	<E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>
Message-ID: <E1530F66-4021-4D96-BA01-C84751E918F6@acm.org>

On Sep 21, 2008, at 7:24 AM, Barry Warsaw wrote:
> I notice that for 2.5, we zip'd and tar-bz2'd them.  Do we want to  
> also support tgz?  We'll have to hack the release script to build  
> the doc zips.  I'll try to test drive this whole process soon.


We specifically decided to drop .tgz since it didn't really make sense  
to have both .tgz and .tar.bz2, and the software to handle .tar.bz2 is  
widely deployed.


   -Fred

-- 
Fred Drake   <fdrake at acm.org>


From martin at v.loewis.de  Sun Sep 21 21:12:25 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 21 Sep 2008 21:12:25 +0200
Subject: [Python-Dev] Python documentation
In-Reply-To: <E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>	<gb5663$iqt$1@ger.gmane.org>
	<E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>
Message-ID: <48D69C99.6080708@v.loewis.de>

> I notice that for 2.5, we zip'd and tar-bz2'd them.  Do we want to also
> support tgz?  We'll have to hack the release script to build the doc
> zips.  I'll try to test drive this whole process soon.

In addition to the creation of downloadable scripts, there should also
be a copy of the documentation online, at

http://www.python.org/doc/2.6/
http://www.python.org/doc/3.0/

In addition, some documentation set should show up at

http://docs.python.org/

Not sure whether this should be 2.6 or 3.0.

Regards,
Martin

From musiccomposition at gmail.com  Sun Sep 21 21:25:35 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sun, 21 Sep 2008 14:25:35 -0500
Subject: [Python-Dev] Python documentation
In-Reply-To: <48D69C99.6080708@v.loewis.de>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>
	<gb5663$iqt$1@ger.gmane.org>
	<E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>
	<48D69C99.6080708@v.loewis.de>
Message-ID: <1afaf6160809211225i57de3489i680f0599651abd3e@mail.gmail.com>

On Sun, Sep 21, 2008 at 2:12 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> I notice that for 2.5, we zip'd and tar-bz2'd them.  Do we want to also
>> support tgz?  We'll have to hack the release script to build the doc
>> zips.  I'll try to test drive this whole process soon.
>
> In addition to the creation of downloadable scripts, there should also
> be a copy of the documentation online, at
>
> http://www.python.org/doc/2.6/
> http://www.python.org/doc/3.0/
>
> In addition, some documentation set should show up at
>
> http://docs.python.org/
>
> Not sure whether this should be 2.6 or 3.0.

Almost certainly 2.6.



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From brett at python.org  Sun Sep 21 21:37:32 2008
From: brett at python.org (Brett Cannon)
Date: Sun, 21 Sep 2008 12:37:32 -0700
Subject: [Python-Dev] Deploying a Python Service on Apache Axis2
In-Reply-To: <12e5d0d90809210724i12435ce6r1c80234a43dea4a3@mail.gmail.com>
References: <12e5d0d90809210724i12435ce6r1c80234a43dea4a3@mail.gmail.com>
Message-ID: <bbaeab100809211237ide46d81m8f7ec7f5a6556aa0@mail.gmail.com>

On Sun, Sep 21, 2008 at 7:24 AM, Heshan Suriyaarachchi
<heshan.suri at gmail.com> wrote:
> Hi guys,
>      Apache Axis2/Java, is a popular open source Web service engine. It
> currently supports exposing services written in Java, Javascript as Web
> services. This article [1] discusses the Python data Binding that enable
> exposing Web services written in Python.
>
> [1] - http://wso2.org/library/articles/deploying-python-service-axis2
> [2] - http://heshans.blogspot.com/2008/09/wso2-wsfjython-10-alpha.html
> [3] - http://wso2.org/library/invoking-enterprise-web-services-using-jython
>

Thanks for the links, Heshan, but python-dev is for discussing the
design of Python. For general announcements like this, they should go
to comp.lang.python or comp.lang.python.announce (both of which you
can get to at Google Groups).

-Brett

From Scott.Daniels at Acm.Org  Sun Sep 21 23:26:57 2008
From: Scott.Daniels at Acm.Org (Scott David Daniels)
Date: Sun, 21 Sep 2008 14:26:57 -0700
Subject: [Python-Dev] Possible issue in warnings.py
Message-ID: <gb6del$e8c$1@ger.gmane.org>

Does someone who knows the design of warnings.py a bit better than I
know whether it would be an improvement to switch from:
 >     try:
 >         file.write(formatwarning(message, category, filename,
 >                                  lineno, line))
 >     except IOError:
 >         pass # the file (probably stderr) is invalid
 >              # - this warning gets lost.

to:
 >     complaint = formatwarning(message, category, filename,
 >                               lineno, line)
 >     try:
 >         file.write(complaint)
 >     except IOError:
 >         pass # the file (probably stderr) is invalid
 >              # - this warning gets lost.

on the grounds that you might not want failures in the
linecache code to behave the same as failures in writing
the complaint to the target area?

I'm working on a patch where the Idle warnings code seems to
accidentally escalating warnings into errors, and it looked
to me like this would accidentally swallow errors getting
warning context and make them fail silently.  The Idle issue
that I'm fiddling with is that it doesn't take the new
showwarning call format, and it looked like this should
possibly be fixed at the same time.


--Scott David Daniels
Scott.Daniels at Acm.Org


From musiccomposition at gmail.com  Sun Sep 21 23:17:31 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sun, 21 Sep 2008 16:17:31 -0500
Subject: [Python-Dev] Possible issue in warnings.py
In-Reply-To: <gb6del$e8c$1@ger.gmane.org>
References: <gb6del$e8c$1@ger.gmane.org>
Message-ID: <1afaf6160809211417xcc94bc1m1465a7368c019250@mail.gmail.com>

On Sun, Sep 21, 2008 at 4:26 PM, Scott David Daniels
<Scott.Daniels at acm.org> wrote:
> Does someone who knows the design of warnings.py a bit better than I
> know whether it would be an improvement to switch from:
>>     try:
>>         file.write(formatwarning(message, category, filename,
>>                                  lineno, line))
>>     except IOError:
>>         pass # the file (probably stderr) is invalid
>>              # - this warning gets lost.
>
> to:
>>     complaint = formatwarning(message, category, filename,
>>                               lineno, line)
>>     try:
>>         file.write(complaint)
>>     except IOError:
>>         pass # the file (probably stderr) is invalid
>>              # - this warning gets lost.
>
> on the grounds that you might not want failures in the
> linecache code to behave the same as failures in writing
> the complaint to the target area?
>
> I'm working on a patch where the Idle warnings code seems to
> accidentally escalating warnings into errors, and it looked
> to me like this would accidentally swallow errors getting
> warning context and make them fail silently.  The Idle issue
> that I'm fiddling with is that it doesn't take the new
> showwarning call format, and it looked like this should
> possibly be fixed at the same time.

Sounds reasonable. Please file a ticket on the tracker.



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From ncoghlan at gmail.com  Mon Sep 22 00:11:50 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 22 Sep 2008 08:11:50 +1000
Subject: [Python-Dev] Python documentation
In-Reply-To: <1afaf6160809211225i57de3489i680f0599651abd3e@mail.gmail.com>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>	<gb5663$iqt$1@ger.gmane.org>	<E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>	<48D69C99.6080708@v.loewis.de>
	<1afaf6160809211225i57de3489i680f0599651abd3e@mail.gmail.com>
Message-ID: <48D6C6A6.1000806@gmail.com>

Benjamin Peterson wrote:
> On Sun, Sep 21, 2008 at 2:12 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>> I notice that for 2.5, we zip'd and tar-bz2'd them.  Do we want to also
>>> support tgz?  We'll have to hack the release script to build the doc
>>> zips.  I'll try to test drive this whole process soon.
>> In addition to the creation of downloadable scripts, there should also
>> be a copy of the documentation online, at
>>
>> http://www.python.org/doc/2.6/
>> http://www.python.org/doc/3.0/
>>
>> In addition, some documentation set should show up at
>>
>> http://docs.python.org/
>>
>> Not sure whether this should be 2.6 or 3.0.
> 
> Almost certainly 2.6.

Definitely 2.6 - 2.x is still going to be the "mainstream" Python
version for a good while yet.

Although it may be worth tweaking the main index page for the 2.6 docs
to include a cross-link to the 3.0 docs.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From gregor.lingl at aon.at  Mon Sep 22 00:15:36 2008
From: gregor.lingl at aon.at (Gregor Lingl)
Date: Mon, 22 Sep 2008 00:15:36 +0200
Subject: [Python-Dev] turtle.Screen.__init__ issue
Message-ID: <48D6C788.2050400@aon.at>

Hello there,

its high time to resolve an issue, which I have already addressed twice 
some weeks ago. (You can find a more elaborate description in my former 
posting cited below)

There is a tiny difference (also in behaviour!) in 
turtle.Screen.__init__() between the versions for 2.6 and 3.0. The 
difference results from the fact, that I submitted the 3.0 version 
approx. a week later, after having it ported to 3.0. In this process I 
had found what I now consider to be a bug in 2.6 and changed it 
accordingly.  Shortly:

If you have already a Screen object containing some turtles and some 
graphics,

in 2.6: s = Screen() returns an object with identical state and 
behaviour, but clears (re-initializes) the screen and thus destroys the 
content
in 3.0 s = Screen()  returns an object with identical state and 
behaviour, but leaves the content untouched

The difference in code consist only in indenting the call of the 
__init__ method of the parent class, so it will be executed only 
conditionally.

Anyway, as this difference between the two versions is highly 
undesirable there are (imho) three options to proceed:

(1) correct 2.6 in order that it will work like 3.0
(2) undo the change in 3.0 in order that it will work like 2.6
(3) find a different solution for both

I would (like Vern, see below) decisevely prefer option (1), and I 
suppose that there is not enough time left to chose option (3) as this 
would probably need some discussions.

What is your opinion, and who should decide?

For your convenience I've attached a diff-file which also contains the 
description of three other small bugs, which I've found in the meantime 
and which shouldn't cause any controversies.

Regards, Gregor


%%%%%%%%%%

Here follows the answer of Vern Ceder - a long term turtle graphics user 
and author of several patches for the old turtle module - to my former 
posting:

 >> Gregor,
 >>
 >> I don't feel authoritative on the correctness/appropriateness of the 
implementation,
 >> but I do agree completely that behavior b, or what you have in the 
3.0 version,
 >> is vastly preferable.
 >>
 >> Cheers,
 >> Vern



-------- Original-Nachricht --------
Betreff: 	[Python-Dev] turtle.Screen- how to implement best a Singleton
Datum: 	Mon, 18 Aug 2008 10:15:45 +0200
Von: 	Gregor Lingl <gregor.lingl at aon.at>
An: 	python-dev at python.org
CC: 	Toby Donaldson <tjd at sfu.ca>, python-3000 at python.org, 
jjposner at snet.net, Brad Miller <bonelake at gmail.com>, Vern Ceder 
<vceder at canterburyschool.org>



Hi,

this posting - concerning the new turtle module - goes to the Python-Dev 
and Python-3000 lists and to a couple of 'power users' of turtle 
graphics, hoping to recieve feedback from the developer's point of view 
as well as from the user's point of view.

Currently the implementations of the turtle.Screen class for Python 2.6 
and Python 3.0 differ by a 'tiny' detail with an important difference in 
behaviour. So clearly this has to be resolved  before the final 
release.(The origin of this difference is, that when I ported turtle.py 
to Python 3.0 I discovered (and 'fixed') what I now consider to be a bug 
in the 2.6 version.) I'd like to ask you kindly for your advice to 
achieve an optimal solution.

The posting consists of three parts:
1. Exposition of design goals
2. Problem with the implementation 
3. How to solve it?

Preliminary remark:  I've had some discussions on this topic before but 
I still do not see a clear solution. Moreover I'm well aware of the fact 
that using the Singleton pattern is controversial. So ...

1. Exposition of design goals
... why use the Singleton design pattern? The turtle module contains a 
TurtleScreen class, which implements methods to control the drawing area 
the turtle is (turtles are) drawing on. It's constructor needs a Tkinter 
Canvas as argument. In order to avoid the need for users to tinker 
around with Tkinter stuff there is the Screen(TurtleScreen) class, 
designed to be used by beginners(students, kids,...), particularly in 
interactive sessions.

A (THE (!)) Screen object is essentially a window containing a scrolled 
canvas, the TurtleScreen. So it's a ressource which should exist only 
once. It can be constructed in several ways:
- implicitely by calling an arbitrary function derived from a 
Turtle-method, such as forward(100) or by constructing a Turtle such as 
bob = Turtle()
- implicitely by calling an arbitrary function derived from a Screen 
method, such as bgcolor("red")
- explicitely by calling it's constructor such as s = Screen()
Anyway this construction should only happen if a Screen object doesn't 
exist yet.
Now for the pending question: What should happen, when s = Screen() is 
called explicitely and there exists already 'the' Screen object.
(i) Clearly s should get a reference to the existing Screen object, but ...
(ii) (a)... should s be reinitialized (this is the case now in Python 
2.6), or
    (b)... should s be left untouched (this is the case now in Python 3.0)

I, for my part, prefer the latter solution (b). Example: a student, 
having (interactively) produced some design using some turtle t = 
Turtle() decides spontaneously to change backgroundcolor. s = Screen(); 
s.bgcolor("pink") should do this for her - instead of deleting her 
design and moreover her turtle. To reinitialize the screen she still can 
use s.clear().

Of course, there are workarounds to achieve the same effect also with 
solution (a), for instance by assigning s = Screen() *before* drawing 
anything or by assigning s = t.getscreen(). But imho (which derives 
itself from my experience as a teacher) solution (b) supports better the 
oop-view as well as experimenting spontaneously in interactive sessions.

2. Problem with the implementation
The task is to derive a Singleton class from a Nonsingleton class 
(Screen from TurtleScreen). The current implementations of the Screen 
'Singleton' both use the Borg idiom.  Just for *explaining* the 
difference between the two versions of class Screen here concisely,  
I'll use a 'standard' Singleton pattern (roughly equivalent to the Borg 
idiom):

class Spam(object):
   def __init__(self, s):
       self.s = s

class SingleSpam(Spam):
   _inst = None
   def __new__(cls, *args, **kwargs):       
       if cls != type(cls._inst):
           cls._inst = Spam.__new__(cls, *args, **kwargs)
       return cls._inst
   def __init__(self, s):
       if vars(self): return    ######  should this be here???
       Spam.__init__(self, s)

Shortly, this means that SingleSpam.__init__() acts like an empty method 
whenever a (the!) SingleSpam object already exists. 3.0 version of 
Screen acts like this. By contrast 2.6 version of Screen acts as if the 
butlast line were not there and thus reinitializes the Screen object.

3. How to solve it?

Main question: which *behaviour* of the Screen class should be 
preferred.  If  3.0, is it feasible and correct not to call the 
constructor of the parent class if the object already exists?

Additional question: Do you consider the Borg idiom a good solution for 
this task or should the standard singleton pattern as shown above be 
preferred. Or would you suggest a solution/an approach different from both?

Thanks for your patience, and - in advance - for your assistance

Regard,
Gregor




_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/gregor.lingl%40aon.at



-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: turtle_patch_rc2-diff.txt
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080922/ac8279f3/attachment.txt>

From g.brandl at gmx.net  Mon Sep 22 00:37:06 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 22 Sep 2008 00:37:06 +0200
Subject: [Python-Dev] Python documentation
In-Reply-To: <E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>	<gb5663$iqt$1@ger.gmane.org>
	<E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>
Message-ID: <gb6iam$ptl$1@ger.gmane.org>

Barry Warsaw schrieb:
> On Sep 21, 2008, at 6:03 AM, Georg Brandl wrote:
> 
>> Barry Warsaw schrieb:
>>> Martin points out that in the past, as part of the release process,
>>> we've built separate downloadable documentation.
>>>
>>> Do we still want to do that for Python 2.6 and 3.0, and if so, how do
>>> we go about doing that?  I have this feeling that building the
>>> documentation is much different now than in the past, and I don't
>>> really have a good feel for how it's done now.
>>>
>>> If you think we should release separate downloadable documentation  
>>> and
>>> can help integrate that into the release project, you just might be a
>>> Documentation Expert <wink>.  Please let me know if you can help.
> 
>> There is almost everything ready for doing this. There is a "download"
>> page in the HTML docs (that at the moment contains nothing).
> 
>> If you tell me where the downloadable files will be on python.org, I  
>> can
>> add them to the "download" page and you only need to build the docs  
>> and
>> put them in that location. I've just added a "dist" target to the Doc/
>> Makefile, so a "make dist" should place all needed files in the Doc/ 
>> dist
>> directory, from where you can copy them to the desired location.
> 
> Benjamin has hacked on the release.py script to build and sign the  
> documentation.  I haven't tried it yet but it looks like it does a  
> 'make html' and exports that.
> 
> Given the above, we should change that to 'make dist' and update the  
> PEP to describe scp'ing them to dinsdale:/ftp/python/doc/X.Y using the  
> templates on this page: http://docs.python.org/download.html

I've now completed the download page for the new docs. The file names it
expects are currently

http://docs.python.org/ftp/python/doc/$version/python-docs-$format.$ext

which is what "make dist" generates.  If you want to have the version in
the filename as well, use "make dist DISTVERSION=2.6" and I'll adapt the
download page accordingly.

> I notice that for 2.5, we zip'd and tar-bz2'd them.  Do we want to  
> also support tgz?  We'll have to hack the release script to build the  
> doc zips.  I'll try to test drive this whole process soon.

As Fred says, bz2 should suffice. gzip offers no compression advantage
over zip either.

I've also dropped the PostScript download versions; I see no compelling
reason to use them instead of PDF.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From martin at v.loewis.de  Mon Sep 22 07:07:28 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Mon, 22 Sep 2008 07:07:28 +0200
Subject: [Python-Dev] turtle.Screen.__init__ issue
In-Reply-To: <48D6C788.2050400@aon.at>
References: <48D6C788.2050400@aon.at>
Message-ID: <48D72810.8090409@v.loewis.de>

> What is your opinion, and who should decide?

Please don't post patches to the mailing list. Post them exclusively
to bugs.python.org instead.

Also, don't post unrelated patches in a single message. Create separate
issues in the bug tracker for them.

Regards,
Martin

From gregor.lingl at aon.at  Mon Sep 22 08:41:15 2008
From: gregor.lingl at aon.at (Gregor Lingl)
Date: Mon, 22 Sep 2008 08:41:15 +0200
Subject: [Python-Dev] turtle.Screen.__init__ issue
In-Reply-To: <48D72810.8090409@v.loewis.de>
References: <48D6C788.2050400@aon.at> <48D72810.8090409@v.loewis.de>
Message-ID: <48D73E0B.50602@aon.at>



Martin v. L?wis schrieb:
>> What is your opinion, and who should decide?
>>     
>
> Please don't post patches to the mailing list. Post them exclusively
> to bugs.python.org instead.
>   
Ok. But this was meant for illustrative purposes only and not as a 
patch-submission, which
I'll do eventually separately.

Sorry
Gregor
> Also, don't post unrelated patches in a single message. Create separate
> issues in the bug tracker for them.
>
> Regards,
> Martin
>
>
>   

From steve at holdenweb.com  Mon Sep 22 13:17:38 2008
From: steve at holdenweb.com (Steve Holden)
Date: Mon, 22 Sep 2008 07:17:38 -0400
Subject: [Python-Dev] Python documentation
In-Reply-To: <48D6C6A6.1000806@gmail.com>
References: <880B19BE-9665-4E27-AE01-FEFCBA51B138@python.org>	<gb5663$iqt$1@ger.gmane.org>	<E4F83E84-A58D-4B5A-AE75-B1E97F76CD68@python.org>	<48D69C99.6080708@v.loewis.de>	<1afaf6160809211225i57de3489i680f0599651abd3e@mail.gmail.com>
	<48D6C6A6.1000806@gmail.com>
Message-ID: <gb7usk$ra$1@ger.gmane.org>

Nick Coghlan wrote:
> Benjamin Peterson wrote:
>> On Sun, Sep 21, 2008 at 2:12 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
[...]
>>> In addition, some documentation set should show up at
>>>
>>> http://docs.python.org/
>>>
>>> Not sure whether this should be 2.6 or 3.0.
>> Almost certainly 2.6.
> 
> Definitely 2.6 - 2.x is still going to be the "mainstream" Python
> version for a good while yet.
> 
> Although it may be worth tweaking the main index page for the 2.6 docs
> to include a cross-link to the 3.0 docs.
> 
Indeed. Having spent efforts to persuade the community that 3.0 isn't
intended to replace 2.6 for production use it would be a shame to have
docs.python.org counteract that.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From jcea at jcea.es  Mon Sep 22 19:47:39 2008
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 22 Sep 2008 19:47:39 +0200
Subject: [Python-Dev] Patch review needed
Message-ID: <48D7DA3B.4030408@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Sorry to bother you, but I would like a patch review, since this code
must be merged for 2.6.

http://bugs.python.org/issue3885

Thanks in advance.

- --
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.8 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iQCVAwUBSNfaK5lgi5GaxT1NAQJCOAP+IVtGPcjjzWT/AefvEzxPajRmqf0H+MVn
5TO0STwkWuJHyYdv/z4OtA9UD6NTQ3Z4gLPOc5ZwlvhTW1gd856IBhVEFy1JARoJ
ZERR7cp/WcdUSksHlC5qgTd2ZD1Y1VYcgCwTkL59D8R7I2wq752om6ZzhKs3SKC7
SG8kLYMlo9I=
=P9mB
-----END PGP SIGNATURE-----

From skip at pobox.com  Mon Sep 22 21:46:14 2008
From: skip at pobox.com (skip at pobox.com)
Date: Mon, 22 Sep 2008 14:46:14 -0500
Subject: [Python-Dev] Release blocker with no assignee (now me)
Message-ID: <18647.62982.763289.55574@montanaro-dyndns-org.local>

Can someone comment on this ticket?

    http://bugs.python.org/issue3666

It's marked as a release blocker for 3.0.  I added a patch then refined it
based on a comment by Christian Heimes.  It was unassigned.  I just took it
and will apply the patch if it gets a thumbs up from another developer.

Skip


From brenocon at gmail.com  Tue Sep 23 02:15:32 2008
From: brenocon at gmail.com (Brendan O'Connor)
Date: Mon, 22 Sep 2008 17:15:32 -0700
Subject: [Python-Dev] Status of dictnotes.txt?
Message-ID: <fba9be1d0809221715g6ff9baf3jb315ab4e5bab0e58@mail.gmail.com>

Hi everyone,

I was enjoying reading Raymond Hettinger's Objects/dictnotes.txt, and
was wondering, which (if any) of its suggestions implemented?  I see
most of it was written back in 2003.

Thanks,
Brendan

From josiah.carlson at gmail.com  Tue Sep 23 04:15:14 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Mon, 22 Sep 2008 19:15:14 -0700
Subject: [Python-Dev] Status of dictnotes.txt?
In-Reply-To: <fba9be1d0809221715g6ff9baf3jb315ab4e5bab0e58@mail.gmail.com>
References: <fba9be1d0809221715g6ff9baf3jb315ab4e5bab0e58@mail.gmail.com>
Message-ID: <e6511dbf0809221915m5d8ccdd6t1d81295a587d5f52@mail.gmail.com>

On Mon, Sep 22, 2008 at 5:15 PM, Brendan O'Connor <brenocon at gmail.com> wrote:
> Hi everyone,
>
> I was enjoying reading Raymond Hettinger's Objects/dictnotes.txt, and
> was wondering, which (if any) of its suggestions implemented?  I see
> most of it was written back in 2003.

If I remember correctly (and based on a skimming of dictnotes.txt),
those aren't suggestions, it is documentation about the actual
implementation and why those decisions were made.

 - Josiah

From eckhardt at satorlaser.com  Tue Sep 23 10:52:36 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Tue, 23 Sep 2008 10:52:36 +0200
Subject: [Python-Dev] Status of MS Windows CE port
Message-ID: <200809231052.36720.eckhardt@satorlaser.com>

Greetings!

I'm currently trying to assess the effort required for a CE port. I'm already 
aware of the project at http://pythonce.sourceforge.net, but I find it a 
waste of time to have two separate projects which then require syncing 
changes back and forth.

Questions:
- What are the feelings towards a port in the mainline? I'm aware that you are 
trying to get 2.6 and 3.0 out of the door and are probably not willing to 
risk such changes to the release code right now, but in general?
- Is anyone already working on this? I know that Brad Clements made an attempt 
some years ago but didn't get it integrated, are there any others? Maybe even 
any ongoing work?
- I see that you are using Subversion and Bazaar, would it be possible to get 
a feature branch for the CE port? I'm thinking about the effort going on at 
pythonce.sf.net and some further porting that I need for my specific 
platform. IMHO both would be better off in the same repository as your 
upstream sources, since merging changes back and forth is much easier that 
way.

Thanks!

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From brenocon at gmail.com  Tue Sep 23 19:34:09 2008
From: brenocon at gmail.com (Brendan O'Connor)
Date: Tue, 23 Sep 2008 10:34:09 -0700
Subject: [Python-Dev] Status of dictnotes.txt?
In-Reply-To: <e6511dbf0809221915m5d8ccdd6t1d81295a587d5f52@mail.gmail.com>
References: <fba9be1d0809221715g6ff9baf3jb315ab4e5bab0e58@mail.gmail.com>
	<e6511dbf0809221915m5d8ccdd6t1d81295a587d5f52@mail.gmail.com>
Message-ID: <fba9be1d0809231034v6ae1a25aj99df8e7becb57ae@mail.gmail.com>

> If I remember correctly (and based on a skimming of dictnotes.txt),
> those aren't suggestions, it is documentation about the actual
> implementation and why those decisions were made.

Well, some of the document is written as "if we did X, then we'd see
Y" so it's not always clear.  Some parts are clearly not implemented,
e.g. read-only freezing.  Some parts are implemented but slightly
differently -- e.g. small dicts are specialized to be faster, but the
document talks about specializing lookdict() and lookdict_string()
with linear search; however, as far as I can tell,
PyDictObject->ma_smalltable is used for small dictionaries, with the
standard hashing algorithm.

This is a little confusing because the ma_lookup() function pointer is
used as indirection; I was thinking that maybe there was a linear
search version, but actually it looks like it's just set to
lookdict_string() then switches to lookdict() if there's ever a
non-string key.  lookdict[_string]() always uses hashing.

Anyway, if anyone who knows about this or dictnotes.txt in general is
reading, feel free to pipe in :)

Brendan

From skip at pobox.com  Tue Sep 23 22:00:28 2008
From: skip at pobox.com (skip at pobox.com)
Date: Tue, 23 Sep 2008 15:00:28 -0500
Subject: [Python-Dev] okay to add new test cases to 2.6?
Message-ID: <18649.19164.853803.31148@montanaro-dyndns-org.local>

Benjamin Peterson wound up writing a test case for the new C atexit module
on the py3k branch.  A similar test, though different in detail, makes sense
for the Python atexit module on trunk.  Is this something I can check in or
should I just wait until after 2.6 is released?

    def test_badargs(self):
        s = StringIO.StringIO()
        sys.stdout = sys.stderr = s
        save_handlers = atexit._exithandlers
        atexit._exithandlers = []
        try:
            atexit.register(lambda: 1, 0, 0, (x for x in (1,2)), 0, 0)
            self.assertRaises(TypeError, atexit._run_exitfuncs)
        finally:
            sys.stdout = sys.__stdout__
            sys.stderr = sys.__stderr__
            atexit._exithandlers = save_handlers

Thx,

Skip

From barry at python.org  Tue Sep 23 22:31:25 2008
From: barry at python.org (Barry Warsaw)
Date: Tue, 23 Sep 2008 16:31:25 -0400
Subject: [Python-Dev] okay to add new test cases to 2.6?
In-Reply-To: <18649.19164.853803.31148@montanaro-dyndns-org.local>
References: <18649.19164.853803.31148@montanaro-dyndns-org.local>
Message-ID: <8DD625E3-043F-4582-9496-9B971447E9F8@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 23, 2008, at 4:00 PM, skip at pobox.com wrote:

> Benjamin Peterson wound up writing a test case for the new C atexit  
> module
> on the py3k branch.  A similar test, though different in detail,  
> makes sense
> for the Python atexit module on trunk.  Is this something I can  
> check in or
> should I just wait until after 2.6 is released?
>
>    def test_badargs(self):
>        s = StringIO.StringIO()
>        sys.stdout = sys.stderr = s
>        save_handlers = atexit._exithandlers
>        atexit._exithandlers = []
>        try:
>            atexit.register(lambda: 1, 0, 0, (x for x in (1,2)), 0, 0)
>            self.assertRaises(TypeError, atexit._run_exitfuncs)
>        finally:
>            sys.stdout = sys.__stdout__
>            sys.stderr = sys.__stderr__
>            atexit._exithandlers = save_handlers

If you don't increase the redness of the buildbots, I think it's fine.

- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSNlSHXEjvBPtnXfVAQKQ3gP/XJr97CdjZitwSQxrL28xCuLyPfKmtBVA
2hK2ewQxH9Rxrl0BsFZlnBz2vEygPmPWnduu8VT+sjsrFu/Ua9d1Xo06lQ6/fFck
Tp/en0KlAyJ6Pjf/1pWahn4ttwk74YG02LiQgUsfOaIUG+8cMfGfrtyUzwGBoKhW
4RBuwpPjsBo=
=7Brm
-----END PGP SIGNATURE-----

From nnorwitz at gmail.com  Wed Sep 24 06:41:53 2008
From: nnorwitz at gmail.com (Neal Norwitz)
Date: Tue, 23 Sep 2008 21:41:53 -0700
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <200809231052.36720.eckhardt@satorlaser.com>
References: <200809231052.36720.eckhardt@satorlaser.com>
Message-ID: <ee2a432c0809232141t68e1b44fx6750a5451d7e2881@mail.gmail.com>

On Tue, Sep 23, 2008 at 1:52 AM, Ulrich Eckhardt
<eckhardt at satorlaser.com> wrote:
> Greetings!
>
> I'm currently trying to assess the effort required for a CE port. I'm already
> aware of the project at http://pythonce.sourceforge.net, but I find it a
> waste of time to have two separate projects which then require syncing
> changes back and forth.
>
> Questions:
> - What are the feelings towards a port in the mainline? I'm aware that you are
> trying to get 2.6 and 3.0 out of the door and are probably not willing to
> risk such changes to the release code right now, but in general?

In general, we try to ensure that there is an active maintainer,
preferably more than one.  As long as it doesn't make the code that
much harder to maintain, it is desirable to support more platforms.

> - Is anyone already working on this? I know that Brad Clements made an attempt
> some years ago but didn't get it integrated, are there any others? Maybe even
> any ongoing work?

I'm not sure if anyone here knows.  I haven't seen anything on this
list, but that doesn't mean there isn't another list either on
python.org or elsewhere that does have the information.  Try searching
if you haven't.

> - I see that you are using Subversion and Bazaar, would it be possible to get
> a feature branch for the CE port? I'm thinking about the effort going on at
> pythonce.sf.net and some further porting that I need for my specific
> platform. IMHO both would be better off in the same repository as your
> upstream sources, since merging changes back and forth is much easier that
> way.

It's possible.  The best way to get the ball rolling is to produce
some patches against the mainline.  Demonstrate what you can do and
privileges will follow.  It's a similar process to becoming a
python-committer.

Perhaps the best way to start is using Bazaar or some distributed
version control system for your patches.  Once you've demonstrated
your ability and intent, we can make a branch.

Cheers,
n

From bob at redivi.com  Wed Sep 24 06:51:18 2008
From: bob at redivi.com (Bob Ippolito)
Date: Tue, 23 Sep 2008 21:51:18 -0700
Subject: [Python-Dev] json decoder speedups, any time left for 2.6?
Message-ID: <6a36e7290809232151k21824aabw62293dc871dac098@mail.gmail.com>

I'm out of town this week for a conference (ICFP/CUFP in Victoria) and
my hotel's connection has been bad enough such that I can't get any
Real Work done so I've managed to hammer on the json library's
decoding quite a bit instead. I just released simplejson 1.9.3 which
improves decoding performance by about 2x and I've got some more
changes along the way in trunk for 1.9.4 that will increase it even
further (over 3x my original 1.9.2 benchmark perf). How much time do I
have left to get this into Python 2.6?

FWIW the changes are all on the Python side, no C code has been harmed
(yet). The test suite still passes of course.

-bob

From ncoghlan at gmail.com  Wed Sep 24 11:47:07 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 Sep 2008 19:47:07 +1000
Subject: [Python-Dev] json decoder speedups, any time left for 2.6?
In-Reply-To: <6a36e7290809232151k21824aabw62293dc871dac098@mail.gmail.com>
References: <6a36e7290809232151k21824aabw62293dc871dac098@mail.gmail.com>
Message-ID: <48DA0C9B.6030707@gmail.com>

Bob Ippolito wrote:
> How much time do I
> have left to get this into Python 2.6?

Zero I'm afraid - with rc1 out, it's release blocker bugs only. Anything
which can be deferred to the 2.6.1 release without causing any major
harm is definitely out - and while a 2x speedup is very nice, it isn't
something to be squeezing in post-rc1.

Still, that should make for a nice incremental improvement when 2.6.1
rolls around.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From barry at python.org  Wed Sep 24 15:14:36 2008
From: barry at python.org (Barry Warsaw)
Date: Wed, 24 Sep 2008 09:14:36 -0400
Subject: [Python-Dev] json decoder speedups, any time left for 2.6?
In-Reply-To: <48DA0C9B.6030707@gmail.com>
References: <6a36e7290809232151k21824aabw62293dc871dac098@mail.gmail.com>
	<48DA0C9B.6030707@gmail.com>
Message-ID: <F4A9FA8C-9020-4B19-8AEA-D6F60A32D505@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Sep 24, 2008, at 5:47 AM, Nick Coghlan wrote:

> Bob Ippolito wrote:
>> How much time do I
>> have left to get this into Python 2.6?
>
> Zero I'm afraid - with rc1 out, it's release blocker bugs only.  
> Anything
> which can be deferred to the 2.6.1 release without causing any major
> harm is definitely out - and while a 2x speedup is very nice, it isn't
> something to be squeezing in post-rc1.
>
> Still, that should make for a nice incremental improvement when 2.6.1
> rolls around.

I concur.
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSNo9PHEjvBPtnXfVAQKJZQP/dGcmjO7P9RKIcYsFAwmDdVlCfn2WiGQt
WlNgJiIq5tnq4WSyRQMNeDAWZfheq/vHqlDMU6OunIz5jcv0cVNmKkfcW+pTl9Yc
7Ur6mejeAUYrg2jHkimhMv0wpX/lJ+7CrWD1c+3PK+tQD0GgWfen+/xzCkGqlbc8
ompjNE1WPPs=
=Jm8s
-----END PGP SIGNATURE-----

From bob at redivi.com  Wed Sep 24 17:08:43 2008
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 24 Sep 2008 08:08:43 -0700
Subject: [Python-Dev] json decoder speedups, any time left for 2.6?
In-Reply-To: <F4A9FA8C-9020-4B19-8AEA-D6F60A32D505@python.org>
References: <6a36e7290809232151k21824aabw62293dc871dac098@mail.gmail.com>
	<48DA0C9B.6030707@gmail.com>
	<F4A9FA8C-9020-4B19-8AEA-D6F60A32D505@python.org>
Message-ID: <6a36e7290809240808y547587dcj94b339a5d6ed987b@mail.gmail.com>

On Wed, Sep 24, 2008 at 6:14 AM, Barry Warsaw <barry at python.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On Sep 24, 2008, at 5:47 AM, Nick Coghlan wrote:
>
>> Bob Ippolito wrote:
>>>
>>> How much time do I
>>> have left to get this into Python 2.6?
>>
>> Zero I'm afraid - with rc1 out, it's release blocker bugs only. Anything
>> which can be deferred to the 2.6.1 release without causing any major
>> harm is definitely out - and while a 2x speedup is very nice, it isn't
>> something to be squeezing in post-rc1.
>>
>> Still, that should make for a nice incremental improvement when 2.6.1
>> rolls around.
>
> I concur.

Ok, no problem. The speedup is about 3x now on the trunk ;) I think
that further optimization will require some more C hacking, but 2.6.1
should give me plenty of time to get around to some of that.

-bob

From aleaxit at gmail.com  Wed Sep 24 17:42:36 2008
From: aleaxit at gmail.com (Alex Martelli)
Date: Wed, 24 Sep 2008 08:42:36 -0700
Subject: [Python-Dev] json decoder speedups, any time left for 2.6?
In-Reply-To: <6a36e7290809240808y547587dcj94b339a5d6ed987b@mail.gmail.com>
References: <6a36e7290809232151k21824aabw62293dc871dac098@mail.gmail.com>
	<48DA0C9B.6030707@gmail.com>
	<F4A9FA8C-9020-4B19-8AEA-D6F60A32D505@python.org>
	<6a36e7290809240808y547587dcj94b339a5d6ed987b@mail.gmail.com>
Message-ID: <e8a0972d0809240842j248528d1x5279464199def385@mail.gmail.com>

Meanwhile, can you please release (wherever you normally release
things;-) the pure-Python version as well?  I'd like to play around
with it in Google App Engine opensource sandboxes (e.g., cfr.
gae-json-rest -- I'll be delighted to add you to that project if you
want of course;-) and that requires Python 2.5 and only pure-Python
add-ons... thanks!

Alex


On Wed, Sep 24, 2008 at 8:08 AM, Bob Ippolito <bob at redivi.com> wrote:
> On Wed, Sep 24, 2008 at 6:14 AM, Barry Warsaw <barry at python.org> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On Sep 24, 2008, at 5:47 AM, Nick Coghlan wrote:
>>
>>> Bob Ippolito wrote:
>>>>
>>>> How much time do I
>>>> have left to get this into Python 2.6?
>>>
>>> Zero I'm afraid - with rc1 out, it's release blocker bugs only. Anything
>>> which can be deferred to the 2.6.1 release without causing any major
>>> harm is definitely out - and while a 2x speedup is very nice, it isn't
>>> something to be squeezing in post-rc1.
>>>
>>> Still, that should make for a nice incremental improvement when 2.6.1
>>> rolls around.
>>
>> I concur.
>
> Ok, no problem. The speedup is about 3x now on the trunk ;) I think
> that further optimization will require some more C hacking, but 2.6.1
> should give me plenty of time to get around to some of that.
>
> -bob
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com
>

From bob at redivi.com  Wed Sep 24 18:02:10 2008
From: bob at redivi.com (Bob Ippolito)
Date: Wed, 24 Sep 2008 09:02:10 -0700
Subject: [Python-Dev] json decoder speedups, any time left for 2.6?
In-Reply-To: <e8a0972d0809240842j248528d1x5279464199def385@mail.gmail.com>
References: <6a36e7290809232151k21824aabw62293dc871dac098@mail.gmail.com>
	<48DA0C9B.6030707@gmail.com>
	<F4A9FA8C-9020-4B19-8AEA-D6F60A32D505@python.org>
	<6a36e7290809240808y547587dcj94b339a5d6ed987b@mail.gmail.com>
	<e8a0972d0809240842j248528d1x5279464199def385@mail.gmail.com>
Message-ID: <6a36e7290809240902n2acf8aafqdd77b1c46cd9961@mail.gmail.com>

http://pypi.python.org/pypi/simplejson

The _speedups module is optional.

On Wed, Sep 24, 2008 at 8:42 AM, Alex Martelli <aleaxit at gmail.com> wrote:
> Meanwhile, can you please release (wherever you normally release
> things;-) the pure-Python version as well?  I'd like to play around
> with it in Google App Engine opensource sandboxes (e.g., cfr.
> gae-json-rest -- I'll be delighted to add you to that project if you
> want of course;-) and that requires Python 2.5 and only pure-Python
> add-ons... thanks!
>
> Alex
>
>
> On Wed, Sep 24, 2008 at 8:08 AM, Bob Ippolito <bob at redivi.com> wrote:
>> On Wed, Sep 24, 2008 at 6:14 AM, Barry Warsaw <barry at python.org> wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA1
>>>
>>> On Sep 24, 2008, at 5:47 AM, Nick Coghlan wrote:
>>>
>>>> Bob Ippolito wrote:
>>>>>
>>>>> How much time do I
>>>>> have left to get this into Python 2.6?
>>>>
>>>> Zero I'm afraid - with rc1 out, it's release blocker bugs only. Anything
>>>> which can be deferred to the 2.6.1 release without causing any major
>>>> harm is definitely out - and while a 2x speedup is very nice, it isn't
>>>> something to be squeezing in post-rc1.
>>>>
>>>> Still, that should make for a nice incremental improvement when 2.6.1
>>>> rolls around.
>>>
>>> I concur.
>>
>> Ok, no problem. The speedup is about 3x now on the trunk ;) I think
>> that further optimization will require some more C hacking, but 2.6.1
>> should give me plenty of time to get around to some of that.
>>
>> -bob
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com
>>
>

From jason.orendorff at gmail.com  Thu Sep 25 01:13:25 2008
From: jason.orendorff at gmail.com (Jason Orendorff)
Date: Wed, 24 Sep 2008 18:13:25 -0500
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <ca471dc20809142112j6a930a85o5e7bd621182e80b7@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<48CDBA02.9040802@canterbury.ac.nz> <48CDD859.90407@trueblade.com>
	<cc7430500809142055y462d6e88v86a4e9938548042b@mail.gmail.com>
	<ca471dc20809142112j6a930a85o5e7bd621182e80b7@mail.gmail.com>
Message-ID: <bb8868b90809241613m3421f4e0nba03b4c54bda6643@mail.gmail.com>

What I really want is for the need to be less common.  What if assert
recognized certain commonly used expression types and actually
generated appropriate error messages?

  >>> assert foo.answer == 42
  AssertionError: expected foo.answer == 42; actual: 'a suffusion of yellow'

Maybe that's too magical.  :(  Failing that, I wish the message could
look sort of like a comment.

    assert cond || message

Yes, I know that symbol is used for something else in other languages...

-j

From eldonz at atlanticdb.com  Thu Sep 25 12:23:52 2008
From: eldonz at atlanticdb.com (Eldon Ziegler)
Date: Thu, 25 Sep 2008 06:23:52 -0400
Subject: [Python-Dev] Update to httplib.py
Message-ID: <1222338232.3594.10.camel@localhost6.localdomain6>

I updated httplib.py, python 2.4, to be able to bind to a specific IP
address when connecting to a remote site. Would there be any interest in
making this available to others? If so, are there instructions on how to
post an update?

Eldon Ziegler



From eric at trueblade.com  Thu Sep 25 12:54:28 2008
From: eric at trueblade.com (Eric Smith)
Date: Thu, 25 Sep 2008 06:54:28 -0400
Subject: [Python-Dev] Update to httplib.py
In-Reply-To: <1222338232.3594.10.camel@localhost6.localdomain6>
References: <1222338232.3594.10.camel@localhost6.localdomain6>
Message-ID: <48DB6DE4.7020605@trueblade.com>

Eldon Ziegler wrote:
> I updated httplib.py, python 2.4, to be able to bind to a specific IP
> address when connecting to a remote site. Would there be any interest in
> making this available to others? If so, are there instructions on how to
> post an update?

Create an issue at http://bugs.python.org/, mark it as a feature 
request, and attach the patch. That is, assuming this feature is still 
missing in 2.6 and 3.0. I don't know enough about it to say.

Eric.

From eckhardt at satorlaser.com  Thu Sep 25 13:47:24 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Thu, 25 Sep 2008 13:47:24 +0200
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <ee2a432c0809232141t68e1b44fx6750a5451d7e2881@mail.gmail.com>
References: <200809231052.36720.eckhardt@satorlaser.com> 
	<ee2a432c0809232141t68e1b44fx6750a5451d7e2881@mail.gmail.com>
Message-ID: <200809251347.24163.eckhardt@satorlaser.com>

On Wednesday 24 September 2008, Neal Norwitz wrote:
> In general, we try to ensure that there is an active maintainer,
> preferably more than one.  As long as it doesn't make the code that
> much harder to maintain, it is desirable to support more platforms.

Actually, looking at the sources, there are several places where even existing 
win32 ports would benefit. There are four main differences between CE and the 
desktop variants:
1. TCHAR is always WCHAR
2. some APIs are missing
3. the vendor-supplied C or C++ stdlibs are rather minimal
4. you always cross-compile

The first point is interesting to the desktop win32 variants because 
everything since NT4 (or even 3.x) also used UCS2 or UTF-16 internally. Only 
win9x and ME(?) use 8-bit chars, and since support for them is being dropped, 
this would save a few conversions while allowing better Unicode support. I 
think getting Python to compile with _UNICODE #defined will be my first step.

The second point is mainly about features that don't exist like a current 
directory for a process or environment variables. Some other APIs don't 
support all parameters or similar differences. This should be rather easy to 
do.

The third means that functions like e.g. system() are simply not available in 
the supplied C API. If the functionality itself exists, it can be built on 
top of the win32 API. There might be a few cases where the desktop variants 
would benefit from one less adaption layer, too.

The last one only affects the build system and possibly the integration of 
unit testing. pythonce.sf.net already have one based on SCons, which is nice 
and easily extended. Being able to build 64 bit variants on 32 bit systems 
and vice versa might be another benefit.

> It's possible.  The best way to get the ball rolling is to produce
> some patches against the mainline.

Hmmm, which mainline? 2.5, 2.6 or 3.0? Up to now I have only looked at 2.5, 
because that one is stable. However, quite a few of above changes are not 
suitable for a stable version though, so I'll either have to restrict them to 
only CE or rather apply them to version 3. We can discuss that when I have 
actual patches.

> Perhaps the best way to start is using Bazaar or some distributed
> version control system for your patches.  Once you've demonstrated
> your ability and intent, we can make a branch.

I'll be digging into Bazaar, looks like an interesting topic by itself anyway. 
You'll hear from me!

cheers

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From janssen at parc.com  Thu Sep 25 18:00:56 2008
From: janssen at parc.com (Bill Janssen)
Date: Thu, 25 Sep 2008 09:00:56 PDT
Subject: [Python-Dev] Update to httplib.py
In-Reply-To: <1222338232.3594.10.camel@localhost6.localdomain6>
References: <1222338232.3594.10.camel@localhost6.localdomain6>
Message-ID: <6803.1222358456@parc.com>

Eldon Ziegler <eldonz at atlanticdb.com> wrote:

> I updated httplib.py, python 2.4, to be able to bind to a specific IP
> address when connecting to a remote site.

Would that be something like

  conn = httplib.HTTPConnection('82.94.237.218', 80)

Doesn't it already do that?

Bill

From phd at phd.pp.ru  Thu Sep 25 18:04:50 2008
From: phd at phd.pp.ru (Oleg Broytmann)
Date: Thu, 25 Sep 2008 20:04:50 +0400
Subject: [Python-Dev] Update to httplib.py
In-Reply-To: <6803.1222358456@parc.com>
References: <1222338232.3594.10.camel@localhost6.localdomain6>
	<6803.1222358456@parc.com>
Message-ID: <20080925160450.GD2454@phd.pp.ru>

On Thu, Sep 25, 2008 at 09:00:56AM -0700, Bill Janssen wrote:
> Eldon Ziegler <eldonz at atlanticdb.com> wrote:
> > I updated httplib.py, python 2.4, to be able to bind to a specific IP
> > address when connecting to a remote site.
> 
> Would that be something like
> 
>   conn = httplib.HTTPConnection('82.94.237.218', 80)
> 
> Doesn't it already do that?

   It's the destination address. AFAIU OP said about source address - in
case the program runs on a computer with many addresses - multihomed server
or such...

Oleg.
-- 
     Oleg Broytmann            http://phd.pp.ru/            phd at phd.pp.ru
           Programmers don't die, they just GOSUB without RETURN.

From martin at v.loewis.de  Thu Sep 25 22:53:19 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 25 Sep 2008 22:53:19 +0200
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <200809231052.36720.eckhardt@satorlaser.com>
References: <200809231052.36720.eckhardt@satorlaser.com>
Message-ID: <48DBFA3F.1040807@v.loewis.de>

> - Is anyone already working on this? I know that Brad Clements made an attempt 
> some years ago but didn't get it integrated, are there any others? Maybe even 
> any ongoing work?

My understanding is that Python works fairly well on CE as-is, so no
significant patches are needed.

If that understanding is incorrect, it may be because it broke over
time, or it may be that my understanding was incorrect to begin with.

Regards,
Martin

From martin at v.loewis.de  Thu Sep 25 23:01:16 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Thu, 25 Sep 2008 23:01:16 +0200
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <200809251347.24163.eckhardt@satorlaser.com>
References: <200809231052.36720.eckhardt@satorlaser.com>
	<ee2a432c0809232141t68e1b44fx6750a5451d7e2881@mail.gmail.com>
	<200809251347.24163.eckhardt@satorlaser.com>
Message-ID: <48DBFC1C.3080008@v.loewis.de>

> 1. TCHAR is always WCHAR

Python shouldn't be using TCHAR at all; that getpathp uses it is
misguided (IMO).

> The first point is interesting to the desktop win32 variants because 
> everything since NT4 (or even 3.x) also used UCS2 or UTF-16 internally. Only 
> win9x and ME(?) use 8-bit chars, and since support for them is being dropped, 
> this would save a few conversions while allowing better Unicode support. I 
> think getting Python to compile with _UNICODE #defined will be my first step.

Please don't. Whether or not _UNICODE is defined should have no effect.

> The third means that functions like e.g. system() are simply not available in 
> the supplied C API.

In this case, not defining HAVE_SYSTEM would be already sufficient, no?

> If the functionality itself exists, it can be built on 
> top of the win32 API. There might be a few cases where the desktop variants 
> would benefit from one less adaption layer, too.

Again, in the specific case of system(), the replacement is already
there, in form of the subprocess module? There might be other places
also, but apart from stdio (which is mostly gone in 3.0) and memory
management, I can't think of any.

> The last one only affects the build system and possibly the integration of 
> unit testing. pythonce.sf.net already have one based on SCons, which is nice 
> and easily extended. Being able to build 64 bit variants on 32 bit systems 
> and vice versa might be another benefit.

Wouldn't it be easier to extend the VS projects? VS targets CE fairly
well, no?

>> It's possible.  The best way to get the ball rolling is to produce
>> some patches against the mainline.
> 
> Hmmm, which mainline? 2.5, 2.6 or 3.0?

The mainline is 2.6 (i.e. trunk). 2.5 and 3.0 are branches.

> Up to now I have only looked at 2.5, 
> because that one is stable. However, quite a few of above changes are not 
> suitable for a stable version though, so I'll either have to restrict them to 
> only CE or rather apply them to version 3. We can discuss that when I have 
> actual patches.

I guess none of your changes are suitable for 2.5 (as they probably
constitute new features). You should really target trunk (i.e. 2.7) when
you produce patches.

Regards,
Martin


From fuzzyman at voidspace.org.uk  Thu Sep 25 23:21:07 2008
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Thu, 25 Sep 2008 22:21:07 +0100
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <48DBFA3F.1040807@v.loewis.de>
References: <200809231052.36720.eckhardt@satorlaser.com>
	<48DBFA3F.1040807@v.loewis.de>
Message-ID: <48DC00C3.40409@voidspace.org.uk>

Martin v. L?wis wrote:
>> - Is anyone already working on this? I know that Brad Clements made an attempt 
>> some years ago but didn't get it integrated, are there any others? Maybe even 
>> any ongoing work?
>>     
>
> My understanding is that Python works fairly well on CE as-is, so no
> significant patches are needed.
>
> If that understanding is incorrect, it may be because it broke over
> time, or it may be that my understanding was incorrect to begin with.
>   

I used to follow the Python-CE list. I believe that Luke Dunstan (who 
has done the most recent work on a port) discussed a previous attempt to 
get the patch-set accepted into trunk but it was rejected because it 
required too *many* changes.

Michael

> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
>   


-- 
http://www.ironpythoninaction.com/
http://www.voidspace.org.uk/
http://www.trypython.org/
http://www.ironpython.info/
http://www.theotherdelia.co.uk/
http://www.resolverhacks.net/


From martin at v.loewis.de  Fri Sep 26 07:13:12 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 26 Sep 2008 07:13:12 +0200
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <48DC00C3.40409@voidspace.org.uk>
References: <200809231052.36720.eckhardt@satorlaser.com>
	<48DBFA3F.1040807@v.loewis.de> <48DC00C3.40409@voidspace.org.uk>
Message-ID: <48DC6F68.8080407@v.loewis.de>

> I used to follow the Python-CE list. I believe that Luke Dunstan (who
> has done the most recent work on a port) discussed a previous attempt to
> get the patch-set accepted into trunk but it was rejected because it
> required too *many* changes.

Perhaps. But I also recall committing patches that
suggested a number of changes necessary for CE (e.g. r46819,
from #1495999 by Lukas Dunstan, and r46064, from #1492356,
by the same author). I don't recall rejecting a patch from
him, but I may have forgotten, or somebody else may have
rejected such a patch.

I do recall a certain unhappiness with the patches. If you
look at http://bugs.python.org/issue1492356, it starts with
"This patch contains part of the changes". What would be
the rationale for including part of the changes, but not all
of them? This appears to be a widespread view of port maintainers:
let's try to get some changes in. Apparently, some believe that
suggesting fewer changes might be more successful. I feel
uncomfortable with that, because I then don't know how many
more times they will approach me with another collection
of a few changes, and whether learning what the later changes
are might have impact on the earlier changes (which I had
already accepted).

Regards,
Martin

From eckhardt at satorlaser.com  Fri Sep 26 09:21:35 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Fri, 26 Sep 2008 09:21:35 +0200
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <48DBFC1C.3080008@v.loewis.de>
References: <200809231052.36720.eckhardt@satorlaser.com> 
	<200809251347.24163.eckhardt@satorlaser.com> 
	<48DBFC1C.3080008@v.loewis.de>
Message-ID: <200809260921.36012.eckhardt@satorlaser.com>

On Thursday 25 September 2008, Martin v. L?wis wrote:
> > 1. TCHAR is always WCHAR
>
> Python shouldn't be using TCHAR at all; that getpathp uses it is
> misguided (IMO).

It already does, see below...

> > The first point is interesting to the desktop win32 variants because
> > everything since NT4 (or even 3.x) also used UCS2 or UTF-16 internally.
> > Only win9x and ME(?) use 8-bit chars, and since support for them is being
> > dropped, this would save a few conversions while allowing better Unicode
> > support. I think getting Python to compile with _UNICODE #defined will be
> > my first step.
>
> Please don't. Whether or not _UNICODE is defined should have no effect.

Well, currently it does make a difference. Simple example: CreateFile(). One 
of the parameters is actually documented at LPCTSTR, which is a TCHAR string. 
That means that under CE, CreateFile(), which is actually a macro, resolves 
to CreateFileW and the existing code that passes a char-string doesn't work.

> > The third means that functions like e.g. system() are simply not
> > available in the supplied C API.
>
> In this case, not defining HAVE_SYSTEM would be already sufficient, no?

I'll take a look, I actually don't know.


> > The last one only affects the build system and possibly the integration
> > of unit testing. pythonce.sf.net already have one based on SCons, which
> > is nice and easily extended. Being able to build 64 bit variants on 32
> > bit systems and vice versa might be another benefit.
>
> Wouldn't it be easier to extend the VS projects? VS targets CE fairly
> well, no?

VS2005 works really good with CE5 and later. My problem here is that I can't 
provide a projectfile that suits everyone, because different devices have 
different SDKs and each configuration/SDK/CPU is another entry in that file. 
Further, the SDKs or devices differ in what they support, the platform CE as 
such doesn't exist.

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From status at bugs.python.org  Fri Sep 26 18:06:35 2008
From: status at bugs.python.org (Python tracker)
Date: Fri, 26 Sep 2008 18:06:35 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20080926160635.1403F78525@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (09/19/08 - 09/26/08)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue 
number.  Do NOT respond to this message.


 2052 open (+35) / 13746 closed (+32) / 15798 total (+67)

Open issues with patches:   660

Average duration of open issues: 713 days.
Median duration of open issues: 1818 days.

Open Issues Breakdown
   open  2037 (+34)
pending    15 ( +1)

Issues Created Or Reopened (69)
_______________________________

turtle in the tkinter package?                                   09/20/08
CLOSED http://bugs.python.org/issue3884    reopened loewis                    
       patch                                                                   

subprocess failing in GUI applications on Windows                09/21/08
       http://bugs.python.org/issue3905    reopened georg.brandl              
                                                                               

Building PDF documentation from tex files                        09/19/08
       http://bugs.python.org/issue3909    created  wplappert                 
                                                                               

2.6 regression in socket.ssl method                              09/19/08
       http://bugs.python.org/issue3910    created  matejcik                  
       patch                                                                   

ftplib.FTP.makeport() bug                                        09/19/08
       http://bugs.python.org/issue3911    created  giampaolo.rodola          
       patch                                                                   

unittest. assertAlmostEqual() documentation incomplete           09/20/08
CLOSED http://bugs.python.org/issue3912    created  roysmith                  
       patch                                                                   

compound_stmt syntax includes 'decorated'                        09/20/08
CLOSED http://bugs.python.org/issue3913    created  dangyogi                  
                                                                               

augop definition does not include "//="                          09/20/08
CLOSED http://bugs.python.org/issue3914    created  dangyogi                  
                                                                               

Broken link in 14.7 curses, Python Library Reference             09/20/08
CLOSED http://bugs.python.org/issue3915    created  jasoneth                  
                                                                               

layout of build directories for Windows not current              09/20/08
CLOSED http://bugs.python.org/issue3916    created  gagenellina               
       patch                                                                   

set_display definition allows empty '{' '}' in Language Definiti 09/21/08
CLOSED http://bugs.python.org/issue3917    created  dangyogi                  
                                                                               

random.uniform suprisingly (good kind) does not work as document 09/21/08
CLOSED http://bugs.python.org/issue3918    created  xuinkrbin                 
                                                                               

PySys_SetObject crashes after Py_NewInterpreter().               09/21/08
CLOSED http://bugs.python.org/issue3919    created  grahamd                   
                                                                               

OpenBSD 4.4 still doesn't support _XOPEN_SOURCE                  09/21/08
       http://bugs.python.org/issue3920    created  loewis                    
       patch, needs review                                                     

smtplib cannot sendmail over TLS                                 09/21/08
       http://bugs.python.org/issue3921    created  shidot                    
                                                                               

3.0rc1 missing tk lib in sys.path?                               09/21/08
CLOSED http://bugs.python.org/issue3922    created  jmfauth                   
                                                                               

'genexpr_for' in definition of 'call' in Language Reference.     09/21/08
CLOSED http://bugs.python.org/issue3923    created  dangyogi                  
                                                                               

cookielib chokes on non-integer cookie version, should ignore it 09/21/08
       http://bugs.python.org/issue3924    created  DenNukem                  
                                                                               

test_distutils fails on cygwin                                   09/21/08
CLOSED http://bugs.python.org/issue3925    created  ocean-city                
       patch, easy                                                             

Idle doesn't obey the new improved warnings arguements           09/21/08
       http://bugs.python.org/issue3926    created  scott_daniels             
                                                                               

dummy multiprocessing needs to use properties                    09/22/08
CLOSED http://bugs.python.org/issue3927    created  benjamin.peterson         
       patch                                                                   

os.mknod missing on Solaris                                      09/22/08
       http://bugs.python.org/issue3928    created  sandberg                  
                                                                               

Incorrect exception raising in dbm.open on non-existing DB       09/22/08
CLOSED http://bugs.python.org/issue3929    created  hagen                     
       patch, needs review                                                     

urllib.request.urlopen() different on Windows than Linux         09/22/08
CLOSED http://bugs.python.org/issue3930    created  mark                      
                                                                               

codecs.charmap_build is untested and undocumented                09/22/08
       http://bugs.python.org/issue3931    created  fijal                     
                                                                               

HTMLParser cannot handle '&' and non-ascii characters in attribu 09/22/08
       http://bugs.python.org/issue3932    created  yanne                     
                                                                               

presence of .pythonstartup.py causes __main__.__file__ to be set 09/22/08
CLOSED http://bugs.python.org/issue3933    created  exarkun                   
                                                                               

sphinx - building muppy docs fails on Linux                      09/22/08
CLOSED http://bugs.python.org/issue3934    created  robwolfe                  
       patch                                                                   

bisect insort C implementation ignores methods on list subclasse 09/22/08
       http://bugs.python.org/issue3935    created  jek                       
                                                                               

Faulty suppression of 'as' keyword warning                       09/22/08
CLOSED http://bugs.python.org/issue3936    created  tjreedy                   
       patch                                                                   

platform.dist(): detect Linux distribution version in a robust,  09/22/08
CLOSED http://bugs.python.org/issue3937    created  zooko                     
       patch                                                                   

Clearing globals; interpreter -- IDLE difference                 09/22/08
       http://bugs.python.org/issue3938    created  tjreedy                   
                                                                               

Patch to implement a real ftplib test suite                      09/22/08
       http://bugs.python.org/issue3939    created  giampaolo.rodola          
       patch                                                                   

turtle.py - minor bugfixes                                       09/23/08
CLOSED http://bugs.python.org/issue3940    created  gregorlingl               
       patch                                                                   

Help in IDLE should open the chm file                            09/23/08
       http://bugs.python.org/issue3941    created  gregorlingl               
                                                                               

Usability issue from not being able to use defined start and end 09/23/08
CLOSED http://bugs.python.org/issue3942    created  Mez                       
                                                                               

IDLE won't start in 3.0rc1 "Subprocess didn't make connection... 09/23/08
CLOSED http://bugs.python.org/issue3943    created  rbtyod                    
                                                                               

faster long multiplication                                       09/23/08
       http://bugs.python.org/issue3944    created  pernici                   
       patch                                                                   

compile error in _fileio.c (cygwin)                              09/23/08
CLOSED http://bugs.python.org/issue3945    created  ocean-city                
       patch, easy                                                             

PyObject_CheckReadBuffer crashes on memoryview object            09/23/08
       http://bugs.python.org/issue3946    created  rupole                    
       patch, needs review                                                     

configure --with-threads on cygwin => crash on thread related te 09/23/08
       http://bugs.python.org/issue3947    created  ocean-city                
                                                                               

readline steals sigwinch                                         09/23/08
       http://bugs.python.org/issue3948    created  shish                     
                                                                               

curses' sigwinch handler isn't visible from python               09/23/08
       http://bugs.python.org/issue3949    created  shish                     
                                                                               

turtle.py: bug in TurtleScreenBase._drawimage                    09/23/08
CLOSED http://bugs.python.org/issue3950    created  gregorlingl               
       patch                                                                   

Disable Py_USING_MEMORY_DEBUGGER!                                09/24/08
       http://bugs.python.org/issue3951    created  haypo                     
       patch                                                                   

_lsprof: clear() should call flush_unmatched()                   09/24/08
       http://bugs.python.org/issue3952    created  haypo                     
       patch                                                                   

__cmp__ still documented in 3.0rc1?                              09/24/08
CLOSED http://bugs.python.org/issue3953    created  hniksic                   
                                                                               

_hotshot: invalid error control in logreader()                   09/24/08
       http://bugs.python.org/issue3954    created  haypo                     
       patch                                                                   

maybe doctest doesn't understand unicode_literals?               09/24/08
       http://bugs.python.org/issue3955    created  mark                      
                                                                               

turtle.py - bug in Screen.__init__()                             09/24/08
       http://bugs.python.org/issue3956    created  gregorlingl               
       patch                                                                   

[contextlib] Reverse order of locking                            09/24/08
       http://bugs.python.org/issue3957    created  DragonSA                  
                                                                               

strage default value behaviour                                   09/24/08
CLOSED http://bugs.python.org/issue3958    created  dominik.holler            
                                                                               

Add Google's ipaddr.py to the stdlib                             09/24/08
       http://bugs.python.org/issue3959    created  gvanrossum                
                                                                               

Section permalink html anchors are wrong                         09/25/08
CLOSED http://bugs.python.org/issue3960    created  ronny                     
                                                                               

Arrows key do not browse in the IDLE                             09/25/08
       http://bugs.python.org/issue3961    created  italian-boy               
                                                                               

single architecture framework build fails on OS X 10.5           09/25/08
       http://bugs.python.org/issue3962    created  cekees                    
                                                                               

Problems when calling exec from a function                       09/25/08
CLOSED http://bugs.python.org/issue3963    created  sandberg                  
                                                                               

quiet the freeze makefile                                        09/25/08
       http://bugs.python.org/issue3964    created  docwhat                   
       patch                                                                   

2.6rc2 crashes when trying to open unicode filename with unprint 09/25/08
CLOSED http://bugs.python.org/issue3965    created  giltay                    
       patch                                                                   

Win32ErrorTests from test_os.py                                  09/25/08
       http://bugs.python.org/issue3966    created  rpetrov                   
                                                                               

bytearray().count()                                              09/25/08
       http://bugs.python.org/issue3967    created  haypo                     
       patch, needs review                                                     

turtle.py: fill() and end_fill() do not work correctly           09/25/08
CLOSED http://bugs.python.org/issue3968    created  gregorlingl               
       patch                                                                   

turtle.py - setup() doesn't work correctly                       09/25/08
CLOSED http://bugs.python.org/issue3969    created  gregorlingl               
       patch                                                                   

Tix Tree widget no longer instantiable.                          09/26/08
CLOSED http://bugs.python.org/issue3970    created  ronLongo                  
                                                                               

s_push: parser stack overflow MemoryError                        09/26/08
       http://bugs.python.org/issue3971    created  netcaf                    
                                                                               

Add Option to Bind to a Local IP Address in httplib.py           09/26/08
       http://bugs.python.org/issue3972    created  eldonz at atlanticdb.com     
       patch                                                                   

Invalid line number in Exception traceback with header # -*- cod 09/26/08
CLOSED http://bugs.python.org/issue3973    created  haypo                     
                                                                               

collections.namedtuple uses exec to create new classes           09/26/08
       http://bugs.python.org/issue3974    created  audax                     
       patch                                                                   

PyTraceBack_Print() doesn't respect # coding: xxx header         09/26/08
       http://bugs.python.org/issue3975    created  haypo                     
                                                                               



Issues Now Closed (55)
______________________

Incorrectly displayed non ascii characters in prompt using "inpu  274 days
       http://bugs.python.org/issue1688    amaury.forgeotdarc        
       patch                                                                   

ctypes.wstring_at and string_at call Python API without the GIL    40 days
       http://bugs.python.org/issue3554    haypo                     
                                                                               

subprocess + stdout redirection + wait + svn= hang                 34 days
       http://bugs.python.org/issue3593    gregory.p.smith           
                                                                               

IDLE does not run with Py30b3                                      30 days
       http://bugs.python.org/issue3628    benjamin.peterson         
                                                                               

Duplicated test name in regex test script                          29 days
       http://bugs.python.org/issue3654    timehorse                 
                                                                               

sqlite: enumeration value 'TYPE_STRING' not handled in switch      29 days
       http://bugs.python.org/issue3659    ghaering                  
       patch, easy                                                             

atexit.register with bad input segfaults on exit                   29 days
       http://bugs.python.org/issue3666    skip.montanaro            
       patch                                                                   

Ignored LDFLAGS during linking libpython$(VERSION).so              27 days
       http://bugs.python.org/issue3678    gregory.p.smith           
       patch, easy                                                             

test_tarfile error on cygwin (Directory not empty)                 10 days
       http://bugs.python.org/issue3838    ocean-city                
       patch, easy                                                             

kqueue.control requires 2 params while docs say max_events (the     8 days
       http://bugs.python.org/issue3852    georg.brandl              
       patch                                                                   

provide a "cmem" role                                               9 days
       http://bugs.python.org/issue3875    georg.brandl              
                                                                               

2.6 regression in urllib.getproxies_environment                     5 days
       http://bugs.python.org/issue3879    benjamin.peterson         
       patch, needs review                                                     

turtle in the tkinter package?                                      2 days
       http://bugs.python.org/issue3884    kirill_simonov            
       patch                                                                   

errors on _bsddb creation and dealloc                               7 days
       http://bugs.python.org/issue3885    haypo                     
       patch, needs review                                                     

Python 2.6 doesn't run after installation on amd64                  3 days
       http://bugs.python.org/issue3887    jpe                       
                                                                               

Demo/parser/unparse.py                                              4 days
       http://bugs.python.org/issue3889    georg.brandl              
       patch                                                                   

collections.deque should have empty() method                        2 days
       http://bugs.python.org/issue3891    rhettinger                
                                                                               

collections                                                         3 days
       http://bugs.python.org/issue3897    georg.brandl              
                                                                               

Slight readme.txt fix (VC9)                                         3 days
       http://bugs.python.org/issue3901    georg.brandl              
       patch                                                                   

Strange heapq behavior on Python 3.0 when overriding __le__         0 days
       http://bugs.python.org/issue3908    giampaolo.rodola          
                                                                               

unittest. assertAlmostEqual() documentation incomplete              1 days
       http://bugs.python.org/issue3912    georg.brandl              
       patch                                                                   

compound_stmt syntax includes 'decorated'                           1 days
       http://bugs.python.org/issue3913    georg.brandl              
                                                                               

augop definition does not include "//="                             1 days
       http://bugs.python.org/issue3914    georg.brandl              
                                                                               

Broken link in 14.7 curses, Python Library Reference                1 days
       http://bugs.python.org/issue3915    georg.brandl              
                                                                               

layout of build directories for Windows not current                 0 days
       http://bugs.python.org/issue3916    georg.brandl              
       patch                                                                   

set_display definition allows empty '{' '}' in Language Definiti    0 days
       http://bugs.python.org/issue3917    georg.brandl              
                                                                               

random.uniform suprisingly (good kind) does not work as document    0 days
       http://bugs.python.org/issue3918    georg.brandl              
                                                                               

PySys_SetObject crashes after Py_NewInterpreter().                  1 days
       http://bugs.python.org/issue3919    benjamin.peterson         
                                                                               

3.0rc1 missing tk lib in sys.path?                                  0 days
       http://bugs.python.org/issue3922    loewis                    
                                                                               

'genexpr_for' in definition of 'call' in Language Reference.        0 days
       http://bugs.python.org/issue3923    georg.brandl              
                                                                               

test_distutils fails on cygwin                                      0 days
       http://bugs.python.org/issue3925    ocean-city                
       patch, easy                                                             

dummy multiprocessing needs to use properties                       1 days
       http://bugs.python.org/issue3927    benjamin.peterson         
       patch                                                                   

Incorrect exception raising in dbm.open on non-existing DB          4 days
       http://bugs.python.org/issue3929    hagen                     
       patch, needs review                                                     

urllib.request.urlopen() different on Windows than Linux            0 days
       http://bugs.python.org/issue3930    amaury.forgeotdarc        
                                                                               

presence of .pythonstartup.py causes __main__.__file__ to be set    0 days
       http://bugs.python.org/issue3933    loewis                    
                                                                               

sphinx - building muppy docs fails on Linux                         1 days
       http://bugs.python.org/issue3934    schuppenies               
       patch                                                                   

Faulty suppression of 'as' keyword warning                          3 days
       http://bugs.python.org/issue3936    benjamin.peterson         
       patch                                                                   

platform.dist(): detect Linux distribution version in a robust,     1 days
       http://bugs.python.org/issue3937    lemburg                   
       patch                                                                   

turtle.py - minor bugfixes                                          0 days
       http://bugs.python.org/issue3940    loewis                    
       patch                                                                   

Usability issue from not being able to use defined start and end    0 days
       http://bugs.python.org/issue3942    benjamin.peterson         
                                                                               

IDLE won't start in 3.0rc1 "Subprocess didn't make connection...    3 days
       http://bugs.python.org/issue3943    amaury.forgeotdarc        
                                                                               

compile error in _fileio.c (cygwin)                                 0 days
       http://bugs.python.org/issue3945    ocean-city                
       patch, easy                                                             

turtle.py: bug in TurtleScreenBase._drawimage                       1 days
       http://bugs.python.org/issue3950    georg.brandl              
       patch                                                                   

__cmp__ still documented in 3.0rc1?                                 0 days
       http://bugs.python.org/issue3953    georg.brandl              
                                                                               

strage default value behaviour                                      0 days
       http://bugs.python.org/issue3958    dominik.holler            
                                                                               

Section permalink html anchors are wrong                            1 days
       http://bugs.python.org/issue3960    ronny                     
                                                                               

Problems when calling exec from a function                          1 days
       http://bugs.python.org/issue3963    sandberg                  
                                                                               

2.6rc2 crashes when trying to open unicode filename with unprint    0 days
       http://bugs.python.org/issue3965    giltay                    
       patch                                                                   

turtle.py: fill() and end_fill() do not work correctly              0 days
       http://bugs.python.org/issue3968    georg.brandl              
       patch                                                                   

turtle.py - setup() doesn't work correctly                          0 days
       http://bugs.python.org/issue3969    georg.brandl              
       patch                                                                   

Tix Tree widget no longer instantiable.                             0 days
       http://bugs.python.org/issue3970    amaury.forgeotdarc        
                                                                               

Invalid line number in Exception traceback with header # -*- cod    0 days
       http://bugs.python.org/issue3973    pitrou                    
                                                                               

SRE: x++ isn't supported                                         2658 days
       http://bugs.python.org/issue433031  georg.brandl              
                                                                               

have a way to search backwards for re                            2413 days
       http://bugs.python.org/issue516762  benjamin.peterson         
                                                                               

Add current dir when running try_run test program                1282 days
       http://bugs.python.org/issue1168055 lianos                    
       patch                                                                   



Top Issues Most Discussed (10)
______________________________

 20 Regexp 2.7 (modifications to current re 2.2.2)                   164 days
open    http://bugs.python.org/issue2636   

 11 Major reworking of Python 2.5.2 re module                         17 days
open    http://bugs.python.org/issue3825   

 11 Failed to build Python 2.5.1 with sqlite3                        521 days
open    http://bugs.python.org/issue1706863

 10 zero-length match confuses re.finditer()                         606 days
open    http://bugs.python.org/issue1647489

  9 26.rc1: test_signal issue on FreeBSD 6.3                          12 days
open    http://bugs.python.org/issue3864   

  9 test_tarfile fails on cygwin (unicode decode error)               17 days
open    http://bugs.python.org/issue3824   

  9 Ctypes is confused by bitfields of varying integer types          45 days
open    http://bugs.python.org/issue3547   

  8 subprocess failing in GUI applications on Windows                  5 days
open    http://bugs.python.org/issue3905   

  8 platform.dist() has unpredictable result under Linux             338 days
open    http://bugs.python.org/issue1322   

  7 Faulty suppression of 'as' keyword warning                         3 days
closed  http://bugs.python.org/issue3936   




From lkcl at lkcl.net  Fri Sep 26 17:37:30 2008
From: lkcl at lkcl.net (Luke Kenneth Casson Leighton)
Date: Fri, 26 Sep 2008 15:37:30 +0000
Subject: [Python-Dev] preliminary findings of compiling python to javascript
	and running it under google "V8"
Message-ID: <ced5f0f60809260837t42624b6bt81b64441b2af26bb@mail.gmail.com>

http://groups.google.com/group/pyjamas-dev/browse_thread/thread/5e14ac70508112e5/53ca0b8190f35e21?hl=en#53ca0b8190f35e21

may be of interest to some people.

From martin at v.loewis.de  Fri Sep 26 23:23:51 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Fri, 26 Sep 2008 23:23:51 +0200
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <200809260921.36012.eckhardt@satorlaser.com>
References: <200809231052.36720.eckhardt@satorlaser.com>
	<200809251347.24163.eckhardt@satorlaser.com>
	<48DBFC1C.3080008@v.loewis.de>
	<200809260921.36012.eckhardt@satorlaser.com>
Message-ID: <48DD52E7.8060408@v.loewis.de>

>> Please don't. Whether or not _UNICODE is defined should have no effect.
> 
> Well, currently it does make a difference. Simple example: CreateFile().

It's not so simple: Python doesn't actually call CreateFile, AFAICT
(well, it does, in _msi.c, but I hope you aren't worried about that
module on CE).

> One 
> of the parameters is actually documented at LPCTSTR, which is a TCHAR string. 
> That means that under CE, CreateFile(), which is actually a macro, resolves 
> to CreateFileW and the existing code that passes a char-string doesn't work.

True, but irrelevant: Python doesn't actually use CreateFile. Instead,
it calls both CreateFileA and CreateFileW (in posixmodule.c).

> VS2005 works really good with CE5 and later. My problem here is that I can't 
> provide a projectfile that suits everyone, because different devices have 
> different SDKs and each configuration/SDK/CPU is another entry in that file. 
> Further, the SDKs or devices differ in what they support, the platform CE as 
> such doesn't exist.

Hmm. And you *can* provide an SCons file that supports all the SDKs?

Regards,
Martin

From barry at python.org  Sat Sep 27 00:24:37 2008
From: barry at python.org (Barry Warsaw)
Date: Fri, 26 Sep 2008 18:24:37 -0400
Subject: [Python-Dev] Reminder: Python 2.6 final next Wednesday
Message-ID: <AA1F5430-661B-46AE-86C8-3E9CC9CE26C8@python.org>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

This is a reminder that Python 2.6 final is schedule for release next  
Wednesday, October 1st.

Once again, I've gone through the release blocker issues and knocked  
anything that doesn't specifically affect 2.6 to deferred blocker.   
This leaves us with  7 open blocking issues.

Please spend some time over the next several days reviewing patches,  
making comments and working toward closing these issues.  Email me  
directly if you have any questions, or ping me on irc.  Unfortunately,  
my 'net connection may be a little bit flakey until Wednesday, but I  
will do my best to get online and follow up as needed.

Please pitch in to help get Python 2.6 released on time!

Thanks,
- -Barry

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)

iQCVAwUBSN1hJnEjvBPtnXfVAQIQfQQAgYr0tuzJhm3LZX/1SaCwRJcX09UvNH1I
CjZHs2TKS22MjF9d3mmBgcEJPl9AwGE+6EF6OiSgrsNRoRtnN0MMT3nQo+deRkan
P3jUgMFJMFkA7Uq5MmuNnEnKZXa/bsu/8Om/4wqHqvtDXbUQkZPfyE8BFwBzJJSM
Aa6Wp3wieFs=
=r4iD
-----END PGP SIGNATURE-----

From aahz at pythoncraft.com  Sat Sep 27 01:25:50 2008
From: aahz at pythoncraft.com (Aahz)
Date: Fri, 26 Sep 2008 16:25:50 -0700
Subject: [Python-Dev] PyCon 2009 Call for Proposals
Message-ID: <20080926232550.GA29031@panix.com>

Call for proposals -- PyCon 2009 -- <http://us.pycon.org/2009/>
===============================================================

Want to share your experience and expertise? PyCon 2009 is looking for
proposals to fill the formal presentation tracks. The PyCon conference
days will be March 27-29, 2009 in Chicago, Illinois, preceded by the
tutorial days (March 25-26), and followed by four days of development
sprints (March 30-April 2).

Previous PyCon conferences have had a broad range of presentations,
from reports on academic and commercial projects to tutorials and case
studies. We hope to continue that tradition this year.

Online proposal submission will open on September 29, 2008. Proposals
will be accepted through November 03, with acceptance notifications
coming out on December 15. For the detailed call for proposals,
please see:

   <http://us.pycon.org/2009/conference/proposals/>

We look forward to seeing you in Chicago!
-- 
Aahz (aahz at pythoncraft.com)           <*>         http://www.pythoncraft.com/

"Argue for your limitations, and sure enough they're yours."  --Richard Bach

From musiccomposition at gmail.com  Sat Sep 27 03:23:36 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Fri, 26 Sep 2008 20:23:36 -0500
Subject: [Python-Dev] call for help: C-API 2 to 3 porting guide
Message-ID: <1afaf6160809261823w7e9698dfm799fa45976a82fcb@mail.gmail.com>

Hi guys,
I remember a while back we were bemoaning the fact that little
documentation exists for the extension module transition between 2.x
and 3.x. In the name of this effort, I've created a c porting howto
(Doc/howto/cporting.rst). I have started a few sections, but I can't
do it all alone. Please consider writing a section.

-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From brett at python.org  Sat Sep 27 04:36:16 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 26 Sep 2008 19:36:16 -0700
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <20080926232550.GA29031@panix.com>
References: <20080926232550.GA29031@panix.com>
Message-ID: <bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>

On Fri, Sep 26, 2008 at 4:25 PM, Aahz <aahz at pythoncraft.com> wrote:
> Call for proposals -- PyCon 2009 -- <http://us.pycon.org/2009/>
> ===============================================================
>
> Want to share your experience and expertise? PyCon 2009 is looking for
> proposals to fill the formal presentation tracks. The PyCon conference
> days will be March 27-29, 2009 in Chicago, Illinois, preceded by the
> tutorial days (March 25-26), and followed by four days of development
> sprints (March 30-April 2).
>

I am thinking of organizing a panel this year for python-dev (much
like the one I organized in 2007). Who would be willing to be on the
panel with me if I did this?

-Brett

From musiccomposition at gmail.com  Sat Sep 27 04:37:47 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Fri, 26 Sep 2008 21:37:47 -0500
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
References: <20080926232550.GA29031@panix.com>
	<bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
Message-ID: <1afaf6160809261937q2edc9ecdub75a1d6d9b5a866a@mail.gmail.com>

On Fri, Sep 26, 2008 at 9:36 PM, Brett Cannon <brett at python.org> wrote:
> On Fri, Sep 26, 2008 at 4:25 PM, Aahz <aahz at pythoncraft.com> wrote:
>> Call for proposals -- PyCon 2009 -- <http://us.pycon.org/2009/>
>> ===============================================================
>>
>> Want to share your experience and expertise? PyCon 2009 is looking for
>> proposals to fill the formal presentation tracks. The PyCon conference
>> days will be March 27-29, 2009 in Chicago, Illinois, preceded by the
>> tutorial days (March 25-26), and followed by four days of development
>> sprints (March 30-April 2).
>>
>
> I am thinking of organizing a panel this year for python-dev (much
> like the one I organized in 2007). Who would be willing to be on the
> panel with me if I did this?

Could you explain what this is to us a little more, please? :)


-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From ncoghlan at gmail.com  Sat Sep 27 04:56:26 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 27 Sep 2008 12:56:26 +1000
Subject: [Python-Dev] [Python-3000] Reminder: Python 2.6 final next
	Wednesday
In-Reply-To: <AA1F5430-661B-46AE-86C8-3E9CC9CE26C8@python.org>
References: <AA1F5430-661B-46AE-86C8-3E9CC9CE26C8@python.org>
Message-ID: <48DDA0DA.2010200@gmail.com>

Barry Warsaw wrote:
> This is a reminder that Python 2.6 final is schedule for release next
> Wednesday, October 1st.
> 
> Once again, I've gone through the release blocker issues and knocked
> anything that doesn't specifically affect 2.6 to deferred blocker.  This
> leaves us with  7 open blocking issues.

Have a couple of these been fixed already? The showstoppers search on
the bug tracker is only giving me 5 hits.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From brett at python.org  Sat Sep 27 07:02:32 2008
From: brett at python.org (Brett Cannon)
Date: Fri, 26 Sep 2008 22:02:32 -0700
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <1afaf6160809261937q2edc9ecdub75a1d6d9b5a866a@mail.gmail.com>
References: <20080926232550.GA29031@panix.com>
	<bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
	<1afaf6160809261937q2edc9ecdub75a1d6d9b5a866a@mail.gmail.com>
Message-ID: <bbaeab100809262202r6bbeb7c1jfeec9008e39732e5@mail.gmail.com>

On Fri, Sep 26, 2008 at 7:37 PM, Benjamin Peterson
<musiccomposition at gmail.com> wrote:
> On Fri, Sep 26, 2008 at 9:36 PM, Brett Cannon <brett at python.org> wrote:
>> On Fri, Sep 26, 2008 at 4:25 PM, Aahz <aahz at pythoncraft.com> wrote:
>>> Call for proposals -- PyCon 2009 -- <http://us.pycon.org/2009/>
>>> ===============================================================
>>>
>>> Want to share your experience and expertise? PyCon 2009 is looking for
>>> proposals to fill the formal presentation tracks. The PyCon conference
>>> days will be March 27-29, 2009 in Chicago, Illinois, preceded by the
>>> tutorial days (March 25-26), and followed by four days of development
>>> sprints (March 30-April 2).
>>>
>>
>> I am thinking of organizing a panel this year for python-dev (much
>> like the one I organized in 2007). Who would be willing to be on the
>> panel with me if I did this?
>
> Could you explain what this is to us a little more, please? :)
>

You sit in front of a bunch of people answering questions asked by the
audience. You know, a panel. =) It's just a Q&A session so that PyCon
attendees can ask python-dev a bunch of random questions. Demystifies
some things and puts faces to python-dev.

-Brett

From ziade.tarek at gmail.com  Sat Sep 27 12:25:34 2008
From: ziade.tarek at gmail.com (=?ISO-8859-1?Q?Tarek_Ziad=E9?=)
Date: Sat, 27 Sep 2008 12:25:34 +0200
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <bbaeab100809262202r6bbeb7c1jfeec9008e39732e5@mail.gmail.com>
References: <20080926232550.GA29031@panix.com>
	<bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
	<1afaf6160809261937q2edc9ecdub75a1d6d9b5a866a@mail.gmail.com>
	<bbaeab100809262202r6bbeb7c1jfeec9008e39732e5@mail.gmail.com>
Message-ID: <94bdd2610809270325u49c52a3dhd874f6ff432ebc13@mail.gmail.com>

On Sat, Sep 27, 2008 at 7:02 AM, Brett Cannon <brett at python.org> wrote:

> On Fri, Sep 26, 2008 at 7:37 PM, Benjamin Peterson
> <musiccomposition at gmail.com> wrote:
> > On Fri, Sep 26, 2008 at 9:36 PM, Brett Cannon <brett at python.org> wrote:
> >> On Fri, Sep 26, 2008 at 4:25 PM, Aahz <aahz at pythoncraft.com> wrote:
> >>> Call for proposals -- PyCon 2009 -- <http://us.pycon.org/2009/>
> >>> ===============================================================
> >>>
> >>> Want to share your experience and expertise? PyCon 2009 is looking for
> >>> proposals to fill the formal presentation tracks. The PyCon conference
> >>> days will be March 27-29, 2009 in Chicago, Illinois, preceded by the
> >>> tutorial days (March 25-26), and followed by four days of development
> >>> sprints (March 30-April 2).
> >>>
> >>
> >> I am thinking of organizing a panel this year for python-dev (much
> >> like the one I organized in 2007). Who would be willing to be on the
> >> panel with me if I did this?
> >
> > Could you explain what this is to us a little more, please? :)
> >
>
> You sit in front of a bunch of people answering questions asked by the
> audience. You know, a panel. =) It's just a Q&A session so that PyCon
> attendees can ask python-dev a bunch of random questions. Demystifies
> some things and puts faces to python-dev.


>From a non-core developer point of view:

What could be great imho would be to have a short "How Python is developed"
presentation
just before the panel starts.

Tarek

-- 
Tarek Ziad? | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080927/9f54f466/attachment.htm>

From victor.stinner at haypocalc.com  Sat Sep 27 14:04:25 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Sat, 27 Sep 2008 14:04:25 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
Message-ID: <200809271404.25654.victor.stinner@haypocalc.com>

Hi,

I read that Python 2.6 is planned to Wednesday. One bug is is still open and 
important for me: Python 2.6/3.0 are unable to use filename as byte strings.
    http://bugs.python.org/issue3187

The problem
===========

On Windows, all filenames are unicode strings (I guess UTF-16-LE), but on UNIX 
for historical reasons, filenames are byte strings. On Linux, you can expect 
UTF-8 valid filenames but sometimes (eg. copy from a FAT32 USB key to an ext3 
filesystem) you get invalid filename (byte string in a different charset than 
your default filesystem encoding (utf8)).


Python functions using filenames
================================

In Python, you have (incomplete list):
 - filename producer: os.listdir(), os.walk(), glob.glob()
 - filename manipulation: os.path.*()
 - access file: open(), os.unlink(), shutil.rmtree()

If you give unicode to producer, they return unicode _or_ byte strings (type 
may change for each filename :-/). Guido proposed to break this behaviour: 
raise an exception if unicode conversion fails. We may consider an option 
like "skip invalid".

If you give bytes to producer, they only return byte strings. Great.

Filename manipulation: in python 2.6/3.0, os.path.*() is not compatible with 
the type "bytes". So you can not use os.path.join(<your unicode path>, <bytes 
filename>) *nor* os.path.join(<your bytes path>, <bytes filename>) because 
os.path.join() (eg. with the posix version) uses path.endswith('/').

Access file: open() rejects the type bytes (it's just a test, open() supports 
bytes if you remove the test). As I remember, unlink() is compatible with 
bytes. But rmtree() fails because it uses os.path.join() (even if you give 
bytes directory, join() fails).


Solutions
=========

 - producer: unicode => *only* unicode // bytes => bytes
 - manipulation: support both unicode and bytes but avoid (when it's possible)
   to mix bytes and characters
 - open(): allow bytes

I implemented these solutions as a patch set attached to the issue #3187:
 * posix_path_bytes.patch: fix posixpath.join() to support bytes
 * io_byte_filename.patch: open() allows bytes filename
 * fnmatch_bytes.patch: patch fnmatch.filter() to accept bytes filenames
 * glob1_bytes.patch: fix glob.glob() to accept invalid directory name

Mmmh, there is no patch for stop os.listdir() on invalid filename.


Priority
========

I think that the problem is important because it's a regression from 2.5 to 
2.6/3.0. Python 2.5 uses bytes filename, so it was possible to 
open/unlink "invalid" unicode strings (since it's not unicode but bytes).

Well, if it's too late for the final versions, this problem should be at least 
fixed quickly.


Test the problem
================

Example to create invalid filenames on Linux:

$ mkdir /tmp/test
$ cd /tmp/test
$ touch $(echo -e "a\xffb")
$ mkdir $(echo -e "dir\xffname")
$ touch $(echo -e "dir\xffname/file")
$ find
.
./a?b
./dir?name
./dir?name/file

Python 2.5:
>>> import os
>>> os.listdir('.')
['a\xffb', 'dir\xffname']
>>> open(os.listdir('.')[0]).close()  # open file: ok
>>> os.unlink(os.listdir('.')[0])     # remove file: ok
>>> os.listdir('.')
['dir\xffname']
>>> shutil.rmtree(os.listdir('.')[0])  # remove dir: ok


Wrong solutions
===============

New type
--------

I proposed an ugly type "InvalidFilename" mixing bytes and characters. As 
everybody using unicode knows, it's a bad idea :-) (and it introduces a new 
type).

Convert bytes to unicode (replace)
----------------------------------

unicode_filename = unicode(bytes_filename, charset, "replace")

Ok, you will get valid unicode strings which can be used in os.path.join() & 
friends, but open() or unlink() will fails because this filename doesn't 
exist!

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From eric at trueblade.com  Sat Sep 27 14:24:07 2008
From: eric at trueblade.com (Eric Smith)
Date: Sat, 27 Sep 2008 08:24:07 -0400
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
References: <20080926232550.GA29031@panix.com>
	<bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
Message-ID: <48DE25E7.9010302@trueblade.com>

Brett Cannon wrote:
> I am thinking of organizing a panel this year for python-dev (much
> like the one I organized in 2007). Who would be willing to be on the
> panel with me if I did this?

If you're looking for the perspective of someone who's relatively new to 
Python core programming, I'll do it. You won't offend me by picking 
someone else, though.

Eric.


From musiccomposition at gmail.com  Sat Sep 27 14:40:05 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sat, 27 Sep 2008 07:40:05 -0500
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <bbaeab100809262202r6bbeb7c1jfeec9008e39732e5@mail.gmail.com>
References: <20080926232550.GA29031@panix.com>
	<bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
	<1afaf6160809261937q2edc9ecdub75a1d6d9b5a866a@mail.gmail.com>
	<bbaeab100809262202r6bbeb7c1jfeec9008e39732e5@mail.gmail.com>
Message-ID: <1afaf6160809270540x294d8f77jc6ffcd6f4c99deb5@mail.gmail.com>

On Sat, Sep 27, 2008 at 12:02 AM, Brett Cannon <brett at python.org> wrote:
> On Fri, Sep 26, 2008 at 7:37 PM, Benjamin Peterson
> <musiccomposition at gmail.com> wrote:
>> On Fri, Sep 26, 2008 at 9:36 PM, Brett Cannon <brett at python.org> wrote:
>>>>
>>>
>>> I am thinking of organizing a panel this year for python-dev (much
>>> like the one I organized in 2007). Who would be willing to be on the
>>> panel with me if I did this?
>>
>> Could you explain what this is to us a little more, please? :)
>>
>
> You sit in front of a bunch of people answering questions asked by the
> audience. You know, a panel. =) It's just a Q&A session so that PyCon
> attendees can ask python-dev a bunch of random questions. Demystifies
> some things and puts faces to python-dev.

Sure. I would be interested in that.



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From amauryfa at gmail.com  Sat Sep 27 14:41:39 2008
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sat, 27 Sep 2008 14:41:39 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809271404.25654.victor.stinner@haypocalc.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
Message-ID: <e27efe130809270541l30bc9a3fu2273f9d5d742b57c@mail.gmail.com>

Hello Victor,

2008/9/27 Victor Stinner <victor.stinner at haypocalc.com>:
> Hi,
>
> I read that Python 2.6 is planned to Wednesday. One bug is is still open and
> important for me: Python 2.6/3.0 are unable to use filename as byte strings.
>    http://bugs.python.org/issue3187
[...]

Is it really a 2.6 problem? I could not find any difference with 2.5
in this regard,
the tests you propose still pass.

> Filename manipulation: in python 2.6/3.0, os.path.*() is not compatible with
> the type "bytes"

With python 2.6,
>>> bytes is str
True


But I agree that this is THE unresolved issue of python 3.0.

-- 
Amaury Forgeot d'Arc

From fredrik at pythonware.com  Sat Sep 27 15:01:32 2008
From: fredrik at pythonware.com (Fredrik Lundh)
Date: Sat, 27 Sep 2008 15:01:32 +0200
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <bbaeab100809262202r6bbeb7c1jfeec9008e39732e5@mail.gmail.com>
References: <20080926232550.GA29031@panix.com>	<bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>	<1afaf6160809261937q2edc9ecdub75a1d6d9b5a866a@mail.gmail.com>
	<bbaeab100809262202r6bbeb7c1jfeec9008e39732e5@mail.gmail.com>
Message-ID: <gblarb$rvg$2@ger.gmane.org>

Brett Cannon wrote:

> You sit in front of a bunch of people answering questions asked by the
> audience. You know, a panel. =) It's just a Q&A session so that PyCon
> attendees can ask python-dev a bunch of random questions. Demystifies
> some things and puts faces to python-dev.

and using moderator.appspot.com to gather questions from off-site 
attendees, perhaps?

</F>


From lists at cheimes.de  Sat Sep 27 15:09:29 2008
From: lists at cheimes.de (Christian Heimes)
Date: Sat, 27 Sep 2008 15:09:29 +0200
Subject: [Python-Dev] call for help: C-API 2 to 3 porting guide
In-Reply-To: <1afaf6160809261823w7e9698dfm799fa45976a82fcb@mail.gmail.com>
References: <1afaf6160809261823w7e9698dfm799fa45976a82fcb@mail.gmail.com>
Message-ID: <48DE3089.7040308@cheimes.de>

Benjamin Peterson wrote:
> Hi guys,
> I remember a while back we were bemoaning the fact that little
> documentation exists for the extension module transition between 2.x
> and 3.x. In the name of this effort, I've created a c porting howto
> (Doc/howto/cporting.rst). I have started a few sections, but I can't
> do it all alone. Please consider writing a section.

I'll try to find some time over the weekend to assist you. I'm going to 
  contact the Cython developers, too. They have a great deal of 
experience with Python 2.x and 3.0 compatible C code. Perhaps they can 
give us some hints and tell us about gotchas and tricks.

Christian



From victor.stinner at haypocalc.com  Sat Sep 27 16:44:50 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Sat, 27 Sep 2008 16:44:50 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809271404.25654.victor.stinner@haypocalc.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
Message-ID: <200809271644.50501.victor.stinner@haypocalc.com>

Le Saturday 27 September 2008 14:04:25 Victor Stinner, vous avez ?crit?:
> I read that Python 2.6 is planned to Wednesday. One bug is is still open
> and important for me: Python 2.6/3.0 are unable to use filename as byte
> strings. http://bugs.python.org/issue3187

Ooops, as amaury noticed, the problem is specific to Python 3.0. My example 
works correctly with Python 2.6:
----------
$ find
.
./a?b
./dir?name
./dir?name/file

$ ~/prog/python-trunk/python
Python 2.6rc2+ (trunk:66627M, Sep 26 2008, 19:03:31)
>>> import os, shutil
>>> os.listdir('.')
['a\xffb', 'dir\xffname']
>>> open(os.listdir('.')[0]).close()
>>> os.unlink(os.listdir('.')[0])
>>> os.listdir('.')
['dir\xffname']
>>> shutil.rmtree(os.listdir('.')[0])
----------


Same test with Python 3.0:
----------
$ pwd
/tmp/test

$ find
.
./a?b
./dir?name
./dir?name/file

$ ~/prog/py3k/python
Python 3.0rc1+ (py3k:66627M, Sep 26 2008, 18:10:03)
>>> import os, shutil
>>> os.listdir('.')
[b'a\xffb', b'dir\xffname']
>>> open(os.listdir('.')[0]).close()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NOT FOUNT
>>> os.unlink(os.listdir('.')[0])
>>> os.listdir('.')
[b'dir\xffname']
>>> shutil.rmtree(os.listdir('.')[0])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NOT FOUNT
----------

Results:
 * open() doesn't support bytes
 * unlink() supports bytes
 * shutil.rmtree() doesn't support bytes


Another example to test chdir()/getcwd():
----------
$ pwd
/tmp/test

$ ~/prog/py3k/python
Python 3.0rc1+ (py3k:66627M, Sep 26 2008, 18:10:03)
>>> import os, shutil
>>> os.getcwd()
'/tmp/test'
>>> os.chdir(b'/tmp/test/dir\xffname')
>>> os.getcwd()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
NOT FOUNT
----------

Results:
 * chdir() supports byte filename
 * getcwd() fails

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From victor.stinner at haypocalc.com  Sat Sep 27 17:32:19 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Sat, 27 Sep 2008 17:32:19 +0200
Subject: [Python-Dev] I would like an Python account
Message-ID: <200809271732.19317.victor.stinner@haypocalc.com>

Hi,

Would it possible to get more permissions on Python bugtracker, especially to 
add keywords, close a duplicate bug, etc.?

I also would like an SVN account for Python trunk and Python 3000 branch. I 
know that both branches are near "frozen" and commit are only allowed after 
patch review and/or discussion (on the mailing list or the bug tracker. So I 
will after the final versions (2.6 and 3.0) to commit. Or would it be 
possible to have a "mentor"? Someone to ask questions about Python SVN/bug 
tracker, or someone who reviews my patchs/commits.

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From musiccomposition at gmail.com  Sat Sep 27 17:35:50 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sat, 27 Sep 2008 10:35:50 -0500
Subject: [Python-Dev] I would like an Python account
In-Reply-To: <200809271732.19317.victor.stinner@haypocalc.com>
References: <200809271732.19317.victor.stinner@haypocalc.com>
Message-ID: <1afaf6160809270835w20f3bb8teca2e5cfdd7c98ee@mail.gmail.com>

On Sat, Sep 27, 2008 at 10:32 AM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Hi,
>
> Would it possible to get more permissions on Python bugtracker, especially to
> add keywords, close a duplicate bug, etc.?

+1

In the time Victor has been contributing, he's found copious amounts
of bugs and contributed excellent patches for almost every one.

>
> I also would like an SVN account for Python trunk and Python 3000 branch. I
> know that both branches are near "frozen" and commit are only allowed after
> patch review and/or discussion (on the mailing list or the bug tracker. So I
> will after the final versions (2.6 and 3.0) to commit. Or would it be
> possible to have a "mentor"? Someone to ask questions about Python SVN/bug
> tracker, or someone who reviews my patchs/commits.
>
> --
> Victor Stinner aka haypo
> http://www.haypocalc.com/blog/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From victor.stinner at haypocalc.com  Sat Sep 27 17:38:42 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Sat, 27 Sep 2008 17:38:42 +0200
Subject: [Python-Dev] 2to3 and reST text format
Message-ID: <200809271738.42156.victor.stinner@haypocalc.com>

Hi,

As I remember, last month 2to3 was able to fix my reST documentation 
using "-d" option. But trunk version fails to parse my reST documents. Should 
I run fsck on my brain or is it really a regression?

Short example:
-----------------
Title
=====

Some text before the example:

    >>> print "Hello world!"
    Hello world!

Verbatim block: ::

    >>> print "Hello world!"
    Hello world!
-----------------

The document can be checked with:
   python2.5 -c "import doctest; doctest.testfile('test.rst')"

2to3 output:
-----------------
$ 2to3 -d test.rst
(...)
RefactoringTool: Can't parse test.rst: ParseError: bad input: type=28, 
value='==', context=('', (2, 0))
(...)
-----------------

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From musiccomposition at gmail.com  Sat Sep 27 17:45:26 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sat, 27 Sep 2008 10:45:26 -0500
Subject: [Python-Dev] 2to3 and reST text format
In-Reply-To: <200809271738.42156.victor.stinner@haypocalc.com>
References: <200809271738.42156.victor.stinner@haypocalc.com>
Message-ID: <1afaf6160809270845q26814ae5ka11e82c3b299d6b2@mail.gmail.com>

On Sat, Sep 27, 2008 at 10:38 AM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Hi,
>
> As I remember, last month 2to3 was able to fix my reST documentation
> using "-d" option. But trunk version fails to parse my reST documents. Should
> I run fsck on my brain or is it really a regression?

Looks like it's my fault. Fixed in r66644.

>
> Short example:
> -----------------
> Title
> =====
>
> Some text before the example:
>
>    >>> print "Hello world!"
>    Hello world!
>
> Verbatim block: ::
>
>    >>> print "Hello world!"
>    Hello world!
> -----------------
>
> The document can be checked with:
>   python2.5 -c "import doctest; doctest.testfile('test.rst')"
>
> 2to3 output:
> -----------------
> $ 2to3 -d test.rst
> (...)
> RefactoringTool: Can't parse test.rst: ParseError: bad input: type=28,
> value='==', context=('', (2, 0))
> (...)
> -----------------
>
> --
> Victor Stinner aka haypo
> http://www.haypocalc.com/blog/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/musiccomposition%40gmail.com
>



-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From victor.stinner at haypocalc.com  Sat Sep 27 17:54:28 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Sat, 27 Sep 2008 17:54:28 +0200
Subject: [Python-Dev] Python security team
Message-ID: <200809271754.29068.victor.stinner@haypocalc.com>

Hi,

I would like to know if a Python security team does exist. I sent an email 
about an imageop issue, and I didn't get any answer. Later I learned that a 
security ticket was created, I don't have access to it.

First, I would like to access to these informations. Not only this issue, but 
all security related issues. I have some knowledges about security and I can 
help to resolve issues and/or estimate the criticity of an issue.

Second, I would like to help to fix all Python security issues. It looks like 
Python community isn't very reactive (proactive?) about security. Eg. a DoS 
was reported in smtpd server (integrated to Python)... 15 months ago. A patch 
is available but it's not applied in Python trunk.

Third, I'm also looking for a document explaining "how Python is secure" (!). 
If an user can run arbitrary Python code, we know that it can do anything 
(read/remove any file, create/kill any process, read/write anywhere in 
memory, etc.). Brett wrote a paper about CPython sandboxing. PyPy is also 
working on sandboxing using two interpreters: one has high priviledge and 
execute instructions from the second interpreter (after checking the 
permissions and arguments). So is there somewhere a document to explain to 
current status of Python security?

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From martin at v.loewis.de  Sat Sep 27 19:41:50 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sat, 27 Sep 2008 19:41:50 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809271404.25654.victor.stinner@haypocalc.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
Message-ID: <48DE705E.6050405@v.loewis.de>

> I think that the problem is important because it's a regression from 2.5 to 
> 2.6/3.0. Python 2.5 uses bytes filename, so it was possible to 
> open/unlink "invalid" unicode strings (since it's not unicode but bytes).

I'd like to stress that the problem is *not* a regression from 2.5 to 2.6.

As for 3.0, I'd like to argue that the problem is a minor issue. Even
though you may run into file names that can't be decoded, that happening
really indicates some bigger problem in the management of the system
where this happens, and the proper solution (IMO) should be to change
the system (leaving open the question whether or not Python should
be also changed to work with such broken systems).

Regards,
Martin

From bob at redivi.com  Sat Sep 27 19:50:31 2008
From: bob at redivi.com (Bob Ippolito)
Date: Sat, 27 Sep 2008 10:50:31 -0700
Subject: [Python-Dev] json decoder speedups, any time left for 2.6?
In-Reply-To: <6a36e7290809240902n2acf8aafqdd77b1c46cd9961@mail.gmail.com>
References: <6a36e7290809232151k21824aabw62293dc871dac098@mail.gmail.com>
	<48DA0C9B.6030707@gmail.com>
	<F4A9FA8C-9020-4B19-8AEA-D6F60A32D505@python.org>
	<6a36e7290809240808y547587dcj94b339a5d6ed987b@mail.gmail.com>
	<e8a0972d0809240842j248528d1x5279464199def385@mail.gmail.com>
	<6a36e7290809240902n2acf8aafqdd77b1c46cd9961@mail.gmail.com>
Message-ID: <6a36e7290809271050q32bd1bbfk26c3b2a51210fe8d@mail.gmail.com>

simplejson 2.0.0 is now released which is about as optimized as I can
be bothered to make it. It's about 4x faster than cPickle for encoding
and just a little slower and decoding, which is good enough for now ;)
The pure Python source is much uglier now (to avoid global lookups,
etc.), but also several times faster than it was.

http://pypi.python.org/pypi/simplejson

One of the optimizations I made probably isn't good for Py3k, it will
return ASCII strings as str objects instead of converting to unicode,
but that shouldn't be too much work to port (though I haven't looked
at the current _json.c port for Py3k).

I also converted over to using Sphinx documentation, which was nice
because I was able to just re-use the docs that were already in Python
trunk after changing the module name around.

All of the work should be easy to merge back into trunk so I'll try
and take care of that quickly after Python 2.6 is released.

On Wed, Sep 24, 2008 at 9:02 AM, Bob Ippolito <bob at redivi.com> wrote:
> http://pypi.python.org/pypi/simplejson
>
> The _speedups module is optional.
>
> On Wed, Sep 24, 2008 at 8:42 AM, Alex Martelli <aleaxit at gmail.com> wrote:
>> Meanwhile, can you please release (wherever you normally release
>> things;-) the pure-Python version as well?  I'd like to play around
>> with it in Google App Engine opensource sandboxes (e.g., cfr.
>> gae-json-rest -- I'll be delighted to add you to that project if you
>> want of course;-) and that requires Python 2.5 and only pure-Python
>> add-ons... thanks!
>>
>> Alex
>>
>>
>> On Wed, Sep 24, 2008 at 8:08 AM, Bob Ippolito <bob at redivi.com> wrote:
>>> On Wed, Sep 24, 2008 at 6:14 AM, Barry Warsaw <barry at python.org> wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA1
>>>>
>>>> On Sep 24, 2008, at 5:47 AM, Nick Coghlan wrote:
>>>>
>>>>> Bob Ippolito wrote:
>>>>>>
>>>>>> How much time do I
>>>>>> have left to get this into Python 2.6?
>>>>>
>>>>> Zero I'm afraid - with rc1 out, it's release blocker bugs only. Anything
>>>>> which can be deferred to the 2.6.1 release without causing any major
>>>>> harm is definitely out - and while a 2x speedup is very nice, it isn't
>>>>> something to be squeezing in post-rc1.
>>>>>
>>>>> Still, that should make for a nice incremental improvement when 2.6.1
>>>>> rolls around.
>>>>
>>>> I concur.
>>>
>>> Ok, no problem. The speedup is about 3x now on the trunk ;) I think
>>> that further optimization will require some more C hacking, but 2.6.1
>>> should give me plenty of time to get around to some of that.
>>>
>>> -bob
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> http://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/aleaxit%40gmail.com
>>>
>>
>

From josiah.carlson at gmail.com  Sat Sep 27 20:04:06 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Sat, 27 Sep 2008 11:04:06 -0700
Subject: [Python-Dev] Python security team
In-Reply-To: <200809271754.29068.victor.stinner@haypocalc.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
Message-ID: <e6511dbf0809271104y6fc22c29m608ea24fb8417781@mail.gmail.com>

On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Second, I would like to help to fix all Python security issues. It looks like
> Python community isn't very reactive (proactive?) about security. Eg. a DoS
> was reported in smtpd server (integrated to Python)... 15 months ago. A patch
> is available but it's not applied in Python trunk.

The smtpd module is not meant to be used without modification.  It is
the responsibility of the application writer to decide the limitations
of the emails they want to allow sending, and subsequently handle the
case where emails overrun that limit.  That the bug wasn't assigned to
me outright (I am the maintainer of asyncore, asynchat, and smtpd) was
an understandable mistake.

 - Josiah

From hodgestar+pythondev at gmail.com  Sun Sep 28 01:13:43 2008
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Sun, 28 Sep 2008 01:13:43 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48DE705E.6050405@v.loewis.de>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48DE705E.6050405@v.loewis.de>
Message-ID: <fb73205e0809271613h5597e3cci1e24c7630840ad21@mail.gmail.com>

On Sat, Sep 27, 2008 at 7:41 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> As for 3.0, I'd like to argue that the problem is a minor issue. Even
> though you may run into file names that can't be decoded, that happening
> really indicates some bigger problem in the management of the system
> where this happens, and the proper solution (IMO) should be to change
> the system (leaving open the question whether or not Python should
> be also changed to work with such broken systems).

I can't agree here. File handling is a fundamental operation and I
would expect something like:

>>> for fname in os.listdir('.'):
...       if os.path.isfile(fname):
...          file(fname)

to work for all files.  To have to know to put in special handling for
certain corner case filenames or worse to not be able to open some
files at all would be a serious loss. It would also complicate
migrating code correctly to 3.0.

Regardless of whose fault the underlying issue is, someone has to deal
with the problem and if core Python doesn't, each developer who
encounters the problem will have to come up with his/her own solution.

From brett at python.org  Sun Sep 28 02:21:35 2008
From: brett at python.org (Brett Cannon)
Date: Sat, 27 Sep 2008 17:21:35 -0700
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <gblarb$rvg$2@ger.gmane.org>
References: <20080926232550.GA29031@panix.com>
	<bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
	<1afaf6160809261937q2edc9ecdub75a1d6d9b5a866a@mail.gmail.com>
	<bbaeab100809262202r6bbeb7c1jfeec9008e39732e5@mail.gmail.com>
	<gblarb$rvg$2@ger.gmane.org>
Message-ID: <bbaeab100809271721n48f87f94t6e5188956fbe2f2b@mail.gmail.com>

On Sat, Sep 27, 2008 at 6:01 AM, Fredrik Lundh <fredrik at pythonware.com> wrote:
> Brett Cannon wrote:
>
>> You sit in front of a bunch of people answering questions asked by the
>> audience. You know, a panel. =) It's just a Q&A session so that PyCon
>> attendees can ask python-dev a bunch of random questions. Demystifies
>> some things and puts faces to python-dev.
>
> and using moderator.appspot.com to gather questions from off-site attendees,
> perhaps?
>

That was my thinking. That way I don't have to do much moderating
beyond what the community voted for.

-Brett

From brett at python.org  Sun Sep 28 02:22:40 2008
From: brett at python.org (Brett Cannon)
Date: Sat, 27 Sep 2008 17:22:40 -0700
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <94bdd2610809270325u49c52a3dhd874f6ff432ebc13@mail.gmail.com>
References: <20080926232550.GA29031@panix.com>
	<bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
	<1afaf6160809261937q2edc9ecdub75a1d6d9b5a866a@mail.gmail.com>
	<bbaeab100809262202r6bbeb7c1jfeec9008e39732e5@mail.gmail.com>
	<94bdd2610809270325u49c52a3dhd874f6ff432ebc13@mail.gmail.com>
Message-ID: <bbaeab100809271722l4e968a06n1b0aac88bb686bd9@mail.gmail.com>

On Sat, Sep 27, 2008 at 3:25 AM, Tarek Ziad? <ziade.tarek at gmail.com> wrote:
>
>
> On Sat, Sep 27, 2008 at 7:02 AM, Brett Cannon <brett at python.org> wrote:
>>
>> On Fri, Sep 26, 2008 at 7:37 PM, Benjamin Peterson
>> <musiccomposition at gmail.com> wrote:
>> > On Fri, Sep 26, 2008 at 9:36 PM, Brett Cannon <brett at python.org> wrote:
>> >> On Fri, Sep 26, 2008 at 4:25 PM, Aahz <aahz at pythoncraft.com> wrote:
>> >>> Call for proposals -- PyCon 2009 -- <http://us.pycon.org/2009/>
>> >>> ===============================================================
>> >>>
>> >>> Want to share your experience and expertise? PyCon 2009 is looking for
>> >>> proposals to fill the formal presentation tracks. The PyCon conference
>> >>> days will be March 27-29, 2009 in Chicago, Illinois, preceded by the
>> >>> tutorial days (March 25-26), and followed by four days of development
>> >>> sprints (March 30-April 2).
>> >>>
>> >>
>> >> I am thinking of organizing a panel this year for python-dev (much
>> >> like the one I organized in 2007). Who would be willing to be on the
>> >> panel with me if I did this?
>> >
>> > Could you explain what this is to us a little more, please? :)
>> >
>>
>> You sit in front of a bunch of people answering questions asked by the
>> audience. You know, a panel. =) It's just a Q&A session so that PyCon
>> attendees can ask python-dev a bunch of random questions. Demystifies
>> some things and puts faces to python-dev.
>
> From a non-core developer point of view:
>
> What could be great imho would be to have a short "How Python is developed"
> presentation
> just before the panel starts.
>

I was already planning on giving my "how Python is developed" talk
anyway, and I would do my best to make sure they were run
back-to-back.

-Brett

From brett at python.org  Sun Sep 28 02:24:57 2008
From: brett at python.org (Brett Cannon)
Date: Sat, 27 Sep 2008 17:24:57 -0700
Subject: [Python-Dev] PyCon 2009 Call for Proposals
In-Reply-To: <48DE25E7.9010302@trueblade.com>
References: <20080926232550.GA29031@panix.com>
	<bbaeab100809261936l3b417ab7yd17889b44bc39e2e@mail.gmail.com>
	<48DE25E7.9010302@trueblade.com>
Message-ID: <bbaeab100809271724kec8931dh47bf2331e90586b3@mail.gmail.com>

On Sat, Sep 27, 2008 at 5:24 AM, Eric Smith <eric at trueblade.com> wrote:
> Brett Cannon wrote:
>>
>> I am thinking of organizing a panel this year for python-dev (much
>> like the one I organized in 2007). Who would be willing to be on the
>> panel with me if I did this?
>
> If you're looking for the perspective of someone who's relatively new to
> Python core programming, I'll do it. You won't offend me by picking someone
> else, though.
>

I don't mind it, but it needs to be balanced. The panel should only be
4 - 6 people in the end and I would like a good balance of someone
from PythonLabs (covered by Barry), long-term, mid-term (covered by
me), and a newbie like yourself. =) I will see who steps forward and I
will pull 3-5 people from those who volunteer. So please speak up
still if you are interested in participating.

-Brett

From brett at python.org  Sun Sep 28 02:35:17 2008
From: brett at python.org (Brett Cannon)
Date: Sat, 27 Sep 2008 17:35:17 -0700
Subject: [Python-Dev] I would like an Python account
In-Reply-To: <200809271732.19317.victor.stinner@haypocalc.com>
References: <200809271732.19317.victor.stinner@haypocalc.com>
Message-ID: <bbaeab100809271735q71a41323i6296397b2c80f6bc@mail.gmail.com>

On Sat, Sep 27, 2008 at 8:32 AM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Hi,
>
> Would it possible to get more permissions on Python bugtracker, especially to
> add keywords, close a duplicate bug, etc.?
>
> I also would like an SVN account for Python trunk and Python 3000 branch. I
> know that both branches are near "frozen" and commit are only allowed after
> patch review and/or discussion (on the mailing list or the bug tracker. So I
> will after the final versions (2.6 and 3.0) to commit. Or would it be
> possible to have a "mentor"? Someone to ask questions about Python SVN/bug
> tracker, or someone who reviews my patchs/commits.

If you were given commit privs, Victor, everyone would be your mentor
to make sure you didn't mess up. =) And python-dev or
python-committers (which you will be subscribed to if you get the
privs) are always available to ask questions.

-Brett

>
> --
> Victor Stinner aka haypo
> http://www.haypocalc.com/blog/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/brett%40python.org
>

From brett at python.org  Sun Sep 28 02:45:14 2008
From: brett at python.org (Brett Cannon)
Date: Sat, 27 Sep 2008 17:45:14 -0700
Subject: [Python-Dev] Python security team
In-Reply-To: <200809271754.29068.victor.stinner@haypocalc.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
Message-ID: <bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>

On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Hi,
>
> I would like to know if a Python security team does exist. I sent an email
> about an imageop issue, and I didn't get any answer. Later I learned that a
> security ticket was created, I don't have access to it.
>

Yes, the PSRT (Python Security Response Team) does exist. We did get
your email; sorry we didn't respond. There are very few members on
that list and most of them are extremely busy. Responding to your
email just slipped through the cracks. I believe Benjamin was the last
person to work on your submitted patch.

> First, I would like to access to these informations. Not only this issue, but
> all security related issues. I have some knowledges about security and I can
> help to resolve issues and/or estimate the criticity of an issue.
>

That would require commit privileges first. Don't know if the group
requires that a person have a decent amount of time committing to the
core first (I just joined the list in late July).

> Second, I would like to help to fix all Python security issues. It looks like
> Python community isn't very reactive (proactive?) about security. Eg. a DoS
> was reported in smtpd server (integrated to Python)... 15 months ago. A patch
> is available but it's not applied in Python trunk.
>

Historically we have not been proactive. No one on the core team (that
I know of) would claim they are a security expert. And with Python not
making any claims to being secure, we just don't worry about DoS
stuff, etc.; only the severe buffer overflow attacks that get reported
and such typically get immediate attention. Considering we have a
Crashers directory in the test suite I think that shows we are not
stressed over plugging every potential crash (although we obviously
would like to).

> Third, I'm also looking for a document explaining "how Python is secure" (!).
> If an user can run arbitrary Python code, we know that it can do anything
> (read/remove any file, create/kill any process, read/write anywhere in
> memory, etc.). Brett wrote a paper about CPython sandboxing. PyPy is also
> working on sandboxing using two interpreters: one has high priviledge and
> execute instructions from the second interpreter (after checking the
> permissions and arguments). So is there somewhere a document to explain to
> current status of Python security?
>

Nope. I think my paper and blog posts are about the best you are going
to find since we removed Bastion/rexec. Basically the philosophy has
been "fix privilege escalation stuff immediately, fix crashers when
the fix is simple or someone has the time to fix the complicated
ones". Or at least that is the philosophy I personally have followed.

-Brett

From divinekid at gmail.com  Sun Sep 28 05:53:07 2008
From: divinekid at gmail.com (Haoyu Bai)
Date: Sun, 28 Sep 2008 11:53:07 +0800
Subject: [Python-Dev] call for help: C-API 2 to 3 porting guide
In-Reply-To: <48DE3089.7040308@cheimes.de>
References: <1afaf6160809261823w7e9698dfm799fa45976a82fcb@mail.gmail.com>
	<48DE3089.7040308@cheimes.de>
Message-ID: <1d7983e80809272053wa52f0dcq20fec7dbecc3cb29@mail.gmail.com>

On Sat, Sep 27, 2008 at 9:09 PM, Christian Heimes <lists at cheimes.de> wrote:
> Benjamin Peterson wrote:
>>
>> Hi guys,
>> I remember a while back we were bemoaning the fact that little
>> documentation exists for the extension module transition between 2.x
>> and 3.x. In the name of this effort, I've created a c porting howto
>> (Doc/howto/cporting.rst). I have started a few sections, but I can't
>> do it all alone. Please consider writing a section.
>
> I'll try to find some time over the weekend to assist you. I'm going to
>  contact the Cython developers, too. They have a great deal of experience
> with Python 2.x and 3.0 compatible C code. Perhaps they can give us some
> hints and tell us about gotchas and tricks.
>

Also, there's a wiki page for some py3k extension moduls migration
tips: http://wiki.python.org/moin/Py3kExtensionModules

Hope it helps.

-- 
Haoyu Bai

From steve at holdenweb.com  Sun Sep 28 15:39:40 2008
From: steve at holdenweb.com (Steve Holden)
Date: Sun, 28 Sep 2008 09:39:40 -0400
Subject: [Python-Dev] Python security team
In-Reply-To: <bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
Message-ID: <48DF891C.2070902@holdenweb.com>

Brett Cannon wrote:
> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
> <victor.stinner at haypocalc.com> wrote:
>> Hi,
>>
>> I would like to know if a Python security team does exist. I sent an email
>> about an imageop issue, and I didn't get any answer. Later I learned that a
>> security ticket was created, I don't have access to it.
>>
> 
> Yes, the PSRT (Python Security Response Team) does exist. We did get
> your email; sorry we didn't respond. There are very few members on
> that list and most of them are extremely busy. Responding to your
> email just slipped through the cracks. I believe Benjamin was the last
> person to work on your submitted patch.
> 
[...]

If we don't have a documented procedure, or if we do have a procedure
and it isn't being followed, we can't be said to be taking security
seriously, which I would find disappointing. This is one of the few
areas where we probably *do* need to be meticulous, and the absence of a
reply to a security report isn't really satisfactory.

Perhaps if the PSF does eventually hire some paid help, running the
secretarial and administrative portions of the security team would help
the busy members to avoid such issues dropping through the cracks in future.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From steve at holdenweb.com  Sun Sep 28 15:39:40 2008
From: steve at holdenweb.com (Steve Holden)
Date: Sun, 28 Sep 2008 09:39:40 -0400
Subject: [Python-Dev] Python security team
In-Reply-To: <bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
Message-ID: <48DF891C.2070902@holdenweb.com>

Brett Cannon wrote:
> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
> <victor.stinner at haypocalc.com> wrote:
>> Hi,
>>
>> I would like to know if a Python security team does exist. I sent an email
>> about an imageop issue, and I didn't get any answer. Later I learned that a
>> security ticket was created, I don't have access to it.
>>
> 
> Yes, the PSRT (Python Security Response Team) does exist. We did get
> your email; sorry we didn't respond. There are very few members on
> that list and most of them are extremely busy. Responding to your
> email just slipped through the cracks. I believe Benjamin was the last
> person to work on your submitted patch.
> 
[...]

If we don't have a documented procedure, or if we do have a procedure
and it isn't being followed, we can't be said to be taking security
seriously, which I would find disappointing. This is one of the few
areas where we probably *do* need to be meticulous, and the absence of a
reply to a security report isn't really satisfactory.

Perhaps if the PSF does eventually hire some paid help, running the
secretarial and administrative portions of the security team would help
the busy members to avoid such issues dropping through the cracks in future.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From victor.stinner at haypocalc.com  Sun Sep 28 17:04:52 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Sun, 28 Sep 2008 17:04:52 +0200
Subject: [Python-Dev] Python security team
In-Reply-To: <bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
Message-ID: <200809281704.52162.victor.stinner@haypocalc.com>

Le Sunday 28 September 2008 02:45:14 Brett Cannon, vous avez ?crit?:
> > Second, I would like to help to fix all Python security issues. It looks
> > like Python community isn't very reactive (proactive?) about security.
> > Eg. a DoS was reported in smtpd server (integrated to Python)... 15
> > months ago. A patch is available but it's not applied in Python trunk.
>
> Historically we have not been proactive.

Well, I just asked to know the current status. First step for a secure program 
is to know its limits ;)

> Nope. I think my paper and blog posts are about the best you are going
> to find

I will try to write a document about Python and security next week.


-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From victor.stinner at haypocalc.com  Sun Sep 28 17:14:48 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Sun, 28 Sep 2008 17:14:48 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48DE705E.6050405@v.loewis.de>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48DE705E.6050405@v.loewis.de>
Message-ID: <200809281714.48630.victor.stinner@haypocalc.com>

Le Saturday 27 September 2008 19:41:50 Martin v. L?wis, vous avez ?crit?:
> > I think that the problem is important because it's a regression from 2.5
> > to 2.6/3.0. Python 2.5 uses bytes filename, so it was possible to
> > open/unlink "invalid" unicode strings (since it's not unicode but bytes).
>
> I'd like to stress that the problem is *not* a regression from 2.5 to 2.6.

Sorry, 2.6 has no problem. This issue is a regression from Python2 to Python3.

> Even though you may run into file names that can't be decoded, 
> that happening really indicates some bigger problem in the management 
> of the system where this happens, and the proper solution (IMO) should 
> be to change the system

In the *real world*, people are using different file systems, different 
operations systems, and some broken programs and/or operating system create 
invalid filenames. It could be a configuration problem (wrong charset 
definition in /etc/fstab) or the charset autodetection failure, but who 
cares? Sometimes you don't care that your music directory contains some 
strange filenames, you just want to hear the music. Or maybe you would like 
to *fix* the encoding problem, which is not possible using Python3 trunk.

People having this problem are, for example, people who write or use a backup 
program. This week someone asked me (on IRC) how to manage filenames in pure 
unicode with python 2.5 and Linux... which was impossible because on of his 
filename was invalid (maybe a file from a Windows system). So he switched to 
raw (bytes) filenames.

In a perfect world, everybody uses Linux with utf-8 filenames and only 
programs in Python using space indentation :-D

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From greg at krypto.org  Sun Sep 28 22:34:50 2008
From: greg at krypto.org (Gregory P. Smith)
Date: Sun, 28 Sep 2008 13:34:50 -0700
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48DE705E.6050405@v.loewis.de>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48DE705E.6050405@v.loewis.de>
Message-ID: <52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>

On 9/27/08, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> I think that the problem is important because it's a regression from 2.5
>> to
>> 2.6/3.0. Python 2.5 uses bytes filename, so it was possible to
>> open/unlink "invalid" unicode strings (since it's not unicode but bytes).
>
> I'd like to stress that the problem is *not* a regression from 2.5 to 2.6.
>
> As for 3.0, I'd like to argue that the problem is a minor issue. Even
> though you may run into file names that can't be decoded, that happening
> really indicates some bigger problem in the management of the system
> where this happens, and the proper solution (IMO) should be to change
> the system (leaving open the question whether or not Python should
> be also changed to work with such broken systems).
>
> Regards,
> Martin

Note:  bcc python-dev,cc: python-3000

"broken" systems will always exist.  Code to deal with them must be
possible to write in python 3.0.

since any given path (not just fs) can have its own encoding it makes
the most sense to me to let the OS deal with the errors and not try to
enforce bytes vs string encoding type at the python lib. level.

-gps

From martin at v.loewis.de  Sun Sep 28 23:19:03 2008
From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=)
Date: Sun, 28 Sep 2008 23:19:03 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <fb73205e0809271613h5597e3cci1e24c7630840ad21@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>	<48DE705E.6050405@v.loewis.de>
	<fb73205e0809271613h5597e3cci1e24c7630840ad21@mail.gmail.com>
Message-ID: <48DFF4C7.2080504@v.loewis.de>

> I can't agree here. File handling is a fundamental operation and I
> would expect something like:
> 
>>>> for fname in os.listdir('.'):
> ...       if os.path.isfile(fname):
> ...          file(fname)
> 
> to work for all files.

I agree. However, if it fails: is the bug of the Python, or of the
system administrator maintaining it?

> To have to know to put in special handling for
> certain corner case filenames or worse to not be able to open some
> files at all would be a serious loss. It would also complicate
> migrating code correctly to 3.0.

I agree completely. Unfortunately, all proposed solutions *do* require
special handling for certain corner cases.

> Regardless of whose fault the underlying issue is, someone has to deal
> with the problem and if core Python doesn't, each developer who
> encounters the problem will have to come up with his/her own solution.

This is quite in the abstract. Can you be more specific?

Regards,
Martin

From brett at python.org  Mon Sep 29 00:43:08 2008
From: brett at python.org (Brett Cannon)
Date: Sun, 28 Sep 2008 15:43:08 -0700
Subject: [Python-Dev] Python security team
In-Reply-To: <48DF891C.2070902@holdenweb.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
	<48DF891C.2070902@holdenweb.com>
Message-ID: <bbaeab100809281543s2403b852t38aa659dc0bd7bec@mail.gmail.com>

On Sun, Sep 28, 2008 at 6:39 AM, Steve Holden <steve at holdenweb.com> wrote:
> Brett Cannon wrote:
>> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
>> <victor.stinner at haypocalc.com> wrote:
>>> Hi,
>>>
>>> I would like to know if a Python security team does exist. I sent an email
>>> about an imageop issue, and I didn't get any answer. Later I learned that a
>>> security ticket was created, I don't have access to it.
>>>
>>
>> Yes, the PSRT (Python Security Response Team) does exist. We did get
>> your email; sorry we didn't respond. There are very few members on
>> that list and most of them are extremely busy. Responding to your
>> email just slipped through the cracks. I believe Benjamin was the last
>> person to work on your submitted patch.
>>
> [...]
>
> If we don't have a documented procedure, or if we do have a procedure
> and it isn't being followed, we can't be said to be taking security
> seriously, which I would find disappointing. This is one of the few
> areas where we probably *do* need to be meticulous, and the absence of a
> reply to a security report isn't really satisfactory.
>
> Perhaps if the PSF does eventually hire some paid help, running the
> secretarial and administrative portions of the security team would help
> the busy members to avoid such issues dropping through the cracks in future.
>

That actually would be extremely beneficial since as right now a big
problem we have is writing up the official announcement that some
security issue has been plugged and then sticking up the patches
online for people to download.

-Brett

From dirkjan at ochtman.nl  Mon Sep 29 00:20:28 2008
From: dirkjan at ochtman.nl (Dirkjan Ochtman)
Date: Mon, 29 Sep 2008 00:20:28 +0200
Subject: [Python-Dev] Reminder: Python 2.6 final next Wednesday
In-Reply-To: <AA1F5430-661B-46AE-86C8-3E9CC9CE26C8@python.org>
References: <AA1F5430-661B-46AE-86C8-3E9CC9CE26C8@python.org>
Message-ID: <48E0032C.4070604@ochtman.nl>

Barry Warsaw wrote:
> Once again, I've gone through the release blocker issues and knocked 
> anything that doesn't specifically affect 2.6 to deferred blocker.  This 
> leaves us with  7 open blocking issues.

I noticed you deferred issue3723. I wonder how this "doesn't affect" 2.6 
when it appears I can't even use mod_wsgi (for Apache) with 2.6 if this 
doesn't get fixed. It's a pretty bad regression, right?

Cheers,

Dirkjan

From musiccomposition at gmail.com  Mon Sep 29 02:25:58 2008
From: musiccomposition at gmail.com (Benjamin Peterson)
Date: Sun, 28 Sep 2008 19:25:58 -0500
Subject: [Python-Dev] Reminder: Python 2.6 final next Wednesday
In-Reply-To: <48E0032C.4070604@ochtman.nl>
References: <AA1F5430-661B-46AE-86C8-3E9CC9CE26C8@python.org>
	<48E0032C.4070604@ochtman.nl>
Message-ID: <1afaf6160809281725x215e6939w1eafb8b0bea2bc1f@mail.gmail.com>

On Sun, Sep 28, 2008 at 5:20 PM, Dirkjan Ochtman <dirkjan at ochtman.nl> wrote:
> Barry Warsaw wrote:
>>
>> Once again, I've gone through the release blocker issues and knocked
>> anything that doesn't specifically affect 2.6 to deferred blocker.  This
>> leaves us with  7 open blocking issues.
>
> I noticed you deferred issue3723. I wonder how this "doesn't affect" 2.6
> when it appears I can't even use mod_wsgi (for Apache) with 2.6 if this
> doesn't get fixed. It's a pretty bad regression, right?

It doesn't affect 2.6. Only 3.0.

-- 
Cheers,
Benjamin Peterson
"There's no place like 127.0.0.1."

From eckhardt at satorlaser.com  Mon Sep 29 09:39:59 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Mon, 29 Sep 2008 09:39:59 +0200
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <48DD52E7.8060408@v.loewis.de>
References: <200809231052.36720.eckhardt@satorlaser.com> 
	<200809260921.36012.eckhardt@satorlaser.com> 
	<48DD52E7.8060408@v.loewis.de>
Message-ID: <200809290939.59963.eckhardt@satorlaser.com>

On Friday 26 September 2008, Martin v. L?wis wrote:
> >> Please don't. Whether or not _UNICODE is defined should have no effect.
> >
> > Well, currently it does make a difference. Simple example: CreateFile().
>
> It's not so simple: Python doesn't actually call CreateFile, AFAICT
> (well, it does, in _msi.c, but I hope you aren't worried about that
> module on CE).

Martin, CreateFile() was just used as an example. You can substitute it with 
LoadString() or CreateProcess() if you like, the problem remains the same.


[about using SCons for building]
> And you *can* provide an SCons file that supports all the SDKs?

No, but I can provide one that allows parametrisation. ;)

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From mithrandi-python-dev at mithrandi.za.net  Mon Sep 29 11:21:11 2008
From: mithrandi-python-dev at mithrandi.za.net (Tristan Seligmann)
Date: Mon, 29 Sep 2008 11:21:11 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48DE705E.6050405@v.loewis.de>
	<52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>
Message-ID: <20080929092110.GB4469@mithrandi.net>

* Gregory P. Smith <greg at krypto.org> [2008-09-28 13:34:50 -0700]:

> since any given path (not just fs) can have its own encoding it makes
> the most sense to me to let the OS deal with the errors and not try to
> enforce bytes vs string encoding type at the python lib. level.

But the underlying APIs differ; Linux uses bytestrings for filenames,
whereas I believe the native Windows APIs take "wide" (ie. Unicode)
strings.
-- 
mithrandi, i Ainil en-Balandor, a faer Ambar
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080929/7d5f719b/attachment.pgp>

From eckhardt at satorlaser.com  Mon Sep 29 12:50:03 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Mon, 29 Sep 2008 12:50:03 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com> 
	<48DE705E.6050405@v.loewis.de> 
	<52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>
Message-ID: <200809291250.03291.eckhardt@satorlaser.com>

On Sunday 28 September 2008, Gregory P. Smith wrote:
> "broken" systems will always exist.  Code to deal with them must be
> possible to write in python 3.0.
>
> since any given path (not just fs) can have its own encoding it makes
> the most sense to me to let the OS deal with the errors and not try to
> enforce bytes vs string encoding type at the python lib. level.

Actually I'm afraid that that isn't really useful. I, too, would like to kick 
peoples' back in order to get the to fix their systems or use the proper 
codepage while mounting etc, etc, but that is not going to happen soon. Just 
ignoring those broken systems is tempting, but alienating a large group of 
users isn't IMHO worth it.

Instead, I'd like to present a different approach:

1. For POSIX platforms (using a byte string for the path):
Here, the first approach is to convert the path to Unicode, according to the 
locale's CTYPE category. Hopefully, it will be UTF-8, but also codepages 
should work. If there is a segment (a byte sequence between two path 
separators) where it doesn't work, it uses an ASCII mapping where possible 
and codepoints from the "Private Use Area" (PUA) of Unicode for the 
non-decodable bytes.
In order to pass this path to fopen(), each segment would be converted to a 
byte string again, using the locale's CTYPE category except for segments 
which use the PUA where it simply encodes the original bytes.

2. For win32 platforms, the path is already Unicode (UTF-16) and the whole 
problem is solved or not solved by the OS.

In the end, both approaches yield a path represented by a Unicode string for 
intermediate use, which provides maximum flexibility. Further, it 
preserves "broken" encodings by simply mapping their byte-values to the PUA 
of Unicode. Maybe not using a string to represent a path would be a good 
idea, too. At least it would make it very clear that the string is not 
completely free-form.

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From mal at egenix.com  Mon Sep 29 13:16:11 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Mon, 29 Sep 2008 13:16:11 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809291250.03291.eckhardt@satorlaser.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48DE705E.6050405@v.loewis.de>
	<52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>
	<200809291250.03291.eckhardt@satorlaser.com>
Message-ID: <48E0B8FB.9070701@egenix.com>

On 2008-09-29 12:50, Ulrich Eckhardt wrote:
> On Sunday 28 September 2008, Gregory P. Smith wrote:
>> "broken" systems will always exist.  Code to deal with them must be
>> possible to write in python 3.0.
>>
>> since any given path (not just fs) can have its own encoding it makes
>> the most sense to me to let the OS deal with the errors and not try to
>> enforce bytes vs string encoding type at the python lib. level.
> 
> Actually I'm afraid that that isn't really useful. I, too, would like to kick 
> peoples' back in order to get the to fix their systems or use the proper 
> codepage while mounting etc, etc, but that is not going to happen soon. Just 
> ignoring those broken systems is tempting, but alienating a large group of 
> users isn't IMHO worth it.
> 
> Instead, I'd like to present a different approach:
> 
> 1. For POSIX platforms (using a byte string for the path):
> Here, the first approach is to convert the path to Unicode, according to the 
> locale's CTYPE category. Hopefully, it will be UTF-8, but also codepages 
> should work. If there is a segment (a byte sequence between two path 
> separators) where it doesn't work, it uses an ASCII mapping where possible 
> and codepoints from the "Private Use Area" (PUA) of Unicode for the 
> non-decodable bytes.
> In order to pass this path to fopen(), each segment would be converted to a 
> byte string again, using the locale's CTYPE category except for segments 
> which use the PUA where it simply encodes the original bytes.

I'm not sure how this would work. How would you map the private use
code points back to bytes ? Using a special codec that knows about
these code points ? How would the fopen() know to use that special
codec instead of e.g. the UTF-8 codec ?

BTW: Private use areas in Unicode are meant for e.g. company specific
code points. Using them for escaping purposes is likely to cause problems
due to assignment clashes.

Regarding the subject of file names:

On Unix, it's well possible to have to deal with 2-3 different file
systems mounted on a machine. Each of those may use a different file name
encoding or not support file name encoding at all.

If the OS doesn't guarantee a consistent file name encoding, then
why should Python try to emulate this on top of the OS ?

I think it's more important to be able to open a file, than to have
a readable file name when printing it to stdout, e.g. I wouldn't be able
to tell whether some Chinese file name makes sense or not, but if I know
that all files in a directory are meant for processing I should be able
to iterate over them regardless of whether they make sense or not.

> 2. For win32 platforms, the path is already Unicode (UTF-16) and the whole 
> problem is solved or not solved by the OS.
> 
> In the end, both approaches yield a path represented by a Unicode string for 
> intermediate use, which provides maximum flexibility. Further, it 
> preserves "broken" encodings by simply mapping their byte-values to the PUA 
> of Unicode. Maybe not using a string to represent a path would be a good 
> idea, too. At least it would make it very clear that the string is not 
> completely free-form.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 29 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From glyph at divmod.com  Mon Sep 29 13:28:51 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Mon, 29 Sep 2008 11:28:51 -0000
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809291250.03291.eckhardt@satorlaser.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48DE705E.6050405@v.loewis.de>
	<52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>
	<200809291250.03291.eckhardt@satorlaser.com>
Message-ID: <20080929112851.31635.999142795.divmod.xquotient.274@weber.divmod.com>

On 10:50 am, eckhardt at satorlaser.com wrote:
>On Sunday 28 September 2008, Gregory P. Smith wrote:
>>"broken" systems will always exist.  Code to deal with them must be
>>possible to write in python 3.0.

>>since any given path (not just fs) can have its own encoding it makes
>>the most sense to me to let the OS deal with the errors and not try to
>>enforce bytes vs string encoding type at the python lib. level.

>Actually I'm afraid that that isn't really useful. I, too, would like 
>to kick
>peoples' back in order to get the to fix their systems or use the 
>proper
>codepage while mounting etc, etc, but that is not going to happen soon. 
>Just
>ignoring those broken systems is tempting, but alienating a large group 
>of
>users isn't IMHO worth it.
>
>Instead, I'd like to present a different approach:

>1. For POSIX platforms (using a byte string for the path):
>Here, the first approach is to convert the path to Unicode, according 
>to the
>locale's CTYPE category. Hopefully, it will be UTF-8, but also 
>codepages
>should work. If there is a segment (a byte sequence between two path
>separators) where it doesn't work, it uses an ASCII mapping where 
>possible
>and codepoints from the "Private Use Area" (PUA) of Unicode for the
>non-decodable bytes.

>In order to pass this path to fopen(), each segment would be converted 
>to a
>byte string again, using the locale's CTYPE category except for 
>segments
>which use the PUA where it simply encodes the original bytes.

That's a cool idea, but this encoding hack would need to be clearly 
documented and exposed for when you need to talk to another piece of 
software about pathnames.  Consider a Python implementation of "xargs". 
Right now this can be implemented as a pretty simple for loop which 
eventually invokes 'subprocess.call' or similar.

http://docs.python.org/dev/3.0/library/os.html#process-management 
doesn't say what the type of the arguments to the various 'exec' 
variants are - one presumes they'd have to be bytes.  Not all arguments 
to subprocesses need to be filenames, but when they are they need to be 
encoded appropriately.

Also, consider the following nightmare scenario: a system which has two 
users with incompatible locales.  One wishes to write a "text" (ha ha) 
file with a list of pathnames in it to share with the other.  What 
encoding should that file be in?  How should the other user know how to 
interpret it?  (And of course: what if that user is going to be piping 
that file to "xargs", or the original file came out of "find"?)  I don't 
think that you can do encoding a segment at a time here, at least not at 
the API level; however, the whole file could be written in the py-posix- 
paths encoding which does exactly what you propose.
>2. For win32 platforms, the path is already Unicode (UTF-16) and the 
>whole
>problem is solved or not solved by the OS.

If the "or not solved" part of that is true then this probably bears 
further investigation.  I suspect that the OS *always* provides some 
solution, even if it's the wrong solution, though.

Also, what about MacOS X?
>In the end, both approaches yield a path represented by a Unicode 
>string for
>intermediate use, which provides maximum flexibility. Further, it
>preserves "broken" encodings by simply mapping their byte-values to the 
>PUA
>of Unicode. Maybe not using a string to represent a path would be a 
>good
>idea, too. At least it would make it very clear that the string is not
>completely free-form.

Personally, I plan to use this:

http://twistedmatrix.com/documents/8.1.0/api/twisted.python.filepath.FilePath.html

for all of my file I/O in the future.

For what it's worth, this object _doesn't_ handle unicode properly and 
it's been a thorn in our side for quite a while.  We have plans to 
implement some kind of unicode-friendly API which is compatible with 
2.6; if we have any brilliant ideas I'll let you know, but I doubt 
they'll be in time.

The general idea right now is that we'll keep around the original bytes 
returned from filesystem inspection and provide some context-sensitive 
encoding/decoding APIs for different applications.  The PUA approach 
would allow us to maintain an API compatible with that.  I would not 
actually mind if there were a POSIX-specific module we had to use to get 
every arcane nuance of brokenness of writing pathnames into text files 
to be correct, since Windows needs to come up with _some_ valid unicode 
filename for every file in the system (even if it's improperly decoded).

From victor.stinner at haypocalc.com  Mon Sep 29 13:37:29 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Mon, 29 Sep 2008 13:37:29 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809291250.03291.eckhardt@satorlaser.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>
	<200809291250.03291.eckhardt@satorlaser.com>
Message-ID: <200809291337.29898.victor.stinner@haypocalc.com>

Le Monday 29 September 2008 12:50:03 Ulrich Eckhardt, vous avez ?crit?:
> (...) uses an ASCII mapping where possible and codepoints from the 
> "Private Use Area" (PUA) of Unicode for the non-decodable bytes.

That sounds to me like a very *ugly* hack.

It remembers me my proposition to create an object have the API of both bytes 
and str types: str(<Filename object>) = human representation of the filename, 
bytes(<Filename>) = original bytes filename. As I wrote in the first email of 
this thread, it's not a good idea to mix bytes and characters.

Why trying to convert bytes to characters when the operating system expects 
bytes? To get the best compatibility, we have to use the native types, at 
least when str(filename, fs_encoding) fails and/or str(filename, 
fs_encoding).encode(fs_encoding) != filename.

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From eckhardt at satorlaser.com  Mon Sep 29 13:59:06 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Mon, 29 Sep 2008 13:59:06 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48E0B8FB.9070701@egenix.com>
References: <200809271404.25654.victor.stinner@haypocalc.com> 
	<200809291250.03291.eckhardt@satorlaser.com> 
	<48E0B8FB.9070701@egenix.com>
Message-ID: <200809291359.06334.eckhardt@satorlaser.com>

On Monday 29 September 2008, M.-A. Lemburg wrote:
> On 2008-09-29 12:50, Ulrich Eckhardt wrote:
> > 1. For POSIX platforms (using a byte string for the path):
> > Here, the first approach is to convert the path to Unicode, according to
> > the locale's CTYPE category. Hopefully, it will be UTF-8, but also
> > codepages should work. If there is a segment (a byte sequence between two
> > path separators) where it doesn't work, it uses an ASCII mapping where
> > possible and codepoints from the "Private Use Area" (PUA) of Unicode for
> > the non-decodable bytes.
> > In order to pass this path to fopen(), each segment would be converted to
> > a byte string again, using the locale's CTYPE category except for
> > segments which use the PUA where it simply encodes the original bytes.
>
> I'm not sure how this would work. How would you map the private use
> code points back to bytes ? Using a special codec that knows about
> these code points ? How would the fopen() know to use that special
> codec instead of e.g. the UTF-8 codec ?

Sorry, I wasn't clear enough. I'll try to explain further...

Let's assume we have a filename like this:

  0xc2 0xa9 0x2f 0x7f

The first two bytes are the copyright sign encoded in UTF-8, followed by a 
slash (0x2f, path separator) and a character encoded in an unknown codepage 
(0x7f is not ASCII!). The first thing when receiving that path from the 
system would be to split it into segments, here we would get two of them, one 
with 0xc2 0xa9 and the other with 0x7f. This uses the fact that the separator 
(slash/0x2f) is rather universal (Note: I'm not sure about encodings like 
BIG5, i.e. ones that are neither UTF-8 nor derived from ASCII).

For each segment, we would apply the locale's CTYPE facet and get the Unicode 
codepoint 0xa9 for the first segment, while the second one fails to convert. 
So, for the second one, we simply check for each byte if it is valid and 
printable ASCII (0x7f isn't). If it is, we emit the byte as Unicode 
codepoint. Otherwise, we map to the PUA.

The PUA reserves 0xe000 to 0xf8ff for private uses. I would simply encode the 
byte 0x7f as 0xe07f, i.e. map it to the beginning of that range. Eventually, 
we would end up with the following Unicode codepoints:

  0xa9, 0x2f, 0xe07f

When converting to a byte string for use with fopen(), we simply inspect the 
supplied string again. If a segment contains elements of the PUA, we simply 
reverse the mapping for those and leave the others in that segment as-is. For 
all other segments, we apply the CTYPE conversion.


Notes:
* This effectively converts the current path representation (a string) into a 
sequence of segments where each segment can either be a fully Unicode-capable 
string or a raw byte string without any known interpretation. However, 
instead of using an array for that, it uses a string, which is what most 
people's code expects anyway.
* You could also work on a byte-base instead of splitting the path in segments 
first. I just assumed that a single segment will not contain valid UTF-8 
sequences mixed with invalid ones. A path however can contain both correctly 
and incorrectly encoded segments.


> BTW: Private use areas in Unicode are meant for e.g. company specific
> code points. Using them for escaping purposes is likely to cause problems
> due to assignment clashes.

I'm not sure if the use I proposed is correct according to the intended use of 
the PUA. I know that ideally no such string would escape from Python, i.e. it 
should only be visible internally. I would guess that that is something the 
PUA was intended for.

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From victor.stinner at haypocalc.com  Mon Sep 29 14:07:55 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Mon, 29 Sep 2008 14:07:55 +0200
Subject: [Python-Dev] New proposition for Python3 bytes filename issue
Message-ID: <200809291407.55291.victor.stinner@haypocalc.com>

Hi,

After reading the previous discussion, here is new proposition.

Python 2.x and Windows are not affected by this issue. Only Python3 on POSIX 
(eg. Linux or *BSD) is affected.

Some system are broken, but Python have to be able to open/copy/move/remove 
files with an "invalid filename".

The issue can wait for Python 3.0.1 / 3.1.

Windows
-------

On Windows, we might reject bytes filenames for all file operations: open(), 
unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError)

POSIX OS
--------

The default behaviour should be to use unicode and raise an error if 
conversion to unicode fails. It should also be possible to use bytes using 
bytes arguments and optional arguments (for getcwd).

 - listdir(unicode) -> unicode and raise an error on invalid filename
 - listdir(bytes) -> bytes
 - getcwd() -> unicode
 - getcwd(bytes=True) -> bytes
 - open(): accept bytes or unicode

os.path.*() should accept operations on bytes filenames, but maybe not on 
bytes+unicode arguments. os.path.join('directory', b'filename'): raise an 
error (or use *implicit* conversion to bytes)?

When the user wants to display a filename to the screen, he can uses:
   text = str(filename, fs_encoding, "replace")

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From victor.stinner at haypocalc.com  Mon Sep 29 14:12:07 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Mon, 29 Sep 2008 14:12:07 +0200
Subject: [Python-Dev] [Python-3000] Filename as byte string in python
	2.6 or 3.0?
In-Reply-To: <96AAA50A-8C20-4320-A3C7-58B4C33D091D@fuhm.net>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<52dc1c820809281621l3beb260ahec22988a05e74327@mail.gmail.com>
	<96AAA50A-8C20-4320-A3C7-58B4C33D091D@fuhm.net>
Message-ID: <200809291412.07840.victor.stinner@haypocalc.com>

Le Monday 29 September 2008 06:43:55, vous avez ?crit?:
> It will make users happy, and it's simple enough to implement for
> python 3.0.

I dislike your argument. A "quick and dirty hack" is always faster to 
implement than a real solution, but we may hits later new issues if we don't 
choose the right solution.

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From jan.matejek at novell.com  Mon Sep 29 14:11:55 2008
From: jan.matejek at novell.com (Jan Matejek)
Date: Mon, 29 Sep 2008 14:11:55 +0200
Subject: [Python-Dev] Python security team
In-Reply-To: <bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
Message-ID: <48E0C60B.5060006@novell.com>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Brett Cannon napsal(a):
> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
> <victor.stinner at haypocalc.com> wrote:
>> First, I would like to access to these informations. Not only this issue, but
>> all security related issues. I have some knowledges about security and I can
>> help to resolve issues and/or estimate the criticity of an issue.
>>
> 
> That would require commit privileges first. Don't know if the group
> requires that a person have a decent amount of time committing to the
> core first (I just joined the list in late July).

commit privileges?
I would be interested in joining the PSRT list too - as a python
maintainer for openSUSE, i think that it would be beneficial for both my
and your work. And i can imagine that maintainers from other
distributions have similar opinion on this ;)
And that does not necessarily mean commit privileges, right?

Or is this an issue of trust, where "we trust you enough to make changes
to the core" equals "we also trust you enough to see the security issues" ?

regards
jan matejek
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkjgxgsACgkQjBrWA+AvBr+8IACfdh6ia9btlB4YrD+FI49CI5rv
8PcAoKQJVdie4YKDzLxaJCE33/TakcdW
=Y8Ck
-----END PGP SIGNATURE-----

From Jack.Jansen at cwi.nl  Mon Sep 29 14:24:11 2008
From: Jack.Jansen at cwi.nl (Jack Jansen)
Date: Mon, 29 Sep 2008 14:24:11 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809291250.03291.eckhardt@satorlaser.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48DE705E.6050405@v.loewis.de>
	<52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>
	<200809291250.03291.eckhardt@satorlaser.com>
Message-ID: <C823EDDC-BE02-413A-B427-83CAF8A4E42D@cwi.nl>

I'm a bit late to join in this discussion, but if unicode filenames  
are going to be the normal mode, how about this whole normalized/ 
canonical business?

This is a headache that has shown up on the Mac a couple of times,  
because MacOS prefers filenames to be NFC, whereas Python prefers its  
Unicode to be NFD (or the other way around, I keep forgetting the  
details).

To make the problem worse, even though MacOS prefers its filenames in  
the one form, it will allow filenames in the other form (which can  
happen if you mount a foreign filesystem, for example over the net).  
The fact that "incorrect" filenames can exist mean that the simple  
solution of converting NFC<->NFD in Python's open() and friends won't  
work (or, at least, it'll make some filenames inaccessible, and  
listdir() may return filenames that don't exist).


--
Jack Jansen, <Jack.Jansen at cwi.nl>, http://www.cwi.nl/~jack
If I can't dance I don't want to be part of your revolution -- Emma  
Goldman



From eckhardt at satorlaser.com  Mon Sep 29 14:34:07 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Mon, 29 Sep 2008 14:34:07 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <20080929112851.31635.999142795.divmod.xquotient.274@weber.divmod.com>
References: <200809271404.25654.victor.stinner@haypocalc.com> 
	<200809291250.03291.eckhardt@satorlaser.com> 
	<20080929112851.31635.999142795.divmod.xquotient.274@weber.divmod.com>
Message-ID: <200809291434.07263.eckhardt@satorlaser.com>

On Monday 29 September 2008, glyph at divmod.com wrote:
> Also, what about MacOS X?

AFAIK, OS X guarantees UTF-8 for filesystem encodings. So the OS also provides 
Unicode filenames and how it deals with broken or legacy media is left up to 
the OS.

Uli


-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From exarkun at divmod.com  Mon Sep 29 14:46:27 2008
From: exarkun at divmod.com (Jean-Paul Calderone)
Date: Mon, 29 Sep 2008 08:46:27 -0400
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809291434.07263.eckhardt@satorlaser.com>
Message-ID: <20080929124627.29191.636938224.divmod.quotient.31787@ohm>

On Mon, 29 Sep 2008 14:34:07 +0200, Ulrich Eckhardt <eckhardt at satorlaser.com> wrote:
>On Monday 29 September 2008, glyph at divmod.com wrote:
>> Also, what about MacOS X?
>
>AFAIK, OS X guarantees UTF-8 for filesystem encodings. So the OS also provides
>Unicode filenames and how it deals with broken or legacy media is left up to
>the OS.

Read Jack Jansen's recent email about NFC vs NFD.

Jean-Paul

From victor.stinner at haypocalc.com  Mon Sep 29 15:23:04 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Mon, 29 Sep 2008 15:23:04 +0200
Subject: [Python-Dev] New proposition for Python3 bytes filename issue
In-Reply-To: <200809291407.55291.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
Message-ID: <200809291523.04421.victor.stinner@haypocalc.com>

Patches are already avaible in the issue #3187 (os.listdir):

Le Monday 29 September 2008 14:07:55 Victor Stinner, vous avez ?crit?:
>  - listdir(unicode) -> unicode and raise an error on invalid filename

Need raise_decoding_errors.patch (don't clear Unicode error

>  - listdir(bytes) -> bytes

Always working.

>  - getcwd() -> unicode
>  - getcwd(bytes=True) -> bytes

Need merge_os_getcwd_getcwdu.patch

Note that current implement of getcwd() uses PyUnicode_FromString() to encode 
the directory, whereas getcwdu() uses the correct code (PyUnicode_Decode). So 
I merged both functions to keep only the correct version: getcwdu() => 
getcwd().

>  - open(): accept bytes or unicode

Need io_byte_filename.patch (just remove a check)

> os.path.*() should accept operations on bytes filenames, but maybe not on
> bytes+unicode arguments. os.path.join('directory', b'filename'): raise an
> error (or use *implicit* conversion to bytes)?

os.path.join() already reject mixing bytes + str.

But os.path.join(), glob.glob(), fnmatch.*(), etc. doesn't support bytes. I 
wrote some patches like:
 - glob1_bytes.patch: Fix glob.glob() to accept invalid directory name
 - fnmatch_bytes.patch: Patch fnmatch.filter() to accept bytes filenames

But I dislike both patches since they mix bytes and str. So this part still 
need some work.

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From glyph at divmod.com  Mon Sep 29 16:01:33 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Mon, 29 Sep 2008 14:01:33 -0000
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809291359.06334.eckhardt@satorlaser.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<48E0B8FB.9070701@egenix.com>
	<200809291359.06334.eckhardt@satorlaser.com>
Message-ID: <20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>


On 11:59 am, eckhardt at satorlaser.com wrote:
>Sorry, I wasn't clear enough. I'll try to explain further...
>
>Let's assume we have a filename like this:
>
>  0xc2 0xa9 0x2f 0x7f
>
>The first two bytes are the copyright sign encoded in UTF-8, followed 
>by a
>slash (0x2f, path separator) and a character encoded in an unknown 
>codepage
>(0x7f is not ASCII!).

Originally I thought that this was a valid idea, but then it became 
clear that this could be a problem.  Consider a filename which includes 
a UTF-8 encoding of a PUA code point.
>I'm not sure if the use I proposed is correct according to the intended 
>use of
>the PUA. I know that ideally no such string would escape from Python, 
>i.e. it
>should only be visible internally. I would guess that that is something 
>the
>PUA was intended for.

Viewing the PUA with GNOME charmap, I can see that many code points 
there have character renderings on my Ubuntu system.  I have to assume, 
therefore, that there are other (and potentially conflicting) uses for 
this unicode feature.

From michele.simionato at gmail.com  Mon Sep 29 16:12:13 2008
From: michele.simionato at gmail.com (Michele Simionato)
Date: Mon, 29 Sep 2008 16:12:13 +0200
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <48CD7B2B.2090000@v.loewis.de>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
	<48CD77F3.7020009@v.loewis.de> <20080914205225.GA15493@idyll.org>
	<48CD7B2B.2090000@v.loewis.de>
Message-ID: <4edc17eb0809290712x62275543uf9ee9ff8c95744f3@mail.gmail.com>

I like Martin's proposals (use a function, remove -O) very much.
Actually I wanted
to propose the same months ago. Here is my take at the assert function, which
I would like to be able to raise even exceptions different from AssertionError:

def assert_(cond, exc, *args):
    """Raise an exception if cond is not satisfied. exc can be a template
    string (then args are the interpolation arguments) or an exception
    class (then args are passed to the constructor). Here are a few
    examples:

    >>> assert_(False, 'ahia!')
    Traceback (most recent call last):
       ...
    AssertionError: ahia!

    >>> assert_(False, ValueError)
    Traceback (most recent call last):
      ...
    ValueError

    >>> a = 1
    >>> assert_(isinstance(a, str), TypeError, '%r is not a string' % a)
    Traceback (most recent call last):
    TypeError: 1 is not a string

    """
    if isinstance(exc, basestring):
        raise AssertionError(exc % args)
    elif inspect.isclass(exc) and issubclass(exc, Exception):
        raise exc(*args)
    else:
        raise TypeError('The second argument of assert_ must be a string '
        'or an exception class, not %r' % exc)

 M. Simionato

From eckhardt at satorlaser.com  Mon Sep 29 17:00:38 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Mon, 29 Sep 2008 17:00:38 +0200
Subject: [Python-Dev] Broken link to NASM download location
Message-ID: <200809291700.38946.eckhardt@satorlaser.com>

Hi!

In trunk/PCbuild/readme.txt it says NASM is available from kernel.org. The 
project has moved to Sourceforge though, please replace the link with

  http://nasm.sf.net

Thanks!

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From steven.bethard at gmail.com  Mon Sep 29 17:16:47 2008
From: steven.bethard at gmail.com (Steven Bethard)
Date: Mon, 29 Sep 2008 09:16:47 -0600
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <200809291407.55291.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
Message-ID: <d11dcfba0809290816k40b75e84pf6df8cfd6263ab15@mail.gmail.com>

On Mon, Sep 29, 2008 at 6:07 AM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> The default behaviour should be to use unicode and raise an error if
> conversion to unicode fails. It should also be possible to use bytes using
> bytes arguments and optional arguments (for getcwd).
>
>  - listdir(unicode) -> unicode and raise an error on invalid filename
>  - listdir(bytes) -> bytes
>  - getcwd() -> unicode
>  - getcwd(bytes=True) -> bytes

Please let's not introduce boolean flags like this. How about
``getcwdb`` in parallel with the old ``getcwdu``?

Steve
-- 
I'm not *in*-sane. Indeed, I am so far *out* of sane that you appear a
tiny blip on the distant coast of sanity.
        --- Bucky Katt, Get Fuzzy

From victor.stinner at haypocalc.com  Mon Sep 29 18:00:18 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Mon, 29 Sep 2008 18:00:18 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <d11dcfba0809290816k40b75e84pf6df8cfd6263ab15@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<d11dcfba0809290816k40b75e84pf6df8cfd6263ab15@mail.gmail.com>
Message-ID: <200809291800.18911.victor.stinner@haypocalc.com>

Le Monday 29 September 2008 17:16:47 Steven Bethard, vous avez ?crit?:
> >  - getcwd() -> unicode
> >  - getcwd(bytes=True) -> bytes
>
> Please let's not introduce boolean flags like this. How about
> ``getcwdb`` in parallel with the old ``getcwdu``?

Yeah, you're right. So i wrote a new patch: os_getcwdb.patch

With my patch we get (Python3):
 * os.getcwd() -> unicode
 * os.getcwdb() -> bytes

Previously in Python2 it was:
 * os.getcwd() -> str (bytes)
 * os.getcwdu() -> unicode

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From dstanek at dstanek.com  Mon Sep 29 18:19:04 2008
From: dstanek at dstanek.com (David Stanek)
Date: Mon, 29 Sep 2008 12:19:04 -0400
Subject: [Python-Dev] Python security team
In-Reply-To: <bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
Message-ID: <de32cc030809290919q2f05a9b3s2e6b4d5a45550211@mail.gmail.com>

On Sat, Sep 27, 2008 at 8:45 PM, Brett Cannon <brett at python.org> wrote:
> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
> <victor.stinner at haypocalc.com> wrote:
>>
>> I would like to know if a Python security team does exist. I sent an email
>> about an imageop issue, and I didn't get any answer. Later I learned that a
>> security ticket was created, I don't have access to it.
>>
>
> Yes, the PSRT (Python Security Response Team) does exist. We did get
> your email; sorry we didn't respond. There are very few members on
> that list and most of them are extremely busy. Responding to your
> email just slipped through the cracks. I believe Benjamin was the last
> person to work on your submitted patch.
>

I would be interested in participating. Is there any documentation
about the team or the processes? My Google search just turned up a
bunch of mailing list posts looking for team members.

-- 
David
http://www.traceback.org

From g.brandl at gmx.net  Mon Sep 29 18:51:55 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 29 Sep 2008 18:51:55 +0200
Subject: [Python-Dev] Broken link to NASM download location
In-Reply-To: <200809291700.38946.eckhardt@satorlaser.com>
References: <200809291700.38946.eckhardt@satorlaser.com>
Message-ID: <gbr142$jif$1@ger.gmane.org>

Ulrich Eckhardt schrieb:
> Hi!
> 
> In trunk/PCbuild/readme.txt it says NASM is available from kernel.org. The 
> project has moved to Sourceforge though, please replace the link with
> 
>   http://nasm.sf.net

Fixed in r66681. Thanks! (In the future, it might be better to open an issue
at bugs.python.org for this kind of reports since it can't get lost so easily ;)

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From guido at python.org  Mon Sep 29 19:06:01 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 29 Sep 2008 10:06:01 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbr0nv$iqu$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
Message-ID: <ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>

> Victor Stinner schrieb:

(Thanks Victor for moving this to the list. Having a discussion in the
tracker is really painful, I find.)

>> POSIX OS
>> --------
>>
>> The default behaviour should be to use unicode and raise an error if
>> conversion to unicode fails. It should also be possible to use bytes using
>> bytes arguments and optional arguments (for getcwd).
>>
>>  - listdir(unicode) -> unicode and raise an error on invalid filename

I know I keep flipflopping on this one, but the more I think about it
the more I believe it is better to drop those names than to raise an
exception. Otherwise a "naive" program that happens to use
os.listdir() can be rendered completely useless by a single non-UTF-8
filename. Consider the use of os.listdir() by the glob module. If I am
globbing for *.py, why should the presence of a file named b'\xff'
cause it to fail?

Robust programs using os.listdir() should use the bytes->bytes version.

>>  - listdir(bytes) -> bytes
>>  - getcwd() -> unicode
>>  - getcwd(bytes=True) -> bytes
>>  - open(): accept bytes or unicode
>>
>> os.path.*() should accept operations on bytes filenames, but maybe not on
>> bytes+unicode arguments. os.path.join('directory', b'filename'): raise an
>> error (or use *implicit* conversion to bytes)?

(Yeah, it should be all bytes or all strings.)

On Mon, Sep 29, 2008 at 9:45 AM, Georg Brandl <g.brandl at gmx.net> wrote:

> This approach (changing all path-handling functions to accept either bytes
> or string, but not both) is doomed in my eyes. First, there are lots of them,
> second, they are not only in os.path but in many modules and also in user
> code, and third, I see no clean way of implementing them in the specified way.
> (Just try to do it with os.path.join as an example; I couldn't find the
> good way to write it, only the bad and the ugly...)

It doesn't have to be supported for all operations -- just enough to
be able to access all the system calls. and do the most basic pathname
manipulations (split and join -- almost everything else can be built
out of those).

> If I had to choose, I'd still argue for the modified UTF-8 as filesystem
> encoding (if it were UTF-8 otherwise), despite possible surprises when a
> such-encoded filename escapes from Python.

I'm having a hard time finding info about UTF-8b. Does anyone have a
decent link?

I noticed that OSX has a different approach yet. I believe it insists
on valid UTF-8 filenames. It may even require some normalization but I
don't know if the kernel enforces this. I tried to create a file named
b'\xff' and it came out as %ff. Then "rm %ff" worked. So I think it
may be replacing all bad UTF8 sequences with their % encoding.

The "set filesystem encoding to be Latin-1" approach has a certain
charm as well, but clearly would be a mistake on OSX, and probably on
other systems too (whenever the user doesn't think in Latin-1).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Mon Sep 29 19:16:23 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 29 Sep 2008 10:16:23 -0700
Subject: [Python-Dev] Python security team
In-Reply-To: <48E0C60B.5060006@novell.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>
	<48E0C60B.5060006@novell.com>
Message-ID: <ca471dc20809291016l516cae9ev8b9676ecef60f40@mail.gmail.com>

On Mon, Sep 29, 2008 at 5:11 AM, Jan Matejek <jan.matejek at novell.com> wrote:
> Brett Cannon napsal(a):
>> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
>> <victor.stinner at haypocalc.com> wrote:
>>> First, I would like to access to these informations. Not only this issue, but
>>> all security related issues. I have some knowledges about security and I can
>>> help to resolve issues and/or estimate the criticity of an issue.
>>>
>>
>> That would require commit privileges first. Don't know if the group
>> requires that a person have a decent amount of time committing to the
>> core first (I just joined the list in late July).
>
> commit privileges?
> I would be interested in joining the PSRT list too - as a python
> maintainer for openSUSE, i think that it would be beneficial for both my
> and your work. And i can imagine that maintainers from other
> distributions have similar opinion on this ;)
> And that does not necessarily mean commit privileges, right?
>
> Or is this an issue of trust, where "we trust you enough to make changes
> to the core" equals "we also trust you enough to see the security issues" ?

Traditionally we have been extremely careful in selecting people to
join the PSRT -- basically people that have many years of reputation
*within the Python community*.

I think we may have to expand our selection creteria, since the
existing approach has led to a small PSRT whose members are all too
busy to do the necessary legwork. At the same time we need to remain
selective -- I don't think having a crowd of hundreds would be
productive, and we need to be sure that every single member can
absolutely be trusted to take security seriously.

To answer your question directly, I don't think that just being the
Python maintainer for some Linux distribution is enough to qualify --
if our process worked well enough, you'd be getting the patches from
us via some downstream-flowing distribution mechanism that reaches
only trusted people within each vendor organization. I don't happen to
know you personally -- but perhaps other current members of the PSRT
do and that could be enough to secure an invitation.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Mon Sep 29 19:47:20 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 29 Sep 2008 10:47:20 -0700
Subject: [Python-Dev] ',' precedence in documentation
In-Reply-To: <4edc17eb0809290712x62275543uf9ee9ff8c95744f3@mail.gmail.com>
References: <20080914193101.GB30564@idyll.org> <gajpob$2n6$1@ger.gmane.org>
	<ca471dc20809141300m6cde172cpadc274a736d8b8b7@mail.gmail.com>
	<1afaf6160809141306i73c4225am244b029c987bad01@mail.gmail.com>
	<48CD77F3.7020009@v.loewis.de> <20080914205225.GA15493@idyll.org>
	<48CD7B2B.2090000@v.loewis.de>
	<4edc17eb0809290712x62275543uf9ee9ff8c95744f3@mail.gmail.com>
Message-ID: <ca471dc20809291047n4d4b7533he028c8fe5461ee64@mail.gmail.com>

On Mon, Sep 29, 2008 at 7:12 AM, Michele Simionato
<michele.simionato at gmail.com> wrote:
> I like Martin's proposals (use a function, remove -O) very much.

That's too bad, since I don't like it at all :-). You can write your
own function trivially that does this; however IMO the *language*
should support something that can be disabled to the point where no
code is generated for it. Most languages have this (in fact Java
*added* it). I'll concede that -O is not necessary the right flag to
pass, but I do want to be able to write asserts that can be turned
into nothing completely, and unless we were to add macros for just
this purpose, it'll have to be something recognized by the compiler,
like a statement. Most other details are negotiable (like the exact
syntax, or the flag used to disable it). However I don't like changing
the syntax so that it resembles a function call -- that's not going to
resolve the existing confusion and will add more confusion ("why can't
I write my own assert() function?").

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From gnewsg at gmail.com  Mon Sep 29 21:02:16 2008
From: gnewsg at gmail.com (Giampaolo Rodola')
Date: Mon, 29 Sep 2008 12:02:16 -0700 (PDT)
Subject: [Python-Dev] Python security team
In-Reply-To: <e6511dbf0809271104y6fc22c29m608ea24fb8417781@mail.gmail.com>
References: <200809271754.29068.victor.stinner@haypocalc.com> 
	<e6511dbf0809271104y6fc22c29m608ea24fb8417781@mail.gmail.com>
Message-ID: <cc471a31-aafd-4eb8-89c4-8d85a630bb3e@m44g2000hsc.googlegroups.com>



On 27 Set, 20:04, "Josiah Carlson" <josiah.carl... at gmail.com> wrote:
> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
>
> <victor.stin... at haypocalc.com> wrote:
> > Second, I would like to help to fix all Python security issues. It looks like
> > Python community isn't very reactive (proactive?) about security. Eg. a DoS
> > was reported in smtpd server (integrated to Python)... 15 months ago. A patch
> > is available but it's not applied in Python trunk.
>
> The smtpd module is not meant to be used without modification. ?It is
> the responsibility of the application writer to decide the limitations
> of the emails they want to allow sending, and subsequently handle the
> case where emails overrun that limit. ?

The issue does not concern the emails but the buffer used internally
to store the received raw data sent by client.
The user who wants to fix the issue (#1745035) should override the
collect_incoming_data method which is usually not meant to be
modified.
Moreover, there are two RFCs which state that extremely long lines
must be truncated and an error reply must be returned.

--- Giampaolo
http://code.google.com/p/pyftpdlib/

From josiah.carlson at gmail.com  Mon Sep 29 22:44:16 2008
From: josiah.carlson at gmail.com (Josiah Carlson)
Date: Mon, 29 Sep 2008 13:44:16 -0700
Subject: [Python-Dev] Python security team
In-Reply-To: <cc471a31-aafd-4eb8-89c4-8d85a630bb3e@m44g2000hsc.googlegroups.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>
	<e6511dbf0809271104y6fc22c29m608ea24fb8417781@mail.gmail.com>
	<cc471a31-aafd-4eb8-89c4-8d85a630bb3e@m44g2000hsc.googlegroups.com>
Message-ID: <e6511dbf0809291344j4aea9315m47d4baef8942941b@mail.gmail.com>

On Mon, Sep 29, 2008 at 12:02 PM, Giampaolo Rodola' <gnewsg at gmail.com> wrote:
> On 27 Set, 20:04, "Josiah Carlson" <josiah.carl... at gmail.com> wrote:
>> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
>>
>> <victor.stin... at haypocalc.com> wrote:
>> > Second, I would like to help to fix all Python security issues. It looks like
>> > Python community isn't very reactive (proactive?) about security. Eg. a DoS
>> > was reported in smtpd server (integrated to Python)... 15 months ago. A patch
>> > is available but it's not applied in Python trunk.
>>
>> The smtpd module is not meant to be used without modification.  It is
>> the responsibility of the application writer to decide the limitations
>> of the emails they want to allow sending, and subsequently handle the
>> case where emails overrun that limit.
>
> The issue does not concern the emails but the buffer used internally
> to store the received raw data sent by client.
> The user who wants to fix the issue (#1745035) should override the
> collect_incoming_data method which is usually not meant to be
> modified.
> Moreover, there are two RFCs which state that extremely long lines
> must be truncated and an error reply must be returned.

We can and should discuss the specifics of this item in the bug report
itself.  I should have replied there instead.

 - Josiah

From gnewsg at gmail.com  Mon Sep 29 23:23:49 2008
From: gnewsg at gmail.com (Giampaolo Rodola')
Date: Mon, 29 Sep 2008 14:23:49 -0700 (PDT)
Subject: [Python-Dev] Python security team
In-Reply-To: <e6511dbf0809291344j4aea9315m47d4baef8942941b@mail.gmail.com>
References: <200809271754.29068.victor.stinner@haypocalc.com> 
	<e6511dbf0809271104y6fc22c29m608ea24fb8417781@mail.gmail.com> 
	<cc471a31-aafd-4eb8-89c4-8d85a630bb3e@m44g2000hsc.googlegroups.com> 
	<e6511dbf0809291344j4aea9315m47d4baef8942941b@mail.gmail.com>
Message-ID: <0ef5fa6c-1c96-4d4e-a613-35612663df76@k13g2000hse.googlegroups.com>

Yeah, right. Let's continue there.

--- Giampaolo
http://code.google.com/p/pyftpdlib



On 29 Set, 22:44, "Josiah Carlson" <josiah.carl... at gmail.com> wrote:
> On Mon, Sep 29, 2008 at 12:02 PM, Giampaolo Rodola' <gne... at gmail.com> wrote:
> > On 27 Set, 20:04, "Josiah Carlson" <josiah.carl... at gmail.com> wrote:
> >> On Sat, Sep 27, 2008 at 8:54 AM, Victor Stinner
>
> >> <victor.stin... at haypocalc.com> wrote:
> >> > Second, I would like to help to fix all Python security issues. It looks like
> >> > Python community isn't very reactive (proactive?) about security. Eg. a DoS
> >> > was reported in smtpd server (integrated to Python)... 15 months ago. A patch
> >> > is available but it's not applied in Python trunk.
>
> >> The smtpd module is not meant to be used without modification. ?It is
> >> the responsibility of the application writer to decide the limitations
> >> of the emails they want to allow sending, and subsequently handle the
> >> case where emails overrun that limit.
>
> > The issue does not concern the emails but the buffer used internally
> > to store the received raw data sent by client.
> > The user who wants to fix the issue (#1745035) should override the
> > collect_incoming_data method which is usually not meant to be
> > modified.
> > Moreover, there are two RFCs which state that extremely long lines
> > must be truncated and an error reply must be returned.
>
> We can and should discuss the specifics of this item in the bug report
> itself. ?I should have replied there instead.
>
> ?- Josiah
> _______________________________________________
> Python-Dev mailing list
> Python-... at python.orghttp://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:http://mail.python.org/mailman/options/python-dev/python-dev2-garchiv...- Nascondi testo citato
>
> - Mostra testo citato -

From rhamph at gmail.com  Tue Sep 30 00:04:38 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Mon, 29 Sep 2008 16:04:38 -0600
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <200809291800.18911.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<d11dcfba0809290816k40b75e84pf6df8cfd6263ab15@mail.gmail.com>
	<200809291800.18911.victor.stinner@haypocalc.com>
Message-ID: <aac2c7cb0809291504k7f19e190ga9f0dd82587d4448@mail.gmail.com>

On Mon, Sep 29, 2008 at 10:00 AM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Le Monday 29 September 2008 17:16:47 Steven Bethard, vous avez ?crit :
>> >  - getcwd() -> unicode
>> >  - getcwd(bytes=True) -> bytes
>>
>> Please let's not introduce boolean flags like this. How about
>> ``getcwdb`` in parallel with the old ``getcwdu``?
>
> Yeah, you're right. So i wrote a new patch: os_getcwdb.patch
>
> With my patch we get (Python3):
>  * os.getcwd() -> unicode
>  * os.getcwdb() -> bytes
>
> Previously in Python2 it was:
>  * os.getcwd() -> str (bytes)
>  * os.getcwdu() -> unicode

Why not do:
 * os.getcwd() -> unicode
 * posix.getcwdb() -> bytes

os gets the standard version and posix has an (unambiguously named)
platform-specific version.


-- 
Adam Olsen, aka Rhamphoryncus

From rhamph at gmail.com  Tue Sep 30 00:17:20 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Mon, 29 Sep 2008 16:17:20 -0600
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
Message-ID: <aac2c7cb0809291517h13a631bbqd4ea49ec757f36e1@mail.gmail.com>

On Mon, Sep 29, 2008 at 11:06 AM, Guido van Rossum <guido at python.org> wrote:
> On Mon, Sep 29, 2008 at 9:45 AM, Georg Brandl <g.brandl at gmx.net> wrote:
>
>> This approach (changing all path-handling functions to accept either bytes
>> or string, but not both) is doomed in my eyes. First, there are lots of them,
>> second, they are not only in os.path but in many modules and also in user
>> code, and third, I see no clean way of implementing them in the specified way.
>> (Just try to do it with os.path.join as an example; I couldn't find the
>> good way to write it, only the bad and the ugly...)
>
> It doesn't have to be supported for all operations -- just enough to
> be able to access all the system calls. and do the most basic pathname
> manipulations (split and join -- almost everything else can be built
> out of those).
>
>> If I had to choose, I'd still argue for the modified UTF-8 as filesystem
>> encoding (if it were UTF-8 otherwise), despite possible surprises when a
>> such-encoded filename escapes from Python.
>
> I'm having a hard time finding info about UTF-8b. Does anyone have a
> decent link?

http://mail.nl.linux.org/linux-utf8/2000-07/msg00040.html

Scroll down to item D, near the bottom.

It turns malformed bytes into lone (therefor malformed) surrogates.


> I noticed that OSX has a different approach yet. I believe it insists
> on valid UTF-8 filenames. It may even require some normalization but I
> don't know if the kernel enforces this. I tried to create a file named
> b'\xff' and it came out as %ff. Then "rm %ff" worked. So I think it
> may be replacing all bad UTF8 sequences with their % encoding.

I suspect linux will eventually take this route as well.  If ext3 had
an option for UTF-8 validation I know I'd want it on.  That'd move the
error to the program creating bogus file names, rather than those
trying to read, display, and manage them.


> The "set filesystem encoding to be Latin-1" approach has a certain
> charm as well, but clearly would be a mistake on OSX, and probably on
> other systems too (whenever the user doesn't think in Latin-1).

Aye, it's a better hack than UTF-8b, but adding byte functions is even better.


-- 
Adam Olsen, aka Rhamphoryncus

From foom at fuhm.net  Tue Sep 30 00:28:31 2008
From: foom at fuhm.net (James Y Knight)
Date: Mon, 29 Sep 2008 18:28:31 -0400
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <aac2c7cb0809291517h13a631bbqd4ea49ec757f36e1@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
	<aac2c7cb0809291517h13a631bbqd4ea49ec757f36e1@mail.gmail.com>
Message-ID: <D6599E0D-3744-4D08-913D-544BB13743E9@fuhm.net>

On Sep 29, 2008, at 6:17 PM, Adam Olsen wrote:
> I suspect linux will eventually take this route as well.  If ext3 had
> an option for UTF-8 validation I know I'd want it on.  That'd move the
> error to the program creating bogus file names, rather than those
> trying to read, display, and manage them.

Of course, even on Mac OS X, or a theoretical UTF-8-enforcing ext3,  
random byte strings are still possible in your program's argv, in  
environment variables, and as arguments to subprocesses.

So python still needs to do something...

James

From martin at v.loewis.de  Tue Sep 30 00:42:08 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 00:42:08 +0200
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <200809290939.59963.eckhardt@satorlaser.com>
References: <200809231052.36720.eckhardt@satorlaser.com>
	<200809260921.36012.eckhardt@satorlaser.com>
	<48DD52E7.8060408@v.loewis.de>
	<200809290939.59963.eckhardt@satorlaser.com>
Message-ID: <48E159C0.3020407@v.loewis.de>

Ulrich Eckhardt wrote:
>>> Well, currently it does make a difference. Simple example: CreateFile().
>> It's not so simple: Python doesn't actually call CreateFile
> 
> Martin, CreateFile() was just used as an example. You can substitute it with 
> LoadString() or CreateProcess() if you like, the problem remains the same.

However, the solution should be different from the one you propose. I
don't know what call of CreateProcess you are referring to specifically,
but I think they should all be changed to call CreateProcessW.

Again, whether or not _UNICODE is defined should have no effect. If it
does, it's a bug, and the solution is not to sprinkle TCHAR all over the
place.

> [about using SCons for building]
>> And you *can* provide an SCons file that supports all the SDKs?
> 
> No, but I can provide one that allows parametrisation. ;)

And, with proper parametrization, then supports all SDKs?

Regards,
Martin

From martin at v.loewis.de  Tue Sep 30 00:49:39 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 00:49:39 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>	<200809291250.03291.eckhardt@satorlaser.com>	<48E0B8FB.9070701@egenix.com>	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
Message-ID: <48E15B83.9040205@v.loewis.de>

> Originally I thought that this was a valid idea, but then it became
> clear that this could be a problem.  Consider a filename which includes
> a UTF-8 encoding of a PUA code point.

I still think it's a valid idea. For non-UTF-8 file system encodings,
use PUA characters, and generate them through an error handler.

If the file system encoding is UTF-8, use UTF-8b instead as the
file system encoding.

> Viewing the PUA with GNOME charmap, I can see that many code points
> there have character renderings on my Ubuntu system.  I have to assume,
> therefore, that there are other (and potentially conflicting) uses for
> this unicode feature.

Depends on how you use it. If you use the PUA block 1 (i.e.
U+E000..U+F8FF), there is a realistic chance of collision.

If you use the Plane 15 or Plane 16 PUA blocks, there is currently
zero chance of collision (AFAIK). PUA has a wide use for additional
characters in TrueType, but I don't think many tools even support
plane 15 and 16 for generating fonts, or rendering them (it may even
that the TrueType/OpenType format doesn't support them in the first
place). However, Python can make use of these planes fairly easily,
even in 2-byte mode (through UTF-16).

Regards,
Martin

From martin at v.loewis.de  Tue Sep 30 00:51:55 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 00:51:55 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <C823EDDC-BE02-413A-B427-83CAF8A4E42D@cwi.nl>
References: <200809271404.25654.victor.stinner@haypocalc.com>	<48DE705E.6050405@v.loewis.de>	<52dc1c820809281334t36086001ie7b87f618b949bdb@mail.gmail.com>	<200809291250.03291.eckhardt@satorlaser.com>
	<C823EDDC-BE02-413A-B427-83CAF8A4E42D@cwi.nl>
Message-ID: <48E15C0B.1020006@v.loewis.de>

Jack Jansen wrote:
> I'm a bit late to join in this discussion, but if unicode filenames are
> going to be the normal mode, how about this whole normalized/canonical
> business?

I don't think there is a change in the current implementation. Users
interested in this issue should contribute code that normalizes file
names appropriately on systems that require such normalization.

Regards,
Martin

From martin at v.loewis.de  Tue Sep 30 00:56:18 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 00:56:18 +0200
Subject: [Python-Dev] New proposition for Python3 bytes filename issue
In-Reply-To: <200809291407.55291.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
Message-ID: <48E15D12.40009@v.loewis.de>


> The default behaviour should be to use unicode and raise an error if 
> conversion to unicode fails. It should also be possible to use bytes using 
> bytes arguments and optional arguments (for getcwd).

I'm still opposed to allowing bytes as file names at all in 3k. Python
should really strive for providing a uniform datatype, and that should
be the character string type.

For applications that cannot trust that the conversion works always
correctly on POSIX systems, sys.setfilesystemencoding should be
provided.

In the long run, need for explicit calls to this function should be
reduced, by
a) systems getting more consistent in their file name encoding, and
b) Python providing better defaults for detecting the file name
   encoding, and better round-trip support for non-encodable bytes.
Part b) is probably out-of-scope for 3.0 now, but should be reconsidered
for 3.1

Regards,
Martin

From victor.stinner at haypocalc.com  Tue Sep 30 01:05:53 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 30 Sep 2008 01:05:53 +0200
Subject: [Python-Dev] Real segmentation fault handler
Message-ID: <200809300105.53473.victor.stinner@haypocalc.com>

Hi,

I would like to be able to catch SIGSEGV in my Python code! So I started to 
hack Python trunk to support this feature. The idea is to use a signal 
handler which call longjmp(), and add setjmp() at Py_EvalFrameEx() enter.

See attached ("small") patch: segfault.patch

Example read.py with the *evil* ctypes module of invalid memory read:
------------------- 8< --------------
from ctypes import string_at

def fault():
    text = string_at(1, 10)
    print("text = {0!r}".format(text))

def test():
    print("test: 1")
    try:
        fault()
    except MemoryError, err:
        print "ooops!"
        print err

    print("test: 2")
    try:
        fault()
    except MemoryError, err:
        print "ooops!"
        print err

    print("test: end")

def main():
    test()

if __name__ == "__main__":
    main()
------------------- 8< --------------

Result:
------------------- 8< --------------
$ python read.py
test: 1
sizeof()=160
ooops!
segmentation fault
test: 2
sizeof()=160
ooops!
segmentation fault
test: end
------------------- 8< --------------

Example bug1.py of a stack overflow:
----------
loop = None,
for i in xrange(10**5):
    loop = loop, None
----------

Result:
----------
$ python -i bug1.py
>>> print loop
(((((((((...Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: segmentation fault
>>>
----------

Python is able to restore a valid state (stack/heap) after a segmentation 
fault and raise a classical Python exception (I choosed MemoryError, but it 
could be a specific exception).

On my computer (Ubuntu Gutsy/i386), each segfault_frame takes 
sizeof(sigjmpbuf) + sizeof(void*) = 160 bytes, allocated on the stack. I 
don't know if it's huge or not, but that will limit the number of recursive 
calls. The feature can be optional if we add a configure option and some 
#ifdef/#endif. A dedicated stack is needed to be call the signal handler on 
stack overflow error. I choosed 4 KB, but since I only call longjmp(), 
smaller stack might also works.

Does other VM support such feature? JVM, Mono, .NET, etc. ?

I had the idea of catching SIGSEGV after reading the issue 1069092 (stack 
overflow because of too many recursive calls).

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: segfault.patch
Type: text/x-diff
Size: 4195 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080930/d2f489cd/attachment-0001.patch>

From brett at python.org  Tue Sep 30 01:23:33 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Sep 2008 16:23:33 -0700
Subject: [Python-Dev] I would like an Python account
In-Reply-To: <200809271732.19317.victor.stinner@haypocalc.com>
References: <200809271732.19317.victor.stinner@haypocalc.com>
Message-ID: <bbaeab100809291623y64f8fcf9m79b336dfadb1f4bf@mail.gmail.com>

On Sat, Sep 27, 2008 at 8:32 AM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Hi,
>
> Would it possible to get more permissions on Python bugtracker, especially to
> add keywords, close a duplicate bug, etc.?
>

Let's start off with giving you Developer permissions on the tracker
to start and then you can work up to commit privileges, Victor. What
is your username on the tracker?

-Brett

From victor.stinner at haypocalc.com  Tue Sep 30 01:29:24 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 30 Sep 2008 01:29:24 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
Message-ID: <200809300129.24972.victor.stinner@haypocalc.com>

Le Monday 29 September 2008 19:06:01 Guido van Rossum, vous avez ?crit?:
> >>  - listdir(unicode) -> unicode and raise an error on invalid filename
>
> I know I keep flipflopping on this one, but the more I think about it
> the more I believe it is better to drop those names than to raise an
> exception. Otherwise a "naive" program that happens to use
> os.listdir() can be rendered completely useless by a single non-UTF-8
> filename. Consider the use of os.listdir() by the glob module. If I am
> globbing for *.py, why should the presence of a file named b'\xff'
> cause it to fail?

It would be hard for a newbie programmer to understand why he's unable to find 
his very important file ("important r?port.doc") using os.listdir(). And yes, 
if your file system is broken, glob(<unicode>) will fail.

If we choose to support bytes on Linux, a robust and portable program have to 
use only bytes filenames on Linux to always be able to list and open files.

A full example to list files and display filenames:

  import os
  import os.path
  import sys
  if os.path.supports_unicode_filenames:
     cwd = getcwd()
  else:
     cwd = getcwdb()
     encoding = sys.getfilesystemencoding()
  for filename in os.listdir(cwd):
     if os.path.supports_unicode_filenames:
        text = str(filename, encoding, "replace)
     else:
        text = filename
     print("=== File {0} ===".format(text))
     for line in open(filename):
        ...

We need an "if" to choose the directory. The second "if" is only needed to 
display the filename. Using bytes, it would be possible to write better code 
detect the real charset (eg. ISO-8859-1 in a UTF-8 file system) and so 
display correctly the filename and/or propose to rename the file. Would it 
possible using UTF-8b / PUA hacks?

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From rhamph at gmail.com  Tue Sep 30 01:31:45 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Mon, 29 Sep 2008 17:31:45 -0600
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48E15B83.9040205@v.loewis.de>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<48E0B8FB.9070701@egenix.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
Message-ID: <aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>

On Mon, Sep 29, 2008 at 4:49 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> Originally I thought that this was a valid idea, but then it became
>> clear that this could be a problem.  Consider a filename which includes
>> a UTF-8 encoding of a PUA code point.
>
> I still think it's a valid idea. For non-UTF-8 file system encodings,
> use PUA characters, and generate them through an error handler.
>
> If the file system encoding is UTF-8, use UTF-8b instead as the
> file system encoding.
>
>> Viewing the PUA with GNOME charmap, I can see that many code points
>> there have character renderings on my Ubuntu system.  I have to assume,
>> therefore, that there are other (and potentially conflicting) uses for
>> this unicode feature.
>
> Depends on how you use it. If you use the PUA block 1 (i.e.
> U+E000..U+F8FF), there is a realistic chance of collision.
>
> If you use the Plane 15 or Plane 16 PUA blocks, there is currently
> zero chance of collision (AFAIK). PUA has a wide use for additional
> characters in TrueType, but I don't think many tools even support
> plane 15 and 16 for generating fonts, or rendering them (it may even
> that the TrueType/OpenType format doesn't support them in the first
> place). However, Python can make use of these planes fairly easily,
> even in 2-byte mode (through UTF-16).

An example where lossy conversion fails:

1) create file using UTF-8 app with PUA (or ambiguous scalar of
choice) filename.
2) list dir in python.  file name is now a unicode object with PUA.
3) attempt to open.  file name gets converted to malformed UTF-8
sequence.  Doesn't match the name on disk, so opening fails

Lossy conversion just moves around what gets treated as garbage.  As
all valid unicode scalars can be round tripped, there's no way to
create a valid unicode file name without being lossy.  The alternative
is not be valid unicode, but since we can't use such objects with
external libs, can't even print them, we might as well call them
something else.  We already have a name for that: bytes.


-- 
Adam Olsen, aka Rhamphoryncus

From greg.ewing at canterbury.ac.nz  Tue Sep 30 01:32:42 2008
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 30 Sep 2008 11:32:42 +1200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <200809291434.07263.eckhardt@satorlaser.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<20080929112851.31635.999142795.divmod.xquotient.274@weber.divmod.com>
	<200809291434.07263.eckhardt@satorlaser.com>
Message-ID: <48E1659A.8020403@canterbury.ac.nz>

Ulrich Eckhardt wrote:

> AFAIK, OS X guarantees UTF-8 for filesystem encodings. So the OS also provides 
> Unicode filenames and how it deals with broken or legacy media is left up to 
> the OS.

Does this mean that the OS always returns valid utf-8 strings
from filesystem calls, even if the media is broken or legacy?

-- 
Greg

From guido at python.org  Tue Sep 30 01:41:36 2008
From: guido at python.org (Guido van Rossum)
Date: Mon, 29 Sep 2008 16:41:36 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <200809300129.24972.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
	<200809300129.24972.victor.stinner@haypocalc.com>
Message-ID: <ca471dc20809291641m3b563b8ar6521b4ec94f0e120@mail.gmail.com>

On Mon, Sep 29, 2008 at 4:29 PM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Le Monday 29 September 2008 19:06:01 Guido van Rossum, vous avez ?crit :
>> >>  - listdir(unicode) -> unicode and raise an error on invalid filename
>>
>> I know I keep flipflopping on this one, but the more I think about it
>> the more I believe it is better to drop those names than to raise an
>> exception. Otherwise a "naive" program that happens to use
>> os.listdir() can be rendered completely useless by a single non-UTF-8
>> filename. Consider the use of os.listdir() by the glob module. If I am
>> globbing for *.py, why should the presence of a file named b'\xff'
>> cause it to fail?
>
> It would be hard for a newbie programmer to understand why he's unable to find
> his very important file ("important r?port.doc") using os.listdir().

*Every* failure in this scenario will be hard to understand for a
newbie programmer. We can just document the fact.

> And yes,
> if your file system is broken, glob(<unicode>) will fail.

Why should it?

> If we choose to support bytes on Linux, a robust and portable program have to
> use only bytes filenames on Linux to always be able to list and open files.

Right. But such robustness is only needed to support certain odd cases
and we cannot demand that most people bother to write robust code all
the time.

> A full example to list files and display filenames:
>
>  import os
>  import os.path
>  import sys
>  if os.path.supports_unicode_filenames:

This is backwards -- the Unicode API is always supported, the bytes
API only on Linux (and perhaps some other other Unixes).

>     cwd = getcwd()
>  else:
>     cwd = getcwdb()
>     encoding = sys.getfilesystemencoding()
>  for filename in os.listdir(cwd):
>     if os.path.supports_unicode_filenames:
>        text = str(filename, encoding, "replace)
>     else:
>        text = filename
>     print("=== File {0} ===".format(text))
>     for line in open(filename):
>        ...
>
> We need an "if" to choose the directory. The second "if" is only needed to
> display the filename. Using bytes, it would be possible to write better code
> detect the real charset (eg. ISO-8859-1 in a UTF-8 file system) and so
> display correctly the filename and/or propose to rename the file. Would it
> possible using UTF-8b / PUA hacks?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From victor.stinner at haypocalc.com  Tue Sep 30 02:02:38 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 30 Sep 2008 02:02:38 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbr0nv$iqu$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
Message-ID: <200809300202.38574.victor.stinner@haypocalc.com>

Le Monday 29 September 2008 18:45:28 Georg Brandl, vous avez ?crit?:
> If I had to choose, I'd still argue for the modified UTF-8 as filesystem
> encoding (if it were UTF-8 otherwise), despite possible surprises when a
> such-encoded filename escapes from Python.

If I understand correctly this solution. The idea is to change the default 
file system encoding, right? Eg. if your filesystem is UTF-8, use ISO-8859-1 
to make sure that UTF-8 conversion will never fail.

Let's try with an ugly directory on my UTF-8 file system:
$ find
.
./t?ste
./?
./a?b
./dossi?
./dossi?/abc
./dir?name
./dir?name/xyz

Python3 using encoding=ISO-8859-1:
>>> import os; os.listdir(b'.')
[b't\xc3\xaaste', b'\xc3\xb4', b'a\xffb', b'dossi\xc3\xa9', b'dir\xffname']
>>> files=os.listdir('.'); files
['t??ste', '??', 'a?b', 'dossi??', 'dir?name']
>>> open(files[0]).close()
>>> os.listdir(files[-1])
['xyz']

Ok, I have unicode filenames and I'm able to open a file and list a directory. 
The problem is now to display correctly the filenames.

For me "unicode" sounds like "text (characters) encoded in the correct 
charset". In this case, unicode is just a storage for *bytes* in a custom 
charset.

How can we mix <custom unicode (bytes encoded in ISO-8859-1)> with <real 
unicode>? Eg. os.path.join('dossi??', "fichi?") : first argument is encoded 
in ISO-8859-1 whereas the second argument is encoding in Unicode. It's 
something like that:
   str(b'dossi\xc3\xa9', 'ISO-8859-1') + '/' + 'fichi\xe9'

Whereas the correct (unicode) result should be: 
   'dossi?/fichi?'
as bytes in ISO-8859-1:
   b'dossi\xc3\xa9/fichi\xc3\xa9'
as bytes in UTF-8:
   b'dossi\xe9/fichi\xe9'

Change the default file system encoding to store bytes in Unicode is like 
introducing a new Python type: <fake Unicode for filename hacks>.

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From martin at v.loewis.de  Tue Sep 30 02:07:23 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 02:07:23 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <200809300129.24972.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
	<200809300129.24972.victor.stinner@haypocalc.com>
Message-ID: <48E16DBB.1030805@v.loewis.de>

>   import os
>   import os.path
>   import sys
>   if os.path.supports_unicode_filenames:
>      cwd = getcwd()
>   else:
>      cwd = getcwdb()
>      encoding = sys.getfilesystemencoding()
>   for filename in os.listdir(cwd):
>      if os.path.supports_unicode_filenames:
>         text = str(filename, encoding, "replace)
>      else:
>         text = filename
>      print("=== File {0} ===".format(text))
>      for line in open(filename):
>         ...
> 
> We need an "if" to choose the directory. The second "if" is only needed to 
> display the filename. Using bytes, it would be possible to write better code 
> detect the real charset (eg. ISO-8859-1 in a UTF-8 file system) and so 
> display correctly the filename and/or propose to rename the file. Would it 
> possible using UTF-8b / PUA hacks?

Not sure what "it" is: to write the code above using the PUA hack:

for filename in os.listdir(os.getcwd())
    text = repr(filename)
    print("=== File {0} ===".format(text))
    for line in open(filenmae):
        ...

If "it" is "display the filename": sure, see above. If "it" is "detect
the real charset": sure, why not?

Regards,
Martin

From rhamph at gmail.com  Tue Sep 30 02:08:41 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Mon, 29 Sep 2008 18:08:41 -0600
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <200809300129.24972.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
	<200809300129.24972.victor.stinner@haypocalc.com>
Message-ID: <aac2c7cb0809291708u2fc03647sa5a09af4a15719ba@mail.gmail.com>

On Mon, Sep 29, 2008 at 5:29 PM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Le Monday 29 September 2008 19:06:01 Guido van Rossum, vous avez ?crit :
>> >>  - listdir(unicode) -> unicode and raise an error on invalid filename
>>
>> I know I keep flipflopping on this one, but the more I think about it
>> the more I believe it is better to drop those names than to raise an
>> exception. Otherwise a "naive" program that happens to use
>> os.listdir() can be rendered completely useless by a single non-UTF-8
>> filename. Consider the use of os.listdir() by the glob module. If I am
>> globbing for *.py, why should the presence of a file named b'\xff'
>> cause it to fail?
>
> It would be hard for a newbie programmer to understand why he's unable to find
> his very important file ("important r?port.doc") using os.listdir(). And yes,
> if your file system is broken, glob(<unicode>) will fail.

Imagine a program that list all files in a dir, as well as their file
size.  If we return bytes we'll print the name wrong.  If we return
lossy unicode we'll be unable to get the size of some files.  If we
return a malformed unicode we'll be unable to print at all (and what
if this is a GUI app?)

The common use cases need unicode, so the best options for them are to
fail outright or skip bad filenames.

The uncommon use cases need bytes, and they could do an explicit lossy
decode for printing, while still keeping the internal file name as
bytes.


Failing outright does have the advantage that the resulting exception
should have a half-decent approximation of the bad filename.  (Thanks
to the recent choices on unicode repr() and having stderr do escapes.)


-- 
Adam Olsen, aka Rhamphoryncus

From victor.stinner at haypocalc.com  Tue Sep 30 02:09:33 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 30 Sep 2008 02:09:33 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
Message-ID: <200809300209.33636.victor.stinner@haypocalc.com>

Le Tuesday 30 September 2008 01:31:45 Adam Olsen, vous avez ?crit?:
> The alternative is not be valid unicode, but since we can't use such 
> objects with external libs, can't even print them, we might as well 
> call them something else.  We already have a name for that: bytes.

:-)

From victor.stinner at haypocalc.com  Tue Sep 30 02:47:20 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 30 Sep 2008 02:47:20 +0200
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
Message-ID: <200809300247.20349.victor.stinner@haypocalc.com>

Hi,

See attached patch: python3_bytes_filename.patch

Using the patch, you will get:
 - open() support bytes
 - listdir(unicode) -> only unicode, *skip* invalid filenames 
   (as asked by Guido)
 - remove os.getcwdu()
 - create os.getcwdb() -> bytes
 - glob.glob() support bytes
 - fnmatch.filter() support bytes
 - posixpath.join() and posixpath.split() support bytes

Mixing bytes and str is invalid. Examples raising a TypeError:
 - posixpath.join(b'x', 'y')
 - fnmatch.filter([b'x', 'y'], '*')
 - fnmatch.filter([b'x', b'y'], '*')
 - glob.glob1('.', b'*')
 - glob.glob1(b'.', '*')

$ diffstat ~/python3_bytes_filename.patch
 Lib/fnmatch.py        |    7 +++-
 Lib/glob.py           |   15 ++++++---
 Lib/io.py             |    2 -
 Lib/posixpath.py      |   20 ++++++++----
 Modules/posixmodule.c |   83 
++++++++++++++++++--------------------------------
 5 files changed, 62 insertions(+), 65 deletions(-)

TODO:
 - review this patch :-)
 - support non-ASCII bytes in fnmatch.filter()
 - fix other functions, eg. posixpath.isabs() and fnmatch.fnmatchcase()
 - fix functions written in C: grep FileSystemDefaultEncoding
 - make sure that mixing bytes and str is rejected

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: python3_bytes_filename.patch
Type: text/x-diff
Size: 6732 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20080930/e8998338/attachment.patch>

From stephen at xemacs.org  Tue Sep 30 03:13:11 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 30 Sep 2008 10:13:11 +0900
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48E1659A.8020403@canterbury.ac.nz>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<20080929112851.31635.999142795.divmod.xquotient.274@weber.divmod.com>
	<200809291434.07263.eckhardt@satorlaser.com>
	<48E1659A.8020403@canterbury.ac.nz>
Message-ID: <87r672e8rc.fsf@xemacs.org>

Greg Ewing writes:
 > Ulrich Eckhardt wrote:
 > 
 > > AFAIK, OS X guarantees UTF-8 for filesystem encodings. So the OS
 > > also provides Unicode filenames and how it deals with broken or
 > > legacy media is left up to the OS.
 > 
 > Does this mean that the OS always returns valid utf-8 strings
 > from filesystem calls, even if the media is broken or legacy?

No, this means Ulrich is wrong.  NFD-normalized UTF-8 is more or less
enforced by the default filesystem, but Mac OS X up to 10.4 at least
also supports the FreeBSD filesystems, and some of those can have any
encoding you like or none at all (ie, KOI8-R and Shift JIS in the same
directory is possible).

If you have a Mac it's easy enough to test by creating a disk image
with a non-default file system.

From rwgk at yahoo.com  Tue Sep 30 04:06:07 2008
From: rwgk at yahoo.com (Ralf W. Grosse-Kunstleve)
Date: Mon, 29 Sep 2008 19:06:07 -0700 (PDT)
Subject: [Python-Dev] Real segmentation fault handler
Message-ID: <566534.24492.qm@web31405.mail.mud.yahoo.com>

FWIW: I didn't have much luck translating segfaults into exceptions. It (seemed) to work on
some platforms, but not others; this was in the context of C++.
In my experience, it is more useful to generate Python and C stack traces and bail out.
I also do this for floating-point exceptions. The handlers are installed at runtime
from a low-level extension module:

http://cctbx.svn.sourceforge.net/viewvc/cctbx/trunk/boost_adaptbx/meta_ext.cpp?view=markup

Example output is below. It works under Linux and partially under Mac OS X.

Ralf


% boost_adaptbx.segmentation_fault 
Now dereferencing null-pointer ...
show_stack(1): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/segmentation_fault.py(10) run
show_stack(2): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/segmentation_fault.py(14) <module>
libc backtrace (18 frames, most recent call last):
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python [0x4118e9]
  /lib64/libc.so.6(__libc_start_main+0xf4) [0x363241e074]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(Py_Main+0x935) [0x4123c5]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_SimpleFileExFlags+0x1a0) [0x4a8860]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_FileExFlags+0x10e) [0x4a85ce]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCode+0x32) [0x487402]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCodeEx+0x81f) [0x4873bf]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x6bc1) [0x486541]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x2bb9) [0x482539]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyObject_Call+0x13) [0x415ae3]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7c6f7]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x28) [0x2aaaaba87148]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::function0<void>::operator()() const+0x19e) [0x2aaaaba8816e]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7fef8]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0x7d) [0x2aaaaba7fb5d]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so(boost::python::objects::caller_py_function_impl<boost::python::detail::caller<char (*)(char const*), boost::python::default_call_policies, boost::mpl::vector2<char, char const*> > >::operator()(_object*, _object*)+0x29) [0x2aaaab8470a9]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so [0x2aaaab843790]
  /lib64/libc.so.6 [0x3632430f30]
Segmentation fault (Python and libc call stacks above)


% boost_adaptbx.divide_by_zero 
Now dividing by zero (in C++) ...
show_stack(1): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/divide_by_zero.py(10) run
show_stack(2): /net/chevy/raid1/rwgk/dist/boost_adaptbx/command_line/divide_by_zero.py(14) <module>
libc backtrace (18 frames, most recent call last):
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python [0x4118e9]
  /lib64/libc.so.6(__libc_start_main+0xf4) [0x363241e074]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(Py_Main+0x935) [0x4123c5]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_SimpleFileExFlags+0x1a0) [0x4a8860]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyRun_FileExFlags+0x10e) [0x4a85ce]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCode+0x32) [0x487402]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalCodeEx+0x81f) [0x4873bf]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x6bc1) [0x486541]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyEval_EvalFrameEx+0x2bb9) [0x482539]
  /net/chevy/raid1/rwgk/bintbx_py252/base/bin/python(PyObject_Call+0x13) [0x415ae3]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7c6f7]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::handle_exception_impl(boost::function0<void>)+0x28) [0x2aaaaba87148]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::function0<void>::operator()() const+0x19e) [0x2aaaaba8816e]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so [0x2aaaaba7fef8]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/libboost_python.so(boost::python::objects::function::call(_object*, _object*) const+0x7d) [0x2aaaaba7fb5d]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so(boost::python::objects::caller_py_function_impl<boost::python::detail::caller<double (*)(double const&, double const&), boost::python::default_call_policies, boost::mpl::vector3<double, double const&, double const&> > >::operator()(_object*, _object*)+0x12a) [0x2aaaab84759a]
  /net/chevy/raid1/rwgk/bintbx_py252/lib/boost_python_meta_ext.so [0x2aaaab8437a4]
  /lib64/libc.so.6 [0x3632430f30]
Floating-point error (Python and libc call stacks above)




----- Original Message ----
From: Victor Stinner <victor.stinner at haypocalc.com>
To: python-dev at python.org
Sent: Monday, September 29, 2008 4:05:53 PM
Subject: [Python-Dev] Real segmentation fault handler

Hi,

I would like to be able to catch SIGSEGV in my Python code! So I started to 
hack Python trunk to support this feature. The idea is to use a signal 
handler which call longjmp(), and add setjmp() at Py_EvalFrameEx() enter.

See attached ("small") patch: segfault.patch

Example read.py with the *evil* ctypes module of invalid memory read:
------------------- 8< --------------
from ctypes import string_at

def fault():
    text = string_at(1, 10)
    print("text = {0!r}".format(text))

def test():
    print("test: 1")
    try:
        fault()
    except MemoryError, err:
        print "ooops!"
        print err

    print("test: 2")
    try:
        fault()
    except MemoryError, err:
        print "ooops!"
        print err

    print("test: end")

def main():
    test()

if __name__ == "__main__":
    main()
------------------- 8< --------------

Result:
------------------- 8< --------------
$ python read.py
test: 1
sizeof()=160
ooops!
segmentation fault
test: 2
sizeof()=160
ooops!
segmentation fault
test: end
------------------- 8< --------------

Example bug1.py of a stack overflow:
----------
loop = None,
for i in xrange(10**5):
    loop = loop, None
----------

Result:
----------
$ python -i bug1.py
>>> print loop
(((((((((...Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: segmentation fault
>>>
----------

Python is able to restore a valid state (stack/heap) after a segmentation 
fault and raise a classical Python exception (I choosed MemoryError, but it 
could be a specific exception).

On my computer (Ubuntu Gutsy/i386), each segfault_frame takes 
sizeof(sigjmpbuf) + sizeof(void*) = 160 bytes, allocated on the stack. I 
don't know if it's huge or not, but that will limit the number of recursive 
calls. The feature can be optional if we add a configure option and some 
#ifdef/#endif. A dedicated stack is needed to be call the signal handler on 
stack overflow error. I choosed 4 KB, but since I only call longjmp(), 
smaller stack might also works.

Does other VM support such feature? JVM, Mono, .NET, etc. ?

I had the idea of catching SIGSEGV after reading the issue 1069092 (stack 
overflow because of too many recursive calls).

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/


From stephen at xemacs.org  Tue Sep 30 04:24:29 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 30 Sep 2008 11:24:29 +0900
Subject: [Python-Dev] [Python-3000] New proposition for Python3
	bytes	filename issue
In-Reply-To: <ca471dc20809291641m3b563b8ar6521b4ec94f0e120@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
	<200809300129.24972.victor.stinner@haypocalc.com>
	<ca471dc20809291641m3b563b8ar6521b4ec94f0e120@mail.gmail.com>
Message-ID: <87prmme5gi.fsf@xemacs.org>

Guido van Rossum writes:
 > On Mon, Sep 29, 2008 at 4:29 PM, Victor Stinner
 > <victor.stinner at haypocalc.com> wrote:

 > > It would be hard for a newbie programmer to understand why he's
 > > unable to find his very important file ("important r?port.doc")
 > > using os.listdir().

 > *Every* failure in this scenario will be hard to understand for a
 > newbie programmer. We can just document the fact.

Guido is absolutely right.  The Emacs/Mule people have been trying to
solve this kind of problem for 20 years, and the best they've come up
with is Martin's strategy: if you need really robust decoding, force
ISO 8859/1 (which for historical reasons uses all 256 octets) to get a
lossless internal text representation, and decode from that and *track
the encoding used* at the application level.  The email-sig/Mailman
people will testify how hard this is to do well, even when you have a
handful of RFCs that specify how it is to be done!

On the other hand, this kind of robustness is almost never needed in
"general newbie programming", except when you are writing a program to
be used to clean up after an undisciplined administration, or some
other system disaster.  Under normal circumstances the system encoding
is well-known and conformance is universal.

The best you can do for a general programming system is to
heuristically determine a single system encoding and raise an error if
the decoding fails.


From brett at python.org  Tue Sep 30 05:14:00 2008
From: brett at python.org (Brett Cannon)
Date: Mon, 29 Sep 2008 20:14:00 -0700
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <200809300247.20349.victor.stinner@haypocalc.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
Message-ID: <bbaeab100809292014y275c801fwa27e98ba8c5066f3@mail.gmail.com>

On Mon, Sep 29, 2008 at 5:47 PM, Victor Stinner
<victor.stinner at haypocalc.com> wrote:
> Hi,
>
> See attached patch: python3_bytes_filename.patch
>

Patches should go on the tracker, not the mailing list. Otherwise it
will just get lost.

-Brett

From tjreedy at udel.edu  Tue Sep 30 05:55:09 2008
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 29 Sep 2008 23:55:09 -0400
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <200809300129.24972.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
	<200809300129.24972.victor.stinner@haypocalc.com>
Message-ID: <gbs7up$1q5$1@ger.gmane.org>


> Le Monday 29 September 2008 19:06:01 Guido van Rossum, vous avez ?crit :

>> I know I keep flipflopping on this one, but the more I think about it
>> the more I believe it is better to drop those names than to raise an
>> exception. Otherwise a "naive" program that happens to use
>> os.listdir() can be rendered completely useless by a single non-UTF-8
>> filename. Consider the use of os.listdir() by the glob module. If I am
>> globbing for *.py, why should the presence of a file named b'\xff'
>> cause it to fail?

To avoid silent skipping, is it possible to drop 'unreadable' names, 
issue a warning (instead of exception), and continue to completion?
"Warning: unreadable filename skipped; see PyWiki/UnreadableFilenames"


From martin at v.loewis.de  Tue Sep 30 08:00:55 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 08:00:55 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <200809300202.38574.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
Message-ID: <48E1C097.8030309@v.loewis.de>

> Change the default file system encoding to store bytes in Unicode is like 
> introducing a new Python type: <fake Unicode for filename hacks>.

Exactly. Seems like the best solution to me, despite your polemics.

Regards,
Martin


From rhamph at gmail.com  Tue Sep 30 08:52:21 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Tue, 30 Sep 2008 00:52:21 -0600
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbsgk6$kc1$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<gbsgk6$kc1$1@ger.gmane.org>
Message-ID: <aac2c7cb0809292352l40a3cca1h64a8046fd7cd4364@mail.gmail.com>

On Tue, Sep 30, 2008 at 12:22 AM, Georg Brandl <g.brandl at gmx.net> wrote:
> Victor Stinner schrieb:
>> Le Monday 29 September 2008 18:45:28 Georg Brandl, vous avez ?crit :
>>> If I had to choose, I'd still argue for the modified UTF-8 as filesystem
>>> encoding (if it were UTF-8 otherwise), despite possible surprises when a
>>> such-encoded filename escapes from Python.
>>
>> If I understand correctly this solution. The idea is to change the default
>> file system encoding, right? Eg. if your filesystem is UTF-8, use ISO-8859-1
>> to make sure that UTF-8 conversion will never fail.
>
> No, that was not what I meant (although it is another possibility). As I wrote,
> Martin's proposal that I support here is using the modified UTF-8 codec that
> successfully roundtrips otherwise invalid UTF-8 data.
>
> You seem to forget that (disregarding OSX here, since it already enforces
> UTF-8) the majority of file names on Posix systems will be encoded correctly.
>
>> Let's try with an ugly directory on my UTF-8 file system:
>> $ find
>> ..
>> ../t?ste
>> ../?
>> ../a?b
>> ../dossi?
>> ../dossi?/abc
>> ../dir?name
>> ../dir?name/xyz
>>
>> Python3 using encoding=ISO-8859-1:
>>>>> import os; os.listdir(b'.')
>> [b't\xc3\xaaste', b'\xc3\xb4', b'a\xffb', b'dossi\xc3\xa9', b'dir\xffname']
>>>>> files=os.listdir('.'); files
>> ['t??ste', '??', 'a?b', 'dossi?(c)', 'dir?name']
>>>>> open(files[0]).close()
>>>>> os.listdir(files[-1])
>> ['xyz']
>>
>> Ok, I have unicode filenames and I'm able to open a file and list a directory.
>> The problem is now to display correctly the filenames.
>>
>> For me "unicode" sounds like "text (characters) encoded in the correct
>> charset". In this case, unicode is just a storage for *bytes* in a custom
>> charset.
>
>> How can we mix <custom unicode (bytes encoded in ISO-8859-1)> with <real
>> unicode>? Eg. os.path.join('dossi?(c)', "fichi?") : first argument is encoded
>> in ISO-8859-1 whereas the second argument is encoding in Unicode. It's
>> something like that:
>>    str(b'dossi\xc3\xa9', 'ISO-8859-1') + '/' + 'fichi\xe9'
>>
>> Whereas the correct (unicode) result should be:
>>    'dossi?/fichi?'
>> as bytes in ISO-8859-1:
>>    b'dossi\xc3\xa9/fichi\xc3\xa9'
>> as bytes in UTF-8:
>>    b'dossi\xe9/fichi\xe9'
>
> With the filenames decoded by UTF-8, your files named t?ste, ?, dossi? will
> be displayed and handled correctly. The others are *invalid* in the filesystem
> encoding UTF-8 and therefore would be represented by something like
>
> u'dir\uXXffname' where XX is some private use Unicode namespace. It won't look
> pretty when printed, but then, what do other applications do? They e.g. display
> a question mark as you show above, which is not better in terms of readability.
>
> But it will work when given to a filename-handling function. Valid filenames
> can be compared to Unicode strings.
>
> A real-world example: OpenOffice can't open files with invalid bytes in their
> name. They are displayed in the "Open file" dialog, but trying to open fails.
> This regularly drives me crazy. Let's not make Python not work this way too,
> or, even worse, not even display those filenames.

The only way to display that file would be to transform it into some
other valid unicode string.  However, as that string is already valid,
you've just made any files named after it impossible to open.  If you
extend unicode then you're unable to display that extended name[1].

I think Guido's right on this one.  If I have to choose between
openoffice crashing or skipping my file, I'd vastly prefer it skip it.
 A warning would be a nice bonus (from python or from openoffice),
telling me there's a buggered file I should go fix.  Renaming the file
is the end solution.


[1] You could argue that Unicode should add new scalars to handle all
currently invalid UTF-8 sequences.  They could then output to their
original forms if in UTF-8, or a mundane form in UTF-16 and UTF-32.
However, I suspect "we don't want to add validation to linux" will not
be a very persuasive argument.

-- 
Adam Olsen, aka Rhamphoryncus

From eckhardt at satorlaser.com  Tue Sep 30 09:31:58 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Tue, 30 Sep 2008 09:31:58 +0200
Subject: [Python-Dev] Status of MS Windows CE port
In-Reply-To: <48E159C0.3020407@v.loewis.de>
References: <200809231052.36720.eckhardt@satorlaser.com> 
	<200809290939.59963.eckhardt@satorlaser.com> 
	<48E159C0.3020407@v.loewis.de>
Message-ID: <200809300931.58364.eckhardt@satorlaser.com>

On Tuesday 30 September 2008, Martin v. L?wis wrote:
> Ulrich Eckhardt wrote:
> >>> Well, currently it does make a difference. Simple example:
> >>> CreateFile().
> >>
> >> It's not so simple: Python doesn't actually call CreateFile
> >
> > Martin, CreateFile() was just used as an example. You can substitute it
> > with LoadString() or CreateProcess() if you like, the problem remains the
> > same.
>
> However, the solution should be different from the one you propose. I
> don't know what call of CreateProcess you are referring to specifically,
> but I think they should all be changed to call CreateProcessW.
>
> Again, whether or not _UNICODE is defined should have no effect. If it
> does, it's a bug, and the solution is not to sprinkle TCHAR all over the
> place.

I think we're misunderstanding each other, because that is exactly the 
solution I'm targetting. I'm aware that TCHAR is just a hack to ease 
transition between obsolete MS Windows versions and NT and later. However, 
current state is that Python uses CreateProcessA() and changing that is not 
always trivial.

Therefore, my first step in porting is also to remove the dependency on TCHAR, 
i.e. replace things like CreateProcess() with preferably CreateProcessW() or 
alternatively CreateProcessA(). Just #defining _UNICODE in the build already 
turns up around 50 places in pythoncore that need work. I'll send patches 
soon.

> > [about using SCons for building]
> >
> >> And you *can* provide an SCons file that supports all the SDKs?
> >
> > No, but I can provide one that allows parametrisation. ;)
>
> And, with proper parametrization, then supports all SDKs?

Hopefully, yes, but I'm not going to make any claims which I'm not sure about. 
SCons is just convenient because the PythonCE project already uses it but I'm 
not adamant on that matter. The approach of Jack Jansen also sounds good, 
generating the projectfiles (which are XML, btw) from a Python script also 
sounds nice.

cheers

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From ncoghlan at gmail.com  Tue Sep 30 11:45:31 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 Sep 2008 19:45:31 +1000
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>	<200809291250.03291.eckhardt@satorlaser.com>	<48E0B8FB.9070701@egenix.com>	<200809291359.06334.eckhardt@satorlaser.com>	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
Message-ID: <48E1F53B.7030901@gmail.com>

Adam Olsen wrote:
> Lossy conversion just moves around what gets treated as garbage.  As
> all valid unicode scalars can be round tripped, there's no way to
> create a valid unicode file name without being lossy.  The alternative
> is not be valid unicode, but since we can't use such objects with
> external libs, can't even print them, we might as well call them
> something else.  We already have a name for that: bytes.

To my mind, there are two kinds of app in the world when it comes to
file paths:
1) "Normal" apps (e.g. a word processor), that are only interested in
files with sane, well-formed file names that can be properly decoded to
Unicode with the filesystem encoding identified by Python. If there is
invalid data on the filesystem, they don't care and don't want to see it
or have to deal with it.
2) "Filesystem" apps (e.g. a filesystem explorer), that need to be able
to deal with malformed filenames that may not decode properly using the
identified filesystem encoding.

For the former category of apps, the presence of a malformed filename
should NOT disrupt the processing of well-formed files and directories.
Those applications should "just work", even if the underlying filesystem
has a few broken filenames.

The latter category of applications need some way of defining their own
application-specific handling of malformed names.

That screams "callback" to me - and one mechanism to achieve that would
be to expose the unicode "errors" argument for filesystem operations
that return file paths (e.g. os.getcwd(), os.listdir(), os.readlink(),
os.walk()).

Once that was exposed, the existing error handling machinery in the
codecs module could be used to allow applications to define their own
custom error handling for Unicode decode errors in these operations.
(e.g. set "codecs.register_error('bad_filepath',
handle_filepath_error)", then use "errors='bad_filepath'" in the
relevant os API calls)

The default handling could be left at "strict", with os.listdir() and
os.walk() specifically ignoring path entries that trigger
UnicodeDecodeError.

getcwd() and readlink() could just propagate the exception, since they
have no other information to return.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From mal at egenix.com  Tue Sep 30 12:31:51 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 30 Sep 2008 12:31:51 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <48E1C097.8030309@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de>
Message-ID: <48E20017.3020405@egenix.com>

On 2008-09-30 08:00, Martin v. L?wis wrote:
>> Change the default file system encoding to store bytes in Unicode is like 
>> introducing a new Python type: <fake Unicode for filename hacks>.
> 
> Exactly. Seems like the best solution to me, despite your polemics.

Not a bad idea... have os.listdir() return Unicode subclasses that work
like file handles, ie. they have an extra buffer that holds the original
bytes value received from the underlying C API.

Passing these handles to open() would then do the right thing by using
whatever os.listdir() got back from the file system to open the file,
while still providing a sane way to display the filename, e.g. using
question marks for the invalid characters.

The only problem with this approach is concatenation of such handles
to form pathnames, but then perhaps those concatenations could just
work on the bytes value as well (I don't know of any OS that uses non-
ASCII path separators).

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 30 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From stephen at xemacs.org  Tue Sep 30 13:24:45 2008
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Tue, 30 Sep 2008 20:24:45 +0900
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <aac2c7cb0809292352l40a3cca1h64a8046fd7cd4364@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<gbsgk6$kc1$1@ger.gmane.org>
	<aac2c7cb0809292352l40a3cca1h64a8046fd7cd4364@mail.gmail.com>
Message-ID: <87iqsdev0i.fsf@xemacs.org>

Adam Olsen writes:

 > [1] You could argue that Unicode should add new scalars to handle all
 > currently invalid UTF-8 sequences.

AFAIK there are about 2^31 of these, though!


From hrvoje.niksic at avl.com  Tue Sep 30 12:52:56 2008
From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=)
Date: Tue, 30 Sep 2008 12:52:56 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48E1F53B.7030901@gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>	<48E0B8FB.9070701@egenix.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com>
Message-ID: <1222771976.2598.39.camel@localhost>

On Tue, 2008-09-30 at 19:45 +1000, Nick Coghlan wrote:
> To my mind, there are two kinds of app in the world when it comes to
> file paths:
> 1) "Normal" apps (e.g. a word processor), that are only interested in
> files with sane, well-formed file names that can be properly decoded to
> Unicode with the filesystem encoding identified by Python. If there is
> invalid data on the filesystem, they don't care and don't want to see it
> or have to deal with it.

I am not convinced that a word processor can just ignore files with
(what it thinks are) undecodable file names.  In countries with a
history of incompatible national encodings, such file names crop up very
often, sometimes as a natural consequence of data migrating from older
systems to newer ones.  You can and do encounter "invalid" file names in
the filesystems of mainstream users even without them using buggy or
obsolete software.



From jmatejek at suse.cz  Tue Sep 30 13:27:33 2008
From: jmatejek at suse.cz (=?ISO-8859-1?Q?Jan_Mate=28jek?=)
Date: Tue, 30 Sep 2008 13:27:33 +0200
Subject: [Python-Dev] Python security team
In-Reply-To: <ca471dc20809291016l516cae9ev8b9676ecef60f40@mail.gmail.com>
References: <200809271754.29068.victor.stinner@haypocalc.com>	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>	<48E0C60B.5060006@novell.com>
	<ca471dc20809291016l516cae9ev8b9676ecef60f40@mail.gmail.com>
Message-ID: <48E20D25.4090503@suse.cz>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Guido van Rossum napsal(a):
> I think we may have to expand our selection creteria, since the
> existing approach has led to a small PSRT whose members are all too
> busy to do the necessary legwork. At the same time we need to remain
> selective -- I don't think having a crowd of hundreds would be
> productive, and we need to be sure that every single member can
> absolutely be trusted to take security seriously.

of course

> 
> To answer your question directly, I don't think that just being the
> Python maintainer for some Linux distribution is enough to qualify --
> if our process worked well enough, you'd be getting the patches from
> us via some downstream-flowing distribution mechanism that reaches
> only trusted people within each vendor organization. I don't happen to

Thanks for your answer. I guess the process is the real problem then.
- From what i could observe, the connection between vendor-sec and PSRT is
not really working as it should.
(And then of course you need some kind of upstream flow too, because not
everyone reports to PSRT.)

> know you personally -- but perhaps other current members of the PSRT
> do and that could be enough to secure an invitation.
> 
No, i don't think that i'm known well enough to earn the invitation
(yet), this was more of a "so how the hell does it really work" question.


regards,
jan matejek
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.9 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org

iEYEARECAAYFAkjiDSUACgkQjBrWA+AvBr+zVwCfRGPsDUjREfUKBk7/9yzxDTRN
egUAoLQlQe1qJHU9IkbigpevDme6OqwT
=BYl7
-----END PGP SIGNATURE-----

From steve at holdenweb.com  Tue Sep 30 14:15:39 2008
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 30 Sep 2008 08:15:39 -0400
Subject: [Python-Dev] Python security team
In-Reply-To: <48E20D25.4090503@suse.cz>
References: <200809271754.29068.victor.stinner@haypocalc.com>	<bbaeab100809271745p2679ed20y7816bab079b052fd@mail.gmail.com>	<48E0C60B.5060006@novell.com>	<ca471dc20809291016l516cae9ev8b9676ecef60f40@mail.gmail.com>
	<48E20D25.4090503@suse.cz>
Message-ID: <gbt59b$ilt$1@ger.gmane.org>

Jan Mate wrote:
> Guido van Rossum napsal(a):
[...]
>> know you personally -- but perhaps other current members of the PSRT
>> do and that could be enough to secure an invitation.
> 
> No, i don't think that i'm known well enough to earn the invitation
> (yet), this was more of a "so how the hell does it really work" question.
> 
I haven't yet heard anyone make a convincing case that it does. It is a
great idea, and we *do* need to take security seriously, but at present
all we have is a bunch of well-intentioned and over-committed volunteers.

regards
 Steve
-- 
Steve Holden        +1 571 484 6266   +1 800 494 3119
Holden Web LLC              http://www.holdenweb.com/


From eckhardt at satorlaser.com  Tue Sep 30 14:30:35 2008
From: eckhardt at satorlaser.com (Ulrich Eckhardt)
Date: Tue, 30 Sep 2008 14:30:35 +0200
Subject: [Python-Dev] when is path==NULL?
Message-ID: <200809301430.35410.eckhardt@satorlaser.com>

Hi!

I'm looking at trunk/Python/sysmodule.c, function PySys_SetArgv(). In that 
function, there is code like this:

  PyObject* path = PySys_GetObject("path");
  ...
  if (path != NULL) {
    ...
  }

My intuition says that if path==NULL, something is very wrong. At least I 
would expect to get 'None', but never NULL, except when out of memory. So, 
for the case that path==NULL', I would simply invoke Py_FatalError("no mem 
for sys.path"), similarly to the other call there.

Sounds reasonable?

Uli

-- 
Sator Laser GmbH
Gesch?ftsf?hrer: Thorsten F?cking, Amtsgericht Hamburg HR B62 932

**************************************************************************************
           Visit our website at <http://www.satorlaser.de/>
**************************************************************************************
Diese E-Mail einschlie?lich s?mtlicher Anh?nge ist nur f?r den Adressaten bestimmt und kann vertrauliche Informationen enthalten. Bitte benachrichtigen Sie den Absender umgehend, falls Sie nicht der beabsichtigte Empf?nger sein sollten. Die E-Mail ist in diesem Fall zu l?schen und darf weder gelesen, weitergeleitet, ver?ffentlicht oder anderweitig benutzt werden.
E-Mails k?nnen durch Dritte gelesen werden und Viren sowie nichtautorisierte ?nderungen enthalten. Sator Laser GmbH ist f?r diese Folgen nicht verantwortlich.

**************************************************************************************


From rhamph at gmail.com  Tue Sep 30 14:36:51 2008
From: rhamph at gmail.com (Adam Olsen)
Date: Tue, 30 Sep 2008 06:36:51 -0600
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <87iqsdev0i.fsf@xemacs.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<gbsgk6$kc1$1@ger.gmane.org>
	<aac2c7cb0809292352l40a3cca1h64a8046fd7cd4364@mail.gmail.com>
	<87iqsdev0i.fsf@xemacs.org>
Message-ID: <aac2c7cb0809300536l3004acees8790141f6da04b44@mail.gmail.com>

On Tue, Sep 30, 2008 at 5:24 AM, Stephen J. Turnbull <stephen at xemacs.org> wrote:
> Adam Olsen writes:
>
>  > [1] You could argue that Unicode should add new scalars to handle all
>  > currently invalid UTF-8 sequences.
>
> AFAIK there are about 2^31 of these, though!

They've promised to never allocate above U+10FFFF (0 to 1114111).  Not
sure that makes new additions easier or harder. ;)

-- 
Adam Olsen, aka Rhamphoryncus

From lists at cheimes.de  Tue Sep 30 14:48:40 2008
From: lists at cheimes.de (Christian Heimes)
Date: Tue, 30 Sep 2008 14:48:40 +0200
Subject: [Python-Dev] when is path==NULL?
In-Reply-To: <200809301430.35410.eckhardt@satorlaser.com>
References: <200809301430.35410.eckhardt@satorlaser.com>
Message-ID: <gbt777$p1i$1@ger.gmane.org>

Ulrich Eckhardt wrote:
> Hi!
> 
> I'm looking at trunk/Python/sysmodule.c, function PySys_SetArgv(). In that 
> function, there is code like this:
> 
>   PyObject* path = PySys_GetObject("path");
>   ...
>   if (path != NULL) {
>     ...
>   }
> 
> My intuition says that if path==NULL, something is very wrong. At least I 
> would expect to get 'None', but never NULL, except when out of memory. So, 
> for the case that path==NULL', I would simply invoke Py_FatalError("no mem 
> for sys.path"), similarly to the other call there.

PySys_GetObject may return NULL after the user has removed sys.path with 
delattr(sys, 'path'). There are valid applications for removing sys.path.

Christian


From tom at vector-seven.com  Tue Sep 30 14:42:11 2008
From: tom at vector-seven.com (Thomas Lee)
Date: Tue, 30 Sep 2008 22:42:11 +1000
Subject: [Python-Dev] when is path==NULL?
In-Reply-To: <200809301430.35410.eckhardt@satorlaser.com>
References: <200809301430.35410.eckhardt@satorlaser.com>
Message-ID: <48E21EA3.70607@vector-seven.com>

Ulrich Eckhardt wrote:
> Hi!
>
> I'm looking at trunk/Python/sysmodule.c, function PySys_SetArgv(). In that 
> function, there is code like this:
>
>   PyObject* path = PySys_GetObject("path");
>   ...
>   if (path != NULL) {
>     ...
>   }
>
> My intuition says that if path==NULL, something is very wrong. At least I 
> would expect to get 'None', but never NULL, except when out of memory. So, 
> for the case that path==NULL', I would simply invoke Py_FatalError("no mem 
> for sys.path"), similarly to the other call there.
>
> Sounds reasonable?
>
> Uli
>
>   
Maybe it's just being safe?

 From Python/sysmodule.c:

    PyThreadState *tstate = PyThreadState_GET();
    PyObject *sd = tstate->interp->sysdict;
    if (sd == NULL)
        return NULL;
    return PyDict_GetItemString(sd, name);


So if tstate->interp->sysdict is NULL, we return NULL. That's probably a 
bit unlikely.

However, PyDict_GetItemString attempts to allocate a new PyString from 
the given char* key. If that fails, PySys_GetObject will also return 
NULL -- just like most functions in the code base that hit an out of 
memory error:

PyObject *
PyDict_GetItemString(PyObject *v, const char *key)
{
    PyObject *kv, *rv;
    kv = PyString_FromString(key);
    if (kv == NULL)
        return NULL;
    rv = PyDict_GetItem(v, kv);
    Py_DECREF(kv);
    return rv;
}

Seems perfectly reasonable for it to return NULL in this situation.

Cheers,
T


From tom at vector-seven.com  Tue Sep 30 14:46:08 2008
From: tom at vector-seven.com (Thomas Lee)
Date: Tue, 30 Sep 2008 22:46:08 +1000
Subject: [Python-Dev] when is path==NULL?
In-Reply-To: <200809301430.35410.eckhardt@satorlaser.com>
References: <200809301430.35410.eckhardt@satorlaser.com>
Message-ID: <48E21F90.6050904@vector-seven.com>

Ulrich Eckhardt wrote:
> Hi!
>
> I'm looking at trunk/Python/sysmodule.c, function PySys_SetArgv(). In that 
> function, there is code like this:
>
>   PyObject* path = PySys_GetObject("path");
>   ...
>   if (path != NULL) {
>     ...
>   }
>
> My intuition says that if path==NULL, something is very wrong. At least I 
> would expect to get 'None', but never NULL, except when out of memory. So, 
> for the case that path==NULL', I would simply invoke Py_FatalError("no mem 
> for sys.path"), similarly to the other call there.
>
> Sounds reasonable?
>
> Uli
>
>   
I also meant to mention that there might be a reason why we want the out 
of memory error to bubble up to the caller should that happen while 
attempting to allocate the PyString in PyDict_GetItemString, rather than 
just bailing out with a generic FatalError.

Cheers,
T

From glyph at divmod.com  Tue Sep 30 15:21:51 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Tue, 30 Sep 2008 13:21:51 -0000
Subject: [Python-Dev] Patch for an initial support of bytes filename
	in	Python3
In-Reply-To: <200809300247.20349.victor.stinner@haypocalc.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
Message-ID: <20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>

On 12:47 am, victor.stinner at haypocalc.com wrote:

This is the most sane contribution I've seen so far :).
>See attached patch: python3_bytes_filename.patch
>
>Using the patch, you will get:
>- open() support bytes
>- listdir(unicode) -> only unicode, *skip* invalid filenames
>   (as asked by Guido)

Forgive me for being a bit dense, but I couldn't find this hunk in the 
patch.  Do I understand properly that (listdir(bytes) -> bytes)?

If so, this seems basically sane to me, since it provides text behavior 
where possible and allows more sophisticated filesystem wrappers (i.e. 
Twisted's FilePath, Will McGugan's "FS") to do more tricky things, 
separating filenames for display to the user and filenames for exchange 
with the FS.
>- remove os.getcwdu()
>- create os.getcwdb() -> bytes
>- glob.glob() support bytes
>- fnmatch.filter() support bytes
>- posixpath.join() and posixpath.split() support bytes

It sounds like maybe there should be some 2to3 fixers in here somewhere, 
too?  Not necessarily as part of this patch, but somewhere related?  I 
don't know what they would do, but it does seem quite likely that code 
which was previously correct under 2.6 (using bytes) would suddenly be 
mixing bytes and unicode with these APIs.

From guido at python.org  Tue Sep 30 15:50:10 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 06:50:10 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbs7up$1q5$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
	<200809300129.24972.victor.stinner@haypocalc.com>
	<gbs7up$1q5$1@ger.gmane.org>
Message-ID: <ca471dc20809300650i7cb5c9adl38f5cbe7aa9fc18@mail.gmail.com>

On Mon, Sep 29, 2008 at 8:55 PM, Terry Reedy <tjreedy at udel.edu> wrote:
>
>> Le Monday 29 September 2008 19:06:01 Guido van Rossum, vous avez ?crit :
>
>>> I know I keep flipflopping on this one, but the more I think about it
>>> the more I believe it is better to drop those names than to raise an
>>> exception. Otherwise a "naive" program that happens to use
>>> os.listdir() can be rendered completely useless by a single non-UTF-8
>>> filename. Consider the use of os.listdir() by the glob module. If I am
>>> globbing for *.py, why should the presence of a file named b'\xff'
>>> cause it to fail?
>
> To avoid silent skipping, is it possible to drop 'unreadable' names, issue a
> warning (instead of exception), and continue to completion?
> "Warning: unreadable filename skipped; see PyWiki/UnreadableFilenames"

That would be annoying as hell in most cases.

I consider the dropping of unreadable names similar to the suppression
of "hidden" files by various operating systems.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 15:53:09 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 06:53:09 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E1C097.8030309@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de>
Message-ID: <ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>

On Mon, Sep 29, 2008 at 11:00 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>> Change the default file system encoding to store bytes in Unicode is like
>> introducing a new Python type: <fake Unicode for filename hacks>.
>
> Exactly. Seems like the best solution to me, despite your polemics.

Martin, I don't understand why you are in favor of storing raw bytes
encoded as Latin-1 in Unicode string objects, which clearly gives rise
to mojibake. In the past you have always been staunchly opposed to API
changes or practices that could lead to mojibake (and you had me quite
convinced).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From victor.stinner at haypocalc.com  Tue Sep 30 15:54:20 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 30 Sep 2008 15:54:20 +0200
Subject: [Python-Dev]
	=?iso-8859-1?q?Patch_for_an_initial_support_of_bytes?=
	=?iso-8859-1?q?_filename_in=09Python3?=
In-Reply-To: <20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
Message-ID: <200809301554.21222.victor.stinner@haypocalc.com>

Hi,

> This is the most sane contribution I've seen so far :).

Oh thanks.

> Do I understand properly that (listdir(bytes) -> bytes)?

Yes, os.listdir(bytes)->bytes. It's already the current behaviour.

But with Python3 trunk, os.listdir(str) -> str ... or bytes (if unicode 
conversion fails).

> If so, this seems basically sane to me, since it provides text behavior
> where possible and allows more sophisticated filesystem wrappers (i.e.
> Twisted's FilePath, Will McGugan's "FS") to do more tricky things,
> separating filenames for display to the user and filenames for exchange
> with the FS.

It's the goal of my patch. Let people do what you want with bytes: rename the 
file, try the best charset to display the filename, etc.

> >- remove os.getcwdu()
> >- create os.getcwdb() -> bytes
> >- glob.glob() support bytes
> >- fnmatch.filter() support bytes
> >- posixpath.join() and posixpath.split() support bytes
>
> It sounds like maybe there should be some 2to3 fixers in here somewhere,
> too?

IMHO a programmer should not use bytes for filenames. Only specific programs 
used to fix a broken system (eg. convmv program), a backup program, etc. 
should use bytes. So the "default" type (type and not charset) for filenames 
should be str in Python3.

If my patch would be applied, 2to3 have to replace getcwdu() to getcwd(). 
That's all.

> Not necessarily as part of this patch, but somewhere related?  I 
> don't know what they would do, but it does seem quite likely that code
> which was previously correct under 2.6 (using bytes) would suddenly be
> mixing bytes and unicode with these APIs.

It looks like 2to3 convert all text '...' or u'...' to unicode (str). So 
converted programs will use str for filenames.

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From guido at python.org  Tue Sep 30 15:59:42 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 06:59:42 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbsgk6$kc1$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<gbsgk6$kc1$1@ger.gmane.org>
Message-ID: <ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>

On Mon, Sep 29, 2008 at 11:22 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> No, that was not what I meant (although it is another possibility). As I wrote,
> Martin's proposal that I support here is using the modified UTF-8 codec that
> successfully roundtrips otherwise invalid UTF-8 data.

I thought that the "successful rountripping" pretty much stopped as
soon as the unicode data is exported to somewhere else -- doesn't it
contain invalid surrogate sequences?

In general, I'm very reluctant to use utf-8b given that it doesn't
seem to be well documented as a standard anywhere. Providing some
minimal APIs that can process raw-bytes filenames still makes more
sense -- it is mostly analogous of our treatment of text files, where
the underlying binary data is also accessible.

> You seem to forget that (disregarding OSX here, since it already enforces
> UTF-8) the majority of file names on Posix systems will be encoded correctly.

Apparently under certain circumstances (external FS mounted) OSX can
also have non-UTF-8 filenames.

[...]

> With the filenames decoded by UTF-8, your files named t?ste, ?, dossi? will
> be displayed and handled correctly. The others are *invalid* in the filesystem
> encoding UTF-8 and therefore would be represented by something like
>
> u'dir\uXXffname' where XX is some private use Unicode namespace. It won't look
> pretty when printed, but then, what do other applications do? They e.g. display
> a question mark as you show above, which is not better in terms of readability.
>
> But it will work when given to a filename-handling function. Valid filenames
> can be compared to Unicode strings.
>
> A real-world example: OpenOffice can't open files with invalid bytes in their
> name. They are displayed in the "Open file" dialog, but trying to open fails.
> This regularly drives me crazy. Let's not make Python not work this way too,
> or, even worse, not even display those filenames.

How can it *regularly* drive you crazy when "the majority of fie names
[...] encoded correctly" (as you assert above)?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 16:05:58 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 07:05:58 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E20017.3020405@egenix.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de> <48E20017.3020405@egenix.com>
Message-ID: <ca471dc20809300705x58aa87acn5e3760891e6b57b9@mail.gmail.com>

On Tue, Sep 30, 2008 at 3:31 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> On 2008-09-30 08:00, Martin v. L?wis wrote:
>>> Change the default file system encoding to store bytes in Unicode is like
>>> introducing a new Python type: <fake Unicode for filename hacks>.
>>
>> Exactly. Seems like the best solution to me, despite your polemics.
>
> Not a bad idea... have os.listdir() return Unicode subclasses that work
> like file handles, ie. they have an extra buffer that holds the original
> bytes value received from the underlying C API.
>
> Passing these handles to open() would then do the right thing by using
> whatever os.listdir() got back from the file system to open the file,
> while still providing a sane way to display the filename, e.g. using
> question marks for the invalid characters.
>
> The only problem with this approach is concatenation of such handles
> to form pathnames, but then perhaps those concatenations could just
> work on the bytes value as well (I don't know of any OS that uses non-
> ASCII path separators).

While this seems to work superficially I expect an infinite number of
problems caused by code that doesn't understand this subclass. You are
hinting at this in your last paragraph.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From victor.stinner at haypocalc.com  Tue Sep 30 16:11:02 2008
From: victor.stinner at haypocalc.com (Victor Stinner)
Date: Tue, 30 Sep 2008 16:11:02 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de>
	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
Message-ID: <200809301611.03027.victor.stinner@haypocalc.com>

Le Tuesday 30 September 2008 15:53:09 Guido van Rossum, vous avez ?crit?:
> On Mon, Sep 29, 2008 at 11:00 PM, "Martin v. L?wis" <martin at v.loewis.de> 
wrote:
> >> Change the default file system encoding to store bytes in Unicode is
> >> like introducing a new Python type: <fake Unicode for filename hacks>.
> >
> > Exactly. Seems like the best solution to me, despite your polemics.
>
> Martin, I don't understand why you are in favor of storing raw bytes
> encoded as Latin-1 in Unicode string objects, which clearly gives rise
> to mojibake. In the past you have always been staunchly opposed to API
> changes or practices that could lead to mojibake (and you had me quite
> convinced).

If I understood correctly, the goal of Python3 is the clear *separation* of 
bytes and characters. Store bytes in Unicode is pratical because it doesn't 
need to change the existing code, but it doesn't fix the problem, it's just 
move problems which be raised later.

I didn't get an answer to my question: what is the result <bytes (fake 
characters) stored in unicode> + <real unicode>? I guess that the result is 
<mixed "bytes" and characters in unicode> instead of raising an error 
(invalid types). So again: why introducing a new type instead of reusing 
existing Python types?

-- 
Victor Stinner aka haypo
http://www.haypocalc.com/blog/

From guido at python.org  Tue Sep 30 16:22:11 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 07:22:11 -0700
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <48E1F53B.7030901@gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<48E0B8FB.9070701@egenix.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com>
Message-ID: <ca471dc20809300722x4a700e85v153732843f43c920@mail.gmail.com>

On Tue, Sep 30, 2008 at 2:45 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Adam Olsen wrote:
>> Lossy conversion just moves around what gets treated as garbage.  As
>> all valid unicode scalars can be round tripped, there's no way to
>> create a valid unicode file name without being lossy.  The alternative
>> is not be valid unicode, but since we can't use such objects with
>> external libs, can't even print them, we might as well call them
>> something else.  We already have a name for that: bytes.
>
> To my mind, there are two kinds of app in the world when it comes to
> file paths:
> 1) "Normal" apps (e.g. a word processor), that are only interested in
> files with sane, well-formed file names that can be properly decoded to
> Unicode with the filesystem encoding identified by Python. If there is
> invalid data on the filesystem, they don't care and don't want to see it
> or have to deal with it.
> 2) "Filesystem" apps (e.g. a filesystem explorer), that need to be able
> to deal with malformed filenames that may not decode properly using the
> identified filesystem encoding.
>
> For the former category of apps, the presence of a malformed filename
> should NOT disrupt the processing of well-formed files and directories.
> Those applications should "just work", even if the underlying filesystem
> has a few broken filenames.

Right. Totally agreed.

> The latter category of applications need some way of defining their own
> application-specific handling of malformed names.

Agreed again.

> That screams "callback" to me - and one mechanism to achieve that would
> be to expose the unicode "errors" argument for filesystem operations
> that return file paths (e.g. os.getcwd(), os.listdir(), os.readlink(),
> os.walk()).

Hm. This doesn't scream callback to me at all. I would never have
thought of callbacks for this use case -- and I don't think it's a
good idea. The callback would either be an extra argument to all
system calls (bad, ugly etc., and why not go with the existing unicode
encoding and error flags if we're adding extra args?) or would be
global, where I'd be worried that it might interfere with the proper
operation of library code that is several abstractions away from
whoever installed the callback, not under their control, and not
expecting the callback.

I suppose I may have totally misunderstood your proposal, but in
general I find callbacks unwieldy.

> Once that was exposed, the existing error handling machinery in the
> codecs module could be used to allow applications to define their own
> custom error handling for Unicode decode errors in these operations.
> (e.g. set "codecs.register_error('bad_filepath',
> handle_filepath_error)", then use "errors='bad_filepath'" in the
> relevant os API calls)
>
> The default handling could be left at "strict", with os.listdir() and
> os.walk() specifically ignoring path entries that trigger
> UnicodeDecodeError.
>
> getcwd() and readlink() could just propagate the exception, since they
> have no other information to return.



-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 16:26:13 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 07:26:13 -0700
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <1222771976.2598.39.camel@localhost>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<48E0B8FB.9070701@egenix.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com> <1222771976.2598.39.camel@localhost>
Message-ID: <ca471dc20809300726g2d3b0d43qcaad50608d0283b3@mail.gmail.com>

On Tue, Sep 30, 2008 at 3:52 AM, Hrvoje Nik?i? <hrvoje.niksic at avl.com> wrote:
> On Tue, 2008-09-30 at 19:45 +1000, Nick Coghlan wrote:
>> To my mind, there are two kinds of app in the world when it comes to
>> file paths:
>> 1) "Normal" apps (e.g. a word processor), that are only interested in
>> files with sane, well-formed file names that can be properly decoded to
>> Unicode with the filesystem encoding identified by Python. If there is
>> invalid data on the filesystem, they don't care and don't want to see it
>> or have to deal with it.
>
> I am not convinced that a word processor can just ignore files with
> (what it thinks are) undecodable file names.  In countries with a
> history of incompatible national encodings, such file names crop up very
> often, sometimes as a natural consequence of data migrating from older
> systems to newer ones.  You can and do encounter "invalid" file names in
> the filesystems of mainstream users even without them using buggy or
> obsolete software.

This is a quality of implementation issue. Either the word processor
is written to support "undecodable" files, or it isn't. If it isn't,
there's nothing that can be done about it (short of buying another
wordprocessor) and it shouldn't be crippled by the mere *presence* of
an undecodable file in a directory. I can think of lots of apps that
have a sufficiently small or homogeneous audience (e.g. lots of
in-house apps) that they don't need to care about such files, and
these shouldn't break when they are used in the vicinity of an
undecodable filename -- it's enough if they just ignore it.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 16:27:53 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 07:27:53 -0700
Subject: [Python-Dev] when is path==NULL?
In-Reply-To: <gbt777$p1i$1@ger.gmane.org>
References: <200809301430.35410.eckhardt@satorlaser.com>
	<gbt777$p1i$1@ger.gmane.org>
Message-ID: <ca471dc20809300727y78803c45j8b905dd9c3a10cf2@mail.gmail.com>

On Tue, Sep 30, 2008 at 5:48 AM, Christian Heimes <lists at cheimes.de> wrote:
> Ulrich Eckhardt wrote:
>>
>> Hi!
>>
>> I'm looking at trunk/Python/sysmodule.c, function PySys_SetArgv(). In that
>> function, there is code like this:
>>
>>  PyObject* path = PySys_GetObject("path");
>>  ...
>>  if (path != NULL) {
>>    ...
>>  }
>>
>> My intuition says that if path==NULL, something is very wrong. At least I
>> would expect to get 'None', but never NULL, except when out of memory. So,
>> for the case that path==NULL', I would simply invoke Py_FatalError("no mem
>> for sys.path"), similarly to the other call there.
>
> PySys_GetObject may return NULL after the user has removed sys.path with
> delattr(sys, 'path'). There are valid applications for removing sys.path.

Or before sys.path is initialized using PySys_SetPath(). Trust me,
this code is as it should be.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 16:32:38 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 07:32:38 -0700
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
Message-ID: <ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>

On Tue, Sep 30, 2008 at 6:21 AM,  <glyph at divmod.com> wrote:
> On 12:47 am, victor.stinner at haypocalc.com wrote:
>
> This is the most sane contribution I've seen so far :).

Thanks. I'll review it later today (after coffee+breakfast :) and will
apply it assuming the code is reasonably sane, otherwise I'll go
around with Victor until it is to my satisfaction.

>> See attached patch: python3_bytes_filename.patch
>>
>> Using the patch, you will get:
>> - open() support bytes
>> - listdir(unicode) -> only unicode, *skip* invalid filenames
>>  (as asked by Guido)
>
> Forgive me for being a bit dense, but I couldn't find this hunk in the
> patch.  Do I understand properly that (listdir(bytes) -> bytes)?
>
> If so, this seems basically sane to me, since it provides text behavior
> where possible and allows more sophisticated filesystem wrappers (i.e.
> Twisted's FilePath, Will McGugan's "FS") to do more tricky things,
> separating filenames for display to the user and filenames for exchange with
> the FS.
>>
>> - remove os.getcwdu()
>> - create os.getcwdb() -> bytes
>> - glob.glob() support bytes
>> - fnmatch.filter() support bytes
>> - posixpath.join() and posixpath.split() support bytes
>
> It sounds like maybe there should be some 2to3 fixers in here somewhere,
> too?  Not necessarily as part of this patch, but somewhere related?  I don't
> know what they would do, but it does seem quite likely that code which was
> previously correct under 2.6 (using bytes) would suddenly be mixing bytes
> and unicode with these APIs.

Doesn't seem easy for 2to3 to recognize such cases.

If 2.6 weren't pretty much released already I'd ask to add
os.getcwdb() there, as an alias for os.getcwd(), and add a 2to3 fixer
that converts os.getcwdu() to os.getcwd(), leaves os.getcwd() alone
(benefit of the doubt) and leaves os.getcwdb() alone as well (a strong
indication the user meant to get bytes in the 3.x version of their
code. (Similar to using bytes instead of str in 2.6 even though they
mean the same thing there -- they will be properly separated in 3.x.)

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From hrvoje.niksic at avl.com  Tue Sep 30 16:39:45 2008
From: hrvoje.niksic at avl.com (Hrvoje =?UTF-8?Q?Nik=C5=A1i=C4=87?=)
Date: Tue, 30 Sep 2008 16:39:45 +0200
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <ca471dc20809300726g2d3b0d43qcaad50608d0283b3@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<48E0B8FB.9070701@egenix.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com> <1222771976.2598.39.camel@localhost>
	<ca471dc20809300726g2d3b0d43qcaad50608d0283b3@mail.gmail.com>
Message-ID: <1222785585.2598.45.camel@localhost>

On Tue, 2008-09-30 at 07:26 -0700, Guido van Rossum wrote:
> > I am not convinced that a word processor can just ignore files with
> > (what it thinks are) undecodable file names.  In countries with a
> > history of incompatible national encodings, such file names crop up very
> > often, sometimes as a natural consequence of data migrating from older
> > systems to newer ones.  You can and do encounter "invalid" file names in
> > the filesystems of mainstream users even without them using buggy or
> > obsolete software.
> 
> This is a quality of implementation issue. Either the word processor
> is written to support "undecodable" files, or it isn't. If it isn't,
> there's nothing that can be done about it (short of buying another
> wordprocessor)

I agree with this.  I just believe the underlying python APIs shouldn't
make it impossible (or unnecessarily hard) for the word processor to
implement showing of files with undecodable names.

For example, implementing os.listdir to return the file names as Unicode
subclasses with ability to access the underlying bytes (automatically
recognized by open and friends) sounds like a good compromise that
allows the word processor to both have the cake and eat it.


From steve at pearwood.info  Tue Sep 30 16:53:10 2008
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 1 Oct 2008 00:53:10 +1000
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <ca471dc20809300650i7cb5c9adl38f5cbe7aa9fc18@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbs7up$1q5$1@ger.gmane.org>
	<ca471dc20809300650i7cb5c9adl38f5cbe7aa9fc18@mail.gmail.com>
Message-ID: <200810010053.10890.steve@pearwood.info>

On Tue, 30 Sep 2008 11:50:10 pm Guido van Rossum wrote:

> > To avoid silent skipping, is it possible to drop 'unreadable'
> > names, issue a warning (instead of exception), and continue to
> > completion? "Warning: unreadable filename skipped; see
> > PyWiki/UnreadableFilenames"
>
> That would be annoying as hell in most cases.

Doesn't the warning module default to only displaying the warning once 
per session? I don't see that it would be annoying as hell to be 
notified once per session that an error has occurred.

What I'd find annoying as hell would be something like this:

$ ls . | wc -l
25
$ python
...
>>> import os
>>> len(os.listdir('.')
24


Give me a nice clear error, or even a warning. Don't let the error pass 
silently, unless I explicitly silence it.


> I consider the dropping of unreadable names similar to the
> suppression of "hidden" files by various operating systems.

With the exception of '.' and '..', I consider "hidden" files to be a 
serious design mistake, but at least most operating systems give the 
user a way to easily see all such hidden files if you ask.

(Almost all. Windows has "superhidden" files that remain hidden even 
when the user asks to see hidden files, all the better to hide malware. 
But that's a rant for another list.)

 

-- 
Steven

From mal at egenix.com  Tue Sep 30 17:20:42 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 30 Sep 2008 17:20:42 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <ca471dc20809300705x58aa87acn5e3760891e6b57b9@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>	<48E1C097.8030309@v.loewis.de>
	<48E20017.3020405@egenix.com>
	<ca471dc20809300705x58aa87acn5e3760891e6b57b9@mail.gmail.com>
Message-ID: <48E243CA.1090604@egenix.com>

On 2008-09-30 16:05, Guido van Rossum wrote:
> On Tue, Sep 30, 2008 at 3:31 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> On 2008-09-30 08:00, Martin v. L?wis wrote:
>>>> Change the default file system encoding to store bytes in Unicode is like
>>>> introducing a new Python type: <fake Unicode for filename hacks>.
>>> Exactly. Seems like the best solution to me, despite your polemics.
>> Not a bad idea... have os.listdir() return Unicode subclasses that work
>> like file handles, ie. they have an extra buffer that holds the original
>> bytes value received from the underlying C API.
>>
>> Passing these handles to open() would then do the right thing by using
>> whatever os.listdir() got back from the file system to open the file,
>> while still providing a sane way to display the filename, e.g. using
>> question marks for the invalid characters.
>>
>> The only problem with this approach is concatenation of such handles
>> to form pathnames, but then perhaps those concatenations could just
>> work on the bytes value as well (I don't know of any OS that uses non-
>> ASCII path separators).
> 
> While this seems to work superficially I expect an infinite number of
> problems caused by code that doesn't understand this subclass. You are
> hinting at this in your last paragraph.

Well, to some extent Unicode objects themselves already implement
such a strategy: the default encoded bytes object basically provides
the low-level interfacing value.

But I agree, the approach is not foolproof.

In the end, I think it's better not to be clever and just return
the filenames that cannot be decoded as bytes objects in os.listdir().

Passing those to open() will then open the files as expected, in most
other cases the application will have to provide explicit conversions
in whatever way best fits the application.

Also note that os.listdir() isn't the only source of filesnames. You
often read them from a file, a database, some socket, etc, so letting
the application decide what to do is not asking too much, IMHO.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 30 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From janssen at parc.com  Tue Sep 30 17:47:30 2008
From: janssen at parc.com (Bill Janssen)
Date: Tue, 30 Sep 2008 08:47:30 PDT
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <200809300247.20349.victor.stinner@haypocalc.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
Message-ID: <58953.1222789650@parc.com>

Victor Stinner <victor.stinner at haypocalc.com> wrote:

>  - listdir(unicode) -> only unicode, *skip* invalid filenames 
>    (as asked by Guido)

Is there an option listdir(bytes) which will return *all* filenames (as
byte sequences)?  Otherwise, this seems troubling to me; *something*
should be returned for filenames which can't be represented, even if
it's only None.

Bill

From guido at python.org  Tue Sep 30 18:27:26 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 09:27:26 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <200810010053.10890.steve@pearwood.info>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbs7up$1q5$1@ger.gmane.org>
	<ca471dc20809300650i7cb5c9adl38f5cbe7aa9fc18@mail.gmail.com>
	<200810010053.10890.steve@pearwood.info>
Message-ID: <ca471dc20809300927j46b0c47ds9dfb7f008660afb1@mail.gmail.com>

On Tue, Sep 30, 2008 at 7:53 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Tue, 30 Sep 2008 11:50:10 pm Guido van Rossum wrote:
>
>> > To avoid silent skipping, is it possible to drop 'unreadable'
>> > names, issue a warning (instead of exception), and continue to
>> > completion? "Warning: unreadable filename skipped; see
>> > PyWiki/UnreadableFilenames"
>>
>> That would be annoying as hell in most cases.
>
> Doesn't the warning module default to only displaying the warning once
> per session? I don't see that it would be annoying as hell to be
> notified once per session that an error has occurred.
>
> What I'd find annoying as hell would be something like this:
>
> $ ls . | wc -l
> 25
> $ python
> ...
>>>> import os
>>>> len(os.listdir('.')
> 24

And yet similar discrepancies happen all the time -- ls suppresses
filenames starting with '.', while os.listdir() shows them (except '.'
and '..' themselves). The Mac Finder and its Windows equivalent hide
lots of files from you. And have you considered mount points (on
Unix)?

Face it. Filesystems are black boxes. They have roughly specified
behavior, but calls into them can fail or seem inconsistent for many
reasons -- concurrent changes by other processes, hidden files
(Windows), files that exist but can't be opened due to kernel-level
locking, etc. It's best not to worry too much about this.

Here's another anomaly:

>>> import os
>>> '.snapshot' in os.listdir('.')
False
>>> os.chdir('.snapshot')
>>> os.getcwd()
'/home/guido/bin/.snapshot'
>>>

IOW there's a hidden .snapshot directory that os.listdir() doesn't
return -- but it exists! This is a standard feature on NetApp filers.
(The reason this file is extra hidden is that it gives access to an
infinite set of backups that you don't want to be found by find(1),
os.walk() and their kin.)

> Give me a nice clear error, or even a warning. Don't let the error pass
> silently, unless I explicitly silence it.

Depends on your use case. We're talking here of a family of APIs where
different programs have different needs. I assert that most programs
are best served by an API that doesn't give them surprising and
irrelevant errors, as long as there's also an API for the few that
want to get to the bottom of things (or as close as they can get --
see above '.snapshot' example).

>> I consider the dropping of unreadable names similar to the
>> suppression of "hidden" files by various operating systems.
>
> With the exception of '.' and '..', I consider "hidden" files to be a
> serious design mistake, but at least most operating systems give the
> user a way to easily see all such hidden files if you ask.
>
> (Almost all. Windows has "superhidden" files that remain hidden even
> when the user asks to see hidden files, all the better to hide malware.
> But that's a rant for another list.)

Rant all you want, it won't go away.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 18:46:00 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 09:46:00 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E243CA.1090604@egenix.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de> <48E20017.3020405@egenix.com>
	<ca471dc20809300705x58aa87acn5e3760891e6b57b9@mail.gmail.com>
	<48E243CA.1090604@egenix.com>
Message-ID: <ca471dc20809300946n275084cep66ba478ac93b9de9@mail.gmail.com>

On Tue, Sep 30, 2008 at 8:20 AM, M.-A. Lemburg <mal at egenix.com> wrote:
> In the end, I think it's better not to be clever and just return
> the filenames that cannot be decoded as bytes objects in os.listdir().

Unfortunately that's going to break most code that is using
os.listdir(), so it's hardly an improved experience.

> Passing those to open() will then open the files as expected, in most
> other cases the application will have to provide explicit conversions
> in whatever way best fits the application.

In most cases the app will try to concatenate a pathname given as a
string and then it will fail.

> Also note that os.listdir() isn't the only source of filesnames. You
> often read them from a file, a database, some socket, etc, so letting
> the application decide what to do is not asking too much, IMHO.

In all those cases, the code that reads them is responsible for
picking an encoding or relying on a default encoding, and the
resulting filenames are always expressed as text, not bytes. I don't
think it's the same at all.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 18:47:10 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 09:47:10 -0700
Subject: [Python-Dev] [Python-3000] Patch for an initial support of
	bytes filename in Python3
In-Reply-To: <58953.1222789650@parc.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<58953.1222789650@parc.com>
Message-ID: <ca471dc20809300947q6ba8c4aw1285e84b0e124919@mail.gmail.com>

On Tue, Sep 30, 2008 at 8:47 AM, Bill Janssen <janssen at parc.com> wrote:
> Victor Stinner <victor.stinner at haypocalc.com> wrote:
>
>>  - listdir(unicode) -> only unicode, *skip* invalid filenames
>>    (as asked by Guido)
>
> Is there an option listdir(bytes) which will return *all* filenames (as
> byte sequences)?  Otherwise, this seems troubling to me; *something*
> should be returned for filenames which can't be represented, even if
> it's only None.

Yes, os.listdir() becomes polymorphic -- if you pass it a pathname in
bytes the output is in bytes and it will return everything exactly as
the underlying syscall returns it to you.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From g.brandl at gmx.net  Tue Sep 30 19:28:23 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 30 Sep 2008 19:28:23 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>	<gbsgk6$kc1$1@ger.gmane.org>
	<ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>
Message-ID: <gbtnjo$quh$1@ger.gmane.org>

Guido van Rossum schrieb:

>> With the filenames decoded by UTF-8, your files named t?ste, ?, dossi? will
>> be displayed and handled correctly. The others are *invalid* in the filesystem
>> encoding UTF-8 and therefore would be represented by something like
>>
>> u'dir\uXXffname' where XX is some private use Unicode namespace. It won't look
>> pretty when printed, but then, what do other applications do? They e.g. display
>> a question mark as you show above, which is not better in terms of readability.
>>
>> But it will work when given to a filename-handling function. Valid filenames
>> can be compared to Unicode strings.
>>
>> A real-world example: OpenOffice can't open files with invalid bytes in their
>> name. They are displayed in the "Open file" dialog, but trying to open fails.
>> This regularly drives me crazy. Let's not make Python not work this way too,
>> or, even worse, not even display those filenames.
> 
> How can it *regularly* drive you crazy when "the majority of fie names
> [...] encoded correctly" (as you assert above)?

Because Office files are a) often named with long, seemingly descriptive
filenames, which invariably means umlauts in German, and b) often sent around
between systems, creating encoding problems.

Having seen how much controversy returning an invalid Unicode string sparks,
and given that it really isn't obvious to the newbie either, I think I now agree
that dropping filenames when calling a listdir() that returns Unicode filenames
is the best solution. I'm a little uneasy with having one function for both
bytes and Unicode return, because that kind of str/unicode mixing I thought we
had left behind in 2.x, but of course can live with it.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From g.brandl at gmx.net  Tue Sep 30 19:32:34 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 30 Sep 2008 19:32:34 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <200810010053.10890.steve@pearwood.info>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbs7up$1q5$1@ger.gmane.org>	<ca471dc20809300650i7cb5c9adl38f5cbe7aa9fc18@mail.gmail.com>
	<200810010053.10890.steve@pearwood.info>
Message-ID: <gbtnrj$quh$2@ger.gmane.org>

Steven D'Aprano schrieb:
> On Tue, 30 Sep 2008 11:50:10 pm Guido van Rossum wrote:
> 
>> > To avoid silent skipping, is it possible to drop 'unreadable'
>> > names, issue a warning (instead of exception), and continue to
>> > completion? "Warning: unreadable filename skipped; see
>> > PyWiki/UnreadableFilenames"
>>
>> That would be annoying as hell in most cases.
> 
> Doesn't the warning module default to only displaying the warning once 
> per session? I don't see that it would be annoying as hell to be 
> notified once per session that an error has occurred.
> 
> What I'd find annoying as hell would be something like this:
> 
> $ ls . | wc -l
> 25
> $ python
> ....
>>>> import os
>>>> len(os.listdir('.')
> 24
> 
> 
> Give me a nice clear error, or even a warning. Don't let the error pass 
> silently, unless I explicitly silence it.

Just another data point: I've just looked at Qt, which provides a filesystem
API and whose strings are Unicode, and it seems to drop undecodable filenames
as well.

Georg


-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From janssen at parc.com  Tue Sep 30 19:41:05 2008
From: janssen at parc.com (Bill Janssen)
Date: Tue, 30 Sep 2008 10:41:05 PDT
Subject: [Python-Dev] [Python-3000] Patch for an initial support of
	bytes filename in Python3
In-Reply-To: <ca471dc20809300947q6ba8c4aw1285e84b0e124919@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<58953.1222789650@parc.com>
	<ca471dc20809300947q6ba8c4aw1285e84b0e124919@mail.gmail.com>
Message-ID: <61658.1222796465@parc.com>

Guido van Rossum <guido at python.org> wrote:
> On Tue, Sep 30, 2008 at 8:47 AM, Bill Janssen <janssen at parc.com> wrote:
> > Victor Stinner <victor.stinner at haypocalc.com> wrote:
> >
> >>  - listdir(unicode) -> only unicode, *skip* invalid filenames
> >>    (as asked by Guido)
> >
> > Is there an option listdir(bytes) which will return *all* filenames (as
> > byte sequences)?  Otherwise, this seems troubling to me; *something*
> > should be returned for filenames which can't be represented, even if
> > it's only None.
> 
> Yes, os.listdir() becomes polymorphic -- if you pass it a pathname in
> bytes the output is in bytes and it will return everything exactly as
> the underlying syscall returns it to you.

What about everything else?  For instance, if I call
os.path.join(<bytes>, <bytes>), I presume I get back a <bytes> which can
be passed to os.listdir() to retrieve the contents of that directory.

Bill

From guido at python.org  Tue Sep 30 19:45:55 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 10:45:55 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbtnjo$quh$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<gbsgk6$kc1$1@ger.gmane.org>
	<ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>
	<gbtnjo$quh$1@ger.gmane.org>
Message-ID: <ca471dc20809301045r59251402g3fe947dec3bc7f22@mail.gmail.com>

On Tue, Sep 30, 2008 at 10:28 AM, Georg Brandl <g.brandl at gmx.net> wrote:
>> How can it *regularly* drive you crazy when "the majority of fie names
>> [...] encoded correctly" (as you assert above)?
>
> Because Office files are a) often named with long, seemingly descriptive
> filenames, which invariably means umlauts in German, and b) often sent around
> between systems, creating encoding problems.

Gotcha.

> Having seen how much controversy returning an invalid Unicode string sparks,
> and given that it really isn't obvious to the newbie either, I think I now agree
> that dropping filenames when calling a listdir() that returns Unicode filenames
> is the best solution. I'm a little uneasy with having one function for both
> bytes and Unicode return, because that kind of str/unicode mixing I thought we
> had left behind in 2.x, but of course can live with it.

Well, the *current* Py3k behavior where it may return a mix of bytes
and str instances is really messy, and likely to trip up most code
that doesn't expect it in a way that makes it hard to debug. However
the *proposed* behavior (returns bytes if the arg was bytes, and
returns str when the arg was str) is IMO sane, and no different than
the polymorphism found in len() or many builtin operations.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From mal at egenix.com  Tue Sep 30 19:50:37 2008
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 30 Sep 2008 19:50:37 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <ca471dc20809300946n275084cep66ba478ac93b9de9@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>	<48E1C097.8030309@v.loewis.de>
	<48E20017.3020405@egenix.com>	<ca471dc20809300705x58aa87acn5e3760891e6b57b9@mail.gmail.com>	<48E243CA.1090604@egenix.com>
	<ca471dc20809300946n275084cep66ba478ac93b9de9@mail.gmail.com>
Message-ID: <48E266ED.9020902@egenix.com>

On 2008-09-30 18:46, Guido van Rossum wrote:
> On Tue, Sep 30, 2008 at 8:20 AM, M.-A. Lemburg <mal at egenix.com> wrote:
>> In the end, I think it's better not to be clever and just return
>> the filenames that cannot be decoded as bytes objects in os.listdir().
> 
> Unfortunately that's going to break most code that is using
> os.listdir(), so it's hardly an improved experience.

Right, but this also signals a problem to the application and the
application is in the best position to determine a proper
work-around.

>> Passing those to open() will then open the files as expected, in most
>> other cases the application will have to provide explicit conversions
>> in whatever way best fits the application.
> 
> In most cases the app will try to concatenate a pathname given as a
> string and then it will fail.

True, and that's the right thing to do in those cases.
The application will have to deal with the problem, e.g. convert
the path to bytes and retry the joining, or convert the bytes string
to Latin-1 and then convert the result back to bytes (using Latin-1)
for passing it to open() (which will of course only work if there are
no non-Latin-1 characters in the path dir), or apply a different
filename encoding based on the path and then retry to convert the
bytes filename into Unicode, or ask the user what to do, etc.

There are many possibilities to solve the problem, apply a work-around,
or inform the user of ways to correct it.

>> Also note that os.listdir() isn't the only source of filesnames. You
>> often read them from a file, a database, some socket, etc, so letting
>> the application decide what to do is not asking too much, IMHO.
> 
> In all those cases, the code that reads them is responsible for
> picking an encoding or relying on a default encoding, and the
> resulting filenames are always expressed as text, not bytes. I don't
> think it's the same at all.

What I was trying to say is that you run into the same problem
in other places as well. Trying to have os.listdir() implement
some strategy is not going to solve the problem at large.

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Source  (#1, Sep 30 2008)
>>> Python/Zope Consulting and Support ...        http://www.egenix.com/
>>> mxODBC.Zope.Database.Adapter ...             http://zope.egenix.com/
>>> mxODBC, mxDateTime, mxTextTools ...        http://python.egenix.com/
________________________________________________________________________

:::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,MacOSX for free ! ::::


   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611

From g.brandl at gmx.net  Tue Sep 30 19:52:55 2008
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 30 Sep 2008 19:52:55 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <ca471dc20809301045r59251402g3fe947dec3bc7f22@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>	<gbsgk6$kc1$1@ger.gmane.org>	<ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>	<gbtnjo$quh$1@ger.gmane.org>
	<ca471dc20809301045r59251402g3fe947dec3bc7f22@mail.gmail.com>
Message-ID: <gbtp1o$kq$1@ger.gmane.org>

Guido van Rossum schrieb:
> On Tue, Sep 30, 2008 at 10:28 AM, Georg Brandl <g.brandl at gmx.net> wrote:
>>> How can it *regularly* drive you crazy when "the majority of fie names
>>> [...] encoded correctly" (as you assert above)?
>>
>> Because Office files are a) often named with long, seemingly descriptive
>> filenames, which invariably means umlauts in German, and b) often sent around
>> between systems, creating encoding problems.
> 
> Gotcha.

Which means?

>> Having seen how much controversy returning an invalid Unicode string sparks,
>> and given that it really isn't obvious to the newbie either, I think I now agree
>> that dropping filenames when calling a listdir() that returns Unicode filenames
>> is the best solution. I'm a little uneasy with having one function for both
>> bytes and Unicode return, because that kind of str/unicode mixing I thought we
>> had left behind in 2.x, but of course can live with it.
> 
> Well, the *current* Py3k behavior where it may return a mix of bytes
> and str instances is really messy, and likely to trip up most code
> that doesn't expect it in a way that makes it hard to debug. However
> the *proposed* behavior (returns bytes if the arg was bytes, and
> returns str when the arg was str) is IMO sane, and no different than
> the polymorphism found in len() or many builtin operations.

I agree that everything is better than the current behavior.

Georg

-- 
Thus spake the Lord: Thou shalt indent with four spaces. No more, no less.
Four shall be the number of spaces thou shalt indent, and the number of thy
indenting shall be four. Eight shalt thou not indent, nor either indent thou
two, excepting that thou then proceed to four. Tabs are right out.


From guido at python.org  Tue Sep 30 19:54:15 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 10:54:15 -0700
Subject: [Python-Dev] [Python-3000] Patch for an initial support of
	bytes filename in Python3
In-Reply-To: <61658.1222796465@parc.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<58953.1222789650@parc.com>
	<ca471dc20809300947q6ba8c4aw1285e84b0e124919@mail.gmail.com>
	<61658.1222796465@parc.com>
Message-ID: <ca471dc20809301054x21082592i5a8a58e81223c4f1@mail.gmail.com>

On Tue, Sep 30, 2008 at 10:41 AM, Bill Janssen <janssen at parc.com> wrote:
> Guido van Rossum <guido at python.org> wrote:
>> On Tue, Sep 30, 2008 at 8:47 AM, Bill Janssen <janssen at parc.com> wrote:
>> > Victor Stinner <victor.stinner at haypocalc.com> wrote:
>> >
>> >>  - listdir(unicode) -> only unicode, *skip* invalid filenames
>> >>    (as asked by Guido)
>> >
>> > Is there an option listdir(bytes) which will return *all* filenames (as
>> > byte sequences)?  Otherwise, this seems troubling to me; *something*
>> > should be returned for filenames which can't be represented, even if
>> > it's only None.
>>
>> Yes, os.listdir() becomes polymorphic -- if you pass it a pathname in
>> bytes the output is in bytes and it will return everything exactly as
>> the underlying syscall returns it to you.
>
> What about everything else?  For instance, if I call
> os.path.join(<bytes>, <bytes>), I presume I get back a <bytes> which can
> be passed to os.listdir() to retrieve the contents of that directory.

Yeah, Victor's code at http://bugs.python.org/issue3187 (file
python3_bytes_filename.patch) does this. More needs to be done but
it's a start.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From glyph at divmod.com  Tue Sep 30 19:59:32 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Tue, 30 Sep 2008 17:59:32 -0000
Subject: [Python-Dev] Patch for an initial support of bytes filename
	in	Python3
In-Reply-To: <ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
Message-ID: <20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>

On 02:32 pm, guido at python.org wrote:
>On Tue, Sep 30, 2008 at 6:21 AM,  <glyph at divmod.com> wrote:
>>On 12:47 am, victor.stinner at haypocalc.com wrote:

>>It sounds like maybe there should be some 2to3 fixers in here 
>>somewhere,
>>too?  Not necessarily as part of this patch, but somewhere related?  I 
>>don't
>>know what they would do, but it does seem quite likely that code which 
>>was
>>previously correct under 2.6 (using bytes) would suddenly be mixing 
>>bytes
>>and unicode with these APIs.
>
>Doesn't seem easy for 2to3 to recognize such cases.

Actually I think I'm wrong.  As far as dealing with glob(), listdir() 
and friends, I suppose that other bytes/text fixers will already have 
had their opportunity to deal with getting the type to be the 
appropriate thing, and if you have glob(<something that 2to3 understands 
should be bytes>) it will work as expected in 3.0.  (I am really just 
confirming that I have nothing useful to say here, using too many words 
to do it: at least, I hope that nobody will waste further time thinking 
about it as a result.)
>If 2.6 weren't pretty much released already I'd ask to add
>os.getcwdb() there, as an alias for os.getcwd(), and add a 2to3 fixer
>that converts os.getcwdu() to os.getcwd(), leaves os.getcwd() alone
>(benefit of the doubt) and leaves os.getcwdb() alone as well (a strong
>indication the user meant to get bytes in the 3.x version of their
>code. (Similar to using bytes instead of str in 2.6 even though they
>mean the same thing there -- they will be properly separated in 3.x.)

In the absence of a 2.6 getcwdb, perhaps the fixer could just drop the 
"benefit of the doubt" case?  It could always be added to 2.7, and the 
parity release of 2to3 could have a --2.7 switch that would modify the 
behavior of this and other fixers.

From guido at python.org  Tue Sep 30 19:56:45 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 10:56:45 -0700
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
Message-ID: <ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>

On Tue, Sep 30, 2008 at 10:59 AM,  <glyph at divmod.com> wrote:
> On 02:32 pm, guido at python.org wrote:
>> If 2.6 weren't pretty much released already I'd ask to add
>> os.getcwdb() there, as an alias for os.getcwd(), and add a 2to3 fixer
>> that converts os.getcwdu() to os.getcwd(), leaves os.getcwd() alone
>> (benefit of the doubt) and leaves os.getcwdb() alone as well (a strong
>> indication the user meant to get bytes in the 3.x version of their
>> code. (Similar to using bytes instead of str in 2.6 even though they
>> mean the same thing there -- they will be properly separated in 3.x.)
>
> In the absence of a 2.6 getcwdb, perhaps the fixer could just drop the
> "benefit of the doubt" case?  It could always be added to 2.7, and the
> parity release of 2to3 could have a --2.7 switch that would modify the
> behavior of this and other fixers.

I'm not sure what you're proposing. *My* proposal is that 2to3 changes
os.getcwdu() calls to os.getcwd() and leaves os.getcwd() calls alone
-- there's no way to tell whether os.getcwdb() would be a better
match, and for portable code, it won't be (since os.getcwdb() is a
Unix-only thing).

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From glyph at divmod.com  Tue Sep 30 20:12:31 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Tue, 30 Sep 2008 18:12:31 -0000
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <1222785585.2598.45.camel@localhost>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291250.03291.eckhardt@satorlaser.com>
	<48E0B8FB.9070701@egenix.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com> <1222771976.2598.39.camel@localhost>
	<ca471dc20809300726g2d3b0d43qcaad50608d0283b3@mail.gmail.com>
	<1222785585.2598.45.camel@localhost>
Message-ID: <20080930181231.31635.1557225685.divmod.xquotient.503@weber.divmod.com>

On 02:39 pm, hrvoje.niksic at avl.com wrote:
>For example, implementing os.listdir to return the file names as 
>Unicode
>subclasses with ability to access the underlying bytes (automatically
>recognized by open and friends) sounds like a good compromise that
>allows the word processor to both have the cake and eat it.

It really seems like the strategy of the current patch (which I believe 
Guido proposed) makes the most sense.  Programs pass different arguments 
for different things:

listdir(text) -> I am thinking in unicode and I do not know about 
encodings, please give me only things that are proper unicode, because I 
don't want to deal with that.

listdir(bytes) -> I am thinking about bytes, I know about encodings. 
Just give me filenames as bytes and I will decode them myself or do 
other fancy things.

You can argue about whether this should really be 'listdiru' or 'globu' 
for explicitness, but when such a simple strategy with unambiguous types 
works, there's no reason to introduce some weird hybrid bytes/text type 
that will inevitably be a bug attractor.

Python's path abstractions have never been particularly high level, nor 
do I think they necessarily should be - at least, not until there's some 
community consensus about what a "high level path abstraction" really 
looks like.  We're still wrestling with it in Twisted, and I can think 
of at least three ways that ours is wrong.  And ours is the one that's 
doing the best, as far as I can tell :).

This proposal gives higher level software the information that it needs 
to construct appropriate paths.

The one thing it doesn't do is expose the decoding rules for the higher- 
level applications to deal with.  I am pretty sure I don't understand 
how the interaction between filesystem encoding and user locale works in 
that case, though, so I can't immediately recommend a way to do it.

From guido at python.org  Tue Sep 30 20:16:31 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 11:16:31 -0700
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <20080930181231.31635.1557225685.divmod.xquotient.503@weber.divmod.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com> <1222771976.2598.39.camel@localhost>
	<ca471dc20809300726g2d3b0d43qcaad50608d0283b3@mail.gmail.com>
	<1222785585.2598.45.camel@localhost>
	<20080930181231.31635.1557225685.divmod.xquotient.503@weber.divmod.com>
Message-ID: <ca471dc20809301116y480bd028qbcdca44f05c62596@mail.gmail.com>

On Tue, Sep 30, 2008 at 11:12 AM,  <glyph at divmod.com> wrote:

> The one thing it doesn't do is expose the decoding rules for the higher-
> level applications to deal with.  I am pretty sure I don't understand how
> the interaction between filesystem encoding and user locale works in that
> case, though, so I can't immediately recommend a way to do it.

You can ask what the filesystem encoding is with
sys.getfilesystemencoding(). On my Linux box I can make this return
anything I like by setting LC_CTYPE=en_US.<whatever> (as long as
<whatever> is a recognized encoding). There are probably 5 other
environment variables to influence this. :-(

Of course that doesn't help for undecodable filenames, and in that
case I don't think *anything* can help you unless you have a lot of
additional knowledge about what the user might be doing, e.g. you know
a few other encodings to try that make sense for their environment.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 20:20:22 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 11:20:22 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbtq8t$3dl$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbtq8t$3dl$1@ger.gmane.org>
Message-ID: <ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>

On Tue, Sep 30, 2008 at 11:13 AM, Georg Brandl <g.brandl at gmx.net> wrote:
> Victor Stinner schrieb:
>> On Windows, we might reject bytes filenames for all file operations: open(),
>> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError)
>
> Since I've seen no objections to this yet: please no. If we offer a
> "lower-level" bytes filename API, it should work for all platforms.

I'm not sure either way. I've heard it claim that Windows filesystem
APIs use Unicode natively. Does Python 3.0 on Windows currently
support filenames expressed as bytes? Are they encoded first before
passing to the Unicode APIs? Using what encoding?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From glyph at divmod.com  Tue Sep 30 20:42:58 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Tue, 30 Sep 2008 18:42:58 -0000
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <ca471dc20809301116y480bd028qbcdca44f05c62596@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<200809291359.06334.eckhardt@satorlaser.com>
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com> <1222771976.2598.39.camel@localhost>
	<ca471dc20809300726g2d3b0d43qcaad50608d0283b3@mail.gmail.com>
	<1222785585.2598.45.camel@localhost>
	<20080930181231.31635.1557225685.divmod.xquotient.503@weber.divmod.com>
	<ca471dc20809301116y480bd028qbcdca44f05c62596@mail.gmail.com>
Message-ID: <20080930184258.31635.76988507.divmod.xquotient.511@weber.divmod.com>

On 06:16 pm, guido at python.org wrote:
>On Tue, Sep 30, 2008 at 11:12 AM,  <glyph at divmod.com> wrote:
>>The one thing it doesn't do is expose the decoding rules for the 
>>higher-
>>level applications to deal with.  I am pretty sure I don't understand 
>>how
>>the interaction between filesystem encoding and user locale works in 
>>that
>>case, though, so I can't immediately recommend a way to do it.
>
>You can ask what the filesystem encoding is with
>sys.getfilesystemencoding(). On my Linux box I can make this return
>anything I like by setting LC_CTYPE=en_US.<whatever> (as long as
><whatever> is a recognized encoding). There are probably 5 other
>environment variables to influence this. :-(

Only 5?  Great! :-)
>Of course that doesn't help for undecodable filenames, and in that
>case I don't think *anything* can help you unless you have a lot of
>additional knowledge about what the user might be doing, e.g. you know
>a few other encodings to try that make sense for their environment.

There are other ways to glean this knowledge; for example, looking at 
the 'iocharset' or 'nls' mount options supplied to mount various 
filesystems.  I thought maybe Python (or some C library call) might be 
invoking some logic that did something with data like that; if not, 
great, one day when I have some free time (meaning: never) I can 
implement that logic myself without duplicating a bunch of work.

From glyph at divmod.com  Tue Sep 30 20:47:51 2008
From: glyph at divmod.com (glyph at divmod.com)
Date: Tue, 30 Sep 2008 18:47:51 -0000
Subject: [Python-Dev] Patch for an initial support of bytes filename
	in	Python3
In-Reply-To: <ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
Message-ID: <20080930184751.31635.1484325691.divmod.xquotient.520@weber.divmod.com>


On 05:56 pm, guido at python.org wrote:
>On Tue, Sep 30, 2008 at 10:59 AM,  <glyph at divmod.com> wrote:
>>On 02:32 pm, guido at python.org wrote:

>>In the absence of a 2.6 getcwdb, perhaps the fixer could just drop the
>>"benefit of the doubt" case?  It could always be added to 2.7, and the
>>parity release of 2to3 could have a --2.7 switch that would modify the
>>behavior of this and other fixers.
>
>I'm not sure what you're proposing. *My* proposal is that 2to3 changes
>os.getcwdu() calls to os.getcwd() and leaves os.getcwd() calls alone
>-- there's no way to tell whether os.getcwdb() would be a better
>match, and for portable code, it won't be (since os.getcwdb() is a
>Unix-only thing).

My proposal is simply to change getcwd to getcwdb, and getcwdu to 
getcwd.  This preserves whatever bytes/text behavior you are expecting 
from 2.6 into 3.0.  Granted, the fact that unicode is really always the 
right thing to do on Windows complicates things.

I already tend to avoid os.getcwd() though, and this is just one more 
reason to avoid it.  In the rare cases where I really do need it, it 
looks like os.path.abspath(b".") / os.path.abspath(u".") will provide 
the clarity that I want.

From hodgestar+pythondev at gmail.com  Tue Sep 30 21:07:54 2008
From: hodgestar+pythondev at gmail.com (Simon Cross)
Date: Tue, 30 Sep 2008 21:07:54 +0200
Subject: [Python-Dev] Patch for an initial support of bytes filename in
	Python3
In-Reply-To: <ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
References: <200809300247.20349.victor.stinner@haypocalc.com>
	<20080930132151.31635.132601277.divmod.xquotient.434@weber.divmod.com>
	<ca471dc20809300732r456678fcgb8caeb369a6cf349@mail.gmail.com>
	<20080930175932.31635.989735053.divmod.xquotient.478@weber.divmod.com>
	<ca471dc20809301056j6800b6e1nca9a9ec5a52e8445@mail.gmail.com>
Message-ID: <fb73205e0809301207g4e212aa8lc5ae2981795d7744@mail.gmail.com>

On Tue, Sep 30, 2008 at 7:56 PM, Guido van Rossum <guido at python.org> wrote:
> (since os.getcwdb() is a Unix-only thing).

I would be happier if all the Unix byte functions existed on Windows
fell back to something like encoding the filenames to/from UTF-8. Then
at least it would be possible for programs to support reading all
files on both Unix and Windows without having to perform some sort of
explicit check to see whether os.getcwdb() and friends are supported.

From martin at v.loewis.de  Tue Sep 30 22:04:42 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 22:04:42 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	
	<gbr0nv$iqu$1@ger.gmane.org>	
	<200809300202.38574.victor.stinner@haypocalc.com>	
	<48E1C097.8030309@v.loewis.de>
	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
Message-ID: <48E2865A.3010404@v.loewis.de>

Guido van Rossum wrote:
> On Mon, Sep 29, 2008 at 11:00 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>> Change the default file system encoding to store bytes in Unicode is like
>>> introducing a new Python type: <fake Unicode for filename hacks>.
>> Exactly. Seems like the best solution to me, despite your polemics.
> 
> Martin, I don't understand why you are in favor of storing raw bytes
> encoded as Latin-1 in Unicode string objects, which clearly gives rise
> to mojibake. In the past you have always been staunchly opposed to API
> changes or practices that could lead to mojibake (and you had me quite
> convinced).

True. I try to outweigh the need for simplicity in the API against the
need to support all cases. So I see two solutions:

a) support bytes as file names. Supports all cases, but complicates
   the API very much, by pervasively bringing bytes into the status
   of a character data type. IMO, this must be prevented at all costs.

b) make character (Unicode) strings the only string type. Does not
   immediately support all cases, so some hacks are needed. However,
   even with the hacks, it preserves the simplicity of the API; the
   hacks then should ideally be limited to the applications that need
   it. On this side, I see the following approaches:
   1. try to automatically embed non-representable characters into
      the Unicode strings, e.g. by using PUA characters. Reduces
      the amount of moji-bake, but produces a lot of difficult issues.
   2. let applications that desire so access all file names in a
      uniform manner, at the cost of producing tons of moji-bake

In this case, I think moji-bake is unavoidable: it is just a plain
flaw in the POSIX implementations (not the API or specification) that
you can run into file names where you can't come up with the right
rendering. Even for solution a), the resulting data cannot
be displayed "correctly" in all cases.

Currently, I favor b2, but haven't given up on b1, and they don't
exclude each other. b2 is simple to implement, and delegates the
choice between legible file names and universal access to all files
to the application. Given the way Unix works, this is the most sensible
choice, IMO: by default, Python should try to make file names legible,
but stuff like backup applications should be implementable also -
and they don't need legible file names.

I think option a) will hunt us forever. People will ask for more and
more features in the bytes type, eventually asking "give us Python
2.x strings back". It already starts: see #3982, where Benjamin
asks to have .format added to bytes (for a reason unrelated to file
names).

Regards,
Martin



From martin at v.loewis.de  Tue Sep 30 22:22:07 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 22:22:07 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <200809301611.03027.victor.stinner@haypocalc.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<48E1C097.8030309@v.loewis.de>	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
	<200809301611.03027.victor.stinner@haypocalc.com>
Message-ID: <48E28A6F.8030604@v.loewis.de>

> I didn't get an answer to my question: what is the result <bytes (fake 
> characters) stored in unicode> + <real unicode>? I guess that the result is 
> <mixed "bytes" and characters in unicode> instead of raising an error 
> (invalid types). So again: why introducing a new type instead of reusing 
> existing Python types?

I didn't mean to introduce a new data type in the strict sense - merely
to pass through undecodable bytes through the regular Unicode type.
So the result of adding them is a regular Unicode string.

Regards,
Martin


From martin at v.loewis.de  Tue Sep 30 22:29:37 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 22:29:37 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <ca471dc20809301045r59251402g3fe947dec3bc7f22@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbr0nv$iqu$1@ger.gmane.org>	<200809300202.38574.victor.stinner@haypocalc.com>	<gbsgk6$kc1$1@ger.gmane.org>	<ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>	<gbtnjo$quh$1@ger.gmane.org>
	<ca471dc20809301045r59251402g3fe947dec3bc7f22@mail.gmail.com>
Message-ID: <48E28C31.6060606@v.loewis.de>

Guido van Rossum wrote:
> However
> the *proposed* behavior (returns bytes if the arg was bytes, and
> returns str when the arg was str) is IMO sane, and no different than
> the polymorphism found in len() or many builtin operations.

My concern still is that it brings the bytes type into the status of
another character string type, which is really bad, and will require
further modifications to Python for the lifetime of 3.x.

This is because applications will then regularly use byte strings for
file names on Unix, and regular strings on Windows, and then expect
the program to work the same without further modifications. The next
question then will be environment variables and command line arguments,
for which we then should provide two versions (e.g. sys.argv and
sys.argvb; for os.environ, os.environ["PATH"] could mean something
different from os.environ[b"PATH"]). And so on (passwd/group file,
Tkinter, ...)

Regards,
Martin



From martin at v.loewis.de  Tue Sep 30 22:45:55 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 22:45:55 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
 filename issue
In-Reply-To: <ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbtq8t$3dl$1@ger.gmane.org>
	<ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>
Message-ID: <48E29003.1010703@v.loewis.de>

> I'm not sure either way. I've heard it claim that Windows filesystem
> APIs use Unicode natively. Does Python 3.0 on Windows currently
> support filenames expressed as bytes?

Yes, it does (at least, os.open, os.stat support them, builtin open
doesn't).

> Are they encoded first before
> passing to the Unicode APIs? Using what encoding?

They aren't passed to the Unicode (W) APIs (by Python). Instead, they
are passed to the "ANSI" (A) APIs (i.e. CP_ACP APIs). On Windows NT+,
that API then converts it to Unicode through the CP_ACP (aka "mbcs")
encoding; this is inside the system DLLs.

CP_ACP is a lossy encoding (from Unicode to bytes): Microsoft uses
replacement characters if they can, starting with similarly-looking
characters, and falling back to question marks.

Regards,
Martin


From guido at python.org  Tue Sep 30 23:06:31 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 14:06:31 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbtveu$ocl$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbtq8t$3dl$1@ger.gmane.org>
	<ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>
	<gbtveu$ocl$1@ger.gmane.org>
Message-ID: <ca471dc20809301406s286e2328u853dbc453f1faf13@mail.gmail.com>

On Tue, Sep 30, 2008 at 12:42 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> Guido van Rossum wrote:
>>
>> On Tue, Sep 30, 2008 at 11:13 AM, Georg Brandl <g.brandl at gmx.net> wrote:
>>>
>>> Victor Stinner schrieb:
>>>>
>>>> On Windows, we might reject bytes filenames for all file operations:
>>>> open(),
>>>> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError)
>>>
>>> Since I've seen no objections to this yet: please no. If we offer a
>>> "lower-level" bytes filename API, it should work for all platforms.
>>
>> I'm not sure either way. I've heard it claim that Windows filesystem
>> APIs use Unicode natively. Does Python 3.0 on Windows currently
>> support filenames expressed as bytes? Are they encoded first before
>> passing to the Unicode APIs? Using what encoding?
>
> In 3.0rc1, the listdir doc needs updating:
> "os.listdir(path)
> Return a list containing the names of the entries in the directory. The list
> is in arbitrary order. It does not include the special entries '.' and '..'
> even if they are present in the directory. Availability: Unix, Windows.
>
> On Windows NT/2k/XP and Unix, if path is a Unicode object, the result will
> be a list of Unicode objects."
>
> s/Unicode/bytes/ at least for Windows.
>
>>>> os.listdir(b'.')
> [b'countries.txt', b'multeetest.py', b't1.py', b't1.pyc', b't2.py', b'tem',
> b'temp.py', b'temp.pyc', b'temp2.py', b'temp3.py', b'temp4.py', b'test.py',
> b'z', b'z.txt']
>
> The bytes names do not work however:
>
>>>> t=open(b'tem')
> Traceback (most recent call last):
>  File "<pyshell#23>", line 1, in <module>
>    t=open(b'tem')
>  File "C:\Programs\Python30\lib\io.py", line 284, in __new__
>    return open(*args, **kwargs)
>  File "C:\Programs\Python30\lib\io.py", line 184, in open
>    raise TypeError("invalid file: %r" % file)
> TypeError: invalid file: b'tem'
>
> Is this what you were asking?

No, that's because bytes is missing from the explicit list of
allowable types in io.open. Victor has a one-line trivial patch for
this. Could you try this though?

>>> import _fileio
>>> _fileio._FileIO(b'tem')

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 23:22:11 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 14:22:11 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E2865A.3010404@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<48E1C097.8030309@v.loewis.de>
	<ca471dc20809300653m4e79dcd7y818b624f9ecd8f5e@mail.gmail.com>
	<48E2865A.3010404@v.loewis.de>
Message-ID: <ca471dc20809301422u1e797dacm8a19fd9b4e3e74e6@mail.gmail.com>

On Tue, Sep 30, 2008 at 1:04 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Guido van Rossum wrote:
>> On Mon, Sep 29, 2008 at 11:00 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
>>>> Change the default file system encoding to store bytes in Unicode is like
>>>> introducing a new Python type: <fake Unicode for filename hacks>.
>>> Exactly. Seems like the best solution to me, despite your polemics.
>>
>> Martin, I don't understand why you are in favor of storing raw bytes
>> encoded as Latin-1 in Unicode string objects, which clearly gives rise
>> to mojibake. In the past you have always been staunchly opposed to API
>> changes or practices that could lead to mojibake (and you had me quite
>> convinced).
>
> True. I try to outweigh the need for simplicity in the API against the
> need to support all cases. So I see two solutions:
>
> a) support bytes as file names. Supports all cases, but complicates
>   the API very much, by pervasively bringing bytes into the status
>   of a character data type. IMO, this must be prevented at all costs.

That's a matter of opinion. I would also like to point out that it is
in fact already supported by the system calls. io.open() doesn't, but
that's a wrapper around _fileio._FileIO which does support bytes. All
other syscalls already do the right thing (even readlink()!) except
os.listdir(), which returns a mixture of bytes and str values (which
is horrible) and os.getcwd() which needs a bytes equivalent. Victor's
patch addresses all these issues.

Victor's patch also tries to fix glob.py, fnmatch.py, and
posixpath.py. That is more debatable, because this might be the start
of a never-ending project. OTOH we have precedents, e.g. the re module
similarly supports both bytes and unicode (and makes an effort to
avoid mixing them).

> b) make character (Unicode) strings the only string type. Does not
>   immediately support all cases, so some hacks are needed. However,
>   even with the hacks, it preserves the simplicity of the API; the
>   hacks then should ideally be limited to the applications that need
>   it. On this side, I see the following approaches:
>   1. try to automatically embed non-representable characters into
>      the Unicode strings, e.g. by using PUA characters. Reduces
>      the amount of moji-bake, but produces a lot of difficult issues.
>   2. let applications that desire so access all file names in a
>      uniform manner, at the cost of producing tons of moji-bake
>
> In this case, I think moji-bake is unavoidable: it is just a plain
> flaw in the POSIX implementations (not the API or specification) that
> you can run into file names where you can't come up with the right
> rendering. Even for solution a), the resulting data cannot
> be displayed "correctly" in all cases.

But I still like the ultimate solution to displaying names for (a)
better: if it's not decodable, display it as the repr() of a bytes
object. (Which happens to be its str() as well.)

> Currently, I favor b2, but haven't given up on b1, and they don't
> exclude each other. b2 is simple to implement, and delegates the
> choice between legible file names and universal access to all files
> to the application. Given the way Unix works, this is the most sensible
> choice, IMO: by default, Python should try to make file names legible,
> but stuff like backup applications should be implementable also -
> and they don't need legible file names.

I don't believe that an application-wide choice is safe. For example
the tempfile module manipulates filenames (at least for
NamedTemporaryFile) and I think it would be wrong if it were affected
by such a global setting. (E.g. the user could pass a suffix argument
containing Unicode characters outside Latin-1.)

> I think option a) will hunt us forever. People will ask for more and
> more features in the bytes type, eventually asking "give us Python
> 2.x strings back". It already starts: see #3982, where Benjamin
> asks to have .format added to bytes (for a reason unrelated to file
> names).

I'm not so worried about feature requests for the bytes type unrelated
to filesystems; we can either grant them or not, and I am actually in
many cases in favor of granting them -- just like we support bytes in
the re module as I already mentioned above. The bytes and str types
have intentionally similar APIs, because they have similar structure,
and even somewhat similar semantics (b'ABC' and 'ABC' have related
meanings even if there are subtle differences).

I am also encouraged by Glyph's support for (a). He has a lot of
practical experience.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 23:24:31 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 14:24:31 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbu173$vnq$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbtq8t$3dl$1@ger.gmane.org>
	<ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>
	<gbtveu$ocl$1@ger.gmane.org> <gbu173$vnq$1@ger.gmane.org>
Message-ID: <ca471dc20809301424u523554d4jf322865d143ab638@mail.gmail.com>

On Tue, Sep 30, 2008 at 1:12 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> Terry Reedy wrote:
>>
>> Guido van Rossum wrote:
>
>>> I'm not sure either way. I've heard it claim that Windows filesystem
>>> APIs use Unicode natively. Does Python 3.0 on Windows currently
>>> support filenames expressed as bytes? Are they encoded first before
>>> passing to the Unicode APIs? Using what encoding?
>
>> [os.listdir(bytes) returns list of bytes, open(bytes) fails]
>
> More:
>
> The path functions seem also do not work:
>
>>>> op.abspath(b'tem')
> ...
>    path = path.replace("/", "\\")
> TypeError: expected an object with the buffer interface
>
> The error message is a bit cryptic given that the problem is that the
> arguments to replace should be bytes instead of strings for a bytes path.
>
> .basename fails with
> ...
>   while i and p[i-1] not in '/\\':
> TypeError: 'in <string>' requires string as left operand, not int
>
> os.rename, os.stat, os.mkdir, os.rmdir work.  I presume same is true for
> others that normally work on windows.

It looks roughly like the system calls do support bytes (using what
encoding?) but the Python code in os.path doesn't. This is the same as
the status quo on Linux.

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 23:34:36 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 14:34:36 -0700
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E28C31.6060606@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<200809300202.38574.victor.stinner@haypocalc.com>
	<gbsgk6$kc1$1@ger.gmane.org>
	<ca471dc20809300659g608f8c14g29ba2b30def1be1f@mail.gmail.com>
	<gbtnjo$quh$1@ger.gmane.org>
	<ca471dc20809301045r59251402g3fe947dec3bc7f22@mail.gmail.com>
	<48E28C31.6060606@v.loewis.de>
Message-ID: <ca471dc20809301434u6116391cje5778bcef5048cc9@mail.gmail.com>

On Tue, Sep 30, 2008 at 1:29 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> Guido van Rossum wrote:
>> However
>> the *proposed* behavior (returns bytes if the arg was bytes, and
>> returns str when the arg was str) is IMO sane, and no different than
>> the polymorphism found in len() or many builtin operations.
>
> My concern still is that it brings the bytes type into the status of
> another character string type, which is really bad, and will require
> further modifications to Python for the lifetime of 3.x.

I'd like to understand why this is "really bad". I though it was by
design that the str and bytes types behave pretty similarly. You can
use both as dict keys.

> This is because applications will then regularly use byte strings for
> file names on Unix, and regular strings on Windows, and then expect
> the program to work the same without further modifications.

It seems that bytes arguments actually *do* work on Windows -- somehow
they get decoded. (Unless Terry's report was from 2.x.)

> The next
> question then will be environment variables and command line arguments,
> for which we then should provide two versions (e.g. sys.argv and
> sys.argvb; for os.environ, os.environ["PATH"] could mean something
> different from os.environ[b"PATH"]).

Actually something like that may not be a bad idea. Ian Bicking's
webob supports similar double APIs for getting the request parameters
out of a request object; I believe request.GET['x'] is a text object
and request.GET_str['x'] is the corresponding uninterpreted bytes
sequence. I would prefer to have os.environb over os.environ[b"PATH"]
though.

> And so on (passwd/group file, Tkinter, ...)

I assume at some point we can stop and have sufficiently low-level
interfaces that everyone can agree are in bytes only. Bytes aren't
going away. How does Java deal with this? Its File class doesn't seem
to deal in bytes at all. What would its listFiles() method do with
undecodable filenames?

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From guido at python.org  Tue Sep 30 23:37:26 2008
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 Sep 2008 14:37:26 -0700
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <20080930184258.31635.76988507.divmod.xquotient.511@weber.divmod.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>
	<48E15B83.9040205@v.loewis.de>
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>
	<48E1F53B.7030901@gmail.com> <1222771976.2598.39.camel@localhost>
	<ca471dc20809300726g2d3b0d43qcaad50608d0283b3@mail.gmail.com>
	<1222785585.2598.45.camel@localhost>
	<20080930181231.31635.1557225685.divmod.xquotient.503@weber.divmod.com>
	<ca471dc20809301116y480bd028qbcdca44f05c62596@mail.gmail.com>
	<20080930184258.31635.76988507.divmod.xquotient.511@weber.divmod.com>
Message-ID: <ca471dc20809301437y36d389bbn932ab2cf810727f0@mail.gmail.com>

On Tue, Sep 30, 2008 at 11:42 AM,  <glyph at divmod.com> wrote:
> There are other ways to glean this knowledge; for example, looking at the
> 'iocharset' or 'nls' mount options supplied to mount various filesystems.  I
> thought maybe Python (or some C library call) might be invoking some logic
> that did something with data like that; if not, great, one day when I have
> some free time (meaning: never) I can implement that logic myself without
> duplicating a bunch of work.

I know we could do a better job, but absent anyone who knows what
they're doing we've chosen a fairly conservative approach. I certainly
hope that someone will contribute some mean encoding-guessing code to
the stdlib that users can use. I'm not sure if I'll ever endorse doing
this automatically in io.open(), though I'd be fine with a convention
like passing encoding="guess".

-- 
--Guido van Rossum (home page: http://www.python.org/~guido/)

From martin at v.loewis.de  Tue Sep 30 23:40:01 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 23:40:01 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbtq8t$3dl$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbtq8t$3dl$1@ger.gmane.org>
Message-ID: <48E29CB1.5010309@v.loewis.de>

>> On Windows, we might reject bytes filenames for all file operations: open(), 
>> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError)
> 
> Since I've seen no objections to this yet: please no. If we offer a
> "lower-level" bytes filename API, it should work for all platforms.

Unfortunately, it can't. You cannot represent all possible file names
in a byte string in Windows (just as you can't do so in a Unicode
string on Unix).

So using byte strings on Windows would work for some files, but fail
for others. In particular, listdir might give you a list of file names
which you then can't open/stat/recurse into.

(of course, you could use UTF-8 as the file system encoding on Windows,
but then you will have to rewrite a lot of C code first)

Regards,
Martin

From martin at v.loewis.de  Tue Sep 30 23:42:19 2008
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 30 Sep 2008 23:42:19 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <gbtvd3$na4$1@ger.gmane.org>
References: <200809291407.55291.victor.stinner@haypocalc.com>	<gbtq8t$3dl$1@ger.gmane.org>	<ca471dc20809301120y5149d346s31b0027b7bdd529e@mail.gmail.com>
	<gbtvd3$na4$1@ger.gmane.org>
Message-ID: <48E29D3B.5030900@v.loewis.de>

> Oh, ok. I had assumed Windows just uses a fixed encoding without the problem
> of misencoded filenames.

It's the other way 'round: On Windows, Unicode file names are the
natural choice, and byte strings have limitations. In a sense, Windows
got it right - but then, they started later. Unix missed the opportunity
of declaring that all file APIs are UTF-8 (except for Plan-9 and OS X,
neither being "true" Unix).

Regards,
Martin

From ncoghlan at gmail.com  Tue Sep 30 23:43:26 2008
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 01 Oct 2008 07:43:26 +1000
Subject: [Python-Dev] Filename as byte string in python 2.6 or 3.0?
In-Reply-To: <ca471dc20809300722x4a700e85v153732843f43c920@mail.gmail.com>
References: <200809271404.25654.victor.stinner@haypocalc.com>	
	<200809291250.03291.eckhardt@satorlaser.com>	
	<48E0B8FB.9070701@egenix.com>	
	<200809291359.06334.eckhardt@satorlaser.com>	
	<20080929140133.31635.1170224088.divmod.xquotient.314@weber.divmod.com>	
	<48E15B83.9040205@v.loewis.de>	
	<aac2c7cb0809291631sf25a0cahe355fc6cc5816ff7@mail.gmail.com>	
	<48E1F53B.7030901@gmail.com>
	<ca471dc20809300722x4a700e85v153732843f43c920@mail.gmail.com>
Message-ID: <48E29D7E.6080406@gmail.com>

Guido van Rossum wrote:
> The callback would either be an extra argument to all
> system calls (bad, ugly etc., and why not go with the existing unicode
> encoding and error flags if we're adding extra args?) or would be
> global, where I'd be worried that it might interfere with the proper
> operation of library code that is several abstractions away from
> whoever installed the callback, not under their control, and not
> expecting the callback.
> 
> I suppose I may have totally misunderstood your proposal, but in
> general I find callbacks unwieldy.

Not really - later in the email, I actually pointed out that exposing
the unicode errors flag for the implicit PyUnicode_Decode invocations
would be enough to enable a callback mechanism.

However, James's post pointing out that this is a problem that also
affects environment variables and command line arguments, not just file
paths completely kills any hope of purely callback based approach - that
processing needs to "just work" without any additional intervention from
the application.

Of the suggestions I've seen so far, I like Marcin's Mono-inspired
NULL-escape codec idea the best. Since these strings all come from parts
of the environment where NULLs are not permitted, a simple "'\0' in
text" check will immediately identify any strings where decoding failed
(for applications which care about the difference and want to try to do
better), while applications which don't care will receive perfectly
valid Python strings that can be passed around and manipulated as if the
decoding error never happened.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------
            http://www.boredomandlaziness.org

From qrczak at knm.org.pl  Tue Sep 30 23:34:37 2008
From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk)
Date: Tue, 30 Sep 2008 23:34:37 +0200
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E29335.7090102@g.nevcal.com>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbr0nv$iqu$1@ger.gmane.org>
	<ca471dc20809291006n93de9c6y2aad06d59b22aca3@mail.gmail.com>
	<48E29335.7090102@g.nevcal.com>
Message-ID: <3f4107910809301434j2e23d5f5l84ef14a1d248659b@mail.gmail.com>

2008/9/30 Glenn Linderman <v+python at g.nevcal.com>:

> So the problem is that a Unicode file system interface can't deal with
> non-UTF-8 byte streams as file names.
>
> So it seems there are four suggested approaches, all of which have aspects
> that are inconvenient.

Let's not forget what happens when a non-UTF-8 file name is read from
a file or written to a file, under the assumption that the filename is
written to the file directly (which probably breaks for filenames
containing newlines or such).

> 4) Use of bytes APIs on FS interfaces.  This seems to be the "solution"
> adopted by Posix that creates the "problem" encountered by Unicode-native
> applications.  It is cumbersome to deal with within applications that
> attempt to display the names.  What do Posix-style "open file" dialog boxes
> do in this case?

http://library.gnome.org/devel/glib/stable/glib-Character-Set-Conversion.html#g-filename-display-name

I used to observe three different ways to display such filenames
within gedit (including %xx and \xx escapes), but now it is
consistent, probably because it switched to using the above function
everywhere:
$ touch $'abc\xffz'
$ gedit
The Open dialog shows:
   abc?z (invalid encoding)
When the file is open, the window title and the tab title show:
   abc?z
and the same is in recent file list.

It has a bug: it appends " (invalid encoding)" even if the filename
contains a correctly encoded U+FFFD character. Nautilus has the same
behavior and the same bug because this is a design bug of that
function which does not allow to tell whether the conversion was
successful.

A filename containing a newline is sometimes displayed in two lines,
and sometimes with a U+000A character from a fallback font (hex
character number in a box).

-- 
Marcin Kowalczyk
qrczak at knm.org.pl
http://qrnik.knm.org.pl/~qrczak/

From foom at fuhm.net  Tue Sep 30 23:59:10 2008
From: foom at fuhm.net (James Y Knight)
Date: Tue, 30 Sep 2008 17:59:10 -0400
Subject: [Python-Dev] [Python-3000] New proposition for Python3 bytes
	filename issue
In-Reply-To: <48E29CB1.5010309@v.loewis.de>
References: <200809291407.55291.victor.stinner@haypocalc.com>
	<gbtq8t$3dl$1@ger.gmane.org> <48E29CB1.5010309@v.loewis.de>
Message-ID: <83758335-97EA-441B-A783-05F16EBE6D7A@fuhm.net>

On Sep 30, 2008, at 5:40 PM, Martin v. L?wis wrote:
>>> On Windows, we might reject bytes filenames for all file  
>>> operations: open(),
>>> unlink(), os.path.join(), etc. (raise a TypeError or UnicodeError)
>>
>> Since I've seen no objections to this yet: please no. If we offer a
>> "lower-level" bytes filename API, it should work for all platforms.
>
> Unfortunately, it can't. You cannot represent all possible file names
> in a byte string in Windows (just as you can't do so in a Unicode
> string on Unix).

As you mention in the parenthetical below, of course it can.

> So using byte strings on Windows would work for some files, but fail
> for others. In particular, listdir might give you a list of file names
> which you then can't open/stat/recurse into.
>
> (of course, you could use UTF-8 as the file system encoding on  
> Windows,
> but then you will have to rewrite a lot of C code first)

Yes! If there is a byte-string access method for Windows, pretty  
please make it decode from UTF-8 internally and call the Unicode  
version of the Windows APIs. The non-unicode windows APIs are pretty  
much just broken -- Ideally, Python should never be calling those.

But, I still don't like the idea of propagating the "sometimes a  
string, sometimes bytes" APIs...One or the other, please. Either  
always strings (if and only if a method for assuring decoding always  
succeeds), or always bytes.

James