From paul at boddie.org.uk  Sun Jul  1 02:22:47 2012
From: paul at boddie.org.uk (Paul Boddie)
Date: Sun, 1 Jul 2012 02:22:47 +0200
Subject: [Python-Dev] Announcing the python-static-type-checking google
	group
Message-ID: <201207010222.48167.paul@boddie.org.uk>

Edward K. Ream wrote:
> Hello all,
>
> GvR has asked me to announce the python-static-type-checking google
> group http://groups.google.com/group/python-static-type-checking to
> python-dev.
>
> Consider it announced.  Anyone from python-dev who likes may become a
> member.

Is there any reason why the compiler-sig mailing list wasn't chosen as a venue 
for such discussions? I know it has "compiler" in the title, but the mandate 
overlaps significantly with what you intend to discuss.

It's obviously your choice where you host discussions and who you invite, and 
I know that the special interest group mailing lists aren't exactly well 
advertised these days, what with the lack of agility around updating the Web 
content that advertises such things (or the lack of visibility of the Wiki 
content), but I feel that you might have a more productive discussion if you 
don't insist on Google Groups membership and also allow a wider selection of 
participants.

Please consider this as friendly advice: I too would like to see progress in 
the area concerned.

Paul

From raymond.hettinger at gmail.com  Sun Jul  1 09:07:56 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sun, 1 Jul 2012 00:07:56 -0700
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <jsoo4p$via$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<jsoo4p$via$1@dough.gmane.org>
Message-ID: <2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com>


On Jun 30, 2012, at 10:44 PM, Stefan Behnel wrote:

>> 
>> Another addition could be a new subsection on grouping (chunking) that
>> would discuss post-processing of grouper (as discussed above), as well as
>> other recipes, including ones specific to strings and sequences. It would
>> essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or
>> Chunking Sequences and Iterables". The synonyms will help external
>> searching. A toc would let people who have found this doc know to look for
>> this at the bottom.
> 
> If it really is such an important use case for so many people, I agree that
> it's worth special casing it in the docs. It's not a trivial algorithmic
> step from a sequential iterable to a grouped iterable.

I'm not too keen on adding a section like this to the itertools docs.

Instead, I would be open adding "further reading" section with external links 
to interesting iterator writeups in blogs, cookbooks, stack overflow answers, wikis, etc.

If one of you wants to craft an elegant blog post on "Grouping, Blocking, or
Chunking Sequences and Iterables", I would be happy to link to it.


Raymond




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120701/0532fc7f/attachment.html>

From g.brandl at gmx.net  Sun Jul  1 10:03:00 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 01 Jul 2012 10:03:00 +0200
Subject: [Python-Dev] cpython: Add syntax highlighter tool
In-Reply-To: <E1Sl7Yl-0005JD-RR@dinsdale.python.org>
References: <E1Sl7Yl-0005JD-RR@dinsdale.python.org>
Message-ID: <4FF00434.5020604@gmx.net>

On 01.07.2012 01:58, raymond.hettinger wrote:
> http://hg.python.org/cpython/rev/da4dd603030b
> changeset:   77899:da4dd603030b
> user:        Raymond Hettinger <python at rcn.com>
> date:        Sat Jun 30 16:58:06 2012 -0700
> summary:
>    Add syntax highlighter tool
>
> files:
>    Tools/scripts/pycolorize.py |  109 ++++++++++++++++++++++++
>    1 files changed, 109 insertions(+), 0 deletions(-)

Uh, this looks quite a lot like a new feature...

Since it's in Tools, I'm not going to veto it, just as with the
improvements to the gdb helper, but it would have been nice to
at least *ask*...

Georg


From doko at ubuntu.com  Sun Jul  1 10:22:10 2012
From: doko at ubuntu.com (Matthias Klose)
Date: Sun, 01 Jul 2012 10:22:10 +0200
Subject: [Python-Dev] cross-compiling patches
In-Reply-To: <20120630231742.2c896682@pitrou.net>
References: <20120630231742.2c896682@pitrou.net>
Message-ID: <4FF008B2.8080807@ubuntu.com>

On 30.06.2012 23:17, Antoine Pitrou wrote:
> 
> Hello,
> 
> I think these patches are premature (they break compilation on OS X,
> and they break ctypes configure on my Linux box).

that was unrelated. fixed last night.

> Furthermore, they
> were committed post-beta, which means they should probably have waited
> for after the 3.3 release. So I propose for these commits to be
> reverted.

somebody (?) asked on irc that either Martin or I should update the internal
copies, and someone suggested to do the same with the expat sources. And I asked
our release manager if he was ok with this post beta change.

> (to be clear, I'm talking about all configure / Makefile / setup.py /
> libffi changes since and including
> http://hg.python.org/cpython/rev/e6e99d449bdc876fa57111e7e534c44ecbc3bcbd
> )

the first cross-build fixes went in in April, please consider these fixes for
the then incomplete cros-build fixes. The build issues you did see last night,
were fixed for the OS X build, and I reverted the update for the compiler search
paths for the native build. So please lets keep this cross-build support for
3.3. It's working, tested with a arm-linux-gnueabi build on x86_64-linux-gnu.

  Matthias

From ncoghlan at gmail.com  Sun Jul  1 11:13:47 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 1 Jul 2012 19:13:47 +1000
Subject: [Python-Dev] cross-compiling patches
In-Reply-To: <4FF008B2.8080807@ubuntu.com>
References: <20120630231742.2c896682@pitrou.net> <4FF008B2.8080807@ubuntu.com>
Message-ID: <CADiSq7dD+qbDP-GDY3KwPCdkFbFcNGMo09BLjFtnM7oxFWd2HQ@mail.gmail.com>

On Sun, Jul 1, 2012 at 6:22 PM, Matthias Klose <doko at ubuntu.com> wrote:
> the first cross-build fixes went in in April, please consider these fixes for
> the then incomplete cros-build fixes. The build issues you did see last night,
> were fixed for the OS X build, and I reverted the update for the compiler search
> paths for the native build. So please lets keep this cross-build support for
> 3.3. It's working, tested with a arm-linux-gnueabi build on x86_64-linux-gnu.

Indeed, especially with the variety of ARM mini-boards coming onto the
market at the moment.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From g.brandl at gmx.net  Sun Jul  1 11:37:50 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 01 Jul 2012 11:37:50 +0200
Subject: [Python-Dev] cross-compiling patches
In-Reply-To: <4FF008B2.8080807@ubuntu.com>
References: <20120630231742.2c896682@pitrou.net> <4FF008B2.8080807@ubuntu.com>
Message-ID: <jsp5o8$cs7$1@dough.gmane.org>

On 01.07.2012 10:22, Matthias Klose wrote:
> On 30.06.2012 23:17, Antoine Pitrou wrote:
>>
>> Hello,
>>
>> I think these patches are premature (they break compilation on OS X,
>> and they break ctypes configure on my Linux box).
>
> that was unrelated. fixed last night.

It's also something the buildbots can catch easily.

>> Furthermore, they
>> were committed post-beta, which means they should probably have waited
>> for after the 3.3 release. So I propose for these commits to be
>> reverted.
>
> somebody (?) asked on irc that either Martin or I should update the internal
> copies, and someone suggested to do the same with the expat sources. And I asked
> our release manager if he was ok with this post beta change.
>
>> (to be clear, I'm talking about all configure / Makefile / setup.py /
>> libffi changes since and including
>> http://hg.python.org/cpython/rev/e6e99d449bdc876fa57111e7e534c44ecbc3bcbd
>> )
>
> the first cross-build fixes went in in April, please consider these fixes for
> the then incomplete cros-build fixes. The build issues you did see last night,
> were fixed for the OS X build, and I reverted the update for the compiler search
> paths for the native build. So please lets keep this cross-build support for
> 3.3. It's working, tested with a arm-linux-gnueabi build on x86_64-linux-gnu.

FWIW, I agree that build fixes for platforms we already support are not a new
feature.  I am not happy about the amount of changes, and I hope that no such
big changes will come until final now...

Georg


From solipsis at pitrou.net  Sun Jul  1 13:43:49 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 1 Jul 2012 13:43:49 +0200
Subject: [Python-Dev] cross-compiling patches
References: <20120630231742.2c896682@pitrou.net> <4FF008B2.8080807@ubuntu.com>
	<jsp5o8$cs7$1@dough.gmane.org>
Message-ID: <20120701134349.5390b738@pitrou.net>

On Sun, 01 Jul 2012 11:37:50 +0200
Georg Brandl <g.brandl at gmx.net> wrote:
> >
> > the first cross-build fixes went in in April, please consider these fixes for
> > the then incomplete cros-build fixes. The build issues you did see last night,
> > were fixed for the OS X build, and I reverted the update for the compiler search
> > paths for the native build. So please lets keep this cross-build support for
> > 3.3. It's working, tested with a arm-linux-gnueabi build on x86_64-linux-gnu.
> 
> FWIW, I agree that build fixes for platforms we already support are not a new
> feature.  I am not happy about the amount of changes, and I hope that no such
> big changes will come until final now...

I don't think we have ever supported cross-compiling officially (which
is different from supporting a given platform, e.g. ARM). But fair
enough.

Regards

Antoine.



From stefan_ml at behnel.de  Sun Jul  1 14:01:36 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 01 Jul 2012 14:01:36 +0200
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<jsoo4p$via$1@dough.gmane.org>
	<2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com>
Message-ID: <jspe70$4gq$1@dough.gmane.org>

Hi Raymond,

Raymond Hettinger, 01.07.2012 09:07:
> On Jun 30, 2012, at 10:44 PM, Stefan Behnel wrote:
>>> Another addition could be a new subsection on grouping (chunking) that
>>> would discuss post-processing of grouper (as discussed above), as well as
>>> other recipes, including ones specific to strings and sequences. It would
>>> essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or
>>> Chunking Sequences and Iterables". The synonyms will help external
>>> searching. A toc would let people who have found this doc know to look for
>>> this at the bottom.
>>
>> If it really is such an important use case for so many people, I agree that
>> it's worth special casing it in the docs. It's not a trivial algorithmic
>> step from a sequential iterable to a grouped iterable.
> 
> I'm not too keen on adding a section like this to the itertools docs.

I've only just seen that the recipes section is part of the same page since
the 2.6 documentation was sphinxified. I had remembered it being on a
separate page before. That resolves most of my original concerns. Sorry,
should have looked earlier.

To address the main problem of users not finding what they need, what about
simply extending the docstring of the grouper() function with a sentence
like this:

"This functionality is also called 'chunking' or 'blocking' and can be used
for load distribution and sharding."

That would make it easy for users to find what they are looking for when
they search the page for "chunk". I find that a much more common and less
ambiguous name than "grouping", which reminds me more of "group by".

It might be a good idea in general to add a short comment on a use case to
each recipe where it's not immediately obvious or where there is a use case
with a well-known name, simply to aid in text searches over the page.


> Instead, I would be open adding "further reading" section with external links 
> to interesting iterator writeups in blogs, cookbooks, stack overflow answers, wikis, etc.
> 
> If one of you wants to craft an elegant blog post on "Grouping, Blocking, or
> Chunking Sequences and Iterables", I would be happy to link to it.

That could be done in addition, but it bares the risk of bit rotting the
documentation by links dying, blogs moving or texts changing.

Stefan


From guido at python.org  Sun Jul  1 17:23:57 2012
From: guido at python.org (Guido van Rossum)
Date: Sun, 1 Jul 2012 17:23:57 +0200
Subject: [Python-Dev] Announcing the python-static-type-checking google
	group
In-Reply-To: <201207010222.48167.paul@boddie.org.uk>
References: <201207010222.48167.paul@boddie.org.uk>
Message-ID: <CAP7+vJKUdcmrDNkjYhdLCgGPBBdcdJt5ThkZ_UajRn+YN-ZROQ@mail.gmail.com>

Hey Paul,

This list is going to be high volume super focused and run by Edward;
so far he's mostly been thinking out loud. I don't think that Edward
would be comfortable doing that quite in the same way on th existing
compiler-list. But by all means sign up for his list!

--Guido

On Sun, Jul 1, 2012 at 2:22 AM, Paul Boddie <paul at boddie.org.uk> wrote:
> Edward K. Ream wrote:
>> Hello all,
>>
>> GvR has asked me to announce the python-static-type-checking google
>> group http://groups.google.com/group/python-static-type-checking to
>> python-dev.
>>
>> Consider it announced.  Anyone from python-dev who likes may become a
>> member.
>
> Is there any reason why the compiler-sig mailing list wasn't chosen as a venue
> for such discussions? I know it has "compiler" in the title, but the mandate
> overlaps significantly with what you intend to discuss.
>
> It's obviously your choice where you host discussions and who you invite, and
> I know that the special interest group mailing lists aren't exactly well
> advertised these days, what with the lack of agility around updating the Web
> content that advertises such things (or the lack of visibility of the Wiki
> content), but I feel that you might have a more productive discussion if you
> don't insist on Google Groups membership and also allow a wider selection of
> participants.
>
> Please consider this as friendly advice: I too would like to see progress in
> the area concerned.
>
> Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From stefan_ml at behnel.de  Sun Jul  1 17:56:28 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 01 Jul 2012 17:56:28 +0200
Subject: [Python-Dev] Announcing the python-static-type-checking google
	group
In-Reply-To: <201207010222.48167.paul@boddie.org.uk>
References: <201207010222.48167.paul@boddie.org.uk>
Message-ID: <jsprvd$28j$1@dough.gmane.org>

Paul Boddie, 01.07.2012 02:22:
> Is there any reason why the compiler-sig mailing list wasn't chosen as a venue 

Even I didn't know that this list even existed. And looking at the archives
now, it's hard to see any relevant discussion in all the spam it received
until it apparently died away in (almost) silence a couple of years ago.


> It's obviously your choice where you host discussions and who you invite, and 
> I know that the special interest group mailing lists aren't exactly well 
> advertised these days

True, but many (most?) of them are simply not very well frequented, which
reduces the interest in joining them even further. Both SIG mailing lists
that I read only receive a mail every so many months, often enough without
any reply. And almost all of these mails deal with questions that would
better be discussed on python-list to leverage the substantially higher
number of eyeballs there.

I think that's the basic problem: as long as more experts are lurking on
python-list than on the dedicated SIG-ML, it's better not to use the SIG-ML
for discussions but to go to python-list (or maybe python-ideas or
python-dev) straight away.

Stefan


From paul at boddie.org.uk  Sun Jul  1 23:43:45 2012
From: paul at boddie.org.uk (Paul Boddie)
Date: Sun, 1 Jul 2012 23:43:45 +0200
Subject: [Python-Dev] Announcing the python-static-type-checking google
	group
Message-ID: <201207012343.45808.paul@boddie.org.uk>

Stefan Behnel wrote:
> Paul Boddie, 01.07.2012 02:22:
> > Is there any reason why the compiler-sig mailing list wasn't chosen as a
> > venue
>
> Even I didn't know that this list even existed. And looking at the archives
> now, it's hard to see any relevant discussion in all the spam it received
> until it apparently died away in (almost) silence a couple of years ago.

Yes, although the mailing lists for special interest groups are advertised on 
python.org, there is no longer the focus on steering discussion to those 
lists. And I see that the compiler SIG is "retired", as is the related types 
SIG:

http://www.python.org/community/sigs/retired/

I seem to remember various procedures about SIGs and their retirement, but I 
don't really recall much discussion of such things recently. Still, the 
compiler SIG matches the scope of the Google group pretty well:

http://www.python.org/community/sigs/retired/compiler-sig/

There's even a link to discussion of some tools you may be familiar with.

> > It's obviously your choice where you host discussions and who you invite,
> > and I know that the special interest group mailing lists aren't exactly
> > well advertised these days
>
> True, but many (most?) of them are simply not very well frequented, which
> reduces the interest in joining them even further. Both SIG mailing lists
> that I read only receive a mail every so many months, often enough without
> any reply. And almost all of these mails deal with questions that would
> better be discussed on python-list to leverage the substantially higher
> number of eyeballs there.

Special interest group lists were always meant to be used as focused channels 
of communication where people are actively trying to get stuff done. The 
unfortunate thing is that they aren't as well known as they were. Another 
unfortunate thing is that getting stuff done of mutual benefit is frequently 
something that takes second place to whatever other motivations and goals 
people have, for whatever reason, good or bad. Thus, traffic drops away as 
people either do other things entirely or instead promote any related work in 
other channels instead.

> I think that's the basic problem: as long as more experts are lurking on
> python-list than on the dedicated SIG-ML, it's better not to use the SIG-ML
> for discussions but to go to python-list (or maybe python-ideas or
> python-dev) straight away.

I think we really have to sort out what python-dev is for, because currently 
there's a tendency to target the list when any kind of "expert" discussion is 
required, but there are a number of people who would rather see only 
CPython-related discussion here.

Another matter is that static analysis of Python is a topic that frequently 
hits the end of the road when one cannot, by definition, analyze Python in 
its most dynamic form, and when people refuse to accept that such analysis 
has anything to do with Python in its most pure, undiluted (and most 
contrived) form.

But as I wrote, I still intend to follow the newly created group and see what 
people have to say.

Paul

From solipsis at pitrou.net  Sun Jul  1 23:39:59 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 1 Jul 2012 23:39:59 +0200
Subject: [Python-Dev] hg.python.org migrated
Message-ID: <20120701233959.745ec8a6@pitrou.net>


Hello,

hg.python.org has just been migrated to a new (virtual) machine hosted
by OSU OSL (*). Until the domain name fully propagates, you won't be
able to push or pull from the repositories using the ssh protocol.
If you notice other issues, don't hesitate to mention them.

(*) http://osuosl.org/

Regards

Antoine.



From roundup-admin at psf.upfronthosting.co.za  Mon Jul  2 20:36:31 2012
From: roundup-admin at psf.upfronthosting.co.za (Python tracker)
Date: Mon, 02 Jul 2012 18:36:31 +0000
Subject: [Python-Dev] Failed issue tracker submission
Message-ID: <20120702183631.DFC031CC82@psf.upfronthosting.co.za>



There was a problem with the message you sent:
   This issue can't be closed until issue <a href="http://bugs.python.org/issue15031">15031</a> is closed.



Mail Gateway Help
=================
Incoming messages are examined for multiple parts:
 . In a multipart/mixed message or part, each subpart is extracted and
   examined. The text/plain subparts are assembled to form the textual
   body of the message, to be stored in the file associated with a "msg"
   class node. Any parts of other types are each stored in separate files
   and given "file" class nodes that are linked to the "msg" node.
 . In a multipart/alternative message or part, we look for a text/plain
   subpart and ignore the other parts.
 . A message/rfc822 is treated similar tomultipart/mixed (except for
   special handling of the first text part) if unpack_rfc822 is set in
   the mailgw config section.

Summary
-------
The "summary" property on message nodes is taken from the first non-quoting
section in the message body. The message body is divided into sections by
blank lines. Sections where the second and all subsequent lines begin with
a ">" or "|" character are considered "quoting sections". The first line of
the first non-quoting section becomes the summary of the message.

Addresses
---------
All of the addresses in the To: and Cc: headers of the incoming message are
looked up among the user nodes, and the corresponding users are placed in
the "recipients" property on the new "msg" node. The address in the From:
header similarly determines the "author" property of the new "msg"
node. The default handling for addresses that don't have corresponding
users is to create new users with no passwords and a username equal to the
address. (The web interface does not permit logins for users with no
passwords.) If we prefer to reject mail from outside sources, we can simply
register an auditor on the "user" class that prevents the creation of user
nodes with no passwords.

Actions
-------
The subject line of the incoming message is examined to determine whether
the message is an attempt to create a new item or to discuss an existing
item. A designator enclosed in square brackets is sought as the first thing
on the subject line (after skipping any "Fwd:" or "Re:" prefixes).

If an item designator (class name and id number) is found there, the newly
created "msg" node is added to the "messages" property for that item, and
any new "file" nodes are added to the "files" property for the item.

If just an item class name is found there, we attempt to create a new item
of that class with its "messages" property initialized to contain the new
"msg" node and its "files" property initialized to contain any new "file"
nodes.

Triggers
--------
Both cases may trigger detectors (in the first case we are calling the
set() method to add the message to the item's spool; in the second case we
are calling the create() method to create a new node). If an auditor raises
an exception, the original message is bounced back to the sender with the
explanatory message given in the exception.

$Id: mailgw.py,v 1.196 2008-07-23 03:04:44 richard Exp $
-------------- next part --------------
Return-Path: <python-dev at python.org>
X-Original-To: report at bugs.python.org
Delivered-To: roundup+tracker at psf.upfronthosting.co.za
Received: from mail.python.org (mail.python.org [82.94.164.166])
	by psf.upfronthosting.co.za (Postfix) with ESMTPS id 5E7D11CC7C
	for <report at bugs.python.org>; Mon,  2 Jul 2012 20:36:31 +0200 (CEST)
Received: from albatross.python.org (localhost [127.0.0.1])
	by mail.python.org (Postfix) with ESMTP id 3WQxxp54vszPJ6
	for <report at bugs.python.org>; Mon,  2 Jul 2012 20:36:30 +0200 (CEST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=python.org; s=200901;
	t=1341254190; bh=bcoPtckr6VOn/nnWeOnKJxf0E8057TGiXziLdKxmDAU=;
	h=MIME-Version:Content-Type:Content-Transfer-Encoding:From:To:
	 Subject:Message-Id:Date;
	b=LeXZP20cZnfbA2FHjnPNH2KEX5ZCGryZdBwT5XwAqbn2PrdQCxB0OJ766NdEkuI5N
	 RlaB95Vz83qouHfh8sVxO8welo9E4UpoXVEEvwFt5/XYUtDG3eoVp6ufKpWCxut1Sj
	 PDwgNWINBY5UPYtDB92G1S7YTLttEiQ/jt2a/vaE=
Received: from localhost (HELO mail.python.org) (127.0.0.1)
  by albatross.python.org with SMTP; 02 Jul 2012 20:36:30 +0200
Received: from virt-7yvsjn.psf.osuosl.org (virt-7yvsjn.psf.osuosl.org [140.211.10.72])
	by mail.python.org (Postfix) with ESMTP
	for <report at bugs.python.org>; Mon,  2 Jul 2012 20:36:30 +0200 (CEST)
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: base64
From: python-dev at python.org
To: report at bugs.python.org
Subject: [issue15030] [status=closed; resolution=fixed;
 stage=committed/rejected]
Message-Id: <3WQxxp54vszPJ6 at mail.python.org>
Date: Mon,  2 Jul 2012 20:36:30 +0200 (CEST)

TmV3IGNoYW5nZXNldCBiNzQ2M2VjMTk4MGMgYnkgQnJldHQgQ2Fubm9uIGluIGJyYW5jaCAnZGVm
YXVsdCc6CkNsb3NlcyAjMTUwMzA6IE1ha2UgaW1wb3J0bGliLmFiYy5QeVB5Y0xvYWRlciByZXNw
ZWN0IHRoZSBuZXcgLnB5YwpodHRwOi8vaGcucHl0aG9uLm9yZy9jcHl0aG9uL3Jldi9iNzQ2M2Vj
MTk4MGMK

From solipsis at pitrou.net  Mon Jul  2 20:51:35 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 2 Jul 2012 20:51:35 +0200
Subject: [Python-Dev] New stable buildbots
Message-ID: <20120702205135.1d9607c4@pitrou.net>


Hello,

I have decided to add two new buildslaves to the stable buildbots fleet:
- ?ukasz Langa's AMD64 OS Lion buildbot (using clang as compiler)
- Jeremy Kloth's AMD64 Windows7 buildbot (our first 64-bit Windows
  buildbot!)

They bring the number of stable buildbots to twelve: 4 Windows, 5
Linux and 3 other Unices:
http://buildbot.python.org/all/waterfall?category=3.x.stable

Regards

Antoine.



From tjreedy at udel.edu  Mon Jul  2 21:14:38 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 02 Jul 2012 15:14:38 -0400
Subject: [Python-Dev] New stable buildbots
In-Reply-To: <20120702205135.1d9607c4@pitrou.net>
References: <20120702205135.1d9607c4@pitrou.net>
Message-ID: <jssrv1$hf7$1@dough.gmane.org>

On 7/2/2012 2:51 PM, Antoine Pitrou wrote:
>
> Hello,
>
> I have decided to add two new buildslaves to the stable buildbots fleet:
> - ?ukasz Langa's AMD64 OS Lion buildbot (using clang as compiler)
> - Jeremy Kloth's AMD64 Windows7 buildbot (our first 64-bit Windows
>    buildbot!)

Great.

> They bring the number of stable buildbots to twelve: 4 Windows,

and all seems to have failed compiling a1c8302e6b27 after succeeding.
Is that from the repository migration?

> http://buildbot.python.org/all/waterfall?category=3.x.stable

-- 
Terry Jan Reedy





From solipsis at pitrou.net  Mon Jul  2 21:54:53 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 2 Jul 2012 21:54:53 +0200
Subject: [Python-Dev] New stable buildbots
References: <20120702205135.1d9607c4@pitrou.net> <jssrv1$hf7$1@dough.gmane.org>
Message-ID: <20120702215453.07aedcf8@pitrou.net>

On Mon, 02 Jul 2012 15:14:38 -0400
Terry Reedy <tjreedy at udel.edu> wrote:

> On 7/2/2012 2:51 PM, Antoine Pitrou wrote:
> >
> > Hello,
> >
> > I have decided to add two new buildslaves to the stable buildbots fleet:
> > - ?ukasz Langa's AMD64 OS Lion buildbot (using clang as compiler)
> > - Jeremy Kloth's AMD64 Windows7 buildbot (our first 64-bit Windows
> >    buildbot!)
> 
> Great.
> 
> > They bring the number of stable buildbots to twelve: 4 Windows,
> 
> and all seems to have failed compiling a1c8302e6b27 after succeeding.
> Is that from the repository migration?

No, it's from the buildmaster restart.
However, David Bolen's Windows 7 buildbot seems to show weird and
persistent compile errors unrelated to Windows' file locking:


..\PC\python_nt.rc(35): error RC2104: undefined keyword or key name:
  MS_DLL_ID
  [D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pythoncore.vcxproj]

..\PC\pylauncher.rc(16): error RC2104: undefined
keyword or key name: FIELD3
[D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pylauncher.vcxproj]

..\PC\pylauncher.rc(16): error RC2104: undefined
keyword or key name: FIELD3
[D:\cygwin\home\db3l\buildarea\3.x.bolen-windows7\build\PCbuild\pywlauncher.vcxproj]


Regards

Antoine.



From solipsis at pitrou.net  Mon Jul  2 23:20:27 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 2 Jul 2012 23:20:27 +0200
Subject: [Python-Dev] cpython: Issue #15210: If _frozen_importlib is not
 found in sys.modules by
References: <3WQyK70dd5zPCD@mail.python.org>
Message-ID: <20120702232027.5be1f0e2@pitrou.net>

On Mon,  2 Jul 2012 20:53:15 +0200 (CEST)
brett.cannon <python-checkins at python.org> wrote:

> http://hg.python.org/cpython/rev/818db871d29a
> changeset:   77916:818db871d29a
> user:        Brett Cannon <brett at python.org>
> date:        Mon Jul 02 14:53:10 2012 -0400
> summary:
>   Issue #15210: If _frozen_importlib is not found in sys.modules by
> importlib.__init__, then catch the KeyError raised, not ImportError.

There's a test failure here:

http://buildbot.python.org/all/builders/AMD64%20Fedora%20without%20threads%203.x/builds/2774/steps/test/logs/stdio

Regards

Antoine.



From raymond.hettinger at gmail.com  Tue Jul  3 06:23:29 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 2 Jul 2012 21:23:29 -0700
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <jspe70$4gq$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<jsoo4p$via$1@dough.gmane.org>
	<2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com>
	<jspe70$4gq$1@dough.gmane.org>
Message-ID: <877C3C8D-1DE3-4DB0-812A-B352E8608364@gmail.com>


On Jul 1, 2012, at 5:01 AM, Stefan Behnel wrote:

> To address the main problem of users not finding what they need, what about
> simply extending the docstring of the grouper()


Here's  a small change to the docstring:  http://hg.python.org/cpython/rev/d32f21d87363

FWIW, if you're interested in load balancing applications, George Sakkis's itertools 
recipe for roundrobin() may be of interest.

Another interesting iterator technique that is not well known is the two-argument 
form of iter() which is a marvel for transforming callables into iterators:

    for block in iter(partial(f.read, 1024), ''):
           ...

    for diceroll in iter(partial(randrange(1, 7), 4):
           ...


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120702/8550f12c/attachment.html>

From techtonik at gmail.com  Wed Jul  4 10:42:38 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 4 Jul 2012 11:42:38 +0300
Subject: [Python-Dev] GitHub mirror (Was:  Bitbucket mirror?)
Message-ID: <CAPkN8xKN5y7LX9uQ=sPfDt-F3FkKYd0jjAJ6+aPaaCDAL8VTqw@mail.gmail.com>

On Fri, Jun 29, 2012 at 6:58 AM, Eli Bendersky <eliben at gmail.com> wrote:
>
> The devguide (http://docs.python.org/devguide/committing.html) says:
>
> Bitbucket also maintain an up to date clone of the main cpython repository
> that can be used as the basis for a new clone or patch queue.
>
> [the link goes to https://bitbucket.org/mirror/cpython/overview]

This one is better than Atlassian's (looks nice, shorter URL) and
worked long before Atlassian promotion as "makers of Bitbucket", which
is dead wrong BTW, because everybody knows they bought BB. I don't
remember if I commented on the Python Insider or if it was censored.
Whatever.. it hurts Atlassian, not me.

On the subject. Is there a mirror of CPython on GitHub?
I abandoned my efforts to create online editor for docs, which should
have been both web fronted independent, maintainable and clean. It
appeared much easier to just use GitHub - so far it is the only one
from open source code hosting services that directly commits online
changes to repository (and allows anonymous to do this). I've made
more than a dozen proposal for fixing docs, because as a matter of
fact - filling a bug AND explaining why docs are wrong, why they need
to be fixed, what should be added - all of this is a way *much easier*
(and less time consuming!) than just fixing them. Unfortunately.

From ralf at systemexit.de  Wed Jul  4 11:51:25 2012
From: ralf at systemexit.de (Ralf Schmitt)
Date: Wed, 04 Jul 2012 11:51:25 +0200
Subject: [Python-Dev] GitHub mirror
In-Reply-To: <CAPkN8xKN5y7LX9uQ=sPfDt-F3FkKYd0jjAJ6+aPaaCDAL8VTqw@mail.gmail.com>
	(anatoly techtonik's message of "Wed, 4 Jul 2012 11:42:38 +0300")
References: <CAPkN8xKN5y7LX9uQ=sPfDt-F3FkKYd0jjAJ6+aPaaCDAL8VTqw@mail.gmail.com>
Message-ID: <8762a326si.fsf@winserver.brainbot.com>

anatoly techtonik <techtonik at gmail.com> writes:

> On the subject. Is there a mirror of CPython on GitHub?

https://github.com/schmir/python

-- 
Cheers
Ralf

From techtonik at gmail.com  Wed Jul  4 11:57:20 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Wed, 4 Jul 2012 12:57:20 +0300
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <jsl3cb$knn$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org>
Message-ID: <CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>

On Fri, Jun 29, 2012 at 11:32 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> On 26.06.2012 10:03, anatoly techtonik wrote:
>>
>> Now that Python 3 is all about iterators (which is a user killer
>> feature for Python according to StackOverflow -
>> http://stackoverflow.com/questions/tagged/python) would it be nice to
>> introduce more first class functions to work with them? One function
>> to be exact to split string into chunks.
>>
>>      itertools.chunks(iterable, size, fill=None)
>>
>> Which is the 33th most voted Python question on SO -
>>
>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464
>>
>> P.S. CC'ing to python-dev@ to notify about the thread in python-ideas.
>>
>
> Anatoly, so far there were no negative votes -- would you care to go
> another step and propose a patch?

Was about to say "no problem", but in fact - there is. Sorry from
whining from my side and thanks for nudging. The only thought that a
simple task of copy/pasting relevant code from
http://docs.python.org/library/itertools.html?highlight=itertools#recipes
will require a few hours waiting of download (still not everybody has
a high-speed internet) makes me switch to other less time consuming
tasks before getting around to it. These tasks become more important
in a few hours, and basically I've passed through this many times
before. It then becomes quite hard to switch back.

I absolutely don't mind someone else being credited for the idea,
because ideas usually worthless without implementation. It will be
interesting to design how the process could work in a separate thread.
For now the best thing I can do (I don't risk even to mention anything
with 3.3) is to copy/paste code from the docs here:

from itertools import izip_longest
def chunks(iterable, size, fill=None):
    """Split an iterable into blocks of fixed-length"""
    # chunks('ABCDEFG', 3, 'x') --> ABC DEF Gxx
    args = [iter(iterable)] * size
    return izip_longest(fillvalue=fill, *args)

BTW, this doesn't work as expected (at least for strings). Expected is:
  chunks('ABCDEFG', 3, 'x') --> 'ABC' 'DEF' 'Gxx'
got:
  chunks('ABCDEFG', 3, 'x') --> ('A' 'B' 'C') ('D' 'E' 'F') ('G' 'x' 'x')

Needs more round tuits definitely.

From stefan_ml at behnel.de  Wed Jul  4 14:37:12 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 04 Jul 2012 14:37:12 +0200
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org>
	<CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>
Message-ID: <jt1ddo$8hb$1@dough.gmane.org>

anatoly techtonik, 04.07.2012 11:57:
> On Fri, Jun 29, 2012 at 11:32 PM, Georg Brandl wrote:
>> On 26.06.2012 10:03, anatoly techtonik wrote:
>>> Now that Python 3 is all about iterators (which is a user killer
>>> feature for Python according to StackOverflow -
>>> http://stackoverflow.com/questions/tagged/python) would it be nice to
>>> introduce more first class functions to work with them? One function
>>> to be exact to split string into chunks.
>>>
>>>      itertools.chunks(iterable, size, fill=None)
>>>
>>> Which is the 33th most voted Python question on SO -
>>>
>>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464
>>>
>>> P.S. CC'ing to python-dev@ to notify about the thread in python-ideas.
>>>
>>
>> Anatoly, so far there were no negative votes -- would you care to go
>> another step and propose a patch?
> 
> Was about to say "no problem", but in fact - there is. Sorry from
> whining from my side and thanks for nudging. The only thought that a
> simple task of copy/pasting relevant code from
> http://docs.python.org/library/itertools.html?highlight=itertools#recipes
> will require a few hours waiting of download (still not everybody has
> a high-speed internet) makes me switch to other less time consuming
> tasks before getting around to it. These tasks become more important
> in a few hours, and basically I've passed through this many times
> before. It then becomes quite hard to switch back.
> 
> I absolutely don't mind someone else being credited for the idea,
> because ideas usually worthless without implementation. It will be
> interesting to design how the process could work in a separate thread.
> For now the best thing I can do (I don't risk even to mention anything
> with 3.3) is to copy/paste code from the docs here:
> 
> from itertools import izip_longest
> def chunks(iterable, size, fill=None):
>     """Split an iterable into blocks of fixed-length"""
>     # chunks('ABCDEFG', 3, 'x') --> ABC DEF Gxx
>     args = [iter(iterable)] * size
>     return izip_longest(fillvalue=fill, *args)

I think Raymond's change fixes this issue quite nicely, no need to touch at
the module code.

Stefan


From larry at hastings.org  Wed Jul  4 18:46:21 2012
From: larry at hastings.org (Larry Hastings)
Date: Wed, 04 Jul 2012 18:46:21 +0200
Subject: [Python-Dev] EuroPython 2012 Language Summit Is In JEOPARDY *gasp*
Message-ID: <4FF4735D.5020801@hastings.org>



So far I've had exactly four reservations for the Language Summit at 
EuroPython 2012.  One of them is Guido--but he's threatening to skip it 
if we don't get more people and just go to the sprints.

Also, honestly I have next-to-nothing on the docket.  At this point 
we've just hit feature freeze for Python 3.3--or at least that's what 
people tell me. In general, anyone who wanted to get a language change 
in have already either just succeeded or just failed.  So after a flurry 
of activity I feel like we've hit a quiet period.  I mean, the trunk for 
3.4 won't even be open for a month or two.

So: if you're a core developer, and you're interested in attending the 
Language Summit in Florence on Saturday, please email me your RSVP.  
Also, if you have suggestions for things we should discuss, send those 
along too.  Please send all these emails directly to me, *off-list*, 
tonight or Thursday.

I'll send another email on Friday morning decreeing the fate of the 
Language Summit.

Thanks!


//arry/

p.s. I'm assured the room we'd have for the Language Summit has 
excellent air conditioning.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120704/db89d289/attachment.html>

From tjreedy at udel.edu  Wed Jul  4 20:31:19 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 04 Jul 2012 14:31:19 -0400
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org>
	<CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>
Message-ID: <jt225u$iih$1@dough.gmane.org>

On 7/4/2012 5:57 AM, anatoly techtonik wrote:
> On Fri, Jun 29, 2012 at 11:32 PM, Georg Brandl <g.brandl at gmx.net> wrote:

>> Anatoly, so far there were no negative votes -- would you care to go
>> another step and propose a patch?
>
> Was about to say "no problem",

Did you read that there *are* strong negative votes? And that this idea 
has been rejected before? I summarized the objections in my two 
responses and pointed to the tracker issues. One of the objections is 
that there are 4 different things one might want if the sequence length 
is not an even multiple of the chunk size. Your original 'idea' did not 
specify.

> For now the best thing I can do (I don't risk even to mention anything
> with 3.3) is to copy/paste code from the docs here:
>
> from itertools import izip_longest
> def chunks(iterable, size, fill=None):
>      """Split an iterable into blocks of fixed-length"""
>      # chunks('ABCDEFG', 3, 'x') --> ABC DEF Gxx
>      args = [iter(iterable)] * size
>      return izip_longest(fillvalue=fill, *args)

Python ideas is about Python 3 ideas. Please post Python 3 code.

This is actually a one liner

     return zip_longest(*[iter(iterable)]*size, fillvalue=file)

We don't generally add such to the stdlib.

> BTW, this doesn't work as expected (at least for strings). Expected is:
>    chunks('ABCDEFG', 3, 'x') --> 'ABC' 'DEF' 'Gxx'
> got:
>    chunks('ABCDEFG', 3, 'x') --> ('A' 'B' 'C') ('D' 'E' 'F') ('G' 'x' 'x')

One of the problems with idea of 'add a chunker' is that there are at 
least a dozen variants that different people want. I discussed the 
problem of return types issue in my responses. I showed how to get the 
'expected' response above using grouper, but also suggested that it is 
the wrong basis for splitting strings. Repeated slicing make more sense 
for concrete sequence types.

def seqchunk_odd(s, size):
     # include odd size left over
     for i in range(0, len(s), size):
         yield s[i:i+size]

print(list(seqchunk_odd('ABCDEFG', 3)))
#
['ABC', 'DEF', 'G']

def seqchunk_even(s, size):
     # only include even chunks
     for i in range(0, size*(len(s)//size), size):
         yield s[i:i+size]

print(list(seqchunk_even('ABCDEFG', 3)))
#
['ABC', 'DEF']

def strchunk_fill(s, size, fill):
     # fill odd chunks
     q, r = divmod(len(s), size)
     even = size * q
     for i in range(0, even, size):
         yield s[i:i+size]
     if size != even:
         yield s[even:] + fill * (size - r)

print(list(strchunk_fill('ABCDEFG', 3, 'x')))
#
['ABC', 'DEF', 'Gxx']

Because the 'fill' value is necessarily a sequence for strings, 
strchunk_fill would only work for lists and tuples if the fill value 
were either required to be given as a tuple or list of length 1 or if it 
were internally converted inside the function. Skipping that for now.

Having written the fill version based on the even version, it is easy to 
select among the three behaviors by modifying the fill version.

def strchunk(s, size, fill=NotImplemented):
     # fill odd chunks
     q, r = divmod(len(s), size)
     even = size * q
     for i in range(0, even, size):
         yield s[i:i+size]
     if size != even and fill is not NotImplemented:
         yield s[even:] + fill * (size - r)

print(*strchunk('ABCDEFG', 3))
print(*strchunk('ABCDEFG', 3, ''))
print(*strchunk('ABCDEFG', 3, 'x'))
#
ABC DEF
ABC DEF G
ABC DEF Gxx

I already described how something similar could be done by checking each 
grouper output tuple for a fill value, but that requires that the fill 
value be a sentinal that could not otherwise appear in the tuple. One 
could modify grouper to fill with a private object() and check the last 
item of each group for that sentinal and act accordingly (delete, 
truncate, or replace). A generic api needs some thought, though.

---
An issue I did not previously mention is that people sometimes want 
overlapping chunks rather than contiguous disjoint chunks. The slice 
approach trivially adapts to that.

def seqlap(s, size):
     for i in range(len(s)-size+1):
         yield s[i:i+size]

print(*seqlap('ABCDEFG', 3))
#
ABC BCD CDE DEF EFG

A sliding window for a generic iterable requires a deque or ring buffer 
approach that is quite different from the zip-longest -- grouper approach.

-- 
Terry Jan Reedy




From petri at digip.org  Thu Jul  5 14:49:52 2012
From: petri at digip.org (Petri Lehtinen)
Date: Thu, 5 Jul 2012 15:49:52 +0300
Subject: [Python-Dev] GitHub mirror (Was:  Bitbucket mirror?)
In-Reply-To: <CAPkN8xKN5y7LX9uQ=sPfDt-F3FkKYd0jjAJ6+aPaaCDAL8VTqw@mail.gmail.com>
References: <CAPkN8xKN5y7LX9uQ=sPfDt-F3FkKYd0jjAJ6+aPaaCDAL8VTqw@mail.gmail.com>
Message-ID: <20120705124952.GA17866@chang>

anatoly techtonik wrote:
> On the subject. Is there a mirror of CPython on GitHub?

https://github.com/akheron/cpython

> changes to repository (and allows anonymous to do this). I've made
> more than a dozen proposal for fixing docs, because as a matter of
> fact - filling a bug AND explaining why docs are wrong, why they need
> to be fixed, what should be added - all of this is a way *much easier*
> (and less time consuming!) than just fixing them. Unfortunately.

You won't get any changes in to CPython by creating pull requests. We
use http://bugs.python.org/ for that, sorry.

From techtonik at gmail.com  Thu Jul  5 15:36:24 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Jul 2012 16:36:24 +0300
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <4FEF6B10.9040409@udel.edu>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
Message-ID: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>

Before anything else I must apologize for significant lags in my
replies. I can not read all of them to hold in my head, so I reply one
by one as it goes trying not to miss a single point out there. It
would be much easier to do this in unified interface for threaded
discussions, but for now there is no capabilities for that neither in
Mailman nor in GMail. And when it turns out that the amount of text is
too big, and I spend a lot of time trying to squeeze it down and then
it becomes pointless to send at all.

Now back on the topic:

On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 6/29/2012 4:32 PM, Georg Brandl wrote:
>>
>> On 26.06.2012 10:03, anatoly techtonik wrote:
>>>
>>> Now that Python 3 is all about iterators (which is a user killer
>>> feature for Python according to StackOverflow -
>>> http://stackoverflow.com/questions/tagged/python) would it be nice to
>>> introduce more first class functions to work with them? One function
>>> to be exact to split string into chunks.
>
> Nothing special about strings.

It seemed so, but it just appeared that grouper recipe didn't work for me.

>>>      itertools.chunks(iterable, size, fill=None)
>
> This is a renaming of itertools.grouper in 9.1.2. Itertools Recipes. You
> should have mentioned this. I think of 'blocks' rather than 'chunks', but I
> notice several SO questions with 'chunk(s)' in the title.

I guess `block` gives too low signal/noize ration in search results.
That's why it probably also called chunks in other languages, where
`block` stand for something else (I speak of Ruby blocks).

>>> Which is the 33th most voted Python question on SO -
>>>
>>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464
>
> I am curious how you get that number. I do note that there are about 15
> other Python SO questions that seem to be variations on the theme. There
> might be more if 'blocks' and 'groups' were searched for.

It's easy:
1. Go http://stackoverflow.com/
2. Search [python]
3. Click `votes` tab
4. Choose `30 per page` at the bottom
5. Jump to the second page, there it is 4th from the top:
http://stackoverflow.com/questions/tagged/python?page=2&sort=votes&pagesize=30

As for duplicates - feel free to mark them as such. SO allows
everybody to do this (unlike Roundup).

>> Anatoly, so far there were no negative votes -- would you care to go
>> another step and propose a patch?
>
> That is because Raymond H. is not reading either list right now ;-)
> Hence the Cc:. Also because I did not yet respond to a vague, very
> incomplete idea.
>
> From Raymond's first message on http://bugs.python.org/issue6021 , add
> grouper:
>
> "This has been rejected before.

I quite often see such arguments and I can't stand to repeat that
these are not arguments. It is good to know, but when people use that
as a reason to close tickets - that's just disgusting. To the
Raymond's honor he cares to explain.

> * It is not a fundamental itertool primitive.  The recipes section in
> the docs shows a clean, fast implementation derived from zip_longest().

What is the definition of 'fundamental primitive'?
To me the fact that top answer for chunking strings on SO has 2+ times
more votes than itertools versions is a clear 5 sigma indicator that
something is wrong with this Standard model without chunks boson.

> * There is some debate on a correct API for odd lengths.  Some people
> want an exception, some want fill-in values, some want truncation, and
> some want a partially filled-in tuple.  The alone is reason enough not
> to set one behavior in stone.

use case 3.1: odd lengths exception (CHOOSE ONE)
1. I see that no itertools function throws exceptions, check manually:
    len(iterable) / float(size) ==  len(iterable) // float(size)
2. Explicitly
  -  itertools.chunks(iterable, size, fill=None)
  +  itertools.chunks(iterable, size, fill=None, exception=False)

use case 3.2. fill in value. it is here (SOLVED)

use case 3.3: truncation
no itertools support truncation, do manually
   chunks(iter, size)[:len(iter)//size)

use case 4: partially filled-in tuple
  What should be there?
   >>> chunks('ABCDEFG', 3, 'x')
   >>> |


More replies and workarounds to some of the raised points are below.

> * There is an issue with having too many itertools.  The module taken as
> a whole becomes more difficult to use as new tools are added."

There can be only two reasons to that:
* chosen basis is bad (many functions that are rarely used or easily emulated)
* basis is good, but insufficient, because iterators universe is more
complicated
  than we think

> This is not to say that the question should not be re-considered. Given the
> StackOverflow experience in addition to that of the tracker and python-list
> (and maybe python-ideas), a special exception might be made in relation to
> points 1 and 3.

--[offtopic about Python enhancements / proposals feedback]--
Yes, without SO I probably wouldn't trigger this at all. Because
tracker doesn't help with raising importance - there are no votes, no
feature proposals, no "stars". And what I "like" the most is that very
"nice" resolution status - "committed/rejected" - which doesn't say
anything at all. Python list? I try not to disrupt the frequency
there. Python ideas? Too low participation level for gathering
signals. There are many people that read, support, but don't want to
reply (don't want to stand out or just lazy). There are many outside
who don't want to be subscribed at all. There are 2000+ people
spending time on Python conferences all over the world each year we
see only a couple of reactions for every Python idea here. Quite often
there are mistakes and omissions that would be nice to correct and you
can't. So StackOverflow really helps here, but it is a Q&A tool, which
is still much better than ML that are solely for chatting,
brainstorming and all the crazy reading / writing stuff. They don't
help to develop ideas collaboratively. Quite often I am just lost in
amount of text to handle.
--[/offtopic]--

> It regard to point 2: many 'proposals', including Anatoly's, neglect this
> detail. But the function has to do *something* when seqlen % grouplen != 0.
> So an 'idea' is not really a concrete programmable proposal until
> 'something' is specified.
>
> Exception -- not possible for an itertool until the end of the iteration
> (see below). To raise immediately for sequences, one could wrap grouper.
>
> def exactgrouper(sequence, k):  # untested
>   if len(sequence) % k:
>     raise ValueError('Sequence length {} must be a multiple of group length
> {}'.format(len(sequence), k)
>   else:
>     return itertools.grouper(sequence, k)

Right. Iterator is not a sequence, because it doesn't know the length
of its sequence. The method should not belong to itertools at all
then.

Python 3 is definitely become more complicated. I'd prefer to keep
separated from iterator stuff, but it seems more harder with every
iteration.

> Of course, sequences can also be directly sequentially sliced (but should
> the result be an iterable or sequence of blocks?). But we do not have a
> seqtools module and I do not think there should be another method added to
> the seq protocol.

I'd expect strings chunked into strings and lists into lists. Don't
want to know anything about protocols.

> Fill -- grouper always does this, with a default of None.
>
> Truncate, Remainder -- grouper (zip_longest) cannot directly do this and no
> recipes are given in the itertools docs. (More could be, see below.)
>
> Discussions on python-list gives various implementations either for
> sequences or iterables. For the latter, one approach is "it =
> iter(iterable)" followed by repeated islice of the first n items. Another is
> to use a sentinal for the 'fill' to detect a final incomplete block (tuple
> for grouper).
>
> def grouper_x(n, iterable):  # untested
>   sentinal = object()
>   for g in grouper(n, iterable, sentinal):
>     if g[-1] != sentinal:
>       yield g
>     else:
>       # pass to truncate
>       # yield g[:g.index(sentinal) for remainer
>       # raise ValueError for delayed exception

We need a simple function to split a sequence into chunks(). Now we
face with the problem to apply that technique to a sequence of
infinite length when a last element of infinite sequence is
encountered. You might be thinking now that this is a reduction to
absurdity. But I'd say it is an exit from the trap. Mathematically
this problem can't be solved. I am not ignoring your solution - I
think it's quite feasible, but isn't it an overcomplication?

I mean 160 people out of 149 who upvoted the question are pretty happy
with an answer that just outputs the last chunk as-is:
http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python

   chunks('ABCDEFG', 3) --> 'ABC' 'DEF' 'G'

And it is quite nice solution to me, because you're free to do
anything you'd like if you expect you data to be odd:

   for chunk in chunks('ABCDEFG', size):
     if len(chunk) < size:
       raise Tail

You can make a helper iterator out of it too.

> ---
> The above discussion of point 2 touches on point 4, which Raymond neglected
> in the particular message above but which has come up before: What are the
> allowed input and output types? An idea is not a programmable proposal until
> the domain, range, and mapping are specified.

Domain? Mapping? I am not ignoring existing knowledge and experience.
I just don't want to complicate and don't see appropriate `import
usecase` in current context, so I won't try to guess what this means.

in string -> out list of strings
in list -> out list of lists

> Possible inputs are a specific sequence (string, for instance), any
> sequence, any iterable. Possible outputs are a sequence or iterator of
> sequence or iterator. The various python-list and stackoverflow posts
> questions asks for various combinations. zip_longest and hence grouper takes
> any iterable and returns an iterator of tuples. (An iterator of maps might
> be more useful as a building block.) This is not what one usually wants with
> string input, for instance, nor with range input. To illustrate:

Allright. Got it. Sequences have a length and can be sliced with
[i:j], iterator can't be sliced (and hence no chunks can be made). So
this function doesn't belong to itertools - it is a missing string or
sequence method. We can't have a chunk with an iterator, because
iterator over a string decomposes it into a group of pieces with no
reverse function. We can have a group and then join the group into
something. But this requires the knowledge of appropriate join()
function for the iterator, and probably not efficient. As there are no
such function (must be that Mapping you referenced above) - the
recomposition into chunks is impossible.

> import itertools as it
>
> def grouper(n, iterable, fillvalue=None):
>     "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
>     args = [iter(iterable)] * n
>     return it.zip_longest(*args, fillvalue=fillvalue)
>
> print(*(grouper(3, 'ABCDEFG', 'x')))  # probably not wanted
> print(*(''.join(g) for g in grouper(3, 'ABCDEFG', 'x')))
> #
> ('A', 'B', 'C') ('D', 'E', 'F') ('G', 'x', 'x')
> ABC DEF Gxx
>
> --
> What to do? One could easily write 20 different functions. So more thought
> is needed before adding anything. -1 on the idea as is.

I've learned a new English type of argument - "straw man" (I used to
call this "hijacking"). This -1 doesn't belong to original idea. It
belongs to proposal of  itertools.chunks() with a long list of above
points and completely different user stories (i.e. not "split string
into chunks"). I hope you still +1 with 160 people on SO that think
Python needs an easy way to chunk sequences.

> For the doc, I think it would be helpful here and in most module subchapters
> if there were a subchapter table of contents at the top (under 9.1 in this
> case). Even though just 2 lines here (currently, but see below), it would
> let people know that there *is* a recipes section. After the appropriate
> tables, mention that there are example uses in the recipe section. Possibly
> add similar tables in the recipe section.

Unfortunately, it appeared that grouper() is not chunks(). It doesn't
delivers list of list of chars given string as an input instead of
list of chunks.

> Another addition could be a new subsection on grouping (chunking) that would
> discuss post-processing of grouper (as discussed above), as well as other
> recipes, including ones specific to strings and sequences. It would
> essentially be a short how-to. Call it 9.1.3 "Grouping, Blocking, or
> Chunking Sequences and Iterables". The synonyms will help external
> searching. A toc would let people who have found this doc know to look for
> this at the bottom.

This makes matters pretty ugly. In ideal language there should be less
docs, not more.

From techtonik at gmail.com  Thu Jul  5 15:47:53 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Jul 2012 16:47:53 +0300
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <jspe70$4gq$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<jsoo4p$via$1@dough.gmane.org>
	<2E44286F-D3DB-4739-89AF-3A8EAA9D8901@gmail.com>
	<jspe70$4gq$1@dough.gmane.org>
Message-ID: <CAPkN8xJrKgAUaLw3siTpzAjUB=eg1S08oWu-Z=7ThHqgU0eM+w@mail.gmail.com>

On Sun, Jul 1, 2012 at 3:01 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>
> To address the main problem of users not finding what they need, what about
> simply extending the docstring of the grouper() function with a sentence
> like this:
>
> "This functionality is also called 'chunking' or 'blocking' and can be used
> for load distribution and sharding."
>
> That would make it easy for users to find what they are looking for when
> they search the page for "chunk". I find that a much more common and less
> ambiguous name than "grouping", which reminds me more of "group by".

In appeared that "chunking" and "grouping" are different kind of
tasks. You can chunk a sequence (sting) by slicing it into smaller
sequences, but you can not chunk in iterable - you can only group it.

There is an data loss about the structure that occurs when a sequence
(string) becomes an iterator:
 chunks ABCDE -> AB CD E
 group ABCDE -> A B C D E -> (A B) (C D) (D E)

From barry at python.org  Thu Jul  5 15:52:49 2012
From: barry at python.org (Barry Warsaw)
Date: Thu, 5 Jul 2012 09:52:49 -0400
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
 fill=None)
In-Reply-To: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
Message-ID: <20120705095249.79ca136c@limelight.wooz.org>

On Jul 05, 2012, at 04:36 PM, anatoly techtonik wrote:

>It would be much easier to do this in unified interface for threaded
>discussions, but for now there is no capabilities for that neither in Mailman
>nor in GMail.

You might like to read the mailing lists via NNTP on Gmane.

Cheers,
-Barry

From techtonik at gmail.com  Thu Jul  5 16:33:19 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Jul 2012 17:33:19 +0300
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <jt225u$iih$1@dough.gmane.org>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org>
	<CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>
	<jt225u$iih$1@dough.gmane.org>
Message-ID: <CAPkN8x+C-z6Xma44Qm4R-KzHid_pZ86eoJFKo7fq=BtjjdNsrg@mail.gmail.com>

On Wed, Jul 4, 2012 at 9:31 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 7/4/2012 5:57 AM, anatoly techtonik wrote:
>>
>> On Fri, Jun 29, 2012 at 11:32 PM, Georg Brandl <g.brandl at gmx.net> wrote:
>
>
>>> Anatoly, so far there were no negative votes -- would you care to go
>>> another step and propose a patch?
>>
>>
>> Was about to say "no problem",
>
>
> Did you read that there *are* strong negative votes? And that this idea has
> been rejected before? I summarized the objections in my two responses and
> pointed to the tracker issues. One of the objections is that there are 4
> different things one might want if the sequence length is not an even
> multiple of the chunk size. Your original 'idea' did not specify.

I actually meant that there is a problem to propose a patch in the
sense of getting checkout, working on a diff, sending it by attaching
to bug tracker as developer guide says.

>> For now the best thing I can do (I don't risk even to mention anything
>> with 3.3) is to copy/paste code from the docs here:
>>
>> from itertools import izip_longest
>> def chunks(iterable, size, fill=None):
>>      """Split an iterable into blocks of fixed-length"""
>>      # chunks('ABCDEFG', 3, 'x') --> ABC DEF Gxx
>>      args = [iter(iterable)] * size
>>      return izip_longest(fillvalue=fill, *args)
>
>
> Python ideas is about Python 3 ideas. Please post Python 3 code.
>
> This is actually a one liner
>
>     return zip_longest(*[iter(iterable)]*size, fillvalue=file)
>
> We don't generally add such to the stdlib.

Can you figure out from the code what this stuff does?
It doesn't give chunks of strings.

>> BTW, this doesn't work as expected (at least for strings). Expected is:
>>    chunks('ABCDEFG', 3, 'x') --> 'ABC' 'DEF' 'Gxx'
>> got:
>>    chunks('ABCDEFG', 3, 'x') --> ('A' 'B' 'C') ('D' 'E' 'F') ('G' 'x' 'x')
>
>
> One of the problems with idea of 'add a chunker' is that there are at least
> a dozen variants that different people want.

That's not the problem. People always want something extra. The
problem that we don't have a real wish distribution. If 1000 people
want chunks and 1 wants groups with exception - we still account these
as equal variants.

Therefore my idea is deliberately limited to "string to chunks" user
story, and SO implementation proposal.

> I discussed the problem of
> return types issue in my responses. I showed how to get the 'expected'
> response above using grouper, but also suggested that it is the wrong basis
> for splitting strings. Repeated slicing make more sense for concrete
> sequence types.
>
> def seqchunk_odd(s, size):
>     # include odd size left over
>     for i in range(0, len(s), size):
>         yield s[i:i+size]
>
> print(list(seqchunk_odd('ABCDEFG', 3)))
> #
> ['ABC', 'DEF', 'G']

Right. That's the top answer on SO that people think should be in
stdlib. Great we are talking about the same thing actually.

> def seqchunk_even(s, size):
>     # only include even chunks
>     for i in range(0, size*(len(s)//size), size):
>         yield s[i:i+size]
>
> print(list(seqchunk_even('ABCDEFG', 3)))
> #
> ['ABC', 'DEF']

This is deducible from seqchunk_odd(s, size)

> def strchunk_fill(s, size, fill):
>     # fill odd chunks
>     q, r = divmod(len(s), size)
>     even = size * q
>     for i in range(0, even, size):
>         yield s[i:i+size]
>     if size != even:
>         yield s[even:] + fill * (size - r)
>
> print(list(strchunk_fill('ABCDEFG', 3, 'x')))
> #
> ['ABC', 'DEF', 'Gxx']

Also deducible from seqchunk_odd(s, size)

> Because the 'fill' value is necessarily a sequence for strings,
> strchunk_fill would only work for lists and tuples if the fill value were
> either required to be given as a tuple or list of length 1 or if it were
> internally converted inside the function. Skipping that for now.
>
> Having written the fill version based on the even version, it is easy to
> select among the three behaviors by modifying the fill version.
>
> def strchunk(s, size, fill=NotImplemented):
>     # fill odd chunks
>     q, r = divmod(len(s), size)
>     even = size * q
>     for i in range(0, even, size):
>         yield s[i:i+size]
>     if size != even and fill is not NotImplemented:
>         yield s[even:] + fill * (size - r)
>
> print(*strchunk('ABCDEFG', 3))
> print(*strchunk('ABCDEFG', 3, ''))
> print(*strchunk('ABCDEFG', 3, 'x'))
> #
> ABC DEF
> ABC DEF G
> ABC DEF Gxx

I now don't even think that fill value is needed as argument.
if len(chunk) < size:
  chunk.extend( [fill] * ( size - len(chunk)) )

> I already described how something similar could be done by checking each
> grouper output tuple for a fill value, but that requires that the fill value
> be a sentinal that could not otherwise appear in the tuple. One could modify
> grouper to fill with a private object() and check the last item of each
> group for that sentinal and act accordingly (delete, truncate, or replace).
> A generic api needs some thought, though.

I just need to chunk strings and sequences. Generic API is too complex
without counting all usecases and iterating over them.

> An issue I did not previously mention is that people sometimes want
> overlapping chunks rather than contiguous disjoint chunks. The slice
> approach trivially adapts to that.
>
> def seqlap(s, size):
>     for i in range(len(s)-size+1):
>         yield s[i:i+size]
>
> print(*seqlap('ABCDEFG', 3))
> #
> ABC BCD CDE DEF EFG
>
> A sliding window for a generic iterable requires a deque or ring buffer
> approach that is quite different from the zip-longest -- grouper approach.

That's why I'd like to drastically reduce the scope of proposal.
itertools doesn't seem to be the best place anymore. How about
sequence method?

   string.chunks(size)  -> ABC DEF G
   list.chunks(size) -> [A,B,C], [C,D,E],[G]

If somebody needs a keyword argument - this can come later without
breaking compatibility.

From steve at pearwood.info  Thu Jul  5 17:57:17 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 06 Jul 2012 01:57:17 +1000
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
 fill=None)
In-Reply-To: <CAPkN8x+C-z6Xma44Qm4R-KzHid_pZ86eoJFKo7fq=BtjjdNsrg@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>	<jsl3cb$knn$1@dough.gmane.org>	<CAPkN8xJShvv1sMHLs9vREBSih1TGzYsK+V68LKe59aHTXmCjHw@mail.gmail.com>	<jt225u$iih$1@dough.gmane.org>
	<CAPkN8x+C-z6Xma44Qm4R-KzHid_pZ86eoJFKo7fq=BtjjdNsrg@mail.gmail.com>
Message-ID: <4FF5B95D.3060208@pearwood.info>

anatoly techtonik wrote:
> On Wed, Jul 4, 2012 at 9:31 PM, Terry Reedy <tjreedy at udel.edu> wrote:

>> A sliding window for a generic iterable requires a deque or ring buffer
>> approach that is quite different from the zip-longest -- grouper approach.
> 
> That's why I'd like to drastically reduce the scope of proposal.
> itertools doesn't seem to be the best place anymore. How about
> sequence method?
> 
>    string.chunks(size)  -> ABC DEF G
>    list.chunks(size) -> [A,B,C], [C,D,E],[G]

-1

This is a fairly trivial problem to solve, and there are many variations on 
it. Many people will not find the default behaviour helpful, and will need to 
write their own. Why complicate the API for all sequence types with this?

I don't believe that we should enshrine one variation as a built-in method, 
without any evidence that it is the most useful or common variation. Even if 
there is one variation far more useful than the others, that doesn't 
necessarily mean we ought to make it a builtin method unless it is a 
fundamental sequence operation, has wide applicability, and is genuinely hard 
to write. I don't believe chunking meets *any* of those criteria, let alone 
all three.

Not every six line function needs to be a builtin.

I believe that splitting a sequence (or a string) into fixed-size chunks is 
more of a programming exercise problem than a genuinely useful tool. That does 
not mean that there is never any real use-cases for splitting into fixed-size 
chunks, only that this is the function that *seems* more useful in theory than 
it turns out in practice.

Compare this with more useful sequence/iteration tools, like (say) zip. You 
can hardly write a hundred lines of code without using zip at least once. But 
I bet you can write tens of thousands of lines of code without needing to 
split sequences into fixed chunks like this.

Besides, the name "chunks" is more general than how you are using it. For 
example, I consider chunking to be splitting a sequence up at a various 
delimiters or separators, not at fixed character positions. E.g. "the third 
word of item two of the fourth line" is a chunk.

This fits more with the non-programming use of the term chunk or chunking, and 
has precedence in Apple's Hypertalk language, which literally allowed you to 
talk about words, items and lines of text, each of which are described as chunks.

This might be a good candidate for a utility module made up of assorted useful 
functions, but not for the string and sequence APIs.



-- 
Steven


From ctb at msu.edu  Thu Jul  5 18:01:26 2012
From: ctb at msu.edu (C. Titus Brown)
Date: Thu, 5 Jul 2012 09:01:26 -0700
Subject: [Python-Dev] GitHub mirror (Was:  Bitbucket mirror?)
In-Reply-To: <20120705124952.GA17866@chang>
References: <CAPkN8xKN5y7LX9uQ=sPfDt-F3FkKYd0jjAJ6+aPaaCDAL8VTqw@mail.gmail.com>
	<20120705124952.GA17866@chang>
Message-ID: <20120705160126.GG31815@idyll.org>

On Thu, Jul 05, 2012 at 03:49:52PM +0300, Petri Lehtinen wrote:
> anatoly techtonik wrote:
> > On the subject. Is there a mirror of CPython on GitHub?
> 
> https://github.com/akheron/cpython
> 
> > changes to repository (and allows anonymous to do this). I've made
> > more than a dozen proposal for fixing docs, because as a matter of
> > fact - filling a bug AND explaining why docs are wrong, why they need
> > to be fixed, what should be added - all of this is a way *much easier*
> > (and less time consuming!) than just fixing them. Unfortunately.
> 
> You won't get any changes in to CPython by creating pull requests. We
> use http://bugs.python.org/ for that, sorry.

Question -- is there a reason to abide by this rule for docs?  That is, if we
could get a sympathetic core dev to look at pull requests for docs as part of 
a streamlined process, would it cause problems?

(What I'm really asking is whether or the bugs.python.org process is
considered critical for potentially minor doc changes and additions.)

thanks,
--titus
-- 
C. Titus Brown, ctb at msu.edu

From steve at pearwood.info  Thu Jul  5 18:09:44 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Fri, 06 Jul 2012 02:09:44 +1000
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
 fill=None)
In-Reply-To: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>	<jsl3cb$knn$1@dough.gmane.org>
	<4FEF6B10.9040409@udel.edu>
	<CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
Message-ID: <4FF5BC48.9080302@pearwood.info>

anatoly techtonik wrote:
>>>> Which is the 33th most voted Python question on SO -
>>>>
>>>> http://stackoverflow.com/questions/312443/how-do-you-split-a-list-into-evenly-sized-chunks-in-python/312464
>> I am curious how you get that number. I do note that there are about 15
>> other Python SO questions that seem to be variations on the theme. There
>> might be more if 'blocks' and 'groups' were searched for.
> 
> It's easy:
> 1. Go http://stackoverflow.com/
> 2. Search [python]
> 3. Click `votes` tab
> 4. Choose `30 per page` at the bottom
> 5. Jump to the second page, there it is 4th from the top:
> http://stackoverflow.com/questions/tagged/python?page=2&sort=votes&pagesize=30

Yes. I don't think this is particularly significant. Have a look at some of 
the questions with roughly the same number of votes:

#26 "How can I remove (chomp) a newline in Python?" 176 votes

#33 "How do you split a list into evenly sized chunks in Python?" 149 votes

#36 "Accessing the index in Python for loops" 144 votes


Being 33rd most voted question doesn't really mean much.


By the way, why is this discussion going to both python-dev and python-ideas?



-- 
Steven

From stefan_ml at behnel.de  Thu Jul  5 18:50:32 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jul 2012 18:50:32 +0200
Subject: [Python-Dev] [Python-ideas] itertools.chunks(iterable, size,
	fill=None)
In-Reply-To: <CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
References: <CAPkN8xKq7sfWQt9P2Vaw6B_ZwqKtNLiGdavPT=Y84yBcV08uEg@mail.gmail.com>
	<jsl3cb$knn$1@dough.gmane.org> <4FEF6B10.9040409@udel.edu>
	<CAPkN8xL3UKBZDotaPWtr3WP+wzPVJ87M7KKKuc-uoU+E+WExpA@mail.gmail.com>
Message-ID: <jt4gko$ahr$1@dough.gmane.org>

anatoly techtonik, 05.07.2012 15:36:
> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
>> From Raymond's first message on http://bugs.python.org/issue6021 , add
>> grouper:
>>
>> "This has been rejected before.
> 
> I quite often see such arguments and I can't stand to repeat that
> these are not arguments. It is good to know, but when people use that
> as a reason to close tickets - that's just disgusting.

The *real* problem is that people keep bringing up topics (and even spell
them out in the bug tracker) without searching for existing discussions
and/or tickets first. That's why those who do such a search (or who know
what they are talking about anyway) close these tickets with the remark
"this has been rejected before", instead of repeating an entire heap of
arguments all over again to feed a discussion that would only lead to the
same result as it did before, often several times before.

Stefan


From martin at v.loewis.de  Thu Jul  5 20:27:02 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Thu, 05 Jul 2012 20:27:02 +0200
Subject: [Python-Dev] GitHub mirror (Was: Bitbucket mirror?)
In-Reply-To: <20120705160126.GG31815@idyll.org>
References: <CAPkN8xKN5y7LX9uQ=sPfDt-F3FkKYd0jjAJ6+aPaaCDAL8VTqw@mail.gmail.com>
	<20120705124952.GA17866@chang> <20120705160126.GG31815@idyll.org>
Message-ID: <20120705202702.Horde.Yh-RBqGZi1VP9dx2H7Nj-nA@webmail.df.eu>

>> You won't get any changes in to CPython by creating pull requests. We
>> use http://bugs.python.org/ for that, sorry.
>
> Question -- is there a reason to abide by this rule for docs?  That is, if we
> could get a sympathetic core dev to look at pull requests for docs as part of
> a streamlined process, would it cause problems?

How do you communicate a "pull request"? On bitbucket, there is a  
"pull request"
UI resulting in a tracker item being generated (and an email being sent), but
hg.python.org doesn't have a notion of pull requests. Of course, you could
use any communication means (email, telephone call, carrier pigeon) to request
a pull from a "sympathetic core dev".

> (What I'm really asking is whether or the bugs.python.org process is
> considered critical for potentially minor doc changes and additions.)

The sympathetic core dev is mostly free to bypass any submission process
initially; commits that bypass established procedures will likely be  
questioned
only after the fact.

In the specific case, I'd be worried to verify that the submitter has provided
a contributor form. That's easy to do in the bug tracker, but difficult to do
in an offline pull request. Of course, for a really minor doc change  
(e.g. typo
fixes), no contrib form is necessary.

Regards,
Martin



From techtonik at gmail.com  Thu Jul  5 21:41:29 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Thu, 5 Jul 2012 22:41:29 +0300
Subject: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
 itertools.chunks(iterable, size, fill=None))
Message-ID: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>

On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> anatoly techtonik, 05.07.2012 15:36:
>> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
>>> From Raymond's first message on http://bugs.python.org/issue6021 , add
>>> grouper:
>>>
>>> "This has been rejected before.
>>
>> I quite often see such arguments and I can't stand to repeat that
>> these are not arguments. It is good to know, but when people use that
>> as a reason to close tickets - that's just disgusting.
>
> The *real* problem is that people keep bringing up topics (and even spell
> them out in the bug tracker) without searching for existing discussions
> and/or tickets first. That's why those who do such a search (or who know
> what they are talking about anyway) close these tickets with the remark
> "this has been rejected before", instead of repeating an entire heap of
> arguments all over again to feed a discussion that would only lead to the
> same result as it did before, often several times before.

Make the bloody FAQ and summarize this stuff? Why waste each others
time? If people don't enjoy repeating themselves over and over - there
is a bloody wiki. What should happen to people to start extracting
gems of knowledge from piles of dusty sheets called list "archives"
for others to admire.

No, it is easier to say "it was already discussed many times", "why
don't you Google yourself", "so far you're only complaining", etc. If
people can't find anything - why everybody thinks they are ignorant
and lazy. Even if it so, why nobody thinks that maybe that bloody
Xapian index is dead again for a bloody amount of moons nobody knows
why and how many exactly? Why nobody thinks that lazy coders can also
help with development? Maybe that laziness is the primary reason some
major groups actually prefer Python to Java, C++ and other more
interesting languages (such as PHP) when it comes to typing? Make it
easy and the patches will follow. Answers like "this was discussed
before" don't make it easy to understand, and leaving users rereading
old 19xx archives that people don't reread themselves will likely make
users bounce and never (NEVER!) come up with some proposal again. An
"organic" way to keep traffic low.

Miscommunication is a bad experience for users, bad experience for
developers, everybody is annoyed and as a result such nice language as
Python loses points on TIOBE (and convenient chunk() functions to
munch-munch on the sequence data).

Wheew. :-F

From breamoreboy at yahoo.co.uk  Thu Jul  5 21:55:09 2012
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Thu, 05 Jul 2012 20:55:09 +0100
Subject: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
 itertools.chunks(iterable, size, fill=None))
In-Reply-To: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>
References: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>
Message-ID: <jt4re5$3gs$1@dough.gmane.org>

On 05/07/2012 20:41, anatoly techtonik wrote:
> On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
>> anatoly techtonik, 05.07.2012 15:36:
>>> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
>>>>  From Raymond's first message on http://bugs.python.org/issue6021 , add
>>>> grouper:
>>>>
>>>> "This has been rejected before.
>>>
>>> I quite often see such arguments and I can't stand to repeat that
>>> these are not arguments. It is good to know, but when people use that
>>> as a reason to close tickets - that's just disgusting.
>>
>> The *real* problem is that people keep bringing up topics (and even spell
>> them out in the bug tracker) without searching for existing discussions
>> and/or tickets first. That's why those who do such a search (or who know
>> what they are talking about anyway) close these tickets with the remark
>> "this has been rejected before", instead of repeating an entire heap of
>> arguments all over again to feed a discussion that would only lead to the
>> same result as it did before, often several times before.
>
> Make the bloody FAQ and summarize this stuff? Why waste each others
> time? If people don't enjoy repeating themselves over and over - there
> is a bloody wiki. What should happen to people to start extracting
> gems of knowledge from piles of dusty sheets called list "archives"
> for others to admire.
>
> No, it is easier to say "it was already discussed many times", "why
> don't you Google yourself", "so far you're only complaining", etc. If
> people can't find anything - why everybody thinks they are ignorant
> and lazy. Even if it so, why nobody thinks that maybe that bloody
> Xapian index is dead again for a bloody amount of moons nobody knows
> why and how many exactly? Why nobody thinks that lazy coders can also
> help with development? Maybe that laziness is the primary reason some
> major groups actually prefer Python to Java, C++ and other more
> interesting languages (such as PHP) when it comes to typing? Make it
> easy and the patches will follow. Answers like "this was discussed
> before" don't make it easy to understand, and leaving users rereading
> old 19xx archives that people don't reread themselves will likely make
> users bounce and never (NEVER!) come up with some proposal again. An
> "organic" way to keep traffic low.
>
> Miscommunication is a bad experience for users, bad experience for
> developers, everybody is annoyed and as a result such nice language as
> Python loses points on TIOBE (and convenient chunk() functions to
> munch-munch on the sequence data).
>
> Wheew. :-F
>

Can I safely assume that you are volunteering to do the work required?

-- 
Cheers.

Mark Lawrence.




From stefan_ml at behnel.de  Thu Jul  5 21:58:52 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Thu, 05 Jul 2012 21:58:52 +0200
Subject: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
 itertools.chunks(iterable, size, fill=None))
In-Reply-To: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>
References: <CAPkN8x+A-OYWNLNKDH=6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA@mail.gmail.com>
Message-ID: <jt4rlt$45k$1@dough.gmane.org>

anatoly techtonik, 05.07.2012 21:41:
> On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel wrote:
>> anatoly techtonik, 05.07.2012 15:36:
>>> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
>>>> From Raymond's first message on http://bugs.python.org/issue6021 , add
>>>> grouper:
>>>>
>>>> "This has been rejected before.
>>>
>>> I quite often see such arguments and I can't stand to repeat that
>>> these are not arguments. It is good to know, but when people use that
>>> as a reason to close tickets - that's just disgusting.
>>
>> The *real* problem is that people keep bringing up topics (and even spell
>> them out in the bug tracker) without searching for existing discussions
>> and/or tickets first. That's why those who do such a search (or who know
>> what they are talking about anyway) close these tickets with the remark
>> "this has been rejected before", instead of repeating an entire heap of
>> arguments all over again to feed a discussion that would only lead to the
>> same result as it did before, often several times before.
> 
> Make the bloody FAQ and summarize this stuff? Why waste each others
> time?

Yes, that is exactly the question.

It takes time to write things up nicely. I mean, once someone has pointed
out to you that this has been discussed before, you could just go, look it
up (or search for it), and then put it into a Wiki or blog post yourself,
or sum it up and send it to the mailing list as a reply. Why rely on others
to do it for you?

Stefan


From paul at boddie.org.uk  Thu Jul  5 23:11:46 2012
From: paul at boddie.org.uk (Paul Boddie)
Date: Thu, 5 Jul 2012 23:11:46 +0200
Subject: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
	itertools.chunks(iterable, size, fill=None))
Message-ID: <201207052311.46867.paul@boddie.org.uk>

Stefan Behnel wrote:
> anatoly techtonik, 05.07.2012 21:41:
> >
> > Make the bloody FAQ and summarize this stuff? Why waste each others
> > time?
>
> Yes, that is exactly the question.
>
> It takes time to write things up nicely. I mean, once someone has pointed
> out to you that this has been discussed before, you could just go, look it
> up (or search for it), and then put it into a Wiki or blog post yourself,
> or sum it up and send it to the mailing list as a reply. Why rely on others
> to do it for you?

To be fair, Anatoly has done quite a bit of maintenance on some of the Wiki 
content around various aspects of the project, so it's not as if he's 
demanding anything out of the ordinary or asking for others to do things that 
he isn't already doing in some sense. My experience is that there usually 
needs to be some willingness on the other end of the transaction, and if it 
takes repetition to encourage it amongst those who don't see the current 
situation as a problem for them, then so be it.

Of course, this kind of documentation activity, where one gathers together 
historical decisions and the consensus from long-forgotten discussions, is 
pretty thankless work. I occasionally regard it as worthwhile if only to 
bring up something someone said as an inconvenient interruption in any 
current discussion, but that's a pretty minimal reward for all the effort 
unless one has such work as part of one's daily routine.

Paul

From larry at hastings.org  Fri Jul  6 08:47:30 2012
From: larry at hastings.org (Larry Hastings)
Date: Fri, 06 Jul 2012 08:47:30 +0200
Subject: [Python-Dev] EuroPython 2012 Language Summit is Canceled.
Message-ID: <4FF68A02.8000500@hastings.org>



I only got one more RSVP and zero topics for the docket.  So let's 
sprint instead.

See you at the PyCon 2013 Language Summit,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120706/f13295aa/attachment.html>

From merwok at netwok.org  Fri Jul  6 16:54:39 2012
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Fri, 06 Jul 2012 10:54:39 -0400
Subject: [Python-Dev] [Python-checkins] peps: specify Provides-Extra
In-Reply-To: <3WTJQY0ZPZzNND@mail.python.org>
References: <3WTJQY0ZPZzNND@mail.python.org>
Message-ID: <4FF6FC2F.2070208@netwok.org>

Le 06/07/2012 10:36, benjamin.peterson a ?crit :
> http://hg.python.org/peps/rev/4dc72bbf9ed2
> changeset:   4483:4dc72bbf9ed2
> user:        Daniel Holth<dholth at fastmail.fm>
> date:        Wed Jul 04 07:30:08 2012 -0400
> summary:
>    specify Provides-Extra

Sorry, but this was premature.  The mail on distutils-sig only describes 
the syntax for a new field without any explanation of the semantics and 
use case, not the relation to the Setup-Requires and Test-Requires 
fields that we?ve been discussing for some time (and which would make 
part of the Extra proposal obsolete).  I know Alexis and I are not as 
active as other devs in mailing lists and PEPs, but these things are on 
our radar and we want to push them forward.  Please allow the discussion 
to take place before editing the packaging PEPs.

Regards

From status at bugs.python.org  Fri Jul  6 18:07:18 2012
From: status at bugs.python.org (Python tracker)
Date: Fri,  6 Jul 2012 18:07:18 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20120706160718.9092F1CD07@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2012-06-29 - 2012-07-06)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    3495 (+10)
  closed 23546 (+31)
  total  27041 (+41)

Open issues with patches: 1462 


Issues opened (32)
==================

#10571: "setup.py upload --sign" broken: TypeError: 'str' does not sup
http://bugs.python.org/issue10571  reopened by eric.araujo

#15223: datetime instances lack __module__ attribute
http://bugs.python.org/issue15223  opened by cjerdonek

#15227: Fatal Python error: PyEval_RestoreThread: NULL tstate on examp
http://bugs.python.org/issue15227  opened by tmaslach

#15228: os.utime() docs not clear on behavior on nonexistant files
http://bugs.python.org/issue15228  opened by dewin

#15230: runpy.run_path broken: Breaks scoping; pollutes sys.modules; d
http://bugs.python.org/issue15230  opened by Benjamin.S.Wolf

#15231: update PyPI upload doc to say --no-raw passed to rst2html.py
http://bugs.python.org/issue15231  opened by cjerdonek

#15232: email.generator.Generator doesn't mangle "From " lines in MIME
http://bugs.python.org/issue15232  opened by Chris.Pickett

#15233: atexit: guarantee order of execution of registered functions?
http://bugs.python.org/issue15233  opened by georg.brandl

#15234: avoid runtime library path for extensions found in system dire
http://bugs.python.org/issue15234  opened by doko

#15235: allow newer berkley db versions
http://bugs.python.org/issue15235  opened by doko

#15237: Add capsule API to _decimal
http://bugs.python.org/issue15237  opened by skrah

#15238: shutil.copystat should copy Linux extended attributes
http://bugs.python.org/issue15238  opened by larry

#15239: Abandoned Tools/unicode/mkstringprep.py
http://bugs.python.org/issue15239  opened by storchaka

#15242: PyImport_GetMagicTag() should use the same const char * as sys
http://bugs.python.org/issue15242  opened by eric.snow

#15243: Misleading documentation for __prepare__
http://bugs.python.org/issue15243  opened by William.Schwartz

#15244: Support for opening files with FILE_SHARE_DELETE on Windows
http://bugs.python.org/issue15244  opened by sbt

#15245: ast.literal_eval fails on some literals
http://bugs.python.org/issue15245  opened by JBernardo

#15246: Line coverage for collectionts.abc.Set
http://bugs.python.org/issue15246  opened by agentultra

#15247: io.open() is inconsistent re os.open()
http://bugs.python.org/issue15247  opened by apalala

#15248: In "TypeError: 'tuple' object is not callable", explain that a
http://bugs.python.org/issue15248  opened by o11c

#15249: email.generator.BytesGenerator doesn't mangle "From " lines wh
http://bugs.python.org/issue15249  opened by petri.lehtinen

#15250: document that filecmp.dircmp comparisons are "shallow"
http://bugs.python.org/issue15250  opened by cjerdonek

#15251: new.code and new.function crashes Python iterpretter
http://bugs.python.org/issue15251  opened by jeffdharper

#15255: spam / garbage report
http://bugs.python.org/issue15255  opened by andisthermal555

#15256: Typo in error message
http://bugs.python.org/issue15256  opened by Marc.Abramowitz

#15257: Misc/.gdbinit:pystack is too brittle
http://bugs.python.org/issue15257  opened by rian

#15258: argparse documentation: Improve optparse section regarding all
http://bugs.python.org/issue15258  opened by nejucomo

#15259: "Helping with Documentation" references missing dailybuild.py
http://bugs.python.org/issue15259  opened by cjerdonek

#15260: Mention how to order Misc/NEWS entries
http://bugs.python.org/issue15260  opened by cjerdonek

#15262: Idle does not show traceback in other threads
http://bugs.python.org/issue15262  opened by Mark

#15263: Guard against invalid file handles in os functions
http://bugs.python.org/issue15263  opened by larry

#1207613: Bottom Scroll Bar
http://bugs.python.org/issue1207613  reopened by terry.reedy



Most recent 15 issues with no replies (15)
==========================================

#15262: Idle does not show traceback in other threads
http://bugs.python.org/issue15262

#15260: Mention how to order Misc/NEWS entries
http://bugs.python.org/issue15260

#15259: "Helping with Documentation" references missing dailybuild.py
http://bugs.python.org/issue15259

#15258: argparse documentation: Improve optparse section regarding all
http://bugs.python.org/issue15258

#15257: Misc/.gdbinit:pystack is too brittle
http://bugs.python.org/issue15257

#15256: Typo in error message
http://bugs.python.org/issue15256

#15250: document that filecmp.dircmp comparisons are "shallow"
http://bugs.python.org/issue15250

#15244: Support for opening files with FILE_SHARE_DELETE on Windows
http://bugs.python.org/issue15244

#15234: avoid runtime library path for extensions found in system dire
http://bugs.python.org/issue15234

#15230: runpy.run_path broken: Breaks scoping; pollutes sys.modules; d
http://bugs.python.org/issue15230

#15228: os.utime() docs not clear on behavior on nonexistant files
http://bugs.python.org/issue15228

#15227: Fatal Python error: PyEval_RestoreThread: NULL tstate on examp
http://bugs.python.org/issue15227

#15201: C argument errors and Python arguments error are different
http://bugs.python.org/issue15201

#15199: Default mimetype for javascript should be application/javascri
http://bugs.python.org/issue15199

#15195: test_distutils fails when ARCHFLAGS is set on a Mac
http://bugs.python.org/issue15195



Most recent 15 issues waiting for review (15)
=============================================

#15257: Misc/.gdbinit:pystack is too brittle
http://bugs.python.org/issue15257

#15250: document that filecmp.dircmp comparisons are "shallow"
http://bugs.python.org/issue15250

#15249: email.generator.BytesGenerator doesn't mangle "From " lines wh
http://bugs.python.org/issue15249

#15246: Line coverage for collectionts.abc.Set
http://bugs.python.org/issue15246

#15245: ast.literal_eval fails on some literals
http://bugs.python.org/issue15245

#15244: Support for opening files with FILE_SHARE_DELETE on Windows
http://bugs.python.org/issue15244

#15242: PyImport_GetMagicTag() should use the same const char * as sys
http://bugs.python.org/issue15242

#15239: Abandoned Tools/unicode/mkstringprep.py
http://bugs.python.org/issue15239

#15238: shutil.copystat should copy Linux extended attributes
http://bugs.python.org/issue15238

#15234: avoid runtime library path for extensions found in system dire
http://bugs.python.org/issue15234

#15233: atexit: guarantee order of execution of registered functions?
http://bugs.python.org/issue15233

#15231: update PyPI upload doc to say --no-raw passed to rst2html.py
http://bugs.python.org/issue15231

#15220: Reduce parsing overhead in email.feedparser.BufferedSubFile
http://bugs.python.org/issue15220

#15209: Re-raising exceptions from an expression
http://bugs.python.org/issue15209

#15207: mimetypes.read_windows_registry() uses the wrong regkey, creat
http://bugs.python.org/issue15207



Top 10 most discussed issues (10)
=================================

#14243: tempfile.NamedTemporaryFile not particularly useful on	Windows
http://bugs.python.org/issue14243  15 msgs

#3754: cross-compilation support for python build
http://bugs.python.org/issue3754  12 msgs

#3871: cross and native build of python for mingw32 with packaging
http://bugs.python.org/issue3871  11 msgs

#15202: followlinks/follow_symlinks/symlinks flags unification
http://bugs.python.org/issue15202  11 msgs

#15233: atexit: guarantee order of execution of registered functions?
http://bugs.python.org/issue15233  11 msgs

#15245: ast.literal_eval fails on some literals
http://bugs.python.org/issue15245  11 msgs

#15180: Cryptic traceback from os.path.join when mixing str & bytes
http://bugs.python.org/issue15180   6 msgs

#15198: multiprocessing Pipe send of non-picklable objects doesn't rai
http://bugs.python.org/issue15198   6 msgs

#1602: windows console doesn't print or input Unicode
http://bugs.python.org/issue1602   5 msgs

#3561: Windows installer should add Python and Scripts directories to
http://bugs.python.org/issue3561   5 msgs



Issues closed (28)
==================

#1677: Ctrl-C will exit out of Python interpreter in Windows
http://bugs.python.org/issue1677  closed by tim.golden

#2345: Using an exception variable outside an 'except' clause should 
http://bugs.python.org/issue2345  closed by brett.cannon

#4485: fast swap of "default" Windows python versions
http://bugs.python.org/issue4485  closed by tim.golden

#9239: zipfile: truncating comment can corrupt the zipfile
http://bugs.python.org/issue9239  closed by pitrou

#14330: don't use host python, use host search paths for host compiler
http://bugs.python.org/issue14330  closed by doko

#14591: Value returned by random.random() out of valid range on 64-bit
http://bugs.python.org/issue14591  closed by python-dev

#14902: test_logging failed
http://bugs.python.org/issue14902  closed by python-dev

#14967: distutils2.utils.resolve_name cannot be implemented to give co
http://bugs.python.org/issue14967  closed by eric.araujo

#15020: default value for progname in pythonrun.c should be python3 fo
http://bugs.python.org/issue15020  closed by pitrou

#15030: PyPycLoader can't read cached .pyc files
http://bugs.python.org/issue15030  closed by brett.cannon

#15033: Different exit status when using -m
http://bugs.python.org/issue15033  closed by orsenthil

#15141: IDLE horizontal scroll bar missing (Win-XPsp3)
http://bugs.python.org/issue15141  closed by terry.reedy

#15166: Implement imp.get_tag() using sys.implementation
http://bugs.python.org/issue15166  closed by brett.cannon

#15174: amd64\python_d.exe -m test fails
http://bugs.python.org/issue15174  closed by skrah

#15210: importlib.__init__ checks for the wrong exception when looking
http://bugs.python.org/issue15210  closed by brett.cannon

#15212: Rename SC_GLOBAL_EXPLICT to SC_GLOBAL_EXPLICIT in compiler mod
http://bugs.python.org/issue15212  closed by pitrou

#15224: Range: Additional Methods (min/max/__and__)
http://bugs.python.org/issue15224  closed by mark.dickinson

#15225: Add negative tests for passing str to hmac.HMAC and hmac.new
http://bugs.python.org/issue15225  closed by pitrou

#15226: max( str ) should be fast with PEP 393
http://bugs.python.org/issue15226  closed by loewis

#15229: stringification of subclasses of OSError can cause crash
http://bugs.python.org/issue15229  closed by pitrou

#15236: SEGFAULT in visit_decref
http://bugs.python.org/issue15236  closed by amaury.forgeotdarc

#15240: ctype Structure keeps reference to function pointers
http://bugs.python.org/issue15240  closed by vpelletier

#15241: venv module and pyvenv script documented, but not python behav
http://bugs.python.org/issue15241  closed by python-dev

#15252: Delivery reports about your e-mail
http://bugs.python.org/issue15252  closed by benjamin.peterson

#15253: xmlrpclib.ServerProxy does not support 2-tuple value for uri p
http://bugs.python.org/issue15253  closed by loewis

#15254: 08 is invalid token in lists.
http://bugs.python.org/issue15254  closed by tim.golden

#15261: os.stat(fd) crashes on Windows if fd does not exist
http://bugs.python.org/issue15261  closed by sbt

#798876: windows sys.path contains nonexistant directory
http://bugs.python.org/issue798876  closed by tim.golden

From g.brandl at gmx.net  Fri Jul  6 21:27:19 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 06 Jul 2012 21:27:19 +0200
Subject: [Python-Dev] GitHub mirror (Was:  Bitbucket mirror?)
In-Reply-To: <CAPkN8xKN5y7LX9uQ=sPfDt-F3FkKYd0jjAJ6+aPaaCDAL8VTqw@mail.gmail.com>
References: <CAPkN8xKN5y7LX9uQ=sPfDt-F3FkKYd0jjAJ6+aPaaCDAL8VTqw@mail.gmail.com>
Message-ID: <jt7e5b$duo$1@dough.gmane.org>

On 04.07.2012 10:42, anatoly techtonik wrote:
> On Fri, Jun 29, 2012 at 6:58 AM, Eli Bendersky <eliben at gmail.com> wrote:
>>
>> The devguide (http://docs.python.org/devguide/committing.html) says:
>>
>> Bitbucket also maintain an up to date clone of the main cpython repository
>> that can be used as the basis for a new clone or patch queue.
>>
>> [the link goes to https://bitbucket.org/mirror/cpython/overview]
>
> This one is better than Atlassian's (looks nice, shorter URL) and
> worked long before Atlassian promotion as "makers of Bitbucket", which
> is dead wrong BTW, because everybody knows they bought BB. I don't
> remember if I commented on the Python Insider or if it was censored.
> Whatever.. it hurts Atlassian, not me.
>
> On the subject. Is there a mirror of CPython on GitHub?
> I abandoned my efforts to create online editor for docs, which should
> have been both web fronted independent, maintainable and clean. It
> appeared much easier to just use GitHub - so far it is the only one
> from open source code hosting services that directly commits online
> changes to repository (and allows anonymous to do this). I've made
> more than a dozen proposal for fixing docs, because as a matter of
> fact - filling a bug AND explaining why docs are wrong, why they need
> to be fixed, what should be added - all of this is a way *much easier*
> (and less time consuming!) than just fixing them. Unfortunately.

Then you didn't read http://docs.python.org/bugs -- an email to docs at python.org
is enough.  If the bug/change is minor, it is usually taken care of right away,
if it is major, we (mostly Sandro -- thanks Sandro -- and I) create a tracker
item for you.

Georg


From g.brandl at gmx.net  Fri Jul  6 21:36:27 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 06 Jul 2012 21:36:27 +0200
Subject: [Python-Dev] cpython (2.7): Make it easier to search for the
	grouper() recipe.
In-Reply-To: <3WRBfX2mWbzN2y@mail.python.org>
References: <3WRBfX2mWbzN2y@mail.python.org>
Message-ID: <jt7eme$duo$3@dough.gmane.org>

On 03.07.2012 06:09, raymond.hettinger wrote:
> http://hg.python.org/cpython/rev/d32f21d87363
> changeset:   77921:d32f21d87363
> branch:      2.7
> parent:      77914:366df7ba1616
> user:        Raymond Hettinger <python at rcn.com>
> date:        Mon Jul 02 21:08:45 2012 -0700
> summary:
>    Make it easier to search for the grouper() recipe.
>
> files:
>    Doc/library/itertools.rst |  3 ++-
>    1 files changed, 2 insertions(+), 1 deletions(-)
>
>
> diff --git a/Doc/library/itertools.rst b/Doc/library/itertools.rst
> --- a/Doc/library/itertools.rst
> +++ b/Doc/library/itertools.rst
> @@ -733,7 +733,8 @@
>          return izip(a, b)
>
>      def grouper(n, iterable, fillvalue=None):
> -       "grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
> +       "Collect data into fixed-length chunks or blocks"
> +       # grouper(3, 'ABCDEFG', 'x') --> ABC DEF Gxx"
>          args = [iter(iterable)] * n
>          return izip_longest(fillvalue=fillvalue, *args)

Will you please apply this also to 3.x?

Georg


From nad at acm.org  Sat Jul  7 03:49:34 2012
From: nad at acm.org (Ned Deily)
Date: Fri, 06 Jul 2012 18:49:34 -0700
Subject: [Python-Dev] Problem with hg hook for devguide repo?
Message-ID: <nad-B27479.18493406072012@news.gmane.org>

pushing to ssh://pyhg/devguide
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 1 changes to 1 files
remote: sent email to roundup at report at bugs.python.org
remote: notified python-checkins at python.org of incoming changeset 
bbe197bf57a1
remote: 1 files updated, 0 files merged, 0 files removed, 0 files 
unresolved
remote: sh: 1: source: not found
remote: sh: 1: sphinx-build: not found
remote: warning: incoming.sphinxbuild hook exited with status 127

-- 
 Ned Deily,
 nad at acm.org


From benjamin at python.org  Sat Jul  7 05:44:44 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 6 Jul 2012 22:44:44 -0500
Subject: [Python-Dev] Problem with hg hook for devguide repo?
In-Reply-To: <nad-B27479.18493406072012@news.gmane.org>
References: <nad-B27479.18493406072012@news.gmane.org>
Message-ID: <CAPZV6o-WNyVOEwrtdJLzMc_9fzj-BukTe+JicniO2KbMpaP6PA@mail.gmail.com>

2012/7/6 Ned Deily <nad at acm.org>:
> pushing to ssh://pyhg/devguide
> searching for changes
> remote: adding changesets
> remote: adding manifests
> remote: adding file changes
> remote: added 1 changesets with 1 changes to 1 files
> remote: sent email to roundup at report at bugs.python.org
> remote: notified python-checkins at python.org of incoming changeset
> bbe197bf57a1
> remote: 1 files updated, 0 files merged, 0 files removed, 0 files
> unresolved
> remote: sh: 1: source: not found
> remote: sh: 1: sphinx-build: not found
> remote: warning: incoming.sphinxbuild hook exited with status 127

Yes, this is because hg was migrated to OSUL. This is being discussed
on the infrastructure list.


-- 
Regards,
Benjamin

From solipsis at pitrou.net  Sat Jul  7 15:55:35 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 7 Jul 2012 15:55:35 +0200
Subject: [Python-Dev] cpython: Issue 14814: Correctly return
 NotImplemented from ipaddress._BaseNetwork.__eq__
References: <3WTtRG6h3vzN4C@mail.python.org>
Message-ID: <20120707155535.1f7c4778@pitrou.net>

On Sat,  7 Jul 2012 15:08:42 +0200 (CEST)
nick.coghlan <python-checkins at python.org> wrote:
>  
>      def __eq__(self, other):
> -        if not isinstance(other, _BaseNetwork):
> -            raise TypeError('%s and %s are not of the same type' % (
> -                             self, other))
> -        return (self._version == other._version and
> -                self.network_address == other.network_address and
> -                int(self.netmask) == int(other.netmask))
> +        try:
> +            return (self._version == other._version and
> +                    self.network_address == other.network_address and
> +                    int(self.netmask) == int(other.netmask))
> +        except AttributeError:
> +            return NotImplemented

I think the isinstance() test was correct. If you have an object which
duck-types IPNetwork, you probably want its __eq__ to be called, not
yours.

Regards

Antoine.



From ncoghlan at gmail.com  Sat Jul  7 16:59:51 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 8 Jul 2012 00:59:51 +1000
Subject: [Python-Dev] cpython: Issue 14814: Correctly return
 NotImplemented from ipaddress._BaseNetwork.__eq__
In-Reply-To: <20120707155535.1f7c4778@pitrou.net>
References: <3WTtRG6h3vzN4C@mail.python.org>
	<20120707155535.1f7c4778@pitrou.net>
Message-ID: <CADiSq7fVVSUnPJJQO=FORFswv5d=eb-+O5a4pXf1OsVB9+hQXQ@mail.gmail.com>

On Sat, Jul 7, 2012 at 11:55 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> I think the isinstance() test was correct. If you have an object which
> duck-types IPNetwork, you probably want its __eq__ to be called, not
> yours.

This change was just to bring IPNetwork in line with the other types
in the module and to stop it throwing TypeError itself, which meant
the RHS was never even getting a chance to affect the result.

Ducktyping and operator overloading has always been a tricky area
though. If you use isinstance() checks, then the other side has to
know how to reimplement your equality check, or temporarily create a
real instance to do the comparison. If you use ducktyping internally,
then the other side *has* to use inheritance if they want to
completely control the result, but also have the option to just expose
the appropriate attributes in order to interoperate with your class.

The standard library tends to be a mixture of both based on how
integral the author feels the ordering and comparison behaviour is to
the classes involved. In this case, I currently think internal
ducktyping is the right answer, but I'm open to being persuaded
otherwise.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Jul  7 17:18:22 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 8 Jul 2012 01:18:22 +1000
Subject: [Python-Dev] cpython: Issue 14814: Correctly return
 NotImplemented from ipaddress._BaseNetwork.__eq__
In-Reply-To: <CADiSq7fVVSUnPJJQO=FORFswv5d=eb-+O5a4pXf1OsVB9+hQXQ@mail.gmail.com>
References: <3WTtRG6h3vzN4C@mail.python.org>
	<20120707155535.1f7c4778@pitrou.net>
	<CADiSq7fVVSUnPJJQO=FORFswv5d=eb-+O5a4pXf1OsVB9+hQXQ@mail.gmail.com>
Message-ID: <CADiSq7c4_m5Y9EEUhfRguMZ7GUVzT67o6wXBZ9z-eQbH2gW+Wg@mail.gmail.com>

On Sun, Jul 8, 2012 at 12:59 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sat, Jul 7, 2012 at 11:55 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> I think the isinstance() test was correct. If you have an object which
>> duck-types IPNetwork, you probably want its __eq__ to be called, not
>> yours.
>
> This change was just to bring IPNetwork in line with the other types
> in the module and to stop it throwing TypeError itself, which meant
> the RHS was never even getting a chance to affect the result.

Hmm, I just noticed the __lt__ implementations still throw TypeError
directly (at least in the IPNetwork case). Looks like some more
cleanups are still needed in this area :P

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Sun Jul  8 02:27:46 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 8 Jul 2012 02:27:46 +0200
Subject: [Python-Dev] cpython (3.2): Fix issue14826 - make
 urllib.request.Request quoted url consistent with
References: <3WV9D85ygYzNdL@mail.python.org>
Message-ID: <20120708022746.041e64c7@pitrou.net>

>  
> +    def test_quote_url(self):
> +        Request = urllib.request.Request
> +        request = Request("http://www.python.org/foo bar")
> +        self.assertEqual(request.full_url, "http://www.python.org/foo%20bar")
> +

This means someone who already quotes URL will get a double escaping,
right?

Regards

Antoine.



From g.brandl at gmx.net  Sun Jul  8 10:12:37 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 08 Jul 2012 10:12:37 +0200
Subject: [Python-Dev] cpython: Fix 5931 - Python runtime hardcoded in
 wsgiref.simple_server - Now it specifies
In-Reply-To: <3WV5Xh3VfXzNcF@mail.python.org>
References: <3WV5Xh3VfXzNcF@mail.python.org>
Message-ID: <jtbfc8$opc$1@dough.gmane.org>

On 07.07.2012 23:29, senthil.kumaran wrote:
> http://hg.python.org/cpython/rev/34e705fa4da4
> changeset:   77983:34e705fa4da4
> user:        Senthil Kumaran <senthil at uthcode.com>
> date:        Sat Jul 07 14:29:58 2012 -0700
> summary:
>    Fix 5931 - Python runtime hardcoded in wsgiref.simple_server - Now it specifies an implementation specific term.
>
> files:
>    Doc/library/wsgiref.rst      |   5 +++++
>    Lib/test/test_wsgiref.py     |  10 ++++++++--
>    Lib/wsgiref/simple_server.py |   3 ++-
>    Misc/NEWS                    |   3 +++
>    4 files changed, 18 insertions(+), 3 deletions(-)
>
>
> diff --git a/Doc/library/wsgiref.rst b/Doc/library/wsgiref.rst
> --- a/Doc/library/wsgiref.rst
> +++ b/Doc/library/wsgiref.rst
> @@ -609,6 +609,11 @@
>         as :class:`BaseCGIHandler` and :class:`CGIHandler`) that are not HTTP origin
>         servers.
>
> +      .. versionchanged:: 3.3
> +
> +      The term "Python" is replaced with implementation specific term like
> +      "CPython", "Jython" etc.

Please fix the markup: no blank line, but indented. E.g.

.. versionchanged:: 3.3
    Blah, blah.

Georg


From senthil at uthcode.com  Sun Jul  8 10:34:19 2012
From: senthil at uthcode.com (Senthil Kumaran)
Date: Sun, 8 Jul 2012 01:34:19 -0700
Subject: [Python-Dev] cpython: Fix 5931 - Python runtime hardcoded in
 wsgiref.simple_server - Now it specifies
In-Reply-To: <jtbfc8$opc$1@dough.gmane.org>
References: <3WV5Xh3VfXzNcF@mail.python.org>
	<jtbfc8$opc$1@dough.gmane.org>
Message-ID: <CAPOVWOSAmFE5vnAmdKqXG1qtAKXj8hwY+C60PRQbhFtU8ekBxA@mail.gmail.com>

On Sun, Jul 8, 2012 at 1:12 AM, Georg Brandl <g.brandl at gmx.net> wrote:
>
> Please fix the markup: no blank line, but indented. E.g.
>
> .. versionchanged:: 3.3
>    Blah, blah.

Done. Sorry for that, I only built the docs and saw if the directive
was affected and assumed it okay.

Fixed that.

Thanks,
Senthil

From ncoghlan at gmail.com  Mon Jul  9 04:13:24 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 9 Jul 2012 12:13:24 +1000
Subject: [Python-Dev] [Python-checkins] cpython: Issue 14814: The new
 systematic tests aren't just about error reporting any
In-Reply-To: <4FF9C1E3.2000501@udel.edu>
References: <3WVVWD22TgzNt6@mail.python.org>
	<4FF9C1E3.2000501@udel.edu>
Message-ID: <CADiSq7eUsG34rKV-qTcq7AnVNbT=ejEKwaHRU7Tv7SM6=E-B+A@mail.gmail.com>

On Mon, Jul 9, 2012 at 3:22 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 7/8/2012 9:14 AM, nick.coghlan wrote:
> If you want to report both errors, when present:

I don't really - when there are multiple things wrong with the
address, ipaddress just reports the first one it notices (e.g.
IPv4Address("google.com") will complain about the lack of "."
separators rather than the fact that "google" and "com" are not valid
octets).

This was just a case where I started to reorder the checks to do the
faster one first, and then realised it was more user friendly to check
the other way around (since the success case will always include both
checks, it only makes a difference to the speed of rejection of some
invalid cases). Most of the other validation checks have a more
obvious suitable order (for example, there's no reason to start
checking individual parts if we can't even split the parts up
appropriately).

>> +class BaseTestCase(unittest.TestCase):
>>       # One big change in ipaddress over the original ipaddr module is
>>       # error reporting that tries to assume users *don't know the rules*
>>       # for what constitutes an RFC compliant IP address
>
>
> Good assumption. There *will* be people who learn by trial and error.

*cough* It's conceivable I started down this path due to using the
REPL for exploration while making an initial attempt at writing method
and data attribute docs over the weekend. I'm firmly in the category
of IPv4 network programming engineers with next to no IPv6 experience
:)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Mon Jul  9 11:14:15 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 9 Jul 2012 11:14:15 +0200
Subject: [Python-Dev] cpython: Issue #15283: Updated pyvenv
 documentation to expand on activation.
References: <3WW0466grWzMhd@mail.python.org>
Message-ID: <20120709111415.596e0047@pitrou.net>

On Mon,  9 Jul 2012 10:25:58 +0200 (CEST)
vinay.sajip <python-checkins at python.org> wrote:
> +
> +   Common installation tools such as ``distribute`` and ``pip`` work as
> +   expected with venvs - i.e. when a venv is active, they install Python
> +   packages into the venv without needing to be told to do so explicitly.

Not exactly. They will do so if you *install* them in the venv. A
system-wide distribute or pip, AFAIK, will still install packages into
the system-wide locations.

Also, I wonder why you are duplicating the exact same docs in two files.
That's a maintenance burden.

Regards

Antoine.



From g.brandl at gmx.net  Mon Jul  9 12:21:29 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 09 Jul 2012 12:21:29 +0200
Subject: [Python-Dev] cpython: Issue 15265: document the exception
 raised for invalid sample sizes.
In-Reply-To: <3WVlYQ41nmzNLp@mail.python.org>
References: <3WVlYQ41nmzNLp@mail.python.org>
Message-ID: <jteb9s$gj0$1@dough.gmane.org>

On 09.07.2012 01:02, raymond.hettinger wrote:
> http://hg.python.org/cpython/rev/72174d8af3ba
> changeset:   78015:72174d8af3ba
> user:        Raymond Hettinger <python at rcn.com>
> date:        Sun Jul 08 16:01:53 2012 -0700
> summary:
>    Issue 15265: document the exception raised for invalid sample sizes.
>
> files:
>    Doc/library/random.rst |  3 +++
>    1 files changed, 3 insertions(+), 0 deletions(-)
>
>
> diff --git a/Doc/library/random.rst b/Doc/library/random.rst
> --- a/Doc/library/random.rst
> +++ b/Doc/library/random.rst
> @@ -150,6 +150,9 @@
>      argument.  This is especially fast and space efficient for sampling from a large
>      population:  ``sample(range(10000000), 60)``.
>
> +   If the sample size is larger than the population size, a :exc:``ValueError``

Should be :exc:`ValueError` (one set of quotes only).

Georg


From msabramo at gmail.com  Tue Jul 10 18:41:50 2012
From: msabramo at gmail.com (Marc Abramowitz)
Date: Tue, 10 Jul 2012 09:41:50 -0700
Subject: [Python-Dev] Testing 3.3 framework builds on OS X
Message-ID: <CAAgNOZ31o0Rpm8Uby14BzeTXrJ4TT2LFi0_sg8dNsgB+H9oBJQ@mail.gmail.com>

Anyone know how to build and run tests on an OS X build with
?enable-framework, without installing the framework? I want to test
that something works in my tree without polluting /Library/Frameworks
with stuff that could mess up future tests.

I've been mucking around with DYLD_FRAMEWORK_PATH but not having much luck.

If I build with ./configure --enable-framework --with-dtrace && make
and use DYLD_FRAMEWORK_PATH=. then I suspect it's still loading from
/Library/Frameworks, because my tests fail in a way that suggests it's
not finding the new bits in my working directory.

If I build with ./configure --enable-framework=. --with-dtrace && make
and use DYLD_FRAMEWORK_PATH=., then it fails with:

dyld: Library not loaded: ./Python.framework/Versions/3.3/Python

presumably because ./Python.framework doesn't have any .dylib's or .so's.

Thanks,
Marc

From nad at acm.org  Tue Jul 10 19:27:31 2012
From: nad at acm.org (Ned Deily)
Date: Tue, 10 Jul 2012 10:27:31 -0700
Subject: [Python-Dev] Testing 3.3 framework builds on OS X
References: <CAAgNOZ31o0Rpm8Uby14BzeTXrJ4TT2LFi0_sg8dNsgB+H9oBJQ@mail.gmail.com>
Message-ID: <nad-EC34FD.10273110072012@news.gmane.org>

In article 
<CAAgNOZ31o0Rpm8Uby14BzeTXrJ4TT2LFi0_sg8dNsgB+H9oBJQ at mail.gmail.com>,
 Marc Abramowitz <msabramo at gmail.com> wrote:
> Anyone know how to build and run tests on an OS X build with
> ?enable-framework, without installing the framework? I want to test
> that something works in my tree without polluting /Library/Frameworks
> with stuff that could mess up future tests.
> 
> I've been mucking around with DYLD_FRAMEWORK_PATH but not having much luck.
> 
> If I build with ./configure --enable-framework --with-dtrace && make
> and use DYLD_FRAMEWORK_PATH=. then I suspect it's still loading from
> /Library/Frameworks, because my tests fail in a way that suggests it's
> not finding the new bits in my working directory.
> 
> If I build with ./configure --enable-framework=. --with-dtrace && make
> and use DYLD_FRAMEWORK_PATH=., then it fails with:
> 
> dyld: Library not loaded: ./Python.framework/Versions/3.3/Python
> 
> presumably because ./Python.framework doesn't have any .dylib's or .so's.

Read the updated Mac/README for more information about framework builds.   
Here's a sample configure I use to minimize the chances of pollution 
from other installed Pythons:

BASEDIR="$(dirname $(pwd -P))"
[ -n "${BASEDIR}" ] && ./configure --enable-universalsdk 
--with-universal-archs=intel 
--enable-framework="${BASEDIR}"/root/Library/Frameworks 
--with-framework-name=pytest_10_7 --with-pydebug && rm -rf 
"${BASEDIR}"/root/* || echo "ERROR: configure failed"

This is for 3.3 on 10.7.  For 3.2 and 2.7 at the moment you also need to 
explicitly set MACOSX_DEPLOYMENT_TARGET=10.7 and CC=clang.

The framework path probably needs to be an absolute path, BTW.

-- 
 Ned Deily,
 nad at acm.org


From nad at acm.org  Tue Jul 10 19:35:50 2012
From: nad at acm.org (Ned Deily)
Date: Tue, 10 Jul 2012 10:35:50 -0700
Subject: [Python-Dev] Testing 3.3 framework builds on OS X
References: <CAAgNOZ31o0Rpm8Uby14BzeTXrJ4TT2LFi0_sg8dNsgB+H9oBJQ@mail.gmail.com>
	<nad-EC34FD.10273110072012@news.gmane.org>
Message-ID: <nad-B5125B.10355010072012@news.gmane.org>

In article <nad-EC34FD.10273110072012 at news.gmane.org>,
 Ned Deily <nad at acm.org> wrote:
> Read the updated Mac/README for more information about framework builds.   
> Here's a sample configure I use to minimize the chances of pollution 
> from other installed Pythons:
> 
> BASEDIR="$(dirname $(pwd -P))"
> [ -n "${BASEDIR}" ] && ./configure --enable-universalsdk 
> --with-universal-archs=intel 
> --enable-framework="${BASEDIR}"/root/Library/Frameworks 
> --with-framework-name=pytest_10_7 --with-pydebug && rm -rf 
> "${BASEDIR}"/root/* || echo "ERROR: configure failed"

This does require that you run "make install" but everything is safely 
installed at ../root, including Library/Frameworks, usr/bin, and 
Applications directories.

-- 
 Ned Deily,
 nad at acm.org


From ryanpaullin at gmail.com  Tue Jul 10 23:04:07 2012
From: ryanpaullin at gmail.com (Ryan Paullin)
Date: Tue, 10 Jul 2012 14:04:07 -0700
Subject: [Python-Dev] Python-Dev Digest, Vol 108, Issue 7
In-Reply-To: <mailman.67.1341568803.26719.python-dev@python.org>
References: <mailman.67.1341568803.26719.python-dev@python.org>
Message-ID: <CANzqOBSTKaigO3DM38kEjden2pbS8b3EcYqR_4=rtCc6bdxNeA@mail.gmail.com>

thanks for the reply hastings ive been working on a loopback interface its
done

On Fri, Jul 6, 2012 at 3:00 AM, <python-dev-request at python.org> wrote:

> Send Python-Dev mailing list submissions to
>         python-dev at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mail.python.org/mailman/listinfo/python-dev
> or, via email, send a message with subject or body 'help' to
>         python-dev-request at python.org
>
> You can reach the person managing the list at
>         python-dev-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Python-Dev digest..."
>
>
> Today's Topics:
>
>    1. Re: GitHub mirror (Was: Bitbucket mirror?) (martin at v.loewis.de)
>    2. Bloody FAQ (Was: [Python-ideas] itertools.chunks(iterable,
>       size, fill=None)) (anatoly techtonik)
>    3. Re: Bloody FAQ (Was: [Python-ideas]
>       itertools.chunks(iterable, size, fill=None)) (Mark Lawrence)
>    4. Re: Bloody FAQ (Was: [Python-ideas]
>       itertools.chunks(iterable, size, fill=None)) (Stefan Behnel)
>    5. Re: Bloody FAQ (Was: [Python-ideas]
>       itertools.chunks(iterable, size, fill=None)) (Paul Boddie)
>    6. EuroPython 2012 Language Summit is Canceled. (Larry Hastings)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Thu, 05 Jul 2012 20:27:02 +0200
> From: martin at v.loewis.de
> To: python-dev at python.org
> Subject: Re: [Python-Dev] GitHub mirror (Was: Bitbucket mirror?)
> Message-ID:
>         <20120705202702.Horde.Yh-RBqGZi1VP9dx2H7Nj-nA at webmail.df.eu>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; DelSp=Yes
>
> >> You won't get any changes in to CPython by creating pull requests. We
> >> use http://bugs.python.org/ for that, sorry.
> >
> > Question -- is there a reason to abide by this rule for docs?  That is,
> if we
> > could get a sympathetic core dev to look at pull requests for docs as
> part of
> > a streamlined process, would it cause problems?
>
> How do you communicate a "pull request"? On bitbucket, there is a
> "pull request"
> UI resulting in a tracker item being generated (and an email being sent),
> but
> hg.python.org doesn't have a notion of pull requests. Of course, you could
> use any communication means (email, telephone call, carrier pigeon) to
> request
> a pull from a "sympathetic core dev".
>
> > (What I'm really asking is whether or the bugs.python.org process is
> > considered critical for potentially minor doc changes and additions.)
>
> The sympathetic core dev is mostly free to bypass any submission process
> initially; commits that bypass established procedures will likely be
> questioned
> only after the fact.
>
> In the specific case, I'd be worried to verify that the submitter has
> provided
> a contributor form. That's easy to do in the bug tracker, but difficult to
> do
> in an offline pull request. Of course, for a really minor doc change
> (e.g. typo
> fixes), no contrib form is necessary.
>
> Regards,
> Martin
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Thu, 5 Jul 2012 22:41:29 +0300
> From: anatoly techtonik <techtonik at gmail.com>
> To: Stefan Behnel <stefan_ml at behnel.de>
> Cc: python-ideas at python.org, python-dev at python.org
> Subject: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
>         itertools.chunks(iterable, size, fill=None))
> Message-ID:
>         <CAPkN8x+A-OYWNLNKDH=
> 6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA at mail.gmail.com>
> Content-Type: text/plain; charset=UTF-8
>
> On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> > anatoly techtonik, 05.07.2012 15:36:
> >> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
> >>> From Raymond's first message on http://bugs.python.org/issue6021 , add
> >>> grouper:
> >>>
> >>> "This has been rejected before.
> >>
> >> I quite often see such arguments and I can't stand to repeat that
> >> these are not arguments. It is good to know, but when people use that
> >> as a reason to close tickets - that's just disgusting.
> >
> > The *real* problem is that people keep bringing up topics (and even spell
> > them out in the bug tracker) without searching for existing discussions
> > and/or tickets first. That's why those who do such a search (or who know
> > what they are talking about anyway) close these tickets with the remark
> > "this has been rejected before", instead of repeating an entire heap of
> > arguments all over again to feed a discussion that would only lead to the
> > same result as it did before, often several times before.
>
> Make the bloody FAQ and summarize this stuff? Why waste each others
> time? If people don't enjoy repeating themselves over and over - there
> is a bloody wiki. What should happen to people to start extracting
> gems of knowledge from piles of dusty sheets called list "archives"
> for others to admire.
>
> No, it is easier to say "it was already discussed many times", "why
> don't you Google yourself", "so far you're only complaining", etc. If
> people can't find anything - why everybody thinks they are ignorant
> and lazy. Even if it so, why nobody thinks that maybe that bloody
> Xapian index is dead again for a bloody amount of moons nobody knows
> why and how many exactly? Why nobody thinks that lazy coders can also
> help with development? Maybe that laziness is the primary reason some
> major groups actually prefer Python to Java, C++ and other more
> interesting languages (such as PHP) when it comes to typing? Make it
> easy and the patches will follow. Answers like "this was discussed
> before" don't make it easy to understand, and leaving users rereading
> old 19xx archives that people don't reread themselves will likely make
> users bounce and never (NEVER!) come up with some proposal again. An
> "organic" way to keep traffic low.
>
> Miscommunication is a bad experience for users, bad experience for
> developers, everybody is annoyed and as a result such nice language as
> Python loses points on TIOBE (and convenient chunk() functions to
> munch-munch on the sequence data).
>
> Wheew. :-F
>
>
> ------------------------------
>
> Message: 3
> Date: Thu, 05 Jul 2012 20:55:09 +0100
> From: Mark Lawrence <breamoreboy at yahoo.co.uk>
> To: python-dev at python.org
> Cc: python-ideas at python.org
> Subject: Re: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
>         itertools.chunks(iterable, size, fill=None))
> Message-ID: <jt4re5$3gs$1 at dough.gmane.org>
> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> On 05/07/2012 20:41, anatoly techtonik wrote:
> > On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel <stefan_ml at behnel.de>
> wrote:
> >> anatoly techtonik, 05.07.2012 15:36:
> >>> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
> >>>>  From Raymond's first message on http://bugs.python.org/issue6021 ,
> add
> >>>> grouper:
> >>>>
> >>>> "This has been rejected before.
> >>>
> >>> I quite often see such arguments and I can't stand to repeat that
> >>> these are not arguments. It is good to know, but when people use that
> >>> as a reason to close tickets - that's just disgusting.
> >>
> >> The *real* problem is that people keep bringing up topics (and even
> spell
> >> them out in the bug tracker) without searching for existing discussions
> >> and/or tickets first. That's why those who do such a search (or who know
> >> what they are talking about anyway) close these tickets with the remark
> >> "this has been rejected before", instead of repeating an entire heap of
> >> arguments all over again to feed a discussion that would only lead to
> the
> >> same result as it did before, often several times before.
> >
> > Make the bloody FAQ and summarize this stuff? Why waste each others
> > time? If people don't enjoy repeating themselves over and over - there
> > is a bloody wiki. What should happen to people to start extracting
> > gems of knowledge from piles of dusty sheets called list "archives"
> > for others to admire.
> >
> > No, it is easier to say "it was already discussed many times", "why
> > don't you Google yourself", "so far you're only complaining", etc. If
> > people can't find anything - why everybody thinks they are ignorant
> > and lazy. Even if it so, why nobody thinks that maybe that bloody
> > Xapian index is dead again for a bloody amount of moons nobody knows
> > why and how many exactly? Why nobody thinks that lazy coders can also
> > help with development? Maybe that laziness is the primary reason some
> > major groups actually prefer Python to Java, C++ and other more
> > interesting languages (such as PHP) when it comes to typing? Make it
> > easy and the patches will follow. Answers like "this was discussed
> > before" don't make it easy to understand, and leaving users rereading
> > old 19xx archives that people don't reread themselves will likely make
> > users bounce and never (NEVER!) come up with some proposal again. An
> > "organic" way to keep traffic low.
> >
> > Miscommunication is a bad experience for users, bad experience for
> > developers, everybody is annoyed and as a result such nice language as
> > Python loses points on TIOBE (and convenient chunk() functions to
> > munch-munch on the sequence data).
> >
> > Wheew. :-F
> >
>
> Can I safely assume that you are volunteering to do the work required?
>
> --
> Cheers.
>
> Mark Lawrence.
>
>
>
>
>
> ------------------------------
>
> Message: 4
> Date: Thu, 05 Jul 2012 21:58:52 +0200
> From: Stefan Behnel <stefan_ml at behnel.de>
> To: python-dev at python.org
> Cc: python-ideas at python.org
> Subject: Re: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
>         itertools.chunks(iterable, size, fill=None))
> Message-ID: <jt4rlt$45k$1 at dough.gmane.org>
> Content-Type: text/plain; charset=UTF-8
>
> anatoly techtonik, 05.07.2012 21:41:
> > On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel wrote:
> >> anatoly techtonik, 05.07.2012 15:36:
> >>> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
> >>>> From Raymond's first message on http://bugs.python.org/issue6021 ,
> add
> >>>> grouper:
> >>>>
> >>>> "This has been rejected before.
> >>>
> >>> I quite often see such arguments and I can't stand to repeat that
> >>> these are not arguments. It is good to know, but when people use that
> >>> as a reason to close tickets - that's just disgusting.
> >>
> >> The *real* problem is that people keep bringing up topics (and even
> spell
> >> them out in the bug tracker) without searching for existing discussions
> >> and/or tickets first. That's why those who do such a search (or who know
> >> what they are talking about anyway) close these tickets with the remark
> >> "this has been rejected before", instead of repeating an entire heap of
> >> arguments all over again to feed a discussion that would only lead to
> the
> >> same result as it did before, often several times before.
> >
> > Make the bloody FAQ and summarize this stuff? Why waste each others
> > time?
>
> Yes, that is exactly the question.
>
> It takes time to write things up nicely. I mean, once someone has pointed
> out to you that this has been discussed before, you could just go, look it
> up (or search for it), and then put it into a Wiki or blog post yourself,
> or sum it up and send it to the mailing list as a reply. Why rely on others
> to do it for you?
>
> Stefan
>
>
>
> ------------------------------
>
> Message: 5
> Date: Thu, 5 Jul 2012 23:11:46 +0200
> From: Paul Boddie <paul at boddie.org.uk>
> To: python-dev at python.org
> Subject: Re: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
>         itertools.chunks(iterable, size, fill=None))
> Message-ID: <201207052311.46867.paul at boddie.org.uk>
> Content-Type: text/plain;  charset="utf-8"
>
> Stefan Behnel wrote:
> > anatoly techtonik, 05.07.2012 21:41:
> > >
> > > Make the bloody FAQ and summarize this stuff? Why waste each others
> > > time?
> >
> > Yes, that is exactly the question.
> >
> > It takes time to write things up nicely. I mean, once someone has pointed
> > out to you that this has been discussed before, you could just go, look
> it
> > up (or search for it), and then put it into a Wiki or blog post yourself,
> > or sum it up and send it to the mailing list as a reply. Why rely on
> others
> > to do it for you?
>
> To be fair, Anatoly has done quite a bit of maintenance on some of the Wiki
> content around various aspects of the project, so it's not as if he's
> demanding anything out of the ordinary or asking for others to do things
> that
> he isn't already doing in some sense. My experience is that there usually
> needs to be some willingness on the other end of the transaction, and if it
> takes repetition to encourage it amongst those who don't see the current
> situation as a problem for them, then so be it.
>
> Of course, this kind of documentation activity, where one gathers together
> historical decisions and the consensus from long-forgotten discussions, is
> pretty thankless work. I occasionally regard it as worthwhile if only to
> bring up something someone said as an inconvenient interruption in any
> current discussion, but that's a pretty minimal reward for all the effort
> unless one has such work as part of one's daily routine.
>
> Paul
>
>
> ------------------------------
>
> Message: 6
> Date: Fri, 06 Jul 2012 08:47:30 +0200
> From: Larry Hastings <larry at hastings.org>
> To: python-dev at python.org, python-committers at python.org
> Subject: [Python-Dev] EuroPython 2012 Language Summit is Canceled.
> Message-ID: <4FF68A02.8000500 at hastings.org>
> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
>
>
>
> I only got one more RSVP and zero topics for the docket.  So let's
> sprint instead.
>
> See you at the PyCon 2013 Language Summit,
>
>
> //arry/
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.python.org/pipermail/python-dev/attachments/20120706/f13295aa/attachment-0001.html
> >
>
> ------------------------------
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
>
>
> End of Python-Dev Digest, Vol 108, Issue 7
> ******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120710/4dbfe67a/attachment.html>

From ryanpaullin at gmail.com  Tue Jul 10 23:04:24 2012
From: ryanpaullin at gmail.com (Ryan Paullin)
Date: Tue, 10 Jul 2012 14:04:24 -0700
Subject: [Python-Dev] Python-Dev Digest, Vol 108, Issue 7
In-Reply-To: <CANzqOBSTKaigO3DM38kEjden2pbS8b3EcYqR_4=rtCc6bdxNeA@mail.gmail.com>
References: <mailman.67.1341568803.26719.python-dev@python.org>
	<CANzqOBSTKaigO3DM38kEjden2pbS8b3EcYqR_4=rtCc6bdxNeA@mail.gmail.com>
Message-ID: <CANzqOBT9ZUJ1Os78q82iFsEqb2C4fVOfea+g__FLuroXZSJSqw@mail.gmail.com>

spoke too early on its done sorry

On Tue, Jul 10, 2012 at 2:04 PM, Ryan Paullin <ryanpaullin at gmail.com> wrote:

> thanks for the reply hastings ive been working on a loopback interface its
> done
>
>
> On Fri, Jul 6, 2012 at 3:00 AM, <python-dev-request at python.org> wrote:
>
>> Send Python-Dev mailing list submissions to
>>         python-dev at python.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         http://mail.python.org/mailman/listinfo/python-dev
>> or, via email, send a message with subject or body 'help' to
>>         python-dev-request at python.org
>>
>> You can reach the person managing the list at
>>         python-dev-owner at python.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Python-Dev digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Re: GitHub mirror (Was: Bitbucket mirror?) (martin at v.loewis.de)
>>    2. Bloody FAQ (Was: [Python-ideas] itertools.chunks(iterable,
>>       size, fill=None)) (anatoly techtonik)
>>    3. Re: Bloody FAQ (Was: [Python-ideas]
>>       itertools.chunks(iterable, size, fill=None)) (Mark Lawrence)
>>    4. Re: Bloody FAQ (Was: [Python-ideas]
>>       itertools.chunks(iterable, size, fill=None)) (Stefan Behnel)
>>    5. Re: Bloody FAQ (Was: [Python-ideas]
>>       itertools.chunks(iterable, size, fill=None)) (Paul Boddie)
>>    6. EuroPython 2012 Language Summit is Canceled. (Larry Hastings)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Thu, 05 Jul 2012 20:27:02 +0200
>> From: martin at v.loewis.de
>> To: python-dev at python.org
>> Subject: Re: [Python-Dev] GitHub mirror (Was: Bitbucket mirror?)
>> Message-ID:
>>         <20120705202702.Horde.Yh-RBqGZi1VP9dx2H7Nj-nA at webmail.df.eu>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed; DelSp=Yes
>>
>> >> You won't get any changes in to CPython by creating pull requests. We
>> >> use http://bugs.python.org/ for that, sorry.
>> >
>> > Question -- is there a reason to abide by this rule for docs?  That is,
>> if we
>> > could get a sympathetic core dev to look at pull requests for docs as
>> part of
>> > a streamlined process, would it cause problems?
>>
>> How do you communicate a "pull request"? On bitbucket, there is a
>> "pull request"
>> UI resulting in a tracker item being generated (and an email being sent),
>> but
>> hg.python.org doesn't have a notion of pull requests. Of course, you
>> could
>> use any communication means (email, telephone call, carrier pigeon) to
>> request
>> a pull from a "sympathetic core dev".
>>
>> > (What I'm really asking is whether or the bugs.python.org process is
>> > considered critical for potentially minor doc changes and additions.)
>>
>> The sympathetic core dev is mostly free to bypass any submission process
>> initially; commits that bypass established procedures will likely be
>> questioned
>> only after the fact.
>>
>> In the specific case, I'd be worried to verify that the submitter has
>> provided
>> a contributor form. That's easy to do in the bug tracker, but difficult
>> to do
>> in an offline pull request. Of course, for a really minor doc change
>> (e.g. typo
>> fixes), no contrib form is necessary.
>>
>> Regards,
>> Martin
>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 2
>> Date: Thu, 5 Jul 2012 22:41:29 +0300
>> From: anatoly techtonik <techtonik at gmail.com>
>> To: Stefan Behnel <stefan_ml at behnel.de>
>> Cc: python-ideas at python.org, python-dev at python.org
>> Subject: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
>>         itertools.chunks(iterable, size, fill=None))
>> Message-ID:
>>         <CAPkN8x+A-OYWNLNKDH=
>> 6GnQn+o_Tb3LMnimHYs9zkYmWR1GTgA at mail.gmail.com>
>> Content-Type: text/plain; charset=UTF-8
>>
>> On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel <stefan_ml at behnel.de>
>> wrote:
>> > anatoly techtonik, 05.07.2012 15:36:
>> >> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
>> >>> From Raymond's first message on http://bugs.python.org/issue6021 ,
>> add
>> >>> grouper:
>> >>>
>> >>> "This has been rejected before.
>> >>
>> >> I quite often see such arguments and I can't stand to repeat that
>> >> these are not arguments. It is good to know, but when people use that
>> >> as a reason to close tickets - that's just disgusting.
>> >
>> > The *real* problem is that people keep bringing up topics (and even
>> spell
>> > them out in the bug tracker) without searching for existing discussions
>> > and/or tickets first. That's why those who do such a search (or who know
>> > what they are talking about anyway) close these tickets with the remark
>> > "this has been rejected before", instead of repeating an entire heap of
>> > arguments all over again to feed a discussion that would only lead to
>> the
>> > same result as it did before, often several times before.
>>
>> Make the bloody FAQ and summarize this stuff? Why waste each others
>> time? If people don't enjoy repeating themselves over and over - there
>> is a bloody wiki. What should happen to people to start extracting
>> gems of knowledge from piles of dusty sheets called list "archives"
>> for others to admire.
>>
>> No, it is easier to say "it was already discussed many times", "why
>> don't you Google yourself", "so far you're only complaining", etc. If
>> people can't find anything - why everybody thinks they are ignorant
>> and lazy. Even if it so, why nobody thinks that maybe that bloody
>> Xapian index is dead again for a bloody amount of moons nobody knows
>> why and how many exactly? Why nobody thinks that lazy coders can also
>> help with development? Maybe that laziness is the primary reason some
>> major groups actually prefer Python to Java, C++ and other more
>> interesting languages (such as PHP) when it comes to typing? Make it
>> easy and the patches will follow. Answers like "this was discussed
>> before" don't make it easy to understand, and leaving users rereading
>> old 19xx archives that people don't reread themselves will likely make
>> users bounce and never (NEVER!) come up with some proposal again. An
>> "organic" way to keep traffic low.
>>
>> Miscommunication is a bad experience for users, bad experience for
>> developers, everybody is annoyed and as a result such nice language as
>> Python loses points on TIOBE (and convenient chunk() functions to
>> munch-munch on the sequence data).
>>
>> Wheew. :-F
>>
>>
>> ------------------------------
>>
>> Message: 3
>> Date: Thu, 05 Jul 2012 20:55:09 +0100
>> From: Mark Lawrence <breamoreboy at yahoo.co.uk>
>> To: python-dev at python.org
>> Cc: python-ideas at python.org
>> Subject: Re: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
>>         itertools.chunks(iterable, size, fill=None))
>> Message-ID: <jt4re5$3gs$1 at dough.gmane.org>
>> Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>>
>> On 05/07/2012 20:41, anatoly techtonik wrote:
>> > On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel <stefan_ml at behnel.de>
>> wrote:
>> >> anatoly techtonik, 05.07.2012 15:36:
>> >>> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
>> >>>>  From Raymond's first message on http://bugs.python.org/issue6021 ,
>> add
>> >>>> grouper:
>> >>>>
>> >>>> "This has been rejected before.
>> >>>
>> >>> I quite often see such arguments and I can't stand to repeat that
>> >>> these are not arguments. It is good to know, but when people use that
>> >>> as a reason to close tickets - that's just disgusting.
>> >>
>> >> The *real* problem is that people keep bringing up topics (and even
>> spell
>> >> them out in the bug tracker) without searching for existing discussions
>> >> and/or tickets first. That's why those who do such a search (or who
>> know
>> >> what they are talking about anyway) close these tickets with the remark
>> >> "this has been rejected before", instead of repeating an entire heap of
>> >> arguments all over again to feed a discussion that would only lead to
>> the
>> >> same result as it did before, often several times before.
>> >
>> > Make the bloody FAQ and summarize this stuff? Why waste each others
>> > time? If people don't enjoy repeating themselves over and over - there
>> > is a bloody wiki. What should happen to people to start extracting
>> > gems of knowledge from piles of dusty sheets called list "archives"
>> > for others to admire.
>> >
>> > No, it is easier to say "it was already discussed many times", "why
>> > don't you Google yourself", "so far you're only complaining", etc. If
>> > people can't find anything - why everybody thinks they are ignorant
>> > and lazy. Even if it so, why nobody thinks that maybe that bloody
>> > Xapian index is dead again for a bloody amount of moons nobody knows
>> > why and how many exactly? Why nobody thinks that lazy coders can also
>> > help with development? Maybe that laziness is the primary reason some
>> > major groups actually prefer Python to Java, C++ and other more
>> > interesting languages (such as PHP) when it comes to typing? Make it
>> > easy and the patches will follow. Answers like "this was discussed
>> > before" don't make it easy to understand, and leaving users rereading
>> > old 19xx archives that people don't reread themselves will likely make
>> > users bounce and never (NEVER!) come up with some proposal again. An
>> > "organic" way to keep traffic low.
>> >
>> > Miscommunication is a bad experience for users, bad experience for
>> > developers, everybody is annoyed and as a result such nice language as
>> > Python loses points on TIOBE (and convenient chunk() functions to
>> > munch-munch on the sequence data).
>> >
>> > Wheew. :-F
>> >
>>
>> Can I safely assume that you are volunteering to do the work required?
>>
>> --
>> Cheers.
>>
>> Mark Lawrence.
>>
>>
>>
>>
>>
>> ------------------------------
>>
>> Message: 4
>> Date: Thu, 05 Jul 2012 21:58:52 +0200
>> From: Stefan Behnel <stefan_ml at behnel.de>
>> To: python-dev at python.org
>> Cc: python-ideas at python.org
>> Subject: Re: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
>>         itertools.chunks(iterable, size, fill=None))
>> Message-ID: <jt4rlt$45k$1 at dough.gmane.org>
>> Content-Type: text/plain; charset=UTF-8
>>
>> anatoly techtonik, 05.07.2012 21:41:
>> > On Thu, Jul 5, 2012 at 7:50 PM, Stefan Behnel wrote:
>> >> anatoly techtonik, 05.07.2012 15:36:
>> >>> On Sun, Jul 1, 2012 at 12:09 AM, Terry Reedy wrote:
>> >>>> From Raymond's first message on http://bugs.python.org/issue6021 ,
>> add
>> >>>> grouper:
>> >>>>
>> >>>> "This has been rejected before.
>> >>>
>> >>> I quite often see such arguments and I can't stand to repeat that
>> >>> these are not arguments. It is good to know, but when people use that
>> >>> as a reason to close tickets - that's just disgusting.
>> >>
>> >> The *real* problem is that people keep bringing up topics (and even
>> spell
>> >> them out in the bug tracker) without searching for existing discussions
>> >> and/or tickets first. That's why those who do such a search (or who
>> know
>> >> what they are talking about anyway) close these tickets with the remark
>> >> "this has been rejected before", instead of repeating an entire heap of
>> >> arguments all over again to feed a discussion that would only lead to
>> the
>> >> same result as it did before, often several times before.
>> >
>> > Make the bloody FAQ and summarize this stuff? Why waste each others
>> > time?
>>
>> Yes, that is exactly the question.
>>
>> It takes time to write things up nicely. I mean, once someone has pointed
>> out to you that this has been discussed before, you could just go, look it
>> up (or search for it), and then put it into a Wiki or blog post yourself,
>> or sum it up and send it to the mailing list as a reply. Why rely on
>> others
>> to do it for you?
>>
>> Stefan
>>
>>
>>
>> ------------------------------
>>
>> Message: 5
>> Date: Thu, 5 Jul 2012 23:11:46 +0200
>> From: Paul Boddie <paul at boddie.org.uk>
>> To: python-dev at python.org
>> Subject: Re: [Python-Dev] Bloody FAQ (Was: [Python-ideas]
>>         itertools.chunks(iterable, size, fill=None))
>> Message-ID: <201207052311.46867.paul at boddie.org.uk>
>> Content-Type: text/plain;  charset="utf-8"
>>
>> Stefan Behnel wrote:
>> > anatoly techtonik, 05.07.2012 21:41:
>> > >
>> > > Make the bloody FAQ and summarize this stuff? Why waste each others
>> > > time?
>> >
>> > Yes, that is exactly the question.
>> >
>> > It takes time to write things up nicely. I mean, once someone has
>> pointed
>> > out to you that this has been discussed before, you could just go, look
>> it
>> > up (or search for it), and then put it into a Wiki or blog post
>> yourself,
>> > or sum it up and send it to the mailing list as a reply. Why rely on
>> others
>> > to do it for you?
>>
>> To be fair, Anatoly has done quite a bit of maintenance on some of the
>> Wiki
>> content around various aspects of the project, so it's not as if he's
>> demanding anything out of the ordinary or asking for others to do things
>> that
>> he isn't already doing in some sense. My experience is that there usually
>> needs to be some willingness on the other end of the transaction, and if
>> it
>> takes repetition to encourage it amongst those who don't see the current
>> situation as a problem for them, then so be it.
>>
>> Of course, this kind of documentation activity, where one gathers together
>> historical decisions and the consensus from long-forgotten discussions, is
>> pretty thankless work. I occasionally regard it as worthwhile if only to
>> bring up something someone said as an inconvenient interruption in any
>> current discussion, but that's a pretty minimal reward for all the effort
>> unless one has such work as part of one's daily routine.
>>
>> Paul
>>
>>
>> ------------------------------
>>
>> Message: 6
>> Date: Fri, 06 Jul 2012 08:47:30 +0200
>> From: Larry Hastings <larry at hastings.org>
>> To: python-dev at python.org, python-committers at python.org
>> Subject: [Python-Dev] EuroPython 2012 Language Summit is Canceled.
>> Message-ID: <4FF68A02.8000500 at hastings.org>
>> Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
>>
>>
>>
>> I only got one more RSVP and zero topics for the docket.  So let's
>> sprint instead.
>>
>> See you at the PyCon 2013 Language Summit,
>>
>>
>> //arry/
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://mail.python.org/pipermail/python-dev/attachments/20120706/f13295aa/attachment-0001.html
>> >
>>
>> ------------------------------
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>>
>>
>> End of Python-Dev Digest, Vol 108, Issue 7
>> ******************************************
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120710/df9496e0/attachment-0001.html>

From ronaldoussoren at mac.com  Wed Jul 11 07:30:43 2012
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 11 Jul 2012 07:30:43 +0200
Subject: [Python-Dev] Testing 3.3 framework builds on OS X
In-Reply-To: <CAAgNOZ31o0Rpm8Uby14BzeTXrJ4TT2LFi0_sg8dNsgB+H9oBJQ@mail.gmail.com>
References: <CAAgNOZ31o0Rpm8Uby14BzeTXrJ4TT2LFi0_sg8dNsgB+H9oBJQ@mail.gmail.com>
Message-ID: <A7DAF37B-4BA0-4382-93B5-9DB9EF7A2257@mac.com>


On 10 Jul, 2012, at 18:41, Marc Abramowitz wrote:

> Anyone know how to build and run tests on an OS X build with
> ?enable-framework, without installing the framework? I want to test
> that something works in my tree without polluting /Library/Frameworks
> with stuff that could mess up future tests.
> 
> I've been mucking around with DYLD_FRAMEWORK_PATH but not having much luck.
> 

Setting DYLD_FRAMEWORK_PATH should work and is whtat's used for RUNSHARED in the Makefile. 

I regularly build using 'configure --enable-framework' and then use 'make test' to test that build. This runs the python.exe in the build directory with DYLD_FRAMEWORK_PATH set to the build directory (as an absolute path).

Ronald

> 
> Thanks,
> Marc
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/ronaldoussoren%40mac.com

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4788 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120711/3f1df402/attachment.bin>

From steve at holdenweb.com  Thu Jul 12 02:54:33 2012
From: steve at holdenweb.com (Steve Holden)
Date: Wed, 11 Jul 2012 19:54:33 -0500
Subject: [Python-Dev] =?windows-1252?q?Fwd=3A_How_to_recruit_open-source_c?=
 =?windows-1252?q?ontributors_=96_The_Story_of_Data?=
Message-ID: <44E4E015-4C35-4265-A4EF-51B2D3A64086@holdenweb.com>

Interesting article about Gentoo's approach to developer recruitment from GSoC:

http://redmonk.com/dberkholz/2012/07/10/how-to-recruit-open-source-contributors/ 

S
--
Steve Holden steve at holdenweb.com,  Holden Web, LLC http://holdenweb.com/
Python classes (and much more) through the web http://oreillyschool.com/
Conferences and technical event management at http://theopenbastion.com/
Next:           DjangoCon US Sep 6-8, Washington DC http://djangocon.us/



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120711/0a6e9085/attachment.html>

From ryanpaullin at gmail.com  Thu Jul 12 12:30:59 2012
From: ryanpaullin at gmail.com (Ryan Paullin)
Date: Thu, 12 Jul 2012 03:30:59 -0700
Subject: [Python-Dev] Python-Dev Digest, Vol 108, Issue 14
In-Reply-To: <mailman.75.1342087204.7267.python-dev@python.org>
References: <mailman.75.1342087204.7267.python-dev@python.org>
Message-ID: <CANzqOBR_DHZ==bojqAG7tPjZOAONR8uBQrrxBuw_qLe-_zRZvw@mail.gmail.com>

looks like theres no forgiveness except for dj yoda

On Thu, Jul 12, 2012 at 3:00 AM, <python-dev-request at python.org> wrote:

> Send Python-Dev mailing list submissions to
>         python-dev at python.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         http://mail.python.org/mailman/listinfo/python-dev
> or, via email, send a message with subject or body 'help' to
>         python-dev-request at python.org
>
> You can reach the person managing the list at
>         python-dev-owner at python.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Python-Dev digest..."
>
>
> Today's Topics:
>
>    1. Fwd: How to recruit open-source contributors ? The Story of
>       Data (Steve Holden)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Wed, 11 Jul 2012 19:54:33 -0500
> From: Steve Holden <steve at holdenweb.com>
> To: Arc Riley <arcriley at gmail.com>
> Cc: python-dev at python.org
> Subject: [Python-Dev] Fwd: How to recruit open-source contributors ?
>         The Story of Data
> Message-ID: <44E4E015-4C35-4265-A4EF-51B2D3A64086 at holdenweb.com>
> Content-Type: text/plain; charset="us-ascii"
>
> Interesting article about Gentoo's approach to developer recruitment from
> GSoC:
>
>
> http://redmonk.com/dberkholz/2012/07/10/how-to-recruit-open-source-contributors/
>
> S
> --
> Steve Holden steve at holdenweb.com,  Holden Web, LLC http://holdenweb.com/
> Python classes (and much more) through the web http://oreillyschool.com/
> Conferences and technical event management at http://theopenbastion.com/
> Next:           DjangoCon US Sep 6-8, Washington DC http://djangocon.us/
>
>
>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> http://mail.python.org/pipermail/python-dev/attachments/20120711/0a6e9085/attachment-0001.html
> >
>
> ------------------------------
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
>
>
> End of Python-Dev Digest, Vol 108, Issue 14
> *******************************************
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120712/dd155fb2/attachment.html>

From ryanpaullin at gmail.com  Thu Jul 12 12:31:18 2012
From: ryanpaullin at gmail.com (Ryan Paullin)
Date: Thu, 12 Jul 2012 03:31:18 -0700
Subject: [Python-Dev] Python-Dev Digest, Vol 108, Issue 14
In-Reply-To: <CANzqOBR_DHZ==bojqAG7tPjZOAONR8uBQrrxBuw_qLe-_zRZvw@mail.gmail.com>
References: <mailman.75.1342087204.7267.python-dev@python.org>
	<CANzqOBR_DHZ==bojqAG7tPjZOAONR8uBQrrxBuw_qLe-_zRZvw@mail.gmail.com>
Message-ID: <CANzqOBSLL=1jY4=TDuzk1pNo+rDURGJSMb9eqCaM_t9rdmbk1w@mail.gmail.com>

<- its just my gmail face

On Thu, Jul 12, 2012 at 3:30 AM, Ryan Paullin <ryanpaullin at gmail.com> wrote:

> looks like theres no forgiveness except for dj yoda
>
>
> On Thu, Jul 12, 2012 at 3:00 AM, <python-dev-request at python.org> wrote:
>
>> Send Python-Dev mailing list submissions to
>>         python-dev at python.org
>>
>> To subscribe or unsubscribe via the World Wide Web, visit
>>         http://mail.python.org/mailman/listinfo/python-dev
>> or, via email, send a message with subject or body 'help' to
>>         python-dev-request at python.org
>>
>> You can reach the person managing the list at
>>         python-dev-owner at python.org
>>
>> When replying, please edit your Subject line so it is more specific
>> than "Re: Contents of Python-Dev digest..."
>>
>>
>> Today's Topics:
>>
>>    1. Fwd: How to recruit open-source contributors ? The Story of
>>       Data (Steve Holden)
>>
>>
>> ----------------------------------------------------------------------
>>
>> Message: 1
>> Date: Wed, 11 Jul 2012 19:54:33 -0500
>> From: Steve Holden <steve at holdenweb.com>
>> To: Arc Riley <arcriley at gmail.com>
>> Cc: python-dev at python.org
>> Subject: [Python-Dev] Fwd: How to recruit open-source contributors ?
>>         The Story of Data
>> Message-ID: <44E4E015-4C35-4265-A4EF-51B2D3A64086 at holdenweb.com>
>> Content-Type: text/plain; charset="us-ascii"
>>
>> Interesting article about Gentoo's approach to developer recruitment from
>> GSoC:
>>
>>
>> http://redmonk.com/dberkholz/2012/07/10/how-to-recruit-open-source-contributors/
>>
>> S
>> --
>> Steve Holden steve at holdenweb.com,  Holden Web, LLC http://holdenweb.com/
>> Python classes (and much more) through the web http://oreillyschool.com/
>> Conferences and technical event management at http://theopenbastion.com/
>> Next:           DjangoCon US Sep 6-8, Washington DC http://djangocon.us/
>>
>>
>>
>> -------------- next part --------------
>> An HTML attachment was scrubbed...
>> URL: <
>> http://mail.python.org/pipermail/python-dev/attachments/20120711/0a6e9085/attachment-0001.html
>> >
>>
>> ------------------------------
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>>
>>
>> End of Python-Dev Digest, Vol 108, Issue 14
>> *******************************************
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120712/23e3d828/attachment.html>

From ryanpaullin at gmail.com  Thu Jul 12 12:34:20 2012
From: ryanpaullin at gmail.com (Ryan Paullin)
Date: Thu, 12 Jul 2012 03:34:20 -0700
Subject: [Python-Dev] Python-Dev Digest, Vol 108, Issue 14
In-Reply-To: <CANzqOBSLL=1jY4=TDuzk1pNo+rDURGJSMb9eqCaM_t9rdmbk1w@mail.gmail.com>
References: <mailman.75.1342087204.7267.python-dev@python.org>
	<CANzqOBR_DHZ==bojqAG7tPjZOAONR8uBQrrxBuw_qLe-_zRZvw@mail.gmail.com>
	<CANzqOBSLL=1jY4=TDuzk1pNo+rDURGJSMb9eqCaM_t9rdmbk1w@mail.gmail.com>
Message-ID: <CANzqOBTiA=H4P8sX0nT9z3Nk7cD5bh_EWVpw8y9zWFnW5oqViQ@mail.gmail.com>

$ python -m timeit -s 'import test' 'list(test.grouper(2,"abcdef"))'
100000 loops, best of 3: 5.34 usec per loop
$ python -m timeit -s 'import test' 'test.grouper(2,"abcdef")'
100000 loops, best of 3: 2.19 usec per loop
$ python -m timeit -s 'import test' 'list(test.chunks(2,"abcdef"))'
100000 loops, best of 3: 2.85 usec per loop
$ python -m timeit -s 'import test' 'test.chunks(2,"abcdef")'
1000000 loops, best of 3: 0.685 usec per loop

some woman wrote this



On Thu, Jul 12, 2012 at 3:31 AM, Ryan Paullin <ryanpaullin at gmail.com> wrote:

> <- its just my gmail face
>
>
> On Thu, Jul 12, 2012 at 3:30 AM, Ryan Paullin <ryanpaullin at gmail.com>wrote:
>
>> looks like theres no forgiveness except for dj yoda
>>
>>
>> On Thu, Jul 12, 2012 at 3:00 AM, <python-dev-request at python.org> wrote:
>>
>>> Send Python-Dev mailing list submissions to
>>>         python-dev at python.org
>>>
>>> To subscribe or unsubscribe via the World Wide Web, visit
>>>         http://mail.python.org/mailman/listinfo/python-dev
>>> or, via email, send a message with subject or body 'help' to
>>>         python-dev-request at python.org
>>>
>>> You can reach the person managing the list at
>>>         python-dev-owner at python.org
>>>
>>> When replying, please edit your Subject line so it is more specific
>>> than "Re: Contents of Python-Dev digest..."
>>>
>>>
>>> Today's Topics:
>>>
>>>    1. Fwd: How to recruit open-source contributors ? The Story of
>>>       Data (Steve Holden)
>>>
>>>
>>> ----------------------------------------------------------------------
>>>
>>> Message: 1
>>> Date: Wed, 11 Jul 2012 19:54:33 -0500
>>> From: Steve Holden <steve at holdenweb.com>
>>> To: Arc Riley <arcriley at gmail.com>
>>> Cc: python-dev at python.org
>>> Subject: [Python-Dev] Fwd: How to recruit open-source contributors ?
>>>         The Story of Data
>>> Message-ID: <44E4E015-4C35-4265-A4EF-51B2D3A64086 at holdenweb.com>
>>> Content-Type: text/plain; charset="us-ascii"
>>>
>>> Interesting article about Gentoo's approach to developer recruitment
>>> from GSoC:
>>>
>>>
>>> http://redmonk.com/dberkholz/2012/07/10/how-to-recruit-open-source-contributors/
>>>
>>> S
>>> --
>>> Steve Holden steve at holdenweb.com,  Holden Web, LLC http://holdenweb.com/
>>> Python classes (and much more) through the web http://oreillyschool.com/
>>> Conferences and technical event management at http://theopenbastion.com/
>>> Next:           DjangoCon US Sep 6-8, Washington DC http://djangocon.us/
>>>
>>>
>>>
>>> -------------- next part --------------
>>> An HTML attachment was scrubbed...
>>> URL: <
>>> http://mail.python.org/pipermail/python-dev/attachments/20120711/0a6e9085/attachment-0001.html
>>> >
>>>
>>> ------------------------------
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> http://mail.python.org/mailman/listinfo/python-dev
>>>
>>>
>>> End of Python-Dev Digest, Vol 108, Issue 14
>>> *******************************************
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120712/a103d264/attachment.html>

From rdmurray at bitdance.com  Thu Jul 12 15:46:27 2012
From: rdmurray at bitdance.com (R. David Murray)
Date: Thu, 12 Jul 2012 09:46:27 -0400
Subject: [Python-Dev] Python-Dev Digest, Vol 108, Issue 14
In-Reply-To: <CANzqOBTiA=H4P8sX0nT9z3Nk7cD5bh_EWVpw8y9zWFnW5oqViQ@mail.gmail.com>
References: <mailman.75.1342087204.7267.python-dev@python.org>
	<CANzqOBR_DHZ==bojqAG7tPjZOAONR8uBQrrxBuw_qLe-_zRZvw@mail.gmail.com>
	<CANzqOBSLL=1jY4=TDuzk1pNo+rDURGJSMb9eqCaM_t9rdmbk1w@mail.gmail.com>
	<CANzqOBTiA=H4P8sX0nT9z3Nk7cD5bh_EWVpw8y9zWFnW5oqViQ@mail.gmail.com>
Message-ID: <20120712134628.2A523250823@webabinitio.net>

On Thu, 12 Jul 2012 03:34:20 -0700, Ryan Paullin <ryanpaullin at gmail.com> wrote:
> $ python -m timeit -s 'import test' 'list(test.grouper(2,"abcdef"))'
> 100000 loops, best of 3: 5.34 usec per loop
> $ python -m timeit -s 'import test' 'test.grouper(2,"abcdef")'
> 100000 loops, best of 3: 2.19 usec per loop
> $ python -m timeit -s 'import test' 'list(test.chunks(2,"abcdef"))'
> 100000 loops, best of 3: 2.85 usec per loop
> $ python -m timeit -s 'import test' 'test.chunks(2,"abcdef")'
> 1000000 loops, best of 3: 0.685 usec per loop
> 
> some woman wrote this

I don't have any idea what any of your recent posts mean, but this
one appears to be offensive.  I would appreciate it if you would stop
posting until you have something substantive to say, and can do so in
a civil fashion.

--David

From solipsis at pitrou.net  Thu Jul 12 18:20:32 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 12 Jul 2012 18:20:32 +0200
Subject: [Python-Dev]
 =?utf-8?q?How_to_recruit_open-source_contributors_?=
 =?utf-8?q?=E2=80=93_The_Story_of_Data?=
References: <44E4E015-4C35-4265-A4EF-51B2D3A64086@holdenweb.com>
Message-ID: <20120712182032.3d68e891@pitrou.net>

On Wed, 11 Jul 2012 19:54:33 -0500
Steve Holden <steve at holdenweb.com> wrote:
> Interesting article about Gentoo's approach to developer recruitment from GSoC:
> 
> http://redmonk.com/dberkholz/2012/07/10/how-to-recruit-open-source-contributors/

This is quite an interesting article, but I don't think we are that
high in pyramid of needs; we would first need someone dedicated enough
to keep track of all GSoC projects.
(and we would also need mentoring of potential mentors)

Regards

Antoine.



-- 
Software development and contracting: http://pro.pitrou.net



From status at bugs.python.org  Fri Jul 13 18:07:18 2012
From: status at bugs.python.org (Python tracker)
Date: Fri, 13 Jul 2012 18:07:18 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20120713160718.808121C984@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2012-07-06 - 2012-07-13)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    3520 (+25)
  closed 23603 (+57)
  total  27123 (+82)

Open issues with patches: 1485 


Issues opened (58)
==================

#14826: urllib2.urlopen fails to load URL
http://bugs.python.org/issue14826  reopened by rosslagerwall

#15264: PyErr_SetFromErrnoWithFilenameObject() undocumented
http://bugs.python.org/issue15264  opened by pitrou

#15266: Perform the same checks as PyPI for Description field
http://bugs.python.org/issue15266  opened by cjerdonek

#15267: tempfile.TemporaryFile and httplib incompatibility
http://bugs.python.org/issue15267  opened by tzs

#15268: curses configure checks fail if only /usr/include/ncursesw/cur
http://bugs.python.org/issue15268  opened by doko

#15269: Document dircmp.left and dircmp.right
http://bugs.python.org/issue15269  opened by cjerdonek

#15270: "Economy of Expression" section outdated
http://bugs.python.org/issue15270  opened by cjerdonek

#15271: argparse: repeatedly specifying the same argument ignores the 
http://bugs.python.org/issue15271  opened by mapleoin

#15272: pkgutil.find_loader accepts invalid module names
http://bugs.python.org/issue15272  opened by ncoghlan

#15273: Remove unnecessarily random behavior from test_unparse.py
http://bugs.python.org/issue15273  opened by larry

#15275: isinstance is called a more times that needed in ntpath
http://bugs.python.org/issue15275  opened by mandel

#15276: unicode format does not really work in Python 2.x
http://bugs.python.org/issue15276  opened by Ariel.Ben-Yehuda

#15278: UnicodeDecodeError when readline in codecs.py
http://bugs.python.org/issue15278  opened by lovelylain

#15279: Spurious unittest warnings
http://bugs.python.org/issue15279  opened by lukasz.langa

#15280: Don't use builtins as variable names in urllib.request
http://bugs.python.org/issue15280  opened by bbrazil

#15285: test_timeout failure when system on IPv4 10.x.x.x subnet
http://bugs.python.org/issue15285  opened by flox

#15286: normpath does not work with local literal paths
http://bugs.python.org/issue15286  opened by mandel

#15292: import hook behavior documentation improvement
http://bugs.python.org/issue15292  opened by iko

#15295: Document PEP 420 namespace packages
http://bugs.python.org/issue15295  opened by brett.cannon

#15297: pkgutil.iter_importers() includes an ImpImporter
http://bugs.python.org/issue15297  opened by cjerdonek

#15298: _sysconfigdata is generated in srcdir, not builddir
http://bugs.python.org/issue15298  opened by doko

#15299: pkgutil.ImpImporter(None).iter_modules() does not search sys.p
http://bugs.python.org/issue15299  opened by cjerdonek

#15301: os.chown: OverflowError: Python int too large to convert to C 
http://bugs.python.org/issue15301  opened by do1

#15302: Use argparse instead of getopt in test.regrtest
http://bugs.python.org/issue15302  opened by cjerdonek

#15303: Minor revision to the <BaseWidget._setup> method in Tkinter
http://bugs.python.org/issue15303  opened by Drew.French

#15304: Wrong path in test.support.temp_cwd() error message
http://bugs.python.org/issue15304  opened by cjerdonek

#15305: Test harness unnecessarily disambiguating twice
http://bugs.python.org/issue15305  opened by cjerdonek

#15307: Patch for --symlink support in pyvenv with framework python
http://bugs.python.org/issue15307  opened by ronaldoussoren

#15308: IDLE - add an "Interrupt Execution" to shell menu
http://bugs.python.org/issue15308  opened by serwy

#15310: urllib: Support for multiple WWW-Authenticate headers and/or m
http://bugs.python.org/issue15310  opened by almost

#15311: Developer Guide doesn't get updated once a day
http://bugs.python.org/issue15311  opened by cjerdonek

#15313: IDLE - remove all bare excepts
http://bugs.python.org/issue15313  opened by serwy

#15314: Use importlib instead of pkgutil in runpy
http://bugs.python.org/issue15314  opened by ncoghlan

#15315: Can't build Python extension with mingw32 on Windows
http://bugs.python.org/issue15315  opened by cmcqueen1975

#15317: Source installation sets incorrect permissions for Grammar3.2.
http://bugs.python.org/issue15317  opened by tpievila

#15318: IDLE - sys.stdin is writeable
http://bugs.python.org/issue15318  opened by serwy

#15320: thread-safety issue in regrtest.main()
http://bugs.python.org/issue15320  opened by cjerdonek

#15321: bdist_wininst installers may terminate with "close failed in f
http://bugs.python.org/issue15321  opened by mhammond

#15322: sysconfig.get_config_var('srcdir') returns unexpected value
http://bugs.python.org/issue15322  opened by cjerdonek

#15323: Provide target name in output message when Mock.assert_called_
http://bugs.python.org/issue15323  opened by Brian.Jones

#15324: --match does not work for regrtest
http://bugs.python.org/issue15324  opened by cjerdonek

#15325: --fromfile does not work for regrtest
http://bugs.python.org/issue15325  opened by cjerdonek

#15326: --random does not work for regrtest
http://bugs.python.org/issue15326  opened by cjerdonek

#15327: Argparse: main arguments and subparser arguments indistinguish
http://bugs.python.org/issue15327  opened by Ingo.Fischer

#15328: datetime.strptime slow
http://bugs.python.org/issue15328  opened by Lars.Nordin

#15329: clarify which deque methods are thread-safe
http://bugs.python.org/issue15329  opened by cjerdonek

#15331: Missing codec aliases for bytes-bytes codecs
http://bugs.python.org/issue15331  opened by ncoghlan

#15332: 2to3 should fix bad indentation (or warn about it)
http://bugs.python.org/issue15332  opened by jwilk

#15334: access denied for HKEY_PERFORMANCE_DATA
http://bugs.python.org/issue15334  opened by jkloth

#15335: IDLE - debugger steps through run.py internals
http://bugs.python.org/issue15335  opened by serwy

#15336: Argparse required arguments incorrectly displayed as optional 
http://bugs.python.org/issue15336  opened by rhettinger

#15337: The cmd module incorrectly lists "help" as an undocument comma
http://bugs.python.org/issue15337  opened by rhettinger

#15338: test_UNC_path failure in test_import
http://bugs.python.org/issue15338  opened by pitrou

#15339: document the threading "facts of life" in Python
http://bugs.python.org/issue15339  opened by cjerdonek

#15340: OSError with "import random" when /dev/urandom doesn't exist (
http://bugs.python.org/issue15340  opened by iwienand

#15343: "pydoc -w <package>" writes out page with empty "Package Conte
http://bugs.python.org/issue15343  opened by christopherthemagnificent

#15344: devinabox: failure when running make_a_box multiple times
http://bugs.python.org/issue15344  opened by eric.snow

#15345: HOWTOs Argparse tutorial - code example raises SyntaxError
http://bugs.python.org/issue15345  opened by simon.hayward



Most recent 15 issues with no replies (15)
==========================================

#15345: HOWTOs Argparse tutorial - code example raises SyntaxError
http://bugs.python.org/issue15345

#15343: "pydoc -w <package>" writes out page with empty "Package Conte
http://bugs.python.org/issue15343

#15340: OSError with "import random" when /dev/urandom doesn't exist (
http://bugs.python.org/issue15340

#15337: The cmd module incorrectly lists "help" as an undocument comma
http://bugs.python.org/issue15337

#15336: Argparse required arguments incorrectly displayed as optional 
http://bugs.python.org/issue15336

#15334: access denied for HKEY_PERFORMANCE_DATA
http://bugs.python.org/issue15334

#15327: Argparse: main arguments and subparser arguments indistinguish
http://bugs.python.org/issue15327

#15326: --random does not work for regrtest
http://bugs.python.org/issue15326

#15325: --fromfile does not work for regrtest
http://bugs.python.org/issue15325

#15321: bdist_wininst installers may terminate with "close failed in f
http://bugs.python.org/issue15321

#15303: Minor revision to the <BaseWidget._setup> method in Tkinter
http://bugs.python.org/issue15303

#15280: Don't use builtins as variable names in urllib.request
http://bugs.python.org/issue15280

#15278: UnicodeDecodeError when readline in codecs.py
http://bugs.python.org/issue15278

#15275: isinstance is called a more times that needed in ntpath
http://bugs.python.org/issue15275

#15269: Document dircmp.left and dircmp.right
http://bugs.python.org/issue15269



Most recent 15 issues waiting for review (15)
=============================================

#15345: HOWTOs Argparse tutorial - code example raises SyntaxError
http://bugs.python.org/issue15345

#15334: access denied for HKEY_PERFORMANCE_DATA
http://bugs.python.org/issue15334

#15323: Provide target name in output message when Mock.assert_called_
http://bugs.python.org/issue15323

#15320: thread-safety issue in regrtest.main()
http://bugs.python.org/issue15320

#15318: IDLE - sys.stdin is writeable
http://bugs.python.org/issue15318

#15311: Developer Guide doesn't get updated once a day
http://bugs.python.org/issue15311

#15310: urllib: Support for multiple WWW-Authenticate headers and/or m
http://bugs.python.org/issue15310

#15308: IDLE - add an "Interrupt Execution" to shell menu
http://bugs.python.org/issue15308

#15307: Patch for --symlink support in pyvenv with framework python
http://bugs.python.org/issue15307

#15304: Wrong path in test.support.temp_cwd() error message
http://bugs.python.org/issue15304

#15302: Use argparse instead of getopt in test.regrtest
http://bugs.python.org/issue15302

#15299: pkgutil.ImpImporter(None).iter_modules() does not search sys.p
http://bugs.python.org/issue15299

#15298: _sysconfigdata is generated in srcdir, not builddir
http://bugs.python.org/issue15298

#15286: normpath does not work with local literal paths
http://bugs.python.org/issue15286

#15280: Don't use builtins as variable names in urllib.request
http://bugs.python.org/issue15280



Top 10 most discussed issues (10)
=================================

#15318: IDLE - sys.stdin is writeable
http://bugs.python.org/issue15318  22 msgs

#14814: Implement PEP 3144 (the ipaddress module)
http://bugs.python.org/issue14814  20 msgs

#14826: urllib2.urlopen fails to load URL
http://bugs.python.org/issue14826  17 msgs

#15320: thread-safety issue in regrtest.main()
http://bugs.python.org/issue15320  14 msgs

#15302: Use argparse instead of getopt in test.regrtest
http://bugs.python.org/issue15302  10 msgs

#4832: IDLE does not supply a default ext of .py on Windows or OS X f
http://bugs.python.org/issue4832   9 msgs

#15144: Possible integer overflow in operations with addresses and siz
http://bugs.python.org/issue15144   9 msgs

#15231: update PyPI upload doc to say --no-raw passed to rst2html.py
http://bugs.python.org/issue15231   9 msgs

#15338: test_UNC_path failure in test_import
http://bugs.python.org/issue15338   9 msgs

#15285: test_timeout failure when system on IPv4 10.x.x.x subnet
http://bugs.python.org/issue15285   8 msgs



Issues closed (53)
==================

#5931: Python runtime name hardcoded in wsgiref.simple_server
http://bugs.python.org/issue5931  closed by orsenthil

#9867: Interrupted system calls are not retried
http://bugs.python.org/issue9867  closed by pitrou

#10248: Fix resource warnings in test_xmlrpclib
http://bugs.python.org/issue10248  closed by bbrazil

#11153: urllib2 basic auth parser handle unquoted realm in WWW-Authent
http://bugs.python.org/issue11153  closed by orsenthil

#11319: Command line option -t (and -tt) does not work for a particula
http://bugs.python.org/issue11319  closed by gvanrossum

#11624: distutils should support a custom list of exported symbols for
http://bugs.python.org/issue11624  closed by dholth

#11796: Comprehensions in a class definition mostly cannot access clas
http://bugs.python.org/issue11796  closed by flox

#12081: Remove distributed copy of libffi
http://bugs.python.org/issue12081  closed by loewis

#12927: test_ctypes: segfault with suncc
http://bugs.python.org/issue12927  closed by skrah

#13532: In IDLE,	sys.stdout and sys.stderr can write any pickleable ob
http://bugs.python.org/issue13532  closed by loewis

#13686: Some notes on the docs of multiprocessing
http://bugs.python.org/issue13686  closed by eli.bendersky

#13959: Re-implement parts of imp in pure Python
http://bugs.python.org/issue13959  closed by brett.cannon

#14190: Minor C API documentation bugs
http://bugs.python.org/issue14190  closed by eli.bendersky

#14241: io.UnsupportedOperation.__new__(io.UnsupportedOperation) fails
http://bugs.python.org/issue14241  closed by Mark.Shannon

#14590: ConfigParser doesn't strip inline comment when delimiter occur
http://bugs.python.org/issue14590  closed by lukasz.langa

#14990: detect_encoding should fail with SyntaxError on invalid encodi
http://bugs.python.org/issue14990  closed by flox

#15053: imp.lock_held() "Changed in Python 3.3" mention accidentally o
http://bugs.python.org/issue15053  closed by brett.cannon

#15056: Have imp.cache_from_source() raise NotImplementedError when ca
http://bugs.python.org/issue15056  closed by brett.cannon

#15110: strange Tracebacks with importlib
http://bugs.python.org/issue15110  closed by pitrou

#15111: Wrong ImportError message with importlib
http://bugs.python.org/issue15111  closed by brett.cannon

#15167: Re-implement imp.get_magic() in pure Python
http://bugs.python.org/issue15167  closed by brett.cannon

#15242: PyImport_GetMagicTag() should use the same const char * as sys
http://bugs.python.org/issue15242  closed by eric.snow

#15247: io.open() is inconsistent re os.open()
http://bugs.python.org/issue15247  closed by pitrou

#15256: Typo in error message
http://bugs.python.org/issue15256  closed by brett.cannon

#15259: "Helping with Documentation" references missing dailybuild.py
http://bugs.python.org/issue15259  closed by ned.deily

#15260: Mention how to order Misc/NEWS entries
http://bugs.python.org/issue15260  closed by ned.deily

#15262: Idle does not show traceback in other threads
http://bugs.python.org/issue15262  closed by terry.reedy

#15265: random.sample() docs unclear on k < len(population)
http://bugs.python.org/issue15265  closed by rhettinger

#15274: Patch for issue 5765: stack overflow evaluating eval("()" * 30
http://bugs.python.org/issue15274  closed by ag6502

#15277: Fix resource leak in support.py:_is_ipv6_enabled
http://bugs.python.org/issue15277  closed by rosslagerwall

#15281: pyvenv --symlinks option is a no-op?
http://bugs.python.org/issue15281  closed by python-dev

#15282: pysetup still installed
http://bugs.python.org/issue15282  closed by pitrou

#15283: pyvenv says nothing on success
http://bugs.python.org/issue15283  closed by vinay.sajip

#15284: Handle ipv6 not being enabled in test_socket
http://bugs.python.org/issue15284  closed by bbrazil

#15287: support.TESTFN was modified by test_builtin
http://bugs.python.org/issue15287  closed by flox

#15288: Clarify the pkgutil.walk_packages() note
http://bugs.python.org/issue15288  closed by brett.cannon

#15289: Adding __getitem__ as a class method doesn't work as expected
http://bugs.python.org/issue15289  closed by eric.snow

#15290: setAttribute() can fail
http://bugs.python.org/issue15290  closed by loewis

#15291: test_ast leaks memory a lot
http://bugs.python.org/issue15291  closed by pitrou

#15293: AST nodes do not support garbage collection
http://bugs.python.org/issue15293  closed by python-dev

#15294: regression with nested namespace packages
http://bugs.python.org/issue15294  closed by pitrou

#15296: Minidom can't create ASCII representation
http://bugs.python.org/issue15296  closed by eli.bendersky

#15300: test directory doubly-nested running tests with -j/--multiproc
http://bugs.python.org/issue15300  closed by pitrou

#15306: Python3 segfault? (works in Python2)
http://bugs.python.org/issue15306  closed by amaury.forgeotdarc

#15309: buffer/memoryview slice assignment uses only memcpy
http://bugs.python.org/issue15309  closed by skrah

#15312: Serial library not found
http://bugs.python.org/issue15312  closed by ezio.melotti

#15316: runpy swallows ImportError information with relative imports
http://bugs.python.org/issue15316  closed by amaury.forgeotdarc

#15319: IDLE - readline, isatty, and input broken
http://bugs.python.org/issue15319  closed by loewis

#15330: allow deque to act as a thread-safe circular buffer
http://bugs.python.org/issue15330  closed by rhettinger

#15333: import on Windows will recompile a pyc file created on Unix
http://bugs.python.org/issue15333  closed by pitrou

#15341: Cplex and python
http://bugs.python.org/issue15341  closed by amaury.forgeotdarc

#15342: os.path.join behavior
http://bugs.python.org/issue15342  closed by ned.deily

#1616125: Cached globals+builtins lookup optimization
http://bugs.python.org/issue1616125  closed by ag6502

From alex.gaynor at gmail.com  Sun Jul 15 00:11:46 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 14 Jul 2012 22:11:46 +0000 (UTC)
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
Message-ID: <loom.20120715T000919-641@post.gmane.org>

Hi all,

I've just submitted a PEP proposing making __length_hint__ a public API for 
users to define and other VMs to implement:

PEP: 424
Title: A method for exposing a length hint
Version: $Revision$
Last-Modified: $Date
Author: Alex Gaynor <alex.gaynor at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 14-July-2012
Python-Version: 3.4

Abstract
========

CPython currently defines an ``__length_hint__`` method on several types, such
as various iterators. This method is then used by various other functions (such 
as
``map``) to presize lists based on the estimated returned by
``__length_hint__``. Types can then define ``__length_hint__`` which are not
sized, and thus should not define ``__len__``, but can estimate or compute a
size (such as many iterators).

Proposal
========

This PEP proposes formally documenting ``__length_hint__`` for other
interpreter and non-standard library Python to implement.

``__length_hint__`` must return an integer, and is not required to be accurate.
It may return a value that is either larger or smaller than the actual size of
the container. It may raise a ``TypeError`` if a specific instance cannot have
its length estimated. It may not return a negative value.

Rationale
=========

Being able to pre-allocate lists based on the expected size, as estimated by 
``__length_hint__``, can be a significant optimization. CPython has been
observed to run some code faster than PyPy, purely because of this optimization
being present.

Open questions
==============

There are two open questions for this PEP:

* Should ``list`` expose a kwarg in it's constructor for supplying a length
  hint.
* Should a function be added either to ``builtins`` or some other module which
  calls ``__length_hint__``, like ``builtins.len`` calls ``__len__``.

Copyright
=========

This document has been placed into the public domain.

..
    Local Variables:
    mode: indented-text
    indent-tabs-mode: nil
    sentence-end-double-space: t
    fill-column: 70
    coding: utf-8




Alex


From benjamin at python.org  Sun Jul 15 01:18:38 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 14 Jul 2012 16:18:38 -0700
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <loom.20120715T000919-641@post.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
Message-ID: <CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>

2012/7/14 Alex Gaynor <alex.gaynor at gmail.com>:
>
> Proposal
> ========
>
> This PEP proposes formally documenting ``__length_hint__`` for other
> interpreter and non-standard library Python to implement.
>
> ``__length_hint__`` must return an integer, and is not required to be accurate.
> It may return a value that is either larger or smaller than the actual size of
> the container. It may raise a ``TypeError`` if a specific instance cannot have
> its length estimated. It may not return a negative value.

And what happens if you return a negative value?

>
> Rationale
> =========
>
> Being able to pre-allocate lists based on the expected size, as estimated by
> ``__length_hint__``, can be a significant optimization. CPython has been
> observed to run some code faster than PyPy, purely because of this optimization
> being present.
>
> Open questions
> ==============
>
> There are two open questions for this PEP:
>
> * Should ``list`` expose a kwarg in it's constructor for supplying a length
>   hint.
> * Should a function be added either to ``builtins`` or some other module which
>   calls ``__length_hint__``, like ``builtins.len`` calls ``__len__``.

Let's try to keep this as limited as possible for a public API.


-- 
Regards,
Benjamin

From alex.gaynor at gmail.com  Sun Jul 15 01:21:45 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 14 Jul 2012 16:21:45 -0700
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
Message-ID: <CAFRnB2Vbjf5s9nOc2CR8F=_bfhAHYz01miTLMGz63VqcYhJBMA@mail.gmail.com>

On Sat, Jul 14, 2012 at 4:18 PM, Benjamin Peterson <benjamin at python.org>wrote:

> 2012/7/14 Alex Gaynor <alex.gaynor at gmail.com>:
> >
> > Proposal
> > ========
> >
> > This PEP proposes formally documenting ``__length_hint__`` for other
> > interpreter and non-standard library Python to implement.
> >
> > ``__length_hint__`` must return an integer, and is not required to be
> accurate.
> > It may return a value that is either larger or smaller than the actual
> size of
> > the container. It may raise a ``TypeError`` if a specific instance
> cannot have
> > its length estimated. It may not return a negative value.
>
> And what happens if you return a negative value?
>
>
ValueError, the same as with len.


> >
> > Rationale
> > =========
> >
> > Being able to pre-allocate lists based on the expected size, as
> estimated by
> > ``__length_hint__``, can be a significant optimization. CPython has been
> > observed to run some code faster than PyPy, purely because of this
> optimization
> > being present.
> >
> > Open questions
> > ==============
> >
> > There are two open questions for this PEP:
> >
> > * Should ``list`` expose a kwarg in it's constructor for supplying a
> length
> >   hint.
> > * Should a function be added either to ``builtins`` or some other module
> which
> >   calls ``__length_hint__``, like ``builtins.len`` calls ``__len__``.
>
> Let's try to keep this as limited as possible for a public API.
>
>
Sounds reasonable to me!  Should we just go ahead and strip those out now?


>
> --
> Regards,
> Benjamin
>

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120714/c8cfb6d7/attachment.html>

From alexandre.zani at gmail.com  Sun Jul 15 01:28:07 2012
From: alexandre.zani at gmail.com (Alexandre Zani)
Date: Sat, 14 Jul 2012 16:28:07 -0700
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CAFRnB2Vbjf5s9nOc2CR8F=_bfhAHYz01miTLMGz63VqcYhJBMA@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CAFRnB2Vbjf5s9nOc2CR8F=_bfhAHYz01miTLMGz63VqcYhJBMA@mail.gmail.com>
Message-ID: <CAJVMporo4OtqmQbwsYxY55tsQsoG-OQqNDobnNo6v8NP8CBvXA@mail.gmail.com>

On Sat, Jul 14, 2012 at 4:21 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>
>
> On Sat, Jul 14, 2012 at 4:18 PM, Benjamin Peterson <benjamin at python.org>
> wrote:
>>
>> 2012/7/14 Alex Gaynor <alex.gaynor at gmail.com>:
>> >
>> > Proposal
>> > ========
>> >
>> > This PEP proposes formally documenting ``__length_hint__`` for other
>> > interpreter and non-standard library Python to implement.
>> >
>> > ``__length_hint__`` must return an integer, and is not required to be
>> > accurate.
>> > It may return a value that is either larger or smaller than the actual
>> > size of
>> > the container. It may raise a ``TypeError`` if a specific instance
>> > cannot have
>> > its length estimated. It may not return a negative value.
>>
>> And what happens if you return a negative value?
>>
>
> ValueError, the same as with len.
>
>>
>> >
>> > Rationale
>> > =========
>> >
>> > Being able to pre-allocate lists based on the expected size, as
>> > estimated by
>> > ``__length_hint__``, can be a significant optimization. CPython has been
>> > observed to run some code faster than PyPy, purely because of this
>> > optimization
>> > being present.
>> >
>> > Open questions
>> > ==============
>> >
>> > There are two open questions for this PEP:
>> >
>> > * Should ``list`` expose a kwarg in it's constructor for supplying a
>> > length
>> >   hint.
>> > * Should a function be added either to ``builtins`` or some other module
>> > which
>> >   calls ``__length_hint__``, like ``builtins.len`` calls ``__len__``.
>>
>> Let's try to keep this as limited as possible for a public API.
>>
>
> Sounds reasonable to me!  Should we just go ahead and strip those out now?

I'm +1 on not having a public API for this. Ultimately the contract
for a length hint will depend heavily upon what you need it for. Some
applications would require a length hint to be an "at least" others an
"at most" and others something else entirely. Given that the contract
here appears to be >=0, I don't think the length hint is particularly
useful to the public at large.

>
>>
>>
>> --
>> Regards,
>> Benjamin
>
>
> Alex
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/alexandre.zani%40gmail.com
>

From benjamin at python.org  Sun Jul 15 01:37:18 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 14 Jul 2012 16:37:18 -0700
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CAFRnB2Vbjf5s9nOc2CR8F=_bfhAHYz01miTLMGz63VqcYhJBMA@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CAFRnB2Vbjf5s9nOc2CR8F=_bfhAHYz01miTLMGz63VqcYhJBMA@mail.gmail.com>
Message-ID: <CAPZV6o_kTTYkWBx0HzPOg_Cr83+zpKSAmFd8DdsBxpdbXP_LzQ@mail.gmail.com>

2012/7/14 Alex Gaynor <alex.gaynor at gmail.com>:
>
>
> On Sat, Jul 14, 2012 at 4:18 PM, Benjamin Peterson <benjamin at python.org>
> wrote:
>>
>> 2012/7/14 Alex Gaynor <alex.gaynor at gmail.com>:
>> >
>> > Proposal
>> > ========
>> >
>> > This PEP proposes formally documenting ``__length_hint__`` for other
>> > interpreter and non-standard library Python to implement.
>> >
>> > ``__length_hint__`` must return an integer, and is not required to be
>> > accurate.
>> > It may return a value that is either larger or smaller than the actual
>> > size of
>> > the container. It may raise a ``TypeError`` if a specific instance
>> > cannot have
>> > its length estimated. It may not return a negative value.
>>
>> And what happens if you return a negative value?
>>
>
> ValueError, the same as with len.

CPython will probably have to updated to not ignore it if you return "melons".

>
>>
>> >
>> > Rationale
>> > =========
>> >
>> > Being able to pre-allocate lists based on the expected size, as
>> > estimated by
>> > ``__length_hint__``, can be a significant optimization. CPython has been
>> > observed to run some code faster than PyPy, purely because of this
>> > optimization
>> > being present.
>> >
>> > Open questions
>> > ==============
>> >
>> > There are two open questions for this PEP:
>> >
>> > * Should ``list`` expose a kwarg in it's constructor for supplying a
>> > length
>> >   hint.
>> > * Should a function be added either to ``builtins`` or some other module
>> > which
>> >   calls ``__length_hint__``, like ``builtins.len`` calls ``__len__``.
>>
>> Let's try to keep this as limited as possible for a public API.
>>
>
> Sounds reasonable to me!  Should we just go ahead and strip those out now?

Certainly.


-- 
Regards,
Benjamin

From tjreedy at udel.edu  Sun Jul 15 03:03:31 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Sat, 14 Jul 2012 21:03:31 -0400
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <loom.20120715T000919-641@post.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
Message-ID: <jtt4tr$qun$1@dough.gmane.org>

On 7/14/2012 6:11 PM, Alex Gaynor wrote:
...

Various thoughts:

"This method is then used by various other functions (such +as ``map``) 
to presize lists"
  -- map no longer produces lists. This only makes sense in 3.x if you 
mean that map can pass along the value of its inputs.

"Types can then define ``__length_hint__`` which are not
+sized, and thus should not define ``__len__``,"
is awkwardly phrased. I think you mean
"Types that are not sized and should not define __len__ can then define 
__length_hint__.

What do 'sized' and 'should' mean? Some iterators know exactly how many 
items they have yet to yield. The main implication of having a __len__ 
versus __length_hint__ methods seems to be it bool() value when empty.

If lists were to get a new keyword arg, so should the other classes 
based on one internal array. I see this has been removed.

Generator functions are the nicest way to define iterators in Python. 
Generator instances returned from generator functions cannot be given a 
length hint. They are not directly helped. However ...

Not addressed in the PEP: do consumers of __length_hint look for it (and 
__length__ before or after calling iter(input), or both? If before, then 
the following should work.

class gwlh: # generator with length hint
     def __init__(self, gen, len):
         self.gen = gen
         self.len = len
     def __iter__(self):
         return self.gen
     def __length_hint__(self):
         return len

Do transformation iterators pass through hints from inputs? Does map(f, 
iterable) look for len or hint on iterable? Ditto for some itertools, 
like chain (add lengths). Any guidelines in the PEP

-- 
Terry Jan Reedy




From ncoghlan at gmail.com  Sun Jul 15 07:16:02 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 15 Jul 2012 15:16:02 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
Message-ID: <CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>

On Sun, Jul 15, 2012 at 9:18 AM, Benjamin Peterson <benjamin at python.org> wrote:
>> Open questions
>> ==============
>>
>> There are two open questions for this PEP:
>>
>> * Should ``list`` expose a kwarg in it's constructor for supplying a length
>>   hint.
>> * Should a function be added either to ``builtins`` or some other module which
>>   calls ``__length_hint__``, like ``builtins.len`` calls ``__len__``.
>
> Let's try to keep this as limited as possible for a public API.

Length hints are very useful for *any* container implementation,
whether those containers are in the standard library or not. Just as
we exposed operator.index when __index__ was added, we should expose
an "operator.length_hint" function with the following semantics:

    def length_hint(obj):
        """Return an estimate of the number of items in obj. This is
useful for presizing containers when building from an iterable.

        If the object supports len(), the result will be exact.
Otherwise, it may over or underestimate by an arbitrary amount. The
        result will be an integer >= 0.
        """
        try:
            return len(obj)
        except TypeError:
            try:
                get_hint = obj.__length_hint__
            except AttributeError:
                return 0
            hint = get_hint()
            if not isinstance(hint, int):
                raise TypeError("Length hint must be an integer, not
%r" % type(hint))
            if hint < 0:
                raise ValueError("Length hint (%r) must be >= 0" % hint)
            return hint

There's no reason to make pure Python container implementations
reimplement all that for themselves.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From alex.gaynor at gmail.com  Sun Jul 15 07:20:03 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 14 Jul 2012 22:20:03 -0700
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
Message-ID: <CAFRnB2WoU=9XaFfT6ZCkCUcJjPfzva5pb-RGPvtGA5kh=LDO7Q@mail.gmail.com>

On Sat, Jul 14, 2012 at 10:16 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Sun, Jul 15, 2012 at 9:18 AM, Benjamin Peterson <benjamin at python.org>
> wrote:
> >> Open questions
> >> ==============
> >>
> >> There are two open questions for this PEP:
> >>
> >> * Should ``list`` expose a kwarg in it's constructor for supplying a
> length
> >>   hint.
> >> * Should a function be added either to ``builtins`` or some other
> module which
> >>   calls ``__length_hint__``, like ``builtins.len`` calls ``__len__``.
> >
> > Let's try to keep this as limited as possible for a public API.
>
> Length hints are very useful for *any* container implementation,
> whether those containers are in the standard library or not. Just as
> we exposed operator.index when __index__ was added, we should expose
> an "operator.length_hint" function with the following semantics:
>
>     def length_hint(obj):
>         """Return an estimate of the number of items in obj. This is
> useful for presizing containers when building from an iterable.
>
>         If the object supports len(), the result will be exact.
> Otherwise, it may over or underestimate by an arbitrary amount. The
>         result will be an integer >= 0.
>         """
>         try:
>             return len(obj)
>         except TypeError:
>             try:
>                 get_hint = obj.__length_hint__
>             except AttributeError:
>                 return 0
>             hint = get_hint()
>             if not isinstance(hint, int):
>                 raise TypeError("Length hint must be an integer, not
> %r" % type(hint))
>             if hint < 0:
>                 raise ValueError("Length hint (%r) must be >= 0" % hint)
>             return hint
>
> There's no reason to make pure Python container implementations
> reimplement all that for themselves.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>

Sounds reasonable to me, the only issue with your psuedocode (err... I mean
Python ;)), is that there's no way for the __lenght_hint__ to specify that
that particular instance can't have a length hint computed.  e.g. imagine
some sort of lazy stream that cached itself, and only wanted to offer a
length hint if it had already been evaluated.  Without an exception to
raise, it has to return whatever the magic value for length_hint is (in
your impl it appears to be 0, the current _PyObject_LengthHint method in
CPython has a required `default` parameter).  The PEP proposes using
TypeError for that.

Anyways that code looks good, do you want to add it to the PEP?

Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120714/37f52bd4/attachment.html>

From steve at pearwood.info  Sun Jul 15 10:21:13 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 15 Jul 2012 18:21:13 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
Message-ID: <50027D79.8010500@pearwood.info>

Nick Coghlan wrote:
> On Sun, Jul 15, 2012 at 9:18 AM, Benjamin Peterson <benjamin at python.org> wrote:
>>> Open questions
>>> ==============
>>>
>>> There are two open questions for this PEP:
>>>
>>> * Should ``list`` expose a kwarg in it's constructor for supplying a length
>>>   hint.
>>> * Should a function be added either to ``builtins`` or some other module which
>>>   calls ``__length_hint__``, like ``builtins.len`` calls ``__len__``.
>> Let's try to keep this as limited as possible for a public API.
> 
> Length hints are very useful for *any* container implementation,
> whether those containers are in the standard library or not. Just as
> we exposed operator.index when __index__ was added, we should expose
> an "operator.length_hint" function with the following semantics:
[...]

As given, length_hint gives no way of distinguishing between iterables and 
non-iterables:

py> length_hint([])
0
py> length_hint(42)
0

nor does it give iterable objects a way to indicate that either they don't 
know their length, or that they are infinite.

I suggest:

* object (and hence all other types that don't explicitly override it)
   should have a __length_hint__ that raises TypeError;

* __length_hint__ should be allowed to return None to indicate "don't know"
   or -1 to indicate "infinite".

Presumably anything that wishes to create a list or other sequence from an 
object with a hint of -1 could then raise an exception immediately.




-- 
Steven

From ncoghlan at gmail.com  Sun Jul 15 10:47:38 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 15 Jul 2012 18:47:38 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <50027D79.8010500@pearwood.info>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
Message-ID: <CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>

On Sun, Jul 15, 2012 at 6:21 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> I suggest:
>
> * object (and hence all other types that don't explicitly override it)
>   should have a __length_hint__ that raises TypeError;

We can keep it simpler than that just by changing the order of the checks.

> * __length_hint__ should be allowed to return None to indicate "don't know"
>   or -1 to indicate "infinite".
>
> Presumably anything that wishes to create a list or other sequence from an
> object with a hint of -1 could then raise an exception immediately.

I'm not seeing the value in returning None over 0 for the don't know
case - it just makes the API harder to use. Declaring negative results
as meaning "I'm infinite" sounds reasonable, though:

    def length_hint(obj):
        """Return an estimate of the number of items in obj.

        This is useful for presizing containers when building from an iterable.

        If the object supports len(), the result will be exact. Otherwise,
        it may over or underestimate by an arbitrary amount.
        """
        try:
            get_hint = obj.__length_hint__
        except AttributeError:
            return len(obj)
        hint = get_hint()
        if not isinstance(hint, int):
            msg = "Length hint must be an integer, not %r"
            raise TypeError(msg % type(hint))
        if hint < 0:
            raise ValueError("%r is an infinite iterator" % (obj,))
        return hint

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From stefan_ml at behnel.de  Sun Jul 15 11:11:50 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Sun, 15 Jul 2012 11:11:50 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CAFRnB2WoU=9XaFfT6ZCkCUcJjPfzva5pb-RGPvtGA5kh=LDO7Q@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<CAFRnB2WoU=9XaFfT6ZCkCUcJjPfzva5pb-RGPvtGA5kh=LDO7Q@mail.gmail.com>
Message-ID: <jtu1gl$qo3$1@dough.gmane.org>

Alex Gaynor, 15.07.2012 07:20:
> there's no way for the __lenght_hint__ to specify that
> that particular instance can't have a length hint computed.  e.g. imagine
> some sort of lazy stream that cached itself, and only wanted to offer a
> length hint if it had already been evaluated.  Without an exception to
> raise, it has to return whatever the magic value for length_hint is (in
> your impl it appears to be 0, the current _PyObject_LengthHint method in
> CPython has a required `default` parameter).  The PEP proposes using
> TypeError for that.

Yes, that's a major issue. I've been planning to add a length hint to
Cython's generator expressions for a while, but the problem is really that
in most cases it is only known at runtime if the underlying iterable has a
length hint, so propagating it needs a way to say "sorry, I thought I might
know, but I don't". It would be even better if this way was efficient.
Since we're at a point of making this an official protocol, why not change
the current behaviour and return -1 (or even just 0) to explicitly state
that "we don't know"?

The problem with an exception here is that it might have been raised
accidentally inside of the __length_hint__() implementation that is being
asked. Swallowing it just because it happened to be a TypeError rather than
something else may end up covering bugs. We had a similar issue with
hasattr() in the past.

Also, it would be nice if this became a type slot rather than requiring a
dict lookup and Python function call.

Stefan


From solipsis at pitrou.net  Sun Jul 15 14:36:41 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Jul 2012 14:36:41 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
Message-ID: <20120715143641.3d2bb83d@pitrou.net>

On Sun, 15 Jul 2012 18:47:38 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> > * __length_hint__ should be allowed to return None to indicate "don't know"
> >   or -1 to indicate "infinite".
> >
> > Presumably anything that wishes to create a list or other sequence from an
> > object with a hint of -1 could then raise an exception immediately.
> 
> I'm not seeing the value in returning None over 0 for the don't know
> case - it just makes the API harder to use.

The point is that 0 is a legitimate value for a length hint. Simple
implementations of __length_hint__ will start returning 0 as a
legitimate value and you will wrongly interpret that as "don't know",
which kinds of defeat the purpose of __length-hint__ ;)

That said, I don't think a special value for "is infinite" is useful.
Just make -1 mean "I don't know".

Regards

Antoine.



From solipsis at pitrou.net  Sun Jul 15 14:42:25 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Jul 2012 14:42:25 +0200
Subject: [Python-Dev] cpython: Take the first step in resolving the
 messy pkgutil vs importlib edge cases by
References: <3WZgR32c8lzPBc@mail.python.org>
Message-ID: <20120715144225.79966de4@pitrou.net>

On Sun, 15 Jul 2012 10:10:07 +0200 (CEST)
nick.coghlan <python-checkins at python.org> wrote:
>  
>  int
> +set_main_loader(PyObject *d, const char *filename, const char *loader_name)
> +{

This function should be static.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From solipsis at pitrou.net  Sun Jul 15 14:43:31 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Jul 2012 14:43:31 +0200
Subject: [Python-Dev] cpython: Actually initialize __main__.__loader__
 with loader instances, not the
References: <3WZhn613RWzPJM@mail.python.org>
Message-ID: <20120715144331.16d7461c@pitrou.net>

On Sun, 15 Jul 2012 11:10:50 +0200 (CEST)
nick.coghlan <python-checkins at python.org> wrote:
>      tstate = PyThreadState_GET();
>      interp = tstate->interp;
> -    loader = PyObject_GetAttrString(interp->importlib, loader_name);
> +    loader_type = PyObject_GetAttrString(interp->importlib, loader_name);
> +    if (loader_type == NULL) {
> +        return -1;
> +    }
> +    loader = PyObject_CallFunction(loader_type, "ss", "__main__", filename);

I think you may have a refleak on loader_type here.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From steve at pearwood.info  Sun Jul 15 15:47:07 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 15 Jul 2012 23:47:07 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <20120715143641.3d2bb83d@pitrou.net>
References: <loom.20120715T000919-641@post.gmane.org>	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>	<50027D79.8010500@pearwood.info>	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
Message-ID: <5002C9DB.2000409@pearwood.info>

Antoine Pitrou wrote:

> The point is that 0 is a legitimate value for a length hint. Simple
> implementations of __length_hint__ will start returning 0 as a
> legitimate value and you will wrongly interpret that as "don't know",
> which kinds of defeat the purpose of __length-hint__  ;) 

> That said, I don't think a special value for "is infinite" is useful.
> Just make -1 mean "I don't know".

You've obviously never accidentally called list on an infinite iterator *wink*

It's not the (eventual) MemoryError that is the problem. On some systems, this 
can cause the PC  to become unresponsive as the OS tries to free an 
ever-increasing amount of memory. Been there, done that, on a production 
system. I had to do a hard reboot to fix it.

I think having a hint that says "there's no way this can succeed, fail 
immediately" is more useful than caring about the difference between a hint of 
0 and a hint of 1.



-- 
Steven

From ncoghlan at gmail.com  Sun Jul 15 16:08:41 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Jul 2012 00:08:41 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <5002C9DB.2000409@pearwood.info>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
Message-ID: <CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>

Right, I agree on the value in being able to return something to say "this
cannot be converted to a concrete container".

I still haven't seen a use case where the appropriate response to "I don't
know" differs from the appropriate response to a hint of zero - that is,
you don't preallocate, you just start iterating.

Cheers,
Nick.

--
Sent from my phone, thus the relative brevity :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120716/071f1196/attachment.html>

From solipsis at pitrou.net  Sun Jul 15 16:22:32 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Jul 2012 16:22:32 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
Message-ID: <20120715162232.25c040b7@pitrou.net>

On Mon, 16 Jul 2012 00:08:41 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> Right, I agree on the value in being able to return something to say "this
> cannot be converted to a concrete container".

Who would be able to return that, apart from trivial cases like
itertools.cycle()?

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From lists at cheimes.de  Sun Jul 15 16:33:23 2012
From: lists at cheimes.de (Christian Heimes)
Date: Sun, 15 Jul 2012 16:33:23 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <20120715162232.25c040b7@pitrou.net>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<20120715162232.25c040b7@pitrou.net>
Message-ID: <jtukbj$jje$1@dough.gmane.org>

Am 15.07.2012 16:22, schrieb Antoine Pitrou:
> On Mon, 16 Jul 2012 00:08:41 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Right, I agree on the value in being able to return something to say "this
>> cannot be converted to a concrete container".
> 
> Who would be able to return that, apart from trivial cases like
> itertools.cycle()?

For example most numerical sequence iterators like Fibonacci generator,
prime number sequence generator and even trivial cases like even natural
number generator. IMO it's a good idea to have a notation for infinitive
iterators that can't be materialized as finite containers.

+1

Christian


From mark at hotpy.org  Sun Jul 15 16:39:07 2012
From: mark at hotpy.org (Mark Shannon)
Date: Sun, 15 Jul 2012 15:39:07 +0100
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>	<50027D79.8010500@pearwood.info>	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>	<20120715143641.3d2bb83d@pitrou.net>	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
Message-ID: <5002D60B.4000203@hotpy.org>

Nick Coghlan wrote:
> Right, I agree on the value in being able to return something to say 
> "this cannot be converted to a concrete container".
> 
> I still haven't seen a use case where the appropriate response to "I 
> don't know" differs from the appropriate response to a hint of zero - 
> that is, you don't preallocate, you just start iterating.
> 

There seem to be 5 possible classes values of __length_hint__ that an
iterator object can provide:

1. Don't implement it at all.

2. Implement __length_hint__() but don't want to return any value.
    Either raise an exception (TypeError) -- As suggested in the PEP.
    or return NotImplemented -- my preferred option.

3. Return a "don't know" value:
    Returning 0 would be fine for this, but the VM might want to respond
    differently to "don't know" and 0.
	__length_hint__() == 0             container should be minimum size.
         __length_hint__() == "unknown"     container starts at default size.

4. Infinite iterator:
    Could return float('inf'), but given this is a "hint" then
    returning sys.maxsize or sys.maxsize + 1 might be OK.
    Alternatively raise an OverflowError

5. A meaningful length. No problem :)

Also, what are the allowable return types?

1. int only
2. Any number (ie any type with a __int__() method)?
3. Or any integer-like object (ie a type with a __index__() method)?

My suggestion:

a) Don't want to return any value or "don't know": return NotImplemented
b) For infinite iterators: raise an OverflowError
c) All other cases: return an int or a type with a __index__() method.

Cheers,
Mark.


From mark at hotpy.org  Sun Jul 15 16:14:59 2012
From: mark at hotpy.org (Mark Shannon)
Date: Sun, 15 Jul 2012 15:14:59 +0100
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <loom.20120715T000919-641@post.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
Message-ID: <5002D063.1060401@hotpy.org>

Alex Gaynor wrote:
> Hi all,
> 
> I've just submitted a PEP proposing making __length_hint__ a public API for 
> users to define and other VMs to implement:

These seems back-to-front.
__length_hint__ is *used* by the VM, not provided by it.
It should be part of the object model, rather than the API.

> 
> PEP: 424
> Title: A method for exposing a length hint
> Version: $Revision$
> Last-Modified: $Date
> Author: Alex Gaynor <alex.gaynor at gmail.com>
> Status: Draft
> Type: Standards Track
> Content-Type: text/x-rst
> Created: 14-July-2012
> Python-Version: 3.4
> 
> Abstract
> ========
> 
> CPython currently defines an ``__length_hint__`` method on several types, such
> as various iterators. This method is then used by various other functions (such 
> as
> ``map``) to presize lists based on the estimated returned by

Don't use "map" as an example.
map returns an iterator so it doesn't need __length_hint__

> ``__length_hint__``. Types can then define ``__length_hint__`` which are not
> sized, and thus should not define ``__len__``, but can estimate or compute a
> size (such as many iterators).
> 
> Proposal
> ========
> 
> This PEP proposes formally documenting ``__length_hint__`` for other
> interpreter and non-standard library Python to implement.
> 
> ``__length_hint__`` must return an integer, and is not required to be accurate.
> It may return a value that is either larger or smaller than the actual size of
> the container. It may raise a ``TypeError`` if a specific instance cannot have
> its length estimated. It may not return a negative value.

Rather than raising a TypeError, why not return NotImplemented?

> 
> Rationale
> =========
> 
> Being able to pre-allocate lists based on the expected size, as estimated by 
> ``__length_hint__``, can be a significant optimization. CPython has been
> observed to run some code faster than PyPy, purely because of this optimization
> being present.
> 
> Open questions
> ==============
> 
> There are two open questions for this PEP:
> 
> * Should ``list`` expose a kwarg in it's constructor for supplying a length
>   hint.
> * Should a function be added either to ``builtins`` or some other module which
>   calls ``__length_hint__``, like ``builtins.len`` calls ``__len__``.
> 
> Copyright
> =========
> 
> This document has been placed into the public domain.
> 
> ..
>     Local Variables:
>     mode: indented-text
>     indent-tabs-mode: nil
>     sentence-end-double-space: t
>     fill-column: 70
>     coding: utf-8
> 
> 
> 
> 
> Alex
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/mark%40hotpy.org


From brett at python.org  Sun Jul 15 16:47:09 2012
From: brett at python.org (Brett Cannon)
Date: Sun, 15 Jul 2012 10:47:09 -0400
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <5002D60B.4000203@hotpy.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net> <5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<5002D60B.4000203@hotpy.org>
Message-ID: <CAP1=2W7xS-gieivi0s3d514b8g2g_TJiWNXx8M96JmKJ4G9PNA@mail.gmail.com>

On Sun, Jul 15, 2012 at 10:39 AM, Mark Shannon <mark at hotpy.org> wrote:

> Nick Coghlan wrote:
>
>> Right, I agree on the value in being able to return something to say
>> "this cannot be converted to a concrete container".
>>
>> I still haven't seen a use case where the appropriate response to "I
>> don't know" differs from the appropriate response to a hint of zero - that
>> is, you don't preallocate, you just start iterating.
>>
>>
> There seem to be 5 possible classes values of __length_hint__ that an
> iterator object can provide:
>
> 1. Don't implement it at all.
>
> 2. Implement __length_hint__() but don't want to return any value.
>    Either raise an exception (TypeError) -- As suggested in the PEP.
>    or return NotImplemented -- my preferred option.
>
> 3. Return a "don't know" value:
>    Returning 0 would be fine for this, but the VM might want to respond
>    differently to "don't know" and 0.
>         __length_hint__() == 0             container should be minimum
> size.
>         __length_hint__() == "unknown"     container starts at default
> size.


> 4. Infinite iterator:
>    Could return float('inf'), but given this is a "hint" then
>    returning sys.maxsize or sys.maxsize + 1 might be OK.
>    Alternatively raise an OverflowError
>

I am really having a hard time differentiating infinity with "I don't know"
since they are both accurate from the point of view of __length_hint__ and
its typical purpose of allocation. You have no clue how many values will be
grabbed from an infinite iterator, so it's the same as just not knowing
upfront how long the iterator will be, infinite or not, and thus not worth
distinguishing.


>
> 5. A meaningful length. No problem :)
>
> Also, what are the allowable return types?
>
> 1. int only
> 2. Any number (ie any type with a __int__() method)?
> 3. Or any integer-like object (ie a type with a __index__() method)?
>
> My suggestion:
>
> a) Don't want to return any value or "don't know": return NotImplemented
> b) For infinite iterators: raise an OverflowError
> c) All other cases: return an int or a type with a __index__() method.
>

I'm fine with (a), drop (b), and for (c) use what we allow for __len__()
since, as Nick's operator.length_hint pseudo-code suggests, people will
call this as a fallback if __len__ isn't defined.

-Brett



>
> Cheers,
> Mark.
>
>
> ______________________________**_________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/**mailman/listinfo/python-dev<http://mail.python.org/mailman/listinfo/python-dev>
> Unsubscribe: http://mail.python.org/**mailman/options/python-dev/**
> brett%40python.org<http://mail.python.org/mailman/options/python-dev/brett%40python.org>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120715/34922c83/attachment.html>

From lists at cheimes.de  Sun Jul 15 16:56:39 2012
From: lists at cheimes.de (Christian Heimes)
Date: Sun, 15 Jul 2012 16:56:39 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <5002D60B.4000203@hotpy.org>
References: <loom.20120715T000919-641@post.gmane.org>	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>	<50027D79.8010500@pearwood.info>	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>	<20120715143641.3d2bb83d@pitrou.net>	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<5002D60B.4000203@hotpy.org>
Message-ID: <jtuln7$sua$1@dough.gmane.org>

Am 15.07.2012 16:39, schrieb Mark Shannon:
> 1. Don't implement it at all.
> 
> 2. Implement __length_hint__() but don't want to return any value.
>    Either raise an exception (TypeError) -- As suggested in the PEP.
>    or return NotImplemented -- my preferred option.

How is this different from "don't know"? What's the use case for knowing
that the object doesn't want to say anything or doesn't know its
possible length.

> 3. Return a "don't know" value:
>    Returning 0 would be fine for this, but the VM might want to respond
>    differently to "don't know" and 0.

How about None? It's the logical choice, simple and easy to test for in
Python and C code.

0 is a valid number for "I know that's I'll return nothing".

> 4. Infinite iterator:
>    Could return float('inf'), but given this is a "hint" then
>    returning sys.maxsize or sys.maxsize + 1 might be OK.
>    Alternatively raise an OverflowError

Too complex, hard to remember and even harder to check for. Since a
length is always positive or zero, -1 is a good return value for infinite.

> a) Don't want to return any value or "don't know": return NotImplemented

+1

> b) For infinite iterators: raise an OverflowError

-1, I'm for -1. ;) I'm not a fan of using exception for valid and
correct return values.

> c) All other cases: return an int or a type with a __index__() method.

+1

Christian


From mark at hotpy.org  Sun Jul 15 17:08:00 2012
From: mark at hotpy.org (Mark Shannon)
Date: Sun, 15 Jul 2012 16:08:00 +0100
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CAP1=2W7xS-gieivi0s3d514b8g2g_TJiWNXx8M96JmKJ4G9PNA@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<5002D60B.4000203@hotpy.org>
	<CAP1=2W7xS-gieivi0s3d514b8g2g_TJiWNXx8M96JmKJ4G9PNA@mail.gmail.com>
Message-ID: <5002DCD0.6040107@hotpy.org>

Brett Cannon wrote:
> 
> 
> On Sun, Jul 15, 2012 at 10:39 AM, Mark Shannon <mark at hotpy.org 
> <mailto:mark at hotpy.org>> wrote:
> 
>     Nick Coghlan wrote:
> 
>         Right, I agree on the value in being able to return something to
>         say "this cannot be converted to a concrete container".
> 
>         I still haven't seen a use case where the appropriate response
>         to "I don't know" differs from the appropriate response to a
>         hint of zero - that is, you don't preallocate, you just start
>         iterating.
> 
> 
>     There seem to be 5 possible classes values of __length_hint__ that an
>     iterator object can provide:
> 
>     1. Don't implement it at all.
> 
>     2. Implement __length_hint__() but don't want to return any value.
>        Either raise an exception (TypeError) -- As suggested in the PEP.
>        or return NotImplemented -- my preferred option.
> 
>     3. Return a "don't know" value:
>        Returning 0 would be fine for this, but the VM might want to respond
>        differently to "don't know" and 0.
>             __length_hint__() == 0             container should be
>     minimum size.
>             __length_hint__() == "unknown"     container starts at
>     default size.
> 
> 
>     4. Infinite iterator:
>        Could return float('inf'), but given this is a "hint" then
>        returning sys.maxsize or sys.maxsize + 1 might be OK.
>        Alternatively raise an OverflowError
> 
> 
> I am really having a hard time differentiating infinity with "I don't 
> know" since they are both accurate from the point of view of 
> __length_hint__ and its typical purpose of allocation. You have no clue 
> how many values will be grabbed from an infinite iterator, so it's the 
> same as just not knowing upfront how long the iterator will be, infinite 
> or not, and thus not worth distinguishing.
>  
> 
> 
>     5. A meaningful length. No problem :)
> 
>     Also, what are the allowable return types?
> 
>     1. int only
>     2. Any number (ie any type with a __int__() method)?
>     3. Or any integer-like object (ie a type with a __index__() method)?
> 
>     My suggestion:
> 
>     a) Don't want to return any value or "don't know": return NotImplemented
>     b) For infinite iterators: raise an OverflowError
>     c) All other cases: return an int or a type with a __index__() method.
> 
> 
> I'm fine with (a), drop (b), and for (c) use what we allow for __len__() 
> since, as Nick's operator.length_hint pseudo-code suggests, people will 
> call this as a fallback if __len__ isn't defined.

So how does an iterator express infinite length?

What should happen if I am silly enough to do this:
 >>> list(itertools.count())

This will fail; it should fail quickly.

Cheers,
Mark.

From solipsis at pitrou.net  Sun Jul 15 17:06:28 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Jul 2012 17:06:28 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<20120715162232.25c040b7@pitrou.net> <jtukbj$jje$1@dough.gmane.org>
Message-ID: <20120715170628.1f6fcc04@pitrou.net>

On Sun, 15 Jul 2012 16:33:23 +0200
Christian Heimes <lists at cheimes.de> wrote:
> Am 15.07.2012 16:22, schrieb Antoine Pitrou:
> > On Mon, 16 Jul 2012 00:08:41 +1000
> > Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> Right, I agree on the value in being able to return something to say "this
> >> cannot be converted to a concrete container".
> > 
> > Who would be able to return that, apart from trivial cases like
> > itertools.cycle()?
> 
> For example most numerical sequence iterators like Fibonacci generator,
> prime number sequence generator and even trivial cases like even natural
> number generator.

First, you can't implement __length_hint__ for a generator, which is the
preferred (the most practical) way of writing iterators in pure Python.

Second, not all iterators will implement __length_hint__ (because it's
optional and, really, of rather little use). So, as a user, you cannot
hope that `list(some_iterator)` will always raise instead of filling
your memory with an infinite stream of values: you have to be careful
anyway.

Even if __length_hint__ is implemented, its result may be wrong.
That's the whole point: it's a *hint*; an iterator might tell you it's
finite while it's infinite, or the reverse.


My conclusion is that an infinite iterator is a documentation issue.
Just tell the user that it doesn't stop, and let them shoot themselves
in the foot in they want to.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From solipsis at pitrou.net  Sun Jul 15 17:14:30 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Jul 2012 17:14:30 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<5002D60B.4000203@hotpy.org>
	<CAP1=2W7xS-gieivi0s3d514b8g2g_TJiWNXx8M96JmKJ4G9PNA@mail.gmail.com>
	<5002DCD0.6040107@hotpy.org>
Message-ID: <20120715171430.0b1b1ddc@pitrou.net>

On Sun, 15 Jul 2012 16:08:00 +0100
Mark Shannon <mark at hotpy.org> wrote:
> 
> What should happen if I am silly enough to do this:
>  >>> list(itertools.count())
> 
> This will fail; it should fail quickly.

Why should it? AFAIK it's not a common complaint. You said it yourself:
it's a silly thing to do.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From alexandre.zani at gmail.com  Sun Jul 15 17:38:20 2012
From: alexandre.zani at gmail.com (Alexandre Zani)
Date: Sun, 15 Jul 2012 08:38:20 -0700
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <5002DCD0.6040107@hotpy.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net> <5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<5002D60B.4000203@hotpy.org>
	<CAP1=2W7xS-gieivi0s3d514b8g2g_TJiWNXx8M96JmKJ4G9PNA@mail.gmail.com>
	<5002DCD0.6040107@hotpy.org>
Message-ID: <CAJVMportDg_Zd-iaUssdj_mxtU9HP6OBkpjW7Yv6_uq__uypBg@mail.gmail.com>

On Sun, Jul 15, 2012 at 8:08 AM, Mark Shannon <mark at hotpy.org> wrote:
> Brett Cannon wrote:
>
>>
>>
>> On Sun, Jul 15, 2012 at 10:39 AM, Mark Shannon <mark at hotpy.org
>> <mailto:mark at hotpy.org>> wrote:
>>
>>     Nick Coghlan wrote:
>>
>>         Right, I agree on the value in being able to return something to
>>         say "this cannot be converted to a concrete container".
>>
>>         I still haven't seen a use case where the appropriate response
>>         to "I don't know" differs from the appropriate response to a
>>         hint of zero - that is, you don't preallocate, you just start
>>         iterating.
>>
>>
>>     There seem to be 5 possible classes values of __length_hint__ that an
>>     iterator object can provide:
>>
>>     1. Don't implement it at all.
>>
>>     2. Implement __length_hint__() but don't want to return any value.
>>        Either raise an exception (TypeError) -- As suggested in the PEP.
>>        or return NotImplemented -- my preferred option.
>>
>>     3. Return a "don't know" value:
>>        Returning 0 would be fine for this, but the VM might want to
>> respond
>>        differently to "don't know" and 0.
>>             __length_hint__() == 0             container should be
>>     minimum size.
>>             __length_hint__() == "unknown"     container starts at
>>     default size.
>>
>>
>>     4. Infinite iterator:
>>        Could return float('inf'), but given this is a "hint" then
>>        returning sys.maxsize or sys.maxsize + 1 might be OK.
>>        Alternatively raise an OverflowError
>>
>>
>> I am really having a hard time differentiating infinity with "I don't
>> know" since they are both accurate from the point of view of __length_hint__
>> and its typical purpose of allocation. You have no clue how many values will
>> be grabbed from an infinite iterator, so it's the same as just not knowing
>> upfront how long the iterator will be, infinite or not, and thus not worth
>> distinguishing.
>>
>>
>>     5. A meaningful length. No problem :)
>>
>>     Also, what are the allowable return types?
>>
>>     1. int only
>>     2. Any number (ie any type with a __int__() method)?
>>     3. Or any integer-like object (ie a type with a __index__() method)?
>>
>>     My suggestion:
>>
>>     a) Don't want to return any value or "don't know": return
>> NotImplemented
>>     b) For infinite iterators: raise an OverflowError
>>     c) All other cases: return an int or a type with a __index__() method.
>>
>>
>> I'm fine with (a), drop (b), and for (c) use what we allow for __len__()
>> since, as Nick's operator.length_hint pseudo-code suggests, people will call
>> this as a fallback if __len__ isn't defined.
>
>
> So how does an iterator express infinite length?
>
> What should happen if I am silly enough to do this:
>>>> list(itertools.count())
>
> This will fail; it should fail quickly.
>
>
> Cheers,
> Mark.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/alexandre.zani%40gmail.com

The PEP so far says: "It may raise a ``TypeError`` if a specific
instance cannot have
its length estimated." In many ways, "I don't know" is the same as
this "specific instance cannot have its length estimated". Why not
just raise a TypeError?

Also, regarding the code Nick posted above, I'm a little concerned
about calling len as the first thing to try. That means that if I
implement both __len__ and __len_hint__ (perhaps because __len__ is
very expensive) __len_hint__ will never be used. It's relatively easy
to say:

try:
  hint = len_hint(l)
except TypeError:
  hint = len(l)

From steve at pearwood.info  Sun Jul 15 17:55:09 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 16 Jul 2012 01:55:09 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <5002DCD0.6040107@hotpy.org>
References: <loom.20120715T000919-641@post.gmane.org>	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>	<50027D79.8010500@pearwood.info>	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>	<20120715143641.3d2bb83d@pitrou.net>	<5002C9DB.2000409@pearwood.info>	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>	<5002D60B.4000203@hotpy.org>	<CAP1=2W7xS-gieivi0s3d514b8g2g_TJiWNXx8M96JmKJ4G9PNA@mail.gmail.com>
	<5002DCD0.6040107@hotpy.org>
Message-ID: <5002E7DD.7030400@pearwood.info>

Mark Shannon wrote:

> So how does an iterator express infinite length?

The suggestion was it should return -1.


> What should happen if I am silly enough to do this:
>  >>> list(itertools.count())
> 
> This will fail; it should fail quickly.

That depends on your OS. I've just tested it now on Linux Mint, and the Python 
process was terminated within seconds.

I've also inadvertently done it on a Fedora system, which became completely 
unresponsive to user-input (including ctrl-alt-delete) within a few minutes. I 
let it run overnight (16 hours) before literally pulling the plug.

(I expect the difference in behaviour is due to the default ulimit under 
Debian/Mint and RedHat/Fedora systems.)

Ignoring OS-specific features, the promise[1] of the language is that list 
will try to allocate enough space for every item yielded by the iterator, or 
fail with a MemoryError. No promise is made as to how long that will take: it 
could take hours, or days, depending on how badly memory allocation 
performance drops when faced with unreasonably large requests. You can't 
expect it to fail either quickly or with an exception.

With a length hint, we could strengthen that promise:

"if __length_hint__ returns a negative number, list, tuple and set will fail 
immediately with MemoryError"

which I think is a good safety feature for some things which cannot possibly 
succeed, but risk DOSing your system. Does it prevent every possible failure 
mode? No, of course not. But just because you can't prevent *every* problem 
doesn't mean you should prevent the ones which you can.


[1] I think. I'm sure I read this somewhere in the docs, but I can't find it now.


-- 
Steven

From rosuav at gmail.com  Sun Jul 15 18:00:58 2012
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 16 Jul 2012 02:00:58 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <5002E7DD.7030400@pearwood.info>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<5002D60B.4000203@hotpy.org>
	<CAP1=2W7xS-gieivi0s3d514b8g2g_TJiWNXx8M96JmKJ4G9PNA@mail.gmail.com>
	<5002DCD0.6040107@hotpy.org> <5002E7DD.7030400@pearwood.info>
Message-ID: <CAPTjJmogOtbjUjt2P8xxwbzLjBcRUkJ_u209DcKgkemf01sTrA@mail.gmail.com>

On Mon, Jul 16, 2012 at 1:55 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> (I expect the difference in behaviour is due to the default ulimit under
> Debian/Mint and RedHat/Fedora systems.)

Possibly also virtual memory settings. Allocating gobs of memory with
a huge page file slows everything down without raising an error.

And since it's possible to have non-infinite but ridiculous-sized
iterators, I'd not bother putting too much effort into protecting
infinite iterators - although the "huge but not infinite" case is,
admittedly, rather rarer than either "reasonable-sized" or "actually
infinite".

ChrisA

From steve at pearwood.info  Sun Jul 15 18:21:20 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 16 Jul 2012 02:21:20 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <20120715170628.1f6fcc04@pitrou.net>
References: <loom.20120715T000919-641@post.gmane.org>	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>	<50027D79.8010500@pearwood.info>	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>	<20120715143641.3d2bb83d@pitrou.net>	<5002C9DB.2000409@pearwood.info>	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>	<20120715162232.25c040b7@pitrou.net>
	<jtukbj$jje$1@dough.gmane.org> <20120715170628.1f6fcc04@pitrou.net>
Message-ID: <5002EE00.7000305@pearwood.info>

Antoine Pitrou wrote:

> First, you can't implement __length_hint__ for a generator, which is the
> preferred (the most practical) way of writing iterators in pure Python.

Limitations of generators are no reason for not improving iterators which are 
not generators. __length_hint__ already exists; this proposal simply proposes 
making it documented and officially supported.

py> iter([]).__length_hint__
<built-in method __length_hint__ of list_iterator object at 0xb7bcf98c>


> Even if __length_hint__ is implemented, its result may be wrong.
> That's the whole point: it's a *hint*; an iterator might tell you it's
> finite while it's infinite, or the reverse.

If it claims to be infinite, I see no reason to disbelieve it on the 
off-chance that it is actually both finite and small enough to fit into memory 
  without crashing my system. If it claims to be finite, but is actually 
infinite, well that's not much of a hint, is it? There's an implied promise 
that the hint will be close to the real value, not infinitely distant.


> My conclusion is that an infinite iterator is a documentation issue.
> Just tell the user that it doesn't stop, and let them shoot themselves
> in the foot in they want to.

Buffer overflows are a documentation issue. Just tell the user not to 
overwrite memory they don't mean to, and let them shoot themselves in the foot 
if they want.

*wink*



-- 
Steven


From solipsis at pitrou.net  Sun Jul 15 18:19:47 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Jul 2012 18:19:47 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<5002D60B.4000203@hotpy.org>
	<CAP1=2W7xS-gieivi0s3d514b8g2g_TJiWNXx8M96JmKJ4G9PNA@mail.gmail.com>
	<5002DCD0.6040107@hotpy.org> <5002E7DD.7030400@pearwood.info>
	<CAPTjJmogOtbjUjt2P8xxwbzLjBcRUkJ_u209DcKgkemf01sTrA@mail.gmail.com>
Message-ID: <20120715181947.5c3a06e4@pitrou.net>

On Mon, 16 Jul 2012 02:00:58 +1000
Chris Angelico <rosuav at gmail.com> wrote:
> On Mon, Jul 16, 2012 at 1:55 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> > (I expect the difference in behaviour is due to the default ulimit under
> > Debian/Mint and RedHat/Fedora systems.)
> 
> Possibly also virtual memory settings. Allocating gobs of memory with
> a huge page file slows everything down without raising an error.
> 
> And since it's possible to have non-infinite but ridiculous-sized
> iterators, I'd not bother putting too much effort into protecting
> infinite iterators - although the "huge but not infinite" case is,
> admittedly, rather rarer than either "reasonable-sized" or "actually
> infinite".

In the real world, I'm sure "huge but not infinite" is much more
frequent than "actually infinite". Trying to list() an infinite
iterator is a programming error, so it shouldn't end up in production
code. However, data that grows bigger than expected (or that gets
disposed of too late) is quite a common thing.

<hint>
When hg.python.org died of OOM two weeks ago, it wasn't because of an
infinite iterator:
http://mail.python.org/pipermail/python-committers/2012-July/002084.html
</hint>

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From solipsis at pitrou.net  Sun Jul 15 18:26:19 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 15 Jul 2012 18:26:19 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<20120715162232.25c040b7@pitrou.net> <jtukbj$jje$1@dough.gmane.org>
	<20120715170628.1f6fcc04@pitrou.net>
	<5002EE00.7000305@pearwood.info>
Message-ID: <20120715182619.201e6d1f@pitrou.net>

On Mon, 16 Jul 2012 02:21:20 +1000
Steven D'Aprano <steve at pearwood.info> wrote:
> 
> > My conclusion is that an infinite iterator is a documentation issue.
> > Just tell the user that it doesn't stop, and let them shoot themselves
> > in the foot in they want to.
> 
> Buffer overflows are a documentation issue. Just tell the user not to 
> overwrite memory they don't mean to, and let them shoot themselves in the foot 
> if they want.

No, buffer overflows are bugs and they get fixed.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From steve at pearwood.info  Sun Jul 15 18:39:01 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 16 Jul 2012 02:39:01 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <5002E7DD.7030400@pearwood.info>
References: <loom.20120715T000919-641@post.gmane.org>	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>	<50027D79.8010500@pearwood.info>	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>	<20120715143641.3d2bb83d@pitrou.net>	<5002C9DB.2000409@pearwood.info>	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>	<5002D60B.4000203@hotpy.org>	<CAP1=2W7xS-gieivi0s3d514b8g2g_TJiWNXx8M96JmKJ4G9PNA@mail.gmail.com>	<5002DCD0.6040107@hotpy.org>
	<5002E7DD.7030400@pearwood.info>
Message-ID: <5002F225.4080806@pearwood.info>

Steven D'Aprano wrote:

> With a length hint, we could strengthen that promise:
> 
> "if __length_hint__ returns a negative number, list, tuple and set will 
> fail immediately with MemoryError"
> 
> which I think is a good safety feature for some things which cannot 
> possibly succeed, but risk DOSing your system. Does it prevent every 
> possible failure mode? No, of course not. But just because you can't 
> prevent *every* problem doesn't mean you should prevent the ones which 
> you can.

Gah, I messed that last sentence up. It should read:

just because you can't prevent *every* problem doesn't mean you SHOULDN'T 
prevent the ones which you can.


-- 
Steven


From stephen at xemacs.org  Mon Jul 16 05:52:18 2012
From: stephen at xemacs.org (Stephen J. Turnbull)
Date: Mon, 16 Jul 2012 12:52:18 +0900
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
Message-ID: <87zk702wi5.fsf@uwakimon.sk.tsukuba.ac.jp>

Nick Coghlan writes:

 > Right, I agree on the value in being able to return something to say "this
 > cannot be converted to a concrete container".
 > 
 > I still haven't seen a use case where the appropriate response to "I don't
 > know" differs from the appropriate response to a hint of zero - that is,
 > you don't preallocate, you just start iterating.

Why wouldn't one just believe the hint and jump past the iteration?

What about an alternative API such as length_hint(iter, bound)
returning 'cannot say' (if no hint is available), 'small' (if the
estimated length is less than bound), and 'large' (if it's greater
than the bound or infinite)?  (Or None, True, False which would give
the boolean interpretation "do I know I'm small enough to be converted
to a concrete container?")

The point is that I don't really see the value in returning a precise
estimate that cannot be relied on to be accurate.  OK, Python is a
"consenting adults" language, but returning an integer here seems like
invitation to abuse.

From ncoghlan at gmail.com  Mon Jul 16 06:20:17 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Jul 2012 14:20:17 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <87zk702wi5.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<87zk702wi5.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7c8aJuF3qb2yDQAsp9Nivhx91MpBod7Yb0ZjPWspmqL4A@mail.gmail.com>

On Jul 16, 2012 1:52 PM, "Stephen J. Turnbull" <stephen at xemacs.org> wrote:
> The point is that I don't really see the value in returning a precise
> estimate that cannot be relied on to be accurate.  OK, Python is a
> "consenting adults" language, but returning an integer here seems like
> invitation to abuse.

Because preallocating memory is ridiculously faster than doing multiple
resizes. That's all this API is for: how many objects should a container
constructor preallocate space for when building from an iterable. It's an
important optimisation in CPython when using itertools, and PyPy is
planning to adopt it as well. Alex is doing the right thing in attempting
to standardise it rather than risk the two implementations using subtly
incompatible definitions.

Skipping the iteration in the zero case is a pointless micro-optimisation
that just makes the API more complex for no good reason. Allowing a
negative hint to mean "infinite", on the other hand, avoids certain
categories of errors without making the API any harder to use (since
negatives have to be rejected anyway).

--
Sent from my phone, thus the relative brevity :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120716/79bf6cc7/attachment.html>

From steve at pearwood.info  Mon Jul 16 06:22:36 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Mon, 16 Jul 2012 14:22:36 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <87zk702wi5.fsf@uwakimon.sk.tsukuba.ac.jp>
References: <loom.20120715T000919-641@post.gmane.org>	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>	<50027D79.8010500@pearwood.info>	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>	<20120715143641.3d2bb83d@pitrou.net>	<5002C9DB.2000409@pearwood.info>	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<87zk702wi5.fsf@uwakimon.sk.tsukuba.ac.jp>
Message-ID: <5003970C.6040606@pearwood.info>

Stephen J. Turnbull wrote:

> The point is that I don't really see the value in returning a precise
> estimate that cannot be relied on to be accurate.  OK, Python is a
> "consenting adults" language, but returning an integer here seems like
> invitation to abuse.

Since __length_hint__ already exists and is already used, we should probably 
hear from somebody who knows how it is used and what problems and/or benefits 
it leads to.



-- 
Steven

From stefan_ml at behnel.de  Mon Jul 16 08:54:27 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Jul 2012 08:54:27 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <5002D063.1060401@hotpy.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<5002D063.1060401@hotpy.org>
Message-ID: <ju0dr4$5kb$1@dough.gmane.org>

Mark Shannon, 15.07.2012 16:14:
> Alex Gaynor wrote:
>> CPython currently defines an ``__length_hint__`` method on several types,
>> such
>> as various iterators. This method is then used by various other functions
>> (such as ``map``) to presize lists based on the estimated returned by
> 
> Don't use "map" as an example.
> map returns an iterator so it doesn't need __length_hint__

Right. It's a good example for something else, though. As I mentioned
before, iterators should be able to propagate the length hint of an
underlying iterator, e.g. in generator expressions or map(). I consider
that an important feature that the protocol must support.

Stefan


From stefan_ml at behnel.de  Mon Jul 16 09:10:45 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Jul 2012 09:10:45 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <20120715170628.1f6fcc04@pitrou.net>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<20120715162232.25c040b7@pitrou.net> <jtukbj$jje$1@dough.gmane.org>
	<20120715170628.1f6fcc04@pitrou.net>
Message-ID: <ju0epl$bu4$1@dough.gmane.org>

Antoine Pitrou, 15.07.2012 17:06:
> On Sun, 15 Jul 2012 16:33:23 +0200
> Christian Heimes wrote:
>> Am 15.07.2012 16:22, schrieb Antoine Pitrou:
>>> On Mon, 16 Jul 2012 00:08:41 +1000
>>> Nick Coghlan wrote:
>>>> Right, I agree on the value in being able to return something to say "this
>>>> cannot be converted to a concrete container".
>>>
>>> Who would be able to return that, apart from trivial cases like
>>> itertools.cycle()?
>>
>> For example most numerical sequence iterators like Fibonacci generator,
>> prime number sequence generator and even trivial cases like even natural
>> number generator.
> 
> First, you can't implement __length_hint__ for a generator, which is the
> preferred (the most practical) way of writing iterators in pure Python.

It can be implemented for generator expressions without a conditional,
though, including the case of comprehensions. I wanted to do this in Cython
for a while, but the protocol wasn't very well adapted to that use case.
The "don't know" case was just too common and inefficient.

For the other points, I agree with the already presented counterarguments.
Being able to prevent some obvious traps is a good thing, even if you can't
prevent all of them.

Stefan


From stefan_ml at behnel.de  Mon Jul 16 09:36:57 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Jul 2012 09:36:57 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <loom.20120715T000919-641@post.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
Message-ID: <ju0gaq$mci$1@dough.gmane.org>

Alex Gaynor, 15.07.2012 00:11:
> CPython currently defines an ``__length_hint__`` method on several types, such
> as various iterators. This method is then used by various other functions (such 
> as ``map``) to presize lists based on the estimated returned by
> ``__length_hint__``. Types can then define ``__length_hint__`` which are not
> sized, and thus should not define ``__len__``, but can estimate or compute a
> size (such as many iterators).
> 
> Proposal
> ========
> 
> This PEP proposes formally documenting ``__length_hint__`` for other
> interpreter and non-standard library Python to implement.
> 
> ``__length_hint__`` must return an integer, and is not required to be accurate.
> It may return a value that is either larger or smaller than the actual size of
> the container. It may raise a ``TypeError`` if a specific instance cannot have
> its length estimated. It may not return a negative value.

I'd like to more visibly repeat my suggestion to make this a slot method
"tp_length_hint()" in extension types that returns a Py_ssize_t.

That suggests that a negative return value would have a special meaning
instead of relying on return values like NotImplemented. The Python wrapper
of that slot method could still implement a mapping for this.

Return values could be -1 for "don't know" and -2 for "infinite" at the C
level, and NotImplemented for "don't know" at the Python level. Not sure
about a good Python value for "infinite".

Maybe return -1 for "infinite" at both levels and -2/NotImplemented for
"don't know" in C/Python? That would suggest -3 to propagate exceptions at
the C level.

Stefan


From ncoghlan at gmail.com  Mon Jul 16 10:26:11 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Jul 2012 18:26:11 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <ju0gaq$mci$1@dough.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<ju0gaq$mci$1@dough.gmane.org>
Message-ID: <CADiSq7fYD8vACAJ-B2yNpoBo3FNFmE_juF3ZRN+Akxi0zeUECQ@mail.gmail.com>

Proposing anything substantially more complicated than what is currently
implemented in CPython will just get the idea rejected. The substantial
immediate gain for PyPy is in skipping the memory resizing when building
containers from itertools iterators, not anywhere else.

--
Sent from my phone, thus the relative brevity :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120716/38854e40/attachment.html>

From mark at hotpy.org  Mon Jul 16 10:37:33 2012
From: mark at hotpy.org (Mark Shannon)
Date: Mon, 16 Jul 2012 09:37:33 +0100
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424: A
 method for exposing a length hint)
In-Reply-To: <loom.20120715T000919-641@post.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
Message-ID: <5003D2CD.2030903@hotpy.org>

To quote from PEP 424:

> Rationale
> =========
> 
> Being able to pre-allocate lists based on the expected size, as estimated by 
> ``__length_hint__``, can be a significant optimization. CPython has been
> observed to run some code faster than PyPy, purely because of this optimization
> being present.
> 

Why is it a significant optimisation?

How much slower is it?
Where is the data?

*If* resizing list is so slow, then why not make it faster?

To quote wikipedia (http://en.wikipedia.org/wiki/Dynamic_array)

"""
As n elements are inserted, the capacities form a geometric progression.
Expanding the array by any constant proportion ensures that inserting n elements
takes O(n) time overall, meaning that each insertion takes amortized constant time.
The value of this proportion a leads to a time-space tradeoff: the average time per
insertion operation is about a/(a-1), while the number of wasted cells is bounded
above by (a-1)n. The choice of a depends on the library or application: some textbooks
use a = 2, but Java's ArrayList implementation uses a = 3/2 and the C implementation
of Python's list data structure uses a = 9/8.
"""

If resizing of lists is too slow, then we should reconsider the 9/8 factor
and/or look to tweak the resizing code.

Cheers,
Mark.


From stefan_ml at behnel.de  Mon Jul 16 10:53:54 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Jul 2012 10:53:54 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CADiSq7fYD8vACAJ-B2yNpoBo3FNFmE_juF3ZRN+Akxi0zeUECQ@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<ju0gaq$mci$1@dough.gmane.org>
	<CADiSq7fYD8vACAJ-B2yNpoBo3FNFmE_juF3ZRN+Akxi0zeUECQ@mail.gmail.com>
Message-ID: <ju0kr2$p08$1@dough.gmane.org>

Nick Coghlan, 16.07.2012 10:26:
> Proposing anything substantially more complicated than what is currently
> implemented in CPython will just get the idea rejected. The substantial
> immediate gain for PyPy is in skipping the memory resizing when building
> containers from itertools iterators, not anywhere else.

The same applies to Cython, where the extension types that implement
generator expressions can benefit from propagating the length hint of the
underlying iterator. A type slot would help in making this more efficient
overall, also for CPython itself.

Stefan


From stefan_ml at behnel.de  Mon Jul 16 11:02:42 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Jul 2012 11:02:42 +0200
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
 A method for exposing a length hint)
In-Reply-To: <5003D2CD.2030903@hotpy.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org>
Message-ID: <ju0lbj$t15$1@dough.gmane.org>

Mark Shannon, 16.07.2012 10:37:
> If resizing of lists is too slow, then we should reconsider the 9/8 factor
> and/or look to tweak the resizing code.

The thing is that the performance is platform specific. On systems with a
fast memory allocator, especially on Linux, the difference is negligible.
However, with a slow memory allocator, especially a slow realloc(), e.g. on
Windows or Solaris, this can substantially hurt the performance, up to a
quadratically increasing runtime in the worst case.

The length hint was implemented specifically to work around this problem.

Stefan


From raymond.hettinger at gmail.com  Mon Jul 16 11:05:08 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 16 Jul 2012 02:05:08 -0700
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
	A method for exposing a length hint)
In-Reply-To: <5003D2CD.2030903@hotpy.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org>
Message-ID: <AB773278-83E2-434C-AE8C-50E8E256BD96@gmail.com>


On Jul 16, 2012, at 1:37 AM, Mark Shannon wrote:

> To quote from PEP 424:
> 
>> Rationale
>> =========
>> Being able to pre-allocate lists based on the expected size, as estimated by ``__length_hint__``, can be a significant optimization. CPython has been
>> observed to run some code faster than PyPy, purely because of this optimization
>> being present.
> 
> Why is it a significant optimisation?

Unless pre-sized by with a length prediction, a growing list
periodically needs to call realloc() which can move all the
data to a new location in memory.    Pre-sizing avoids that entirely.

> If resizing of lists is too slow, then we should reconsider the 9/8 factor
> and/or look to tweak the resizing code.

A great deal of thought and care went into the current design.
It has already been "tweaked".


Raymond


P.S.  The dictionary code also uses presizing for copies, updates, set conversion, etc.
It is a perfectly reasonable technique to pre-allocate the correct size container
when the ultimate length is knowable in advance.



From fijall at gmail.com  Mon Jul 16 11:14:54 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Mon, 16 Jul 2012 11:14:54 +0200
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
 A method for exposing a length hint)
In-Reply-To: <ju0lbj$t15$1@dough.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org> <ju0lbj$t15$1@dough.gmane.org>
Message-ID: <CAK5idxTC-gtR78d1SO2QG0OnB-PrWB5gwBUY9bH0_6TfwFniUg@mail.gmail.com>

On Mon, Jul 16, 2012 at 11:02 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Mark Shannon, 16.07.2012 10:37:
> > If resizing of lists is too slow, then we should reconsider the 9/8
> factor
> > and/or look to tweak the resizing code.
>
> The thing is that the performance is platform specific. On systems with a
> fast memory allocator, especially on Linux, the difference is negligible.
> However, with a slow memory allocator, especially a slow realloc(), e.g. on
> Windows or Solaris, this can substantially hurt the performance, up to a
> quadratically increasing runtime in the worst case.
>
> The length hint was implemented specifically to work around this problem.
>
> Stefan
>
>
It's not the actual allocation (typically), it's the copying that's the
problem.

As far as data goes - preallocation can make 4x difference (on PyPy,
although the dominant operation is not different on CPython) on
''.join(some-iterable). It depends grossly on the sizes of the list, so you
can't claim that there is a precise speedup of a constant factor, however,
there are cases where it *can* be significant (as in the copying is by far
the dominating operation), most notable giant templates with iterators
written in C.

Speaking of which - I find this bikeshed disgusting. The purpose of the PEP
is to codify whatever is already written in code in CPython. If you guys
don't want it, we'll just implement it anyway and try to follow the CPython
current implementation from 2.7.

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120716/cf774556/attachment.html>

From mail at timgolden.me.uk  Mon Jul 16 11:21:52 2012
From: mail at timgolden.me.uk (Tim Golden)
Date: Mon, 16 Jul 2012 10:21:52 +0100
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
 A method for exposing a length hint)
In-Reply-To: <CAK5idxTC-gtR78d1SO2QG0OnB-PrWB5gwBUY9bH0_6TfwFniUg@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org> <ju0lbj$t15$1@dough.gmane.org>
	<CAK5idxTC-gtR78d1SO2QG0OnB-PrWB5gwBUY9bH0_6TfwFniUg@mail.gmail.com>
Message-ID: <5003DD30.4080407@timgolden.me.uk>

> Speaking of which - I find this bikeshed disgusting. 

Disgusting? Irritating, perhaps, but why should a PEP -- even one whose
purpose is to codify existing practice -- not result in discussions
about its subject matter?

The final P stands for Proposal, not for Pronouncement.

TJG

From stefan_ml at behnel.de  Mon Jul 16 11:37:30 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Mon, 16 Jul 2012 11:37:30 +0200
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
 A method for exposing a length hint)
In-Reply-To: <CAK5idxTC-gtR78d1SO2QG0OnB-PrWB5gwBUY9bH0_6TfwFniUg@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org> <ju0lbj$t15$1@dough.gmane.org>
	<CAK5idxTC-gtR78d1SO2QG0OnB-PrWB5gwBUY9bH0_6TfwFniUg@mail.gmail.com>
Message-ID: <ju0ncq$dfm$1@dough.gmane.org>

Maciej Fijalkowski, 16.07.2012 11:14:
> On Mon, Jul 16, 2012 at 11:02 AM, Stefan Behnel wrote:
>> Mark Shannon, 16.07.2012 10:37:
>>> If resizing of lists is too slow, then we should reconsider the 9/8
>>> factor and/or look to tweak the resizing code.
>>
>> The thing is that the performance is platform specific. On systems with a
>> fast memory allocator, especially on Linux, the difference is negligible.
>> However, with a slow memory allocator, especially a slow realloc(), e.g. on
>> Windows or Solaris, this can substantially hurt the performance, up to a
>> quadratically increasing runtime in the worst case.
>>
>> The length hint was implemented specifically to work around this problem.
>
> It's not the actual allocation (typically), it's the copying that's the
> problem.

Note that a call to realloc() includes that part and can avoid copying if
possible. A good realloc() implementation can make this use case run in
amortised linear time, at least on a system with sufficient memory. A bad
one can result in quadratic runtime, which is way more than a change by a
constant factor. Thus my above comment that it's platform specific.


> there are cases where it *can* be significant (as in the copying is by far
> the dominating operation), most notable giant templates with iterators
> written in C.

Absolutely. This is particularly visible at the C level because C
implemented iterators have a very low overhead overall.


> Speaking of which - I find this bikeshed disgusting. The purpose of the PEP
> is to codify whatever is already written in code in CPython. If you guys
> don't want it, we'll just implement it anyway and try to follow the CPython
> current implementation from 2.7.

The idea behind this bikeshedding is that at the moment where we make this
an official protocol rather than an implementation detail, it should be
able to communicate the different states on the callee side of such a
protocol. I.e. it needs a better design than the current one.

Stefan


From raymond.hettinger at gmail.com  Mon Jul 16 11:38:00 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 16 Jul 2012 02:38:00 -0700
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <20120715162232.25c040b7@pitrou.net>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
	<5002C9DB.2000409@pearwood.info>
	<CADiSq7dXMsHgoyXRRBnpdCKGr_aPeOsQqvbs_vZhR06QCOe3=w@mail.gmail.com>
	<20120715162232.25c040b7@pitrou.net>
Message-ID: <1766DA38-1DBC-452A-9442-4DA5DD4E70BF@gmail.com>


On Jul 15, 2012, at 7:22 AM, Antoine Pitrou wrote:

> On Mon, 16 Jul 2012 00:08:41 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Right, I agree on the value in being able to return something to say "this
>> cannot be converted to a concrete container".
> 
> Who would be able to return that, apart from trivial cases like
> itertools.cycle()?


FWIW, here are the notes from the docstring in Lib/test/test_iterlen.py:

""" Test Iterator Length Transparency

Some functions or methods which accept general iterable arguments have
optional, more efficient code paths if they know how many items to expect.
For instance, map(func, iterable), will pre-allocate the exact amount of
space required whenever the iterable can report its length.

The desired invariant is:  len(it)==len(list(it)).

A complication is that an iterable and iterator can be the same object. To
maintain the invariant, an iterator needs to dynamically update its length.
For instance, an iterable such as xrange(10) always reports its length as ten,
but it=iter(xrange(10)) starts at ten, and then goes to nine after it.next().
Having this capability means that map() can ignore the distinction between
map(func, iterable) and map(func, iter(iterable)).

When the iterable is immutable, the implementation can straight-forwardly
report the original length minus the cumulative number of calls to next().
This is the case for tuples, xrange objects, and itertools.repeat().

Some containers become temporarily immutable during iteration.  This includes
dicts, sets, and collections.deque.  Their implementation is equally simple
though they need to permanently set their length to zero whenever there is
an attempt to iterate after a length mutation.

The situation slightly more involved whenever an object allows length mutation
during iteration.  Lists and sequence iterators are dynamically updatable.
So, if a list is extended during iteration, the iterator will continue through
the new items.  If it shrinks to a point before the most recent iteration,
then no further items are available and the length is reported at zero.

Reversed objects can also be wrapped around mutable objects; however, any
appends after the current position are ignored.  Any other approach leads
to confusion and possibly returning the same item more than once.

The iterators not listed above, such as enumerate and the other itertools,
are not length transparent because they have no way to distinguish between
iterables that report static length and iterators whose length changes with
each call (i.e. the difference between enumerate('abc') and
enumerate(iter('abc')).

"""


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120716/76aad833/attachment.html>

From storchaka at gmail.com  Mon Jul 16 11:42:09 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 16 Jul 2012 12:42:09 +0300
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <ju0gaq$mci$1@dough.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<ju0gaq$mci$1@dough.gmane.org>
Message-ID: <ju0nk0$fad$1@dough.gmane.org>

On 16.07.12 10:36, Stefan Behnel wrote:
> Return values could be -1 for "don't know" and -2 for "infinite" at the C
> level, and NotImplemented for "don't know" at the Python level.

PY_SSIZE_T_MAX is better value for "infinite". In any case no difference 
for consumer between PY_SSIZE_T_MAX and a real infinity.


From kristjan at ccpgames.com  Mon Jul 16 11:42:15 2012
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Mon, 16 Jul 2012 09:42:15 +0000
Subject: [Python-Dev] early startup error reporting failure
Message-ID: <EFE3877620384242A686D52278B7CCD34278E8@RKV-IT-EXCH104.ccp.ad.local>

Hi there.

I've been busy taking the current beta candidate and merging it into the stackless repo.

As expected, things don't just go smoothly and there are the usual startup errors, this being a rather intrusive patch and all that.



However, I found that early startup errors were not being reported correctly, so I had do make some changes to fix that.  I'm not sure these are the correct fixes, so I'd like to start this here and see if anyone feels responsible.



Right:  The initial error occurs here:

if (PyImport_ImportFrozenModule("_frozen_importlib") <= 0) {

  Py_FatalError("Py_Initialize: can't import _frozen_importlib");

My problem was that the actual exception was not being reported along with the FatalError message.



Digging around a bit, I found the cause here:

fileobject.c, PyFile_WriteString()

  }
    else if (!PyErr_Occurred()) {


That is, this function declines to write anything if there is an exception present.

My quick and dirty fix was to remove this test and just print even with a present exception.  That fixes the issue.

But perhaps the _correct_ way is to suppress the exception higher up in the callchain, which is this:

> python33_d.dll!PyFile_WriteString(const char * s, _object * f)  Line 179 C
  python33_d.dll!PyTraceBack_Print(_object * v, _object * f)  Line 415 + 0x11 bytes C
  python33_d.dll!print_exception(_object * f, _object * value)  Line 1748 + 0x12 bytes C
  python33_d.dll!print_exception_recursive(_object * f, _object * value, _object * seen)  Line 1889 C
  python33_d.dll!PyErr_Display(_object * exception, _object * value, _object * tb)  Line 1913 C
  python33_d.dll!sys_excepthook(_object * self, _object * args)  Line 197 C
  python33_d.dll!PyCFunction_Call(_object * func, _object * arg, _object * kw)  Line 99 + 0x46 bytes C
  python33_d.dll!PyObject_Call(_object * func, _object * arg, _object * kw)  Line 2149 + 0x48 bytes C
  python33_d.dll!PyEval_CallObjectWithKeywords(_object * func, _object * arg, _object * kw)  Line 4584 C
  python33_d.dll!PyErr_PrintEx(int set_sys_last_vars)  Line 1686 + 0x12 bytes C
  python33_d.dll!Py_FatalError(const char * msg)  Line 2358 C

Perhaps error should be fetched and restored in PyTraceback_Print(), since it already does some exception juggling, obviously assuming that an exception state can be present that it is worthwhile to preserve.



Ok, then I came to the second issue.

When printing the tracebacks, this early in the process, I hit upon this code, in

traceback.c, tb_displayline(), I made this change (line 344):

-    return _Py_DisplaySourceLine(f, filename, lineno, 4);
+ /* ignore IO errors here, IO may not be ready yet */
+ if ( _Py_DisplaySourceLine(f, filename, lineno, 4) )
+  PyErr_Clear();
+ return err;



This early in the process, IO cannot be imported so it is impossible to output source line.  The source line output is a "bonus" feature anyway and we shouldn't, IMHO, fail outputting tracebacks if we cannot read the code.



The actual failure was importing the IO library.  Perhaps an alternative fix, then, is to fix the _Py_DisplaySourceLine() so that it deals with failure to import IO in the same way as failure to read the file, i.e. just returns a "success" value of 0.



With these changes, I was able to successfully output the error.  Hopefully I will be able to debug it too :)



Any thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120716/376933bd/attachment.html>

From raymond.hettinger at gmail.com  Mon Jul 16 11:51:39 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Mon, 16 Jul 2012 02:51:39 -0700
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <ju0gaq$mci$1@dough.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<ju0gaq$mci$1@dough.gmane.org>
Message-ID: <9DBB02E3-26DE-4567-9F08-611E18730BD4@gmail.com>


On Jul 16, 2012, at 12:36 AM, Stefan Behnel wrote:

> I'd like to more visibly repeat my suggestion to make this a slot method
> "tp_length_hint()" in extension types that returns a Py_ssize_t.

That is merely an implementation detail, but it would be a nice improvement.


Raymond

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120716/ec7a620a/attachment.html>

From kristjan at ccpgames.com  Mon Jul 16 12:20:43 2012
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Mon, 16 Jul 2012 10:20:43 +0000
Subject: [Python-Dev] early startup error reporting failure
In-Reply-To: <EFE3877620384242A686D52278B7CCD34278E8@RKV-IT-EXCH104.ccp.ad.local>
References: <EFE3877620384242A686D52278B7CCD34278E8@RKV-IT-EXCH104.ccp.ad.local>
Message-ID: <EFE3877620384242A686D52278B7CCD342799E@RKV-IT-EXCH104.ccp.ad.local>

Looking better at the code, the fileobject change isn't necessary.  A simpler fix is to just ignore and clear errors from _Py_DisplaySourceLine.

I?ll prepare a defect/patch



K

________________________________
Fr?: python-dev-bounces+kristjan=ccpgames.com at python.org [python-dev-bounces+kristjan=ccpgames.com at python.org] fyrir h?nd Kristj?n Valur J?nsson [kristjan at ccpgames.com]
Sent: 16. j?l? 2012 09:42
To: python-dev at python.org
Efni: [Python-Dev] early startup error reporting failure


Hi there.

I've been busy taking the current beta candidate and merging it into the stackless repo.

As expected, things don't just go smoothly and there are the usual startup errors, this being a rather intrusive patch and all that.



However, I found that early startup errors were not being reported correctly, so I had do make some changes to fix that.  I'm not sure these are the correct fixes, so I'd like to start this here and see if anyone feels responsible.



Right:  The initial error occurs here:

if (PyImport_ImportFrozenModule("_frozen_importlib") <= 0) {

  Py_FatalError("Py_Initialize: can't import _frozen_importlib");

My problem was that the actual exception was not being reported along with the FatalError message.



Digging around a bit, I found the cause here:

fileobject.c, PyFile_WriteString()

  }
    else if (!PyErr_Occurred()) {


That is, this function declines to write anything if there is an exception present.

My quick and dirty fix was to remove this test and just print even with a present exception.  That fixes the issue.

But perhaps the _correct_ way is to suppress the exception higher up in the callchain, which is this:

> python33_d.dll!PyFile_WriteString(const char * s, _object * f)  Line 179 C
  python33_d.dll!PyTraceBack_Print(_object * v, _object * f)  Line 415 + 0x11 bytes C
  python33_d.dll!print_exception(_object * f, _object * value)  Line 1748 + 0x12 bytes C
  python33_d.dll!print_exception_recursive(_object * f, _object * value, _object * seen)  Line 1889 C
  python33_d.dll!PyErr_Display(_object * exception, _object * value, _object * tb)  Line 1913 C
  python33_d.dll!sys_excepthook(_object * self, _object * args)  Line 197 C
  python33_d.dll!PyCFunction_Call(_object * func, _object * arg, _object * kw)  Line 99 + 0x46 bytes C
  python33_d.dll!PyObject_Call(_object * func, _object * arg, _object * kw)  Line 2149 + 0x48 bytes C
  python33_d.dll!PyEval_CallObjectWithKeywords(_object * func, _object * arg, _object * kw)  Line 4584 C
  python33_d.dll!PyErr_PrintEx(int set_sys_last_vars)  Line 1686 + 0x12 bytes C
  python33_d.dll!Py_FatalError(const char * msg)  Line 2358 C

Perhaps error should be fetched and restored in PyTraceback_Print(), since it already does some exception juggling, obviously assuming that an exception state can be present that it is worthwhile to preserve.



Ok, then I came to the second issue.

When printing the tracebacks, this early in the process, I hit upon this code, in

traceback.c, tb_displayline(), I made this change (line 344):

-    return _Py_DisplaySourceLine(f, filename, lineno, 4);
+ /* ignore IO errors here, IO may not be ready yet */
+ if ( _Py_DisplaySourceLine(f, filename, lineno, 4) )
+  PyErr_Clear();
+ return err;



This early in the process, IO cannot be imported so it is impossible to output source line.  The source line output is a "bonus" feature anyway and we shouldn't, IMHO, fail outputting tracebacks if we cannot read the code.



The actual failure was importing the IO library.  Perhaps an alternative fix, then, is to fix the _Py_DisplaySourceLine() so that it deals with failure to import IO in the same way as failure to read the file, i.e. just returns a "success" value of 0.



With these changes, I was able to successfully output the error.  Hopefully I will be able to debug it too :)



Any thoughts?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120716/e4c39496/attachment.html>

From victor.stinner at gmail.com  Mon Jul 16 13:05:20 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 16 Jul 2012 13:05:20 +0200
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
 A method for exposing a length hint)
In-Reply-To: <5003D2CD.2030903@hotpy.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org>
Message-ID: <CAMpsgwaLLJPtjJksD=tBrvC4=Gb8o8KcgdnBUw0Z-d6VSb4FsA@mail.gmail.com>

> *If* resizing list is so slow, then why not make it faster?

A simple solution to speed up such problem is to change the
overallocation factor, but it may waste memory. Python tries to be
fast and to not waste too much memory.

> Why is it a significant optimisation?
> How much slower is it?
> Where is the data?

I worked recently on optimizing str%args and str.format(args). Handle
correctly the memory allocation is just critical for performances,
especially for str with the PEP 393, because we have to shrink the
buffer to the exact string length with the formatting function is
done. I tried various overallocation factors and I chose 1.25 (5/4)
because it was the fastest. See for example this issue for benchmark
numbers:
http://bugs.python.org/issue14687

The PyUnicodeWriter internal object uses various options to choose how
many bytes should be allocated:
 * an overallocation flag to disable overallocation when we know that
we are writing the last character/string into be buffer
 * a "minimal length" used for the first allocation
 * an hardcoded overallocation factor (1.25)

PyUnicodeWriter is a little bit different than the __length_hint__
issue because PyUnicodeWriter has to shrink the buffer when it is
done, but I can say that overallocation is very useful for speed.

Victor

From brett at python.org  Mon Jul 16 15:01:14 2012
From: brett at python.org (Brett Cannon)
Date: Mon, 16 Jul 2012 09:01:14 -0400
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <ju0gaq$mci$1@dough.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<ju0gaq$mci$1@dough.gmane.org>
Message-ID: <CAP1=2W7wjbBU0z10Y_CZsADUhDXWmBU5sjbd6shtdX8CqPgqpw@mail.gmail.com>

On Mon, Jul 16, 2012 at 3:36 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> Alex Gaynor, 15.07.2012 00:11:
> > CPython currently defines an ``__length_hint__`` method on several
> types, such
> > as various iterators. This method is then used by various other
> functions (such
> > as ``map``) to presize lists based on the estimated returned by
> > ``__length_hint__``. Types can then define ``__length_hint__`` which are
> not
> > sized, and thus should not define ``__len__``, but can estimate or
> compute a
> > size (such as many iterators).
> >
> > Proposal
> > ========
> >
> > This PEP proposes formally documenting ``__length_hint__`` for other
> > interpreter and non-standard library Python to implement.
> >
> > ``__length_hint__`` must return an integer, and is not required to be
> accurate.
> > It may return a value that is either larger or smaller than the actual
> size of
> > the container. It may raise a ``TypeError`` if a specific instance
> cannot have
> > its length estimated. It may not return a negative value.
>
> I'd like to more visibly repeat my suggestion to make this a slot method
> "tp_length_hint()" in extension types that returns a Py_ssize_t.
>
> That suggests that a negative return value would have a special meaning
> instead of relying on return values like NotImplemented. The Python wrapper
> of that slot method could still implement a mapping for this.
>
> Return values could be -1 for "don't know" and -2 for "infinite" at the C
> level, and NotImplemented for "don't know" at the Python level. Not sure
> about a good Python value for "infinite".
>

Gods no. Making the return value different in C vs. Python code is just
asking for trouble in terms of having to remember that specific difference
while coding. Plus asking for people to check for an explicit negative
values instead of just >= 0 would be problematic and prone to error.


>
> Maybe return -1 for "infinite" at both levels and -2/NotImplemented for
> "don't know" in C/Python? That would suggest -3 to propagate exceptions at
> the C level.
>

See above.

This is another reason why I don't think the infinite iterator concept is
worth expressin. It's just mucking things up for no good reason. "infinite"
== "I don't know" in the case of pre-allocation of a list. Just raise an
exception or return None and be done with it. Nice and simple. And my vote
is for an exception as EAFP.

-Brett
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120716/ce8d0025/attachment.html>

From ncoghlan at gmail.com  Mon Jul 16 15:23:18 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 16 Jul 2012 23:23:18 +1000
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
 A method for exposing a length hint)
In-Reply-To: <5003DD30.4080407@timgolden.me.uk>
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org> <ju0lbj$t15$1@dough.gmane.org>
	<CAK5idxTC-gtR78d1SO2QG0OnB-PrWB5gwBUY9bH0_6TfwFniUg@mail.gmail.com>
	<5003DD30.4080407@timgolden.me.uk>
Message-ID: <CADiSq7c1miPXUK2Nsg8r7rw265KmaCrxoapb1z7zkSvB7T2E3w@mail.gmail.com>

On Mon, Jul 16, 2012 at 7:21 PM, Tim Golden <mail at timgolden.me.uk> wrote:
>> Speaking of which - I find this bikeshed disgusting.
>
> Disgusting? Irritating, perhaps, but why should a PEP -- even one whose
> purpose is to codify existing practice -- not result in discussions
> about its subject matter?
>
> The final P stands for Proposal, not for Pronouncement.

Indeed - I'd be worried if any PEP sailed through python-dev review
without a thorough kicking of the tires. Yes, it can be annoying
having to bring people up to speed on issues that they aren't familiar
with, but that's generally a sign that there is relevant background
information *missing from the PEP*.

PEP's aren't supposed to be written just for people that are already
intimately familiar with a problem - they're supposed to provide
enough background that they stand on their own.

In this case, the key points that I think need to be added:

- more background on why the __length_hint__ API exists in CPython in
the first place: to minimise potentially expensive data copies (due to
memory reallocation) when creating a concrete container from an
iterator. This includes when creating them from another concrete
container via an intermediate iterator. This is why at least the
following produce objects that define __length_hint__ in CPython:

    reversed
    itertools.repeat
    iter(dict())
    iter(list())
    iter(tuple())
    iter(str())
    iter(bytes())
    iter(bytearray())
    iter(set())
    iter(frozenset())
    dict.values()
    dict.keys()

As well as any user defined sequence that relies on the default
sequence iterator:
    >>> class MySeq():
    ...     def __getitem__(self, idx):
    ...         return idx
    ...     def __len__(self):
    ...         return 10
    ...
    >>> hasattr(iter(MySeq()), "__length_hint__")
    True

- clarification on the implications of it only being a "hint":
specifically, as it may be an over- or underestimate, you *cannot*
rely on the hint to decide whether or not to iterate over the object
when a valid length is returned (as a value of zero may be an
underestimate). However, it does allow you to presize your container
more appropriately than just starting at zero as usual, thus achieving
the aim of reducing the risk of unnecessary memory copies.

That's the basic proposal. Separate from that, there are a few
suggestions for *enhancement* beyond what CPython currently uses (and
has demonstrated a clear need for):

- adding operator.length_hint(). This makes sense to me, as it makes
it much easier to use the API when implementing a container type in
Python. It's also a pretty simple addition.

- adding a C level type slot. I'm personally -1 on this one in the
context of the PEP. I don't think the current PEP (which is really
aimed at standardisation for PyPy's benefit) should be weighed down
with this CPython specific implementation detail.  As a separate
proposal, independent of this PEP, from someone that cares
specifically about micro-optimising this API for CPython, and
(preferably) can produce benchmark numbers to show the additional
implementation complexity is worthwhile, then I wouldn't object. I
just don't want this orthogonal concern to derail the standardisation
effort.

- distinguishing between different reasons for saying "don't
preallocate any space" (i.e. returning zero). I still haven't heard a
convincing rationale for this one - it seems to be based on the notion
that the case of skipping the iteration step for a genuinely empty
iterable is worth optimising. This idea just doesn't make sense when
any legitimate length value that is returned can't be trusted to be
completely accurate - you have to iterate to confirm the actual
number.

- making it possible to fail *fast* when a known infinite iterator
(like itertools.cycle or itertools.count) is passed to a concrete
container. I think this is best covered in the PEP by explicitly
stating that some types may implement __length_hint__ to always raise
TypeError rather than defining a magic return value that means "I'm
infinite".

- making it possible for iterators like enumerate, map and filter to
delegate __length_hint__ to their underlying iterator. This seems
potentially worth doing, but requires resolving the problem that
Raymond noted with handling the difference in internal behaviour
between enumerate("abc") and enumerate(iter("abc")). Again, it would
be unfortunate to see the PEP held up over this.

- making it possible to define __length_hint__ for generator-iterator
objects. While this is a nice idea, again, I don't think it's
something that this particular PEP should be burdened with.

My main point is that the current __length_hint__ behaviour has
already proven its value in the real world. The PyPy team would like
that behaviour codified, so they can be reasonably sure both
implementations are doing the same thing. Many of the enhancements I
have listed above may be suitable material for future enhancement
proposals, but I'm not seeing any requested functionality that would
be actively *blocked* by simply codifying the status quo.

The PEP itself already has this general tone, but I think that it
should be even more emphatic that it's about codifying the status quo,
*not* about modifying it with ideas haven't been proven useful through
past experience.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Mon Jul 16 16:01:54 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Jul 2012 16:01:54 +0200
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
 A method for exposing a length hint)
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org> <ju0lbj$t15$1@dough.gmane.org>
	<CAK5idxTC-gtR78d1SO2QG0OnB-PrWB5gwBUY9bH0_6TfwFniUg@mail.gmail.com>
	<5003DD30.4080407@timgolden.me.uk>
	<CADiSq7c1miPXUK2Nsg8r7rw265KmaCrxoapb1z7zkSvB7T2E3w@mail.gmail.com>
Message-ID: <20120716160154.5aaa2591@pitrou.net>

On Mon, 16 Jul 2012 23:23:18 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> - distinguishing between different reasons for saying "don't
> preallocate any space" (i.e. returning zero). I still haven't heard a
> convincing rationale for this one

The point is that zero is a valid value for a length hint. By making it
a special value for "don't know", you are making the protocol
potentially confusing, and you are also departing from the current
semantics.

(and, yes, I think distinguishing between zero and "don't know" is
useful: imagine a container that would preallocate 256 entries by
default when the answer is "don't know")

> The PEP itself already has this general tone, but I think that it
> should be even more emphatic that it's about codifying the status quo,
> *not* about modifying it with ideas haven't been proven useful through
> past experience.

Then the PEP shouldn't address infinite iterators at all.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From ncoghlan at gmail.com  Mon Jul 16 16:18:55 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 17 Jul 2012 00:18:55 +1000
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
 A method for exposing a length hint)
In-Reply-To: <20120716160154.5aaa2591@pitrou.net>
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org> <ju0lbj$t15$1@dough.gmane.org>
	<CAK5idxTC-gtR78d1SO2QG0OnB-PrWB5gwBUY9bH0_6TfwFniUg@mail.gmail.com>
	<5003DD30.4080407@timgolden.me.uk>
	<CADiSq7c1miPXUK2Nsg8r7rw265KmaCrxoapb1z7zkSvB7T2E3w@mail.gmail.com>
	<20120716160154.5aaa2591@pitrou.net>
Message-ID: <CADiSq7dYw17k+-E2JxWYeD2czrcNpjHuAEYAVY1jSAuyab6N3w@mail.gmail.com>

On Tue, Jul 17, 2012 at 12:01 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Mon, 16 Jul 2012 23:23:18 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> - distinguishing between different reasons for saying "don't
>> preallocate any space" (i.e. returning zero). I still haven't heard a
>> convincing rationale for this one
>
> The point is that zero is a valid value for a length hint. By making it
> a special value for "don't know", you are making the protocol
> potentially confusing, and you are also departing from the current
> semantics.

No, it just means the default estimate is always zero. If you don't do
that, then *every* client of __length_hint__ has to check for the
magic value. It's making the API harder to use for no good reason.

> (and, yes, I think distinguishing between zero and "don't know" is
> useful: imagine a container that would preallocate 256 entries by
> default when the answer is "don't know")

Such a container has to already deal with the case when it
overestimates severely. The only cost of using zero as a default
estimate is that such hypothetical containers will overallocate when
they technically didn't need to, which will already happen for any
empty iterator that doesn't provide __length_hint__ at all.

Given that all standard library containers default to assuming a size
of zero (absent information indicating otherwise), and a special value
would need to be special cased by *every* client of the API (and
almost always defaulted to zero), it's simply not a good trade-off.

>> The PEP itself already has this general tone, but I think that it
>> should be even more emphatic that it's about codifying the status quo,
>> *not* about modifying it with ideas haven't been proven useful through
>> past experience.
>
> Then the PEP shouldn't address infinite iterators at all.

Noting that infinite iterators are free to define __length_hint__ to
always throw an exception *is* the status quo. We just haven't done it
for the stdlib ones.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From solipsis at pitrou.net  Mon Jul 16 16:40:04 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 16 Jul 2012 16:40:04 +0200
Subject: [Python-Dev] Do we need __length_hint__ at all? (Was PEP 0424:
 A method for exposing a length hint)
In-Reply-To: <CADiSq7dYw17k+-E2JxWYeD2czrcNpjHuAEYAVY1jSAuyab6N3w@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<5003D2CD.2030903@hotpy.org> <ju0lbj$t15$1@dough.gmane.org>
	<CAK5idxTC-gtR78d1SO2QG0OnB-PrWB5gwBUY9bH0_6TfwFniUg@mail.gmail.com>
	<5003DD30.4080407@timgolden.me.uk>
	<CADiSq7c1miPXUK2Nsg8r7rw265KmaCrxoapb1z7zkSvB7T2E3w@mail.gmail.com>
	<20120716160154.5aaa2591@pitrou.net>
	<CADiSq7dYw17k+-E2JxWYeD2czrcNpjHuAEYAVY1jSAuyab6N3w@mail.gmail.com>
Message-ID: <20120716164004.544d692a@pitrou.net>

On Tue, 17 Jul 2012 00:18:55 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Given that all standard library containers default to assuming a size
> of zero (absent information indicating otherwise), and a special value
> would need to be special cased by *every* client of the API (and
> almost always defaulted to zero), it's simply not a good trade-off.

Actually, dict and set default to a non-zero internal size, but I agree
making containers harder to implement is not a good thing.

> >> The PEP itself already has this general tone, but I think that it
> >> should be even more emphatic that it's about codifying the status quo,
> >> *not* about modifying it with ideas haven't been proven useful through
> >> past experience.
> >
> > Then the PEP shouldn't address infinite iterators at all.
> 
> Noting that infinite iterators are free to define __length_hint__ to
> always throw an exception *is* the status quo.

Being "free" to do unexpected or unconventional things is not the same
thing as codifying those behaviours in a PEP, especially when noone is
actually doing them. __length_hint__ is supposed to be informative, it
shouldn't error out on you. So still -1 from me.

Regards

Antoine.

From mstefanro at gmail.com  Mon Jul 16 21:36:21 2012
From: mstefanro at gmail.com (M Stefan)
Date: Mon, 16 Jul 2012 22:36:21 +0300
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <ju0dr4$5kb$1@dough.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<5002D063.1060401@hotpy.org> <ju0dr4$5kb$1@dough.gmane.org>
Message-ID: <50046D35.3000708@gmail.com>

On 7/16/2012 9:54 AM, Stefan Behnel wrote:
> Mark Shannon, 15.07.2012 16:14:
>> Alex Gaynor wrote:
>>> CPython currently defines an ``__length_hint__`` method on several types,
>>> such
>>> as various iterators. This method is then used by various other functions
>>> (such as ``map``) to presize lists based on the estimated returned by
>> Don't use "map" as an example.
>> map returns an iterator so it doesn't need __length_hint__
> Right. It's a good example for something else, though. As I mentioned
> before, iterators should be able to propagate the length hint of an
> underlying iterator, e.g. in generator expressions or map(). I consider
> that an important feature that the protocol must support.
>
> Stefan
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/mstefanro%40gmail.com
>
map() is quite problematic in this matter, and may actually
benefit from the existence of __length_hint__.
It is very easy to create an infinite loop currently by doing stuff like 
x=[1]; x+=map(str,x)
[61081 refs]
 >>> x=[1]; x+=map(str,x)
Traceback (most recent call last):
   ...
MemoryError
[120959834 refs]
 >>> len(x)
120898752

Obviously, this won't cause an infinite loop in Python2 where map is 
non-lazy.
Also, this won't work for all mutable containers, because not all of 
them permit
adding elements while iterating:
 >>> s=set([1]); s.update(map(str,s))
Traceback (most recent call last):
   ...
RuntimeError: Set changed size during iteration
[61101 refs]
 >>> s
{1, '1'}
[61101 refs]
 >>> del s
[61099 refs]

If map objects were to disallow changing the size of the container
while iterating (I can't really think of an use-case in which such a
limitation would be harmful), it might as well be with __length_hint__.

Also, what would iter([1,2,3]).__length_hint__() return? 3 or unknown?
If 3, then the semantics of l=[1,2,3]; l += iter(l) will change 
(infinite loop
without __length_hint__ vs. list of 6 elements with __length_hint__).
If unknown, then it doesn't seem like there are very many places where
__length_hint__ can return anything but unknown.

Regards,
   Stefan M

From ethan at stoneleaf.us  Mon Jul 16 21:59:20 2012
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 16 Jul 2012 12:59:20 -0700
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <50046D35.3000708@gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>	<5002D063.1060401@hotpy.org>
	<ju0dr4$5kb$1@dough.gmane.org> <50046D35.3000708@gmail.com>
Message-ID: <50047298.9040806@stoneleaf.us>

M Stefan wrote:
> Also, what would iter([1,2,3]).__length_hint__() return? 3 or unknown?
> If 3, then the semantics of l=[1,2,3]; l += iter(l) will change 
> (infinite loop without __length_hint__ vs. list of 6 elements with __length_hint__).

What __length_hint__ returns is irrelevant -- it's only a hint.  Python 
will have to loop over all the items.  So you would still get an 
infinite loop with the above code.

~Ethan~

From alex.gaynor at gmail.com  Tue Jul 17 05:03:08 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Tue, 17 Jul 2012 03:03:08 +0000 (UTC)
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
References: <loom.20120715T000919-641@post.gmane.org>
	<ju0gaq$mci$1@dough.gmane.org>
	<CAP1=2W7wjbBU0z10Y_CZsADUhDXWmBU5sjbd6shtdX8CqPgqpw@mail.gmail.com>
Message-ID: <loom.20120717T050122-455@post.gmane.org>

I've updated the PEP to reflect the discussion. There are two major changes:

1) NotImplemented may be used by __length_hint__ to indicate that there is no 
   finite length hint available.
2) callers of operator.length_hint() must provide their own default value, this
   is also required by the current C _PyObject_LengthHint implementation.

There are no provisions for infinite iterators, that is not within the scope of
this proposal.

Alex


From ncoghlan at gmail.com  Tue Jul 17 05:19:55 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 17 Jul 2012 13:19:55 +1000
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <loom.20120717T050122-455@post.gmane.org>
References: <loom.20120715T000919-641@post.gmane.org>
	<ju0gaq$mci$1@dough.gmane.org>
	<CAP1=2W7wjbBU0z10Y_CZsADUhDXWmBU5sjbd6shtdX8CqPgqpw@mail.gmail.com>
	<loom.20120717T050122-455@post.gmane.org>
Message-ID: <CADiSq7evUu_MYYmkWi27J4KZsmMM3nHSJ+fq5YcbsX0LW_j5gA@mail.gmail.com>

On Tue, Jul 17, 2012 at 1:03 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> I've updated the PEP to reflect the discussion. There are two major changes:
>
> 1) NotImplemented may be used by __length_hint__ to indicate that there is no
>    finite length hint available.

I've been thinking about this a bit more, and I think it does provide
good scope for eventually adding __length_hint__ to more iterators
(including map, filter and enumerate).

> 2) callers of operator.length_hint() must provide their own default value, this
>    is also required by the current C _PyObject_LengthHint implementation.

And this makes it explicit that API users need to deal with the
AttributeError/NotImplemented case, whilst making it easy to do so.
Good call.

> There are no provisions for infinite iterators, that is not within the scope of
> this proposal.

I'll repeat my observation that remaining silent on this point is
effectively identical to blessing the practice of raising an exception
in __length_hint__ to force fast failure of attempts to convert an
infinite iterator to a concrete container. Rather than leaving people
to figure this out on their own, we may as well make it official that
TypeError can be raised in __length_hint__ to block conversion to
concrete containers that use a preallocation strategy.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From eliben at gmail.com  Tue Jul 17 05:34:14 2012
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 17 Jul 2012 06:34:14 +0300
Subject: [Python-Dev] io.BytesIO slower than monkey-patching io.RawIOBase
Message-ID: <CAF-Rda-nXNDejQFBQygnRUuiBgoEvgMJtF1PYVx59AfnfnxOPg@mail.gmail.com>

While working on #1767933, Serhiy came up with an observation that
"monkey-patching" one of the base classes of io is faster than using
BytesIO when in need of a file-like object for writing into.

I've distilled it into this standalone test:

import io

data = [b'a'*10, b'bb'*5, b'ccc'*5] * 10000

def withbytesio():
    bio = io.BytesIO()
    for i in data:
        bio.write(i)
    return bio.getvalue()

def monkeypatching():
    mydata = []
    file = io.RawIOBase()
    file.writable = lambda: True
    file.write = mydata.append

    for i in data:
        file.write(i)
    return b''.join(mydata)

The second approach is consistently 10-20% faster than the first one
(depending on input) for trunk Python 3.3

Is there any reason for this to be so? What does BytesIO give us that the
second approach does not (I tried adding more methods to the patched
RawIOBase to make it more functional, like seekable() and tell(), and it
doesn't affect performance)?

This also raises a "moral" question - should I be using the second approach
deep inside the stdlib (ET.tostring) just because it's faster?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120717/922d5d0d/attachment.html>

From jxo6948 at rit.edu  Tue Jul 17 06:57:04 2012
From: jxo6948 at rit.edu (John O'Connor)
Date: Tue, 17 Jul 2012 00:57:04 -0400
Subject: [Python-Dev] io.BytesIO slower than monkey-patching io.RawIOBase
In-Reply-To: <CAF-Rda-nXNDejQFBQygnRUuiBgoEvgMJtF1PYVx59AfnfnxOPg@mail.gmail.com>
References: <CAF-Rda-nXNDejQFBQygnRUuiBgoEvgMJtF1PYVx59AfnfnxOPg@mail.gmail.com>
Message-ID: <CABCbifVN2tK+jAof3bEnpHjFUFdzTkW6A1-PPe12M9kpQRtCfg@mail.gmail.com>

>
> The second approach is consistently 10-20% faster than the first one
> (depending on input) for trunk Python 3.3
>

I think the difference is that StringIO spends extra time reallocating
memory during the write loop as it grows, whereas bytes.join computes
the allocation size first since it already knows the final length.

From ncoghlan at gmail.com  Tue Jul 17 07:48:44 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 17 Jul 2012 15:48:44 +1000
Subject: [Python-Dev] io.BytesIO slower than monkey-patching io.RawIOBase
In-Reply-To: <CABCbifVN2tK+jAof3bEnpHjFUFdzTkW6A1-PPe12M9kpQRtCfg@mail.gmail.com>
References: <CAF-Rda-nXNDejQFBQygnRUuiBgoEvgMJtF1PYVx59AfnfnxOPg@mail.gmail.com>
	<CABCbifVN2tK+jAof3bEnpHjFUFdzTkW6A1-PPe12M9kpQRtCfg@mail.gmail.com>
Message-ID: <CADiSq7fGxwGi=Co-ztP94yoCMigiAcnJaw+yD4U7PYd7_HJ5wg@mail.gmail.com>

On Tue, Jul 17, 2012 at 2:57 PM, John O'Connor <jxo6948 at rit.edu> wrote:
>>
>> The second approach is consistently 10-20% faster than the first one
>> (depending on input) for trunk Python 3.3
>>
>
> I think the difference is that StringIO spends extra time reallocating
> memory during the write loop as it grows, whereas bytes.join computes
> the allocation size first since it already knows the final length.

BytesIO is actually missing an optimisation that is already used in
StringIO: the StringIO C implementation uses a fragment accumulator
internally, and collapses that into a single string object when
getvalue() is called. BytesIO is still using the old
"resize-the-buffer-as-you-go" strategy, and thus ends up repeatedly
reallocating the buffer as the data sequence grows incrementally.

It should be optimised to work the same way StringIO does (which is
effectively the same way that the monkeypatched version works)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dickinsm at gmail.com  Tue Jul 17 09:49:26 2012
From: dickinsm at gmail.com (Mark Dickinson)
Date: Tue, 17 Jul 2012 08:49:26 +0100
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <20120715143641.3d2bb83d@pitrou.net>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CADiSq7dg_F+5u30xucXnJOadQn2U4ThQ7LoKu3HX2qHOsCfcQw@mail.gmail.com>
	<50027D79.8010500@pearwood.info>
	<CADiSq7exHaQWXpdtz0W9iRMW+atvLsfK5KS1kqzwW6cbih8_+A@mail.gmail.com>
	<20120715143641.3d2bb83d@pitrou.net>
Message-ID: <CAAu3qLU56hXGoxQRa4RapGT5Xy4iVLfmqLUALXsLpZcRmk-vcw@mail.gmail.com>

On Sun, Jul 15, 2012 at 1:36 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 15 Jul 2012 18:47:38 +1000
> Nick Coghlan <ncoghlan at gmail.com> wrote:
>> I'm not seeing the value in returning None over 0 for the don't know
>> case - it just makes the API harder to use.
>
> The point is that 0 is a legitimate value for a length hint. Simple
> implementations of __length_hint__ will start returning 0 as a
> legitimate value and you will wrongly interpret that as "don't know",
> which kinds of defeat the purpose of __length-hint__ ;)

I agree with this: giving special meaning to what's already a valid
length value seems wrong.

Mark

From solipsis at pitrou.net  Tue Jul 17 10:30:41 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 17 Jul 2012 10:30:41 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
References: <loom.20120715T000919-641@post.gmane.org>
	<ju0gaq$mci$1@dough.gmane.org>
	<CAP1=2W7wjbBU0z10Y_CZsADUhDXWmBU5sjbd6shtdX8CqPgqpw@mail.gmail.com>
	<loom.20120717T050122-455@post.gmane.org>
	<CADiSq7evUu_MYYmkWi27J4KZsmMM3nHSJ+fq5YcbsX0LW_j5gA@mail.gmail.com>
Message-ID: <20120717103041.2902d4b5@pitrou.net>

On Tue, 17 Jul 2012 13:19:55 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> > There are no provisions for infinite iterators, that is not within the scope of
> > this proposal.
> 
> I'll repeat my observation that remaining silent on this point is
> effectively identical to blessing the practice of raising an exception
> in __length_hint__ to force fast failure of attempts to convert an
> infinite iterator to a concrete container.

And I'll repeat that it is false ;) Being silent is certainly not the
same thing as blessing a non-existent practice.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From solipsis at pitrou.net  Tue Jul 17 10:35:02 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 17 Jul 2012 10:35:02 +0200
Subject: [Python-Dev] io.BytesIO slower than monkey-patching io.RawIOBase
References: <CAF-Rda-nXNDejQFBQygnRUuiBgoEvgMJtF1PYVx59AfnfnxOPg@mail.gmail.com>
Message-ID: <20120717103502.6c9cad90@pitrou.net>

On Tue, 17 Jul 2012 06:34:14 +0300
Eli Bendersky <eliben at gmail.com> wrote:
> 
> Is there any reason for this to be so? What does BytesIO give us that the
> second approach does not (I tried adding more methods to the patched
> RawIOBase to make it more functional, like seekable() and tell(), and it
> doesn't affect performance)?

Well, try implementing non-trivial methods such as readline() or
seek(), and writing in the middle rather than at the end. 

As Nick said, we could implement the same optimization as in StringIO,
i.e. only materialize the buffer when necessary.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From storchaka at gmail.com  Tue Jul 17 13:59:20 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 17 Jul 2012 14:59:20 +0300
Subject: [Python-Dev] io.BytesIO slower than monkey-patching io.RawIOBase
In-Reply-To: <CAF-Rda-nXNDejQFBQygnRUuiBgoEvgMJtF1PYVx59AfnfnxOPg@mail.gmail.com>
References: <CAF-Rda-nXNDejQFBQygnRUuiBgoEvgMJtF1PYVx59AfnfnxOPg@mail.gmail.com>
Message-ID: <ju3k16$jie$1@dough.gmane.org>

On 17.07.12 06:34, Eli Bendersky wrote:
> The second approach is consistently 10-20% faster than the first one
> (depending on input) for trunk Python 3.3
>
> Is there any reason for this to be so? What does BytesIO give us that
> the second approach does not (I tried adding more methods to the patched
> RawIOBase to make it more functional, like seekable() and tell(), and it
> doesn't affect performance)?

BytesIO resizes underlying buffer if it overflowed (overallocating 1/8 
of size and copying old content to new buffer). Total it makes 
log[9/8](N) allocations and copy 8*N bytes (for large N). List uses the 
same strategy, but number of chunks usually significantly less than 
number of bytes. At the end all this chunks concatenated by join, which 
calculates sum of chunk lengths and allocate the resulting array with 
the desired size. That is why append/join is faster than BytesIO in this 
case.

There are other note, about ElementTree.tostringlist(). Creating 
DataStream class in every function call is too expensive, and that is 
why "monkeypatched" version several times is faster than DataStream 
version for small data. But for long data it is faster too, because 
data.append() is on one lookup slower than "monkeypatched" 
write=data.append.

> This also raises a "moral" question - should I be using the second
> approach deep inside the stdlib (ET.tostring) just because it's faster?

Please note that the previous version of Python used a "monkeypatching".


From eliben at gmail.com  Tue Jul 17 14:14:55 2012
From: eliben at gmail.com (Eli Bendersky)
Date: Tue, 17 Jul 2012 15:14:55 +0300
Subject: [Python-Dev] io.BytesIO slower than monkey-patching io.RawIOBase
In-Reply-To: <ju3k16$jie$1@dough.gmane.org>
References: <CAF-Rda-nXNDejQFBQygnRUuiBgoEvgMJtF1PYVx59AfnfnxOPg@mail.gmail.com>
	<ju3k16$jie$1@dough.gmane.org>
Message-ID: <CAF-Rda-6giMXquLBPKJTk0DxRTEknhY17Ae-Mc7eDT86GiPWRw@mail.gmail.com>

On Tue, Jul 17, 2012 at 2:59 PM, Serhiy Storchaka <storchaka at gmail.com>wrote:

> On 17.07.12 06:34, Eli Bendersky wrote:
>
>> The second approach is consistently 10-20% faster than the first one
>> (depending on input) for trunk Python 3.3
>>
>> Is there any reason for this to be so? What does BytesIO give us that
>> the second approach does not (I tried adding more methods to the patched
>> RawIOBase to make it more functional, like seekable() and tell(), and it
>> doesn't affect performance)?
>>
>
> BytesIO resizes underlying buffer if it overflowed (overallocating 1/8 of
> size and copying old content to new buffer). Total it makes log[9/8](N)
> allocations and copy 8*N bytes (for large N). List uses the same strategy,
> but number of chunks usually significantly less than number of bytes. At
> the end all this chunks concatenated by join, which calculates sum of chunk
> lengths and allocate the resulting array with the desired size. That is why
> append/join is faster than BytesIO in this case.
>
>
I've created http://bugs.python.org/issue15381 to track this (optimizing
BytesIO).


> There are other note, about ElementTree.tostringlist(). Creating
> DataStream class in every function call is too expensive, and that is why
> "monkeypatched" version several times is faster than DataStream version for
> small data. But for long data it is faster too, because data.append() is on
> one lookup slower than "monkeypatched" write=data.append.
>

I updated tostringlist() to use an outside class. This brings performance
back to the old code.

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120717/99491312/attachment.html>

From regebro at gmail.com  Tue Jul 17 16:02:05 2012
From: regebro at gmail.com (Lennart Regebro)
Date: Tue, 17 Jul 2012 16:02:05 +0200
Subject: [Python-Dev] PEP 0424: A method for exposing a length hint
In-Reply-To: <CAJVMporo4OtqmQbwsYxY55tsQsoG-OQqNDobnNo6v8NP8CBvXA@mail.gmail.com>
References: <loom.20120715T000919-641@post.gmane.org>
	<CAPZV6o-ViCYLuOHer88kpUk42ySv2qoUN9D_kUQ=e=iTA2-3=w@mail.gmail.com>
	<CAFRnB2Vbjf5s9nOc2CR8F=_bfhAHYz01miTLMGz63VqcYhJBMA@mail.gmail.com>
	<CAJVMporo4OtqmQbwsYxY55tsQsoG-OQqNDobnNo6v8NP8CBvXA@mail.gmail.com>
Message-ID: <CAL0kPAU1hi82GXOHgaHenJwwEd_jSPSN5jxOoxfDYHn1rtBEMA@mail.gmail.com>

On Sun, Jul 15, 2012 at 1:28 AM, Alexandre Zani
<alexandre.zani at gmail.com> wrote:
> I'm +1 on not having a public API for this. Ultimately the contract
> for a length hint will depend heavily upon what you need it for. Some
> applications would require a length hint to be an "at least" others an
> "at most" and others something else entirely. Given that the contract
> here appears to be >=0, I don't think the length hint is particularly
> useful to the public at large.

Other possible related uses could be to get an approximate number of
results for a query without having to actually go through the whole
query, useful for databases and search engines. But then you *do* want
__len__ as well, so that also doesn't fit with the current PEP.

But maybe that's a completely different usecase, even though it seems
related to me?

//Lennart

From victor.stinner at gmail.com  Tue Jul 17 19:34:05 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Jul 2012 19:34:05 +0200
Subject: [Python-Dev] Use function names instead of functions for
	os.supports_dir_fd?
Message-ID: <CAMpsgwaNK+Hot3moRo4BYgh9BQJHfJmRDJOANB2HsnNMwEK3nA@mail.gmail.com>

Hi,

Python 3.3 introduced os.supports_dir_fd to check if some os functions
do accept a file descriptor instead of a path. The problem is that
os.supports_dir_fd is a list of functions, not a list of function
names. If os functions are monkey patched, you cannot test anymore if
a function supports file descriptor.

Monkey patching is a common practice in Python. test_os.py replaces
os.exec*() functions temporary for example.

It's also inconsistent with the new time.get_clock_info() function
which expects the name of a time function, not the function directly.

Victor

From victor.stinner at gmail.com  Tue Jul 17 20:38:37 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 17 Jul 2012 20:38:37 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
Message-ID: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>

Hi,

I would like to write yet another JIT compiler for CPython. Before
writing anything, I would like your opinion because I don't know well
other Python compilers. I also want to prepare a possible integration
into CPython since the beginning of the project, or at least stay very
close to the CPython project (and CPython developers!). I did not
understand exactly why Unladen Swallow and psyco projects failed, so
please tell me if you think that my project is going to fail too!


== Why? ==

CPython is still the reference implementation, new features are first
added to this implementation (ex: PyPy is not supporting Python 3 yet,
but there is a project to support Python 3). Some projects still rely
on low level properties of CPython, especially its C API (ex: numpy;
PyPy has a cpyext module to emulate the CPython C API).

A JIT is the most promising solution to speed up the main evaluation
loop: using a JIT, it is possible to compile a function for a specific
type on the fly and so enable deeper optimizations.

psyco is no more maintained. It had its own JIT which is complex to
maintain. For example, it is hard to port it to a new hardware.

LLVM is fast and the next version will be faster. LLVM has a
community, a documentation, a lot of tools and is active.

There are many Python compilers which are very fast, but most of them
only support a subset of Python or require to modify the code (ex:
specify the type of all parameters and variables). For example, you
cannot run Django with Shredskin.

IMO PyPy is complex and hard to maintain. PyPy has a design completly
different than CPython and is much faster and has a better memory
footprint. I don't expect to be as fast as PyPy, just faster than
CPython.


== General idea ==

I don't want to replace CPython. This is an important point. All
others Python compilers try to write something completly new, which is
an huge task and is a problem to stay compatible with CPython. I would
like to reuse as much as possible code of CPython and don't try to
fight against the GIL or reference counting, but try to cooperate
instead.

I would like to use a JIT to generate specialized functions for a
combinaison of arguments types. Specialization enables more
optimizations. I would like to use LLVM because LLVM is an active
project, has many developers and users, is fast and the next version
will be faster! LLVM already supports common optimizations like
inlining.

My idea is to emit the same code than ceval.c from the bytecode to be
fully compatible with CPython, and then write a JIT to optimize
functions for a specific type.


== Roadmap ==

-- Milestone 1: Proof of concept --

 * Use the bytecode produced by CPython parser and compiler
 * Only compile a single function
 * Emit the same code than ceval.c using LLVM, but without tracing,
exceptions nor signal handling (they will be added later)
 * Support compiling and calling the following functions: "def func(a,
b): return a+b"

The pymothoa project can be used as a base to implement quickly such
proof of concept.

-- Milestone 2: Specialized function for the int type --

 * Use type annotation to generate specialized functions for the int type
 * Use C int with a guard detecting integer overflow to fallback on Python int

-- Milestone 3: JIT --

 * Depending on the type seen at runtime, recompile the function to
generate specialized functions
 * Use guard to fallback to a generic implementation if the type is
not the expected type
 * Drop maybe the code using function annotations

At this step, we can start to benchmark to check if the (JIT) compiler
is faster than CPython.

-- Later (unsorted ideas) --

 * Support exceptions
 * Full support of Python

  - classes
  - list comprehension
  - etc.

 * Optimizations:

   - avoid reference counting when possible
   - avoid temporary objects when possible
   - release the GIL when possible
   - inlining: should be very interesting with list comprehension
   - unroll loops?
   - lazy creation of the frame?

 * Use registers instead of a stack in the "evaluation loop"?
 * Add code to allow tracing and profiling
 * Add code to handle signals (pending calls)
 * Write a compiler using the AST, with a fallback to the bytecode?
(would it be faster? easier or more complex to maintain?)
 * Test LLVM optimizers
 * Compile a whole module or even a whole program
 * Reduce memory footprint
 * Type annotation to help the optimizer? (with guards?)
 * "const" annotation to help the optimizer? (with guards?)
 * Support any build option of Python:

   - support Python 2 (2.5, 2.6, 2.7) and 3 (3.1, 3.2, 3.3, 3.4)
   - support narrow and wide mode: flag at runtime?
   - support debug and release mode: flag at runtime?
   - support 32 and 64 bits mode on Windows?


== Other Python VM and compilers ==

-- Fully Python compliant --

 * `PyPy <http://pypy.org/>`_
 * `Jython <http://www.jython.org/>`_ based on the JVM
 * `IronPython <http://ironpython.net/>`_ based on the .NET VM
 * `Unladen Swallow <http://code.google.com/p/unladen-swallow/>`_ fork
of CPython 2.6 using LLVM

   - `Unladen Swallow Retrospective
     <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_
   - `PEP 3146 <http://python.org/dev/peps/pep-3146/>`_

 * `psyco <http://psyco.sourceforge.net/>`_ (fully Python compliant?),
no more maintained

-- Subset of Python to C++ --

 * `Nuitka <http://www.nuitka.net/pages/overview.html>`_
 * `Python2C <http://strout.net/info/coding/python/ai/python2c.py>`_
 * `Shedskin <http://code.google.com/p/shedskin/>`_
 * `pythran <https://github.com/serge-sans-paille/pythran>`_ (no
class, set, dict, exception, file handling, ...)

-- Subset of Python --

 * `pymothoa <http://code.google.com/p/pymothoa/>`_: use LLVM; don't
support classes nor exceptions.
 * `unpython <http://code.google.com/p/unpython/>`_: Python to C
 * `Perthon <http://perthon.sourceforge.net/>`_: Python to Perl
 * `Copperhead <http://copperhead.github.com/>`_: Python to GPU (Nvidia)

-- Language very close to Python --

 * `Cython <http://www.cython.org/>`_: "Cython is a programming
language based on Python, with extra syntax allowing for optional
static type declarations." Based on `Pyrex
<http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_


== See also ==

 * `Volunteer developed free-threaded cross platform virtual machines?
   <http://www.boredomandlaziness.org/2012/07/volunteer-supported-free-threaded-cross.html>`_

Victor Stinner

From stefan_ml at behnel.de  Tue Jul 17 20:58:55 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jul 2012 20:58:55 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
Message-ID: <ju4clf$9fr$1@dough.gmane.org>

Victor Stinner, 17.07.2012 20:38:
> -- Subset of Python --
> 
>  * `pymothoa <http://code.google.com/p/pymothoa/>`_: use LLVM; don't
> support classes nor exceptions.
>  * `unpython <http://code.google.com/p/unpython/>`_: Python to C
>  * `Perthon <http://perthon.sourceforge.net/>`_: Python to Perl
>  * `Copperhead <http://copperhead.github.com/>`_: Python to GPU (Nvidia)

You might also want to add numexpr and numba to that list. Numba might
actually be quite close to pymothoa (hadn't heard of it before).

Personally, I like the idea of having a JIT compiler more or less as an
extension module at hand. Sort-of like a co-processor, just in software.
Lets you run your code either interpreter or JITed, just as you need.

Note that the Cython project is working on a protocol to efficiently call
external C implemented Python functions by effectively unboxing them. That
explicitly includes JIT compiled code, and a JIT compiler could obviously
make good use of it from the other side as well.

Stefan


From d.s.seljebotn at astro.uio.no  Tue Jul 17 20:44:19 2012
From: d.s.seljebotn at astro.uio.no (Dag Sverre Seljebotn)
Date: Tue, 17 Jul 2012 20:44:19 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
Message-ID: <5005B283.3030007@astro.uio.no>

I'll admit I didn't read through your email, but you should absolutely 
check out Numba which is ramping up just now to do this:

https://github.com/numba

(I'm CC-ing their mailing list, perhaps some of them will read this and 
respond.)

It is probably much less ambitious but that hopefully shouldn't stop you 
cooperating.

It's started by Travis Oliphant (who started NumPy); here's his thoughts 
on PyPy and NumPy which provides some of the background for this project.

http://technicaldiscovery.blogspot.no/2011/10/thoughts-on-porting-numpy-to-pypy.html

Dag

On 07/17/2012 08:38 PM, Victor Stinner wrote:
> Hi,
>
> I would like to write yet another JIT compiler for CPython. Before
> writing anything, I would like your opinion because I don't know well
> other Python compilers. I also want to prepare a possible integration
> into CPython since the beginning of the project, or at least stay very
> close to the CPython project (and CPython developers!). I did not
> understand exactly why Unladen Swallow and psyco projects failed, so
> please tell me if you think that my project is going to fail too!
>
>
> == Why? ==
>
> CPython is still the reference implementation, new features are first
> added to this implementation (ex: PyPy is not supporting Python 3 yet,
> but there is a project to support Python 3). Some projects still rely
> on low level properties of CPython, especially its C API (ex: numpy;
> PyPy has a cpyext module to emulate the CPython C API).
>
> A JIT is the most promising solution to speed up the main evaluation
> loop: using a JIT, it is possible to compile a function for a specific
> type on the fly and so enable deeper optimizations.
>
> psyco is no more maintained. It had its own JIT which is complex to
> maintain. For example, it is hard to port it to a new hardware.
>
> LLVM is fast and the next version will be faster. LLVM has a
> community, a documentation, a lot of tools and is active.
>
> There are many Python compilers which are very fast, but most of them
> only support a subset of Python or require to modify the code (ex:
> specify the type of all parameters and variables). For example, you
> cannot run Django with Shredskin.
>
> IMO PyPy is complex and hard to maintain. PyPy has a design completly
> different than CPython and is much faster and has a better memory
> footprint. I don't expect to be as fast as PyPy, just faster than
> CPython.
>
>
> == General idea ==
>
> I don't want to replace CPython. This is an important point. All
> others Python compilers try to write something completly new, which is
> an huge task and is a problem to stay compatible with CPython. I would
> like to reuse as much as possible code of CPython and don't try to
> fight against the GIL or reference counting, but try to cooperate
> instead.
>
> I would like to use a JIT to generate specialized functions for a
> combinaison of arguments types. Specialization enables more
> optimizations. I would like to use LLVM because LLVM is an active
> project, has many developers and users, is fast and the next version
> will be faster! LLVM already supports common optimizations like
> inlining.
>
> My idea is to emit the same code than ceval.c from the bytecode to be
> fully compatible with CPython, and then write a JIT to optimize
> functions for a specific type.
>
>
> == Roadmap ==
>
> -- Milestone 1: Proof of concept --
>
>   * Use the bytecode produced by CPython parser and compiler
>   * Only compile a single function
>   * Emit the same code than ceval.c using LLVM, but without tracing,
> exceptions nor signal handling (they will be added later)
>   * Support compiling and calling the following functions: "def func(a,
> b): return a+b"
>
> The pymothoa project can be used as a base to implement quickly such
> proof of concept.
>
> -- Milestone 2: Specialized function for the int type --
>
>   * Use type annotation to generate specialized functions for the int type
>   * Use C int with a guard detecting integer overflow to fallback on Python int
>
> -- Milestone 3: JIT --
>
>   * Depending on the type seen at runtime, recompile the function to
> generate specialized functions
>   * Use guard to fallback to a generic implementation if the type is
> not the expected type
>   * Drop maybe the code using function annotations
>
> At this step, we can start to benchmark to check if the (JIT) compiler
> is faster than CPython.
>
> -- Later (unsorted ideas) --
>
>   * Support exceptions
>   * Full support of Python
>
>    - classes
>    - list comprehension
>    - etc.
>
>   * Optimizations:
>
>     - avoid reference counting when possible
>     - avoid temporary objects when possible
>     - release the GIL when possible
>     - inlining: should be very interesting with list comprehension
>     - unroll loops?
>     - lazy creation of the frame?
>
>   * Use registers instead of a stack in the "evaluation loop"?
>   * Add code to allow tracing and profiling
>   * Add code to handle signals (pending calls)
>   * Write a compiler using the AST, with a fallback to the bytecode?
> (would it be faster? easier or more complex to maintain?)
>   * Test LLVM optimizers
>   * Compile a whole module or even a whole program
>   * Reduce memory footprint
>   * Type annotation to help the optimizer? (with guards?)
>   * "const" annotation to help the optimizer? (with guards?)
>   * Support any build option of Python:
>
>     - support Python 2 (2.5, 2.6, 2.7) and 3 (3.1, 3.2, 3.3, 3.4)
>     - support narrow and wide mode: flag at runtime?
>     - support debug and release mode: flag at runtime?
>     - support 32 and 64 bits mode on Windows?
>
>
> == Other Python VM and compilers ==
>
> -- Fully Python compliant --
>
>   * `PyPy<http://pypy.org/>`_
>   * `Jython<http://www.jython.org/>`_ based on the JVM
>   * `IronPython<http://ironpython.net/>`_ based on the .NET VM
>   * `Unladen Swallow<http://code.google.com/p/unladen-swallow/>`_ fork
> of CPython 2.6 using LLVM
>
>     - `Unladen Swallow Retrospective
>       <http://qinsb.blogspot.com.au/2011/03/unladen-swallow-retrospective.html>`_
>     - `PEP 3146<http://python.org/dev/peps/pep-3146/>`_
>
>   * `psyco<http://psyco.sourceforge.net/>`_ (fully Python compliant?),
> no more maintained
>
> -- Subset of Python to C++ --
>
>   * `Nuitka<http://www.nuitka.net/pages/overview.html>`_
>   * `Python2C<http://strout.net/info/coding/python/ai/python2c.py>`_
>   * `Shedskin<http://code.google.com/p/shedskin/>`_
>   * `pythran<https://github.com/serge-sans-paille/pythran>`_ (no
> class, set, dict, exception, file handling, ...)
>
> -- Subset of Python --
>
>   * `pymothoa<http://code.google.com/p/pymothoa/>`_: use LLVM; don't
> support classes nor exceptions.
>   * `unpython<http://code.google.com/p/unpython/>`_: Python to C
>   * `Perthon<http://perthon.sourceforge.net/>`_: Python to Perl
>   * `Copperhead<http://copperhead.github.com/>`_: Python to GPU (Nvidia)
>
> -- Language very close to Python --
>
>   * `Cython<http://www.cython.org/>`_: "Cython is a programming
> language based on Python, with extra syntax allowing for optional
> static type declarations." Based on `Pyrex
> <http://www.cosc.canterbury.ac.nz/greg.ewing/python/Pyrex/>`_
>
>
> == See also ==
>
>   * `Volunteer developed free-threaded cross platform virtual machines?
>     <http://www.boredomandlaziness.org/2012/07/volunteer-supported-free-threaded-cross.html>`_
>
> Victor Stinner
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/d.s.seljebotn%40astro.uio.no


From amauryfa at gmail.com  Tue Jul 17 21:16:30 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Tue, 17 Jul 2012 21:16:30 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
Message-ID: <CAGmFidYr8cJoz0tu3i4JDggwd0b_aEd=Z9gfM5bQN4mFhEtjPg@mail.gmail.com>

Hi,

2012/7/17 Victor Stinner <victor.stinner at gmail.com>:
> -- Milestone 3: JIT --
>
>  * Depending on the type seen at runtime, recompile the function to
> generate specialized functions
>  * Use guard to fallback to a generic implementation if the type is
> not the expected type

>From my understanding, psyco did exactly this.

-- 
Amaury Forgeot d'Arc

From fijall at gmail.com  Tue Jul 17 21:16:32 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Tue, 17 Jul 2012 21:16:32 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
Message-ID: <CAK5idxSyMrmTFzG8BfTgDDNjpbWx-N1hBosb3WmZ=_m6T=rVVA@mail.gmail.com>

Hi Victor.

I'm willing to explain to you details why having LLVM does not solve almost
any issues and why PyPy is complex, or why you think it's complex. Find me
on IRC if you want (fijal, can be found on #pypy on freenode for example).
In our opinion something like psyco that gets brought to the levels of
speed of pypy would be massively more complex than PyPy, most importantly
it would be incredibly fragile. It's possible, but it's lots and lots of
work. I don't think it possibly can be done with one person.

Speaking about compatible with cpython and yet fast - I would strongly
recommend talking to Mark Shannon (the author of HotPy). He's by far the
best person who can answer some questions and have a rough plan how to go
forward. It would be much better to concentrate efforts rather than write
yet another half-finished JIT (because reading code is hard).

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120717/809da929/attachment-0001.html>

From stefan_ml at behnel.de  Tue Jul 17 21:28:41 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Tue, 17 Jul 2012 21:28:41 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAK5idxSyMrmTFzG8BfTgDDNjpbWx-N1hBosb3WmZ=_m6T=rVVA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<CAK5idxSyMrmTFzG8BfTgDDNjpbWx-N1hBosb3WmZ=_m6T=rVVA@mail.gmail.com>
Message-ID: <ju4ed9$nub$1@dough.gmane.org>

Maciej Fijalkowski, 17.07.2012 21:16:
> It would be much better to concentrate efforts rather than write
> yet another half-finished JIT (because reading code is hard).

+1

Stefan


From glyph at twistedmatrix.com  Tue Jul 17 21:43:33 2012
From: glyph at twistedmatrix.com (Glyph)
Date: Tue, 17 Jul 2012 12:43:33 -0700
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
Message-ID: <D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>


On Jul 17, 2012, at 11:38 AM, Victor Stinner <victor.stinner at gmail.com> wrote:

> IMO PyPy is complex and hard to maintain. PyPy has a design completly
> different than CPython and is much faster and has a better memory
> footprint. I don't expect to be as fast as PyPy, just faster than
> CPython.

I think this criticism is misguided.

Let's grant for the moment that you're right, and PyPy is complex and hard to maintain.  If a high-level Python parser and JIT compiler written in Python came out as complex and unmaintainable, why do you believe that they'll be easy to write in C?

You are correct that it has a different architecture than CPython: it has a different architecture because CPython's architecture is limiting because of its simplicity and makes it difficult to do things like write JIT compilers.  The output of the Unladen Swallow project was illuminating in that regard.  (Please note I said "output" and not "failure", the Unladen Swallow folks did the community a great service and produced many useful artifacts, even if they didn't meet their original goal.)

Polluting the straightforward, portable architecture of CPython with significant machine-specific optimizations to bolt on extra features that are already being worked on elsewhere seems like a waste of effort to me.  You could, instead, go work on documenting PyPy's architecture so it seems less arcane to newcomers.  Some of the things in there which look like hideous black magic are actually fairly straightforward when explained, as I have learned by being lucky enough to receive explanations in person from Maciej, Benjamin and Alex at various conferences.

I mean, don't get me wrong, if this worked out, I'd love a faster CPython, I do still use use many tools which don't support PyPy yet, so I can see the appeal of greater runtime compatibility with CPython than CPyExt offers.  I just think that it will end up being a big expenditure of effort for relatively little return.

If you disagree, you should feel no need to convince me; just go do it and prove me wrong, which I will be quite happy to be.  I would just like to think about whether this is the best use of your energy first.

But definitely listen to Maciej's suggestion about concentrating efforts with other people engaged in similar efforts, regardless :).  As your original message shows, there has already been enough duplication of effort in this area.

-glyph
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120717/807365a7/attachment.html>

From martin at v.loewis.de  Wed Jul 18 00:04:53 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Wed, 18 Jul 2012 00:04:53 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
Message-ID: <20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>

> I would like to write yet another JIT compiler for CPython.

FWIW, so do I.

> I did not
> understand exactly why Unladen Swallow and psyco projects failed, so
> please tell me if you think that my project is going to fail too!

It may well happen that your project fails, or doesn't even start. Mine
didn't start for the last two years (but now may eventually do start).

I'm not sure psyco really "failed"; if it did, it was because of PyPy:
PyPy was created to do the same stuff as psyco, just better. It was
abandoned in favor of PyPy - whether that's a failure of psyco, I
don't know. IMO, the psyco implementation itself failed because it was
unmaintainable, containing very complicated code that nobody but its
authors could understand.

Also, I know for a fact that Unladen Swallow (the project) didn't fail;
some interesting parts were contributed to Python and are now part of its
code base. It's the JIT compiler of Unladen Swallow that "failed"; in
my understanding because LLVM is crap (i.e. it is slow, memory-consuming,
and buggy) - as a low-level virtual machine; it may be ok as a compiler
backend (but I still think it is buggy there as well).

> psyco is no more maintained.

I think this is factually incorrect: Christian Tismer maintains it (IIUC).

> I would like to use a JIT to generate specialized functions for a
> combinaison of arguments types.

I think history has moved past specializing JITs. Tracing JITs are the
status quo; they provide specialization as a side effect.

Regards,
Martin



From martin at v.loewis.de  Wed Jul 18 00:12:26 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Wed, 18 Jul 2012 00:12:26 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
Message-ID: <20120718001226.Horde.kvoaR6GZi1VQBeNKvDRTJ7A@webmail.df.eu>

> If you disagree, you should feel no need to convince me; just go do  
> it and prove me wrong, which I will be quite happy to be.  I would  
> just like to think about whether this is the best use of your energy  
> first.

While I follow most of your reasoning, I think this is a flaw in your
logic. This is free software: the only person to decide where energy is
best used is the person providing the energy.

It may well be that Victor gives up after the first three steps, or
it may be that he comes back with a working prototype in August. He may
well find that his energy is *best* spent in this project, since it may
get him a highly-payed job, a university diploma, or reputation. If not
that, he'll learn a lot.

> But definitely listen to Maciej's suggestion about concentrating  
> efforts with other people engaged in similar efforts, regardless :).

Again, this thinking is flawed, IMO. It might be in the community's
interest if people coordinate, but not in the interest of the individual
contributor.

> As your original message shows, there has already been enough  
> duplication of effort in this area.

And that's not really a problem, IMO.

Regards,
Martin



From amauryfa at gmail.com  Wed Jul 18 00:14:36 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 18 Jul 2012 00:14:36 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
Message-ID: <CAGmFidYGVPEtox9kH+zpqNTpp5mWTXsGHkKkd+1_SgsSZfpyaA@mail.gmail.com>

2012/7/18  <martin at v.loewis.de>:
> > I would like to write yet another JIT compiler for CPython.
>
> FWIW, so do I.

I don't know whether it's good news (that Martin wants to put his
expertise in this area)
or a bad sign (that he did not start after so many years of Python
development - the problem becomes more and more difficult each time
one thinks about it)

-- 
Amaury Forgeot d'Arc

From victor.stinner at gmail.com  Wed Jul 18 00:15:53 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Jul 2012 00:15:53 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <ju4clf$9fr$1@dough.gmane.org>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<ju4clf$9fr$1@dough.gmane.org>
Message-ID: <CAMpsgwanLh6NZSJC+taVXWk0p1-yLP3P0AVypPSj4AyZ8iTN8A@mail.gmail.com>

> Personally, I like the idea of having a JIT compiler more or less as an
> extension module at hand. Sort-of like a co-processor, just in software.
> Lets you run your code either interpreter or JITed, just as you need.

Me too, so something like psyco. LLVM is written in C++ and may have
license issue, so I don't really want to add a dependency to LLVM to
CPython. For an experimental project, a third party module is also
more convinient.

Victor

From victor.stinner at gmail.com  Wed Jul 18 00:20:01 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Jul 2012 00:20:01 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
Message-ID: <CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>

> It's the JIT compiler of Unladen Swallow that "failed"; in
> my understanding because LLVM is crap (i.e. it is slow, memory-consuming,
> and buggy) - as a low-level virtual machine; it may be ok as a compiler
> backend (but I still think it is buggy there as well).

What is the status of LLVM nowadays? Is it not a good solution to
write a portable JIT?

I don't want to write my own library to generate machine code.

>> psyco is no more maintained.
>
> I think this is factually incorrect: Christian Tismer maintains it (IIUC).

http://psyco.sourceforge.net/ says:

"News, 12 March 2012

Psyco is unmaintained and dead. Please look at PyPy for the
state-of-the-art in JIT compilers for Python."

Victor

From ncoghlan at gmail.com  Wed Jul 18 00:28:11 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 18 Jul 2012 08:28:11 +1000
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
Message-ID: <CADiSq7e7R-LB2TUtkD5QrddbtoCcdH+_AJROBjQ3qwdYXhsF=A@mail.gmail.com>

On the cpyext front, it would be rather helpful if developers interested in
a high speed Python interpreter with good C extension compatibility worked
with Dave Malcolm on his static analyser for Python C extensions. One of
the reasons cpyext has trouble is that many refcounting bugs in extensions
aren't fatal on CPython?s due to additional internal references - a
refcount of 1 when it should be 2 is survivable in a way that 0 vs 1 is not.

Get rid of that drudgery from hacking on cpyext and it becomes
significantly easier to expand the number of extensions that will work
across multiple implementations of the API.

Cheers,
Nick.

--
Sent from my phone, thus the relative brevity :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120718/29eb33c2/attachment.html>

From jeanpierreda at gmail.com  Wed Jul 18 00:38:59 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Tue, 17 Jul 2012 18:38:59 -0400
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
Message-ID: <CABicbJ+nAJ_u=cgn88VsddGchOZrzSBv2NnpPhZ2G1EF5Sq+OQ@mail.gmail.com>

On Tue, Jul 17, 2012 at 6:20 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> What is the status of LLVM nowadays? Is it not a good solution to
> write a portable JIT?
>
> I don't want to write my own library to generate machine code.

You don't have to, even if you don't want to use LLVM. There are
plenty of "ligher-weight" approaches to that. For example, GNU
Lightning [1] or sljit [2].

[1] http://www.gnu.org/software/lightning/
[2] http://sljit.sourceforge.net/

-- Devin

From amauryfa at gmail.com  Wed Jul 18 00:49:18 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Wed, 18 Jul 2012 00:49:18 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CADiSq7e7R-LB2TUtkD5QrddbtoCcdH+_AJROBjQ3qwdYXhsF=A@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
	<CADiSq7e7R-LB2TUtkD5QrddbtoCcdH+_AJROBjQ3qwdYXhsF=A@mail.gmail.com>
Message-ID: <CAGmFida9Z1CoqZG29SEMC7BUvegFH7xkXJFtse6NXw4dw6ci8w@mail.gmail.com>

2012/7/18 Nick Coghlan <ncoghlan at gmail.com>:
> On the cpyext front, it would be rather helpful if developers interested in
> a high speed Python interpreter with good C extension compatibility worked
> with Dave Malcolm on his static analyser for Python C extensions. One of the
> reasons cpyext has trouble is that many refcounting bugs in extensions
> aren't fatal on CPython?s due to additional internal references - a refcount
> of 1 when it should be 2 is survivable in a way that 0 vs 1 is not.

It's not only about bugs. Even when reference counts are correctly
managed, cpyext is slow:

- each time an object crosses the C|pypy boundary, there is a dict lookup (!)

- each time a new object is passed or returned to C, a PyObject
structure must be allocated
(and sometime much more, specially for strings and types). Py_DECREF
will of course
free the PyObject, so next time will allocate the object again.

- borrowed references are a nightmare.

> Get rid of that drudgery from hacking on cpyext and it becomes significantly
> easier to expand the number of extensions that will work across multiple
> implementations of the API.

There are also some extension modules that play tricky games with the API;
PyQt for example uses metaclasses with a custom tp_alloc slot, to have access
to the PyTypeObject structure during the construction of the type...
The Python C API is quite complete, but some use cases are still
poorly supported.

-- 
Amaury Forgeot d'Arc

From yselivanov.ml at gmail.com  Wed Jul 18 01:07:50 2012
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Tue, 17 Jul 2012 19:07:50 -0400
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CABicbJ+nAJ_u=cgn88VsddGchOZrzSBv2NnpPhZ2G1EF5Sq+OQ@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<CABicbJ+nAJ_u=cgn88VsddGchOZrzSBv2NnpPhZ2G1EF5Sq+OQ@mail.gmail.com>
Message-ID: <02FB69A0-6FCE-4528-B88A-1B68DDDA4532@gmail.com>

On 2012-07-17, at 6:38 PM, Devin Jeanpierre wrote:

> On Tue, Jul 17, 2012 at 6:20 PM, Victor Stinner
> <victor.stinner at gmail.com> wrote:
>> What is the status of LLVM nowadays? Is it not a good solution to
>> write a portable JIT?
>> 
>> I don't want to write my own library to generate machine code.
> 
> You don't have to, even if you don't want to use LLVM. There are
> plenty of "ligher-weight" approaches to that. For example, GNU
> Lightning [1] or sljit [2].
> 
> [1] http://www.gnu.org/software/lightning/
> [2] http://sljit.sourceforge.net/

And, there is also DynASM [1], [2].  This one was built for
LuaJIT and is under MIT licence.

[1] http://luajit.org/dynasm.html
[2] https://github.com/LuaDist/luajit/tree/master/dynasm

-
Yury

From victor.stinner at gmail.com  Wed Jul 18 01:09:23 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Jul 2012 01:09:23 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
Message-ID: <CAMpsgwbMR9bewQGZeJYnuUMWVgDTHBH1uENXhjjP4nziWXEyFA@mail.gmail.com>

> As your original message shows, there has already been
> enough duplication of effort in this area.

I didn't find yet a project reusing ceval.c: most projects implement
their own eval loop and don't use CPython at all.

My idea is not to write something new, but just try to optimize the
existing ceval.c code. Pseudo-code:

 * read the bytecode of a function
 * replace each bytecode by its "C code"
 * optimize
 * compile the "C code" to machine code

(I don't know if "C code" is the right expression here, it's just for
the example)

Dummy example:
----
def mysum(a, b):
   return a+b
----

Python compiles it to bytecode as:
----
>>> dis.dis(mysum)
0 LOAD_FAST                0 (a)
3 LOAD_FAST                1 (b)
6 BINARY_ADD
7 RETURN_VALUE
----

The bytecode can be compiled to something like:
----
x = GETLOCAL(0); # "a"
if (x == NULL) /* error */
Py_INCREF(x);
PUSH(x);

x = GETLOCAL(1); # "b"
if (x == NULL) /* error */
Py_INCREF(x);
PUSH(x);

w = POP();
v = TOP();
x = PyNumber_Add(v, w);
Py_DECREF(v);
Py_DECREF(w);
if (x == NULL) /* error */
SET_TOP(x);

retval = POP();

return retval;
----

The calls to Py_INCREF() and Py_DEREF() can be removed. The code is no
more based on a loop: CPU prefers sequential code. The stack can be
replaced variables: the compiler (LLVM?) knows how to replace many
variables with a few variables, or even use CPU registers instead.

Example:
----
a = GETLOCAL(0); # "a"
if (a == NULL) /* error */
b = GETLOCAL(1); # "b"
if (b == NULL) /* error */
return PyNumber_Add(a, b);
----

I don't expect to run a program 10x faster, but I would be happy if I
can run arbitrary Python code 25% faster.

--

Specialization / tracing JIT can be seen as another project, or at
least added later.

Victor

From breamoreboy at yahoo.co.uk  Wed Jul 18 01:10:33 2012
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Wed, 18 Jul 2012 00:10:33 +0100
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
Message-ID: <ju4rbq$r14$1@dough.gmane.org>

On 17/07/2012 23:20, Victor Stinner wrote:
>
> http://psyco.sourceforge.net/ says:
>
> "News, 12 March 2012
>
> Psyco is unmaintained and dead. Please look at PyPy for the
> state-of-the-art in JIT compilers for Python."
>
> Victor
>

A search on pypi for JIT compilers gives no matches.

-- 
Cheers.

Mark Lawrence.




From larry at hastings.org  Wed Jul 18 01:13:33 2012
From: larry at hastings.org (Larry Hastings)
Date: Tue, 17 Jul 2012 16:13:33 -0700
Subject: [Python-Dev] Use function names instead of functions for
	os.supports_dir_fd?
In-Reply-To: <CAMpsgwaNK+Hot3moRo4BYgh9BQJHfJmRDJOANB2HsnNMwEK3nA@mail.gmail.com>
References: <CAMpsgwaNK+Hot3moRo4BYgh9BQJHfJmRDJOANB2HsnNMwEK3nA@mail.gmail.com>
Message-ID: <5005F19D.80707@hastings.org>

On 07/17/2012 10:34 AM, Victor Stinner wrote:
> Python 3.3 introduced os.supports_dir_fd to check if some os functions
> do accept a file descriptor instead of a path. The problem is that
> os.supports_dir_fd is a list of functions, not a list of function
> names. If os functions are monkey patched, you cannot test anymore if
> a function supports file descriptor.

If you're monkey-patching the function, you can monkey-patch 
os.supports_dir_fd too.

> Monkey patching is a common practice in Python. test_os.py replaces
> os.exec*() functions temporary for example.

For testing, yes.  It's not recommended for production code.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120717/f9d63070/attachment.html>

From steve at pearwood.info  Wed Jul 18 01:34:15 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 18 Jul 2012 09:34:15 +1000
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
Message-ID: <5005F677.20906@pearwood.info>

Victor Stinner wrote:

> == Other Python VM and compilers ==

As far as I know, these are all still active, although possibly experimental:

Pynie               (Python for the Parrot VM)
WPython             (16-bit word-codes instead of byte-codes)
HotPy               (high-performance optimizing VM for Python)
Skulpt              (Javascript implementation)
HoPe                (Python in Haskell)
Berp                (another Python in Haskell)


WPython in particular seems to be very promising, and quite fast. I don't 
understand why it doesn't get more attention (although I admit I can't 
criticise, since I haven't installed or used it myself).


http://www.pycon.it/media/stuff/slides/beyond-bytecode-a-wordcode-based-python.pdf


In the Java world, there are byte-code optimizers such as Soot, BLOAT and 
ProGuard which apparently can speed up Java significantly. As far as I can 
tell, in the Python world byte-code optimization is a severely neglected area. 
For good reason? No idea.



-- 
Steven

From steve at pearwood.info  Wed Jul 18 01:45:28 2012
From: steve at pearwood.info (Steven D'Aprano)
Date: Wed, 18 Jul 2012 09:45:28 +1000
Subject: [Python-Dev] Use function names instead of functions
	for	os.supports_dir_fd?
In-Reply-To: <CAMpsgwaNK+Hot3moRo4BYgh9BQJHfJmRDJOANB2HsnNMwEK3nA@mail.gmail.com>
References: <CAMpsgwaNK+Hot3moRo4BYgh9BQJHfJmRDJOANB2HsnNMwEK3nA@mail.gmail.com>
Message-ID: <5005F918.1050708@pearwood.info>

Victor Stinner wrote:
> Hi,
> 
> Python 3.3 introduced os.supports_dir_fd to check if some os functions
> do accept a file descriptor instead of a path. The problem is that
> os.supports_dir_fd is a list of functions, not a list of function
> names. If os functions are monkey patched, you cannot test anymore if
> a function supports file descriptor.

One of the dangers of monkey-patching.


> Monkey patching is a common practice in Python. test_os.py replaces
> os.exec*() functions temporary for example.


Perhaps for testing, but I don't think monkey-patching is common in production 
code. Perhaps you are thinking of Ruby :)


> It's also inconsistent with the new time.get_clock_info() function
> which expects the name of a time function, not the function directly.

Since functions are first-class objects in Python, and people should be used 
to passing functions around as parameters, perhaps it is better to say that 
get_clock_info is inconsistent with supports_dir_fd.

Personally, I prefer passing function objects rather than names, since the 
*name* of the function shouldn't matter. But since I recognise that other 
people may think differently, I would probably support passing both the name 
or the function object itself.



-- 
Steven


From larry at hastings.org  Wed Jul 18 01:55:23 2012
From: larry at hastings.org (Larry Hastings)
Date: Tue, 17 Jul 2012 16:55:23 -0700
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <5005F677.20906@pearwood.info>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<5005F677.20906@pearwood.info>
Message-ID: <5005FB6B.80304@hastings.org>

On 07/17/2012 04:34 PM, Steven D'Aprano wrote:
> As far as I know, these are all still active, although possibly 
> experimental:
>
> Pynie               (Python for the Parrot VM)
> WPython             (16-bit word-codes instead of byte-codes)
> [...]
> WPython in particular seems to be very promising, and quite fast. I 
> don't understand why it doesn't get more attention (although I admit I 
> can't criticise, since I haven't installed or used it myself).

Cesar (sp?) was at Mark's talk on HotPy at EuroPython.  We asked him if 
WPython was still active, and he said, nope, no community interest.

IIRC Pynie is basically dead too.

I don't know about the others,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120717/47602a17/attachment.html>

From victor.stinner at gmail.com  Wed Jul 18 02:26:14 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 18 Jul 2012 02:26:14 +0200
Subject: [Python-Dev] Use function names instead of functions for
	os.supports_dir_fd?
In-Reply-To: <5005F918.1050708@pearwood.info>
References: <CAMpsgwaNK+Hot3moRo4BYgh9BQJHfJmRDJOANB2HsnNMwEK3nA@mail.gmail.com>
	<5005F918.1050708@pearwood.info>
Message-ID: <CAMpsgwb5VaBnN7R=QKJpn+YAyXEThEttqrCW4mgCiz5Rjd6tQA@mail.gmail.com>

>> Monkey patching is a common practice in Python. test_os.py replaces
>> os.exec*() functions temporary for example.
>
> Perhaps for testing, but I don't think monkey-patching is common in
> production code. Perhaps you are thinking of Ruby :)

The gevent library does monkey-patch os.fork (and time.sleep and many
other functions), but gevent is maybe not ready for production? :-)

Victor

From alex.gaynor at gmail.com  Wed Jul 18 03:24:49 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Wed, 18 Jul 2012 01:24:49 +0000 (UTC)
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
	<CAMpsgwbMR9bewQGZeJYnuUMWVgDTHBH1uENXhjjP4nziWXEyFA@mail.gmail.com>
Message-ID: <loom.20120718T032212-695@post.gmane.org>

Victor Stinner <victor.stinner <at> gmail.com> writes:


> Example:
> ----
> a = GETLOCAL(0); # "a"
> if (a == NULL) /* error */
> b = GETLOCAL(1); # "b"
> if (b == NULL) /* error */
> return PyNumber_Add(a, b);
> ----
> 
> I don't expect to run a program 10x faster, but I would be happy if I
> can run arbitrary Python code 25% faster.
> 
> --
> 
> Specialization / tracing JIT can be seen as another project, or at
> least added later.
> 
> Victor
> 

This is almost exactly what Unladen Swallow originally did.  First, LLVM will 
not
do all of the optimizations you are expecting it to do out of the box.  It will
still have all the stack accesses, and it will have all of the ref counting
operations.  You can get a small speed boost from removing the interpretation
dispatch overhead, but you also explode your memory usage, and the speedups are
tiny.

Please, learn from Unladen Swallow and other's experiences, otherwise they're
for naught, and frankly we (python-dev) waste a lot of time.

Alex


From stefan_ml at behnel.de  Wed Jul 18 06:23:02 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 18 Jul 2012 06:23:02 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwanLh6NZSJC+taVXWk0p1-yLP3P0AVypPSj4AyZ8iTN8A@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<ju4clf$9fr$1@dough.gmane.org>
	<CAMpsgwanLh6NZSJC+taVXWk0p1-yLP3P0AVypPSj4AyZ8iTN8A@mail.gmail.com>
Message-ID: <ju5dn6$cvn$1@dough.gmane.org>

Victor Stinner, 18.07.2012 00:15:
>> Personally, I like the idea of having a JIT compiler more or less as an
>> extension module at hand. Sort-of like a co-processor, just in software.
>> Lets you run your code either interpreter or JITed, just as you need.
> 
> Me too, so something like psyco.

In the sense that it's a third party module, yes. Not in the sense of how
it hooks into the runtime. The intention would be that users explicitly run
their code in a JIT compiled environment, e.g. their template processing or
math code. The runtime wouldn't switch to a JIT compiler automatically for
"normal" code.

I mean, that could still become a feature at some point, but I find a
decorator or an exec-like interface quite acceptable, as long as it fails
loudly with "can't do that" if the JIT compiler doesn't support a specific
language feature.

Stefan


From stefan_ml at behnel.de  Wed Jul 18 07:14:25 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 18 Jul 2012 07:14:25 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <loom.20120718T032212-695@post.gmane.org>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
	<CAMpsgwbMR9bewQGZeJYnuUMWVgDTHBH1uENXhjjP4nziWXEyFA@mail.gmail.com>
	<loom.20120718T032212-695@post.gmane.org>
Message-ID: <ju5gng$uvd$1@dough.gmane.org>

Alex Gaynor, 18.07.2012 03:24:
> Victor Stinner writes:
>> Example:
>> ----
>> a = GETLOCAL(0); # "a"
>> if (a == NULL) /* error */
>> b = GETLOCAL(1); # "b"
>> if (b == NULL) /* error */
>> return PyNumber_Add(a, b);
>> ----
>>
>> I don't expect to run a program 10x faster, but I would be happy if I
>> can run arbitrary Python code 25% faster.
>>
>> --
>>
>> Specialization / tracing JIT can be seen as another project, or at
>> least added later.
> 
> This is almost exactly what Unladen Swallow originally did.  First, LLVM will 
> not
> do all of the optimizations you are expecting it to do out of the box.  It will
> still have all the stack accesses, and it will have all of the ref counting
> operations.  You can get a small speed boost from removing the interpretation
> dispatch overhead, but you also explode your memory usage, and the speedups are
> tiny.

My experience with Cython tells me that even if you move the entire
interpretation overhead out of the way, you'd only get some 5-20% speedup
for real code, rarely more if you have some really tight loops. Adding a
full-blown JIT compiler to the dependencies just for that is usually not
worth it, and Unladen Swallow succeeded in showing that pretty clearly.
It's when you start specialising and optimising code patterns that it
becomes really interesting, but you can do that statically at build time or
compile time in most cases (at least in the more interesting ones) and
Cython is one way to do it. Again, no need to add a JIT compiler.

The nice thing about JIT compilers is that you can give them your code and
they'll try to optimise it for you without further interaction. That
doesn't mean you get the fastest code ever, it just means that they do all
the profiling for you and try to figure it out all by themselves. That may
or may not work out, but it usually works quite ok (and you'll love JIT
compilers for it) and only rarely gets seriously in the way (and that's
when you'll hate JIT compilers).

However, it requires that the JIT compiler knows about a lot of
optimisations. PyPy's JIT is full of those. It's not the fact that it has a
JIT compiler at all that makes it fast and not the fact that they compile
Python to machine code, it's the fact that they came up with a huge bunch
of specialisations that makes lots of code patterns fast once it detected
them. LLVM (or any other low-level JIT compiler) won't help at all with that.

Stefan


From cesare.di.mauro at gmail.com  Wed Jul 18 07:50:23 2012
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Wed, 18 Jul 2012 07:50:23 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwbMR9bewQGZeJYnuUMWVgDTHBH1uENXhjjP4nziWXEyFA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
	<CAMpsgwbMR9bewQGZeJYnuUMWVgDTHBH1uENXhjjP4nziWXEyFA@mail.gmail.com>
Message-ID: <CAP7v7k7gD9b4faEeJNGaePrub0ZDcyrbh1GvRYpGKJq7G8zErQ@mail.gmail.com>

2012/7/18 Victor Stinner <victor.stinner at gmail.com>

> I don't expect to run a program 10x faster, but I would be happy if I
> can run arbitrary Python code 25% faster.
>

If that's your target, you don't need to resort to a
bytecode-to-binary-equivalent compiler. WPython already gave similar
results with Python 2.6.

The idea behind is that using an hybrid stack-register VM, you'll spend
less time on the ceval loop "constant stuff" (checking for events / GIL
release, etc.). That's because superinstructions aggregates more bytecodes
into a single "wordcode", which requires only one decoding phase, avoids
many pushes/pops, and some unnecessary inc/decr reference counting. A
better peephole optimizer is provided, and some other optimizations as well.

There's also room for more optimizations. I have many ideas to improve both
WPython or just the ceval loop. For example, at the last EuroPython sprint
I was working to a ceval optimization that gave about 10% speed improvement
for the CPython 3.3 beta trunk (on my old MacBook Air, running 32-bit
Windows 8 preview), but still needs to be checked for correctness (I'm
spending much more time running and checking the standard tests than for
its implementation ;-)

In the end, I think that a lot can be done to improve the good old CPython
VM, without resorting to a JIT compiler. Lack of time is the enemy...

Regards,
Cesare


> --
>
> Specialization / tracing JIT can be seen as another project, or at
> least added later.
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/cesare.di.mauro%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120718/41d75c78/attachment.html>

From martin at v.loewis.de  Wed Jul 18 07:53:14 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Wed, 18 Jul 2012 07:53:14 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
Message-ID: <20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>

> What is the status of LLVM nowadays? Is it not a good solution to
> write a portable JIT?

I don't think it is. It is still slow and memory hungry. The fact that
the version that Apple ships with Xcode still miscompiles Python 3.3
tells me that it is still buggy.

> I don't want to write my own library to generate machine code.

I plan to use nanojit.

Regards,
Martin



From martin at v.loewis.de  Wed Jul 18 07:55:01 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Wed, 18 Jul 2012 07:55:01 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <ju4rbq$r14$1@dough.gmane.org>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<ju4rbq$r14$1@dough.gmane.org>
Message-ID: <20120718075501.Horde.GV5ecLuWis5QBk_1SHahyAA@webmail.df.eu>


Zitat von Mark Lawrence <breamoreboy at yahoo.co.uk>:

> On 17/07/2012 23:20, Victor Stinner wrote:
>>
>> http://psyco.sourceforge.net/ says:
>>
>> "News, 12 March 2012
>>
>> Psyco is unmaintained and dead. Please look at PyPy for the
>> state-of-the-art in JIT compilers for Python."
>>
>> Victor
>>
>
> A search on pypi for JIT compilers gives no matches.

I think you misread: PyPy, not pypi.

Regards,
Martin



From martin at v.loewis.de  Wed Jul 18 07:56:45 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Wed, 18 Jul 2012 07:56:45 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <loom.20120718T032212-695@post.gmane.org>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
	<CAMpsgwbMR9bewQGZeJYnuUMWVgDTHBH1uENXhjjP4nziWXEyFA@mail.gmail.com>
	<loom.20120718T032212-695@post.gmane.org>
Message-ID: <20120718075645.Horde.57dfEruWis5QBlAduIbRwiA@webmail.df.eu>

> Please, learn from Unladen Swallow and other's experiences, otherwise they're
> for naught, and frankly we (python-dev) waste a lot of time.

Again: we (python-dev) won't waste much time (unless we chose to in  
discussions);
Victor may lose time, but then he may not.

Regards,
Martin



From cesare.di.mauro at gmail.com  Wed Jul 18 08:05:29 2012
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Wed, 18 Jul 2012 08:05:29 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <5005F677.20906@pearwood.info>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<5005F677.20906@pearwood.info>
Message-ID: <CAP7v7k6s6XRFrriE0cboMAWo4aiu26r1SfUhS7XaHzH6_t4AnA@mail.gmail.com>

2012/7/18 Steven D'Aprano <steve at pearwood.info>

> WPython in particular seems to be very promising, and quite fast. I don't
> understand why it doesn't get more attention (although I admit I can't
> criticise, since I haven't installed or used it myself).
>
>
> http://www.pycon.it/media/stuff/slides/beyond-bytecode-a-wordcode-based-python.pdf


Yes, that was the reason that brought me to stop the project: lack of
interest from python community. But at the last EuroPython I had the
opportunity to talk to Guido, so I think that I can try to port WPython
(and check for some ideas).

However the fault with WPython was mine: it wasn't a simple patch, so it
was very difficult to review. My bad.

In the Java world, there are byte-code optimizers such as Soot, BLOAT and
> ProGuard which apparently can speed up Java significantly. As far as I can
> tell, in the Python world byte-code optimization is a severely neglected
> area. For good reason? No idea.
>
> --
>  Steven
>

I think that Python case is different. You can't spend so many times on
optimizing the generating code, because usually the code is compiled at the
execution time. Startup time is an issue for Python, which is influenced so
much by the source-to-bytecode compilation.

Java is statically compiled, and then executed. So you can think about
using better optimizers until the code will finally run.

Regards,
Cesare
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120718/e3db2db0/attachment.html>

From stefan_ml at behnel.de  Wed Jul 18 08:27:14 2012
From: stefan_ml at behnel.de (Stefan Behnel)
Date: Wed, 18 Jul 2012 08:27:14 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>
Message-ID: <ju5l02$pu3$1@dough.gmane.org>

martin at v.loewis.de, 18.07.2012 07:53:
> [Victor Stinner]
>> I don't want to write my own library to generate machine code.
> 
> I plan to use nanojit.

As I said, generating machine code is the uninteresting part of it and
won't give you much of a win. The changes you make to the way the code
works while walking the path from seeing the Python code to generating
machine code is what makes the difference.

You could even skip the machine code generation all together and just go
with triggering pre-implemented high-level patterns from the interpreter.
If you code up the right patterns, your win would be bigger than with a
bare 1:1 mapping from Python code to machine code. Both Cython and WPython
are clear examples for that.

Stefan


From ncoghlan at gmail.com  Wed Jul 18 08:50:24 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 18 Jul 2012 16:50:24 +1000
Subject: [Python-Dev] Use function names instead of functions for
	os.supports_dir_fd?
In-Reply-To: <CAMpsgwb5VaBnN7R=QKJpn+YAyXEThEttqrCW4mgCiz5Rjd6tQA@mail.gmail.com>
References: <CAMpsgwaNK+Hot3moRo4BYgh9BQJHfJmRDJOANB2HsnNMwEK3nA@mail.gmail.com>
	<5005F918.1050708@pearwood.info>
	<CAMpsgwb5VaBnN7R=QKJpn+YAyXEThEttqrCW4mgCiz5Rjd6tQA@mail.gmail.com>
Message-ID: <CADiSq7eMXJamJ9G-YrTg0Cv9=n+M91J1BaBk4=7cw_Qp29BxPA@mail.gmail.com>

On Wed, Jul 18, 2012 at 10:26 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
>>> Monkey patching is a common practice in Python. test_os.py replaces
>>> os.exec*() functions temporary for example.
>>
>> Perhaps for testing, but I don't think monkey-patching is common in
>> production code. Perhaps you are thinking of Ruby :)
>
> The gevent library does monkey-patch os.fork (and time.sleep and many
> other functions), but gevent is maybe not ready for production? :-)

In many cases, it isn't. Quite often third party libraries require
updates to be compatible with gevent's modifications (and if they make
a blocking call to an API that gevent doesn't patch, then things
really don't work). However, if you don't have third party
dependencies that conflict with gevent, then gevent is a brilliant
approach to scaling network IO bound applications.

Monkey patching needs to be recognised for what it is: a fork of the
monkey-patched project. It's a fork that happens at runtime, but it's
a fork nonetheless. It comes with all the increased coupling that a
fork implies, just without the distribution complexity of trying to
have two versions of the monkey-patched project installed at the same
time.

In this case, it's a *good* thing that monkey-patching will change the
answers from the query functions, because we *don't know* if any
monkey-patched versions will support the additional arguments (in
fact, they probably won't). So, unless the monkey-patching project:
1. Updates their monkey-patched versions to support the additional
parameters; and
2. Correctly registers their patched versions in the query sets

Then 3.3 will actually give the right answer by assuming that any
monkey-patched versions don't support the new parameters. When they
*do* support the API additions, then they're easy to register (by
adding them to the relevant sets) when putting the monkey-patch in
place.

Using the functions directly also avoids the classic problem of
triggering the fallback path *all* the time by making a typo in the
query string (we actually had exactly that problem during the 3.3.
development cycle when the initial hasattr() checks in shutil degraded
gracefully when the previously separate functions were replaced with
the dir_fd keyword argument support).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From breamoreboy at yahoo.co.uk  Wed Jul 18 09:55:11 2012
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Wed, 18 Jul 2012 08:55:11 +0100
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <20120718075501.Horde.GV5ecLuWis5QBk_1SHahyAA@webmail.df.eu>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<ju4rbq$r14$1@dough.gmane.org>
	<20120718075501.Horde.GV5ecLuWis5QBk_1SHahyAA@webmail.df.eu>
Message-ID: <ju5q4h$tfi$1@dough.gmane.org>

On 18/07/2012 06:55, martin at v.loewis.de wrote:
>
> Zitat von Mark Lawrence <breamoreboy at yahoo.co.uk>:
>
>> On 17/07/2012 23:20, Victor Stinner wrote:
>>>
>>> http://psyco.sourceforge.net/ says:
>>>
>>> "News, 12 March 2012
>>>
>>> Psyco is unmaintained and dead. Please look at PyPy for the
>>> state-of-the-art in JIT compilers for Python."
>>>
>>> Victor
>>>
>>
>> A search on pypi for JIT compilers gives no matches.
>
> I think you misread: PyPy, not pypi.
>
> Regards,
> Martin
>
>

No think about it I did misread, my apologies for time wasting :-(

-- 
Cheers.

Mark Lawrence.




From nad at acm.org  Wed Jul 18 10:55:13 2012
From: nad at acm.org (Ned Deily)
Date: Wed, 18 Jul 2012 01:55:13 -0700
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>
Message-ID: <nad-0B2CFF.01551318072012@news.gmane.org>

In article 
<20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA at webmail.df.eu>,
 martin at v.loewis.de wrote:
> I don't think it is. It is still slow and memory hungry. The fact that
> the version that Apple ships with Xcode still miscompiles Python 3.3
> tells me that it is still buggy.

Whether LLVM is suitable for a JIT is an interesting question but it's 
not LLVM per se that is the problem with compiling 3.3.  Apple ships two 
C compiler chains with Xcode 4 for OS X 10.7, both of them are based on 
LLVM.  It's the Apple transitional gcc-4.2 frontend with an old LLVM 
backend that is problematic (and not to be confused with the "pure" 
gcc-4.2 shipped with Xcode 3).  That compiler was the default compiler 
for early releases of Xcode 4 and for building OS X 10.7.  It has been 
frozen for a long time because Apple's efforts have been going into 
transitioning the OS X world to the new compiler: a clang frontend with 
a more current LLVM backend.  The latest releases of Xcode 4 now use 
clang-llvm as the default and that's what we now use as a default for 
building Python 3.3 with Xcode 4.  That transition will be complete with 
the imminent release of OS X 10.8 Mountain Lion when the whole OS is 
built with clang-llvm. The iOS world is already there.

-- 
 Ned Deily,
 nad at acm.org


From fijall at gmail.com  Wed Jul 18 11:10:32 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 18 Jul 2012 11:10:32 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <ju5gng$uvd$1@dough.gmane.org>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<D2D7E7BE-3C2E-4C6D-AD1F-11516E77AF74@twistedmatrix.com>
	<CAMpsgwbMR9bewQGZeJYnuUMWVgDTHBH1uENXhjjP4nziWXEyFA@mail.gmail.com>
	<loom.20120718T032212-695@post.gmane.org>
	<ju5gng$uvd$1@dough.gmane.org>
Message-ID: <CAK5idxSE08c91S5qmf5aLCLSmxuaVW8coch9g-+4gydv=X4+tQ@mail.gmail.com>

>
>
> However, it requires that the JIT compiler knows about a lot of
> optimisations. PyPy's JIT is full of those. It's not the fact that it has a
> JIT compiler at all that makes it fast and not the fact that they compile
> Python to machine code, it's the fact that they came up with a huge bunch
> of specialisations that makes lots of code patterns fast once it detected
> them. LLVM (or any other low-level JIT compiler) won't help at all with
> that.
>
> Stefan
>

Very good point Stefan

I would just like to add that a lot of those also require changes in the
object model which might render changes in CPython C API (like the
introduction of maps). Certainly you can't keep the current C structures,
which will already break some code.

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120718/61d22294/attachment.html>

From fijall at gmail.com  Wed Jul 18 11:14:33 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Wed, 18 Jul 2012 11:14:33 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <ju5l02$pu3$1@dough.gmane.org>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>
	<ju5l02$pu3$1@dough.gmane.org>
Message-ID: <CAK5idxSeE9AT5RjkbmVf3DKHwKLPpipW+znBR6-AkVRJ-m6huQ@mail.gmail.com>

On Wed, Jul 18, 2012 at 8:27 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:

> martin at v.loewis.de, 18.07.2012 07:53:
> > [Victor Stinner]
> >> I don't want to write my own library to generate machine code.
> >
> > I plan to use nanojit.
>
> As I said, generating machine code is the uninteresting part of it and
> won't give you much of a win. The changes you make to the way the code
> works while walking the path from seeing the Python code to generating
> machine code is what makes the difference.
>
> You could even skip the machine code generation all together and just go
> with triggering pre-implemented high-level patterns from the interpreter.
> If you code up the right patterns, your win would be bigger than with a
> bare 1:1 mapping from Python code to machine code. Both Cython and WPython
> are clear examples for that.
>
> Stefan
>
>
It's uninteresting but it's completely necessary and it's still quite a bit
of work. For the PyPy needs llvm failed to provide some features (besides
being buggy), like dynamic patching of compiled assembler (you kind of need
that for the tracing JIT when you discover new paths) or speed of execution.

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120718/4d7973cf/attachment.html>

From mark at hotpy.org  Wed Jul 18 11:45:25 2012
From: mark at hotpy.org (Mark Shannon)
Date: Wed, 18 Jul 2012 10:45:25 +0100
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
Message-ID: <500685B5.2020507@hotpy.org>

Some of my (reasonably well informed) opinions on this subject...

The theory
----------

Don't think in terms of speeding up your program.
Think in terms of reducing the time spent executing your program.

Performance is improved by removing aspects of the execution overhead.
In a talk I gave at EuroPython 2010(1), I divided the overhead into 5 parts:

Interpretive
Imprecise type information
Parameter Handling & Call overhead
Lookups (globals/builtins/attributes)
Memory management (garbage collection)

For optimising CPython, we cannot change the GC from ref-counting,
but the other 4 apply, plus boxing and unboxing of floats and ints.

Compilation (by which I assume people mean converting bytecodes to machine code)
addresses only the first point (by definition).

I worry that Victor is proposing to make the same mistake made by Unladen Swallow,
which is to attack the interpretive overhead first, then attack the other overheads.
This is the wrong way around. If you want good performance, JIT compilation should
come last not first.

Results from my PhD thesis(2) show that the original HotPy without any JIT
compilation outperformed Unladen Swallow using JIT compilation.
In other words, an optimising interpreter for Python will be faster than a
naive JIT compiler. The optimised bytecode traces in an optimising interpreter
are much better input for a JIT compiler than the original bytecodes.

The practice
------------

If you want modest speedup for modest effort, then look at Cesare's WPython.
Also take a look at Stefan Brunthaler's work on inline caching in an interpreter.

If you want a larger speedup then you need to tackle most or all of the causes of
execution overhead listed above.
HotPy (version 2, a fork of CPython) aims to tackle all of these causes except
the GC overhead. As far as I am aware, it is the only project that does so.

Please take a look at www.hotpy.org for more information on HotPy.
You can see my talk from EuroPython 2011(3) on the ideas behind it and
from EuroPython 2012(4) on the current implementation.



Finally, a defence of LLVM.

LLVM is a quality piece of software. It may have some bugs, so does all software.
The code-generation components are designed with static compilation in mind,
so they do use a lot of memory and run slowly for a JIT compiler,
but they produce excellent quality code.
And don't forget the old saying about blaming your tools ;)

If HotPy (version 2) were to have an (optional) JIT I would expect it to be LLVM based.
The JIT can run in a separate thread, while the optimised code continues to run
in the interpreter, patching in the machine code when it is complete.

Cheers,
Mark.

1) Talk at EuroPython 2010
Slides: www.dcs.gla.ac.uk/~marks/comparison.pdf
Video: http://blip.tv/europythonvideos/mark_shannon-_hotpy_a_comparison-3999872
The information in the talk is a bit out of date; PyPy now includes out-of-line guards.

2) theses.gla.ac.uk/2975/01/2011shannonphd.pdf

3) Talk at EuroPython 2011
https://ep2012.europython.eu/conference/talks/making-cpython-fast-using-trace-based-optimisations

4) Talk at EuroPythnon 2012
https://ep2012.europython.eu/conference/talks/hotpy-2-a-high-performance-binary-compatible-virtual-machine-for-python





From ncoghlan at gmail.com  Wed Jul 18 12:30:04 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 18 Jul 2012 20:30:04 +1000
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <500685B5.2020507@hotpy.org>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<500685B5.2020507@hotpy.org>
Message-ID: <CADiSq7ffwJWKJcxGC6UM0dYmXLfvv=BvUqLx1WX0Bhybd4pr4g@mail.gmail.com>

On Wed, Jul 18, 2012 at 7:45 PM, Mark Shannon <mark at hotpy.org> wrote:
> The practice
> ------------
>
> If you want modest speedup for modest effort, then look at Cesare's WPython.
> Also take a look at Stefan Brunthaler's work on inline caching in an
> interpreter.
>
> If you want a larger speedup then you need to tackle most or all of the
> causes of
> execution overhead listed above.
> HotPy (version 2, a fork of CPython) aims to tackle all of these causes
> except
> the GC overhead. As far as I am aware, it is the only project that does so.

Indeed, there's a lot that could be done in the CPython compiler and
eval loop, and the bytecode design used to communicate between them.
Thanks for summarising that so clearly.

There are a couple of other compiler related patches that anyone
interested in optimisation of CPython should at least glance at:

- Dave Malcolm's patch that adds a Python level AST optimisation step
to the compiler (effectively trading slower compile times for faster
execution of the compiled bytecode)
(http://bugs.python.org/issue10399)
- Eugene Toder's patch to add an AST optimisation step to the compiler
chain (http://bugs.python.org/issue11549) (I've asked Eugene about
this patch more recently and his current thought is that subsequent
improvements to the peephole optimisation have rendered it less
valuable. However, the patch is still a potentially useful resource
for anyone investigating bytecode optimisation ideas)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ronaldoussoren at mac.com  Wed Jul 18 13:12:54 2012
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 18 Jul 2012 13:12:54 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>
Message-ID: <ED701F1C-14A2-43F5-A125-32E5AAD467B1@mac.com>


On 18 Jul, 2012, at 7:53, martin at v.loewis.de wrote:

>> What is the status of LLVM nowadays? Is it not a good solution to
>> write a portable JIT?
> 
> I don't think it is. It is still slow and memory hungry. The fact that
> the version that Apple ships with Xcode still miscompiles Python 3.3
> tells me that it is still buggy.

Does it miscompile? I regularly run the 3.3 testsuite using the latest Xcode from the Appstore on a OSX Lion machine and that works properly. The only unexpected test failures are in ctypes, which is probably caused by the way the clang developers interpret the ABI w.r.t. passing small values.

Ronald
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4788 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120718/109173e1/attachment.bin>

From eliben at gmail.com  Wed Jul 18 14:46:38 2012
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 18 Jul 2012 15:46:38 +0300
Subject: [Python-Dev] PyBytes_Join
In-Reply-To: <CAF-Rda8YLdOqF4OHr5wTd2S9ND8SYNFSK8ZGdpyWC01=iQtw6Q@mail.gmail.com>
References: <CAF-Rda8YLdOqF4OHr5wTd2S9ND8SYNFSK8ZGdpyWC01=iQtw6Q@mail.gmail.com>
Message-ID: <CAF-Rda9tgdxC7upUECp0Eu0acibCtMQTgF82m84AX4NtDVq-yg@mail.gmail.com>

On Wed, Jul 18, 2012 at 3:35 PM, Eli Bendersky <eliben at gmail.com> wrote:

> Hi,
>
> PyUnicode_Join is a public C API function. However, its cousin for the
> bytes object is tucked privately in Objects/bytesobject.c as _PyBytes_Join.
> Is there any harm in exposing it publicly?
>
>
A more correctly formulated question would be: why is _PyBytes_Join part of
the limited API, while PyUnicode_Join is in the full API. Is there a reason
for the former to be less stable than the latter?

Thanks in advance,
Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120718/bbd1ac3a/attachment.html>

From eliben at gmail.com  Wed Jul 18 14:35:15 2012
From: eliben at gmail.com (Eli Bendersky)
Date: Wed, 18 Jul 2012 15:35:15 +0300
Subject: [Python-Dev] PyBytes_Join
Message-ID: <CAF-Rda8YLdOqF4OHr5wTd2S9ND8SYNFSK8ZGdpyWC01=iQtw6Q@mail.gmail.com>

Hi,

PyUnicode_Join is a public C API function. However, its cousin for the
bytes object is tucked privately in Objects/bytesobject.c as _PyBytes_Join.
Is there any harm in exposing it publicly?

Eli
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120718/432aba9f/attachment.html>

From martin at v.loewis.de  Wed Jul 18 16:59:14 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Wed, 18 Jul 2012 16:59:14 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <ED701F1C-14A2-43F5-A125-32E5AAD467B1@mac.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>
	<ED701F1C-14A2-43F5-A125-32E5AAD467B1@mac.com>
Message-ID: <20120718165914.Horde.a4ccVklCcOxQBs9CvpTDH4A@webmail.df.eu>


Zitat von Ronald Oussoren <ronaldoussoren at mac.com>:

> On 18 Jul, 2012, at 7:53, martin at v.loewis.de wrote:
>
>>> What is the status of LLVM nowadays? Is it not a good solution to
>>> write a portable JIT?
>>
>> I don't think it is. It is still slow and memory hungry. The fact that
>> the version that Apple ships with Xcode still miscompiles Python 3.3
>> tells me that it is still buggy.
>
> Does it miscompile?

I'm talking about the bug in

http://mail.python.org/pipermail/python-dev/2011-September/113731.html

> I regularly run the 3.3 testsuite using the latest Xcode from the  
> Appstore on a OSX Lion machine and that works properly.

I'm not actually using the latest Xcode. So if you could test my test
program, that would be much appreciated.

Regards,
Martin



From ronaldoussoren at mac.com  Wed Jul 18 17:15:18 2012
From: ronaldoussoren at mac.com (Ronald Oussoren)
Date: Wed, 18 Jul 2012 17:15:18 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <20120718165914.Horde.a4ccVklCcOxQBs9CvpTDH4A@webmail.df.eu>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>
	<ED701F1C-14A2-43F5-A125-32E5AAD467B1@mac.com>
	<20120718165914.Horde.a4ccVklCcOxQBs9CvpTDH4A@webmail.df.eu>
Message-ID: <33283836-894E-4018-8B3F-9F799DB8B064@mac.com>


On 18 Jul, 2012, at 16:59, martin at v.loewis.de wrote:

> 
> Zitat von Ronald Oussoren <ronaldoussoren at mac.com>:
> 
>> On 18 Jul, 2012, at 7:53, martin at v.loewis.de wrote:
>> 
>>>> What is the status of LLVM nowadays? Is it not a good solution to
>>>> write a portable JIT?
>>> 
>>> I don't think it is. It is still slow and memory hungry. The fact that
>>> the version that Apple ships with Xcode still miscompiles Python 3.3
>>> tells me that it is still buggy.
>> 
>> Does it miscompile?
> 
> I'm talking about the bug in
> 
> http://mail.python.org/pipermail/python-dev/2011-September/113731.html
> 
>> I regularly run the 3.3 testsuite using the latest Xcode from the Appstore on a OSX Lion machine and that works properly.
> 
> I'm not actually using the latest Xcode. So if you could test my test
> program, that would be much appreciated.

That bug in llvm-gcc still exists, and is unlikely to get fixed.  That's a bug in the integretion of the GCC frontend and LLVM backend, clang (LLVM project frontend + LLVM backend) does work.

Ronald

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4788 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120718/83af4f5a/attachment.bin>

From jimjjewett at gmail.com  Wed Jul 18 18:50:30 2012
From: jimjjewett at gmail.com (Jim Jewett)
Date: Wed, 18 Jul 2012 12:50:30 -0400
Subject: [Python-Dev] [Python-checkins] cpython: Close #15387:
 inspect.getmodulename() now uses a new
In-Reply-To: <3Wcf3h3GZTzPFg@mail.python.org>
References: <3Wcf3h3GZTzPFg@mail.python.org>
Message-ID: <CA+OGgf6uNMtL9AkUAroopDTA-qvpPtT85h8nAnkGVTJA-41LbQ@mail.gmail.com>

Why is inspect.getmoduleinfo() deprecated?  Is it just to remove
circular dependencies?

FWIW, I much prefer an API like:

    tell_me_about(object)

to one like:

    for test_data in (X, Y, Z):
        usable = tester(object, test_data)
        if valid(usable):
            return possible_results[test_data]

and to me, inspect.getmoduleinfo(path) looks like the first, while
checking the various import.machinery.*SUFFIXES looks like the second.

-jJ

On 7/18/12, nick.coghlan <python-checkins at python.org> wrote:
> http://hg.python.org/cpython/rev/af7961e1c362
> changeset:   78161:af7961e1c362
> user:        Nick Coghlan <ncoghlan at gmail.com>
> date:        Wed Jul 18 23:14:57 2012 +1000
> summary:
>   Close #15387: inspect.getmodulename() now uses a new
> importlib.machinery.all_suffixes() API rather than the deprecated
> inspect.getmoduleinfo()
>
> files:
>   Doc/library/importlib.rst  |  13 ++++++++++++-
>   Doc/library/inspect.rst    |  15 ++++++++++++---
>   Lib/importlib/machinery.py |   4 ++++
>   Lib/inspect.py             |  11 +++++++++--
>   Misc/NEWS                  |   3 +++
>   5 files changed, 40 insertions(+), 6 deletions(-)
>
>
> diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
> --- a/Doc/library/importlib.rst
> +++ b/Doc/library/importlib.rst
> @@ -533,12 +533,23 @@
>
>  .. attribute:: EXTENSION_SUFFIXES
>
> -   A list of strings representing the the recognized file suffixes for
> +   A list of strings representing the recognized file suffixes for
>     extension modules.
>
>     .. versionadded:: 3.3
>
>
> +.. func:: all_suffixes()
> +
> +   Returns a combined list of strings representing all file suffixes for
> +   Python modules recognized by the standard import machinery. This is a
> +   helper for code which simply needs to know if a filesystem path
> +   potentially represents a Python module (for example,
> +   :func:`inspect.getmodulename`)
> +
> +   .. versionadded:: 3.3
> +
> +
>  .. class:: BuiltinImporter
>
>      An :term:`importer` for built-in modules. All known built-in modules
> are
> diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
> --- a/Doc/library/inspect.rst
> +++ b/Doc/library/inspect.rst
> @@ -198,9 +198,18 @@
>  .. function:: getmodulename(path)
>
>     Return the name of the module named by the file *path*, without
> including the
> -   names of enclosing packages.  This uses the same algorithm as the
> interpreter
> -   uses when searching for modules.  If the name cannot be matched
> according to the
> -   interpreter's rules, ``None`` is returned.
> +   names of enclosing packages. The file extension is checked against all
> of
> +   the entries in :func:`importlib.machinery.all_suffixes`. If it matches,
> +   the final path component is returned with the extension removed.
> +   Otherwise, ``None`` is returned.
> +
> +   Note that this function *only* returns a meaningful name for actual
> +   Python modules - paths that potentially refer to Python packages will
> +   still return ``None``.
> +
> +   .. versionchanged:: 3.3
> +      This function is now based directly on :mod:`importlib` rather than
> the
> +      deprecated :func:`getmoduleinfo`.
>
>
>  .. function:: ismodule(object)
> diff --git a/Lib/importlib/machinery.py b/Lib/importlib/machinery.py
> --- a/Lib/importlib/machinery.py
> +++ b/Lib/importlib/machinery.py
> @@ -13,3 +13,7 @@
>  from ._bootstrap import ExtensionFileLoader
>
>  EXTENSION_SUFFIXES = _imp.extension_suffixes()
> +
> +def all_suffixes():
> +    """Returns a list of all recognized module suffixes for this process"""
> +    return SOURCE_SUFFIXES + BYTECODE_SUFFIXES + EXTENSION_SUFFIXES
> diff --git a/Lib/inspect.py b/Lib/inspect.py
> --- a/Lib/inspect.py
> +++ b/Lib/inspect.py
> @@ -450,8 +450,15 @@
>
>  def getmodulename(path):
>      """Return the module name for a given file, or None."""
> -    info = getmoduleinfo(path)
> -    if info: return info[0]
> +    fname = os.path.basename(path)
> +    # Check for paths that look like an actual module file
> +    suffixes = [(-len(suffix), suffix)
> +                    for suffix in importlib.machinery.all_suffixes()]
> +    suffixes.sort() # try longest suffixes first, in case they overlap
> +    for neglen, suffix in suffixes:
> +        if fname.endswith(suffix):
> +            return fname[:neglen]
> +    return None
>
>  def getsourcefile(object):
>      """Return the filename that can be used to locate an object's source.
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -41,6 +41,9 @@
>  Library
>  -------
>
> +- Issue #15397: inspect.getmodulename() is now based directly on importlib
> +  via a new importlib.machinery.all_suffixes() API.
> +
>  - Issue #14635: telnetlib will use poll() rather than select() when
> possible
>    to avoid failing due to the select() file descriptor limit.
>
>
> --
> Repository URL: http://hg.python.org/cpython
>

From ncoghlan at gmail.com  Thu Jul 19 01:31:52 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 19 Jul 2012 09:31:52 +1000
Subject: [Python-Dev] [Python-checkins] cpython: Close #15387:
 inspect.getmodulename() now uses a new
In-Reply-To: <CA+OGgf6uNMtL9AkUAroopDTA-qvpPtT85h8nAnkGVTJA-41LbQ@mail.gmail.com>
References: <3Wcf3h3GZTzPFg@mail.python.org>
	<CA+OGgf6uNMtL9AkUAroopDTA-qvpPtT85h8nAnkGVTJA-41LbQ@mail.gmail.com>
Message-ID: <CADiSq7dgajCEr4V-PQ1PcGQLHU_xgBON3EvxGBijLNFm2H6YRg@mail.gmail.com>

Because the concepts it is based on are no longer used internally -
determining the kind of module is now the province of importlib's finders
and loaders.

--
Sent from my phone, thus the relative brevity :)
On Jul 19, 2012 2:50 AM, "Jim Jewett" <jimjjewett at gmail.com> wrote:

> Why is inspect.getmoduleinfo() deprecated?  Is it just to remove
> circular dependencies?
>
> FWIW, I much prefer an API like:
>
>     tell_me_about(object)
>
> to one like:
>
>     for test_data in (X, Y, Z):
>         usable = tester(object, test_data)
>         if valid(usable):
>             return possible_results[test_data]
>
> and to me, inspect.getmoduleinfo(path) looks like the first, while
> checking the various import.machinery.*SUFFIXES looks like the second.
>
> -jJ
>
> On 7/18/12, nick.coghlan <python-checkins at python.org> wrote:
> > http://hg.python.org/cpython/rev/af7961e1c362
> > changeset:   78161:af7961e1c362
> > user:        Nick Coghlan <ncoghlan at gmail.com>
> > date:        Wed Jul 18 23:14:57 2012 +1000
> > summary:
> >   Close #15387: inspect.getmodulename() now uses a new
> > importlib.machinery.all_suffixes() API rather than the deprecated
> > inspect.getmoduleinfo()
> >
> > files:
> >   Doc/library/importlib.rst  |  13 ++++++++++++-
> >   Doc/library/inspect.rst    |  15 ++++++++++++---
> >   Lib/importlib/machinery.py |   4 ++++
> >   Lib/inspect.py             |  11 +++++++++--
> >   Misc/NEWS                  |   3 +++
> >   5 files changed, 40 insertions(+), 6 deletions(-)
> >
> >
> > diff --git a/Doc/library/importlib.rst b/Doc/library/importlib.rst
> > --- a/Doc/library/importlib.rst
> > +++ b/Doc/library/importlib.rst
> > @@ -533,12 +533,23 @@
> >
> >  .. attribute:: EXTENSION_SUFFIXES
> >
> > -   A list of strings representing the the recognized file suffixes for
> > +   A list of strings representing the recognized file suffixes for
> >     extension modules.
> >
> >     .. versionadded:: 3.3
> >
> >
> > +.. func:: all_suffixes()
> > +
> > +   Returns a combined list of strings representing all file suffixes for
> > +   Python modules recognized by the standard import machinery. This is a
> > +   helper for code which simply needs to know if a filesystem path
> > +   potentially represents a Python module (for example,
> > +   :func:`inspect.getmodulename`)
> > +
> > +   .. versionadded:: 3.3
> > +
> > +
> >  .. class:: BuiltinImporter
> >
> >      An :term:`importer` for built-in modules. All known built-in modules
> > are
> > diff --git a/Doc/library/inspect.rst b/Doc/library/inspect.rst
> > --- a/Doc/library/inspect.rst
> > +++ b/Doc/library/inspect.rst
> > @@ -198,9 +198,18 @@
> >  .. function:: getmodulename(path)
> >
> >     Return the name of the module named by the file *path*, without
> > including the
> > -   names of enclosing packages.  This uses the same algorithm as the
> > interpreter
> > -   uses when searching for modules.  If the name cannot be matched
> > according to the
> > -   interpreter's rules, ``None`` is returned.
> > +   names of enclosing packages. The file extension is checked against
> all
> > of
> > +   the entries in :func:`importlib.machinery.all_suffixes`. If it
> matches,
> > +   the final path component is returned with the extension removed.
> > +   Otherwise, ``None`` is returned.
> > +
> > +   Note that this function *only* returns a meaningful name for actual
> > +   Python modules - paths that potentially refer to Python packages will
> > +   still return ``None``.
> > +
> > +   .. versionchanged:: 3.3
> > +      This function is now based directly on :mod:`importlib` rather
> than
> > the
> > +      deprecated :func:`getmoduleinfo`.
> >
> >
> >  .. function:: ismodule(object)
> > diff --git a/Lib/importlib/machinery.py b/Lib/importlib/machinery.py
> > --- a/Lib/importlib/machinery.py
> > +++ b/Lib/importlib/machinery.py
> > @@ -13,3 +13,7 @@
> >  from ._bootstrap import ExtensionFileLoader
> >
> >  EXTENSION_SUFFIXES = _imp.extension_suffixes()
> > +
> > +def all_suffixes():
> > +    """Returns a list of all recognized module suffixes for this
> process"""
> > +    return SOURCE_SUFFIXES + BYTECODE_SUFFIXES + EXTENSION_SUFFIXES
> > diff --git a/Lib/inspect.py b/Lib/inspect.py
> > --- a/Lib/inspect.py
> > +++ b/Lib/inspect.py
> > @@ -450,8 +450,15 @@
> >
> >  def getmodulename(path):
> >      """Return the module name for a given file, or None."""
> > -    info = getmoduleinfo(path)
> > -    if info: return info[0]
> > +    fname = os.path.basename(path)
> > +    # Check for paths that look like an actual module file
> > +    suffixes = [(-len(suffix), suffix)
> > +                    for suffix in importlib.machinery.all_suffixes()]
> > +    suffixes.sort() # try longest suffixes first, in case they overlap
> > +    for neglen, suffix in suffixes:
> > +        if fname.endswith(suffix):
> > +            return fname[:neglen]
> > +    return None
> >
> >  def getsourcefile(object):
> >      """Return the filename that can be used to locate an object's
> source.
> > diff --git a/Misc/NEWS b/Misc/NEWS
> > --- a/Misc/NEWS
> > +++ b/Misc/NEWS
> > @@ -41,6 +41,9 @@
> >  Library
> >  -------
> >
> > +- Issue #15397: inspect.getmodulename() is now based directly on
> importlib
> > +  via a new importlib.machinery.all_suffixes() API.
> > +
> >  - Issue #14635: telnetlib will use poll() rather than select() when
> > possible
> >    to avoid failing due to the select() file descriptor limit.
> >
> >
> > --
> > Repository URL: http://hg.python.org/cpython
> >
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120719/4847022e/attachment-0001.html>

From stefan at epy.co.at  Thu Jul 19 14:28:15 2012
From: stefan at epy.co.at (Stefan H. Holek)
Date: Thu, 19 Jul 2012 14:28:15 +0200
Subject: [Python-Dev] Why no venv in existing directory?
Message-ID: <26A4F829-8091-4CA4-A4F5-A18B5813C1CC@epy.co.at>

Hi All,

While trying 3.3 beta I found that I cannot use my favorite virtualenv pattern with pyvenv:

   $ virtualenv .
   Installing.....done.

   $ pyvenv .
   Error: Directory exists: /Users/stefan/sandbox/foo

I appreciate that this behavior is documented and was in the PEP from the start:
"If the target directory already exists an error will be raised, unless the --clear or --upgrade option was provided."

Still, I am curious what the rationale behind this restriction is. For me, being able to "bless" an existing directory with a virtualenv has always been one of its attractions. 

Thanks,
Stefan

-- 
Stefan H. Holek
stefan at epy.co.at


From andrew.svetlov at gmail.com  Thu Jul 19 18:26:42 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Thu, 19 Jul 2012 19:26:42 +0300
Subject: [Python-Dev] venv scripts for fish and csh shells
Message-ID: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>

virtualenv has virtualenv.csh and virtualenv.fish files.
Is there any reason for restricting venv to bash/zsh only?

-- 
Thanks,
Andrew Svetlov

From carl at oddbird.net  Thu Jul 19 18:39:31 2012
From: carl at oddbird.net (Carl Meyer)
Date: Thu, 19 Jul 2012 10:39:31 -0600
Subject: [Python-Dev] Why no venv in existing directory?
In-Reply-To: <26A4F829-8091-4CA4-A4F5-A18B5813C1CC@epy.co.at>
References: <26A4F829-8091-4CA4-A4F5-A18B5813C1CC@epy.co.at>
Message-ID: <50083843.2040105@oddbird.net>

Hi Stefan,

On 07/19/2012 06:28 AM, Stefan H. Holek wrote:
> While trying 3.3 beta I found that I cannot use my favorite
> virtualenv pattern with pyvenv:
> 
> $ virtualenv . Installing.....done.
> 
> $ pyvenv . Error: Directory exists: /Users/stefan/sandbox/foo
> 
> I appreciate that this behavior is documented and was in the PEP from
> the start: "If the target directory already exists an error will be
> raised, unless the --clear or --upgrade option was provided."
> 
> Still, I am curious what the rationale behind this restriction is.

I'd have no problem with lifting the restriction.

I don't recall any clear rationale; I think it was probably just the
simplest implementation initially, and no one ever raised it as an issue
in the PEP process.

Carl



From carl at oddbird.net  Thu Jul 19 18:31:45 2012
From: carl at oddbird.net (Carl Meyer)
Date: Thu, 19 Jul 2012 10:31:45 -0600
Subject: [Python-Dev] venv scripts for fish and csh shells
In-Reply-To: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
References: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
Message-ID: <50083671.6050903@oddbird.net>

On 07/19/2012 10:26 AM, Andrew Svetlov wrote:
> virtualenv has virtualenv.csh and virtualenv.fish files.
> Is there any reason for restricting venv to bash/zsh only?

No. As far as I'm concerned, a patch to port the virtualenv csh and fish
activate scripts to pyvenv would be welcome (though I can't commit said
patch, so it might be good to hear if Vinay has a different opinion).

Carl


From mstefanro at gmail.com  Thu Jul 19 18:53:27 2012
From: mstefanro at gmail.com (M Stefan)
Date: Thu, 19 Jul 2012 19:53:27 +0300
Subject: [Python-Dev] Unbinding of methods
Message-ID: <50083B87.3050502@gmail.com>

Hey,

As part of pickle4, I found it interesting to add the possibility
of pickling bound functions (instance methods). This is done by
pickling f.__self__ and f.__func__ separately, and then adding
a BIND opcode to tie them together.

While this appears to work fine for python methods (non-builtin), some
issues arise with builtins. These are partly caused because
not all builtin function types support __func__, partly because
not all of them fill __module__ when they should and partly
because there are many (7) types a function can actually have:

ClassMethodDescriptorType = type(??)
BuiltinFunctionType = type(len)
FunctionType = type(f)
MethodType = type(A().f())
MethodDescriptorType = type(list.append)
WrapperDescriptorType = type(list.__add__)
MethodWrapperType = type([].__add__)

AllFunctionTypes = (ClassMethodDescriptorType, BuiltinFunctionType,
                     FunctionType, MethodType, MethodDescriptorType,
                     WrapperDescriptorType, MethodWrapperType)
repr(AllFunctionTypes) = (
     <class 'classmethod_descriptor'>,
     <class 'builtin_function_or_method'>, <class 'function'>,
     <class 'method'>, <class 'method_descriptor'>,
     <class 'wrapper_descriptor'>, <class 'method-wrapper'>)

I have created a patch at [1], which adds __func__ to some other
function types, as well as:
  1) adds AllFunctionTypes etc. to Lib/types.py
  2) inspect.isanyfunction(), inspect.isanyboundfunction(),
     inspect.isanyunboundfunction()
  3) functools.unbind
Note that I am not knowledgeable of cpython internals and therefore
the patch needs to be carefully reviewed.

Possible issues: Should classmethods be considered bound or unbound?
If cm is a classmethod, then should
cm.__func__.__self__ = cm.__self__ or cm.__func__.__self__ = None?
Currently does the latter:
     >>> cm.__self__, hasattr(cm,'__self__'), hasattr(cm.__func__, 
'__self__')
     (<class '__main__.A'>, True, False)
This requires treating classmethods separately when pickling,
so I'm not sure if this is ideal.

Let me know if I should have opened an issue instead. I look
forward to hearing your opinions/suggestions on this matter.

Regards,
   Stefan M

[1] https://gist.github.com/3145210

From brett at python.org  Thu Jul 19 19:00:47 2012
From: brett at python.org (Brett Cannon)
Date: Thu, 19 Jul 2012 13:00:47 -0400
Subject: [Python-Dev] Unbinding of methods
In-Reply-To: <50083B87.3050502@gmail.com>
References: <50083B87.3050502@gmail.com>
Message-ID: <CAP1=2W6kQpPdKQ88gv2JyYQf0Qp+2HU_kq=1TuHWxvrMVF3NRQ@mail.gmail.com>

On Thu, Jul 19, 2012 at 12:53 PM, M Stefan <mstefanro at gmail.com> wrote:

> Hey,
>
> As part of pickle4, I found it interesting to add the possibility
> of pickling bound functions (instance methods). This is done by
> pickling f.__self__ and f.__func__ separately, and then adding
> a BIND opcode to tie them together.
>
> While this appears to work fine for python methods (non-builtin), some
> issues arise with builtins. These are partly caused because
> not all builtin function types support __func__, partly because
> not all of them fill __module__ when they should and partly
> because there are many (7) types a function can actually have:
>
> ClassMethodDescriptorType = type(??)
> BuiltinFunctionType = type(len)
> FunctionType = type(f)
> MethodType = type(A().f())
> MethodDescriptorType = type(list.append)
> WrapperDescriptorType = type(list.__add__)
> MethodWrapperType = type([].__add__)
>
> AllFunctionTypes = (ClassMethodDescriptorType, BuiltinFunctionType,
>                     FunctionType, MethodType, MethodDescriptorType,
>                     WrapperDescriptorType, MethodWrapperType)
> repr(AllFunctionTypes) = (
>     <class 'classmethod_descriptor'>,
>     <class 'builtin_function_or_method'>, <class 'function'>,
>     <class 'method'>, <class 'method_descriptor'>,
>     <class 'wrapper_descriptor'>, <class 'method-wrapper'>)
>
> I have created a patch at [1], which adds __func__ to some other
> function types, as well as:
>  1) adds AllFunctionTypes etc. to Lib/types.py
>  2) inspect.isanyfunction(), inspect.isanyboundfunction(),
>     inspect.isanyunboundfunction()
>  3) functools.unbind
> Note that I am not knowledgeable of cpython internals and therefore
> the patch needs to be carefully reviewed.
>
> Possible issues: Should classmethods be considered bound or unbound?
> If cm is a classmethod, then should
> cm.__func__.__self__ = cm.__self__ or cm.__func__.__self__ = None?
> Currently does the latter:
>     >>> cm.__self__, hasattr(cm,'__self__'), hasattr(cm.__func__,
> '__self__')
>     (<class '__main__.A'>, True, False)
> This requires treating classmethods separately when pickling,
> so I'm not sure if this is ideal.
>
> Let me know if I should have opened an issue instead. I look
> forward to hearing your opinions/suggestions on this matter.
>

Yes, open an issue for your patch and reply here with the issue #.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120719/3cfb03a6/attachment.html>

From brett at python.org  Thu Jul 19 19:43:52 2012
From: brett at python.org (Brett Cannon)
Date: Thu, 19 Jul 2012 13:43:52 -0400
Subject: [Python-Dev] Unbinding of methods
In-Reply-To: <CAP1=2W6kQpPdKQ88gv2JyYQf0Qp+2HU_kq=1TuHWxvrMVF3NRQ@mail.gmail.com>
References: <50083B87.3050502@gmail.com>
	<CAP1=2W6kQpPdKQ88gv2JyYQf0Qp+2HU_kq=1TuHWxvrMVF3NRQ@mail.gmail.com>
Message-ID: <CAP1=2W5AZa95mfvH7u9CmNAWMW+1agD0xgyguZrCetxVT4pp1Q@mail.gmail.com>

The issue is http://bugs.python.org/issue15397 [Stefan accidentally replied
privately to me]

On Thu, Jul 19, 2012 at 1:00 PM, Brett Cannon <brett at python.org> wrote:

>
>
> On Thu, Jul 19, 2012 at 12:53 PM, M Stefan <mstefanro at gmail.com> wrote:
>
>> Hey,
>>
>> As part of pickle4, I found it interesting to add the possibility
>> of pickling bound functions (instance methods). This is done by
>> pickling f.__self__ and f.__func__ separately, and then adding
>> a BIND opcode to tie them together.
>>
>> While this appears to work fine for python methods (non-builtin), some
>> issues arise with builtins. These are partly caused because
>> not all builtin function types support __func__, partly because
>> not all of them fill __module__ when they should and partly
>> because there are many (7) types a function can actually have:
>>
>> ClassMethodDescriptorType = type(??)
>> BuiltinFunctionType = type(len)
>> FunctionType = type(f)
>> MethodType = type(A().f())
>> MethodDescriptorType = type(list.append)
>> WrapperDescriptorType = type(list.__add__)
>> MethodWrapperType = type([].__add__)
>>
>> AllFunctionTypes = (ClassMethodDescriptorType, BuiltinFunctionType,
>>                     FunctionType, MethodType, MethodDescriptorType,
>>                     WrapperDescriptorType, MethodWrapperType)
>> repr(AllFunctionTypes) = (
>>     <class 'classmethod_descriptor'>,
>>     <class 'builtin_function_or_method'>, <class 'function'>,
>>     <class 'method'>, <class 'method_descriptor'>,
>>     <class 'wrapper_descriptor'>, <class 'method-wrapper'>)
>>
>> I have created a patch at [1], which adds __func__ to some other
>> function types, as well as:
>>  1) adds AllFunctionTypes etc. to Lib/types.py
>>  2) inspect.isanyfunction(), inspect.isanyboundfunction(),
>>     inspect.isanyunboundfunction()
>>  3) functools.unbind
>> Note that I am not knowledgeable of cpython internals and therefore
>> the patch needs to be carefully reviewed.
>>
>> Possible issues: Should classmethods be considered bound or unbound?
>> If cm is a classmethod, then should
>> cm.__func__.__self__ = cm.__self__ or cm.__func__.__self__ = None?
>> Currently does the latter:
>>     >>> cm.__self__, hasattr(cm,'__self__'), hasattr(cm.__func__,
>> '__self__')
>>     (<class '__main__.A'>, True, False)
>> This requires treating classmethods separately when pickling,
>> so I'm not sure if this is ideal.
>>
>> Let me know if I should have opened an issue instead. I look
>> forward to hearing your opinions/suggestions on this matter.
>>
>
> Yes, open an issue for your patch and reply here with the issue #.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120719/c3ce47f5/attachment.html>

From solipsis at pitrou.net  Thu Jul 19 20:44:09 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 19 Jul 2012 20:44:09 +0200
Subject: [Python-Dev] clang
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
	<20120718075314.Horde.ty5wC7uWis5QBk9Krz1hyUA@webmail.df.eu>
	<ED701F1C-14A2-43F5-A125-32E5AAD467B1@mac.com>
	<20120718165914.Horde.a4ccVklCcOxQBs9CvpTDH4A@webmail.df.eu>
	<33283836-894E-4018-8B3F-9F799DB8B064@mac.com>
Message-ID: <20120719204409.19c9996d@pitrou.net>

On Wed, 18 Jul 2012 17:15:18 +0200
Ronald Oussoren <ronaldoussoren at mac.com> wrote:
> > 
> >> I regularly run the 3.3 testsuite using the latest Xcode from the Appstore on a OSX Lion machine and that works properly.
> > 
> > I'm not actually using the latest Xcode. So if you could test my test
> > program, that would be much appreciated.
> 
> That bug in llvm-gcc still exists, and is unlikely to get fixed.  That's a bug in the integretion of the GCC frontend and LLVM backend, clang (LLVM project frontend + LLVM backend) does work.

Not only clang seems to work, but we have a stable buildbot running on
it:

http://buildbot.python.org/all/buildslaves/langa-lion

Regards

Antoine.



From solipsis at pitrou.net  Thu Jul 19 20:48:21 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 19 Jul 2012 20:48:21 +0200
Subject: [Python-Dev] Use function names instead of functions for
	os.supports_dir_fd?
References: <CAMpsgwaNK+Hot3moRo4BYgh9BQJHfJmRDJOANB2HsnNMwEK3nA@mail.gmail.com>
	<5005F918.1050708@pearwood.info>
	<CAMpsgwb5VaBnN7R=QKJpn+YAyXEThEttqrCW4mgCiz5Rjd6tQA@mail.gmail.com>
Message-ID: <20120719204821.4654b294@pitrou.net>

On Wed, 18 Jul 2012 02:26:14 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:

> >> Monkey patching is a common practice in Python. test_os.py replaces
> >> os.exec*() functions temporary for example.
> >
> > Perhaps for testing, but I don't think monkey-patching is common in
> > production code. Perhaps you are thinking of Ruby :)
> 
> The gevent library does monkey-patch os.fork (and time.sleep and many
> other functions), but gevent is maybe not ready for production? :-)

Extensive monkey-patching of core OS functions would certainly make me
weary of using such a third-party library, even if it claims to be
"serious".

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From solipsis at pitrou.net  Thu Jul 19 20:54:08 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 19 Jul 2012 20:54:08 +0200
Subject: [Python-Dev] Unbinding of methods
References: <50083B87.3050502@gmail.com>
Message-ID: <20120719205408.17bd5ceb@pitrou.net>

On Thu, 19 Jul 2012 19:53:27 +0300
M Stefan <mstefanro at gmail.com> wrote:
> Hey,
> 
> As part of pickle4, I found it interesting to add the possibility
> of pickling bound functions (instance methods). This is done by
> pickling f.__self__ and f.__func__ separately, and then adding
> a BIND opcode to tie them together.

Instead of a specific opcode, can't you use a suitable __reduce__ magic
(or __getnewargs__, perhaps)? We want to limit the number of opcodes
except for performance-critical types (and I don't think bound methods
are performance-critical for the purpose of serialization).

> I have created a patch at [1], which adds __func__ to some other
> function types, as well as:
>   1) adds AllFunctionTypes etc. to Lib/types.py
>   2) inspect.isanyfunction(), inspect.isanyboundfunction(),
>      inspect.isanyunboundfunction()
>   3) functools.unbind

That sounds like a lot of changes if the goal is simply to make those
types picklable.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From shibturn at gmail.com  Thu Jul 19 21:51:39 2012
From: shibturn at gmail.com (Richard Oudkerk)
Date: Thu, 19 Jul 2012 20:51:39 +0100
Subject: [Python-Dev] Unbinding of methods
In-Reply-To: <20120719205408.17bd5ceb@pitrou.net>
References: <50083B87.3050502@gmail.com> <20120719205408.17bd5ceb@pitrou.net>
Message-ID: <ju9ogk$bf9$1@dough.gmane.org>

On 19/07/2012 7:54pm, Antoine Pitrou wrote:
 > Instead of a specific opcode, can't you use a suitable __reduce__
 > magic (or __getnewargs__, perhaps)? We want to limit the number of
 > opcodes except for performance-critical types (and I don't think
 > bound methods are performance-critical for the purpose of
 > serialization).

The one wrinkle is that BuiltinFunctionType is special cased to be 
pickled with save_global, and has no fallback to 
__reduce__/__reduce_ex__/copyreg.  (The C implementation for 
FunctionType *does* have such a fallback, whereas the Python 
implementation doesn't -- see bug http://bugs.python.org/issue14336.)

If the fallback is added for BuiltinFunctionType then "__reduce__ magic" 
should be enough.

The following code works as expected:

import pickle
import copyreg

class A(object):
     def f(self):
         pass
     @classmethod
     def g(cls):
         pass

def f(self):
     pass

ClassMethodDescriptorType = type(A.g)
BuiltinFunctionType = type(len)
FunctionType = type(f)
MethodType = type(A().f)
MethodDescriptorType = type(list.append)
WrapperDescriptorType = type(list.__add__)
MethodWrapperType = type([].__add__)

obj_list = [A.g, len, f, A().f, list.append, list.__add__, [].__add__]

assert ClassMethodDescriptorType is MethodType

def reduce_self(self):
     return getattr, (self.__self__, self.__name__)

def reduce_objclass(self):
     return getattr, (self.__objclass__, self.__name__)

copyreg.pickle(MethodType, reduce_self)
copyreg.pickle(BuiltinFunctionType, reduce_self)
copyreg.pickle(MethodWrapperType, reduce_self)
copyreg.pickle(MethodDescriptorType, reduce_objclass)
copyreg.pickle(WrapperDescriptorType, reduce_objclass)

for obj in obj_list:
     data = pickle.dumps(obj)
     new_obj = pickle.loads(data)
     print('%s\n%s\n' % (obj, new_obj))



From mstefanro at gmail.com  Fri Jul 20 00:32:58 2012
From: mstefanro at gmail.com (M Stefan)
Date: Fri, 20 Jul 2012 01:32:58 +0300
Subject: [Python-Dev] Unbinding of methods
In-Reply-To: <20120719205408.17bd5ceb@pitrou.net>
References: <50083B87.3050502@gmail.com> <20120719205408.17bd5ceb@pitrou.net>
Message-ID: <50088B1A.8040705@gmail.com>

On 7/19/2012 9:54 PM, Antoine Pitrou wrote:
> On Thu, 19 Jul 2012 19:53:27 +0300
> M Stefan <mstefanro at gmail.com> wrote:
>> Hey,
>>
>> As part of pickle4, I found it interesting to add the possibility
>> of pickling bound functions (instance methods). This is done by
>> pickling f.__self__ and f.__func__ separately, and then adding
>> a BIND opcode to tie them together.
> Instead of a specific opcode, can't you use a suitable __reduce__ magic
> (or __getnewargs__, perhaps)? We want to limit the number of opcodes
> except for performance-critical types (and I don't think bound methods
> are performance-critical for the purpose of serialization).
Yes, I agree that doing it with __reduce__ would be better approach than
adding a new opcode, I'll consider switching.
>> I have created a patch at [1], which adds __func__ to some other
>> function types, as well as:
>>    1) adds AllFunctionTypes etc. to Lib/types.py
>>    2) inspect.isanyfunction(), inspect.isanyboundfunction(),
>>       inspect.isanyunboundfunction()
>>    3) functools.unbind
> That sounds like a lot of changes if the goal is simply to make those
> types picklable.
>
> Regards
>
> Antoine.
>
>
Indeed they are, I just thought there may be a chance this code would be 
used elsewhere
too. It's a bit weird that you can use inspect to check for certain 
types of functions but
not others, as well as be able to "unbind" certain types of methods but 
not others.
Admittedly, these changes have little use-case and are not a priority.


Yours,
  Stefan M


From ncoghlan at gmail.com  Fri Jul 20 02:30:16 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 20 Jul 2012 10:30:16 +1000
Subject: [Python-Dev] Runtime forks through monkeypatching (was Re: Use
 function names instead of functions for os.supports_dir_fd?)
Message-ID: <CADiSq7cP9ntMcv0RiWNUSRSAcsqtf1=U2=nY8=UZVKKnw6oROw@mail.gmail.com>

On Fri, Jul 20, 2012 at 4:48 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 18 Jul 2012 02:26:14 +0200
> Victor Stinner <victor.stinner at gmail.com> wrote:
>
>> >> Monkey patching is a common practice in Python. test_os.py replaces
>> >> os.exec*() functions temporary for example.
>> >
>> > Perhaps for testing, but I don't think monkey-patching is common in
>> > production code. Perhaps you are thinking of Ruby :)
>>
>> The gevent library does monkey-patch os.fork (and time.sleep and many
>> other functions), but gevent is maybe not ready for production? :-)
>
> Extensive monkey-patching of core OS functions would certainly make me
> weary of using such a third-party library, even if it claims to be
> "serious".

gevent is one of the well-behaved players in that game - it doesn't
monkeypatch anything unless you explicitly tell it to [1]. You do need
to have some idea what you're doing if using the monkey patching
aspect (as not all third party modules will behave well in that case -
just as not all third party modules behave well on other Python
implementations), but you can also just use gevent as a normal library
without monkey patching anything (however, adhering strictly to that
rule leaves you in the same situation as Twisted, where modules that
use the standard blocking APIs are completely useless to you. Thus, if
your goal is "pure async" code without any need for monkeypatching,
Twisted is a much better bet than gevent, just because it's been
playing that game for longer and has more fully async aware components
available).

That's why I call features like the gevent.monkey module "runtime
forks" - when they're implemented well, they effectively give you two
Python implementations without the complexity of actually having
parallel Python installations (compare the complexity of using
Stackless, a traditional fork, vs gevent.monkey, a runtime fork -
there's a reason the core greenlet component was lifted out of
Stackless and made available as a CPython extension module).
Applications can decide at startup which Python variant (standard
CPython, or CPython+gevent) they want to use without affecting other
Python applications installed on the system and without needing to
duplicating all installed Python modules into a second interpreter.

It's only when libraries monkeypatch standard interfaces *implicitly*
as a side effect of import that you start to get seriously broken
"action at a distance" behaviour. Or, as happened with distutils, you
get a third party project (setuptools) effectively freezing
implementation details of the stdlib version - when monkeypatching
starts delving too deep into implementation details, it's time to
switch to a traditional fork and start figuring out better ways to
solve the problem (which process is now grinding away on the
distutils2/packaging front).

Monkeypatching is a powerful tool - used well, it lets you achieve
impressive things that would otherwise be substantially more difficult
to either create in the first place or maintain in the long run. As
with many powerful tools though, used badly it can lead to a lot of
pain. Static languages like Java decide the risk of action at a
distance from monkeypatching isn't worth the power it gives you to
work around issues in third party libraries without resorting to
creating a traditional fork, so they strictly enforce access controls
and have completely closed class definitions. Python permits
monkeypatching at a language level (because when you need it, you
really need it), but discourages it a social level (because there is
usually a less magical technique, such as creating and using a wrapper
or subclass, that will achieve the desired effect).

Cheers,
Nick.

[1] http://www.gevent.org/intro.html#monkey-patching

P.S. for a non-monkeypatching approach to reusing a synchronous
library when creating an asynchronous equivalent, take a look at
10gen's attempt to create a *maintainable* Tornado-compatible async
wrapper for PyMongo by using greenlets ([2][3]). It's a clever idea,
but still a lot more work than just using PyMongo itself with gevent
if you're only interested in the "async" part rather than the
"Tornado-compatible" part.

[2] http://emptysquare.net/blog/motor-internals-how-i-asynchronized-a-synchronous-library/
[3] http://emptysquare.net/blog/motor-four-strategies-for-maintainability/


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From fuzzyman at voidspace.org.uk  Fri Jul 20 14:55:53 2012
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Fri, 20 Jul 2012 13:55:53 +0100
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
Message-ID: <BE3A8A3B-F69C-4F84-9F99-3B1CCCCA3A21@voidspace.org.uk>


On 17 Jul 2012, at 23:04, martin at v.loewis.de wrote:

>> [snip...]
> 
>> I would like to use a JIT to generate specialized functions for a
>> combinaison of arguments types.
> 
> I think history has moved past specializing JITs. Tracing JITs are the
> status quo; they provide specialization as a side effect.
> 

Mozilla implemented a method-JIT (compile whole methods) on top of their tracing JIT because a tracing JIT only optimises part of your code (only in loops and only if executed more times than the threshold) and there are further performance improvements to be had. So tracing JITs are not the *whole* of the state of the art.

Michael


> Regards,
> Martin
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
> 


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From alex.gaynor at gmail.com  Fri Jul 20 17:01:56 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Fri, 20 Jul 2012 15:01:56 +0000 (UTC)
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<BE3A8A3B-F69C-4F84-9F99-3B1CCCCA3A21@voidspace.org.uk>
Message-ID: <loom.20120720T165907-171@post.gmane.org>

That's not, strictly speaking, true. Mozilla added a method-JIT (Jaegermonkey)
and then added another one (IonMonkey) because their tracing JIT (Tracemonkey)
was bad.  There's no fundamental reason that tracing has to only cover loops,
indeed PyPy's tracing has been generalized to compile individual functions,
recursion, etc. And any profiling JIT, in practice, needs a compile heuristic
for how many calls must occur before a unit is compiled, even the Hotspot JVM
has  one.

Alex


From jyasskin at gmail.com  Fri Jul 20 17:59:34 2012
From: jyasskin at gmail.com (Jeffrey Yasskin)
Date: Fri, 20 Jul 2012 08:59:34 -0700
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<CAMpsgwan=zKUt496Nfdvda78OtN75YTGr3Qc9q1k_FO3RDexKw@mail.gmail.com>
Message-ID: <CA+6j2ggh5u8y-eMFrD0-mfFadFcs2GcR57HrBS7whhX=Z0PuUg@mail.gmail.com>

On Tue, Jul 17, 2012 at 3:20 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
>> It's the JIT compiler of Unladen Swallow that "failed"; in
>> my understanding because LLVM is crap (i.e. it is slow, memory-consuming,
>> and buggy) - as a low-level virtual machine; it may be ok as a compiler
>> backend (but I still think it is buggy there as well).
>
> What is the status of LLVM nowadays? Is it not a good solution to
> write a portable JIT?

Its code generator is still fairly slow. You could probably get a
faster one committed, but you'd have to write it. LLVM also still
doesn't have great profile-guided optimizations (what you need in a
JIT), although the infrastructure is starting to be built. You'd
probably have to contribute to that too. It's probably a better use of
your time to contribute to PyPy.

Jeffrey

From status at bugs.python.org  Fri Jul 20 18:07:38 2012
From: status at bugs.python.org (Python tracker)
Date: Fri, 20 Jul 2012 18:07:38 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20120720160738.1ABCB1CCCC@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2012-07-13 - 2012-07-20)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    3528 ( +8)
  closed 23658 (+55)
  total  27186 (+63)

Open issues with patches: 1501 


Issues opened (39)
==================

#9374: urlparse should parse query and fragment for arbitrary schemes
http://bugs.python.org/issue9374  reopened by eric.araujo

#9803: IDLE closes with save while breakpoint open
http://bugs.python.org/issue9803  reopened by serwy

#15346: Tkinter dnd has no documentation
http://bugs.python.org/issue15346  opened by weirdink13

#15347: IDLE - does not close if the debugger was active
http://bugs.python.org/issue15347  opened by serwy

#15348: IDLE - shell becomes unresponsive if debugger windows is close
http://bugs.python.org/issue15348  opened by serwy

#15350: {urllib,urllib.parse}.urlencode.__doc__ is unclear
http://bugs.python.org/issue15350  opened by samwyse

#15351: Add to unittest.TestCase support for using context managers
http://bugs.python.org/issue15351  opened by cjerdonek

#15352: importlib.h should be regenerated when the marshaling code cha
http://bugs.python.org/issue15352  opened by meador.inge

#15353: ld: library not found for -lpython3.3m because of wrong LDFLAG
http://bugs.python.org/issue15353  opened by Jerry.Jacobs

#15355: generator.__next__() docs should mention exception if already 
http://bugs.python.org/issue15355  opened by cjerdonek

#15358: Test pkgutil.walk_packages in test_pkgutil instead of test_run
http://bugs.python.org/issue15358  opened by ncoghlan

#15359: Sockets support for CAN_BCM
http://bugs.python.org/issue15359  opened by Thorney

#15360: Behavior of assigning to __dict__ is not documented
http://bugs.python.org/issue15360  opened by davide.rizzo

#15362: pyport.h includes antiquated UTF handling for FreeBSD
http://bugs.python.org/issue15362  opened by JohnSchneider

#15363: Idle/tkinter ~x.py 'save as' fails. closes idle
http://bugs.python.org/issue15363  opened by terry.reedy

#15364: sysconfig confused by relative paths
http://bugs.python.org/issue15364  opened by sbt

#15365: Traceback reporting can fail if IO cannot be imported
http://bugs.python.org/issue15365  opened by kristjan.jonsson

#15369: pybench and test.pystone poorly documented
http://bugs.python.org/issue15369  opened by flox

#15370: test_runpy should include namespace package tests
http://bugs.python.org/issue15370  opened by ncoghlan

#15371: test_cmd_line_script should include namespace package tests
http://bugs.python.org/issue15371  opened by ncoghlan

#15373: copy.copy() does not properly copy os.environment
http://bugs.python.org/issue15373  opened by htgoebel

#15376: Refactor the test_runpy walk_package support code into a commo
http://bugs.python.org/issue15376  opened by cjerdonek

#15378: Underported Tools/unicode/comparecodecs.py
http://bugs.python.org/issue15378  opened by storchaka

#15379: Charmap decoding of no-BMP characters
http://bugs.python.org/issue15379  opened by storchaka

#15381: Optimize BytesIO to so less reallocations when written, simila
http://bugs.python.org/issue15381  opened by eli.bendersky

#15382: os.utime() mishandles some arguments
http://bugs.python.org/issue15382  opened by Arfrever

#15388: SAX parse (ExpatParser) leaks file handle when given filename 
http://bugs.python.org/issue15388  opened by mth

#15389: PEP 3121, 384 refactoring applied to curses module
http://bugs.python.org/issue15389  opened by Robin.Schreiber

#15390: PEP 3121, 384 refactoring applied to datetime module
http://bugs.python.org/issue15390  opened by Robin.Schreiber

#15392: Create a unittest framework for IDLE
http://bugs.python.org/issue15392  opened by terry.reedy

#15393: JSONDecoder.raw_decode breaks on leading whitespace
http://bugs.python.org/issue15393  opened by aalien

#15397: Unbinding of methods
http://bugs.python.org/issue15397  opened by mstefanro

#15398: intermittence on UnicodeFileTests.test_rename at test_pep277 o
http://bugs.python.org/issue15398  opened by flavio.ribeiro

#15402: Correct __sizeof__ support for struct
http://bugs.python.org/issue15402  opened by storchaka

#15403: Refactor package creation support code into a common location
http://bugs.python.org/issue15403  opened by cjerdonek

#15406: Deprecation Warnings fixes on test suite
http://bugs.python.org/issue15406  opened by flavio.ribeiro

#15407: CSV parser fails to iterate properly on 2.6.6
http://bugs.python.org/issue15407  opened by sleepycal

#15408: os.fork/os.popen behaviour change between 2.7 and 3.2
http://bugs.python.org/issue15408  opened by doerwalter

#1492704: distinct error type if shutil.copyfile() fails because of src 
http://bugs.python.org/issue1492704  reopened by hynek



Most recent 15 issues with no replies (15)
==========================================

#15408: os.fork/os.popen behaviour change between 2.7 and 3.2
http://bugs.python.org/issue15408

#15390: PEP 3121, 384 refactoring applied to datetime module
http://bugs.python.org/issue15390

#15388: SAX parse (ExpatParser) leaks file handle when given filename 
http://bugs.python.org/issue15388

#15371: test_cmd_line_script should include namespace package tests
http://bugs.python.org/issue15371

#15370: test_runpy should include namespace package tests
http://bugs.python.org/issue15370

#15362: pyport.h includes antiquated UTF handling for FreeBSD
http://bugs.python.org/issue15362

#15355: generator.__next__() docs should mention exception if already 
http://bugs.python.org/issue15355

#15353: ld: library not found for -lpython3.3m because of wrong LDFLAG
http://bugs.python.org/issue15353

#15350: {urllib,urllib.parse}.urlencode.__doc__ is unclear
http://bugs.python.org/issue15350

#15348: IDLE - shell becomes unresponsive if debugger windows is close
http://bugs.python.org/issue15348

#15347: IDLE - does not close if the debugger was active
http://bugs.python.org/issue15347

#15346: Tkinter dnd has no documentation
http://bugs.python.org/issue15346

#15340: OSError with "import random" when /dev/urandom doesn't exist (
http://bugs.python.org/issue15340

#15336: Argparse required arguments incorrectly displayed as optional 
http://bugs.python.org/issue15336

#15326: --random does not work for regrtest
http://bugs.python.org/issue15326



Most recent 15 issues waiting for review (15)
=============================================

#15406: Deprecation Warnings fixes on test suite
http://bugs.python.org/issue15406

#15403: Refactor package creation support code into a common location
http://bugs.python.org/issue15403

#15402: Correct __sizeof__ support for struct
http://bugs.python.org/issue15402

#15397: Unbinding of methods
http://bugs.python.org/issue15397

#15393: JSONDecoder.raw_decode breaks on leading whitespace
http://bugs.python.org/issue15393

#15390: PEP 3121, 384 refactoring applied to datetime module
http://bugs.python.org/issue15390

#15389: PEP 3121, 384 refactoring applied to curses module
http://bugs.python.org/issue15389

#15382: os.utime() mishandles some arguments
http://bugs.python.org/issue15382

#15381: Optimize BytesIO to so less reallocations when written, simila
http://bugs.python.org/issue15381

#15379: Charmap decoding of no-BMP characters
http://bugs.python.org/issue15379

#15378: Underported Tools/unicode/comparecodecs.py
http://bugs.python.org/issue15378

#15373: copy.copy() does not properly copy os.environment
http://bugs.python.org/issue15373

#15365: Traceback reporting can fail if IO cannot be imported
http://bugs.python.org/issue15365

#15364: sysconfig confused by relative paths
http://bugs.python.org/issue15364

#15363: Idle/tkinter ~x.py 'save as' fails. closes idle
http://bugs.python.org/issue15363



Top 10 most discussed issues (10)
=================================

#1492704: distinct error type if shutil.copyfile() fails because of src 
http://bugs.python.org/issue1492704  21 msgs

#15364: sysconfig confused by relative paths
http://bugs.python.org/issue15364  17 msgs

#15398: intermittence on UnicodeFileTests.test_rename at test_pep277 o
http://bugs.python.org/issue15398  14 msgs

#15381: Optimize BytesIO to so less reallocations when written, simila
http://bugs.python.org/issue15381  11 msgs

#15352: importlib.h should be regenerated when the marshaling code cha
http://bugs.python.org/issue15352  10 msgs

#15301: os.chown: OverflowError: Python int too large to convert to C 
http://bugs.python.org/issue15301   9 msgs

#14988: _elementtree: Raise ImportError when importing of pyexpat fail
http://bugs.python.org/issue14988   8 msgs

#15379: Charmap decoding of no-BMP characters
http://bugs.python.org/issue15379   7 msgs

#15393: JSONDecoder.raw_decode breaks on leading whitespace
http://bugs.python.org/issue15393   7 msgs

#4130: Intel icc 9.1 does not support __int128_t used by ctypes
http://bugs.python.org/issue4130   6 msgs



Issues closed (53)
==================

#6068: ctypes is not correctly handling bitfields backed by 64 bit in
http://bugs.python.org/issue6068  closed by meador.inge

#6257: Idle terminates on source save while debugging
http://bugs.python.org/issue6257  closed by serwy

#6493: Can not set value for structure members larger than 32 bits
http://bugs.python.org/issue6493  closed by meador.inge

#9254: __import__ docstring should recommend importlib.import_module(
http://bugs.python.org/issue9254  closed by python-dev

#9458: xml.etree.ElementTree.ElementTree.write(): encoding handling p
http://bugs.python.org/issue9458  closed by eli.bendersky

#9591: kqueue not reporting EOF under certain circumstances
http://bugs.python.org/issue9591  closed by Volodymyr.Kostyrko

#13823: xml.etree.ElementTree.ElementTree.write - argument checking
http://bugs.python.org/issue13823  closed by eli.bendersky

#14600: Change ImportError reference handling, naming
http://bugs.python.org/issue14600  closed by brian.curtin

#14635: telnetlib uses select instead of poll - limited to FD_SETSIZE 
http://bugs.python.org/issue14635  closed by gregory.p.smith

#14790: use packaging in setup.py
http://bugs.python.org/issue14790  closed by loewis

#15169: Clear C code under PyImport_ExecCodeModuleObject()
http://bugs.python.org/issue15169  closed by brett.cannon

#15170: Fix 64-bit building for buildbot scripts (2.7)
http://bugs.python.org/issue15170  closed by skrah

#15171: Fix 64-bit building for buildbot scripts (3.2)
http://bugs.python.org/issue15171  closed by skrah

#15180: Cryptic traceback from os.path.join when mixing str & bytes
http://bugs.python.org/issue15180  closed by hynek

#15230: runpy.run_path doesn't set __package__ correctly
http://bugs.python.org/issue15230  closed by python-dev

#15238: shutil.copystat should copy Linux extended attributes
http://bugs.python.org/issue15238  closed by larry

#15297: pkgutil.iter_importers() includes an ImpImporter
http://bugs.python.org/issue15297  closed by cjerdonek

#15299: pkgutil.ImpImporter(None).iter_modules() does not search sys.p
http://bugs.python.org/issue15299  closed by ncoghlan

#15307: Patch for --symlink support in pyvenv with framework python
http://bugs.python.org/issue15307  closed by python-dev

#15314: Use importlib instead of pkgutil in runpy
http://bugs.python.org/issue15314  closed by ncoghlan

#15334: access denied for HKEY_PERFORMANCE_DATA
http://bugs.python.org/issue15334  closed by pitrou

#15337: The cmd module incorrectly lists "help" as an undocumented com
http://bugs.python.org/issue15337  closed by rhettinger

#15338: test_UNC_path failure in test_import
http://bugs.python.org/issue15338  closed by pitrou

#15343: "pydoc -w <package>" writes out page with empty "Package Conte
http://bugs.python.org/issue15343  closed by ncoghlan

#15345: HOWTOs Argparse tutorial - code example raises SyntaxError
http://bugs.python.org/issue15345  closed by ezio.melotti

#15349: SyntaxError b0rked grammar
http://bugs.python.org/issue15349  closed by serwy

#15354: _PyObject_LengthHint only accepts longs
http://bugs.python.org/issue15354  closed by python-dev

#15356: '\xa0' isspace returns true while compiling python on solaris 
http://bugs.python.org/issue15356  closed by r.david.murray

#15357: Deprecate redundant pieces of pkgutil
http://bugs.python.org/issue15357  closed by ncoghlan

#15361: venv's Activate.ps1 causes broken prompt with powershell
http://bugs.python.org/issue15361  closed by python-dev

#15366: venv assumes header files in sys._home + '/Include'
http://bugs.python.org/issue15366  closed by python-dev

#15367: build_ext in a venv on Windows assumes pyconfig.h in sys.exec_
http://bugs.python.org/issue15367  closed by python-dev

#15368: bytecode generation is not deterministic
http://bugs.python.org/issue15368  closed by meador.inge

#15372: Python is missing alternative for common quoting character
http://bugs.python.org/issue15372  closed by r.david.murray

#15374: venv environment variable should follow the conventions
http://bugs.python.org/issue15374  closed by pitrou

#15375: Trivial for fix in the subprocess documentation
http://bugs.python.org/issue15375  closed by r.david.murray

#15377: os.path.join() error misleading with path1=None
http://bugs.python.org/issue15377  closed by hynek

#15380: bytes/str mismatch in distribute
http://bugs.python.org/issue15380  closed by ned.deily

#15383: Autocompletion crashes Python if the __builtins__ module canno
http://bugs.python.org/issue15383  closed by r.david.murray

#15384: pkgutil importlib migration test was failing on Windows buildb
http://bugs.python.org/issue15384  closed by ncoghlan

#15385: Behaviour change in runpy for __file__ attributes between 3.2 
http://bugs.python.org/issue15385  closed by ncoghlan

#15386: Still getting two copies of importlib._bootstrap
http://bugs.python.org/issue15386  closed by python-dev

#15387: pkgutil.walk_packages is using a deprecated API
http://bugs.python.org/issue15387  closed by python-dev

#15391: Add bitlength function to the math module
http://bugs.python.org/issue15391  closed by jcea

#15394: memory leak in PyModule_Create2
http://bugs.python.org/issue15394  closed by meador.inge

#15395: memory leaks in selectmodule.c
http://bugs.python.org/issue15395  closed by python-dev

#15396: memory leak in tkinter
http://bugs.python.org/issue15396  closed by python-dev

#15399: processName key is un-/mis-documented in 2.6 and up
http://bugs.python.org/issue15399  closed by python-dev

#15400: int('12345L', 10) raises ValueError
http://bugs.python.org/issue15400  closed by jcea

#15401: Typo in inspect.getclosurevars docstring
http://bugs.python.org/issue15401  closed by meador.inge

#15404: Refleak in PyMethodObject repr
http://bugs.python.org/issue15404  closed by asvetlov

#15405: Invitation to connect on LinkedIn
http://bugs.python.org/issue15405  closed by nadeem.vawda

#1767933: Badly formed XML using etree and utf-16
http://bugs.python.org/issue1767933  closed by python-dev

From fijall at gmail.com  Fri Jul 20 18:50:44 2012
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Fri, 20 Jul 2012 18:50:44 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <BE3A8A3B-F69C-4F84-9F99-3B1CCCCA3A21@voidspace.org.uk>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<BE3A8A3B-F69C-4F84-9F99-3B1CCCCA3A21@voidspace.org.uk>
Message-ID: <CAK5idxR2-av6n7eXc9qovSU5e7qw549Rd5Psd90os-b03YWO_g@mail.gmail.com>

On Fri, Jul 20, 2012 at 2:55 PM, Michael Foord <fuzzyman at voidspace.org.uk>wrote:

>
> On 17 Jul 2012, at 23:04, martin at v.loewis.de wrote:
>
> >> [snip...]
> >
> >> I would like to use a JIT to generate specialized functions for a
> >> combinaison of arguments types.
> >
> > I think history has moved past specializing JITs. Tracing JITs are the
> > status quo; they provide specialization as a side effect.
> >
>
> Mozilla implemented a method-JIT (compile whole methods) on top of their
> tracing JIT because a tracing JIT only optimises part of your code (only in
> loops and only if executed more times than the threshold) and there are
> further performance improvements to be had. So tracing JITs are not the
> *whole* of the state of the art.
>
> Michael
>

I'm sorry michael but you're like a 100th person I have to explain this to.
The pure reason that mozilla did not make a tracing JIT work does not mean
the entire approach is horribly doomed as many people would like to assume.
The reasons are multiple, but a lot of them are connected to poor
engineering (for example the part inherited from adobe is notoriously bad,
have a look if you want).

Cheers,
fijal
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120720/9f0ba343/attachment.html>

From solipsis at pitrou.net  Fri Jul 20 22:12:26 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Fri, 20 Jul 2012 22:12:26 +0200
Subject: [Python-Dev] cpython: Issue #15168: Move importlb.test to
 test.test_importlib.
References: <3Wf1Mz3DGzzPPy@mail.python.org>
Message-ID: <20120720221226.3323fc33@pitrou.net>

On Fri, 20 Jul 2012 20:49:03 +0200 (CEST)
brett.cannon <python-checkins at python.org> wrote:
> diff --git a/Lib/importlib/test/__init__.py b/Lib/test/test_importlib/__init__.py
> rename from Lib/importlib/test/__init__.py
> rename to Lib/test/test_importlib/__init__.py
> --- a/Lib/importlib/test/__init__.py
> +++ b/Lib/test/test_importlib/__init__.py
> @@ -1,5 +1,6 @@
>  import os
>  import sys
> +from .. import support

I don't know if I'm only speaking for myself, but I really have trouble
parsing non-trivial relative imports, and I personally prefer when
people use absolute imports (e.g. "from test import support").

cheers

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From rdmurray at bitdance.com  Fri Jul 20 22:25:04 2012
From: rdmurray at bitdance.com (R. David Murray)
Date: Fri, 20 Jul 2012 16:25:04 -0400
Subject: [Python-Dev] cpython: Issue #15168: Move importlb.test to
	test.test_importlib.
In-Reply-To: <20120720221226.3323fc33@pitrou.net>
References: <3Wf1Mz3DGzzPPy@mail.python.org>
	<20120720221226.3323fc33@pitrou.net>
Message-ID: <20120720202505.2572C2502C2@webabinitio.net>

On Fri, 20 Jul 2012 22:12:26 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Fri, 20 Jul 2012 20:49:03 +0200 (CEST)
> brett.cannon <python-checkins at python.org> wrote:
> > diff --git a/Lib/importlib/test/__init__.py b/Lib/test/test_importlib/__init__.py
> > rename from Lib/importlib/test/__init__.py
> > rename to Lib/test/test_importlib/__init__.py
> > --- a/Lib/importlib/test/__init__.py
> > +++ b/Lib/test/test_importlib/__init__.py
> > @@ -1,5 +1,6 @@
> >  import os
> >  import sys
> > +from .. import support
> 
> I don't know if I'm only speaking for myself, but I really have trouble
> parsing non-trivial relative imports, and I personally prefer when
> people use absolute imports (e.g. "from test import support").

Agreed.  I don't see that there is any reason to use relative
imports in the stdlib.

--David

From andrew.svetlov at gmail.com  Fri Jul 20 22:25:48 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Fri, 20 Jul 2012 23:25:48 +0300
Subject: [Python-Dev] cpython: Issue #15168: Move importlb.test to
	test.test_importlib.
In-Reply-To: <20120720202505.2572C2502C2@webabinitio.net>
References: <3Wf1Mz3DGzzPPy@mail.python.org>
	<20120720221226.3323fc33@pitrou.net>
	<20120720202505.2572C2502C2@webabinitio.net>
Message-ID: <CAL3CFcWp=r5b5BkDymTB+SA6=UVfdPJgRfU7GV3NGRB9tRd76w@mail.gmail.com>

+1

On Fri, Jul 20, 2012 at 11:25 PM, R. David Murray <rdmurray at bitdance.com> wrote:
> On Fri, 20 Jul 2012 22:12:26 +0200, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Fri, 20 Jul 2012 20:49:03 +0200 (CEST)
>> brett.cannon <python-checkins at python.org> wrote:
>> > diff --git a/Lib/importlib/test/__init__.py b/Lib/test/test_importlib/__init__.py
>> > rename from Lib/importlib/test/__init__.py
>> > rename to Lib/test/test_importlib/__init__.py
>> > --- a/Lib/importlib/test/__init__.py
>> > +++ b/Lib/test/test_importlib/__init__.py
>> > @@ -1,5 +1,6 @@
>> >  import os
>> >  import sys
>> > +from .. import support
>>
>> I don't know if I'm only speaking for myself, but I really have trouble
>> parsing non-trivial relative imports, and I personally prefer when
>> people use absolute imports (e.g. "from test import support").
>
> Agreed.  I don't see that there is any reason to use relative
> imports in the stdlib.
>
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov

From raymond.hettinger at gmail.com  Sat Jul 21 01:59:35 2012
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Fri, 20 Jul 2012 16:59:35 -0700
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CADiSq7ffwJWKJcxGC6UM0dYmXLfvv=BvUqLx1WX0Bhybd4pr4g@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<500685B5.2020507@hotpy.org>
	<CADiSq7ffwJWKJcxGC6UM0dYmXLfvv=BvUqLx1WX0Bhybd4pr4g@mail.gmail.com>
Message-ID: <20C8EA74-683B-4E3B-AA53-1795CE0437AA@gmail.com>


On Jul 18, 2012, at 3:30 AM, Nick Coghlan wrote:

> - Eugene Toder's patch to add an AST optimisation step to the compiler
> chain (http://bugs.python.org/issue11549) (I've asked Eugene about
> this patch more recently and his current thought is that subsequent
> improvements to the peephole optimisation have rendered it less
> valuable. However, the patch is still a potentially useful resource
> for anyone investigating bytecode optimisation ideas)


+1 for furthering Eugene's patch.  The AST is the correct place
to do some of the optimizations currently done by the peepholer.


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120720/868501e8/attachment-0001.html>

From fuzzyman at voidspace.org.uk  Sat Jul 21 12:45:21 2012
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Sat, 21 Jul 2012 11:45:21 +0100
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
In-Reply-To: <CAK5idxR2-av6n7eXc9qovSU5e7qw549Rd5Psd90os-b03YWO_g@mail.gmail.com>
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<BE3A8A3B-F69C-4F84-9F99-3B1CCCCA3A21@voidspace.org.uk>
	<CAK5idxR2-av6n7eXc9qovSU5e7qw549Rd5Psd90os-b03YWO_g@mail.gmail.com>
Message-ID: <166C01FF-BE24-4904-9E87-A3B5696399D9@voidspace.org.uk>


On 20 Jul 2012, at 17:50, Maciej Fijalkowski wrote:

> On Fri, Jul 20, 2012 at 2:55 PM, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
> 
> On 17 Jul 2012, at 23:04, martin at v.loewis.de wrote:
> 
> >> [snip...]
> >
> >> I would like to use a JIT to generate specialized functions for a
> >> combinaison of arguments types.
> >
> > I think history has moved past specializing JITs. Tracing JITs are the
> > status quo; they provide specialization as a side effect.
> >
> 
> Mozilla implemented a method-JIT (compile whole methods) on top of their tracing JIT because a tracing JIT only optimises part of your code (only in loops and only if executed more times than the threshold) and there are further performance improvements to be had. So tracing JITs are not the *whole* of the state of the art.
> 
> Michael
> 
> I'm sorry michael but you're like a 100th person I have to explain this to. The pure reason that mozilla did not make a tracing JIT work does not mean the entire approach is horribly doomed as many people would like to assume. The reasons are multiple, but a lot of them are connected to poor engineering (for example the part inherited from adobe is notoriously bad, have a look if you want).


Well, that isn't how they describe it. If it is the case, it's *still* interesting that rather than putting their efforts into improving the tracing JIT they put them into adding a method-JIT *as well*. Also note that where I said "tracing JITs  are not the whole of the state of the art" you somehow managed to translate this into "the entire approach is horribly doomed". That seems an ungenerous reading of what I wrote... :-)

Michael



> 
> Cheers,
> fijal
> 
> 


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From solipsis at pitrou.net  Sat Jul 21 12:56:29 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 21 Jul 2012 12:56:29 +0200
Subject: [Python-Dev] A new JIT compiler for a faster CPython?
References: <CAMpsgwYczn48GTRK8zYV03Q=Kv=WqGgOy8YEg8LqNzARdY6veA@mail.gmail.com>
	<20120718000453.Horde.VBVcWaGZi1VQBeGFK6UzHFA@webmail.df.eu>
	<BE3A8A3B-F69C-4F84-9F99-3B1CCCCA3A21@voidspace.org.uk>
	<CAK5idxR2-av6n7eXc9qovSU5e7qw549Rd5Psd90os-b03YWO_g@mail.gmail.com>
	<166C01FF-BE24-4904-9E87-A3B5696399D9@voidspace.org.uk>
Message-ID: <20120721125629.1b4d47f0@pitrou.net>

On Sat, 21 Jul 2012 11:45:21 +0100
Michael Foord <fuzzyman at voidspace.org.uk> wrote:
> 
> On 20 Jul 2012, at 17:50, Maciej Fijalkowski wrote:
> 
> > On Fri, Jul 20, 2012 at 2:55 PM, Michael Foord <fuzzyman at voidspace.org.uk> wrote:
> > 
> > On 17 Jul 2012, at 23:04, martin at v.loewis.de wrote:
> > 
> > >> [snip...]
> > >
> > >> I would like to use a JIT to generate specialized functions for a
> > >> combinaison of arguments types.
> > >
> > > I think history has moved past specializing JITs. Tracing JITs are the
> > > status quo; they provide specialization as a side effect.
> > >
> > 
> > Mozilla implemented a method-JIT (compile whole methods) on top of their tracing JIT because a tracing JIT only optimises part of your code (only in loops and only if executed more times than the threshold) and there are further performance improvements to be had. So tracing JITs are not the *whole* of the state of the art.
> > 
> > Michael
> > 
> > I'm sorry michael but you're like a 100th person I have to explain this to. The pure reason that mozilla did not make a tracing JIT work does not mean the entire approach is horribly doomed as many people would like to assume. The reasons are multiple, but a lot of them are connected to poor engineering (for example the part inherited from adobe is notoriously bad, have a look if you want).
> 
> 
> Well, that isn't how they describe it. If it is the case, it's *still* interesting 
> that rather than putting their efforts into improving the tracing JIT they put them 
> into adding a method-JIT *as well*.

Honestly I'm not sure that's a very interesting discussion. First,
Javascript performance is not based on the same prorities as Python
performance: for the former, startup time is key. Second, whether
method-based or tracing-based, a well-written JIT would certainly bring
significant performance improvements over a bytecode interpreter anyway.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From brett at python.org  Sat Jul 21 15:56:48 2012
From: brett at python.org (Brett Cannon)
Date: Sat, 21 Jul 2012 09:56:48 -0400
Subject: [Python-Dev] cpython: Issue #15168: Move importlb.test to
	test.test_importlib.
In-Reply-To: <20120720221226.3323fc33@pitrou.net>
References: <3Wf1Mz3DGzzPPy@mail.python.org>
	<20120720221226.3323fc33@pitrou.net>
Message-ID: <CAP1=2W5CCoAC4O3yt_w2GohkSh2AvLG7BxixhwK+eZ3=LMWSAg@mail.gmail.com>

On Fri, Jul 20, 2012 at 4:12 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Fri, 20 Jul 2012 20:49:03 +0200 (CEST)
> brett.cannon <python-checkins at python.org> wrote:
> > diff --git a/Lib/importlib/test/__init__.py
> b/Lib/test/test_importlib/__init__.py
> > rename from Lib/importlib/test/__init__.py
> > rename to Lib/test/test_importlib/__init__.py
> > --- a/Lib/importlib/test/__init__.py
> > +++ b/Lib/test/test_importlib/__init__.py
> > @@ -1,5 +1,6 @@
> >  import os
> >  import sys
> > +from .. import support
>
> I don't know if I'm only speaking for myself, but I really have trouble
> parsing non-trivial relative imports,


I obviously don't, but then again I would hope I of all people could
comprehend them after all this time. =)


> and I personally prefer when
> people use absolute imports (e.g. "from test import support").
>
>
So I changed the import above, but I'm not going to go through and change
every other relative import that stays within test.test_importlib. I'm fine
if someone else wants to change them, but I leave that as an exercise for
the reader.

-Brett


> cheers
>
> Antoine.
>
>
> --
> Software development and contracting: http://pro.pitrou.net
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120721/4af3f7c4/attachment.html>

From g.brandl at gmx.net  Sat Jul 21 16:41:51 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 21 Jul 2012 16:41:51 +0200
Subject: [Python-Dev] cpython (3.2): ISsue #14988: restore Python 2's
 behavior of raising ImportError when unable to
In-Reply-To: <3Wbzgn72ckzPJr@mail.python.org>
References: <3Wbzgn72ckzPJr@mail.python.org>
Message-ID: <juef3e$j4$1@dough.gmane.org>

This looks like it will give an "error return without exception set"
if the "if (expat_capi)" directly before the change is true, and
expat_capi is then set to NULL in the branch.

Georg

On 07/17/2012 01:25 PM, eli.bendersky wrote:
> http://hg.python.org/cpython/rev/d896fd0a8ba7
> changeset:   78152:d896fd0a8ba7
> branch:      3.2
> parent:      78142:71537aba3a0a
> user:        Eli Bendersky <eliben at gmail.com>
> date:        Tue Jul 17 14:20:38 2012 +0300
> summary:
>   ISsue #14988: restore Python 2's behavior of raising ImportError when unable to load pyexpat, instead of a SystemError/RuntimeError
> 
> files:
>   Modules/_elementtree.c |  2 ++
>   1 files changed, 2 insertions(+), 0 deletions(-)
> 
> 
> diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c
> --- a/Modules/_elementtree.c
> +++ b/Modules/_elementtree.c
> @@ -3117,6 +3117,8 @@
>              expat_capi->MICRO_VERSION != XML_MICRO_VERSION)
>              expat_capi = NULL;
>      }
> +    if (!expat_capi)
> +        return NULL;
>  #endif
>  
>      elementtree_parseerror_obj = PyErr_NewException(
> 
> 
> 
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
> 



From g.brandl at gmx.net  Sat Jul 21 16:43:02 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Sat, 21 Jul 2012 16:43:02 +0200
Subject: [Python-Dev] cpython: Optimize tostringlist by taking the
 stream class outside the function. It's now
In-Reply-To: <3Wc0gJ62Z5zPBM@mail.python.org>
References: <3Wc0gJ62Z5zPBM@mail.python.org>
Message-ID: <juef5l$j4$2@dough.gmane.org>

Uh, optimizations are not exactly what I want to see during feature freeze.

Georg

On 07/17/2012 02:10 PM, eli.bendersky wrote:
> http://hg.python.org/cpython/rev/51978f89e5ed
> changeset:   78156:51978f89e5ed
> user:        Eli Bendersky <eliben at gmail.com>
> date:        Tue Jul 17 15:09:12 2012 +0300
> summary:
>   Optimize tostringlist by taking the stream class outside the function. It's now 2x faster on short calls. Related to #1767933
> 
> files:
>   Lib/xml/etree/ElementTree.py |  38 +++++++++++++----------
>   1 files changed, 22 insertions(+), 16 deletions(-)
> 
> 
> diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py
> --- a/Lib/xml/etree/ElementTree.py
> +++ b/Lib/xml/etree/ElementTree.py
> @@ -1184,23 +1184,29 @@
>  # @defreturn sequence
>  # @since 1.3
>  
> +class _ListDataStream(io.BufferedIOBase):
> +    """ An auxiliary stream accumulating into a list reference
> +    """
> +    def __init__(self, lst):
> +        self.lst = lst
> +        
> +    def writable(self):
> +        return True
> +
> +    def seekable(self):
> +        return True
> +
> +    def write(self, b):
> +        self.lst.append(b)
> +
> +    def tell(self):
> +        return len(self.lst)
> +
>  def tostringlist(element, encoding=None, method=None):
> -    data = []
> -    class DataStream(io.BufferedIOBase):
> -        def writable(self):
> -            return True
> -
> -        def seekable(self):
> -            return True
> -
> -        def write(self, b):
> -            data.append(b)
> -
> -        def tell(self):
> -            return len(data)
> -
> -    ElementTree(element).write(DataStream(), encoding, method=method)
> -    return data
> +    lst = []
> +    stream = _ListDataStream(lst)
> +    ElementTree(element).write(stream, encoding, method=method)
> +    return lst
>  
>  ##
>  # Writes an element tree or element structure to sys.stdout.  This
> 
> 
> 
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
> 



From eliben at gmail.com  Sat Jul 21 19:11:52 2012
From: eliben at gmail.com (Eli Bendersky)
Date: Sat, 21 Jul 2012 20:11:52 +0300
Subject: [Python-Dev] cpython: Optimize tostringlist by taking the
 stream class outside the function. It's now
In-Reply-To: <juef5l$j4$2@dough.gmane.org>
References: <3Wc0gJ62Z5zPBM@mail.python.org> <juef5l$j4$2@dough.gmane.org>
Message-ID: <CAF-Rda9+wVJzN+11BVOSStnkH0E09X+3rD+VNogGaHVanzMNpg@mail.gmail.com>

On Sat, Jul 21, 2012 at 5:43 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> Uh, optimizations are not exactly what I want to see during feature freeze.
>

This is not a new optimization. It fixes a performance regression
introduced by an earlier bugfix changeset (for
http://bugs.python.org/issue1767933).

Eli

From eliben at gmail.com  Sat Jul 21 19:17:21 2012
From: eliben at gmail.com (Eli Bendersky)
Date: Sat, 21 Jul 2012 20:17:21 +0300
Subject: [Python-Dev] cpython (3.2): ISsue #14988: restore Python 2's
 behavior of raising ImportError when unable to
In-Reply-To: <juef3e$j4$1@dough.gmane.org>
References: <3Wbzgn72ckzPJr@mail.python.org> <juef3e$j4$1@dough.gmane.org>
Message-ID: <CAF-Rda_p0QYxv+WDJuZ=2-WBLCcUmDNnQi5xF6iC5grApw_jqA@mail.gmail.com>

On Sat, Jul 21, 2012 at 5:41 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> This looks like it will give an "error return without exception set"
> if the "if (expat_capi)" directly before the change is true, and
> expat_capi is then set to NULL in the branch.
>
> Georg
>

Good catch! I'll fix it soon.
Eli



> On 07/17/2012 01:25 PM, eli.bendersky wrote:
>> http://hg.python.org/cpython/rev/d896fd0a8ba7
>> changeset:   78152:d896fd0a8ba7
>> branch:      3.2
>> parent:      78142:71537aba3a0a
>> user:        Eli Bendersky <eliben at gmail.com>
>> date:        Tue Jul 17 14:20:38 2012 +0300
>> summary:
>>   ISsue #14988: restore Python 2's behavior of raising ImportError when unable to load pyexpat, instead of a SystemError/RuntimeError
>>
>> files:
>>   Modules/_elementtree.c |  2 ++
>>   1 files changed, 2 insertions(+), 0 deletions(-)
>>
>>
>> diff --git a/Modules/_elementtree.c b/Modules/_elementtree.c
>> --- a/Modules/_elementtree.c
>> +++ b/Modules/_elementtree.c
>> @@ -3117,6 +3117,8 @@
>>              expat_capi->MICRO_VERSION != XML_MICRO_VERSION)
>>              expat_capi = NULL;
>>      }
>> +    if (!expat_capi)
>> +        return NULL;
>>  #endif
>>
>>      elementtree_parseerror_obj = PyErr_NewException(
>>
>>
>>
>> _______________________________________________
>> Python-checkins mailing list
>> Python-checkins at python.org
>> http://mail.python.org/mailman/listinfo/python-checkins
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/eliben%40gmail.com

From chris.jerdonek at gmail.com  Sat Jul 21 21:11:56 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sat, 21 Jul 2012 12:11:56 -0700
Subject: [Python-Dev] four easy documentation patches
Message-ID: <CAOTb1wdR9WEjdHob2iyCZrPhM0NrC88vSB6tW2t7wo-Qn=Q86A@mail.gmail.com>

I was wondering if someone could review the following minor
documentation patches:

Document that filecmp.dircmp comparisons are "shallow" (7/5/2012):
http://bugs.python.org/issue15250

Document dircmp.left and dircmp.right (7/6/2012):
http://bugs.python.org/issue15269

Update "PyPI package display" documentation re: rst2html.py (7/5/2012):
http://bugs.python.org/issue15231

[Dev Guide] Update documentation around Dev Guide process (7/10/2012):
http://bugs.python.org/issue15311

Thanks,
--Chris

From kristjan at ccpgames.com  Sat Jul 21 23:29:50 2012
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Sat, 21 Jul 2012 21:29:50 +0000
Subject: [Python-Dev] test_hashlib
Message-ID: <EFE3877620384242A686D52278B7CCD3432157@RKV-IT-EXCH104.ccp.ad.local>

I was hit by this today.

in test_hashlib.py there is this:



def test_unknown_hash(self):

    self.assertRaises(ValueError, hashlib.new, 'spam spam spam spam spam')

    self.assertRaises(TypeError, hashlib.new, 1)



but in hashlib.py, there is this code:



except ImportError:

    pass # no extension module, this hash is unsupported.

raise ValueError('unsupported hash type %s' % name)





The code will raise ValueError when int(1) is passed in, but the unittests expect a TypeError.

So, which is correct?



K
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120721/62747a7b/attachment.html>

From solipsis at pitrou.net  Sat Jul 21 23:58:36 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 21 Jul 2012 23:58:36 +0200
Subject: [Python-Dev] test_hashlib
References: <EFE3877620384242A686D52278B7CCD3432157@RKV-IT-EXCH104.ccp.ad.local>
Message-ID: <20120721235836.7947cc54@pitrou.net>

On Sat, 21 Jul 2012 21:29:50 +0000
Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote:
> 
> The code will raise ValueError when int(1) is passed in, but the
> unittests expect a TypeError.

Well, if test_hashlib passes, surely your analysis is wrong, no?


-- 
Software development and contracting: http://pro.pitrou.net



From amauryfa at gmail.com  Sun Jul 22 00:56:55 2012
From: amauryfa at gmail.com (Amaury Forgeot d'Arc)
Date: Sun, 22 Jul 2012 00:56:55 +0200
Subject: [Python-Dev] test_hashlib
In-Reply-To: <20120721235836.7947cc54@pitrou.net>
References: <EFE3877620384242A686D52278B7CCD3432157@RKV-IT-EXCH104.ccp.ad.local>
	<20120721235836.7947cc54@pitrou.net>
Message-ID: <CAGmFidaWPmWZo=AFVDCPRh6jUu9SpjQF7RkifzpAQu+Cgh5pTg@mail.gmail.com>

2012/7/21 Antoine Pitrou <solipsis at pitrou.net>:
> Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote:
>>
>> The code will raise ValueError when int(1) is passed in, but the
>> unittests expect a TypeError.
>
> Well, if test_hashlib passes, surely your analysis is wrong, no?

In the normal case, yes:

>>> import hashlib
>>> hashlib.new(1)
TypeError: name must be a string

But if the _hashlib extension module is not available, the python
version is used and ValueError is raised:

>>> import sys
>>> sys.modules['_hashlib'] = None
>>> import hashlib
>>> hashlib.new(1)
ValueError: unsupported hash type 1

-- 
Amaury Forgeot d'Arc

From guido at python.org  Sun Jul 22 02:08:01 2012
From: guido at python.org (Guido van Rossum)
Date: Sat, 21 Jul 2012 17:08:01 -0700
Subject: [Python-Dev] test_hashlib
In-Reply-To: <CAGmFidaWPmWZo=AFVDCPRh6jUu9SpjQF7RkifzpAQu+Cgh5pTg@mail.gmail.com>
References: <EFE3877620384242A686D52278B7CCD3432157@RKV-IT-EXCH104.ccp.ad.local>
	<20120721235836.7947cc54@pitrou.net>
	<CAGmFidaWPmWZo=AFVDCPRh6jUu9SpjQF7RkifzpAQu+Cgh5pTg@mail.gmail.com>
Message-ID: <CAP7+vJ+CE1xp7QeFfyxhEq+46ATx9mnO4Adcw8BU-CdfC=uJTw@mail.gmail.com>

I think I see Kristj?n's point: the pure Python implementation handles
errors differently than the C implementation, so the unittest fails if
the pure Python version is enabled. I imagine this is a general
problem that often occurs when a pure Python version is normally
shadowed by a C extension, unless the unittest is rigged so that it
tests the pure Python version as well as the C version. But it still
remains a matter of judgment whether in a particular case the unittest
is overspecified or the Python version is incorrect. I think that in
this case Kristj?n's hunch is correct, and the pure Python version
needs to be adjusted to pass the test. I also think this is a low
priority issue since it only affects behavior of error cases.

--Guido

On Sat, Jul 21, 2012 at 3:56 PM, Amaury Forgeot d'Arc
<amauryfa at gmail.com> wrote:
> 2012/7/21 Antoine Pitrou <solipsis at pitrou.net>:
>> Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote:
>>>
>>> The code will raise ValueError when int(1) is passed in, but the
>>> unittests expect a TypeError.
>>
>> Well, if test_hashlib passes, surely your analysis is wrong, no?
>
> In the normal case, yes:
>
>>>> import hashlib
>>>> hashlib.new(1)
> TypeError: name must be a string
>
> But if the _hashlib extension module is not available, the python
> version is used and ValueError is raised:
>
>>>> import sys
>>>> sys.modules['_hashlib'] = None
>>>> import hashlib
>>>> hashlib.new(1)
> ValueError: unsupported hash type 1
>
> --
> Amaury Forgeot d'Arc
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From andrew.svetlov at gmail.com  Sun Jul 22 02:12:00 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sun, 22 Jul 2012 03:12:00 +0300
Subject: [Python-Dev] venv scripts for fish and csh shells
In-Reply-To: <50083671.6050903@oddbird.net>
References: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
	<50083671.6050903@oddbird.net>
Message-ID: <CAL3CFcWrvrRhZHb+f=cN6D8sHa47S_=EOeCTX_A8BuhcyJRW4w@mail.gmail.com>

I created http://bugs.python.org/issue15417 with required activation scripts.
If there are no objections I like to commit it after, say, three days.

On Thu, Jul 19, 2012 at 7:31 PM, Carl Meyer <carl at oddbird.net> wrote:
> On 07/19/2012 10:26 AM, Andrew Svetlov wrote:
>> virtualenv has virtualenv.csh and virtualenv.fish files.
>> Is there any reason for restricting venv to bash/zsh only?
>
> No. As far as I'm concerned, a patch to port the virtualenv csh and fish
> activate scripts to pyvenv would be welcome (though I can't commit said
> patch, so it might be good to hear if Vinay has a different opinion).
>
> Carl
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov

From kristjan at ccpgames.com  Sun Jul 22 03:05:35 2012
From: kristjan at ccpgames.com (=?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?=)
Date: Sun, 22 Jul 2012 01:05:35 +0000
Subject: [Python-Dev] test_hashlib
In-Reply-To: <CAGmFidaWPmWZo=AFVDCPRh6jUu9SpjQF7RkifzpAQu+Cgh5pTg@mail.gmail.com>
References: <EFE3877620384242A686D52278B7CCD3432157@RKV-IT-EXCH104.ccp.ad.local>
	<20120721235836.7947cc54@pitrou.net>,
	<CAGmFidaWPmWZo=AFVDCPRh6jUu9SpjQF7RkifzpAQu+Cgh5pTg@mail.gmail.com>
Message-ID: <EFE3877620384242A686D52278B7CCD3432805@RKV-IT-EXCH104.ccp.ad.local>

Indeed, shame on me for not mentioning this.
I rarely have the full complement of externals available when I'm doing python work, and it struck me that this unitest was failing.
I suppose it should be possible to write unittests that test more than one particular implementation.

K

________________________________________
Fr?: python-dev-bounces+kristjan=ccpgames.com at python.org [python-dev-bounces+kristjan=ccpgames.com at python.org] fyrir h&#246;nd Amaury Forgeot d'Arc [amauryfa at gmail.com]
Sent: 21. j?l? 2012 22:56
To: Antoine Pitrou
Cc: python-dev at python.org
Efni: Re: [Python-Dev] test_hashlib

2012/7/21 Antoine Pitrou <solipsis at pitrou.net>:
> Kristj?n Valur J?nsson <kristjan at ccpgames.com> wrote:
>>
>> The code will raise ValueError when int(1) is passed in, but the
>> unittests expect a TypeError.
>
> Well, if test_hashlib passes, surely your analysis is wrong, no?

In the normal case, yes:

>>> import hashlib
>>> hashlib.new(1)
TypeError: name must be a string

But if the _hashlib extension module is not available, the python
version is used and ValueError is raised:

>>> import sys
>>> sys.modules['_hashlib'] = None
>>> import hashlib
>>> hashlib.new(1)
ValueError: unsupported hash type 1

--
Amaury Forgeot d'Arc
_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/kristjan%40ccpgames.com

From rdmurray at bitdance.com  Sun Jul 22 04:10:55 2012
From: rdmurray at bitdance.com (R. David Murray)
Date: Sat, 21 Jul 2012 22:10:55 -0400
Subject: [Python-Dev] test_hashlib
In-Reply-To: <EFE3877620384242A686D52278B7CCD3432805@RKV-IT-EXCH104.ccp.ad.local>
References: <EFE3877620384242A686D52278B7CCD3432157@RKV-IT-EXCH104.ccp.ad.local>
	<20120721235836.7947cc54@pitrou.net>,
	<CAGmFidaWPmWZo=AFVDCPRh6jUu9SpjQF7RkifzpAQu+Cgh5pTg@mail.gmail.com>
	<EFE3877620384242A686D52278B7CCD3432805@RKV-IT-EXCH104.ccp.ad.local>
Message-ID: <20120722021056.256732502C2@webabinitio.net>

On Sun, 22 Jul 2012 01:05:35 -0000, =?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?= <kristjan at ccpgames.com> wrote:
> Indeed, shame on me for not mentioning this.
> I rarely have the full complement of externals available when I'm doing python work, and it struck me that this unitest was failing.
> I suppose it should be possible to write unittests that test more than one particular implementation.

It is indeed, and we have been moving the unit tests to do exactly
that for the cases where there is a Python implementation as well as
a C implementation.  It seems like it would be appropriate to open an
issue for doing that for hashlib, in addition to one for fixing this
particular issue with the Python version.

--David

From greg at krypto.org  Sun Jul 22 06:25:18 2012
From: greg at krypto.org (Gregory P. Smith)
Date: Sat, 21 Jul 2012 21:25:18 -0700
Subject: [Python-Dev] test_hashlib
In-Reply-To: <20120722021056.256732502C2@webabinitio.net>
References: <EFE3877620384242A686D52278B7CCD3432157@RKV-IT-EXCH104.ccp.ad.local>
	<20120721235836.7947cc54@pitrou.net>
	<CAGmFidaWPmWZo=AFVDCPRh6jUu9SpjQF7RkifzpAQu+Cgh5pTg@mail.gmail.com>
	<EFE3877620384242A686D52278B7CCD3432805@RKV-IT-EXCH104.ccp.ad.local>
	<20120722021056.256732502C2@webabinitio.net>
Message-ID: <CAGE7PNJwKqh-TL3spss3UqxG9YcLjsDeyRpu0QyyyXyRLtM=bg@mail.gmail.com>

Fixed. The TypeError in this nonsense never gonna work use case is now
consistent in 2.7, 3.2 and 3.3.

On Sat, Jul 21, 2012 at 7:10 PM, R. David Murray <rdmurray at bitdance.com>wrote:

> On Sun, 22 Jul 2012 01:05:35 -0000,
> =?iso-8859-1?Q?Kristj=E1n_Valur_J=F3nsson?= <kristjan at ccpgames.com> wrote:
> > Indeed, shame on me for not mentioning this.
> > I rarely have the full complement of externals available when I'm doing
> python work, and it struck me that this unitest was failing.
> > I suppose it should be possible to write unittests that test more than
> one particular implementation.
>
> It is indeed, and we have been moving the unit tests to do exactly
> that for the cases where there is a Python implementation as well as
> a C implementation.  It seems like it would be appropriate to open an
> issue for doing that for hashlib, in addition to one for fixing this
> particular issue with the Python version.
>
> --David
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120721/1acfb23b/attachment.html>

From techtonik at gmail.com  Sun Jul 22 10:21:38 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Sun, 22 Jul 2012 11:21:38 +0300
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
Message-ID: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>

http://docs.python.org/py3k/howto/pyporting.html#during-installation

What's the point in making implicit Python 3 check here:
try:  # Python 3
  from distutils.command.build_py import build_py_2to3 as build_py
except ImportError:  # Python 2
  from distutils.command.build_py import build_py

instead of explicit check like:
import sys
if sys.version_info[0] >= 3:
    from distutils.command.build_py import build_py_2to3 as build_py

?
--
anatoly t.

From techtonik at gmail.com  Sun Jul 22 12:18:35 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Sun, 22 Jul 2012 13:18:35 +0300
Subject: [Python-Dev] Print policy for deprecated modules
Message-ID: <CAPkN8x+yQGkf9LtXj_zfRK5BB8U53+=AUHq+6hcBJe4RV9rz4Q@mail.gmail.com>

What is a print policy for deprecated modules? "new" module is
deprecated in 2.6, but 2.7.3 doesn't print any warnings. Is it a bug?
    python -Wd -c "import new"
--
anatoly t.

From rdmurray at bitdance.com  Sun Jul 22 15:08:48 2012
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 22 Jul 2012 09:08:48 -0400
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
Message-ID: <20120722130849.A2968250064@webabinitio.net>

On Sun, 22 Jul 2012 11:21:38 +0300, anatoly techtonik <techtonik at gmail.com> wrote:
> http://docs.python.org/py3k/howto/pyporting.html#during-installation
> 
> What's the point in making implicit Python 3 check here:
> try:  # Python 3
>   from distutils.command.build_py import build_py_2to3 as build_py
> except ImportError:  # Python 2
>   from distutils.command.build_py import build_py
> 
> instead of explicit check like:
> import sys
> if sys.version_info[0] >= 3:
>     from distutils.command.build_py import build_py_2to3 as build_py

It's called testing for the thing that actually matters, rather than
testing a constant with a much broader meaning.  Yes, in this case the
results are the same, but IMO it is better programming practice to test
the thing that actually matters when you can.

--David

From g.brandl at gmx.net  Sun Jul 22 17:21:54 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 22 Jul 2012 17:21:54 +0200
Subject: [Python-Dev] venv scripts for fish and csh shells
In-Reply-To: <CAL3CFcWrvrRhZHb+f=cN6D8sHa47S_=EOeCTX_A8BuhcyJRW4w@mail.gmail.com>
References: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
	<50083671.6050903@oddbird.net>
	<CAL3CFcWrvrRhZHb+f=cN6D8sHa47S_=EOeCTX_A8BuhcyJRW4w@mail.gmail.com>
Message-ID: <juh5qj$q9n$1@dough.gmane.org>

These are new features, please do not commit before 3.3 is released.

Georg

On 07/22/2012 02:12 AM, Andrew Svetlov wrote:
> I created http://bugs.python.org/issue15417 with required activation scripts.
> If there are no objections I like to commit it after, say, three days.
> 
> On Thu, Jul 19, 2012 at 7:31 PM, Carl Meyer <carl at oddbird.net> wrote:
>> On 07/19/2012 10:26 AM, Andrew Svetlov wrote:
>>> virtualenv has virtualenv.csh and virtualenv.fish files.
>>> Is there any reason for restricting venv to bash/zsh only?
>>
>> No. As far as I'm concerned, a patch to port the virtualenv csh and fish
>> activate scripts to pyvenv would be welcome (though I can't commit said
>> patch, so it might be good to hear if Vinay has a different opinion).
>>
>> Carl
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
> 
> 
> 



From andrew.svetlov at gmail.com  Sun Jul 22 18:10:59 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sun, 22 Jul 2012 19:10:59 +0300
Subject: [Python-Dev] venv scripts for fish and csh shells
In-Reply-To: <juh5qj$q9n$1@dough.gmane.org>
References: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
	<50083671.6050903@oddbird.net>
	<CAL3CFcWrvrRhZHb+f=cN6D8sHa47S_=EOeCTX_A8BuhcyJRW4w@mail.gmail.com>
	<juh5qj$q9n$1@dough.gmane.org>
Message-ID: <CAL3CFcWXB3e=-68SQwvBkg=sLvzL+1mBtoFCOu204Wiawphayg@mail.gmail.com>

Georg, sorry, I've committed it before you answered.

If you want, I will revert it back.
But, please, describe me why those scripts are to late for 3.3?
I strongly checked correctness of both activation script for fish and
csh/tsch shells.
Works fine.

It's not *new feature*, it is just extending support to csh and fish shells.
Virtualenv (base for venv) already contains this activation scripts.
I just converted those files to venv with replacing virtualenv
specific environment variables to schema supported by venv.
I see nothing bad for adding to venv support for fish and csh as well
as already supported bash/zsh

Say again, I triple checked correctness of added scripts. With
defining VIRTUAL_ENV_DISABLE_PROMPT as well as without it.
My addition is safe, trivial, correct and it is not new feature ? just
fixing of missing support for widely used shells.

Anyway.
You are Release Manager ? the main person for 3.3 release.
If you feel what my changes is potentially dangerous and should be
removed from 3.3 ? I'll do it.
But, please, do benevolent.



On Sun, Jul 22, 2012 at 6:21 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> These are new features, please do not commit before 3.3 is released.
>
> Georg
>
> On 07/22/2012 02:12 AM, Andrew Svetlov wrote:
>> I created http://bugs.python.org/issue15417 with required activation scripts.
>> If there are no objections I like to commit it after, say, three days.
>>
>> On Thu, Jul 19, 2012 at 7:31 PM, Carl Meyer <carl at oddbird.net> wrote:
>>> On 07/19/2012 10:26 AM, Andrew Svetlov wrote:
>>>> virtualenv has virtualenv.csh and virtualenv.fish files.
>>>> Is there any reason for restricting venv to bash/zsh only?
>>>
>>> No. As far as I'm concerned, a patch to port the virtualenv csh and fish
>>> activate scripts to pyvenv would be welcome (though I can't commit said
>>> patch, so it might be good to hear if Vinay has a different opinion).
>>>
>>> Carl
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> http://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com
>>
>>
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov

From g.brandl at gmx.net  Sun Jul 22 18:50:24 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Sun, 22 Jul 2012 18:50:24 +0200
Subject: [Python-Dev] venv scripts for fish and csh shells
In-Reply-To: <CAL3CFcWXB3e=-68SQwvBkg=sLvzL+1mBtoFCOu204Wiawphayg@mail.gmail.com>
References: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
	<50083671.6050903@oddbird.net>
	<CAL3CFcWrvrRhZHb+f=cN6D8sHa47S_=EOeCTX_A8BuhcyJRW4w@mail.gmail.com>
	<juh5qj$q9n$1@dough.gmane.org>
	<CAL3CFcWXB3e=-68SQwvBkg=sLvzL+1mBtoFCOu204Wiawphayg@mail.gmail.com>
Message-ID: <juhb0h$i0$1@dough.gmane.org>

On 07/22/2012 06:10 PM, Andrew Svetlov wrote:
> Georg, sorry, I've committed it before you answered.

After you wrote that you'd wait three days...  anyway, I can't find the
commit in the repository, so that's fine.

> If you want, I will revert it back.
> But, please, describe me why those scripts are to late for 3.3?
> I strongly checked correctness of both activation script for fish and
> csh/tsch shells.
> Works fine.

"Works fine" is not good enough.  There are hundreds of potential patches
on the tracker implementing new features of which the same could be said.

> It's not *new feature*, it is just extending support to csh and fish shells.

That is a new feature in my view.  If it was an important feature, someone
would have thought of it while features were allowed.

> Virtualenv (base for venv) already contains this activation scripts.
> I just converted those files to venv with replacing virtualenv
> specific environment variables to schema supported by venv.
> I see nothing bad for adding to venv support for fish and csh as well
> as already supported bash/zsh
> 
> Say again, I triple checked correctness of added scripts. With
> defining VIRTUAL_ENV_DISABLE_PROMPT as well as without it.
> My addition is safe, trivial, correct and it is not new feature ? just
> fixing of missing support for widely used shells.
> 
> Anyway.
> You are Release Manager ? the main person for 3.3 release.
> If you feel what my changes is potentially dangerous and should be
> removed from 3.3 ? I'll do it.
> But, please, do benevolent.

The point of feature freeze is that there are no new features.  Where
can I draw the line, when everybody argues like that and I agree?

Georg


From andrew.svetlov at gmail.com  Sun Jul 22 19:39:15 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Sun, 22 Jul 2012 20:39:15 +0300
Subject: [Python-Dev] venv scripts for fish and csh shells
In-Reply-To: <juhb0h$i0$1@dough.gmane.org>
References: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
	<50083671.6050903@oddbird.net>
	<CAL3CFcWrvrRhZHb+f=cN6D8sHa47S_=EOeCTX_A8BuhcyJRW4w@mail.gmail.com>
	<juh5qj$q9n$1@dough.gmane.org>
	<CAL3CFcWXB3e=-68SQwvBkg=sLvzL+1mBtoFCOu204Wiawphayg@mail.gmail.com>
	<juhb0h$i0$1@dough.gmane.org>
Message-ID: <CAL3CFcVy=j2hyqWZwripmfzjMpQHdCrduG-61+9aTeyZe+pR=g@mail.gmail.com>

Ok.
Sorry for my mistake ? there are really no commits for
http://bugs.python.org/issue15417
It's look important for me ? but you are release manager.
If you consider the patch as potentially dangerous ? I have to agree with you.
You are the master :)

On Sun, Jul 22, 2012 at 7:50 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> On 07/22/2012 06:10 PM, Andrew Svetlov wrote:
>> Georg, sorry, I've committed it before you answered.
>
> After you wrote that you'd wait three days...  anyway, I can't find the
> commit in the repository, so that's fine.
>
>> If you want, I will revert it back.
>> But, please, describe me why those scripts are to late for 3.3?
>> I strongly checked correctness of both activation script for fish and
>> csh/tsch shells.
>> Works fine.
>
> "Works fine" is not good enough.  There are hundreds of potential patches
> on the tracker implementing new features of which the same could be said.
>
>> It's not *new feature*, it is just extending support to csh and fish shells.
>
> That is a new feature in my view.  If it was an important feature, someone
> would have thought of it while features were allowed.
>
>> Virtualenv (base for venv) already contains this activation scripts.
>> I just converted those files to venv with replacing virtualenv
>> specific environment variables to schema supported by venv.
>> I see nothing bad for adding to venv support for fish and csh as well
>> as already supported bash/zsh
>>
>> Say again, I triple checked correctness of added scripts. With
>> defining VIRTUAL_ENV_DISABLE_PROMPT as well as without it.
>> My addition is safe, trivial, correct and it is not new feature ? just
>> fixing of missing support for widely used shells.
>>
>> Anyway.
>> You are Release Manager ? the main person for 3.3 release.
>> If you feel what my changes is potentially dangerous and should be
>> removed from 3.3 ? I'll do it.
>> But, please, do benevolent.
>
> The point of feature freeze is that there are no new features.  Where
> can I draw the line, when everybody argues like that and I agree?
>
> Georg
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov

From solipsis at pitrou.net  Sun Jul 22 19:54:40 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 22 Jul 2012 19:54:40 +0200
Subject: [Python-Dev] venv scripts for fish and csh shells
References: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
	<50083671.6050903@oddbird.net>
	<CAL3CFcWrvrRhZHb+f=cN6D8sHa47S_=EOeCTX_A8BuhcyJRW4w@mail.gmail.com>
	<juh5qj$q9n$1@dough.gmane.org>
	<CAL3CFcWXB3e=-68SQwvBkg=sLvzL+1mBtoFCOu204Wiawphayg@mail.gmail.com>
	<juhb0h$i0$1@dough.gmane.org>
	<CAL3CFcVy=j2hyqWZwripmfzjMpQHdCrduG-61+9aTeyZe+pR=g@mail.gmail.com>
Message-ID: <20120722195440.2cfc10a2@pitrou.net>

On Sun, 22 Jul 2012 20:39:15 +0300
Andrew Svetlov <andrew.svetlov at gmail.com> wrote:
> Ok.
> Sorry for my mistake ? there are really no commits for
> http://bugs.python.org/issue15417
> It's look important for me ? but you are release manager.
> If you consider the patch as potentially dangerous ? I have to agree with you.
> You are the master :)

This is not because Georg is the master.  When a release is nearing we
think it is important to avoid introducing potential regressions,
except when fixing existing bugs.  That's why we have a feature freeze
which extends to many kinds of "enhancements", including performance
improvements: really, it is more of a "bugfix-only period".

One could propose other mechanisms for release preparation, but in the
meantime, it is important as a community that we all follow similar
rules.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From oscar.benjamin at bristol.ac.uk  Sun Jul 22 21:23:42 2012
From: oscar.benjamin at bristol.ac.uk (Oscar Benjamin)
Date: Sun, 22 Jul 2012 20:23:42 +0100
Subject: [Python-Dev]  2to3 porting HOWTO: setup.py question
In-Reply-To: <CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
Message-ID: <CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>

On 22 July 2012 14:08, R. David Murray <rdmurray at bitdance.com> wrote:

> On Sun, 22 Jul 2012 11:21:38 +0300, anatoly techtonik <techtonik at gmail.com>
> wrote:
> > http://docs.python.org/py3k/howto/pyporting.html#during-installation
> >
> > What's the point in making implicit Python 3 check here:
> > try:  # Python 3
> >   from distutils.command.build_py import build_py_2to3 as build_py
> > except ImportError:  # Python 2
> >   from distutils.command.build_py import build_py
> >
> > instead of explicit check like:
> > import sys
> > if sys.version_info[0] >= 3:
> >     from distutils.command.build_py import build_py_2to3 as build_py
>
> It's called testing for the thing that actually matters, rather than
> testing a constant with a much broader meaning.  Yes, in this case the
> results are the same, but IMO it is better programming practice to test
> the thing that actually matters when you can.


I recently changed a setup.py from try/ImportError to an explicit
sys.version_info check. I'm not totally sure how to reproduce this but I
had a problem where I was installing into a 2.x virtualenv and it was
running 2to3 during install and subsequently failing to import the 3.x code
(the problem didn't occur when using the same python that generated the
virtualenv).

I may be wrong but I imagined that sometimes build_py_2to3 is importable on
2.x, perhaps for cross-building or something. In any case 'testing the
thing that matters' means testing what version of Python you are about to
install into not whether the python version supports running 2to3.

Cheers,
Oscar.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120722/e2c1f59d/attachment.html>

From rdmurray at bitdance.com  Sun Jul 22 21:57:10 2012
From: rdmurray at bitdance.com (R. David Murray)
Date: Sun, 22 Jul 2012 15:57:10 -0400
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
Message-ID: <20120722195711.16863250054@webabinitio.net>

Benjamin sent me this message separately(*) privately and I responded
privately.  Here is my response.

(*) or his mailer did

On Sun, 22 Jul 2012 20:22:50 +0100, Oscar Benjamin
<oscar.benjamin at bristol.ac.uk> wrote:
> On 22 July 2012 14:08, R. David Murray <rdmurray at bitdance.com> wrote:
>
> > On Sun, 22 Jul 2012 11:21:38 +0300, anatoly techtonik <techtonik at gmail.com>
> > wrote:
> > > http://docs.python.org/py3k/howto/pyporting.html#during-installation
> > >
> > > What's the point in making implicit Python 3 check here:
> > > try:  # Python 3
> > >   from distutils.command.build_py import build_py_2to3 as build_py
> > > except ImportError:  # Python 2
> > >   from distutils.command.build_py import build_py
> > >
> > > instead of explicit check like:
> > > import sys
> > > if sys.version_info[0] >= 3:
> > >     from distutils.command.build_py import build_py_2to3 as build_py
> >
> > It's called testing for the thing that actually matters, rather than
> > testing a constant with a much broader meaning.  Yes, in this case the
> > results are the same, but IMO it is better programming practice to test
> > the thing that actually matters when you can.
>
>
> I recently changed a setup.py from try/ImportError to an explicit
> sys.version_info check. I'm not totally sure how to reproduce this but I
> had a problem where I was installing into a 2.x virtualenv and it was
> running 2to3 during install and subsequently failing to import the 3.x code
> (the problem didn't occur when using the same python that generated the
> virtualenv).
>
> I may be wrong but I imagined that sometimes build_py_2to3 is importable on
> 2.x, perhaps for cross-building or something. In any case 'testing the
> thing that matters' means testing what version of Python you are about to
> install into not whether the python version supports running 2to3.

I'm not familiar with distutils, really, so you could be right about
what it is important to test.  I was commenting based on the code
snippet presented, which just deciding which "build" object to use.
If build_py_2to3 can be imported by python2 and subsequently screws up
the build, then yes the logic is incorrect.  But I have to defer to the
packaging people on that.  (I wish I had time to help with packaging
because it is important, but it doesn't seem like a sensible place for
me personally to spend my currently available time.)

--David

From ezio.melotti at gmail.com  Sun Jul 22 23:06:42 2012
From: ezio.melotti at gmail.com (Ezio Melotti)
Date: Sun, 22 Jul 2012 23:06:42 +0200
Subject: [Python-Dev] Print policy for deprecated modules
In-Reply-To: <CAPkN8x+yQGkf9LtXj_zfRK5BB8U53+=AUHq+6hcBJe4RV9rz4Q@mail.gmail.com>
References: <CAPkN8x+yQGkf9LtXj_zfRK5BB8U53+=AUHq+6hcBJe4RV9rz4Q@mail.gmail.com>
Message-ID: <CACBhJdGUrZRm-Ra1Z=9_StXEQTtWW4QCfN30C=vTr8Fw9Nupvw@mail.gmail.com>

On Sun, Jul 22, 2012 at 12:18 PM, anatoly techtonik <techtonik at gmail.com> wrote:
> What is a print policy for deprecated modules? "new" module is
> deprecated in 2.6, but 2.7.3 doesn't print any warnings. Is it a bug?
>     python -Wd -c "import new"

In theory this should show a warning, but for some reason it doesn't.
Reading the messages on http://bugs.python.org/issue1247765 it seems
that there wasn't a clear consensus about the deprecation schedule, so
that might be the reason.
If the warning is missing just because no one remembered to add it, I
guess it can still be fixed on 2.7, but for 2.6 is too late now.

FWIW you get a warning if you use the -3 flag:
  $ python -Wd -3 -c "import new"
  -c:1: DeprecationWarning: The 'new' module has been removed in
Python 3.0; use the 'types' module instead.

From oscar.j.benjamin at gmail.com  Sun Jul 22 23:21:04 2012
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Sun, 22 Jul 2012 22:21:04 +0100
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <20120722195711.16863250054@webabinitio.net>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
Message-ID: <CAHVvXxSVPWS+mgP9ch0V9H2UHH0vFhfufuV5fx=BoASz3AntPw@mail.gmail.com>

On 22 July 2012 20:57, R. David Murray <rdmurray at bitdance.com> wrote:

> Benjamin sent me this message separately(*) privately and I responded
> privately.  Here is my response.
>

(*) or his mailer did
>

I think I accidentally replied from my work email address (which is not
subscribed to python-dev) and so the second copy I sent to python-dev
wasn't posted.


>
> On Sun, 22 Jul 2012 20:22:50 +0100, Oscar Benjamin
> <oscar.benjamin at bristol.ac.uk> wrote:
> > On 22 July 2012 14:08, R. David Murray <rdmurray at bitdance.com> wrote:
> >
> > > On Sun, 22 Jul 2012 11:21:38 +0300, anatoly techtonik <
> techtonik at gmail.com>
> > > wrote:
> > > > http://docs.python.org/py3k/howto/pyporting.html#during-installation
> > > >
> > > > What's the point in making implicit Python 3 check here:
> > > > try:  # Python 3
> > > >   from distutils.command.build_py import build_py_2to3 as build_py
> > > > except ImportError:  # Python 2
> > > >   from distutils.command.build_py import build_py
> > > >
> > > > instead of explicit check like:
> > > > import sys
> > > > if sys.version_info[0] >= 3:
> > > >     from distutils.command.build_py import build_py_2to3 as build_py
> > >
> > > It's called testing for the thing that actually matters, rather than
> > > testing a constant with a much broader meaning.  Yes, in this case the
> > > results are the same, but IMO it is better programming practice to test
> > > the thing that actually matters when you can.
> >
> >
> > I recently changed a setup.py from try/ImportError to an explicit
> > sys.version_info check. I'm not totally sure how to reproduce this but I
> > had a problem where I was installing into a 2.x virtualenv and it was
> > running 2to3 during install and subsequently failing to import the 3.x
> code
> > (the problem didn't occur when using the same python that generated the
> > virtualenv).
> >
> > I may be wrong but I imagined that sometimes build_py_2to3 is importable
> on
> > 2.x, perhaps for cross-building or something. In any case 'testing the
> > thing that matters' means testing what version of Python you are about to
> > install into not whether the python version supports running 2to3.
>
> I'm not familiar with distutils, really, so you could be right about
> what it is important to test.  I was commenting based on the code
> snippet presented, which just deciding which "build" object to use.
> If build_py_2to3 can be imported by python2 and subsequently screws up
> the build, then yes the logic is incorrect.  But I have to defer to the
> packaging people on that.  (I wish I had time to help with packaging
> because it is important, but it doesn't seem like a sensible place for
> me personally to spend my currently available time.)
>

I'm not currently able to reproduce the problem on this machine. I think I
was using pip/easy_install to install distribution X from PyPI that
depended on distribution Y also on PyPI into an isolated 2.x virtualenv and
found that distribution Y was converted with 2to3 when it was automatically
installed. It could be a bug but I'm not confident enough with virtualenv
to say that it wasn't just me messing things up somehow.

Either way I still think that in this particular case a version check is
the most explicit and appropriate thing to do. The author of a distribution
that is distributed as Python 2.x code and installed on Python 3.x using
2to3 knows precisely when they want 2to3 to run and when they don't so why
not make that explicit?

As an aside, I find the check slightly easier to read if it is written like:

if sys.version_info >= (3, 0):
    from distutils.build_py import build_py_2to3 as build_py

Cheers,
Oscar.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120722/37eb4461/attachment-0001.html>

From guido at python.org  Mon Jul 23 00:38:48 2012
From: guido at python.org (Guido van Rossum)
Date: Sun, 22 Jul 2012 15:38:48 -0700
Subject: [Python-Dev] Print policy for deprecated modules
In-Reply-To: <CACBhJdGUrZRm-Ra1Z=9_StXEQTtWW4QCfN30C=vTr8Fw9Nupvw@mail.gmail.com>
References: <CAPkN8x+yQGkf9LtXj_zfRK5BB8U53+=AUHq+6hcBJe4RV9rz4Q@mail.gmail.com>
	<CACBhJdGUrZRm-Ra1Z=9_StXEQTtWW4QCfN30C=vTr8Fw9Nupvw@mail.gmail.com>
Message-ID: <CAP7+vJ+MQin1UoGcz57TxBgEgxYHLxRYt3QdjDMcT=SgHuSgLQ@mail.gmail.com>

On Sun, Jul 22, 2012 at 2:06 PM, Ezio Melotti <ezio.melotti at gmail.com> wrote:
> On Sun, Jul 22, 2012 at 12:18 PM, anatoly techtonik <techtonik at gmail.com> wrote:
>> What is a print policy for deprecated modules? "new" module is
>> deprecated in 2.6, but 2.7.3 doesn't print any warnings. Is it a bug?
>>     python -Wd -c "import new"
>
> In theory this should show a warning, but for some reason it doesn't.
> Reading the messages on http://bugs.python.org/issue1247765 it seems
> that there wasn't a clear consensus about the deprecation schedule, so
> that might be the reason.
> If the warning is missing just because no one remembered to add it, I
> guess it can still be fixed on 2.7, but for 2.6 is too late now.
>
> FWIW you get a warning if you use the -3 flag:
>   $ python -Wd -3 -c "import new"
>   -c:1: DeprecationWarning: The 'new' module has been removed in
> Python 3.0; use the 'types' module instead.

IIRC we decided not to add new deprecations to 2.7, since the
deprecation would only affect the Python 3 line anyway -- nothing that
exists in Python 2 is going away in Python 2...

-- 
--Guido van Rossum (python.org/~guido)

From meadori at gmail.com  Mon Jul 23 07:34:02 2012
From: meadori at gmail.com (Meador Inge)
Date: Mon, 23 Jul 2012 00:34:02 -0500
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.2 -> default):
 Merge #15232: correctly mangle From lines in MIME preamble and epilogue
In-Reply-To: <3WgQlD5GcszPGf@mail.python.org>
References: <3WgQlD5GcszPGf@mail.python.org>
Message-ID: <CAK1QooqfEo1e=H6RU0hE18jm7K9q2C5DdYYVmAwZ_B-OQ9qjSA@mail.gmail.com>

On Sun, Jul 22, 2012 at 8:55 PM, r.david.murray
<python-checkins at python.org> wrote:

> http://hg.python.org/cpython/rev/80b81658455b
> changeset:   78246:80b81658455b
> parent:      78244:c43d73277756
> parent:      78245:b97f65f2298d
> user:        R David Murray <rdmurray at bitdance.com>
> date:        Sun Jul 22 21:53:54 2012 -0400
> summary:
>   Merge #15232: correctly mangle From lines in MIME preamble and epilogue
>
> files:
>   Lib/email/generator.py            |  12 ++++++++-
>   Lib/test/test_email/test_email.py |  22 +++++++++++++++++++
>   Misc/NEWS                         |   3 ++
>   3 files changed, 35 insertions(+), 2 deletions(-)

I'm not quite sure what happened, but something seems to have gone wrong
with this merge.  After doing the following while on the "default" branch:

$ hg merge 3.2

I see:

$ hg st
M Lib/email/generator.py
M Lib/test/test_email/test_email.py
M Misc/NEWS

and a bunch of conflicts in 'Misc/NEWS'.

>
> diff --git a/Lib/email/generator.py b/Lib/email/generator.py
> --- a/Lib/email/generator.py
> +++ b/Lib/email/generator.py
> @@ -252,7 +252,11 @@
>              msg.set_boundary(boundary)
>          # If there's a preamble, write it out, with a trailing CRLF
>          if msg.preamble is not None:
> -            self.write(msg.preamble + self._NL)
> +            if self._mangle_from_:
> +                preamble = fcre.sub('>From ', msg.preamble)
> +            else:
> +                preamble = msg.preamble
> +            self.write(preamble + self._NL)
>          # dash-boundary transport-padding CRLF
>          self.write('--' + boundary + self._NL)
>          # body-part
> @@ -270,7 +274,11 @@
>          self.write(self._NL + '--' + boundary + '--')
>          if msg.epilogue is not None:
>              self.write(self._NL)
> -            self.write(msg.epilogue)
> +            if self._mangle_from_:
> +                epilogue = fcre.sub('>From ', msg.epilogue)
> +            else:
> +                epilogue = msg.epilogue
> +            self.write(epilogue)
>
>      def _handle_multipart_signed(self, msg):
>          # The contents of signed parts has to stay unmodified in order to keep
> diff --git a/Lib/test/test_email/test_email.py b/Lib/test/test_email/test_email.py
> --- a/Lib/test/test_email/test_email.py
> +++ b/Lib/test/test_email/test_email.py
> @@ -1283,6 +1283,28 @@
>  Blah blah blah
>  """)
>
> +    def test_mangle_from_in_preamble_and_epilog(self):
> +        s = StringIO()
> +        g = Generator(s, mangle_from_=True)
> +        msg = email.message_from_string(textwrap.dedent("""\
> +            From: foo at bar.com
> +            Mime-Version: 1.0
> +            Content-Type: multipart/mixed; boundary=XXX
> +
> +            From somewhere unknown
> +
> +            --XXX
> +            Content-Type: text/plain
> +
> +            foo
> +
> +            --XXX--
> +
> +            From somewhere unknowable
> +            """))
> +        g.flatten(msg)
> +        self.assertEqual(len([1 for x in s.getvalue().split('\n')
> +                                  if x.startswith('>From ')]), 2)
>
>
>  # Test the basic MIMEAudio class
> diff --git a/Misc/NEWS b/Misc/NEWS
> --- a/Misc/NEWS
> +++ b/Misc/NEWS
> @@ -52,6 +52,9 @@
>  Library
>  -------
>
> +- Issue #15232: when mangle_from is True, email.Generator now correctly mangles
> +  lines that start with 'From' that occur in a MIME preamble or epilogue.
> +
>  - Issue #15094: Incorrectly placed #endif in _tkinter.c.
>    Patch by Serhiy Storchaka.
>
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>



-- 
# Meador

From stefan at epy.co.at  Mon Jul 23 09:09:52 2012
From: stefan at epy.co.at (Stefan H. Holek)
Date: Mon, 23 Jul 2012 09:09:52 +0200
Subject: [Python-Dev] Why no venv in existing directory?
In-Reply-To: <50083843.2040105@oddbird.net>
References: <26A4F829-8091-4CA4-A4F5-A18B5813C1CC@epy.co.at>
	<50083843.2040105@oddbird.net>
Message-ID: <F899DB32-14B4-471E-91D3-46EEFD3DB7CE@epy.co.at>

Hi Carl, 

On 19.07.2012, at 18:39, Carl Meyer wrote:

> Hi Stefan,
> 
> On 07/19/2012 06:28 AM, Stefan H. Holek wrote:
>> While trying 3.3 beta I found that I cannot use my favorite
>> virtualenv pattern with pyvenv:
> 
> I'd have no problem with lifting the restriction.
> 
> I don't recall any clear rationale; I think it was probably just the
> simplest implementation initially, and no one ever raised it as an issue
> in the PEP process.

Thanks for your reply. The feature certainly is on *my* wish-list but I might be alone here. ;-)

Stefan

-- 
Stefan H. Holek
stefan at epy.co.at


From lukasz at langa.pl  Mon Jul 23 09:52:23 2012
From: lukasz at langa.pl (=?iso-8859-2?Q?=A3ukasz_Langa?=)
Date: Mon, 23 Jul 2012 09:52:23 +0200
Subject: [Python-Dev] Why no venv in existing directory?
In-Reply-To: <F899DB32-14B4-471E-91D3-46EEFD3DB7CE@epy.co.at>
References: <26A4F829-8091-4CA4-A4F5-A18B5813C1CC@epy.co.at>
	<50083843.2040105@oddbird.net>
	<F899DB32-14B4-471E-91D3-46EEFD3DB7CE@epy.co.at>
Message-ID: <1979E575-41E7-4DCD-A0A7-D3E03EDCA310@langa.pl>

Wiadomo?? napisana przez Stefan H. Holek w dniu 23 lip 2012, o godz. 09:09:

> Hi Carl, 
> 
> On 19.07.2012, at 18:39, Carl Meyer wrote:
> 
>> Hi Stefan,
>> 
>> On 07/19/2012 06:28 AM, Stefan H. Holek wrote:
>>> While trying 3.3 beta I found that I cannot use my favorite
>>> virtualenv pattern with pyvenv:
>> 
>> I'd have no problem with lifting the restriction.
>> 
>> I don't recall any clear rationale; I think it was probably just the
>> simplest implementation initially, and no one ever raised it as an issue
>> in the PEP process.
> 
> Thanks for your reply. The feature certainly is on *my* wish-list but I might be alone here. ;-)


You are not.

-- 
Best regards,
?ukasz Langa
Senior Systems Architecture Engineer

IT Infrastructure Department
Grupa Allegro Sp. z o.o.

http://lukasz.langa.pl/
+48 791 080 144

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120723/6868ec68/attachment.html>

From meadori at gmail.com  Mon Jul 23 16:17:33 2012
From: meadori at gmail.com (Meador Inge)
Date: Mon, 23 Jul 2012 09:17:33 -0500
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.2 -> default):
 Merge #15232: correctly mangle From lines in MIME preamble and epilogue
In-Reply-To: <CAK1QooqfEo1e=H6RU0hE18jm7K9q2C5DdYYVmAwZ_B-OQ9qjSA@mail.gmail.com>
References: <3WgQlD5GcszPGf@mail.python.org>
	<CAK1QooqfEo1e=H6RU0hE18jm7K9q2C5DdYYVmAwZ_B-OQ9qjSA@mail.gmail.com>
Message-ID: <CAK1QooqMVy+LvBS9LFcdCroxifaed01pM60iwqy14D0FYtRpEA@mail.gmail.com>

On Mon, Jul 23, 2012 at 12:34 AM, Meador Inge <meadori at gmail.com> wrote:

> On Sun, Jul 22, 2012 at 8:55 PM, r.david.murray
> <python-checkins at python.org> wrote:
>
>> http://hg.python.org/cpython/rev/80b81658455b
>> changeset:   78246:80b81658455b
>> parent:      78244:c43d73277756
>> parent:      78245:b97f65f2298d
>> user:        R David Murray <rdmurray at bitdance.com>
>> date:        Sun Jul 22 21:53:54 2012 -0400
>> summary:
>>   Merge #15232: correctly mangle From lines in MIME preamble and epilogue
>>
>> files:
>>   Lib/email/generator.py            |  12 ++++++++-
>>   Lib/test/test_email/test_email.py |  22 +++++++++++++++++++
>>   Misc/NEWS                         |   3 ++
>>   3 files changed, 35 insertions(+), 2 deletions(-)
>
> I'm not quite sure what happened, but something seems to have gone wrong
> with this merge.  After doing the following while on the "default" branch:
>
> $ hg merge 3.2
>
> I see:
>
> $ hg st
> M Lib/email/generator.py
> M Lib/test/test_email/test_email.py
> M Misc/NEWS
>
> and a bunch of conflicts in 'Misc/NEWS'.

Hmmm, actually it looks like this head merge that Senthil performed:
http://hg.python.org/cpython/rev/af2e044609ca.

Anyway, I resolved the conflicts.

-- Meador

From andrew.svetlov at gmail.com  Mon Jul 23 18:24:15 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Mon, 23 Jul 2012 19:24:15 +0300
Subject: [Python-Dev] venv scripts for fish and csh shells
In-Reply-To: <20120722195440.2cfc10a2@pitrou.net>
References: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
	<50083671.6050903@oddbird.net>
	<CAL3CFcWrvrRhZHb+f=cN6D8sHa47S_=EOeCTX_A8BuhcyJRW4w@mail.gmail.com>
	<juh5qj$q9n$1@dough.gmane.org>
	<CAL3CFcWXB3e=-68SQwvBkg=sLvzL+1mBtoFCOu204Wiawphayg@mail.gmail.com>
	<juhb0h$i0$1@dough.gmane.org>
	<CAL3CFcVy=j2hyqWZwripmfzjMpQHdCrduG-61+9aTeyZe+pR=g@mail.gmail.com>
	<20120722195440.2cfc10a2@pitrou.net>
Message-ID: <CAL3CFcXW+WnQFvX6v2fW1ST__t9JCdJ-B-+-e6J44Gu7Wuoa8Q@mail.gmail.com>

I thought my proposition is minor change, but if it's too late for 3.3
? I'm ok.

On Sun, Jul 22, 2012 at 8:54 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sun, 22 Jul 2012 20:39:15 +0300
> Andrew Svetlov <andrew.svetlov at gmail.com> wrote:
>> Ok.
>> Sorry for my mistake ? there are really no commits for
>> http://bugs.python.org/issue15417
>> It's look important for me ? but you are release manager.
>> If you consider the patch as potentially dangerous ? I have to agree with you.
>> You are the master :)
>
> This is not because Georg is the master.  When a release is nearing we
> think it is important to avoid introducing potential regressions,
> except when fixing existing bugs.  That's why we have a feature freeze
> which extends to many kinds of "enhancements", including performance
> improvements: really, it is more of a "bugfix-only period".
>
> One could propose other mechanisms for release preparation, but in the
> meantime, it is important as a community that we all follow similar
> rules.
>
> Regards
>
> Antoine.
>
>
> --
> Software development and contracting: http://pro.pitrou.net
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov

From meadori at gmail.com  Mon Jul 23 18:27:44 2012
From: meadori at gmail.com (Meador Inge)
Date: Mon, 23 Jul 2012 11:27:44 -0500
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.2 -> default):
 MERGE: Better test for Issue #15402: Add a __sizeof__ method to
 struct.Struct
In-Reply-To: <3Wgnsd0h5RzNS2@mail.python.org>
References: <3Wgnsd0h5RzNS2@mail.python.org>
Message-ID: <CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>

On Mon, Jul 23, 2012 at 11:17 AM, jesus.cea <python-checkins at python.org> wrote:

> http://hg.python.org/cpython/rev/b9a3ed1b14b9
> changeset:   78260:b9a3ed1b14b9
> parent:      78257:03063e718f5f
> parent:      78259:1911e192af0d
> user:        Jesus Cea <jcea at jcea.es>
> date:        Mon Jul 23 18:16:18 2012 +0200
> summary:
>   MERGE: Better test for Issue #15402: Add a __sizeof__ method to struct.Struct
>
> files:
>   Doc/ACKS.txt            |  1 +
>   Lib/test/test_struct.py |  8 ++++----
>   2 files changed, 5 insertions(+), 4 deletions(-)

Jes?s,

Doc/ACKS.txt is *only* for acknowledging documentation contributions.
Serhiy is already in Misc/ACKS.  No need to add him to Doc/ACKS.txt.

As for the tests, I intentionally kept them the way that Serhiy contributed
them -- using >= instead of >.  I kept them this way because we also
discussed in issue14596 the prospect of optimizing the way repeat counts
are handled.  These tests would start failing if (when) that optimization
happens.

So, neither of these changes are really necessary.  Although, it wouldn't
hurt to have *additional* tests using the > relation.

>
> diff --git a/Doc/ACKS.txt b/Doc/ACKS.txt
> --- a/Doc/ACKS.txt
> +++ b/Doc/ACKS.txt
> @@ -205,6 +205,7 @@
>     * Anthony Starks
>     * Greg Stein
>     * Peter Stoehr
> +   * Serhiy Storchaka
>     * Mark Summerfield
>     * Reuben Sumner
>     * Kalle Svensson
> diff --git a/Lib/test/test_struct.py b/Lib/test/test_struct.py
> --- a/Lib/test/test_struct.py
> +++ b/Lib/test/test_struct.py
> @@ -575,12 +575,12 @@
>      def test_sizeof(self):
>          self.assertGreater(sys.getsizeof(struct.Struct('BHILfdspP')),
>                             sys.getsizeof(struct.Struct('B')))
> -        self.assertGreaterEqual(sys.getsizeof(struct.Struct('123B')),
> +        self.assertGreater(sys.getsizeof(struct.Struct('123B')),
>                                  sys.getsizeof(struct.Struct('B')))
> -        self.assertGreaterEqual(sys.getsizeof(struct.Struct('B' * 123)),
> +        self.assertGreater(sys.getsizeof(struct.Struct('B' * 1234)),
>                                  sys.getsizeof(struct.Struct('123B')))
> -        self.assertGreaterEqual(sys.getsizeof(struct.Struct('123xB')),
> -                                sys.getsizeof(struct.Struct('B')))
> +        self.assertGreater(sys.getsizeof(struct.Struct('1234B')),
> +                                sys.getsizeof(struct.Struct('123B')))
>
>  def test_main():
>      run_unittest(StructTest)
>
> --
> Repository URL: http://hg.python.org/cpython
>
> _______________________________________________
> Python-checkins mailing list
> Python-checkins at python.org
> http://mail.python.org/mailman/listinfo/python-checkins
>



-- 
# Meador

From brian at python.org  Mon Jul 23 18:34:57 2012
From: brian at python.org (Brian Curtin)
Date: Mon, 23 Jul 2012 11:34:57 -0500
Subject: [Python-Dev] venv scripts for fish and csh shells
In-Reply-To: <CAL3CFcXW+WnQFvX6v2fW1ST__t9JCdJ-B-+-e6J44Gu7Wuoa8Q@mail.gmail.com>
References: <CAL3CFcX2hpG4pSKu_BP8jC8d+zpq47-sABHfQCVS+GqFMC-FXQ@mail.gmail.com>
	<50083671.6050903@oddbird.net>
	<CAL3CFcWrvrRhZHb+f=cN6D8sHa47S_=EOeCTX_A8BuhcyJRW4w@mail.gmail.com>
	<juh5qj$q9n$1@dough.gmane.org>
	<CAL3CFcWXB3e=-68SQwvBkg=sLvzL+1mBtoFCOu204Wiawphayg@mail.gmail.com>
	<juhb0h$i0$1@dough.gmane.org>
	<CAL3CFcVy=j2hyqWZwripmfzjMpQHdCrduG-61+9aTeyZe+pR=g@mail.gmail.com>
	<20120722195440.2cfc10a2@pitrou.net>
	<CAL3CFcXW+WnQFvX6v2fW1ST__t9JCdJ-B-+-e6J44Gu7Wuoa8Q@mail.gmail.com>
Message-ID: <CAD+XWwqoY_kF9djKhGDqo+BvrJTtE-TbBYKjwVeydg0M+bcJVA@mail.gmail.com>

On Mon, Jul 23, 2012 at 11:24 AM, Andrew Svetlov
<andrew.svetlov at gmail.com> wrote:
> I thought my proposition is minor change, but if it's too late for 3.3
> ? I'm ok.

Very simply, the first beta is when feature freeze goes into effect.
This is a really common policy that has been in effect for a long time
in CPython and most projects.

From jcea at jcea.es  Mon Jul 23 18:38:30 2012
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 23 Jul 2012 18:38:30 +0200
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.2 -> default):
 MERGE: Better test for Issue #15402: Add a __sizeof__ method to
 struct.Struct
In-Reply-To: <CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
Message-ID: <500D7E06.4020606@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23/07/12 18:27, Meador Inge wrote:
> Doc/ACKS.txt is *only* for acknowledging documentation
> contributions. Serhiy is already in Misc/ACKS.  No need to add him
> to Doc/ACKS.txt.

Oh, I missed that. Thanks for the head up.

> As for the tests, I intentionally kept them the way that Serhiy
> contributed them -- using >= instead of >.  I kept them this way
> because we also discussed in issue14596 the prospect of optimizing
> the way repeat counts are handled.  These tests would start failing
> if (when) that optimization happens.

The problem is that if we do ">=", then an unpatched python
interpreter could pass the test too. So we are not actually testing
the feature.

If the repeat counters are going to be optimized, the obvious step
would be to upgrade the test to do something like "BHHIL" instead of
"123B". I would wait until this feature is implemented to update the test.

What do you think?.

> 
> So, neither of these changes are really necessary.  Although, it
> wouldn't hurt to have *additional* tests using the > relation.
> 
>> 
>> diff --git a/Doc/ACKS.txt b/Doc/ACKS.txt --- a/Doc/ACKS.txt +++
>> b/Doc/ACKS.txt @@ -205,6 +205,7 @@ * Anthony Starks * Greg Stein 
>> * Peter Stoehr +   * Serhiy Storchaka * Mark Summerfield * Reuben
>> Sumner * Kalle Svensson diff --git a/Lib/test/test_struct.py
>> b/Lib/test/test_struct.py --- a/Lib/test/test_struct.py +++
>> b/Lib/test/test_struct.py @@ -575,12 +575,12 @@ def
>> test_sizeof(self): 
>> self.assertGreater(sys.getsizeof(struct.Struct('BHILfdspP')), 
>> sys.getsizeof(struct.Struct('B'))) -
>> self.assertGreaterEqual(sys.getsizeof(struct.Struct('123B')), +
>> self.assertGreater(sys.getsizeof(struct.Struct('123B')), 
>> sys.getsizeof(struct.Struct('B'))) -
>> self.assertGreaterEqual(sys.getsizeof(struct.Struct('B' * 123)), 
>> +        self.assertGreater(sys.getsizeof(struct.Struct('B' *
>> 1234)), sys.getsizeof(struct.Struct('123B'))) -
>> self.assertGreaterEqual(sys.getsizeof(struct.Struct('123xB')), -
>> sys.getsizeof(struct.Struct('B'))) +
>> self.assertGreater(sys.getsizeof(struct.Struct('1234B')), +
>> sys.getsizeof(struct.Struct('123B')))
>> 
>> def test_main(): run_unittest(StructTest)
>> 
>> -- Repository URL: http://hg.python.org/cpython
>> 
>> _______________________________________________ Python-checkins
>> mailing list Python-checkins at python.org 
>> http://mail.python.org/mailman/listinfo/python-checkins
>> 
> 
> 
> 

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBUA1+Bplgi5GaxT1NAQJtSAQAkv5DyoQ1N1YdOH2QLHnFbOsvp/1aG0Vy
hHMlD6cu/L7Ub+gyWWo65v9Dp4sLahV+CYem1wL4Fzd2QyBNQdg+BNou9eqoDzGF
IJbY2HALwOwz1vgeBiamFOSvpyWya/hzXR9I7rkBqXdR9c2Njdl/ioZQNKETO05k
TRfd/BQas4k=
=TKFO
-----END PGP SIGNATURE-----

From solipsis at pitrou.net  Mon Jul 23 19:17:32 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 23 Jul 2012 19:17:32 +0200
Subject: [Python-Dev] Doc/ACKS and Misc/ACKS
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es>
Message-ID: <20120723191732.7e1093b6@pitrou.net>

On Mon, 23 Jul 2012 18:38:30 +0200
Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 23/07/12 18:27, Meador Inge wrote:
> > Doc/ACKS.txt is *only* for acknowledging documentation
> > contributions. Serhiy is already in Misc/ACKS.  No need to add him
> > to Doc/ACKS.txt.
> 
> Oh, I missed that. Thanks for the head up.

That said, we could probably merge Doc/ACKS and Misc/ACKS (*). There
doesn't seem to be any strong argument for separating doc contributions
from other contributions.

(*) I think perhaps ?ric already proposed it at some point

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From eliben at gmail.com  Mon Jul 23 19:30:00 2012
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 23 Jul 2012 20:30:00 +0300
Subject: [Python-Dev] Doc/ACKS and Misc/ACKS
In-Reply-To: <20120723191732.7e1093b6@pitrou.net>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es> <20120723191732.7e1093b6@pitrou.net>
Message-ID: <CAF-Rda_4v8bxsaoLmFXSjnnecFfke0K643Wov_cSqqRQ0GB_FQ@mail.gmail.com>

>> On 23/07/12 18:27, Meador Inge wrote:
>> > Doc/ACKS.txt is *only* for acknowledging documentation
>> > contributions. Serhiy is already in Misc/ACKS.  No need to add him
>> > to Doc/ACKS.txt.
>>
>> Oh, I missed that. Thanks for the head up.
>
> That said, we could probably merge Doc/ACKS and Misc/ACKS (*). There
> doesn't seem to be any strong argument for separating doc contributions
> from other contributions.
>
> (*) I think perhaps ?ric already proposed it at some point
>

+1

From meadori at gmail.com  Mon Jul 23 20:05:27 2012
From: meadori at gmail.com (Meador Inge)
Date: Mon, 23 Jul 2012 13:05:27 -0500
Subject: [Python-Dev] Doc/ACKS and Misc/ACKS
In-Reply-To: <20120723191732.7e1093b6@pitrou.net>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es> <20120723191732.7e1093b6@pitrou.net>
Message-ID: <CAK1Qooqa0kYWVbJJCLk38-JeaHSpWutG8J3wTdWa3oJzyjYOCA@mail.gmail.com>

On Mon, Jul 23, 2012 at 12:17 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 23 Jul 2012 18:38:30 +0200
> Jesus Cea <jcea at jcea.es> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 23/07/12 18:27, Meador Inge wrote:
>> > Doc/ACKS.txt is *only* for acknowledging documentation
>> > contributions. Serhiy is already in Misc/ACKS.  No need to add him
>> > to Doc/ACKS.txt.
>>
>> Oh, I missed that. Thanks for the head up.
>
> That said, we could probably merge Doc/ACKS and Misc/ACKS (*). There
> doesn't seem to be any strong argument for separating doc contributions
> from other contributions.
>
> (*) I think perhaps ?ric already proposed it at some point

+1

-- Meador

From meadori at gmail.com  Mon Jul 23 20:19:51 2012
From: meadori at gmail.com (Meador Inge)
Date: Mon, 23 Jul 2012 13:19:51 -0500
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.2 -> default):
 MERGE: Better test for Issue #15402: Add a __sizeof__ method to
 struct.Struct
In-Reply-To: <500D7E06.4020606@jcea.es>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es>
Message-ID: <CAK1QoorTQvxOMHWpw0-SMqi_+P+j5WtXjM00zRnzJ1Lh0vZ7sw@mail.gmail.com>

On Mon, Jul 23, 2012 at 11:38 AM, Jesus Cea <jcea at jcea.es> wrote:

>> As for the tests, I intentionally kept them the way that Serhiy
>> contributed them -- using >= instead of >.  I kept them this way
>> because we also discussed in issue14596 the prospect of optimizing
>> the way repeat counts are handled.  These tests would start failing
>> if (when) that optimization happens.
>
> The problem is that if we do ">=", then an unpatched python
> interpreter could pass the test too. So we are not actually testing
> the feature.

We are testing the feature because the first test looks like:

        self.assertGreater(sys.getsizeof(struct.Struct('BHILfdspP')),
                           sys.getsizeof(struct.Struct('B')))

The way things were written 'sys.getsizeof' would returns the same
answer regardless of the format string.

The remaining tests looked like:

        self.assertGreaterEqual(sys.getsizeof(struct.Struct('123B')),
                                sys.getsizeof(struct.Struct('B')))
        self.assertGreaterEqual(sys.getsizeof(struct.Struct('B' * 123)),
                                sys.getsizeof(struct.Struct('123B')))
        self.assertGreaterEqual(sys.getsizeof(struct.Struct('123xB')),
                                sys.getsizeof(struct.Struct('B')))

and while they didn't fail without the patch I felt they were still useful in
documenting that there is nothing that guarantees 'sizeof("123B") > sizeof("B")'
'sizeof("B" * 123) > sizeof("123B")', or 'sizeof("123xB") > sizeof("B")'.

> If the repeat counters are going to be optimized, the obvious step
> would be to upgrade the test to do something like "BHHIL" instead of
> "123B". I would wait until this feature is implemented to update the test.

That is what the first test basically already does :-)

> What do you think?.

It isn't that big of a deal.  We can just leave the tests as you changed them.
In the future it would probably be better to hash this stuff out in the tracker.
The patch was out for review for several days ...

Thanks,

-- Meador

From jcea at jcea.es  Mon Jul 23 20:28:04 2012
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 23 Jul 2012 20:28:04 +0200
Subject: [Python-Dev] Doc/ACKS and Misc/ACKS
In-Reply-To: <CAF-Rda_4v8bxsaoLmFXSjnnecFfke0K643Wov_cSqqRQ0GB_FQ@mail.gmail.com>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es> <20120723191732.7e1093b6@pitrou.net>
	<CAF-Rda_4v8bxsaoLmFXSjnnecFfke0K643Wov_cSqqRQ0GB_FQ@mail.gmail.com>
Message-ID: <500D97B4.90005@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23/07/12 19:30, Eli Bendersky wrote:
>> That said, we could probably merge Doc/ACKS and Misc/ACKS (*).
>> There doesn't seem to be any strong argument for separating doc
>> contributions from other contributions.
>> 
>> (*) I think perhaps ?ric already proposed it at some point
>> 
> 
> +1

+1 too.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBUA2XtJlgi5GaxT1NAQKiXQQAnOmVaALBmcAbEK7vImQ03m6tdh86ZyU/
VyRuoHVgHxsOn83h2VG+94zjNutedIMK9rq1hEhhPApJcXnYwftMpgEwlyj7vLFA
RUz8c02sKpoi/T8BGv2xVdW09yeMCUwzTDAuaS73NqscwcGplibaSPU5oKOjqetc
NhS0JdGQcr8=
=Ifpc
-----END PGP SIGNATURE-----

From brian at python.org  Mon Jul 23 20:32:02 2012
From: brian at python.org (Brian Curtin)
Date: Mon, 23 Jul 2012 13:32:02 -0500
Subject: [Python-Dev] Doc/ACKS and Misc/ACKS
In-Reply-To: <500D97B4.90005@jcea.es>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es> <20120723191732.7e1093b6@pitrou.net>
	<CAF-Rda_4v8bxsaoLmFXSjnnecFfke0K643Wov_cSqqRQ0GB_FQ@mail.gmail.com>
	<500D97B4.90005@jcea.es>
Message-ID: <CAD+XWwo6s42-0F440odRKowyt8j1tsuBz=MV_zpfJo4jKt3Z6w@mail.gmail.com>

On Mon, Jul 23, 2012 at 1:28 PM, Jesus Cea <jcea at jcea.es> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> On 23/07/12 19:30, Eli Bendersky wrote:
>>> That said, we could probably merge Doc/ACKS and Misc/ACKS (*).
>>> There doesn't seem to be any strong argument for separating doc
>>> contributions from other contributions.
>>>
>>> (*) I think perhaps ?ric already proposed it at some point
>>>
>>
>> +1
>
> +1 too.

Before everyone else on the list just writes a two character "+1"
response, can we just assume that if you don't speak up, you're ok
with it?

Especially when it's about an ack file...

From jcea at jcea.es  Mon Jul 23 20:35:53 2012
From: jcea at jcea.es (Jesus Cea)
Date: Mon, 23 Jul 2012 20:35:53 +0200
Subject: [Python-Dev] [Python-checkins] cpython (merge 3.2 -> default):
 MERGE: Better test for Issue #15402: Add a __sizeof__ method to
 struct.Struct
In-Reply-To: <CAK1QoorTQvxOMHWpw0-SMqi_+P+j5WtXjM00zRnzJ1Lh0vZ7sw@mail.gmail.com>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es>
	<CAK1QoorTQvxOMHWpw0-SMqi_+P+j5WtXjM00zRnzJ1Lh0vZ7sw@mail.gmail.com>
Message-ID: <500D9989.1050506@jcea.es>

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 23/07/12 20:19, Meador Inge wrote:
> self.assertGreaterEqual(sys.getsizeof(struct.Struct('123B')),
[...]
> and while they didn't fail without the patch I felt they were still
> useful in documenting that there is nothing that guarantees
> 'sizeof("123B") > sizeof("B")' 'sizeof("B" * 123) >
> sizeof("123B")', or 'sizeof("123xB") > sizeof("B")'.

No garantee, but I would find "interesting" that
"sizeof("1234B")==sizeof("B")".

If someday we implement some clever idea here (like the repeat counter
optimization discussed), we can simply change "sizeof("123B")" to
"sizeof("12345B")", or to "sizeof("BHBHBHBH"), etc.

> It isn't that big of a deal.  We can just leave the tests as you
> changed them. In the future it would probably be better to hash
> this stuff out in the tracker. The patch was out for review for
> several days ...

I agree. I should have raised this issue in the tracker. The fact is
that I was checking the patch carefully today, when we collided
mid-air working in the same issue both of us :-). I disliked the
proposed tests at that time.

Thanks for raising the issue. I will try to be more careful.

- -- 
Jesus Cea Avion                         _/_/      _/_/_/        _/_/_/
jcea at jcea.es - http://www.jcea.es/     _/_/    _/_/  _/_/    _/_/  _/_/
jabber / xmpp:jcea at jabber.org         _/_/    _/_/          _/_/_/_/_/
.                              _/_/  _/_/    _/_/          _/_/  _/_/
"Things are not so easy"      _/_/  _/_/    _/_/  _/_/    _/_/  _/_/
"My name is Dump, Core Dump"   _/_/_/        _/_/_/      _/_/  _/_/
"El amor es poner tu felicidad en la felicidad de otro" - Leibniz
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.10 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQCVAwUBUA2ZiZlgi5GaxT1NAQKwLQP/RqrP5qbvUtZ9MCuyTaT45l8+7QzqlJrx
Nyh2t98jWVxiso0FDyT2vw839lX0CwssuKyNPFkXzKicNiX4mW0rC1uxNajCk0kG
aVHKL6aC+65iJhA7+9uOW1yfRFyhqQbUc3aRlvg7UJMj4YEfB82Okk/2Wu0hgyiU
4Ti5VvFuOZ8=
=G/WJ
-----END PGP SIGNATURE-----

From storchaka at gmail.com  Mon Jul 23 20:49:56 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 23 Jul 2012 21:49:56 +0300
Subject: [Python-Dev] cpython (merge 3.2 -> default): MERGE: Better test
 for Issue #15402: Add a __sizeof__ method to struct.Struct
In-Reply-To: <500D7E06.4020606@jcea.es>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es>
Message-ID: <juk6co$h1v$1@dough.gmane.org>

On 23.07.12 19:38, Jesus Cea wrote:
> The problem is that if we do ">=", then an unpatched python
> interpreter could pass the test too. So we are not actually testing
> the feature.
>
> If the repeat counters are going to be optimized, the obvious step
> would be to upgrade the test to do something like "BHHIL" instead of
> "123B". I would wait until this feature is implemented to update the test.
>
> What do you think?.

I think any __sizeof__ tests are meaningless, because any result is 
implementation detail. For other implementations we get other values and 
other relations. Any of our a priori assumptions could be incorrect. 
Even my first assert may fail, if implementation uses a continuous array 
with overallocation.

I am now prepared a set of 14 __sizeof__ patches (should it be one issue 
or 14 individual issues in bugtracker?), and I feel a great desire not 
to write tests at all.


From storchaka at gmail.com  Mon Jul 23 20:59:40 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 23 Jul 2012 21:59:40 +0300
Subject: [Python-Dev] cpython (merge 3.2 -> default): MERGE: Better test
 for Issue #15402: Add a __sizeof__ method to struct.Struct
In-Reply-To: <CAK1QoorTQvxOMHWpw0-SMqi_+P+j5WtXjM00zRnzJ1Lh0vZ7sw@mail.gmail.com>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es>
	<CAK1QoorTQvxOMHWpw0-SMqi_+P+j5WtXjM00zRnzJ1Lh0vZ7sw@mail.gmail.com>
Message-ID: <juk6v7$ldr$1@dough.gmane.org>

On 23.07.12 21:19, Meador Inge wrote:
> The patch was out for review for several days ...

Actually for several months (in issue14596).


From fuzzyman at voidspace.org.uk  Mon Jul 23 21:46:44 2012
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Mon, 23 Jul 2012 20:46:44 +0100
Subject: [Python-Dev] cpython (merge 3.2 -> default): MERGE: Better test
	for Issue #15402: Add a __sizeof__ method to struct.Struct
In-Reply-To: <juk6co$h1v$1@dough.gmane.org>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es> <juk6co$h1v$1@dough.gmane.org>
Message-ID: <531E3DFF-5302-4045-BE27-1AD61D3713F0@voidspace.org.uk>


On 23 Jul 2012, at 19:49, Serhiy Storchaka wrote:

> On 23.07.12 19:38, Jesus Cea wrote:
>> The problem is that if we do ">=", then an unpatched python
>> interpreter could pass the test too. So we are not actually testing
>> the feature.
>> 
>> If the repeat counters are going to be optimized, the obvious step
>> would be to upgrade the test to do something like "BHHIL" instead of
>> "123B". I would wait until this feature is implemented to update the test.
>> 
>> What do you think?.
> 
> I think any __sizeof__ tests are meaningless, because any result is implementation detail. For other implementations we get other values and other relations. Any of our a priori assumptions could be incorrect. Even my first assert may fail, if implementation uses a continuous array with overallocation.
> 
> I am now prepared a set of 14 __sizeof__ patches (should it be one issue or 14 individual issues in bugtracker?), and I feel a great desire not to write tests at all.
> 


Without tests how can you have any confidence the patches are correct (or will continue to be correct)? Just because something is implementation dependent doesn't mean it should not be tested - instead the tests should be marked as cpython specific so that they aren't executed on other implementations.

All the best,

Michael

> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
> 


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From chris.jerdonek at gmail.com  Mon Jul 23 22:11:13 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Mon, 23 Jul 2012 13:11:13 -0700
Subject: [Python-Dev]  Doc/ACKS and Misc/ACKS
Message-ID: <CAOTb1we1L+1Z_n6AZn4auF7oyD67s6QHkM7+ifpZO9zM6Z0Dyg@mail.gmail.com>

> Date: Mon, 23 Jul 2012 19:17:32 +0200
> From: Antoine Pitrou <solipsis at pitrou.net>
> To: python-dev at python.org
> Subject: [Python-Dev] Doc/ACKS and Misc/ACKS
>
>> > Doc/ACKS.txt is *only* for acknowledging documentation
>> > contributions. Serhiy is already in Misc/ACKS.  No need to add him
>> > to Doc/ACKS.txt.
>>
> That said, we could probably merge Doc/ACKS and Misc/ACKS (*). There
> doesn't seem to be any strong argument for separating doc contributions
> from other contributions.
>
> (*) I think perhaps ?ric already proposed it at some point

I created an issue for this here:

http://bugs.python.org/issue15437

--Chris

From storchaka at gmail.com  Mon Jul 23 22:31:14 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 23 Jul 2012 23:31:14 +0300
Subject: [Python-Dev] cpython (merge 3.2 -> default): MERGE: Better test
 for Issue #15402: Add a __sizeof__ method to struct.Struct
In-Reply-To: <531E3DFF-5302-4045-BE27-1AD61D3713F0@voidspace.org.uk>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es> <juk6co$h1v$1@dough.gmane.org>
	<531E3DFF-5302-4045-BE27-1AD61D3713F0@voidspace.org.uk>
Message-ID: <jukcal$84i$1@dough.gmane.org>

On 23.07.12 22:46, Michael Foord wrote:
>> I am now prepared a set of 14 __sizeof__ patches (should it be one issue or 14 individual issues in bugtracker?), and I feel a great desire not to write tests at all.
>
> Without tests how can you have any confidence the patches are correct (or will continue to be correct)?

Tests may not provide this. If we add a new dynamically allocated data 
or change the size of the old and forgot to reflect this in __sizeof__, 
then non modified tests not notice this. __sizeof__ returns 42. What is 42?


From ethan at stoneleaf.us  Mon Jul 23 22:46:29 2012
From: ethan at stoneleaf.us (Ethan Furman)
Date: Mon, 23 Jul 2012 13:46:29 -0700
Subject: [Python-Dev] Doc/ACKS and Misc/ACKS
In-Reply-To: <CAD+XWwo6s42-0F440odRKowyt8j1tsuBz=MV_zpfJo4jKt3Z6w@mail.gmail.com>
References: <3Wgnsd0h5RzNS2@mail.python.org>	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>	<500D7E06.4020606@jcea.es>
	<20120723191732.7e1093b6@pitrou.net>	<CAF-Rda_4v8bxsaoLmFXSjnnecFfke0K643Wov_cSqqRQ0GB_FQ@mail.gmail.com>	<500D97B4.90005@jcea.es>
	<CAD+XWwo6s42-0F440odRKowyt8j1tsuBz=MV_zpfJo4jKt3Z6w@mail.gmail.com>
Message-ID: <500DB825.9030908@stoneleaf.us>

Brian Curtin wrote:
> On Mon, Jul 23, 2012 at 1:28 PM, Jesus Cea <jcea at jcea.es> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> On 23/07/12 19:30, Eli Bendersky wrote:
>>>> That said, we could probably merge Doc/ACKS and Misc/ACKS (*).
>>>> There doesn't seem to be any strong argument for separating doc
>>>> contributions from other contributions.
>>>>
>>>> (*) I think perhaps ?ric already proposed it at some point
>>>>
>>> +1
>> +1 too.
> 
> Before everyone else on the list just writes a two character "+1"
> response, can we just assume that if you don't speak up, you're ok
> with it?

You mean I don't get to be clever and say

+1 + too == +3

?

;)

~Ethan~

From martin at v.loewis.de  Mon Jul 23 23:44:47 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Mon, 23 Jul 2012 23:44:47 +0200
Subject: [Python-Dev] cpython (merge 3.2 -> default): MERGE: Better test
 for Issue #15402: Add a __sizeof__ method to struct.Struct
In-Reply-To: <jukcal$84i$1@dough.gmane.org>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es> <juk6co$h1v$1@dough.gmane.org>
	<531E3DFF-5302-4045-BE27-1AD61D3713F0@voidspace.org.uk>
	<jukcal$84i$1@dough.gmane.org>
Message-ID: <20120723234447.Horde.9EtVNtjz9kRQDcXPZFbhwBA@webmail.df.eu>


Zitat von Serhiy Storchaka <storchaka at gmail.com>:

> On 23.07.12 22:46, Michael Foord wrote:
>>> I am now prepared a set of 14 __sizeof__ patches (should it be one  
>>> issue or 14 individual issues in bugtracker?), and I feel a great  
>>> desire not to write tests at all.
>>
>> Without tests how can you have any confidence the patches are  
>> correct (or will continue to be correct)?
>
> Tests may not provide this. If we add a new dynamically allocated  
> data or change the size of the old and forgot to reflect this in  
> __sizeof__, then non modified tests not notice this. __sizeof__  
> returns 42. What is 42?

42 is most likely not the right answer, as the size should be a
multiple of four.

The point of writing tests for the sizeof code is that you detect
your own mistakes *in writing the test case*. The value of the
test case is not so much that it provides detection of regressions,
but that you notice the bug once you write the test case.

You may argue that then there is no value in committing the test
case, but there is also no harm in doing so.

In addition, it *may* catch regressions: if somebody changes
the layout of an object, the corresponding test most likely
fails. Maybe the fix is trivial and just in the test, but maybe
the layout changed fundamentally, and the author forgot to change
sizeof along with the structure change.

You wouldn't have to write 14 patches if implementing sizeof
correctly was easy.

Please see the existing extensive tests for inspiration on how
to add new ones.

Regards,
Martin



From techtonik at gmail.com  Mon Jul 23 23:50:52 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 24 Jul 2012 00:50:52 +0300
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <CAHVvXxSVPWS+mgP9ch0V9H2UHH0vFhfufuV5fx=BoASz3AntPw@mail.gmail.com>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
	<CAHVvXxSVPWS+mgP9ch0V9H2UHH0vFhfufuV5fx=BoASz3AntPw@mail.gmail.com>
Message-ID: <CAPkN8x+2j4bpsCrRtiPXAK-1GhB4maOocwGxvaHEFKwC=mcdoA@mail.gmail.com>

On Mon, Jul 23, 2012 at 12:21 AM, Oscar Benjamin
<oscar.j.benjamin at gmail.com> wrote:
>> On Sun, 22 Jul 2012 20:22:50 +0100, Oscar Benjamin
>> <oscar.benjamin at bristol.ac.uk> wrote:
>> > On 22 July 2012 14:08, R. David Murray <rdmurray at bitdance.com> wrote:
>> >
>> > > On Sun, 22 Jul 2012 11:21:38 +0300, anatoly techtonik
>> > > <techtonik at gmail.com>
>> > > wrote:
>> > > > http://docs.python.org/py3k/howto/pyporting.html#during-installation
>> > > >
>> > > > What's the point in making implicit Python 3 check here:
>> > > > try:  # Python 3
>> > > >   from distutils.command.build_py import build_py_2to3 as build_py
>> > > > except ImportError:  # Python 2
>> > > >   from distutils.command.build_py import build_py
>> > > >
>> > > > instead of explicit check like:
>> > > > import sys
>> > > > if sys.version_info[0] >= 3:
>> > > >     from distutils.command.build_py import build_py_2to3 as build_py
>> > >
>> > > It's called testing for the thing that actually matters, rather than
>> > > testing a constant with a much broader meaning.  Yes, in this case the
>> > > results are the same, but IMO it is better programming practice to
>> > > test
>> > > the thing that actually matters when you can.
>> >
>> >
>> > I recently changed a setup.py from try/ImportError to an explicit
>> > sys.version_info check. I'm not totally sure how to reproduce this but I
>> > had a problem where I was installing into a 2.x virtualenv and it was
>> > running 2to3 during install and subsequently failing to import the 3.x
>> > code
>> > (the problem didn't occur when using the same python that generated the
>> > virtualenv).
>> >
>> > I may be wrong but I imagined that sometimes build_py_2to3 is importable
>> > on
>> > 2.x, perhaps for cross-building or something. In any case 'testing the
>> > thing that matters' means testing what version of Python you are about
>> > to
>> > install into not whether the python version supports running 2to3.
>>
>> I'm not familiar with distutils, really, so you could be right about
>> what it is important to test.  I was commenting based on the code
>> snippet presented, which just deciding which "build" object to use.
>> If build_py_2to3 can be imported by python2 and subsequently screws up
>> the build, then yes the logic is incorrect.  But I have to defer to the
>> packaging people on that.  (I wish I had time to help with packaging
>> because it is important, but it doesn't seem like a sensible place for
>> me personally to spend my currently available time.)
>
>
> I'm not currently able to reproduce the problem on this machine. I think I
> was using pip/easy_install to install distribution X from PyPI that depended
> on distribution Y also on PyPI into an isolated 2.x virtualenv and found
> that distribution Y was converted with 2to3 when it was automatically
> installed. It could be a bug but I'm not confident enough with virtualenv to
> say that it wasn't just me messing things up somehow.
>
> Either way I still think that in this particular case a version check is the
> most explicit and appropriate thing to do. The author of a distribution that
> is distributed as Python 2.x code and installed on Python 3.x using 2to3
> knows precisely when they want 2to3 to run and when they don't so why not
> make that explicit?
>
> As an aside, I find the check slightly easier to read if it is written like:
>
> if sys.version_info >= (3, 0):
>     from distutils.build_py import build_py_2to3 as build_py

Yes. This looks better. If we reached consensus, I wonder how hard
is it to find somebody who have the rights and able to fix the
documentation:
http://docs.python.org/py3k/howto/pyporting.html#during-installation
--
anatoly t.

From merwok at netwok.org  Tue Jul 24 00:27:54 2012
From: merwok at netwok.org (=?UTF-8?B?w4lyaWMgQXJhdWpv?=)
Date: Mon, 23 Jul 2012 18:27:54 -0400
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <20120722195711.16863250054@webabinitio.net>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
Message-ID: <500DCFEA.8030503@netwok.org>

On 22/07/2012 15:57, R. David Murray wrote:
> I'm not familiar with distutils, really, so you could be right about
> what it is important to test.  I was commenting based on the code
> snippet presented, which just deciding which "build" object to use.
> If build_py_2to3 can be imported by python2 and subsequently screws up
> the build, then yes the logic is incorrect.

That can?t happen.  The *_2to3 classes (don?t forget build_scripts_2to3) 
only exist in 3.x and work with a version check or an import with 
fallback.  There is no cross-version-build at all in distutils.

Regards

From oscar.benjamin at bristol.ac.uk  Tue Jul 24 00:41:40 2012
From: oscar.benjamin at bristol.ac.uk (Oscar Benjamin)
Date: Mon, 23 Jul 2012 23:41:40 +0100
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <500DCFEA.8030503@netwok.org>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
	<500DCFEA.8030503@netwok.org>
Message-ID: <CAHVvXxThn34eSd5riY46x-YJ=YXQN=4U53RDgPzpoLHoyf0UBQ@mail.gmail.com>

On 23 July 2012 23:27, ?ric Araujo <merwok at netwok.org> wrote:

> On 22/07/2012 15:57, R. David Murray wrote:
>
>> I'm not familiar with distutils, really, so you could be right about
>> what it is important to test.  I was commenting based on the code
>> snippet presented, which just deciding which "build" object to use.
>> If build_py_2to3 can be imported by python2 and subsequently screws up
>> the build, then yes the logic is incorrect.
>>
>
> That can?t happen.  The *_2to3 classes (don?t forget build_scripts_2to3)
> only exist in 3.x and work with a version check or an import with fallback.
>  There is no cross-version-build at all in distutils.
>

I'm regretting the fact that I didn't keep notes of exactly what I was
doing and I can't reproduce this now but this did happen to me when using
one of pip/easy_install in a virtualenv. As I said earlier it may have been
a mistake on my part as I'm not confident with virtualenv.

At the time when I looked at the files in my pretty much empty virtualenv
the main things I could see where related to setuptools so I guessed that
some kind of setuptools monkey-patch had made build_py_2to3 importable and
changed the setup.py to an explicit version check which fixed the problem
in that environment.

Oscar.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120723/90896dab/attachment.html>

From techtonik at gmail.com  Tue Jul 24 06:44:47 2012
From: techtonik at gmail.com (anatoly techtonik)
Date: Tue, 24 Jul 2012 07:44:47 +0300
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <500DCFEA.8030503@netwok.org>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
	<500DCFEA.8030503@netwok.org>
Message-ID: <CAPkN8xKwSZc-uz+QS=MLAi9eiP4qPNWfMZ6LBG_=Y=vtTNbVqg@mail.gmail.com>

On Tue, Jul 24, 2012 at 1:27 AM, ?ric Araujo <merwok at netwok.org> wrote:
> On 22/07/2012 15:57, R. David Murray wrote:
>>
>> I'm not familiar with distutils, really, so you could be right about
>> what it is important to test.  I was commenting based on the code
>> snippet presented, which just deciding which "build" object to use.
>> If build_py_2to3 can be imported by python2 and subsequently screws up
>> the build, then yes the logic is incorrect.
>
>
> That can?t happen.  The *_2to3 classes (don?t forget build_scripts_2to3)
> only exist in 3.x and work with a version check or an import with fallback.
> There is no cross-version-build at all in distutils.

Python 3 check explicitly tells the reader that 2to3 should only be
used in Python 3. Otherwise everybody need to guess when this *_2to3
tools are triggered. As for me, I see no technical limitations why
*_2to3 can not be run by Python 2 (PyPy, RPython or whatever). Maybe I
don't have Python3, but want to build my package for Python 3. In
ideal world it is possible.
--
anatoly t.

From storchaka at gmail.com  Tue Jul 24 09:49:38 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 24 Jul 2012 10:49:38 +0300
Subject: [Python-Dev] cpython (merge 3.2 -> default): MERGE: Better test
 for Issue #15402: Add a __sizeof__ method to struct.Struct
In-Reply-To: <20120723234447.Horde.9EtVNtjz9kRQDcXPZFbhwBA@webmail.df.eu>
References: <3Wgnsd0h5RzNS2@mail.python.org>
	<CAK1Qoorf3Q9caStXd=q3-P5mRZQ7ZBDt9RZO5ed-iOuHmvi9cw@mail.gmail.com>
	<500D7E06.4020606@jcea.es> <juk6co$h1v$1@dough.gmane.org>
	<531E3DFF-5302-4045-BE27-1AD61D3713F0@voidspace.org.uk>
	<jukcal$84i$1@dough.gmane.org>
	<20120723234447.Horde.9EtVNtjz9kRQDcXPZFbhwBA@webmail.df.eu>
Message-ID: <julk2m$op2$1@dough.gmane.org>

On 24.07.12 00:44, martin at v.loewis.de wrote:
> 42 is most likely not the right answer, as the size should be a
> multiple of four.

 >>> ''.__sizeof__()
25


From tjreedy at udel.edu  Tue Jul 24 11:30:43 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 24 Jul 2012 05:30:43 -0400
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <CAPkN8xKwSZc-uz+QS=MLAi9eiP4qPNWfMZ6LBG_=Y=vtTNbVqg@mail.gmail.com>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
	<500DCFEA.8030503@netwok.org>
	<CAPkN8xKwSZc-uz+QS=MLAi9eiP4qPNWfMZ6LBG_=Y=vtTNbVqg@mail.gmail.com>
Message-ID: <julq0k$5q9$1@dough.gmane.org>

On 7/24/2012 12:44 AM, anatoly techtonik wrote:

> Python 3 check explicitly tells the reader that 2to3 should only be
> used in Python 3. Otherwise everybody need to guess when this *_2to3
> tools are triggered. As for me, I see no technical limitations why
> *_2to3 can not be run by Python 2 (PyPy, RPython or whatever). Maybe I
> don't have Python3, but want to build my package for Python 3. In
> ideal world it is possible.

This is not an ideal world and 2to3 is not good enough to convert files 
without further intervention and testing.

-- 
Terry Jan Reedy




From fuzzyman at voidspace.org.uk  Tue Jul 24 12:07:42 2012
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Tue, 24 Jul 2012 11:07:42 +0100
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <julq0k$5q9$1@dough.gmane.org>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
	<500DCFEA.8030503@netwok.org>
	<CAPkN8xKwSZc-uz+QS=MLAi9eiP4qPNWfMZ6LBG_=Y=vtTNbVqg@mail.gmail.com>
	<julq0k$5q9$1@dough.gmane.org>
Message-ID: <A1C47918-5215-4D63-93F7-1A8CD1E3848A@voidspace.org.uk>


On 24 Jul 2012, at 10:30, Terry Reedy wrote:

> On 7/24/2012 12:44 AM, anatoly techtonik wrote:
> 
>> Python 3 check explicitly tells the reader that 2to3 should only be
>> used in Python 3. Otherwise everybody need to guess when this *_2to3
>> tools are triggered. As for me, I see no technical limitations why
>> *_2to3 can not be run by Python 2 (PyPy, RPython or whatever). Maybe I
>> don't have Python3, but want to build my package for Python 3. In
>> ideal world it is possible.
> 
> This is not an ideal world and 2to3 is not good enough to convert files without further intervention and testing.


It is if you design your code *to be converted* by 2to3 and do regular testing of the result.

Michael

> 
> -- 
> Terry Jan Reedy
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/fuzzyman%40voidspace.org.uk
> 


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From oscar.j.benjamin at gmail.com  Tue Jul 24 12:14:49 2012
From: oscar.j.benjamin at gmail.com (Oscar Benjamin)
Date: Tue, 24 Jul 2012 11:14:49 +0100
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <julq0k$5q9$1@dough.gmane.org>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
	<500DCFEA.8030503@netwok.org>
	<CAPkN8xKwSZc-uz+QS=MLAi9eiP4qPNWfMZ6LBG_=Y=vtTNbVqg@mail.gmail.com>
	<julq0k$5q9$1@dough.gmane.org>
Message-ID: <CAHVvXxTyJFHC23aUbPk1Em7a7FvinMeCi4Wmy+vH8f9L_cbntw@mail.gmail.com>

On Jul 24, 2012 10:32 AM, "Terry Reedy" <tjreedy at udel.edu> wrote:
>
> On 7/24/2012 12:44 AM, anatoly techtonik wrote:
>
>> Python 3 check explicitly tells the reader that 2to3 should only be
>> used in Python 3. Otherwise everybody need to guess when this *_2to3
>> tools are triggered. As for me, I see no technical limitations why
>> *_2to3 can not be run by Python 2 (PyPy, RPython or whatever). Maybe I
>> don't have Python3, but want to build my package for Python 3. In
>> ideal world it is possible.
>
>
> This is not an ideal world and 2to3 is not good enough to convert files
without further intervention and testing.

Which is exactly why it's use should be explicit. To go back to the
original question is it not better to be explicit about the version check?
The try/importerror snippet in setup.py is often accompanied by a comment
that explains the fact that it is implicitly performing a version check
whereas I find the explicit version check self documenting.

I know python users often frown upon explicitly checking with if/else,
preferring the flexibility afforded by duck typing and the possibility of
monkey patching but I don't see any advantage in this case. As said above,
"it's called checking the thing that matters" which is definitely the
python version.

Oscar

>
> --
> Terry Jan Reedy
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/oscar.j.benjamin%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120724/177b5ee0/attachment.html>

From jeanpierreda at gmail.com  Tue Jul 24 12:52:16 2012
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Tue, 24 Jul 2012 06:52:16 -0400
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <A1C47918-5215-4D63-93F7-1A8CD1E3848A@voidspace.org.uk>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
	<500DCFEA.8030503@netwok.org>
	<CAPkN8xKwSZc-uz+QS=MLAi9eiP4qPNWfMZ6LBG_=Y=vtTNbVqg@mail.gmail.com>
	<julq0k$5q9$1@dough.gmane.org>
	<A1C47918-5215-4D63-93F7-1A8CD1E3848A@voidspace.org.uk>
Message-ID: <CABicbJKChaWhiRALD=LhMDqCKXEE4jXakhV8f0mOmJUoGUZgww@mail.gmail.com>

On Tue, Jul 24, 2012 at 6:07 AM, Michael Foord
<fuzzyman at voidspace.org.uk> wrote:
>> This is not an ideal world and 2to3 is not good enough to convert files without further intervention and testing.
>
> It is if you design your code *to be converted* by 2to3 and do regular testing of the result.

That's hardly without testing!

-- Devin

From fuzzyman at voidspace.org.uk  Tue Jul 24 13:09:24 2012
From: fuzzyman at voidspace.org.uk (Michael Foord)
Date: Tue, 24 Jul 2012 12:09:24 +0100
Subject: [Python-Dev] 2to3 porting HOWTO: setup.py question
In-Reply-To: <CABicbJKChaWhiRALD=LhMDqCKXEE4jXakhV8f0mOmJUoGUZgww@mail.gmail.com>
References: <CAPkN8x+Bc_dmfT6XUOOsxM0XRrqGkC2AaToC2Bvvo8=Ei2H_Dg@mail.gmail.com>
	<20120722130849.A2968250064@webabinitio.net>
	<CAHVvXxTVOBm5ToENS56Li=AG3uBWqxdAsqt_ORX6gJ+NjdZ9tw@mail.gmail.com>
	<CAHVvXxQ_C4Oxv_FbBFYENZ6jBh=p8ss47J0UPFfc0zTPqf5Q8Q@mail.gmail.com>
	<20120722195711.16863250054@webabinitio.net>
	<500DCFEA.8030503@netwok.org>
	<CAPkN8xKwSZc-uz+QS=MLAi9eiP4qPNWfMZ6LBG_=Y=vtTNbVqg@mail.gmail.com>
	<julq0k$5q9$1@dough.gmane.org>
	<A1C47918-5215-4D63-93F7-1A8CD1E3848A@voidspace.org.uk>
	<CABicbJKChaWhiRALD=LhMDqCKXEE4jXakhV8f0mOmJUoGUZgww@mail.gmail.com>
Message-ID: <4A1B3EA8-3FDE-4FAA-B097-1D2921C6E186@voidspace.org.uk>


On 24 Jul 2012, at 11:52, Devin Jeanpierre wrote:

> On Tue, Jul 24, 2012 at 6:07 AM, Michael Foord
> <fuzzyman at voidspace.org.uk> wrote:
>>> This is not an ideal world and 2to3 is not good enough to convert files without further intervention and testing.
>> 
>> It is if you design your code *to be converted* by 2to3 and do regular testing of the result.
> 
> That's hardly without testing!
> 

Well, *no* code can safely be created without testing. The OP did say intervention *and* testing...

Michael

> -- Devin
> 


--
http://www.voidspace.org.uk/


May you do good and not evil
May you find forgiveness for yourself and forgive others
May you share freely, never taking more than you give.
-- the sqlite blessing 
http://www.sqlite.org/different.html






From wimcolgate at gmail.com  Tue Jul 24 20:02:40 2012
From: wimcolgate at gmail.com (Wim Colgate)
Date: Tue, 24 Jul 2012 11:02:40 -0700
Subject: [Python-Dev] Building python 2.7.3 with Visual Studio 2012 (VS11.0)
Message-ID: <CAKUmoVhJLM2e2k=Fn0XFnkgNyid-B_2qekNMJwFQkOVtMVZmjQ@mail.gmail.com>

Please forgive me if this is not the prescribed method for asking this
question.

For various reasons, I would like to build python 2.7.3 from source
using the latest VS tools (VS11.0 is in RC -- which is close enough
for my purposes). I have seen the various sub-directories (VC6, VS7.1
and VS8.0) in the sources for specific VS tool chains. I have also
seen the patch for VS 10.0 (http://wiki.python.org/moin/VS2010).

If building with VS11.0, is there more than just applying the
equivalent VS10.0 patch to also include VS11? Are the other VS
sub-directories unneeded?

Regards,

Wim

From brian at python.org  Tue Jul 24 21:05:45 2012
From: brian at python.org (Brian Curtin)
Date: Tue, 24 Jul 2012 14:05:45 -0500
Subject: [Python-Dev] Building python 2.7.3 with Visual Studio 2012
	(VS11.0)
In-Reply-To: <CAKUmoVhJLM2e2k=Fn0XFnkgNyid-B_2qekNMJwFQkOVtMVZmjQ@mail.gmail.com>
References: <CAKUmoVhJLM2e2k=Fn0XFnkgNyid-B_2qekNMJwFQkOVtMVZmjQ@mail.gmail.com>
Message-ID: <CAD+XWwqd2fxrWbsPJxN9yp-PsCd3MkLtU6ja=crRM+oySpF6sg@mail.gmail.com>

On Tue, Jul 24, 2012 at 1:02 PM, Wim Colgate <wimcolgate at gmail.com> wrote:
> Please forgive me if this is not the prescribed method for asking this
> question.
>
> For various reasons, I would like to build python 2.7.3 from source
> using the latest VS tools (VS11.0 is in RC -- which is close enough
> for my purposes). I have seen the various sub-directories (VC6, VS7.1
> and VS8.0) in the sources for specific VS tool chains. I have also
> seen the patch for VS 10.0 (http://wiki.python.org/moin/VS2010).
>
> If building with VS11.0, is there more than just applying the
> equivalent VS10.0 patch to also include VS11? Are the other VS
> sub-directories unneeded?

If you can get it working on VS2010 first, VS2012 can read that
project file, but without converting it'll just run the 2010 compiler
and allow you to use the 2012 IDE.

Competing the actual port from 2010 to 2012 did not appear to be very
hard, but I didn't look to far into it.


You don't need the old VS sub-directories unless you are compiling
with those versions.

From martin at v.loewis.de  Tue Jul 24 21:20:46 2012
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Tue, 24 Jul 2012 21:20:46 +0200
Subject: [Python-Dev] Building python 2.7.3 with Visual Studio 2012
	(VS11.0)
In-Reply-To: <CAKUmoVhJLM2e2k=Fn0XFnkgNyid-B_2qekNMJwFQkOVtMVZmjQ@mail.gmail.com>
References: <CAKUmoVhJLM2e2k=Fn0XFnkgNyid-B_2qekNMJwFQkOVtMVZmjQ@mail.gmail.com>
Message-ID: <500EF58E.7030007@v.loewis.de>

> If building with VS11.0, is there more than just applying the
> equivalent VS10.0 patch to also include VS11?

I think nobody *really* knows at this point. Microsoft has a tradition
of breaking Python with every VS release, by making slight incompatible
changes in the C library. With VS 2012, on the one hand, they give
explicit consideration to VS 2010 and continued use of its tools; OTOH,
they also deliberately broke XP support in the CRT.

So you have to try for yourself. If Python passes the test suite (as
good as the official release), then the build was successful.

A different matter is dependent libraries (zlib, openssl, Tcl/Tk, ...).
You also have to build those with VS 2012 (if you want to use them),
each one likely posing its own challenges.

If you manage to succeed, don't forget to post your findings here.
Also if you fail.

Good luck,
Martin


From vincenzo.ampolo at gmail.com  Wed Jul 25 02:58:34 2012
From: vincenzo.ampolo at gmail.com (Vincenzo Ampolo)
Date: Tue, 24 Jul 2012 17:58:34 -0700
Subject: [Python-Dev] datetime nanosecond support
Message-ID: <500F44BA.1060802@gmail.com>

Hi all,

This is the first time I write to this list so thank you for considering
this message (if you will) :)

I know that this has been debated many times but until now there was no
a real use case. If you look on google about "python datetime
nanosecond" you can find more than 141k answer about that. They all say
that "you can't due to hardware imprecisions" or "you don't need it"
even if there is a good amount of people looking for this feature.

But let me explain my use case:

most OSes let users capture network packets (using tools like tcpdump or
wireshark) and store them using file formats like pcap or pcap-ng. These
formats include a timestamp for each of the captured packets, and this
timestamp usually has nanosecond precision. The reason is that on
gigabit and 10 gigabit networks the frame rate is so high that
microsecond precision is not enough to tell two frames apart.
pcap (and now pcap-ng) are extremely popular file formats, with millions
of files stored around the world. Support for nanoseconds in datetime
would make it possible to properly parse these files inside python to
compute precise statistics, for example network delays or round trip times.

More about this issue at http://bugs.python.org/issue15443

I completely agree with the YAGNI principle that seems to have driven
decisions in this area until now but It is the case to reconsider it
since this real use case has shown up?

Thank you for your attention

Best Regards,
-- 
Vincenzo Ampolo
http://vincenzo-ampolo.net
http://goshawknest.wordpress.com

From guido at python.org  Wed Jul 25 03:46:43 2012
From: guido at python.org (Guido van Rossum)
Date: Tue, 24 Jul 2012 18:46:43 -0700
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <500F44BA.1060802@gmail.com>
References: <500F44BA.1060802@gmail.com>
Message-ID: <CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>

On Tue, Jul 24, 2012 at 5:58 PM, Vincenzo Ampolo
<vincenzo.ampolo at gmail.com> wrote:
> Hi all,
>
> This is the first time I write to this list so thank you for considering
> this message (if you will) :)

You're welcome.

> I know that this has been debated many times but until now there was no
> a real use case. If you look on google about "python datetime
> nanosecond" you can find more than 141k answer about that. They all say
> that "you can't due to hardware imprecisions" or "you don't need it"
> even if there is a good amount of people looking for this feature.

Have you read PEP 410 and my rejection of it
(http://mail.python.org/pipermail/python-dev/2012-February/116837.html)?
Even though that's about using Decimal for timestamps, it could still
be considered related.

> But let me explain my use case:
>
> most OSes let users capture network packets (using tools like tcpdump or
> wireshark) and store them using file formats like pcap or pcap-ng. These
> formats include a timestamp for each of the captured packets, and this
> timestamp usually has nanosecond precision. The reason is that on
> gigabit and 10 gigabit networks the frame rate is so high that
> microsecond precision is not enough to tell two frames apart.
> pcap (and now pcap-ng) are extremely popular file formats, with millions
> of files stored around the world. Support for nanoseconds in datetime
> would make it possible to properly parse these files inside python to
> compute precise statistics, for example network delays or round trip times.
>
> More about this issue at http://bugs.python.org/issue15443
>
> I completely agree with the YAGNI principle that seems to have driven
> decisions in this area until now but It is the case to reconsider it
> since this real use case has shown up?

Not every use case deserves an API change. :-)

First you will have to show how you'd have to code this *without*
nanosecond precision in datetime and how tedious that is. (I expect
that representing the timestamp as a long integer expressing a posix
timestamp times a billion would be very reasonable.)

I didn't read the entire bug, but it mentioned something about storing
datetimes in databases. Do databases support nanosecond precision?

-- 
--Guido van Rossum (python.org/~guido)

From chris at kateandchris.net  Wed Jul 25 05:12:05 2012
From: chris at kateandchris.net (Chris Lambacher)
Date: Tue, 24 Jul 2012 23:12:05 -0400
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
Message-ID: <CAAXXHgJgUdY-T8OcRJWDG-Zhmg4BMSVS22eA=edq64myUX-4xA@mail.gmail.com>

On Tue, Jul 24, 2012 at 9:46 PM, Guido van Rossum <guido at python.org> wrote:

> I didn't read the entire bug, but it mentioned something about storing
> datetimes in databases. Do databases support nanosecond precision?
>

MS SQL Server 2008 R2 has the datetime2 data type which supports 100
nanosecond (.1 microsecond) precision:
http://msdn.microsoft.com/en-us/library/bb677335(v=sql.105)

PostgreSQL does 1 microsecond:
http://www.postgresql.org/docs/8.0/static/datatype-datetime.html

If I am reading this correctly the Oracle TIMESTAMP type allows up to 9
digits of fractional seconds (1 nanosecond):
http://docs.oracle.com/cd/B19306_01/server.102/b14195/sqlqr06.htm#r9c1-t3

-Chris

-- 
Christopher Lambacher
chris at kateandchris.net
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120724/7c2287a3/attachment.html>

From vincenzo.ampolo at gmail.com  Wed Jul 25 05:25:38 2012
From: vincenzo.ampolo at gmail.com (Vincenzo Ampolo)
Date: Tue, 24 Jul 2012 20:25:38 -0700
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
Message-ID: <500F6732.8050604@gmail.com>

On 07/24/2012 06:46 PM, Guido van Rossum wrote:
> 
> You're welcome.

Hi Guido,

I'm glad you spent your time reading my mail. I would have never
imagined that my mail could come to your attention.

> Have you read PEP 410 and my rejection of it
> (http://mail.python.org/pipermail/python-dev/2012-February/116837.html)?
> Even though that's about using Decimal for timestamps, it could still
> be considered related.

I've read it and point 5 is very like in this issue. You said:

"[...]
I see only one real use case for nanosecond precision: faithful
copying of the mtime/atime recorded by filesystems, in cases where the
filesystem (like e.g. ext4) records these times with nanosecond
precision. Even if such timestamps can't be trusted to be accurate,
converting them to floats and back loses precision, and verification
using tools not written in Python will flag the difference. But for
this specific use case a much simpler set of API changes will suffice;
only os.stat() and os.utime() need to change slightly (and variants of
os.stat() like os.fstat()).
[...]"

I think that's based on a wrong hypothesis: just one case -> let's
handle in a different way (modifying os.stat() and os.utime()).
I would say: It's not just one case, there are at lest other two
scenarios. One is like mine, parsing network packets, the other one is
in parsing stock trading data.
But in this case there is no os.stat() or os.utime() that can be
modified. I've to write my own class to deal with time and loose all the
power and flexibility that the datetime module adds to the python language.

> Not every use case deserves an API change. :-)
> 
> First you will have to show how you'd have to code this *without*
> nanosecond precision in datetime and how tedious that is. (I expect
> that representing the timestamp as a long integer expressing a posix
> timestamp times a billion would be very reasonable.)

Yeah that's exactly how we built our Time class to handle this, and we
wrote also a Duration class to represent timedelta.
The code we developed is 383 python lines long but is not comparable
with all the functionalities that the datetime module offers and it's
also really slow compared to native datetime module which is written in C.

As you may think using that approach in a web application is very
limiting since there is no strftime() in this custom class.

I cannot share the code right now since It's copyrighted by the company
I work for but I've asked permission to do so. I just need to wait
tomorrow morning (PDT time) so they approve my request. Looking at the
code you can see how tedious is to try to remake all the conversions
that are already implemented on the datetime module.
Just let me know if you actually want to have a look at the code.

> 
> I didn't read the entire bug, but it mentioned something about storing
> datetimes in databases. Do databases support nanosecond precision?
> 

Yeah. According to
http://wiki.ispirer.com/sqlways/postgresql/data-types/timestamp at least
Oracle support timestamps with nanoseconds accuracy, SQL server supports
100 nanosecond accuracy.
Since I use Postgresql personally the best way to accomplish it (also
suggested by the #postgresql on freenode) is to store the timestamp with
nanosecond (like 1343158283.880338907242) as bigint and let the ORM (so
a python ORM) do all the conversion job.
An yet again, having nanosecond support in datetime would make things
much more easy.

While writing this mail Chris Lambacher answered with more data about
nanosecond support on databases

Best Regards,
-- 
Vincenzo Ampolo
http://vincenzo-ampolo.net
http://goshawknest.wordpress.com

From guido at python.org  Wed Jul 25 05:47:54 2012
From: guido at python.org (Guido van Rossum)
Date: Tue, 24 Jul 2012 20:47:54 -0700
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <500F6732.8050604@gmail.com>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500F6732.8050604@gmail.com>
Message-ID: <CAP7+vJL=EoM6yUmoZXaSkowQPfqjHLAS9gj_ks=tyLOOZbzz8w@mail.gmail.com>

On Tue, Jul 24, 2012 at 8:25 PM, Vincenzo Ampolo
<vincenzo.ampolo at gmail.com> wrote:
> On 07/24/2012 06:46 PM, Guido van Rossum wrote:
>>
>> You're welcome.
>
> Hi Guido,
>
> I'm glad you spent your time reading my mail. I would have never
> imagined that my mail could come to your attention.

Stop brownnosing already. :-) If you'd followed python-dev you'd known
I read it.

>> Have you read PEP 410 and my rejection of it
>> (http://mail.python.org/pipermail/python-dev/2012-February/116837.html)?
>> Even though that's about using Decimal for timestamps, it could still
>> be considered related.
>
> I've read it and point 5 is very like in this issue. You said:
>
> "[...]
> I see only one real use case for nanosecond precision: faithful
> copying of the mtime/atime recorded by filesystems, in cases where the
> filesystem (like e.g. ext4) records these times with nanosecond
> precision. Even if such timestamps can't be trusted to be accurate,
> converting them to floats and back loses precision, and verification
> using tools not written in Python will flag the difference. But for
> this specific use case a much simpler set of API changes will suffice;
> only os.stat() and os.utime() need to change slightly (and variants of
> os.stat() like os.fstat()).
> [...]"
>
> I think that's based on a wrong hypothesis: just one case -> let's
> handle in a different way (modifying os.stat() and os.utime()).
> I would say: It's not just one case, there are at lest other two
> scenarios. One is like mine, parsing network packets, the other one is
> in parsing stock trading data.
> But in this case there is no os.stat() or os.utime() that can be
> modified. I've to write my own class to deal with time and loose all the
> power and flexibility that the datetime module adds to the python language.

Also, this use case is unlike the PEP 410 use case, because the
timestamps there use a numeric type, not datetime (and that was
separately argued).

>> Not every use case deserves an API change. :-)
>>
>> First you will have to show how you'd have to code this *without*
>> nanosecond precision in datetime and how tedious that is. (I expect
>> that representing the timestamp as a long integer expressing a posix
>> timestamp times a billion would be very reasonable.)
>
> Yeah that's exactly how we built our Time class to handle this, and we
> wrote also a Duration class to represent timedelta.
> The code we developed is 383 python lines long but is not comparable
> with all the functionalities that the datetime module offers and it's
> also really slow compared to native datetime module which is written in C.

So what functionality specifically do you require? You speak in
generalities but I need specifics.

> As you may think using that approach in a web application is very
> limiting since there is no strftime() in this custom class.

Apparently you didn't need it? :-) Web frameworks usually have their
own date/time formatting anyway.

> I cannot share the code right now since It's copyrighted by the company
> I work for but I've asked permission to do so. I just need to wait
> tomorrow morning (PDT time) so they approve my request. Looking at the
> code you can see how tedious is to try to remake all the conversions
> that are already implemented on the datetime module.
> Just let me know if you actually want to have a look at the code.

I believe you.

>> I didn't read the entire bug, but it mentioned something about storing
>> datetimes in databases. Do databases support nanosecond precision?
>>
>
> Yeah. According to
> http://wiki.ispirer.com/sqlways/postgresql/data-types/timestamp at least
> Oracle support timestamps with nanoseconds accuracy, SQL server supports
> 100 nanosecond accuracy.
> Since I use Postgresql personally the best way to accomplish it (also
> suggested by the #postgresql on freenode) is to store the timestamp with
> nanosecond (like 1343158283.880338907242) as bigint and let the ORM (so
> a python ORM) do all the conversion job.
> An yet again, having nanosecond support in datetime would make things
> much more easy.

How so, given that the database you use doesn't support it?

> While writing this mail Chris Lambacher answered with more data about
> nanosecond support on databases

Thanks, Chris.

TBH, I think that adding nanosecond precision to the datetime type is
not unthinkable. You'll have to come up with some clever backward
compatibility in the API though, and that will probably be a bit ugly
(you'd have a microsecond parameter with a range of 0-1000000 and a
nanosecond parameter with a range of 0-1000). Also the space it takes
in memory would probably increase (there's no room for an extra 10
bits in the carefully arranged 8-byte internal representation).

But let me be clear -- are you willing to help implement any of this?
You can't just order a feature, you know...

-- 
--Guido van Rossum (python.org/~guido)

From vincenzo.ampolo at gmail.com  Wed Jul 25 10:03:58 2012
From: vincenzo.ampolo at gmail.com (Vincenzo Ampolo)
Date: Wed, 25 Jul 2012 01:03:58 -0700
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <CAP7+vJL=EoM6yUmoZXaSkowQPfqjHLAS9gj_ks=tyLOOZbzz8w@mail.gmail.com>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500F6732.8050604@gmail.com>
	<CAP7+vJL=EoM6yUmoZXaSkowQPfqjHLAS9gj_ks=tyLOOZbzz8w@mail.gmail.com>
Message-ID: <500FA86E.3080405@gmail.com>

On 07/24/2012 08:47 PM, Guido van Rossum wrote:
>
> So what functionality specifically do you require? You speak in
> generalities but I need specifics.

The ability of thinking datetime.datetime as a flexible class that can
give you the representation you need when you need. To be more specific
think about this case:

User selects year, month, day, hour, minute, millisecond, nanosecond of
a network event from a browser the javascript code does a ajax call with
time of this format (variant of iso8601):
YYYY-MM-DDTHH:MM:SS.mmmmmmnnn (where nnn is the nanosecond representation).
The python server takes that string, converts to a datetime, does all
the math with its data and gives the output back using labeling data
with int(nano_datetime.strftime('MMSSmmmmmmnnn')) so I've a sequence
number that javascript can sort and handle easily.
It's this flexibility of conversion I'm talking about.

>
>> As you may think using that approach in a web application is very
>> limiting since there is no strftime() in this custom class.
>
> Apparently you didn't need it? :-) Web frameworks usually have their
> own date/time formatting anyway.

Which is usually derived from python's datetime, like in web2py (
http://web2py.com/books/default/chapter/29/6#Record-representation ) in
which timestamps are real python datetime objects and It's ORM
responsability to find the right representation of that data at database
level.
This lead, as you know, to one of the main advantages of any ORM:
abstract from the database layer and the SQL syntax.

The same applies for another well known framework, Django ( your
personal favorite :) ) in which DateTimeField (
https://docs.djangoproject.com/en/dev/ref/models/fields/#django.db.models.DateTimeField
) is a date and time
represented in Python by a datetime.datetime instance.

We didn't need to build a webapp yet. I've been hired for it :) So I'll
do very soon. Unluckly if datetime does not support nanoseconds, I
cannot blame any ORM for not supporting it natively.

>
>>> I didn't read the entire bug, but it mentioned something about storing
>>> datetimes in databases. Do databases support nanosecond precision?
>>>
>>
>> Yeah. According to
>> http://wiki.ispirer.com/sqlways/postgresql/data-types/timestamp at least
>> Oracle support timestamps with nanoseconds accuracy, SQL server supports
>> 100 nanosecond accuracy.
>> Since I use Postgresql personally the best way to accomplish it (also
>> suggested by the #postgresql on freenode) is to store the timestamp with
>> nanosecond (like 1343158283.880338907242) as bigint and let the ORM (so
>> a python ORM) do all the conversion job.
>> An yet again, having nanosecond support in datetime would make things
>> much more easy.
>
> How so, given that the database you use doesn't support it?

Wasn't the job of an ORM to abstract from actual database (either
relational or not relational) such that people who use the ORM do not
care about how data is represented behind it?

If yes It's job of the ORM to figure out what's the best representation
of a data on the given relational or non relational database.

>
> TBH, I think that adding nanosecond precision to the datetime type is
> not unthinkable. You'll have to come up with some clever backward
> compatibility in the API though, and that will probably be a bit ugly
> (you'd have a microsecond parameter with a range of 0-1000000 and a
> nanosecond parameter with a range of 0-1000). Also the space it takes
> in memory would probably increase (there's no room for an extra 10
> bits in the carefully arranged 8-byte internal representation).

Sure, that are all open issues but as soon as you are in favour of
adding nanosecond support we can start addressing them. I'm sure there
would be other people here that would like to participate to those
issues too.

>
> But let me be clear -- are you willing to help implement any of this?
> You can't just order a feature, you know...
>

Of course, as I wrote in my second message in the issue (
http://bugs.python.org/issue15443#msg166333 ) I'm ready and excited to
contribute to the python core if I can.

Best Regards,
-- 
Vincenzo Ampolo
http://vincenzo-ampolo.net
http://goshawknest.wordpress.com

From lists at cheimes.de  Wed Jul 25 11:24:14 2012
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 25 Jul 2012 11:24:14 +0200
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
Message-ID: <500FBB3E.60404@cheimes.de>

Am 25.07.2012 03:46, schrieb Guido van Rossum:
> First you will have to show how you'd have to code this *without*
> nanosecond precision in datetime and how tedious that is. (I expect
> that representing the timestamp as a long integer expressing a posix
> timestamp times a billion would be very reasonable.)

I'd vote for two separate numbers, the first similar to JDN (Julian Day
Number [1]), the second for nanoseconds per day. 3600 * 1000000 fit
nicely into an unsigned 32bit int. This approach has the neat benefit
that we'd get rid of the timestamp_t limitations and year 2038 bug at
once. IIRC datetime used to break for dates before 1970 on some system
because timestamp_t was unsigned. Python could finally support dates BC!

JDN is widely used by astronomers and historians to supports a wide
range of years as well as convert between calendar systems. Its day 0 is
January 1, 4713 BC in proleptic Julian calendar. The conversion between
Julian and Gregorian calendar makes JDN hard to use. Rata Die (Januar 1,
1 AD at midnight in proleptic Gregorian calender) sounds like a good idea.

People in need for a high precision timer should also consinder TAI [2]
instead of UTC as TAI doesn't have leap seconds. DJB's daemontools
specifies a tai64n log format [3] that is similar to your idea.

Christian

[1] http://en.wikipedia.org/wiki/Julian_Day_Number
[2] http://en.wikipedia.org/wiki/International_Atomic_Time
[3] http://cr.yp.to/daemontools/tai64n.html

From solipsis at pitrou.net  Wed Jul 25 13:48:53 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Wed, 25 Jul 2012 13:48:53 +0200
Subject: [Python-Dev] datetime nanosecond support
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de>
Message-ID: <20120725134853.3d98b87f@pitrou.net>

On Wed, 25 Jul 2012 11:24:14 +0200
Christian Heimes <lists at cheimes.de> wrote:
> Am 25.07.2012 03:46, schrieb Guido van Rossum:
> > First you will have to show how you'd have to code this *without*
> > nanosecond precision in datetime and how tedious that is. (I expect
> > that representing the timestamp as a long integer expressing a posix
> > timestamp times a billion would be very reasonable.)
> 
> I'd vote for two separate numbers, the first similar to JDN (Julian Day
> Number [1]), the second for nanoseconds per day. 3600 * 1000000 fit
> nicely into an unsigned 32bit int.

But 24 * 3600 * 1e9 doesn't. Perhaps I didn't understand your proposal.

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From ncoghlan at gmail.com  Wed Jul 25 14:11:13 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 25 Jul 2012 22:11:13 +1000
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <500FBB3E.60404@cheimes.de>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de>
Message-ID: <CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>

On Wed, Jul 25, 2012 at 7:24 PM, Christian Heimes <lists at cheimes.de> wrote:
> Am 25.07.2012 03:46, schrieb Guido van Rossum:
> I'd vote for two separate numbers, the first similar to JDN (Julian Day
> Number [1]), the second for nanoseconds per day. 3600 * 1000000 fit
> nicely into an unsigned 32bit int. This approach has the neat benefit
> that we'd get rid of the timestamp_t limitations and year 2038 bug at
> once. IIRC datetime used to break for dates before 1970 on some system
> because timestamp_t was unsigned. Python could finally support dates BC!

Alternatively, use Decimal as the internal representation (backed by
cdecimal if additional speed is needed).

However, rather than getting buried in the weeds right here:

1. For the reasons presented, I think it's worth attempting to define
a common API that is based on datetime, but is tailored towards high
precision time operations (at least using a different internal
representation, perhaps supporting TAI).

2. I don't think the stdlib is the right place to define the initial
version of this. It seems most sensible to first fork the pure Python
version of datetime, figure out the details to get that working as a
new distribution on PyPI, and then fork the C implementation to make
the PyPI version faster. Assuming it can be completed in time, the
revised API could then be brought back as a PEP (alternatively,
depending on the details of the proposal, the use case may be deemed
sufficiently rare that it is just kept as a specialist module on
PyPI).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From lists at cheimes.de  Wed Jul 25 15:59:03 2012
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 25 Jul 2012 15:59:03 +0200
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <20120725134853.3d98b87f@pitrou.net>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de> <20120725134853.3d98b87f@pitrou.net>
Message-ID: <juou37$kh1$1@dough.gmane.org>

Am 25.07.2012 13:48, schrieb Antoine Pitrou:
>> I'd vote for two separate numbers, the first similar to JDN (Julian Day
>> Number [1]), the second for nanoseconds per day. 3600 * 1000000 fit
>> nicely into an unsigned 32bit int.
> 
> But 24 * 3600 * 1e9 doesn't. Perhaps I didn't understand your proposal.

What the h... was I thinking? I confused nano with micro and forgot the
hours, how embarrassing. :(

days
----
32bit signed integer

numbers of days since Jan 1, 1 AD in proleptic Gregorian calendar (aka
modern civil calendar). That's Rata Die minus one day since it defines
Jan 1, 1 AD as day 1. This allows days between year 5.8 Mio in the past
and 5.8 Mio in the future ((1<<31) // 365.242 ~ 5879618).

nanoseconds
-----------
64bit signed or unsigned integer

more than enough for nanosecond granularity (47bits), we could easily
push it to pico seconds resolution (57bits) in the future.


Christian



From lists at cheimes.de  Wed Jul 25 16:11:25 2012
From: lists at cheimes.de (Christian Heimes)
Date: Wed, 25 Jul 2012 16:11:25 +0200
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de>
	<CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>
Message-ID: <500FFE8D.8040207@cheimes.de>

Am 25.07.2012 14:11, schrieb Nick Coghlan:
> 1. For the reasons presented, I think it's worth attempting to define
> a common API that is based on datetime, but is tailored towards high
> precision time operations (at least using a different internal
> representation, perhaps supporting TAI).

This is a great opportunity to implement two requests at once. Some
people want high precision datetime objects while others would like to
see a datetime implementation that works with dates BC.

>>> from datetime import datetime, timedelta
>>> epoch = datetime.datetime(1970, 1, 1)
>>> epoc - timedelta(days=1969*365.242)
datetime.datetime(1, 1, 1, 12, 2, 52, 799998)
>>> epoch - timedelta(days=1970*365.242)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: date value out of range


> 2. I don't think the stdlib is the right place to define the initial
> version of this. 

+1


From charlesc-lists-python-dev2 at pyropus.ca  Wed Jul 25 16:21:25 2012
From: charlesc-lists-python-dev2 at pyropus.ca (Charles Cazabon)
Date: Wed, 25 Jul 2012 08:21:25 -0600
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <juou37$kh1$1@dough.gmane.org>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de> <20120725134853.3d98b87f@pitrou.net>
	<juou37$kh1$1@dough.gmane.org>
Message-ID: <20120725142125.GA29390@pyropus.ca>

Christian Heimes <lists at cheimes.de> wrote:
> 
> days
> ----
> 32bit signed integer
> 
> numbers of days since Jan 1, 1 AD in proleptic Gregorian calendar (aka
> modern civil calendar). That's Rata Die minus one day since it defines
> Jan 1, 1 AD as day 1. This allows days between year 5.8 Mio in the past
> and 5.8 Mio in the future ((1<<31) // 365.242 ~ 5879618).
> 
> nanoseconds
> -----------
> 64bit signed or unsigned integer
> 
> more than enough for nanosecond granularity (47bits), we could easily
> push it to pico seconds resolution (57bits) in the future.

An alternate strategy might be to use tai64/tai64n/tai64na, which can
represent any time over the course of a few hundred billion years to
second/nanosecond/attosecond, respectively.  They're well-defined, and there's
a fair bit of software that can use or manipulate dates in these formats.
tai64 is defined similar to propleptic Gregorian in that it uses an idealized
24*60*60 second day, etc.

Charles
-- 
-----------------------------------------------------------------------
Charles Cazabon
GPL'ed software available at:               http://pyropus.ca/software/
-----------------------------------------------------------------------

From guido at python.org  Wed Jul 25 16:38:52 2012
From: guido at python.org (Guido van Rossum)
Date: Wed, 25 Jul 2012 07:38:52 -0700
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <500FFE8D.8040207@cheimes.de>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de>
	<CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>
	<500FFE8D.8040207@cheimes.de>
Message-ID: <CAP7+vJJf8FgvJeTG-naQdpwP2JZhf0R=5+TVsULuv8nakkkh1A@mail.gmail.com>

On Wed, Jul 25, 2012 at 7:11 AM, Christian Heimes <lists at cheimes.de> wrote:
> Am 25.07.2012 14:11, schrieb Nick Coghlan:
>> 1. For the reasons presented, I think it's worth attempting to define
>> a common API that is based on datetime, but is tailored towards high
>> precision time operations (at least using a different internal
>> representation, perhaps supporting TAI).
>
> This is a great opportunity to implement two requests at once. Some
> people want high precision datetime objects while others would like to
> see a datetime implementation that works with dates BC.

Beware, people requesting dates BC rarely know what they are asking
for. (E.g. Jesus wasn't born on 12/25/0001.) The calendrical
ambiguities are such that representing dates that far in the past is
better left to a specialized class. Read the original discussions
about the datetime type; it loses meaning for dates long ago even if
it can represent them, but the choice was made to ignore these and to
offer a uniform abstraction for 1 <= year <= 9999. TBH I'm more
worried about years >= 10000. :-)

>>>> from datetime import datetime, timedelta
>>>> epoch = datetime.datetime(1970, 1, 1)
>>>> epoc - timedelta(days=1969*365.242)
> datetime.datetime(1, 1, 1, 12, 2, 52, 799998)
>>>> epoch - timedelta(days=1970*365.242)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> OverflowError: date value out of range
>
>
>> 2. I don't think the stdlib is the right place to define the initial
>> version of this.
>
> +1

+1

-- 
--Guido van Rossum (python.org/~guido)

From martin at v.loewis.de  Wed Jul 25 19:02:09 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Wed, 25 Jul 2012 19:02:09 +0200
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de>
	<CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>
Message-ID: <20120725190209.Horde.z0JGVKGZi1VQECaRKA_jMDA@webmail.df.eu>

> 1. For the reasons presented, I think it's worth attempting to define
> a common API that is based on datetime, but is tailored towards high
> precision time operations (at least using a different internal
> representation, perhaps supporting TAI).

IIUC, the concern is about the DB-API, which does sort-of mandate that
the datetime module is used to represent SQL time types. So not basing
this off datetime isn't really an option.

Providing a subtype should work, though.

Regards,
Martin



From chris.jerdonek at gmail.com  Thu Jul 26 01:28:38 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Wed, 25 Jul 2012 16:28:38 -0700
Subject: [Python-Dev] feature freeze questions
Message-ID: <CAOTb1wf6FQFFuL-2VWq1B583mvrsr=fSxw-Hx8VtT6Bz=Xz2yA@mail.gmail.com>

I have a few questions about feature freeze:

(1) Is increasing test coverage acceptable during feature freeze, even
if it does not involve fixing a bug?

(2) When adding new tests (e.g. in the course of fixing a bug or
increasing test coverage), are we allowed to refactor other tests so
that supporting test code can be shared?  Or should the tests be added
in a less DRY fashion and refactored only after the branch goes back
to pre-alpha?

(3) What types of documentation changes are allowed during feature
freeze?  For example, are we only allowed to fix incorrect
information, or is it acceptable to improve or add to the information
about existing functionality?

Thanks,
--Chris

From benjamin at python.org  Thu Jul 26 01:29:45 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Wed, 25 Jul 2012 16:29:45 -0700
Subject: [Python-Dev] feature freeze questions
In-Reply-To: <CAOTb1wf6FQFFuL-2VWq1B583mvrsr=fSxw-Hx8VtT6Bz=Xz2yA@mail.gmail.com>
References: <CAOTb1wf6FQFFuL-2VWq1B583mvrsr=fSxw-Hx8VtT6Bz=Xz2yA@mail.gmail.com>
Message-ID: <CAPZV6o-WaQqFYOT12qsw5L00sg84Mj+pQhcsPREpOqXQ5mWvRQ@mail.gmail.com>

2012/7/25 Chris Jerdonek <chris.jerdonek at gmail.com>:
> I have a few questions about feature freeze:
>
> (1) Is increasing test coverage acceptable during feature freeze, even
> if it does not involve fixing a bug?
>
> (2) When adding new tests (e.g. in the course of fixing a bug or
> increasing test coverage), are we allowed to refactor other tests so
> that supporting test code can be shared?  Or should the tests be added
> in a less DRY fashion and refactored only after the branch goes back
> to pre-alpha?

You can do basically anything you want to tests just as long as you
don't make them less stable.

>
> (3) What types of documentation changes are allowed during feature
> freeze?  For example, are we only allowed to fix incorrect
> information, or is it acceptable to improve or add to the information
> about existing functionality?

All documentation changes are accepted.


-- 
Regards,
Benjamin

From ncoghlan at gmail.com  Thu Jul 26 03:03:05 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Jul 2012 11:03:05 +1000
Subject: [Python-Dev] [Python-checkins] cpython (3.2): Issue #15445:
 Updated logging configuration documentation to highlight
In-Reply-To: <3Wj4WF1jHjzPCy@mail.python.org>
References: <3Wj4WF1jHjzPCy@mail.python.org>
Message-ID: <CADiSq7cCeVjYKu06WEbiA7-tNZhPLQbexRQuFwfBGqX7JJ2=_A@mail.gmail.com>

On Thu, Jul 26, 2012 at 4:20 AM, vinay.sajip <python-checkins at python.org> wrote:
> +   .. note:: Because portions of the configuration are passed through
> +      :func:`eval`, use of this function may open its users to a security risk.
> +      While the function only binds to a socket on ``localhost``, and so does
> +      not accept connections from remote machines, there are scenarios where
> +      untrusted code could be run under the account of the process which calls
> +      :func:`listen`. Specifically, if the process calling :func:`listen` runs
> +      on a multi-user machine where users cannot trust each other, then a
> +      malicious user could arrange to run essentially arbitrary code in a
> +      victim user's process, simply by connecting to the victim's
> +      :func:`listen` socket and sending a configuration which runs whatever
> +      code the attacker wants to have executed in the victim's process. This is
> +      especially easy to do if the default port is used, but not hard even if a
> +      different port is used).

Looking at PEP 391, it appears it should be possible to replace the
current use of eval() with a combination of the much safer
ast.literal_eval() and the str.format attribute/item access
micro-language.

Worth exploring for 3.4 (http://bugs.python.org/issue15452), as it
would be better to actually try to close this attack vector rather
than just documenting that it exists.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From chris at simplistix.co.uk  Thu Jul 26 09:59:19 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 26 Jul 2012 08:59:19 +0100
Subject: [Python-Dev] Who's maintaining Sphinx nowadays?
Message-ID: <5010F8D7.6090502@simplistix.co.uk>

Hi All,

Sorry if this is considered slightly off topic, but Sphinx is the tool 
we use for the Python core docs so:

Who's looking after Sphinx nowadays? I've hit what I consider to be a bug:

https://groups.google.com/group/sphinx-dev/browse_thread/thread/197fc26ba570913d?hl=en

So I forked on bitbucket and wanted to work up a fix, but running the 
unit tests on my fresh fork was problematic:

https://groups.google.com/group/sphinx-dev/browse_thread/thread/1e1f45978a84577?hl=en

The sphinx-dev group seems devoid of sphinx developers currently, so 
thought I'd ask here to see if anyone could point me in the right 
direction ;-)

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
             - http://www.simplistix.co.uk

From eliben at gmail.com  Thu Jul 26 10:24:26 2012
From: eliben at gmail.com (Eli Bendersky)
Date: Thu, 26 Jul 2012 11:24:26 +0300
Subject: [Python-Dev] feature freeze questions
In-Reply-To: <CAPZV6o-WaQqFYOT12qsw5L00sg84Mj+pQhcsPREpOqXQ5mWvRQ@mail.gmail.com>
References: <CAOTb1wf6FQFFuL-2VWq1B583mvrsr=fSxw-Hx8VtT6Bz=Xz2yA@mail.gmail.com>
	<CAPZV6o-WaQqFYOT12qsw5L00sg84Mj+pQhcsPREpOqXQ5mWvRQ@mail.gmail.com>
Message-ID: <CAF-Rda_N7NYshKaw0R+r9UwVA+UUp2zpxcExSYBU16vj1R1wmA@mail.gmail.com>

>> I have a few questions about feature freeze:
>>
>> (1) Is increasing test coverage acceptable during feature freeze, even
>> if it does not involve fixing a bug?
>>
>> (2) When adding new tests (e.g. in the course of fixing a bug or
>> increasing test coverage), are we allowed to refactor other tests so
>> that supporting test code can be shared?  Or should the tests be added
>> in a less DRY fashion and refactored only after the branch goes back
>> to pre-alpha?
>
> You can do basically anything you want to tests just as long as you
> don't make them less stable.
>
>>
>> (3) What types of documentation changes are allowed during feature
>> freeze?  For example, are we only allowed to fix incorrect
>> information, or is it acceptable to improve or add to the information
>> about existing functionality?
>
> All documentation changes are accepted.

A follow-up question: during the RC stage, do documentation & test
changes require code review by an additional core dev?

This also leads me to think about minor (= bugfix) releases. For
example, current commits to the 3.2 branch will (presumably) go into
3.2.4 once it's out. Should these commits have the same restrictions
as commits to an upcoming release that's in RC stage? Common sense
would suggest they should (since any change can break existing
functionality and lead to regressions in a previously stable release),
but in practice I don't think this is followed. At least not to the
letter, with the devguide stating:

    You cannot skip the peer review during an RC, no matter how small!
Even if it is a simple copy-and-paste change, everything requires peer
review from a core developer.

Eli

From ncoghlan at gmail.com  Thu Jul 26 10:40:40 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 26 Jul 2012 18:40:40 +1000
Subject: [Python-Dev] feature freeze questions
In-Reply-To: <CAF-Rda_N7NYshKaw0R+r9UwVA+UUp2zpxcExSYBU16vj1R1wmA@mail.gmail.com>
References: <CAOTb1wf6FQFFuL-2VWq1B583mvrsr=fSxw-Hx8VtT6Bz=Xz2yA@mail.gmail.com>
	<CAPZV6o-WaQqFYOT12qsw5L00sg84Mj+pQhcsPREpOqXQ5mWvRQ@mail.gmail.com>
	<CAF-Rda_N7NYshKaw0R+r9UwVA+UUp2zpxcExSYBU16vj1R1wmA@mail.gmail.com>
Message-ID: <CADiSq7c9bEj=s+E5Pj6SEzk1z3R7PsYD3qTpxTZ9_je1gy8D_w@mail.gmail.com>

On Thu, Jul 26, 2012 at 6:24 PM, Eli Bendersky <eliben at gmail.com> wrote:
>     You cannot skip the peer review during an RC, no matter how small!
> Even if it is a simple copy-and-paste change, everything requires peer
> review from a core developer.

The extra restrictions during RC are more about not doing things that
might force the release to be postponed while a broken change gets
unwound. During normal maintenance, we have a bit more time to recover
based on buildbot failures or post-checkin review.

Another way of reading that paragraph is "once we hit RC, start
postponing everything to the next bugfix release, *unless* at least 2
core devs agree it really needs to be in the current release"

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From lists at cheimes.de  Thu Jul 26 15:00:52 2012
From: lists at cheimes.de (Christian Heimes)
Date: Thu, 26 Jul 2012 15:00:52 +0200
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <CAP7+vJJf8FgvJeTG-naQdpwP2JZhf0R=5+TVsULuv8nakkkh1A@mail.gmail.com>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de>
	<CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>
	<500FFE8D.8040207@cheimes.de>
	<CAP7+vJJf8FgvJeTG-naQdpwP2JZhf0R=5+TVsULuv8nakkkh1A@mail.gmail.com>
Message-ID: <50113F84.5060608@cheimes.de>

Am 25.07.2012 16:38, schrieb Guido van Rossum:
> Beware, people requesting dates BC rarely know what they are asking
> for. (E.g. Jesus wasn't born on 12/25/0001.) The calendrical
> ambiguities are such that representing dates that far in the past is
> better left to a specialized class. Read the original discussions
> about the datetime type; it loses meaning for dates long ago even if
> it can represent them, but the choice was made to ignore these and to
> offer a uniform abstraction for 1 <= year <= 9999. 

For starters. Calendars have more subtle edge cases, for example TAI has
a 10 second offset from UTC plus 15 leap seconds. Or the leap year
errors in Julian calendar that are handled differently in proleptic
Julian calendar which has unsystematic leap years between 45 BC and 4
AD. The rotation velocity of the Earth isn't constant, too. It's a major
PITB!

> TBH I'm more worried about years >= 10000. :-)

Why life in the past? The future is ... err the future! :)

Christian

From theller at ctypes.org  Thu Jul 26 19:56:12 2012
From: theller at ctypes.org (Thomas Heller)
Date: Thu, 26 Jul 2012 19:56:12 +0200
Subject: [Python-Dev] 2.7 releases
Message-ID: <jus0br$j6h$1@dough.gmane.org>

Will there be more 2.7 bugfix releases, and when the next one?

In other words; if I submit a patch and it is accepted, can I
expect that patch be committed also to the 2.7 branch?

Thanks,
Thomas

Been a long time that I've been here - but still using Python (2.7 now)
more and more...


From senthil at uthcode.com  Thu Jul 26 20:15:52 2012
From: senthil at uthcode.com (Senthil Kumaran)
Date: Thu, 26 Jul 2012 11:15:52 -0700
Subject: [Python-Dev] 2.7 releases
In-Reply-To: <jus0br$j6h$1@dough.gmane.org>
References: <jus0br$j6h$1@dough.gmane.org>
Message-ID: <CAPOVWORm5dRYK8dgDKK0JJYDqAAka115-+Hw3TKxOj3+A7JT9Q@mail.gmail.com>

On Thu, Jul 26, 2012 at 10:56 AM, Thomas Heller <theller at ctypes.org> wrote:

> Will there be more 2.7 bugfix releases, and when the next one?
>
> In other words; if I submit a patch and it is accepted, can I
> expect that patch be committed also to the 2.7 branch?
>
>
We are still back-porting bug fixes to 2.7 branch. Which means that yes, we
may definitely see a release.
Benjamin might be right person to answer the question on when.

Thanks,
Senthil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120726/4a1d99cb/attachment.html>

From martin at v.loewis.de  Thu Jul 26 20:16:00 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Thu, 26 Jul 2012 20:16:00 +0200
Subject: [Python-Dev] 2.7 releases
In-Reply-To: <jus0br$j6h$1@dough.gmane.org>
References: <jus0br$j6h$1@dough.gmane.org>
Message-ID: <20120726201600.Horde.lF6WBtjz9kRQEYlgtzJ3-KA@webmail.df.eu>

> Will there be more 2.7 bugfix releases

Yes.

> and when the next one?

That's up for Benjamin to decide. My view is that one bugfix
release every year is more than enough.

> In other words; if I submit a patch and it is accepted, can I
> expect that patch be committed also to the 2.7 branch?

Don't you have commit rights still?

Regards,
Martin



From theller at ctypes.org  Thu Jul 26 20:50:21 2012
From: theller at ctypes.org (Thomas Heller)
Date: Thu, 26 Jul 2012 20:50:21 +0200
Subject: [Python-Dev] 2.7 releases
In-Reply-To: <20120726201600.Horde.lF6WBtjz9kRQEYlgtzJ3-KA@webmail.df.eu>
References: <jus0br$j6h$1@dough.gmane.org>
	<20120726201600.Horde.lF6WBtjz9kRQEYlgtzJ3-KA@webmail.df.eu>
Message-ID: <jus3hc$bqk$1@dough.gmane.org>

Am 26.07.2012 20:16, schrieb martin at v.loewis.de:
>> Will there be more 2.7 bugfix releases
>
> Yes.
>
>> and when the next one?
>
> That's up for Benjamin to decide. My view is that one bugfix
> release every year is more than enough.

Ok.  I expect we will still be using 2.7 next year in my company.

>> In other words; if I submit a patch and it is accepted, can I
>> expect that patch be committed also to the 2.7 branch?
>
> Don't you have commit rights still?

I dont't know.  IIRC, I have asked for them to be retracted some years
ago.  Anyway, I do know nearly nothing about hg and don't have time to
learn it.

So, I have uploaded a patch and asked for review (since I'm not 1000%
sure that it is absolutely correct):

http://bugs.python.org/issue15459

Thomas



From tjreedy at udel.edu  Thu Jul 26 21:47:28 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 26 Jul 2012 15:47:28 -0400
Subject: [Python-Dev] 2.7 releases
In-Reply-To: <jus3hc$bqk$1@dough.gmane.org>
References: <jus0br$j6h$1@dough.gmane.org>
	<20120726201600.Horde.lF6WBtjz9kRQEYlgtzJ3-KA@webmail.df.eu>
	<jus3hc$bqk$1@dough.gmane.org>
Message-ID: <jus6t7$aje$1@dough.gmane.org>

On 7/26/2012 2:50 PM, Thomas Heller wrote:
> Am 26.07.2012 20:16, schrieb martin at v.loewis.de:

>> Don't you have commit rights still?
>
> I dont't know.

The tracker thinks you do. That is what the Python logo next to your 
name means.

>  Anyway, I do know nearly nothing about hg and don't have time to
> learn it.

I started from nothing too. TortoiseHG and its GUI Workbench makes the 
basics pretty easy.

-- 
Terry Jan Reedy


From g.brandl at gmx.net  Thu Jul 26 22:07:16 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Thu, 26 Jul 2012 22:07:16 +0200
Subject: [Python-Dev] Who's maintaining Sphinx nowadays?
In-Reply-To: <5010F8D7.6090502@simplistix.co.uk>
References: <5010F8D7.6090502@simplistix.co.uk>
Message-ID: <jus812$je2$1@dough.gmane.org>

Am 26.07.2012 09:59, schrieb Chris Withers:
> Hi All,
> 
> Sorry if this is considered slightly off topic, but Sphinx is the tool 
> we use for the Python core docs so:
> 
> Who's looking after Sphinx nowadays? I've hit what I consider to be a bug:
> 
> https://groups.google.com/group/sphinx-dev/browse_thread/thread/197fc26ba570913d?hl=en
> 
> So I forked on bitbucket and wanted to work up a fix, but running the 
> unit tests on my fresh fork was problematic:
> 
> https://groups.google.com/group/sphinx-dev/browse_thread/thread/1e1f45978a84577?hl=en
> 
> The sphinx-dev group seems devoid of sphinx developers currently, so 
> thought I'd ask here to see if anyone could point me in the right 
> direction ;-)

If you're patient enough, I'll take care of your problem eventually.

If not, and you're looking for a project to co-maintain, we can set
something up :)

Georg


From chris at simplistix.co.uk  Fri Jul 27 00:03:41 2012
From: chris at simplistix.co.uk (Chris Withers)
Date: Thu, 26 Jul 2012 23:03:41 +0100
Subject: [Python-Dev] Who's maintaining Sphinx nowadays?
In-Reply-To: <jus812$je2$1@dough.gmane.org>
References: <5010F8D7.6090502@simplistix.co.uk> <jus812$je2$1@dough.gmane.org>
Message-ID: <5011BEBD.8090908@simplistix.co.uk>

Hi Georg,

On 26/07/2012 21:07, Georg Brandl wrote:
> If you're patient enough, I'll take care of your problem eventually.
>
> If not, and you're looking for a project to co-maintain, we can set
> something up :)

I'm certainly interested in helping out where I can, Sphinx has been a 
fantastic tool...

It's mainly the test issues I'm worried about, the change in return code 
is the bug I was happy to work on.

Does Sphinx have any CI anywhere?

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk

From g.brandl at gmx.net  Fri Jul 27 07:14:24 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Fri, 27 Jul 2012 07:14:24 +0200
Subject: [Python-Dev] feature freeze questions
In-Reply-To: <CAF-Rda_N7NYshKaw0R+r9UwVA+UUp2zpxcExSYBU16vj1R1wmA@mail.gmail.com>
References: <CAOTb1wf6FQFFuL-2VWq1B583mvrsr=fSxw-Hx8VtT6Bz=Xz2yA@mail.gmail.com>
	<CAPZV6o-WaQqFYOT12qsw5L00sg84Mj+pQhcsPREpOqXQ5mWvRQ@mail.gmail.com>
	<CAF-Rda_N7NYshKaw0R+r9UwVA+UUp2zpxcExSYBU16vj1R1wmA@mail.gmail.com>
Message-ID: <jut82u$dh7$1@dough.gmane.org>

Am 26.07.2012 10:24, schrieb Eli Bendersky:
>>> I have a few questions about feature freeze:
>>>
>>> (1) Is increasing test coverage acceptable during feature freeze, even
>>> if it does not involve fixing a bug?
>>>
>>> (2) When adding new tests (e.g. in the course of fixing a bug or
>>> increasing test coverage), are we allowed to refactor other tests so
>>> that supporting test code can be shared?  Or should the tests be added
>>> in a less DRY fashion and refactored only after the branch goes back
>>> to pre-alpha?
>>
>> You can do basically anything you want to tests just as long as you
>> don't make them less stable.
>>
>>>
>>> (3) What types of documentation changes are allowed during feature
>>> freeze?  For example, are we only allowed to fix incorrect
>>> information, or is it acceptable to improve or add to the information
>>> about existing functionality?
>>
>> All documentation changes are accepted.
> 
> A follow-up question: during the RC stage, do documentation & test
> changes require code review by an additional core dev?

Test changes: yes, doc changes: only if you think it's necessary.

The thing about the docs is: we can fix them basically at any time,
since most people use the online version anyway, which is rebuilt
daily.

> This also leads me to think about minor (= bugfix) releases. For
> example, current commits to the 3.2 branch will (presumably) go into
> 3.2.4 once it's out. Should these commits have the same restrictions
> as commits to an upcoming release that's in RC stage? Common sense
> would suggest they should (since any change can break existing
> functionality and lead to regressions in a previously stable release),

But that can also happen for 3.2 -> 3.3.

> but in practice I don't think this is followed. At least not to the
> letter, with the devguide stating:
> 
>     You cannot skip the peer review during an RC, no matter how small!
> Even if it is a simple copy-and-paste change, everything requires peer
> review from a core developer.

That is only for RC phase, yes.  Reviews are always useful, of course
(calling them "peer review" sounds funny to me though), and should be
always done for larger patches -- which we are quite used to nowadays.

Georg


From status at bugs.python.org  Fri Jul 27 18:07:10 2012
From: status at bugs.python.org (Python tracker)
Date: Fri, 27 Jul 2012 18:07:10 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20120727160710.3F4291C87E@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2012-07-20 - 2012-07-27)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    3532 ( +4)
  closed 23716 (+58)
  total  27248 (+62)

Open issues with patches: 1514 


Issues opened (48)
==================

#14330: don't use host python, use host search paths for host compiler
http://bugs.python.org/issue14330  reopened by ezio.melotti

#15231: update PyPI upload doc to say --no-raw passed to rst2html.py
http://bugs.python.org/issue15231  reopened by eric.araujo

#15402: Correct __sizeof__ support for struct
http://bugs.python.org/issue15402  reopened by jcea

#15411: os.chmod() does not follow symlinks on Windows
http://bugs.python.org/issue15411  opened by ishimoto

#15412: Note in documentation for weakrefs
http://bugs.python.org/issue15412  opened by sbt

#15414: os.path.join behavior on Windows (ntpath.join) is unexpected a
http://bugs.python.org/issue15414  opened by r.david.murray

#15415: Add temp_dir() and change_cwd() to test.support
http://bugs.python.org/issue15415  opened by cjerdonek

#15417: Add support for csh and fish in venv activation scripts
http://bugs.python.org/issue15417  opened by asvetlov

#15418: 2to3 docs should mention setup.py fixes required to install co
http://bugs.python.org/issue15418  opened by techtonik

#15419: distutils: build_py_2to3 should use a separate build directory
http://bugs.python.org/issue15419  opened by techtonik

#15421: Calendar.itermonthdates OverflowError
http://bugs.python.org/issue15421  opened by ced

#15422: Get rid of PyCFunction_New macro
http://bugs.python.org/issue15422  opened by asvetlov

#15424: __sizeof__ of array should include size of items
http://bugs.python.org/issue15424  opened by Pankrat

#15425: Another strange Tracebacks with importlib
http://bugs.python.org/issue15425  opened by amaury.forgeotdarc

#15426: On a x86_64 Linux workstation, the build-from-source is borked
http://bugs.python.org/issue15426  opened by Shocked.Pancake

#15427: Describe use of args parameter of argparse.ArgumentParser.pars
http://bugs.python.org/issue15427  opened by bethard

#15428: add "Name Collision" section to argparse docs
http://bugs.python.org/issue15428  opened by bethard

#15430: Improve filecmp documentation
http://bugs.python.org/issue15430  opened by orsenthil

#15431: Cannot build importlib.h on Windows
http://bugs.python.org/issue15431  opened by amaury.forgeotdarc

#15432: gzip.py: mtime argument only since python 2.7
http://bugs.python.org/issue15432  opened by gomyhr

#15436: __sizeof__ is not documented
http://bugs.python.org/issue15436  opened by storchaka

#15437: Merge Doc/ACKS.txt names into Misc/ACKS
http://bugs.python.org/issue15437  opened by cjerdonek

#15438: Incredible issue in math.pow
http://bugs.python.org/issue15438  opened by andrea.bergamini

#15439: Include Misc/ACKS names into the documentation
http://bugs.python.org/issue15439  opened by cjerdonek

#15440: multiprocess fails to re-raise exception which has mandatory a
http://bugs.python.org/issue15440  opened by Bala.FA

#15441: test_posixpath fails on Japanese edition of Windows
http://bugs.python.org/issue15441  opened by ishimoto

#15442: Expand the list of default dirs filecmp.dircmp ignores
http://bugs.python.org/issue15442  opened by eli.bendersky

#15443: datetime module has no support for nanoseconds
http://bugs.python.org/issue15443  opened by goshawk

#15444: Incorrectly written contributor's names
http://bugs.python.org/issue15444  opened by storchaka

#15447: A file is not properly closed by webbrowser._invoke
http://bugs.python.org/issue15447  opened by anton.barkovsky

#15448: utimes() functions fail with ENOSYS even when detected by conf
http://bugs.python.org/issue15448  opened by richmose

#15450: Allow dircmp.subdirs to behave well under subclassing
http://bugs.python.org/issue15450  opened by cjerdonek

#15451: PATH is not honored in subprocess.Popen in win32
http://bugs.python.org/issue15451  opened by Grissiom.Gu

#15452: Eliminate the use of eval() in the logging config implementati
http://bugs.python.org/issue15452  opened by ncoghlan

#15453: ctype with packed bitfields does not match native compiler
http://bugs.python.org/issue15453  opened by Mike.Castle

#15454: Allow dircmp.report() output stream to be customized
http://bugs.python.org/issue15454  opened by cjerdonek

#15455: index entries not showing up in glossary
http://bugs.python.org/issue15455  opened by cjerdonek

#15457: consistent treatment of generator terminology
http://bugs.python.org/issue15457  opened by cjerdonek

#15459: ctypes Structures with subclassed types in the _fields_ list
http://bugs.python.org/issue15459  opened by theller

#15460: SQLite cursor.description is not DB-API compatible
http://bugs.python.org/issue15460  opened by CharlieX

#15463: test_faulthandler can fail if install path is too long
http://bugs.python.org/issue15463  opened by ned.deily

#15464: ssl: add set_msg_callback function
http://bugs.python.org/issue15464  opened by tweksteen

#15465: Improved documentation for C API version info
http://bugs.python.org/issue15465  opened by ncoghlan

#15466: Python/importlib.h is different on 32bit and 64bit
http://bugs.python.org/issue15466  opened by amaury.forgeotdarc

#15467: Updating __sizeof__ tests
http://bugs.python.org/issue15467  opened by storchaka

#15468: Edit docs to hide hashlib.md5()
http://bugs.python.org/issue15468  opened by dholth

#15469: Correct __sizeof__ support for deque
http://bugs.python.org/issue15469  opened by storchaka

#15470: Stuck/hang when reading ssl object
http://bugs.python.org/issue15470  opened by seamus.mckenna



Most recent 15 issues with no replies (15)
==========================================

#15470: Stuck/hang when reading ssl object
http://bugs.python.org/issue15470

#15469: Correct __sizeof__ support for deque
http://bugs.python.org/issue15469

#15468: Edit docs to hide hashlib.md5()
http://bugs.python.org/issue15468

#15467: Updating __sizeof__ tests
http://bugs.python.org/issue15467

#15465: Improved documentation for C API version info
http://bugs.python.org/issue15465

#15460: SQLite cursor.description is not DB-API compatible
http://bugs.python.org/issue15460

#15455: index entries not showing up in glossary
http://bugs.python.org/issue15455

#15448: utimes() functions fail with ENOSYS even when detected by conf
http://bugs.python.org/issue15448

#15442: Expand the list of default dirs filecmp.dircmp ignores
http://bugs.python.org/issue15442

#15440: multiprocess fails to re-raise exception which has mandatory a
http://bugs.python.org/issue15440

#15432: gzip.py: mtime argument only since python 2.7
http://bugs.python.org/issue15432

#15428: add "Name Collision" section to argparse docs
http://bugs.python.org/issue15428

#15427: Describe use of args parameter of argparse.ArgumentParser.pars
http://bugs.python.org/issue15427

#15425: Another strange Tracebacks with importlib
http://bugs.python.org/issue15425

#15422: Get rid of PyCFunction_New macro
http://bugs.python.org/issue15422



Most recent 15 issues waiting for review (15)
=============================================

#15469: Correct __sizeof__ support for deque
http://bugs.python.org/issue15469

#15467: Updating __sizeof__ tests
http://bugs.python.org/issue15467

#15464: ssl: add set_msg_callback function
http://bugs.python.org/issue15464

#15459: ctypes Structures with subclassed types in the _fields_ list
http://bugs.python.org/issue15459

#15457: consistent treatment of generator terminology
http://bugs.python.org/issue15457

#15450: Allow dircmp.subdirs to behave well under subclassing
http://bugs.python.org/issue15450

#15448: utimes() functions fail with ENOSYS even when detected by conf
http://bugs.python.org/issue15448

#15447: A file is not properly closed by webbrowser._invoke
http://bugs.python.org/issue15447

#15444: Incorrectly written contributor's names
http://bugs.python.org/issue15444

#15441: test_posixpath fails on Japanese edition of Windows
http://bugs.python.org/issue15441

#15437: Merge Doc/ACKS.txt names into Misc/ACKS
http://bugs.python.org/issue15437

#15431: Cannot build importlib.h on Windows
http://bugs.python.org/issue15431

#15430: Improve filecmp documentation
http://bugs.python.org/issue15430

#15426: On a x86_64 Linux workstation, the build-from-source is borked
http://bugs.python.org/issue15426

#15424: __sizeof__ of array should include size of items
http://bugs.python.org/issue15424



Top 10 most discussed issues (10)
=================================

#15402: Correct __sizeof__ support for struct
http://bugs.python.org/issue15402  35 msgs

#15438: Incredible issue in math.pow
http://bugs.python.org/issue15438  21 msgs

#15441: test_posixpath fails on Japanese edition of Windows
http://bugs.python.org/issue15441  17 msgs

#15437: Merge Doc/ACKS.txt names into Misc/ACKS
http://bugs.python.org/issue15437  15 msgs

#15439: Include Misc/ACKS names into the documentation
http://bugs.python.org/issue15439  14 msgs

#15443: datetime module has no support for nanoseconds
http://bugs.python.org/issue15443  14 msgs

#14578: importlib doesn't check Windows registry for paths
http://bugs.python.org/issue14578  13 msgs

#15381: Optimize BytesIO to so less reallocations when written, simila
http://bugs.python.org/issue15381  11 msgs

#15444: Incorrectly written contributor's names
http://bugs.python.org/issue15444  11 msgs

#1692335: Fix exception pickling: Move initial args assignment to BaseEx
http://bugs.python.org/issue1692335  10 msgs



Issues closed (56)
==================

#4221: inconsistent exception from int is confusing
http://bugs.python.org/issue4221  closed by flox

#7003: finish documentation of user local paths
http://bugs.python.org/issue7003  closed by orsenthil

#7163: IDLE suppresses sys.stdout.write() return value
http://bugs.python.org/issue7163  closed by loewis

#7578: Behavior of operations on a closed file object is not document
http://bugs.python.org/issue7578  closed by pitrou

#7710: Inconsistent Exception for int() conversion
http://bugs.python.org/issue7710  closed by flox

#7996: concurrency problem in regrtest -jX
http://bugs.python.org/issue7996  closed by rosslagerwall

#9610: buildbot: uncaptured python exception (smtpd), but no failure 
http://bugs.python.org/issue9610  closed by flox

#9692: UnicodeDecodeError in ElementTree.tostring()
http://bugs.python.org/issue9692  closed by flox

#9772: test_pep277 failure on AMD64 debian parallel buildbot
http://bugs.python.org/issue9772  closed by flox

#9773: test_tarfile fails because of inaccurate mtime on AMD64 debian
http://bugs.python.org/issue9773  closed by flox

#9783: _elementtree.c warnings under 64-bit Windows
http://bugs.python.org/issue9783  closed by flox

#10017: pprint.pprint raises TypeError on dictionaries with user-defin
http://bugs.python.org/issue10017  closed by flox

#12353: argparse cannot handle empty arguments
http://bugs.python.org/issue12353  closed by r.david.murray

#13818: argparse: -h listening required options under optional argumen
http://bugs.python.org/issue13818  closed by bethard

#13922: argparse handling multiple "--" in args improperly
http://bugs.python.org/issue13922  closed by r.david.murray

#14197: OS X framework builds do not create ABI-suffixed libpython3.x
http://bugs.python.org/issue14197  closed by ned.deily

#14392: type=bool doesn't raise error in argparse.Action
http://bugs.python.org/issue14392  closed by shima__shima

#14449: argparse optional arguments should follow getopt_long(3)
http://bugs.python.org/issue14449  closed by bethard

#14579: CVE-2012-2135: Vulnerability in the utf-16 decoder after error
http://bugs.python.org/issue14579  closed by pitrou

#14895: test_warnings.py EnvironmentVariableTests is a bad test
http://bugs.python.org/issue14895  closed by brett.cannon

#14998: pprint._safe_key is not always safe enough
http://bugs.python.org/issue14998  closed by flox

#15041: tkinter documentation: update  "see also" list
http://bugs.python.org/issue15041  closed by asvetlov

#15063: Source code links for JSON documentation
http://bugs.python.org/issue15063  closed by orsenthil

#15091: ImportError when package is symlinked on Unix
http://bugs.python.org/issue15091  closed by brett.cannon

#15093: ntpath.isdir returns False for directory symlinks
http://bugs.python.org/issue15093  closed by jason.coombs

#15094: Incorrectly placed #endif in _tkinter.c.
http://bugs.python.org/issue15094  closed by asvetlov

#15168: Move importlib.test to test.importlib
http://bugs.python.org/issue15168  closed by brett.cannon

#15184: Test failure in test_sysconfig_module
http://bugs.python.org/issue15184  closed by ned.deily

#15188: test_ldshared_value failure on OS X using python.org Pythons
http://bugs.python.org/issue15188  closed by ned.deily

#15195: test_distutils fails when ARCHFLAGS is set on a Mac
http://bugs.python.org/issue15195  closed by ned.deily

#15232: email.generator.Generator doesn't mangle "From " lines in MIME
http://bugs.python.org/issue15232  closed by r.david.murray

#15250: document that filecmp.dircmp comparisons are "shallow"
http://bugs.python.org/issue15250  closed by orsenthil

#15271: argparse: repeatedly specifying the same argument ignores the 
http://bugs.python.org/issue15271  closed by bethard

#15320: thread-safety issue in regrtest.main()
http://bugs.python.org/issue15320  closed by pitrou

#15327: Argparse: main arguments and subparser arguments indistinguish
http://bugs.python.org/issue15327  closed by bethard

#15336: Argparse required arguments incorrectly displayed as optional 
http://bugs.python.org/issue15336  closed by bethard

#15353: ld: library not found for -lpython3.3m because of wrong LDFLAG
http://bugs.python.org/issue15353  closed by ned.deily

#15406: Deprecation Warning fix on ctypes module test
http://bugs.python.org/issue15406  closed by meador.inge

#15407: CSV parser fails to iterate properly on 2.6.6
http://bugs.python.org/issue15407  closed by r.david.murray

#15409: Deprecation Warning fix on cookiejar module
http://bugs.python.org/issue15409  closed by meador.inge

#15410: urllib.request.Request.is_unverifiable deprecation documentati
http://bugs.python.org/issue15410  closed by meador.inge

#15413: os.times() disappeared under Windows
http://bugs.python.org/issue15413  closed by pitrou

#15416: 3 * [] gives a list of 3 cross-referenced lists, a[1]='blah' w
http://bugs.python.org/issue15416  closed by amaury.forgeotdarc

#15420: issue6964 reminder
http://bugs.python.org/issue15420  closed by loewis

#15423: Minidom tries to parse tags inside of <script> tag
http://bugs.python.org/issue15423  closed by r.david.murray

#15429: types.NoneType missing
http://bugs.python.org/issue15429  closed by r.david.murray

#15433: argparse usage line is wrong
http://bugs.python.org/issue15433  closed by bethard

#15434: __import__() problem in 3.3
http://bugs.python.org/issue15434  closed by brett.cannon

#15435: Strange behavior when AttributeError raise inside a property g
http://bugs.python.org/issue15435  closed by ncoghlan

#15445: Ability to do code injection via logging module configuration 
http://bugs.python.org/issue15445  closed by python-dev

#15446: Eval Recursion SIGSEGV
http://bugs.python.org/issue15446  closed by r.david.murray

#15449: Optional size argument for readline()
http://bugs.python.org/issue15449  closed by pitrou

#15456: Correct __sizeof__ support for code objects
http://bugs.python.org/issue15456  closed by loewis

#15458: Add python-config --configdir option
http://bugs.python.org/issue15458  closed by barry

#15461: os.stat() 's inappropriate behavior when dealing with a broken
http://bugs.python.org/issue15461  closed by r.david.murray

#15462: UTF8 BOM incorrectly prepended syslog messages when using rsys
http://bugs.python.org/issue15462  closed by r.david.murray

From meadori at gmail.com  Fri Jul 27 20:12:22 2012
From: meadori at gmail.com (Meador Inge)
Date: Fri, 27 Jul 2012 13:12:22 -0500
Subject: [Python-Dev] 2.7 releases
In-Reply-To: <jus3hc$bqk$1@dough.gmane.org>
References: <jus0br$j6h$1@dough.gmane.org>
	<20120726201600.Horde.lF6WBtjz9kRQEYlgtzJ3-KA@webmail.df.eu>
	<jus3hc$bqk$1@dough.gmane.org>
Message-ID: <CAK1QooreFSW_dtyDws5eA4S2E6tndCBNOMd_aJy3VY8ZT-83-A@mail.gmail.com>

On Thu, Jul 26, 2012 at 1:50 PM, Thomas Heller <theller at ctypes.org> wrote:

> So, I have uploaded a patch and asked for review (since I'm not 1000%
> sure that it is absolutely correct):
>
> http://bugs.python.org/issue15459

I'll take a look at this in the next few days.

-- Meador

From benjamin at python.org  Fri Jul 27 21:12:45 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 27 Jul 2012 12:12:45 -0700
Subject: [Python-Dev] 2.7 releases
In-Reply-To: <jus0br$j6h$1@dough.gmane.org>
References: <jus0br$j6h$1@dough.gmane.org>
Message-ID: <CAPZV6o8Layp66Xt27_kcELPa0wXgEQDVnn6xWEOknne=ewR6pQ@mail.gmail.com>

2012/7/26 Thomas Heller <theller at ctypes.org>:
> Will there be more 2.7 bugfix releases, and when the next one?

Probably late fall or early 2013.



-- 
Regards,
Benjamin

From python at mrabarnett.plus.com  Fri Jul 27 21:45:36 2012
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 27 Jul 2012 20:45:36 +0100
Subject: [Python-Dev] 2.7 releases
In-Reply-To: <CAPZV6o8Layp66Xt27_kcELPa0wXgEQDVnn6xWEOknne=ewR6pQ@mail.gmail.com>
References: <jus0br$j6h$1@dough.gmane.org>
	<CAPZV6o8Layp66Xt27_kcELPa0wXgEQDVnn6xWEOknne=ewR6pQ@mail.gmail.com>
Message-ID: <5012EFE0.7080101@mrabarnett.plus.com>

On 27/07/2012 20:12, Benjamin Peterson wrote:
> 2012/7/26 Thomas Heller <theller at ctypes.org>:
>> Will there be more 2.7 bugfix releases, and when the next one?
>
> Probably late fall or early 2013.
>
Or, for those in the southern hemisphere, late spring or early 2013.

From barry at python.org  Sat Jul 28 01:10:41 2012
From: barry at python.org (Barry Warsaw)
Date: Fri, 27 Jul 2012 19:10:41 -0400
Subject: [Python-Dev] Issue 15295: New import documentation
Message-ID: <20120727191041.4272e3f3@resist.wooz.org>

Back when Eric, Jason, and I sprinted on the PEP 420 implementation, I
threatened to actually document how the import machinery worked, updating and
reorganizing to include the new PEP 420 protocols.  Issue 15295 tracks this:

http://bugs.python.org/issue15295

I now have a first draft of the new import machinery documentation.  This
refactors the import statement docs, moving everything concerning the finding
and loading of modules into a separate language reference section.  The import
statement section then only describes the name binding operations that occur
after the module has been loaded.

I've also added a bunch of new glossary terms, and updated documentation and
code related to the importlib ABCs.

Everything lives in the features/pep-420 clone, in the importdocs branch:

http://hg.python.org/features/pep-420/shortlog/e6e77d8bb729

I really don't want to extract a patch for the tracker issue, since that will
be much less convenient to update with the inevitable deluge of corrections
I'm sure I'll get <wink>.  Please do review the updated documentation, and
send comments either here or in the tracker issue.  Given Python's release
schedule and my own summer schedule, review is fairly urgent.

OTOH, this is just documentation, so if Georg agrees, and the changes look
mostly sane, then I can commit it and we can fix up the nits later.  Either
way, I'd like to commit the docs by mid-next week at the latest (so after rc1
unfortunately).

Cheers,
-Barry
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120727/49446528/attachment.pgp>

From benjamin at python.org  Sat Jul 28 08:01:17 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Fri, 27 Jul 2012 23:01:17 -0700
Subject: [Python-Dev] Issue 15295: New import documentation
In-Reply-To: <20120727191041.4272e3f3@resist.wooz.org>
References: <20120727191041.4272e3f3@resist.wooz.org>
Message-ID: <CAPZV6o_rM7rujZR_9KkXxZudAr-zU1XC65ezkZbfU_JzLu4oOA@mail.gmail.com>

2012/7/27 Barry Warsaw <barry at python.org>:
> Back when Eric, Jason, and I sprinted on the PEP 420 implementation, I
> threatened to actually document how the import machinery worked, updating and
> reorganizing to include the new PEP 420 protocols.  Issue 15295 tracks this:
>
> http://bugs.python.org/issue15295
>
> I now have a first draft of the new import machinery documentation.

That's great! Thanks a lot for doing this. It is desparately in need
of documentation.



-- 
Regards,
Benjamin

From vincenzo.ampolo at gmail.com  Sat Jul 28 10:50:20 2012
From: vincenzo.ampolo at gmail.com (Vincenzo Ampolo)
Date: Sat, 28 Jul 2012 01:50:20 -0700
Subject: [Python-Dev] datetime nanosecond support. list of proposals
In-Reply-To: <50113F84.5060608@cheimes.de>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de>
	<CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>
	<500FFE8D.8040207@cheimes.de>
	<CAP7+vJJf8FgvJeTG-naQdpwP2JZhf0R=5+TVsULuv8nakkkh1A@mail.gmail.com>
	<50113F84.5060608@cheimes.de>
Message-ID: <5013A7CC.2050702@gmail.com>

Hi all again,

I've been quite busy these days and I collected all the suggestions
about the proposal. Here is a small summary:

Christian Heimes:
    two numbers:
        Julian Day Number (Rata Die) 32 bit signed integer
        nanoseconds in a day 64 bit signed or unsigned integer
    pro:
        fix 2038 bug
    cons:
        hard conversion to Gregorian calendar

Charles Cazabon:
    use tai64/tai64n/tai64na
    pro:
        well defined
        libraries available
    cons:
    	?

As ways to implement the idea there are these advices:

Nick Coghlan:
    define common API based on datetime
    maybe use TAI
    fork the pure Python version of datetime, then fork the C
implementation to make PyPI version faster, then make a PEP

Guido van Rossum:
    must do:
        clever backward compatibility
        use fewer bits as possible
    stdlib is not the right place for first implementation

Since I'm not a big expert of calendars and date representation I'm
going to study the Julian Calendar and the TAI representation.
As a first read from Wikipedia the TAI solution looks very promising.

For the ways to implement the idea I also believe that It's better to
have a pure python implementation (so It can be used on python 2.x and
distributed on PyPI) and then a Python 3.x C implementation and a PEP
submission.

I'm open to any other idea/advice. If there are other people that would
like to implement this with me, just write me a mail.

Thank you.

Best Regards,
-- 
Vincenzo Ampolo
http://vincenzo-ampolo.net
http://goshawknest.wordpress.com

From storchaka at gmail.com  Sat Jul 28 10:54:24 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 28 Jul 2012 11:54:24 +0300
Subject: [Python-Dev] __sizeof__ tests
Message-ID: <jv09c0$dgd$1@dough.gmane.org>

I have a set of patches about the correct __sizeof__ implementations 
(one has already been committed (issue #15456), 13 left, 2 of them 
published (#15469 for deque and #15475 for itertools)). The most tedious 
is to write tests for them, especially paranoid tests, on which Martin 
v. L?wis insists. For not to duplicate the supporting code, I suggest 
add (move from test_sys test) three utility functions to test.support 
module (issue #15467). Also from the use of the common code will benefit 
issues #15402 (for struct) and #15424 (for array). The testing for 
cElemetTree also will be easier.

Martin remarked that this adds new features to test.support and can only 
be done in 3.4. Question to release managers. Whether this is 
acceptable, if we do these functions private (with names that begin with 
an underscore)? Or would we have to copy the supporting code in every 
test file with __sizeof__ test?


From chris.jerdonek at gmail.com  Sat Jul 28 11:58:05 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sat, 28 Jul 2012 02:58:05 -0700
Subject: [Python-Dev]  __sizeof__ tests
Message-ID: <CAOTb1wctUCwAHYF3BCR-6ygSOZYUro3_P=Vu7=eVAkkz2yzzgA@mail.gmail.com>

On Sat Jul 28 10:54:24 CEST 2012, Serhiy Storchaka storchaka at gmail.com wrote:

> For not to duplicate the supporting code, I suggest
> add (move from test_sys test) three utility functions to test.support
> module (issue #15467). Also from the use of the common code will benefit
> issues #15402 (for struct) and #15424 (for array). The testing for
> cElemetTree also will be easier.
>
> Martin remarked that this adds new features to test.support and can only
> be done in 3.4. Question to release managers. Whether this is
> acceptable, if we do these functions private (with names that begin with
> an underscore)? Or would we have to copy the supporting code in every
> test file with __sizeof__ test?

This is nearly the same as the question I asked below about feature
freeze (or that I was trying to ask), to which I thought I received an
answer different from the one stated above:

"(2) When adding new tests (e.g. in the course of fixing a bug or
increasing test coverage), are we allowed to refactor other tests so
that supporting test code can be shared?  Or should the tests be added
in a less DRY fashion and refactored only after the branch goes back
to pre-alpha?"

http://mail.python.org/pipermail/python-dev/2012-July/121137.html

I don't feel test.support should be held to the same standard as the
public modules based in part on the documentation note below:

"Note: test.support is not a public module. It is documented here to
help Python developers write tests. The API of this module is subject
to change without backwards compatibility concerns between releases."

http://docs.python.org/dev/library/test.html#module-test.support

I think it benefits us to have a test support API that we can change
at any time.  This lets us improve the quality and maintainability of
our tests (e.g. by making them more DRY) with fewer obstacles.
Otherwise, we are creating barriers for ourselves to increasing code
coverage of the code that we really want to keep unchanged.

--Chris

From chris.jerdonek at gmail.com  Sat Jul 28 12:25:24 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sat, 28 Jul 2012 03:25:24 -0700
Subject: [Python-Dev] feature freeze questions
In-Reply-To: <CAPZV6o-WaQqFYOT12qsw5L00sg84Mj+pQhcsPREpOqXQ5mWvRQ@mail.gmail.com>
References: <CAOTb1wf6FQFFuL-2VWq1B583mvrsr=fSxw-Hx8VtT6Bz=Xz2yA@mail.gmail.com>
	<CAPZV6o-WaQqFYOT12qsw5L00sg84Mj+pQhcsPREpOqXQ5mWvRQ@mail.gmail.com>
Message-ID: <CAOTb1wdaOU3Y=uv=PiYw+18sHJfc2BX70spDYE0DMRkOkKFJeg@mail.gmail.com>

On Wed, Jul 25, 2012 at 4:29 PM, Benjamin Peterson <benjamin at python.org> wrote:
>> (2) When adding new tests (e.g. in the course of fixing a bug or
>> increasing test coverage), are we allowed to refactor other tests so
>> that supporting test code can be shared?  Or should the tests be added
>> in a less DRY fashion and refactored only after the branch goes back
>> to pre-alpha?
>
> You can do basically anything you want to tests just as long as you
> don't make them less stable.

Also, this response seems somewhat at odds to the response I got to a
patch that refactors part of regrest, fixes some bugs in it, and adds
its first unit tests (in particular, increasing code coverage):

http://bugs.python.org/issue15302

The regrtest module only affects tests (which I would think qualifies
it as "tests"), and this is borne out by the following documentation
note (which is essentially the same as the note for test.support):

"Note: The test package is meant for internal use by Python only. It
is documented for the benefit of the core developers of Python. Any
use of this package outside of Python?s standard library is
discouraged as code mentioned here can change or be removed without
notice between releases of Python.

"The test package contains all regression tests for Python as well as
the modules test.support and test.regrtest. test.support is used to
enhance your tests while test.regrtest drives the testing suite."

(from http://docs.python.org/dev/library/test.html#module-test )

--Chris

From ncoghlan at gmail.com  Sat Jul 28 15:05:46 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 28 Jul 2012 23:05:46 +1000
Subject: [Python-Dev] __sizeof__ tests
In-Reply-To: <jv09c0$dgd$1@dough.gmane.org>
References: <jv09c0$dgd$1@dough.gmane.org>
Message-ID: <CADiSq7d6n6GsK89TBabe15kyEQHCQ+7t4wNvtWT=r_51ew-ZoA@mail.gmail.com>

On Sat, Jul 28, 2012 at 6:54 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> I have a set of patches about the correct __sizeof__ implementations (one
> has already been committed (issue #15456), 13 left, 2 of them published
> (#15469 for deque and #15475 for itertools)). The most tedious is to write
> tests for them, especially paranoid tests, on which Martin v. L?wis insists.
> For not to duplicate the supporting code, I suggest add (move from test_sys
> test) three utility functions to test.support module (issue #15467). Also
> from the use of the common code will benefit issues #15402 (for struct) and
> #15424 (for array). The testing for cElemetTree also will be easier.
>
> Martin remarked that this adds new features to test.support and can only be
> done in 3.4. Question to release managers. Whether this is acceptable, if we
> do these functions private (with names that begin with an underscore)? Or
> would we have to copy the supporting code in every test file with __sizeof__
> test?

No, Martin is incorrect on this one. test.support is strictly our turf
and not covered by the normal standard library rules - the only reason
it is documented at all is because *we* want docs for it so people
writing tests don't reinvent too many wheels. Since we reserve the
right to delete stuff without notice, we can certainly add new stuff.

>From the top of the "test" package docs:
=====
The test package is meant for internal use by Python only. It is
documented for the benefit of the core developers of Python. Any use
of this package outside of Python?s standard library is discouraged as
code mentioned here can change or be removed without notice between
releases of Python.
=====

Cheers,
Nick.


-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Jul 28 15:21:04 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 28 Jul 2012 23:21:04 +1000
Subject: [Python-Dev] feature freeze questions
In-Reply-To: <CAOTb1wdaOU3Y=uv=PiYw+18sHJfc2BX70spDYE0DMRkOkKFJeg@mail.gmail.com>
References: <CAOTb1wf6FQFFuL-2VWq1B583mvrsr=fSxw-Hx8VtT6Bz=Xz2yA@mail.gmail.com>
	<CAPZV6o-WaQqFYOT12qsw5L00sg84Mj+pQhcsPREpOqXQ5mWvRQ@mail.gmail.com>
	<CAOTb1wdaOU3Y=uv=PiYw+18sHJfc2BX70spDYE0DMRkOkKFJeg@mail.gmail.com>
Message-ID: <CADiSq7cUjovwggrr971guTj-TRnVMMcKLrzYbkpDikHxarkU7A@mail.gmail.com>

On Sat, Jul 28, 2012 at 8:25 PM, Chris Jerdonek
<chris.jerdonek at gmail.com> wrote:
> Also, this response seems somewhat at odds to the response I got to a
> patch that refactors part of regrest, fixes some bugs in it, and adds
> its first unit tests (in particular, increasing code coverage):
>
> http://bugs.python.org/issue15302
>
> The regrtest module only affects tests (which I would think qualifies
> it as "tests"), and this is borne out by the following documentation
> note (which is essentially the same as the note for test.support):

Asking that #15302 be postponed is a risk judgement taking into
account "what do we have time to review before rc1?". Cleaning up the
argument parsing in regrtest is definitely a good thing to do, but
it's not an urgent fix.

A similar judgement call will be made for any test suite changes
between now and rc1: weighing up the risk of causing problems with the
release process vs having improved tests at the time of the release.
Anyone with commit rights gets to make that call, and we have three
options: yes, no, or ask the Release Manager (Georg for 3.3). Getting
to make that final call is one of the main responsibilities of the RM
(the default answer is usually "No", given that whichever of us asks
the question clearly has misgivings about the idea of including the
change).

Once we hit rc1, I expect Georg will fork the 3.3 maintenance branch,
and any code changes between then and the final release will only
happen specifically at his request (the docs will remain open to any
committer).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Sat Jul 28 15:37:10 2012
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 28 Jul 2012 16:37:10 +0300
Subject: [Python-Dev] __sizeof__ tests
In-Reply-To: <CAOTb1wctUCwAHYF3BCR-6ygSOZYUro3_P=Vu7=eVAkkz2yzzgA@mail.gmail.com>
References: <CAOTb1wctUCwAHYF3BCR-6ygSOZYUro3_P=Vu7=eVAkkz2yzzgA@mail.gmail.com>
Message-ID: <jv0pub$p0e$1@dough.gmane.org>

On 28.07.12 12:58, Chris Jerdonek wrote:
> This is nearly the same as the question I asked below about feature
> freeze (or that I was trying to ask), to which I thought I received an
> answer different from the one stated above:

Thank you, Chris.



From alex.gaynor at gmail.com  Sat Jul 28 19:05:01 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Sat, 28 Jul 2012 17:05:01 +0000 (UTC)
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
Message-ID: <loom.20120728T190358-272@post.gmane.org>

Hi all,

The discussion on PEP 0424 seems to have subsided (and I haven't gotten angry 
emails in a week!). So I would like to request a BDFL or BDFP pronouncement
on PEP 0424, text available here: http://hg.python.org/peps/file/tip/pep-0424.txt

Alex


From guido at python.org  Sat Jul 28 19:37:45 2012
From: guido at python.org (Guido van Rossum)
Date: Sat, 28 Jul 2012 10:37:45 -0700
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <loom.20120728T190358-272@post.gmane.org>
References: <loom.20120728T190358-272@post.gmane.org>
Message-ID: <CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>

I'll pronounce. Give me a few days.

On Sat, Jul 28, 2012 at 10:05 AM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> Hi all,
>
> The discussion on PEP 0424 seems to have subsided (and I haven't gotten angry
> emails in a week!). So I would like to request a BDFL or BDFP pronouncement
> on PEP 0424, text available here: http://hg.python.org/peps/file/tip/pep-0424.txt

-- 
--Guido van Rossum (python.org/~guido)

From chris.jerdonek at gmail.com  Sat Jul 28 19:50:11 2012
From: chris.jerdonek at gmail.com (Chris Jerdonek)
Date: Sat, 28 Jul 2012 10:50:11 -0700
Subject: [Python-Dev] feature freeze questions
In-Reply-To: <CADiSq7cUjovwggrr971guTj-TRnVMMcKLrzYbkpDikHxarkU7A@mail.gmail.com>
References: <CAOTb1wf6FQFFuL-2VWq1B583mvrsr=fSxw-Hx8VtT6Bz=Xz2yA@mail.gmail.com>
	<CAPZV6o-WaQqFYOT12qsw5L00sg84Mj+pQhcsPREpOqXQ5mWvRQ@mail.gmail.com>
	<CAOTb1wdaOU3Y=uv=PiYw+18sHJfc2BX70spDYE0DMRkOkKFJeg@mail.gmail.com>
	<CADiSq7cUjovwggrr971guTj-TRnVMMcKLrzYbkpDikHxarkU7A@mail.gmail.com>
Message-ID: <CAOTb1wdGBxZT8BAGMN4FVskgpiGFbsReniU7h=YQVSwAZKTr_Q@mail.gmail.com>

On Sat, Jul 28, 2012 at 6:21 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On Sat, Jul 28, 2012 at 8:25 PM, Chris Jerdonek
> <chris.jerdonek at gmail.com> wrote:
>> Also, this response seems somewhat at odds to the response I got to a
>> patch that refactors part of regrest, fixes some bugs in it, and adds
>> its first unit tests (in particular, increasing code coverage):
>>
>> http://bugs.python.org/issue15302
>
> Asking that #15302 be postponed is a risk judgement taking into
> account "what do we have time to review before rc1?". Cleaning up the
> argument parsing in regrtest is definitely a good thing to do, but
> it's not an urgent fix.
>
> A similar judgement call will be made for any test suite changes
> between now and rc1: weighing up the risk of causing problems with the
> release process vs having improved tests at the time of the release.

I guess I'm still not clear on the restrictions to making changes to
test-related code -- whether these restrictions apply just to the RC
phase, or also to the bugfix releases in maintenance branches.

In a previous e-mail, you said in response to the question:

> > Martin remarked that this adds new features to test.support and can
> > only be done in 3.4. Question to release managers. Whether this is
> > acceptable, if we do these functions private (with names that begin
> > with an underscore)? Or would we have to copy the supporting code in
> > every test file with __sizeof__ test?
>
> No, Martin is incorrect on this one. test.support is strictly our turf
> and not covered by the normal standard library rules - the only reason
> it is documented at all is because *we* want docs for it so people
> writing tests don't reinvent too many wheels. Since we reserve the
> right to delete stuff without notice, we can certainly add new stuff.

(from http://mail.python.org/pipermail/python-dev/2012-July/121162.html )

This e-mail is also relevant:

> The extra restrictions during RC are more about not doing things that
> might force the release to be postponed while a broken change gets
> unwound. During normal maintenance, we have a bit more time to recover
> based on buildbot failures or post-checkin review.

http://mail.python.org/pipermail/python-dev/2012-July/121142.html

But with regard to the regrtest changes in issue 15302, which do
fix three bugs (it was marked as a superseder to three issues), Georg commented:

> Please leave this for Python 3.4 -- it is not a bugfix.

It seems like the following would be a very common scenario:

We have bug fixes that could be made to the Python 3.3 maintenance
branch (for example), but the bug fixes are written in a way that
change or otherwise enhance existing test code like test.support or
regrtest.  If enhancements to our test support code can only go into
Python 3.4, then it seems like we are creating an incentive to write
lower quality test code (e.g. avoiding refactoring, and encouraging
practices like cut and paste) -- even for our next minor version.

This creates a trade-off that it doesn't seem like we need to have or
be imposing on ourselves: between having bugs fixed in an earlier
version, and writing higher quality tests for our future versions.

There is also a possible cumulative effect: whereby not committing
test-support enhancements to maintenance branches can also prevent
future bug fixes that depend on that support code from being able to
go in those maintenance branches.

--Chris

From guido at python.org  Sat Jul 28 20:35:24 2012
From: guido at python.org (Guido van Rossum)
Date: Sat, 28 Jul 2012 20:35:24 +0200
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
References: <loom.20120728T190358-272@post.gmane.org>
	<CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
Message-ID: <CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>

Looks good to me, so accepted.

But why isn't it visible on python.org/dev/peps/ yet?

On Sat, Jul 28, 2012 at 7:37 PM, Guido van Rossum <guido at python.org> wrote:

> I'll pronounce. Give me a few days.
>
> On Sat, Jul 28, 2012 at 10:05 AM, Alex Gaynor <alex.gaynor at gmail.com>
> wrote:
> > Hi all,
> >
> > The discussion on PEP 0424 seems to have subsided (and I haven't gotten
> angry
> > emails in a week!). So I would like to request a BDFL or BDFP
> pronouncement
> > on PEP 0424, text available here:
> http://hg.python.org/peps/file/tip/pep-0424.txt
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120728/8d5d03f8/attachment.html>

From benjamin at python.org  Sat Jul 28 20:38:07 2012
From: benjamin at python.org (Benjamin Peterson)
Date: Sat, 28 Jul 2012 11:38:07 -0700
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
References: <loom.20120728T190358-272@post.gmane.org>
	<CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
	<CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
Message-ID: <CAPZV6o9D3VK=e8gsrAEmUz0K29DFD7iQoE=qpqoaXg391bKgKg@mail.gmail.com>

2012/7/28 Guido van Rossum <guido at python.org>:
> Looks good to me, so accepted.
>
> But why isn't it visible on python.org/dev/peps/ yet?

The rebuilding hook is broken.



-- 
Regards,
Benjamin

From solipsis at pitrou.net  Sat Jul 28 20:51:49 2012
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sat, 28 Jul 2012 20:51:49 +0200
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
References: <loom.20120728T190358-272@post.gmane.org>
Message-ID: <20120728205149.38f4b328@pitrou.net>

On Sat, 28 Jul 2012 17:05:01 +0000 (UTC)
Alex Gaynor <alex.gaynor at gmail.com> wrote:

> Hi all,
> 
> The discussion on PEP 0424 seems to have subsided (and I haven't gotten angry 
> emails in a week!). So I would like to request a BDFL or BDFP pronouncement
> on PEP 0424, text available here: http://hg.python.org/peps/file/tip/pep-0424.txt

"It may raise a ``TypeError`` if a 32 specific instance cannot have its
length estimated."

-> what is the consumer supposed to do in this case? Propagate the
error, or use a default value?

Regards

Antoine.


-- 
Software development and contracting: http://pro.pitrou.net



From martin at v.loewis.de  Sat Jul 28 22:14:43 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Sat, 28 Jul 2012 22:14:43 +0200
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
References: <loom.20120728T190358-272@post.gmane.org>
	<CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
	<CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
Message-ID: <20120728221443.Horde.3_FXC8L8999QFEgzt2LUPlA@webmail.df.eu>

> But why isn't it visible on python.org/dev/peps/ yet?

Because the build process of the PEPs broke when hg.python.org became
its own system.

I'll try to look into that tomorrow, since unfortunately nobody else  
volunteered
to fix it.

Regards,
Martin


From francismb at email.de  Sun Jul 29 12:13:02 2012
From: francismb at email.de (francis)
Date: Sun, 29 Jul 2012 12:13:02 +0200
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <loom.20120728T190358-272@post.gmane.org>
References: <loom.20120728T190358-272@post.gmane.org>
Message-ID: <50150CAE.9050907@email.de>

Hi Alex,
just a small info (view pep-0424.txt @ 4491:7838a83c3ad1):

- Section Proposal:

[...] than the actual size >>> ofthe <<< container. [...]

- Section Rationale:
The first line is really long (seems to need a newline before 
``__length_hint__``)

Regards

francis



From martin at v.loewis.de  Sun Jul 29 13:18:24 2012
From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=)
Date: Sun, 29 Jul 2012 13:18:24 +0200
Subject: [Python-Dev] PEP updating
Message-ID: <50151C00.7030408@v.loewis.de>

I added a cron job on www.python.org that checks every 5 minutes for
a modified PEP. So automatic updating should work again.

Regards,
Martin

From ncoghlan at gmail.com  Sun Jul 29 13:30:43 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 29 Jul 2012 21:30:43 +1000
Subject: [Python-Dev] PEP updating
In-Reply-To: <50151C00.7030408@v.loewis.de>
References: <50151C00.7030408@v.loewis.de>
Message-ID: <CADiSq7fDa78oS-CoDo5oRhWE4=X3V6fyKX1LR8=8OUHmAoODDg@mail.gmail.com>

On Sun, Jul 29, 2012 at 9:18 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> I added a cron job on www.python.org that checks every 5 minutes for
> a modified PEP. So automatic updating should work again.

Thanks! Will be very helpful to see the updates as Georg tweaks the
release schedule in response to 3.3 release blocking issue reports.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From barry at python.org  Sun Jul 29 22:42:58 2012
From: barry at python.org (Barry Warsaw)
Date: Sun, 29 Jul 2012 16:42:58 -0400
Subject: [Python-Dev] python-checkins moderation
Message-ID: <20120729164258.2f1d01be@limelight.wooz.org>

I just freed up a bunch, but not all, of the messages to python-checkins being
held for moderator approval.  We could use some additional moderator
volunteers.  I'm not sure Fred is still moderating the list, and I suck at it.

Please email me directly if you'd like to help out, have experience moderating
lists, and at least vaguely <wink> understand what's on and off topic for
python-checkins.

Cheers,
-Barry

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120729/4e69a7f3/attachment.pgp>

From victor.stinner at gmail.com  Sun Jul 29 23:40:28 2012
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 29 Jul 2012 23:40:28 +0200
Subject: [Python-Dev] PEP updating
In-Reply-To: <50151C00.7030408@v.loewis.de>
References: <50151C00.7030408@v.loewis.de>
Message-ID: <CAMpsgwajSffs0=F06m2OY10xT_FZHLLN1G5Fs_GX9yQ3Nwwwog@mail.gmail.com>

Oh, thanks Martin!

Victor
Le 29 juil. 2012 13:20, Martin v. L?wis <martin at v.loewis.de> a ?crit :
>
> I added a cron job on www.python.org that checks every 5 minutes for
> a modified PEP. So automatic updating should work again.
>
> Regards,
> Martin
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
http://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120729/60d0b471/attachment.html>

From eliben at gmail.com  Sun Jul 29 23:45:01 2012
From: eliben at gmail.com (Eli Bendersky)
Date: Mon, 30 Jul 2012 00:45:01 +0300
Subject: [Python-Dev] PEP updating
In-Reply-To: <50151C00.7030408@v.loewis.de>
References: <50151C00.7030408@v.loewis.de>
Message-ID: <CAF-Rda8OVkXdW8gTsZR76WTVCnZgB846UAz9=A0bRyuQHY6F5A@mail.gmail.com>

On Sun, Jul 29, 2012 at 2:18 PM, "Martin v. L?wis" <martin at v.loewis.de> wrote:
> I added a cron job on www.python.org that checks every 5 minutes for
> a modified PEP. So automatic updating should work again.
>
> Regards,
> Martin
> _______

Martin, I have just received a huge pile of email saying this:

"
/home/docs/build-devguide: line 3: hg: command not found
"

>From "Cron Daemon root at python.org"

Could this be related to the recent fix you made?

Eli

From martin at v.loewis.de  Mon Jul 30 00:14:39 2012
From: martin at v.loewis.de (martin at v.loewis.de)
Date: Mon, 30 Jul 2012 00:14:39 +0200
Subject: [Python-Dev] PEP updating
In-Reply-To: <CAF-Rda8OVkXdW8gTsZR76WTVCnZgB846UAz9=A0bRyuQHY6F5A@mail.gmail.com>
References: <50151C00.7030408@v.loewis.de>
	<CAF-Rda8OVkXdW8gTsZR76WTVCnZgB846UAz9=A0bRyuQHY6F5A@mail.gmail.com>
Message-ID: <20120730001439.Horde.Wv9mdUlCcOxQFbXPiPeDsoA@webmail.df.eu>

> Martin, I have just received a huge pile of email saying this:
>
> "
> /home/docs/build-devguide: line 3: hg: command not found
> "
>
> From "Cron Daemon root at python.org"
>
> Could this be related to the recent fix you made?

Remotely related. This is really the cron job for updating the
devguide, not the one for updating the PEPs.

Howerver, it should have stopped now.

Regards,
Martin



From alex.gaynor at gmail.com  Mon Jul 30 03:11:52 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 30 Jul 2012 01:11:52 +0000 (UTC)
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
References: <loom.20120728T190358-272@post.gmane.org>
	<CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
	<CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
Message-ID: <loom.20120730T031103-33@post.gmane.org>

Guido van Rossum <guido <at> python.org> writes:

> 
> 
> Looks good to me, so accepted.But why isn't it visible on python.org/dev/peps/ 
yet?

I just realized the text in the python.org repo did not match what I had locally.
I've pushed what I intended to be the latest text, if everyone could take a new
look at that I would be very grateful. Sorry for the mixup.

Alex


From tjreedy at udel.edu  Mon Jul 30 03:15:51 2012
From: tjreedy at udel.edu (Terry Reedy)
Date: Sun, 29 Jul 2012 21:15:51 -0400
Subject: [Python-Dev] python-checkins moderation
In-Reply-To: <20120729164258.2f1d01be@limelight.wooz.org>
References: <20120729164258.2f1d01be@limelight.wooz.org>
Message-ID: <jv4n89$rqo$1@dough.gmane.org>

On 7/29/2012 4:42 PM, Barry Warsaw wrote:
> I just freed up a bunch, but not all, of the messages to python-checkins being
> held for moderator approval.  We could use some additional moderator
> volunteers.  I'm not sure Fred is still moderating the list, and I suck at it.

It seems that me that messages from people with commit rights should be 
automatic along with those from the repository.

> Please email me directly if you'd like to help out,

yes

> have experience moderating lists,

no

It seems that responses from non-committers are rare to non-existent. 
Will the list email a moderator when sucy arrive?

> and at least vaguely <wink> understand what's on and off topic for
> python-checkins.

yes

---
Terry Jan Reedy



From stuart at stuartbishop.net  Mon Jul 30 06:56:53 2012
From: stuart at stuartbishop.net (Stuart Bishop)
Date: Mon, 30 Jul 2012 11:56:53 +0700
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <500FFE8D.8040207@cheimes.de>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de>
	<CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>
	<500FFE8D.8040207@cheimes.de>
Message-ID: <CADmi=6PWrvTkeO4+BoAN2pBQ60NFUdqbH88XaP28VYDTtG+1qg@mail.gmail.com>

On Wed, Jul 25, 2012 at 9:11 PM, Christian Heimes <lists at cheimes.de> wrote:
> Am 25.07.2012 14:11, schrieb Nick Coghlan:
>> 1. For the reasons presented, I think it's worth attempting to define
>> a common API that is based on datetime, but is tailored towards high
>> precision time operations (at least using a different internal
>> representation, perhaps supporting TAI).
>
> This is a great opportunity to implement two requests at once. Some
> people want high precision datetime objects while others would like to
> see a datetime implementation that works with dates BC.

Back when the datetime library was being designed, a limiting factor
was size of the pickle (for reasons that I think no longer apply).
Support for the is_dst flag was never in there, only because the extra
single bit required overflowed the pickle size limit. If api changes
are being considered, please consider adding this bit back to match
the standard libraries. This will let me make the pytz timezone
library's API saner, and allow Python to do wallclock datetime
arithmetic without ambiguous cases.


-- 
Stuart Bishop <stuart at stuartbishop.net>
http://www.stuartbishop.net/

From guido at python.org  Mon Jul 30 07:10:32 2012
From: guido at python.org (Guido van Rossum)
Date: Mon, 30 Jul 2012 07:10:32 +0200
Subject: [Python-Dev] datetime nanosecond support
In-Reply-To: <CADmi=6PWrvTkeO4+BoAN2pBQ60NFUdqbH88XaP28VYDTtG+1qg@mail.gmail.com>
References: <500F44BA.1060802@gmail.com>
	<CAP7+vJ+YTK_dwxJjpdS_pCgGVTmwzmbbWcfbAvAFmrGjh+ZPxg@mail.gmail.com>
	<500FBB3E.60404@cheimes.de>
	<CADiSq7c9giSejxOiDuPXv5MO-qq+ZyQ51V2EoHKn+SmtgMsStg@mail.gmail.com>
	<500FFE8D.8040207@cheimes.de>
	<CADmi=6PWrvTkeO4+BoAN2pBQ60NFUdqbH88XaP28VYDTtG+1qg@mail.gmail.com>
Message-ID: <CAP7+vJ+rxrLyiLQe=gzBqEnzxPo43+3v=AMeiFdDBZdkTjiGVw@mail.gmail.com>

It can't be *that* easy. DST never is... For one, the dst flag is two
bits -- it can be on, off, or undefined. Also it should probably only
apply when a tzinfo is present. I don't recall that pickling ever was
the reason, but it could have been the size of the in-memory
representation (for reasons I can't fully recall we were *very*
concerned about the memory size of datetime objects). Anyway, I don't
want to be the limiting factor here, and I think this (as well as
nanoseconds) should be considered, but I don't want to have to
hand-hold the design. A PEP is in order.

--Guido

On Mon, Jul 30, 2012 at 6:56 AM, Stuart Bishop <stuart at stuartbishop.net> wrote:
> On Wed, Jul 25, 2012 at 9:11 PM, Christian Heimes <lists at cheimes.de> wrote:
>> Am 25.07.2012 14:11, schrieb Nick Coghlan:
>>> 1. For the reasons presented, I think it's worth attempting to define
>>> a common API that is based on datetime, but is tailored towards high
>>> precision time operations (at least using a different internal
>>> representation, perhaps supporting TAI).
>>
>> This is a great opportunity to implement two requests at once. Some
>> people want high precision datetime objects while others would like to
>> see a datetime implementation that works with dates BC.
>
> Back when the datetime library was being designed, a limiting factor
> was size of the pickle (for reasons that I think no longer apply).
> Support for the is_dst flag was never in there, only because the extra
> single bit required overflowed the pickle size limit. If api changes
> are being considered, please consider adding this bit back to match
> the standard libraries. This will let me make the pytz timezone
> library's API saner, and allow Python to do wallclock datetime
> arithmetic without ambiguous cases.
>
>
> --
> Stuart Bishop <stuart at stuartbishop.net>
> http://www.stuartbishop.net/



-- 
--Guido van Rossum (python.org/~guido)

From andrew.svetlov at gmail.com  Mon Jul 30 11:23:51 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Mon, 30 Jul 2012 12:23:51 +0300
Subject: [Python-Dev] [Python-checkins] peps: Update PEP-0424 with
	respect to feedback received.
In-Reply-To: <3WljQP1gshzPXX@mail.python.org>
References: <3WljQP1gshzPXX@mail.python.org>
Message-ID: <CAL3CFcVRocqPGD=Mtou9DufwWyS6BehJwX5oMnBF-FxE1PsSsg@mail.gmail.com>

On Mon, Jul 30, 2012 at 4:10 AM, alex.gaynor <python-checkins at python.org> wrote:

> +In addition, a new function ``operator.length`` hint is added

``operator.length`` or ``operator.length_hint``?

> +    def length_hint(obj, default):
Is there default value for `default` parameter?
For example:

def length_hint(obj, default=0):
    pass

or something like that?

-- 
Thanks,
Andrew Svetlov

From andrew.svetlov at gmail.com  Mon Jul 30 12:21:36 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Mon, 30 Jul 2012 13:21:36 +0300
Subject: [Python-Dev] Pushing issue #12288
Message-ID: <CAL3CFcVPRSuN4=BL1CVsJXCRMq=TF2zWqnYT1o3a+f5XLoPnZA@mail.gmail.com>

Is there ok to push http://bugs.python.org/issue12288 in 3.3?
It looks very easy and straightforward.
If now is too alte ? I'm ok.

Georg Brandl, what's your decision?

-- 
Thanks,
Andrew Svetlov

From peck at us.ibm.com  Mon Jul 30 12:22:08 2012
From: peck at us.ibm.com (Jon K Peck)
Date: Mon, 30 Jul 2012 04:22:08 -0600
Subject: [Python-Dev] AUTO: Jon K Peck is out of the office
Message-ID: <OF5DEEFC6F.FAF7E24C-ON87257A4B.0038F562-87257A4B.0038F562@us.ibm.com>



I am out of the office until 08/02/2012.

I will be out of the office from July 29 through Aug 2.  I expect to have
some access to email but will be delayed in responding.


Note: This is an automated response to your message  "Python-Dev Digest,
Vol 108, Issue 46" sent on 07/30/2012 4:00:03.

This is the only notification you will receive while this person is away.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120730/f83ecc8f/attachment.html>

From g.brandl at gmx.net  Mon Jul 30 18:19:44 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 30 Jul 2012 18:19:44 +0200
Subject: [Python-Dev] Pushing issue #12288
In-Reply-To: <CAL3CFcVPRSuN4=BL1CVsJXCRMq=TF2zWqnYT1o3a+f5XLoPnZA@mail.gmail.com>
References: <CAL3CFcVPRSuN4=BL1CVsJXCRMq=TF2zWqnYT1o3a+f5XLoPnZA@mail.gmail.com>
Message-ID: <jv6c6f$gn2$1@dough.gmane.org>

Am 30.07.2012 12:21, schrieb Andrew Svetlov:
> Is there ok to push http://bugs.python.org/issue12288 in 3.3?
> It looks very easy and straightforward.
> If now is too alte ? I'm ok.
> 
> Georg Brandl, what's your decision?

Sure, go ahead.  Nobody (I hope) said that every tiny bugfix needs
my approval.

Georg


From guido at python.org  Mon Jul 30 18:51:09 2012
From: guido at python.org (Guido van Rossum)
Date: Mon, 30 Jul 2012 09:51:09 -0700
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <loom.20120730T031103-33@post.gmane.org>
References: <loom.20120728T190358-272@post.gmane.org>
	<CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
	<CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
	<loom.20120730T031103-33@post.gmane.org>
Message-ID: <CAP7+vJJWOmLR=o68oKUWGsLPrf1V8-eDAcPv+jz-co_P7_Lj7Q@mail.gmail.com>

On Sun, Jul 29, 2012 at 6:11 PM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
> Guido van Rossum <guido <at> python.org> writes:
>
>>
>>
>> Looks good to me, so accepted.But why isn't it visible on python.org/dev/peps/
> yet?
>
> I just realized the text in the python.org repo did not match what I had locally.
> I've pushed what I intended to be the latest text, if everyone could take a new
> look at that I would be very grateful. Sorry for the mixup.

NP.

I look a careful look at what's in Hg (still totally different from
what python.org displays) and am proposing a few editorial changes;
please see the review at http://codereview.appspot.com/6447061

Also, I have a few content quibbles:

- Is it really worth flagging a negative return value with ValueError?
I'd just as well clip this to zero. What's the worry? That the
computed value is wrong? But it's only meant to be a hint, and why
would -1 be any more wrong than e.g. 1000000000?

- Did you mean to define operator.length_hint()?

- The default can be zero with no semantic impact, so I think there's
no need to require the caller to specify a default.

- Most importantly: calling len(obj) and catching TypeError can only
be a substitute for the real implementation, which IMO ought to check
for the presence of a tp_len slot. Alas, checking hasattr(obj,
'__len__') doesn't quite cut it either, since this returns true for a
class object that defines a __len__ method for its instances (the
class itself doesn't have a length). Still, I worry that calling
len(obj) and catching all TypeErrors overspecifies the desired
behavior; what I *want* to happen is to check if there is a __len__
method, and if so, call it and let any exceptions bubble through. It
may be best to add a comment explaining that am implementation doesn't
have to follow the letter of the Python code in the PEP, in
particular, if obj *has* a __len__() method but calling it raises an
exception, then length_hint(obj) may (ought to?) pass this exception
on instead of calling obj.__length_hint__().

-- 
--Guido van Rossum (python.org/~guido)

From alex.gaynor at gmail.com  Mon Jul 30 18:58:21 2012
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Mon, 30 Jul 2012 09:58:21 -0700
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <CAP7+vJJWOmLR=o68oKUWGsLPrf1V8-eDAcPv+jz-co_P7_Lj7Q@mail.gmail.com>
References: <loom.20120728T190358-272@post.gmane.org>
	<CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
	<CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
	<loom.20120730T031103-33@post.gmane.org>
	<CAP7+vJJWOmLR=o68oKUWGsLPrf1V8-eDAcPv+jz-co_P7_Lj7Q@mail.gmail.com>
Message-ID: <CAFRnB2WBS93trUHsih+xepZmDpOvuaKGNCPz-eQ0rTP8Sw-s4Q@mail.gmail.com>

On Mon, Jul 30, 2012 at 9:51 AM, Guido van Rossum <guido at python.org> wrote:
>
> Also, I have a few content quibbles:
>
> - Is it really worth flagging a negative return value with ValueError?
> I'd just as well clip this to zero. What's the worry? That the
> computed value is wrong? But it's only meant to be a hint, and why
> would -1 be any more wrong than e.g. 1000000000?
>
>
This was done for consistency with len(), I'm not particularly attached to
any behavior.


> - Did you mean to define operator.length_hint()?
>
>
Of course :)


> - The default can be zero with no semantic impact, so I think there's
> no need to require the caller to specify a default.
>
>
I suppose that's fair.


> - Most importantly: calling len(obj) and catching TypeError can only
> be a substitute for the real implementation, which IMO ought to check
> for the presence of a tp_len slot. Alas, checking hasattr(obj,
> '__len__') doesn't quite cut it either, since this returns true for a
> class object that defines a __len__ method for its instances (the
> class itself doesn't have a length). Still, I worry that calling
> len(obj) and catching all TypeErrors overspecifies the desired
> behavior; what I *want* to happen is to check if there is a __len__
> method, and if so, call it and let any exceptions bubble through. It
> may be best to add a comment explaining that am implementation doesn't
> have to follow the letter of the Python code in the PEP, in
> particular, if obj *has* a __len__() method but calling it raises an
> exception, then length_hint(obj) may (ought to?) pass this exception
> on instead of calling obj.__length_hint__().
>
>
Seems reasonable, rather than try to spec that out precisely in the
pseudocode (aka Python ;)) a note like you suggest sounds good.


> --
> --Guido van Rossum (python.org/~guido)
>


Alex

-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120730/52f7bc3b/attachment.html>

From andrew.svetlov at gmail.com  Mon Jul 30 19:08:14 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Mon, 30 Jul 2012 20:08:14 +0300
Subject: [Python-Dev] Pushing issue #12288
In-Reply-To: <jv6c6f$gn2$1@dough.gmane.org>
References: <CAL3CFcVPRSuN4=BL1CVsJXCRMq=TF2zWqnYT1o3a+f5XLoPnZA@mail.gmail.com>
	<jv6c6f$gn2$1@dough.gmane.org>
Message-ID: <CAL3CFcU=VVXYPJdYszZzkog-y172jbKdZBBGtcbVbuzofEtiVA@mail.gmail.com>

Pushed. Thank you.

On Mon, Jul 30, 2012 at 7:19 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> Am 30.07.2012 12:21, schrieb Andrew Svetlov:
>> Is there ok to push http://bugs.python.org/issue12288 in 3.3?
>> It looks very easy and straightforward.
>> If now is too alte ? I'm ok.
>>
>> Georg Brandl, what's your decision?
>
> Sure, go ahead.  Nobody (I hope) said that every tiny bugfix needs
> my approval.
>
> Georg
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: http://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com



-- 
Thanks,
Andrew Svetlov

From guido at python.org  Mon Jul 30 19:09:17 2012
From: guido at python.org (Guido van Rossum)
Date: Mon, 30 Jul 2012 10:09:17 -0700
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <CAFRnB2WBS93trUHsih+xepZmDpOvuaKGNCPz-eQ0rTP8Sw-s4Q@mail.gmail.com>
References: <loom.20120728T190358-272@post.gmane.org>
	<CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
	<CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
	<loom.20120730T031103-33@post.gmane.org>
	<CAP7+vJJWOmLR=o68oKUWGsLPrf1V8-eDAcPv+jz-co_P7_Lj7Q@mail.gmail.com>
	<CAFRnB2WBS93trUHsih+xepZmDpOvuaKGNCPz-eQ0rTP8Sw-s4Q@mail.gmail.com>
Message-ID: <CAP7+vJJkqm29bTF4kQB7QdacXX8nobFa=Ku+7VHEBMUovx5SdQ@mail.gmail.com>

If you give my second patch an LGTM I'll submit it and you're done.

On Mon, Jul 30, 2012 at 9:58 AM, Alex Gaynor <alex.gaynor at gmail.com> wrote:
>
>
> On Mon, Jul 30, 2012 at 9:51 AM, Guido van Rossum <guido at python.org> wrote:
>>
>> Also, I have a few content quibbles:
>>
>> - Is it really worth flagging a negative return value with ValueError?
>> I'd just as well clip this to zero. What's the worry? That the
>> computed value is wrong? But it's only meant to be a hint, and why
>> would -1 be any more wrong than e.g. 1000000000?
>>
>
> This was done for consistency with len(), I'm not particularly attached to
> any behavior.
>
>>
>> - Did you mean to define operator.length_hint()?
>>
>
> Of course :)
>
>>
>> - The default can be zero with no semantic impact, so I think there's
>> no need to require the caller to specify a default.
>>
>
> I suppose that's fair.
>
>>
>> - Most importantly: calling len(obj) and catching TypeError can only
>> be a substitute for the real implementation, which IMO ought to check
>> for the presence of a tp_len slot. Alas, checking hasattr(obj,
>> '__len__') doesn't quite cut it either, since this returns true for a
>> class object that defines a __len__ method for its instances (the
>> class itself doesn't have a length). Still, I worry that calling
>> len(obj) and catching all TypeErrors overspecifies the desired
>> behavior; what I *want* to happen is to check if there is a __len__
>> method, and if so, call it and let any exceptions bubble through. It
>> may be best to add a comment explaining that am implementation doesn't
>> have to follow the letter of the Python code in the PEP, in
>> particular, if obj *has* a __len__() method but calling it raises an
>> exception, then length_hint(obj) may (ought to?) pass this exception
>> on instead of calling obj.__length_hint__().
>>
>
> Seems reasonable, rather than try to spec that out precisely in the
> pseudocode (aka Python ;)) a note like you suggest sounds good.
>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>
>
>
> Alex
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
>



-- 
--Guido van Rossum (python.org/~guido)

From iftecan2000 at gmail.com  Mon Jul 30 20:49:32 2012
From: iftecan2000 at gmail.com (Ifthikhan Nazeem)
Date: Mon, 30 Jul 2012 20:49:32 +0200
Subject: [Python-Dev] [Python-checkins] peps: Update PEP-0424 with
 respect to feedback received.
In-Reply-To: <CAL3CFcVRocqPGD=Mtou9DufwWyS6BehJwX5oMnBF-FxE1PsSsg@mail.gmail.com>
References: <3WljQP1gshzPXX@mail.python.org>
	<CAL3CFcVRocqPGD=Mtou9DufwWyS6BehJwX5oMnBF-FxE1PsSsg@mail.gmail.com>
Message-ID: <CAAax9+qtR68Bis_HO=DiVwfyb_s-Cy3EN+_C-pC2=giG8FPZAg@mail.gmail.com>

The question is not very clear. Are you asking whether function arguments
can have default parameters?

--
Thanks
Iftikhan Nazeem
*LinkedIn* : http://ae.linkedin.com/in/ifthikhan<http://ae.linkedin.com/in/iftikhan>


On Mon, Jul 30, 2012 at 11:23 AM, Andrew Svetlov
<andrew.svetlov at gmail.com>wrote:

> On Mon, Jul 30, 2012 at 4:10 AM, alex.gaynor <python-checkins at python.org>
> wrote:
>
> > +In addition, a new function ``operator.length`` hint is added
>
> ``operator.length`` or ``operator.length_hint``?
>
> > +    def length_hint(obj, default):
> Is there default value for `default` parameter?
> For example:
>
> def length_hint(obj, default=0):
>     pass
>
> or something like that?
>
> --
> Thanks,
> Andrew Svetlov
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> http://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> http://mail.python.org/mailman/options/python-dev/iftecan2000%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120730/deda9854/attachment.html>

From andrew.svetlov at gmail.com  Mon Jul 30 20:58:59 2012
From: andrew.svetlov at gmail.com (Andrew Svetlov)
Date: Mon, 30 Jul 2012 21:58:59 +0300
Subject: [Python-Dev] [Python-checkins] peps: Update PEP-0424 with
 respect to feedback received.
In-Reply-To: <CAAax9+qtR68Bis_HO=DiVwfyb_s-Cy3EN+_C-pC2=giG8FPZAg@mail.gmail.com>
References: <3WljQP1gshzPXX@mail.python.org>
	<CAL3CFcVRocqPGD=Mtou9DufwWyS6BehJwX5oMnBF-FxE1PsSsg@mail.gmail.com>
	<CAAax9+qtR68Bis_HO=DiVwfyb_s-Cy3EN+_C-pC2=giG8FPZAg@mail.gmail.com>
Message-ID: <CAL3CFcVWg7Q5+P7X_jCx5adm4HHSWCugPHXxJk_JWVZuWb5B+Q@mail.gmail.com>

Sure, GvR covered my question in much more clean way in
http://code.activestate.com/lists/python-dev/117152/

On Mon, Jul 30, 2012 at 9:49 PM, Ifthikhan Nazeem <iftecan2000 at gmail.com> wrote:
> The question is not very clear. Are you asking whether function arguments
> can have default parameters?
>
> --
> Thanks
> Iftikhan Nazeem
> LinkedIn : http://ae.linkedin.com/in/ifthikhan
>
>
> On Mon, Jul 30, 2012 at 11:23 AM, Andrew Svetlov <andrew.svetlov at gmail.com>
> wrote:
>>
>> On Mon, Jul 30, 2012 at 4:10 AM, alex.gaynor <python-checkins at python.org>
>> wrote:
>>
>> > +In addition, a new function ``operator.length`` hint is added
>>
>> ``operator.length`` or ``operator.length_hint``?
>>
>> > +    def length_hint(obj, default):
>> Is there default value for `default` parameter?
>> For example:
>>
>> def length_hint(obj, default=0):
>>     pass
>>
>> or something like that?
>>
>> --
>> Thanks,
>> Andrew Svetlov
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> http://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> http://mail.python.org/mailman/options/python-dev/iftecan2000%40gmail.com
>
>



-- 
Thanks,
Andrew Svetlov

From yselivanov.ml at gmail.com  Mon Jul 30 21:42:39 2012
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Mon, 30 Jul 2012 15:42:39 -0400
Subject: [Python-Dev] python 3.3 b2
Message-ID: <7E4C776A-2456-4FDC-A86F-E1F0851DEB25@gmail.com>

Hi all,

What's the status of Python 3.3 beta 2?

Thanks,
Yury

From stefan at bytereef.org  Mon Jul 30 22:14:03 2012
From: stefan at bytereef.org (Stefan Krah)
Date: Mon, 30 Jul 2012 22:14:03 +0200
Subject: [Python-Dev] python 3.3 b2
In-Reply-To: <7E4C776A-2456-4FDC-A86F-E1F0851DEB25@gmail.com>
References: <7E4C776A-2456-4FDC-A86F-E1F0851DEB25@gmail.com>
Message-ID: <20120730201403.GA17899@sleipnir.bytereef.org>

Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> What's the status of Python 3.3 beta 2?

It's postponed until next weekend.


Stefan Krah



From g.brandl at gmx.net  Mon Jul 30 22:22:25 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Mon, 30 Jul 2012 22:22:25 +0200
Subject: [Python-Dev] python 3.3 b2
In-Reply-To: <7E4C776A-2456-4FDC-A86F-E1F0851DEB25@gmail.com>
References: <7E4C776A-2456-4FDC-A86F-E1F0851DEB25@gmail.com>
Message-ID: <jv6qdh$av4$1@dough.gmane.org>

Am 30.07.2012 21:42, schrieb Yury Selivanov:
> Hi all,
> 
> What's the status of Python 3.3 beta 2?

As I've explained on python-committers, it's currently on hold pending
the resolution of some importlib issues as well as a bug with the
cross-compiling code.  I won't issue a concrete date, but I expect the
release to be made some time before next Sunday.

cheers,
Georg


From barry at python.org  Mon Jul 30 22:59:08 2012
From: barry at python.org (Barry Warsaw)
Date: Mon, 30 Jul 2012 16:59:08 -0400
Subject: [Python-Dev] python 3.3 b2
In-Reply-To: <jv6qdh$av4$1@dough.gmane.org>
References: <7E4C776A-2456-4FDC-A86F-E1F0851DEB25@gmail.com>
	<jv6qdh$av4$1@dough.gmane.org>
Message-ID: <20120730165908.25ad5c12@limelight.wooz.org>

On Jul 30, 2012, at 10:22 PM, Georg Brandl wrote:

>As I've explained on python-committers, it's currently on hold pending
>the resolution of some importlib issues as well as a bug with the
>cross-compiling code.  I won't issue a concrete date, but I expect the
>release to be made some time before next Sunday.

I expect to land the new import documentation (issue 15295) within the next
day or so, after I slog through the remaining comments.  I have unpushed local
changes that still need a little polishing, but I don't want it to linger too
long.

-Barry

From pje at telecommunity.com  Tue Jul 31 05:25:28 2012
From: pje at telecommunity.com (PJ Eby)
Date: Mon, 30 Jul 2012 23:25:28 -0400
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <CAP7+vJJWOmLR=o68oKUWGsLPrf1V8-eDAcPv+jz-co_P7_Lj7Q@mail.gmail.com>
References: <loom.20120728T190358-272@post.gmane.org>
	<CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
	<CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
	<loom.20120730T031103-33@post.gmane.org>
	<CAP7+vJJWOmLR=o68oKUWGsLPrf1V8-eDAcPv+jz-co_P7_Lj7Q@mail.gmail.com>
Message-ID: <CALeMXf7+QT1-jU4o=h=JwyDfxoSaMW5FwV2dk7TpyGgFTHNJ2Q@mail.gmail.com>

On Mon, Jul 30, 2012 at 12:51 PM, Guido van Rossum <guido at python.org> wrote:
> - Most importantly: calling len(obj) and catching TypeError can only
> be a substitute for the real implementation, which IMO ought to check
> for the presence of a tp_len slot. Alas, checking hasattr(obj,
> '__len__') doesn't quite cut it either, since this returns true for a
> class object that defines a __len__ method for its instances (the
> class itself doesn't have a length).

This isn't the only place this pattern comes up; maybe a hasmethod()
function somewhere (builtin, operator, inspect?) for this would be a
good idea.  (i.e., something that returns true only if the method is
for the instance.)

(But perhaps that's a python-ideas topic, since it raises the question
of whether it should really be something more like instancehasattr(),
or whether it should be limited to special slots or something else.)

From guido at python.org  Tue Jul 31 06:17:25 2012
From: guido at python.org (Guido van Rossum)
Date: Tue, 31 Jul 2012 06:17:25 +0200
Subject: [Python-Dev] Requesting pronouncement on PEP 0424
In-Reply-To: <CALeMXf7+QT1-jU4o=h=JwyDfxoSaMW5FwV2dk7TpyGgFTHNJ2Q@mail.gmail.com>
References: <loom.20120728T190358-272@post.gmane.org>
	<CAP7+vJKFdoH+kyQVsVu5fOxe+z+a2g7nQ1oQ2Vyj5_ueJrhJ6g@mail.gmail.com>
	<CAP7+vJLE11GwFG8uoAyr1cyPb7dZn=L5D-fAC=MKAQzMfaasog@mail.gmail.com>
	<loom.20120730T031103-33@post.gmane.org>
	<CAP7+vJJWOmLR=o68oKUWGsLPrf1V8-eDAcPv+jz-co_P7_Lj7Q@mail.gmail.com>
	<CALeMXf7+QT1-jU4o=h=JwyDfxoSaMW5FwV2dk7TpyGgFTHNJ2Q@mail.gmail.com>
Message-ID: <CAP7+vJKB-D7sTDUWLf1A41a+vCs0gnJqXic+C9jX9RcnvkWsiA@mail.gmail.com>

On Tue, Jul 31, 2012 at 5:25 AM, PJ Eby <pje at telecommunity.com> wrote:
> On Mon, Jul 30, 2012 at 12:51 PM, Guido van Rossum <guido at python.org> wrote:
>> - Most importantly: calling len(obj) and catching TypeError can only
>> be a substitute for the real implementation, which IMO ought to check
>> for the presence of a tp_len slot. Alas, checking hasattr(obj,
>> '__len__') doesn't quite cut it either, since this returns true for a
>> class object that defines a __len__ method for its instances (the
>> class itself doesn't have a length).
>
> This isn't the only place this pattern comes up; maybe a hasmethod()
> function somewhere (builtin, operator, inspect?) for this would be a
> good idea.  (i.e., something that returns true only if the method is
> for the instance.)
>
> (But perhaps that's a python-ideas topic, since it raises the question
> of whether it should really be something more like instancehasattr(),
> or whether it should be limited to special slots or something else.)

Yes, please redirect / repost; I read p-ideas too. It's an interesting
topic, if very specialized.

-- 
--Guido van Rossum (python.org/~guido)

From g.brandl at gmx.net  Tue Jul 31 06:46:40 2012
From: g.brandl at gmx.net (Georg Brandl)
Date: Tue, 31 Jul 2012 06:46:40 +0200
Subject: [Python-Dev] python 3.3 b2
In-Reply-To: <20120730165908.25ad5c12@limelight.wooz.org>
References: <7E4C776A-2456-4FDC-A86F-E1F0851DEB25@gmail.com>
	<jv6qdh$av4$1@dough.gmane.org>
	<20120730165908.25ad5c12@limelight.wooz.org>
Message-ID: <jv7nuv$j49$1@dough.gmane.org>

Am 30.07.2012 22:59, schrieb Barry Warsaw:
> On Jul 30, 2012, at 10:22 PM, Georg Brandl wrote:
> 
>>As I've explained on python-committers, it's currently on hold pending
>>the resolution of some importlib issues as well as a bug with the
>>cross-compiling code.  I won't issue a concrete date, but I expect the
>>release to be made some time before next Sunday.
> 
> I expect to land the new import documentation (issue 15295) within the next
> day or so, after I slog through the remaining comments.  I have unpushed local
> changes that still need a little polishing, but I don't want it to linger too
> long.

That's good news; thank you Barry!

And of course big kudos go to Nick, who's sorting out a lot of the latest
blockers more or less single-handedly :)

Georg


From ncoghlan at gmail.com  Tue Jul 31 07:56:49 2012
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 31 Jul 2012 15:56:49 +1000
Subject: [Python-Dev] python 3.3 b2
In-Reply-To: <jv7nuv$j49$1@dough.gmane.org>
References: <7E4C776A-2456-4FDC-A86F-E1F0851DEB25@gmail.com>
	<jv6qdh$av4$1@dough.gmane.org>
	<20120730165908.25ad5c12@limelight.wooz.org>
	<jv7nuv$j49$1@dough.gmane.org>
Message-ID: <CADiSq7d2JW866giVqDsALodEViQ-XvBguCctoapHc6CyvTKT0w@mail.gmail.com>

On Tue, Jul 31, 2012 at 2:46 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> Am 30.07.2012 22:59, schrieb Barry Warsaw:
>> On Jul 30, 2012, at 10:22 PM, Georg Brandl wrote:
>>
>>>As I've explained on python-committers, it's currently on hold pending
>>>the resolution of some importlib issues as well as a bug with the
>>>cross-compiling code.  I won't issue a concrete date, but I expect the
>>>release to be made some time before next Sunday.
>>
>> I expect to land the new import documentation (issue 15295) within the next
>> day or so, after I slog through the remaining comments.  I have unpushed local
>> changes that still need a little polishing, but I don't want it to linger too
>> long.
>
> That's good news; thank you Barry!
>
> And of course big kudos go to Nick, who's sorting out a lot of the latest
> blockers more or less single-handedly :)

With Brett doing so much of the heavy lifting to get importlib
bootstrapped in the first place, helping to file off the last few
rough edges seems like the least I can do :)

Something that has definitely suffered for it is the ipaddress docs,
though - they're still missing the method and attribute details for
the various types (see http://bugs.python.org/issue14814).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From brett at python.org  Tue Jul 31 15:16:16 2012
From: brett at python.org (Brett Cannon)
Date: Tue, 31 Jul 2012 09:16:16 -0400
Subject: [Python-Dev] python 3.3 b2
In-Reply-To: <CADiSq7d2JW866giVqDsALodEViQ-XvBguCctoapHc6CyvTKT0w@mail.gmail.com>
References: <7E4C776A-2456-4FDC-A86F-E1F0851DEB25@gmail.com>
	<jv6qdh$av4$1@dough.gmane.org>
	<20120730165908.25ad5c12@limelight.wooz.org>
	<jv7nuv$j49$1@dough.gmane.org>
	<CADiSq7d2JW866giVqDsALodEViQ-XvBguCctoapHc6CyvTKT0w@mail.gmail.com>
Message-ID: <CAP1=2W541J_LYahjW=BM54OXcvLZZSKtgmNVVRzQtpVnBCthiw@mail.gmail.com>

On Tue, Jul 31, 2012 at 1:56 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On Tue, Jul 31, 2012 at 2:46 PM, Georg Brandl <g.brandl at gmx.net> wrote:
> > Am 30.07.2012 22:59, schrieb Barry Warsaw:
> >> On Jul 30, 2012, at 10:22 PM, Georg Brandl wrote:
> >>
> >>>As I've explained on python-committers, it's currently on hold pending
> >>>the resolution of some importlib issues as well as a bug with the
> >>>cross-compiling code.  I won't issue a concrete date, but I expect the
> >>>release to be made some time before next Sunday.
> >>
> >> I expect to land the new import documentation (issue 15295) within the
> next
> >> day or so, after I slog through the remaining comments.  I have
> unpushed local
> >> changes that still need a little polishing, but I don't want it to
> linger too
> >> long.
> >
> > That's good news; thank you Barry!
> >
> > And of course big kudos go to Nick, who's sorting out a lot of the latest
> > blockers more or less single-handedly :)
>
> With Brett doing so much of the heavy lifting to get importlib
> bootstrapped in the first place, helping to file off the last few
> rough edges seems like the least I can do :)
>

Yes, definite thanks to Nick for dealing with a bunch of pkgutil bugs and
such, along with Barry pushing the new import docs forward,  Eric Snow for
helping discuss all of the issues, and Eric Smith chiming in as well for b2
doc stuff.

My personal life got thrown a curveball last week that is going to take a
month before the turmoil from it settles down completely and my free time
isn't spent on work stuff (no need to worry, everyone is fine and there
won't be any long term effects) so I definitely appreciate the help with
all of this.


>
> Something that has definitely suffered for it is the ipaddress docs,
> though - they're still missing the method and attribute details for
> the various types (see http://bugs.python.org/issue14814).
>

Heh, the PEP 362 Signature docs have not been checked in yet either (
http://bugs.python.org/issue15151), although Yury has a draft done. So you
are not the only one behind in docs stuff for 3.3. =)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20120731/655d684c/attachment.html>