From tismer at stackless.com  Sat Oct  1 07:39:45 2016
From: tismer at stackless.com (Christian Tismer)
Date: Sat, 1 Oct 2016 13:39:45 +0200
Subject: [Python-Dev] __future__ and eval()
Message-ID: <13646f28-1be0-396d-6888-ad8ce9c1fb75@stackless.com>

Hi guys,

when developing my dedent tool, I stumbled over the following
behavior:

In dedent, I had the line

     from __future__ import print_function

Later in the script, I do some

     exec(the_script)

and I was surprised:
The exec() script inherited the __future__ statement!
It behaved like the future statement were implicitly there.

Is that a bug or a feature?

You can try the effect by "pip install dedent" and adding
the future statement there.

I'd like to know if this is a bug (and I think so)

-- 
Christian Tismer             :^)   tismer at stackless.com
Software Consulting          :     http://www.stackless.com/
Karl-Liebknecht-Str. 121     :     https://github.com/PySide
14482 Potsdam                :     GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023

From rosuav at gmail.com  Sat Oct  1 08:17:41 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 1 Oct 2016 22:17:41 +1000
Subject: [Python-Dev] __future__ and eval()
In-Reply-To: <13646f28-1be0-396d-6888-ad8ce9c1fb75@stackless.com>
References: <13646f28-1be0-396d-6888-ad8ce9c1fb75@stackless.com>
Message-ID: <CAPTjJmo96EPUyjtx_xw--K6ksiCAdpCp2GB14-cEFLUGu+5Esw@mail.gmail.com>

On Sat, Oct 1, 2016 at 9:39 PM, Christian Tismer <tismer at stackless.com> wrote:
> The exec() script inherited the __future__ statement!
> It behaved like the future statement were implicitly there.
>
> Is that a bug or a feature?

It's documented, but not very noisily.

https://docs.python.org/2/reference/simple_stmts.html#future-statements

So if you want to isolate the execution environments, you can use
compile() explicitly. Without isolation:

Python 2.7.12+ (default, Sep  1 2016, 20:27:38)
[GCC 6.2.0 20160822] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exec(compile("print('hello','world')", "-", "exec"))
('hello', 'world')
>>> exec(compile("from __future__ import print_function; print('hello','world')", "-", "exec"))
hello world
>>> from __future__ import print_function
>>> exec(compile("print('hello','world')", "-", "exec"))
hello world
>>> exec(compile("from __future__ import print_function; print('hello','world')", "-", "exec"))
hello world

With isolation:

>>> exec(compile("print('hello','world')", "-", "exec", 0, 1))
('hello', 'world')
>>> exec(compile("from __future__ import print_function; print('hello','world')", "-", "exec", 0, 1))
hello world

So I'd call it a feature, but possibly one that warrants a mention in
the exec and eval docs. Maybe something like:

https://docs.python.org/2/reference/simple_stmts.html#the-exec-statement

[2] When strings are executed, __future__ directives active in the
surrounding context will be active for the compiled code also. If this
is not desired, see the compile() function's dont_inherit parameter.

Would that clarify?

ChrisA

From rosuav at gmail.com  Sat Oct  1 08:18:44 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Sat, 1 Oct 2016 22:18:44 +1000
Subject: [Python-Dev] __future__ and eval()
In-Reply-To: <CAPTjJmo96EPUyjtx_xw--K6ksiCAdpCp2GB14-cEFLUGu+5Esw@mail.gmail.com>
References: <13646f28-1be0-396d-6888-ad8ce9c1fb75@stackless.com>
 <CAPTjJmo96EPUyjtx_xw--K6ksiCAdpCp2GB14-cEFLUGu+5Esw@mail.gmail.com>
Message-ID: <CAPTjJmqP3tJC5i02TVKCzAttcyE=wt-rxUfFJYvu1kZ7GEbQMQ@mail.gmail.com>

On Sat, Oct 1, 2016 at 10:17 PM, Chris Angelico <rosuav at gmail.com> wrote:
> So I'd call it a feature, but possibly one that warrants a mention in
> the exec and eval docs.

To clarify: This *is* documented under __future__, but not under
exec/eval. I'm just suggesting adding another cross-reference.

ChrisA

From tismer at stackless.com  Sat Oct  1 08:46:09 2016
From: tismer at stackless.com (Christian Tismer)
Date: Sat, 1 Oct 2016 14:46:09 +0200
Subject: [Python-Dev] __future__ and eval()
In-Reply-To: <CAPTjJmo96EPUyjtx_xw--K6ksiCAdpCp2GB14-cEFLUGu+5Esw@mail.gmail.com>
References: <13646f28-1be0-396d-6888-ad8ce9c1fb75@stackless.com>
 <CAPTjJmo96EPUyjtx_xw--K6ksiCAdpCp2GB14-cEFLUGu+5Esw@mail.gmail.com>
Message-ID: <edf65e9f-95e8-0630-9b99-3e063e8529b9@stackless.com>

Ah, interesting!

Thanks for the clarification. So it is really possible to write
code with an implicit future statement in it, or to switch the behavior
off.
Good to know. I will probably not use it, since I can't decide on a good
default, but getting rid of print_statement is tempting...

 > https://docs.python.org/2/reference/simple_stmts.html#the-exec-statement
 >
 > [2] When strings are executed, __future__ directives active in the
 > surrounding context will be active for the compiled code also. If this
 > is not desired, see the compile() function's dont_inherit parameter.
 >
 > Would that clarify?

Yes please, that would be a good place to document it. For some reason
I did not look up __future__.

Thanks -- Chris


On 01/10/16 14:17, Chris Angelico wrote:
> On Sat, Oct 1, 2016 at 9:39 PM, Christian Tismer <tismer at stackless.com> wrote:
>> The exec() script inherited the __future__ statement!
>> It behaved like the future statement were implicitly there.
>>
>> Is that a bug or a feature?
>
> It's documented, but not very noisily.
>
> https://docs.python.org/2/reference/simple_stmts.html#future-statements
>
> So if you want to isolate the execution environments, you can use
> compile() explicitly. Without isolation:
>
> Python 2.7.12+ (default, Sep  1 2016, 20:27:38)
> [GCC 6.2.0 20160822] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> exec(compile("print('hello','world')", "-", "exec"))
> ('hello', 'world')
>>>> exec(compile("from __future__ import print_function; print('hello','world')", "-", "exec"))
> hello world
>>>> from __future__ import print_function
>>>> exec(compile("print('hello','world')", "-", "exec"))
> hello world
>>>> exec(compile("from __future__ import print_function; print('hello','world')", "-", "exec"))
> hello world
>
> With isolation:
>
>>>> exec(compile("print('hello','world')", "-", "exec", 0, 1))
> ('hello', 'world')
>>>> exec(compile("from __future__ import print_function; print('hello','world')", "-", "exec", 0, 1))
> hello world
>
> So I'd call it a feature, but possibly one that warrants a mention in
> the exec and eval docs. Maybe something like:
>
> https://docs.python.org/2/reference/simple_stmts.html#the-exec-statement
>
> [2] When strings are executed, __future__ directives active in the
> surrounding context will be active for the compiled code also. If this
> is not desired, see the compile() function's dont_inherit parameter.
>
> Would that clarify?
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com
>


-- 
Christian Tismer             :^)   tismer at stackless.com
Software Consulting          :     http://www.stackless.com/
Karl-Liebknecht-Str. 121     :     https://github.com/PySide
14482 Potsdam                :     GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023

From ncoghlan at gmail.com  Sun Oct  2 03:22:07 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 2 Oct 2016 17:22:07 +1000
Subject: [Python-Dev] __future__ and eval()
In-Reply-To: <CAPTjJmqP3tJC5i02TVKCzAttcyE=wt-rxUfFJYvu1kZ7GEbQMQ@mail.gmail.com>
References: <13646f28-1be0-396d-6888-ad8ce9c1fb75@stackless.com>
 <CAPTjJmo96EPUyjtx_xw--K6ksiCAdpCp2GB14-cEFLUGu+5Esw@mail.gmail.com>
 <CAPTjJmqP3tJC5i02TVKCzAttcyE=wt-rxUfFJYvu1kZ7GEbQMQ@mail.gmail.com>
Message-ID: <CADiSq7edk5keohNyZnsEdQJt0Ya3KbKgQ+303wLatVJLF=kuSQ@mail.gmail.com>

On 1 October 2016 at 22:18, Chris Angelico <rosuav at gmail.com> wrote:
> On Sat, Oct 1, 2016 at 10:17 PM, Chris Angelico <rosuav at gmail.com> wrote:
>> So I'd call it a feature, but possibly one that warrants a mention in
>> the exec and eval docs.
>
> To clarify: This *is* documented under __future__, but not under
> exec/eval. I'm just suggesting adding another cross-reference.

The most appropriate cross-reference would probably be to the
compile() docs rather than the language reference though:
https://docs.python.org/2/library/functions.html#compile

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Mon Oct  3 03:58:51 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 3 Oct 2016 09:58:51 +0200
Subject: [Python-Dev] Need a review on a patch related to weak references
 and the garbage collector
Message-ID: <CAMpsgwY3z-3pXvKTUwOrKQ5u-emTGgg4Hf5C0mawz_gV0jZpgw@mail.gmail.com>

Hi,

Can someone please review the patch attached to:
http://bugs.python.org/issue26617

The patch seems safe and well defined, but it would be nice to have
more reviews on it.

It's a crash (assertion error) in Python >= 3.4 which occurs
"sometimes" in asyncio.

Victor

From skip.montanaro at gmail.com  Mon Oct  3 09:37:10 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 3 Oct 2016 08:37:10 -0500
Subject: [Python-Dev] Missing PY_ prefixes in structmember.h
Message-ID: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>

I'm sure this has a simple explanation (and is probably only of historical
interest at this point), but ...

While starting to port the Python Sybase module to Python 3, among other
hurdles, I noticed that RO is no longer defined. Looking in structmember.h,
I see that most of the macros defined there are missing a PY_ prefix. Given
that these are macros which are likely to be used by extension modules and
are thus really part of the C API, shouldn't they have PY_ or _PY_
prefixes? This file got its start in 1990, so would surely have been around
for the great renaming
<http://python-history.blogspot.com/2009/03/great-or-grand-renaming.html>.
Looking at the documentation on defining new types, I saw no mention of
these peculiarly named constants, though they are clearly documented.

Thanks,

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161003/7b841249/attachment.html>

From victor.stinner at gmail.com  Mon Oct  3 11:39:12 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 3 Oct 2016 17:39:12 +0200
Subject: [Python-Dev] Missing PY_ prefixes in structmember.h
In-Reply-To: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>
References: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>
Message-ID: <CAMpsgwYH4Lo4RyCtLv+w9w70SoLQtbDmaY8qJAh_A+HAhRutkw@mail.gmail.com>

2016-10-03 15:37 GMT+02:00 Skip Montanaro <skip.montanaro at gmail.com>:
> While starting to port the Python Sybase module to Python 3, among other
> hurdles, I noticed that RO is no longer defined. Looking in structmember.h,

RO was an alias to READONLY. READONLY still exists in Python 3, just
use this name.

You might create an alias in your code if it is missing:

#ifndef RO
#  define RO READONLY
#endif

Victor

From skip.montanaro at gmail.com  Mon Oct  3 13:11:30 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 3 Oct 2016 12:11:30 -0500
Subject: [Python-Dev] Missing PY_ prefixes in structmember.h
In-Reply-To: <CAMpsgwYH4Lo4RyCtLv+w9w70SoLQtbDmaY8qJAh_A+HAhRutkw@mail.gmail.com>
References: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>
 <CAMpsgwYH4Lo4RyCtLv+w9w70SoLQtbDmaY8qJAh_A+HAhRutkw@mail.gmail.com>
Message-ID: <CANc-5UzO98e0kYcckfd0WGS2Co4CZtR9FA4-qPKPozyWKYy2qg@mail.gmail.com>

Thanks, Victor, but that's not quite what I was asking. Replacing "RO"
with "READONLY" is clearly no big deal. My question was more wondering
why I didn't find myself replacing PY_RO with PY_READONLY". Both names
were part of the Public API in the early 90s, I suspect, and one of
that set of flags does have a "PY_" prefix. Why didn't these flags
(and the T_* flags in structmember.h) get swept up in all the hubbub
around the grand renaming.

Skip


On Mon, Oct 3, 2016 at 10:39 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 2016-10-03 15:37 GMT+02:00 Skip Montanaro <skip.montanaro at gmail.com>:
>> While starting to port the Python Sybase module to Python 3, among other
>> hurdles, I noticed that RO is no longer defined. Looking in structmember.h,
>
> RO was an alias to READONLY. READONLY still exists in Python 3, just
> use this name.
>
> You might create an alias in your code if it is missing:
>
> #ifndef RO
> #  define RO READONLY
> #endif
>
> Victor

From gvanrossum at gmail.com  Mon Oct  3 13:20:18 2016
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Mon, 3 Oct 2016 10:20:18 -0700
Subject: [Python-Dev] Missing PY_ prefixes in structmember.h
In-Reply-To: <CANc-5UzO98e0kYcckfd0WGS2Co4CZtR9FA4-qPKPozyWKYy2qg@mail.gmail.com>
References: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>
 <CAMpsgwYH4Lo4RyCtLv+w9w70SoLQtbDmaY8qJAh_A+HAhRutkw@mail.gmail.com>
 <CANc-5UzO98e0kYcckfd0WGS2Co4CZtR9FA4-qPKPozyWKYy2qg@mail.gmail.com>
Message-ID: <CAP7+vJJTsCuJtpHRHZZ5++zoGit35HwQ7UXJtrAE2QTwVhfHBw@mail.gmail.com>

Sounds like an oversight in a backwater of a header file. We should add the
PY symbols and deprecate the others.

--Guido (mobile)

On Oct 3, 2016 10:17 AM, "Skip Montanaro" <skip.montanaro at gmail.com> wrote:

> Thanks, Victor, but that's not quite what I was asking. Replacing "RO"
> with "READONLY" is clearly no big deal. My question was more wondering
> why I didn't find myself replacing PY_RO with PY_READONLY". Both names
> were part of the Public API in the early 90s, I suspect, and one of
> that set of flags does have a "PY_" prefix. Why didn't these flags
> (and the T_* flags in structmember.h) get swept up in all the hubbub
> around the grand renaming.
>
> Skip
>
>
> On Mon, Oct 3, 2016 at 10:39 AM, Victor Stinner
> <victor.stinner at gmail.com> wrote:
> > 2016-10-03 15:37 GMT+02:00 Skip Montanaro <skip.montanaro at gmail.com>:
> >> While starting to port the Python Sybase module to Python 3, among other
> >> hurdles, I noticed that RO is no longer defined. Looking in
> structmember.h,
> >
> > RO was an alias to READONLY. READONLY still exists in Python 3, just
> > use this name.
> >
> > You might create an alias in your code if it is missing:
> >
> > #ifndef RO
> > #  define RO READONLY
> > #endif
> >
> > Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161003/6c58a85d/attachment.html>

From alexander.belopolsky at gmail.com  Mon Oct  3 13:27:20 2016
From: alexander.belopolsky at gmail.com (Alexander Belopolsky)
Date: Mon, 3 Oct 2016 13:27:20 -0400
Subject: [Python-Dev] Missing PY_ prefixes in structmember.h
In-Reply-To: <CANc-5UzO98e0kYcckfd0WGS2Co4CZtR9FA4-qPKPozyWKYy2qg@mail.gmail.com>
References: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>
 <CAMpsgwYH4Lo4RyCtLv+w9w70SoLQtbDmaY8qJAh_A+HAhRutkw@mail.gmail.com>
 <CANc-5UzO98e0kYcckfd0WGS2Co4CZtR9FA4-qPKPozyWKYy2qg@mail.gmail.com>
Message-ID: <CAP7h-xZojvYWTGei7JVaBzSgXAkqUzSouL6u5DcizagqA1BAiA@mail.gmail.com>

On Mon, Oct 3, 2016 at 1:11 PM, Skip Montanaro <skip.montanaro at gmail.com>
wrote:

> Why didn't these flags
> (and the T_* flags in structmember.h) get swept up in all the hubbub
> around the grand renaming.
>

Note that structmember.h is semi-private - it is not included in Python.h.
See discussion at <http://bugs.python.org/issue2897>.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161003/19e9a113/attachment.html>

From naitikchandak213 at gmail.com  Mon Oct  3 13:23:11 2016
From: naitikchandak213 at gmail.com (Naitik Chandak)
Date: Mon, 3 Oct 2016 22:53:11 +0530
Subject: [Python-Dev] Missing PY_ prefixes in structmember.h
In-Reply-To: <CAP7+vJJTsCuJtpHRHZZ5++zoGit35HwQ7UXJtrAE2QTwVhfHBw@mail.gmail.com>
References: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>
 <CAMpsgwYH4Lo4RyCtLv+w9w70SoLQtbDmaY8qJAh_A+HAhRutkw@mail.gmail.com>
 <CANc-5UzO98e0kYcckfd0WGS2Co4CZtR9FA4-qPKPozyWKYy2qg@mail.gmail.com>
 <CAP7+vJJTsCuJtpHRHZZ5++zoGit35HwQ7UXJtrAE2QTwVhfHBw@mail.gmail.com>
Message-ID: <CAKtJGMWvG0Fb8Rp2YEHOanPTqtNR2dmimxagJZ3Z3+KZ3kDsnA@mail.gmail.com>

hello Members,

If any one is working AIML library for Python 3.5/3.6 version? Please let
me know about it,

thanks

On Mon, Oct 3, 2016 at 10:50 PM, Guido van Rossum <gvanrossum at gmail.com>
wrote:

> Sounds like an oversight in a backwater of a header file. We should add
> the PY symbols and deprecate the others.
>
> --Guido (mobile)
>
> On Oct 3, 2016 10:17 AM, "Skip Montanaro" <skip.montanaro at gmail.com>
> wrote:
>
>> Thanks, Victor, but that's not quite what I was asking. Replacing "RO"
>> with "READONLY" is clearly no big deal. My question was more wondering
>> why I didn't find myself replacing PY_RO with PY_READONLY". Both names
>> were part of the Public API in the early 90s, I suspect, and one of
>> that set of flags does have a "PY_" prefix. Why didn't these flags
>> (and the T_* flags in structmember.h) get swept up in all the hubbub
>> around the grand renaming.
>>
>> Skip
>>
>>
>> On Mon, Oct 3, 2016 at 10:39 AM, Victor Stinner
>> <victor.stinner at gmail.com> wrote:
>> > 2016-10-03 15:37 GMT+02:00 Skip Montanaro <skip.montanaro at gmail.com>:
>> >> While starting to port the Python Sybase module to Python 3, among
>> other
>> >> hurdles, I noticed that RO is no longer defined. Looking in
>> structmember.h,
>> >
>> > RO was an alias to READONLY. READONLY still exists in Python 3, just
>> > use this name.
>> >
>> > You might create an alias in your code if it is missing:
>> >
>> > #ifndef RO
>> > #  define RO READONLY
>> > #endif
>> >
>> > Victor
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
>> 40python.org
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> naitikchandak213%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161003/ddd04482/attachment.html>

From guido at python.org  Mon Oct  3 13:41:23 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 3 Oct 2016 10:41:23 -0700
Subject: [Python-Dev] Missing PY_ prefixes in structmember.h
In-Reply-To: <CAP7h-xZojvYWTGei7JVaBzSgXAkqUzSouL6u5DcizagqA1BAiA@mail.gmail.com>
References: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>
 <CAMpsgwYH4Lo4RyCtLv+w9w70SoLQtbDmaY8qJAh_A+HAhRutkw@mail.gmail.com>
 <CANc-5UzO98e0kYcckfd0WGS2Co4CZtR9FA4-qPKPozyWKYy2qg@mail.gmail.com>
 <CAP7h-xZojvYWTGei7JVaBzSgXAkqUzSouL6u5DcizagqA1BAiA@mail.gmail.com>
Message-ID: <CAP7+vJ+BOWHN5RVKFKmBoyMUC_kbDjP7pFpWwdjL+o79EKdBEw@mail.gmail.com>

But they most certainly are documented! So Skip's point is valid.
MvL's suggestion for how to evolve this API is still the best IMO.

On Mon, Oct 3, 2016 at 10:27 AM, Alexander Belopolsky
<alexander.belopolsky at gmail.com> wrote:
>
> On Mon, Oct 3, 2016 at 1:11 PM, Skip Montanaro <skip.montanaro at gmail.com>
> wrote:
>>
>> Why didn't these flags
>> (and the T_* flags in structmember.h) get swept up in all the hubbub
>> around the grand renaming.
>
>
> Note that structmember.h is semi-private - it is not included in Python.h.
> See discussion at <http://bugs.python.org/issue2897>.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From storchaka at gmail.com  Mon Oct  3 14:10:37 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 3 Oct 2016 21:10:37 +0300
Subject: [Python-Dev] Missing PY_ prefixes in structmember.h
In-Reply-To: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>
References: <CANc-5Uw_=DaVzcwx8eBTs245Fp85upWtGdc6DjTpzF0Q_HBjYg@mail.gmail.com>
Message-ID: <nsu6us$2lu$1@blaine.gmane.org>

On 03.10.16 16:37, Skip Montanaro wrote:
> I'm sure this has a simple explanation (and is probably only of
> historical interest at this point), but ...
>
> While starting to port the Python Sybase module to Python 3, among other
> hurdles, I noticed that RO is no longer defined. Looking in
> structmember.h, I see that most of the macros defined there are missing
> a PY_ prefix. Given that these are macros which are likely to be used by
> extension modules and are thus really part of the C API, shouldn't they
> have PY_ or _PY_ prefixes? This file got its start in 1990, so would
> surely have been around for the great renaming
> <http://python-history.blogspot.com/2009/03/great-or-grand-renaming.html>.
> Looking at the documentation on defining new types, I saw no mention of
> these peculiarly named constants, though they are clearly documented.

structmember.h is not included in Python.h. I think it was not 
considered as the part of public C API.

But PyMemberDef and constants are documented and structmember.h is used 
in C API examples.

There are other problems with these constants. Opened an issue for this: 
http://bugs.python.org/issue28349 .



From michael at felt.demon.nl  Thu Oct  6 06:30:45 2016
From: michael at felt.demon.nl (Michael Felt)
Date: Thu, 6 Oct 2016 12:30:45 +0200
Subject: [Python-Dev] C99
In-Reply-To: <m2shwtnzhy.fsf@news.realpath.org>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <CA+eR4cF56FBdd86inH8NBzPLcL4W+Nb2ikixBn=xUsoOCmhhLg@mail.gmail.com>
 <m2shwtnzhy.fsf@news.realpath.org>
Message-ID: <457d5f0d-c5a1-5699-d705-fdf2985573d4@felt.demon.nl>



On 04-Jun-16 10:12, Sebastian Krause wrote:
>> Sounds good for features that are well-supported by compilers that
>> >people use. (Are there other compilers used than just GCC and MSVC?)
Just thought I'd mention that "little ole me" is using IBM vac on AIX 
rather than gcc.
This has more to do with maintaining the gcc rte requirements of 
anything built by gcc rather compiler features.

There is one gcc compiler feature that has proved a show-stopper - 
constructor - in glibc (Gnome). There they also make an exception for 
Windows, but not for "other" compilers.

FYI: calling the IBM C compiler as c99 magically makes it c99 compliant, 
as xlc makes it c99 plus several extensions - and take note - as cc it 
is "pre-c89". In other words, calling IBM compiler as "cc" is not a wise 
thing to do - I recommend to anyone listing to use either "xlc" or "xlc_r".

Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161006/b34a1961/attachment.html>

From njs at pobox.com  Fri Oct  7 01:10:54 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 6 Oct 2016 22:10:54 -0700
Subject: [Python-Dev] C99
In-Reply-To: <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
Message-ID: <CAPJVwB=o8iD04VggM9Trp7zGbmEr2LR2Tr6p2Chpyty_+TJnMw@mail.gmail.com>

On Fri, Aug 5, 2016 at 8:02 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 6 August 2016 at 12:15, Steve Dower <steve.dower at python.org> wrote:
>> FYI, it's MSVC 14.0 (which is included in VS 2015).
>>
>> Personally I'd like to see it restricted to the common subset of C99 and
>> some version of C++ (which is obviously mostly C and includes no C++),
>> because I know there are a few things in C99 only that are very obscure
>> because they aren't also in C++.
>
> As a pragmatic requirement, what if we went with:
>
> - must compile with the Python 3.5 Windows compiler (so MSVC 14.0)
> - must compile with the Python 3.5 Mac OS X compiler (some version of clang?)
> - must compile with the manylinux1 gcc compiler (gcc 4.8.2, as per PEP 513)
>
> The reason I like the pragmatic definition is that it's one we can
> readily automate by ensuring these 3 configurations are always present
> in the stable buildbot fleet, while the formal standards based version
> is technically more precise, but not in a way we can easily test.

This approach makes a lot of sense to me. In particular it seems like
it can avoid getting bogged down in a lot of questions about the
theoretical idea of C99, which is often somewhat unclear.

I wrote a little test file for all the C99 features that were
mentioned in this thread (I think):
  https://gist.github.com/njsmith/f666671de31f35644bb91a815c9cad75

Ned reports that our OS X compiler is currently GCC 4.2, and that this
doesn't officially support "C99" -- but I just checked and it happily
builds that file :-). (Judging from [1], it looks like the major
feature missing in 4.2 was support for C99's more obscure inlining
semantics; but I think we only really care about 'static inline'
specifically, and that appears to be unchanged [2].)

> Phrasing the policy that way would mean moving future iterations of
> the manylinux spec out of distutils-sig only territory into one which
> needs to be reviewed by both distutils-sig and python-dev (since it
> would now also modify CPython's compiler compatibility requirements in
> PEP 7), but I think that's a reasonable position to adopt, and
> unlikely to cause any major problems (manylinux is a *trailing* edge
> definition that is only likely to get updated as the oldest enterprise
> Linux definitions drop out of commercial support)

With my manylinux maintainer hat on, I don't think there'd be any
problem with bringing python-dev into those discussions.

But note that technically, the manylinux1 spec doesn't say what
compiler you can use; it says that whatever compiler you use has to
generate binaries that work on CentOS 5. (Why CentOS 5? Because it's
the oldest non-EOL linux distribution in common use; manylinux2 will
presumably bump the requirement to CentOS 6 once CentOS 5 goes EOL).

The reason this matters is that the official vendor compiler on RHEL 5
is gcc 4.1, but there's also a separately-distributed version of gcc
4.8.2 that can target it. If a packager trying to build manylinux
wheels wants a more modern gcc, then it's reasonable to ask them to
get this patched gcc. But for an end-user who's just trying to build
CPython on their machine, you might or might not consider this an
acceptable request -- maybe CPython wants to work on default vendor
compiler to imposing that on users.

And in practice this almost certainly doesn't matter -- the only
reason people jump through hoops to get gcc 4.8 is for its improved
C++ support. I just tried my c99 test file on CentOS 5's default gcc
4.1 and it was fine. And since RHEL 5 is going EOL in March 2017, for
3.6 purposes we only care about RHEL 6+, which ships with gcc 4.4,
which is even less of a worry.

tl;dr: for purposes of 3.6 on linux, "CPython has to build on gcc 4.4"
should be a fine rule that works for everyone. But I guess we probably
also want a CentOS 5+gcc 4.1 buildbot for testing 3.5 point releases.

-n

[1] https://gcc.gnu.org/c99status.html
[2] "The [different versions] of inlining behave similarly ... when
the inline keyword is used on a static function" --
https://gcc.gnu.org/onlinedocs/gcc/Inline.html

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Fri Oct  7 01:33:08 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Thu, 6 Oct 2016 22:33:08 -0700
Subject: [Python-Dev] C99
In-Reply-To: <CAPJVwB=o8iD04VggM9Trp7zGbmEr2LR2Tr6p2Chpyty_+TJnMw@mail.gmail.com>
References: <1465020691.2818312.627646289.6A6F4D74@webmail.messagingengine.com>
 <niv32b$d7k$1@ger.gmane.org>
 <CAP7+vJKNu58iKeXJ-=pZeXp_c0B=JERPgHjxYiQTf9X7sbJRqA@mail.gmail.com>
 <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <CAPJVwB=o8iD04VggM9Trp7zGbmEr2LR2Tr6p2Chpyty_+TJnMw@mail.gmail.com>
Message-ID: <CAPJVwBm84FYTjWC0eLDcFQtcmmRFcL2SzhaHL_QPT4bzfm0vGw@mail.gmail.com>

On Thu, Oct 6, 2016 at 10:10 PM, Nathaniel Smith <njs at pobox.com> wrote:
[...]
> And in practice this almost certainly doesn't matter -- the only
> reason people jump through hoops to get gcc 4.8 is for its improved
> C++ support. I just tried my c99 test file on CentOS 5's default gcc
> 4.1 and it was fine. And since RHEL 5 is going EOL in March 2017, for
> 3.6 purposes we only care about RHEL 6+, which ships with gcc 4.4,
> which is even less of a worry.
>
> tl;dr: for purposes of 3.6 on linux, "CPython has to build on gcc 4.4"
> should be a fine rule that works for everyone. But I guess we probably
> also want a CentOS 5+gcc 4.1 buildbot for testing 3.5 point releases.

Doh, I have an off-by-one error, and wrote 3.5/3.6 where I meant 3.6/3.7.

If the change is on the table for 3.6, then I guess I'd suggest
continuing to support gcc 4.1 through the 3.6 support lifecycle, since
I believe CentOS/RHEL 5 are considered supported platforms for 3.6,
and the burden of doing this is probably very small. In particular, it
doesn't seem to block any C99 features that we care about.

(Is the list of supported platforms for Linux documented anywhere? PEP
11 documents the rule for Windows, and I'm assuming that the same rule
applies to RHEL.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From storchaka at gmail.com  Fri Oct  7 09:36:55 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 7 Oct 2016 16:36:55 +0300
Subject: [Python-Dev] Is explicit registration of Iterators needed?
Message-ID: <nt88dn$k72$1@blaine.gmane.org>

A number of builtin iterator classes (but not all builtin iterator 
classes) are registered with the Iterator ABC in 
Lib/_collections_abc.py.  But isinstance(it, Iterable) check works 
without explicit registration, because Iterable has __subclasshook__ 
that checks iterator methods.  Is there a need in explicit 
registrations?  Or their can be safely removed?


From guido at python.org  Fri Oct  7 10:37:58 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Oct 2016 07:37:58 -0700
Subject: [Python-Dev] Is explicit registration of Iterators needed?
In-Reply-To: <nt88dn$k72$1@blaine.gmane.org>
References: <nt88dn$k72$1@blaine.gmane.org>
Message-ID: <CAP7+vJKf=1ER8Y+i5c6QjvJ3T3oMtxUohhh1FtiO52XEDRURhQ@mail.gmail.com>

On Fri, Oct 7, 2016 at 6:36 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> A number of builtin iterator classes (but not all builtin iterator classes)
> are registered with the Iterator ABC in Lib/_collections_abc.py.  But
> isinstance(it, Iterable) check works without explicit registration, because
> Iterable has __subclasshook__ that checks iterator methods.  Is there a need
> in explicit registrations?  Or their can be safely removed?

The preferred apprach is actually inheritance; registration comes
next; the __subclasshook__ is a final compromise to the tradition of
duck typing. I think the registrations should stay.

-- 
--Guido van Rossum (python.org/~guido)

From storchaka at gmail.com  Fri Oct  7 10:47:24 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Fri, 7 Oct 2016 17:47:24 +0300
Subject: [Python-Dev] Is explicit registration of Iterators needed?
In-Reply-To: <CAP7+vJKf=1ER8Y+i5c6QjvJ3T3oMtxUohhh1FtiO52XEDRURhQ@mail.gmail.com>
References: <nt88dn$k72$1@blaine.gmane.org>
 <CAP7+vJKf=1ER8Y+i5c6QjvJ3T3oMtxUohhh1FtiO52XEDRURhQ@mail.gmail.com>
Message-ID: <nt8chq$s15$1@blaine.gmane.org>

On 07.10.16 17:37, Guido van Rossum wrote:
> On Fri, Oct 7, 2016 at 6:36 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> A number of builtin iterator classes (but not all builtin iterator classes)
>> are registered with the Iterator ABC in Lib/_collections_abc.py.  But
>> isinstance(it, Iterable) check works without explicit registration, because
>> Iterable has __subclasshook__ that checks iterator methods.  Is there a need
>> in explicit registrations?  Or their can be safely removed?
>
> The preferred apprach is actually inheritance; registration comes
> next; the __subclasshook__ is a final compromise to the tradition of
> duck typing. I think the registrations should stay.

Should we register missed builtin iterators? For example longrange_iterator.


From guido at python.org  Fri Oct  7 11:08:29 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 7 Oct 2016 08:08:29 -0700
Subject: [Python-Dev] Is explicit registration of Iterators needed?
In-Reply-To: <nt8chq$s15$1@blaine.gmane.org>
References: <nt88dn$k72$1@blaine.gmane.org>
 <CAP7+vJKf=1ER8Y+i5c6QjvJ3T3oMtxUohhh1FtiO52XEDRURhQ@mail.gmail.com>
 <nt8chq$s15$1@blaine.gmane.org>
Message-ID: <CAP7+vJKbHPoBpykaijX1d7wRyjXc3Oy3zjhjyS09bq=WSOJRiA@mail.gmail.com>

On Fri, Oct 7, 2016 at 7:47 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 07.10.16 17:37, Guido van Rossum wrote:
>>
>> On Fri, Oct 7, 2016 at 6:36 AM, Serhiy Storchaka <storchaka at gmail.com>
>> wrote:
>>>
>>> A number of builtin iterator classes (but not all builtin iterator
>>> classes)
>>> are registered with the Iterator ABC in Lib/_collections_abc.py.  But
>>> isinstance(it, Iterable) check works without explicit registration,
>>> because
>>> Iterable has __subclasshook__ that checks iterator methods.  Is there a
>>> need
>>> in explicit registrations?  Or their can be safely removed?
>>
>>
>> The preferred apprach is actually inheritance; registration comes
>> next; the __subclasshook__ is a final compromise to the tradition of
>> duck typing. I think the registrations should stay.
>
>
> Should we register missed builtin iterators? For example longrange_iterator.

I don't feel strongly about this either way. Let sleeping dogs lie,
etc. (Is this related to issue 26906?)

-- 
--Guido van Rossum (python.org/~guido)

From status at bugs.python.org  Fri Oct  7 12:08:49 2016
From: status at bugs.python.org (Python tracker)
Date: Fri,  7 Oct 2016 18:08:49 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20161007160849.55A65560D4@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-09-30 - 2016-10-07)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5517 ( -2)
  closed 34624 (+69)
  total  40141 (+67)

Open issues with patches: 2383 


Issues opened (33)
==================

#6280: calendar.timegm() belongs in time module, next to time.gmtime(
http://bugs.python.org/issue6280  reopened by belopolsky

#28315: incorrect "in ?" output in 'divide' example at "Defining Clean
http://bugs.python.org/issue28315  reopened by terry.reedy

#28320: Hostname validation is False by default in imaplib
http://bugs.python.org/issue28320  opened by maciej.szulik

#28326: multiprocessing.Process depends on sys.stdout being open
http://bugs.python.org/issue28326  opened by Valentin.Lorentz

#28327: statistics.geometric_mean gives incorrect results for mixed in
http://bugs.python.org/issue28327  opened by mark.dickinson

#28328: statistics.geometric_mean has no tests. Defer to 3.7?
http://bugs.python.org/issue28328  opened by mark.dickinson

#28330: Use simpler and faster sched.Event definition
http://bugs.python.org/issue28330  opened by Satoru Logic

#28331: "CPython implementation detail:" is removed when contents is t
http://bugs.python.org/issue28331  opened by inada.naoki

#28333: input() with Unicode prompt produces mojibake on Windows
http://bugs.python.org/issue28333  opened by Drekin

#28334: netrc does not work if $HOME is not set
http://bugs.python.org/issue28334  opened by Dimitri Merejkowsky

#28339: "TypeError: Parameterized generics cannot be used with class o
http://bugs.python.org/issue28339  opened by Arfrever

#28340: TextIOWrapper.tell extremely slow
http://bugs.python.org/issue28340  opened by rmalouf

#28343: Bad encoding alias cp936 -> gbk: euro sign
http://bugs.python.org/issue28343  opened by Artoria2e5

#28344: Python 3.5.2 installer hangs when run in session 0
http://bugs.python.org/issue28344  opened by spooja

#28351: statistics.geometric_mean can enter infinite loop for Decimal 
http://bugs.python.org/issue28351  opened by mark.dickinson

#28352: winfo_pathname(..) | window id "xyz" doesn't exist in this app
http://bugs.python.org/issue28352  opened by Tyler Sweeden

#28353: os.fwalk() unhandled exception when error occurs accessing sym
http://bugs.python.org/issue28353  opened by Samson Lee

#28355: wsgiref simple_server PATH_INFO treats slashes and %2F the sam
http://bugs.python.org/issue28355  opened by tdammers

#28356: Windows: os.rename different in python 2.7.12 and python 3.5.2
http://bugs.python.org/issue28356  opened by stephan

#28364: Windows - Popen (subprocess.py) does not call _handle.Close() 
http://bugs.python.org/issue28364  opened by Mateusz Klatt

#28367: Add more standard baud rate constants to "termios"
http://bugs.python.org/issue28367  opened by Andrey Smirnov

#28373: input() prints to original stdout even if sys.stdout is wrappe
http://bugs.python.org/issue28373  opened by wiggin15

#28374: SyntaxError: invalid token in python2.7/test/test_grammar.py
http://bugs.python.org/issue28374  opened by welker

#28375: cgi.py spam in Apache server logs
http://bugs.python.org/issue28375  opened by srittau

#28376: rangeiter_new fails when creating a range of step 0
http://bugs.python.org/issue28376  opened by Oren Milman

#28377: struct.unpack of Bool
http://bugs.python.org/issue28377  opened by Weihong Guan

#28378: urllib2 does not handle cookies with `,`
http://bugs.python.org/issue28378  opened by Grzegorz Sikorski

#28379: PyUnicode_CopyCharacters could lead to undefined behaviour
http://bugs.python.org/issue28379  opened by xiang.zhang

#28381: Add a "starcaller" function
http://bugs.python.org/issue28381  opened by josh.r

#28382: Possible deadlock after many multiprocessing.Process are launc
http://bugs.python.org/issue28382  opened by Hadhoke

#28383: __hash__ documentation recommends naive XOR to combine but thi
http://bugs.python.org/issue28383  opened by Kevin.Norris

#28384: hmac cannot be used with shake algorithms
http://bugs.python.org/issue28384  opened by minrk

#28385: Bytes objects should reject all formatting codes with an error
http://bugs.python.org/issue28385  opened by mmarkk



Most recent 15 issues with no replies (15)
==========================================

#28384: hmac cannot be used with shake algorithms
http://bugs.python.org/issue28384

#28383: __hash__ documentation recommends naive XOR to combine but thi
http://bugs.python.org/issue28383

#28382: Possible deadlock after many multiprocessing.Process are launc
http://bugs.python.org/issue28382

#28375: cgi.py spam in Apache server logs
http://bugs.python.org/issue28375

#28367: Add more standard baud rate constants to "termios"
http://bugs.python.org/issue28367

#28355: wsgiref simple_server PATH_INFO treats slashes and %2F the sam
http://bugs.python.org/issue28355

#28351: statistics.geometric_mean can enter infinite loop for Decimal 
http://bugs.python.org/issue28351

#28343: Bad encoding alias cp936 -> gbk: euro sign
http://bugs.python.org/issue28343

#28340: TextIOWrapper.tell extremely slow
http://bugs.python.org/issue28340

#28333: input() with Unicode prompt produces mojibake on Windows
http://bugs.python.org/issue28333

#28320: Hostname validation is False by default in imaplib
http://bugs.python.org/issue28320

#28317: Improve support of FORMAT_VALUE in dis
http://bugs.python.org/issue28317

#28312: Minor change - more direct hint re: multiple machine sizes and
http://bugs.python.org/issue28312

#28309: Accelerate string.Template by using formatted string literals
http://bugs.python.org/issue28309

#28287: Refactor subprocess.Popen to let a subclass handle IO asynchro
http://bugs.python.org/issue28287



Most recent 15 issues waiting for review (15)
=============================================

#28379: PyUnicode_CopyCharacters could lead to undefined behaviour
http://bugs.python.org/issue28379

#28376: rangeiter_new fails when creating a range of step 0
http://bugs.python.org/issue28376

#28367: Add more standard baud rate constants to "termios"
http://bugs.python.org/issue28367

#28353: os.fwalk() unhandled exception when error occurs accessing sym
http://bugs.python.org/issue28353

#28334: netrc does not work if $HOME is not set
http://bugs.python.org/issue28334

#28331: "CPython implementation detail:" is removed when contents is t
http://bugs.python.org/issue28331

#28330: Use simpler and faster sched.Event definition
http://bugs.python.org/issue28330

#28328: statistics.geometric_mean has no tests. Defer to 3.7?
http://bugs.python.org/issue28328

#28327: statistics.geometric_mean gives incorrect results for mixed in
http://bugs.python.org/issue28327

#28326: multiprocessing.Process depends on sys.stdout being open
http://bugs.python.org/issue28326

#28317: Improve support of FORMAT_VALUE in dis
http://bugs.python.org/issue28317

#28315: incorrect "in ?" output in 'divide' example at "Defining Clean
http://bugs.python.org/issue28315

#28314: ElementTree: Element.getiterator(tag) broken in 3.6
http://bugs.python.org/issue28314

#28309: Accelerate string.Template by using formatted string literals
http://bugs.python.org/issue28309

#28298: can't set big int-like objects to items in array 'Q', 'L' and 
http://bugs.python.org/issue28298



Top 10 most discussed issues (10)
=================================

#10716: Modernize pydoc to use better HTML and separate CSS
http://bugs.python.org/issue10716  11 msgs

#28259: Ctypes bug windows
http://bugs.python.org/issue28259  10 msgs

#2897: Deprecate structmember.h
http://bugs.python.org/issue2897   8 msgs

#25720: Fix curses module compilation with ncurses6
http://bugs.python.org/issue25720   8 msgs

#27972: Confusing error during cyclic yield
http://bugs.python.org/issue27972   7 msgs

#28328: statistics.geometric_mean has no tests. Defer to 3.7?
http://bugs.python.org/issue28328   7 msgs

#28376: rangeiter_new fails when creating a range of step 0
http://bugs.python.org/issue28376   7 msgs

#26439: ctypes.util.find_library fails when ldconfig/glibc not availab
http://bugs.python.org/issue26439   6 msgs

#28353: os.fwalk() unhandled exception when error occurs accessing sym
http://bugs.python.org/issue28353   6 msgs

#28356: Windows: os.rename different in python 2.7.12 and python 3.5.2
http://bugs.python.org/issue28356   6 msgs



Issues closed (66)
==================

#12274: "Print window" menu on IDLE aborts whole application
http://bugs.python.org/issue12274  closed by berker.peksag

#13756: Python3.2.2 make fail on cygwin
http://bugs.python.org/issue13756  closed by zach.ware

#17345: Portable and extended type specifiers for array module
http://bugs.python.org/issue17345  closed by berker.peksag

#20254: Duplicate bytearray test on test_socket.py
http://bugs.python.org/issue20254  closed by berker.peksag

#21085: Cygwin does not provide siginfo_t.si_band
http://bugs.python.org/issue21085  closed by zach.ware

#21626: Add options width and compact to pickle cli
http://bugs.python.org/issue21626  closed by berker.peksag

#24065: Outdated *_RESTRICTED flags in structmember.h
http://bugs.python.org/issue24065  closed by belopolsky

#24697: Add CoroutineReturn and CoroutineExit builtin exceptions for c
http://bugs.python.org/issue24697  closed by yselivanov

#24870: Optimize ascii and latin1 decoder with surrogateescape and sur
http://bugs.python.org/issue24870  closed by inada.naoki

#26617: Assertion failed in gc with __del__ and weakref
http://bugs.python.org/issue26617  closed by python-dev

#27168: Comprehensions and await need more unittests
http://bugs.python.org/issue27168  closed by yselivanov

#27231: Support the fspath protocol in the posixpath module
http://bugs.python.org/issue27231  closed by berker.peksag

#27358: BUILD_MAP_UNPACK_WITH_CALL is slow
http://bugs.python.org/issue27358  closed by serhiy.storchaka

#27392: Add a server_side keyword parameter to create_connection
http://bugs.python.org/issue27392  closed by yselivanov

#27511: Add PathLike objects support to BZ2File
http://bugs.python.org/issue27511  closed by berker.peksag

#27859: argparse - subparsers does not retain namespace
http://bugs.python.org/issue27859  closed by ned.deily

#28201: dict: perturb shift should be done when first conflict
http://bugs.python.org/issue28201  closed by inada.naoki

#28217: Add interactive console tests
http://bugs.python.org/issue28217  closed by steve.dower

#28218: Windows docs have wrong versionadded description
http://bugs.python.org/issue28218  closed by steve.dower

#28222: test_distutils fails
http://bugs.python.org/issue28222  closed by berker.peksag

#28225: bz2 does not support pathlib
http://bugs.python.org/issue28225  closed by berker.peksag

#28226: compileall does not support pathlib
http://bugs.python.org/issue28226  closed by berker.peksag

#28227: gzip does not support pathlib
http://bugs.python.org/issue28227  closed by berker.peksag

#28228: imghdr does not support pathlib
http://bugs.python.org/issue28228  closed by berker.peksag

#28229: lzma does not support pathlib
http://bugs.python.org/issue28229  closed by berker.peksag

#28251: Help manuals do not appear in Windows search
http://bugs.python.org/issue28251  closed by steve.dower

#28277: ./Modules/_io/_iomodule.c build failure on AIX (beta1) while (
http://bugs.python.org/issue28277  closed by berker.peksag

#28294: HTTPServer server.py assumes sys.stderr != None
http://bugs.python.org/issue28294  closed by r.david.murray

#28295: PyUnicode_AsUCS4 doc and impl conflict on exception
http://bugs.python.org/issue28295  closed by serhiy.storchaka

#28301: python3.4-config --extension-suffix reports '.cpython-34m.so'
http://bugs.python.org/issue28301  closed by berker.peksag

#28318: Python unittest.mock.mock_calls stores references to arguments
http://bugs.python.org/issue28318  closed by michael.foord

#28319: typo in lzma module documentation
http://bugs.python.org/issue28319  closed by berker.peksag

#28321: Plistlib: Half of the double width characters are missing when
http://bugs.python.org/issue28321  closed by serhiy.storchaka

#28322: chain.__setstate__ Type Confusion
http://bugs.python.org/issue28322  closed by serhiy.storchaka

#28323: Remove MacOS 9-specific codes from exit() and quit()
http://bugs.python.org/issue28323  closed by ned.deily

#28324: Clean up MacOS 9-specific description in the docstring of os.p
http://bugs.python.org/issue28324  closed by ned.deily

#28325: Remove MacOS 9-specific module macurl2path.py
http://bugs.python.org/issue28325  closed by ned.deily

#28329: Add support for customizing scheduler's timefunc and delayfunc
http://bugs.python.org/issue28329  closed by rhettinger

#28332: silent truncations in socket.htons and socket.ntohs
http://bugs.python.org/issue28332  closed by serhiy.storchaka

#28335: Exception reporting in `logging.config`
http://bugs.python.org/issue28335  closed by python-dev

#28336: Slicing (operation) is not symmetrical with respect to suffixe
http://bugs.python.org/issue28336  closed by serhiy.storchaka

#28337: Segfault in _struct.unpack_iterator
http://bugs.python.org/issue28337  closed by zach.ware

#28338: test_pdb doctests have been removed from its test suite
http://bugs.python.org/issue28338  closed by xdegaye

#28341: cpython tip fails to build on Mac OS X 10.11.6 w/ XCode 8.0
http://bugs.python.org/issue28341  closed by ned.deily

#28342: OSX 10.12 crash in urllib.request getproxies_macosx_sysconf an
http://bugs.python.org/issue28342  closed by ned.deily

#28345: 8/3 is calculated incorrectly
http://bugs.python.org/issue28345  closed by benjamin.peterson

#28346: 3.6 source doesn't build on centos 6
http://bugs.python.org/issue28346  closed by lurker10

#28348: Doc typo in asyncio.Task
http://bugs.python.org/issue28348  closed by berker.peksag

#28349: Issues with PyMemberDef flags
http://bugs.python.org/issue28349  closed by belopolsky

#28350: Interning string constants with null character
http://bugs.python.org/issue28350  closed by serhiy.storchaka

#28354: DeprecationWarning not reported for invalid escape sequences
http://bugs.python.org/issue28354  closed by serhiy.storchaka

#28357: Spam
http://bugs.python.org/issue28357  closed by zach.ware

#28358: Spam
http://bugs.python.org/issue28358  closed by berker.peksag

#28359: Spam
http://bugs.python.org/issue28359  closed by berker.peksag

#28360: Spam
http://bugs.python.org/issue28360  closed by berker.peksag

#28361: BETA report: Python3.6 names pip pip3.6 (and why is the other 
http://bugs.python.org/issue28361  closed by zach.ware

#28362: Deprecation warning doesn't stand out enough
http://bugs.python.org/issue28362  closed by r.david.murray

#28363: -D_LARGE_FILES not exported to modules (for AIX)
http://bugs.python.org/issue28363  closed by Michael.Felt

#28365: IDLE: don't offer to save text files as .py
http://bugs.python.org/issue28365  closed by terry.reedy

#28366: Syntax issue
http://bugs.python.org/issue28366  closed by zach.ware

#28368: Refuse monitoring processes if the child watcher has no loop a
http://bugs.python.org/issue28368  closed by yselivanov

#28369: Raise RuntimeError when transport's FD is used with add_reader
http://bugs.python.org/issue28369  closed by yselivanov

#28370: Speedup asyncio.StreamReader.readexactly
http://bugs.python.org/issue28370  closed by yselivanov

#28371: Deprecate passing asyncio.Handles to run_in_executor
http://bugs.python.org/issue28371  closed by yselivanov

#28372: Fix asyncio to support formatting of non-python coroutines
http://bugs.python.org/issue28372  closed by yselivanov

#28380: Mock functions with autospec don't support assert_called_once,
http://bugs.python.org/issue28380  closed by gregory.p.smith

From levkivskyi at gmail.com  Fri Oct  7 18:52:50 2016
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sat, 8 Oct 2016 00:52:50 +0200
Subject: [Python-Dev] Is explicit registration of Iterators needed?
In-Reply-To: <CAP7+vJKbHPoBpykaijX1d7wRyjXc3Oy3zjhjyS09bq=WSOJRiA@mail.gmail.com>
References: <nt88dn$k72$1@blaine.gmane.org>
 <CAP7+vJKf=1ER8Y+i5c6QjvJ3T3oMtxUohhh1FtiO52XEDRURhQ@mail.gmail.com>
 <nt8chq$s15$1@blaine.gmane.org>
 <CAP7+vJKbHPoBpykaijX1d7wRyjXc3Oy3zjhjyS09bq=WSOJRiA@mail.gmail.com>
Message-ID: <CAOMjWkm7tKJ+rHpUPLYONEvqEDR1w9pP5JQo-XzdKw_y5OjXPA@mail.gmail.com>

On 7 October 2016 at 17:08, Guido van Rossum <guido at python.org> wrote:

> On Fri, Oct 7, 2016 at 7:47 AM, Serhiy Storchaka <storchaka at gmail.com>
> wrote:
> > On 07.10.16 17:37, Guido van Rossum wrote:
> >>
> >> On Fri, Oct 7, 2016 at 6:36 AM, Serhiy Storchaka <storchaka at gmail.com>
> >> wrote:
> >>>
> >>> A number of builtin iterator classes (but not all builtin iterator
> >>> classes)
> >>> are registered with the Iterator ABC in Lib/_collections_abc.py.  But
> >>> isinstance(it, Iterable) check works without explicit registration,
> >>> because
> >>> Iterable has __subclasshook__ that checks iterator methods.  Is there a
> >>> need
> >>> in explicit registrations?  Or their can be safely removed?
> >>
> >>
> >> The preferred apprach is actually inheritance; registration comes
> >> next; the __subclasshook__ is a final compromise to the tradition of
> >> duck typing. I think the registrations should stay.
>

I have a question about the registration of builtins. Currently, typing.py
contains this line:

    ByteString.register(type(memoryview(b'')))

But there are two test lines in test_collections.py

    self.assertNotIsInstance(memoryview(b""), ByteString)
    self.assertFalse(issubclass(memoryview, ByteString))

This looks like a contradiction. Which one is right?
Should these tests be removed or the registration in typing.py?

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161008/b87a2fac/attachment.html>

From storchaka at gmail.com  Sat Oct  8 05:46:21 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 8 Oct 2016 12:46:21 +0300
Subject: [Python-Dev] Is explicit registration of Iterators needed?
In-Reply-To: <CAP7+vJKbHPoBpykaijX1d7wRyjXc3Oy3zjhjyS09bq=WSOJRiA@mail.gmail.com>
References: <nt88dn$k72$1@blaine.gmane.org>
 <CAP7+vJKf=1ER8Y+i5c6QjvJ3T3oMtxUohhh1FtiO52XEDRURhQ@mail.gmail.com>
 <nt8chq$s15$1@blaine.gmane.org>
 <CAP7+vJKbHPoBpykaijX1d7wRyjXc3Oy3zjhjyS09bq=WSOJRiA@mail.gmail.com>
Message-ID: <ntaf9b$lqa$1@blaine.gmane.org>

On 07.10.16 18:08, Guido van Rossum wrote:
> On Fri, Oct 7, 2016 at 7:47 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> Should we register missed builtin iterators? For example longrange_iterator.
>
> I don't feel strongly about this either way. Let sleeping dogs lie,
> etc. (Is this related to issue 26906?)

Not directly. If remove explicit iterator registrations some tests 
become failing due to the bug 26906. After fixing issue26906 the tests 
are passed again. Thus the bug 26906 could be found earlier if iterators 
were not registered.


From storchaka at gmail.com  Sat Oct  8 06:01:27 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sat, 8 Oct 2016 13:01:27 +0300
Subject: [Python-Dev] Check dict implementation details
Message-ID: <ntag5m$kdb$1@blaine.gmane.org>

Since dict is ordered in CPython 3.6, it can be used instead of 
OrderedDict in some places (e.g. for implementing simple limited 
caches). But since this is implementation detail, it can't be used in 
the stdlib unconditionally. Needed a way to check whether dict is ordered.

Actually there are two levels of "ordering".

1. Dict without deletions is iterated in the order of adding items. 
Raymond's original compact dict implementation satisfied this claim.

2. In addition the order is preserved after deletion operations. Naoki's 
implementation satisfies this more strong claim.


From steve at pearwood.info  Sat Oct  8 12:23:43 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Sun, 9 Oct 2016 03:23:43 +1100
Subject: [Python-Dev] C99
In-Reply-To: <CAPJVwB=o8iD04VggM9Trp7zGbmEr2LR2Tr6p2Chpyty_+TJnMw@mail.gmail.com>
References: <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <CAPJVwB=o8iD04VggM9Trp7zGbmEr2LR2Tr6p2Chpyty_+TJnMw@mail.gmail.com>
Message-ID: <20161008162342.GL22471@ando.pearwood.info>

On Thu, Oct 06, 2016 at 10:10:54PM -0700, Nathaniel Smith wrote:

> The reason this matters is that the official vendor compiler on RHEL 5
> is gcc 4.1, but there's also a separately-distributed version of gcc
> 4.8.2 that can target it. 

Where can I get that 4.8 for RHEL 5?

I'm using Centos 5, which ought to be the same as RHEL 5, and the 
standard gcc is 4.1, with 4.4 available through yum. If 4.8 is available 
anywhere, I haven't been able to find it. And as far as I can see, 3.6 
won't build under either 4.1 or 4.4 on Centos 5.


> If a packager trying to build manylinux
> wheels wants a more modern gcc, then it's reasonable to ask them to
> get this patched gcc. But for an end-user who's just trying to build
> CPython on their machine, you might or might not consider this an
> acceptable request -- maybe CPython wants to work on default vendor
> compiler to imposing that on users.
> 
> And in practice this almost certainly doesn't matter -- the only
> reason people jump through hoops to get gcc 4.8 is for its improved
> C++ support. I just tried my c99 test file on CentOS 5's default gcc
> 4.1 and it was fine. 

Can you try building Python 3.6? Because it fails for me, and the 
discussion here:

http://bugs.python.org/issue28092

concluded that 4.1 is not supported and I'm right out of luck until I 
can upgrade.




-- 
Steve

From njs at pobox.com  Sat Oct  8 12:42:34 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 8 Oct 2016 09:42:34 -0700
Subject: [Python-Dev] C99
In-Reply-To: <20161008162342.GL22471@ando.pearwood.info>
References: <CAMpsgwbq6e9Va9koS884LuR5Ly9wmoDhuv=CTGJMONiO152XFw@mail.gmail.com>
 <837505337487020527.659630sturla.molden-gmail.com@news.gmane.org>
 <CAP7+vJJuSreWFH=ojBphsYsBXt8cz2zoc3=6Pmh=KPu7E-VOaA@mail.gmail.com>
 <CAMpsgwaj1aLDTzUrHYczY65m38yKT9aCvZ+R0mBpRmTrSqESZQ@mail.gmail.com>
 <CAP1=2W4HvYCcbA0ha+L4c0TO+byNOQ9V2VvEDwaTcK1=aWGpsQ@mail.gmail.com>
 <CAP7+vJ+XQSGQqWuWjSnqeveDxHrKThHPK3PTkFobph-Ra3_D0A@mail.gmail.com>
 <CAP1=2W4Kx5P78i30qky_CfQwyC52Wh1cGzXbTsH3CD7cAo0QTA@mail.gmail.com>
 <E1bVr9P-0003d4-Kf@se2-syd.hostedmail.net.au>
 <CADiSq7f5m8ZMkdnyh67aqEKREK4j=6qJEuRqWigj5Y0qdUdnrQ@mail.gmail.com>
 <CAPJVwB=o8iD04VggM9Trp7zGbmEr2LR2Tr6p2Chpyty_+TJnMw@mail.gmail.com>
 <20161008162342.GL22471@ando.pearwood.info>
Message-ID: <CAPJVwB=f_=f1vs7bPROg+0kJd7v9qcfyGWMy_WA+xo4OSs_GUA@mail.gmail.com>

On Sat, Oct 8, 2016 at 9:23 AM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Thu, Oct 06, 2016 at 10:10:54PM -0700, Nathaniel Smith wrote:
>
>> The reason this matters is that the official vendor compiler on RHEL 5
>> is gcc 4.1, but there's also a separately-distributed version of gcc
>> 4.8.2 that can target it.
>
> Where can I get that 4.8 for RHEL 5?
>
> I'm using Centos 5, which ought to be the same as RHEL 5, and the
> standard gcc is 4.1, with 4.4 available through yum. If 4.8 is available
> anywhere, I haven't been able to find it. And as far as I can see, 3.6
> won't build under either 4.1 or 4.4 on Centos 5.

It's RH's "devtoolset" release. The build scripts for the manylinux
docker images demonstrate how to install them:

https://github.com/pypa/manylinux/blob/master/docker/build_scripts/build.sh#L38

(Unfortunately this involves installing them from a random CentOS
dev's home directory over plain-HTTP, but that's the only way to get
them if you don't have the appropriate RHEL subscription -- see
https://github.com/pypa/manylinux/issues/46#issuecomment-205054077)

>> If a packager trying to build manylinux
>> wheels wants a more modern gcc, then it's reasonable to ask them to
>> get this patched gcc. But for an end-user who's just trying to build
>> CPython on their machine, you might or might not consider this an
>> acceptable request -- maybe CPython wants to work on default vendor
>> compiler to imposing that on users.
>>
>> And in practice this almost certainly doesn't matter -- the only
>> reason people jump through hoops to get gcc 4.8 is for its improved
>> C++ support. I just tried my c99 test file on CentOS 5's default gcc
>> 4.1 and it was fine.
>
> Can you try building Python 3.6? Because it fails for me, and the
> discussion here:
>
> http://bugs.python.org/issue28092
>
> concluded that 4.1 is not supported and I'm right out of luck until I
> can upgrade.

I don't have time to follow up right now, but something like

  docker run -v $PWD:/io -it --rm quay.io/pypa/manylinux1_x86_64 /bin/bash

should drop you into a shell in a temporary centos5 VM where 'gcc' is 4.8.2.

>From the bug report it sounds like the issue is that my comments above
about 'we only care about static inline' were wrong, or at least,
Benjamin disagrees :-). I guess the revised version is
- if we insist on full C99 inline support, then that means dropping
support for building with vendor gcc on CentOS 5 and OS X (and
possibly dropping support for older OS X altogether?)
- if we are willing to restrict ourselves to 'static inline' and forgo
the other 'inline' variants, then all the other C99 things we care
about are fine and we can keep support for vendor gcc on CentOS 5 and
OS X.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From raymond.hettinger at gmail.com  Sat Oct  8 14:57:55 2016
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Sat, 8 Oct 2016 11:57:55 -0700
Subject: [Python-Dev] Check dict implementation details
In-Reply-To: <ntag5m$kdb$1@blaine.gmane.org>
References: <ntag5m$kdb$1@blaine.gmane.org>
Message-ID: <055EAD0E-D816-4486-AA00-46856E361619@gmail.com>


> On Oct 8, 2016, at 3:01 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> 
> Since dict is ordered in CPython 3.6, it can be used instead of OrderedDict in some places (e.g. for implementing simple limited caches). But since this is implementation detail, it can't be used in the stdlib unconditionally. Needed a way to check whether dict is ordered.

I think this is premature and shouldn't be done.  Instead, everyone (including us) should follow Guido's directive that for now, the right way to provide an ordered dictionary is to use collections.OrderedDict.

In the future, we may reimplement collections.OrderedDict() in terms of the new compact dict and everything that uses OrderedDict will benefit from the change.  But for now, I think you should avoid wholesale rewrites of everything in the standard library that uses OrderedDict.

It seems to me that "needed a way to check whether dict is ordered" is just an end-run around Guido's decision.  Also, from a maintenance standpoint, we don't really want two code paths.  It is better to just let the new dict implementation play itself out.   There is no reason to rush to change lots of code that is currently working just fine.


Raymond

From guido at python.org  Sat Oct  8 22:55:25 2016
From: guido at python.org (Guido van Rossum)
Date: Sat, 8 Oct 2016 19:55:25 -0700
Subject: [Python-Dev] Is explicit registration of Iterators needed?
In-Reply-To: <CAOMjWkm7tKJ+rHpUPLYONEvqEDR1w9pP5JQo-XzdKw_y5OjXPA@mail.gmail.com>
References: <nt88dn$k72$1@blaine.gmane.org>
 <CAP7+vJKf=1ER8Y+i5c6QjvJ3T3oMtxUohhh1FtiO52XEDRURhQ@mail.gmail.com>
 <nt8chq$s15$1@blaine.gmane.org>
 <CAP7+vJKbHPoBpykaijX1d7wRyjXc3Oy3zjhjyS09bq=WSOJRiA@mail.gmail.com>
 <CAOMjWkm7tKJ+rHpUPLYONEvqEDR1w9pP5JQo-XzdKw_y5OjXPA@mail.gmail.com>
Message-ID: <CAP7+vJJn+SwN6wNUFBB3YDmXppUPsNexZ-v8DaLbcYpZsbLqnA@mail.gmail.com>

On Fri, Oct 7, 2016 at 3:52 PM, Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
> I have a question about the registration of builtins. Currently, typing.py
> contains this line:
>
>     ByteString.register(type(memoryview(b'')))
>
> But there are two test lines in test_collections.py
>
>     self.assertNotIsInstance(memoryview(b""), ByteString)
>     self.assertFalse(issubclass(memoryview, ByteString))
>
> This looks like a contradiction. Which one is right?
> Should these tests be removed or the registration in typing.py?

Looks like the registration is in error. The stubs (and hence mypy)
don't consider memoryview consistent with ByteString.

-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Sat Oct  8 23:14:42 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 9 Oct 2016 13:14:42 +1000
Subject: [Python-Dev] Check dict implementation details
In-Reply-To: <ntag5m$kdb$1@blaine.gmane.org>
References: <ntag5m$kdb$1@blaine.gmane.org>
Message-ID: <CADiSq7fTnaoKoqWtrNevKE4up+56C8CzkaPvEudDq+XwUzg=_A@mail.gmail.com>

On 8 October 2016 at 20:01, Serhiy Storchaka <storchaka at gmail.com> wrote:
> Since dict is ordered in CPython 3.6, it can be used instead of OrderedDict
> in some places (e.g. for implementing simple limited caches). But since this
> is implementation detail, it can't be used in the stdlib unconditionally.
> Needed a way to check whether dict is ordered.

As Raymond suggests, if order actually matters for a given use case,
then use collections.OrderedDict unconditionally without worrying
about the behaviour of the default dict implementation.

In addition to reducing code churn and improving cross-version and
cross-implementation compatibility, doing that also lets the *reader*
of the code know that the key iteration order matters.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From leewangzhong+python at gmail.com  Sun Oct  9 19:44:29 2016
From: leewangzhong+python at gmail.com (Franklin? Lee)
Date: Sun, 9 Oct 2016 19:44:29 -0400
Subject: [Python-Dev] Check dict implementation details
In-Reply-To: <ntag5m$kdb$1@blaine.gmane.org>
References: <ntag5m$kdb$1@blaine.gmane.org>
Message-ID: <CAB_e7izUf3r++SX7M8xsfWM145GNRmexp96=E0dCo6-_odoEVg@mail.gmail.com>

On Sat, Oct 8, 2016 at 6:01 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> Since dict is ordered in CPython 3.6, it can be used instead of OrderedDict
> in some places (e.g. for implementing simple limited caches). But since this
> is implementation detail, it can't be used in the stdlib unconditionally.
> Needed a way to check whether dict is ordered.
>
> Actually there are two levels of "ordering".
>
> 1. Dict without deletions is iterated in the order of adding items.
> Raymond's original compact dict implementation satisfied this claim.
>
> 2. In addition the order is preserved after deletion operations. Naoki's
> implementation satisfies this more strong claim.

Sidenote: OrderedDict, unlike dict, is a sequential container (though
not a Sequence), so order matters when doing comparisons, and
OrderedDicts can be reverse-iterated. That might keep dict from
replacing OrderedDict in some cases. Something to keep in mind if this
topic is revisited.

From larry at hastings.org  Mon Oct 10 04:17:33 2016
From: larry at hastings.org (Larry Hastings)
Date: Mon, 10 Oct 2016 10:17:33 +0200
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference from...
 whom?
Message-ID: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>


The documentation for PyWeakref_GetObject() states:

    Return value: Borrowed reference.

    Return the referenced object from a weak reference, ref. If the
    referent is no longer live, returns Py_None.

Given that the weakref doesn't have a reference to the object--merely a 
weak reference, different thing--whose reference is it borrowing?

FWIW, yes, this is playing merry hell with the Gilectomy.  If there are 
two threads, and one calls PyWeakref_GetObject(obj), and there's only 
one reference to obj, and the other thread blows it away... now what?  
It's my contention that this API is simply untenable under the 
Gilectomy, and that it needs to change to returning a new (strong) 
reference.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/f96b2ce1/attachment.html>

From jeanpierreda at gmail.com  Mon Oct 10 05:05:01 2016
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Mon, 10 Oct 2016 02:05:01 -0700
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
Message-ID: <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>

> It's my contention that this API is simply untenable under the Gilectomy

Yeah, I agree with your contention. The only circumstance PyWeakref_GetObject
would still be valid without a GIL is if you happened to know you had a
reference elsewhere that was keeping it alive. But if you did, you probably
wouldn't need to use the weakref. If you don't, then with GILectomy you
have no thread-safe options.

Trivial bikeshedding:  rather than change the existing API, wouldn't it be
better to add a new API, and not support the old one in GILectomized
CPython? Introducing memory leaks would be a little unfortunate, and it's
hard to audit C code for correct use of refcounted pointers even *before*
you change the ownership of returned pointers across versions.

w.r.t. the exact question:

> whose reference is it borrowing?

I think this is a red herring sort of question. "borrowed" only means
"unowned". But anyway, we borrowed from the weakref. (Borrowing from
somebody doesn't imply they own a reference -- we can borrow from a
borrowed reference, for example, and this is common.)

The term "borrowed" is supposed to imply a sensible scope during which
you're free to use the object, and weakrefs don't have that (except for
what is granted by the GIL), so this does sound wacky. I bet it was for
performance.

-- Devin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/f62c7018/attachment.html>

From larry at hastings.org  Mon Oct 10 05:35:31 2016
From: larry at hastings.org (Larry Hastings)
Date: Mon, 10 Oct 2016 11:35:31 +0200
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
Message-ID: <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>



On 10/10/2016 11:05 AM, Devin Jeanpierre wrote:
> > whose reference is it borrowing?
>
> I think this is a red herring sort of question. "borrowed" only means 
> "unowned". But anyway, we borrowed from the weakref. (Borrowing from 
> somebody doesn't imply they own a reference -- we can borrow from a 
> borrowed reference, for example, and this is common.)

Huh?  In all other circumstances, a "borrowed" reference is exactly 
that: X has a reference, and you are relying on X's reference to keep 
the object alive.  Borrowing from a borrowed reference is simply a chain 
of these; Z borrows from Y, Y borrows from X, and X is the original 
person who did the incref.  But you're borrowing from something 
specific, somebody who the API guarantees has a legitimate reference on 
the object and won't drop it while you're using it.  I bet for every 
other spot in the API I can tell you from whom you're borrowing the 
reference.

In contrast, the "borrowed" reference returned by PyWeakRef_GetObject() 
seems to be "borrowed" from some unspecified entity.  The fact that the 
object is live in Python directly implies that, yes, *somebody* must 
have a reference, somewhere.  But ISTM (and apparently you) that this is 
relying on the GIL preventing that unknown other actor from dropping 
their reference while you've borrow it.  A guarantee that the 
post-Gilectomy Python interpreter can no longer make!

In any case, I see nothing in the documentation that suggests "borrowed 
only means unowned" as you suggest.  In contrast, the documentation 
seems to suggest that the metaphor is how I understood it; that when you 
"borrow" a reference, there is another object who has a reference and 
you're relying on their reference to keep the object alive.

    https://docs.python.org/3.6/extending/extending.html#reference-counts



//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/f80ea4f7/attachment.html>

From rosuav at gmail.com  Mon Oct 10 05:45:19 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Mon, 10 Oct 2016 20:45:19 +1100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
Message-ID: <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>

On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings <larry at hastings.org> wrote:
> Huh?  In all other circumstances, a "borrowed" reference is exactly that: X
> has a reference, and you are relying on X's reference to keep the object
> alive.  Borrowing from a borrowed reference is simply a chain of these; Z
> borrows from Y, Y borrows from X, and X is the original person who did the
> incref.  But you're borrowing from something specific, somebody who the API
> guarantees has a legitimate reference on the object and won't drop it while
> you're using it.  I bet for every other spot in the API I can tell you from
> whom you're borrowing the reference.

Okay. Here's a test:

PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index)
Return value: Borrowed reference.

Presumably you own a reference to the list itself before you call
this, and the list has a reference to its items. But suppose another
thread clear()s the list immediately after you call this; whose
reference are you borrowing now? The list's is gone.

Or is this another API that will have to change post gilectomy?

ChrisA

From greg.ewing at canterbury.ac.nz  Mon Oct 10 06:34:48 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 10 Oct 2016 23:34:48 +1300
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
Message-ID: <57FB6EC8.8010605@canterbury.ac.nz>

Larry Hastings wrote:
> In contrast, the "borrowed" reference returned by PyWeakRef_GetObject() 
> seems to be "borrowed" from some unspecified entity.

It's a delocalised quantum reference, borrowing a little
bit from all strong references in existence. :-)

-- 
Greg

From catch-all at masklinn.net  Mon Oct 10 07:06:20 2016
From: catch-all at masklinn.net (Xavier Morel)
Date: Mon, 10 Oct 2016 13:06:20 +0200
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
Message-ID: <1A5BB933-3203-4EFC-A150-188ABBD03973@masklinn.net>


> On 2016-10-10, at 11:05 , Devin Jeanpierre <jeanpierreda at gmail.com> wrote:
> The term "borrowed" is supposed to imply a sensible scope during which you're free to use the object, and weakrefs don't have that (except for what is granted by the GIL), so this does sound wacky. I bet it was for performance.

Especially as it handles both getting an object from a weakref and
checking whether the weakref is still alive.

OTOH it could be an enshrined bug, http://bugs.python.org/issue520087
fixed a discrepancy between the doc and the implementation by matching
the doc to the implementation (of returning a borrowed ref').

Also of note, pypy developers have been reporting issues with that
specific API since ~2010[0][1], and IIRC they have added a
PyWeakref_LockObject to cpyext.

[0] http://bugs.python.org/issue8578
[1] http://bugs.python.org/issue16602#msg177272
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/26c0be80/attachment.html>

From fred at fdrake.net  Mon Oct 10 08:29:50 2016
From: fred at fdrake.net (Fred Drake)
Date: Mon, 10 Oct 2016 08:29:50 -0400
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
Message-ID: <CAFT4OTHKHAsKdzBLhaQfWZkymUJ=64JY+H4=F+kV5HTR263TEg@mail.gmail.com>

On Mon, Oct 10, 2016 at 4:17 AM, Larry Hastings <larry at hastings.org> wrote:
> Given that the weakref doesn't have a reference to the object--merely a weak
> reference, different thing--whose reference is it borrowing?

As others have said, it doesn't really matter who's reference it was;
just that there was another at the time it was returned.  Clearly it
can't be considered valid once additional Python code might be run.

> FWIW, yes, this is playing merry hell with the Gilectomy.  If there are two
> threads, and one calls PyWeakref_GetObject(obj), and there's only one
> reference to obj, and the other thread blows it away... now what?  It's my
> contention that this API is simply untenable under the Gilectomy, and that
> it needs to change to returning a new (strong) reference.

+1 for this change.


  -Fred

-- 
Fred L. Drake, Jr.    <fred at fdrake.net>
"A storm broke loose in my mind."  --Albert Einstein

From g.rodola at gmail.com  Mon Oct 10 09:34:35 2016
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Mon, 10 Oct 2016 15:34:35 +0200
Subject: [Python-Dev] pythonhosted.org upload no longer works
Message-ID: <CAFYqXL-=4PA9odjH11TT4DDRZnmz==W0OVMeTO_08OjURg7F=Q@mail.gmail.com>

This is what I've bumped into just now:

python setup.py upload_sphinx --upload-dir=docs/_build/html
running upload_sphinx
Submitting documentation to https://upload.pypi.org/legacy/
Upload failed (410): Uploading documentation is no longer supported, we
recommend using https://readthedocs.org/.

Personally I prefer pythonhosted over readthedocs as I can provide the same
style as official Python's doc (see https://pythonhosted.org/psutil/) and,
most importantly, because I can automatize doc upload just by running "make
doc-upload".
Is there a reason upload functionality has been disabled?
Is pythonhosted.org going to be dismissed or something?

-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/76de2418/attachment.html>

From p.f.moore at gmail.com  Mon Oct 10 10:41:39 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 10 Oct 2016 15:41:39 +0100
Subject: [Python-Dev] pythonhosted.org upload no longer works
In-Reply-To: <CAFYqXL-=4PA9odjH11TT4DDRZnmz==W0OVMeTO_08OjURg7F=Q@mail.gmail.com>
References: <CAFYqXL-=4PA9odjH11TT4DDRZnmz==W0OVMeTO_08OjURg7F=Q@mail.gmail.com>
Message-ID: <CACac1F9hfkyYYz4xoSezLOV+RprUCJ-gVi7eknyCh_GUMLFYGg@mail.gmail.com>

On 10 October 2016 at 14:34, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> This is what I've bumped into just now:
>
> python setup.py upload_sphinx --upload-dir=docs/_build/html
> running upload_sphinx
> Submitting documentation to https://upload.pypi.org/legacy/
> Upload failed (410): Uploading documentation is no longer supported, we
> recommend using https://readthedocs.org/.
>
> Personally I prefer pythonhosted over readthedocs as I can provide the same
> style as official Python's doc (see https://pythonhosted.org/psutil/) and,
> most importantly, because I can automatize doc upload just by running "make
> doc-upload".
> Is there a reason upload functionality has been disabled?
> Is pythonhosted.org going to be dismissed or something?

This was discussed on distutils-sig back in May 2015. The thread
starts here: https://mail.python.org/pipermail/distutils-sig/2015-May/026327.html.
One of the actions in the final proposal
(https://mail.python.org/pipermail/distutils-sig/2015-May/026381.html)
included a step to contact projects that were using pythonhosted.org.
Did you not get any contact?

It might be worth picking this up on distutils-sig, as that's where
the original discussion occurred. I've copied this reply to there, and
suggest followups go to that list.

Paul

From elliot.gorokhovsky at gmail.com  Mon Oct 10 02:18:35 2016
From: elliot.gorokhovsky at gmail.com (Elliot Gorokhovsky)
Date: Mon, 10 Oct 2016 06:18:35 +0000
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
Message-ID: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>

Hi,

I posted here a while back asking what you guys thought about implementing
radix sort for strings in list.sort(). You gave me a lot of reasons why
that would be a bad idea. However, it got me thinking, and I came up with
something that you may find interesting.

First, some simple benchmark results (numbers are seconds, check out the
extension module at https://github.com/embg/python-fast-listsort.git):

*** 1e3 ints ***
F.fastsort(): 0.00018930435180664062
F.sort(): 0.0002830028533935547
*** 1e3 strings ***
F.fastsort(): 0.0003533363342285156
F.sort(): 0.00044846534729003906
*** 1e7 ints ***
F.fastsort(): 5.479267358779907
F.sort(): 8.063318014144897
*** 1e7 strings ***
F.fastsort(): 9.992833137512207
F.sort(): 13.730914115905762

The optimization uses the fact that, in practice, we are almost always
sorting keys of the same type (note: not objects of the same type, *keys*
of the same type; we could have a complex key function like str that takes
many different types, but the keys are still all the same type). So we can
just do one typecheck up front and then do unsafe comparisons during the
sort (if our initial check passes). Specifically, we check that for all the
PyObject* in saved_ob_item, the ob->ob_type are the same. Then we replace
PyObject_RichCompare with ob_type->tp_richcompare. Additionally, we can
check for the very common cases of ints and strings and give optimized
comparison functions for those cases. It might not seem like this would
save a lot of time, but it turns out that PyObject_RichCompare is a massive
clusterf**k that has to test tons of type stuff before it can actually
compare. Cutting that out ends up saving a *lot* of time, as the benchmark
demonstrates.

What do you think? I'm planning on writing this up into a patch, but wanted
to get some feedback on the implementation and ideas for improvement first.

Thanks,
Elliot
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/1feca58f/attachment.html>

From g.rodola at gmail.com  Mon Oct 10 11:17:23 2016
From: g.rodola at gmail.com (Giampaolo Rodola')
Date: Mon, 10 Oct 2016 17:17:23 +0200
Subject: [Python-Dev] pythonhosted.org upload no longer works
In-Reply-To: <CACac1F9hfkyYYz4xoSezLOV+RprUCJ-gVi7eknyCh_GUMLFYGg@mail.gmail.com>
References: <CAFYqXL-=4PA9odjH11TT4DDRZnmz==W0OVMeTO_08OjURg7F=Q@mail.gmail.com>
 <CACac1F9hfkyYYz4xoSezLOV+RprUCJ-gVi7eknyCh_GUMLFYGg@mail.gmail.com>
Message-ID: <CAFYqXL_jCQ1UYehmQ_+JS8=+c7yX8jqEy1-_+Ybj1SPcQ45Pmg@mail.gmail.com>

Thanks Paul,
no I wasn't aware of that, and I will subscribe to distutils-sig from now
on.
I don't remember ever receiving a warrant about this decision but it's
entirely possible I may have forgot. =)
So long story short is that I will have to move the doc on readthedocs,
correct?
Does pythonhosted provide an automatic redirect or I'll have to upload a
static page which states the doc has been moved?

On Mon, Oct 10, 2016 at 4:41 PM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 10 October 2016 at 14:34, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> > This is what I've bumped into just now:
> >
> > python setup.py upload_sphinx --upload-dir=docs/_build/html
> > running upload_sphinx
> > Submitting documentation to https://upload.pypi.org/legacy/
> > Upload failed (410): Uploading documentation is no longer supported, we
> > recommend using https://readthedocs.org/.
> >
> > Personally I prefer pythonhosted over readthedocs as I can provide the
> same
> > style as official Python's doc (see https://pythonhosted.org/psutil/)
> and,
> > most importantly, because I can automatize doc upload just by running
> "make
> > doc-upload".
> > Is there a reason upload functionality has been disabled?
> > Is pythonhosted.org going to be dismissed or something?
>
> This was discussed on distutils-sig back in May 2015. The thread
> starts here: https://mail.python.org/pipermail/distutils-sig/2015-
> May/026327.html.
> One of the actions in the final proposal
> (https://mail.python.org/pipermail/distutils-sig/2015-May/026381.html)
> included a step to contact projects that were using pythonhosted.org.
> Did you not get any contact?
>
> It might be worth picking this up on distutils-sig, as that's where
> the original discussion occurred. I've copied this reply to there, and
> suggest followups go to that list.
>
> Paul
>



-- 
Giampaolo - http://grodola.blogspot.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/71d44115/attachment.html>

From mafagafogigante at gmail.com  Mon Oct 10 11:27:54 2016
From: mafagafogigante at gmail.com (Bernardo Sulzbach)
Date: Mon, 10 Oct 2016 12:27:54 -0300
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
Message-ID: <08d462a8-99e8-2b46-67bf-67349cbf8e66@gmail.com>

On 10/10/2016 03:18 AM, Elliot Gorokhovsky wrote:
> Hi,
>
> I posted here a while back asking what you guys thought about
> implementing radix sort for strings in list.sort(). You gave me a lot of
> reasons why that would be a bad idea. However, it got me thinking, and I
> came up with something that you may find interesting.
>
> First, some simple benchmark results (numbers are seconds, check out the
> extension module at https://github.com/embg/python-fast-listsort.git):
>
> *** 1e3 ints ***
> F.fastsort(): 0.00018930435180664062
> F.sort(): 0.0002830028533935547
> *** 1e3 strings ***
> F.fastsort(): 0.0003533363342285156
> F.sort(): 0.00044846534729003906
> *** 1e7 ints ***
> F.fastsort(): 5.479267358779907
> F.sort(): 8.063318014144897
> *** 1e7 strings ***
> F.fastsort(): 9.992833137512207
> F.sort(): 13.730914115905762
>

The numbers are good. How does this interfere with sorting very small 
lists? Obviously, even if it does not help with very small lists, we can 
always put a threshold on the size and take this path or not.

-- 
Bernardo Sulzbach
http://www.mafagafogigante.org/
mafagafogigante at mafagafogigante.org

From rosuav at gmail.com  Mon Oct 10 11:29:09 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 11 Oct 2016 02:29:09 +1100
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
Message-ID: <CAPTjJmrcwLJY5ZuFKnsrOyFSHE5bbJMfkp6TB_KPe_fqYXECgg@mail.gmail.com>

On Mon, Oct 10, 2016 at 5:18 PM, Elliot Gorokhovsky
<elliot.gorokhovsky at gmail.com> wrote:
> First, some simple benchmark results (numbers are seconds, check out the
> extension module at https://github.com/embg/python-fast-listsort.git):
>
> *** 1e3 ints ***
> F.fastsort(): 0.00018930435180664062
> F.sort(): 0.0002830028533935547
> *** 1e3 strings ***
> F.fastsort(): 0.0003533363342285156
> F.sort(): 0.00044846534729003906
> *** 1e7 ints ***
> F.fastsort(): 5.479267358779907
> F.sort(): 8.063318014144897
> *** 1e7 strings ***
> F.fastsort(): 9.992833137512207
> F.sort(): 13.730914115905762
>
> The optimization uses the fact that, in practice, we are almost always
> sorting keys of the same type

(Might be a topic for python-ideas rather than python-dev.)

Can you pick up a standard benchmark suite and use that instead of
these simplistic operations? How much slower are sorts of
heterogeneous lists - what's the impact/cost on those? If a large test
suite ("make test" if you don't have a nice benchmark handy - the
CPython test suite is a solid mass of code) shows an overall slowdown,
this probably isn't worth pursuing; if it shows a minor but
insignificant increase, there might be something worth looking into;
and if it shows a huge increase... well, to be honest, if it shows a
huge increase, I'd suspect a fault in the measurements, because
sorting isn't a significant part of most test suites :)

ChrisA

From donald at stufft.io  Mon Oct 10 12:06:37 2016
From: donald at stufft.io (Donald Stufft)
Date: Mon, 10 Oct 2016 12:06:37 -0400
Subject: [Python-Dev] [Distutils] pythonhosted.org upload no longer works
In-Reply-To: <CAFYqXL_jCQ1UYehmQ_+JS8=+c7yX8jqEy1-_+Ybj1SPcQ45Pmg@mail.gmail.com>
References: <CAFYqXL-=4PA9odjH11TT4DDRZnmz==W0OVMeTO_08OjURg7F=Q@mail.gmail.com>
 <CACac1F9hfkyYYz4xoSezLOV+RprUCJ-gVi7eknyCh_GUMLFYGg@mail.gmail.com>
 <CAFYqXL_jCQ1UYehmQ_+JS8=+c7yX8jqEy1-_+Ybj1SPcQ45Pmg@mail.gmail.com>
Message-ID: <3E64EF6B-3F35-4BFB-BADF-EA7743A560DD@stufft.io>

You?re using the new PyPI (either by getting a new default from a new version of Python or Twine OR explicitly switching to it yourself) which has this disabled. You can still upload to legacy PyPI by switching the default back. That will restore the ability to upload docs to pythonhosted.org.

We do plan on adding the ability to provide an automatic redirect to the new PyPI, but it?s not there yet.

> On Oct 10, 2016, at 11:17 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
> 
> Thanks Paul,
> no I wasn't aware of that, and I will subscribe to distutils-sig from now on. 
> I don't remember ever receiving a warrant about this decision but it's entirely possible I may have forgot. =)
> So long story short is that I will have to move the doc on readthedocs, correct?
> Does pythonhosted provide an automatic redirect or I'll have to upload a static page which states the doc has been moved?
> 
> On Mon, Oct 10, 2016 at 4:41 PM, Paul Moore <p.f.moore at gmail.com <mailto:p.f.moore at gmail.com>> wrote:
> On 10 October 2016 at 14:34, Giampaolo Rodola' <g.rodola at gmail.com <mailto:g.rodola at gmail.com>> wrote:
> > This is what I've bumped into just now:
> >
> > python setup.py upload_sphinx --upload-dir=docs/_build/html
> > running upload_sphinx
> > Submitting documentation to https://upload.pypi.org/legacy/ <https://upload.pypi.org/legacy/>
> > Upload failed (410): Uploading documentation is no longer supported, we
> > recommend using https://readthedocs.org/ <https://readthedocs.org/>.
> >
> > Personally I prefer pythonhosted over readthedocs as I can provide the same
> > style as official Python's doc (see https://pythonhosted.org/psutil/ <https://pythonhosted.org/psutil/>) and,
> > most importantly, because I can automatize doc upload just by running "make
> > doc-upload".
> > Is there a reason upload functionality has been disabled?
> > Is pythonhosted.org <http://pythonhosted.org/> going to be dismissed or something?
> 
> This was discussed on distutils-sig back in May 2015. The thread
> starts here: https://mail.python.org/pipermail/distutils-sig/2015-May/026327.html <https://mail.python.org/pipermail/distutils-sig/2015-May/026327.html>.
> One of the actions in the final proposal
> (https://mail.python.org/pipermail/distutils-sig/2015-May/026381.html <https://mail.python.org/pipermail/distutils-sig/2015-May/026381.html>)
> included a step to contact projects that were using pythonhosted.org <http://pythonhosted.org/>.
> Did you not get any contact?
> 
> It might be worth picking this up on distutils-sig, as that's where
> the original discussion occurred. I've copied this reply to there, and
> suggest followups go to that list.
> 
> Paul
> 
> 
> 
> -- 
> Giampaolo - http://grodola.blogspot.com <http://grodola.blogspot.com/>
> 
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org <mailto:Distutils-SIG at python.org>
> https://mail.python.org/mailman/listinfo/distutils-sig <https://mail.python.org/mailman/listinfo/distutils-sig>

?
Donald Stufft



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/4a09428c/attachment.html>

From guido at python.org  Mon Oct 10 12:27:02 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 10 Oct 2016 09:27:02 -0700
Subject: [Python-Dev] [Distutils] pythonhosted.org upload no longer works
In-Reply-To: <3E64EF6B-3F35-4BFB-BADF-EA7743A560DD@stufft.io>
References: <CAFYqXL-=4PA9odjH11TT4DDRZnmz==W0OVMeTO_08OjURg7F=Q@mail.gmail.com>
 <CACac1F9hfkyYYz4xoSezLOV+RprUCJ-gVi7eknyCh_GUMLFYGg@mail.gmail.com>
 <CAFYqXL_jCQ1UYehmQ_+JS8=+c7yX8jqEy1-_+Ybj1SPcQ45Pmg@mail.gmail.com>
 <3E64EF6B-3F35-4BFB-BADF-EA7743A560DD@stufft.io>
Message-ID: <CAP7+vJLQqM8ow1cNZmR4QQ6_4F+X8c13RHS5_b84JyRdNgD-Sw@mail.gmail.com>

Honestly I like readthedocs a lot, and I actually *don't* like docs
that look too much like the standard Python docs -- it should be clear
to readers (subliminally, through page style, not just be parsing the
URL) that they're reading third party docs.

On Mon, Oct 10, 2016 at 9:06 AM, Donald Stufft <donald at stufft.io> wrote:
> You?re using the new PyPI (either by getting a new default from a new
> version of Python or Twine OR explicitly switching to it yourself) which has
> this disabled. You can still upload to legacy PyPI by switching the default
> back. That will restore the ability to upload docs to pythonhosted.org.
>
> We do plan on adding the ability to provide an automatic redirect to the new
> PyPI, but it?s not there yet.
>
> On Oct 10, 2016, at 11:17 AM, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>
> Thanks Paul,
> no I wasn't aware of that, and I will subscribe to distutils-sig from now
> on.
> I don't remember ever receiving a warrant about this decision but it's
> entirely possible I may have forgot. =)
> So long story short is that I will have to move the doc on readthedocs,
> correct?
> Does pythonhosted provide an automatic redirect or I'll have to upload a
> static page which states the doc has been moved?
>
> On Mon, Oct 10, 2016 at 4:41 PM, Paul Moore <p.f.moore at gmail.com> wrote:
>>
>> On 10 October 2016 at 14:34, Giampaolo Rodola' <g.rodola at gmail.com> wrote:
>> > This is what I've bumped into just now:
>> >
>> > python setup.py upload_sphinx --upload-dir=docs/_build/html
>> > running upload_sphinx
>> > Submitting documentation to https://upload.pypi.org/legacy/
>> > Upload failed (410): Uploading documentation is no longer supported, we
>> > recommend using https://readthedocs.org/.
>> >
>> > Personally I prefer pythonhosted over readthedocs as I can provide the
>> > same
>> > style as official Python's doc (see https://pythonhosted.org/psutil/)
>> > and,
>> > most importantly, because I can automatize doc upload just by running
>> > "make
>> > doc-upload".
>> > Is there a reason upload functionality has been disabled?
>> > Is pythonhosted.org going to be dismissed or something?
>>
>> This was discussed on distutils-sig back in May 2015. The thread
>> starts here:
>> https://mail.python.org/pipermail/distutils-sig/2015-May/026327.html.
>> One of the actions in the final proposal
>> (https://mail.python.org/pipermail/distutils-sig/2015-May/026381.html)
>> included a step to contact projects that were using pythonhosted.org.
>> Did you not get any contact?
>>
>> It might be worth picking this up on distutils-sig, as that's where
>> the original discussion occurred. I've copied this reply to there, and
>> suggest followups go to that list.
>>
>> Paul
>
>
>
>
> --
> Giampaolo - http://grodola.blogspot.com
>
> _______________________________________________
> Distutils-SIG maillist  -  Distutils-SIG at python.org
> https://mail.python.org/mailman/listinfo/distutils-sig
>
>
>
> ?
> Donald Stufft
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From guido at python.org  Mon Oct 10 12:37:44 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 10 Oct 2016 09:37:44 -0700
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAFT4OTHKHAsKdzBLhaQfWZkymUJ=64JY+H4=F+kV5HTR263TEg@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CAFT4OTHKHAsKdzBLhaQfWZkymUJ=64JY+H4=F+kV5HTR263TEg@mail.gmail.com>
Message-ID: <CAP7+vJL54KQ2t2ARwa==JFeatPSV9UqsQp8MiHw-=ObYSJH0HA@mail.gmail.com>

Modified +1: you can't change the behavior of the existing API, but
you can deprecate it and introduce a better one with a different name.
We'll have until Python 4.0 to carry through the deprecation anyways.
And I doubt this is the only C API change needed for happy gil-free
coding...

On Mon, Oct 10, 2016 at 5:29 AM, Fred Drake <fred at fdrake.net> wrote:
> On Mon, Oct 10, 2016 at 4:17 AM, Larry Hastings <larry at hastings.org> wrote:
>> Given that the weakref doesn't have a reference to the object--merely a weak
>> reference, different thing--whose reference is it borrowing?
>
> As others have said, it doesn't really matter who's reference it was;
> just that there was another at the time it was returned.  Clearly it
> can't be considered valid once additional Python code might be run.
>
>> FWIW, yes, this is playing merry hell with the Gilectomy.  If there are two
>> threads, and one calls PyWeakref_GetObject(obj), and there's only one
>> reference to obj, and the other thread blows it away... now what?  It's my
>> contention that this API is simply untenable under the Gilectomy, and that
>> it needs to change to returning a new (strong) reference.
>
> +1 for this change.
>
>
>   -Fred
>
> --
> Fred L. Drake, Jr.    <fred at fdrake.net>
> "A storm broke loose in my mind."  --Albert Einstein
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From python at mrabarnett.plus.com  Mon Oct 10 12:49:35 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 10 Oct 2016 17:49:35 +0100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
Message-ID: <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>

On 2016-10-10 10:45, Chris Angelico wrote:
> On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings <larry at hastings.org> wrote:
>> Huh?  In all other circumstances, a "borrowed" reference is exactly that: X
>> has a reference, and you are relying on X's reference to keep the object
>> alive.  Borrowing from a borrowed reference is simply a chain of these; Z
>> borrows from Y, Y borrows from X, and X is the original person who did the
>> incref.  But you're borrowing from something specific, somebody who the API
>> guarantees has a legitimate reference on the object and won't drop it while
>> you're using it.  I bet for every other spot in the API I can tell you from
>> whom you're borrowing the reference.
>
> Okay. Here's a test:
>
> PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index)
> Return value: Borrowed reference.
>
> Presumably you own a reference to the list itself before you call
> this, and the list has a reference to its items. But suppose another
> thread clear()s the list immediately after you call this; whose
> reference are you borrowing now? The list's is gone.
>
> Or is this another API that will have to change post gilectomy?
>
Couldn't the reference to the list also be borrowed?

If you lookup something in a dict, it'll be a borrowed reference.

If the dict is globals() and there's no GIL, another thread could delete 
the item while your code had the borrowed reference.

It looks like there might be a lot that will need to changed post gilectomy!


From p.f.moore at gmail.com  Mon Oct 10 13:03:45 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 10 Oct 2016 18:03:45 +0100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
Message-ID: <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>

On 10 October 2016 at 17:49, MRAB <python at mrabarnett.plus.com> wrote:
> If you lookup something in a dict, it'll be a borrowed reference.
>
> If the dict is globals() and there's no GIL, another thread could delete the
> item while your code had the borrowed reference.
>
> It looks like there might be a lot that will need to changed post gilectomy!

It seems to me that the whole concept of a borrowed reference may be
risky in a post-GIL world. There may be occasional cases where it's
possible to prove that borrowing is safe, but I suspect they'll be
pretty rare.

Paul

From rosuav at gmail.com  Mon Oct 10 13:06:26 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 11 Oct 2016 04:06:26 +1100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
Message-ID: <CAPTjJmpgTTcqmfzbsNKFH3vmh9-08PkSiqCUZe9M-Kr2EC=4Rw@mail.gmail.com>

On Tue, Oct 11, 2016 at 3:49 AM, MRAB <python at mrabarnett.plus.com> wrote:
> On 2016-10-10 10:45, Chris Angelico wrote:
>>
>> On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings <larry at hastings.org>
>> wrote:
>>>
>>> Huh?  In all other circumstances, a "borrowed" reference is exactly that:
>>> X
>>> has a reference, and you are relying on X's reference to keep the object
>>> alive.  Borrowing from a borrowed reference is simply a chain of these; Z
>>> borrows from Y, Y borrows from X, and X is the original person who did
>>> the
>>> incref.  But you're borrowing from something specific, somebody who the
>>> API
>>> guarantees has a legitimate reference on the object and won't drop it
>>> while
>>> you're using it.  I bet for every other spot in the API I can tell you
>>> from
>>> whom you're borrowing the reference.
>>
>>
>> Okay. Here's a test:
>>
>> PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index)
>> Return value: Borrowed reference.
>>
>> Presumably you own a reference to the list itself before you call
>> this, and the list has a reference to its items. But suppose another
>> thread clear()s the list immediately after you call this; whose
>> reference are you borrowing now? The list's is gone.
>>
>> Or is this another API that will have to change post gilectomy?
>>
> Couldn't the reference to the list also be borrowed?
>
> If you lookup something in a dict, it'll be a borrowed reference.
>
> If the dict is globals() and there's no GIL, another thread could delete the
> item while your code had the borrowed reference.
>
> It looks like there might be a lot that will need to changed post gilectomy!

If you're trying to manipulate a list, you should probably own a
reference to it. Otherwise, you're taking great risks. The trouble is
that there's time enough for a context switch between PyList_GetItem
and even an immediately-following incref, so you could have a junk
pointer despite your best efforts.

Contrast:

PyObject* PySequence_GetItem(PyObject *o, Py_ssize_t i)
Return value: New reference.

If you use sequence protocol, you are guaranteed to have a valid
reference to the object. To be sure, the item might no longer be in
the sequence by the time you look at it... but you know you're not
looking at junk memory. With PyList_GetItem, it would be possible for
you to have a dud pointer.

Also contrast:

PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
Return value: Borrowed reference.

As long as you own a reference to the tuple itself, you can be
confident that the borrowed reference will still be valid. There's no
way that the object's refcount can validly hit zero so long as the
tuple exists. (I'm ignoring PyTuple_SetItem here, which basically is
for initialization.) It's only with lists that you have this risk.

PyObject* PyDict_GetItem(PyObject *p, PyObject *key)
Return value: Borrowed reference.

... okay, it's only with lists and dictionaries. Here come the other
lot called 'Python', I guess.

ChrisA

From rosuav at gmail.com  Mon Oct 10 13:14:20 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 11 Oct 2016 04:14:20 +1100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
Message-ID: <CAPTjJmoZ8C4STFGRc_Qx_WnS0BwqQ=kOH3-w7xf=P3pA3U+f-g@mail.gmail.com>

On Tue, Oct 11, 2016 at 4:03 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 10 October 2016 at 17:49, MRAB <python at mrabarnett.plus.com> wrote:
>> If you lookup something in a dict, it'll be a borrowed reference.
>>
>> If the dict is globals() and there's no GIL, another thread could delete the
>> item while your code had the borrowed reference.
>>
>> It looks like there might be a lot that will need to changed post gilectomy!
>
> It seems to me that the whole concept of a borrowed reference may be
> risky in a post-GIL world. There may be occasional cases where it's
> possible to prove that borrowing is safe, but I suspect they'll be
> pretty rare.

Not really. If you have an object that forever owns a reference to
another object, it's safe to return borrowed references. A quick
search for 'borrowed' showed up these perfectly safe[1] examples:

PyObject* PyFunction_GetGlobals(PyObject *op)
Return value: Borrowed reference.
-- a function's __globals__ attribute is read-only
-- though its __code__ is not, so possibly PyFunction_GetCode may be dangerous??

PyObject* PyTuple_GetItem(PyObject *p, Py_ssize_t pos)
Return value: Borrowed reference.
-- as mentioned previously, tuples are immutable and thus safe

PyObject* PyMethod_Function(PyObject *meth)
Return value: Borrowed reference.
PyObject* PyMethod_Self(PyObject *meth)
Return value: Borrowed reference.
-- a bound method's __func__ and __self__ attributes are read-only

In these cases, it's easy to see what reference you're borrowing, and
it's not a problem.

ChrisA

[1] As far as I know!

From njs at pobox.com  Mon Oct 10 13:50:25 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 10 Oct 2016 10:50:25 -0700
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
Message-ID: <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>

On Mon, Oct 10, 2016 at 10:03 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 10 October 2016 at 17:49, MRAB <python at mrabarnett.plus.com> wrote:
>> If you lookup something in a dict, it'll be a borrowed reference.
>>
>> If the dict is globals() and there's no GIL, another thread could delete the
>> item while your code had the borrowed reference.
>>
>> It looks like there might be a lot that will need to changed post gilectomy!
>
> It seems to me that the whole concept of a borrowed reference may be
> risky in a post-GIL world. There may be occasional cases where it's
> possible to prove that borrowing is safe, but I suspect they'll be
> pretty rare.

I don't think it's that bad...

In a post-GIL world, the problem cases we're talking about like
deleting items from containers all require some sort of locking, even
before we start thinking about borrowed references. If two threads
start mucking around resizing the internals of a list or dict at the
same time, then you are unlikely to go to space today. IIRC to handle
this gilectomy adds per-object mutexes that you have to hold whenever
you're mucking around with that object's internals.

If we say that borrowing reference from a dict is one of the things
that counts as mucking about with that dict, and thus requires you to
hold the dict lock for as long as you hold the borrowed reference,
then all should be well.

I assume Larry is way ahead of us on this, which is why he's suddenly
confusing us all by emphasizing that when you borrow a reference you
should know who you're borrowing it from, so you know which lock to
hold.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From p.f.moore at gmail.com  Mon Oct 10 13:55:33 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 10 Oct 2016 18:55:33 +0100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
Message-ID: <CACac1F_EaG0KOiDNu5UL2MEqgMLHNc1jsGnrSZc+UtNE-kUkYQ@mail.gmail.com>

On 10 October 2016 at 18:50, Nathaniel Smith <njs at pobox.com> wrote:
> I assume Larry is way ahead of us on this,

Yeah, I'd imagine you're right on that :-)
Paul

From python at mrabarnett.plus.com  Mon Oct 10 14:04:30 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 10 Oct 2016 19:04:30 +0100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
Message-ID: <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>

On 2016-10-10 18:50, Nathaniel Smith wrote:
> On Mon, Oct 10, 2016 at 10:03 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> > On 10 October 2016 at 17:49, MRAB <python at mrabarnett.plus.com> wrote:
> >> If you lookup something in a dict, it'll be a borrowed reference.
> >>
> >> If the dict is globals() and there's no GIL, another thread could delete the
> >> item while your code had the borrowed reference.
> >>
> >> It looks like there might be a lot that will need to changed post gilectomy!
> >
> > It seems to me that the whole concept of a borrowed reference may be
> > risky in a post-GIL world. There may be occasional cases where it's
> > possible to prove that borrowing is safe, but I suspect they'll be
> > pretty rare.
>
> I don't think it's that bad...
>
> In a post-GIL world, the problem cases we're talking about like
> deleting items from containers all require some sort of locking, even
> before we start thinking about borrowed references. If two threads
> start mucking around resizing the internals of a list or dict at the
> same time, then you are unlikely to go to space today. IIRC to handle
> this gilectomy adds per-object mutexes that you have to hold whenever
> you're mucking around with that object's internals.
>
> If we say that borrowing reference from a dict is one of the things
> that counts as mucking about with that dict, and thus requires you to
> hold the dict lock for as long as you hold the borrowed reference,
> then all should be well.
>
> I assume Larry is way ahead of us on this, which is why he's suddenly
> confusing us all by emphasizing that when you borrow a reference you
> should know who you're borrowing it from, so you know which lock to
> hold.
>
Instead of locking the object, could we keep the GIL, but have it 
normally released?

A thread could then still call a function such as PyWeakref_GetObject() 
that returns a borrowed reference, but only if it's holding the GIL. It 
would be able to INCREF the reference before releasing the GIL again.

There could still be a new function that returned a strong reference.


From random832 at fastmail.com  Mon Oct 10 14:24:20 2016
From: random832 at fastmail.com (Random832)
Date: Mon, 10 Oct 2016 14:24:20 -0400
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
Message-ID: <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>

On Mon, Oct 10, 2016, at 14:04, MRAB wrote:
> Instead of locking the object, could we keep the GIL, but have it 
> normally released?
> 
> A thread could then still call a function such as PyWeakref_GetObject() 
> that returns a borrowed reference, but only if it's holding the GIL. It 
> would be able to INCREF the reference before releasing the GIL again.

So, what stops the other thread which never asks for the GIL from
blowing away the reference? Or is this a special kind of lock that you
can "assert isn't locked" without locking it for yourself, and
INCREF/DECREF does so?

From rosuav at gmail.com  Mon Oct 10 15:36:59 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 11 Oct 2016 06:36:59 +1100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
 <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
Message-ID: <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>

On Tue, Oct 11, 2016 at 5:24 AM, Random832 <random832 at fastmail.com> wrote:
> On Mon, Oct 10, 2016, at 14:04, MRAB wrote:
>> Instead of locking the object, could we keep the GIL, but have it
>> normally released?
>>
>> A thread could then still call a function such as PyWeakref_GetObject()
>> that returns a borrowed reference, but only if it's holding the GIL. It
>> would be able to INCREF the reference before releasing the GIL again.
>
> So, what stops the other thread which never asks for the GIL from
> blowing away the reference? Or is this a special kind of lock that you
> can "assert isn't locked" without locking it for yourself, and
> INCREF/DECREF does so?

"assert isn't locked" is pretty cheap (race conditions wouldn't be
applicable here, as you would have to completely obtain the GIL before
even attempting a dangerous operation), but what would INCREF/DECREF
do if the GIL is locked by another thread?

Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory
deallocation lock". While it is held (by any thread - it's a guard,
more than a lock), Py_DECREF will not actually deallocate memory -
objects can fall to zero references without being wiped. Once the
lock/guard is freed/cleared, anything that had fallen to zero is now
deallocated. This probably would mean stuffing them onto a list of
"doomed objects", and upon release of the guard, any doomed objects
that still have no refs would get deallocated.

That would, by definition, make *all* borrowed refs legal; you can
either use them and finish, or INCREF them before releasing the
deallocation lock, and either way, you have a guarantee that (a)
you're not messing with junk memory, and (b) it really is the object
you think it is. There should be no risk of race conditions, because
you would completely claim the deallocation lock before calling
something that gives you a borrowed ref, meaning that the whole time
the ref is borrowed, you must have that lock.

But I'm guessing other people have thought far more about this than I have.

ChrisA

From python at mrabarnett.plus.com  Mon Oct 10 16:27:38 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 10 Oct 2016 21:27:38 +0100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
 <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
 <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
Message-ID: <fc990253-3e84-8933-7e3e-143c58db4b92@mrabarnett.plus.com>

On 2016-10-10 20:36, Chris Angelico wrote:
> On Tue, Oct 11, 2016 at 5:24 AM, Random832 <random832 at fastmail.com> wrote:
>> On Mon, Oct 10, 2016, at 14:04, MRAB wrote:
>>> Instead of locking the object, could we keep the GIL, but have it
>>> normally released?
>>>
>>> A thread could then still call a function such as PyWeakref_GetObject()
>>> that returns a borrowed reference, but only if it's holding the GIL. It
>>> would be able to INCREF the reference before releasing the GIL again.
>>
>> So, what stops the other thread which never asks for the GIL from
>> blowing away the reference? Or is this a special kind of lock that you
>> can "assert isn't locked" without locking it for yourself, and
>> INCREF/DECREF does so?
>
> "assert isn't locked" is pretty cheap (race conditions wouldn't be
> applicable here, as you would have to completely obtain the GIL before
> even attempting a dangerous operation), but what would INCREF/DECREF
> do if the GIL is locked by another thread?
>
> Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory
> deallocation lock". While it is held (by any thread - it's a guard,
> more than a lock), Py_DECREF will not actually deallocate memory -
> objects can fall to zero references without being wiped. Once the
> lock/guard is freed/cleared, anything that had fallen to zero is now
> deallocated. This probably would mean stuffing them onto a list of
> "doomed objects", and upon release of the guard, any doomed objects
> that still have no refs would get deallocated.
>
> That would, by definition, make *all* borrowed refs legal; you can
> either use them and finish, or INCREF them before releasing the
> deallocation lock, and either way, you have a guarantee that (a)
> you're not messing with junk memory, and (b) it really is the object
> you think it is. There should be no risk of race conditions, because
> you would completely claim the deallocation lock before calling
> something that gives you a borrowed ref, meaning that the whole time
> the ref is borrowed, you must have that lock.
>
> But I'm guessing other people have thought far more about this than I have.
>
I have to admit that the GIL thing was just a vague feeling I had, and I 
hadn't thought too deeply about it, but I think that you've crystallised 
it. :-)


From elliot.gorokhovsky at gmail.com  Mon Oct 10 16:42:37 2016
From: elliot.gorokhovsky at gmail.com (Elliot Gorokhovsky)
Date: Mon, 10 Oct 2016 20:42:37 +0000
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
Message-ID: <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>

I'd like to reply to both questions with one email, if that's all right.

Bernardo Sulzbach asked, "How does this interfere with sorting very small
lists?", and ChrisA asked, "How much slower are sorts of heterogeneous
lists?". I modified the benchmark to answer both questions, the former at
the top, and the latter at the bottom (see the git repo):

*** 10 ints ***
F.fastsort(): 2.86102294921875e-06
F.sort(): 3.337860107421875e-06
*** 10 strings ***
F.fastsort(): 1.6689300537109375e-06
F.sort(): 1.6689300537109375e-06
*** 1e3 ints ***
F.fastsort(): 0.00013589859008789062
F.sort(): 0.00018906593322753906
*** 1e3 strings ***
F.fastsort(): 0.0002529621124267578
F.sort(): 0.0002772808074951172
*** 1e7 ints ***
F.fastsort(): 5.472854375839233
F.sort(): 7.8826072216033936
*** 1e7 strings ***
F.fastsort(): 10.104042053222656
F.sort(): 13.139304876327515
*** 1e7 ints + 1 float (to disable the optimization while keeping the
precheck)***
F.fastsort(): 7.57237982749939
F.sort(): 7.666172504425049

ChrisA suggested I also try "make test" or something to get a more
realistic benchmark. I will do that once I implement this as a patch, right
now it's an extension module that subclasses list, so I can't just drop it
into existing code without modification.

Let me know if you have any other questions/suggestions!

On Mon, Oct 10, 2016 at 12:18 AM Elliot Gorokhovsky <
elliot.gorokhovsky at gmail.com> wrote:

> Hi,
>
> I posted here a while back asking what you guys thought about implementing
> radix sort for strings in list.sort(). You gave me a lot of reasons why
> that would be a bad idea. However, it got me thinking, and I came up with
> something that you may find interesting.
>
> First, some simple benchmark results (numbers are seconds, check out the
> extension module at https://github.com/embg/python-fast-listsort.git):
>
> *** 1e3 ints ***
> F.fastsort(): 0.00018930435180664062
> F.sort(): 0.0002830028533935547
> *** 1e3 strings ***
> F.fastsort(): 0.0003533363342285156
> F.sort(): 0.00044846534729003906
> *** 1e7 ints ***
> F.fastsort(): 5.479267358779907
> F.sort(): 8.063318014144897
> *** 1e7 strings ***
> F.fastsort(): 9.992833137512207
> F.sort(): 13.730914115905762
>
> The optimization uses the fact that, in practice, we are almost always
> sorting keys of the same type (note: not objects of the same type, *keys*
> of the same type; we could have a complex key function like str that takes
> many different types, but the keys are still all the same type). So we can
> just do one typecheck up front and then do unsafe comparisons during the
> sort (if our initial check passes). Specifically, we check that for all the
> PyObject* in saved_ob_item, the ob->ob_type are the same. Then we replace
> PyObject_RichCompare with ob_type->tp_richcompare. Additionally, we can
> check for the very common cases of ints and strings and give optimized
> comparison functions for those cases. It might not seem like this would
> save a lot of time, but it turns out that PyObject_RichCompare is a massive
> clusterf**k that has to test tons of type stuff before it can actually
> compare. Cutting that out ends up saving a *lot* of time, as the benchmark
> demonstrates.
>
> What do you think? I'm planning on writing this up into a patch, but
> wanted to get some feedback on the implementation and ideas for improvement
> first.
>
> Thanks,
> Elliot
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/2b73f811/attachment.html>

From njs at pobox.com  Mon Oct 10 16:53:19 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 10 Oct 2016 13:53:19 -0700
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
Message-ID: <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>

On Mon, Oct 10, 2016 at 1:42 PM, Elliot Gorokhovsky
<elliot.gorokhovsky at gmail.com> wrote:
> *** 10 strings ***
> F.fastsort(): 1.6689300537109375e-06
> F.sort(): 1.6689300537109375e-06

I think something has gone wrong with your timing harness...

For accurately timing microbenchmarks, you should use timeit, or
better yet Victor Stinner's perf package:
  https://perf.readthedocs.io/

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From larry at hastings.org  Mon Oct 10 16:57:02 2016
From: larry at hastings.org (Larry Hastings)
Date: Mon, 10 Oct 2016 22:57:02 +0200
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAP7+vJL54KQ2t2ARwa==JFeatPSV9UqsQp8MiHw-=ObYSJH0HA@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CAFT4OTHKHAsKdzBLhaQfWZkymUJ=64JY+H4=F+kV5HTR263TEg@mail.gmail.com>
 <CAP7+vJL54KQ2t2ARwa==JFeatPSV9UqsQp8MiHw-=ObYSJH0HA@mail.gmail.com>
Message-ID: <4f233082-4006-2d0e-8cfc-e7d7632e425c@hastings.org>


On 10/10/2016 06:37 PM, Guido van Rossum wrote:
> Modified +1: you can't change the behavior of the existing API, but
> you can deprecate it and introduce a better one with a different name.
> We'll have until Python 4.0 to carry through the deprecation anyways.
> And I doubt this is the only C API change needed for happy gil-free
> coding...

First, "deprecate" won't work for these semantics for the Gilectomy 
branch.  I simply cannot safely support the semantics of the existing 
function.  All call sites need to change.

Second, consider that every function that returns a borrowed 
reference--PyDict_GetItem(), PyList_GetItem()--has to change too, to 
return an actual (non-borrowed) reference.  It's going to be a major 
upheaval in the C API.


For now I'm going to leave the names as-is and just change the semantics 
everywhere.  If this approach really works, and if we decide we want to 
merge it back into trunk--and those are both still big if's--we can 
revisit this decision.


I haven't yet ruled out abandoning reference counting completely and 
going to 100% tracing garbage collecting,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/530ec3c3/attachment-0001.html>

From larry at hastings.org  Mon Oct 10 17:14:02 2016
From: larry at hastings.org (Larry Hastings)
Date: Mon, 10 Oct 2016 23:14:02 +0200
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
 <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
 <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
Message-ID: <660f23eb-0cb4-f1c7-af57-889275dc55b0@hastings.org>


On 10/10/2016 09:36 PM, Chris Angelico wrote:
> Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory
> deallocation lock". While it is held (by any thread - it's a guard,
> more than a lock), Py_DECREF will not actually deallocate memory -
> objects can fall to zero references without being wiped. Once the
> lock/guard is freed/cleared, anything that had fallen to zero is now
> deallocated. This probably would mean stuffing them onto a list of
> "doomed objects", and upon release of the guard, any doomed objects
> that still have no refs would get deallocated.

If this worked, this would actually be really easy with my current 
"buffered reference counting" approach.  I literally already have a list 
of doomed objects, which a separate thread queues up for each thread to 
process later.

(This was a necessary part of "buffered reference counting", in which 
reference count changes are stored in a transaction log and later 
committed by a single thread.  Since there's only one thread making 
reference count changes, there's no contention, so it doesn't have to 
use atomic incr/decr, which is a big performance win.)

But I don't think this fixes the problem.  Consider:

 1. Thread A calls Q = PyList_GetItem(L, 0), which returns a borrowed
    reference.  Thread A then gets suspended, before it has a chance to
    Py_INCREF(Q).
 2. Thread B does L.clear(), the reference count of Q goes to 0, Q is
    added to the doomed objects list.
 3. Thread A gets to run again.  It does Py_INCREF(Q); Q's refcount is
    now back up to 1, as if Q had been resurrected.
 4. At some point in the future the "memory deallocation lock" is
    released and Q is deleted.

If you say "well, just look at the reference count, and if it's 1 throw 
it off the doomed objects list", keep in mind that I no longer have 
accurate real-time reference counts.  These hacks where we play games 
with the reference count are mostly removed in my branch.

Also, I don't know when it would ever be safe to release the "memory 
deallocation lock".  Just because it's safe for your thread doesn't mean 
it's safe for another thread.  And if you do it on a thread-by-thread 
basis, in the above example it might be safe from thread B's perspective 
to release its "memory deallocation lock", but as illustrated that can 
have an effect on thread A.

I appreciate the brainstorming but I'm not currently sanguine about this 
idea,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/3b343b29/attachment.html>

From elliot.gorokhovsky at gmail.com  Mon Oct 10 17:16:32 2016
From: elliot.gorokhovsky at gmail.com (Elliot Gorokhovsky)
Date: Mon, 10 Oct 2016 21:16:32 +0000
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
Message-ID: <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>

Hm... that is strange, but I don't think there's anything wrong with the
way I'm timing, though I agree perf/timeit would be better. I ran the
benchmark a couple of times and the numbers seem to exactly line up
something like one in five times; perhaps not that crazy considering
they're executing nearly the same code?

Anyway, benchmarking technique aside, the point is that it it works well
for small lists (i.e. doesn't affect performance).

On Mon, Oct 10, 2016 at 2:53 PM Nathaniel Smith <njs at pobox.com> wrote:

> On Mon, Oct 10, 2016 at 1:42 PM, Elliot Gorokhovsky
> <elliot.gorokhovsky at gmail.com> wrote:
> > *** 10 strings ***
> > F.fastsort(): 1.6689300537109375e-06
> > F.sort(): 1.6689300537109375e-06
>
> I think something has gone wrong with your timing harness...
>
> For accurately timing microbenchmarks, you should use timeit, or
> better yet Victor Stinner's perf package:
>   https://perf.readthedocs.io/
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/23bac208/attachment.html>

From rosuav at gmail.com  Mon Oct 10 17:34:38 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 11 Oct 2016 08:34:38 +1100
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
Message-ID: <CAPTjJmruzC2HzFGXjY17=F3QnT7WgELD-2EW9U8vPgZrx=f2Tg@mail.gmail.com>

On Tue, Oct 11, 2016 at 7:42 AM, Elliot Gorokhovsky
<elliot.gorokhovsky at gmail.com> wrote:
> ChrisA suggested I also try "make test" or something to get a more realistic
> benchmark. I will do that once I implement this as a patch, right now it's
> an extension module that subclasses list, so I can't just drop it into
> existing code without modification.

Oh, okay. If it's done that way, there's (in a way) a guarantee that
it won't worsen anything, because you have to explicitly request the
new behaviour. So if you KNOW that you're going to be doing a ton of
string-only sorting, you can call on this new list subclass, and if
you're not, you simply don't.

ChrisA

From rosuav at gmail.com  Mon Oct 10 17:38:04 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 11 Oct 2016 08:38:04 +1100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <660f23eb-0cb4-f1c7-af57-889275dc55b0@hastings.org>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
 <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
 <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
 <660f23eb-0cb4-f1c7-af57-889275dc55b0@hastings.org>
Message-ID: <CAPTjJmpMZ0fMbO4RhR=_AAHTMEYqvL-=K8LEJgf6+h-eKFJ2Yg@mail.gmail.com>

On Tue, Oct 11, 2016 at 8:14 AM, Larry Hastings <larry at hastings.org> wrote:
> But I don't think this fixes the problem.  Consider:
>
> Thread A calls Q = PyList_GetItem(L, 0), which returns a borrowed reference.
> Thread A then gets suspended, before it has a chance to Py_INCREF(Q).
> Thread B does L.clear(), the reference count of Q goes to 0, Q is added to
> the doomed objects list.
> Thread A gets to run again.  It does Py_INCREF(Q); Q's refcount is now back
> up to 1, as if Q had been resurrected.
> At some point in the future the "memory deallocation lock" is released and Q
> is deleted.
>
> If you say "well, just look at the reference count, and if it's 1 throw it
> off the doomed objects list", keep in mind that I no longer have accurate
> real-time reference counts.  These hacks where we play games with the
> reference count are mostly removed in my branch.

That's exactly what I would have said, because I was assuming that
refcounts would be accurate. I'm not sure what you mean by "play games
with", but it sounds like you have a much more sophisticated system
going on than I had in my head, and you'll need a correspondingly
sophisticated solution to this problem.

Like I said, others have thought a LOT more about this than I have.

ChrisA

From njs at pobox.com  Mon Oct 10 17:49:07 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 10 Oct 2016 14:49:07 -0700
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
Message-ID: <CAPJVwBn3C82faK4eEcaD7qQCVFEavOWVgvNzYWdDAViGDJvoaA@mail.gmail.com>

On Mon, Oct 10, 2016 at 2:16 PM, Elliot Gorokhovsky
<elliot.gorokhovsky at gmail.com> wrote:
> Hm... that is strange, but I don't think there's anything wrong with the way
> I'm timing, though I agree perf/timeit would be better. I ran the benchmark
> a couple of times and the numbers seem to exactly line up something like one
> in five times; perhaps not that crazy considering they're executing nearly
> the same code?

No, computer clocks are precise enough, and CPUs are wonky enough
(cache effects, etc.), that it should be effectively impossible to get
the same timing result twice in row, even for running exactly the same
code. I'm not sure what's going on, but it's weird. Making these kinds
of measurements is much more complicated than it looks and you really
need to use something like timeit or perf if you want trustworthy
results. Fortunately, they're easy to use :-)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From greg.ewing at canterbury.ac.nz  Mon Oct 10 18:22:42 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 11 Oct 2016 11:22:42 +1300
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAPTjJmoZ8C4STFGRc_Qx_WnS0BwqQ=kOH3-w7xf=P3pA3U+f-g@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPTjJmoZ8C4STFGRc_Qx_WnS0BwqQ=kOH3-w7xf=P3pA3U+f-g@mail.gmail.com>
Message-ID: <57FC14B2.8060501@canterbury.ac.nz>

> On Tue, Oct 11, 2016 at 4:03 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> 
>> If you have an object that forever owns a reference to
>> another object, it's safe to return borrowed references.

Even if there are such cases, it seems we're going to
have to be a lot more careful about identifying them.

It might be safer not to have any APIs that return
borrowed references, or at least name them in a way
that make it clear they're dangerous and only to be
used with extreme care.

Having things like PyDict_GetItem where the *normal*
version of the API returns a borrowed reference seems
like a footgun.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon Oct 10 18:27:46 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 11 Oct 2016 11:27:46 +1300
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
Message-ID: <57FC15E2.4000900@canterbury.ac.nz>

Nathaniel Smith wrote:
> IIRC to handle
> this gilectomy adds per-object mutexes that you have to hold whenever
> you're mucking around with that object's internals.

What counts as "mucking around with the object's internals",
though?

If I do the C equivalent of:

    x = somedict[5]
    x.dosomething()

am I mucking around with the internals of somedict?
Intuitively, I would not think so. But the way
PyDict_GetItem currently works, this would be
dangerous.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Mon Oct 10 18:44:55 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 11 Oct 2016 11:44:55 +1300
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
 <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
Message-ID: <57FC19E7.8000405@canterbury.ac.nz>

Random832 wrote:
> Or is this a special kind of lock that you
> can "assert isn't locked" without locking it for yourself, and
> INCREF/DECREF does so?

I don't think that would work. It might be unlocked at
the moment you test it, but someone might lock it between
then and the following INCREF/DECREF.

Locking is like pregnancy -- you can't have just a
little bit of it. :-)

-- 
Greg

From guido at python.org  Mon Oct 10 18:49:25 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 10 Oct 2016 15:49:25 -0700
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <4f233082-4006-2d0e-8cfc-e7d7632e425c@hastings.org>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CAFT4OTHKHAsKdzBLhaQfWZkymUJ=64JY+H4=F+kV5HTR263TEg@mail.gmail.com>
 <CAP7+vJL54KQ2t2ARwa==JFeatPSV9UqsQp8MiHw-=ObYSJH0HA@mail.gmail.com>
 <4f233082-4006-2d0e-8cfc-e7d7632e425c@hastings.org>
Message-ID: <CAP7+vJJg8uRKpVv8FkCP2=M-g5Oah3CAxF07+ohu9zRDKfVmoQ@mail.gmail.com>

On Mon, Oct 10, 2016 at 1:57 PM, Larry Hastings <larry at hastings.org> wrote:
>
> On 10/10/2016 06:37 PM, Guido van Rossum wrote:
>
> Modified +1: you can't change the behavior of the existing API, but
> you can deprecate it and introduce a better one with a different name.
> We'll have until Python 4.0 to carry through the deprecation anyways.
> And I doubt this is the only C API change needed for happy gil-free
> coding...
>
>
> First, "deprecate" won't work for these semantics for the Gilectomy branch.
> I simply cannot safely support the semantics of the existing function.  All
> call sites need to change.

Oh, in the gilectomy you can fix all call sites. Even in CPython. But
3rd party extensions should not have this API removed in Python 3.7.

> Second, consider that every function that returns a borrowed
> reference--PyDict_GetItem(), PyList_GetItem()--has to change too, to return
> an actual (non-borrowed) reference.  It's going to be a major upheaval in
> the C API.

Yeah, this has been discussed to death in the rest of the thread.

> For now I'm going to leave the names as-is and just change the semantics
> everywhere.  If this approach really works, and if we decide we want to
> merge it back into trunk--and those are both still big if's--we can revisit
> this decision.

Changing something as subtle as this to an API sounds really bad even
in an experimental branch. But it's your branch and you can do what
you want (then again, why did you bring it up here? :-).

> I haven't yet ruled out abandoning reference counting completely and going
> to 100% tracing garbage collecting,

I guess that would have its own set of pros and cons.

-- 
--Guido van Rossum (python.org/~guido)

From python at mrabarnett.plus.com  Mon Oct 10 18:52:24 2016
From: python at mrabarnett.plus.com (MRAB)
Date: Mon, 10 Oct 2016 23:52:24 +0100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <660f23eb-0cb4-f1c7-af57-889275dc55b0@hastings.org>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
 <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
 <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
 <660f23eb-0cb4-f1c7-af57-889275dc55b0@hastings.org>
Message-ID: <f09a4a3b-eb64-d217-5487-92172e231992@mrabarnett.plus.com>

On 2016-10-10 22:14, Larry Hastings wrote:
>
> On 10/10/2016 09:36 PM, Chris Angelico wrote:
>> Hmm. Here's a naughty, and maybe dangerous, theory. Obtain a "memory
>> deallocation lock". While it is held (by any thread - it's a guard,
>> more than a lock), Py_DECREF will not actually deallocate memory -
>> objects can fall to zero references without being wiped. Once the
>> lock/guard is freed/cleared, anything that had fallen to zero is now
>> deallocated. This probably would mean stuffing them onto a list of
>> "doomed objects", and upon release of the guard, any doomed objects
>> that still have no refs would get deallocated.
>
> If this worked, this would actually be really easy with my current
> "buffered reference counting" approach.  I literally already have a list
> of doomed objects, which a separate thread queues up for each thread to
> process later.
>
> (This was a necessary part of "buffered reference counting", in which
> reference count changes are stored in a transaction log and later
> committed by a single thread.  Since there's only one thread making
> reference count changes, there's no contention, so it doesn't have to
> use atomic incr/decr, which is a big performance win.)
>
> But I don't think this fixes the problem.  Consider:
>
>  1. Thread A calls Q = PyList_GetItem(L, 0), which returns a borrowed
>     reference.  Thread A then gets suspended, before it has a chance to
>     Py_INCREF(Q).
>  2. Thread B does L.clear(), the reference count of Q goes to 0, Q is
>     added to the doomed objects list.
>  3. Thread A gets to run again.  It does Py_INCREF(Q); Q's refcount is
>     now back up to 1, as if Q had been resurrected.
>  4. At some point in the future the "memory deallocation lock" is
>     released and Q is deleted.
>
> If you say "well, just look at the reference count, and if it's 1 throw
> it off the doomed objects list", keep in mind that I no longer have
> accurate real-time reference counts.  These hacks where we play games
> with the reference count are mostly removed in my branch.
>
> Also, I don't know when it would ever be safe to release the "memory
> deallocation lock".  Just because it's safe for your thread doesn't mean
> it's safe for another thread.  And if you do it on a thread-by-thread
> basis, in the above example it might be safe from thread B's perspective
> to release its "memory deallocation lock", but as illustrated that can
> have an effect on thread A.
>
The deallocation lock could be a counter. Doomed objects would be 
collectable when it's zero.

> I appreciate the brainstorming but I'm not currently sanguine about this
> idea,
>


From rosuav at gmail.com  Mon Oct 10 18:58:38 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 11 Oct 2016 09:58:38 +1100
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <f09a4a3b-eb64-d217-5487-92172e231992@mrabarnett.plus.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
 <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
 <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
 <660f23eb-0cb4-f1c7-af57-889275dc55b0@hastings.org>
 <f09a4a3b-eb64-d217-5487-92172e231992@mrabarnett.plus.com>
Message-ID: <CAPTjJmqaghycbo4NXgn+JTCC8YnQYOq15aF+jaqtpnby9i9giQ@mail.gmail.com>

On Tue, Oct 11, 2016 at 9:52 AM, MRAB <python at mrabarnett.plus.com> wrote:
>> Also, I don't know when it would ever be safe to release the "memory
>> deallocation lock".  Just because it's safe for your thread doesn't mean
>> it's safe for another thread.  And if you do it on a thread-by-thread
>> basis, in the above example it might be safe from thread B's perspective
>> to release its "memory deallocation lock", but as illustrated that can
>> have an effect on thread A.
>>
> The deallocation lock could be a counter. Doomed objects would be
> collectable when it's zero.

Yeah, which is why I described it as a "guard". As long as you have
atomic increment/decrement (ie if two threads simultaneously try to
increment it, it WILL go up by two), it should be fine. Well, that
part should, anyway.

ChrisA

From nad at python.org  Mon Oct 10 21:12:17 2016
From: nad at python.org (Ned Deily)
Date: Mon, 10 Oct 2016 21:12:17 -0400
Subject: [Python-Dev] [RELEASE] Python 3.6.0b2 is now available
Message-ID: <4E66D65C-0EFF-41B9-9B5E-1B3FF60151B4@python.org>

On behalf of the Python development community and the Python 3.6 release
team, I'm happy to announce the availability of Python 3.6.0b2. 3.6.0b2
is the second of four planned beta releases of Python 3.6, the next major
release of Python, and marks the end of the feature development phase
for 3.6.

Among the new major new features in Python 3.6 are:

* PEP 468 - Preserving the order of **kwargs in a function
* PEP 487 - Simpler customization of class creation
* PEP 495 - Local Time Disambiguation
* PEP 498 - Literal String Formatting
* PEP 506 - Adding A Secrets Module To The Standard Library
* PEP 509 - Add a private version to dict
* PEP 515 - Underscores in Numeric Literals
* PEP 519 - Adding a file system path protocol
* PEP 520 - Preserving Class Attribute Definition Order
* PEP 523 - Adding a frame evaluation API to CPython
* PEP 524 - Make os.urandom() blocking on Linux (during system startup)
* PEP 525 - Asynchronous Generators (provisional)
* PEP 526 - Syntax for Variable Annotations (provisional)
* PEP 528 - Change Windows console encoding to UTF-8 (provisional)
* PEP 529 - Change Windows filesystem encoding to UTF-8 (provisional)
* PEP 530 - Asynchronous Comprehensions

Please see "What?s New In Python 3.6" for more information:

https://docs.python.org/3.6/whatsnew/3.6.html

You can find Python 3.6.0b2 here:

https://www.python.org/downloads/release/python-360b2/

Beta releases are intended to give the wider community the opportunity
to test new features and bug fixes and to prepare their projects to
support the new feature release. We strongly encourage maintainers of
third-party Python projects to test with 3.6 during the beta phase and
report issues found to bugs.python.org as soon as possible. While the
release is feature complete entering the beta phase, it is possible that
features may be modified or, in rare cases, deleted up until the start
of the release candidate phase (2016-12-05). Our goal is have no changes
after rc1. To achieve that, it will be extremely important to get as
much exposure for 3.6 as possible during the beta phase. Please keep in
mind that this is a preview release and its use is not recommended for
production environments

The next planned release of Python 3.6 will be 3.6.0b3, currently
scheduled for 2016-10-31. More information about the release schedule
can be found here:

https://www.python.org/dev/peps/pep-0494/

--
  Ned Deily
  nad at python.org -- []


From steve at pearwood.info  Mon Oct 10 20:09:00 2016
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 11 Oct 2016 11:09:00 +1100
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
Message-ID: <20161011000859.GP22471@ando.pearwood.info>

On Mon, Oct 10, 2016 at 09:16:32PM +0000, Elliot Gorokhovsky wrote:

> Anyway, benchmarking technique aside, the point is that it it works well
> for small lists (i.e. doesn't affect performance).

You've been shown that there is something suspicious about your 
benchmarking technique, something that suggests that the timing results 
aren't trustworthy. Until you convince us that your timing results are 
reliable and trustworthy, you shouldn't be drawing *any* conclusions 
about your fastsort versus the standard sort.


-- 
Steve

From guido at python.org  Mon Oct 10 22:05:44 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 10 Oct 2016 19:05:44 -0700
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <20161011000859.GP22471@ando.pearwood.info>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
Message-ID: <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>

So maybe someone should explain to Elliott *why* his own benchmarks
are not trustworthy, rather than just repeat "use perf or timeit".
Actually, there are two things: (a) when something new comes along it
*always* needs to prove beyond a shadow of a doubt that it is actually
an improvement and not a timing artifact or a trick; (b) you can't
time sorting 10 values *once* and get a useful result. You have to do
it many times. And you have to make sure that creating a list of 10
random values isn't taken as part of your test -- that's tricky since
random() isn't all that fast; but it has to be done.

Although Elliott had it coming when he used needlessly offensive
language in his first post.

On Mon, Oct 10, 2016 at 5:09 PM, Steven D'Aprano <steve at pearwood.info> wrote:
> On Mon, Oct 10, 2016 at 09:16:32PM +0000, Elliot Gorokhovsky wrote:
>
>> Anyway, benchmarking technique aside, the point is that it it works well
>> for small lists (i.e. doesn't affect performance).
>
> You've been shown that there is something suspicious about your
> benchmarking technique, something that suggests that the timing results
> aren't trustworthy. Until you convince us that your timing results are
> reliable and trustworthy, you shouldn't be drawing *any* conclusions
> about your fastsort versus the standard sort.
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From njs at pobox.com  Mon Oct 10 22:44:17 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Mon, 10 Oct 2016 19:44:17 -0700
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <57FC15E2.4000900@canterbury.ac.nz>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <57FC15E2.4000900@canterbury.ac.nz>
Message-ID: <CAPJVwBmBx+ts8EVgKsB7y86-1eYHwwEW9dSdJ-rTshfPsUxvaw@mail.gmail.com>

On Mon, Oct 10, 2016 at 3:27 PM, Greg Ewing <greg.ewing at canterbury.ac.nz> wrote:
> Nathaniel Smith wrote:
>>
>> IIRC to handle
>> this gilectomy adds per-object mutexes that you have to hold whenever
>> you're mucking around with that object's internals.
>
>
> What counts as "mucking around with the object's internals",
> though?
>
> If I do the C equivalent of:
>
>    x = somedict[5]
>    x.dosomething()
>
> am I mucking around with the internals of somedict?
> Intuitively, I would not think so. But the way
> PyDict_GetItem currently works, this would be
> dangerous.

I guess you already have to be aware of this to some extent -- e.g.,
the C version of this code is dangerous, even though in Python it
would be fine:

  x = somedict[5]
  del somedict
  x.dosomething()

But Larry pointed out to me separately that while the per-object locks
exist and can be used, the they don't help as much as you'd hope
because the goal is to *not* require C API users to change their code.
So it's safe to have two simultaneous calls to PyDict_Clear, or to any
other single API call, because each PyDict_Clear call wil *internally*
takes the lock, so they won't stomp on each other. But any code
sequence that makes multiple C API calls and assumes that the world
won't shift under its feet becomes problematic. E.g. pre-GILectomy you
can do

Py_ssize_t count = PyList_Size(list);
for (int i = 0; i < count; ++i) {
    PyObject *obj = PyList_Get_Item(list, i);
    /* code which blithely proceeds without checking for obj == NULL */
    ...
}

but post-GILectomy, even putting aside the borrowing issues, this code
might segfault if someone shrinks the list when you aren't looking. I
can't see any way to avoid breaking the API here -- it seems like
callers will just have to take a lock or otherwise update their code.

I guess you can think of borrowed references as a particularly common
and insidious form of this problem -- so the tactical question is
whether it's better to look for a general solution like teaching
everyone to take locks, or try to attack borrowed references
specifically.

(Full API compatibility is never going to be preserved with the
GILectomy anyway -- if nothing else C extensions currently use the GIL
to protect internal globals, and that will have to be replaced by some
kind of more specific locking. So the open question is more like, can
the porting burden be kept low enough to make it viable.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From tim.peters at gmail.com  Mon Oct 10 23:02:12 2016
From: tim.peters at gmail.com (Tim Peters)
Date: Mon, 10 Oct 2016 22:02:12 -0500
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
Message-ID: <CAExdVNmxqDCHZ2tW=7nsc0TOY9YD_3g0UP8t618xOfo949PN1w@mail.gmail.com>

[please restrict follow-ups to python-ideas]

Let's not get hung up on meta-discussion here - I always thought "massive
clusterf**k" was a precise technical term anyway ;-)

While timing certainly needs to be done more carefully, it's obvious to me
that this approach _should_ pay off significantly when it applies.
Comparisons are extraordinarily expensive in Python, precisely because of
the maze of test-and-branch code it requires just to figure out which
bottom-level comparison function to invoke each time.  That's why I spent
months of my life (overall) devising a sequence of sorting algorithms for
Python that reduced the number of comparisons needed.

Note that when Python's current sort was adopted in Java, they still kept a
quicksort variant for "unboxed" builtin types.  The adaptive merge sort
incurs many overheads that often cost more than they save unless
comparisons are in fact very expensive compared to the cost of pointer
copying (and in Java comparison of unboxed types is cheap).  Indeed, for
native numeric types, where comparison is dirt cheap, quicksort generally
runs faster than mergesort despite that the former does _more_ comparisons
(because mergesort does so much more pointer-copying).

I had considered something "like this" for Python 2, but didn't pursue it
because comparison was defined between virtually any two types (34 < [1],
etc), and people were careless about that (both by design and by
accident).  In Python 3, comparison "blows up" for absurdly mixed types, so
specializing for homogeneously-typed lists is a more promising idea on the
face of it.

The comparisons needed to determine _whether_ a list's objects have a
common type is just len(list)-1 C-level pointer comparisons, and so goes
fast.  So I expect that, when it applies, this would speed even sorting an
already-ordered list with at least 2 elements.

For a mixed-type list with at least 2 elements, it will always be pure
loss.  But (a) I expect such lists are uncommon (and especially uncommon in
Python 3); and (b) a one-time scan doing C-level pointer comparisons until
finding a mismatched type is bound to be a relatively tiny cost compared to
the expense of all the "rich comparisons" that follow.

So +1 from me on pursuing this.

Elliot, please:

- Keep this on python-ideas.  python-dev is for current issues in Python
development, not for speculating about changes.

- Open an issue on the tracker:  https://bugs.python.org/

- At least browse the info for developers:
https://docs.python.org/devguide/

- Don't overlook Lib/test/sortperf.py.  As is, it should be a good test of
what your approach so far _doesn't_ help, since it sorts only lists of
floats (& I don't think you're special-casing them).  If the timing results
it reports aren't significantly hurt (and I expect they won't be), then add
specialization for floats too and gloat about the speedup :-)

- I expect tuples will also be worth specializing (complex sort keys are
often implemented as tuples).

Nice start! :-)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161010/31ebba99/attachment.html>

From elliot.gorokhovsky at gmail.com  Mon Oct 10 22:15:25 2016
From: elliot.gorokhovsky at gmail.com (Elliot Gorokhovsky)
Date: Tue, 11 Oct 2016 02:15:25 +0000
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
Message-ID: <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>

Right - sorry about that last bit!

I couldn't figure out how to use timeit/perf for this because the list has
to be reinitialized each time it gets sorted. So with time.time() I can
just wrap the sorting and cut the initialization out (and I could always
throw it in a for loop to do it as many times as necessary). But with
timeit/perf, as I understand, you just give it a number of iterations and a
code snippet and it loops for you. So I don't see how I could get it to cut
out the initialization. There's an option to provide setup code, of course,
but I need to set up before each trial, not just before the loop.

Regardless, one could always wrap the whole contribution with a length test
and disable for lists with less than, say, 1000 elements. And though for
the above reason I couldn't figure out how to benchmark properly for small
lists, it's clear that for large lists this gives a real improvement with
very little change in the code: the fact is that it really doesn't make
sense to do all this typechecking nlogn times when we can just get it over
with once at the beginning. The cost is very, very small, as demonstrated
by the last benchmark (which is for a 1e7 list, so it is probably more
valid with my crude technique), so heterogeneous lists don't get slowed
down perceptibly.

On Mon, Oct 10, 2016 at 8:06 PM Guido van Rossum <guido at python.org> wrote:

> So maybe someone should explain to Elliott *why* his own benchmarks
> are not trustworthy, rather than just repeat "use perf or timeit".
> Actually, there are two things: (a) when something new comes along it
> *always* needs to prove beyond a shadow of a doubt that it is actually
> an improvement and not a timing artifact or a trick; (b) you can't
> time sorting 10 values *once* and get a useful result. You have to do
> it many times. And you have to make sure that creating a list of 10
> random values isn't taken as part of your test -- that's tricky since
> random() isn't all that fast; but it has to be done.
>
> Although Elliott had it coming when he used needlessly offensive
> language in his first post.
>
> On Mon, Oct 10, 2016 at 5:09 PM, Steven D'Aprano <steve at pearwood.info>
> wrote:
> > On Mon, Oct 10, 2016 at 09:16:32PM +0000, Elliot Gorokhovsky wrote:
> >
> >> Anyway, benchmarking technique aside, the point is that it it works well
> >> for small lists (i.e. doesn't affect performance).
> >
> > You've been shown that there is something suspicious about your
> > benchmarking technique, something that suggests that the timing results
> > aren't trustworthy. Until you convince us that your timing results are
> > reliable and trustworthy, you shouldn't be drawing *any* conclusions
> > about your fastsort versus the standard sort.
> >
> >
> > --
> > Steve
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161011/da51ba70/attachment-0001.html>

From elliot.gorokhovsky at gmail.com  Mon Oct 10 22:56:09 2016
From: elliot.gorokhovsky at gmail.com (Elliot Gorokhovsky)
Date: Tue, 11 Oct 2016 02:56:09 +0000
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
Message-ID: <CANZJz4i6PEAoPBpfmLQsXrC50KUZm6Q4bqWUQ+p7ytzfKLHFoQ@mail.gmail.com>

So here's a simple attempt at taking lots of measurements just using
time.time() with lists of ints. The results are great, if they are valid
(which I leave to you to judge); even for lists with just one element, it's
16% faster! The columns are length, number of trials, and percent
improvement:

Integer benchmarks
1 10000000 1-fastsort()/sort(): 0.14896401672523163
5 2000000 1-fastsort()/sort(): 0.2915859704616376
10 1000000 1-fastsort()/sort(): 0.3576859149132914
1000 10000 1-fastsort()/sort(): 0.3230363920681264
1000000 10 1-fastsort()/sort(): 0.292595011903772

And here's the benchmark script:
from fastlist import FastList
import random; random.seed(42)
from time import time

def bench_list(size,  trials):
    L = [random.randrange(size) for _ in range(size)]
    trials = int(trials); size = int(size)

    fastlist_time = 0
    for _ in range(trials):
        F = FastList(L)
        start = time()
        F.fastsort()
        fastlist_time += time() - start


    defaultlist_time = 0
    for _ in range(trials):
        F = FastList(L)
        start = time()
        F.sort()
        defaultlist_time += time() - start


    print(size, trials,
          "1-fastsort()/sort():", 1-fastlist_time/defaultlist_time)

print("Integer benchmarks")
bench_list(1, 1e7)
bench_list(5, 1e7/5)
bench_list(10, 1e7/10)
bench_list(1000, 1e7/1000)
bench_list(1000000, 1e7/1e6)

Is that a valid way to benchmark the implementation?

On Mon, Oct 10, 2016 at 8:15 PM Elliot Gorokhovsky <
elliot.gorokhovsky at gmail.com> wrote:

> Right - sorry about that last bit!
>
> I couldn't figure out how to use timeit/perf for this because the list has
> to be reinitialized each time it gets sorted. So with time.time() I can
> just wrap the sorting and cut the initialization out (and I could always
> throw it in a for loop to do it as many times as necessary). But with
> timeit/perf, as I understand, you just give it a number of iterations and a
> code snippet and it loops for you. So I don't see how I could get it to cut
> out the initialization. There's an option to provide setup code, of course,
> but I need to set up before each trial, not just before the loop.
>
> Regardless, one could always wrap the whole contribution with a length
> test and disable for lists with less than, say, 1000 elements. And though
> for the above reason I couldn't figure out how to benchmark properly for
> small lists, it's clear that for large lists this gives a real improvement
> with very little change in the code: the fact is that it really doesn't
> make sense to do all this typechecking nlogn times when we can just get it
> over with once at the beginning. The cost is very, very small, as
> demonstrated by the last benchmark (which is for a 1e7 list, so it is
> probably more valid with my crude technique), so heterogeneous lists don't
> get slowed down perceptibly.
>
> On Mon, Oct 10, 2016 at 8:06 PM Guido van Rossum <guido at python.org> wrote:
>
> So maybe someone should explain to Elliott *why* his own benchmarks
>
> are not trustworthy, rather than just repeat "use perf or timeit".
>
> Actually, there are two things: (a) when something new comes along it
>
> *always* needs to prove beyond a shadow of a doubt that it is actually
>
> an improvement and not a timing artifact or a trick; (b) you can't
>
> time sorting 10 values *once* and get a useful result. You have to do
>
> it many times. And you have to make sure that creating a list of 10
>
> random values isn't taken as part of your test -- that's tricky since
>
> random() isn't all that fast; but it has to be done.
>
>
>
> Although Elliott had it coming when he used needlessly offensive
>
> language in his first post.
>
>
>
> On Mon, Oct 10, 2016 at 5:09 PM, Steven D'Aprano <steve at pearwood.info>
> wrote:
>
> > On Mon, Oct 10, 2016 at 09:16:32PM +0000, Elliot Gorokhovsky wrote:
>
> >
>
> >> Anyway, benchmarking technique aside, the point is that it it works well
>
> >> for small lists (i.e. doesn't affect performance).
>
> >
>
> > You've been shown that there is something suspicious about your
>
> > benchmarking technique, something that suggests that the timing results
>
> > aren't trustworthy. Until you convince us that your timing results are
>
> > reliable and trustworthy, you shouldn't be drawing *any* conclusions
>
> > about your fastsort versus the standard sort.
>
> >
>
> >
>
> > --
>
> > Steve
>
> > _______________________________________________
>
> > Python-Dev mailing list
>
> > Python-Dev at python.org
>
> > https://mail.python.org/mailman/listinfo/python-dev
>
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>
>
>
>
>
>
>
> --
>
> --Guido van Rossum (python.org/~guido)
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161011/8745f67b/attachment.html>

From tritium-list at sdamon.com  Mon Oct 10 23:16:33 2016
From: tritium-list at sdamon.com (tritium-list at sdamon.com)
Date: Mon, 10 Oct 2016 23:16:33 -0400
Subject: [Python-Dev] [Distutils] pythonhosted.org upload no longer works
In-Reply-To: <CAP7+vJLQqM8ow1cNZmR4QQ6_4F+X8c13RHS5_b84JyRdNgD-Sw@mail.gmail.com>
References: <CAFYqXL-=4PA9odjH11TT4DDRZnmz==W0OVMeTO_08OjURg7F=Q@mail.gmail.com>
 <CACac1F9hfkyYYz4xoSezLOV+RprUCJ-gVi7eknyCh_GUMLFYGg@mail.gmail.com>
 <CAFYqXL_jCQ1UYehmQ_+JS8=+c7yX8jqEy1-_+Ybj1SPcQ45Pmg@mail.gmail.com>
 <3E64EF6B-3F35-4BFB-BADF-EA7743A560DD@stufft.io>
 <CAP7+vJLQqM8ow1cNZmR4QQ6_4F+X8c13RHS5_b84JyRdNgD-Sw@mail.gmail.com>
Message-ID: <0a6601d2236d$de6f28c0$9b4d7a40$@hotmail.com>



> -----Original Message-----
> From: Distutils-SIG [mailto:distutils-sig-bounces+tritium-
> list=sdamon.com at python.org] On Behalf Of Guido van Rossum
> Sent: Monday, October 10, 2016 12:27 PM
> To: Donald Stufft <donald at stufft.io>
> Cc: Distutils <distutils-sig at python.org>; Giampaolo Rodola'
> <g.rodola at gmail.com>; python-dev <python-dev at python.org>
> Subject: Re: [Distutils] [Python-Dev] pythonhosted.org upload no longer
> works
> 
> Honestly I like readthedocs a lot, and I actually *don't* like docs
> that look too much like the standard Python docs -- it should be clear
> to readers (subliminally, through page style, not just be parsing the
> URL) that they're reading third party docs.
> 

That?s quite tangential to pythonhosted.org hosting documentation... But that is a consequence of the theme used by docs.python.org/2/ using the default sphinx theme.  That problem was already solved for python3 docs - they don?t use the default theme.  I really don?t see how this is related at all to the uploading of docs to pythonhosted.org.


From guido at python.org  Tue Oct 11 00:03:11 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 10 Oct 2016 21:03:11 -0700
Subject: [Python-Dev] [Distutils] pythonhosted.org upload no longer works
In-Reply-To: <0a6601d2236d$de6f28c0$9b4d7a40$@hotmail.com>
References: <CAFYqXL-=4PA9odjH11TT4DDRZnmz==W0OVMeTO_08OjURg7F=Q@mail.gmail.com>
 <CACac1F9hfkyYYz4xoSezLOV+RprUCJ-gVi7eknyCh_GUMLFYGg@mail.gmail.com>
 <CAFYqXL_jCQ1UYehmQ_+JS8=+c7yX8jqEy1-_+Ybj1SPcQ45Pmg@mail.gmail.com>
 <3E64EF6B-3F35-4BFB-BADF-EA7743A560DD@stufft.io>
 <CAP7+vJLQqM8ow1cNZmR4QQ6_4F+X8c13RHS5_b84JyRdNgD-Sw@mail.gmail.com>
 <0a6601d2236d$de6f28c0$9b4d7a40$@hotmail.com>
Message-ID: <CAP7+vJ+WHQQ7SjWCH0Bo3po=XoH7JL51Ki49i8nnOi0x-mprOw@mail.gmail.com>

I only mentioned it because the OP mentioned the theme as one of his
reasons to prefer one over the other.

On Mon, Oct 10, 2016 at 8:16 PM,  <tritium-list at sdamon.com> wrote:
>
>
>> -----Original Message-----
>> From: Distutils-SIG [mailto:distutils-sig-bounces+tritium-
>> list=sdamon.com at python.org] On Behalf Of Guido van Rossum
>> Sent: Monday, October 10, 2016 12:27 PM
>> To: Donald Stufft <donald at stufft.io>
>> Cc: Distutils <distutils-sig at python.org>; Giampaolo Rodola'
>> <g.rodola at gmail.com>; python-dev <python-dev at python.org>
>> Subject: Re: [Distutils] [Python-Dev] pythonhosted.org upload no longer
>> works
>>
>> Honestly I like readthedocs a lot, and I actually *don't* like docs
>> that look too much like the standard Python docs -- it should be clear
>> to readers (subliminally, through page style, not just be parsing the
>> URL) that they're reading third party docs.
>>
>
> That?s quite tangential to pythonhosted.org hosting documentation... But that is a consequence of the theme used by docs.python.org/2/ using the default sphinx theme.  That problem was already solved for python3 docs - they don?t use the default theme.  I really don?t see how this is related at all to the uploading of docs to pythonhosted.org.
>



-- 
--Guido van Rossum (python.org/~guido)

From ncoghlan at gmail.com  Tue Oct 11 00:33:03 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 11 Oct 2016 14:33:03 +1000
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAP7+vJJg8uRKpVv8FkCP2=M-g5Oah3CAxF07+ohu9zRDKfVmoQ@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CAFT4OTHKHAsKdzBLhaQfWZkymUJ=64JY+H4=F+kV5HTR263TEg@mail.gmail.com>
 <CAP7+vJL54KQ2t2ARwa==JFeatPSV9UqsQp8MiHw-=ObYSJH0HA@mail.gmail.com>
 <4f233082-4006-2d0e-8cfc-e7d7632e425c@hastings.org>
 <CAP7+vJJg8uRKpVv8FkCP2=M-g5Oah3CAxF07+ohu9zRDKfVmoQ@mail.gmail.com>
Message-ID: <CADiSq7eP0jKAmixkM1Z2fhgD=PGjzY8K=4xZFxrS1zeDFxpxZQ@mail.gmail.com>

On 11 October 2016 at 08:49, Guido van Rossum <guido at python.org> wrote:
> On Mon, Oct 10, 2016 at 1:57 PM, Larry Hastings <larry at hastings.org> wrote:
>> For now I'm going to leave the names as-is and just change the semantics
>> everywhere.  If this approach really works, and if we decide we want to
>> merge it back into trunk--and those are both still big if's--we can revisit
>> this decision.
>
> Changing something as subtle as this to an API sounds really bad even
> in an experimental branch. But it's your branch and you can do what
> you want (then again, why did you bring it up here? :-).

I'm guessing Larry was hoping someone else would see a possible
solution that he'd missed.

On that front, while I don't see an alternative to making borrowed
references real references in a GIL-free Python, I do think we may
need to keep the borrowed/new distinction for the sake of folks
writing cross-version compatible code. To explain why, consider these
scenarios:

* CPython with a GIL:
  - new references MUST be decref'ed
  - borrowed references MUST NOT be decref'ed

* CPython without a GIL:
  - new references MUST be decref'ed
  - borrowed references MUST be decref'ed

Assuming that the GILectomy does end up needing to implicitly treat
all borrowed references as new references, that suggests we're going
to need APIs like "PyBorrowed_DECREF" to provide the "only decref when
built without the GIL" semantics, and the rules for GIL-independent
source code would become:

- new references must be decref'ed using the normal APIs
- borrowed references must be decref'ed using the PyBorrowed* APIs
(with a no-op compatibility shim for older GIL-only CPython versions)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From greg.ewing at canterbury.ac.nz  Tue Oct 11 00:40:02 2016
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 11 Oct 2016 17:40:02 +1300
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
Message-ID: <57FC6D22.2090109@canterbury.ac.nz>

Elliot Gorokhovsky wrote:
> I ran the 
> benchmark a couple of times and the numbers seem to exactly line up 
> something like one in five times; perhaps not that crazy considering 
> they're executing nearly the same code?

Could this be a result of a time being measured in seconds
somewhere and then divided down?

> *** 1e7 ints + 1 float (to disable the optimization while keeping the precheck)***
> F.fastsort(): 7.57237982749939
> F.sort(): 7.666172504425049

This result looks a bit suspicious too -- it's hard to see
how fastsort could be faster even when the optimisation
is not being used.

-- 
Greg


From larry at hastings.org  Tue Oct 11 02:41:46 2016
From: larry at hastings.org (Larry Hastings)
Date: Tue, 11 Oct 2016 08:41:46 +0200
Subject: [Python-Dev] Fwd: Re: PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <f05379f4-3849-7a1f-19df-52f2c8138003@hastings.org>
References: <f05379f4-3849-7a1f-19df-52f2c8138003@hastings.org>
Message-ID: <9a163876-e659-0663-4574-2c41cb6b4f25@hastings.org>


Hit "Reply" instead of "Reply All" last night, oops.  Forwarding to the 
list for posterity's sakes.

/
/arry/

-------- Forwarded Message --------
Subject: 	Re: [Python-Dev] PyWeakref_GetObject() borrows its reference 
from... whom?
Date: 	Mon, 10 Oct 2016 23:01:10 +0200
From: 	Larry Hastings <larry at hastings.org>
To: 	Nathaniel Smith <njs at pobox.com>


On 10/10/2016 07:50 PM, Nathaniel Smith wrote:
> If we say that borrowing reference from a dict is one of the things
> that counts as mucking about with that dict, and thus requires you to
> hold the dict lock for as long as you hold the borrowed reference,
> then all should be well.

That's not how locking works in the Gilectomy right now.  If you call 
PyDict_GetItem(), it locks the dict at the beginning, then looks up the 
thingy, then releases the lock just before returning.  It's hard for me 
to imagine how the dict would magically know when it could drop the 
borrowed reference returned by PyDict_GetItem().


> I assume Larry is way ahead of us on this,

I would say "that's your first mistake!" but I bet you've made others.  
Anyway, tbh most of the time I feel awfully unqualified to work on the 
Gilectomy.


Fools rush in,


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161011/f68baa5d/attachment.html>

From p.f.moore at gmail.com  Tue Oct 11 04:22:13 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 11 Oct 2016 09:22:13 +0100
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
Message-ID: <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>

On 11 October 2016 at 03:15, Elliot Gorokhovsky
<elliot.gorokhovsky at gmail.com> wrote:
> There's an option to provide setup code, of course, but I need to set up
> before each trial, not just before the loop.

Typically, I would just run the benchmark separately for each case,
and then you'd do

# Case 1
python -m perf timeit -s 'setup; code; here' 'code; to; be; timed; here'
[Results 1]
# Case 2
python -m perf timeit -s 'setup; code; here' 'code; to; be; timed; here'
[Results 2]

The other advantage of doing it this way is that you can post your
benchmark command lines, which will allow people to see what you're
timing, and if there *are* any problems (such as a method lookup that
skews the results) people can point them out.

Paul

From jeanpierreda at gmail.com  Tue Oct 11 07:17:28 2016
From: jeanpierreda at gmail.com (Devin Jeanpierre)
Date: Tue, 11 Oct 2016 04:17:28 -0700
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
Message-ID: <CABicbJK5NK=Bu8GBptfQ3xKKPat=8JARaaxr-rb0FBGi0uWOKg@mail.gmail.com>

> Huh?  In all other circumstances, a "borrowed" reference is exactly that:
X has a reference, and you are relying on X's reference to keep the object
alive.  Borrowing from a borrowed reference is simply a chain of these; Z
borrows from Y, Y borrows from X, and X is the original person who did the
incref.  But you're borrowing from something specific, somebody who the API
guarantees has a legitimate reference on the object and won't drop it while
you're using it.  I bet for every other spot in the API I can tell you from
whom you're borrowing the reference.
>
> In contrast, the "borrowed" reference returned by PyWeakRef_GetObject()
seems to be "borrowed" from some unspecified entity.  The fact that the
object is live in Python directly implies that, yes, *somebody* must have a
reference, somewhere.  But ISTM (and apparently you) that this is relying
on the GIL preventing that unknown other actor from dropping their
reference while you've borrow it.  A guarantee that the post-Gilectomy
Python interpreter can no longer make!

Let me try again. The only rule for borrowing: x (borrowed reference) is
only guaranteed to be alive for as long as y (source) is guaranteed to be
alive. At least if you phrase it carefully enough, weakrefs still fit the
bill -- your borrowed reference is alive for as long as the weakref is
alive. Of course, the lifetime for a weakref is completely undefined in
GILectomized Python, and barely controllable even in regular CPython. So
this is not a great API.

At the very least we can contort definitions into making it plausible that
we borrowed from the weakref.  If we can only analyze code through the lens
of borrowing, we have no choice except to look at it this way.

> In any case, I see nothing in the documentation that suggests "borrowed
only means unowned" as you suggest.  In contrast, the documentation seems
to suggest that the metaphor is how I understood it; that when you "borrow"
a reference, there is another object who has a reference and you're relying
on their reference to keep the object alive.

This is its own big tangent. Sorry,

Your link is all I have too. It doesn't spell it out. AFAIK there are
exactly two kinds of references discussed anywhere in the docs: owned
references, where you are obligated to call Py_DECREF, and everything else.
Python exclusively uses the term "borrowed references" for that "everything
else". I don't know why. It's a good way to encourage reasonable practices,
I guess.

As the docs themselves note, this metaphor is useful but flawed: e.g. you
can "borrow" the same thing and the original is still usable. But I'd go in
another direction -- the metaphor is sufficient to show safety of some
code, but some safe code exists where we can't as easily use "borrowing" to
talk about why it's safe, and might need to introduce new concepts or
switch tactics.

PyObject* x = PyList_New(0); // x is a new owned reference.
PyObject* y = x;  // y is a borrowed reference.
PyObject* z = x;  // z is also a borrowed reference.
Py_INCREF(y);  // y has been upgraded to an owned reference.
Py_CLEAR(x);  // the original reference z borrowed from disappeared.
// This is provably safe, but you can't use the "borrowing" metaphor for
that proof without introducing new concepts.

do_stuff(z);


// You might wonder, "why would anyone ever do that?!?"
// One reason is because some functions "steal" references, so you need to
borrow before handing off ownership:

// y is an owned reference.
my_struct->foo = y // borrowed a reference to y.
PyTuple_SetItem(my_strict->some_tuple, 0, y); // stole y.
// Now, in a super-strict interpretation, y is no longer "valid", and the
original borrowing relationship has been broken.
// We should ideally reason "as if" it borrowed from my_struct->some_tuple,
even though it didn't.
// (Obviously, this example is still a little overcomplicated, but you get
how this might happen IRL, yeah?
//  e.g. maybe z already existed and the API stealing a reference doesn't
have a GET_ITEM macro.)
// [Note that this has a similar problem to weakref: another thread could
mutate the tuple and delete the object. Yikes!]


I hope those examples made sense.

weakref is playing with fire in a similar way: PyWeakref_GetObject is safe
because someone still exists, and there is a lock that guarantees they
exist as long as you don't release that lock and don't run any code that
might delete a reference.  (In particular, it's guaranteed safe to
immediately upgrade the borrowed reference -- or is without gilectomy.)
 Unlike the stealing tuple example, it isn't clear where the owned
reference is.

So what I mean by saying it's a red herring, is that the correctness of
code doesn't hinge on how easy it is to apply the concept of borrowing, but
exclusively on lifetimes and whether your borrowed reference can be proven
to lie within the object lifetime. If it could not at all be explained
sensibly as a "borrow" from anyone, it can still be right -- that would
only make it confusing, and dangerous.  (But that's a foregone conclusion
at this point.)

-- Devin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161011/fb6cab7a/attachment-0001.html>

From larry at hastings.org  Tue Oct 11 07:49:07 2016
From: larry at hastings.org (Larry Hastings)
Date: Tue, 11 Oct 2016 13:49:07 +0200
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
Message-ID: <c9dc9377-97fe-98f7-db9c-d564648e9384@hastings.org>


By the way, just to finish playing this new fun game "Who'd I Borrow 
This Reference From?":


On 10/10/2016 11:45 AM, Chris Angelico wrote:
> On Mon, Oct 10, 2016 at 8:35 PM, Larry Hastings <larry at hastings.org> wrote:
>> I bet for every other spot in the API I can tell you from
>> whom you're borrowing the reference.
> Okay. Here's a test:
>
> PyObject* PyList_GetItem(PyObject *list, Py_ssize_t index)
> Return value: Borrowed reference.
>
> Presumably you own a reference to the list itself before you call
> this, and the list has a reference to its items. But suppose another
> thread clear()s the list immediately after you call this; whose
> reference are you borrowing now? The list's is gone.

As you say: you've borrowed the reference from the list.  With the GIL 
this is safe.  After the Gilectomy it's not always safe.  It *can* be: 
for example, if you allocated the list locally in the current thread, 
and it hasn't escaped the thread yet, you can assert that in this case 
reference will never go away on you.  But in general this is an API that 
has to change for the Gilectomy.


//arry/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161011/138bd3d1/attachment.html>

From p.f.moore at gmail.com  Tue Oct 11 09:51:08 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 11 Oct 2016 14:51:08 +0100
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
 <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
 <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>
Message-ID: <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>

On 11 October 2016 at 14:04, Elliot Gorokhovsky
<elliot.gorokhovsky at gmail.com> wrote:
> Right, that sounds good, but there's just one thing I don't understand
> that's keeping me from using it. Namely, I would define a benchmark list L
> in my setup, and then I would have code="F=FastList(L);F.fastsort()". The
> problem here is I'm measuring the constructor time along with the sort time,
> right, so wouldn't that mess up the benchmark? Or does timeit separate the
> times?

That would mess up your times. Put F=FastList(L) in your setup.

Paul

From rosuav at gmail.com  Tue Oct 11 10:00:53 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 12 Oct 2016 01:00:53 +1100
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
 <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
 <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>
 <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>
Message-ID: <CAPTjJmqcP7poqsAdbWdjj8Jhey3sXx-Tz5Cx-FZ+2DrH_srJEQ@mail.gmail.com>

On Wed, Oct 12, 2016 at 12:51 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 11 October 2016 at 14:04, Elliot Gorokhovsky
> <elliot.gorokhovsky at gmail.com> wrote:
>> Right, that sounds good, but there's just one thing I don't understand
>> that's keeping me from using it. Namely, I would define a benchmark list L
>> in my setup, and then I would have code="F=FastList(L);F.fastsort()". The
>> problem here is I'm measuring the constructor time along with the sort time,
>> right, so wouldn't that mess up the benchmark? Or does timeit separate the
>> times?
>
> That would mess up your times. Put F=FastList(L) in your setup.

But then you're resorting an already-sorted list, which may well have
different timings (it certainly does in timsort).

ChrisA

From p.f.moore at gmail.com  Tue Oct 11 10:24:59 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 11 Oct 2016 15:24:59 +0100
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CAPTjJmqcP7poqsAdbWdjj8Jhey3sXx-Tz5Cx-FZ+2DrH_srJEQ@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
 <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
 <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>
 <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>
 <CAPTjJmqcP7poqsAdbWdjj8Jhey3sXx-Tz5Cx-FZ+2DrH_srJEQ@mail.gmail.com>
Message-ID: <CACac1F8gfTjG9N+2VTbr4NThUOZf550BBmDcPpDnyi7qZgffwQ@mail.gmail.com>

On 11 October 2016 at 15:00, Chris Angelico <rosuav at gmail.com> wrote:
> On Wed, Oct 12, 2016 at 12:51 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>> On 11 October 2016 at 14:04, Elliot Gorokhovsky
>> <elliot.gorokhovsky at gmail.com> wrote:
>>> Right, that sounds good, but there's just one thing I don't understand
>>> that's keeping me from using it. Namely, I would define a benchmark list L
>>> in my setup, and then I would have code="F=FastList(L);F.fastsort()". The
>>> problem here is I'm measuring the constructor time along with the sort time,
>>> right, so wouldn't that mess up the benchmark? Or does timeit separate the
>>> times?
>>
>> That would mess up your times. Put F=FastList(L) in your setup.
>
> But then you're resorting an already-sorted list, which may well have
> different timings (it certainly does in timsort).

Why would it be already sorted? I assume FastList(L) is simply a
wrapper round a normal list that has a modified sort method with the
optimisation included.

Of course, that's essentially the point here - without seeing the
code, we're (to an extent) guessing.
Paul

From rosuav at gmail.com  Tue Oct 11 10:26:00 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 12 Oct 2016 01:26:00 +1100
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CACac1F8gfTjG9N+2VTbr4NThUOZf550BBmDcPpDnyi7qZgffwQ@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
 <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
 <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>
 <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>
 <CAPTjJmqcP7poqsAdbWdjj8Jhey3sXx-Tz5Cx-FZ+2DrH_srJEQ@mail.gmail.com>
 <CACac1F8gfTjG9N+2VTbr4NThUOZf550BBmDcPpDnyi7qZgffwQ@mail.gmail.com>
Message-ID: <CAPTjJmoftfN+r5PgNxkm3RLc9ExF9tKmM9tPiL0nxfp4OWWxMg@mail.gmail.com>

On Wed, Oct 12, 2016 at 1:24 AM, Paul Moore <p.f.moore at gmail.com> wrote:
> On 11 October 2016 at 15:00, Chris Angelico <rosuav at gmail.com> wrote:
>> On Wed, Oct 12, 2016 at 12:51 AM, Paul Moore <p.f.moore at gmail.com> wrote:
>>> On 11 October 2016 at 14:04, Elliot Gorokhovsky
>>> <elliot.gorokhovsky at gmail.com> wrote:
>>>> Right, that sounds good, but there's just one thing I don't understand
>>>> that's keeping me from using it. Namely, I would define a benchmark list L
>>>> in my setup, and then I would have code="F=FastList(L);F.fastsort()". The
>>>> problem here is I'm measuring the constructor time along with the sort time,
>>>> right, so wouldn't that mess up the benchmark? Or does timeit separate the
>>>> times?
>>>
>>> That would mess up your times. Put F=FastList(L) in your setup.
>>
>> But then you're resorting an already-sorted list, which may well have
>> different timings (it certainly does in timsort).
>
> Why would it be already sorted? I assume FastList(L) is simply a
> wrapper round a normal list that has a modified sort method with the
> optimisation included.
>
> Of course, that's essentially the point here - without seeing the
> code, we're (to an extent) guessing.

After the first call, the list will be sorted. Any subsequent attempts
will use the sorted list.

ChrisA

From elliot.gorokhovsky at gmail.com  Tue Oct 11 09:04:23 2016
From: elliot.gorokhovsky at gmail.com (Elliot Gorokhovsky)
Date: Tue, 11 Oct 2016 13:04:23 +0000
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
 <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
Message-ID: <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>

Right, that sounds good, but there's just one thing I don't understand
that's keeping me from using it. Namely, I would define a benchmark list L
in my setup, and then I would have code="F=FastList(L);F.fastsort()". The
problem here is I'm measuring the constructor time along with the sort
time, right, so wouldn't that mess up the benchmark? Or does timeit
separate the times?

On Tue, Oct 11, 2016, 2:22 AM Paul Moore <p.f.moore at gmail.com> wrote:

> On 11 October 2016 at 03:15, Elliot Gorokhovsky
> <elliot.gorokhovsky at gmail.com> wrote:
> > There's an option to provide setup code, of course, but I need to set up
> > before each trial, not just before the loop.
>
> Typically, I would just run the benchmark separately for each case,
> and then you'd do
>
> # Case 1
> python -m perf timeit -s 'setup; code; here' 'code; to; be; timed; here'
> [Results 1]
> # Case 2
> python -m perf timeit -s 'setup; code; here' 'code; to; be; timed; here'
> [Results 2]
>
> The other advantage of doing it this way is that you can post your
> benchmark command lines, which will allow people to see what you're
> timing, and if there *are* any problems (such as a method lookup that
> skews the results) people can point them out.
>
> Paul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161011/d14bda25/attachment.html>

From elliot.gorokhovsky at gmail.com  Tue Oct 11 10:32:00 2016
From: elliot.gorokhovsky at gmail.com (Elliot Gorokhovsky)
Date: Tue, 11 Oct 2016 14:32:00 +0000
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
 <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
 <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>
 <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>
Message-ID: <CANZJz4hMvzku+Tg1Y2-gHLnWDy12=4rEky9rNA1aSGtGp=owpg@mail.gmail.com>

But the sort mutates F...does the setup get executed each time? I thought
it's just at the beginning. So then F gets mutated (sorted) and subsequent
sorts time wrong.

On Tue, Oct 11, 2016, 7:51 AM Paul Moore <p.f.moore at gmail.com> wrote:

> On 11 October 2016 at 14:04, Elliot Gorokhovsky
> <elliot.gorokhovsky at gmail.com> wrote:
> > Right, that sounds good, but there's just one thing I don't understand
> > that's keeping me from using it. Namely, I would define a benchmark list
> L
> > in my setup, and then I would have code="F=FastList(L);F.fastsort()". The
> > problem here is I'm measuring the constructor time along with the sort
> time,
> > right, so wouldn't that mess up the benchmark? Or does timeit separate
> the
> > times?
>
> That would mess up your times. Put F=FastList(L) in your setup.
>
> Paul
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161011/615dbb11/attachment.html>

From p.f.moore at gmail.com  Tue Oct 11 11:12:12 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Tue, 11 Oct 2016 16:12:12 +0100
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CANZJz4hMvzku+Tg1Y2-gHLnWDy12=4rEky9rNA1aSGtGp=owpg@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
 <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
 <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>
 <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>
 <CANZJz4hMvzku+Tg1Y2-gHLnWDy12=4rEky9rNA1aSGtGp=owpg@mail.gmail.com>
Message-ID: <CACac1F-B+Nsnk3qM2S2wFQkSueED9mSaOX1OTXYH4WJV+mNqTQ@mail.gmail.com>

On 11 October 2016 at 15:32, Elliot Gorokhovsky
<elliot.gorokhovsky at gmail.com> wrote:
> But the sort mutates F...does the setup get executed each time? I thought
> it's just at the beginning. So then F gets mutated (sorted) and subsequent
> sorts time wrong.

Did I not say earlier - sorry. I'm suggesting that you put each timing
run into a separate Python process.

# Optimised code
python -m perf timeit -s "from mymod import FastList; L=<whatever you
do to create the list>;F=FastList(L)" "F.fastsort()"
# Core sort routine
python -m perf timeit -s "L=<whatever you do to create the list>" "L.sort()"
# Or maybe, if FastList exposes the existing sort function as well
# Non-optimised code
python -m perf timeit -s "from mymod import FastList; L=<whatever you
do to create the list>;F=FastList(L)" "F.sort()"

Paul.

From tjreedy at udel.edu  Tue Oct 11 20:08:22 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Tue, 11 Oct 2016 20:08:22 -0400
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <CAPTjJmoftfN+r5PgNxkm3RLc9ExF9tKmM9tPiL0nxfp4OWWxMg@mail.gmail.com>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
 <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
 <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>
 <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>
 <CAPTjJmqcP7poqsAdbWdjj8Jhey3sXx-Tz5Cx-FZ+2DrH_srJEQ@mail.gmail.com>
 <CACac1F8gfTjG9N+2VTbr4NThUOZf550BBmDcPpDnyi7qZgffwQ@mail.gmail.com>
 <CAPTjJmoftfN+r5PgNxkm3RLc9ExF9tKmM9tPiL0nxfp4OWWxMg@mail.gmail.com>
Message-ID: <ntjutl$h9$1@blaine.gmane.org>

On 10/11/2016 10:26 AM, Chris Angelico wrote:

> After the first call, the list will be sorted. Any subsequent attempts
> will use the sorted list.

This seems like a generic issue with timing mutation methods.  Is the 
mutated output suitable input for another mutation.  With list.reverse, 
the output is suitable.  Ditto for other permutations that are 
independent of the data, including random.shuffle.  With list.pop, the 
number of mutations has to be limited to the length of the list.  With 
list.sort, the list needs to be 'restored' -- either copied or shuffled 
each time.  So it seems that one is stuck with timing 'restore', restore 
+ sort_a, restore + sort_b, subtracting the timing for restore, and 
comparing.  But I am sure Tim worked this out in his test code, which 
should be reused, perhaps updated with Viktor's perf module to get the 
most stable timings possible.

-- 
Terry Jan Reedy


From tim.peters at gmail.com  Tue Oct 11 20:46:54 2016
From: tim.peters at gmail.com (Tim Peters)
Date: Tue, 11 Oct 2016 19:46:54 -0500
Subject: [Python-Dev] Optimizing list.sort() by checking type in advance
In-Reply-To: <ntjutl$h9$1@blaine.gmane.org>
References: <CANZJz4iP4BPn932yV4n3_CkhxXStc9vBg1FZFi+Ohfz0yk4dZw@mail.gmail.com>
 <CANZJz4h21EE-R4d1XPqQy2u8JyWrgid2mC33htC0Lmq3RQQpmg@mail.gmail.com>
 <CAPJVwBnTLG-Kd1ZvTtngK-UVshN13FBJRQSsXi4mdEVFDb+R1A@mail.gmail.com>
 <CANZJz4i2aZeCcdUG3zE8smNxymf7j1+-HCxZ8a4FACeWddFctw@mail.gmail.com>
 <20161011000859.GP22471@ando.pearwood.info>
 <CAP7+vJL2H+x=wCOYEKtGo+0Xc1Opw+mKHF3+u=r1e25bdGaS4Q@mail.gmail.com>
 <CANZJz4ja3PYupucUX9b+HYym=Viez5GfXpBo=UH3Rmw6va8+Sw@mail.gmail.com>
 <CACac1F8Yc=Vtiu=afVt3yK1jaa21wwNtJtS0eaoA-CcLWsYW+w@mail.gmail.com>
 <CANZJz4iFsV=Vb+WCCyLEx-+XJhMhw=v1NNG_=GC8CLCbTwBT4A@mail.gmail.com>
 <CACac1F8kz4_GuXO743MV5Y2xG2xShNRJWzU2mgbxo1wW6__YWg@mail.gmail.com>
 <CAPTjJmqcP7poqsAdbWdjj8Jhey3sXx-Tz5Cx-FZ+2DrH_srJEQ@mail.gmail.com>
 <CACac1F8gfTjG9N+2VTbr4NThUOZf550BBmDcPpDnyi7qZgffwQ@mail.gmail.com>
 <CAPTjJmoftfN+r5PgNxkm3RLc9ExF9tKmM9tPiL0nxfp4OWWxMg@mail.gmail.com>
 <ntjutl$h9$1@blaine.gmane.org>
Message-ID: <CAExdVNn9A-eungF3ecOoTBFKg4sOUdYvpW-nf+OrGXqrGQkHDg@mail.gmail.com>

[Terry Reedy]
>
> This seems like a generic issue with timing mutation methods
> ...
> But I am sure Tim worked this out in his test code, which should be
> reused, perhaps updated with Viktor's perf module to get the most
> stable timings possible.

sortperf.py is older than me ;-)  It's not at all intended to give
fine-grained timings:  it's a sledgehammer than runs each case _only
once_, to confirm/refute _big_ changes.

So it's useful qualitatively when a "big claimed change" is at issue,
but useless for quantifying beyond "wow!" or "ouch!" or "meh".

Since this is a "big claimed change", a "wow!" outcome is good-enough
confirmation.  And a "meh" outcome would be almost good enough for
cases where the new specializations don't apply - although I haven't
seen any numbers from sortperf.py for those cases yet.  I expect a
"meh" outcome for those, based on reasoning - but may be surprised by
reality.

From songofacandy at gmail.com  Wed Oct 12 00:08:05 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 12 Oct 2016 13:08:05 +0900
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467 (was:
 [Python-ideas] Adding bytes.frombuffer() constructor
Message-ID: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>

Hi.

While there were no reply to my previous post on Python-ideas ML,
Now I'm sure about bytes.frombuffer() is worth enough.

Let's describe why I think it's important.


Background
=========

>From Python 3.4, bytearray is good solution for I/O buffer, thanks to
#19087 [1].
Actually, asyncio uses bytearray as I/O buffer often.

When bytearray is used for read buffer, we can parse received data on bytearray
directly, and consume it.  For example, read until '\r\n' is easier
than io.BytesIO().

Sample code:

    def read_line(buf: bytearray) -> bytes:
        try:
            n = buf.index(b'\r\n')
        except ValueError:
            return b''

        line = bytes(buf)[:n]  # bytearray -> bytes -> bytes
        del buf[:n+2]
        return line


    buf = bytearray(b'foo\r\nbar\r\nbaz\r\n')

    while True:
        line = read_line(buf)
        if not line:
            break
        print(line)

As you can see, redundant temporary bytes is used.
This is not ideal for performance and memory efficiency.

Since code like this is typically in lower level code (e.g. asyncio),
performance and
efficiency is important.

[1] https://bugs.python.org/issue19087

(Off topic: bytearray is nice for write buffer too. written =
s.send(buf); del buf[:written];)


Memoryview problem
=================

To avoid redundant copy of `line = bytes(buf)[:n]`, current solution
is using memoryview.

First code I wrote is: `line = bytes(memoryview(buf)[:n])`.

On CPython, it works fine.  But `del buff[:n+2]` in next line may fail
on other Python
implementations.  Changing bytearray size is inhibited while
memoryview is alive.

So right code is:

with memoryview(buf) as m:
    line = bytes(m[:n])

The problem of memoryview approach is:

* Overhead: creating temporary memoryview, __enter__, and __exit__. (see below)

* It isn't "one obvious way": Developers including me may forget to
use context manager.
  And since it works on CPython, it's hard to point it out.


Quick benchmark:

(temporary bytes)
$ python3 -m perf timeit -s 'buf =
bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(buf)[:3]'
....................
Median +- std dev: 652 ns +- 19 ns

(temporary memoryview without "with"
$ python3 -m perf timeit -s 'buf =
bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(memoryview(buf)[:3])'
....................
Median +- std dev: 886 ns +- 26 ns

(temporary memoryview with "with")
$ python3 -m perf timeit -s 'buf = bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- '
with memoryview(buf) as m:
    bytes(m[:3])
'
....................
Median +- std dev: 1.11 us +- 0.03 us


Proposed solution
===============

Adding one more constructor to bytes:

    # when length=-1 (default), use until end of *byteslike*.
    bytes.frombuffer(byteslike, length=-1, offset=0)

With ths API

    with memoryview(buf) as m:
        line = bytes(m[:n])

becomes

    line = bytes.frombuffer(buf, n)


Thanks,

-- 
INADA Naoki  <songofacandy at gmail.com>

From njs at pobox.com  Wed Oct 12 01:03:37 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 11 Oct 2016 22:03:37 -0700
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
Message-ID: <CAPJVwBn4yLo-+Dwk6vsiFLaPdi8wC8D1k-p9=CVzOrO+WpYw-Q@mail.gmail.com>

On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki <songofacandy at gmail.com> wrote:
> From Python 3.4, bytearray is good solution for I/O buffer, thanks to
> #19087 [1].
> Actually, asyncio uses bytearray as I/O buffer often.

Whoa what?! This is awesome, I had no idea that bytearray had O(1)
deletes at the front. I literally reimplemented this myself on type of
bytearray for some 3.5-only code recently because I assumed bytearray
had the same asymptotics as list, and AFAICT this is totally
undocumented. Shouldn't we write this down somewhere? Maybe here? ->
https://docs.python.org/3/library/functions.html#bytearray

>     # when length=-1 (default), use until end of *byteslike*.
>     bytes.frombuffer(byteslike, length=-1, offset=0)

This seems reasonable to me. Mostly I've dealt with the problem by
writing functions like your read_line so that they return bytearray
objects, since that's the only thing you can get out of a bytearray
without double-copying. But returning bytes would feel a bit cleaner,
and this would allow that.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Wed Oct 12 01:07:49 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 12 Oct 2016 15:07:49 +1000
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
Message-ID: <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>

I don't think it makes sense to add any more ideas to PEP 467. That
needed to be a PEP because it proposed breaking backwards
compatibility in a couple of areas, and because of the complex history
of Python 3's "bytes-as-tuple-of-ints" and Python 2's "bytes-as-str"
semantics.

Other enhancements to the binary data handling APIs in Python 3 can be
considered on their own merits.

On 12 October 2016 at 14:08, INADA Naoki <songofacandy at gmail.com> wrote:
> Memoryview problem
> =================
>
> To avoid redundant copy of `line = bytes(buf)[:n]`, current solution
> is using memoryview.
>
> First code I wrote is: `line = bytes(memoryview(buf)[:n])`.
>
> On CPython, it works fine.  But `del buff[:n+2]` in next line may fail
> on other Python
> implementations.  Changing bytearray size is inhibited while
> memoryview is alive.
>
> So right code is:
>
> with memoryview(buf) as m:
>     line = bytes(m[:n])
>
> The problem of memoryview approach is:
>
> * Overhead: creating temporary memoryview, __enter__, and __exit__. (see below)
>
> * It isn't "one obvious way": Developers including me may forget to
> use context manager.
>   And since it works on CPython, it's hard to point it out.

To add to the confusion, there's also
https://docs.python.org/3/library/stdtypes.html#memoryview.tobytes
giving:

    line = memoryview(buf)[:n].tobytes()

However, folks *do* need to learn that many mutable data types will
lock themselves against modification while you have a live memory view
on them, so it's important to release views promptly and reliably when
we don't need them any more.

> Quick benchmark:
>
> (temporary bytes)
> $ python3 -m perf timeit -s 'buf =
> bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(buf)[:3]'
> ....................
> Median +- std dev: 652 ns +- 19 ns
>
> (temporary memoryview without "with"
> $ python3 -m perf timeit -s 'buf =
> bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(memoryview(buf)[:3])'
> ....................
> Median +- std dev: 886 ns +- 26 ns
>
> (temporary memoryview with "with")
> $ python3 -m perf timeit -s 'buf = bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- '
> with memoryview(buf) as m:
>     bytes(m[:3])
> '
> ....................
> Median +- std dev: 1.11 us +- 0.03 us

This is normal though, as memory views trade lower O(N) costs (reduced
data copying) for higher O(1) setup costs (creating and managing the
view, indirection for data access).

> Proposed solution
> ===============
>
> Adding one more constructor to bytes:
>
>     # when length=-1 (default), use until end of *byteslike*.
>     bytes.frombuffer(byteslike, length=-1, offset=0)
>
> With ths API
>
>     with memoryview(buf) as m:
>         line = bytes(m[:n])
>
> becomes
>
>     line = bytes.frombuffer(buf, n)

Does that need to be a method on the builtin rather than a separate
helper function, though? Once you define:

    def snapshot(buf, length=None, offset=0):
        with memoryview(buf) as m:
            return m[offset:length].tobytes()

then that can be replaced by a more optimised C implementation without
users needing to care about the internal details.

That is, getting back to a variant on one of Serhiy's suggestions in
the last PEP 467 discussion, it may make sense for us to offer a
"buffertools" library that's specifically aimed at supporting
efficient buffer manipulation operations that minimise data copying.
The pure Python implementations would work entirely through
memoryview, but we could also have selected C accelerated operations
if that showed a noticeable improvement on asyncio's benchmarks.

Regards,
Nick.

P.S. The length/offset API design is also problematic due to the way
it differs from range() & slice(), but I don't think it makes sense to
get into that kind of detail before discussing the larger question of
adding a new helper module for working efficiently with memory buffers
vs further widening the method API for the builtin bytes type

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Wed Oct 12 01:32:30 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 12 Oct 2016 08:32:30 +0300
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
Message-ID: <ntkhtd$lcd$1@blaine.gmane.org>

On 12.10.16 07:08, INADA Naoki wrote:
> Sample code:
>
>     def read_line(buf: bytearray) -> bytes:
>         try:
>             n = buf.index(b'\r\n')
>         except ValueError:
>             return b''
>
>         line = bytes(buf)[:n]  # bytearray -> bytes -> bytes

Wouldn't be more correct to write this as bytes(buf[:n])?

> Adding one more constructor to bytes:
>
>     # when length=-1 (default), use until end of *byteslike*.
>     bytes.frombuffer(byteslike, length=-1, offset=0)

This interface looks unusual. Would not be better to support the 
interface of buffer in Python 2: buffer(object [, offset[, size]])?



From storchaka at gmail.com  Wed Oct 12 01:53:48 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 12 Oct 2016 08:53:48 +0300
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CAPJVwBn4yLo-+Dwk6vsiFLaPdi8wC8D1k-p9=CVzOrO+WpYw-Q@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CAPJVwBn4yLo-+Dwk6vsiFLaPdi8wC8D1k-p9=CVzOrO+WpYw-Q@mail.gmail.com>
Message-ID: <ntkj59$4ip$1@blaine.gmane.org>

On 12.10.16 08:03, Nathaniel Smith wrote:
> On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki <songofacandy at gmail.com> wrote:
>> From Python 3.4, bytearray is good solution for I/O buffer, thanks to
>> #19087 [1].
>> Actually, asyncio uses bytearray as I/O buffer often.
>
> Whoa what?! This is awesome, I had no idea that bytearray had O(1)
> deletes at the front. I literally reimplemented this myself on type of
> bytearray for some 3.5-only code recently because I assumed bytearray
> had the same asymptotics as list, and AFAICT this is totally
> undocumented. Shouldn't we write this down somewhere? Maybe here? ->
> https://docs.python.org/3/library/functions.html#bytearray

I afraid this is CPython implementation detail (like string 
concatenation optimization). Other implementations can have O(N) deletes 
at the front of bytearray.



From njs at pobox.com  Wed Oct 12 02:31:04 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 11 Oct 2016 23:31:04 -0700
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
Message-ID: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>

On Tue, Oct 11, 2016 at 10:53 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 12.10.16 08:03, Nathaniel Smith wrote:
>>
>> On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki <songofacandy at gmail.com>
>> wrote:
>>>
>>> From Python 3.4, bytearray is good solution for I/O buffer, thanks to
>>> #19087 [1].
>>> Actually, asyncio uses bytearray as I/O buffer often.
>>
>>
>> Whoa what?! This is awesome, I had no idea that bytearray had O(1)
>> deletes at the front. I literally reimplemented this myself on type of
>> bytearray for some 3.5-only code recently because I assumed bytearray
>> had the same asymptotics as list, and AFAICT this is totally
>> undocumented. Shouldn't we write this down somewhere? Maybe here? ->
>> https://docs.python.org/3/library/functions.html#bytearray
>
>
> I afraid this is CPython implementation detail (like string concatenation
> optimization). Other implementations can have O(N) deletes at the front of
> bytearray.

Well, it shouldn't be :-).

The problem with the string concatenation optimization is that to
work, it requires both the use of refcounting GC and that you get
lucky with how the underlying malloc implementation happened to lay
things out in memory. Obviously it shouldn't be part of the language
spec.

But amortized O(1) deletes from the front of bytearray are totally
different, and more like amortized O(1) appends to list: there are
important use cases[1] that simply cannot be implemented without some
feature like this, and putting the implementation inside bytearray is
straightforward, deterministic, and more efficiently than hacking
together something on top. Python should just guarantee it, IMO.

-n

[1] My use case is parsing HTTP out of a receive buffer. If deleting
the first k bytes of an N byte buffer is O(N), then not only does
parsing becomes O(N^2) in the worst case, but it's the sort of O(N^2)
that random untrusted network clients can trigger at will to DoS your
server.

-- 
Nathaniel J. Smith -- https://vorpus.org

From storchaka at gmail.com  Wed Oct 12 03:17:12 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 12 Oct 2016 10:17:12 +0300
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
Message-ID: <ntko1m$tgp$1@blaine.gmane.org>

On 12.10.16 09:31, Nathaniel Smith wrote:
> But amortized O(1) deletes from the front of bytearray are totally
> different, and more like amortized O(1) appends to list: there are
> important use cases[1] that simply cannot be implemented without some
> feature like this, and putting the implementation inside bytearray is
> straightforward, deterministic, and more efficiently than hacking
> together something on top. Python should just guarantee it, IMO.
>
> -n
>
> [1] My use case is parsing HTTP out of a receive buffer. If deleting
> the first k bytes of an N byte buffer is O(N), then not only does
> parsing becomes O(N^2) in the worst case, but it's the sort of O(N^2)
> that random untrusted network clients can trigger at will to DoS your
> server.

Deleting from buffer can be avoided if pass the starting index together 
with the buffer. For example:

     def read_line(buf: bytes, start: int) -> (bytes, int):
         try:
             end = buf.index(b'\r\n', start)
         except ValueError:
             return b'', start

         return buf[start:end], end+2



From njs at pobox.com  Wed Oct 12 04:01:43 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 12 Oct 2016 01:01:43 -0700
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <ntko1m$tgp$1@blaine.gmane.org>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
Message-ID: <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>

On Wed, Oct 12, 2016 at 12:17 AM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 12.10.16 09:31, Nathaniel Smith wrote:
>>
>> But amortized O(1) deletes from the front of bytearray are totally
>> different, and more like amortized O(1) appends to list: there are
>> important use cases[1] that simply cannot be implemented without some
>> feature like this, and putting the implementation inside bytearray is
>> straightforward, deterministic, and more efficiently than hacking
>> together something on top. Python should just guarantee it, IMO.
>>
>> -n
>>
>> [1] My use case is parsing HTTP out of a receive buffer. If deleting
>> the first k bytes of an N byte buffer is O(N), then not only does
>> parsing becomes O(N^2) in the worst case, but it's the sort of O(N^2)
>> that random untrusted network clients can trigger at will to DoS your
>> server.
>
>
> Deleting from buffer can be avoided if pass the starting index together with
> the buffer. For example:
>
>     def read_line(buf: bytes, start: int) -> (bytes, int):
>         try:
>             end = buf.index(b'\r\n', start)
>         except ValueError:
>             return b'', start
>
>         return buf[start:end], end+2

It's more complicated than that -- the right algorithm is the one that
Antoine implemented in 3.4. But yes, having implemented this by hand,
I am aware that it can be implemented by hand :-). My point is that
forcing everyone who writes network code in Python to do that is
silly, especially given that CPython's apparently been shipping this
feature for years.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From songofacandy at gmail.com  Wed Oct 12 05:34:18 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 12 Oct 2016 18:34:18 +0900
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
Message-ID: <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>

On Wed, Oct 12, 2016 at 2:07 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> I don't think it makes sense to add any more ideas to PEP 467. That
> needed to be a PEP because it proposed breaking backwards
> compatibility in a couple of areas, and because of the complex history
> of Python 3's "bytes-as-tuple-of-ints" and Python 2's "bytes-as-str"
> semantics.
>
> Other enhancements to the binary data handling APIs in Python 3 can be
> considered on their own merits.
>

I see.  My proposal should be another PEP (if PEP is required).


>>
>> * It isn't "one obvious way": Developers including me may forget to
>> use context manager.
>>   And since it works on CPython, it's hard to point it out.
>
> To add to the confusion, there's also
> https://docs.python.org/3/library/stdtypes.html#memoryview.tobytes
> giving:
>
>     line = memoryview(buf)[:n].tobytes()
>
> However, folks *do* need to learn that many mutable data types will
> lock themselves against modification while you have a live memory view
> on them, so it's important to release views promptly and reliably when
> we don't need them any more.
>

I agree.
io.TextWrapper objects reports ResourceWarning for unclosed file.
I think same warning for unclosed memoryview objects may help developers.


>> Quick benchmark:
>>
>> (temporary bytes)
>> $ python3 -m perf timeit -s 'buf =
>> bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(buf)[:3]'
>> ....................
>> Median +- std dev: 652 ns +- 19 ns
>>
>> (temporary memoryview without "with"
>> $ python3 -m perf timeit -s 'buf =
>> bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- 'bytes(memoryview(buf)[:3])'
>> ....................
>> Median +- std dev: 886 ns +- 26 ns
>>
>> (temporary memoryview with "with")
>> $ python3 -m perf timeit -s 'buf = bytearray(b"foo\r\nbar\r\nbaz\r\n")' -- '
>> with memoryview(buf) as m:
>>     bytes(m[:3])
>> '
>> ....................
>> Median +- std dev: 1.11 us +- 0.03 us
>
> This is normal though, as memory views trade lower O(N) costs (reduced
> data copying) for higher O(1) setup costs (creating and managing the
> view, indirection for data access).

Yes.  When data is small, benefit of less data copy can be hidden easily.

One big difficulty of I/O frameworks like asyncio is: we can't assume data size.
Framework should be optimized for both of many small chunks and large data.

With memoryview, when we optimize for large data (e.g. downloading large file),
performance for massive small data (e.g. small JSON API) become worse.

Actually, one pull request is gave up to use memoryview because of it.

https://github.com/python/asyncio/pull/395#issuecomment-249044218


>
>> Proposed solution
>> ===============
>>
>> Adding one more constructor to bytes:
>>
>>     # when length=-1 (default), use until end of *byteslike*.
>>     bytes.frombuffer(byteslike, length=-1, offset=0)
>>
>> With ths API
>>
>>     with memoryview(buf) as m:
>>         line = bytes(m[:n])
>>
>> becomes
>>
>>     line = bytes.frombuffer(buf, n)
>
> Does that need to be a method on the builtin rather than a separate
> helper function, though? Once you define:
>
>     def snapshot(buf, length=None, offset=0):
>         with memoryview(buf) as m:
>             return m[offset:length].tobytes()
>
> then that can be replaced by a more optimised C implementation without
> users needing to care about the internal details.

I'm thinking about adding such helper function in asyncio speedup C extension.
But there are some other non-blocking I/O frameworks: Tornado,
Twisted, and curio.

And relying on C extention make harder to optimize for other Python
implementation.
If it is in standard library, PyPy and other Python implementation can
optimize it.


>
> That is, getting back to a variant on one of Serhiy's suggestions in
> the last PEP 467 discussion, it may make sense for us to offer a
> "buffertools" library that's specifically aimed at supporting
> efficient buffer manipulation operations that minimise data copying.
> The pure Python implementations would work entirely through
> memoryview, but we could also have selected C accelerated operations
> if that showed a noticeable improvement on asyncio's benchmarks.
>

It seems nice idea. I'll read the discussion.


> Regards,
> Nick.
>
> P.S. The length/offset API design is also problematic due to the way
> it differs from range() & slice(), but I don't think it makes sense to
> get into that kind of detail before discussing the larger question of
> adding a new helper module for working efficiently with memory buffers
> vs further widening the method API for the builtin bytes type
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

I avoid slice API intentionally, because if it seems like slice,
someone will propose
adding `step` support only for consistency.

But, as Serhiy said, consistent with old buffer API is nice.

-- 
INADA Naoki  <songofacandy at gmail.com>

From songofacandy at gmail.com  Wed Oct 12 05:42:04 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 12 Oct 2016 18:42:04 +0900
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <ntkhtd$lcd$1@blaine.gmane.org>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <ntkhtd$lcd$1@blaine.gmane.org>
Message-ID: <CAEfz+Tzj30t_-9gv27pP_nMDq2-FmgCB38B_g4PbwivR7SWO-w@mail.gmail.com>

On Wed, Oct 12, 2016 at 2:32 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
> On 12.10.16 07:08, INADA Naoki wrote:
>>
>> Sample code:
>>
>>     def read_line(buf: bytearray) -> bytes:
>>         try:
>>             n = buf.index(b'\r\n')
>>         except ValueError:
>>             return b''
>>
>>         line = bytes(buf)[:n]  # bytearray -> bytes -> bytes
>
>
> Wouldn't be more correct to write this as bytes(buf[:n])?

Yes, you're right!
I shouldn't copy whole data only for cast from bytearray to byte.

>
>> Adding one more constructor to bytes:
>>
>>     # when length=-1 (default), use until end of *byteslike*.
>>     bytes.frombuffer(byteslike, length=-1, offset=0)
>
>
> This interface looks unusual. Would not be better to support the interface
> of buffer in Python 2: buffer(object [, offset[, size]])?
>

It looks better.

(Actually speaking, I love deprecated old buffer for simplicity.
memoryview supports non bytes-like complex data types.)

Thanks,

-- 
INADA Naoki  <songofacandy at gmail.com>

From songofacandy at gmail.com  Wed Oct 12 06:28:34 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 12 Oct 2016 19:28:34 +0900
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <ntko1m$tgp$1@blaine.gmane.org>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
Message-ID: <CAEfz+Ty42BpqLo--JaHZ7vRu670pm79cxf=aOs68Ei8Non0LWA@mail.gmail.com>

>>
>> [1] My use case is parsing HTTP out of a receive buffer. If deleting
>> the first k bytes of an N byte buffer is O(N), then not only does
>> parsing becomes O(N^2) in the worst case, but it's the sort of O(N^2)
>> that random untrusted network clients can trigger at will to DoS your
>> server.
>
>
> Deleting from buffer can be avoided if pass the starting index together with
> the buffer. For example:
>
>     def read_line(buf: bytes, start: int) -> (bytes, int):
>         try:
>             end = buf.index(b'\r\n', start)
>         except ValueError:
>             return b'', start
>
>         return buf[start:end], end+2
>

In case of asyncio, we can't assume the order of append and consume.

For example, stream processing HTTP chunked response.
Append to receive buffer and consume a chunk in buffer can happen in
arbitrary order.
That's why bytes is not good for receive buffer. Efficient append is
"must have".

For example, Torando implements receive buffer by deque of bytes. See this code.
https://github.com/tornadoweb/tornado/blob/master/tornado/iostream.py#L784-L817

When Tornado drop Python 2.7 support, they can use bytearray, and
iostream can be more simple and fast.

So I hope "amortized O(1) deletion from the front" is language spec,
at least for Python 3.5+

-- 
INADA Naoki  <songofacandy at gmail.com>

From victor.stinner at gmail.com  Wed Oct 12 07:52:20 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 12 Oct 2016 13:52:20 +0200
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
Message-ID: <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>

2016-10-12 11:34 GMT+02:00 INADA Naoki <songofacandy at gmail.com>:
> I see.  My proposal should be another PEP (if PEP is required).

I don't think that adding a single method deserves its own method.

I like the idea with Serhiy's API (as Python 2 buffer constructor):

bytes.frombuf(buffer, [offset, size])
bytearray.frombuf(buffer, [offset, size])
memoryview.frombuf(buffer, [offset, size])

Victor

From victor.stinner at gmail.com  Wed Oct 12 07:55:34 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 12 Oct 2016 13:55:34 +0200
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
 <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>
Message-ID: <CAMpsgwZVazgFRy3f+SH6+AHmHCvD9s_q8SVCKhQnegdhVWUv-Q@mail.gmail.com>

2016-10-12 10:01 GMT+02:00 Nathaniel Smith <njs at pobox.com>:
> It's more complicated than that -- the right algorithm is the one that
> Antoine implemented in 3.4.
> (...)
> My point is that
> forcing everyone who writes network code in Python to do that is
> silly, especially given that CPython's apparently been shipping this
> feature for years.

"For years" means since March 2014, Python 3.4.0 release, so 2 years ago.

We can document the optimization as a CPython implementation detail
and explain that it's only in Python >= 3.4.

So an application which should work on Python 2.7 as well cannot rely
on this optimization for example.

Victor

From Nikolaus at rath.org  Wed Oct 12 11:35:09 2016
From: Nikolaus at rath.org (Nikolaus Rath)
Date: Wed, 12 Oct 2016 08:35:09 -0700
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
In-Reply-To: <CAPJVwBn4yLo-+Dwk6vsiFLaPdi8wC8D1k-p9=CVzOrO+WpYw-Q@mail.gmail.com>
 (Nathaniel Smith's message of "Tue, 11 Oct 2016 22:03:37 -0700")
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CAPJVwBn4yLo-+Dwk6vsiFLaPdi8wC8D1k-p9=CVzOrO+WpYw-Q@mail.gmail.com>
Message-ID: <877f9dziky.fsf@thinkpad.rath.org>

On Oct 11 2016, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, Oct 11, 2016 at 9:08 PM, INADA Naoki <songofacandy at gmail.com> wrote:
>> From Python 3.4, bytearray is good solution for I/O buffer, thanks to
>> #19087 [1].
>> Actually, asyncio uses bytearray as I/O buffer often.
>
> Whoa what?! This is awesome, I had no idea that bytearray had O(1)
> deletes at the front. I literally reimplemented this myself on type of
> bytearray for some 3.5-only code recently because I assumed bytearray
> had the same asymptotics as list, and AFAICT this is totally
> undocumented. 

Indeed, same here.


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             ?Time flies like an arrow, fruit flies like a Banana.?

From turnbull.stephen.fw at u.tsukuba.ac.jp  Wed Oct 12 12:37:56 2016
From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull)
Date: Thu, 13 Oct 2016 01:37:56 +0900
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
Message-ID: <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>

Victor Stinner writes:
 > 2016-10-12 11:34 GMT+02:00 INADA Naoki <songofacandy at gmail.com>:

 > > I see.  My proposal should be another PEP (if PEP is required).
 > 
 > I don't think that adding a single method deserves its own method.

You mean "deserves own PEP", right?  I interpreted Nick to say that
"the reasons that applied to PEP 367 don't apply here, so you can Just
Do It" (subject to the usual criteria for review, but omit the PEP).

I'm not sure whether he was channeling Guido or that should be
qualified with an IMO or IMHO.




From victor.stinner at gmail.com  Wed Oct 12 12:42:01 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 12 Oct 2016 18:42:01 +0200
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CAMpsgwaU2MUKM6gs8E5Ba9B1ksDKUof-bd4TV9t8+1YJbF__Xw@mail.gmail.com>

Oops, right, I wanted to write "I don't think that adding a single
method deserves its own PEP."

Victor

2016-10-12 18:37 GMT+02:00 Stephen J. Turnbull
<turnbull.stephen.fw at u.tsukuba.ac.jp>:
> Victor Stinner writes:
>  > 2016-10-12 11:34 GMT+02:00 INADA Naoki <songofacandy at gmail.com>:
>
>  > > I see.  My proposal should be another PEP (if PEP is required).
>  >
>  > I don't think that adding a single method deserves its own method.
>
> You mean "deserves own PEP", right?  I interpreted Nick to say that
> "the reasons that applied to PEP 367 don't apply here, so you can Just
> Do It" (subject to the usual criteria for review, but omit the PEP).
>
> I'm not sure whether he was channeling Guido or that should be
> qualified with an IMO or IMHO.
>
>
>

From tjreedy at udel.edu  Wed Oct 12 15:44:19 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 12 Oct 2016 15:44:19 -0400
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CAEfz+Tzj30t_-9gv27pP_nMDq2-FmgCB38B_g4PbwivR7SWO-w@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <ntkhtd$lcd$1@blaine.gmane.org>
 <CAEfz+Tzj30t_-9gv27pP_nMDq2-FmgCB38B_g4PbwivR7SWO-w@mail.gmail.com>
Message-ID: <ntm3qj$54f$1@blaine.gmane.org>

On 10/12/2016 5:42 AM, INADA Naoki wrote:
> On Wed, Oct 12, 2016 at 2:32 PM, Serhiy Storchaka <storchaka at gmail.com> wrote:
>> On 12.10.16 07:08, INADA Naoki wrote:
>>>
>>> Sample code:
>>>
>>>     def read_line(buf: bytearray) -> bytes:
>>>         try:
>>>             n = buf.index(b'\r\n')
>>>         except ValueError:
>>>             return b''
>>>
>>>         line = bytes(buf)[:n]  # bytearray -> bytes -> bytes
>>
>>
>> Wouldn't be more correct to write this as bytes(buf[:n])?
>
> Yes, you're right!
> I shouldn't copy whole data only for cast from bytearray to byte.

Also, why do the conversion from bytearray to bytes?  It is definitely 
not always needed.

 >>> ba = bytearray(b'abc')
 >>> b = b'def'
 >>> ba + b
bytearray(b'abcdef')
 >>> b'%s %s' % (ba, b)
b'abc def'
 >>> b + ba
b'defabc'
 >>> ba.extend(b)
 >>> ba
bytearray(b'abcdef')

Even if it is sometimes needed, why do it always?  The essence of 
read_line is to slice out a line, delete it from the buffer, and return 
the line.  Let the caller explicitly convert when needed.

-- 
Terry Jan Reedy


From njs at pobox.com  Wed Oct 12 17:14:48 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 12 Oct 2016 14:14:48 -0700
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <CAMpsgwZVazgFRy3f+SH6+AHmHCvD9s_q8SVCKhQnegdhVWUv-Q@mail.gmail.com>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
 <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>
 <CAMpsgwZVazgFRy3f+SH6+AHmHCvD9s_q8SVCKhQnegdhVWUv-Q@mail.gmail.com>
Message-ID: <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>

On Wed, Oct 12, 2016 at 4:55 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> 2016-10-12 10:01 GMT+02:00 Nathaniel Smith <njs at pobox.com>:
>> It's more complicated than that -- the right algorithm is the one that
>> Antoine implemented in 3.4.
>> (...)
>> My point is that
>> forcing everyone who writes network code in Python to do that is
>> silly, especially given that CPython's apparently been shipping this
>> feature for years.
>
> "For years" means since March 2014, Python 3.4.0 release, so 2 years ago.
>
> We can document the optimization as a CPython implementation detail
> and explain that it's only in Python >= 3.4.
>
> So an application which should work on Python 2.7 as well cannot rely
> on this optimization for example.

The proposal is that it should be documented as being part of the
language spec starting in 3.4 (or whatever). So applications that
support Python 2.7 can't rely on it, sure. But if I have an
application that requires, say, 3.5+ but I don't want to depend on
CPython-only implementation details, then I'm still allowed to use it.

AFAIK basically the only project that would be affected by this is
PyPy, and I when I asked on #pypy they said:

<cfbolz> njs`: I think we either plan to or already support this

so I'm not sure why this is controversial.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From njs at pobox.com  Wed Oct 12 17:16:16 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Wed, 12 Oct 2016 14:16:16 -0700
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <CAEfz+Ty42BpqLo--JaHZ7vRu670pm79cxf=aOs68Ei8Non0LWA@mail.gmail.com>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
 <CAEfz+Ty42BpqLo--JaHZ7vRu670pm79cxf=aOs68Ei8Non0LWA@mail.gmail.com>
Message-ID: <CAPJVwBksTycN6Q=n9LYWgzAKnuD4Qgw9y=ziObQaWBpDjVT8aQ@mail.gmail.com>

On Wed, Oct 12, 2016 at 3:28 AM, INADA Naoki <songofacandy at gmail.com> wrote:
> When Tornado drop Python 2.7 support, they can use bytearray, and
> iostream can be more simple and fast.

FYI 2.7 does have bytearray. (You still have to implement the O(1)
deletion part as a layer on top, like Victor points out, but I suspect
that'd still be dramatically simpler than what they're doing now...)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From rosuav at gmail.com  Wed Oct 12 20:05:11 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 13 Oct 2016 11:05:11 +1100
Subject: [Python-Dev] Python ROCKS! Thanks guys! [anecdote]
Message-ID: <CAPTjJmoA99s8cO62tqKvTTe1Mm9WJW8rFee+FUCVgzdci7sZdA@mail.gmail.com>

I work with a full-stack web development bootcamp. Most of the course
focuses on JavaScript (Node.js, React, jQuery, etc), but there's a
one-week period in which each student gets to pick some technology to
learn, and at the end of the week, demos to the group some project
s/he has mastered. Two chose to learn Python, and I've been mentoring
them through this week.

The comments from each of them have been fairly glowing. Python is
this incredible thing that has immense power and flexibility;
significant-whitespace hasn't been a cause of confusion (not even
mentioned after the first day); and

The most notable features of Python, for these two
JS-only-up-until-now guys, are the simplicity of the 'for' loop
(including that you don't need lots of different forms - you can
iterate over a dictionary without having to learn some new type of
loop), the list comprehension, and metaprogramming - mainly function
decorator syntax. And both of them are starting to talk about being
"converts" to Python :)

Great job, all. Not that it's particularly difficult to compete with a
language that was originally designed and developed in under two
weeks, but still. :D

ChrisA

From rymg19 at gmail.com  Wed Oct 12 21:01:27 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Wed, 12 Oct 2016 20:01:27 -0500
Subject: [Python-Dev] Python ROCKS! Thanks guys! [anecdote]
In-Reply-To: <CAPTjJmoA99s8cO62tqKvTTe1Mm9WJW8rFee+FUCVgzdci7sZdA@mail.gmail.com>
References: <CAPTjJmoA99s8cO62tqKvTTe1Mm9WJW8rFee+FUCVgzdci7sZdA@mail.gmail.com>
Message-ID: <CAO41-mPEmPxqvrs2T2nRoSfVY86v3xZgCBsn8Ok0yJG0hadSPw@mail.gmail.com>

On Wed, Oct 12, 2016 at 7:05 PM, Chris Angelico <rosuav at gmail.com> wrote:

> I work with a full-stack web development bootcamp. Most of the course
> focuses on *JavaScript (Node.js, React, jQuery, etc),*


Poor students... ;)


> but there's a
> one-week period in which each student gets to pick some technology to
> learn, and at the end of the week, demos to the group some project
> s/he has mastered. Two chose to learn Python, and I've been mentoring
> them through this week.
>
> The comments from each of them have been fairly glowing. Python is
> this incredible thing that has immense power and flexibility;
> significant-whitespace hasn't been a cause of confusion (not even
> mentioned after the first day); and
>
> The most notable features of Python, for these two
> JS-only-up-until-now guys, are the simplicity of the 'for' loop
> (including that you don't need lots of different forms - you can
> iterate over a dictionary without having to learn some new type of
> loop), the list comprehension, and metaprogramming - mainly function
> decorator syntax. And both of them are starting to talk about being
> "converts" to Python :)
>
> Great job, all. Not that it's particularly difficult to compete with a
> language that was originally designed and developed in under two
> weeks, but still. :D
>
>
I think the craziest thing is probably that, based on how you said it,
these two students haven't even begun to enter the entire Python standard
library, which you'd have to download a zillion npm modules (like the
glorious left-pad) in order to match. Once they realize that, they'll never
be going back!


> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> rymg19%40gmail.com
>



-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161012/df395148/attachment.html>

From ncoghlan at gmail.com  Wed Oct 12 22:54:39 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 13 Oct 2016 12:54:39 +1000
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
Message-ID: <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>

On 13 October 2016 at 02:37, Stephen J. Turnbull
<turnbull.stephen.fw at u.tsukuba.ac.jp> wrote:
> Victor Stinner writes:
>  > 2016-10-12 11:34 GMT+02:00 INADA Naoki <songofacandy at gmail.com>:
>
>  > > I see.  My proposal should be another PEP (if PEP is required).
>  >
>  > I don't think that adding a single method deserves its own method.
>
> You mean "deserves own PEP", right?  I interpreted Nick to say that
> "the reasons that applied to PEP 367 don't apply here, so you can Just
> Do It" (subject to the usual criteria for review, but omit the PEP).

Sort of. Adding this to PEP 467 doesn't make sense (as it's not
related to easing migration from Python 2 or addressing the
mutable->immutable design legacy), but I don't have an opinion yet on
whether this should be a PEP or not - that really depends on whether
we tackle it as an implementation detail of asyncio, or as a public
API in its own right.

Method proliferation on builtins is a Big Deal(TM), and efficient
buffer management for IO protocol development is a relatively arcane
speciality (as well as one where there are dedicated OS level
capabilities we may want to exploit some day), which is why I think a
dedicated helper module is likely a better way to go. For example:

- add `asyncio._iobuffers` as a pure Python memoryview based
implementation of the desired buffer management semantics
- add `_iobuffers` as an optional asyncio independent accelerator
module for `asyncio._iobuffers`

If that works out satisfactorily, *then* consider a PEP to either make
`iobuffers` a public module in its own right (ala the `selectors`
module from the original asyncio implementation), or to expose some of
its features directly via the builtin binary data types.

The logical leap I strongly disagree with is going straight from
"asyncio needs some better low level IO buffer manipulation
primitives" to "we should turn the builtin types into low level IO
buffer manipulation primitives that are sufficient for asyncio's
needs". The notion of "we shouldn't need to define our own domain
specific helper libraries" isn't a given for standard library modules
any more than it is for 3rd party ones.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From songofacandy at gmail.com  Wed Oct 12 22:59:00 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 13 Oct 2016 11:59:00 +0900
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <ntm3qj$54f$1@blaine.gmane.org>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <ntkhtd$lcd$1@blaine.gmane.org>
 <CAEfz+Tzj30t_-9gv27pP_nMDq2-FmgCB38B_g4PbwivR7SWO-w@mail.gmail.com>
 <ntm3qj$54f$1@blaine.gmane.org>
Message-ID: <CAEfz+TzdcaYdZvSZ=NJeuWPnYyeywEaf1=fdKdDja+iyrnJvSw@mail.gmail.com>

>
> Also, why do the conversion from bytearray to bytes?  It is definitely not
> always needed.
>
>>>> ba = bytearray(b'abc')
>>>> b = b'def'
>>>> ba + b
> bytearray(b'abcdef')
>>>> b'%s %s' % (ba, b)
> b'abc def'
>>>> b + ba
> b'defabc'
>>>> ba.extend(b)
>>>> ba
> bytearray(b'abcdef')
>
> Even if it is sometimes needed, why do it always?  The essence of read_line
> is to slice out a line, delete it from the buffer, and return the line.  Let
> the caller explicitly convert when needed.
>
> --
> Terry Jan Reedy
>

Because it's public module API.

While bytearray is mostly API compatible (passes duck typing),
isinstance(b, bytes) is False when b is bytearray.

So, I feel changing return type from bytes to bytearray is last option.
I want to return bytes if possible.

-- 
INADA Naoki  <songofacandy at gmail.com>

From ncoghlan at gmail.com  Wed Oct 12 23:31:02 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 13 Oct 2016 13:31:02 +1000
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
Message-ID: <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>

On 13 October 2016 at 12:54, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Method proliferation on builtins is a Big Deal(TM)

I wanted to quantify this concept, so here's a quick metric that helps
convey how every time we add a new builtin method we're immediately
making Python harder to comprehend:

    >>> def get_builtin_types():
    ...     import builtins
    ...     return {name:obj for name, obj in vars(builtins).items()
if isinstance(obj, type) and not (name.startswith("__") or
issubclass(obj, BaseException))}
    ...
    >>> len(get_builtin_types())
    26
    >>> def get_builtin_methods():
    ...     return [(name, method_name) for name, obj in
get_builtin_types().items() for method_name, method in
vars(obj).items() if not method_name.startswith("__")]
    ...
    >>> len(get_builtin_methods())
    230

Putting special purpose functionality behind an import gate helps to
provide a more explicit context of use (in this case, IO buffer
manipulation) vs the relatively domain independent namespace that is
the builtins.

Cheers,
Nick.

P.S. Since I was poking around in the builtins anyway, here are some
other simple language complexity metrics:

    >>> len(vars(builtins))
    151
    >>> def get_interpreter_builtins():
    ...     import builtins
    ...     return {name:obj for name, obj in vars(builtins).items()
if name.startswith("__")}
    ...
    >>> len(get_interpreter_builtins())
    8
    >>> def get_builtin_exceptions():
    ...     import builtins
    ...     return {name:obj for name, obj in vars(builtins).items()
if isinstance(obj, type) and issubclass(obj, BaseException)}
    ...
    >>> len(get_builtin_exceptions())
    65
    >>> def get_builtin_functions():
    ...     import builtins
    ...     return {name:obj for name, obj in vars(builtins).items()
if isinstance(obj, type(repr))}
    ...
    >>> len(get_builtin_functions())
    42
    >>> def get_other_builtins():
    ...     import builtins
    ...     return {name:obj for name, obj in vars(builtins).items()
if not name.startswith("__") and not isinstance(obj, (type,
type(repr)))}
    ...
    >>> len(get_other_builtins())
    12

The "other" builtins are the builtin constants (None, True, False,
Ellipsis, NotImplemented) and various artifacts from doing this at the
interactive prompt (license, credits, copyright, quit, exit, help,
"_")

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu Oct 13 00:31:32 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 13 Oct 2016 14:31:32 +1000
Subject: [Python-Dev] Python ROCKS! Thanks guys! [anecdote]
In-Reply-To: <CAO41-mPEmPxqvrs2T2nRoSfVY86v3xZgCBsn8Ok0yJG0hadSPw@mail.gmail.com>
References: <CAPTjJmoA99s8cO62tqKvTTe1Mm9WJW8rFee+FUCVgzdci7sZdA@mail.gmail.com>
 <CAO41-mPEmPxqvrs2T2nRoSfVY86v3xZgCBsn8Ok0yJG0hadSPw@mail.gmail.com>
Message-ID: <CADiSq7eW7xUF_y-Y4i5k=2fdk5ZytGb_XhzyQVLdQ9DWoXgXRA@mail.gmail.com>

Thanks for passing this feedback along, Chris! It's always wonderful
to see developers feeling empowered by the potential that open source
tools offer them.

On 13 October 2016 at 11:01, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> Poor students... ;)

Folks, as tempting as it may be to make jokes at the expense of other
programming languages, please try to ensure that references to them on
the core Python lists are formulated on the basis of "What can we
learn from their experiences?", rather than as generic putdowns of
entire software development ecosystems. Even as a lighthearted joke
(as here), it isn't helpful to the design process to categorise
programming languages as being generically "better" or "worse" than
each other, rather than seeing them as embodiments of different ways
of thinking about algorithmic problem solving.

In combination with the W3C HTML5 and CSS standardisation work, the
JavaScript community have put together a superb set of tools for
creating user interfaces that are independent of the backend API
server implementation language, as well as useful tools for remote
data access and data transformation pipelines. The fact that all this
work is being done in the open and made freely available as open
source software means that the Python community is able to benefit
from these capabilities as much as anyone.

Regards,
Nick.

P.S. If anyone would like more background on why the "Our language is
universally better than your language" approach can be problematic
(even in jest!), please take a look at Aurynn Shaw's piece on Contempt
Culture in programming communities and the barriers that can create to
effective collaboration: http://blog.aurynn.com/contempt-culture

There's also my own
http://www.curiousefficiency.org/posts/2015/10/languages-to-improve-your-python.html
which looks at some other ways in which dismissing ecosystems out of
hand can inhibit our ability to learn from both their mistakes and
their successes.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From rosuav at gmail.com  Thu Oct 13 00:52:19 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 13 Oct 2016 15:52:19 +1100
Subject: [Python-Dev] Python ROCKS! Thanks guys! [anecdote]
In-Reply-To: <CAO41-mPEmPxqvrs2T2nRoSfVY86v3xZgCBsn8Ok0yJG0hadSPw@mail.gmail.com>
References: <CAPTjJmoA99s8cO62tqKvTTe1Mm9WJW8rFee+FUCVgzdci7sZdA@mail.gmail.com>
 <CAO41-mPEmPxqvrs2T2nRoSfVY86v3xZgCBsn8Ok0yJG0hadSPw@mail.gmail.com>
Message-ID: <CAPTjJmq1j+DXnX_gGqe=6suS4bGcsW5BGwpHwWme_76WOm7j1A@mail.gmail.com>

On Thu, Oct 13, 2016 at 12:01 PM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> On Wed, Oct 12, 2016 at 7:05 PM, Chris Angelico <rosuav at gmail.com> wrote:
>>
>> I work with a full-stack web development bootcamp. Most of the course
>> focuses on JavaScript (Node.js, React, jQuery, etc),
>
>
> Poor students... ;)

The bootcamp guarantees them employment upon graduation (subject to
some constraints, yada yada), so it teaches those skills that are most
likely to be employable. But one of those skills is learning new
technologies, hence the freedom to pick anything they like.

> I think the craziest thing is probably that, based on how you said it, these
> two students haven't even begun to enter the entire Python standard library,
> which you'd have to download a zillion npm modules (like the glorious
> left-pad) in order to match. Once they realize that, they'll never be going
> back!

Yes, this is true; but pip/PyPI is still an important part of web dev
in Python (eg Flask, SQLAlchemy, etc). My explanation to them is that
the dependency tree in Python is not as deep as the equivalent in Ruby
or JavaScript, but it's no less there. But Python definitely does
offer a far richer standard library. It's the little things,
sometimes:

var choice = messages[Math.floor(Math.random() * messages.length)];

choice = random.choice(messages)

The debate on whether it's worth sacrificing this kind of advantage in
order to use the same language on both client and server is unlikely
to be resolved any time soon. In the meantime, I'm very happy to be
able to introduce a few people to the joy of Pythonning.

ChrisA

From storchaka at gmail.com  Thu Oct 13 06:41:38 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 13 Oct 2016 13:41:38 +0300
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
 <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>
 <CAMpsgwZVazgFRy3f+SH6+AHmHCvD9s_q8SVCKhQnegdhVWUv-Q@mail.gmail.com>
 <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>
Message-ID: <ntnod1$7in$1@blaine.gmane.org>

On 13.10.16 00:14, Nathaniel Smith wrote:
> AFAIK basically the only project that would be affected by this is
> PyPy,

And MicroPython.



From songofacandy at gmail.com  Thu Oct 13 06:56:51 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 13 Oct 2016 19:56:51 +0900
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
 <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>
 <CAMpsgwZVazgFRy3f+SH6+AHmHCvD9s_q8SVCKhQnegdhVWUv-Q@mail.gmail.com>
 <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>
Message-ID: <CAEfz+Tw0ZTi6d1DCbx9Ro9ONF8xk6xeDJELhBBNGBMTdV6c3sA@mail.gmail.com>

> AFAIK basically the only project that would be affected by this is
> PyPy, and I when I asked on #pypy they said:
>
> <cfbolz> njs`: I think we either plan to or already support this
>
> so I'm not sure why this is controversial.

FYI, I had filed the feature request on PyPy issue tracker.
https://bitbucket.org/pypy/pypy/issues/2367/del-bytearray-n-should-be-optimized

In case of micropython, it doesn't support deletion for now.
https://github.com/micropython/micropython/blob/b9672bcbe8dd29b61326af8eb026df4d10a8f0ce/py/objarray.c#L371-L378

-- 
INADA Naoki  <songofacandy at gmail.com>

From larry at hastings.org  Thu Oct 13 07:41:14 2016
From: larry at hastings.org (Larry Hastings)
Date: Thu, 13 Oct 2016 12:41:14 +0100
Subject: [Python-Dev] Playing games with reference counts (was Re:
 PyWeakref_GetObject() borrows its reference from... whom?)
In-Reply-To: <CAPTjJmpMZ0fMbO4RhR=_AAHTMEYqvL-=K8LEJgf6+h-eKFJ2Yg@mail.gmail.com>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
 <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
 <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
 <660f23eb-0cb4-f1c7-af57-889275dc55b0@hastings.org>
 <CAPTjJmpMZ0fMbO4RhR=_AAHTMEYqvL-=K8LEJgf6+h-eKFJ2Yg@mail.gmail.com>
Message-ID: <06bcfd5e-00b5-71f9-97e7-0947c92699e2@hastings.org>


On 10/10/2016 10:38 PM, Chris Angelico wrote:
> On Tue, Oct 11, 2016 at 8:14 AM, Larry Hastings <larry at hastings.org> wrote:
>> These hacks where we play games with the
>> reference count are mostly removed in my branch.
> That's exactly what I would have said, because I was assuming that
> refcounts would be accurate. I'm not sure what you mean by "play games
> with",

By "playing games with reference counts", I mean code that purposely 
doesn't follow the rules of reference counting.  Sadly, there are 
special cases that apparently *are* special enough to break the rules.  
Which made implementing "buffered reference counting" that much harder.

I currently know of two examples of this in CPython.  In both instances, 
an object has a reference to another object, but *deliberately* does not 
increase the reference count of the object, in order to prevent keeping 
the other object alive.  The implementation relies on the GIL to 
preserve correctness; without a GIL, it was much harder to ensure this 
code was correct.  (And I'm still not 100% I've done it.  More thinking 
needed.)

Those two examples are:

 1. PyWeakReference objects.  The wr_object pointer--the "reference"
    held by the weak reference object--points to an object, but does not
    increment the reference count.  Worse yet, as already observed,
    PyWeakref_GetObject() and PyWeakref_GET_OBJECT() don't increment the
    reference count, an inconvenient API decision from my perspective.
 2. "Interned mortal" strings.  When a string is both interned *and*
    mortal, it's stored in the static "interned" dict in
    unicodeobject.c--as both key and value--and then its's DECREF'd
    twice so those two references don't count.  When the string is
    destroyed, unicode_dealloc resurrects the string, reinstating those
    two references, then removes it from the "interned" dict, then
    destroys the string as normal.

To support these, I've implemented what is effectively a secondary, 
atomic-only reference count.  It seems to work.  (And yes that means all 
objects are now 8 bytes bigger.  Let me worry about memory consumption 
later, m'kay?)


Resurrecting object also gave me a headache in the Gilectomy with this 
buffered reference counting scheme, but I think I have that figured out 
too.  When you resurrect an object, it's generally because you're going 
to expose it to other subsystems that may incr / decr / otherwise 
inspect the reference count.  Which means that code may buffer reference 
count changes.  Which means you can't immediately destroy the object 
anymore.  So: when you resurrect, you set the new reference count, you 
also set a flag saying "I've already been resurrected", you pass it in 
to that other code, you then drop your references with Py_DECREF, and 
you exit.  Your dealloc function will get called again later; you then 
see you've already done that first resurrection, and you destroy as 
normal.  Curiously enough, the typeobject actually needs to do this 
twice: once for tp_finalize, once for tp_del.  (Assuming I didn't 
completely misunderstand what the code was doing.)


My struggles continue,


//arry/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161013/282f9564/attachment.html>

From solipsis at pitrou.net  Thu Oct 13 08:02:22 2016
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 13 Oct 2016 14:02:22 +0200
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
 <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>
 <CAMpsgwZVazgFRy3f+SH6+AHmHCvD9s_q8SVCKhQnegdhVWUv-Q@mail.gmail.com>
 <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>
Message-ID: <20161013140222.1c016129@fsol>

On Wed, 12 Oct 2016 14:14:48 -0700
Nathaniel Smith <njs at pobox.com> wrote:
> 
> The proposal is that it should be documented as being part of the
> language spec starting in 3.4 (or whatever). So applications that
> support Python 2.7 can't rely on it, sure. But if I have an
> application that requires, say, 3.5+ but I don't want to depend on
> CPython-only implementation details, then I'm still allowed to use it.
> 
> AFAIK basically the only project that would be affected by this is
> PyPy, and I when I asked on #pypy they said:
> 
> <cfbolz> njs`: I think we either plan to or already support this
> 
> so I'm not sure why this is controversial.

The main reason it could be controversial is if someone finds out
another optimization that is made difficult or impossible by the O(1)
delete-at-front guarantee.  That sounds unlikely, though, since the
general structure of a bytearray is constrained by other factors as
well (such as the C API and the buffer API).

By the way, to answer another question, the reason this wasn't made
part of the spec originally is that the feature in itself was already
contentious (see issue tracker discussion).  Now that more people seem
to get interested in network programming, they seem to understand the
point ;-)

Regards

Antoine.



From tismer at stackless.com  Thu Oct 13 13:12:26 2016
From: tismer at stackless.com (Christian Tismer)
Date: Thu, 13 Oct 2016 19:12:26 +0200
Subject: [Python-Dev] Debugging Python scripts with GDB on OSX
In-Reply-To: <BBF024F4-DEA9-48D4-BE6E-298E2951AD4E@qt.io>
References: <BBF024F4-DEA9-48D4-BE6E-298E2951AD4E@qt.io>
Message-ID: <9013A651-56E5-4F73-A06F-77933A47696B@stackless.com>

Hi Alexandru,

I stumbled over this question a little late by chance. 

There is the possibility to use GDB, but it is most likely that you want to use python's pdb module, instead. 

Only in rare cases, when debugging the interpreter itself, you use gdb. For debugging Python code, use pdb or something better. 

Sent from my Ei4Steve

> On Jul 6, 2016, at 18:14, Alexandru Croitor <alexandru.croitor at qt.io> wrote:
> 
> Hello,
> 
> I'm interested to find out if debugging Python scripts with GDB is supported on OSX at all?
> 
> I'm referring to the functionality described on https://wiki.python.org/moin/DebuggingWithGdb and on http://fedoraproject.org/wiki/Features/EasierPythonDebugging.
> 
> I've tried so far various combinations of pre-compiled GDB from the homebrew package manager, locally-compiled GDB from homebrew, as well as locally compiled GDB from MacPorts, together with a pre-compiled Python 2.7, homebrew-compiled 2.7, and custom compiled Python 2.7 from the official source tarball.
> 
> My results so far were not successful. The legacy GDB commands to show a python stack trace or the local variables - do not work. And the new GDB commands (referenced on the Fedora project page) are not present at all in any of the GDB versions.
> 
> I've checked the python CI build bot tests, and it seems the new GDB commands are only successfully tested on Linux machines, and are skipped on FreeBSD, OS X, and Solaris machines.
> 
> Are the new python <-> GDB commands specific to Linux?
> Are there any considerations to take in regards to debug symbols for Python / GDB on OSX?
> 
> Has anyone attempted what I'm trying to do?
> 
> I would be grateful for any advice.
> 
> And I apologize if my choice of the mailing lists is not the best.
> 
> Regards, Alex.
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161013/4b8e5783/attachment.html>

From mariatta.wijaya at gmail.com  Thu Oct 13 14:09:11 2016
From: mariatta.wijaya at gmail.com (Mariatta Wijaya)
Date: Thu, 13 Oct 2016 11:09:11 -0700
Subject: [Python-Dev] Update to PEP 1 re: content type
Message-ID: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>

Hello,

PEP 1 states that plain/text is an acceptable value for a PEP's content
type, and it is the default value if no content type is specified.

May I propose adding something along the line of: "new PEPs should use
restructured Text format", and that reST format becomes the default.

Based on couple tickets in peps repo, it seems like the reST format is
preferred anyway, and there have been thoughts about converting text based
PEPs into reST format
https://github.com/python/peps/issues/4
https://github.com/python/peps/issues/2

I hope this is not too controversial.
Discussed this with Guido earlier. He is supportive, and asked me to send
email to python-dev :)

Thanks


Mariatta Wijaya
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161013/1494ddb6/attachment.html>

From greg at krypto.org  Thu Oct 13 14:33:38 2016
From: greg at krypto.org (Gregory P. Smith)
Date: Thu, 13 Oct 2016 18:33:38 +0000
Subject: [Python-Dev] Playing games with reference counts (was Re:
 PyWeakref_GetObject() borrows its reference from... whom?)
In-Reply-To: <06bcfd5e-00b5-71f9-97e7-0947c92699e2@hastings.org>
References: <d80e62b6-77ec-129a-d192-a58a6cfce744@hastings.org>
 <CABicbJJZiACX=eN7yi-cXWR6n2mb+59JV_gJ8qPjs4e3VO4YPA@mail.gmail.com>
 <60dfba0b-5219-e74f-887c-ebdfcd8edf69@hastings.org>
 <CAPTjJmoYLrQ+grx7eNnnxNZ=nyvXZKxYAaYx9e+qHBVV07BRsA@mail.gmail.com>
 <b8615395-5a7a-cfd6-9917-96648535e34a@mrabarnett.plus.com>
 <CACac1F-EXVgvm-AObsLMHj1hSWftBc5Q7F3U3wJPdXhiRmZm9g@mail.gmail.com>
 <CAPJVwB=GJBcqAy6OVS32LQLiDm+2bqEGA2x9sq4fe9yi2SXj9w@mail.gmail.com>
 <8b111714-f337-f7ab-ec0b-5940a45ef8a6@mrabarnett.plus.com>
 <1476123860.1791233.751518585.648D9391@webmail.messagingengine.com>
 <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
 <660f23eb-0cb4-f1c7-af57-889275dc55b0@hastings.org>
 <CAPTjJmpMZ0fMbO4RhR=_AAHTMEYqvL-=K8LEJgf6+h-eKFJ2Yg@mail.gmail.com>
 <06bcfd5e-00b5-71f9-97e7-0947c92699e2@hastings.org>
Message-ID: <CAGE7PNLdDKkWJPY09g=NWZDAFTbhTpF2bEO=GJG3=yk3mnkxng@mail.gmail.com>

On Thu, Oct 13, 2016 at 4:43 AM Larry Hastings <larry at hastings.org> wrote:

>
> On 10/10/2016 10:38 PM, Chris Angelico wrote:
>
> On Tue, Oct 11, 2016 at 8:14 AM, Larry Hastings <larry at hastings.org> <larry at hastings.org> wrote:
>
> These hacks where we play games with the
> reference count are mostly removed in my branch.
>
> That's exactly what I would have said, because I was assuming that
> refcounts would be accurate. I'm not sure what you mean by "play games
> with",
>
>
> By "playing games with reference counts", I mean code that purposely
> doesn't follow the rules of reference counting.  Sadly, there are special
> cases that apparently *are* special enough to break the rules.  Which made
> implementing "buffered reference counting" that much harder.
>
> I currently know of two examples of this in CPython.  In both instances,
> an object has a reference to another object, but *deliberately* does not
> increase the reference count of the object, in order to prevent keeping the
> other object alive.  The implementation relies on the GIL to preserve
> correctness; without a GIL, it was much harder to ensure this code was
> correct.  (And I'm still not 100% I've done it.  More thinking needed.)
>
> Those two examples are:
>
>    1. PyWeakReference objects.  The wr_object pointer--the "reference"
>    held by the weak reference object--points to an object, but does not
>    increment the reference count.  Worse yet, as already observed,
>    PyWeakref_GetObject() and PyWeakref_GET_OBJECT() don't increment the
>    reference count, an inconvenient API decision from my perspective.
>
> That a PyWeakReference object does not increment the reference count is
the entire point of a weakref. The object wouldn't be destroyed and break
the weak reference otherwise. Weak references could be implemented in a
different manner - coordinate with the garbage collector to consider things
who's only references come from weakrefs as collectable. That'd be an
internal overhaul of the weakref implementation and potentially the gc.

>
>    1.
>    2. "Interned mortal" strings.  When a string is both interned *and*
>    mortal, it's stored in the static "interned" dict in unicodeobject.c--as
>    both key and value--and then its's DECREF'd twice so those two references
>    don't count.  When the string is destroyed, unicode_dealloc resurrects the
>    string, reinstating those two references, then removes it from the
>    "interned" dict, then destroys the string as normal.
>
>
yow. i don't even want to know the history of that one...

Resurrecting object also gave me a headache in the Gilectomy with this
> buffered reference counting scheme, but I think I have that figured out
> too.  When you resurrect an object, it's generally because you're going to
> expose it to other subsystems that may incr / decr / otherwise inspect the
> reference count.  Which means that code may buffer reference count
> changes.  Which means you can't immediately destroy the object anymore.
> So: when you resurrect, you set the new reference count, you also set a
> flag saying "I've already been resurrected", you pass it in to that other
> code, you then drop your references with Py_DECREF, and you exit.  Your
> dealloc function will get called again later; you then see you've already
> done that first resurrection, and you destroy as normal.  Curiously enough,
> the typeobject actually needs to do this twice: once for tp_finalize, once
> for tp_del.  (Assuming I didn't completely misunderstand what the code was
> doing.)
>
> kudos for trying to understand this. resurrection during destruction or
finalization hurts my brain even though in many ways it makes sense.

-gps
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161013/cdcbf52a/attachment.html>

From rosuav at gmail.com  Thu Oct 13 14:43:13 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Fri, 14 Oct 2016 05:43:13 +1100
Subject: [Python-Dev] Update to PEP 1 re: content type
In-Reply-To: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
References: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
Message-ID: <CAPTjJmrct2Sc5eyhu1NH7uxb9oD16Yx8v90ByZPn5f1tYERVdw@mail.gmail.com>

On Fri, Oct 14, 2016 at 5:09 AM, Mariatta Wijaya
<mariatta.wijaya at gmail.com> wrote:
> PEP 1 states that plain/text is an acceptable value for a PEP's content
> type, and it is the default value if no content type is specified.
>
> May I propose adding something along the line of: "new PEPs should use
> restructured Text format", and that reST format becomes the default.

+1. It's already the de facto standard, and I support promoting that to de jure.

ChrisA

From guido at python.org  Thu Oct 13 16:32:57 2016
From: guido at python.org (Guido van Rossum)
Date: Thu, 13 Oct 2016 13:32:57 -0700
Subject: [Python-Dev] Update to PEP 1 re: content type
In-Reply-To: <CAPTjJmrct2Sc5eyhu1NH7uxb9oD16Yx8v90ByZPn5f1tYERVdw@mail.gmail.com>
References: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
 <CAPTjJmrct2Sc5eyhu1NH7uxb9oD16Yx8v90ByZPn5f1tYERVdw@mail.gmail.com>
Message-ID: <CAP7+vJ+UFObdFbMFp98FoUG1=SEq7ra+LDDWYg=4iwTMQjBhrw@mail.gmail.com>

Confirming, +1 from me. Barry, Nick? You two are the most active
authors of PEP 1.

On Thu, Oct 13, 2016 at 11:43 AM, Chris Angelico <rosuav at gmail.com> wrote:
> On Fri, Oct 14, 2016 at 5:09 AM, Mariatta Wijaya
> <mariatta.wijaya at gmail.com> wrote:
>> PEP 1 states that plain/text is an acceptable value for a PEP's content
>> type, and it is the default value if no content type is specified.
>>
>> May I propose adding something along the line of: "new PEPs should use
>> restructured Text format", and that reST format becomes the default.
>
> +1. It's already the de facto standard, and I support promoting that to de jure.
>
> ChrisA
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From barry at python.org  Thu Oct 13 16:38:35 2016
From: barry at python.org (Barry Warsaw)
Date: Thu, 13 Oct 2016 16:38:35 -0400
Subject: [Python-Dev] Update to PEP 1 re: content type
In-Reply-To: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
References: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
Message-ID: <20161013163835.56dfd28c@subdivisions.wooz.org>

On Oct 13, 2016, at 01:32 PM, Guido van Rossum wrote:

>Confirming, +1 from me. Barry, Nick? You two are the most active
>authors of PEP 1.

+1 for text/x-rst being the preferred.  We'd need some time to make it the
default, but I'm generally in favor of that happening.  Is anybody actively
working on issue #4?

https://github.com/python/peps/issues/4

Cheers,
-Barry

From jimjjewett at gmail.com  Thu Oct 13 17:27:19 2016
From: jimjjewett at gmail.com (Jim J. Jewett)
Date: Thu, 13 Oct 2016 14:27:19 -0700 (PDT)
Subject: [Python-Dev] PyWeakref_GetObject() borrows its reference
 from... whom?
In-Reply-To: <CAPTjJmoCPgNsCgZ0BOUkSfkVf6VnifvR=XwCcsk1PqtaNfB3CA@mail.gmail.com>
Message-ID: <57fffc37.43ede90a.27ab7.bb08@mx.google.com>



On Mon, Oct 10, 2016, at 14:04, MRAB wrote:
> Instead of locking the object, could we keep the GIL, but have it
> normally released?

> A thread could then still call a function such as PyWeakref_GetObject()
> that returns a borrowed reference, but only if it's holding the GIL. It
> would be able to INCREF the reference before releasing the GIL again.

So you need to get/release the GIL just to run a slightly faster function
that doesn't bother with an extra incref/defcref pair?  I think anyone
willing to make those changes would be willing to switch to a
non-borrowing version of that same function, and do an explicit
DECREF if that is really what they wanted.

On Tue, Oct 11, 2016 at 5:24 AM, Random832 <random832 at fastmail.com> wrote:
> So, what stops the other thread which never asks for the GIL from
> blowing away the reference? Or is this a special kind of lock that you
> can "assert isn't locked" without locking it for yourself, and
> INCREF/DECREF does so?

On Mon Oct 10 15:36:59 EDT 2016, Chris Angelico wrote:
> "assert isn't locked" is pretty cheap

Yeah, but so is INCREF/DECREF on memory that is almost certainly
in cache anyhow, because you're using the object right next to it.

The write part hurts, particularly when trying to use multiple cores
with shared memory, but any sort of indirection (even separating the
refcount from the object, to allow per-core counters) ... well, it
doesn't take much at all to be worse than INCREF/DECREF in even
the normal case, let alone amortized across the the "drat, this
object now has to be handled specially" cases.

Imagine two memory pools, one for "immortal" objects (such as
None) that won't be collected, and so don't need their memory
dirtied when you INCREF/DECREF.  Alas, now *every* INCREF and DECREF
has to branch on the address to tell whether or not it should be a
no-op.

-jJ

--

If there are still threading problems with my replies, please
email me with details, so that I can try to resolve them.  -jJ

From tjreedy at udel.edu  Thu Oct 13 17:31:44 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 13 Oct 2016 17:31:44 -0400
Subject: [Python-Dev] Update to PEP 1 re: content type
In-Reply-To: <20161013163835.56dfd28c@subdivisions.wooz.org>
References: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
 <20161013163835.56dfd28c@subdivisions.wooz.org>
Message-ID: <ntoufu$ns8$1@blaine.gmane.org>

On 10/13/2016 4:38 PM, Barry Warsaw wrote:
> On Oct 13, 2016, at 01:32 PM, Guido van Rossum wrote:
>
>> Confirming, +1 from me. Barry, Nick? You two are the most active
>> authors of PEP 1.
>
> +1 for text/x-rst being the preferred.  We'd need some time to make it the
> default, but I'm generally in favor of that happening.

The major issue is making knowledge of rst and sphinx a requirement for 
submitting a PEP.  I suggest making sure that PEP 1 exemplifies the 
preferred usage of rst in PEPs and then suggesting using it as  model.

A related issue is plain text for posting.  Initial pre-drafts can be 
posted to python-ideas before being marked up.  Other than copying the 
browser display of the .html conversion (which requires a bit of cleanup 
after pasting), what is the easiest way to get post-markup plain text?

-- 
Terry Jan Reedy



From barry at python.org  Thu Oct 13 17:59:42 2016
From: barry at python.org (Barry Warsaw)
Date: Thu, 13 Oct 2016 17:59:42 -0400
Subject: [Python-Dev] Update to PEP 1 re: content type
In-Reply-To: <ntoufu$ns8$1@blaine.gmane.org>
References: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
 <20161013163835.56dfd28c@subdivisions.wooz.org>
 <ntoufu$ns8$1@blaine.gmane.org>
Message-ID: <20161013175942.11f14f22@subdivisions.wooz.org>

On Oct 13, 2016, at 05:31 PM, Terry Reedy wrote:

>The major issue is making knowledge of rst and sphinx a requirement for
>submitting a PEP.  I suggest making sure that PEP 1 exemplifies the preferred
>usage of rst in PEPs and then suggesting using it as model.

https://github.com/python/peps/pull/115

>A related issue is plain text for posting.  Initial pre-drafts can be posted
>to python-ideas before being marked up.  Other than copying the browser
>display of the .html conversion (which requires a bit of cleanup after
>pasting), what is the easiest way to get post-markup plain text?

I don't understand the question: isn't the whole point of reST that it's a
easily readable markup language?  Just post the reST!

Cheers,
-Barry

From ericsnowcurrently at gmail.com  Thu Oct 13 18:25:19 2016
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 13 Oct 2016 16:25:19 -0600
Subject: [Python-Dev] Update to PEP 1 re: content type
In-Reply-To: <20161013175942.11f14f22@subdivisions.wooz.org>
References: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
 <20161013163835.56dfd28c@subdivisions.wooz.org>
 <ntoufu$ns8$1@blaine.gmane.org>
 <20161013175942.11f14f22@subdivisions.wooz.org>
Message-ID: <CALFfu7DJn8nAKoEVWy2uaDBx1jOE=TNUL=4DgdOL_h+grmMoWA@mail.gmail.com>

On Thu, Oct 13, 2016 at 3:59 PM, Barry Warsaw <barry at python.org> wrote:
> I don't understand the question: isn't the whole point of reST that it's a
> easily readable markup language?  Just post the reST!

+1

-eric

From ja.py at farowl.co.uk  Thu Oct 13 19:28:39 2016
From: ja.py at farowl.co.uk (Jeff Allen)
Date: Fri, 14 Oct 2016 00:28:39 +0100
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <ntnod1$7in$1@blaine.gmane.org>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
 <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>
 <CAMpsgwZVazgFRy3f+SH6+AHmHCvD9s_q8SVCKhQnegdhVWUv-Q@mail.gmail.com>
 <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>
 <ntnod1$7in$1@blaine.gmane.org>
Message-ID: <bd6d2a4e-1337-6901-5c95-aa9eba0430de@farowl.co.uk>

On 13/10/2016 11:41, Serhiy Storchaka wrote:
> On 13.10.16 00:14, Nathaniel Smith wrote:
>> AFAIK basically the only project that would be affected by this is
>> PyPy,
>
> And MicroPython.
>
And Jython, except that from the start its implementation of bytearray 
deferred resizing until the proportion unused space reaches some limit. 
I think that should make it O(log N) on average to delete (or add) a 
byte, at either end of a buffer of size N,. However, observations with 
timeit() look constant up to the point I run out of heap.

Jeff Allen

From tim.mitchell at leapfrog3d.com  Thu Oct 13 21:11:18 2016
From: tim.mitchell at leapfrog3d.com (Tim Mitchell)
Date: Fri, 14 Oct 2016 14:11:18 +1300
Subject: [Python-Dev] single dispatch for instance methods
Message-ID: <CAJjGKEbX=mW=Fb8MHKiW=36RAt2ROfu_SFwYNSWmLK+mLtF8nQ@mail.gmail.com>

Hi All,

It would be nice if the @singledispatch decorator worked on instance
methods.
I have a reference implementation on pypi that does this (
https://pypi.python.org/pypi/methoddispatch).

I posted this message to python ideas a month ago (
https://mail.python.org/pipermail/python-ideas/2016-September/042466.html)
and got 100% support from 1 response.

Not sure if I should proceed with writing a PEP.
I am happy to do all the work if I know it will be accepted.

Cheers
Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161014/49373813/attachment.html>

From vgr255 at live.ca  Thu Oct 13 21:21:12 2016
From: vgr255 at live.ca (Emanuel Barry)
Date: Fri, 14 Oct 2016 01:21:12 +0000
Subject: [Python-Dev] single dispatch for instance methods
In-Reply-To: <CAJjGKEbX=mW=Fb8MHKiW=36RAt2ROfu_SFwYNSWmLK+mLtF8nQ@mail.gmail.com>
References: <CAJjGKEbX=mW=Fb8MHKiW=36RAt2ROfu_SFwYNSWmLK+mLtF8nQ@mail.gmail.com>
Message-ID: <BN6PR13MB185811F04EB0E8EDE6C589DC91DF0@BN6PR13MB1858.namprd13.prod.outlook.com>

Just one reply seems pretty weak of a push. However, you lose nothing by submitting it on the issue tracker: https://bugs.python.org

I don?t have a use case for this myself, but we?ll see :)
-Emanuel

From: Python-Dev [mailto:python-dev-bounces+vgr255=live.ca at python.org] On Behalf Of Tim Mitchell
Sent: Thursday, October 13, 2016 9:11 PM
To: python-dev at python.org
Subject: [Python-Dev] single dispatch for instance methods

Hi All,

It would be nice if the @singledispatch decorator worked on instance methods.
I have a reference implementation on pypi that does this (https://pypi.python.org/pypi/methoddispatch).

I posted this message to python ideas a month ago (https://mail.python.org/pipermail/python-ideas/2016-September/042466.html) and got 100% support from 1 response.

Not sure if I should proceed with writing a PEP.
I am happy to do all the work if I know it will be accepted.

Cheers
Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161014/73ed5d43/attachment.html>

From tjreedy at udel.edu  Fri Oct 14 01:41:10 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 14 Oct 2016 01:41:10 -0400
Subject: [Python-Dev] Update to PEP 1 re: content type
In-Reply-To: <20161013175942.11f14f22@subdivisions.wooz.org>
References: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
 <20161013163835.56dfd28c@subdivisions.wooz.org>
 <ntoufu$ns8$1@blaine.gmane.org>
 <20161013175942.11f14f22@subdivisions.wooz.org>
Message-ID: <ntpr5k$9p4$1@blaine.gmane.org>

On 10/13/2016 5:59 PM, Barry Warsaw wrote:
> On Oct 13, 2016, at 05:31 PM, Terry Reedy wrote:
>
>> The major issue is making knowledge of rst and sphinx a requirement for
>> submitting a PEP.  I suggest making sure that PEP 1 exemplifies the preferred
>> usage of rst in PEPs and then suggesting using it as model.
>
> https://github.com/python/peps/pull/115
>
>> A related issue is plain text for posting.  Initial pre-drafts can be posted
>> to python-ideas before being marked up.  Other than copying the browser
>> display of the .html conversion (which requires a bit of cleanup after
>> pasting), what is the easiest way to get post-markup plain text?
>
> I don't understand the question: isn't the whole point of reST that it's a
> easily readable markup language?  Just post the reST!

OK with me if that becomes the norm.  I don't think it is now.

-- 
Terry Jan Reedy


From tismer at stackless.com  Fri Oct 14 05:56:24 2016
From: tismer at stackless.com (Christian Tismer)
Date: Fri, 14 Oct 2016 11:56:24 +0200
Subject: [Python-Dev] Debugging Python scripts with GDB on OSX
In-Reply-To: <30C9FE3D-40E0-4C01-952E-664DC3D9B52A@qt.io>
References: <BBF024F4-DEA9-48D4-BE6E-298E2951AD4E@qt.io>
 <9013A651-56E5-4F73-A06F-77933A47696B@stackless.com>
 <30C9FE3D-40E0-4C01-952E-664DC3D9B52A@qt.io>
Message-ID: <afd16a6f-4397-ef73-3dc4-68f6ceb0ee1a@stackless.com>

Hi Alexandru,

because I know that you are multi-platform, I can recommend the
debugger integration of PTVS very much.

I am currently using the WingWare debugger for all my projects,
but the Python/C++ integration of PTVS is something that I didn't
find anywhere else!

Please have a look at the following introduction.

https://www.youtube.com/watch?v=JNNAOypc6Ek

Starting with minute 22, you will find the kind of debugging that
you are looking for.

I was pretty amazed by this, and it is probably very helpful in
debugging Qt and PySide. Will give it a try, soon.

Cheers -- Chris


On 14/10/2016 11:12, Alexandru Croitor wrote:
> Hi,
> 
> pdb is fine for pure python scripts.
> 
> I was interested in things like getting the python back trace or local
> variables from inside GDB, when used in conjunction with c++, so that I
> know which parts of C++ calls python functions, and which parts of
> python call c++ functions. You can't do that with pdb.
> 
> 
>> On 13 Oct 2016, at 19:12, Christian Tismer <tismer at stackless.com
>> <mailto:tismer at stackless.com>> wrote:
>>
>> Hi Alexandru,
>>
>> I stumbled over this question a little late by chance. 
>>
>> There is the possibility to use GDB, but it is most likely that you
>> want to use python's pdb module, instead. 
>>
>> Only in rare cases, when debugging the interpreter itself, you use
>> gdb. For debugging Python code, use pdb or something better. 
>>
>> Sent from my Ei4Steve
>>
>> On Jul 6, 2016, at 18:14, Alexandru Croitor <alexandru.croitor at qt.io
>> <mailto:alexandru.croitor at qt.io>> wrote:
>>
>>> Hello,
>>>
>>> I'm interested to find out if debugging Python scripts with GDB is
>>> supported on OSX at all?
>>>
>>> I'm referring to the functionality described
>>> on https://wiki.python.org/moin/DebuggingWithGdb and
>>> on http://fedoraproject.org/wiki/Features/EasierPythonDebugging.
>>>
>>> I've tried so far various combinations of pre-compiled GDB from the
>>> homebrew package manager, locally-compiled GDB from homebrew, as well
>>> as locally compiled GDB from MacPorts, together with a pre-compiled
>>> Python 2.7, homebrew-compiled 2.7, and custom compiled Python 2.7
>>> from the official source tarball.
>>>
>>> My results so far were not successful. The legacy GDB commands to
>>> show a python stack trace or the local variables - do not work. And
>>> the new GDB commands (referenced on the Fedora project page) are not
>>> present at all in any of the GDB versions.
>>>
>>> I've checked the python CI build bot tests, and it seems the new GDB
>>> commands are only successfully tested on Linux machines, and are
>>> skipped on FreeBSD, OS X, and Solaris machines.
>>>
>>> Are the new python <-> GDB commands specific to Linux?
>>> Are there any considerations to take in regards to debug symbols for
>>> Python / GDB on OSX?
>>>
>>> Has anyone attempted what I'm trying to do?
>>>
>>> I would be grateful for any advice.
>>>
>>> And I apologize if my choice of the mailing lists is not the best.
>>>
>>> Regards, Alex.
>>>
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org <mailto:Python-Dev at python.org>
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe:
>>> https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com
> 


-- 
Christian Tismer             :^)   tismer at stackless.com
Software Consulting          :     http://www.stackless.com/
Karl-Liebknecht-Str. 121     :     https://github.com/PySide
14482 Potsdam                :     GPG key -> 0xFB7BEE0E
phone +49 173 24 18 776  fax +49 (30) 700143-0023

From ncoghlan at gmail.com  Fri Oct 14 06:43:10 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 14 Oct 2016 20:43:10 +1000
Subject: [Python-Dev] Update to PEP 1 re: content type
In-Reply-To: <CAP7+vJ+UFObdFbMFp98FoUG1=SEq7ra+LDDWYg=4iwTMQjBhrw@mail.gmail.com>
References: <CAGbohnb2KXVUHOOEMKwf1ax9X6uKzbZp2A7AfcDvotv=EXhZPw@mail.gmail.com>
 <CAPTjJmrct2Sc5eyhu1NH7uxb9oD16Yx8v90ByZPn5f1tYERVdw@mail.gmail.com>
 <CAP7+vJ+UFObdFbMFp98FoUG1=SEq7ra+LDDWYg=4iwTMQjBhrw@mail.gmail.com>
Message-ID: <CADiSq7eCJ7SpZ6J=K2G0RcFkrKjrHUc9OSfFTTj-AjyotpTxpQ@mail.gmail.com>

On 14 October 2016 at 06:32, Guido van Rossum <guido at python.org> wrote:
> Confirming, +1 from me. Barry, Nick? You two are the most active
> authors of PEP 1.

I've merged the PR that Barry wrote to address Mariatta's query, so I
guess that counts as +1's from both of us :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From alexandru.croitor at qt.io  Fri Oct 14 05:12:10 2016
From: alexandru.croitor at qt.io (Alexandru Croitor)
Date: Fri, 14 Oct 2016 09:12:10 +0000
Subject: [Python-Dev] Debugging Python scripts with GDB on OSX
In-Reply-To: <9013A651-56E5-4F73-A06F-77933A47696B@stackless.com>
References: <BBF024F4-DEA9-48D4-BE6E-298E2951AD4E@qt.io>
 <9013A651-56E5-4F73-A06F-77933A47696B@stackless.com>
Message-ID: <30C9FE3D-40E0-4C01-952E-664DC3D9B52A@qt.io>

Hi,

pdb is fine for pure python scripts.

I was interested in things like getting the python back trace or local variables from inside GDB, when used in conjunction with c++, so that I know which parts of C++ calls python functions, and which parts of python call c++ functions. You can't do that with pdb.


On 13 Oct 2016, at 19:12, Christian Tismer <tismer at stackless.com<mailto:tismer at stackless.com>> wrote:

Hi Alexandru,

I stumbled over this question a little late by chance.

There is the possibility to use GDB, but it is most likely that you want to use python's pdb module, instead.

Only in rare cases, when debugging the interpreter itself, you use gdb. For debugging Python code, use pdb or something better.

Sent from my Ei4Steve

On Jul 6, 2016, at 18:14, Alexandru Croitor <alexandru.croitor at qt.io<mailto:alexandru.croitor at qt.io>> wrote:

Hello,

I'm interested to find out if debugging Python scripts with GDB is supported on OSX at all?

I'm referring to the functionality described on https://wiki.python.org/moin/DebuggingWithGdb and on http://fedoraproject.org/wiki/Features/EasierPythonDebugging.

I've tried so far various combinations of pre-compiled GDB from the homebrew package manager, locally-compiled GDB from homebrew, as well as locally compiled GDB from MacPorts, together with a pre-compiled Python 2.7, homebrew-compiled 2.7, and custom compiled Python 2.7 from the official source tarball.

My results so far were not successful. The legacy GDB commands to show a python stack trace or the local variables - do not work. And the new GDB commands (referenced on the Fedora project page) are not present at all in any of the GDB versions.

I've checked the python CI build bot tests, and it seems the new GDB commands are only successfully tested on Linux machines, and are skipped on FreeBSD, OS X, and Solaris machines.

Are the new python <-> GDB commands specific to Linux?
Are there any considerations to take in regards to debug symbols for Python / GDB on OSX?

Has anyone attempted what I'm trying to do?

I would be grateful for any advice.

And I apologize if my choice of the mailing lists is not the best.

Regards, Alex.


_______________________________________________
Python-Dev mailing list
Python-Dev at python.org<mailto:Python-Dev at python.org>
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161014/11d86626/attachment-0001.html>

From status at bugs.python.org  Fri Oct 14 12:08:53 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 14 Oct 2016 18:08:53 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20161014160853.CD65C568F9@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-10-07 - 2016-10-14)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5527 (+10)
  closed 34673 (+49)
  total  40200 (+59)

Open issues with patches: 2397 


Issues opened (37)
==================

#28386: Improve documentation about tzinfo.dst(None)
http://bugs.python.org/issue28386  opened by Daniel Moisset

#28387: double free in io.TextIOWrapper
http://bugs.python.org/issue28387  opened by scufre

#28393: Update encoding lookup docs wrt #27938
http://bugs.python.org/issue28393  opened by scop

#28395: Remove unnecessary semicolons
http://bugs.python.org/issue28395  opened by scop

#28396: Remove *.pyo references from man page
http://bugs.python.org/issue28396  opened by scop

#28398: Return singleton empty string in _PyUnicode_FromASCII
http://bugs.python.org/issue28398  opened by xiang.zhang

#28401: Don't support the PEP384 stable ABI in pydebug builds
http://bugs.python.org/issue28401  opened by stefanor

#28403: Porting guide: disabling & warning on implicit unicode convers
http://bugs.python.org/issue28403  opened by ncoghlan

#28407: Improve coverage of email.utils.make_msgid()
http://bugs.python.org/issue28407  opened by dillon.brock

#28408: Fix redundant code and memory leak in _PyUnicodeWriter_Finish
http://bugs.python.org/issue28408  opened by xiang.zhang

#28409: test.regrtest does not support multiple -x flags
http://bugs.python.org/issue28409  opened by cstratak

#28410: Add convenient C API for "raise ... from ..."
http://bugs.python.org/issue28410  opened by serhiy.storchaka

#28411: Eliminate PyInterpreterState.modules.
http://bugs.python.org/issue28411  opened by eric.snow

#28412: os.path.splitdrive documentation out of date
http://bugs.python.org/issue28412  opened by snooter

#28414: SSL match_hostname fails for internationalized domain names
http://bugs.python.org/issue28414  opened by abracadaber

#28415: PyUnicode_FromFromat interger format handling different from p
http://bugs.python.org/issue28415  opened by xiang.zhang

#28416: defining persistent_id in _pickle.Pickler subclass causes refe
http://bugs.python.org/issue28416  opened by cwitty

#28418: Raise Deprecation warning for tokenize.generate_tokens
http://bugs.python.org/issue28418  opened by mbussonn

#28422: multiprocessing Manager mutable type member access failure
http://bugs.python.org/issue28422  opened by vilnis.termanis

#28424: pkgutil.get_data() doesn't work with namespace packages
http://bugs.python.org/issue28424  opened by dgreiman

#28425: Python3 ignores __init__.py that are links to /dev/null
http://bugs.python.org/issue28425  opened by dgreiman

#28426: PyUnicode_AsDecodedObject can only return unicode now
http://bugs.python.org/issue28426  opened by xiang.zhang

#28427: WeakValueDictionary next bug (with multithreading)
http://bugs.python.org/issue28427  opened by arigo

#28428: Rename _futures module to _asyncio
http://bugs.python.org/issue28428  opened by inada.naoki

#28429: ctypes fails to import with grsecurity's TPE
http://bugs.python.org/issue28429  opened by Glandos

#28430: asyncio: C implemeted Future cause Tornado test fail
http://bugs.python.org/issue28430  opened by inada.naoki

#28431: socket gethostbyaddr returns IPv6 names for 127.0.0.1
http://bugs.python.org/issue28431  opened by carbonin

#28432: Fix doc of PyUnicode_EncodeLocale
http://bugs.python.org/issue28432  opened by xiang.zhang

#28435: test_urllib2_localnet.ProxyAuthTests fails with no_proxy and N
http://bugs.python.org/issue28435  opened by Piotr Szczepaniak

#28436: GzipFile doesn't properly handle short reads and writes on the
http://bugs.python.org/issue28436  opened by abacabadabacaba

#28437: Class definition is not consistent with types.new_class
http://bugs.python.org/issue28437  opened by neil.g

#28439: Remove redundant checks in PyUnicode_EncodeLocale
http://bugs.python.org/issue28439  opened by xiang.zhang

#28440: pip failures on macOS Sierra
http://bugs.python.org/issue28440  opened by Marc.Culler

#28441: sys.executable is ambiguous on Cygwin without .exe suffix
http://bugs.python.org/issue28441  opened by erik.bray

#28442: tuple(a list subclass) does not iterate through the list
http://bugs.python.org/issue28442  opened by siming85

#28443: Logger methods never use kwargs
http://bugs.python.org/issue28443  opened by jb098

#28444: Missing extensions modules when cross compiling python 3.5.2 f
http://bugs.python.org/issue28444  opened by bennykj



Most recent 15 issues with no replies (15)
==========================================

#28444: Missing extensions modules when cross compiling python 3.5.2 f
http://bugs.python.org/issue28444

#28443: Logger methods never use kwargs
http://bugs.python.org/issue28443

#28442: tuple(a list subclass) does not iterate through the list
http://bugs.python.org/issue28442

#28441: sys.executable is ambiguous on Cygwin without .exe suffix
http://bugs.python.org/issue28441

#28440: pip failures on macOS Sierra
http://bugs.python.org/issue28440

#28439: Remove redundant checks in PyUnicode_EncodeLocale
http://bugs.python.org/issue28439

#28435: test_urllib2_localnet.ProxyAuthTests fails with no_proxy and N
http://bugs.python.org/issue28435

#28432: Fix doc of PyUnicode_EncodeLocale
http://bugs.python.org/issue28432

#28430: asyncio: C implemeted Future cause Tornado test fail
http://bugs.python.org/issue28430

#28429: ctypes fails to import with grsecurity's TPE
http://bugs.python.org/issue28429

#28422: multiprocessing Manager mutable type member access failure
http://bugs.python.org/issue28422

#28416: defining persistent_id in _pickle.Pickler subclass causes refe
http://bugs.python.org/issue28416

#28412: os.path.splitdrive documentation out of date
http://bugs.python.org/issue28412

#28408: Fix redundant code and memory leak in _PyUnicodeWriter_Finish
http://bugs.python.org/issue28408

#28407: Improve coverage of email.utils.make_msgid()
http://bugs.python.org/issue28407



Most recent 15 issues waiting for review (15)
=============================================

#28443: Logger methods never use kwargs
http://bugs.python.org/issue28443

#28441: sys.executable is ambiguous on Cygwin without .exe suffix
http://bugs.python.org/issue28441

#28440: pip failures on macOS Sierra
http://bugs.python.org/issue28440

#28439: Remove redundant checks in PyUnicode_EncodeLocale
http://bugs.python.org/issue28439

#28435: test_urllib2_localnet.ProxyAuthTests fails with no_proxy and N
http://bugs.python.org/issue28435

#28432: Fix doc of PyUnicode_EncodeLocale
http://bugs.python.org/issue28432

#28428: Rename _futures module to _asyncio
http://bugs.python.org/issue28428

#28427: WeakValueDictionary next bug (with multithreading)
http://bugs.python.org/issue28427

#28426: PyUnicode_AsDecodedObject can only return unicode now
http://bugs.python.org/issue28426

#28411: Eliminate PyInterpreterState.modules.
http://bugs.python.org/issue28411

#28410: Add convenient C API for "raise ... from ..."
http://bugs.python.org/issue28410

#28409: test.regrtest does not support multiple -x flags
http://bugs.python.org/issue28409

#28408: Fix redundant code and memory leak in _PyUnicodeWriter_Finish
http://bugs.python.org/issue28408

#28407: Improve coverage of email.utils.make_msgid()
http://bugs.python.org/issue28407

#28401: Don't support the PEP384 stable ABI in pydebug builds
http://bugs.python.org/issue28401



Top 10 most discussed issues (10)
=================================

#21720: "TypeError: Item in ``from list'' not a string" message
http://bugs.python.org/issue21720  12 msgs

#28092: Build failure for 3.6 on Centos 5.11
http://bugs.python.org/issue28092  11 msgs

#28409: test.regrtest does not support multiple -x flags
http://bugs.python.org/issue28409  11 msgs

#28339: "TypeError: Parameterized generics cannot be used with class o
http://bugs.python.org/issue28339  10 msgs

#28428: Rename _futures module to _asyncio
http://bugs.python.org/issue28428  10 msgs

#28333: input() with Unicode prompt produces mojibake on Windows
http://bugs.python.org/issue28333   9 msgs

#22635: subprocess.getstatusoutput changed behavior in 3.4 (maybe 3.3.
http://bugs.python.org/issue22635   7 msgs

#27998: Bytes paths now are supported in os.scandir() on Windows
http://bugs.python.org/issue27998   7 msgs

#28214: Improve exception reporting for problematic __set_name__ attri
http://bugs.python.org/issue28214   7 msgs

#28403: Porting guide: disabling & warning on implicit unicode convers
http://bugs.python.org/issue28403   6 msgs



Issues closed (46)
==================

#13646: Document poor interaction between multiprocessing and -m on Wi
http://bugs.python.org/issue13646  closed by ncoghlan

#15332: 2to3 should fix bad indentation (or warn about it)
http://bugs.python.org/issue15332  closed by r.david.murray

#18287: PyType_Ready() should sanity-check the tp_name field
http://bugs.python.org/issue18287  closed by serhiy.storchaka

#18789: XML Vunerability Table Unclear
http://bugs.python.org/issue18789  closed by gvanrossum

#20766: reference leaks in pdb
http://bugs.python.org/issue20766  closed by xdegaye

#21443: asyncio logging documentation clarifications
http://bugs.python.org/issue21443  closed by gvanrossum

#22502: <Ctl-C> after continue in pdb stops in signal.py
http://bugs.python.org/issue22502  closed by xdegaye

#24098: Multiple use after frees in obj2ast_* methods
http://bugs.python.org/issue24098  closed by serhiy.storchaka

#24452: Make webbrowser support Chrome on Mac OS X
http://bugs.python.org/issue24452  closed by gvanrossum

#25783: test_traceback.test_walk_stack() fails when run directly (with
http://bugs.python.org/issue25783  closed by serhiy.storchaka

#26081: Implement asyncio Future in C to improve performance
http://bugs.python.org/issue26081  closed by inada.naoki

#26293: Embedded zipfile fields dependent on absolute position
http://bugs.python.org/issue26293  closed by serhiy.storchaka

#26869: unittest longMessage docs
http://bugs.python.org/issue26869  closed by gvanrossum

#27170: IDLE: remove Toggle Auto Coloring or add to edit menu & doc
http://bugs.python.org/issue27170  closed by terry.reedy

#27588: Type (typing) objects are hashable and comparable for equality
http://bugs.python.org/issue27588  closed by gvanrossum

#27972: Confusing error during cyclic yield
http://bugs.python.org/issue27972  closed by yselivanov

#28100: Refactor error messages in symtable.c
http://bugs.python.org/issue28100  closed by levkivskyi

#28183: Clean up and speed up dict iteration
http://bugs.python.org/issue28183  closed by serhiy.storchaka

#28257: Regression for star argument parameter error messages
http://bugs.python.org/issue28257  closed by serhiy.storchaka

#28317: Improve support of FORMAT_VALUE in dis
http://bugs.python.org/issue28317  closed by serhiy.storchaka

#28376: rangeiter_new fails when creating a range of step 0
http://bugs.python.org/issue28376  closed by serhiy.storchaka

#28377: struct.unpack of Bool
http://bugs.python.org/issue28377  closed by berker.peksag

#28378: urllib2 does not handle cookies with `,`
http://bugs.python.org/issue28378  closed by SilentGhost

#28379: PyUnicode_CopyCharacters could lead to undefined behaviour
http://bugs.python.org/issue28379  closed by serhiy.storchaka

#28388: Update documentation for typing module
http://bugs.python.org/issue28388  closed by gvanrossum

#28389: xmlrpc.client HTTP proxy example code does not work
http://bugs.python.org/issue28389  closed by berker.peksag

#28390: Wrong heading levels in whatsnew/3.6
http://bugs.python.org/issue28390  closed by berker.peksag

#28391: Multiple occurances of: Closing quotes separate words
http://bugs.python.org/issue28391  closed by r.david.murray

#28392: shlex - non-posix-mode: error in Multiple occurances of quotes
http://bugs.python.org/issue28392  closed by r.david.murray

#28394: Spelling fixes
http://bugs.python.org/issue28394  closed by martin.panter

#28397: Faster index range checks
http://bugs.python.org/issue28397  closed by serhiy.storchaka

#28399: Remove UNIX socket from FS before binding
http://bugs.python.org/issue28399  closed by yselivanov

#28400: Remove uncessary checks in unicode_char and resize_copy
http://bugs.python.org/issue28400  closed by serhiy.storchaka

#28402: Add signed catalog files for stdlib on Windows
http://bugs.python.org/issue28402  closed by steve.dower

#28404: Logging SyslogHandler not appending '\n' to the end
http://bugs.python.org/issue28404  closed by vinay.sajip

#28405: Compile error on Modules/_futuresmodule.c
http://bugs.python.org/issue28405  closed by berker.peksag

#28406: Possible PyUnicode_InternInPlace() edge-case bug
http://bugs.python.org/issue28406  closed by larry

#28413: unprefixed global symbol freegrammar
http://bugs.python.org/issue28413  closed by python-dev

#28417: va_end twice in PyUnicode_FromFormatV
http://bugs.python.org/issue28417  closed by python-dev

#28419: List comprehension in class scope does not have access to clas
http://bugs.python.org/issue28419  closed by r.david.murray

#28420: is ok
http://bugs.python.org/issue28420  closed by xiang.zhang

#28421: lib/distutils/sysconfig.py fails to pick up Py_ENABLE_SHARED v
http://bugs.python.org/issue28421  closed by tmatsuzawa

#28423: list.insert(-1,value) is wrong!
http://bugs.python.org/issue28423  closed by tim.peters

#28433: Add sorted (ordered) containers
http://bugs.python.org/issue28433  closed by r.david.murray

#28434: U+1F441 EYE Missing in unicodedata
http://bugs.python.org/issue28434  closed by ned.deily

#28438: Wrong link in pkgutil.get_data doc
http://bugs.python.org/issue28438  closed by orsenthil

From wes.turner at gmail.com  Fri Oct 14 12:23:55 2016
From: wes.turner at gmail.com (Wes Turner)
Date: Fri, 14 Oct 2016 11:23:55 -0500
Subject: [Python-Dev] Debugging Python scripts with GDB on OSX
In-Reply-To: <30C9FE3D-40E0-4C01-952E-664DC3D9B52A@qt.io>
References: <BBF024F4-DEA9-48D4-BE6E-298E2951AD4E@qt.io>
 <9013A651-56E5-4F73-A06F-77933A47696B@stackless.com>
 <30C9FE3D-40E0-4C01-952E-664DC3D9B52A@qt.io>
Message-ID: <CACfEFw_gRmK8+EU90+oCo3bityaDty+ryioxHL5=PpumQQ7arA@mail.gmail.com>

On Friday, October 14, 2016, Alexandru Croitor <alexandru.croitor at qt.io>
wrote:

> Hi,
>
> pdb is fine for pure python scripts.
>
> I was interested in things like getting the python back trace or local
> variables from inside GDB, when used in conjunction with c++, so that I
> know which parts of C++ calls python functions, and which parts of python
> call c++ functions. You can't do that with pdb.
>

>From https://westurner.org/wiki/awesome-python-testing#advanced-debugging :

- https://hg.python.org/cpython/file/tip/Tools/gdb/libpython.py

- https://github.com/lmacken/pyrasite/blob/develop/README.rst mentions
something about codesigned GDB and OSX.
 - https://sourceware.org/gdb/wiki/BuildingOnDarwin

- https://github.com/google/pyringe doesnt mention OSX.

- dtrace probably doesn't help with stack traces or local variables:
  -
https://github.com/benesch/python-dtrace-tracker/blob/master/dtrace-osx-apple.27.patch

Are there error messages?



>
> On 13 Oct 2016, at 19:12, Christian Tismer <tismer at stackless.com
> <javascript:_e(%7B%7D,'cvml','tismer at stackless.com');>> wrote:
>
> Hi Alexandru,
>
> I stumbled over this question a little late by chance.
>
> There is the possibility to use GDB, but it is most likely that you want
> to use python's pdb module, instead.
>
> Only in rare cases, when debugging the interpreter itself, you use gdb.
> For debugging Python code, use pdb or something better.
>
> Sent from my Ei4Steve
>
> On Jul 6, 2016, at 18:14, Alexandru Croitor <alexandru.croitor at qt.io
> <javascript:_e(%7B%7D,'cvml','alexandru.croitor at qt.io');>> wrote:
>
> Hello,
>
> I'm interested to find out if debugging Python scripts with GDB is
> supported on OSX at all?
>
> I'm referring to the functionality described on https://wiki.python.org/
> moin/DebuggingWithGdb and on http://fedoraproject.org/wiki/Features/
> EasierPythonDebugging.
>
> I've tried so far various combinations of pre-compiled GDB from the
> homebrew package manager, locally-compiled GDB from homebrew, as well as
> locally compiled GDB from MacPorts, together with a pre-compiled Python
> 2.7, homebrew-compiled 2.7, and custom compiled Python 2.7 from the
> official source tarball.
>
> My results so far were not successful. The legacy GDB commands to show a
> python stack trace or the local variables - do not work. And the new GDB
> commands (referenced on the Fedora project page) are not present at all in
> any of the GDB versions.
>
> I've checked the python CI build bot tests, and it seems the new GDB
> commands are only successfully tested on Linux machines, and are skipped on
> FreeBSD, OS X, and Solaris machines.
>
> Are the new python <-> GDB commands specific to Linux?
> Are there any considerations to take in regards to debug symbols for
> Python / GDB on OSX?
>
> Has anyone attempted what I'm trying to do?
>
> I would be grateful for any advice.
>
> And I apologize if my choice of the mailing lists is not the best.
>
> Regards, Alex.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> <javascript:_e(%7B%7D,'cvml','Python-Dev at python.org');>
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> tismer%40stackless.com
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161014/e2897f75/attachment.html>

From alexandru.croitor at qt.io  Fri Oct 14 06:34:57 2016
From: alexandru.croitor at qt.io (Alexandru Croitor)
Date: Fri, 14 Oct 2016 10:34:57 +0000
Subject: [Python-Dev] Debugging Python scripts with GDB on OSX
In-Reply-To: <afd16a6f-4397-ef73-3dc4-68f6ceb0ee1a@stackless.com>
References: <BBF024F4-DEA9-48D4-BE6E-298E2951AD4E@qt.io>
 <9013A651-56E5-4F73-A06F-77933A47696B@stackless.com>
 <30C9FE3D-40E0-4C01-952E-664DC3D9B52A@qt.io>
 <afd16a6f-4397-ef73-3dc4-68f6ceb0ee1a@stackless.com>
Message-ID: <71B1B5E4-5019-4BBB-A15A-D28E74A2A395@qt.io>

Hi,

I'm aware of PTVS, but I was hoping of getting a crude version of that for macOS, which you could say that GDB provides.

Unfortunately the GDB functionality only works on Linux, which is why I sent my message in the first place, hoping that I might be missing something.

Regards,

Alex.


> On 14 Oct 2016, at 11:56, Christian Tismer <tismer at stackless.com> wrote:
> 
> Hi Alexandru,
> 
> because I know that you are multi-platform, I can recommend the
> debugger integration of PTVS very much.
> 
> I am currently using the WingWare debugger for all my projects,
> but the Python/C++ integration of PTVS is something that I didn't
> find anywhere else!
> 
> Please have a look at the following introduction.
> 
> https://www.youtube.com/watch?v=JNNAOypc6Ek
> 
> Starting with minute 22, you will find the kind of debugging that
> you are looking for.
> 
> I was pretty amazed by this, and it is probably very helpful in
> debugging Qt and PySide. Will give it a try, soon.
> 
> Cheers -- Chris
> 
> 
> On 14/10/2016 11:12, Alexandru Croitor wrote:
>> Hi,
>> 
>> pdb is fine for pure python scripts.
>> 
>> I was interested in things like getting the python back trace or local
>> variables from inside GDB, when used in conjunction with c++, so that I
>> know which parts of C++ calls python functions, and which parts of
>> python call c++ functions. You can't do that with pdb.
>> 
>> 
>>> On 13 Oct 2016, at 19:12, Christian Tismer <tismer at stackless.com
>>> <mailto:tismer at stackless.com>> wrote:
>>> 
>>> Hi Alexandru,
>>> 
>>> I stumbled over this question a little late by chance. 
>>> 
>>> There is the possibility to use GDB, but it is most likely that you
>>> want to use python's pdb module, instead. 
>>> 
>>> Only in rare cases, when debugging the interpreter itself, you use
>>> gdb. For debugging Python code, use pdb or something better. 
>>> 
>>> Sent from my Ei4Steve
>>> 
>>> On Jul 6, 2016, at 18:14, Alexandru Croitor <alexandru.croitor at qt.io
>>> <mailto:alexandru.croitor at qt.io>> wrote:
>>> 
>>>> Hello,
>>>> 
>>>> I'm interested to find out if debugging Python scripts with GDB is
>>>> supported on OSX at all?
>>>> 
>>>> I'm referring to the functionality described
>>>> on https://wiki.python.org/moin/DebuggingWithGdb and
>>>> on http://fedoraproject.org/wiki/Features/EasierPythonDebugging.
>>>> 
>>>> I've tried so far various combinations of pre-compiled GDB from the
>>>> homebrew package manager, locally-compiled GDB from homebrew, as well
>>>> as locally compiled GDB from MacPorts, together with a pre-compiled
>>>> Python 2.7, homebrew-compiled 2.7, and custom compiled Python 2.7
>>>> from the official source tarball.
>>>> 
>>>> My results so far were not successful. The legacy GDB commands to
>>>> show a python stack trace or the local variables - do not work. And
>>>> the new GDB commands (referenced on the Fedora project page) are not
>>>> present at all in any of the GDB versions.
>>>> 
>>>> I've checked the python CI build bot tests, and it seems the new GDB
>>>> commands are only successfully tested on Linux machines, and are
>>>> skipped on FreeBSD, OS X, and Solaris machines.
>>>> 
>>>> Are the new python <-> GDB commands specific to Linux?
>>>> Are there any considerations to take in regards to debug symbols for
>>>> Python / GDB on OSX?
>>>> 
>>>> Has anyone attempted what I'm trying to do?
>>>> 
>>>> I would be grateful for any advice.
>>>> 
>>>> And I apologize if my choice of the mailing lists is not the best.
>>>> 
>>>> Regards, Alex.
>>>> 
>>>> 
>>>> _______________________________________________
>>>> Python-Dev mailing list
>>>> Python-Dev at python.org <mailto:Python-Dev at python.org>
>>>> https://mail.python.org/mailman/listinfo/python-dev
>>>> Unsubscribe:
>>>> https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com
>> 
> 
> 
> -- 
> Christian Tismer             :^)   tismer at stackless.com
> Software Consulting          :     http://www.stackless.com/
> Karl-Liebknecht-Str. 121     :     https://github.com/PySide
> 14482 Potsdam                :     GPG key -> 0xFB7BEE0E
> phone +49 173 24 18 776  fax +49 (30) 700143-0023


From aric.coady at gmail.com  Sun Oct 16 18:53:04 2016
From: aric.coady at gmail.com (Aric Coady)
Date: Sun, 16 Oct 2016 15:53:04 -0700
Subject: [Python-Dev] single dispatch for instance methods
In-Reply-To: <mailman.3488.1476460164.30832.python-dev@python.org>
References: <mailman.3488.1476460164.30832.python-dev@python.org>
Message-ID: <42464E2A-D0C8-45C0-8281-0254D3FE081F@gmail.com>

On Oct 14, 2016, at 8:49 AM, python-dev-request at python.org wrote:
> Date: Fri, 14 Oct 2016 14:11:18 +1300
> From: Tim Mitchell <tim.mitchell at leapfrog3d.com>
> It would be nice if the @singledispatch decorator worked on instance
> methods.
> I have a reference implementation on pypi that does this (
> https://pypi.python.org/pypi/methoddispatch).

You might be interested in this thread from my multimethod library: https://bitbucket.org/coady/multimethod/issues/2/ <https://bitbucket.org/coady/multimethod/issues/2/>.  It was a feature request for multiple dispatch to work on instance methods.  I implemented a variation where the initial self argument still contributed to the dispatch.

I think the same reasoning would apply in the single dispatch case:  that it?s not obvious the ?dispatch? would occur once on the self argument to find the method, and then again on the ?first? argument.  Especially since Py3 did away with bound methods, so you can call a method through its class and explicitly supply self as the first argument.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161016/068006cc/attachment.html>

From tim.mitchell at leapfrog3d.com  Sun Oct 16 20:55:17 2016
From: tim.mitchell at leapfrog3d.com (Tim Mitchell)
Date: Mon, 17 Oct 2016 13:55:17 +1300
Subject: [Python-Dev] single dispatch for instance methods
In-Reply-To: <42464E2A-D0C8-45C0-8281-0254D3FE081F@gmail.com>
References: <mailman.3488.1476460164.30832.python-dev@python.org>
 <42464E2A-D0C8-45C0-8281-0254D3FE081F@gmail.com>
Message-ID: <CAJjGKEZPirphoLYgtVOjr_=yp3+1x0brE0MTs=v8rMzvpDhF+A@mail.gmail.com>

An interesting idea, however from my point of view in methoddispatch there
is no dispatching on the self argument at all.
Overriding methods simply allows your subclass to have a different dispatch
table for a method than the super class.  This happens when the class is
created, not when the function is called.
When the method is called it is single-dispatched on the (single) argument
after self using the dispatch table for the class of the method.



On 17 October 2016 at 11:53, Aric Coady <aric.coady at gmail.com> wrote:

> On Oct 14, 2016, at 8:49 AM, python-dev-request at python.org wrote:
>
> Date: Fri, 14 Oct 2016 14:11:18 +1300
> From: Tim Mitchell <tim.mitchell at leapfrog3d.com>
>
> It would be nice if the @singledispatch decorator worked on instance
> methods.
> I have a reference implementation on pypi that does this (
> https://pypi.python.org/pypi/methoddispatch).
>
>
> You might be interested in this thread from my multimethod library:
> https://bitbucket.org/coady/multimethod/issues/2/.  It was a feature
> request for multiple dispatch to work on instance methods.  I implemented a
> variation where the initial self argument still contributed to the dispatch.
>
> I think the same reasoning would apply in the single dispatch case:  that
> it?s not obvious the ?dispatch? would occur once on the self argument to
> find the method, and then again on the ?first? argument.  Especially since
> Py3 did away with bound methods, so you can call a method through its class
> and explicitly supply self as the first argument.
>
>
>
------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161017/bd6c776a/attachment.html>

From songofacandy at gmail.com  Mon Oct 17 02:18:38 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Mon, 17 Oct 2016 15:18:38 +0900
Subject: [Python-Dev] [Python-ideas] Show more info when `python -vV`
In-Reply-To: <CAPTjJmoi9YpZ=J5TBpt0=Zg4ANVxfHQ8JWP9rFntkD+pDTWtmQ@mail.gmail.com>
References: <CAEfz+TxOSReGuYXrPMpFKPwxF6gd5+-Q9j-OP114QETtO5hV1Q@mail.gmail.com>
 <ntsm57$p2d$1@blaine.gmane.org>
 <CAEfz+TzN=0xqf8zd1Avxxebcrh4rEJ8J9A+EVb2BV9VvzywbsQ@mail.gmail.com>
 <CADiSq7e0X7+774Zisq-gjtuvEGdYOJX8NR1pRx7gcTu_ZCikow@mail.gmail.com>
 <CAPTjJmqc61PNv_M5v1D8-C5diGTHHuBzqkfTf6uO8XcRVKmgvg@mail.gmail.com>
 <CADiSq7dQqVy-bfdwXPVg-Jt9TfRjnPCZTnbKmGFKYaXQSiJ0_g@mail.gmail.com>
 <CAPTjJmp3kanamX=6Q1jYugHb3F6C6ujmCjWfs4TEUOB4yRJ1xQ@mail.gmail.com>
 <CAEfz+TzSh+o2-Q1sH0kKVG=w0L8uEOzkwM8MzL11aBNw2R+w3Q@mail.gmail.com>
 <CAPTjJmoi9YpZ=J5TBpt0=Zg4ANVxfHQ8JWP9rFntkD+pDTWtmQ@mail.gmail.com>
Message-ID: <CAEfz+TwQrmY9f5P32AXpBy4c5D2fdQf8JuKJpBbVXC6gYyBAgw@mail.gmail.com>

(Added python-dev in CC list, because there are enough +1 already).

On Mon, Oct 17, 2016 at 3:06 PM, Chris Angelico <rosuav at gmail.com> wrote:
> On Mon, Oct 17, 2016 at 5:02 PM, INADA Naoki <songofacandy at gmail.com> wrote:
>> $ ./python.exe -V
>> Python 3.6.0b2+
>>
>> $ ./python.exe -VV
>> Python 3.6.0b2+ (3.6:0b29adb5c804+, Oct 17 2016, 15:00:12)
>> [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.38)]
>
> LGTM.
>
> What's the view on backporting this to 2.7.x? We're still a good few
> years away from its death, and it'd be helpful if recent 2.7s could
> give this info too.
>
> ChrisA


I want to add it at least Python 3.6. Because one reason I want to
propose this is
I can't see exact Python version (commit id) for "nightly" or
"3.6-dev" on Travis-CI test.

But Python 3.6 is beta stage already. If we apply rule strictly, it
should be added
only in default branch (Python 3.7).

So, what version can I add this?

a. Only Python 3.7+
b. (beta) Python 3.6+
c. (maintenance) Python 2.7 and Python 3.5+

-- 
INADA Naoki  <songofacandy at gmail.com>

From ncoghlan at gmail.com  Mon Oct 17 03:01:57 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 17 Oct 2016 17:01:57 +1000
Subject: [Python-Dev] [Python-ideas] Show more info when `python -vV`
In-Reply-To: <CAEfz+TwQrmY9f5P32AXpBy4c5D2fdQf8JuKJpBbVXC6gYyBAgw@mail.gmail.com>
References: <CAEfz+TxOSReGuYXrPMpFKPwxF6gd5+-Q9j-OP114QETtO5hV1Q@mail.gmail.com>
 <ntsm57$p2d$1@blaine.gmane.org>
 <CAEfz+TzN=0xqf8zd1Avxxebcrh4rEJ8J9A+EVb2BV9VvzywbsQ@mail.gmail.com>
 <CADiSq7e0X7+774Zisq-gjtuvEGdYOJX8NR1pRx7gcTu_ZCikow@mail.gmail.com>
 <CAPTjJmqc61PNv_M5v1D8-C5diGTHHuBzqkfTf6uO8XcRVKmgvg@mail.gmail.com>
 <CADiSq7dQqVy-bfdwXPVg-Jt9TfRjnPCZTnbKmGFKYaXQSiJ0_g@mail.gmail.com>
 <CAPTjJmp3kanamX=6Q1jYugHb3F6C6ujmCjWfs4TEUOB4yRJ1xQ@mail.gmail.com>
 <CAEfz+TzSh+o2-Q1sH0kKVG=w0L8uEOzkwM8MzL11aBNw2R+w3Q@mail.gmail.com>
 <CAPTjJmoi9YpZ=J5TBpt0=Zg4ANVxfHQ8JWP9rFntkD+pDTWtmQ@mail.gmail.com>
 <CAEfz+TwQrmY9f5P32AXpBy4c5D2fdQf8JuKJpBbVXC6gYyBAgw@mail.gmail.com>
Message-ID: <CADiSq7eMXsCKv3HpGvq-NmCR=5b6yQdZtwYiKVaPAw+SFE1_kw@mail.gmail.com>

On 17 October 2016 at 16:18, INADA Naoki <songofacandy at gmail.com> wrote:
> (Added python-dev in CC list, because there are enough +1 already).

(switching entirely to python-dev)

Context for python-dev folks:

* Inada-san would like to enable "python -VV" to print the full REPL
header for improved debugging information
* we don't want to mess with "-V" because external scripts may be
relying on it (e.g. to figure out filenames)
* "python -VV" is already permitted, but just means the same thing as
"python -V"

I think this is a good idea, but this is a CPython-the-implementation
feature, rather than a Python-the-language feature, so it isn't clear
what rules apply to adding it.

As a pragmatic matter, I think it would be worth applying it to all
active maintenance branches, as:

- that dramatically reduces the deployment term for it to start being
useful for troubleshooting
- the SSL/TLS changes mean that some redistributed versions of Python
2.7 are starting to accumulate significant backports that aren't
indicated in the nominal upstream version number, but are conveyed in
the full REPL header (which has redistributor build info in it)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From jeremy.kloth at gmail.com  Mon Oct 17 05:43:00 2016
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Mon, 17 Oct 2016 03:43:00 -0600
Subject: [Python-Dev] [Python-checkins] cpython (3.5): remove extra
 PyErr_Format arguments (closes #28454)
In-Reply-To: <20161016224253.33077.69037.67D48F55@psf.io>
References: <20161016224253.33077.69037.67D48F55@psf.io>
Message-ID: <CAGvrs3+71SA1UD9e9Bmgw4L_NVZRAbnsmLuXyfiDRfy7D=8QRg@mail.gmail.com>

On Sun, Oct 16, 2016 at 4:42 PM, benjamin.peterson
<python-checkins at python.org> wrote:
> --- a/Objects/unicodeobject.c
> +++ b/Objects/unicodeobject.c
> @@ -3009,7 +3009,7 @@
>                       "'%.400s' decoder returned '%.400s' instead of 'str'; "
>                       "use codecs.decode() to decode to arbitrary types",
>                       encoding,
> -                     Py_TYPE(unicode)->tp_name, Py_TYPE(unicode)->tp_name);
> +                     Py_TYPE(unicode)->tp_name);
>          Py_DECREF(unicode);
>          goto onError;
>      }

Um, isn't that now an error?  There are 2 format sequences, but only 1 argument.

-- 
Jeremy Kloth

From berker.peksag at gmail.com  Mon Oct 17 06:27:08 2016
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Mon, 17 Oct 2016 13:27:08 +0300
Subject: [Python-Dev] [Python-checkins] cpython (3.5): remove extra
 PyErr_Format arguments (closes #28454)
In-Reply-To: <CAGvrs3+71SA1UD9e9Bmgw4L_NVZRAbnsmLuXyfiDRfy7D=8QRg@mail.gmail.com>
References: <20161016224253.33077.69037.67D48F55@psf.io>
 <CAGvrs3+71SA1UD9e9Bmgw4L_NVZRAbnsmLuXyfiDRfy7D=8QRg@mail.gmail.com>
Message-ID: <CAF4280+i9OJ9YvsqFqj+kM0wHzhVyBnjMTcpv9PdZP85vH3cNA@mail.gmail.com>

On Mon, Oct 17, 2016 at 12:43 PM, Jeremy Kloth <jeremy.kloth at gmail.com> wrote:
> On Sun, Oct 16, 2016 at 4:42 PM, benjamin.peterson
> <python-checkins at python.org> wrote:
>> --- a/Objects/unicodeobject.c
>> +++ b/Objects/unicodeobject.c
>> @@ -3009,7 +3009,7 @@
>>                       "'%.400s' decoder returned '%.400s' instead of 'str'; "
>>                       "use codecs.decode() to decode to arbitrary types",
>>                       encoding,
>> -                     Py_TYPE(unicode)->tp_name, Py_TYPE(unicode)->tp_name);
>> +                     Py_TYPE(unicode)->tp_name);
>>          Py_DECREF(unicode);
>>          goto onError;
>>      }
>
> Um, isn't that now an error?  There are 2 format sequences, but only 1 argument.

Hi Jeremy,

It doesn't look like an error to me. The first argument is 'encoding'
and the second one is 'Py_TYPE(unicode)->tp_name'.

--Berker

From jeremy.kloth at gmail.com  Mon Oct 17 07:05:18 2016
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Mon, 17 Oct 2016 05:05:18 -0600
Subject: [Python-Dev] [Python-checkins] cpython (3.5): remove extra
 PyErr_Format arguments (closes #28454)
In-Reply-To: <CAF4280+i9OJ9YvsqFqj+kM0wHzhVyBnjMTcpv9PdZP85vH3cNA@mail.gmail.com>
References: <20161016224253.33077.69037.67D48F55@psf.io>
 <CAGvrs3+71SA1UD9e9Bmgw4L_NVZRAbnsmLuXyfiDRfy7D=8QRg@mail.gmail.com>
 <CAF4280+i9OJ9YvsqFqj+kM0wHzhVyBnjMTcpv9PdZP85vH3cNA@mail.gmail.com>
Message-ID: <CAGvrs3Ks_nQ+qh91KwPHxjFJrUwWzv_Qr5mOcpM=DtCbQCjOjQ@mail.gmail.com>

Oops, you are right. Silly little comma...

-- 
Jeremy Kloth

From barry at python.org  Mon Oct 17 15:05:45 2016
From: barry at python.org (Barry Warsaw)
Date: Mon, 17 Oct 2016 15:05:45 -0400
Subject: [Python-Dev] [Python-ideas] Show more info when `python -vV`
In-Reply-To: <CADiSq7eMXsCKv3HpGvq-NmCR=5b6yQdZtwYiKVaPAw+SFE1_kw@mail.gmail.com>
References: <CAEfz+TxOSReGuYXrPMpFKPwxF6gd5+-Q9j-OP114QETtO5hV1Q@mail.gmail.com>
 <ntsm57$p2d$1@blaine.gmane.org>
 <CAEfz+TzN=0xqf8zd1Avxxebcrh4rEJ8J9A+EVb2BV9VvzywbsQ@mail.gmail.com>
 <CADiSq7e0X7+774Zisq-gjtuvEGdYOJX8NR1pRx7gcTu_ZCikow@mail.gmail.com>
 <CAPTjJmqc61PNv_M5v1D8-C5diGTHHuBzqkfTf6uO8XcRVKmgvg@mail.gmail.com>
 <CADiSq7dQqVy-bfdwXPVg-Jt9TfRjnPCZTnbKmGFKYaXQSiJ0_g@mail.gmail.com>
 <CAPTjJmp3kanamX=6Q1jYugHb3F6C6ujmCjWfs4TEUOB4yRJ1xQ@mail.gmail.com>
 <CAEfz+TzSh+o2-Q1sH0kKVG=w0L8uEOzkwM8MzL11aBNw2R+w3Q@mail.gmail.com>
 <CAPTjJmoi9YpZ=J5TBpt0=Zg4ANVxfHQ8JWP9rFntkD+pDTWtmQ@mail.gmail.com>
 <CAEfz+TwQrmY9f5P32AXpBy4c5D2fdQf8JuKJpBbVXC6gYyBAgw@mail.gmail.com>
 <CADiSq7eMXsCKv3HpGvq-NmCR=5b6yQdZtwYiKVaPAw+SFE1_kw@mail.gmail.com>
Message-ID: <20161017150545.052006b5@subdivisions.wooz.org>

On Oct 17, 2016, at 05:01 PM, Nick Coghlan wrote:

>* Inada-san would like to enable "python -VV" to print the full REPL
>header for improved debugging information
>* we don't want to mess with "-V" because external scripts may be
>relying on it (e.g. to figure out filenames)
>* "python -VV" is already permitted, but just means the same thing as
>"python -V"
>
>I think this is a good idea

So do I.

>, but this is a CPython-the-implementation feature, rather than a
>Python-the-language feature, so it isn't clear what rules apply to adding it.

Certainly add it to 3.7.  I think the decision whether to add it to 3.6 and
the stable releases should be up to the respective release managers.  As you
say, it's not a language feature, but it's still a new feature and as such it
does fall within a gray area.

I'd personally have no problem with adding it to 3.6 and possibly earlier
stable releases, but I'm not an RM of any active releases any more.  I'd say,
open a bug, post the patch against the versions you want to see it applied to,
go through the review process, and let the RMs decide if its worth the risk.

Cheers,
-Barry

From larry at hastings.org  Mon Oct 17 15:15:55 2016
From: larry at hastings.org (Larry Hastings)
Date: Mon, 17 Oct 2016 20:15:55 +0100
Subject: [Python-Dev] [Python-ideas] Show more info when `python -vV`
In-Reply-To: <20161017150545.052006b5@subdivisions.wooz.org>
References: <CAEfz+TxOSReGuYXrPMpFKPwxF6gd5+-Q9j-OP114QETtO5hV1Q@mail.gmail.com>
 <ntsm57$p2d$1@blaine.gmane.org>
 <CAEfz+TzN=0xqf8zd1Avxxebcrh4rEJ8J9A+EVb2BV9VvzywbsQ@mail.gmail.com>
 <CADiSq7e0X7+774Zisq-gjtuvEGdYOJX8NR1pRx7gcTu_ZCikow@mail.gmail.com>
 <CAPTjJmqc61PNv_M5v1D8-C5diGTHHuBzqkfTf6uO8XcRVKmgvg@mail.gmail.com>
 <CADiSq7dQqVy-bfdwXPVg-Jt9TfRjnPCZTnbKmGFKYaXQSiJ0_g@mail.gmail.com>
 <CAPTjJmp3kanamX=6Q1jYugHb3F6C6ujmCjWfs4TEUOB4yRJ1xQ@mail.gmail.com>
 <CAEfz+TzSh+o2-Q1sH0kKVG=w0L8uEOzkwM8MzL11aBNw2R+w3Q@mail.gmail.com>
 <CAPTjJmoi9YpZ=J5TBpt0=Zg4ANVxfHQ8JWP9rFntkD+pDTWtmQ@mail.gmail.com>
 <CAEfz+TwQrmY9f5P32AXpBy4c5D2fdQf8JuKJpBbVXC6gYyBAgw@mail.gmail.com>
 <CADiSq7eMXsCKv3HpGvq-NmCR=5b6yQdZtwYiKVaPAw+SFE1_kw@mail.gmail.com>
 <20161017150545.052006b5@subdivisions.wooz.org>
Message-ID: <e732fb94-da2f-9635-a0ea-36ee257bcb77@hastings.org>


It is a New Feature and therefore ineligible for adding to 3.5 or 3.4.


//arry/


On 10/17/2016 08:05 PM, Barry Warsaw wrote:
> On Oct 17, 2016, at 05:01 PM, Nick Coghlan wrote:
>
>> * Inada-san would like to enable "python -VV" to print the full REPL
>> header for improved debugging information
>> * we don't want to mess with "-V" because external scripts may be
>> relying on it (e.g. to figure out filenames)
>> * "python -VV" is already permitted, but just means the same thing as
>> "python -V"
>>
>> I think this is a good idea
> So do I.
>
>> , but this is a CPython-the-implementation feature, rather than a
>> Python-the-language feature, so it isn't clear what rules apply to adding it.
> Certainly add it to 3.7.  I think the decision whether to add it to 3.6 and
> the stable releases should be up to the respective release managers.  As you
> say, it's not a language feature, but it's still a new feature and as such it
> does fall within a gray area.
>
> I'd personally have no problem with adding it to 3.6 and possibly earlier
> stable releases, but I'm not an RM of any active releases any more.  I'd say,
> open a bug, post the patch against the versions you want to see it applied to,
> go through the review process, and let the RMs decide if its worth the risk.
>
> Cheers,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/larry%40hastings.org

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161017/e33a62d5/attachment.html>

From nad at python.org  Mon Oct 17 15:48:21 2016
From: nad at python.org (Ned Deily)
Date: Mon, 17 Oct 2016 15:48:21 -0400
Subject: [Python-Dev] [Python-ideas] Show more info when `python -vV`
In-Reply-To: <20161017150545.052006b5@subdivisions.wooz.org>
References: <CAEfz+TxOSReGuYXrPMpFKPwxF6gd5+-Q9j-OP114QETtO5hV1Q@mail.gmail.com>
 <ntsm57$p2d$1@blaine.gmane.org>
 <CAEfz+TzN=0xqf8zd1Avxxebcrh4rEJ8J9A+EVb2BV9VvzywbsQ@mail.gmail.com>
 <CADiSq7e0X7+774Zisq-gjtuvEGdYOJX8NR1pRx7gcTu_ZCikow@mail.gmail.com>
 <CAPTjJmqc61PNv_M5v1D8-C5diGTHHuBzqkfTf6uO8XcRVKmgvg@mail.gmail.com>
 <CADiSq7dQqVy-bfdwXPVg-Jt9TfRjnPCZTnbKmGFKYaXQSiJ0_g@mail.gmail.com>
 <CAPTjJmp3kanamX=6Q1jYugHb3F6C6ujmCjWfs4TEUOB4yRJ1xQ@mail.gmail.com>
 <CAEfz+TzSh+o2-Q1sH0kKVG=w0L8uEOzkwM8MzL11aBNw2R+w3Q@mail.gmail.com>
 <CAPTjJmoi9YpZ=J5TBpt0=Zg4ANVxfHQ8JWP9rFntkD+pDTWtmQ@mail.gmail.com>
 <CAEfz+TwQrmY9f5P32AXpBy4c5D2fdQf8JuKJpBbVXC6gYyBAgw@mail.gmail.com>
 <CADiSq7eMXsCKv3HpGvq-NmCR=5b6yQdZtwYiKVaPAw+SFE1_kw@mail.gmail.com>
 <20161017150545.052006b5@subdivisions.wooz.org>
Message-ID: <nu39u2$9ri$1@blaine.gmane.org>

On 2016-10-17 15:05, Barry Warsaw wrote:
> On Oct 17, 2016, at 05:01 PM, Nick Coghlan wrote: 
>> * Inada-san would like to enable "python -VV" to print the full REPL
>> header for improved debugging information
[...]
> I'd personally have no problem with adding it to 3.6 and possibly earlier
> stable releases, but I'm not an RM of any active releases any more.  I'd say,
> open a bug, post the patch against the versions you want to see it applied to,
> go through the review process, and let the RMs decide if its worth the risk.

I agree with Barry.  I'm open to adding this feature to 3.6.0b3 but
first we need an issue and a final patch to review.




From chris.barker at noaa.gov  Tue Oct 18 11:28:43 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Tue, 18 Oct 2016 08:28:43 -0700
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
 <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>
Message-ID: <-4662978122166531554@unknownmsgid>

>>
>>
>> Method proliferation on builtins is a Big Deal(TM)
>
> I wanted to quantify this concept, so here's a quick metric that helps
> convey how every time we add a new builtin method we're immediately
> making Python harder to comprehend:
>
>>>> def get_builtin_types():
>    ...     import builtins
>    ...     return {name:obj for name, obj in vars(builtins).items()
> if isinstance(obj, type) and not (name.startswith("__") or
> issubclass(obj, BaseException))}
>    ...
>>>> len(get_builtin_types())
>    26

Sure -- adding a new builtin is s big deal.

>>>> def get_builtin_methods():
>    ...     return [(name, method_name) for name, obj in
> get_builtin_types().items() for method_name, method in
> vars(obj).items() if not method_name.startswith("__")]
>    ...
>>>> len(get_builtin_methods())
>    230

So what? No one looks in all the methods of builtins at once. If we
have anything like an OO System (and python builtins only sort of
do...) then folks look for a built in that they need, and only then
look at its methods.

If you need to work with bytes, you'll look at the bytes object and
bytarray object. Having to go find some helper function module to know
to efficiently do something with bytes is VERY non-discoverable!

bytes and bytarray are already low-level objects -- adding low-level
functionality to them makes perfect sense.

And no, this is not just for asycio at all -- it's potentially useful
for any byte manipulation.

+1 on a frombuffer() method.

> Putting special purpose functionality behind an import gate helps to
> provide a more explicit context of use

This is a fine argument for putting bytearray in a separate module --
but that ship has sailed. The method to construct a bytearray from a
buffer belongs with the bytearray object.

-CHB

From chris.barker at noaa.gov  Tue Oct 18 11:57:37 2016
From: chris.barker at noaa.gov (Chris Barker - NOAA Federal)
Date: Tue, 18 Oct 2016 08:57:37 -0700
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
 <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>
 <CAMpsgwZVazgFRy3f+SH6+AHmHCvD9s_q8SVCKhQnegdhVWUv-Q@mail.gmail.com>
 <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>
Message-ID: <5085489373920251619@unknownmsgid>

> The proposal is that it should be documented as being part of the
> language spec starting in 3.4 (or whatever).

Is the performance characteristics of any object part of the language spec?

I.e if someone wrote an implementation with an O(n) insert dict, it
would suck, but wouldn't it still be Python?

-CHB



> So applications that
> support Python 2.7 can't rely on it, sure. But if I have an
> application that requires, say, 3.5+ but I don't want to depend on
> CPython-only implementation details, then I'm still allowed to use it.
>
> AFAIK basically the only project that would be affected by this is
> PyPy, and I when I asked on #pypy they said:
>
> <cfbolz> njs`: I think we either plan to or already support this
>
> so I'm not sure why this is controversial.
>
> -n
>
> --
> Nathaniel J. Smith -- https://vorpus.org
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov

From rosuav at gmail.com  Tue Oct 18 13:03:50 2016
From: rosuav at gmail.com (Chris Angelico)
Date: Wed, 19 Oct 2016 04:03:50 +1100
Subject: [Python-Dev] O(1) deletes from the front of bytearray (was: Re:
 Adding bytes.frombuffer() constructor to PEP 467 (was: [Python-ideas]
 Adding bytes.frombuffer() constructor)
In-Reply-To: <5085489373920251619@unknownmsgid>
References: <CAPJVwBno+=zVNBefVmROafBD781K0ROw5RKykdPEzS0a1=3ULg@mail.gmail.com>
 <ntko1m$tgp$1@blaine.gmane.org>
 <CAPJVwBkXHkWYPc+dUa07bznhAiBjANgF3ydLudnuf39_FPZBUw@mail.gmail.com>
 <CAMpsgwZVazgFRy3f+SH6+AHmHCvD9s_q8SVCKhQnegdhVWUv-Q@mail.gmail.com>
 <CAPJVwBnJG3CgLFmgc9-dcvOy-U3U_N_JcZ0e96ypTBbecE5QLw@mail.gmail.com>
 <5085489373920251619@unknownmsgid>
Message-ID: <CAPTjJmpDK_XjVUj-RpBRN+QqkdJR2pkuCouxoe2ccjZLkuoMaw@mail.gmail.com>

On Wed, Oct 19, 2016 at 2:57 AM, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
>> The proposal is that it should be documented as being part of the
>> language spec starting in 3.4 (or whatever).
>
> Is the performance characteristics of any object part of the language spec?
>
> I.e if someone wrote an implementation with an O(n) insert dict, it
> would suck, but wouldn't it still be Python?

This exact question came up when MicroPython wanted to represent
Unicode strings internally as UTF-8. It was decided that having O(n)
indexing/slicing was acceptable, and was something that the
implementation could judge the value of. (Since uPy is designed for
smaller systems than CPython is, O(n) is less significant.)

ChrisA

From rymg19 at gmail.com  Tue Oct 18 14:08:19 2016
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Tue, 18 Oct 2016 13:08:19 -0500
Subject: [Python-Dev] typing.py doesn't export Pattern and Match;
 is that intentional
Message-ID: <CAO41-mPD9HWb6rSZXWzLbdV_tshoF6jo2n=2OGQjXo6HeEhf8g@mail.gmail.com>

E.g.:


import typing
print(typing.Pattern, typing.Match)  # Works.
from typing import *
print(Pattern, Match)  # NameError: name 'Pattern' is not defined


A quick look shows that typing.py doesn't have Pattern and Match in
__all__. Was this intentional, or just an oversight?


-- 
Ryan
[ERROR]: Your autotools build scripts are 200 lines longer than your
program. Something?s wrong.
http://kirbyfan64.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161018/79b9ab2b/attachment.html>

From guido at python.org  Tue Oct 18 15:20:57 2016
From: guido at python.org (Guido van Rossum)
Date: Tue, 18 Oct 2016 12:20:57 -0700
Subject: [Python-Dev] typing.py doesn't export Pattern and Match;
 is that intentional
In-Reply-To: <CAO41-mPD9HWb6rSZXWzLbdV_tshoF6jo2n=2OGQjXo6HeEhf8g@mail.gmail.com>
References: <CAO41-mPD9HWb6rSZXWzLbdV_tshoF6jo2n=2OGQjXo6HeEhf8g@mail.gmail.com>
Message-ID: <CAP7+vJ+TmOdzgR3wQ0YBELNVEgjB6JWodzcT614cg8NwSBRd8g@mail.gmail.com>

Yes, that's intentional. (We have a separate tracker for this module,
https://github.com/python/typing/issues)

On Tue, Oct 18, 2016 at 11:08 AM, Ryan Gonzalez <rymg19 at gmail.com> wrote:
> E.g.:
>
>
> import typing
> print(typing.Pattern, typing.Match)  # Works.
> from typing import *
> print(Pattern, Match)  # NameError: name 'Pattern' is not defined
>
>
> A quick look shows that typing.py doesn't have Pattern and Match in __all__.
> Was this intentional, or just an oversight?
>
>
> --
> Ryan
> [ERROR]: Your autotools build scripts are 200 lines longer than your
> program. Something?s wrong.
> http://kirbyfan64.github.io/
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)

From victor.stinner at gmail.com  Thu Oct 20 06:56:06 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 20 Oct 2016 12:56:06 +0200
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
Message-ID: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>

Hi,

Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed
results in depth (up to the hardware and kernel drivers!), I wrote new
tools and enhanced existing tools.

* I wrote a new perf module which runs benchmarks in a reliable way
and contains a LOT of features: collect metadata, JSON file format,
commands to compare, render an histogram, etc.

* I rewrote the Python benchmark suite: the old benchmarks Mercurial
repository moved to a new performance GitHub project which uses my
perf module and contains more benchmarks.

* I also made minor enhancements to timeit in Python 3.7 -- some dev
don't want major changes to not "break the backward compatibility".

For timeit, I suggest to use my perf tool which includes a reliable
timeit command and has much more features like --duplicate (repeat the
statements to reduce the cost of the outer loop) and --compare-to
(compare two versions of Python), but also all builtin perf features
(JSON output, statistics, histogram, etc.).

I added benchmarks from PyPy and Pyston benchmark suites to
performance: performance 0.3.1 contains 51 benchmark scripts which run
a total of 121 benchmarks. Example of tested Python modules:

* SQLAlchemy
* Dulwich (full Git implementation in Python)
* Mercurial (currently only the startup time)
* html5lib
* pyaes (AES crypto cipher in pure Python)
* sympy
* Tornado (HTTP client and server)
* Django (sadly, only the template engine right now, Pyston contains
HTTP benchmarks)
* pathlib
* spambayes

More benchmarks will be added later. It would be nice to add
benchmarks on numpy for example, numpy is important for a large part
of our community.

All these (new or updated) tools can now be used to take smarter
decisions on optimizations. Please don't push any optimization anymore
without providing reliable benchmark results!


My first major action was to close the latest attempt to
micro-optimize int+int in Python/ceval.c,
http://bugs.python.org/issue21955 : I closed the issue as rejected,
because there is no significant speedup on benchmarks other than two
(tiny) microbenchmarks. To make sure that no one looses its time on
trying to micro-optimize int+int, I even added a comment to
Python/ceval.c :-)

   https://hg.python.org/cpython/rev/61fcb12a9873
   "Please don't try to micro-optimize int+int"


The perf and performance are now well tested: Travis CI runs tests on
the new commits and pull requests, and the "tox" command can be used
locally to test different Python versions, pep8, doc, ... in a single
command.


Next steps:

* Run performance 0.3.1 on speed.python.org: the benchmark runner is
currently stopped (and still uses the old benchmarks project). The
website part may be updated to allow to download full JSON files which
includes *all* information (all timings, metadata and more).

* I plan to run performance on CPython 2.7, CPython 3.7, PyPy and PyPy
3. Maybe also CPython 3.5 and CPython 3.6 if they don't take too much
resources.

* Later, we can consider adding more implementations of Python:
Jython, IronPython, MicroPython, Pyston, Pyjion, etc. All benchmarks
should be run on the same hardware to be comparable.

* Later, we might also allow other projects to upload their own
benchmark results, but we should find a solution to groups benchmark
results per benchmark runner (ex: at least by the hostname, perf JSON
contains the hostname) to not compare two results from two different
hardware

* We should continue to add more benchmarks to the performance
benchmark suite, especially benchmarks more representative of real
applications (we have enough microbenchmarks!)


Links:

* perf: http://perf.readthedocs.io/
* performance: https://github.com/python/performance
* Python Speed mailing list: https://mail.python.org/mailman/listinfo/speed
* https://speed.python.org/ (currently outdated, and don't use performance yet)

See https://pypi.python.org/pypi/performance which contains even more
links to Python benchmarks (PyPy, Pyston, Numba, Pythran, etc.)

Victor

From skip.montanaro at gmail.com  Thu Oct 20 07:47:42 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 20 Oct 2016 06:47:42 -0500
Subject: [Python-Dev] Have I got my hg dependencies correct?
Message-ID: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>

I've recently run into a problem building the math and cmath modules
for 2.7. (I don't rebuild very often, so this problem might have been
around for awhile.) My hg repos look like this:

* My cpython repo pulls from https://hg.python.org/cpython

* My 2.7 repo (and other non-tip repos) pulls from my cpython repo

I think this setup was recommended way back in the day when hg was new
to the Python toolchain to avoid unnecessary network bandwidth.

So, if I execute

hg pull
hg update

in first cpython, then 2.7 repos I should be up-to-date, correct?
However, rebuilding in my 2.7 repo fails to build math and cmath. The
compiler complains that Modules/_math.o doesn't exist. If I manually
execute

make Modules/_math.o
make

after the failure, then the math and cmath modules build.

Looking on bugs.python.org I saw this closed issue:

http://bugs.python.org/issue24421

which seems related. Is it possible that the fix wasn't propagated to
the 2.7 branch? Or perhaps I've fouled up my hg repo relationships? My
other repos which depend on cpython (3.5, 3.4, 3.3, and 3.2) all build
the math module just fine.

I'm running on an ancient MacBook Pro with OS X 10.11.6 (El Capitan)
and XCode 8.0 installed.

Any suggestions?

Skip

From skip.montanaro at gmail.com  Thu Oct 20 08:13:47 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 20 Oct 2016 07:13:47 -0500
Subject: [Python-Dev] Have I got my hg dependencies correct?
In-Reply-To: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
References: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
Message-ID: <CANc-5Uy+xE3OWaVDNy+EOchCPBhgiHmbetsBQxP9TLVGQo_8ZA@mail.gmail.com>

On Thu, Oct 20, 2016 at 6:47 AM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> Is it possible that the fix wasn't propagated to
> the 2.7 branch? Or perhaps I've fouled up my hg repo relationships?

Either way, I went ahead and opened a ticket:

http://bugs.python.org/issue28487

S

From victor.stinner at gmail.com  Thu Oct 20 08:35:10 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 20 Oct 2016 14:35:10 +0200
Subject: [Python-Dev] Have I got my hg dependencies correct?
In-Reply-To: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
References: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
Message-ID: <CAMpsgwa0eK0nMa2PZk7PjRJGTVu1c5yBuZP0ROH_dSaWH4PyMw@mail.gmail.com>

Are you on the 2.7 branch or the default branch?

You might try to cleanup your checkout:

hg up -C -r 2.7
make distclean
hg purge # WARNING! it removes *all* files not tracked by Mercurial
./configure && make

You should also paste the full error message.

Victor

2016-10-20 13:47 GMT+02:00 Skip Montanaro <skip.montanaro at gmail.com>:
> I've recently run into a problem building the math and cmath modules
> for 2.7. (I don't rebuild very often, so this problem might have been
> around for awhile.) My hg repos look like this:
>
> * My cpython repo pulls from https://hg.python.org/cpython
>
> * My 2.7 repo (and other non-tip repos) pulls from my cpython repo
>
> I think this setup was recommended way back in the day when hg was new
> to the Python toolchain to avoid unnecessary network bandwidth.
>
> So, if I execute
>
> hg pull
> hg update
>
> in first cpython, then 2.7 repos I should be up-to-date, correct?
> However, rebuilding in my 2.7 repo fails to build math and cmath. The
> compiler complains that Modules/_math.o doesn't exist. If I manually
> execute
>
> make Modules/_math.o
> make
>
> after the failure, then the math and cmath modules build.
>
> Looking on bugs.python.org I saw this closed issue:
>
> http://bugs.python.org/issue24421
>
> which seems related. Is it possible that the fix wasn't propagated to
> the 2.7 branch? Or perhaps I've fouled up my hg repo relationships? My
> other repos which depend on cpython (3.5, 3.4, 3.3, and 3.2) all build
> the math module just fine.
>
> I'm running on an ancient MacBook Pro with OS X 10.11.6 (El Capitan)
> and XCode 8.0 installed.
>
> Any suggestions?
>
> Skip
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From fijall at gmail.com  Thu Oct 20 09:15:43 2016
From: fijall at gmail.com (Maciej Fijalkowski)
Date: Thu, 20 Oct 2016 15:15:43 +0200
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
In-Reply-To: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
References: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
Message-ID: <CAK5idxRK1Muf5X8OWot2pkUCvvAG2SgU6D+LDKnnaeUH3P7erA@mail.gmail.com>

Hi Victor

Despite the fact that I was not able to find time to run your stuff
yet, thanks for all the awesome work!

On Thu, Oct 20, 2016 at 12:56 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hi,
>
> Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed
> results in depth (up to the hardware and kernel drivers!), I wrote new
> tools and enhanced existing tools.
>
> * I wrote a new perf module which runs benchmarks in a reliable way
> and contains a LOT of features: collect metadata, JSON file format,
> commands to compare, render an histogram, etc.
>
> * I rewrote the Python benchmark suite: the old benchmarks Mercurial
> repository moved to a new performance GitHub project which uses my
> perf module and contains more benchmarks.
>
> * I also made minor enhancements to timeit in Python 3.7 -- some dev
> don't want major changes to not "break the backward compatibility".
>
> For timeit, I suggest to use my perf tool which includes a reliable
> timeit command and has much more features like --duplicate (repeat the
> statements to reduce the cost of the outer loop) and --compare-to
> (compare two versions of Python), but also all builtin perf features
> (JSON output, statistics, histogram, etc.).
>
> I added benchmarks from PyPy and Pyston benchmark suites to
> performance: performance 0.3.1 contains 51 benchmark scripts which run
> a total of 121 benchmarks. Example of tested Python modules:
>
> * SQLAlchemy
> * Dulwich (full Git implementation in Python)
> * Mercurial (currently only the startup time)
> * html5lib
> * pyaes (AES crypto cipher in pure Python)
> * sympy
> * Tornado (HTTP client and server)
> * Django (sadly, only the template engine right now, Pyston contains
> HTTP benchmarks)
> * pathlib
> * spambayes
>
> More benchmarks will be added later. It would be nice to add
> benchmarks on numpy for example, numpy is important for a large part
> of our community.
>
> All these (new or updated) tools can now be used to take smarter
> decisions on optimizations. Please don't push any optimization anymore
> without providing reliable benchmark results!
>
>
> My first major action was to close the latest attempt to
> micro-optimize int+int in Python/ceval.c,
> http://bugs.python.org/issue21955 : I closed the issue as rejected,
> because there is no significant speedup on benchmarks other than two
> (tiny) microbenchmarks. To make sure that no one looses its time on
> trying to micro-optimize int+int, I even added a comment to
> Python/ceval.c :-)
>
>    https://hg.python.org/cpython/rev/61fcb12a9873
>    "Please don't try to micro-optimize int+int"
>
>
> The perf and performance are now well tested: Travis CI runs tests on
> the new commits and pull requests, and the "tox" command can be used
> locally to test different Python versions, pep8, doc, ... in a single
> command.
>
>
> Next steps:
>
> * Run performance 0.3.1 on speed.python.org: the benchmark runner is
> currently stopped (and still uses the old benchmarks project). The
> website part may be updated to allow to download full JSON files which
> includes *all* information (all timings, metadata and more).
>
> * I plan to run performance on CPython 2.7, CPython 3.7, PyPy and PyPy
> 3. Maybe also CPython 3.5 and CPython 3.6 if they don't take too much
> resources.
>
> * Later, we can consider adding more implementations of Python:
> Jython, IronPython, MicroPython, Pyston, Pyjion, etc. All benchmarks
> should be run on the same hardware to be comparable.
>
> * Later, we might also allow other projects to upload their own
> benchmark results, but we should find a solution to groups benchmark
> results per benchmark runner (ex: at least by the hostname, perf JSON
> contains the hostname) to not compare two results from two different
> hardware
>
> * We should continue to add more benchmarks to the performance
> benchmark suite, especially benchmarks more representative of real
> applications (we have enough microbenchmarks!)
>
>
> Links:
>
> * perf: http://perf.readthedocs.io/
> * performance: https://github.com/python/performance
> * Python Speed mailing list: https://mail.python.org/mailman/listinfo/speed
> * https://speed.python.org/ (currently outdated, and don't use performance yet)
>
> See https://pypi.python.org/pypi/performance which contains even more
> links to Python benchmarks (PyPy, Pyston, Numba, Pythran, etc.)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com

From ericsnowcurrently at gmail.com  Thu Oct 20 11:31:42 2016
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 20 Oct 2016 09:31:42 -0600
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
In-Reply-To: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
References: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
Message-ID: <CALFfu7AZsqrazOvfSDMMruqqTumUvxELWrijV08uAsgnbRnwfw@mail.gmail.com>

On Thu, Oct 20, 2016 at 4:56 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hi,
>
> Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed
> results in depth (up to the hardware and kernel drivers!), I wrote new
> tools and enhanced existing tools.

This is a massive contribution.  Thanks!

> All these (new or updated) tools can now be used to take smarter
> decisions on optimizations. Please don't push any optimization anymore
> without providing reliable benchmark results!

+1

-eric

From skip.montanaro at gmail.com  Thu Oct 20 12:23:18 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Thu, 20 Oct 2016 11:23:18 -0500
Subject: [Python-Dev] Have I got my hg dependencies correct?
In-Reply-To: <CAMpsgwa0eK0nMa2PZk7PjRJGTVu1c5yBuZP0ROH_dSaWH4PyMw@mail.gmail.com>
References: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
 <CAMpsgwa0eK0nMa2PZk7PjRJGTVu1c5yBuZP0ROH_dSaWH4PyMw@mail.gmail.com>
Message-ID: <CANc-5UxbbJDZhG-jdSaxNJwDPC=UUD_ibMnYT+UJ=4u=xpo=qQ@mail.gmail.com>

On Thu, Oct 20, 2016 at 7:35 AM, Victor Stinner <victor.stinner at gmail.com>
wrote:
>
> Are you on the 2.7 branch or the default branch?
>
> You might try to cleanup your checkout:
>
> hg up -C -r 2.7
> make distclean
> hg purge # WARNING! it removes *all* files not tracked by Mercurial
> ./configure && make
>
> You should also paste the full error message.

I am on the 2.7 branch. My tree looks like this:

~/src/hgpython/
               cpython
               3.6
               3.5
               3.4
               3.3
               3.2
               2.7

As I indicated, the cpython repo pulls from the central repo. The 3.6 and
2.7 branches pull from cpython. The other 3.x branches pull from the 3.x+1
branch.

Sorry, I no longer have the error message. I wasn't thinking in the correct
order, and executed the suggested hg purge command before retrieving the
error message from my build.out file.

In any case, there seemed to be something about the hg up command:

% hg up -C -r 2.7
6 files updated, 0 files merged, 0 files removed, 0 files unresolved

A plain hg up command didn't update any files. Looking at 2.7/
Makefile.pre.in, I see it now has the necessary target for _math.o. That
must have been one of the six updated files.

Is there some deficiency in a plain hg update command which suggests I
should always use the more complex command you suggested? It's not a huge
deal typing-wise, as I use a shell script to rebuild my world, doing the
necessary work to update and build each branch.

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161020/8c4f73df/attachment.html>

From yselivanov.ml at gmail.com  Thu Oct 20 12:32:07 2016
From: yselivanov.ml at gmail.com (Yury Selivanov)
Date: Thu, 20 Oct 2016 12:32:07 -0400
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
In-Reply-To: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
References: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
Message-ID: <6894cd5e-2dea-84bf-604d-b772df6d8a4e@gmail.com>

Thank you Victor!  This is a massive amount of work.


On 2016-10-20 6:56 AM, Victor Stinner wrote:
> * I plan to run performance on CPython 2.7, CPython 3.7, PyPy and PyPy
> 3. Maybe also CPython 3.5 and CPython 3.6 if they don't take too much
> resources.

I think it's important to run 3.5 & 3.6 to see if we introduce some 
regressions in performance.

Yury

From ethan at stoneleaf.us  Thu Oct 20 12:38:56 2016
From: ethan at stoneleaf.us (Ethan Furman)
Date: Thu, 20 Oct 2016 09:38:56 -0700
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
In-Reply-To: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
References: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
Message-ID: <5808F320.5050506@stoneleaf.us>

On 10/20/2016 03:56 AM, Victor Stinner wrote:

> Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed
> results in depth (up to the hardware and kernel drivers!), I wrote new
> tools and enhanced existing tools.

Thank you!

--
~Ethan~

From zachary.ware+pydev at gmail.com  Thu Oct 20 12:43:31 2016
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Thu, 20 Oct 2016 11:43:31 -0500
Subject: [Python-Dev] Have I got my hg dependencies correct?
In-Reply-To: <CANc-5UxbbJDZhG-jdSaxNJwDPC=UUD_ibMnYT+UJ=4u=xpo=qQ@mail.gmail.com>
References: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
 <CAMpsgwa0eK0nMa2PZk7PjRJGTVu1c5yBuZP0ROH_dSaWH4PyMw@mail.gmail.com>
 <CANc-5UxbbJDZhG-jdSaxNJwDPC=UUD_ibMnYT+UJ=4u=xpo=qQ@mail.gmail.com>
Message-ID: <CAKJDb-PdP9F_qL+byTf657B=5QJqzWQi3DzKQagZwWo0n-VntQ@mail.gmail.com>

On Thu, Oct 20, 2016 at 11:23 AM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
>
> On Thu, Oct 20, 2016 at 7:35 AM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>>
>> Are you on the 2.7 branch or the default branch?
>>
>> You might try to cleanup your checkout:
>>
>> hg up -C -r 2.7
>> make distclean
>> hg purge # WARNING! it removes *all* files not tracked by Mercurial
>> ./configure && make
>>
>> You should also paste the full error message.
>
> I am on the 2.7 branch. My tree looks like this:
>
> ~/src/hgpython/
>                cpython
>                3.6
>                3.5
>                3.4
>                3.3
>                3.2
>                2.7
>
> As I indicated, the cpython repo pulls from the central repo. The 3.6 and
> 2.7 branches pull from cpython. The other 3.x branches pull from the 3.x+1
> branch.
>
> Sorry, I no longer have the error message. I wasn't thinking in the correct
> order, and executed the suggested hg purge command before retrieving the
> error message from my build.out file.
>
> In any case, there seemed to be something about the hg up command:
>
> % hg up -C -r 2.7
> 6 files updated, 0 files merged, 0 files removed, 0 files unresolved
>
> A plain hg up command didn't update any files. Looking at
> 2.7/Makefile.pre.in, I see it now has the necessary target for _math.o. That
> must have been one of the six updated files.
>
> Is there some deficiency in a plain hg update command which suggests I
> should always use the more complex command you suggested? It's not a huge
> deal typing-wise, as I use a shell script to rebuild my world, doing the
> necessary work to update and build each branch.

Sounds like you had an unexpected patch to Makefile.pre.in (and 5
other files) which broke things.  Plain `hg update` will try to keep
any changes in your working directory intact, whereas `hg update -C`
blows away any changes to tracked files.

-- 
Zach

From storchaka at gmail.com  Thu Oct 20 13:04:02 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 20 Oct 2016 20:04:02 +0300
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
In-Reply-To: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
References: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
Message-ID: <nuate0$vqm$1@blaine.gmane.org>

On 20.10.16 13:56, Victor Stinner wrote:
> Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed
> results in depth (up to the hardware and kernel drivers!), I wrote new
> tools and enhanced existing tools.

Great work! Thank you Victor.



From p.f.moore at gmail.com  Thu Oct 20 14:40:57 2016
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 20 Oct 2016 19:40:57 +0100
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
In-Reply-To: <5808F320.5050506@stoneleaf.us>
References: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
 <5808F320.5050506@stoneleaf.us>
Message-ID: <CACac1F9Wg2EKo7ZCnpkRVOy2zfrjzOyXWonDLO7VKyZo-bbNSw@mail.gmail.com>

On 20 October 2016 at 17:38, Ethan Furman <ethan at stoneleaf.us> wrote:
> On 10/20/2016 03:56 AM, Victor Stinner wrote:
>
>> Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed
>> results in depth (up to the hardware and kernel drivers!), I wrote new
>> tools and enhanced existing tools.
>
>
> Thank you!

Indeed, this is brilliant - and often unappreciated - work, so many
thanks for all the work you've put into this.
Paul

From peter.xihong.wang at intel.com  Thu Oct 20 17:23:49 2016
From: peter.xihong.wang at intel.com (Wang, Peter Xihong)
Date: Thu, 20 Oct 2016 21:23:49 +0000
Subject: [Python-Dev] Python-Dev Digest, Vol 159, Issue 27
In-Reply-To: <mailman.51.1476979205.11932.python-dev@python.org>
References: <mailman.51.1476979205.11932.python-dev@python.org>
Message-ID: <371EBC7881C7844EAAF5556BFF21BCCC42E6D7BA@ORSMSX105.amr.corp.intel.com>

Hi Victor,

Thanks for the great contribution to the unified benchmark development!   In addition to the OutReachy program that we are currently supporting, let us know how else we could help out in this effort.

Other than micros and benchmarking ideas, we'd also like to hear suggestions from the community on workload development around real world use cases, especially in the enterprise world, cloud computing, data analytics, machine learning, high performance computing, etc.

Thanks,

Peter
?

-----Original Message-----
From: Python-Dev [mailto:python-dev-bounces+peter.xihong.wang=intel.com at python.org] On Behalf Of python-dev-request at python.org
Sent: Thursday, October 20, 2016 9:00 AM
To: python-dev at python.org
Subject: Python-Dev Digest, Vol 159, Issue 27

Send Python-Dev mailing list submissions to
	python-dev at python.org

To subscribe or unsubscribe via the World Wide Web, visit
	https://mail.python.org/mailman/listinfo/python-dev
or, via email, send a message with subject or body 'help' to
	python-dev-request at python.org

You can reach the person managing the list at
	python-dev-owner at python.org

When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-Dev digest..."


Today's Topics:

   1. Benchmarking Python and micro-optimizations (Victor Stinner)
   2. Have I got my hg dependencies correct? (Skip Montanaro)
   3. Re: Have I got my hg dependencies correct? (Skip Montanaro)
   4. Re: Have I got my hg dependencies correct? (Victor Stinner)
   5. Re: Benchmarking Python and micro-optimizations
      (Maciej Fijalkowski)
   6. Re: Benchmarking Python and micro-optimizations (Eric Snow)


----------------------------------------------------------------------

Message: 1
Date: Thu, 20 Oct 2016 12:56:06 +0200
From: Victor Stinner <victor.stinner at gmail.com>
To: Python Dev <Python-Dev at python.org>
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
Message-ID:
	<CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi,

Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed results in depth (up to the hardware and kernel drivers!), I wrote new tools and enhanced existing tools.

* I wrote a new perf module which runs benchmarks in a reliable way and contains a LOT of features: collect metadata, JSON file format, commands to compare, render an histogram, etc.

* I rewrote the Python benchmark suite: the old benchmarks Mercurial repository moved to a new performance GitHub project which uses my perf module and contains more benchmarks.

* I also made minor enhancements to timeit in Python 3.7 -- some dev don't want major changes to not "break the backward compatibility".

For timeit, I suggest to use my perf tool which includes a reliable timeit command and has much more features like --duplicate (repeat the statements to reduce the cost of the outer loop) and --compare-to (compare two versions of Python), but also all builtin perf features (JSON output, statistics, histogram, etc.).

I added benchmarks from PyPy and Pyston benchmark suites to
performance: performance 0.3.1 contains 51 benchmark scripts which run a total of 121 benchmarks. Example of tested Python modules:

* SQLAlchemy
* Dulwich (full Git implementation in Python)
* Mercurial (currently only the startup time)
* html5lib
* pyaes (AES crypto cipher in pure Python)
* sympy
* Tornado (HTTP client and server)
* Django (sadly, only the template engine right now, Pyston contains HTTP benchmarks)
* pathlib
* spambayes

More benchmarks will be added later. It would be nice to add benchmarks on numpy for example, numpy is important for a large part of our community.

All these (new or updated) tools can now be used to take smarter decisions on optimizations. Please don't push any optimization anymore without providing reliable benchmark results!


My first major action was to close the latest attempt to micro-optimize int+int in Python/ceval.c,
http://bugs.python.org/issue21955 : I closed the issue as rejected, because there is no significant speedup on benchmarks other than two
(tiny) microbenchmarks. To make sure that no one looses its time on trying to micro-optimize int+int, I even added a comment to Python/ceval.c :-)

   https://hg.python.org/cpython/rev/61fcb12a9873
   "Please don't try to micro-optimize int+int"


The perf and performance are now well tested: Travis CI runs tests on the new commits and pull requests, and the "tox" command can be used locally to test different Python versions, pep8, doc, ... in a single command.


Next steps:

* Run performance 0.3.1 on speed.python.org: the benchmark runner is currently stopped (and still uses the old benchmarks project). The website part may be updated to allow to download full JSON files which includes *all* information (all timings, metadata and more).

* I plan to run performance on CPython 2.7, CPython 3.7, PyPy and PyPy 3. Maybe also CPython 3.5 and CPython 3.6 if they don't take too much resources.

* Later, we can consider adding more implementations of Python:
Jython, IronPython, MicroPython, Pyston, Pyjion, etc. All benchmarks should be run on the same hardware to be comparable.

* Later, we might also allow other projects to upload their own benchmark results, but we should find a solution to groups benchmark results per benchmark runner (ex: at least by the hostname, perf JSON contains the hostname) to not compare two results from two different hardware

* We should continue to add more benchmarks to the performance benchmark suite, especially benchmarks more representative of real applications (we have enough microbenchmarks!)


Links:

* perf: http://perf.readthedocs.io/
* performance: https://github.com/python/performance
* Python Speed mailing list: https://mail.python.org/mailman/listinfo/speed
* https://speed.python.org/ (currently outdated, and don't use performance yet)

See https://pypi.python.org/pypi/performance which contains even more links to Python benchmarks (PyPy, Pyston, Numba, Pythran, etc.)

Victor


------------------------------

Message: 2
Date: Thu, 20 Oct 2016 06:47:42 -0500
From: Skip Montanaro <skip.montanaro at gmail.com>
To: python-dev Dev <python-dev at python.org>
Subject: [Python-Dev] Have I got my hg dependencies correct?
Message-ID:
	<CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

I've recently run into a problem building the math and cmath modules for 2.7. (I don't rebuild very often, so this problem might have been around for awhile.) My hg repos look like this:

* My cpython repo pulls from https://hg.python.org/cpython

* My 2.7 repo (and other non-tip repos) pulls from my cpython repo

I think this setup was recommended way back in the day when hg was new to the Python toolchain to avoid unnecessary network bandwidth.

So, if I execute

hg pull
hg update

in first cpython, then 2.7 repos I should be up-to-date, correct?
However, rebuilding in my 2.7 repo fails to build math and cmath. The compiler complains that Modules/_math.o doesn't exist. If I manually execute

make Modules/_math.o
make

after the failure, then the math and cmath modules build.

Looking on bugs.python.org I saw this closed issue:

http://bugs.python.org/issue24421

which seems related. Is it possible that the fix wasn't propagated to the 2.7 branch? Or perhaps I've fouled up my hg repo relationships? My other repos which depend on cpython (3.5, 3.4, 3.3, and 3.2) all build the math module just fine.

I'm running on an ancient MacBook Pro with OS X 10.11.6 (El Capitan) and XCode 8.0 installed.

Any suggestions?

Skip


------------------------------

Message: 3
Date: Thu, 20 Oct 2016 07:13:47 -0500
From: Skip Montanaro <skip.montanaro at gmail.com>
To: python-dev Dev <python-dev at python.org>
Subject: Re: [Python-Dev] Have I got my hg dependencies correct?
Message-ID:
	<CANc-5Uy+xE3OWaVDNy+EOchCPBhgiHmbetsBQxP9TLVGQo_8ZA at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

On Thu, Oct 20, 2016 at 6:47 AM, Skip Montanaro <skip.montanaro at gmail.com> wrote:
> Is it possible that the fix wasn't propagated to the 2.7 branch? Or 
> perhaps I've fouled up my hg repo relationships?

Either way, I went ahead and opened a ticket:

http://bugs.python.org/issue28487

S


------------------------------

Message: 4
Date: Thu, 20 Oct 2016 14:35:10 +0200
From: Victor Stinner <victor.stinner at gmail.com>
To: Skip Montanaro <skip.montanaro at gmail.com>
Cc: python-dev Dev <python-dev at python.org>
Subject: Re: [Python-Dev] Have I got my hg dependencies correct?
Message-ID:
	<CAMpsgwa0eK0nMa2PZk7PjRJGTVu1c5yBuZP0ROH_dSaWH4PyMw at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Are you on the 2.7 branch or the default branch?

You might try to cleanup your checkout:

hg up -C -r 2.7
make distclean
hg purge # WARNING! it removes *all* files not tracked by Mercurial ./configure && make

You should also paste the full error message.

Victor

2016-10-20 13:47 GMT+02:00 Skip Montanaro <skip.montanaro at gmail.com>:
> I've recently run into a problem building the math and cmath modules 
> for 2.7. (I don't rebuild very often, so this problem might have been 
> around for awhile.) My hg repos look like this:
>
> * My cpython repo pulls from https://hg.python.org/cpython
>
> * My 2.7 repo (and other non-tip repos) pulls from my cpython repo
>
> I think this setup was recommended way back in the day when hg was new 
> to the Python toolchain to avoid unnecessary network bandwidth.
>
> So, if I execute
>
> hg pull
> hg update
>
> in first cpython, then 2.7 repos I should be up-to-date, correct?
> However, rebuilding in my 2.7 repo fails to build math and cmath. The 
> compiler complains that Modules/_math.o doesn't exist. If I manually 
> execute
>
> make Modules/_math.o
> make
>
> after the failure, then the math and cmath modules build.
>
> Looking on bugs.python.org I saw this closed issue:
>
> http://bugs.python.org/issue24421
>
> which seems related. Is it possible that the fix wasn't propagated to 
> the 2.7 branch? Or perhaps I've fouled up my hg repo relationships? My 
> other repos which depend on cpython (3.5, 3.4, 3.3, and 3.2) all build 
> the math module just fine.
>
> I'm running on an ancient MacBook Pro with OS X 10.11.6 (El Capitan) 
> and XCode 8.0 installed.
>
> Any suggestions?
>
> Skip
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gm
> ail.com


------------------------------

Message: 5
Date: Thu, 20 Oct 2016 15:15:43 +0200
From: Maciej Fijalkowski <fijall at gmail.com>
To: Victor Stinner <victor.stinner at gmail.com>
Cc: Python Dev <Python-Dev at python.org>
Subject: Re: [Python-Dev] Benchmarking Python and micro-optimizations
Message-ID:
	<CAK5idxRK1Muf5X8OWot2pkUCvvAG2SgU6D+LDKnnaeUH3P7erA at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi Victor

Despite the fact that I was not able to find time to run your stuff yet, thanks for all the awesome work!

On Thu, Oct 20, 2016 at 12:56 PM, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed 
> results in depth (up to the hardware and kernel drivers!), I wrote new 
> tools and enhanced existing tools.
>
> * I wrote a new perf module which runs benchmarks in a reliable way 
> and contains a LOT of features: collect metadata, JSON file format, 
> commands to compare, render an histogram, etc.
>
> * I rewrote the Python benchmark suite: the old benchmarks Mercurial 
> repository moved to a new performance GitHub project which uses my 
> perf module and contains more benchmarks.
>
> * I also made minor enhancements to timeit in Python 3.7 -- some dev 
> don't want major changes to not "break the backward compatibility".
>
> For timeit, I suggest to use my perf tool which includes a reliable 
> timeit command and has much more features like --duplicate (repeat the 
> statements to reduce the cost of the outer loop) and --compare-to 
> (compare two versions of Python), but also all builtin perf features 
> (JSON output, statistics, histogram, etc.).
>
> I added benchmarks from PyPy and Pyston benchmark suites to
> performance: performance 0.3.1 contains 51 benchmark scripts which run 
> a total of 121 benchmarks. Example of tested Python modules:
>
> * SQLAlchemy
> * Dulwich (full Git implementation in Python)
> * Mercurial (currently only the startup time)
> * html5lib
> * pyaes (AES crypto cipher in pure Python)
> * sympy
> * Tornado (HTTP client and server)
> * Django (sadly, only the template engine right now, Pyston contains 
> HTTP benchmarks)
> * pathlib
> * spambayes
>
> More benchmarks will be added later. It would be nice to add 
> benchmarks on numpy for example, numpy is important for a large part 
> of our community.
>
> All these (new or updated) tools can now be used to take smarter 
> decisions on optimizations. Please don't push any optimization anymore 
> without providing reliable benchmark results!
>
>
> My first major action was to close the latest attempt to 
> micro-optimize int+int in Python/ceval.c,
> http://bugs.python.org/issue21955 : I closed the issue as rejected, 
> because there is no significant speedup on benchmarks other than two
> (tiny) microbenchmarks. To make sure that no one looses its time on 
> trying to micro-optimize int+int, I even added a comment to 
> Python/ceval.c :-)
>
>    https://hg.python.org/cpython/rev/61fcb12a9873
>    "Please don't try to micro-optimize int+int"
>
>
> The perf and performance are now well tested: Travis CI runs tests on 
> the new commits and pull requests, and the "tox" command can be used 
> locally to test different Python versions, pep8, doc, ... in a single 
> command.
>
>
> Next steps:
>
> * Run performance 0.3.1 on speed.python.org: the benchmark runner is 
> currently stopped (and still uses the old benchmarks project). The 
> website part may be updated to allow to download full JSON files which 
> includes *all* information (all timings, metadata and more).
>
> * I plan to run performance on CPython 2.7, CPython 3.7, PyPy and PyPy 
> 3. Maybe also CPython 3.5 and CPython 3.6 if they don't take too much 
> resources.
>
> * Later, we can consider adding more implementations of Python:
> Jython, IronPython, MicroPython, Pyston, Pyjion, etc. All benchmarks 
> should be run on the same hardware to be comparable.
>
> * Later, we might also allow other projects to upload their own 
> benchmark results, but we should find a solution to groups benchmark 
> results per benchmark runner (ex: at least by the hostname, perf JSON 
> contains the hostname) to not compare two results from two different 
> hardware
>
> * We should continue to add more benchmarks to the performance 
> benchmark suite, especially benchmarks more representative of real 
> applications (we have enough microbenchmarks!)
>
>
> Links:
>
> * perf: http://perf.readthedocs.io/
> * performance: https://github.com/python/performance
> * Python Speed mailing list: 
> https://mail.python.org/mailman/listinfo/speed
> * https://speed.python.org/ (currently outdated, and don't use 
> performance yet)
>
> See https://pypi.python.org/pypi/performance which contains even more 
> links to Python benchmarks (PyPy, Pyston, Numba, Pythran, etc.)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/fijall%40gmail.com


------------------------------

Message: 6
Date: Thu, 20 Oct 2016 09:31:42 -0600
From: Eric Snow <ericsnowcurrently at gmail.com>
To: Victor Stinner <victor.stinner at gmail.com>
Cc: Python Dev <Python-Dev at python.org>
Subject: Re: [Python-Dev] Benchmarking Python and micro-optimizations
Message-ID:
	<CALFfu7AZsqrazOvfSDMMruqqTumUvxELWrijV08uAsgnbRnwfw at mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

On Thu, Oct 20, 2016 at 4:56 AM, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed 
> results in depth (up to the hardware and kernel drivers!), I wrote new 
> tools and enhanced existing tools.

This is a massive contribution.  Thanks!

> All these (new or updated) tools can now be used to take smarter 
> decisions on optimizations. Please don't push any optimization anymore 
> without providing reliable benchmark results!

+1

-eric


------------------------------

Subject: Digest Footer

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev


------------------------------

End of Python-Dev Digest, Vol 159, Issue 27
*******************************************

From ncoghlan at gmail.com  Fri Oct 21 02:48:53 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 21 Oct 2016 16:48:53 +1000
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <-4662978122166531554@unknownmsgid>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
 <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>
 <-4662978122166531554@unknownmsgid>
Message-ID: <CADiSq7dTX3KLVnqcovQ4aB7A3iTZOS9+Lb1HUseAXdKtGmfLFw@mail.gmail.com>

On 19 October 2016 at 01:28, Chris Barker - NOAA Federal
<chris.barker at noaa.gov> wrote:
>>>>> def get_builtin_methods():
>>    ...     return [(name, method_name) for name, obj in
>> get_builtin_types().items() for method_name, method in
>> vars(obj).items() if not method_name.startswith("__")]
>>    ...
>>>>> len(get_builtin_methods())
>>    230
>
> So what? No one looks in all the methods of builtins at once.

Yes, Python implementation developers do, which is why it's a useful
part of defining the overall "size" of Python and how that is growing
over time.

When we define a new standard library module (particularly pure Python
ones) rather than new methods on builtin types, we create
substantially less additional work for other implementations, and we
make it easier for educators to decide whether or not they should be
introducing their students to the new capabilities.

That latter aspect is important, as providing functionality as
separate modules means we also gain an enhanced ability to explain
"What is this *for*?", which is something we regularly struggle with
when making changes to the core language to better support relatively
advanced domain specific use cases (see
http://learning-python.com/books/python-changes-2014-plus.html for one
generalist author's perspective on the vast gulf that can arise
between "What professional programmers want" and "What's relevant to
new programmers")

> If we
> have anything like an OO System (and python builtins only sort of
> do...) then folks look for a built in that they need, and only then
> look at its methods.
>
> If you need to work with bytes, you'll look at the bytes object and
> bytarray object. Having to go find some helper function module to know
> to efficiently do something with bytes is VERY non-discoverable!

Which is more comprehensible and discoverable, dict.setdefault(), or
collections.defaultdict()?

Micro-optimisations like dict.setdefault() typically don't make sense
in isolation - they only make sense in the context of a particular
pattern of thought. Now, one approach to such patterns is to say "We
just need to do a better job of teaching people to recognise and use
the pattern!". This approach tends not to work very well - you're
often better off extracting the entire pattern out to a higher level
construct, giving that construct a name, and teaching that, and
letting people worry about how it works internally later.

(For a slightly different example, consider the rationale for adding
the `secrets` module, even though it's mostly just a collection of
relatively thin wrappers around `os.urandom()`)

> bytes and bytarray are already low-level objects -- adding low-level
> functionality to them makes perfect sense.

They're not really that low level. They're *relatively* low level
(especially for Python), but they're still a long way away from the
kind of raw control over memory layout that a language like C or Rust
can give you.

> And no, this is not just for asycio at all -- it's potentially useful
> for any byte manipulation.

Yes, which is why I think the end goal should be a public `iobuffers`
module in the standard library. Doing IO buffer manipulation
efficiently is a complex topic, but it's also one where there are:

- many repeatable patterns for managing IO buffers efficiently that
aren't necessarily applicable to manipulating arbitrary binary data
(ring buffers, ropes, etc)
- many operating system level utilities available to make it even more
efficient that we currently don't use (since we only have general
purpose "bytes" and "bytearray" objects with no "iobuffer" specific
abstraction that could take advantage of those use case specific
features)

> +1 on a frombuffer() method.

Still -1 in the absence of evidence that a good IO buffer abstraction
for asyncio and the standard library can't be written without it
(where the evidence I'll accept is "We already wrote the abstraction
layer, and not having this builtin feature necessarily introduces
inefficiencies or a lack of portability beyond CPython into our
implementation").

>> Putting special purpose functionality behind an import gate helps to
>> provide a more explicit context of use
>
> This is a fine argument for putting bytearray in a separate module --
> but that ship has sailed. The method to construct a bytearray from a
> buffer belongs with the bytearray object.

The bytearray constructor already accepts arbitrary bytes-like
objects. What this proposal is about is a way to *more efficiently*
snapshot a slice of a bytearray object for use in asyncio buffer
manipulation in cases where all of the following constraints apply:

- we don't want to copy the data twice
- we don't want to let a memoryview be cleaned up lazily
- we don't want to incur the readability penalty of explicitly
managing the memoryview

For a great many use cases, we simply don't care about those
constraints (especially the last one), so adding `bytes.frombuffer` is
just confusing: we can readily predict that after adding it, a future
Stack Overflow question will be "When should I use bytes.frombuffer()
in Python instead of the normal bytes constructor?"

By contrast, if we instead say "We want Python to natively support
efficient. readily discoverable, IO buffer manipulation", then folks
can ask "What's preventing us from providing an `iobuffers` module
today?" and start working towards that end goal (just as
the"selectors" module was added as an asyncio-independent abstraction
layer over select, epoll and kqueue, but probably wouldn't have been
without the asyncio use case to drive its design and implementation as
a standard library module)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From njs at pobox.com  Fri Oct 21 03:09:21 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 21 Oct 2016 00:09:21 -0700
Subject: [Python-Dev] Is there any remaining reason why weakref callbacks
 shouldn't be able to access the referenced object?
Message-ID: <CAPJVwB=KfLz1JJC+9Oxry9SR33YTqqZXdkhEJp=jiDU9TwDFoQ@mail.gmail.com>

Hi all,

It's an old feature of the weakref API that you can define an
arbitrary callback to be invoked when the referenced object dies, and
that when this callback is invoked, it gets handed the weakref wrapper
object -- BUT, only after it's been cleared, so that the callback
can't access the originally referenced object. (I.e., this callback
will never raise: def callback(ref): assert ref() is None.)

AFAICT the original motivation for this seems was that if the weakref
callback could get at the object, then the weakref callback would
effectively be another finalizer like __del__, and finalizers and
reference cycles don't mix, so weakref callbacks can't be finalizers.
There's a long document from the 2.4 days about all the terrible
things that could happen if arbitrary code like callbacks could get
unfettered access to cyclic isolates at weakref cleanup time [1].

But that was 2.4. In the mean time, of course, PEP 442 fixed it so
that finalizers and weakrefs mix just fine. In fact, weakref callbacks
are now run *before* __del__ methods [2], so clearly it's now okay for
arbitrary code to touch the objects during that phase of the GC -- at
least in principle.

So what I'm wondering is, would anything terrible happen if we started
passing still-live weakrefs into weakref callbacks, and then clearing
them afterwards? (i.e. making step 1 of the PEP 442 cleanup order be
"run callbacks and then clear weakrefs", instead of the current "clear
weakrefs and then run callbacks"). I skimmed through the PEP 442
discussion, and AFAICT the rationale for keeping the old weakref
behavior was just that no-one could be bothered to mess with it [3].

[The motivation for my question is partly curiosity, and partly that
in the discussion about how to handle GC for async objects, it
occurred to me that it might be very nice if arbitrary classes that
needed access to the event loop during cleanup could do something like

  def __init__(self, ...):
      loop = asyncio.get_event_loop()
      loop.gc_register(self)

  # automatically called by the loop when I am GC'ed; async equivalent
of __del__
  async def aclose(self):
      ...

Right now something *sort* of like this is possible but it requires a
much more cumbersome API, where every class would have to implement
logic to fetch a cleanup callback from the loop, store it, and then
call it from its __del__ method -- like how PEP 525 does it. Delaying
weakref clearing would make this simpler API possible.]

-n

[1] https://github.com/python/cpython/blob/master/Modules/gc_weakref.txt
[2] https://www.python.org/dev/peps/pep-0442/#id7
[3] https://mail.python.org/pipermail/python-dev/2013-May/126592.html

-- 
Nathaniel J. Smith -- https://vorpus.org

From brett at python.org  Fri Oct 21 14:12:57 2016
From: brett at python.org (Brett Cannon)
Date: Fri, 21 Oct 2016 18:12:57 +0000
Subject: [Python-Dev] Have I got my hg dependencies correct?
In-Reply-To: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
References: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
Message-ID: <CAP1=2W6NeUcRe1-EGWyEbiiHV5S69Qn8-K+pNWuhfzMjLOds6A@mail.gmail.com>

On Thu, 20 Oct 2016 at 04:48 Skip Montanaro <skip.montanaro at gmail.com>
wrote:

> I've recently run into a problem building the math and cmath modules
> for 2.7. (I don't rebuild very often, so this problem might have been
> around for awhile.) My hg repos look like this:
>
> * My cpython repo pulls from https://hg.python.org/cpython
>
> * My 2.7 repo (and other non-tip repos) pulls from my cpython repo
>
> I think this setup was recommended way back in the day when hg was new
> to the Python toolchain to avoid unnecessary network bandwidth.
>
> So, if I execute
>
> hg pull
> hg update
>
> in first cpython, then 2.7 repos I should be up-to-date, correct?
>

Nope, you need to execute the same steps in your 2.7 checkout if you're
keeping it in a separate directory from your cpython repo that you're
referring to (you can also use `hg pull -u` to do the two steps above in a
single command).
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161021/b061d837/attachment.html>

From status at bugs.python.org  Fri Oct 21 12:08:48 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 21 Oct 2016 18:08:48 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20161021160848.3875356647@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-10-14 - 2016-10-21)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5524 ( -3)
  closed 34728 (+55)
  total  40252 (+52)

Open issues with patches: 2398 


Issues opened (28)
==================

#28404: Logging SyslogHandler not appending '\n' to the end
http://bugs.python.org/issue28404  reopened by elelement

#28437: Documentation for handling of non-type metaclass hints is uncl
http://bugs.python.org/issue28437  reopened by ncoghlan

#28445: Wrong documentation for GzipFile.peek
http://bugs.python.org/issue28445  opened by abacabadabacaba

#28446: pyvenv generates malformed hashbangs for scripts
http://bugs.python.org/issue28446  opened by alexreg

#28449: tarfile.open(mode = 'r:*', ignore_zeros = True) has 50% chance
http://bugs.python.org/issue28449  opened by Silver Fox

#28450: Misleading/inaccurate documentation about unknown escape seque
http://bugs.python.org/issue28450  opened by lelit

#28451: pydoc.safeimport() raises ErrorDuringImport() if __builtin__._
http://bugs.python.org/issue28451  opened by segfault87

#28453: SSLObject.selected_alpn_protocol() not documented
http://bugs.python.org/issue28453  opened by alex.gronholm

#28457: Make public the current private known hash functions in the C-
http://bugs.python.org/issue28457  opened by rhettinger

#28459: _pyio module broken on Cygwin / setmode not usable
http://bugs.python.org/issue28459  opened by erik.bray

#28460: Minidom, order of attributes, datachars
http://bugs.python.org/issue28460  opened by Petr Pulc

#28462: subprocess pipe can't see EOF from a child in case of a few ch
http://bugs.python.org/issue28462  opened by Vyacheslav Grigoryev

#28463: Email long headers parsing/serialization
http://bugs.python.org/issue28463  opened by ???????????????????? ????????????

#28464: BaseEventLoop.close should shutdown executor before marking it
http://bugs.python.org/issue28464  opened by cmeyer

#28465: python 3.5 magic number
http://bugs.python.org/issue28465  opened by ??????

#28469: timeit: use powers of 2 in autorange(), instead of powers of 1
http://bugs.python.org/issue28469  opened by haypo

#28470: configure.ac -g debug compiler option when not Py_DEBUG
http://bugs.python.org/issue28470  opened by Chris Byers

#28474: WinError(): Python int too large to convert to C long
http://bugs.python.org/issue28474  opened by Kelvin You

#28475: Misleading error on random.sample when k < 0
http://bugs.python.org/issue28475  opened by franciscouzo

#28477: Add optional user argument to pathlib.Path.home()
http://bugs.python.org/issue28477  opened by josh.r

#28478: Built-in module 'time' does not enable functions if -Werror sp
http://bugs.python.org/issue28478  opened by toast12

#28482: test_typing fails if asyncio unavailable
http://bugs.python.org/issue28482  opened by martin.panter

#28485: compileall.compile_dir(workers=<negative>) does not raise Valu
http://bugs.python.org/issue28485  opened by martin.panter

#28488: shutil.make_archive (xxx, zip, root_dir) is adding './' entry 
http://bugs.python.org/issue28488  opened by bialix

#28489: Fix comment in tokenizer.c
http://bugs.python.org/issue28489  opened by Ryan.Gonzalez

#28491: Remove bundled libffi for OSX
http://bugs.python.org/issue28491  opened by zach.ware

#28494: is_zipfile false positives
http://bugs.python.org/issue28494  opened by Thomas.Waldmann

#28496: Mark up constants 0, 1, -1 in C API docs
http://bugs.python.org/issue28496  opened by serhiy.storchaka



Most recent 15 issues with no replies (15)
==========================================

#28485: compileall.compile_dir(workers=<negative>) does not raise Valu
http://bugs.python.org/issue28485

#28470: configure.ac -g debug compiler option when not Py_DEBUG
http://bugs.python.org/issue28470

#28464: BaseEventLoop.close should shutdown executor before marking it
http://bugs.python.org/issue28464

#28460: Minidom, order of attributes, datachars
http://bugs.python.org/issue28460

#28457: Make public the current private known hash functions in the C-
http://bugs.python.org/issue28457

#28446: pyvenv generates malformed hashbangs for scripts
http://bugs.python.org/issue28446

#28439: Remove redundant checks in PyUnicode_EncodeLocale and PyUnicod
http://bugs.python.org/issue28439

#28429: ctypes fails to import with grsecurity's TPE
http://bugs.python.org/issue28429

#28422: multiprocessing Manager mutable type member access failure
http://bugs.python.org/issue28422

#28416: defining persistent_id in _pickle.Pickler subclass causes refe
http://bugs.python.org/issue28416

#28412: os.path.splitdrive documentation out of date
http://bugs.python.org/issue28412

#28408: Fix redundant code and memory leak in _PyUnicodeWriter_Finish
http://bugs.python.org/issue28408

#28407: Improve coverage of email.utils.make_msgid()
http://bugs.python.org/issue28407

#28401: Don't support the PEP384 stable ABI in pydebug builds
http://bugs.python.org/issue28401

#28396: Remove *.pyo references from man page
http://bugs.python.org/issue28396



Most recent 15 issues waiting for review (15)
=============================================

#28496: Mark up constants 0, 1, -1 in C API docs
http://bugs.python.org/issue28496

#28494: is_zipfile false positives
http://bugs.python.org/issue28494

#28491: Remove bundled libffi for OSX
http://bugs.python.org/issue28491

#28489: Fix comment in tokenizer.c
http://bugs.python.org/issue28489

#28488: shutil.make_archive (xxx, zip, root_dir) is adding './' entry 
http://bugs.python.org/issue28488

#28485: compileall.compile_dir(workers=<negative>) does not raise Valu
http://bugs.python.org/issue28485

#28482: test_typing fails if asyncio unavailable
http://bugs.python.org/issue28482

#28475: Misleading error on random.sample when k < 0
http://bugs.python.org/issue28475

#28469: timeit: use powers of 2 in autorange(), instead of powers of 1
http://bugs.python.org/issue28469

#28459: _pyio module broken on Cygwin / setmode not usable
http://bugs.python.org/issue28459

#28451: pydoc.safeimport() raises ErrorDuringImport() if __builtin__._
http://bugs.python.org/issue28451

#28449: tarfile.open(mode = 'r:*', ignore_zeros = True) has 50% chance
http://bugs.python.org/issue28449

#28444: Missing extensions modules when cross compiling python 3.5.2 f
http://bugs.python.org/issue28444

#28443: Logger methods never use kwargs
http://bugs.python.org/issue28443

#28441: Change sys.executable to include executable suffix
http://bugs.python.org/issue28441



Top 10 most discussed issues (10)
=================================

#28444: Missing extensions modules when cross compiling python 3.5.2 f
http://bugs.python.org/issue28444  25 msgs

#28437: Documentation for handling of non-type metaclass hints is uncl
http://bugs.python.org/issue28437  12 msgs

#19795: Formatting of True/False/None in docs
http://bugs.python.org/issue19795  10 msgs

#26240: Docstring of the subprocess module should be cleaned up
http://bugs.python.org/issue26240   8 msgs

#28463: Email long headers parsing/serialization
http://bugs.python.org/issue28463   6 msgs

#28489: Fix comment in tokenizer.c
http://bugs.python.org/issue28489   6 msgs

#28491: Remove bundled libffi for OSX
http://bugs.python.org/issue28491   6 msgs

#23214: BufferedReader.read1(size) signature incompatible with Buffere
http://bugs.python.org/issue23214   5 msgs

#28474: WinError(): Python int too large to convert to C long
http://bugs.python.org/issue28474   5 msgs

#14991: Option for regex groupdict() to show only matching names
http://bugs.python.org/issue14991   4 msgs



Issues closed (54)
==================

#16007: Improved Error message for failing re expressions
http://bugs.python.org/issue16007  closed by serhiy.storchaka

#18219: csv.DictWriter is slow when writing files with large number of
http://bugs.python.org/issue18219  closed by inada.naoki

#21955: ceval.c: implement fast path for integers with a single digit
http://bugs.python.org/issue21955  closed by haypo

#23231: Fix codecs.iterencode/decode() by allowing data parameter to b
http://bugs.python.org/issue23231  closed by martin.panter

#23782: Leak in _PyTraceback_Add
http://bugs.python.org/issue23782  closed by serhiy.storchaka

#24381: Got warning when compiling ffi.c on Mac
http://bugs.python.org/issue24381  closed by ned.deily

#25550: RecursionError in re with '(' * 500
http://bugs.python.org/issue25550  closed by serhiy.storchaka

#26010: document CO_* constants
http://bugs.python.org/issue26010  closed by matrixise

#26436: Add the regex-dna benchmark
http://bugs.python.org/issue26436  closed by haypo

#26942: android: test_ctypes crashes on armv7 and aarch64
http://bugs.python.org/issue26942  closed by xdegaye

#26944: test_posix: Android 'id -G' is entirely wrong or missing the e
http://bugs.python.org/issue26944  closed by xdegaye

#27471: sre_constants.error: bad escape \d
http://bugs.python.org/issue27471  closed by serhiy.storchaka

#27627: clang fails to build ctypes on Android armv7
http://bugs.python.org/issue27627  closed by xdegaye

#27800: Regular expressions with multiple repeat codes
http://bugs.python.org/issue27800  closed by martin.panter

#27844: Python-3.6a4 build messages to stderr (on AIX and xlc compiler
http://bugs.python.org/issue27844  closed by martin.panter

#27896: Allow passing sphinx options to Doc/Makefile
http://bugs.python.org/issue27896  closed by haypo

#28092: Build failure for 3.6 on Centos 5.11
http://bugs.python.org/issue28092  closed by benjamin.peterson

#28099: Drop Mac OS X Tiger support in Python 3.6
http://bugs.python.org/issue28099  closed by ned.deily

#28214: Improve exception reporting for problematic __set_name__ attri
http://bugs.python.org/issue28214  closed by serhiy.storchaka

#28240: Enhance the timeit module: display average +- std dev instead 
http://bugs.python.org/issue28240  closed by haypo

#28256: Cleanup Modules/_math.c
http://bugs.python.org/issue28256  closed by haypo

#28409: test.regrtest does not support multiple -x flags
http://bugs.python.org/issue28409  closed by haypo

#28410: Add convenient C API for "raise ... from ..."
http://bugs.python.org/issue28410  closed by serhiy.storchaka

#28425: Python3 ignores __init__.py that are links to /dev/null
http://bugs.python.org/issue28425  closed by brett.cannon

#28428: Rename _futures module to _asyncio
http://bugs.python.org/issue28428  closed by inada.naoki

#28432: Fix doc of PyUnicode_EncodeLocale
http://bugs.python.org/issue28432  closed by berker.peksag

#28442: tuple(a list subclass) does not iterate through the list
http://bugs.python.org/issue28442  closed by siming85

#28447: socket.getpeername() failure on broken TCP/IP connection
http://bugs.python.org/issue28447  closed by martin.panter

#28448: C implemented Future doesn't work on Windows
http://bugs.python.org/issue28448  closed by inada.naoki

#28452: Remove _asyncio._init_module function
http://bugs.python.org/issue28452  closed by inada.naoki

#28454: Spurious arguments to PyErr_Format in unicodeobject.c
http://bugs.python.org/issue28454  closed by python-dev

#28455: argparse: convert_arg_line_to_args does not actually expect se
http://bugs.python.org/issue28455  closed by berker.peksag

#28456: Test failures under macOS 10.12 Sierra
http://bugs.python.org/issue28456  closed by ned.deily

#28458: from __future__ import print_function does not emulate the flu
http://bugs.python.org/issue28458  closed by berker.peksag

#28461: Replacement of re with the regex package
http://bugs.python.org/issue28461  closed by serhiy.storchaka

#28466: SIGALRM fails to interrupt time.sleep() call on Python 3.5
http://bugs.python.org/issue28466  closed by martin.panter

#28467: Installer should be a 64-bit executable
http://bugs.python.org/issue28467  closed by steve.dower

#28468: Add platform.linux_os_release()
http://bugs.python.org/issue28468  closed by christian.heimes

#28471: Python 3.6b2 crashes with "Python memory allocator called with
http://bugs.python.org/issue28471  closed by yselivanov

#28472: SystemTap usage examples in docs are incorrect
http://bugs.python.org/issue28472  closed by python-dev

#28473: mailbox.MH crashes on certain Claws Mail .mh_sequences files
http://bugs.python.org/issue28473  closed by r.david.murray

#28476: Remove redundant definition of factorial on test_random
http://bugs.python.org/issue28476  closed by python-dev

#28479: Missing indentation in using/windows.rst
http://bugs.python.org/issue28479  closed by python-dev

#28480: Compile error on Modules/socketmodule.c
http://bugs.python.org/issue28480  closed by martin.panter

#28481: Weird string comparison bug
http://bugs.python.org/issue28481  closed by eryksun

#28483: python --version prints output to stderr
http://bugs.python.org/issue28483  closed by SilentGhost

#28484: test_capi, test_regrtest fail when multithreading disabled
http://bugs.python.org/issue28484  closed by martin.panter

#28486: Wrong end index and subgroup for group match
http://bugs.python.org/issue28486  closed by serhiy.storchaka

#28487: missing _math.o target in 2.7 Makefile
http://bugs.python.org/issue28487  closed by haypo

#28490: inappropriate OS.Error "Invalid cross-device link"
http://bugs.python.org/issue28490  closed by steve.newcomb

#28492: C implementation of asyncio.Future doesn't set value of StopIt
http://bugs.python.org/issue28492  closed by yselivanov

#28493: Typo in _asynciomodule.c
http://bugs.python.org/issue28493  closed by yselivanov

#28495: MagickMock created by pat??h annotation refuses to apply funct
http://bugs.python.org/issue28495  closed by r.david.murray

#1520879: make install change: Allow $DESTDIR to be relative
http://bugs.python.org/issue1520879  closed by dgreiman

From skip.montanaro at gmail.com  Fri Oct 21 14:28:42 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Fri, 21 Oct 2016 13:28:42 -0500
Subject: [Python-Dev] Have I got my hg dependencies correct?
In-Reply-To: <CAP1=2W6NeUcRe1-EGWyEbiiHV5S69Qn8-K+pNWuhfzMjLOds6A@mail.gmail.com>
References: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
 <CAP1=2W6NeUcRe1-EGWyEbiiHV5S69Qn8-K+pNWuhfzMjLOds6A@mail.gmail.com>
Message-ID: <CANc-5Uzr9F6ehTd2HWTHAToxAN5M81V+PNE4b=oRPL7j22s2Pw@mail.gmail.com>

On Fri, Oct 21, 2016 at 1:12 PM, Brett Cannon <brett at python.org> wrote:
>> in first cpython, then 2.7 repos I should be up-to-date, correct?
>
>
> Nope, you need to execute the same steps in your 2.7 checkout

"repos" == "checkout" in my message.

So the hg up -C solved my problem, but I'm still a bit confused
(nothing new, in addition to which I only use hg for my Python
repositories)... Why didn't a plain "hg up" tell me it couldn't update
some files because of changes? Or, like git (I think), attempt to
incorporate the upstream changes, then leave conflict markers if that
failed?

Skip

From chris.barker at noaa.gov  Fri Oct 21 17:57:38 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 21 Oct 2016 14:57:38 -0700
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CADiSq7dTX3KLVnqcovQ4aB7A3iTZOS9+Lb1HUseAXdKtGmfLFw@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
 <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>
 <-4662978122166531554@unknownmsgid>
 <CADiSq7dTX3KLVnqcovQ4aB7A3iTZOS9+Lb1HUseAXdKtGmfLFw@mail.gmail.com>
Message-ID: <CALGmxEJkfZ1pAhWYoO+XixCsVQ6LNyLv2MU3MvdVGyBmqYtUhQ@mail.gmail.com>

On Thu, Oct 20, 2016 at 11:48 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> >>>>> len(get_builtin_methods())
> >>    230
> >
> > So what? No one looks in all the methods of builtins at once.
>
> Yes, Python implementation developers do, which is why it's a useful
> part of defining the overall "size" of Python and how that is growing
> over time.
>

sure -- but of course, the trick is that adding *one" new method is never a
big deal by itself.

I'm confused though -- IIUC, you are proposing adding a `iobuffers`
module to the std lib -- how is that not growing the "size" of Python?


I'm still confused about the "io" in "iobuffers" -- I've used buffers a lot
 -- for passing data around between various C libs -- numpy, image
processing, etc... I never really thought of it as IO though. which is why
a  simple frombuffer() seems to make a lot of sense to me, without any
other stuff. (to be honest, I reach for Cyton these days for that sort of
thing though)


>  and we

make it easier for educators to decide whether or not they should be
> introducing their students to the new capabilities.
>

advanced domain specific use cases (see
> http://learning-python.com/books/python-changes-2014-plus.html for one
> generalist author's perspective on the vast gulf that can arise
> between "What professional programmers want" and "What's relevant to
> new programmers")
>

thanks for the link -- I'll need to read the whole thing through -- though
from a glance, I have a slightly different perspective, as an educator as
well:

Python 3, in general, is harder to learn and less suited to scripting,
while potentially more suited to building larer systems.

I came to this conclusion last year when I converted my introductory class
to py3.

Some of it is the redundancy and whatnot talked about in that link --  yes,
those are issue or me. But more of it is real, maybe important change.
Interestingly, the biggest issue with the transition: Unicode, is one thing
that has made life much easier for newbies :-)

But the big ones are things like:

The more to be iterable focused rather than sequence focused -- iterables
really are harder to wrap one's head around when you are first learning.
And I was surprised at how often I had to wrap list() around stuff when
converting my examples and exercise solutions.

I've decided to teach the format() method for string formatting -- but it
is harder to wrap your head around as a newbie.

Even the extra parens in print() makes it a bit harder to script() well.

Use with: -- now I have to explain context managers before they can even
read a file.. (or gloss over it and jsut say " copy this code to open a
file"

Anyway, I've been meaning to write a Blog post about this, that would be
better formed, but you get the idea.

In short, I really appreciate the issues here -- though I really don't see
how adding one method to a fairily obscure builtin really applies -- this
is nothing like having three(!) ways to format strings.

Which is more comprehensible and discoverable, dict.setdefault(), or
> collections.defaultdict()?
>

Well, setdefault is Definitively more discoverable! not sure what your
point is.

As it happens, the homework for my intro class this week can greatly
benefit from setdefault() (or defaultdict() ) -- and in the last few years,
far fewer newbies have discovered defaultdict() for their solutions.
Empirical evidence for discoverability.

As for comprehensible -- I give a slight nod to .setdefault() - my solution
to the HW uses that. I can't say I have a strong argument as to why -- but
having (what looks like) a whole new class for this one extra feature seems
a bit odd, and makes one look carefully to see what else might be different
about it...


> Micro-optimisations like dict.setdefault() typically don't make sense
> in isolation - they only make sense in the context of a particular
> pattern of thought. Now, one approach to such patterns is to say "We
> just need to do a better job of teaching people to recognise and use
> the pattern!". This approach tends not to work very well - you're
> often better off extracting the entire pattern out to a higher level
> construct, giving that construct a name, and teaching that, and
> letting people worry about how it works internally later.
>

hmm -- maybe -- but to me, that example isn't really a pattern of thought
(to me) -- I actually remember my history of learning about setdefault(). I
found myself writing a bunch of code something like:

if key not in a_dict:
    a_dict[key] = something
a_dict['key'].somethign_or_other()

Once I had written that code a few times, I thought: "There has got to be a
cleaner way to do this", looked at the dict methods and eventually found
setdefault() (took an embarrassingly long time). I did think -- "this has
got to be a common enough pattern to be somehow supported" but I will say
that it never, ever dawned on me to think: "this is got to be a common
enough pattern that someone would have made a special kind of dictionary
for it" -- I thought of it as a feature you'd want with a dict -- into a
different kind of dict.

So in the end, if one were to ask ME whether Python should have a
.setdefault() method on dict, or an DefaultDict object -- I'd say the
method.

> bytes and bytarray are already low-level objects -- adding low-level
> > functionality to them makes perfect sense.
>
> They're not really that low level. They're *relatively* low level
> (especially for Python),


sure -- but that is the context here.


> > And no, this is not just for asycio at all -- it's potentially useful
> > for any byte manipulation.
>

Anyway, I think frombuffer() would be a very nice thing to have, and where
it goes is secondary.


> Yes, which is why I think the end goal should be a public `iobuffers`
> module in the standard library.


I guess I'd need to see a proposal for what would go in that module to have
any opinion on  whether it would be a good idea. Is anyone actually working
on such a proposal?

-CHB


-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161021/e2ff7772/attachment.html>

From tjreedy at udel.edu  Fri Oct 21 22:05:26 2016
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 21 Oct 2016 22:05:26 -0400
Subject: [Python-Dev] Have I got my hg dependencies correct?
In-Reply-To: <CAP1=2W6NeUcRe1-EGWyEbiiHV5S69Qn8-K+pNWuhfzMjLOds6A@mail.gmail.com>
References: <CANc-5UwuWxYkN5LgOdDPEO5-7xYDCLK5Xv6dFP=sLBaxbH1K9g@mail.gmail.com>
 <CAP1=2W6NeUcRe1-EGWyEbiiHV5S69Qn8-K+pNWuhfzMjLOds6A@mail.gmail.com>
Message-ID: <nuehh4$q31$1@blaine.gmane.org>

On 10/21/2016 2:12 PM, Brett Cannon wrote:
>
>
> On Thu, 20 Oct 2016 at 04:48 Skip Montanaro <skip.montanaro at gmail.com
> <mailto:skip.montanaro at gmail.com>> wrote:
>
>     I've recently run into a problem building the math and cmath modules
>     for 2.7. (I don't rebuild very often, so this problem might have been
>     around for awhile.) My hg repos look like this:
>
>     * My cpython repo pulls from https://hg.python.org/cpython
>
>     * My 2.7 repo (and other non-tip repos) pulls from my cpython repo
>
>     I think this setup was recommended way back in the day when hg was new
>     to the Python toolchain to avoid unnecessary network bandwidth.
>
>     So, if I execute
>
>     hg pull
>     hg update
>
>     in first cpython, then 2.7 repos I should be up-to-date, correct?
>
>
> Nope, you need to execute the same steps in your 2.7 checkout if you're
> keeping it in a separate directory from your cpython repo that you're
> referring to

If the 2.7 repository shares the default repository, as described in the 
devguide, then only update is needed.  This has worked for me for at 
least two years.


-- 
Terry Jan Reedy


From ncoghlan at gmail.com  Fri Oct 21 23:13:16 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 22 Oct 2016 13:13:16 +1000
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
In-Reply-To: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
References: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
Message-ID: <CADiSq7ekW0cQSiZ-EfKYWM8NZVC+ZJ3Q=-VH3E=8SPTSwPBeQA@mail.gmail.com>

On 20 October 2016 at 20:56, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> Last months, I worked a lot on benchmarks. I ran benchmarks, analyzed
> results in depth (up to the hardware and kernel drivers!), I wrote new
> tools and enhanced existing tools.

Thanks Victor, very cool work!

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri Oct 21 23:32:04 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 22 Oct 2016 13:32:04 +1000
Subject: [Python-Dev] Is there any remaining reason why weakref
 callbacks shouldn't be able to access the referenced object?
In-Reply-To: <CAPJVwB=KfLz1JJC+9Oxry9SR33YTqqZXdkhEJp=jiDU9TwDFoQ@mail.gmail.com>
References: <CAPJVwB=KfLz1JJC+9Oxry9SR33YTqqZXdkhEJp=jiDU9TwDFoQ@mail.gmail.com>
Message-ID: <CADiSq7c-kAyGBeFescTfoY_bz0X6P7xCmLDPrQQvHh7vV=ELTw@mail.gmail.com>

On 21 October 2016 at 17:09, Nathaniel Smith <njs at pobox.com> wrote:
> But that was 2.4. In the mean time, of course, PEP 442 fixed it so
> that finalizers and weakrefs mix just fine. In fact, weakref callbacks
> are now run *before* __del__ methods [2], so clearly it's now okay for
> arbitrary code to touch the objects during that phase of the GC -- at
> least in principle.
>
> So what I'm wondering is, would anything terrible happen if we started
> passing still-live weakrefs into weakref callbacks, and then clearing
> them afterwards?

The weakref-before-__del__ ordering change in
https://www.python.org/dev/peps/pep-0442/#disposal-of-cyclic-isolates
only applies to cyclic garbage collection,so for normal refcount
driven object cleanup in CPython, the __del__ still happens first:

    >>> class C:
    ...     def __del__(self):
    ...         print("__del__ called")
    ...
    >>> c = C()
    >>> import weakref
    >>> def cb():
    ...     print("weakref callback called")
    ...
    >>> weakref.finalize(c, cb)
    <finalize object at 0x7f4300b710a0; for 'C' at 0x7f42f8ae3470>
    >>> del c
    __del__ called
    weakref callback called

This means the main problem with a strong reference being reachable
from the weakref callback object remains: if the callback itself is
reachable, then the original object is reachable, and you don't have a
collectible cycle anymore.

    >>> c = C()
    >>> def cb2(obj):
    ...     print("weakref callback called with object reference")
    ...
    >>> weakref.finalize(c, cb2, c)
    <finalize object at 0x7f4300b710b0; for 'C' at 0x7f42f8ae3470>
    >>> del c
    >>>

Changing that to support resurrecting the object so it can be passed
into the callback without the callback itself holding a strong
reference means losing the main "reasoning about software" benefit
that weakref callbacks offer: they currently can't resurrect the
object they relate to (since they never receive a strong reference to
it), so it nominally doesn't matter if the interpreter calls them
before or after that object has been entirely cleaned up.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From njs at pobox.com  Sat Oct 22 02:05:56 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Fri, 21 Oct 2016 23:05:56 -0700
Subject: [Python-Dev] Is there any remaining reason why weakref
 callbacks shouldn't be able to access the referenced object?
In-Reply-To: <CADiSq7c-kAyGBeFescTfoY_bz0X6P7xCmLDPrQQvHh7vV=ELTw@mail.gmail.com>
References: <CAPJVwB=KfLz1JJC+9Oxry9SR33YTqqZXdkhEJp=jiDU9TwDFoQ@mail.gmail.com>
 <CADiSq7c-kAyGBeFescTfoY_bz0X6P7xCmLDPrQQvHh7vV=ELTw@mail.gmail.com>
Message-ID: <CAPJVwBknpeDk+Y_feYTJX3Et=hnj802rMVzCWtSmG-7U7JzzVg@mail.gmail.com>

On Fri, Oct 21, 2016 at 8:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 21 October 2016 at 17:09, Nathaniel Smith <njs at pobox.com> wrote:
>> But that was 2.4. In the mean time, of course, PEP 442 fixed it so
>> that finalizers and weakrefs mix just fine. In fact, weakref callbacks
>> are now run *before* __del__ methods [2], so clearly it's now okay for
>> arbitrary code to touch the objects during that phase of the GC -- at
>> least in principle.
>>
>> So what I'm wondering is, would anything terrible happen if we started
>> passing still-live weakrefs into weakref callbacks, and then clearing
>> them afterwards?
>
> The weakref-before-__del__ ordering change in
> https://www.python.org/dev/peps/pep-0442/#disposal-of-cyclic-isolates
> only applies to cyclic garbage collection,so for normal refcount
> driven object cleanup in CPython, the __del__ still happens first:
>
>     >>> class C:
>     ...     def __del__(self):
>     ...         print("__del__ called")
>     ...
>     >>> c = C()
>     >>> import weakref
>     >>> def cb():
>     ...     print("weakref callback called")
>     ...
>     >>> weakref.finalize(c, cb)
>     <finalize object at 0x7f4300b710a0; for 'C' at 0x7f42f8ae3470>
>     >>> del c
>     __del__ called
>     weakref callback called

Ah, interesting! And in the old days this was of course the right way
to do it, because until __del__ has completed it's possible that the
object will get resurrected, and you don't want to clear the weakref
until you're certain that it's dead.

But PEP 442 already broke all that :-). Now weakref callbacks can
happen before __del__, and they can happen on objects that are about
to be resurrected. So if we wanted to pursue this then it seems like
it would make sense to standardize on the following sequence for
object teardown:

0) object becomes collectible (either refcount == 0 or it's part of a
cyclic isolate)
1) weakref callbacks fire
2) weakrefs are cleared (unconditionally, so we keep the rule that any
given weakref fires at most once, even if the object is resurrected)
3) if _PyGC_REFS_MASK_FINALIZED isn't set, __del__ fires, and then
_PyGC_REFS_MASK_FINALIZED is set
4) check for resurrection
5) deallocate the object

On further thought, this does still introduce one new edge case, which
is that even if we keep the guarantee that no individual weakref can
fire more than once, it's possible for *new* weakrefs to be registered
after resurrection, so it becomes possible for an object to be
resurrected multiple times. (Currently, resurrection can only happen
once, because __del__ is disabled on resurrected objects and weakrefs
can't resurrect at all.) I'm not actually sure that this is even a
problem, but in any case it's easy to fix by making a rule that you
can't take a weakref to an object whose _PyGC_REFS_MASK_FINALIZED flag
is already set, plus adjust the teardown sequence to be:

0) object becomes collectible (either refcount == 0 or it's part of a
cyclic isolate)
1) if _PyGC_REFS_MASK_FINALIZED is set, then go to step 7. Otherwise:
2) set _PyGC_REFS_MASK_FINALIZED
3) weakref callbacks fire
4) weakrefs are cleared (unconditionally)
5) __del__ fires
6) check for resurrection
7) deallocate the object

There remains one obscure corner case where multiple resurrection is
possible, because the resurrection-prevention flag doesn't exist on
non-GC objects, so you'd still be able to take new weakrefs to those.
But in that case __del__ can already do multiple resurrections, and
some fellow named Nick Coghlan seemed to think that was okay back in
2013 [1], so probably it's not too bad ;-).

[1] https://mail.python.org/pipermail/python-dev/2013-June/126850.html

> This means the main problem with a strong reference being reachable
> from the weakref callback object remains: if the callback itself is
> reachable, then the original object is reachable, and you don't have a
> collectible cycle anymore.
>
>     >>> c = C()
>     >>> def cb2(obj):
>     ...     print("weakref callback called with object reference")
>     ...
>     >>> weakref.finalize(c, cb2, c)
>     <finalize object at 0x7f4300b710b0; for 'C' at 0x7f42f8ae3470>
>     >>> del c
>     >>>
>
> Changing that to support resurrecting the object so it can be passed
> into the callback without the callback itself holding a strong
> reference means losing the main "reasoning about software" benefit
> that weakref callbacks offer: they currently can't resurrect the
> object they relate to (since they never receive a strong reference to
> it), so it nominally doesn't matter if the interpreter calls them
> before or after that object has been entirely cleaned up.

I guess I'm missing the importance of this -- does the interpreter
gain some particular benefit from having flexibility about when to
fire weakref callbacks? Obviously it has to pick one in practice.

(The async use case that got me thinking about this is, of course,
exactly one where we would want a weakref callback to resurrect the
object it refers to. Only once, though.)

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From victor.stinner at gmail.com  Sat Oct 22 02:47:24 2016
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sat, 22 Oct 2016 08:47:24 +0200
Subject: [Python-Dev] Benchmarking Python and micro-optimizations
In-Reply-To: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
References: <CAMpsgwYsAXJgHfJH_hDH6pj13pHacMWYbi_LExLoXdcoUo6F-Q@mail.gmail.com>
Message-ID: <CAMpsgwbyAfovmZ_99q68G8AMa3Xx3ZLpZGLg0HWFk7Fjgf5-BQ@mail.gmail.com>

Hi,

I removed all old benchmarks results and I started to run manually
benchmarks. The timeline view is interesting to investigate
performance regression:
https://speed.python.org/timeline/#/?exe=3&ben=grid&env=1&revs=50&equid=off&quarts=on&extr=on

For example, it seems like call_method became slower between Oct 9 and
Oct 20: 35.9 ms => 59.9 ms:
https://speed.python.org/timeline/#/?exe=3&ben=call_method&env=1&revs=50&equid=off&quarts=on&extr=on

I don't know well the hardware of the benchmark runner, so maybe it's
an issue with the server running benchmarks?

Victor

From ncoghlan at gmail.com  Sat Oct 22 06:26:30 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 22 Oct 2016 20:26:30 +1000
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CALGmxEJkfZ1pAhWYoO+XixCsVQ6LNyLv2MU3MvdVGyBmqYtUhQ@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
 <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>
 <-4662978122166531554@unknownmsgid>
 <CADiSq7dTX3KLVnqcovQ4aB7A3iTZOS9+Lb1HUseAXdKtGmfLFw@mail.gmail.com>
 <CALGmxEJkfZ1pAhWYoO+XixCsVQ6LNyLv2MU3MvdVGyBmqYtUhQ@mail.gmail.com>
Message-ID: <CADiSq7cjfSeuaSKxb_Jd69UEFU8CL3xx8EnqTK3BVzsKHhRrXw@mail.gmail.com>

On 22 October 2016 at 07:57, Chris Barker <chris.barker at noaa.gov> wrote:
> I'm still confused about the "io" in "iobuffers" -- I've used buffers a lot
> -- for passing data around between various C libs -- numpy, image
> processing, etc... I never really thought of it as IO though. which is why a
> simple frombuffer() seems to make a lot of sense to me, without any other
> stuff. (to be honest, I reach for Cyton these days for that sort of thing
> though)

That's the essence of my point though: if you care enough about the
performance of a piece of code for the hidden copy in
"bytes(mydata[start:stop])" to be deemed unacceptable, and also can't
afford the lazy cleanup of the view in
"bytes(memoryview(mydata)[start:stop])", then it seems likely that
you're writing specialist, high performance, low overhead, data
manipulation code, that probably shouldn't be written in Python

In such cases, an extension module written in something like Cython, C
or Rust would be a better fit, as using the more appropriate tool will
give you a range of additional performance improvements (near)
automatically, such as getting to avoid the runtime overhead of
Python's dynamic type system.

At that point, having to write the lowest-available-overhead version
explicitly in Python as:

    with memoryview(mydata) as view:
        return bytes(mydata[start:stop]

is a sign that someone is insisting on staying in pure Python code
when they're do sufficiently low level bit bashing that it probably
isn't the best idea to continue down that path.

>From that perspective, adding "[bytes/bytearray].frombuffer" is adding
complexity to the core language for the sake of giving people one
small additional piece of incremental performance improvement that
they can eke out before they admit to themselves "OK, I'm probably not
using the right language for this part of my application".

By contrast, a library that provided better low level data buffer
manipulation that was suitable for asyncio's needs is *much* easier to
emulate on older versions, and provides more scope for extracting
efficient data manipulation patterns beyond this one very specific
case of more efficiently snapshotting a subset of an existing buffer.

Cheers,
Nick.

P.S. I bring up Rust and the runtime overhead of the type system
specifically here, as Armin Ronacher recently wrote an excellent post
about that in relation to some performance improvement work they were
doing at Sentry:
https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat Oct 22 06:01:49 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 22 Oct 2016 20:01:49 +1000
Subject: [Python-Dev] Is there any remaining reason why weakref
 callbacks shouldn't be able to access the referenced object?
In-Reply-To: <CAPJVwBknpeDk+Y_feYTJX3Et=hnj802rMVzCWtSmG-7U7JzzVg@mail.gmail.com>
References: <CAPJVwB=KfLz1JJC+9Oxry9SR33YTqqZXdkhEJp=jiDU9TwDFoQ@mail.gmail.com>
 <CADiSq7c-kAyGBeFescTfoY_bz0X6P7xCmLDPrQQvHh7vV=ELTw@mail.gmail.com>
 <CAPJVwBknpeDk+Y_feYTJX3Et=hnj802rMVzCWtSmG-7U7JzzVg@mail.gmail.com>
Message-ID: <CADiSq7cpgNQ8kvY65tc4zaS1-Ltgsfxv-r_qQQP9LFRjbvZ8AA@mail.gmail.com>

On 22 October 2016 at 16:05, Nathaniel Smith <njs at pobox.com> wrote:
> On Fri, Oct 21, 2016 at 8:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> But PEP 442 already broke all that :-). Now weakref callbacks can
> happen before __del__, and they can happen on objects that are about
> to be resurrected.

Right, but the resurrection can still only happen *in* __del__, so the
interpreter doesn't need to deal with the case where it happens in a
weakref callback instead - that's where the freedom to do the
callbacks and the __del__ in either order comes from.

> There remains one obscure corner case where multiple resurrection is
> possible, because the resurrection-prevention flag doesn't exist on
> non-GC objects, so you'd still be able to take new weakrefs to those.
> But in that case __del__ can already do multiple resurrections, and
> some fellow named Nick Coghlan seemed to think that was okay back in
> 2013 [1], so probably it's not too bad ;-).
>
> [1] https://mail.python.org/pipermail/python-dev/2013-June/126850.html

Right, that still doesn't bother me.

>> Changing that to support resurrecting the object so it can be passed
>> into the callback without the callback itself holding a strong
>> reference means losing the main "reasoning about software" benefit
>> that weakref callbacks offer: they currently can't resurrect the
>> object they relate to (since they never receive a strong reference to
>> it), so it nominally doesn't matter if the interpreter calls them
>> before or after that object has been entirely cleaned up.
>
> I guess I'm missing the importance of this -- does the interpreter
> gain some particular benefit from having flexibility about when to
> fire weakref callbacks? Obviously it has to pick one in practice.

Sorry, my attempted clarification of one practical implication made it
look like I was defining the phrase I had in quotes. However, the
"reasoning about software" benefit I see is "If you don't define
__del__, you don't need to worry about object resurrection, as it's
categorically impossible when only using weakref callbacks".
Interpreter implementors are just one set of beneficiaries of that
simplification - everyone writing weakref callbacks qualifies as well.

However, if you're happy defining __del__ methods, then PEP 442 means
you can already inject lazy cyclic cleanup that supports resurrection:

    >>> class Target:
    ...     pass
    ...
    >>> class Resurrector:
    ...     def __init__(self, target):
    ...         _self_ref = "_resurrector_{:d}".format(id(self))
    ...         self.target = target
    ...         setattr(target, _self_ref, self)
    ...     def __del__(self):
    ...         globals()["resurrected"] = self.target
    ...
    >>> obj = Target()
    >>> Resurrector(obj)
    <__main__.Resurrector object at 0x7f42f8ae34e0>
    >>> del obj
    >>> resurrected
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    NameError: name 'resurrected' is not defined
    >>> import gc
    >>> gc.collect(); gc.collect(); gc.collect()
    6
    4
    0
    >>> resurrected
    <__main__.Target object at 0x7f42f8ae3438>

Given that, I don't see a lot of benefit in making weakref callbacks
harder to reason about when __del__ + attribute injection already
makes this possible.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From njs at pobox.com  Sat Oct 22 14:22:12 2016
From: njs at pobox.com (Nathaniel Smith)
Date: Sat, 22 Oct 2016 11:22:12 -0700
Subject: [Python-Dev] Is there any remaining reason why weakref
 callbacks shouldn't be able to access the referenced object?
In-Reply-To: <CADiSq7cpgNQ8kvY65tc4zaS1-Ltgsfxv-r_qQQP9LFRjbvZ8AA@mail.gmail.com>
References: <CAPJVwB=KfLz1JJC+9Oxry9SR33YTqqZXdkhEJp=jiDU9TwDFoQ@mail.gmail.com>
 <CADiSq7c-kAyGBeFescTfoY_bz0X6P7xCmLDPrQQvHh7vV=ELTw@mail.gmail.com>
 <CAPJVwBknpeDk+Y_feYTJX3Et=hnj802rMVzCWtSmG-7U7JzzVg@mail.gmail.com>
 <CADiSq7cpgNQ8kvY65tc4zaS1-Ltgsfxv-r_qQQP9LFRjbvZ8AA@mail.gmail.com>
Message-ID: <CAPJVwBmkS1ZhvGCK-iZq-MweDCunew9JiMFE+cV5X5kFyHA1Og@mail.gmail.com>

On Sat, Oct 22, 2016 at 3:01 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 22 October 2016 at 16:05, Nathaniel Smith <njs at pobox.com> wrote:
>> On Fri, Oct 21, 2016 at 8:32 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> But PEP 442 already broke all that :-). Now weakref callbacks can
>> happen before __del__, and they can happen on objects that are about
>> to be resurrected.
>
> Right, but the resurrection can still only happen *in* __del__, so the
> interpreter doesn't need to deal with the case where it happens in a
> weakref callback instead - that's where the freedom to do the
> callbacks and the __del__ in either order comes from.

I think we're probably on the same page here, but to be clear, my
point is that right now the resurrection logic seems to be (a) run
some arbitrary Python code (__del__), (b) run a second check to see if
a resurrection occurred (and the details of that check depend on
whether the object is part of a cyclic isolate). Since these two
phases are already decoupled from each other, it shouldn't cause any
particular difficulty for the interpreter if we add weakref callbacks
to the "run arbitrary code" phase. If we wanted to.

>> There remains one obscure corner case where multiple resurrection is
>> possible, because the resurrection-prevention flag doesn't exist on
>> non-GC objects, so you'd still be able to take new weakrefs to those.
>> But in that case __del__ can already do multiple resurrections, and
>> some fellow named Nick Coghlan seemed to think that was okay back in
>> 2013 [1], so probably it's not too bad ;-).
>>
>> [1] https://mail.python.org/pipermail/python-dev/2013-June/126850.html
>
> Right, that still doesn't bother me.
>
>>> Changing that to support resurrecting the object so it can be passed
>>> into the callback without the callback itself holding a strong
>>> reference means losing the main "reasoning about software" benefit
>>> that weakref callbacks offer: they currently can't resurrect the
>>> object they relate to (since they never receive a strong reference to
>>> it), so it nominally doesn't matter if the interpreter calls them
>>> before or after that object has been entirely cleaned up.
>>
>> I guess I'm missing the importance of this -- does the interpreter
>> gain some particular benefit from having flexibility about when to
>> fire weakref callbacks? Obviously it has to pick one in practice.
>
> Sorry, my attempted clarification of one practical implication made it
> look like I was defining the phrase I had in quotes. However, the
> "reasoning about software" benefit I see is "If you don't define
> __del__, you don't need to worry about object resurrection, as it's
> categorically impossible when only using weakref callbacks".
> Interpreter implementors are just one set of beneficiaries of that
> simplification - everyone writing weakref callbacks qualifies as well.

I do like invariants, but I'm having trouble seeing why this one is
super valuable. I mean, if your object doesn't define __del__, then
it's also impossible to distinguish between a weakref causing
resurrection and a strong reference that prevents the object from
being collected in the first place. And certainly it's harmless in the
use case I have in mind, where normally the weakref would be created
in the object's __init__ anyway :-).

> However, if you're happy defining __del__ methods, then PEP 442 means
> you can already inject lazy cyclic cleanup that supports resurrection:
>
>     >>> class Target:
>     ...     pass
>     ...
>     >>> class Resurrector:
>     ...     def __init__(self, target):
>     ...         _self_ref = "_resurrector_{:d}".format(id(self))
>     ...         self.target = target
>     ...         setattr(target, _self_ref, self)
>     ...     def __del__(self):
>     ...         globals()["resurrected"] = self.target
>     ...
>     >>> obj = Target()
>     >>> Resurrector(obj)
>     <__main__.Resurrector object at 0x7f42f8ae34e0>
>     >>> del obj
>     >>> resurrected
>     Traceback (most recent call last):
>       File "<stdin>", line 1, in <module>
>     NameError: name 'resurrected' is not defined
>     >>> import gc
>     >>> gc.collect(); gc.collect(); gc.collect()
>     6
>     4
>     0
>     >>> resurrected
>     <__main__.Target object at 0x7f42f8ae3438>
>
> Given that, I don't see a lot of benefit in making weakref callbacks
> harder to reason about when __del__ + attribute injection already
> makes this possible.

That's a cute trick :-). But it does have one major downside compared
to allowing weakref callbacks to access the object normally. With
weakrefs you don't interfere with when the object is normally
collected, and in particular for objects that aren't part of cycles,
they're still collected promptly (on CPython). Here every object
becomes part of a cycle, so objects that would otherwise be collected
promptly won't be.

(Remember that the reason I started thinking about this was that I was
wondering if we could have a nice API for the asyncio event loop to
"take over" the job of finalizing an object -- so ideally you'd want
this finalizer to act as much like a regular __del__ method as
possible.)

Anyway I doubt we'll see any changes to this in the immediate future,
but it's nice to get a sense of what the possible design landscape
looks like...

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From barry at barrys-emacs.org  Mon Oct 24 06:43:43 2016
From: barry at barrys-emacs.org (Barry Scott)
Date: Mon, 24 Oct 2016 11:43:43 +0100
Subject: [Python-Dev] Preference for of patch submission?
Message-ID: <6F20C12C-CC59-40DE-9CF0-5368EAF9690D@barrys-emacs.org>

I am over due providing a patch for a doc issue that was discussed on
the ideas list.

What is the preferred way to provide cpython with a patch these days?

bug report + patch?
pull request on github?
something else?

Barry


From guido at python.org  Mon Oct 24 10:50:29 2016
From: guido at python.org (Guido van Rossum)
Date: Mon, 24 Oct 2016 07:50:29 -0700
Subject: [Python-Dev] Preference for of patch submission?
In-Reply-To: <6F20C12C-CC59-40DE-9CF0-5368EAF9690D@barrys-emacs.org>
References: <6F20C12C-CC59-40DE-9CF0-5368EAF9690D@barrys-emacs.org>
Message-ID: <CAP7+vJLyhcvSK==v0vrgBhM1SgdFNoiZSs9rKxK1A=WCXP=Nhw@mail.gmail.com>

The preferred (only) way is upload a patch to bugs.python.org. The
patch should be relative to a recent branch of the Hg master repo. If
the patch applies cleanly to the Hg master repo, a link to the code
review will appear within a few minutes there.

On Mon, Oct 24, 2016 at 3:43 AM, Barry Scott <barry at barrys-emacs.org> wrote:
> I am over due providing a patch for a doc issue that was discussed on
> the ideas list.
>
> What is the preferred way to provide cpython with a patch these days?
>
> bug report + patch?
> pull request on github?
> something else?
>
> Barry
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org



-- 
--Guido van Rossum (python.org/~guido)

From chris.barker at noaa.gov  Mon Oct 24 12:53:49 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 24 Oct 2016 09:53:49 -0700
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CADiSq7cjfSeuaSKxb_Jd69UEFU8CL3xx8EnqTK3BVzsKHhRrXw@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
 <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>
 <-4662978122166531554@unknownmsgid>
 <CADiSq7dTX3KLVnqcovQ4aB7A3iTZOS9+Lb1HUseAXdKtGmfLFw@mail.gmail.com>
 <CALGmxEJkfZ1pAhWYoO+XixCsVQ6LNyLv2MU3MvdVGyBmqYtUhQ@mail.gmail.com>
 <CADiSq7cjfSeuaSKxb_Jd69UEFU8CL3xx8EnqTK3BVzsKHhRrXw@mail.gmail.com>
Message-ID: <CALGmxE+V0kcXVE0nLiCR=ypG-bJ2Zr7Y63jNLjJT52nVO+2OCQ@mail.gmail.com>

Before the horse is totally dead... (maybe it already is), a couple
comments:


> In such cases, an extension module written in something like Cython, C
> or Rust would be a better fit,


well, yes, but:


> From that perspective, adding "[bytes/bytearray].frombuffer"


this would be used for the fairly simple use case of passing stuff around
between different modules, which would probably be written in something
lower lever -- unless they too, only passed data around. Passign data
around is a pretty good use-case for Python.

is adding
> complexity to the core language for the sake of giving people one
> small additional piece of incremental performance improvement


here's the thing: this is a very small increase in complexity in exchange
for a small increase in performance -- really not a big deal either way. If
either of those were large, hte decision would be a no brainer.

By contrast, a library that provided better low level data buffer
> manipulation that was suitable for asyncio's needs is *much* easier to
> emulate on older versions, and provides more scope for extracting
> efficient data manipulation patterns beyond this one very specific
> case of more efficiently snapshotting a subset of an existing buffer.
>


IS this na either-or? IF someone is proposing a nice lib for "low level
data buffer
manipulation", then yes, putting frombuffer() in there would be a fine idea.

But if there is no such proposal on the table, then I think adding a
frombuffer method to the bytes object is a small improvement that we can do
now.

https://blog.sentry.io/2016/10/19/fixing-python-performance-with-rust.html


pretty cool -- I guess I should take a look at Rust...

Thanks,

  -CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161024/5de65b70/attachment.html>

From ncoghlan at gmail.com  Tue Oct 25 03:25:50 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 25 Oct 2016 17:25:50 +1000
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CALGmxE+V0kcXVE0nLiCR=ypG-bJ2Zr7Y63jNLjJT52nVO+2OCQ@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
 <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>
 <-4662978122166531554@unknownmsgid>
 <CADiSq7dTX3KLVnqcovQ4aB7A3iTZOS9+Lb1HUseAXdKtGmfLFw@mail.gmail.com>
 <CALGmxEJkfZ1pAhWYoO+XixCsVQ6LNyLv2MU3MvdVGyBmqYtUhQ@mail.gmail.com>
 <CADiSq7cjfSeuaSKxb_Jd69UEFU8CL3xx8EnqTK3BVzsKHhRrXw@mail.gmail.com>
 <CALGmxE+V0kcXVE0nLiCR=ypG-bJ2Zr7Y63jNLjJT52nVO+2OCQ@mail.gmail.com>
Message-ID: <CADiSq7dLE49QUbiUCnEXgsqKeoEnZGBpxGELO+KC_nNW080Q3A@mail.gmail.com>

On 25 October 2016 at 02:53, Chris Barker <chris.barker at noaa.gov> wrote:
> IS this na either-or? IF someone is proposing a nice lib for "low level data
> buffer
> manipulation", then yes, putting frombuffer() in there would be a fine idea.
>
> But if there is no such proposal on the table, then I think adding a
> frombuffer method to the bytes object is a small improvement that we can do
> now.

The suggestion came from folks working on asyncio performance
improvements, and we already got the entire ``selectors`` abstraction
from the original asyncio implementation work, as well as Yury's new
libuv-based ``uvloop`` asyncio event loop implementation.

Given that "make OpenStack/SDN/NFV run faster" is a key point of
interest for folks like Intel and Red Hat, I'm *absolutely* suggesting
that they put some paid time and energy into a lower level buffer
manipulation library between now and the Python 3.7 feature freeze in
12+ months, as I think that will have longer term pay-offs well beyond
the scope of the original use cases :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue Oct 25 03:27:17 2016
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 25 Oct 2016 17:27:17 +1000
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CADiSq7dLE49QUbiUCnEXgsqKeoEnZGBpxGELO+KC_nNW080Q3A@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
 <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>
 <-4662978122166531554@unknownmsgid>
 <CADiSq7dTX3KLVnqcovQ4aB7A3iTZOS9+Lb1HUseAXdKtGmfLFw@mail.gmail.com>
 <CALGmxEJkfZ1pAhWYoO+XixCsVQ6LNyLv2MU3MvdVGyBmqYtUhQ@mail.gmail.com>
 <CADiSq7cjfSeuaSKxb_Jd69UEFU8CL3xx8EnqTK3BVzsKHhRrXw@mail.gmail.com>
 <CALGmxE+V0kcXVE0nLiCR=ypG-bJ2Zr7Y63jNLjJT52nVO+2OCQ@mail.gmail.com>
 <CADiSq7dLE49QUbiUCnEXgsqKeoEnZGBpxGELO+KC_nNW080Q3A@mail.gmail.com>
Message-ID: <CADiSq7evsoX1M90i80regbupbzcPGX11GU=oyQ22JakDybxBPw@mail.gmail.com>

On 25 October 2016 at 17:25, Nick Coghlan <ncoghlan at gmail.com> wrote:
> as well as Yury's new
> libuv-based ``uvloop`` asyncio event loop implementation.

Oops, I phrased that badly - the default asyncio event loop
implementation is still pure Python, but uvloop is a drop-in Cython
based replacement.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Tue Oct 25 05:37:32 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 25 Oct 2016 12:37:32 +0300
Subject: [Python-Dev] Default formatting
Message-ID: <nun94q$64o$1@blaine.gmane.org>

Classes that doesn't define the __format__ method for custom PEP 3101 
formatting inherits it from parents.

Originally the object.__format__ method was designed as [1]:

     def __format__(self, format_spec):
         return format(str(self), format_spec)

An instance is converted to string and resulting string is formatted 
according to format specifier.

Later this design was reconsidered [2], and now object.__format__ is 
equivalent to:

     def __format__(self, format_spec):
         assert format_spec == ''
         return format(str(self), '')

Non-empty format specifier is rejected.

But why call format() on resulting string? Why not return resulting 
string as is? object.__format__ could be simpler (not just 
implementation, but for understanding):

     def __format__(self, format_spec):
         assert format_spec == ''
         return str(self)

This can change the behaviour in corner case. str(self) can return not 
exact string, but string subclass with overloaded __format__. But I 
think we can ignore such subtle difference.

[1] https://www.python.org/dev/peps/pep-3101/
[2] http://bugs.python.org/issue7994


From chris.barker at noaa.gov  Tue Oct 25 17:25:20 2016
From: chris.barker at noaa.gov (Chris Barker)
Date: Tue, 25 Oct 2016 14:25:20 -0700
Subject: [Python-Dev] Adding bytes.frombuffer() constructor to PEP 467
 (was: [Python-ideas] Adding bytes.frombuffer() constructor
In-Reply-To: <CADiSq7dLE49QUbiUCnEXgsqKeoEnZGBpxGELO+KC_nNW080Q3A@mail.gmail.com>
References: <CAEfz+Tzu0WK7sjWqUeYLp9LZht9SM3uA7S1ZA8G+Y9Wf_2c0yA@mail.gmail.com>
 <CADiSq7cyZMu1sdRVL8zz5Ui253WY8AjdMmjaoB+HJNZ7C8nbpw@mail.gmail.com>
 <CAEfz+Txfk+p2TRHeCQ-PJpnUrSnyV1MLo-6JWJhsoe1KPEEA0Q@mail.gmail.com>
 <CAMpsgwZdQ8q-d5SsYmEr7wRGjXnErEz7++UX8O8eV5-5zVwC1g@mail.gmail.com>
 <22526.26340.634258.888870@turnbull.sk.tsukuba.ac.jp>
 <CADiSq7firs2H8C1UDHiC-XfjaLCpOF_Re-_1okj-tn2FBXMFww@mail.gmail.com>
 <CADiSq7fuqBNxRw5wqbjhO+WpJ+6TyLyhBrphM2etEP_7z-3KzA@mail.gmail.com>
 <-4662978122166531554@unknownmsgid>
 <CADiSq7dTX3KLVnqcovQ4aB7A3iTZOS9+Lb1HUseAXdKtGmfLFw@mail.gmail.com>
 <CALGmxEJkfZ1pAhWYoO+XixCsVQ6LNyLv2MU3MvdVGyBmqYtUhQ@mail.gmail.com>
 <CADiSq7cjfSeuaSKxb_Jd69UEFU8CL3xx8EnqTK3BVzsKHhRrXw@mail.gmail.com>
 <CALGmxE+V0kcXVE0nLiCR=ypG-bJ2Zr7Y63jNLjJT52nVO+2OCQ@mail.gmail.com>
 <CADiSq7dLE49QUbiUCnEXgsqKeoEnZGBpxGELO+KC_nNW080Q3A@mail.gmail.com>
Message-ID: <CALGmxEJeVrrdsvPiJE6K0J+0csYLS8cwNcOzAxVgmTD-nrKZ4g@mail.gmail.com>

On Tue, Oct 25, 2016 at 12:25 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> The suggestion came from folks working on asyncio performance
> improvements,




>  I'm *absolutely* suggesting
> that they put some paid time and energy into a lower level buffer
> manipulation library between now and the Python 3.7 feature freeze in
> 12+ months, as I think that will have longer term pay-offs well beyond
> the scope of the original use cases :)
>

Sounds good -- let's hope something comes of that.

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161025/c21524ed/attachment.html>

From eric at trueblade.com  Wed Oct 26 19:44:51 2016
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 26 Oct 2016 19:44:51 -0400
Subject: [Python-Dev] Default formatting
In-Reply-To: <nun94q$64o$1@blaine.gmane.org>
References: <nun94q$64o$1@blaine.gmane.org>
Message-ID: <c8a6a48c-1560-45e0-6dc8-3fe82cb688b3@trueblade.com>

On 10/25/2016 5:37 AM, Serhiy Storchaka wrote:
> Classes that doesn't define the __format__ method for custom PEP 3101
> formatting inherits it from parents.
>
> Originally the object.__format__ method was designed as [1]:
>
>     def __format__(self, format_spec):
>         return format(str(self), format_spec)
>
> An instance is converted to string and resulting string is formatted
> according to format specifier.
>
> Later this design was reconsidered [2], and now object.__format__ is
> equivalent to:
>
>     def __format__(self, format_spec):
>         assert format_spec == ''
>         return format(str(self), '')
>
> Non-empty format specifier is rejected.
>
> But why call format() on resulting string? Why not return resulting
> string as is? object.__format__ could be simpler (not just
> implementation, but for understanding):
>
>     def __format__(self, format_spec):
>         assert format_spec == ''
>         return str(self)
>
> This can change the behaviour in corner case. str(self) can return not
> exact string, but string subclass with overloaded __format__. But I
> think we can ignore such subtle difference.
>
> [1] https://www.python.org/dev/peps/pep-3101/
> [2] http://bugs.python.org/issue7994
>

I don't feel strongly about this, one way or the other. As you say, it 
would take an unusual case to see this behavior:

 >>> class S(str):
...   def __format__(self, fmt):
...     return 'aha!'
...
 >>> class O:
...   def __str__(self):
...     return S()
...
 >>> format(O(), '')
'aha!'
 >>>

But on the other hand, the existing behavior is well specified and has 
been around since object.__format__ was added. I'm not sure it needs 
changing. What's the harm in leaving it?

Eric.




From storchaka at gmail.com  Thu Oct 27 05:51:43 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 27 Oct 2016 12:51:43 +0300
Subject: [Python-Dev] Default formatting
In-Reply-To: <c8a6a48c-1560-45e0-6dc8-3fe82cb688b3@trueblade.com>
References: <nun94q$64o$1@blaine.gmane.org>
 <c8a6a48c-1560-45e0-6dc8-3fe82cb688b3@trueblade.com>
Message-ID: <nusind$dca$1@blaine.gmane.org>

On 27.10.16 02:44, Eric V. Smith wrote:
> But on the other hand, the existing behavior is well specified and has
> been around since object.__format__ was added. I'm not sure it needs
> changing. What's the harm in leaving it?

More complicated code. And maybe this behavior is less intuitive. It 
contradicts the documentation.

 From the documentation of the format() builtin [1]:

"The default format_spec is an empty string which usually gives the same 
effect as calling str(value)."

 From the description of the format specification mini-language [2]:

"A general convention is that an empty format string ("") produces the 
same result as if you had called str() on the value."

[1] https://docs.python.org/3/library/functions.html#format
[2] https://docs.python.org/3/library/stdtypes.html#str.format


From steve.dower at python.org  Thu Oct 27 12:59:39 2016
From: steve.dower at python.org (Steve Dower)
Date: Thu, 27 Oct 2016 09:59:39 -0700
Subject: [Python-Dev] Default formatting
In-Reply-To: <nusind$dca$1@blaine.gmane.org>
References: <nun94q$64o$1@blaine.gmane.org>
 <c8a6a48c-1560-45e0-6dc8-3fe82cb688b3@trueblade.com>
 <nusind$dca$1@blaine.gmane.org>
Message-ID: <03324fef-4026-da9e-c4e9-557e236fe20f@python.org>

On 27Oct2016 0251, Serhiy Storchaka wrote:
> On 27.10.16 02:44, Eric V. Smith wrote:
>> But on the other hand, the existing behavior is well specified and has
>> been around since object.__format__ was added. I'm not sure it needs
>> changing. What's the harm in leaving it?
>
> More complicated code. And maybe this behavior is less intuitive. It
> contradicts the documentation.
>
> From the documentation of the format() builtin [1]:
>
> "The default format_spec is an empty string which usually gives the same
> effect as calling str(value)."
>
> From the description of the format specification mini-language [2]:
>
> "A general convention is that an empty format string ("") produces the
> same result as if you had called str() on the value."
>
> [1] https://docs.python.org/3/library/functions.html#format
> [2] https://docs.python.org/3/library/stdtypes.html#str.format
>

The only point where this bothers me is that alignments don't work:

 >>>  class F: pass
...
 >>>  '{}'.format(F())
'<__main__.F object at 0x000002148AFE6B70>'
 >>>  '{:100}'.format(F())
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__
 >>>  '{:<100}'.format(F())
Traceback (most recent call last):
   File "<stdin>", line 1, in <module>
TypeError: non-empty format string passed to object.__format__

You need to explicitly include '!s' to be able to align it, which then 
means overriding __format__ in that class later won't have any effect.

 >>>  '{!s:<100}'.format(F())
'<__main__.F object at 0x000002148AFEE240> 
                             '

Having the default __format__ behave like this makes me happiest:

...     def __format__(self, fmt):
...         return format(str(self), fmt)

My 2c. YMMV. etc.

Cheers,
Steve

From storchaka at gmail.com  Thu Oct 27 15:01:14 2016
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 27 Oct 2016 22:01:14 +0300
Subject: [Python-Dev] Default formatting
In-Reply-To: <03324fef-4026-da9e-c4e9-557e236fe20f@python.org>
References: <nun94q$64o$1@blaine.gmane.org>
 <c8a6a48c-1560-45e0-6dc8-3fe82cb688b3@trueblade.com>
 <nusind$dca$1@blaine.gmane.org>
 <03324fef-4026-da9e-c4e9-557e236fe20f@python.org>
Message-ID: <nutitp$l6f$1@blaine.gmane.org>

On 27.10.16 19:59, Steve Dower wrote:
> Having the default __format__ behave like this makes me happiest:
>
> ....     def __format__(self, fmt):
> ....         return format(str(self), fmt)
>
> My 2c. YMMV. etc.

See issue7994 [1] for arguments against this.

[1] http://bugs.python.org/issue7994



From status at bugs.python.org  Fri Oct 28 12:08:53 2016
From: status at bugs.python.org (Python tracker)
Date: Fri, 28 Oct 2016 18:08:53 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20161028160853.B215456638@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2016-10-21 - 2016-10-28)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5525 ( +1)
  closed 34779 (+51)
  total  40304 (+52)

Open issues with patches: 2398 


Issues opened (36)
==================

#5830: heapq item comparison problematic with sched's events
http://bugs.python.org/issue5830  reopened by serhiy.storchaka

#5996: abstract class instantiable when subclassing dict
http://bugs.python.org/issue5996  reopened by gvanrossum

#25002: Deprecate asyncore/asynchat
http://bugs.python.org/issue25002  reopened by serhiy.storchaka

#25152: venv documentation doesn't tell you how to specify a particula
http://bugs.python.org/issue25152  reopened by serhiy.storchaka

#28498: tk busy command
http://bugs.python.org/issue28498  opened by tkinter

#28499: Logging module documentation needs a rework.
http://bugs.python.org/issue28499  opened by Pierre Bousquie

#28501: [Patch] Make os.umask() optional
http://bugs.python.org/issue28501  opened by EdSchouten

#28502: [Patch] Make os.chdir() optional
http://bugs.python.org/issue28502  opened by EdSchouten

#28503: [Patch] '_crypt' module: fix implicit declaration of crypt(), 
http://bugs.python.org/issue28503  opened by EdSchouten

#28506: Multiprocessing Pool starmap - struct.error: 'i' format requir
http://bugs.python.org/issue28506  opened by Justin Ting

#28512: PyErr_SyntaxLocationEx() and PyErr_SyntaxLocationObject() alwa
http://bugs.python.org/issue28512  opened by serhiy.storchaka

#28513: Document zipfile CLI
http://bugs.python.org/issue28513  opened by serhiy.storchaka

#28516: contextlib.ExitStack.__enter__ has trivial but undocumented be
http://bugs.python.org/issue28516  opened by walker.hale.iv

#28518: execute("begin immediate") throwing OperationalError
http://bugs.python.org/issue28518  opened by fschulze

#28519: Update pydoc tool to support generic types
http://bugs.python.org/issue28519  opened by levkivskyi

#28521: _PyEval_RequestCodeExtraIndex should return a globally valid i
http://bugs.python.org/issue28521  opened by jpe

#28522: can't make IDLEX work with python._pth and python-3.6.0b2
http://bugs.python.org/issue28522  opened by Big Stone

#28523: Idlelib.configdialog: use 'color' insteadof 'colour'
http://bugs.python.org/issue28523  opened by terry.reedy

#28524: Set default argument of logging.disable() to logging.CRITICAL
http://bugs.python.org/issue28524  opened by Al.Sweigart

#28528: Pdb.checkline()
http://bugs.python.org/issue28528  opened by takluyver

#28530: Howto detect if an object is of type os.DirEntry
http://bugs.python.org/issue28530  opened by stephan

#28531: Improve utf7 encoder memory usage
http://bugs.python.org/issue28531  opened by xiang.zhang

#28532: Show sys.version when -V option is supplied twice.
http://bugs.python.org/issue28532  opened by inada.naoki

#28533: Replace asyncore
http://bugs.python.org/issue28533  opened by Mariatta

#28534: Replace asynchat
http://bugs.python.org/issue28534  opened by Mariatta

#28536: Show the qualified name when a call fails
http://bugs.python.org/issue28536  opened by Ryan.Gonzalez

#28538: _socket module cross-compilation error on android-24
http://bugs.python.org/issue28538  opened by xdegaye

#28539: httplib/http.client HTTPConnection._set_hostport() regression
http://bugs.python.org/issue28539  opened by cfs-pure

#28540: math.degrees(sys.float_info.max) should throw an OverflowError
http://bugs.python.org/issue28540  opened by franciscouzo

#28541: Improve test coverage for json library - loading bytes
http://bugs.python.org/issue28541  opened by Eric Appelt

#28542: document cross compilation
http://bugs.python.org/issue28542  opened by xdegaye

#28543: Incomplete fast path codecs aliases in codecs doc
http://bugs.python.org/issue28543  opened by xiang.zhang

#28544: Implement asyncio.Task in C
http://bugs.python.org/issue28544  opened by yselivanov

#28546: Better explain setting pdb breakpoints
http://bugs.python.org/issue28546  opened by iank

#28547: Python to use Windows Certificate Store
http://bugs.python.org/issue28547  opened by Jean-Philippe Landry

#28548: http.server parse_request() bug and error reporting
http://bugs.python.org/issue28548  opened by barry



Most recent 15 issues with no replies (15)
==========================================

#28548: http.server parse_request() bug and error reporting
http://bugs.python.org/issue28548

#28546: Better explain setting pdb breakpoints
http://bugs.python.org/issue28546

#28543: Incomplete fast path codecs aliases in codecs doc
http://bugs.python.org/issue28543

#28542: document cross compilation
http://bugs.python.org/issue28542

#28538: _socket module cross-compilation error on android-24
http://bugs.python.org/issue28538

#28534: Replace asynchat
http://bugs.python.org/issue28534

#28533: Replace asyncore
http://bugs.python.org/issue28533

#28532: Show sys.version when -V option is supplied twice.
http://bugs.python.org/issue28532

#28528: Pdb.checkline()
http://bugs.python.org/issue28528

#28521: _PyEval_RequestCodeExtraIndex should return a globally valid i
http://bugs.python.org/issue28521

#28519: Update pydoc tool to support generic types
http://bugs.python.org/issue28519

#28512: PyErr_SyntaxLocationEx() and PyErr_SyntaxLocationObject() alwa
http://bugs.python.org/issue28512

#28503: [Patch] '_crypt' module: fix implicit declaration of crypt(), 
http://bugs.python.org/issue28503

#28502: [Patch] Make os.chdir() optional
http://bugs.python.org/issue28502

#28501: [Patch] Make os.umask() optional
http://bugs.python.org/issue28501



Most recent 15 issues waiting for review (15)
=============================================

#28546: Better explain setting pdb breakpoints
http://bugs.python.org/issue28546

#28544: Implement asyncio.Task in C
http://bugs.python.org/issue28544

#28543: Incomplete fast path codecs aliases in codecs doc
http://bugs.python.org/issue28543

#28542: document cross compilation
http://bugs.python.org/issue28542

#28541: Improve test coverage for json library - loading bytes
http://bugs.python.org/issue28541

#28540: math.degrees(sys.float_info.max) should throw an OverflowError
http://bugs.python.org/issue28540

#28539: httplib/http.client HTTPConnection._set_hostport() regression
http://bugs.python.org/issue28539

#28536: Show the qualified name when a call fails
http://bugs.python.org/issue28536

#28532: Show sys.version when -V option is supplied twice.
http://bugs.python.org/issue28532

#28531: Improve utf7 encoder memory usage
http://bugs.python.org/issue28531

#28528: Pdb.checkline()
http://bugs.python.org/issue28528

#28524: Set default argument of logging.disable() to logging.CRITICAL
http://bugs.python.org/issue28524

#28523: Idlelib.configdialog: use 'color' insteadof 'colour'
http://bugs.python.org/issue28523

#28516: contextlib.ExitStack.__enter__ has trivial but undocumented be
http://bugs.python.org/issue28516

#28513: Document zipfile CLI
http://bugs.python.org/issue28513



Top 10 most discussed issues (10)
=================================

#28498: tk busy command
http://bugs.python.org/issue28498  36 msgs

#28522: can't make IDLEX work with python._pth and python-3.6.0b2
http://bugs.python.org/issue28522  18 msgs

#25002: Deprecate asyncore/asynchat
http://bugs.python.org/issue25002  16 msgs

#23262: webbrowser module broken with Firefox 36+
http://bugs.python.org/issue23262  14 msgs

#28544: Implement asyncio.Task in C
http://bugs.python.org/issue28544  12 msgs

#28539: httplib/http.client HTTPConnection._set_hostport() regression
http://bugs.python.org/issue28539   9 msgs

#28199: Compact dict resizing is doing too much work
http://bugs.python.org/issue28199   7 msgs

#28524: Set default argument of logging.disable() to logging.CRITICAL
http://bugs.python.org/issue28524   7 msgs

#27275: KeyError thrown by optimised collections.OrderedDict.popitem()
http://bugs.python.org/issue27275   6 msgs

#5996: abstract class instantiable when subclassing dict
http://bugs.python.org/issue5996   5 msgs



Issues closed (53)
==================

#3015: tkinter with wantobjects=False has been broken for some time
http://bugs.python.org/issue3015  closed by serhiy.storchaka

#17711: Persistent id in pickle with protocol version 0
http://bugs.python.org/issue17711  closed by serhiy.storchaka

#20357: Mention buildbots in the core dev section of the devguide
http://bugs.python.org/issue20357  closed by berker.peksag

#20491: textwrap: Non-breaking space not honored
http://bugs.python.org/issue20491  closed by serhiy.storchaka

#22757: TclStackFree: incorrect freePtr. Call out of sequence?
http://bugs.python.org/issue22757  closed by serhiy.storchaka

#22949: fnmatch.translate doesn't add ^ at the beginning
http://bugs.python.org/issue22949  closed by serhiy.storchaka

#23214: BufferedReader.read1(size) signature incompatible with Buffere
http://bugs.python.org/issue23214  closed by martin.panter

#25464: Tix HList header_exists should be "exist"
http://bugs.python.org/issue25464  closed by serhiy.storchaka

#25953: re fails to identify invalid numeric group references in repla
http://bugs.python.org/issue25953  closed by serhiy.storchaka

#26240: Docstring of the subprocess module should be cleaned up
http://bugs.python.org/issue26240  closed by martin.panter

#26682: Ttk Notebook tabs do not show with 1-2 char names
http://bugs.python.org/issue26682  closed by ned.deily

#26796: BaseEventLoop.run_in_executor shouldn't specify max_workers fo
http://bugs.python.org/issue26796  closed by yselivanov

#26923: asyncio.gather drops cancellation
http://bugs.python.org/issue26923  closed by yselivanov

#27025: More human readable generated widget names
http://bugs.python.org/issue27025  closed by serhiy.storchaka

#27989: incomplete signature with help function using typing
http://bugs.python.org/issue27989  closed by gvanrossum

#28107: Update typing module documentation for NamedTuple
http://bugs.python.org/issue28107  closed by gvanrossum

#28213: asyncio SSLProtocol _app_transport is private
http://bugs.python.org/issue28213  closed by yselivanov

#28255: TextCalendar.prweek/month/year outputs an extra whitespace cha
http://bugs.python.org/issue28255  closed by serhiy.storchaka

#28314: ElementTree: Element.getiterator(tag) broken in 3.6
http://bugs.python.org/issue28314  closed by serhiy.storchaka

#28330: Use simpler and faster sched.Event definition
http://bugs.python.org/issue28330  closed by rhettinger

#28333: input() with Unicode prompt produces mojibake on Windows
http://bugs.python.org/issue28333  closed by steve.dower

#28353: os.fwalk() unhandled exception when error occurs accessing sym
http://bugs.python.org/issue28353  closed by serhiy.storchaka

#28396: Remove *.pyo references from man page
http://bugs.python.org/issue28396  closed by brett.cannon

#28408: Fix redundant code and memory leak in _PyUnicodeWriter_Finish
http://bugs.python.org/issue28408  closed by serhiy.storchaka

#28426: PyUnicode_AsDecodedObject can only return unicode now
http://bugs.python.org/issue28426  closed by serhiy.storchaka

#28430: asyncio: C implemeted Future cause Tornado test fail
http://bugs.python.org/issue28430  closed by inada.naoki

#28435: test_urllib2_localnet.ProxyAuthTests fails with no_proxy and N
http://bugs.python.org/issue28435  closed by martin.panter

#28439: Remove redundant checks in PyUnicode_EncodeLocale and PyUnicod
http://bugs.python.org/issue28439  closed by serhiy.storchaka

#28465: python 3.5 magic number
http://bugs.python.org/issue28465  closed by brett.cannon

#28469: timeit: use powers of 2 in autorange(), instead of powers of 1
http://bugs.python.org/issue28469  closed by serhiy.storchaka

#28482: test_typing fails if asyncio unavailable
http://bugs.python.org/issue28482  closed by gvanrossum

#28488: shutil.make_archive (xxx, zip, root_dir) is adding './' entry 
http://bugs.python.org/issue28488  closed by serhiy.storchaka

#28496: Mark up constants 0, 1, -1 in C API docs
http://bugs.python.org/issue28496  closed by serhiy.storchaka

#28497: future in tkinter
http://bugs.python.org/issue28497  closed by serhiy.storchaka

#28500: pep 525/asyncio: Handle when async generators are GCed from an
http://bugs.python.org/issue28500  closed by yselivanov

#28504: Cleanup unicode_decode_call_errorhandler_wchar/writer
http://bugs.python.org/issue28504  closed by serhiy.storchaka

#28505: pip installation issues with default path on Windows
http://bugs.python.org/issue28505  closed by eryksun

#28507: Regenerate ./configure on the default branch
http://bugs.python.org/issue28507  closed by ned.deily

#28508: Need way to expose incremental size of key sharing dicts
http://bugs.python.org/issue28508  closed by rhettinger

#28509: dict.update allocates too much
http://bugs.python.org/issue28509  closed by inada.naoki

#28510: PyUnicodeDecodeError_GetObject always return bytes
http://bugs.python.org/issue28510  closed by serhiy.storchaka

#28511: Use the "U" format for parsing Unicode object arg in PyArg_Par
http://bugs.python.org/issue28511  closed by xiang.zhang

#28514: Python (IDLE?) freezes on file save on Windows
http://bugs.python.org/issue28514  closed by SilentGhost

#28515: Py3k warnings in Python 2.7 tests
http://bugs.python.org/issue28515  closed by serhiy.storchaka

#28517: Dead code in wordcode
http://bugs.python.org/issue28517  closed by serhiy.storchaka

#28520: Failed to install Python 3.3.5 on OSX 10.11.6
http://bugs.python.org/issue28520  closed by ned.deily

#28525: Incorrect documented parameter for gc.collect
http://bugs.python.org/issue28525  closed by python-dev

#28526: Use PyUnicode_AsEncodedString instead of PyUnicode_AsEncodedOb
http://bugs.python.org/issue28526  closed by serhiy.storchaka

#28527: Hack in `genericpath.commonprefix()` crashes if `m` argument i
http://bugs.python.org/issue28527  closed by christian.heimes

#28529: 0 ** 0 should raise ArithmeticError
http://bugs.python.org/issue28529  closed by tim.peters

#28535: round seems to provide floor, not proper rounding
http://bugs.python.org/issue28535  closed by r.david.murray

#28537: abc module fails to reject instantiation of some multiply-inhe
http://bugs.python.org/issue28537  closed by r.david.murray

#28545: socket.bind to AF_PACKET should use passed interface name
http://bugs.python.org/issue28545  closed by mturon

From mes at zeroc.com  Fri Oct 28 12:51:07 2016
From: mes at zeroc.com (Mark Spruiell)
Date: Fri, 28 Oct 2016 09:51:07 -0700
Subject: [Python-Dev] Clarification for concurrent.futures.Future
Message-ID: <CABOZqnjvMtzuFTauZ-+25wHLBfH45vZMtx+Qs8VX94FuBjH8-Q@mail.gmail.com>

The docs for this class state:

"Future instances are created by Executor.submit() and should not be
created directly except for testing."

https://docs.python.org/3/library/concurrent.futures.html#future-objects

We have a need for a thread-safe future type in our extension but this
statement makes us hesitate to use it. We don't need the executor
functionality.

We can write our own future class easily enough, we're just wondering what
the justification was for the limitations mentioned in the docs.

Thanks,

Mark Spruiell
ZeroC, Inc.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161028/0f31a137/attachment.html>

From guido at python.org  Fri Oct 28 13:20:13 2016
From: guido at python.org (Guido van Rossum)
Date: Fri, 28 Oct 2016 10:20:13 -0700
Subject: [Python-Dev] Clarification for concurrent.futures.Future
In-Reply-To: <CABOZqnjvMtzuFTauZ-+25wHLBfH45vZMtx+Qs8VX94FuBjH8-Q@mail.gmail.com>
References: <CABOZqnjvMtzuFTauZ-+25wHLBfH45vZMtx+Qs8VX94FuBjH8-Q@mail.gmail.com>
Message-ID: <CAP7+vJKxB1aZ17X3Trp9U0ev5jnb+MVsoJ-oRnNKgzfKSr=HSQ@mail.gmail.com>

It means that the API for setting results/exceptions is essentially
private and may change. So if you use this you risk that your code
will break randomly at some point in the future when you upgrade
Python.

From nad at python.org  Fri Oct 28 20:36:30 2016
From: nad at python.org (Ned Deily)
Date: Fri, 28 Oct 2016 20:36:30 -0400
Subject: [Python-Dev] Reminder: 3.6.0b3 snapshot 2016-10-31 12:00 UTC
Message-ID: <4E2573E2-0ACD-4876-96C4-ADC5B5DEE08D@python.org>

The third, and next-to-last, beta snapshot planned for the 3.6 release cycle is coming up in a few days.  With fewer than 7 weeks remaining until the 3.6.0 release, it is very important that we all focus on stability and correctness.  Please try to make sure that all remaining non-doc issues associated with new features are addressed in b3.  Any remaining 3.6 feature-related issues still open after b3 should be marked in the issue tracker as "release blocker" and *must* be addressed one way or another by b4, 11-21.  All other non-critical bug fixes should also be checked in by b4.  Only release critical and doc fixes will be allowed once we exit the beta phase.  Please plan accordingly.

Please contact me if you have any questions about the 3.6.0 schedule or about whether a change is appropriate at this point in the 3.6.0 cycle.

To recap, the remaining milestones for 3.6.0:

2016-10-31, 1200 UTC: 3.6.0 beta 3 (feature fixes, bug fixes, doc fixes)

2016-11-21: 3.6.0 beta 4 (important bug fixes and doc fixes)

2016-12-05 3.6.0 release candidate 1 (3.6.0 code freeze, release critical bug fixes, doc fixes)

2016-12-16 3.6.0 release (3.6.0rc1 plus any necessary emergency fixes)


Thank you all again for your efforts so far on 3.6!

--Ned

https://www.python.org/dev/peps/pep-0494/

--
  Ned Deily
  nad at python.org -- []


From songofacandy at gmail.com  Sun Oct 30 01:27:00 2016
From: songofacandy at gmail.com (INADA Naoki)
Date: Sun, 30 Oct 2016 14:27:00 +0900
Subject: [Python-Dev] [Python-ideas] Show more info when `python -vV`
In-Reply-To: <nu39u2$9ri$1@blaine.gmane.org>
References: <CAEfz+TxOSReGuYXrPMpFKPwxF6gd5+-Q9j-OP114QETtO5hV1Q@mail.gmail.com>
 <ntsm57$p2d$1@blaine.gmane.org>
 <CAEfz+TzN=0xqf8zd1Avxxebcrh4rEJ8J9A+EVb2BV9VvzywbsQ@mail.gmail.com>
 <CADiSq7e0X7+774Zisq-gjtuvEGdYOJX8NR1pRx7gcTu_ZCikow@mail.gmail.com>
 <CAPTjJmqc61PNv_M5v1D8-C5diGTHHuBzqkfTf6uO8XcRVKmgvg@mail.gmail.com>
 <CADiSq7dQqVy-bfdwXPVg-Jt9TfRjnPCZTnbKmGFKYaXQSiJ0_g@mail.gmail.com>
 <CAPTjJmp3kanamX=6Q1jYugHb3F6C6ujmCjWfs4TEUOB4yRJ1xQ@mail.gmail.com>
 <CAEfz+TzSh+o2-Q1sH0kKVG=w0L8uEOzkwM8MzL11aBNw2R+w3Q@mail.gmail.com>
 <CAPTjJmoi9YpZ=J5TBpt0=Zg4ANVxfHQ8JWP9rFntkD+pDTWtmQ@mail.gmail.com>
 <CAEfz+TwQrmY9f5P32AXpBy4c5D2fdQf8JuKJpBbVXC6gYyBAgw@mail.gmail.com>
 <CADiSq7eMXsCKv3HpGvq-NmCR=5b6yQdZtwYiKVaPAw+SFE1_kw@mail.gmail.com>
 <20161017150545.052006b5@subdivisions.wooz.org>
 <nu39u2$9ri$1@blaine.gmane.org>
Message-ID: <CAEfz+TzbcEkQOYvENKc-LmDyCe-xVi2y+V6Hut=2gUYzWyce_g@mail.gmail.com>

>
> I agree with Barry.  I'm open to adding this feature to 3.6.0b3 but
> first we need an issue and a final patch to review.
>

Here is the issue and patch.  Could someone review it?
http://bugs.python.org/issue28532

-- 
INADA Naoki  <songofacandy at gmail.com>

From skip.montanaro at gmail.com  Mon Oct 31 15:20:00 2016
From: skip.montanaro at gmail.com (Skip Montanaro)
Date: Mon, 31 Oct 2016 14:20:00 -0500
Subject: [Python-Dev] 2.6 branch in Mercurial?
Message-ID: <CANc-5UwVmjtGxXZA2LJSGqkVffNEPL8U=hVz=XDTobX5qbVe8w@mail.gmail.com>

I need to do a little 2.6 spelunking. I don't see a 2.6 branch in the
output of "hg branches". Is "hg clone v2.6.9" the proper incantation to get
the latest version (or perhaps "v2.6")?

Thx,

Skip
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20161031/2a17dcab/attachment.html>

From nad at python.org  Mon Oct 31 15:25:25 2016
From: nad at python.org (Ned Deily)
Date: Mon, 31 Oct 2016 15:25:25 -0400
Subject: [Python-Dev] 2.6 branch in Mercurial?
In-Reply-To: <CANc-5UwVmjtGxXZA2LJSGqkVffNEPL8U=hVz=XDTobX5qbVe8w@mail.gmail.com>
References: <CANc-5UwVmjtGxXZA2LJSGqkVffNEPL8U=hVz=XDTobX5qbVe8w@mail.gmail.com>
Message-ID: <CCE44C6C-1F1A-4E32-872A-AA4C46167F52@python.org>

On Oct 31, 2016, at 15:20, Skip Montanaro <skip.montanaro at gmail.com> wrote:
> I need to do a little 2.6 spelunking. I don't see a 2.6 branch in the output of "hg branches". Is "hg clone v2.6.9" the proper incantation to get the latest version (or perhaps "v2.6")?

hg update -C v2.6.9

--
  Ned Deily
  nad at python.org -- []


From nad at python.org  Mon Oct 31 15:34:32 2016
From: nad at python.org (Ned Deily)
Date: Mon, 31 Oct 2016 15:34:32 -0400
Subject: [Python-Dev] 2.6 branch in Mercurial?
In-Reply-To: <CCE44C6C-1F1A-4E32-872A-AA4C46167F52@python.org>
References: <CANc-5UwVmjtGxXZA2LJSGqkVffNEPL8U=hVz=XDTobX5qbVe8w@mail.gmail.com>
 <CCE44C6C-1F1A-4E32-872A-AA4C46167F52@python.org>
Message-ID: <C4E20221-CF92-4353-AAEF-A6114F7EB8CF@python.org>

On Oct 31, 2016, at 15:25, Ned Deily <nad at python.org> wrote:
> On Oct 31, 2016, at 15:20, Skip Montanaro <skip.montanaro at gmail.com> wrote:
>> I need to do a little 2.6 spelunking. I don't see a 2.6 branch in the output of "hg branches". Is "hg clone v2.6.9" the proper incantation to get the latest version (or perhaps "v2.6")?
> 
> hg update -C v2.6.9

To expand a bit, that command will update to the 2.6.9 release. You don't see a 2.6 branch because it has been marked as closed.  hg branches -c will show all branches.  hg update -C 2.6 will update to the latest head of the 2.6 branch but that should essentially be the same as the tag v2.6.9.

hg help branches

--
  Ned Deily
  nad at python.org -- []


From zachary.ware+pydev at gmail.com  Mon Oct 31 15:34:34 2016
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Mon, 31 Oct 2016 14:34:34 -0500
Subject: [Python-Dev] 2.6 branch in Mercurial?
In-Reply-To: <CANc-5UwVmjtGxXZA2LJSGqkVffNEPL8U=hVz=XDTobX5qbVe8w@mail.gmail.com>
References: <CANc-5UwVmjtGxXZA2LJSGqkVffNEPL8U=hVz=XDTobX5qbVe8w@mail.gmail.com>
Message-ID: <CAKJDb-Pb492fR1-Rj0RMj2mdJvnM1cTmYyQSc=DC+MvNWQ=kPQ@mail.gmail.com>

On Mon, Oct 31, 2016 at 2:20 PM, Skip Montanaro
<skip.montanaro at gmail.com> wrote:
> I need to do a little 2.6 spelunking. I don't see a 2.6 branch in the output
> of "hg branches". Is "hg clone v2.6.9" the proper incantation to get the
> latest version (or perhaps "v2.6")?

2.6 is a closed branch, which pretty much only means it doesn't show
up in `hg branches`.  Use `hg branches -c` to see it (beware that if
you use the 'color' extension, the branch name may still be invisible
due to coloring).

-- 
Zach