From ncoghlan at gmail.com  Mon May  1 02:37:29 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 1 May 2017 16:37:29 +1000
Subject: [Python-Dev] Escaping docs markup in NEWS entries?
Message-ID: <CADiSq7c5GuOMTCJKUrTx=jhaYn7VTho_qDT3FYhOYbD70q2F9Q@mail.gmail.com>

Hi folks,

I'm trying to write a NEWS entry that explains that the
":func:`bytes`" cross-references have changed to refer to the type
descriptions by default (matching other builtin container types), so
you now need to use ``:ref:`func-bytes`" to refer to the old target in
the list of builtin functions (if you really want that for some
reason).

Unfortunately, my first two attempts both cause warnings in "make
suspicious" with the following output:

------------
WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytes\`"
WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytes\`"
WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytearray\`"
WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytearray\`"
WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`"
WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`"
WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`"
WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`"
------------

My first attempt just escaped the nested backticks:

- Issue #30052: the link targets for ``:func:\`bytes\``` and
  ``:func:\`bytearray\``` are now their respective type definitions, rather
  than the corresponding builtin function entries. Use ``:ref:\`func-bytes\```
  and ``:ref:\`func-bytes\``` to reference the latter.

My second attempt escaped the colons as well:

- Issue #30052: the link targets for ``\:func\:\`bytes\``` and
  ``\:func\:\`bytearray\``` are now their respective type definitions, rather
  than the corresponding builtin function entries. Use ``\:ref\:\`func-bytes\```
  and ``\:ref\:\`func-bytes\``` to reference the latter.

My fallback plan is to just include the unescaped markup in the NEWS
entry (rather than trying to make it readable even in rendered form),
but I figured I'd ask for advice here first.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From vadmium+py at gmail.com  Mon May  1 03:13:25 2017
From: vadmium+py at gmail.com (Martin Panter)
Date: Mon, 1 May 2017 07:13:25 +0000
Subject: [Python-Dev] Escaping docs markup in NEWS entries?
In-Reply-To: <CADiSq7c5GuOMTCJKUrTx=jhaYn7VTho_qDT3FYhOYbD70q2F9Q@mail.gmail.com>
References: <CADiSq7c5GuOMTCJKUrTx=jhaYn7VTho_qDT3FYhOYbD70q2F9Q@mail.gmail.com>
Message-ID: <CA+eR4cHSWGpPgYL_eU1G69pfi1h5fjnQqVJfHGjCbzY6ACfE-Q@mail.gmail.com>

On 1 May 2017 at 06:37, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Hi folks,
>
> I'm trying to write a NEWS entry that explains that the
> ":func:`bytes`" cross-references have changed to refer to the type
> descriptions by default (matching other builtin container types), so
> you now need to use ``:ref:`func-bytes`" to refer to the old target in
> the list of builtin functions (if you really want that for some
> reason).
>
> Unfortunately, my first two attempts both cause warnings in "make
> suspicious" with the following output:

What is the full output? Usually it includes instructions to add false
positives to Doc/tools/susp-ignored.csv; maybe that is all you have to
do?

> ------------
> WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytes\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytes\`"
> WARNING: [whatsnew/changelog:986] ":func" found in "\:func\:\`bytearray\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:func\:\`bytearray\`"
> WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`"
> WARNING: [whatsnew/changelog:986] ":ref" found in "\:ref\:\`func-bytes\`"
> WARNING: [whatsnew/changelog:986] "`" found in "\:ref\:\`func-bytes\`"
> ------------
>
> My first attempt just escaped the nested backticks:
>
> - Issue #30052: the link targets for ``:func:\`bytes\``` and
>   ``:func:\`bytearray\``` are now their respective type definitions, rather
>   than the corresponding builtin function entries. Use ``:ref:\`func-bytes\```
>   and ``:ref:\`func-bytes\``` to reference the latter.
>
> My second attempt escaped the colons as well:
>
> - Issue #30052: the link targets for ``\:func\:\`bytes\``` and
>   ``\:func\:\`bytearray\``` are now their respective type definitions, rather
>   than the corresponding builtin function entries. Use ``\:ref\:\`func-bytes\```
>   and ``\:ref\:\`func-bytes\``` to reference the latter.
>
> My fallback plan is to just include the unescaped markup in the NEWS
> entry (rather than trying to make it readable even in rendered form),
> but I figured I'd ask for advice here first.

I thought the NEWS file was mainly regarded as plain text, so it would
be important to avoid ugly RST markup like backslashes if possible.

I am no RST expert, but perhaps it would be clearer to a human RST
parser if you added a space among the last three backticks, where the
underscore is in ``:func:`bytes`_``.

From ncoghlan at gmail.com  Mon May  1 03:24:24 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 1 May 2017 17:24:24 +1000
Subject: [Python-Dev] Escaping docs markup in NEWS entries?
In-Reply-To: <CA+eR4cHSWGpPgYL_eU1G69pfi1h5fjnQqVJfHGjCbzY6ACfE-Q@mail.gmail.com>
References: <CADiSq7c5GuOMTCJKUrTx=jhaYn7VTho_qDT3FYhOYbD70q2F9Q@mail.gmail.com>
 <CA+eR4cHSWGpPgYL_eU1G69pfi1h5fjnQqVJfHGjCbzY6ACfE-Q@mail.gmail.com>
Message-ID: <CADiSq7cLx6184UoJnJ2yx8otagqa-Pqpmgn+nNLpjwbLr=EvmQ@mail.gmail.com>

On 1 May 2017 at 17:13, Martin Panter <vadmium+py at gmail.com> wrote:
> On 1 May 2017 at 06:37, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Hi folks,
>>
>> I'm trying to write a NEWS entry that explains that the
>> ":func:`bytes`" cross-references have changed to refer to the type
>> descriptions by default (matching other builtin container types), so
>> you now need to use ``:ref:`func-bytes`" to refer to the old target in
>> the list of builtin functions (if you really want that for some
>> reason).
>>
>> Unfortunately, my first two attempts both cause warnings in "make
>> suspicious" with the following output:
>
> What is the full output? Usually it includes instructions to add false
> positives to Doc/tools/susp-ignored.csv; maybe that is all you have to
> do?

You're right, that would be likely be the way to go if I decided to
keep the escaped markup.

However...

>> My fallback plan is to just include the unescaped markup in the NEWS
>> entry (rather than trying to make it readable even in rendered form),
>> but I figured I'd ask for advice here first.
>
> I thought the NEWS file was mainly regarded as plain text, so it would
> be important to avoid ugly RST markup like backslashes if possible.

... I think you're right on this point, so it makes more sense to skip
the escaping entirely,
and just use the correct link markup in the NEWS entry.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Mon May  1 05:50:22 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 1 May 2017 12:50:22 +0300
Subject: [Python-Dev] Guarantee the success of some object creation C API
 functions
Message-ID: <CAAuNNr7B1p-hsbwfRMxBidNGTog4smACWBB-z9uF4wYnedZqYA@mail.gmail.com>

I want to add promises to public C API functions that create trivial
instances of immutable basic types (integers 0 and 1, empty tuple,
string and bytes object) -- PyLong_FromLong(0), PyLong_FromLong(1),
PyTuple_New(0), PyUnicode_FromStringAndSize(NULL, 0),
PyUnicode_FromString(""), PyBytes_FromStringAndSize(NULL, 0),
PyBytes_FromString("") -- that they always succeed and never return
NULL. Currently they always or almost always succeed, but this is an
implementation detail. I want to make this promise official and more
strong, so that it is true even at very early stage of the interpreter
intitalization and at very late stage of the finalization. Also I want
to add private references to singletons that represent these values
(_PyLong_Zero, _PyTuple_Empty, etc) similar to Py_None and Py_True.
This could make some code clearer. For example see [1] and [2].

These promises add limitations for future changes of the interpreter,
but the official promise decreases the probability of unintentional
changes that break implicit properties.

Do I have the right to give such promises? Are there any reasons that
might prevent their implementation?

[1] https://bugs.python.org/issue29878
[2] https://bugs.python.org/issue30162

From wes.turner at gmail.com  Mon May  1 06:03:59 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Mon, 1 May 2017 05:03:59 -0500
Subject: [Python-Dev] Escaping docs markup in NEWS entries?
In-Reply-To: <CADiSq7cLx6184UoJnJ2yx8otagqa-Pqpmgn+nNLpjwbLr=EvmQ@mail.gmail.com>
References: <CADiSq7c5GuOMTCJKUrTx=jhaYn7VTho_qDT3FYhOYbD70q2F9Q@mail.gmail.com>
 <CA+eR4cHSWGpPgYL_eU1G69pfi1h5fjnQqVJfHGjCbzY6ACfE-Q@mail.gmail.com>
 <CADiSq7cLx6184UoJnJ2yx8otagqa-Pqpmgn+nNLpjwbLr=EvmQ@mail.gmail.com>
Message-ID: <CACfEFw94Xie5ijCpizEX3sofattBUGsOQkMQpAjUyg9Am3W5Pg@mail.gmail.com>

Where would be a good place for test cases for an rst_escape() function?
Docutils?

https://github.com/westurner/dotfiles/blob/develop/scripts/git-changelog.py

- rst_escape # YMMV
- $ git-changelog.py -r "release/0.3.14" --hdr= "+"`

On Monday, May 1, 2017, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 1 May 2017 at 17:13, Martin Panter <vadmium+py at gmail.com <javascript:;>>
> wrote:
> > On 1 May 2017 at 06:37, Nick Coghlan <ncoghlan at gmail.com <javascript:;>>
> wrote:
> >> Hi folks,
> >>
> >> I'm trying to write a NEWS entry that explains that the
> >> ":func:`bytes`" cross-references have changed to refer to the type
> >> descriptions by default (matching other builtin container types), so
> >> you now need to use ``:ref:`func-bytes`" to refer to the old target in
> >> the list of builtin functions (if you really want that for some
> >> reason).
> >>
> >> Unfortunately, my first two attempts both cause warnings in "make
> >> suspicious" with the following output:
> >
> > What is the full output? Usually it includes instructions to add false
> > positives to Doc/tools/susp-ignored.csv; maybe that is all you have to
> > do?
>
> You're right, that would be likely be the way to go if I decided to
> keep the escaped markup.
>
> However...
>
> >> My fallback plan is to just include the unescaped markup in the NEWS
> >> entry (rather than trying to make it readable even in rendered form),
> >> but I figured I'd ask for advice here first.
> >
> > I thought the NEWS file was mainly regarded as plain text, so it would
> > be important to avoid ugly RST markup like backslashes if possible.
>
> ... I think you're right on this point, so it makes more sense to skip
> the escaping entirely,
> and just use the correct link markup in the NEWS entry.

How convenient.

>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com <javascript:;>   |   Brisbane,
> Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org <javascript:;>
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> wes.turner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170501/1ccd53fb/attachment.html>

From me at louie.lu  Mon May  1 11:07:18 2017
From: me at louie.lu (Louie Lu)
Date: Mon, 1 May 2017 23:07:18 +0800
Subject: [Python-Dev] Request review of cProfile/profile series issue
In-Reply-To: <CADMDV=xeANNRa3DbB0HM6wN73PjNbcK7OYbv5jCWPMV5=KOwtQ@mail.gmail.com>
References: <CADMDV=xeANNRa3DbB0HM6wN73PjNbcK7OYbv5jCWPMV5=KOwtQ@mail.gmail.com>
Message-ID: <CADMDV=ya7uR-P3Cpu3ABtP_k04L-vA7kYB238YKTSgOJqXp6sw@mail.gmail.com>

For who may be reviewing cProfile / profile context manager problem,

I prepare a post for explaining why the problem become,
and why the patch will fix this problem:

post: https://blog.louie.lu/2017/04/23/python-libs-profile-cant-context-manager/


Thanks,
Louie.

2017-04-23 11:21 GMT+08:00 Louie Lu <me at louie.lu>:
> Hi all,
>
> I'm now looking for cProfile/profile lib's issue, and have solve a series of
> dependent problem, here is the list:
>
> #9285 - Add a profile decorator to profile and cProfile
> #30113 - Allow helper functions to wrap sys.setprofile
> #18971 - Use argparse in the profile/cProfile modules
> #30118 - Add unittest for cProfile/profile command line interface
>
> It can divide into two categories, first is the context manager problem, and
> the second is optparse to argparse problem.
>
> 1. context manager problem:
>
> Relative issue: #9285, #30113
> Relative PR: #287, #1212, #1253
>
> This is an issue since 2010, and stop at profile can't simply add a
> __enter__ and __exit__ to make it a context manager. The main problem is,
> sys.setprofile() will hit the return and get bad return in profile
> dispatch_return function. The solution is to insert a simulate call in the
> helper function, to provide the context between helper frame and where the
> profile is defined.
>
> 2. optparse to argparse problem:
>
> Relative issue: #18971, #30118
> Relative PR: #1232
> Relative patch: profile_argparse.patch
>
> Serhiy have provide the patch of replace optparse to argparse in the profile
> and cProfile, but block by Ezio request of unittest for command line
> interface. The unittest is provide at #1232, and need to be reivew. If the
> unittest is add and argparse patch is apply, we can then solve more problem,
> e.g.: #23420, #29238, #21862
>
>
> This is what I've investigated for cProfile / profile library now,
> to be move on, it will need others to review the work.
>
> Thanks!
>
> Best Regards,
> Louie.

From tjreedy at udel.edu  Mon May  1 16:52:49 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 1 May 2017 16:52:49 -0400
Subject: [Python-Dev] Guarantee the success of some object creation C
 API functions
In-Reply-To: <CAAuNNr7B1p-hsbwfRMxBidNGTog4smACWBB-z9uF4wYnedZqYA@mail.gmail.com>
References: <CAAuNNr7B1p-hsbwfRMxBidNGTog4smACWBB-z9uF4wYnedZqYA@mail.gmail.com>
Message-ID: <oe876s$n9o$1@blaine.gmane.org>

On 5/1/2017 5:50 AM, Serhiy Storchaka wrote:
> I want to add promises to public C API functions that create trivial
> instances of immutable basic types (integers 0 and 1, empty tuple,
> string and bytes object) -- PyLong_FromLong(0), PyLong_FromLong(1),
> PyTuple_New(0), PyUnicode_FromStringAndSize(NULL, 0),
> PyUnicode_FromString(""), PyBytes_FromStringAndSize(NULL, 0),
> PyBytes_FromString("") -- that they always succeed and never return
> NULL. Currently they always or almost always succeed, but this is an
> implementation detail. I want to make this promise official and more
> strong, so that it is true even at very early stage of the interpreter
> intitalization and at very late stage of the finalization. Also I want
> to add private references to singletons that represent these values
> (_PyLong_Zero, _PyTuple_Empty, etc) similar to Py_None and Py_True.
> This could make some code clearer. For example see [1] and [2].
> 
> These promises add limitations for future changes of the interpreter,
> but the official promise decreases the probability of unintentional
> changes that break implicit properties.

The promise makes it clear that breaking the property is a bug to be 
fixed.  It only decreases the probability for someone who has read the 
promise.  Unfortunately, 'never fail' is hard to test ;-).

> Do I have the right to give such promises?

By yourself, I would think not.  With a couple of coredev concurrences, 
and none against, maybe.  But I think the following does need to be 
asked publicly, here, so implementors of CPython alternatives have a 
chance to respond.

 > Are there any reasons that might prevent their implementation?
If the answer is no, I think the simplifications in the following are a 
good argument for the change.  On the other hand, I have to wonder why 
there were not made previously.

> [1] https://bugs.python.org/issue29878
> [2] https://bugs.python.org/issue30162

-- 
Terry Jan Reedy


From rosuav at gmail.com  Mon May  1 17:52:26 2017
From: rosuav at gmail.com (Chris Angelico)
Date: Tue, 2 May 2017 07:52:26 +1000
Subject: [Python-Dev] Guarantee the success of some object creation C
 API functions
In-Reply-To: <oe876s$n9o$1@blaine.gmane.org>
References: <CAAuNNr7B1p-hsbwfRMxBidNGTog4smACWBB-z9uF4wYnedZqYA@mail.gmail.com>
 <oe876s$n9o$1@blaine.gmane.org>
Message-ID: <CAPTjJmopDDa+3T9U+UkjnkBgGrA2FFw3J7fPYKAphMjdS1KHXA@mail.gmail.com>

On Tue, May 2, 2017 at 6:52 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> The promise makes it clear that breaking the property is a bug to be fixed.
> It only decreases the probability for someone who has read the promise.
> Unfortunately, 'never fail' is hard to test ;-).
>

Aside from straight-up bugs, how can one of these functions fail? Is
memory allocation failure the only way? If so, the proposed
implementation (private references to pre-created singletons) ought to
guarantee that, to the exact extent that anything else can be
guaranteed.

(Or is that your point - that "never fail" is always "modulo bugs"?)

Incidentally, this guarantee, if implemented the obvious way, will
also mean that (), "", 0, etc are singletons. People talk casually
about the "empty tuple singleton", but I don't think it's actually
guaranteed anywhere.

ChrisA

From jjmaldonis at gmail.com  Mon May  1 23:47:32 2017
From: jjmaldonis at gmail.com (Jason Maldonis)
Date: Mon, 1 May 2017 22:47:32 -0500
Subject: [Python-Dev] __getattribute__'s error is not available in
 __getattr__
Message-ID: <CA+LevYJQ6bHR2BtncCZm_5i7emWUyuuvXfnY1oWNBWbRG6xh1w@mail.gmail.com>

Hi everyone,

If this should be asked in learn python I apologize -- please just tell me
without answering.

I'm working on a large class architecture and I find myself often
overloading __getattr__.  I am continuously running into the issue where I
want __getattr__ to have access to the error that was raised in
__getattribute__, but it seems completely unavailable. Is that true?

One simple case that I'm guessing others have run into, is if __getattr__
fails, the error from __getattribute__ isn't in the stack trace that gets
printed to screen.  To fix this (on occasion) I'll even re-call
__getattribute__ within __getattr__ just to get the error so I can properly
"raise from" the __getattibute__'s error -- although that's probably bad
practice in general.

I'd like to be able to access the error that was raised in __getattribute__
when __getattr__ is called.

Two more quick context comments: python is awesome, thank you all for your
hard work; and I've been writing python almost every day for ~ 5 years now
and I can do all the "black magic" jazz, so I'll be okay with an
implementation that requires that type of stuff if necessary.

Thanks!
Jason
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170501/8d7c2c12/attachment.html>

From ncoghlan at gmail.com  Tue May  2 00:18:42 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 2 May 2017 14:18:42 +1000
Subject: [Python-Dev] Guarantee the success of some object creation C
 API functions
In-Reply-To: <CAPTjJmopDDa+3T9U+UkjnkBgGrA2FFw3J7fPYKAphMjdS1KHXA@mail.gmail.com>
References: <CAAuNNr7B1p-hsbwfRMxBidNGTog4smACWBB-z9uF4wYnedZqYA@mail.gmail.com>
 <oe876s$n9o$1@blaine.gmane.org>
 <CAPTjJmopDDa+3T9U+UkjnkBgGrA2FFw3J7fPYKAphMjdS1KHXA@mail.gmail.com>
Message-ID: <CADiSq7cWsN40gPs=kbdGR5EF9VTKgdkR3FmZiW6vkyeacsoM+A@mail.gmail.com>

On 2 May 2017 at 07:52, Chris Angelico <rosuav at gmail.com> wrote:
> On Tue, May 2, 2017 at 6:52 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>> The promise makes it clear that breaking the property is a bug to be fixed.
>> It only decreases the probability for someone who has read the promise.
>> Unfortunately, 'never fail' is hard to test ;-).
>>
>
> Aside from straight-up bugs, how can one of these functions fail? Is
> memory allocation failure the only way? If so, the proposed
> implementation (private references to pre-created singletons) ought to
> guarantee that, to the exact extent that anything else can be
> guaranteed.
>
> (Or is that your point - that "never fail" is always "modulo bugs"?)
>
> Incidentally, this guarantee, if implemented the obvious way, will
> also mean that (), "", 0, etc are singletons. People talk casually
> about the "empty tuple singleton", but I don't think it's actually
> guaranteed anywhere.

I don't think it is either, and I don't think it's a guarantee we want
to make - even with Serhiy's proposed change, it's still
straightforward to produce non-singleton instances of these values
using the low level C APIs, while the true singletons (True, False,
None, Ellipsis) go out of their way to make it difficult to create new
instances of the corresponding types, even when mucking about at the C
layer.

The assertion Serhiy is proposing to make is much weaker, and would be
a matter of adding something like the following to the C API
reference:

"Similar to the singleton values True, False, None, and Ellipsis,
process global instances of the builtin values (), '', b'', 0, and 1
are pre-allocated at interpreter startup, so APIs returning these
values should never fail, even in low memory conditions. However,
additional empty instances of these types may still be created through
the C API, so they should always be compared by value rather than by
identity."

The specific "never fails as it returns a pointer to a pre-allocated
instance" cases can then be documented on the affected public API
functions.

So +1 from me for making pre-allocation a CPython C API guarantee, -1
for making it a language level singleton commitment.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue May  2 00:44:56 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 2 May 2017 14:44:56 +1000
Subject: [Python-Dev] __getattribute__'s error is not available in
 __getattr__
In-Reply-To: <CA+LevYJQ6bHR2BtncCZm_5i7emWUyuuvXfnY1oWNBWbRG6xh1w@mail.gmail.com>
References: <CA+LevYJQ6bHR2BtncCZm_5i7emWUyuuvXfnY1oWNBWbRG6xh1w@mail.gmail.com>
Message-ID: <CADiSq7cRRgt6-r_GYeokfRvqZyjNyZgyjsPkE0=FhzdYgu7hXQ@mail.gmail.com>

On 2 May 2017 at 13:47, Jason Maldonis <jjmaldonis at gmail.com> wrote:
> Hi everyone,
>
> If this should be asked in learn python I apologize -- please just tell me
> without answering.
>
> I'm working on a large class architecture and I find myself often
> overloading __getattr__.  I am continuously running into the issue where I
> want __getattr__ to have access to the error that was raised in
> __getattribute__, but it seems completely unavailable. Is that true?

__getattr__ can be called *from* __getattribute__, so when it runs,
__getattribute__ hasn't necessarily failed yet - it may just be on its
last resort strategy for attribute retrieval.

If you're sure the base class __getattribute__ doesn't call
__getattr__ directly, you can do:

    def __getattribute__(self, name):
        try:
            return super().__getattribute__(name)
        except AttributeError:
            return self.__getattr__(name)

However, would you mind filing a documentation bug for this? I can't
find anything in the language or library reference that explicitly
states whether or not `object.__getattribute__` itself calls
`__getattr__` directly, and that's a docs limitation which should be
addressed.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Tue May  2 01:49:41 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 2 May 2017 08:49:41 +0300
Subject: [Python-Dev] Guarantee the success of some object creation C
 API functions
In-Reply-To: <CAPTjJmopDDa+3T9U+UkjnkBgGrA2FFw3J7fPYKAphMjdS1KHXA@mail.gmail.com>
References: <CAAuNNr7B1p-hsbwfRMxBidNGTog4smACWBB-z9uF4wYnedZqYA@mail.gmail.com>
 <oe876s$n9o$1@blaine.gmane.org>
 <CAPTjJmopDDa+3T9U+UkjnkBgGrA2FFw3J7fPYKAphMjdS1KHXA@mail.gmail.com>
Message-ID: <oe96lg$i6s$1@blaine.gmane.org>

On 02.05.17 00:52, Chris Angelico wrote:
> Aside from straight-up bugs, how can one of these functions fail? Is
> memory allocation failure the only way?

Yes, memory allocation failure is the only way. In normal Python build 
this can happen only at early stage of the interpreter initialization, 
until the first call of the function allocate a singleton in dynamic 
memory, or, for some instances, at late stage of the interpreter 
finalization after deallocating singletons. In custom Python build with 
disabled free lists for small ints, tuples, etc (if define NSMALLPOSINTS 
or PyTuple_MAXSAVESIZE to 0) this can happen at any time.

> Incidentally, this guarantee, if implemented the obvious way, will
> also mean that (), "", 0, etc are singletons. People talk casually
> about the "empty tuple singleton", but I don't think it's actually
> guaranteed anywhere.

As noted Nick this guarantee doesn't go so far. With C API you can 
modify just created tuple in-place and set its size to 0. This gives you 
an empty tuple different from the "empty tuple singleton". Some C API 
functions actually can return non-singletons equal to singleton objects, 
but usually we try to avoid this. Not because there are any guaranties, 
but for optimization. References to singletons consume less memory and 
compare faster.



From erik.m.bray at gmail.com  Tue May  2 12:01:25 2017
From: erik.m.bray at gmail.com (Erik Bray)
Date: Tue, 2 May 2017 18:01:25 +0200
Subject: [Python-Dev] Possible bug in class-init, lookin for mentors
In-Reply-To: <CABnaYKC+AgqTzKVFb8vxOd7JUrEKGftj-ZbNJU-q7fvOv8qgYg@mail.gmail.com>
References: <CABnaYKCDaW9VGTrhMJR3iODYNXg+UdBZVJ5Tu+JJZCWdb-6AOQ@mail.gmail.com>
 <CABnaYKC+AgqTzKVFb8vxOd7JUrEKGftj-ZbNJU-q7fvOv8qgYg@mail.gmail.com>
Message-ID: <CAOTD34aC6qFUj3LmLFKy5vGgYn-qG3O-GHr=JsKET7-1CoBA5Q@mail.gmail.com>

On Fri, Apr 21, 2017 at 12:09 PM, Justus Schwabedal
<jschwabedal at gmail.com> wrote:
> Hi everyone,
>
> I possibly found a bug in class initialization and would like to fix it.
> Because it's my first journey to core-dev, I would really appreciate the
> help of a mentor that I may ask a few questions to get me up to speed.
>
> To my person, I have previously worked on larger projects in python, c, and
> c++ if that information helps, and I'm really curious to learn more about
> the interiors of the greatest interpreter known to wo-/men.
>
> Here comes the bug-producing example:
>
> `class Foo:
>     def __init__(self, bar=[]):
>         self.list = bar
>
> spam_1 = Foo()
> spam_2 = Foo()
>
> spam_1.list.append(42)
> print(spam_2.list)`
>
> At least I think it's a bug.  Maybe it's a feature..

Sorry to resurrect an old-ish thread; I haven't looked at Python-dev
in several weeks.  I just wanted to point out that while everyone on
this thread pointed out how this isn't a bug (clear to anyone who's
spent enough time with Python), we have here an experienced C/C++
developer who is interested in helping out on Python core devel, and
no one took him up on that offer.

Jus, for what it's worth, there are a slew of *actual* Python bugs to
be worked on--many languishing for years--due to lack of available
developer time.  You can have a good look at the (daunting) list of
bugs at the old bugs.python.org [1].  IIUC Python development is
slowly moving over to GitHub, but the issue list hasn't migrated yet
so that would still be the place to start.

If you find a bug that looks worth your time to try to fix, you should
probably follow up on the issue for that bug itself.  I can't make any
promises anyone will have time for mentorship, but I'd be willing to
point you in the right direction.  I'm not a core developer but I know
Python internals reasonably well and might be able to help *depending*
on the issue.

Best,
Erik

[1] http://bugs.python.org/

From brett at python.org  Tue May  2 12:30:03 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 02 May 2017 16:30:03 +0000
Subject: [Python-Dev] Guarantee the success of some object creation C
 API functions
In-Reply-To: <CADiSq7cWsN40gPs=kbdGR5EF9VTKgdkR3FmZiW6vkyeacsoM+A@mail.gmail.com>
References: <CAAuNNr7B1p-hsbwfRMxBidNGTog4smACWBB-z9uF4wYnedZqYA@mail.gmail.com>
 <oe876s$n9o$1@blaine.gmane.org>
 <CAPTjJmopDDa+3T9U+UkjnkBgGrA2FFw3J7fPYKAphMjdS1KHXA@mail.gmail.com>
 <CADiSq7cWsN40gPs=kbdGR5EF9VTKgdkR3FmZiW6vkyeacsoM+A@mail.gmail.com>
Message-ID: <CAP1=2W79jGozkZ9McYa6rB9U2MvPATDJUY1rb7ixYndvNjyp3A@mail.gmail.com>

On Mon, 1 May 2017 at 21:19 Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 2 May 2017 at 07:52, Chris Angelico <rosuav at gmail.com> wrote:
> > On Tue, May 2, 2017 at 6:52 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> >> The promise makes it clear that breaking the property is a bug to be
> fixed.
> >> It only decreases the probability for someone who has read the promise.
> >> Unfortunately, 'never fail' is hard to test ;-).
> >>
> >
> > Aside from straight-up bugs, how can one of these functions fail? Is
> > memory allocation failure the only way? If so, the proposed
> > implementation (private references to pre-created singletons) ought to
> > guarantee that, to the exact extent that anything else can be
> > guaranteed.
> >
> > (Or is that your point - that "never fail" is always "modulo bugs"?)
> >
> > Incidentally, this guarantee, if implemented the obvious way, will
> > also mean that (), "", 0, etc are singletons. People talk casually
> > about the "empty tuple singleton", but I don't think it's actually
> > guaranteed anywhere.
>
> I don't think it is either, and I don't think it's a guarantee we want
> to make - even with Serhiy's proposed change, it's still
> straightforward to produce non-singleton instances of these values
> using the low level C APIs, while the true singletons (True, False,
> None, Ellipsis) go out of their way to make it difficult to create new
> instances of the corresponding types, even when mucking about at the C
> layer.
>
> The assertion Serhiy is proposing to make is much weaker, and would be
> a matter of adding something like the following to the C API
> reference:
>
> "Similar to the singleton values True, False, None, and Ellipsis,
> process global instances of the builtin values (), '', b'', 0, and 1
> are pre-allocated at interpreter startup, so APIs returning these
> values should never fail, even in low memory conditions. However,
> additional empty instances of these types may still be created through
> the C API, so they should always be compared by value rather than by
> identity."
>
> The specific "never fails as it returns a pointer to a pre-allocated
> instance" cases can then be documented on the affected public API
> functions.
>
> So +1 from me for making pre-allocation a CPython C API guarantee, -1
> for making it a language level singleton commitment.


I agree with Nick's votes.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170502/e68306b9/attachment.html>

From mariatta.wijaya at gmail.com  Tue May  2 12:37:22 2017
From: mariatta.wijaya at gmail.com (Mariatta Wijaya)
Date: Tue, 2 May 2017 09:37:22 -0700
Subject: [Python-Dev] Possible bug in class-init, lookin for mentors
In-Reply-To: <CAOTD34aC6qFUj3LmLFKy5vGgYn-qG3O-GHr=JsKET7-1CoBA5Q@mail.gmail.com>
References: <CABnaYKCDaW9VGTrhMJR3iODYNXg+UdBZVJ5Tu+JJZCWdb-6AOQ@mail.gmail.com>
 <CABnaYKC+AgqTzKVFb8vxOd7JUrEKGftj-ZbNJU-q7fvOv8qgYg@mail.gmail.com>
 <CAOTD34aC6qFUj3LmLFKy5vGgYn-qG3O-GHr=JsKET7-1CoBA5Q@mail.gmail.com>
Message-ID: <CAGbohnar-V26qKDsK4KdZ9R0W_1ZYHE1iHps-HikrHSpSMmKbg@mail.gmail.com>

Justus, welcome.

Consider joining the core-mentorship mailing list. If you have specific
questions on how to contribute or how to get started, we can help you
there. https://mail.python.org/mailman/listinfo/core-mentorship

Thanks, Erik. Yes, CPython has moved to GitHub https://github.com/python/
cpython but we will continue using bugs.python.org as the issue tracker.
We prefer that discussions happen on the issue tracker, and GitHub is for
code reviewing only.

The Dev Guide has been updated with our new GitHub workflow, specifically:
http://cpython-devguide.readthedocs.io/pullrequest.html
http://cpython-devguide.readthedocs.io/gitbootcamp.html

Thanks :)


Mariatta Wijaya

On Tue, May 2, 2017 at 9:01 AM, Erik Bray <erik.m.bray at gmail.com> wrote:

> On Fri, Apr 21, 2017 at 12:09 PM, Justus Schwabedal
> <jschwabedal at gmail.com> wrote:
> > Hi everyone,
> >
> > I possibly found a bug in class initialization and would like to fix it.
> > Because it's my first journey to core-dev, I would really appreciate the
> > help of a mentor that I may ask a few questions to get me up to speed.
> >
> > To my person, I have previously worked on larger projects in python, c,
> and
> > c++ if that information helps, and I'm really curious to learn more about
> > the interiors of the greatest interpreter known to wo-/men.
> >
> > Here comes the bug-producing example:
> >
> > `class Foo:
> >     def __init__(self, bar=[]):
> >         self.list = bar
> >
> > spam_1 = Foo()
> > spam_2 = Foo()
> >
> > spam_1.list.append(42)
> > print(spam_2.list)`
> >
> > At least I think it's a bug.  Maybe it's a feature..
>
> Sorry to resurrect an old-ish thread; I haven't looked at Python-dev
> in several weeks.  I just wanted to point out that while everyone on
> this thread pointed out how this isn't a bug (clear to anyone who's
> spent enough time with Python), we have here an experienced C/C++
> developer who is interested in helping out on Python core devel, and
> no one took him up on that offer.
>
> Jus, for what it's worth, there are a slew of *actual* Python bugs to
> be worked on--many languishing for years--due to lack of available
> developer time.  You can have a good look at the (daunting) list of
> bugs at the old bugs.python.org [1].  IIUC Python development is
> slowly moving over to GitHub, but the issue list hasn't migrated yet
> so that would still be the place to start.
>
> If you find a bug that looks worth your time to try to fix, you should
> probably follow up on the issue for that bug itself.  I can't make any
> promises anyone will have time for mentorship, but I'd be willing to
> point you in the right direction.  I'm not a core developer but I know
> Python internals reasonably well and might be able to help *depending*
> on the issue.
>
> Best,
> Erik
>
> [1] http://bugs.python.org/
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> mariatta.wijaya%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170502/b180f439/attachment.html>

From ethan at stoneleaf.us  Tue May  2 13:06:49 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 02 May 2017 10:06:49 -0700
Subject: [Python-Dev] __getattribute__'s error is not available in
 __getattr__
In-Reply-To: <CA+LevYJQ6bHR2BtncCZm_5i7emWUyuuvXfnY1oWNBWbRG6xh1w@mail.gmail.com>
References: <CA+LevYJQ6bHR2BtncCZm_5i7emWUyuuvXfnY1oWNBWbRG6xh1w@mail.gmail.com>
Message-ID: <5908BCA9.7000303@stoneleaf.us>

On 05/01/2017 08:47 PM, Jason Maldonis wrote:

> If this should be asked in learn python I apologize -- please just tell me without answering.

If you could re-ask this question over on Python List I'd love to discuss which errors you are seeing from 
__getattribute__ (beside AttributeError, of course, as that's the only one I would expect to see).

--
~Ethan~


From goodger at python.org  Tue May  2 14:58:27 2017
From: goodger at python.org (David Goodger)
Date: Tue, 2 May 2017 13:58:27 -0500
Subject: [Python-Dev] Escaping docs markup in NEWS entries?
In-Reply-To: <CACfEFw94Xie5ijCpizEX3sofattBUGsOQkMQpAjUyg9Am3W5Pg@mail.gmail.com>
References: <CADiSq7c5GuOMTCJKUrTx=jhaYn7VTho_qDT3FYhOYbD70q2F9Q@mail.gmail.com>
 <CA+eR4cHSWGpPgYL_eU1G69pfi1h5fjnQqVJfHGjCbzY6ACfE-Q@mail.gmail.com>
 <CADiSq7cLx6184UoJnJ2yx8otagqa-Pqpmgn+nNLpjwbLr=EvmQ@mail.gmail.com>
 <CACfEFw94Xie5ijCpizEX3sofattBUGsOQkMQpAjUyg9Am3W5Pg@mail.gmail.com>
Message-ID: <CABiJP=KmXKXmqG3J5YPuhgB830tPDgDxQ-8VX1SDZ+WqUSRTdA@mail.gmail.com>

On Mon, May 1, 2017 at 5:03 AM, Wes Turner <wes.turner at gmail.com> wrote:
> Where would be a good place for test cases for an rst_escape() function?
> Docutils?
>
> https://github.com/westurner/dotfiles/blob/develop/scripts/git-changelog.py

I'd say that the test cases for any function should reside in the same
project/codebase as the function itself. Where do you intend the
function to reside?

I'm not certain what rst_escape() is supposed to do; what is its
purpose? The docstrings of rst_escape() as well as the
git-changelog.py module lack such explanations.

I am also uncertain if this is necessary, based on the remainder of
the thread. Perhaps it would be better to fix this "make suspicious"
so that it doesn't produce spurious results? (Note: I am unclear on
exactly what "make suspicious" is supposed to do, or why.)

I apologize for the azure hue of my answers, to questions that came
from out of the blue.
;-)

David Goodger
<http://python.net/~goodger>


> - rst_escape # YMMV
> - $ git-changelog.py -r "release/0.3.14" --hdr= "+"`
>
> On Monday, May 1, 2017, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> On 1 May 2017 at 17:13, Martin Panter <vadmium+py at gmail.com> wrote:
>> > On 1 May 2017 at 06:37, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> >> Hi folks,
>> >>
>> >> I'm trying to write a NEWS entry that explains that the
>> >> ":func:`bytes`" cross-references have changed to refer to the type
>> >> descriptions by default (matching other builtin container types), so
>> >> you now need to use ``:ref:`func-bytes`" to refer to the old target in
>> >> the list of builtin functions (if you really want that for some
>> >> reason).
>> >>
>> >> Unfortunately, my first two attempts both cause warnings in "make
>> >> suspicious" with the following output:
>> >
>> > What is the full output? Usually it includes instructions to add false
>> > positives to Doc/tools/susp-ignored.csv; maybe that is all you have to
>> > do?
>>
>> You're right, that would be likely be the way to go if I decided to
>> keep the escaped markup.
>>
>> However...
>>
>> >> My fallback plan is to just include the unescaped markup in the NEWS
>> >> entry (rather than trying to make it readable even in rendered form),
>> >> but I figured I'd ask for advice here first.
>> >
>> > I thought the NEWS file was mainly regarded as plain text, so it would
>> > be important to avoid ugly RST markup like backslashes if possible.
>>
>> ... I think you're right on this point, so it makes more sense to skip
>> the escaping entirely,
>> and just use the correct link markup in the NEWS entry.
>
> How convenient.
>>
>>
>> Cheers,
>> Nick.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe:
>> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com

From wes.turner at gmail.com  Tue May  2 16:45:17 2017
From: wes.turner at gmail.com (Wes Turner)
Date: Tue, 2 May 2017 15:45:17 -0500
Subject: [Python-Dev] Escaping docs markup in NEWS entries?
In-Reply-To: <CABiJP=KmXKXmqG3J5YPuhgB830tPDgDxQ-8VX1SDZ+WqUSRTdA@mail.gmail.com>
References: <CADiSq7c5GuOMTCJKUrTx=jhaYn7VTho_qDT3FYhOYbD70q2F9Q@mail.gmail.com>
 <CA+eR4cHSWGpPgYL_eU1G69pfi1h5fjnQqVJfHGjCbzY6ACfE-Q@mail.gmail.com>
 <CADiSq7cLx6184UoJnJ2yx8otagqa-Pqpmgn+nNLpjwbLr=EvmQ@mail.gmail.com>
 <CACfEFw94Xie5ijCpizEX3sofattBUGsOQkMQpAjUyg9Am3W5Pg@mail.gmail.com>
 <CABiJP=KmXKXmqG3J5YPuhgB830tPDgDxQ-8VX1SDZ+WqUSRTdA@mail.gmail.com>
Message-ID: <CACfEFw_zNTcaxPFL8UeAD-f4P8N5Mgzv9vkZiJ8rRW8r-vXs0w@mail.gmail.com>

On Tue, May 2, 2017 at 1:58 PM, David Goodger <goodger at python.org> wrote:

> On Mon, May 1, 2017 at 5:03 AM, Wes Turner <wes.turner at gmail.com> wrote:
> > Where would be a good place for test cases for an rst_escape() function?
> > Docutils?
> >
> > https://github.com/westurner/dotfiles/blob/develop/scripts/
> git-changelog.py
>
> I'd say that the test cases for any function should reside in the same
> project/codebase as the function itself. Where do you intend the
> function to reside?
>
> I'm not certain what rst_escape() is supposed to do; what is its
> purpose? The docstrings of rst_escape() as well as the
> git-changelog.py module lack such explanations.
>

def rst_escape(_str: str) -> str:
    """"
    Args:
        _str: string to safely escape for literal inclusion in a
ReStructuredText document
    Returns:
        str: safely-escaped ReStructuredText
    """"


>
> I am also uncertain if this is necessary, based on the remainder of
> the thread. Perhaps it would be better to fix this "make suspicious"
> so that it doesn't produce spurious results? (Note: I am unclear on
> exactly what "make suspicious" is supposed to do, or why.)
>

An rst_escape(text: ) function is necessary in order to prevent
ReStructuredText injection (and e.g. * unintentional partial italicization).

There are other directives for including text in a ReStructuredText
document, e.g.:

- ``.. include::``
- ``.. raw::``

And there are extensions for changelogs and sphinx builds:

- ``.. git_changelog::`` -- https://pypi.org/project/sphinx-git/
- ``.. changelog::``, ``.. change::`` -- https://pypi.org/project/changelog/


But, for a README.rst in a git repository, for example, it's far easier to
diff the document with the data included than it is to have to build from a
source .rst to a dest .rst.rst and then diff the .rst.rst files.

There are CWE URLs for describing the type of vulnerability that a
community-reviewed rst_escape() function would be designed to prevent:

- CWE-116: Improper Encoding or Escaping of Output
  https://cwe.mitre.org/data/definitions/116.html
  - CWE-74: Improper Neutralization of Special Elements in Output Used by a
Downstream Component ('Injection')
    https://cwe.mitre.org/data/definitions/74.html#Relationships

    - CWE-89: SQL Injection
      https://cwe.mitre.org/data/definitions/89.htm

      - 2011 Top25 #1 issue
        https://cwe.mitre.org/top25/#CWE-89

    - [ ] CWE-???: RST Injection


Not a bug in docutils; a PEBKAM implementation "BUG,SEC" issue


>
> I apologize for the azure hue of my answers, to questions that came
> from out of the blue.
> ;-)
>
> David Goodger
> <http://python.net/~goodger>
>
>
> > - rst_escape # YMMV
> > - $ git-changelog.py -r "release/0.3.14" --hdr= "+"`
> >
> > On Monday, May 1, 2017, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >>
> >> On 1 May 2017 at 17:13, Martin Panter <vadmium+py at gmail.com> wrote:
> >> > On 1 May 2017 at 06:37, Nick Coghlan <ncoghlan at gmail.com> wrote:
> >> >> Hi folks,
> >> >>
> >> >> I'm trying to write a NEWS entry that explains that the
> >> >> ":func:`bytes`" cross-references have changed to refer to the type
> >> >> descriptions by default (matching other builtin container types), so
> >> >> you now need to use ``:ref:`func-bytes`" to refer to the old target
> in
> >> >> the list of builtin functions (if you really want that for some
> >> >> reason).
> >> >>
> >> >> Unfortunately, my first two attempts both cause warnings in "make
> >> >> suspicious" with the following output:
> >> >
> >> > What is the full output? Usually it includes instructions to add false
> >> > positives to Doc/tools/susp-ignored.csv; maybe that is all you have to
> >> > do?
> >>
> >> You're right, that would be likely be the way to go if I decided to
> >> keep the escaped markup.
> >>
> >> However...
> >>
> >> >> My fallback plan is to just include the unescaped markup in the NEWS
> >> >> entry (rather than trying to make it readable even in rendered form),
> >> >> but I figured I'd ask for advice here first.
> >> >
> >> > I thought the NEWS file was mainly regarded as plain text, so it would
> >> > be important to avoid ugly RST markup like backslashes if possible.
> >>
> >> ... I think you're right on this point, so it makes more sense to skip
> >> the escaping entirely,
> >> and just use the correct link markup in the NEWS entry.
> >
> > How convenient.
> >>
> >>
> >> Cheers,
> >> Nick.
> >>
> >> --
> >> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> >> _______________________________________________
> >> Python-Dev mailing list
> >> Python-Dev at python.org
> >> https://mail.python.org/mailman/listinfo/python-dev
> >> Unsubscribe:
> >> https://mail.python.org/mailman/options/python-dev/
> wes.turner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170502/f8c79e21/attachment-0001.html>

From victor.stinner at gmail.com  Wed May  3 04:22:58 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 3 May 2017 10:22:58 +0200
Subject: [Python-Dev] Status of Python buildbots
Message-ID: <CAMpsgwZ=N1x_YfU7hV-=vKP43AC=uVCPdLkA-=SsSHCBaTQedg@mail.gmail.com>

Hi,

I spent last week working on fixing buildbots:

   https://www.python.org/dev/buildbot/

It should now be able again to rely on them to detect regressions. Changes:

* Fix various bugs (I don't even recall which ones)
* Fix multiple random failures
* Fix dozen of warnings

I also enhanced the configuration of buildbots to search for warnings
in the output of the "tests" step (Python test suite). I use a regular
expression to search for patterns:

https://github.com/python/buildmaster-config/blob/master/master/master.cfg#L93-L125

If a warning is found, the buildbot now becomes orange. As I wrote, I
fixed dozens of warnings, but there are still a few ones (and many
unstable tests failing randomly).

The orange state notify for example when a test failed once, but
passed when it was run again in verbose mode. It helps to detect
unstable tests, but also real bugs.


The "Warning -- xxx was modified by test_xxx" warning now always log
the old and new value to ease debugging these warnings. Example on
Python 2.7:

Warning -- files was modified by test_sax
  Before: []
  After:  ['@test_352_tmp']


TODO:

* Create a mailing list and send an email to this list when the state
of a buildbot changes (only from green to orange or red? similar to
what we have on #python-dev IRC channel). I suggest to start with a
whitelist of buildbots known to be stable... hum, like the list of
"stable buildbots" ;-) Let's start with the 3.x branch, and later
enable it on more branches. The mailing list should help to coordinate
when multiple developers work on buildbot issues in parallel.

* Fix FreeBSD buildbots which fail to compile Python because of
http://bugs.python.org/issue23404 I proposed a change to not
regenerate generated files based on file modification time anymore,
but require an explicit action ("make regen-all"). This change solves
many practical issues.

* Fix remaining warnings.

* Fix unstable tests.

* Add more warnings! I proposed to emit ResourceWarning in
multiprocessing.Queue and concurrent.futures executors ;-)
http://bugs.python.org/issue30171 and
http://bugs.python.org/issue30244

* Add more buildbots! Zachary Ware proposed to add a buildbot running
"regen-all" to check that generated files are up to date.

* Repeat ;-)

Victor

From erik.m.bray at gmail.com  Wed May  3 05:16:33 2017
From: erik.m.bray at gmail.com (Erik Bray)
Date: Wed, 3 May 2017 11:16:33 +0200
Subject: [Python-Dev] Status of Python buildbots
In-Reply-To: <CAMpsgwZ=N1x_YfU7hV-=vKP43AC=uVCPdLkA-=SsSHCBaTQedg@mail.gmail.com>
References: <CAMpsgwZ=N1x_YfU7hV-=vKP43AC=uVCPdLkA-=SsSHCBaTQedg@mail.gmail.com>
Message-ID: <CAOTD34Z1n7+ZDXvjmf12qkZDN-eoOT1t-H-86_8yAGLXhOntnw@mail.gmail.com>

On Wed, May 3, 2017 at 10:22 AM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hi,
>
> I spent last week working on fixing buildbots:
>
>    https://www.python.org/dev/buildbot/
>

Thanks!

> * Add more buildbots! Zachary Ware proposed to add a buildbot running
> "regen-all" to check that generated files are up to date.
>
> * Repeat ;-)

I have been saying for several months now that I would like to set up
a Cygwin buildbot--an important step in making that platform
supportable again.  I now have the infrastructure available to do so
(Windows VM on an OpenStack infrastructure at my university).  I
wanted to wait until the tests on Cygwin were more stable, but since
you allow unstable buildbots I could include it among them for now.
Is the buildbot setup documentation on the wiki page still accurate?

Thanks,
Erik

From ma3yuki.8mamo10 at gmail.com  Wed May  3 04:59:16 2017
From: ma3yuki.8mamo10 at gmail.com (Masayuki YAMAMOTO)
Date: Wed, 3 May 2017 17:59:16 +0900
Subject: [Python-Dev] Removing Python/thread_foobar.h
Message-ID: <CALC+iBwAf-dtYoBFveZjomheC=-jwDGSpA_U8uh7ooGWibX_uA@mail.gmail.com>

Hi, python-dev.

I'm reading the part of threading feature to replace TLS API with TSS API
(PEP 539). Python/thread_foobar.h hasn't be included to Python/thread.c
(its include directive is comment out) and anywhere. This looks like dead
code and we want to delete it, does anyone have a reason we should keep it?

Regards,
Masayuki
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170503/5de4ce9f/attachment.html>

From guido at python.org  Wed May  3 10:35:28 2017
From: guido at python.org (Guido van Rossum)
Date: Wed, 3 May 2017 07:35:28 -0700
Subject: [Python-Dev] Removing Python/thread_foobar.h
In-Reply-To: <CALC+iBwAf-dtYoBFveZjomheC=-jwDGSpA_U8uh7ooGWibX_uA@mail.gmail.com>
References: <CALC+iBwAf-dtYoBFveZjomheC=-jwDGSpA_U8uh7ooGWibX_uA@mail.gmail.com>
Message-ID: <CAP7+vJ+xg+T91PexJWB5d=a-czP84pfMgBotkZ76VhiJAM7uDg@mail.gmail.com>

Good catch! It's actually a template for writing a threading adaptation for
new platforms. But it's probably hopelessly out of date so unless someone
wants to adopt it and keep it up to date, we can delete it. (Also, new
platforms with unique threading APIs don't come along at the rate they did
in the 1990s. :-)

If there's more discussion, it probably belongs on bugs.python.org.

--Guido

On Wed, May 3, 2017 at 1:59 AM, Masayuki YAMAMOTO <ma3yuki.8mamo10 at gmail.com
> wrote:

> Hi, python-dev.
>
> I'm reading the part of threading feature to replace TLS API with TSS API
> (PEP 539). Python/thread_foobar.h hasn't be included to Python/thread.c
> (its include directive is comment out) and anywhere. This looks like dead
> code and we want to delete it, does anyone have a reason we should keep it?
>
> Regards,
> Masayuki
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170503/955a06dd/attachment.html>

From victor.stinner at gmail.com  Wed May  3 11:08:25 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 3 May 2017 17:08:25 +0200
Subject: [Python-Dev] Removing Python/thread_foobar.h
In-Reply-To: <CALC+iBwAf-dtYoBFveZjomheC=-jwDGSpA_U8uh7ooGWibX_uA@mail.gmail.com>
References: <CALC+iBwAf-dtYoBFveZjomheC=-jwDGSpA_U8uh7ooGWibX_uA@mail.gmail.com>
Message-ID: <CAMpsgwZv9NodUASRdnkHzMfj+SiW3ibi2MX-yAJQ3t3sSJeV7g@mail.gmail.com>

You can remove thread_foobar.h. I don't think that anyone still wants
to use this template. The other thread_*.h files can be used as
template as well.

Victor

From brett at python.org  Wed May  3 12:57:38 2017
From: brett at python.org (Brett Cannon)
Date: Wed, 03 May 2017 16:57:38 +0000
Subject: [Python-Dev] Status of Python buildbots
In-Reply-To: <CAMpsgwZ=N1x_YfU7hV-=vKP43AC=uVCPdLkA-=SsSHCBaTQedg@mail.gmail.com>
References: <CAMpsgwZ=N1x_YfU7hV-=vKP43AC=uVCPdLkA-=SsSHCBaTQedg@mail.gmail.com>
Message-ID: <CAP1=2W4w70b7Ab2j87qoVjgXCm=tVY=HtgmaoCedbdQ5Aci-nQ@mail.gmail.com>

On Wed, 3 May 2017 at 01:24 Victor Stinner <victor.stinner at gmail.com> wrote:

> Hi,
>
> I spent last week working on fixing buildbots:
>
>    https://www.python.org/dev/buildbot/
>
> It should now be able again to rely on them to detect regressions.


Thanks, Victor!


> Changes:
>
> * Fix various bugs (I don't even recall which ones)
> * Fix multiple random failures
> * Fix dozen of warnings
>
> I also enhanced the configuration of buildbots to search for warnings
> in the output of the "tests" step (Python test suite). I use a regular
> expression to search for patterns:
>
>
> https://github.com/python/buildmaster-config/blob/master/master/master.cfg#L93-L125
>
> If a warning is found, the buildbot now becomes orange. As I wrote, I
> fixed dozens of warnings, but there are still a few ones (and many
> unstable tests failing randomly).
>
> The orange state notify for example when a test failed once, but
> passed when it was run again in verbose mode. It helps to detect
> unstable tests, but also real bugs.
>
>
> The "Warning -- xxx was modified by test_xxx" warning now always log
> the old and new value to ease debugging these warnings. Example on
> Python 2.7:
>
> Warning -- files was modified by test_sax
>   Before: []
>   After:  ['@test_352_tmp']
>
>
> TODO:
>
> * Create a mailing list and send an email to this list when the state
> of a buildbot changes (only from green to orange or red? similar to
> what we have on #python-dev IRC channel). I suggest to start with a
> whitelist of buildbots known to be stable... hum, like the list of
> "stable buildbots" ;-) Let's start with the 3.x branch, and later
> enable it on more branches. The mailing list should help to coordinate
> when multiple developers work on buildbot issues in parallel.
>

SGTM.

-Brett


>
> * Fix FreeBSD buildbots which fail to compile Python because of
> http://bugs.python.org/issue23404 I proposed a change to not
> regenerate generated files based on file modification time anymore,
> but require an explicit action ("make regen-all"). This change solves
> many practical issues.
>
> * Fix remaining warnings.
>
> * Fix unstable tests.
>
> * Add more warnings! I proposed to emit ResourceWarning in
> multiprocessing.Queue and concurrent.futures executors ;-)
> http://bugs.python.org/issue30171 and
> http://bugs.python.org/issue30244
>
> * Add more buildbots! Zachary Ware proposed to add a buildbot running
> "regen-all" to check that generated files are up to date.
>
> * Repeat ;-)
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170503/ae68bea3/attachment.html>

From zachary.ware+pydev at gmail.com  Wed May  3 13:11:51 2017
From: zachary.ware+pydev at gmail.com (Zachary Ware)
Date: Wed, 3 May 2017 12:11:51 -0500
Subject: [Python-Dev] Status of Python buildbots
In-Reply-To: <CAOTD34Z1n7+ZDXvjmf12qkZDN-eoOT1t-H-86_8yAGLXhOntnw@mail.gmail.com>
References: <CAMpsgwZ=N1x_YfU7hV-=vKP43AC=uVCPdLkA-=SsSHCBaTQedg@mail.gmail.com>
 <CAOTD34Z1n7+ZDXvjmf12qkZDN-eoOT1t-H-86_8yAGLXhOntnw@mail.gmail.com>
Message-ID: <CAKJDb-NC7RTEHh1GXe+_G=Lxy2BGu_Jo23qvWrbFUdhvKbZsZw@mail.gmail.com>

On Wed, May 3, 2017 at 4:16 AM, Erik Bray <erik.m.bray at gmail.com> wrote:
> I have been saying for several months now that I would like to set up
> a Cygwin buildbot--an important step in making that platform
> supportable again.  I now have the infrastructure available to do so
> (Windows VM on an OpenStack infrastructure at my university).  I
> wanted to wait until the tests on Cygwin were more stable, but since
> you allow unstable buildbots I could include it among them for now.

That would be fine :)

> Is the buildbot setup documentation on the wiki page still accurate?

Probably not; the devguide should be more up-to-date:
https://docs.python.org/devguide/buildslave.html

I don't know what unique challenges you may run into setting up a
Cygwin worker, but please submit PRs against that page if you find
anything to be lacking.

-- 
Zach

From brett at python.org  Wed May  3 14:15:06 2017
From: brett at python.org (Brett Cannon)
Date: Wed, 03 May 2017 18:15:06 +0000
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement or
 bugfix?
Message-ID: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>

My allergies have hit me hard so I'm not thinking at full capacity, but did
we ever decide if supporting os.PathLike in the stdlib was viewed as an
enhancement or bugfix? Specifically I'm thinking of
https://bugs.python.org/issue30218 for adding support to
shutil.unpack_archive() and whether it should be backported to 3.6.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170503/8c98134a/attachment.html>

From k7hoven at gmail.com  Wed May  3 17:33:16 2017
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 4 May 2017 00:33:16 +0300
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
Message-ID: <CAMiohoiW5S5TnQKe-98fTp=fc881dBag4e=Tf8vJhMQbExvW+w@mail.gmail.com>

On Wed, May 3, 2017 at 9:15 PM, Brett Cannon <brett at python.org> wrote:
> My allergies have hit me hard so I'm not thinking at full capacity, but did
> we ever decide if supporting os.PathLike in the stdlib was viewed as an
> enhancement or bugfix? Specifically I'm thinking of
> https://bugs.python.org/issue30218 for adding support to
> shutil.unpack_archive() and whether it should be backported to 3.6.
>

Haha, not sure if you have actual allergy, or if you just remembered
the fspath discussions nightmare ;).

You're obviously not asking me, but I'd say anything in the 3.6 stdlib
that takes a path, but not os.PathLike, should count as a bug.
Otherwise there will just be surprises with code that works on 3.7 but
not on the famously fspath-enabled 3.6. I'm not even 100% sure that
backporting to 3.5 would be (or would have been) a bad idea; 3.5 will
stick around for quite some time still.

-- Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +

From tjreedy at udel.edu  Wed May  3 18:07:10 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 3 May 2017 18:07:10 -0400
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
Message-ID: <oedkaa$ckk$1@blaine.gmane.org>

On 5/3/2017 2:15 PM, Brett Cannon wrote:
> My allergies have hit me hard so I'm not thinking at full capacity, but 
> did we ever decide if supporting os.PathLike in the stdlib was viewed as 
> an enhancement or bugfix? Specifically I'm thinking of 
> https://bugs.python.org/issue30218 for adding support to 
> shutil.unpack_archive() and whether it should be backported to 3.6.

On the face of it, that particular issue looks like an enhancement that 
should have gone into 3.6 (if ever), but did not.  I notice that
https://www.python.org/dev/peps/pep-0519/#implementation
did not include "Update shutil", so it was not done, at least not 
completely.

Was shutil updated at all?  Is unpack_archive the only shutil function 
not updated?  If so, I could see the omission as a bug.

If the patch for 30218 were applied in 3.6, would the doc
https://docs.python.org/3/library/shutil.html#shutil.unpack_archive
need to be changed, with a note "Added in 3.6.2: filename can be any 
pathlike object"?  If so, it is an enhancement.

-- 
Terry Jan Reedy


From victor.stinner at gmail.com  Wed May  3 18:25:55 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 4 May 2017 00:25:55 +0200
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
Message-ID: <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>

If you start to backport support for the fspath protocol, be prepared
to have to backport it *many* places. I expect that slowly in the near
future, many functions will be patched to support the fspath protocol.

I suggest to only do that in master. It's not that hard to cast
manually to a string: it's just str(path), no?

Victor

2017-05-03 20:15 GMT+02:00 Brett Cannon <brett at python.org>:
> My allergies have hit me hard so I'm not thinking at full capacity, but did
> we ever decide if supporting os.PathLike in the stdlib was viewed as an
> enhancement or bugfix? Specifically I'm thinking of
> https://bugs.python.org/issue30218 for adding support to
> shutil.unpack_archive() and whether it should be backported to 3.6.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>

From k7hoven at gmail.com  Wed May  3 19:13:20 2017
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 4 May 2017 02:13:20 +0300
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <oedkaa$ckk$1@blaine.gmane.org>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <oedkaa$ckk$1@blaine.gmane.org>
Message-ID: <CAMiohoieBthrQNpPJ71u0x4HirRSR7VH6dVvU20vZwsnRh6fWg@mail.gmail.com>

On Thu, May 4, 2017 at 1:07 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 5/3/2017 2:15 PM, Brett Cannon wrote:
>>
>> My allergies have hit me hard so I'm not thinking at full capacity, but
>> did we ever decide if supporting os.PathLike in the stdlib was viewed as an
>> enhancement or bugfix? Specifically I'm thinking of
>> https://bugs.python.org/issue30218 for adding support to
>> shutil.unpack_archive() and whether it should be backported to 3.6.
>
>
> On the face of it, that particular issue looks like an enhancement that
> should have gone into 3.6

Agreed.

> (if ever), but did not.  I notice that
> https://www.python.org/dev/peps/pep-0519/#implementation
> did not include "Update shutil", so it was not done, at least not
> completely.

Shutil was among the most important to be updated, IMO.

I had made some sort of list of affected modules elsewhere [1]:
ntpath, posixpath, os.scandir, os.[other stuff], DirEntry (tempted to
say os.DirEntry, but that is
not true), shutil.[stuff], (io.)open, fileinput, filecmp, zipfile,
tarfile, tempfile (for the 'dir' keyword arguments), maybe even glob
and fnmatch (are the patterns paths?)

It looks like what made it to PEP519 was mainly this:

"It is expected that most APIs in Python's standard library that
currently accept a file system path will be updated appropriately to
accept path objects (whether that requires code or simply an update to
documentation will vary)."

> Was shutil updated at all?  Is unpack_archive the only shutil function not
> updated?  If so, I could see the omission as a bug.
>
> If the patch for 30218 were applied in 3.6, would the doc
> https://docs.python.org/3/library/shutil.html#shutil.unpack_archive
> need to be changed, with a note "Added in 3.6.2: filename can be any
> pathlike object"?  If so, it is an enhancement.

Regardless of bugfix vs enhancement semantics, that seems like a good
thing to do.

-- Koos

[1] e.g. in this thread somewhere:
https://mail.python.org/pipermail/python-ideas/2016-April/039827.html


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +

From tjreedy at udel.edu  Wed May  3 21:19:02 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 3 May 2017 21:19:02 -0400
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAMiohoieBthrQNpPJ71u0x4HirRSR7VH6dVvU20vZwsnRh6fWg@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <oedkaa$ckk$1@blaine.gmane.org>
 <CAMiohoieBthrQNpPJ71u0x4HirRSR7VH6dVvU20vZwsnRh6fWg@mail.gmail.com>
Message-ID: <oedvi1$h2s$1@blaine.gmane.org>

On 5/3/2017 7:13 PM, Koos Zevenhoven wrote:
> On Thu, May 4, 2017 at 1:07 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>> On 5/3/2017 2:15 PM, Brett Cannon wrote:
>>>
>>> My allergies have hit me hard so I'm not thinking at full capacity, but
>>> did we ever decide if supporting os.PathLike in the stdlib was viewed as an
>>> enhancement or bugfix? Specifically I'm thinking of
>>> https://bugs.python.org/issue30218 for adding support to
>>> shutil.unpack_archive() and whether it should be backported to 3.6.
>>
>>
>> On the face of it, that particular issue looks like an enhancement that
>> should have gone into 3.6
> 
> Agreed.
> 
>> (if ever), but did not.  I notice that
>> https://www.python.org/dev/peps/pep-0519/#implementation
>> did not include "Update shutil", so it was not done, at least not
>> completely.
> 
> Shutil was among the most important to be updated, IMO.
> 
> I had made some sort of list of affected modules elsewhere [1]:
> ntpath, posixpath, os.scandir, os.[other stuff], DirEntry (tempted to
> say os.DirEntry, but that is
> not true), shutil.[stuff], (io.)open, fileinput, filecmp, zipfile,
> tarfile, tempfile (for the 'dir' keyword arguments), maybe even glob
> and fnmatch (are the patterns paths?)

What did not get done for 3.6 should be proposed for 3.7.

> It looks like what made it to PEP519 was mainly this:
> 
> "It is expected that most APIs in Python's standard library that
> currently accept a file system path will be updated appropriately to
> accept path objects (whether that requires code or simply an update to
> documentation will vary)."

To me that says that not being updated (yet) is not a bug.

>> Was shutil updated at all?  Is unpack_archive the only shutil function not
>> updated?  If so, I could see the omission as a bug.
>>
>> If the patch for 30218 were applied in 3.6, would the doc
>> https://docs.python.org/3/library/shutil.html#shutil.unpack_archive
>> need to be changed, with a note "Added in 3.6.2: filename can be any
>> pathlike object"?  If so, it is an enhancement.
> 
> Regardless of bugfix vs enhancement semantics, that seems like a good
> thing to do.

Enhancing public APIs in normal (non-provisional) modules in bugfix 
releases has turned out to be a bad thing to do.  Hence the policy to 
not do that.  The few exceptions have been necessary to fix a bug that 
needed to be fixed, and could not reasonably be fixed otherwise.

> [1] e.g. in this thread somewhere:
> https://mail.python.org/pipermail/python-ideas/2016-April/039827.html


-- 
Terry Jan Reedy


From songofacandy at gmail.com  Wed May  3 22:24:27 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Thu, 4 May 2017 11:24:27 +0900
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
Message-ID: <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>

Hi, Nick and all core devs who are interested in this PEP.

I'm reviewing PEP 538 and I want to accept it in this month.
It will reduces much UnicodeError pains which server-side OPs facing.
Thank you Nick for working on this PEP.

If you have something worrying about this PEP, please post a comment
soon.  If you don't have enough time to read entire this PEP, feel free to
ask a question about you're worrying.


Here is my comments:

>
> Relationship with other PEPs
> ============================
>
> This PEP shares a common problem statement with PEP 540 (improving Python
> 3's
> behaviour in the default C locale), but diverges markedly in the proposed
> solution:
>
> * PEP 540 proposes to entirely decouple CPython's default text encoding from
>   the C locale system in that case, allowing text handling inconsistencies to
>   arise between CPython and other locale-aware components running in the same
>   process and in subprocesses. This approach aims to make CPython behave less
>   like a locale-aware application, and more like locale-independent language
>   runtimes like the JVM, .NET CLR, Go, Node.js, and Rust

https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html says:

> Every instance of the Java virtual machine has a default charset, which may or may not be one of the standard charsets. The default charset is determined during virtual-machine startup and typically depends upon the locale and charset being used by the underlying operating system.

I don't know about .NET runtime on Unix much.  (mono and .NET Core).
"Go, Node.js and Rust" seems enough examples.



> New build-time configuration options
> ------------------------------------
>
> While both of the above behaviours would be enabled by default, they would
> also have new associated configuration options and preprocessor definitions
> for the benefit of redistributors that want to override those default
> settings.
>
> The locale coercion behaviour would be controlled by the flag
> ``--with[out]-c-locale-coercion``, which would set the
> ``PY_COERCE_C_LOCALE``
> preprocessor definition.
>
> The locale warning behaviour would be controlled by the flag
> ``--with[out]-c-locale-warning``, which would set the
> ``PY_WARN_ON_C_LOCALE``
> preprocessor definition.

"locale warning" means warning printed when C locale is used, am I right?

As my understanding, "locale warning" is shown in these cases (all cases implies
under C locale and PYTHONUTF8 is not enabled).

a. C locale is used and locale coercion is disabled by
   ``--without-c-locale-coercion`` configure option.
b. locale coercion is failed since there is none of C.UTF-8, C.utf8,
nor UTF-8 locale.
c. Python is embedded. locale coercion can't be used in this case.

In case of (b), while warning about C locale is not shown, warning
about coercion
is still shown.  So when people don't want to see warning under C
locale and there is no
(C.UTF-8, C.utf8, UTF-8) locales, there are three ways:

* Set PYTHONUTF=1 (if PEP 540 is accepted)
* Set PYTHONCOERCECLOCALE=0.
* Use both of ``--without-c-locale-coercion`` and ``--without-c-locale-warning``
  configure options.

Is my understanding right?

BTW, I prefer PEP 540 provides ``--with-utf8mode`` option which
enables UTF-8 mode
by default.  And if it is added, there are too few use cases for
``--without-c-locale-warning``.

There are some use cases people want to use UTF-8 by default in system
wide. (e.g.
container, webserver in Cent OS, etc...)

On the other hand, most of C locale usage are "per application" basis,
rather than "system wide."
configure option is not suitable for such per application setting, off course.

But I don't propose removing the option from PEP 538.
We can discuss about reducing configure options later.


>
> On platforms where they would have no effect (e.g. Mac OS X, iOS, Android,
> Windows) these preprocessor variables would always be undefined.
>

Why ``--with[out]-c-locale-coercion`` have no effect on macOS, iOS and Android?

On Android, locale coercion fixes readline.  Do you mean locale
coercion happen always
regardless this configuration option?

On macOS, ``LC_ALL=C python`` doesn't make Python's stdio to
``ascii:surrogateescape``?
Even so, locale coercion may fix libraries like readline, curses.
While C locale is less common on macOS, I don't understand any
reason to disable it on macOS.

I know almost nothing about iOS, but it's similar to Android or macOS
in my expectation.


> Improving the handling of the C locale
> --------------------------------------
>
...
> locale settings for locale-aware operations. Both the JVM and the .NET CLR
> use UTF-16-LE as their primary encoding for passing text between applications
> and the underlying platform.

JVM and .NET examples are misleading again.
They just use UTF-16-LE for syscall on Windows, like Python.

I don't know about them much, but I believe they don't use UTF-16 for system
encoding on Linux.


> Defaulting to "surrogateescape" error handling on the standard IO streams
> -------------------------------------------------------------------------
> By coercing the locale away from the legacy C default and its assumption of
> ASCII as the preferred text encoding, this PEP also disables the implicit use
> of the "surrogateescape" error handler on the standard IO streams that was
> introduced in Python 3.5 ([15_]), as well as the automatic use of
> ``surrogateescape`` when operating in PEP 540's UTF-8 mode.
>

I agree that this PEP shouldn't break byte transparent behavior in C locale by
coercing.
But I feel behavior difference between coerced C.UTF-8 locale and usual C.UTF-8
locale can be pitfall.

I read following part of the section and I agree that there is no way to solve
all issue.
But how about using surrogateescape handler in C.* locales like C locale?
It solves Python 3.7 subprocess under Python 3.7 with coerced C.UTF-8 locale
at least.

Anyway, I think https://bugs.python.org/issue15216 should be fixed in
Python 3.7 too.
Python applications which requires byte transparent stdio can use
`set_encoding(errors="surrogateescape")` explicitly.

Regards,

From k7hoven at gmail.com  Thu May  4 10:43:14 2017
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Thu, 4 May 2017 17:43:14 +0300
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <oedvi1$h2s$1@blaine.gmane.org>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <oedkaa$ckk$1@blaine.gmane.org>
 <CAMiohoieBthrQNpPJ71u0x4HirRSR7VH6dVvU20vZwsnRh6fWg@mail.gmail.com>
 <oedvi1$h2s$1@blaine.gmane.org>
Message-ID: <CAMiohojPvc=h4+3bMahfU1m5+ktttFXC3VXkjnzG4NDVgT93GQ@mail.gmail.com>

On Thu, May 4, 2017 at 4:19 AM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 5/3/2017 7:13 PM, Koos Zevenhoven wrote:
>>
[...]

>> Shutil was among the most important to be updated, IMO.
>>
>> I had made some sort of list of affected modules elsewhere [1]:
>> ntpath, posixpath, os.scandir, os.[other stuff], DirEntry (tempted to
>> say os.DirEntry, but that is
>> not true), shutil.[stuff], (io.)open, fileinput, filecmp, zipfile,
>> tarfile, tempfile (for the 'dir' keyword arguments), maybe even glob
>> and fnmatch (are the patterns paths?)
>
>
> What did not get done for 3.6 should be proposed for 3.7.
>

Anyone, feel free. The nightmare part is done, so this could be a case
where a PR actually pays off in terms of being able to use the
feature. There's no need for any unnecessary masochism (should there
ever be?).

[...]
>
> Enhancing public APIs in normal (non-provisional) modules in bugfix releases
> has turned out to be a bad thing to do.  Hence the policy to not do that.
> The few exceptions have been necessary to fix a bug that needed to be fixed,
> and could not reasonably be fixed otherwise.

Such exceptions can of course more easily be made when the adoption of
a version is still small, and almost all users will never see X.Y.0 or
X.Y.1. The fraction of 3.6 users is probably super tiny right now, and
even those users are likely to eagerly update to bugfix releases. For
instance, are there any major (LTS?) linux distros that already come
with 3.6.0 or 3.6.1? Well OK, 3.6.2 may be too late for some.

?Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +

From ncoghlan at gmail.com  Thu May  4 11:01:38 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 5 May 2017 01:01:38 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
Message-ID: <CADiSq7dyk24wfHP46feryXb91BL+ssXf7vdL4aBZxdPBOXJs6g@mail.gmail.com>

On 4 May 2017 at 12:24, INADA Naoki <songofacandy at gmail.com> wrote:
> [PEP 538]
>> * PEP 540 proposes to entirely decouple CPython's default text encoding from
>>   the C locale system in that case, allowing text handling inconsistencies to
>>   arise between CPython and other locale-aware components running in the same
>>   process and in subprocesses. This approach aims to make CPython behave less
>>   like a locale-aware application, and more like locale-independent language
>>   runtimes like the JVM, .NET CLR, Go, Node.js, and Rust
>
> https://docs.oracle.com/javase/7/docs/api/java/nio/charset/Charset.html says:
>
>> Every instance of the Java virtual machine has a default charset, which may or may not be one of the standard charsets. The default charset is determined during virtual-machine startup and typically depends upon the locale and charset being used by the underlying operating system.
>
> I don't know about .NET runtime on Unix much.  (mono and .NET Core).
> "Go, Node.js and Rust" seems enough examples.

I'll push an update to drop the JVM and .NET from the list of examples.

>> New build-time configuration options
>> ------------------------------------
[snip]
> In case of (b), while warning about C locale is not shown, warning
> about coercion
> is still shown.  So when people don't want to see warning under C
> locale and there is no
> (C.UTF-8, C.utf8, UTF-8) locales, there are three ways:
>
> * Set PYTHONUTF=1 (if PEP 540 is accepted)
> * Set PYTHONCOERCECLOCALE=0.
> * Use both of ``--without-c-locale-coercion`` and ``--without-c-locale-warning``
>   configure options.
>
> Is my understanding right?

Yes, that sounds right.

> BTW, I prefer PEP 540 provides ``--with-utf8mode`` option which
> enables UTF-8 mode
> by default.  And if it is added, there are too few use cases for
> ``--without-c-locale-warning``.
>
> There are some use cases people want to use UTF-8 by default in system
> wide. (e.g.
> container, webserver in Cent OS, etc...)
>
> On the other hand, most of C locale usage are "per application" basis,
> rather than "system wide."
> configure option is not suitable for such per application setting, off course.

Yeah, in addition to Barry requesting such an option in one of the
earlier linux-sig reviews, my main rationale for including it is that
providing both config options offers a quick compatibility fix for any
distro where emitting the coercion and/or C locale warning on stderr
causes problems.

The only one of those that Fedora encountered in the F26 alpha was
deemed a bug in the affected application (something in autotools was
checking for "no output on stderr" instead of "subprocess exit code is
0", and the fix was to switch it to check the subprocess exit code),
but there are enough Linux distros and BSD variants out there that I'm
a lot more comfortable shipping the change with straightforward "off"
switches for the stderr output.

> But I don't propose removing the option from PEP 538.
> We can discuss about reducing configure options later.

+1.

>> On platforms where they would have no effect (e.g. Mac OS X, iOS, Android,
>> Windows) these preprocessor variables would always be undefined.
>
> Why ``--with[out]-c-locale-coercion`` have no effect on macOS, iOS and Android?

On these three, we know the system encoding is UTF-8, so we never
interpreted the C locale as meaning "ascii" in the first place.

> On Android, locale coercion fixes readline.  Do you mean locale
> coercion happen always
> regardless this configuration option?

Right, the change for Android is that we switch to calling
'setlocale(LC_ALL, "C.UTF-8")' during interpreter startup instead of
'setlocale(LC_ALL, "")'. That change is guarded by "#ifdef
__ANDROID__", rather than either of the new conditionals.

> On macOS, ``LC_ALL=C python`` doesn't make Python's stdio to
> ``ascii:surrogateescape``?

Similar to Android, CPython itself is hardcoded to assume UTF-8 on Mac
OS X, since that's a platform API guarantee that users can't change.

> Even so, locale coercion may fix libraries like readline, curses.
> While C locale is less common on macOS, I don't understand any
> reason to disable it on macOS.

My understanding is that other libraries and applications also
automatically use UTF-8 for system interfaces on Mac OS X and iOS. It
could be that that understanding is wrong, and locale coercion would
provide a benefit there as well.

(Checking the draft implementation, it turns out I haven't actually
implemented the configure logic to make those config settings platform
dependent yet - they're currently only undefined on Windows by
default, since that doesn't use the autotools based build system)

>
> I know almost nothing about iOS, but it's similar to Android or macOS
> in my expectation.
>
>
>> Improving the handling of the C locale
>> --------------------------------------
>>
> ...
>> locale settings for locale-aware operations. Both the JVM and the .NET CLR
>> use UTF-16-LE as their primary encoding for passing text between applications
>> and the underlying platform.
>
> JVM and .NET examples are misleading again.
> They just use UTF-16-LE for syscall on Windows, like Python.
>
> I don't know about them much, but I believe they don't use UTF-16 for system
> encoding on Linux.

Sorry, this was ambiguous - it's meant to refer to applications
calling in to the JVM or CLR app runtime, not to the JVM or CLR
calling out to the host operating system. I'll try to make it clearer
in the next update.

>> Defaulting to "surrogateescape" error handling on the standard IO streams
>> -------------------------------------------------------------------------
>> By coercing the locale away from the legacy C default and its assumption of
>> ASCII as the preferred text encoding, this PEP also disables the implicit use
>> of the "surrogateescape" error handler on the standard IO streams that was
>> introduced in Python 3.5 ([15_]), as well as the automatic use of
>> ``surrogateescape`` when operating in PEP 540's UTF-8 mode.
>
> I agree that this PEP shouldn't break byte transparent behavior in C locale by
> coercing.
> But I feel behavior difference between coerced C.UTF-8 locale and usual C.UTF-8
> locale can be pitfall.
>
> I read following part of the section and I agree that there is no way to solve
> all issue.
> But how about using surrogateescape handler in C.* locales like C locale?

That would be entirely possible, as the code responsible for that
adjustment is the lines:

            char *loc = setlocale(LC_CTYPE, NULL);
            if (loc != NULL && strcmp(loc, "C") == 0)
                errors = "surrogateescape";

Changing that to include "C.UTF-8" as a second locale that also
implies the use of `surrogateescape` would be low risk, and means we
wouldn't need to call Py_SetStandardStreamEncoding.

As a result, non UTF-8 data (such as latin-1 or GB-18030) would
automatically round-trip, regardless of whether C.UTF-8 was explicitly
set as the locale, or reached as the result of locale coercion.

> It solves Python 3.7 subprocess under Python 3.7 with coerced C.UTF-8 locale
> at least.

It will also extend host/container encoding mismatch compatibility to
containers that explicitly set the C.UTF-8 locale.

That makes me more confident in making that change, as it would be
rather counterproductive if our changes gave base image developers an
incentive *not* to set C.UTF-8 as their default locale.

> Anyway, I think https://bugs.python.org/issue15216 should be fixed in
> Python 3.7 too.
> Python applications which requires byte transparent stdio can use
> `set_encoding(errors="surrogateescape")` explicitly.

Agreed.

Cheers,
Nick.

P.S. I've pushed the JVM/CLR related clarifications, but the standard
stream changes will require a bit more thought and corresponding
updates to the reference implementation - I'll aim to get to that this
weekend.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu May  4 11:13:15 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 5 May 2017 01:13:15 +1000
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
Message-ID: <CADiSq7f-439hu=6euAjn-bQb-+bfZ-ph7+PYfYurvwmd7wOP1A@mail.gmail.com>

On 4 May 2017 at 04:15, Brett Cannon <brett at python.org> wrote:
> My allergies have hit me hard so I'm not thinking at full capacity, but did
> we ever decide if supporting os.PathLike in the stdlib was viewed as an
> enhancement or bugfix? Specifically I'm thinking of
> https://bugs.python.org/issue30218 for adding support to
> shutil.unpack_archive() and whether it should be backported to 3.6.

I'd treat it as an enhancement, similar to adding native context
management support to various APIs. Otherwise, as Terry pointed out,
we're setting folks up for cases where testing on the latest
maintenance release seems fine, but their app breaks when deployed on
a slightly older version.

It's one thing for that to happen when the app is encountering a
genuine standard library bug, but something else entirely when they're
instead inadvertently relying on an enhancement that we could have
deferred to the next feature release.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From storchaka at gmail.com  Thu May  4 11:35:18 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 4 May 2017 18:35:18 +0300
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
Message-ID: <oefhnf$9tm$1@blaine.gmane.org>

On 03.05.17 21:15, Brett Cannon wrote:
> My allergies have hit me hard so I'm not thinking at full capacity, but
> did we ever decide if supporting os.PathLike in the stdlib was viewed as
> an enhancement or bugfix? Specifically I'm thinking
> of https://bugs.python.org/issue30218 for adding support to
> shutil.unpack_archive() and whether it should be backported to 3.6.

I concur with Terry, Victor and Nick.



From solipsis at pitrou.net  Thu May  4 12:25:57 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 4 May 2017 18:25:57 +0200
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
Message-ID: <20170504182557.066f5d9b@fsol>

On Thu, 4 May 2017 11:24:27 +0900
INADA Naoki <songofacandy at gmail.com> wrote:
> Hi, Nick and all core devs who are interested in this PEP.
> 
> I'm reviewing PEP 538 and I want to accept it in this month.
> It will reduces much UnicodeError pains which server-side OPs facing.
> Thank you Nick for working on this PEP.
> 
> If you have something worrying about this PEP, please post a comment
> soon.  If you don't have enough time to read entire this PEP, feel free to
> ask a question about you're worrying.

From my POV, it is problematic that the behaviour outlined in PEP 538
(see Abstract section) varies depending on the adoption of another PEP
(PEP 540).

If we want to adopt PEP 538 before pronouncing on PEP 540, then PEP 538
should remove all points conditional on PEP 540 adoption, and PEP 540
should later be changed to adopt those removed points as PEP
540-specific changes.

Regards

Antoine.



From victor.stinner at gmail.com  Thu May  4 13:13:32 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 4 May 2017 19:13:32 +0200
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
Message-ID: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>

Hi,

tl;dr Are you ok to backport my change replacing "make touch" with
"make regen-all"? (commit a5c62a8e)


Since the creation of CPython, generated files were regenerated
depending on file modification time. For development, that's a
convenient feature. But in practice, it caused a long list of pain
points. It caused me many issues in my experience:

* On Solaris, Python failed to regenerated the AST because only the
system Python was Python 2.6 and the script required Python 2.7 or
newer. The "make touch" workaround didn't help, also because of the
old version of the system Python.

* On FreeBSD, generated files require "python" but only "python2.7" or
"python3.6" programs are available. In The build system was enhanced
to try pythonX.Y and then "python".

* "make touch" workaround requires Mercurial, but also a specific
version of Mercurial: more than once, "make touch" failed because my
Mercurial version was too old.

* Since CPython migrated to Git, "make touch" doesn't work anymore.
Sorry, I didn't check why exactly, but I would prefer to not depend on
Git *and* Mercurial.


For all these reasons, it was decided to modify the CPython (UNIX/BSD)
build system to not regenerate generated files based on file
modification time anymore, but require an explicit action: "make
regen-all". I already pushed my change to the master branch:

https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5e102b

---
commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b
Author: Victor Stinner <victor.stinner at gmail.com>
Date:   Wed May 3 18:21:48 2017 +0200

bpo-23404: make touch becomes make regen-all (#1405)

Don't rebuild generated files based on file modification time
anymore, the action is now explicit. Replace "make touch"
with "make regen-all".

Changes:

* Remove "make touch", Tools/hg/hgtouch.py and .hgtouch
* Add a new "make regen-all" command to rebuild all generated files
* Add subcommands to only generate specific files:

  - regen-ast: Include/Python-ast.h and Python/Python-ast.c
  - regen-grammar: Include/graminit.h and Python/graminit.c
  - regen-importlib: Python/importlib_external.h and Python/importlib.h
  - regen-opcode: Include/opcode.h
  - regen-opcode-targets: Python/opcode_targets.h
  - regen-typeslots: Objects/typeslots.inc

* Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN
* pgen is now only built by by "make regen-grammar"
* Add $(srcdir)/ prefix to paths to source files to handle correctly
  compilation outside the source directory

Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make"
default target building Python.
---


See the issue for the full rationale:

   http://bugs.python.org/issue23404

My commit fixed the two remaining FreeBSD buildbots which were broken
because of broken "make touch". All FreeBSD buildbots are repaired on
master!

Ok, now the question is: are you ok to backport this change to Python
2.7, 3.5 and 3.6?

I started with a backport to 3.6:

https://github.com/python/cpython/pull/1461


See also "Test somehow that generated files are up to date: run make
regen-all" issue:
http://bugs.python.org/issue30259

Victor

From tjreedy at udel.edu  Thu May  4 13:30:10 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Thu, 4 May 2017 13:30:10 -0400
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAMiohojPvc=h4+3bMahfU1m5+ktttFXC3VXkjnzG4NDVgT93GQ@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <oedkaa$ckk$1@blaine.gmane.org>
 <CAMiohoieBthrQNpPJ71u0x4HirRSR7VH6dVvU20vZwsnRh6fWg@mail.gmail.com>
 <oedvi1$h2s$1@blaine.gmane.org>
 <CAMiohojPvc=h4+3bMahfU1m5+ktttFXC3VXkjnzG4NDVgT93GQ@mail.gmail.com>
Message-ID: <oefoeu$74h$1@blaine.gmane.org>

On 5/4/2017 10:43 AM, Koos Zevenhoven wrote:

> On Thu, May 4, 2017 at 4:19 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>> What did not get done for 3.6 should be proposed for 3.7.

> Anyone, feel free. The nightmare part is done, so this could be a case
> where a PR actually pays off in terms of being able to use the
> feature. There's no need for any unnecessary masochism (should there
> ever be?).

I have no idea what this means, but don't need to.

>> Enhancing public APIs in normal (non-provisional) modules in bugfix releases
>> has turned out to be a bad thing to do.  Hence the policy to not do that.
>> The few exceptions have been necessary to fix a bug that needed to be fixed,
>> and could not reasonably be fixed otherwise.
> 
> Such exceptions can of course more easily be made when the adoption of
> a version is still small, and almost all users will never see X.Y.0 or
> X.Y.1.

This is not an allowed excuse for breaking the policy.  The x.y language 
is defined when x.y.0 is released.  Please stop.


-- 
Terry Jan Reedy


From brett at python.org  Thu May  4 13:33:03 2017
From: brett at python.org (Brett Cannon)
Date: Thu, 04 May 2017 17:33:03 +0000
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <oedkaa$ckk$1@blaine.gmane.org>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <oedkaa$ckk$1@blaine.gmane.org>
Message-ID: <CAP1=2W6NB9Ai7+rbRFfa3GdsHtQsbh+omrPXODMJu+4zQ77XYQ@mail.gmail.com>

Thanks for the feedback, everyone. I'll just consider it an enhancement
then.

On Wed, 3 May 2017 at 15:07 Terry Reedy <tjreedy at udel.edu> wrote:

> On 5/3/2017 2:15 PM, Brett Cannon wrote:
> > My allergies have hit me hard so I'm not thinking at full capacity, but
> > did we ever decide if supporting os.PathLike in the stdlib was viewed as
> > an enhancement or bugfix? Specifically I'm thinking of
> > https://bugs.python.org/issue30218 for adding support to
> > shutil.unpack_archive() and whether it should be backported to 3.6.
>
> On the face of it, that particular issue looks like an enhancement that
> should have gone into 3.6 (if ever), but did not.  I notice that
> https://www.python.org/dev/peps/pep-0519/#implementation
> did not include "Update shutil", so it was not done, at least not
> completely.
>
> Was shutil updated at all?


Probably not explicitly. A lot of support just fell through thanks to
updating os and os.path to support path-like objects.


>   Is unpack_archive the only shutil function
> not updated?


Not sure. I opened https://bugs.python.org/issue30235 to track finding out.

-Brett

  If so, I could see the omission as a bug.


> If the patch for 30218 were applied in 3.6, would the doc
> https://docs.python.org/3/library/shutil.html#shutil.unpack_archive
> need to be changed, with a note "Added in 3.6.2: filename can be any
> pathlike object"?  If so, it is an enhancement.
>
> --
> Terry Jan Reedy
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170504/6fced0bf/attachment.html>

From brett at python.org  Thu May  4 13:40:38 2017
From: brett at python.org (Brett Cannon)
Date: Thu, 04 May 2017 17:40:38 +0000
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
In-Reply-To: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
References: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
Message-ID: <CAP1=2W4yy8ZZ8yEBnJkx6sg=Bj8N6rmL1+eu1ZnAOtNv-UQd1w@mail.gmail.com>

I see no issue backporting since I don't think we have any compatibility
promises when it comes to Makefile commands. Plus if the perf changes to
add PGO support could be backported then I don't see why this shouldn't be
allowed.

On Thu, 4 May 2017 at 10:15 Victor Stinner <victor.stinner at gmail.com> wrote:

> Hi,
>
> tl;dr Are you ok to backport my change replacing "make touch" with
> "make regen-all"? (commit a5c62a8e)
>
>
> Since the creation of CPython, generated files were regenerated
> depending on file modification time. For development, that's a
> convenient feature. But in practice, it caused a long list of pain
> points. It caused me many issues in my experience:
>
> * On Solaris, Python failed to regenerated the AST because only the
> system Python was Python 2.6 and the script required Python 2.7 or
> newer. The "make touch" workaround didn't help, also because of the
> old version of the system Python.
>
> * On FreeBSD, generated files require "python" but only "python2.7" or
> "python3.6" programs are available. In The build system was enhanced
> to try pythonX.Y and then "python".
>
> * "make touch" workaround requires Mercurial, but also a specific
> version of Mercurial: more than once, "make touch" failed because my
> Mercurial version was too old.
>
> * Since CPython migrated to Git, "make touch" doesn't work anymore.
> Sorry, I didn't check why exactly, but I would prefer to not depend on
> Git *and* Mercurial.
>
>
> For all these reasons, it was decided to modify the CPython (UNIX/BSD)
> build system to not regenerate generated files based on file
> modification time anymore, but require an explicit action: "make
> regen-all". I already pushed my change to the master branch:
>
>
> https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5e102b
>
> ---
> commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b
> Author: Victor Stinner <victor.stinner at gmail.com>
> Date:   Wed May 3 18:21:48 2017 +0200
>
> bpo-23404: make touch becomes make regen-all (#1405)
>
> Don't rebuild generated files based on file modification time
> anymore, the action is now explicit. Replace "make touch"
> with "make regen-all".
>
> Changes:
>
> * Remove "make touch", Tools/hg/hgtouch.py and .hgtouch
> * Add a new "make regen-all" command to rebuild all generated files
> * Add subcommands to only generate specific files:
>
>   - regen-ast: Include/Python-ast.h and Python/Python-ast.c
>   - regen-grammar: Include/graminit.h and Python/graminit.c
>   - regen-importlib: Python/importlib_external.h and Python/importlib.h
>   - regen-opcode: Include/opcode.h
>   - regen-opcode-targets: Python/opcode_targets.h
>   - regen-typeslots: Objects/typeslots.inc
>
> * Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN
> * pgen is now only built by by "make regen-grammar"
> * Add $(srcdir)/ prefix to paths to source files to handle correctly
>   compilation outside the source directory
>
> Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make"
> default target building Python.
> ---
>
>
> See the issue for the full rationale:
>
>    http://bugs.python.org/issue23404
>
> My commit fixed the two remaining FreeBSD buildbots which were broken
> because of broken "make touch". All FreeBSD buildbots are repaired on
> master!
>
> Ok, now the question is: are you ok to backport this change to Python
> 2.7, 3.5 and 3.6?
>
> I started with a backport to 3.6:
>
> https://github.com/python/cpython/pull/1461
>
>
> See also "Test somehow that generated files are up to date: run make
> regen-all" issue:
> http://bugs.python.org/issue30259
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170504/172a768e/attachment.html>

From raymond.hettinger at gmail.com  Thu May  4 13:51:16 2017
From: raymond.hettinger at gmail.com (Raymond Hettinger)
Date: Thu, 4 May 2017 10:51:16 -0700
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
In-Reply-To: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
References: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
Message-ID: <44E5B9AC-7FCE-4A7E-AE1E-E6676CCD7941@gmail.com>

Yes.  It is perfectly reasonable to backport improvements to the tooling as long as it doesn't break anyone's existing build process. 

Sent from my iPhone

> On May 4, 2017, at 10:13 AM, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> tl;dr Are you ok to backport my change replacing "make touch" with
> "make regen-all"? (commit a5c62a8e)

From berker.peksag at gmail.com  Thu May  4 14:01:04 2017
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Thu, 4 May 2017 21:01:04 +0300
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
Message-ID: <CAF4280JydEmE62wB4Hpt-Pn8oRt84UczYnKQHE9AUMW_dvrHvg@mail.gmail.com>

On Wed, May 3, 2017 at 9:15 PM, Brett Cannon <brett at python.org> wrote:
> My allergies have hit me hard so I'm not thinking at full capacity, but did
> we ever decide if supporting os.PathLike in the stdlib was viewed as an
> enhancement or bugfix? Specifically I'm thinking of
> https://bugs.python.org/issue30218 for adding support to
> shutil.unpack_archive() and whether it should be backported to 3.6.

We've already backported a few patches that improves the PEP 519
support in the stdlib with the permission from the release manager of
3.6. I'd ask Ned whether bpo-30218 qualifies for backporting to 3.6.

--Berker

From storchaka at gmail.com  Thu May  4 14:36:48 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Thu, 4 May 2017 21:36:48 +0300
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAF4280JydEmE62wB4Hpt-Pn8oRt84UczYnKQHE9AUMW_dvrHvg@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAF4280JydEmE62wB4Hpt-Pn8oRt84UczYnKQHE9AUMW_dvrHvg@mail.gmail.com>
Message-ID: <oefsbr$17e$1@blaine.gmane.org>

On 04.05.17 21:01, Berker Peksa? wrote:
> We've already backported a few patches that improves the PEP 519
> support in the stdlib with the permission from the release manager of
> 3.6. I'd ask Ned whether bpo-30218 qualifies for backporting to 3.6.

AFAIK it was before releasing 3.6.1. Some users avoid using x.0 versions 
of software. For 3.6.2 we can have stronger rule.



From rymg19 at gmail.com  Thu May  4 16:38:41 2017
From: rymg19 at gmail.com (Ryan Gonzalez)
Date: Thu, 4 May 2017 15:38:41 -0500
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
In-Reply-To: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
References: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
Message-ID: <CAO41-mOFSnOdUK24JOXMd-aSExGQFGZQ_2wDAB+jBJ=7xEYkHw@mail.gmail.com>

FWIW this will also make cross-compiling a lot easier, since you can't
accidentally overwrite the cross-compiled pgen as easily.

On Thu, May 4, 2017 at 12:13 PM, Victor Stinner
<victor.stinner at gmail.com> wrote:
> Hi,
>
> tl;dr Are you ok to backport my change replacing "make touch" with
> "make regen-all"? (commit a5c62a8e)
>
>
> Since the creation of CPython, generated files were regenerated
> depending on file modification time. For development, that's a
> convenient feature. But in practice, it caused a long list of pain
> points. It caused me many issues in my experience:
>
> * On Solaris, Python failed to regenerated the AST because only the
> system Python was Python 2.6 and the script required Python 2.7 or
> newer. The "make touch" workaround didn't help, also because of the
> old version of the system Python.
>
> * On FreeBSD, generated files require "python" but only "python2.7" or
> "python3.6" programs are available. In The build system was enhanced
> to try pythonX.Y and then "python".
>
> * "make touch" workaround requires Mercurial, but also a specific
> version of Mercurial: more than once, "make touch" failed because my
> Mercurial version was too old.
>
> * Since CPython migrated to Git, "make touch" doesn't work anymore.
> Sorry, I didn't check why exactly, but I would prefer to not depend on
> Git *and* Mercurial.
>
>
> For all these reasons, it was decided to modify the CPython (UNIX/BSD)
> build system to not regenerate generated files based on file
> modification time anymore, but require an explicit action: "make
> regen-all". I already pushed my change to the master branch:
>
> https://github.com/python/cpython/commit/a5c62a8e9f0de6c4133825a5710984a3cd5e102b
>
> ---
> commit a5c62a8e9f0de6c4133825a5710984a3cd5e102b
> Author: Victor Stinner <victor.stinner at gmail.com>
> Date:   Wed May 3 18:21:48 2017 +0200
>
> bpo-23404: make touch becomes make regen-all (#1405)
>
> Don't rebuild generated files based on file modification time
> anymore, the action is now explicit. Replace "make touch"
> with "make regen-all".
>
> Changes:
>
> * Remove "make touch", Tools/hg/hgtouch.py and .hgtouch
> * Add a new "make regen-all" command to rebuild all generated files
> * Add subcommands to only generate specific files:
>
>   - regen-ast: Include/Python-ast.h and Python/Python-ast.c
>   - regen-grammar: Include/graminit.h and Python/graminit.c
>   - regen-importlib: Python/importlib_external.h and Python/importlib.h
>   - regen-opcode: Include/opcode.h
>   - regen-opcode-targets: Python/opcode_targets.h
>   - regen-typeslots: Objects/typeslots.inc
>
> * Rename PYTHON_FOR_GEN to PYTHON_FOR_REGEN
> * pgen is now only built by by "make regen-grammar"
> * Add $(srcdir)/ prefix to paths to source files to handle correctly
>   compilation outside the source directory
>
> Note: $(PYTHON_FOR_REGEN) is no more used nor needed by "make"
> default target building Python.
> ---
>
>
> See the issue for the full rationale:
>
>    http://bugs.python.org/issue23404
>
> My commit fixed the two remaining FreeBSD buildbots which were broken
> because of broken "make touch". All FreeBSD buildbots are repaired on
> master!
>
> Ok, now the question is: are you ok to backport this change to Python
> 2.7, 3.5 and 3.6?
>
> I started with a backport to 3.6:
>
> https://github.com/python/cpython/pull/1461
>
>
> See also "Test somehow that generated files are up to date: run make
> regen-all" issue:
> http://bugs.python.org/issue30259
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com



-- 
Ryan (????)
Yoko Shimomura > ryo (supercell/EGOIST) > Hiroyuki Sawano >> everyone else
http://refi64.com/

From k7hoven at gmail.com  Thu May  4 17:11:33 2017
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Fri, 5 May 2017 00:11:33 +0300
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <oefoeu$74h$1@blaine.gmane.org>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <oedkaa$ckk$1@blaine.gmane.org>
 <CAMiohoieBthrQNpPJ71u0x4HirRSR7VH6dVvU20vZwsnRh6fWg@mail.gmail.com>
 <oedvi1$h2s$1@blaine.gmane.org>
 <CAMiohojPvc=h4+3bMahfU1m5+ktttFXC3VXkjnzG4NDVgT93GQ@mail.gmail.com>
 <oefoeu$74h$1@blaine.gmane.org>
Message-ID: <CAMiohog4Rw+R3zptPZO0LdpyGS6n5w=bNGB9eA96KxCKnFnKCw@mail.gmail.com>

On Thu, May 4, 2017 at 8:30 PM, Terry Reedy <tjreedy at udel.edu> wrote:
> On 5/4/2017 10:43 AM, Koos Zevenhoven wrote:
>> On Thu, May 4, 2017 at 4:19 AM, Terry Reedy <tjreedy at udel.edu> wrote:
>>> Enhancing public APIs in normal (non-provisional) modules in bugfix
>>> releases
>>> has turned out to be a bad thing to do.  Hence the policy to not do that.
>>> The few exceptions have been necessary to fix a bug that needed to be
>>> fixed,
>>> and could not reasonably be fixed otherwise.
>>
>> Such exceptions can of course more easily be made when the adoption of
>> a version is still small, and almost all users will never see X.Y.0 or
>> X.Y.1.
>
> This is not an allowed excuse for breaking the policy.  The x.y language is
> defined when x.y.0 is released.  Please stop.
>

Don't worry, I didn't even start :)

I do think it can cause problems if most of a stdlib module supports
PathLike and some parts do not. People will write code in 3.7
believing it works on 3.6, while it doesn't. Anyway, I'm completely
happy if the policy outweighs this issue, and I have absolutely no
need to argue about the decision.

?Koos


-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +

From a.badger at gmail.com  Thu May  4 21:24:13 2017
From: a.badger at gmail.com (Toshio Kuratomi)
Date: Thu, 4 May 2017 18:24:13 -0700
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
Message-ID: <CABVPEKoKG-D5gGhc0OXjQAThBoqRbEwvoUcfBKUjh99BLjWYbQ@mail.gmail.com>

On Sat, Mar 4, 2017 at 11:50 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>
> Providing implicit locale coercion only when running standalone
> ---------------------------------------------------------------
>
> Over the course of Python 3.x development, multiple attempts have been made
> to improve the handling of incorrect locale settings at the point where the
> Python interpreter is initialised. The problem that emerged is that this is
> ultimately *too late* in the interpreter startup process - data such as
> command
> line arguments and the contents of environment variables may have already
> been
> retrieved from the operating system and processed under the incorrect ASCII
> text encoding assumption well before ``Py_Initialize`` is called.
>
> The problems created by those inconsistencies were then even harder to
> diagnose
> and debug than those created by believing the operating system's claim that
> ASCII was a suitable encoding to use for operating system interfaces. This
> was
> the case even for the default CPython binary, let alone larger C/C++
> applications that embed CPython as a scripting engine.
>
> The approach proposed in this PEP handles that problem by moving the locale
> coercion as early as possible in the interpreter startup sequence when
> running
> standalone: it takes place directly in the C-level ``main()`` function, even
> before calling in to the `Py_Main()`` library function that implements the
> features of the CPython interpreter CLI.
>
> The ``Py_Initialize`` API then only gains an explicit warning (emitted on
> ``stderr``) when it detects use of the ``C`` locale, and relies on the
> embedding application to specify something more reasonable.
>

It feels like having a short section on the caveats of this approach
would help to introduce this section.  Something that says that this
PEP can cause a split in how Python behaves in non-sandalone
applications (mod_wsgi, IDEs where libpython is compiled in, etc) vs
standalone (unless the embedders take similar steps as standalone
python is doing).  Then go on to state that this approach was still
chosen as coercing in Py_Initialize is too late, causing the
inconsistencies and problems listed here.

-Toshio

From ncoghlan at gmail.com  Fri May  5 00:31:19 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 5 May 2017 14:31:19 +1000
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
In-Reply-To: <CAP1=2W4yy8ZZ8yEBnJkx6sg=Bj8N6rmL1+eu1ZnAOtNv-UQd1w@mail.gmail.com>
References: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
 <CAP1=2W4yy8ZZ8yEBnJkx6sg=Bj8N6rmL1+eu1ZnAOtNv-UQd1w@mail.gmail.com>
Message-ID: <CADiSq7fua8Uibvrgi1cN-ntGg+oNtc7NFfgSaU7xfbrLRnsxdQ@mail.gmail.com>

On 5 May 2017 at 03:40, Brett Cannon <brett at python.org> wrote:
> I see no issue backporting since I don't think we have any compatibility
> promises when it comes to Makefile commands. Plus if the perf changes to add
> PGO support could be backported then I don't see why this shouldn't be
> allowed.

For the benefit of Linux distros attempting to ensure they're doing
full "from source" builds, it would be good to note this in a "Notable
changes in maintenance releases", akin to the existing ones for 3.4
and 2.7 (perhaps retitling the latter accordingly).

The note just needs to say that folks that care about doing "complete"
builds need to adjust their command sequence to be "./configure
<options> && make regen-all && make install", rather than the previous
pattern of "./configure <options> && make install".

This should also make bootstrapping easier, since bootstrap builds can
skip the "make regen-all" step and instead rely on the checked in
pre-generated files.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri May  5 00:46:19 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 5 May 2017 14:46:19 +1000
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
Message-ID: <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>

On 4 May 2017 at 08:25, Victor Stinner <victor.stinner at gmail.com> wrote:
> If you start to backport support for the fspath protocol, be prepared
> to have to backport it *many* places. I expect that slowly in the near
> future, many functions will be patched to support the fspath protocol.
>
> I suggest to only do that in master. It's not that hard to cast
> manually to a string: it's just str(path), no?

For 3.6+, "os.fspath(path)" would be more robust - unlike str(), it
will still error out immediately if you pass it something completely
inappropriate.

That improved casting mechanism and the implicit support in the low
level APIs is the main benefit I see in PEP 519, and if we were
talking about an os module API that was missing os.path support, I'd
be more likely to be on the "it's a bug" side of things.

It's only higher level APIs like shutil that I put into the same
category as third party libraries for now: Python 3.6 users shouldn't
expect implicit use of the fspath protocol to be universal yet, so
they may still need some explicit calls to os.fspath() in their own
code to get different APIs to work well together.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Fri May  5 00:57:53 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 5 May 2017 14:57:53 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <20170504182557.066f5d9b@fsol>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <20170504182557.066f5d9b@fsol>
Message-ID: <CADiSq7d3RWhcnVQ5-z-qNKN8fTM6RzfEakPD0m1ZVjn1JhEYuA@mail.gmail.com>

On 5 May 2017 at 02:25, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Thu, 4 May 2017 11:24:27 +0900
> INADA Naoki <songofacandy at gmail.com> wrote:
>> Hi, Nick and all core devs who are interested in this PEP.
>>
>> I'm reviewing PEP 538 and I want to accept it in this month.
>> It will reduces much UnicodeError pains which server-side OPs facing.
>> Thank you Nick for working on this PEP.
>>
>> If you have something worrying about this PEP, please post a comment
>> soon.  If you don't have enough time to read entire this PEP, feel free to
>> ask a question about you're worrying.
>
> From my POV, it is problematic that the behaviour outlined in PEP 538
> (see Abstract section) varies depending on the adoption of another PEP
> (PEP 540).
>
> If we want to adopt PEP 538 before pronouncing on PEP 540, then PEP 538
> should remove all points conditional on PEP 540 adoption, and PEP 540
> should later be changed to adopt those removed points as PEP
> 540-specific changes.

While I won't be certain until I update the PEP and reference
implementation, I'm pretty sure Inada-san's suggestion to replace the
call to Py_SetStandardStreamEncoding with defaulting to
surrogateescape on the standard streams in the C.UTF-8 locale will
remove this current dependency between the PEPs as well as making the
"C.UTF-8 locale" and "C locale coerced to C.UTF-8" behaviour
indistinguishable at runtime (aside from the stderr warning in the
latter case).

It will then be up to Victor to state in PEP 540 how locale coercion
will interact with Python UTF-8 mode (with my recommendation being the
one currently in PEP 538: it should implicitly set the environment
variable, so the mode activation is inherited by subprocesses)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Fri May  5 02:01:07 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 5 May 2017 08:01:07 +0200
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
In-Reply-To: <CADiSq7fua8Uibvrgi1cN-ntGg+oNtc7NFfgSaU7xfbrLRnsxdQ@mail.gmail.com>
References: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
 <CAP1=2W4yy8ZZ8yEBnJkx6sg=Bj8N6rmL1+eu1ZnAOtNv-UQd1w@mail.gmail.com>
 <CADiSq7fua8Uibvrgi1cN-ntGg+oNtc7NFfgSaU7xfbrLRnsxdQ@mail.gmail.com>
Message-ID: <CAMpsgwYuuT+_BweYCfS1u8qVNaNS7=x5Zg4spdGXG7fTWQxZ0g@mail.gmail.com>

Le 5 mai 2017 6:31 AM, "Nick Coghlan" <ncoghlan at gmail.com> a ?crit :

The note just needs to say that folks that care about doing "complete"
builds need to adjust their command sequence to be "./configure
<options> && make regen-all && make install", rather than the previous
pattern of "./configure <options> && make install".


Hum, you only need to regenerate files if you modify Grammar, AST or
opcodes. Do you mean that Linux distro make such downstream changes?

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170505/b6a62da0/attachment.html>

From ncoghlan at gmail.com  Fri May  5 02:39:17 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 5 May 2017 16:39:17 +1000
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
In-Reply-To: <CAMpsgwYuuT+_BweYCfS1u8qVNaNS7=x5Zg4spdGXG7fTWQxZ0g@mail.gmail.com>
References: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
 <CAP1=2W4yy8ZZ8yEBnJkx6sg=Bj8N6rmL1+eu1ZnAOtNv-UQd1w@mail.gmail.com>
 <CADiSq7fua8Uibvrgi1cN-ntGg+oNtc7NFfgSaU7xfbrLRnsxdQ@mail.gmail.com>
 <CAMpsgwYuuT+_BweYCfS1u8qVNaNS7=x5Zg4spdGXG7fTWQxZ0g@mail.gmail.com>
Message-ID: <CADiSq7cUM+mBh+wtMhUoGqoQn_sgcZ0mxmAyvCFhMhn8+ty+vQ@mail.gmail.com>

On 5 May 2017 at 16:01, Victor Stinner <victor.stinner at gmail.com> wrote:
> Le 5 mai 2017 6:31 AM, "Nick Coghlan" <ncoghlan at gmail.com> a ?crit :
>
> The note just needs to say that folks that care about doing "complete"
> builds need to adjust their command sequence to be "./configure
> <options> && make regen-all && make install", rather than the previous
> pattern of "./configure <options> && make install".
>
>
> Hum, you only need to regenerate files if you modify Grammar, AST or
> opcodes. Do you mean that Linux distro make such downstream changes?

No, but at least some of them do have a policy that it be possible to
rebuild the project from its "original sources" in the distro's build
system without relying on cached artifacts provided by the upstream
project. In CPython's case, that means distros with such a policy
should ensure they're running the "regen-all" step in addition to the
C build steps.

To be 100% clear, I agree it makes sense to change the default
behaviour to be to re-use our checked in generated files, and to make
that policy consistent across all active branches.

The request for a note is just to make it clear to redistributors that
CPython now has two potential tiers of build dependencies (those
required to run "make" and those required to run "make regen-all"), so
if a redistributor cares about that kind of thing, they're going to
have to update their build processes accordingly. If they don't make
they change, they may inadvertently lose the ability to run the
"regen-all" step at some point in the future (as an extreme example:
to avoid a tricky bootstrapping problem, while still improving the
tools available for standard library maintenance, we may someday
decide to allow cffi and/or cython as file regeneration dependencies,
while continuing to disallow them as conventional build dependencies).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Fri May  5 03:21:14 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 5 May 2017 09:21:14 +0200
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
In-Reply-To: <CADiSq7fua8Uibvrgi1cN-ntGg+oNtc7NFfgSaU7xfbrLRnsxdQ@mail.gmail.com>
References: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
 <CAP1=2W4yy8ZZ8yEBnJkx6sg=Bj8N6rmL1+eu1ZnAOtNv-UQd1w@mail.gmail.com>
 <CADiSq7fua8Uibvrgi1cN-ntGg+oNtc7NFfgSaU7xfbrLRnsxdQ@mail.gmail.com>
Message-ID: <CAMpsgwbe+RBBN7+rX-dokRATEAd7Y3ETqB1UL98iNHF2pe6E-Q@mail.gmail.com>

2017-05-05 6:31 GMT+02:00 Nick Coghlan <ncoghlan at gmail.com>:
> For the benefit of Linux distros attempting to ensure they're doing
> full "from source" builds, it would be good to note this in a "Notable
> changes in maintenance releases", akin to the existing ones for 3.4
> and 2.7 (perhaps retitling the latter accordingly).

Sorry, I'm not aware of these notes. Where can I found them? Where
should we document the build system change?

You may comment the issue directly: http://bugs.python.org/issue23404

Victor

From erik.m.bray at gmail.com  Fri May  5 05:45:27 2017
From: erik.m.bray at gmail.com (Erik Bray)
Date: Fri, 5 May 2017 11:45:27 +0200
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <20170504182557.066f5d9b@fsol>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <20170504182557.066f5d9b@fsol>
Message-ID: <CAOTD34a-hqOsXbAGpNtkHZDyn0fGWLn4-mnLZv0KjXEpL8YV8w@mail.gmail.com>

On Thu, May 4, 2017 at 6:25 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Thu, 4 May 2017 11:24:27 +0900
> INADA Naoki <songofacandy at gmail.com> wrote:
>> Hi, Nick and all core devs who are interested in this PEP.
>>
>> I'm reviewing PEP 538 and I want to accept it in this month.
>> It will reduces much UnicodeError pains which server-side OPs facing.
>> Thank you Nick for working on this PEP.
>>
>> If you have something worrying about this PEP, please post a comment
>> soon.  If you don't have enough time to read entire this PEP, feel free to
>> ask a question about you're worrying.
>
> From my POV, it is problematic that the behaviour outlined in PEP 538
> (see Abstract section) varies depending on the adoption of another PEP
> (PEP 540).
>
> If we want to adopt PEP 538 before pronouncing on PEP 540, then PEP 538
> should remove all points conditional on PEP 540 adoption, and PEP 540
> should later be changed to adopt those removed points as PEP
> 540-specific changes.

This is kind of an aside, but regardless of the dependency
relationship between PEP 538 and 540, given that they kind of go
hand-in-hand would it make sense to rename them--e.g. have PEP 539 and
PEP 540 trade places, since PEP 539 has nothing to do with this and is
awkwardly nestled between them.  Or would that only confuse matters at
this point?

Thanks,
Erik

From ncoghlan at gmail.com  Fri May  5 06:34:33 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 5 May 2017 20:34:33 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CAOTD34a-hqOsXbAGpNtkHZDyn0fGWLn4-mnLZv0KjXEpL8YV8w@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <20170504182557.066f5d9b@fsol>
 <CAOTD34a-hqOsXbAGpNtkHZDyn0fGWLn4-mnLZv0KjXEpL8YV8w@mail.gmail.com>
Message-ID: <CADiSq7ddJXTgaD8TRJ=ZqeKrYqge-fAGRVS2yq_BinA1mG4e_w@mail.gmail.com>

On 5 May 2017 at 19:45, Erik Bray <erik.m.bray at gmail.com> wrote:
> On Thu, May 4, 2017 at 6:25 PM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> If we want to adopt PEP 538 before pronouncing on PEP 540, then PEP 538
>> should remove all points conditional on PEP 540 adoption, and PEP 540
>> should later be changed to adopt those removed points as PEP
>> 540-specific changes.
>
> This is kind of an aside, but regardless of the dependency
> relationship between PEP 538 and 540, given that they kind of go
> hand-in-hand would it make sense to rename them--e.g. have PEP 539 and
> PEP 540 trade places, since PEP 539 has nothing to do with this and is
> awkwardly nestled between them.  Or would that only confuse matters at
> this point?

While we have renumbered PEPs in the past, it was only in cases where
the PEPs were relatively new, so there weren't many discussions
referencing them under their existing numbers.

In this case, both PEP 539 and 540 have already been discussed
extensively, so renumbering them would cause problems without
providing any corresponding benefit (Python's development is
sufficiently high volume that it isn't unusual for related PEPs to
have non-sequential PEP numbers)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From songofacandy at gmail.com  Fri May  5 09:21:47 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Fri, 5 May 2017 22:21:47 +0900
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <20170504182557.066f5d9b@fsol>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <20170504182557.066f5d9b@fsol>
Message-ID: <CAEfz+TwcR7wbr71SQ-H+s9nd+Z6J3yXV1zr4Cyrm=b5OCTA_bw@mail.gmail.com>

On Fri, May 5, 2017 at 1:25 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Thu, 4 May 2017 11:24:27 +0900
> INADA Naoki <songofacandy at gmail.com> wrote:
>> Hi, Nick and all core devs who are interested in this PEP.
>>
>> I'm reviewing PEP 538 and I want to accept it in this month.
>> It will reduces much UnicodeError pains which server-side OPs facing.
>> Thank you Nick for working on this PEP.
>>
>> If you have something worrying about this PEP, please post a comment
>> soon.  If you don't have enough time to read entire this PEP, feel free to
>> ask a question about you're worrying.
>
> From my POV, it is problematic that the behaviour outlined in PEP 538
> (see Abstract section) varies depending on the adoption of another PEP
> (PEP 540).
>
> If we want to adopt PEP 538 before pronouncing on PEP 540, then PEP 538
> should remove all points conditional on PEP 540 adoption, and PEP 540
> should later be changed to adopt those removed points as PEP
> 540-specific changes.
>
> Regards
>
> Antoine.
>

Fair enough.  I stop hurrying about PEP 538 and start reviewing PEP 540.

Thanks,

From ncoghlan at gmail.com  Fri May  5 10:12:54 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 6 May 2017 00:12:54 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CAEfz+TwcR7wbr71SQ-H+s9nd+Z6J3yXV1zr4Cyrm=b5OCTA_bw@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <20170504182557.066f5d9b@fsol>
 <CAEfz+TwcR7wbr71SQ-H+s9nd+Z6J3yXV1zr4Cyrm=b5OCTA_bw@mail.gmail.com>
Message-ID: <CADiSq7cVrWL2t1Ltmz32dRfTPvuJhB6w8XUB3SC4yoyrCQehrA@mail.gmail.com>

On 5 May 2017 at 23:21, INADA Naoki <songofacandy at gmail.com> wrote:
> On Fri, May 5, 2017 at 1:25 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> If we want to adopt PEP 538 before pronouncing on PEP 540, then PEP 538
>> should remove all points conditional on PEP 540 adoption, and PEP 540
>> should later be changed to adopt those removed points as PEP
>> 540-specific changes.
>
> Fair enough.  I stop hurrying about PEP 538 and start reviewing PEP 540.

Don't forget that Victor's still working on the design of PEP 540, so
it isn't ready for pronouncement yet.

Antoine's request was for me to update PEP *538* to eliminate the
"this will need to change if PEP 540 is accepted" aspects, and I think
your suggestion to make the "C.UTF-8 -> surrogateescape on standard
streams by default" behaviour independent of the locale coercion will
achieve that.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From status at bugs.python.org  Fri May  5 12:09:12 2017
From: status at bugs.python.org (Python tracker)
Date: Fri,  5 May 2017 18:09:12 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20170505160912.99ED256B0E@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2017-04-28 - 2017-05-05)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5929 (+21)
  closed 36109 (+61)
  total  42038 (+82)

Open issues with patches: 2384 


Issues opened (58)
==================

#29094: Regression in zipfile writing in 2.7.13
http://bugs.python.org/issue29094  reopened by gregory.p.smith

#30166: Import command-line parsing modules only when needed
http://bugs.python.org/issue30166  reopened by terry.reedy

#30202: Update test.test_importlib.test_abc to test find_spec()
http://bugs.python.org/issue30202  opened by brett.cannon

#30203: AttributeError in Popen.communicate()
http://bugs.python.org/issue30203  opened by Luke Campagnola

#30210: No Documentation on tkinter dnd module
http://bugs.python.org/issue30210  opened by anthony-flury

#30211: Bdb: add docstrings
http://bugs.python.org/issue30211  opened by csabella

#30212: test_ssl.py is broken in Centos7
http://bugs.python.org/issue30212  opened by david-cpi

#30213: ZipFile from 'a'ppend-mode file generates invalid zip
http://bugs.python.org/issue30213  opened by BoppreH

#30214: make_zip.py lacks command a few line options.
http://bugs.python.org/issue30214  opened by Decorater

#30216: xdrlib.Unpacker.unpack_string returns bytes (docs say should b
http://bugs.python.org/issue30216  opened by rnash

#30217: Missing entry for the tilde (~) operator in the Index
http://bugs.python.org/issue30217  opened by lebigot

#30218: shutil.unpack_archive doesn't support PathLike
http://bugs.python.org/issue30218  opened by Jelle Zijlstra

#30219: webbrowser.open outputs xdg-open deprecation warning
http://bugs.python.org/issue30219  opened by desbma

#30220: Why are the custom messages for ValueError and TypeError suppr
http://bugs.python.org/issue30220  opened by pgacv2

#30222: make_zip.py had a bug when setting up full 64 bit distribution
http://bugs.python.org/issue30222  opened by Decorater

#30223: Add Lib/test/__main__.py in 2.7
http://bugs.python.org/issue30223  opened by serhiy.storchaka

#30224: remove outdated checks in struct
http://bugs.python.org/issue30224  opened by xiang.zhang

#30226: Modernize make_ssl_certs
http://bugs.python.org/issue30226  opened by christian.heimes

#30227: test_site must not write outside the build directory: must not
http://bugs.python.org/issue30227  opened by haypo

#30228: Open a file in text mode requires too many syscalls
http://bugs.python.org/issue30228  opened by haypo

#30229: Closing a BufferedRandom calls lseek() mutliple times
http://bugs.python.org/issue30229  opened by haypo

#30230: Move quick test in PyObject_IsSubClass outside of PyType_Check
http://bugs.python.org/issue30230  opened by Jim Fasarakis-Hilliard

#30231: test_imaplib needs a TLS server accepting self-signed certific
http://bugs.python.org/issue30231  opened by haypo

#30235: Validate shutil supports path-like objects, update docs accord
http://bugs.python.org/issue30235  opened by brett.cannon

#30237: Access violation due to CancelSynchronousIo of console read
http://bugs.python.org/issue30237  opened by eryksun

#30238: 2to3 doesn't detect or fix Exception indexing
http://bugs.python.org/issue30238  opened by dlenski

#30241: Add contextlib.AbstractAsyncContextManager
http://bugs.python.org/issue30241  opened by Jelle Zijlstra

#30242: resolve undefined behaviour in struct
http://bugs.python.org/issue30242  opened by xiang.zhang

#30244: Emit a ResourceWarning in concurrent.futures executor destruct
http://bugs.python.org/issue30244  opened by haypo

#30245: possible overflow when organize struct.pack_into error message
http://bugs.python.org/issue30245  opened by xiang.zhang

#30246: fix several error messages in struct
http://bugs.python.org/issue30246  opened by xiang.zhang

#30247: Make importlib.machinery class handle os.PathLike path
http://bugs.python.org/issue30247  opened by louielu

#30248: Using boolean arguments in the _json module
http://bugs.python.org/issue30248  opened by serhiy.storchaka

#30249: improve struct.unpack_from's error message like struct.pack_in
http://bugs.python.org/issue30249  opened by xiang.zhang

#30250: StreamIO truncate behavior of current position
http://bugs.python.org/issue30250  opened by Albert.Zeyer

#30251: Windows Visual Studio solution does not have an install target
http://bugs.python.org/issue30251  opened by steveire

#30252: Configuration system does not work for Windows/Visual Studio
http://bugs.python.org/issue30252  opened by steveire

#30253: Python does not build without WITH_THREADS defined on Windows/
http://bugs.python.org/issue30253  opened by steveire

#30256: Adding a SyncManager Queue proxy to a SyncManager dict or Name
http://bugs.python.org/issue30256  opened by jjdmon

#30258: [2.7] regrtest: handle child process crash
http://bugs.python.org/issue30258  opened by haypo

#30259: Test somehow that generated files are up to date: run make reg
http://bugs.python.org/issue30259  opened by haypo

#30260: sock_dealloc() may call __repr__ when socket class is already 
http://bugs.python.org/issue30260  opened by asterite

#30262: Don't expose sqlite3 Cache and Statement
http://bugs.python.org/issue30262  opened by palaviv

#30263: regrtest: log the system load?
http://bugs.python.org/issue30263  opened by haypo

#30266: AbstractContextManager should support __method__ = None
http://bugs.python.org/issue30266  opened by Jelle Zijlstra

#30267: Deprecate os.path.commonprefix
http://bugs.python.org/issue30267  opened by Valentin.Lorentz

#30268: Make mimetypes.guess_type accept path-like objects
http://bugs.python.org/issue30268  opened by Valentin.Lorentz

#30270: Remove sqlite3.Cache display method
http://bugs.python.org/issue30270  opened by palaviv

#30271: Make sqlite3 statement cache optional
http://bugs.python.org/issue30271  opened by palaviv

#30272: distutils.version.LooseVersion's compare raises exception in c
http://bugs.python.org/issue30272  opened by ollieparanoid2

#30273: The coverage job is broken: distutils build_ext fails on None
http://bugs.python.org/issue30273  opened by haypo

#30274: Make importlib.abc.ExtensionFileLoader.__init__() documentatio
http://bugs.python.org/issue30274  opened by brett.cannon

#30275: pickle doesn't work in compile/exec
http://bugs.python.org/issue30275  opened by Jeff Zhang

#30276: import hashlib makes many programs slow
http://bugs.python.org/issue30276  opened by bmwiedemann

#30278: The element div#sidebar on documentation requires the attribut
http://bugs.python.org/issue30278  opened by Kyle Thomas

#30280: Warning -- threading._dangling was modified by test_asyncio on
http://bugs.python.org/issue30280  opened by haypo

#30281: set stop default to -PY_SSIZE_T_MAX-1 in PySlice_Unpack
http://bugs.python.org/issue30281  opened by xiang.zhang

#30282: object returned by tarfile.extractfile has an incorrect value 
http://bugs.python.org/issue30282  opened by Ondrej Kubecka



Most recent 15 issues with no replies (15)
==========================================

#30282: object returned by tarfile.extractfile has an incorrect value 
http://bugs.python.org/issue30282

#30280: Warning -- threading._dangling was modified by test_asyncio on
http://bugs.python.org/issue30280

#30278: The element div#sidebar on documentation requires the attribut
http://bugs.python.org/issue30278

#30275: pickle doesn't work in compile/exec
http://bugs.python.org/issue30275

#30274: Make importlib.abc.ExtensionFileLoader.__init__() documentatio
http://bugs.python.org/issue30274

#30272: distutils.version.LooseVersion's compare raises exception in c
http://bugs.python.org/issue30272

#30271: Make sqlite3 statement cache optional
http://bugs.python.org/issue30271

#30266: AbstractContextManager should support __method__ = None
http://bugs.python.org/issue30266

#30256: Adding a SyncManager Queue proxy to a SyncManager dict or Name
http://bugs.python.org/issue30256

#30253: Python does not build without WITH_THREADS defined on Windows/
http://bugs.python.org/issue30253

#30252: Configuration system does not work for Windows/Visual Studio
http://bugs.python.org/issue30252

#30251: Windows Visual Studio solution does not have an install target
http://bugs.python.org/issue30251

#30249: improve struct.unpack_from's error message like struct.pack_in
http://bugs.python.org/issue30249

#30245: possible overflow when organize struct.pack_into error message
http://bugs.python.org/issue30245

#30244: Emit a ResourceWarning in concurrent.futures executor destruct
http://bugs.python.org/issue30244



Most recent 15 issues waiting for review (15)
=============================================

#30281: set stop default to -PY_SSIZE_T_MAX-1 in PySlice_Unpack
http://bugs.python.org/issue30281

#30262: Don't expose sqlite3 Cache and Statement
http://bugs.python.org/issue30262

#30248: Using boolean arguments in the _json module
http://bugs.python.org/issue30248

#30246: fix several error messages in struct
http://bugs.python.org/issue30246

#30242: resolve undefined behaviour in struct
http://bugs.python.org/issue30242

#30224: remove outdated checks in struct
http://bugs.python.org/issue30224

#30219: webbrowser.open outputs xdg-open deprecation warning
http://bugs.python.org/issue30219

#30193: Support the buffer protocol in json.loads()
http://bugs.python.org/issue30193

#30192: hashlib module breaks with 64-bit kernel and 32-bit user space
http://bugs.python.org/issue30192

#30176: curses attribute constants list is incomplete
http://bugs.python.org/issue30176

#30162: Add _PyTuple_Empty and make PyTuple_New(0) never failing
http://bugs.python.org/issue30162

#30156: PYTHONDUMPREFS segfaults on exit
http://bugs.python.org/issue30156

#30152: Reduce the number of imports for argparse
http://bugs.python.org/issue30152

#30143: Using collections ABC from collections.abc rather than collect
http://bugs.python.org/issue30143

#30134: BytesWarning is missing from the documents
http://bugs.python.org/issue30134



Top 10 most discussed issues (10)
=================================

#19903: Idle: Use inspect.signature for calltips
http://bugs.python.org/issue19903  23 msgs

#23404: 'make touch' does not work with git clones of the source repos
http://bugs.python.org/issue23404  15 msgs

#30223: Add Lib/test/__main__.py in 2.7
http://bugs.python.org/issue30223  13 msgs

#29094: Regression in zipfile writing in 2.7.13
http://bugs.python.org/issue29094  11 msgs

#26362: Approved API for creating a temporary file path
http://bugs.python.org/issue26362   9 msgs

#30258: [2.7] regrtest: handle child process crash
http://bugs.python.org/issue30258   8 msgs

#30263: regrtest: log the system load?
http://bugs.python.org/issue30263   8 msgs

#30273: The coverage job is broken: distutils build_ext fails on None
http://bugs.python.org/issue30273   8 msgs

#30276: import hashlib makes many programs slow
http://bugs.python.org/issue30276   8 msgs

#29943: PySlice_GetIndicesEx change broke ABI in 3.5 and 3.6 branches
http://bugs.python.org/issue29943   7 msgs



Issues closed (59)
==================

#15934: flaky test in test_ftplib
http://bugs.python.org/issue15934  closed by haypo

#19133: Transient test failure: test_with_statement (test_ftplib)
http://bugs.python.org/issue19133  closed by haypo

#22410: Locale dependent regexps on different locales
http://bugs.python.org/issue22410  closed by serhiy.storchaka

#22610: test_ftplib fails with sem_init: Too many open files
http://bugs.python.org/issue22610  closed by haypo

#23816: struct.unpack returns null pascal strings - [first] bug report
http://bugs.python.org/issue23816  closed by xiang.zhang

#27593: Deprecate sys._mercurial and create sys._git
http://bugs.python.org/issue27593  closed by brett.cannon

#28149: Incorrect indentation under ???else??? in _bsddb.c
http://bugs.python.org/issue28149  closed by martin.panter

#29448: Implement os.Pathlike for importlib.machinery
http://bugs.python.org/issue29448  closed by serhiy.storchaka

#29621: telnetlib.Telnet.write gives confusing error message when a st
http://bugs.python.org/issue29621  closed by berker.peksag

#29641: ./configure --enable-optimizations && make && make install com
http://bugs.python.org/issue29641  closed by haypo

#29915: Drop Mac OS X Tiger support in Python 3.7?
http://bugs.python.org/issue29915  closed by haypo

#29925: test_uuid fails on OS X Tiger
http://bugs.python.org/issue29925  closed by haypo

#29967: "AMD64 FreeBSD 9.x 3.x" tries to rebuild Include/opcode.h, tim
http://bugs.python.org/issue29967  closed by haypo

#30103: uu package uses old encoding
http://bugs.python.org/issue30103  closed by xiang.zhang

#30104: clang 4.0 miscompiles dtoa.c, giving broken float parsing and 
http://bugs.python.org/issue30104  closed by haypo

#30106: test_asyncore: test_handle_write() fails in tearDown()
http://bugs.python.org/issue30106  closed by haypo

#30107: python.core file created when running tests on AMD64 FreeBSD C
http://bugs.python.org/issue30107  closed by haypo

#30119: (ftplib) A remote attacker could possibly attack by containing
http://bugs.python.org/issue30119  closed by berker.peksag

#30124: Fix C aliasing issue in Python/dtoa.c to use strict aliasing o
http://bugs.python.org/issue30124  closed by haypo

#30125: test_SEH() of test_ctypes logs "Windows fatal exception: acces
http://bugs.python.org/issue30125  closed by haypo

#30126: CheckTraceCallbackContent of test_sqlite3 fails on OS X Tiger
http://bugs.python.org/issue30126  closed by haypo

#30131: test_logging leaks a "dangling" thread
http://bugs.python.org/issue30131  closed by haypo

#30158: Deprecation warnings emitted in test_importlib
http://bugs.python.org/issue30158  closed by serhiy.storchaka

#30169: test_multiprocessing_spawn crashed on AMD64 Windows8.1 Non-Deb
http://bugs.python.org/issue30169  closed by haypo

#30173: x86 Windows7 3.x buildbot has the issue #26624 bug
http://bugs.python.org/issue30173  closed by haypo

#30175: Random test_imaplib.test_logincapa_with_client_certfile failur
http://bugs.python.org/issue30175  closed by haypo

#30184: Add tests for invalid use of PyArg_ParseTupleAndKeywords
http://bugs.python.org/issue30184  closed by serhiy.storchaka

#30185: forkserver process should silence KeyboardInterrupt
http://bugs.python.org/issue30185  closed by pitrou

#30190: unittest's assertAlmostEqual improved error message
http://bugs.python.org/issue30190  closed by berker.peksag

#30196: Add __matmul__ to collections.Counter
http://bugs.python.org/issue30196  closed by rhettinger

#30197: Enhance swap_attr() and swap_item() in test.support
http://bugs.python.org/issue30197  closed by serhiy.storchaka

#30199: Warning -- asyncore.socket_map was modified by test_ssl
http://bugs.python.org/issue30199  closed by haypo

#30200: tkinter ListboxSelect
http://bugs.python.org/issue30200  closed by Frank Pae

#30201: [3.5] RecvmsgIntoSCMRightsStreamTest fails with "OSError: [Err
http://bugs.python.org/issue30201  closed by haypo

#30204: socket.setblocking(0) changes socket.type
http://bugs.python.org/issue30204  closed by giampaolo.rodola

#30205: socket.getsockname() type mismatch with AF_UNIX on Linux
http://bugs.python.org/issue30205  closed by pitrou

#30206: data parameter for binascii.b2a_base64 is not positional-only
http://bugs.python.org/issue30206  closed by xiang.zhang

#30207: Rename test.test_support to test.support in 2.7
http://bugs.python.org/issue30207  closed by serhiy.storchaka

#30208: Small typos in IDLE doc
http://bugs.python.org/issue30208  closed by terry.reedy

#30209: some UTF8 symbols
http://bugs.python.org/issue30209  closed by terry.reedy

#30215: Make re.compile() locale agnostic
http://bugs.python.org/issue30215  closed by serhiy.storchaka

#30221: Deprecate assertNotAlmostEqual
http://bugs.python.org/issue30221  closed by serhiy.storchaka

#30225: EBADF error on x86 Tiger 3.x buildbot
http://bugs.python.org/issue30225  closed by haypo

#30232: configure: support Git worktree
http://bugs.python.org/issue30232  closed by haypo

#30233: [3.5] Warning -- locale was modified by test_idle
http://bugs.python.org/issue30233  closed by terry.reedy

#30234: Remove duplicate checks in test_isinstance
http://bugs.python.org/issue30234  closed by Jim Fasarakis-Hilliard

#30236: Add options --match and --failfast for test.regrtest in 2.7
http://bugs.python.org/issue30236  closed by serhiy.storchaka

#30239: test_asyncore: test_handle_expt() fails on macOS Sierra
http://bugs.python.org/issue30239  closed by haypo

#30240: Add daemon keyword argument to Thread constructor
http://bugs.python.org/issue30240  closed by zach.ware

#30243: Core dump when use uninitialized _json objects
http://bugs.python.org/issue30243  closed by serhiy.storchaka

#30254: [2.7] regrtest announces 401 tests, but there are 304 tests?
http://bugs.python.org/issue30254  closed by haypo

#30255: test_xml_etree: python: Objects/sliceobject.c:176: _PySlice_Ad
http://bugs.python.org/issue30255  closed by haypo

#30257: _bsddb: else misleadingly indented
http://bugs.python.org/issue30257  closed by haypo

#30261: Spam
http://bugs.python.org/issue30261  closed by zach.ware

#30264: [Windows] test_sax: Warning -- files was modified by test_sax
http://bugs.python.org/issue30264  closed by haypo

#30265: [2.7] test.support.unlink() should fail or emit a warning on W
http://bugs.python.org/issue30265  closed by haypo

#30269: [2.7] test_release_task_refs() of test_multiprocessing.py is u
http://bugs.python.org/issue30269  closed by haypo

#30277: Speeds up compiling cases-insensitive regular expressions
http://bugs.python.org/issue30277  closed by serhiy.storchaka

#30279: Remove unused Python/thread_foobar.h
http://bugs.python.org/issue30279  closed by haypo

From victor.stinner at gmail.com  Fri May  5 12:31:37 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 5 May 2017 18:31:37 +0200
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
Message-ID: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>

Hi,

Does someone know who owns the following Git clone of the old
Subversion CPython repository?
https://github.com/python-git/python/

I would suggest to remove it to avoid confusion. A friend pointed to
me this repository and was surprised to see outdated code...

Is https://github.com/python-git a real user or an organization?

Victor

From jcgoble3 at gmail.com  Fri May  5 12:36:20 2017
From: jcgoble3 at gmail.com (Jonathan Goble)
Date: Fri, 05 May 2017 16:36:20 +0000
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
Message-ID: <CAK256p02rjxa3=tNU=iiRM10O2OgqTMZy8Cs3zsGY3gO3cC18g@mail.gmail.com>

On Fri, May 5, 2017 at 12:33 PM Victor Stinner <victor.stinner at gmail.com>
wrote:

> Is https://github.com/python-git a real user or an organization?
>

It appears to me to be an individual user rather than an organization.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170505/715d814d/attachment.html>

From guido at python.org  Fri May  5 12:43:11 2017
From: guido at python.org (Guido van Rossum)
Date: Fri, 5 May 2017 09:43:11 -0700
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
Message-ID: <CAP7+vJ+a9eCFAjPR2gpJ-8ra51-L9K09rpvERvOtkbC0UFnNgw@mail.gmail.com>

I submitted another issue requesting to shut down that repo, but there are
already several other such issues and no response. I think the next step is
to contact GitHub user support and request that the repo be shut down. They
sometimes do such things if it's clear that a username/repo is abandoned.

On Fri, May 5, 2017 at 9:31 AM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> Does someone know who owns the following Git clone of the old
> Subversion CPython repository?
> https://github.com/python-git/python/
>
> I would suggest to remove it to avoid confusion. A friend pointed to
> me this repository and was surprised to see outdated code...
>
> Is https://github.com/python-git a real user or an organization?
>
> Victor
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170505/d62a1dd2/attachment.html>

From berker.peksag at gmail.com  Fri May  5 12:44:35 2017
From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=)
Date: Fri, 5 May 2017 19:44:35 +0300
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
Message-ID: <CAF4280LNARiHMUi6OyMXkaDr8TH22sZ3TfiW9zm9z+Zr3R5e_w@mail.gmail.com>

On Fri, May 5, 2017 at 7:31 PM, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> Does someone know who owns the following Git clone of the old
> Subversion CPython repository?
> https://github.com/python-git/python/
>
> I would suggest to remove it to avoid confusion. A friend pointed to
> me this repository and was surprised to see outdated code...
>
> Is https://github.com/python-git a real user or an organization?

See https://mail.python.org/pipermail/python-dev/2016-February/143224.html
for the previous discussion.

--Berker

From victor.stinner at gmail.com  Fri May  5 12:49:56 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 5 May 2017 18:49:56 +0200
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAK256p02rjxa3=tNU=iiRM10O2OgqTMZy8Cs3zsGY3gO3cC18g@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
 <CAK256p02rjxa3=tNU=iiRM10O2OgqTMZy8Cs3zsGY3gO3cC18g@mail.gmail.com>
Message-ID: <CAMpsgwbA2ViSiDKnYxEg-1RXTmWN_VnXesiSfYd0jgrNaFZUSg@mail.gmail.com>

2017-05-05 18:36 GMT+02:00 Jonathan Goble <jcgoble3 at gmail.com>:
> It appears to me to be an individual user rather than an organization.

Oh nice, glad to meet you :-) So what do you think? Are you ok to
remove this old clone? Or do you have reasons to keep it?

Victor

From brett at python.org  Fri May  5 12:52:16 2017
From: brett at python.org (Brett Cannon)
Date: Fri, 05 May 2017 16:52:16 +0000
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAMpsgwbA2ViSiDKnYxEg-1RXTmWN_VnXesiSfYd0jgrNaFZUSg@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
 <CAK256p02rjxa3=tNU=iiRM10O2OgqTMZy8Cs3zsGY3gO3cC18g@mail.gmail.com>
 <CAMpsgwbA2ViSiDKnYxEg-1RXTmWN_VnXesiSfYd0jgrNaFZUSg@mail.gmail.com>
Message-ID: <CAP1=2W77F+kHuyTKPc5Dp1adMLPTNWBKwjw6RP6GwpkC4r+TtA@mail.gmail.com>

On Fri, 5 May 2017 at 09:50 Victor Stinner <victor.stinner at gmail.com> wrote:

> 2017-05-05 18:36 GMT+02:00 Jonathan Goble <jcgoble3 at gmail.com>:
> > It appears to me to be an individual user rather than an organization.
>
> Oh nice, glad to meet you :-) So what do you think? Are you ok to
> remove this old clone? Or do you have reasons to keep it?
>

I don't think Jonathan was claiming ownership of the python-git account,
just pointed out the account is a personal account of somebody's and not an
organization account.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170505/d9d27ad9/attachment.html>

From jcgoble3 at gmail.com  Fri May  5 12:53:19 2017
From: jcgoble3 at gmail.com (Jonathan Goble)
Date: Fri, 05 May 2017 16:53:19 +0000
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAP1=2W77F+kHuyTKPc5Dp1adMLPTNWBKwjw6RP6GwpkC4r+TtA@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
 <CAK256p02rjxa3=tNU=iiRM10O2OgqTMZy8Cs3zsGY3gO3cC18g@mail.gmail.com>
 <CAMpsgwbA2ViSiDKnYxEg-1RXTmWN_VnXesiSfYd0jgrNaFZUSg@mail.gmail.com>
 <CAP1=2W77F+kHuyTKPc5Dp1adMLPTNWBKwjw6RP6GwpkC4r+TtA@mail.gmail.com>
Message-ID: <CAK256p10azB3fXg8u9=WxS244=yaUS+dVQXcCLcdYdLKD1eQZw@mail.gmail.com>

On Fri, May 5, 2017 at 12:52 PM Brett Cannon <brett at python.org> wrote:

> On Fri, 5 May 2017 at 09:50 Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> 2017-05-05 18:36 GMT+02:00 Jonathan Goble <jcgoble3 at gmail.com>:
>> > It appears to me to be an individual user rather than an organization.
>>
>> Oh nice, glad to meet you :-) So what do you think? Are you ok to
>> remove this old clone? Or do you have reasons to keep it?
>>
>
> I don't think Jonathan was claiming ownership of the python-git account,
> just pointed out the account is a personal account of somebody's and not an
> organization account.
>

Right. I was just answering the "user or organization?" question. I have
only one GItHub account, and that is not it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170505/435f7100/attachment-0001.html>

From guido at python.org  Fri May  5 13:01:38 2017
From: guido at python.org (Guido van Rossum)
Date: Fri, 5 May 2017 10:01:38 -0700
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAP1=2W77F+kHuyTKPc5Dp1adMLPTNWBKwjw6RP6GwpkC4r+TtA@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
 <CAK256p02rjxa3=tNU=iiRM10O2OgqTMZy8Cs3zsGY3gO3cC18g@mail.gmail.com>
 <CAMpsgwbA2ViSiDKnYxEg-1RXTmWN_VnXesiSfYd0jgrNaFZUSg@mail.gmail.com>
 <CAP1=2W77F+kHuyTKPc5Dp1adMLPTNWBKwjw6RP6GwpkC4r+TtA@mail.gmail.com>
Message-ID: <CAP7+vJJrPoPMfMbmPvhCESDvmZb1rKHXW2OHN6zsWG-wF6MkAA@mail.gmail.com>

Let's coordinate who contacts GitHub. Victor, Brett or myself?

On Fri, May 5, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote:

>
>
> On Fri, 5 May 2017 at 09:50 Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> 2017-05-05 18:36 GMT+02:00 Jonathan Goble <jcgoble3 at gmail.com>:
>> > It appears to me to be an individual user rather than an organization.
>>
>> Oh nice, glad to meet you :-) So what do you think? Are you ok to
>> remove this old clone? Or do you have reasons to keep it?
>>
>
> I don't think Jonathan was claiming ownership of the python-git account,
> just pointed out the account is a personal account of somebody's and not an
> organization account.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170505/d31b4eb7/attachment.html>

From chris.barker at noaa.gov  Fri May  5 15:36:29 2017
From: chris.barker at noaa.gov (Chris Barker)
Date: Fri, 5 May 2017 12:36:29 -0700
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
 <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>
Message-ID: <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>

Sorry to come late to the game, It wasn't immediately clear to me what the
implications were of the "enhancement or bugfix" distinction...

On Thu, May 4, 2017 at 9:46 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> That improved casting mechanism and the implicit support in the low
> level APIs is the main benefit I see in PEP 519, and if we were
> talking about an os module API that was missing os.path support, I'd
> be more likely to be on the "it's a bug" side of things.
>

absolutely.


> It's only higher level APIs like shutil that I put into the same
> category as third party libraries for now: Python 3.6 users shouldn't
> expect implicit use of the fspath protocol to be universal yet,


I think stdlib packages, like shutil are very much distinct from third
party  libs, and users certainly expect the entire stdlib to be updated to
support new language features.

We very often make the distinction between third party libs and stdlibs --
in fact, that is one reason we are reluctant to add new packages to the
stdlib...

Indeed, IIRC, that was the entire motivation for PEP 519 -- it started as a
"let's make Path objects work with the stdlib", but when we looked at how
to do that, it became clear that new protocol was teh best way to do that
and also provide flexibility to do that.

And the PEP states:

"""
Changes to Python's standard library are also proposed to utilize this
protocol where appropriate to facilitate the use of path objects where
historically only str and/or bytes file system paths are accepted. The goal
is to facilitate the migration of users towards rich path objects while
providing an easy way to work with code expecting str or bytes .
"""

It doesn't actually say "everywhere possible in the stdlib", but if the
goal is to facilitate migration, as stated, then the any but truly obscure
functions should be covered -- and shutil is certainly not obscure.

So it would be really great if any updates to shutils (and other stdlib
packages) to support the new protocol be back-ported.

Yes, it's "just" adding an extra call, but in my experience, low barrier to
entry are enough to discourage adoption -- and handful of shutil function
failing will certainly be enough to keep some folks from adopting the new
approach.

Is there any chance of it breaking already working code? That would be the
one compelling reason to not back-port.

-CHB



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170505/93009d76/attachment.html>

From eric at trueblade.com  Fri May  5 16:21:36 2017
From: eric at trueblade.com (Eric V. Smith)
Date: Fri, 5 May 2017 16:21:36 -0400
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
 <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>
 <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
Message-ID: <07dd54d3-b4a7-5fb1-4746-df8c53b8c012@trueblade.com>

On 5/5/2017 3:36 PM, Chris Barker wrote:
> Sorry to come late to the game, It wasn't immediately clear to me what 
> the implications were of the "enhancement or bugfix" distinction...
> 
> On Thu, May 4, 2017 at 9:46 PM, Nick Coghlan <ncoghlan at gmail.com 
> <mailto:ncoghlan at gmail.com>> wrote:
> 
>     That improved casting mechanism and the implicit support in the low
>     level APIs is the main benefit I see in PEP 519, and if we were
>     talking about an os module API that was missing os.path support, I'd
>     be more likely to be on the "it's a bug" side of things.
> 
> 
> absolutely.
> 
>     It's only higher level APIs like shutil that I put into the same
>     category as third party libraries for now: Python 3.6 users shouldn't
>     expect implicit use of the fspath protocol to be universal yet,
> 
> 
> I think stdlib packages, like shutil are very much distinct from third 
> party  libs, and users certainly expect the entire stdlib to be updated 
> to support new language features.
> 
> We very often make the distinction between third party libs and stdlibs 
> -- in fact, that is one reason we are reluctant to add new packages to 
> the stdlib...
> 
> Indeed, IIRC, that was the entire motivation for PEP 519 -- it started 
> as a "let's make Path objects work with the stdlib", but when we looked 
> at how to do that, it became clear that new protocol was teh best way to 
> do that and also provide flexibility to do that.
> 
> And the PEP states:
> 
> """
> Changes to Python's standard library are also proposed to utilize this 
> protocol where appropriate to facilitate the use of path objects where 
> historically only str and/or bytes file system paths are accepted. The 
> goal is to facilitate the migration of users towards rich path objects 
> while providing an easy way to work with code expecting str or bytes .
> """
> 
> It doesn't actually say "everywhere possible in the stdlib", but if the 
> goal is to facilitate migration, as stated, then the any but truly 
> obscure functions should be covered -- and shutil is certainly not obscure.
> 
> So it would be really great if any updates to shutils (and other stdlib 
> packages) to support the new protocol be back-ported.
> 
> Yes, it's "just" adding an extra call, but in my experience, low barrier 
> to entry are enough to discourage adoption -- and handful of shutil 
> function failing will certainly be enough to keep some folks from 
> adopting the new approach.
> 
> Is there any chance of it breaking already working code? That would be 
> the one compelling reason to not back-port.

The problem is not breaking existing code. The problem is that when 
someone writes code a year from now and tests it under Python 3.6.2, 
then a customer of theirs finds it doesn't work in 3.6.1. This will 
happen if 3.6.2 supports Path parameters to functions that 3.6.1 does not.

We've burned ourselves before with this, most famously with True and 
False in 2.2.1: 
http://python-history.blogspot.com/2013/11/the-history-of-bool-true-and-false.html

--
Eric.


From tjreedy at udel.edu  Fri May  5 17:34:00 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Fri, 5 May 2017 17:34:00 -0400
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
 <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>
 <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
Message-ID: <oeir44$euo$1@blaine.gmane.org>

On 5/5/2017 3:36 PM, Chris Barker wrote:
> Sorry to come late to the game, It wasn't immediately clear to me what 
> the implications were of the "enhancement or bugfix" distinction...

An enhancement changes the definition of the Python language, a bugfix 
does not.  Enhancements can only be part of a new version of Python.  A 
bugfix can potentially go in any 'active' implementation that has the bug.

'Python 3.6' is a version of Python.  The Python 3.6 docs more-or-less 
define the Python 3.6.  Subsequent doc patches refine and clarify the 
definition, but should not change it.  New doc patches are released 
daily on docs.python.org.

CPython 3.6.0 was the first full implementation of Python 3.6, whose 
definition was mostly fixed at the first beta.  CPython 3.6.1 was the 
next officially released implementation *of the same Python 3.6 
language*, but with some implementation bugs fixed. The same will be 
true of 3.6.2, etcetera.

If someone privately compiles cpython as it is at some time today, it 
will be an implementation of 3.6 that is hopefully better than 3.6.1 and 
worse than the future 3.6.2, but will not be either.  Call it 
3.6.(datetime).  3rd party distributions based on CPython can muddy 
things even further by selectively incorporating upstream patches, so 
that XYZ Python 3.6(.whatever) does not correspond to any date-stamped 
or released version of repository CPython.

> So it would be really great if any updates to shutils (and other stdlib 
> packages) to support the new protocol be back-ported.

Not if they change the language.

-- 
Terry Jan Reedy


From k7hoven at gmail.com  Fri May  5 18:08:53 2017
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Sat, 6 May 2017 01:08:53 +0300
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
 <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>
 <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
Message-ID: <CAMiohoiVPk6HXTbbOZnfqf6Skvf1b3+oWhYgj8fNta2+gMMFbw@mail.gmail.com>

On May 5, 2017 10:39 PM, "Chris Barker" <chris.barker at noaa.gov> wrote:

Sorry to come late to the game, It wasn't immediately clear to me what the
implications were of the "enhancement or bugfix" distinction...

On Thu, May 4, 2017 at 9:46 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> That improved casting mechanism and the implicit support in the low
> level APIs is the main benefit I see in PEP 519, and if we were
> talking about an os module API that was missing os.path support, I'd
> be more likely to be on the "it's a bug" side of things.
>

absolutely.


> It's only higher level APIs like shutil that I put into the same
> category as third party libraries for now: Python 3.6 users shouldn't
> expect implicit use of the fspath protocol to be universal yet,


I think stdlib packages, like shutil are very much distinct from third
party  libs, and users certainly expect the entire stdlib to be updated to
support new language features.

We very often make the distinction between third party libs and stdlibs --
in fact, that is one reason we are reluctant to add new packages to the
stdlib...

Indeed, IIRC, that was the entire motivation for PEP 519 -- it started as a
"let's make Path objects work with the stdlib", but when we looked at how
to do that, it became clear that new protocol was teh best way to do that
and also provide flexibility to do that.

And the PEP states:

"""
Changes to Python's standard library are also proposed to utilize this
protocol where appropriate to facilitate the use of path objects where
historically only str and/or bytes file system paths are accepted. The goal
is to facilitate the migration of users towards rich path objects while
providing an easy way to work with code expecting str or bytes .
"""

It doesn't actually say "everywhere possible in the stdlib", but if the
goal is to facilitate migration, as stated, then the any but truly obscure
functions should be covered -- and shutil is certainly not obscure.


Indeed.

So it would be really great if any updates to shutils (and other stdlib
packages) to support the new protocol be back-ported.

Yes, it's "just" adding an extra call, but in my experience, low barrier to
entry are enough to discourage adoption -- and handful of shutil function
failing will certainly be enough to keep some folks from adopting the new
approach.


I suppose the worst thing to happen is what Eric describes. But technically
speaking, passing os.PathLike to shutil functions might currently be
undefined behavior, so the change is 'legal' if it's not documented ;).

-- Koos (mobile)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170506/ca62f7d4/attachment.html>

From guido at python.org  Fri May  5 18:23:27 2017
From: guido at python.org (Guido van Rossum)
Date: Fri, 5 May 2017 15:23:27 -0700
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAP7+vJJrPoPMfMbmPvhCESDvmZb1rKHXW2OHN6zsWG-wF6MkAA@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
 <CAK256p02rjxa3=tNU=iiRM10O2OgqTMZy8Cs3zsGY3gO3cC18g@mail.gmail.com>
 <CAMpsgwbA2ViSiDKnYxEg-1RXTmWN_VnXesiSfYd0jgrNaFZUSg@mail.gmail.com>
 <CAP1=2W77F+kHuyTKPc5Dp1adMLPTNWBKwjw6RP6GwpkC4r+TtA@mail.gmail.com>
 <CAP7+vJJrPoPMfMbmPvhCESDvmZb1rKHXW2OHN6zsWG-wF6MkAA@mail.gmail.com>
Message-ID: <CAP7+vJL7gy6i8Z-wShD-r18H3q1fpWR7XD1Zgd7H4FY06MBOCg@mail.gmail.com>

OK I'll contact GitHub.

On Fri, May 5, 2017 at 10:01 AM, Guido van Rossum <guido at python.org> wrote:

> Let's coordinate who contacts GitHub. Victor, Brett or myself?
>
> On Fri, May 5, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote:
>
>>
>>
>> On Fri, 5 May 2017 at 09:50 Victor Stinner <victor.stinner at gmail.com>
>> wrote:
>>
>>> 2017-05-05 18:36 GMT+02:00 Jonathan Goble <jcgoble3 at gmail.com>:
>>> > It appears to me to be an individual user rather than an organization.
>>>
>>> Oh nice, glad to meet you :-) So what do you think? Are you ok to
>>> remove this old clone? Or do you have reasons to keep it?
>>>
>>
>> I don't think Jonathan was claiming ownership of the python-git account,
>> just pointed out the account is a personal account of somebody's and not an
>> organization account.
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
>> 40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170505/891fe215/attachment.html>

From ncoghlan at gmail.com  Sat May  6 00:30:57 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 6 May 2017 14:30:57 +1000
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
 <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>
 <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
Message-ID: <CADiSq7ca0XwHG1BJxQYSJwdLKn8oa6yOno_9JYaLhLDSEALoDw@mail.gmail.com>

On 6 May 2017 at 05:36, Chris Barker <chris.barker at noaa.gov> wrote:
> It doesn't actually say "everywhere possible in the stdlib", but if the goal
> is to facilitate migration, as stated, then the any but truly obscure
> functions should be covered -- and shutil is certainly not obscure.
>
> So it would be really great if any updates to shutils (and other stdlib
> packages) to support the new protocol be back-ported.

This argument would apply to all the other protocols we've added over
time, and would lead to significant language churn in maintenance
releases if we regularly relied on it.

As another example from 3.6, we found that the non-type metaclasses in
the standard library (most notably abc.ABCMeta) don't play nice with
the arbitrary keyword support in __init_subclass__, since they don't
pass the additional keywords from the class definition up to
type.__new__ (which in turn would pass them along to
__init_subclass__).

That's an oversight we're definitely going to fix, but we'll fix it in
3.7, not in 3.6.x.

> Yes, it's "just" adding an extra call, but in my experience, low barrier to
> entry are enough to discourage adoption -- and handful of shutil function
> failing will certainly be enough to keep some folks from adopting the new
> approach.

The critical problem prior to 3.6 wasn't the extra call, it was that
the recommended extra call had the undesirable side effect of allowing
*any* object to be used as a "file system path". As a result, you
couldn't easily separate the check out into a helper function without
putting a lot of annoying "isinstance" checks into it (or else relying
on functools.singledispatch).

By contrast, 3.6 users can just unconditionally call os.fspath(), and
know the result will work with all stdlib APIs, without allowing
nonsense like attempting to use "str(str)" as a filesystem path.

Doing that implicitly in various APIs is certainly a nice UX
enhancement, but it's the replacement of str with os.fspath as the
recommended coercion function that removes the major barrier to
pathlib adoption.

> Is there any chance of it breaking already working code? That would be the
> one compelling reason to not back-port.

"Don't make the third party compatibility testing matrix impossibly
complicated" is our other major reason not to backport new features.

We already have platform providers supporting 3.6.0 in the wild
(Heroku and AWS Lambda), and we know 3.6.1 is going to be in at least
Fedora 26, and probably Ubuntu 17.10.

We want CI providers like Travis/GitLab/Circle CI/etc to be able to
provide a *single* 3.6.x release (typically the most recent one), and
have the guarantee be "code that passes on this maintenance release
will only fail on earlier maintenance releases if you hit a genuine
bug in those releases".

Changing a function signature from accepting "Union[str, bytes]" (or
potentially even just "str") to instead accepting "os.PathLike"
doesn't count as fixing a genuine bug in the earlier releases - it
just means we made the API more permissive in a maintenance release.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat May  6 04:00:38 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 6 May 2017 18:00:38 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
Message-ID: <CADiSq7dEKunk_w1jrq39TJNAca4ML_PjZetW661FFS27++PoJA@mail.gmail.com>

On 5 March 2017 at 17:50, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Hi folks,
>
> Late last year I started working on a change to the CPython CLI (*not* the
> shared library) to get it to coerce the legacy C locale to something based
> on UTF-8 when a suitable locale is available.
>
> After a couple of rounds of iteration on linux-sig and python-ideas, I'm now
> bringing it to python-dev as a concrete proposal for Python 3.7.
>
> For most folks, reading the Abstract plus the draft docs updates in the
> reference implementation will tell you everything you need to know (if the
> C.UTF-8, C.utf8 or UTF-8 locales are available, the CLI will automatically
> attempt to coerce the legacy C locale to one of those rather than persisting
> with the latter's default assumption of ASCII as the preferred text
> encoding).

I've just pushed a significant update to the PEP based on the
discussions in this thread:
https://github.com/python/peps/commit/2fb53e7c1bbb04e1321bca11cc0112aec69f6398

The main change at the technical level is to modify the handling of
the coercion target locales such that they *always* lead to
"surrogateescape" being used by default on the standard streams. That
means we don't need to call "Py_SetStandardStreamEncoding" during
startup, that subprocesses will behave the same way as their parent
processes, and that Python in Linux containers will behave
consistently regardless of whether the container locale is set to
"C.UTF-8" explicitly, or is set to "C" and then coerced to "C.UTF-8"
by CPython.

That change also eliminated the behaviour that was contingent on
whether or not PEP 540 was accepted - PEP 540 may still want to have
the coercion target locales imply full UTF-8 mode rather than just
setting the stream error handler differently, but that will be a
question to be considered when reviewing PEP 540 rather than needing
to worry about it now.

The second technical change is that the locale coercion and warning
are now enabled on Android and Mac OS X. For Android, that's a matter
of getting GNU readline to behave sensibly, while for Mac OS X, it's a
matter of simplifying the implementation and improving cross-platform
behavioural consistency (even though we don't expect the coercion to
actually have much impact there).

Beyond that, the PEP update focuses on clarifying a few other points
without actually changing the proposal.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat May  6 04:33:14 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 6 May 2017 18:33:14 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CADiSq7dEKunk_w1jrq39TJNAca4ML_PjZetW661FFS27++PoJA@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CADiSq7dEKunk_w1jrq39TJNAca4ML_PjZetW661FFS27++PoJA@mail.gmail.com>
Message-ID: <CADiSq7cjjWiT8O+UBne7LsjY7Awkj=0uSL_f9JenAhhrMFr-dA@mail.gmail.com>

On 6 May 2017 at 18:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 5 March 2017 at 17:50, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> Hi folks,
>>
>> Late last year I started working on a change to the CPython CLI (*not* the
>> shared library) to get it to coerce the legacy C locale to something based
>> on UTF-8 when a suitable locale is available.
>>
>> After a couple of rounds of iteration on linux-sig and python-ideas, I'm now
>> bringing it to python-dev as a concrete proposal for Python 3.7.
>>
>> For most folks, reading the Abstract plus the draft docs updates in the
>> reference implementation will tell you everything you need to know (if the
>> C.UTF-8, C.utf8 or UTF-8 locales are available, the CLI will automatically
>> attempt to coerce the legacy C locale to one of those rather than persisting
>> with the latter's default assumption of ASCII as the preferred text
>> encoding).
>
> I've just pushed a significant update to the PEP based on the
> discussions in this thread:
> https://github.com/python/peps/commit/2fb53e7c1bbb04e1321bca11cc0112aec69f6398
>
> The main change at the technical level is to modify the handling of
> the coercion target locales such that they *always* lead to
> "surrogateescape" being used by default on the standard streams. That
> means we don't need to call "Py_SetStandardStreamEncoding" during
> startup, that subprocesses will behave the same way as their parent
> processes, and that Python in Linux containers will behave
> consistently regardless of whether the container locale is set to
> "C.UTF-8" explicitly, or is set to "C" and then coerced to "C.UTF-8"
> by CPython.

Working on the revised implementation for this, I've ended up
refactoring it so that all the heavy lifting is done by a single
function exported from the shared library: "_Py_CoerceLegacyLocale()".

The CLI code then just contains the check that says "Are we running in
the legacy C locale? If so, call _Py_CoerceLegacyLocale()", with all
the details of how the coercion actually works being hidden away
inside pylifecycle.c.

That seems like a potential opportunity to make the 3.7 version of
this a public API, using the following pattern:

    if (Py_LegacyLocaleDetected()) {
        Py_CoerceLegacyLocale();
    }

That way applications embedding CPython that wanted to implement the
same locale coercion logic would have an easy way to do so.

Thoughts?

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat May  6 10:24:52 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 7 May 2017 00:24:52 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CADiSq7cjjWiT8O+UBne7LsjY7Awkj=0uSL_f9JenAhhrMFr-dA@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CADiSq7dEKunk_w1jrq39TJNAca4ML_PjZetW661FFS27++PoJA@mail.gmail.com>
 <CADiSq7cjjWiT8O+UBne7LsjY7Awkj=0uSL_f9JenAhhrMFr-dA@mail.gmail.com>
Message-ID: <CADiSq7dmG7BD+HZ0nGCwOrMfcJNmN92Q=HkSHMdpsyWJ-VKD_w@mail.gmail.com>

On 6 May 2017 at 18:33, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 6 May 2017 at 18:00, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 5 March 2017 at 17:50, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>> Hi folks,
>>>
>>> Late last year I started working on a change to the CPython CLI (*not* the
>>> shared library) to get it to coerce the legacy C locale to something based
>>> on UTF-8 when a suitable locale is available.
>>>
>>> After a couple of rounds of iteration on linux-sig and python-ideas, I'm now
>>> bringing it to python-dev as a concrete proposal for Python 3.7.
>>>
>>> For most folks, reading the Abstract plus the draft docs updates in the
>>> reference implementation will tell you everything you need to know (if the
>>> C.UTF-8, C.utf8 or UTF-8 locales are available, the CLI will automatically
>>> attempt to coerce the legacy C locale to one of those rather than persisting
>>> with the latter's default assumption of ASCII as the preferred text
>>> encoding).
>>
>> I've just pushed a significant update to the PEP based on the
>> discussions in this thread:
>> https://github.com/python/peps/commit/2fb53e7c1bbb04e1321bca11cc0112aec69f6398
>>
>> The main change at the technical level is to modify the handling of
>> the coercion target locales such that they *always* lead to
>> "surrogateescape" being used by default on the standard streams. That
>> means we don't need to call "Py_SetStandardStreamEncoding" during
>> startup, that subprocesses will behave the same way as their parent
>> processes, and that Python in Linux containers will behave
>> consistently regardless of whether the container locale is set to
>> "C.UTF-8" explicitly, or is set to "C" and then coerced to "C.UTF-8"
>> by CPython.
>
> Working on the revised implementation for this, I've ended up
> refactoring it so that all the heavy lifting is done by a single
> function exported from the shared library: "_Py_CoerceLegacyLocale()".
>
> The CLI code then just contains the check that says "Are we running in
> the legacy C locale? If so, call _Py_CoerceLegacyLocale()", with all
> the details of how the coercion actually works being hidden away
> inside pylifecycle.c.
>
> That seems like a potential opportunity to make the 3.7 version of
> this a public API, using the following pattern:
>
>     if (Py_LegacyLocaleDetected()) {
>         Py_CoerceLegacyLocale();
>     }
>
> That way applications embedding CPython that wanted to implement the
> same locale coercion logic would have an easy way to do so.

OK, the reference implementation has been updated to match the latest
version of the PEP:
https://github.com/ncoghlan/cpython/commit/188e7807b6d9e49377aacbb287c074e5cabf70c5

For now, the implementation in the standalone CLI looks like this:

    /* [snip] */
    extern int _Py_LegacyLocaleDetected(void);
    extern void _Py_CoerceLegacyLocale(void);
    /* [snip] */
     if (_Py_LegacyLocaleDetected()) {
         _Py_CoerceLegacyLocale();
     }

If we decide to make this a public API for 3.7, the necessary changes would be:

- remove the leading underscore from the function names
- add the function prototypes to the pylifecycle.h header
- add the APIs to the C API documentation in the configuration &
initialization section
- define the APIs in the PEP
- adjust the backport note in the PEP to say that backports should NOT
expose the public C API, but keep it private

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From songofacandy at gmail.com  Sun May  7 01:22:51 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Sun, 7 May 2017 14:22:51 +0900
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CAEfz+TwcR7wbr71SQ-H+s9nd+Z6J3yXV1zr4Cyrm=b5OCTA_bw@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <20170504182557.066f5d9b@fsol>
 <CAEfz+TwcR7wbr71SQ-H+s9nd+Z6J3yXV1zr4Cyrm=b5OCTA_bw@mail.gmail.com>
Message-ID: <CAEfz+TzZiGYZeqHYhGz94gfTWDkcOJ1Vc3rqQUAxHtOoDyuW3g@mail.gmail.com>

Hi, Nick.

After thinking about relationship between PEP 538 and 540 in two days,
I came up with idea which removes locale coercion by default from PEP 538,
it does just enables UTF-8 mode and show warning about C locale.

Of course, this idea is based on PEP 540.  There are no "If PEP 540 is
rejected".

How do you think?

If it make sense, I want to postpone PEP 538 until PEP 540 is
accepted or rejected, or merge PEP 538 into PEP 540.


## Background

Locale coercion in current PEP 538 has some downsides:

* If user set `LANG=C LC_DATE=ja_JP.UTF-8`, locale coercion may
  overrides LC_DATE.

* It makes behavior divergence between standalone and embedded
  Python.

* Parent Python process may use utf-8:surrogateescape, but child process
  Python may use utf-8:strict.  (Python 3.6 uses ascii:surrogateescape in
  both of parent and children).

On the other hand, benefits from locale coercion is restricted:

* When locale coercion succeeds, warning is always shown.
  To hide the warning, user must disable coercion in some way.
  (e.g. use UTF-8 locale explicitly, or set PYTHONCOERCECLOCALE=0).

So I feel benefit / complexity ratio of locale coercion is less than
UTF-8 mode.

But locale coercion works nice on Android.  And there are some Android-like
Unix systems (container or small device) that C.UTF-8 is always proper locale.


## Rough spec

* Make Android-style locale coercion (forced, no warning) is now
  build option.  Some users who build Python for container or small device
  may like it.

* Normal Python build doesn't change locale.  When python executable is
  run in C locale, show locale warning.  locale warning can be disabled
  as current PEP 538.

* User can disable automatic UTF-8 mode by setting PYTHONUTF8=0
  environment variables.  User can hide warning by setting
  PYTHONUTF8=1 too.



On Fri, May 5, 2017 at 10:21 PM, INADA Naoki <songofacandy at gmail.com> wrote:
> On Fri, May 5, 2017 at 1:25 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
>> On Thu, 4 May 2017 11:24:27 +0900
>> INADA Naoki <songofacandy at gmail.com> wrote:
>>> Hi, Nick and all core devs who are interested in this PEP.
>>>
>>> I'm reviewing PEP 538 and I want to accept it in this month.
>>> It will reduces much UnicodeError pains which server-side OPs facing.
>>> Thank you Nick for working on this PEP.
>>>
>>> If you have something worrying about this PEP, please post a comment
>>> soon.  If you don't have enough time to read entire this PEP, feel free to
>>> ask a question about you're worrying.
>>
>> From my POV, it is problematic that the behaviour outlined in PEP 538
>> (see Abstract section) varies depending on the adoption of another PEP
>> (PEP 540).
>>
>> If we want to adopt PEP 538 before pronouncing on PEP 540, then PEP 538
>> should remove all points conditional on PEP 540 adoption, and PEP 540
>> should later be changed to adopt those removed points as PEP
>> 540-specific changes.
>>
>> Regards
>>
>> Antoine.
>>
>
> Fair enough.  I stop hurrying about PEP 538 and start reviewing PEP 540.
>
> Thanks,

From ncoghlan at gmail.com  Mon May  8 01:34:01 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Mon, 8 May 2017 15:34:01 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CAEfz+TzZiGYZeqHYhGz94gfTWDkcOJ1Vc3rqQUAxHtOoDyuW3g@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <20170504182557.066f5d9b@fsol>
 <CAEfz+TwcR7wbr71SQ-H+s9nd+Z6J3yXV1zr4Cyrm=b5OCTA_bw@mail.gmail.com>
 <CAEfz+TzZiGYZeqHYhGz94gfTWDkcOJ1Vc3rqQUAxHtOoDyuW3g@mail.gmail.com>
Message-ID: <CADiSq7eJ_HMxVjQFxy7tq-o-Fg7Owixgr8-aDAP59XZwJ9OoLA@mail.gmail.com>

On 7 May 2017 at 15:22, INADA Naoki <songofacandy at gmail.com> wrote:
> Hi, Nick.
>
> After thinking about relationship between PEP 538 and 540 in two days,
> I came up with idea which removes locale coercion by default from PEP 538,
> it does just enables UTF-8 mode and show warning about C locale.
>
> Of course, this idea is based on PEP 540.  There are no "If PEP 540 is
> rejected".
>
> How do you think?

The main problems I see with this approach are:

1. There's no way to configure earlier Python versions to emulate PEP
540. It's a completely new mode of operation.
2. PEP 540 isn't actually defined yet (Victor is still working on it)
3. Due to 1&2, PEP 540 isn't something 3.6 redistributors can
experiment with backporting to a narrower target audience

By contrast, you can emulate PEP 538 all the way back to Python 3.1 by
setting the following environment variables:

    LC_ALL=C.UTF-8
    LANG=C.UTF-8
    PYTHONIOENCODING=utf-8:surrogateescape

(assuming your platform provides a C.UTF-8 locale and you don't need
to run any Python 2.x components in that same environment)

I think the specific concerns you raise below are valid though, and
I'd be happy to amend PEP 538 to address them all.

> If it make sense, I want to postpone PEP 538 until PEP 540 is
> accepted or rejected, or merge PEP 538 into PEP 540.
>
>
> ## Background
>
> Locale coercion in current PEP 538 has some downsides:
>
> * If user set `LANG=C LC_DATE=ja_JP.UTF-8`, locale coercion may
>   overrides LC_DATE.

The fact it sets "LC_ALL" has previously been raised as a concern with
PEP 538, so it probably makes sense to drop that aspect and just
override "LANG". The scenarios where it makes a difference are
incredibly obscure (involving non-default SSH locale forwarding
settings for folks using SSH on Mac OS X to connect to remote Linux
systems), while just setting "LANG" will be sufficient to address the
"LANG=C" case that is the main driver for the PEP.

That means in the case above, the specific LC_DATE setting would still
take precedence.

> * It makes behavior divergence between standalone and embedded
>   Python.

Such divergence already exists, only in the other direction: embedding
applications may override the runtime's default settings, either by
setting a particular locale, or by using Py_SetStandardStreamEncoding
(which was added specifically to make it easy for Blender to force the
use of UTF-8 on the embedded Python's standard streams, regardless of
the currently locale)

That said, this is also the rationale for my suggestion that we expose
locale coercion as a public API:

    if (Py_LegacyLocaleDetected()) {
        Py_CoerceLegacyLocale();
    }

That would make it straightforward for any embedding application that
wanted to do so to replicate the behaviour of the standard CLI.

The level of divergence is also mitigated by the point in the next section.

> * Parent Python process may use utf-8:surrogateescape, but child process
>   Python may use utf-8:strict.  (Python 3.6 uses ascii:surrogateescape in
>   both of parent and children).

This discrepancy is gone now thanks to your suggestion of making
"surrogateescape" the default standard stream handler when one of the
coercion target locales is explicitly configured - both parent
processes and child processes end up with "utf-8:surrogateescape"
configured on the standard streams.

> On the other hand, benefits from locale coercion is restricted:
>
> * When locale coercion succeeds, warning is always shown.
>   To hide the warning, user must disable coercion in some way.
>   (e.g. use UTF-8 locale explicitly, or set PYTHONCOERCECLOCALE=0).

The current warning is based on what we think is appropriate for
Fedora downstream, but that doesn't necessarily mean its the right
approach for Python upstream, especially if the LC_ALL override is
dropped. We could also opt for a model where Python 3.7 emits the
coercion warning, but Python 3.8 just does the coercion silently (that
rationale would then also apply to PEP 540 - we'd warn on stderr about
the change in default behaviour in 3.7, but take the new behaviour for
granted in 3.8).

The change to make the standard stream error handler setting depend
solely on the currently configured locale also helps here, since it
means it doesn't matter how a process reached the state of having the
locale set to "C.UTF-8". CPython will behave the same way regardless,
so it makes it less import to provide an explicit notice that coercion
took place.

> So I feel benefit / complexity ratio of locale coercion is less than
> UTF-8 mode.

It isn't an either/or though - we're entirely free to do both, one
based solely on the existing configuration options that have been
around since 3.1, and the other going beyond those to also adjust the
default behaviour of other interfaces (like "open()").

> But locale coercion works nice on Android.  And there are some Android-like
> Unix systems (container or small device) that C.UTF-8 is always proper locale.
>
> ## Rough spec
>
> * Make Android-style locale coercion (forced, no warning) is now
>   build option.  Some users who build Python for container or small device
>   may like it.

But do we *want* to support the legacy C locale in 3.7+? I don't think
we do, because it will never work properly for our purposes as long as
it assumes ASCII as the default text encoding.

Part of the motivation for making locale coercion the default is so we
can update PEP 11 to make it clear that running in the legacy C locale
is no longer an officially supported configuration.

> * Normal Python build doesn't change locale.  When python executable is
>   run in C locale, show locale warning.  locale warning can be disabled
>   as current PEP 538.

That still pushes the problem back on end users to fix, though, rather
than just automatically making things like GNU readline integration
work.

> * User can disable automatic UTF-8 mode by setting PYTHONUTF8=0
>   environment variables.  User can hide warning by setting
>   PYTHONUTF8=1 too.

I think I need to better explain in the PEP why PEP 540's UTF-8 mode
on its own won't be enough, as it doesn't necessarily handle
locale-aware extension modules like GNU readline (this came up in the
draft PR review, but I never added anything specifically to the PEP
about it), and also doesn't help at all with invocation of older 3.x
releases in a subprocess.

Here's an interactive session from a PEP 538 enabled CPython, where
each line after the first is executed by doing "up-arrow,
4xleft-arrow, delete, enter"

    $ LANG=C ./python
    Python detected LC_CTYPE=C: LC_ALL & LANG coerced to C.UTF-8 (set
another locale or PYTHONCOERCECLOCALE=0 to disable this locale
coercion behavior).
    Python 3.7.0a0 (heads/pep538-coerce-c-locale:188e780, May  7 2017, 00:21:13)
    [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print("??????")
    ??????
    >>> print("?????")
    ?????
    >>> print("????")
    ????
    >>> print("???")
    ???
    >>> print("??")
    ??
    >>> print("?")
    ?
    >>>

Not exactly exciting, but this is what currently happens on an older
release if you only change the Python level stream encoding settings
without updating the locale settings:

    $ LANG=C PYTHONIOENCODING=utf-8:surrogateescape python3
    Python 3.5.3 (default, Apr 24 2017, 13:32:13)
    [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print("??????")
    ??????
    >>> print("?????")
     File "<stdin>", line 0

       ^
    SyntaxError: 'utf-8' codec can't decode bytes in position 20-21:
invalid continuation byte

That particular misbehaviour is coming from GNU readline, *not*
CPython - because the editing wasn't UTF-8 aware, it corrupted the
history buffer and fed such nonsense to stdin that even the
surrogateescape error handler was bypassed. While PEP 540's UTF-8 mode
could technically be updated to also reconfigure readline, that's
*one* extension module, and only when it's running directly as part of
Python 3.7.

By contrast, using a more appropriate locale setting already gets
readline to play nice, even when its running inside Python 3.5:

    $ LANG=C.UTF-8 python3
    Python 3.5.3 (default, Apr 24 2017, 13:32:13)
    [GCC 6.3.1 20161221 (Red Hat 6.3.1-1)] on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> print("??????")
    ??????
    >>> print("?????")
    ?????
    >>> print("????")
    ????
    >>> print("???")
    ???
    >>> print("??")
    ??
    >>> print("?")
    ?
    >>>

Don't get me wrong, I'm definitely a fan of PEP 540, as it extends
much of what PEP 538 covers beyond the standard streams and also
applies it to other operating system interfaces without relying on the
underlying operating system to provide a UTF-8 based locale. However,
I also expect it to be plagued by extension module compatibility
issues if folks attempt to use it standalone, without locale coercion
to reconfigure the behaviour of extension modules appropriately.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From songofacandy at gmail.com  Mon May  8 04:18:15 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Mon, 8 May 2017 17:18:15 +0900
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CADiSq7dyk24wfHP46feryXb91BL+ssXf7vdL4aBZxdPBOXJs6g@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <CADiSq7dyk24wfHP46feryXb91BL+ssXf7vdL4aBZxdPBOXJs6g@mail.gmail.com>
Message-ID: <CAEfz+Tykh1A4urXf0fh=XRQr3XuaXhg2FPLLjteGz+HL=y4HnA@mail.gmail.com>

>>> On platforms where they would have no effect (e.g. Mac OS X, iOS, Android,
>>> Windows) these preprocessor variables would always be undefined.
>>
>> Why ``--with[out]-c-locale-coercion`` have no effect on macOS, iOS and Android?
>
> On these three, we know the system encoding is UTF-8, so we never
> interpreted the C locale as meaning "ascii" in the first place.
>
>> On Android, locale coercion fixes readline.  Do you mean locale
>> coercion happen always
>> regardless this configuration option?
>
> Right, the change for Android is that we switch to calling
> 'setlocale(LC_ALL, "C.UTF-8")' during interpreter startup instead of
> 'setlocale(LC_ALL, "")'. That change is guarded by "#ifdef
> __ANDROID__", rather than either of the new conditionals.
>
>> On macOS, ``LC_ALL=C python`` doesn't make Python's stdio to
>> ``ascii:surrogateescape``?
>
> Similar to Android, CPython itself is hardcoded to assume UTF-8 on Mac
> OS X, since that's a platform API guarantee that users can't change.
>
>> Even so, locale coercion may fix libraries like readline, curses.
>> While C locale is less common on macOS, I don't understand any
>> reason to disable it on macOS.
>
> My understanding is that other libraries and applications also
> automatically use UTF-8 for system interfaces on Mac OS X and iOS. It
> could be that that understanding is wrong, and locale coercion would
> provide a benefit there as well.
>
> (Checking the draft implementation, it turns out I haven't actually
> implemented the configure logic to make those config settings platform
> dependent yet - they're currently only undefined on Windows by
> default, since that doesn't use the autotools based build system)
>

I tried Python 3.6 on macOS 10.11 El Capitan.

  $ LANG=C python3 -c 'import locale; print(locale.getpreferredencoding())'
  US-ASCII

And interactive shell (which uses readline by default) doesn't accept non-ASCII
input anymore.
https://www.dropbox.com/s/otshuzhnw7a71n5/macos-c-locale-readline.gif?dl=0

I think many problems with C locale are same on macOS too.
So I don't think no special casing is required on macOS.

Regards,

From chris.barker at noaa.gov  Mon May  8 15:04:45 2017
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 8 May 2017 12:04:45 -0700
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <oeir44$euo$1@blaine.gmane.org>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
 <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>
 <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
 <oeir44$euo$1@blaine.gmane.org>
Message-ID: <CALGmxEK3XQqnNUBT=cL2ai1+eWErWfEoYFG9uYR=mUOnD2o=ZQ@mail.gmail.com>

On Fri, May 5, 2017 at 2:34 PM, Terry Reedy <tjreedy at udel.edu> wrote:

> So it would be really great if any updates to shutils (and other stdlib
>> packages) to support the new protocol be back-ported.
>>
>
> Not if they change the language.


we're not talking about language changes here -- we are talking about
updates to the stdlib.

Eric wrote:

The problem is that when someone writes code a year from now and tests it
> under Python 3.6.2, then a customer of theirs finds it doesn't work in
> 3.6.1. This will happen if 3.6.2 supports Path parameters to functions that
> 3.6.1 does not.
>

This does, of course apply to the stdlib -- not just the language.

We've burned ourselves before with this, most famously with True and False
> in 2.2.1: http://python-history.blogspot.com/2013/11/the-history-of-
> bool-true-and-false.html


I'd argue that adding three built-ins is a much bigger deal than enhancing
a subset of one standard package -- not that the same issues wouldn't arise.

I think it's still worth, but I can certainly see why it might not be.

Too bad we didn't do a bit more testing for completely when it was beta....

-CHB




-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170508/281deb0f/attachment.html>

From chris.barker at noaa.gov  Mon May  8 15:08:29 2017
From: chris.barker at noaa.gov (Chris Barker)
Date: Mon, 8 May 2017 12:08:29 -0700
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CADiSq7ca0XwHG1BJxQYSJwdLKn8oa6yOno_9JYaLhLDSEALoDw@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
 <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>
 <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
 <CADiSq7ca0XwHG1BJxQYSJwdLKn8oa6yOno_9JYaLhLDSEALoDw@mail.gmail.com>
Message-ID: <CALGmxEKHk9zWBSCOVsLhHATGCE-bVwLQNTyWOZg-CyMCsH3NWQ@mail.gmail.com>

On Fri, May 5, 2017 at 9:30 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> By contrast, 3.6 users can just unconditionally call os.fspath(), and
> know the result will work with all stdlib APIs, without allowing
> nonsense like attempting to use "str(str)" as a filesystem path.
>


> Doing that implicitly in various APIs is certainly a nice UX
> enhancement, but it's the replacement of str with os.fspath as the
> recommended coercion function that removes the major barrier to
> pathlib adoption.


honestly, I'm not sure about that -- burying a str() call in a lib is
clearly risky, but using one in your user code, not so much -- you usually
know you have a path-like object (if not a Path) itself. the extra need for
wrapping str() around things was a barrier to pathlib adoption -- and it
doesn't take much of a barrier to reduce the adoption of something new.

The goal is clearly that ALL stdlib function take a path_like -- but if
we're not there yet, we're not there yet :-(

-CHB






>
> > Is there any chance of it breaking already working code? That would be
> the
> > one compelling reason to not back-port.
>
> "Don't make the third party compatibility testing matrix impossibly
> complicated" is our other major reason not to backport new features.
>
> We already have platform providers supporting 3.6.0 in the wild
> (Heroku and AWS Lambda), and we know 3.6.1 is going to be in at least
> Fedora 26, and probably Ubuntu 17.10.
>
> We want CI providers like Travis/GitLab/Circle CI/etc to be able to
> provide a *single* 3.6.x release (typically the most recent one), and
> have the guarantee be "code that passes on this maintenance release
> will only fail on earlier maintenance releases if you hit a genuine
> bug in those releases".
>
> Changing a function signature from accepting "Union[str, bytes]" (or
> potentially even just "str") to instead accepting "os.PathLike"
> doesn't count as fixing a genuine bug in the earlier releases - it
> just means we made the API more permissive in a maintenance release.
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>



-- 

Christopher Barker, Ph.D.
Oceanographer

Emergency Response Division
NOAA/NOS/OR&R            (206) 526-6959   voice
7600 Sand Point Way NE   (206) 526-6329   fax
Seattle, WA  98115       (206) 526-6317   main reception

Chris.Barker at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170508/0dc160a4/attachment.html>

From ncoghlan at gmail.com  Mon May  8 23:44:11 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 9 May 2017 13:44:11 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CADiSq7eJ_HMxVjQFxy7tq-o-Fg7Owixgr8-aDAP59XZwJ9OoLA@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <20170504182557.066f5d9b@fsol>
 <CAEfz+TwcR7wbr71SQ-H+s9nd+Z6J3yXV1zr4Cyrm=b5OCTA_bw@mail.gmail.com>
 <CAEfz+TzZiGYZeqHYhGz94gfTWDkcOJ1Vc3rqQUAxHtOoDyuW3g@mail.gmail.com>
 <CADiSq7eJ_HMxVjQFxy7tq-o-Fg7Owixgr8-aDAP59XZwJ9OoLA@mail.gmail.com>
Message-ID: <CADiSq7dYZUDppGL050YoZAPT0X3ARU3rFNJ5k7_J7aj8BPYu0w@mail.gmail.com>

On 8 May 2017 at 15:34, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 7 May 2017 at 15:22, INADA Naoki <songofacandy at gmail.com> wrote:
>> ## Background
>>
>> Locale coercion in current PEP 538 has some downsides:
>>
>> * If user set `LANG=C LC_DATE=ja_JP.UTF-8`, locale coercion may
>>   overrides LC_DATE.
>
> The fact it sets "LC_ALL" has previously been raised as a concern with
> PEP 538, so it probably makes sense to drop that aspect and just
> override "LANG". The scenarios where it makes a difference are
> incredibly obscure (involving non-default SSH locale forwarding
> settings for folks using SSH on Mac OS X to connect to remote Linux
> systems), while just setting "LANG" will be sufficient to address the
> "LANG=C" case that is the main driver for the PEP.

It occurs to me we can even still handle the forwarded
"LC_CTYPE=UTF-8" case by changing the locale coercion to set LC_CTYPE
& LANG, rather than just setting LANG as I suggested above.

That way `LANG=C LC_DATE=ja_JP.UTF-8` would still respect the explicit
LC_DATE setting, `LC_CTYPE=C` would be handled the same way as
`LANG=C`, and LC_ALL=C would continue to provide a way to force the C
locale even for LC_CTYPE without needing to be aware of the Python
specific PYTHONCOERCECLOCALE setting.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue May  9 01:00:48 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 9 May 2017 15:00:48 +1000
Subject: [Python-Dev] Is adding support for os.PathLike an enhancement
 or bugfix?
In-Reply-To: <CALGmxEKHk9zWBSCOVsLhHATGCE-bVwLQNTyWOZg-CyMCsH3NWQ@mail.gmail.com>
References: <CAP1=2W4QnR9uLtm-9Tm0KiU7RyH-YcgoeA27ZsbVkmyC7PZKDg@mail.gmail.com>
 <CAMpsgwZghQaR5wnj5cR+us3O+Uhnz+Df5J6f-zt5Rr84Jr9yWQ@mail.gmail.com>
 <CADiSq7dyM-dGyDubHwXCKnw2UR-rpW00CX7veuxbCkPQhbN+Wg@mail.gmail.com>
 <CALGmxEK-N42g-t5zSAiLiMAZ1=cRPv_K2x+yaq9UT=sRkUEg8w@mail.gmail.com>
 <CADiSq7ca0XwHG1BJxQYSJwdLKn8oa6yOno_9JYaLhLDSEALoDw@mail.gmail.com>
 <CALGmxEKHk9zWBSCOVsLhHATGCE-bVwLQNTyWOZg-CyMCsH3NWQ@mail.gmail.com>
Message-ID: <CADiSq7ddCiQqiDiL5k2Di3X2dqzVHPepe2N9DoD2g2rMOCCtpg@mail.gmail.com>

On 9 May 2017 at 05:08, Chris Barker <chris.barker at noaa.gov> wrote:
> On Fri, May 5, 2017 at 9:30 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
>>
>> By contrast, 3.6 users can just unconditionally call os.fspath(), and
>> know the result will work with all stdlib APIs, without allowing
>> nonsense like attempting to use "str(str)" as a filesystem path.
>>
>> Doing that implicitly in various APIs is certainly a nice UX
>> enhancement, but it's the replacement of str with os.fspath as the
>> recommended coercion function that removes the major barrier to
>> pathlib adoption.
>
>
> honestly, I'm not sure about that -- burying a str() call in a lib is
> clearly risky, but using one in your user code, not so much -- you usually
> know you have a path-like object (if not a Path) itself. the extra need for
> wrapping str() around things was a barrier to pathlib adoption -- and it
> doesn't take much of a barrier to reduce the adoption of something new.

Exactly - the key barrier to adoption was the lack of "os.fspath()",
since it meant that neither the standard library *nor* third party
helper libraries could reliably deal with the conversion task on
behalf of their users. Instead, people had to carefully manage their
own code, ensuring both that they used str() in the appropriate
places, *and* that they didn't accidentally call str() on the wrong
thing.

os.fspath() changes that to be much simpler and friendlier to an
iterative development process:

1. By default, expect things to just work
2. If they blow up with TypeError, wrap the offending argument in a
call to os.fspath()
3. (optional) file an RFE with the maintainers of the affected API to
call os.fspath() implicitly on that argument

The number of APIs that fall into the second category is going to be
non-zero for quite a while (both in the standard library and
elsewhere), but enough folks proceeding to step 3 will bring it down
over time.

Outside customers complaining to their commercial support vendors
though, it's rarely helpful to insist to maintainers (whether of
standard library modules or third party ones) that their failure to
adopt a feature only added in the latest version of Python now counts
as a bug in their software.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Tue May  9 07:06:58 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 9 May 2017 21:06:58 +1000
Subject: [Python-Dev] PEP 538: Coercing the legacy C locale to a UTF-8
 based locale
In-Reply-To: <CADiSq7dYZUDppGL050YoZAPT0X3ARU3rFNJ5k7_J7aj8BPYu0w@mail.gmail.com>
References: <CADiSq7fp7ij1U9vbFpqF6v0E9oW0AXUv21K+38hpe984KuXyvw@mail.gmail.com>
 <CAEfz+Txse29tMM+XTiMduRFXs138duCY58s9s4tZgkiaV3JNhw@mail.gmail.com>
 <20170504182557.066f5d9b@fsol>
 <CAEfz+TwcR7wbr71SQ-H+s9nd+Z6J3yXV1zr4Cyrm=b5OCTA_bw@mail.gmail.com>
 <CAEfz+TzZiGYZeqHYhGz94gfTWDkcOJ1Vc3rqQUAxHtOoDyuW3g@mail.gmail.com>
 <CADiSq7eJ_HMxVjQFxy7tq-o-Fg7Owixgr8-aDAP59XZwJ9OoLA@mail.gmail.com>
 <CADiSq7dYZUDppGL050YoZAPT0X3ARU3rFNJ5k7_J7aj8BPYu0w@mail.gmail.com>
Message-ID: <CADiSq7cesHow_kwnFiZCZyEN+Jrn1NAngvhbb5yANcehEHodUg@mail.gmail.com>

On 9 May 2017 at 13:44, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 8 May 2017 at 15:34, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 7 May 2017 at 15:22, INADA Naoki <songofacandy at gmail.com> wrote:
>>> ## Background
>>>
>>> Locale coercion in current PEP 538 has some downsides:
>>>
>>> * If user set `LANG=C LC_DATE=ja_JP.UTF-8`, locale coercion may
>>>   overrides LC_DATE.
>>
>> The fact it sets "LC_ALL" has previously been raised as a concern with
>> PEP 538, so it probably makes sense to drop that aspect and just
>> override "LANG". The scenarios where it makes a difference are
>> incredibly obscure (involving non-default SSH locale forwarding
>> settings for folks using SSH on Mac OS X to connect to remote Linux
>> systems), while just setting "LANG" will be sufficient to address the
>> "LANG=C" case that is the main driver for the PEP.
>
> It occurs to me we can even still handle the forwarded
> "LC_CTYPE=UTF-8" case by changing the locale coercion to set LC_CTYPE
> & LANG, rather than just setting LANG as I suggested above.
>
> That way `LANG=C LC_DATE=ja_JP.UTF-8` would still respect the explicit
> LC_DATE setting, `LC_CTYPE=C` would be handled the same way as
> `LANG=C`, and LC_ALL=C would continue to provide a way to force the C
> locale even for LC_CTYPE without needing to be aware of the Python
> specific PYTHONCOERCECLOCALE setting.

I've posted an updated reference implementation that works this way,
and it turned out to have some rather nice benefits: not only did it
make the handling of full locales (C.UTF-8, C.utf8) and partial
locales (UTF-8) more consistent (allowing for a net deletion of code),
it also meant I no longer needed a custom test case in _testembed to
check the locale warning. Instead, the affected test cases now just
set "LC_ALL" as a locale override that switches off CPython's locale
coercion without also switching off the locale warning.

Code changes: https://github.com/ncoghlan/cpython/commit/476a78133c94d82e19b89f50036cecd9b4214e7a

Rather than posting the PEP updates here though, I'll start a new
thread that explains what has changed since my initial posting to
python-dev back in March.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Tue May  9 07:20:05 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 9 May 2017 13:20:05 +0200
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
In-Reply-To: <44E5B9AC-7FCE-4A7E-AE1E-E6676CCD7941@gmail.com>
References: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
 <44E5B9AC-7FCE-4A7E-AE1E-E6676CCD7941@gmail.com>
Message-ID: <CAMpsgwbWGkyWE8GysYkOAPiMuGFiD3TEDJ6a0FxijkNRf4BYDw@mail.gmail.com>

2017-05-04 19:51 GMT+02:00 Raymond Hettinger <raymond.hettinger at gmail.com>:
> Yes.  It is perfectly reasonable to backport improvements to the tooling as long as it doesn't break anyone's existing build process.

I pushed my change to 2.7, 3.5, 3.6 and master (3.7) branches: "make"
doesn't try to regenerate generated files based on file modification
time, "make regen-all" is now required instead.

This change broke the Coverage job on Travis CI, because sysconfig.py
uses get_config_var('AST_H_DIR') to build sysconfig.get_python_inc():
http://bugs.python.org/issue30273

sysconfig was modified in 2012. "Include" was replaced with
get_config_var('AST_H_DIR') by:
https://hg.python.org/cpython/rev/998c8a8f2aea
=> see http://bugs.python.org/issue15366

To fix the Coverage job, I reverted this change:
https://github.com/python/cpython/commit/b109a1d3360fc4bb87b9887264e3634632d392ca

I'm unable to reproduce http://bugs.python.org/issue15366 bug: "venv
assumes header files in sys._home + '/Include' ".

It seems like venv and virtualend have been updated in the meanwhile
to add ${venv}/include to the include directories when building an
extension.

Victor

From ncoghlan at gmail.com  Tue May  9 07:57:04 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 9 May 2017 21:57:04 +1000
Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C locale
 to a UTF-8 based locale
Message-ID: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>

Hi folks,

Enough changes have accumulated in PEP 538 since the start of the
previous thread that it seems sensible to me to start a new thread
specifically covering the current design (which aims to address all
the concerns raised in the previous thread).

I haven't requoted the PEP in full since it's so long, but will
instead refer readers to the web version:
https://www.python.org/dev/peps/pep-0538/

I also generated a diff covered the full changes to the PEP text:

* https://gist.github.com/ncoghlan/1067805fe673b3735ac854e195747493/revisions
(this is the diff covering the last few days of changes

Summarising the key technical changes:

* to make the runtime behaviour independent of whether or not locale
coercion took place, stdin and stderr now always have
"surrogateescape" as their error handler in the potential coercion
target locales. This means Python will behave the same way regardless
of whether the locale gets set externally (e.g. by a parent Python
process or a container image definition) or implicitly during CLI
startup
* for the full locales, the interpreter now sets LC_CTYPE and LANG,
*not* LC_ALL. This means LC_ALL is once again a full locale override,
and also means that CPython won't inadvertently interfere with other
locale categories like LC_MONETARY, LC_NUMERIC, etc
* the reference implementation has been refactored so the bulk of the
new code lives in the shared library and is exposed to the linker via
a couple of underscore prefixed API symbols
(_Py_LegacyLocaleDetected() and _Py_CoerceLegacyLocale()). While the
current PEP still keeps them private, it would be straightforward to
make them public for use in embedding applications if we decided we
wanted to do so.
* locale coercion and warnings are now enabled by default on all
platforms that use the autotools-based build chain - the assumption
that some platforms didn't need them turned out to be incorrect

In addition to being updated to cover the above changes, the Rationale
section of the PEP has also been updated to explain why it doesn't
propose setting PYTHONIOENCODING, and to walk through some examples of
the problems with GNU readlines compatibility when the current locale
isn't set correctly.

The essential related changes to the reference implementation can be seen here:

* Always set "surrogateescape" for coercion target locales,
independently of whether or not coercion occurred:
https://github.com/ncoghlan/cpython/commit/188e7807b6d9e49377aacbb287c074e5cabf70c5
* Stop setting LC_ALL:
https://github.com/python/peps/commit/2f530ce0d1fd24835ac0c6f984f40db70482a18f

(There are also some smaller cleanup commits that can be seen by
browsing that branch on GitHub)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From jeremy.kloth at gmail.com  Tue May  9 09:13:36 2017
From: jeremy.kloth at gmail.com (Jeremy Kloth)
Date: Tue, 9 May 2017 07:13:36 -0600
Subject: [Python-Dev] "make touch" replaced with "make regen-all"
In-Reply-To: <CAMpsgwbWGkyWE8GysYkOAPiMuGFiD3TEDJ6a0FxijkNRf4BYDw@mail.gmail.com>
References: <CAMpsgwaFHsRzkf5f4qWTA7B7B7SXdUxodEa65Ti_mV1sWecC6w@mail.gmail.com>
 <44E5B9AC-7FCE-4A7E-AE1E-E6676CCD7941@gmail.com>
 <CAMpsgwbWGkyWE8GysYkOAPiMuGFiD3TEDJ6a0FxijkNRf4BYDw@mail.gmail.com>
Message-ID: <CAGvrs3KqQzG=ENfrqqU2fmM1Ksoi1jEgit-41Gv0NjtvwQPQkw@mail.gmail.com>

On Tue, May 9, 2017 at 5:20 AM, Victor Stinner <victor.stinner at gmail.com> wrote:
> This change broke the Coverage job on Travis CI, because sysconfig.py
> uses get_config_var('AST_H_DIR') to build sysconfig.get_python_inc():
> http://bugs.python.org/issue30273
>
> sysconfig was modified in 2012. "Include" was replaced with
> get_config_var('AST_H_DIR') by:
> https://hg.python.org/cpython/rev/998c8a8f2aea
> => see http://bugs.python.org/issue15366
>
> To fix the Coverage job, I reverted this change:
> https://github.com/python/cpython/commit/b109a1d3360fc4bb87b9887264e3634632d392ca
>
> I'm unable to reproduce http://bugs.python.org/issue15366 bug: "venv
> assumes header files in sys._home + '/Include' ".

To reproduce, Python need to be compiled {builddir} != {srcdir} as
noted in the README.
I've submitted a PR that fixes this (previous?) regression and still
permits the Makefile changes.

> It seems like venv and virtualend have been updated in the meanwhile
> to add ${venv}/include to the include directories when building an
> extension.

Yes, http://bugs.python.org/issue16116

-- 
Jeremy Kloth

From ethan at stoneleaf.us  Tue May  9 13:05:43 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Tue, 09 May 2017 10:05:43 -0700
Subject: [Python-Dev] mention aenum in the Enum docs?
Message-ID: <5911F6E7.2020102@stoneleaf.us>

A comment on a recent SO answer [1] wondered why my aenum library wasn't mentioned in the docs to help guide people that 
needed/wanted more advanced Enum options to it.  I responded that Python was not in the habit of mentioning third-party 
libraries in the docs.

However, I thought I would double-check here to see if it might be a good idea.

Pros:
- drop-in replacement for the stdlib Enum
- has many advanced features such as
   - auto __init__ building
   - multi-value members
   - duplicate value but non-aliasing members
   - etc.
- I'm the primary/only maintainer for both

Cons:
- third-party library

Thoughts?

--
~Ethan~


[1] http://stackoverflow.com/a/43855536/208880

From guido at python.org  Tue May  9 13:19:39 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 9 May 2017 10:19:39 -0700
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
Message-ID: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>

There's a PR to the peps proposal here:
https://github.com/python/peps/pull/242

The full text of the current proposal is below. The motivation for this is
that for complex decorators, even if the type checker can figure out what's
going on (by taking the signature of the decorator into account), it's
sometimes helpful to the human reader of the code to be reminded of the
type after applying the decorators (or a stack thereof). Much discussion
can be found in the PR. Note that we ended up having `Callable` in the type
because there's no rule that says a decorator returns a function type (e.g.
`property` doesn't).

This is a small thing but I'd like to run it by a larger audience than the
core mypy devs who have commented so far. There was a brief discussion on
python-ideas (my original
<https://mail.python.org/pipermail/python-ideas/2017-April/045548.html>,
favorable reply
<https://mail.python.org/pipermail/python-ideas/2017-May/045550.html> by
Nick, my response
<https://mail.python.org/pipermail/python-ideas/2017-May/045557.html>).

Credit for the proposal goes to Naomi Seyfer, with discussion by Ivan
Levkivskyi and Jukka Lehtosalo.

If there's no further debate here I'll merge it into the PEP and an
implementation will hopefully appear in the next version of the typing
module (also hopefully to be included in CPython 3.6.2 and 3.5.4).

Here's the proposed text (wordsmithing suggestions in the PR please):

+Decorators
+----------
+
+Decorators can modify the types of the functions or classes they
+decorate. Use the ``decorated_type`` decorator to declare the type of
+the resulting item after all other decorators have been applied::
+
+ from typing import ContextManager, Iterator, decorated_type
+ from contextlib import contextmanager
+
+ class DatabaseSession: ...
+
+ @decorated_type(Callable[[str], ContextManager[DatabaseSession]])
+ @contextmanager
+ def session(url: str) -> Iterator[DatabaseSession]:
+ s = DatabaseSession(url)
+ try:
+ yield s
+ finally:
+ s.close()
+
+The argument of ``decorated_type`` is a type annotation on the name
+being declared (``session``, in the example above). If you have
+multiple decorators, ``decorated_type`` must be topmost. The
+``decorated_type`` decorator is invalid on a function declaration that
+is also decorated with ``overload``, but you can annotate the
+implementation of the overload series with ``decorated_type``.
+

-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170509/146e3ee7/attachment.html>

From guido at python.org  Tue May  9 13:28:41 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 9 May 2017 10:28:41 -0700
Subject: [Python-Dev] PEP 484 proposal: don't default to Optional if
 argument default is None
Message-ID: <CAP7+vJKFeSVcqEstcPMzk8kJMcae+4SfACcoEAKKJAbb4DokeA@mail.gmail.com>

There's a proposal to change one detail of PEP 484. It currently says:

An optional type is also automatically assumed when the default value is
None, for example::

  def handle_employee(e: Employee = None): ...

This is equivalent to::

  def handle_employee(e: Optional[Employee] = None) -> None: ...


Now that we've got some experience actually using Optional with mypy
(originally mypy ignored Optional), we're beginning to think that this was
a bad idea. There's more discussion at
https://github.com/python/typing/issues/275 and an implementation of the
change (using a command-line flag) in
https://github.com/python/mypy/pull/3248.

Thoughts? Some function declarations will become a bit more verbose, but we
gain clarity (many users of annotations don't seem to be familiar with this
feature) and consistency (since this rule doesn't apply to variable
declarations and class attribute declarations).

-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170509/61e7609d/attachment.html>

From breamoreboy at yahoo.co.uk  Tue May  9 13:58:46 2017
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Tue, 9 May 2017 18:58:46 +0100
Subject: [Python-Dev] mention aenum in the Enum docs?
In-Reply-To: <5911F6E7.2020102@stoneleaf.us>
References: <5911F6E7.2020102@stoneleaf.us>
Message-ID: <oet00l$m1i$1@blaine.gmane.org>

On 09/05/2017 18:05, Ethan Furman wrote:
> A comment on a recent SO answer [1] wondered why my aenum library wasn't 
> mentioned in the docs to help guide people that needed/wanted more 
> advanced Enum options to it.  I responded that Python was not in the 
> habit of mentioning third-party libraries in the docs.
> 
> However, I thought I would double-check here to see if it might be a 
> good idea.
> 
> Pros:
> - drop-in replacement for the stdlib Enum
> - has many advanced features such as
>    - auto __init__ building
>    - multi-value members
>    - duplicate value but non-aliasing members
>    - etc.
> - I'm the primary/only maintainer for both
> 
> Cons:
> - third-party library
> 
> Thoughts?
> 
> -- 
> ~Ethan~
> 
> 
> [1] http://stackoverflow.com/a/43855536/208880

The precedent is all ready set as the third-party regex module gets a 
mention here https://docs.python.org/3/library/re.html and the requests 
package here https://docs.python.org/3/library/urllib.request.html.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From carl at oddbird.net  Tue May  9 14:03:05 2017
From: carl at oddbird.net (Carl Meyer)
Date: Tue, 9 May 2017 11:03:05 -0700
Subject: [Python-Dev] PEP 484 proposal: don't default to Optional if
 argument default is None
In-Reply-To: <CAP7+vJKFeSVcqEstcPMzk8kJMcae+4SfACcoEAKKJAbb4DokeA@mail.gmail.com>
References: <CAP7+vJKFeSVcqEstcPMzk8kJMcae+4SfACcoEAKKJAbb4DokeA@mail.gmail.com>
Message-ID: <ee01b3c6-19be-f131-cc9e-ebccdecfb46a@oddbird.net>

On 05/09/2017 10:28 AM, Guido van Rossum wrote:
> There's a proposal to change one detail of PEP 484. It currently says:
> 
>     An optional type is also automatically assumed when the default value is
>     |None|, for example::
> 
>     |def handle_employee(e: Employee = None): ... |
> 
>     This is equivalent to::
> 
>     |def handle_employee(e: Optional[Employee] = None) -> None: ... |
> 
> 
> Now that we've got some experience actually using Optional with mypy
> (originally mypy ignored Optional), we're beginning to think that this
> was a bad idea. There's more discussion at
> https://github.com/python/typing/issues/275 and an implementation of the
> change (using a command-line flag) in
> https://github.com/python/mypy/pull/3248.
> 
> Thoughts? Some function declarations will become a bit more verbose, but
> we gain clarity (many users of annotations don't seem to be familiar
> with this feature) and consistency (since this rule doesn't apply to
> variable declarations and class attribute declarations).

I've been code-reviewing a lot of diffs adding type coverage over the
last few months, and implicit-Optional has been among the most common
points of confusion. So I favor this change.

It might be nice to have a less verbose syntax for Optional, but that
can be a separate discussion.

Carl

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170509/e850ad49/attachment.sig>

From arj.python at gmail.com  Tue May  9 15:54:26 2017
From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer)
Date: Tue, 9 May 2017 23:54:26 +0400
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
Message-ID: <CADrxXXkEgVH5APWuaaSBT1tNkDAJ1YWnAM6fjA1F9xUFvUmq9Q@mail.gmail.com>

I'm really new to the mailing list. Can someone just summarise the
preceding message in 5 to 10 lines like what it is, what type is it or when
does it happen

Thanks for all,

Abdur-Rahmaan Janhangeer
Vacoas,
Mauritius
https://abdurrahmaanjanhangeer.wordpress.com/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170509/23488e4f/attachment.html>

From brett at python.org  Tue May  9 18:47:59 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 09 May 2017 22:47:59 +0000
Subject: [Python-Dev] mention aenum in the Enum docs?
In-Reply-To: <oet00l$m1i$1@blaine.gmane.org>
References: <5911F6E7.2020102@stoneleaf.us> <oet00l$m1i$1@blaine.gmane.org>
Message-ID: <CAP1=2W5m2=SJVvpSsEnerN1PVr6OLWspwxJr-TwBtTKdYjpS-A@mail.gmail.com>

On Tue, 9 May 2017 at 10:59 Mark Lawrence via Python-Dev <
python-dev at python.org> wrote:

> On 09/05/2017 18:05, Ethan Furman wrote:
> > A comment on a recent SO answer [1] wondered why my aenum library wasn't
> > mentioned in the docs to help guide people that needed/wanted more
> > advanced Enum options to it.  I responded that Python was not in the
> > habit of mentioning third-party libraries in the docs.
> >
> > However, I thought I would double-check here to see if it might be a
> > good idea.
> >
> > Pros:
> > - drop-in replacement for the stdlib Enum
> > - has many advanced features such as
> >    - auto __init__ building
> >    - multi-value members
> >    - duplicate value but non-aliasing members
> >    - etc.
> > - I'm the primary/only maintainer for both
> >
> > Cons:
> > - third-party library
> >
> > Thoughts?
> >
> > --
> > ~Ethan~
> >
> >
> > [1] http://stackoverflow.com/a/43855536/208880
>
> The precedent is all ready set as the third-party regex module gets a
> mention here https://docs.python.org/3/library/re.html and the requests
> package here https://docs.python.org/3/library/urllib.request.html.
>

Mark's right that we already do this for select modules. Since aenum is by
the author of enum to add things that don't fit in the stdlib and has been
out for years (i.e. by you, Ethan :) , I see nothing wrong with linking to
it.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170509/e9beab21/attachment.html>

From brett at python.org  Tue May  9 18:51:34 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 09 May 2017 22:51:34 +0000
Subject: [Python-Dev] PEP 484 proposal: don't default to Optional if
 argument default is None
In-Reply-To: <ee01b3c6-19be-f131-cc9e-ebccdecfb46a@oddbird.net>
References: <CAP7+vJKFeSVcqEstcPMzk8kJMcae+4SfACcoEAKKJAbb4DokeA@mail.gmail.com>
 <ee01b3c6-19be-f131-cc9e-ebccdecfb46a@oddbird.net>
Message-ID: <CAP1=2W6t7M9Ff0KGoCgm1BwHeOTZGwb3HV3AP8L6TcLhmA9aoQ@mail.gmail.com>

On Tue, 9 May 2017 at 11:11 Carl Meyer <carl at oddbird.net> wrote:

> On 05/09/2017 10:28 AM, Guido van Rossum wrote:
> > There's a proposal to change one detail of PEP 484. It currently says:
> >
> >     An optional type is also automatically assumed when the default
> value is
> >     |None|, for example::
> >
> >     |def handle_employee(e: Employee = None): ... |
> >
> >     This is equivalent to::
> >
> >     |def handle_employee(e: Optional[Employee] = None) -> None: ... |
> >
> >
> > Now that we've got some experience actually using Optional with mypy
> > (originally mypy ignored Optional), we're beginning to think that this
> > was a bad idea. There's more discussion at
> > https://github.com/python/typing/issues/275 and an implementation of the
> > change (using a command-line flag) in
> > https://github.com/python/mypy/pull/3248.
> >
> > Thoughts? Some function declarations will become a bit more verbose, but
> > we gain clarity (many users of annotations don't seem to be familiar
> > with this feature) and consistency (since this rule doesn't apply to
> > variable declarations and class attribute declarations).
>
> I've been code-reviewing a lot of diffs adding type coverage over the
> last few months, and implicit-Optional has been among the most common
> points of confusion. So I favor this change.
>

I personally like the shorthand, but the type hints I have written are in
my own projects so my experience with others is zilch and shouldn't count
for much.


>
> It might be nice to have a less verbose syntax for Optional, but that
> can be a separate discussion.
>

You should be able to do that today with `from typing import Optional as
Eh` or whatever your preferred optional/maybe name is. :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170509/7cc4b475/attachment-0001.html>

From brett at python.org  Tue May  9 18:59:14 2017
From: brett at python.org (Brett Cannon)
Date: Tue, 09 May 2017 22:59:14 +0000
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
Message-ID: <CAP1=2W5LNxXA+3tn6Nd7zW+cpLEVaNRuwbrvU06x+C+1Tw=K4A@mail.gmail.com>

The idea seems reasonable to me when viewing type hints as a form of
documentation as it helps remind people how they are expected to call the
final function.

One worry I do have, though, is Callable doesn't support keyword-only
parameters, so declared_type won't work in all cases without Callable
gaining such support (for those that don't know, Callable didn't start with
that support as Callable has been meant for callback scenarios up to this
point).

On Tue, 9 May 2017 at 10:21 Guido van Rossum <guido at python.org> wrote:

> There's a PR to the peps proposal here:
> https://github.com/python/peps/pull/242
>
> The full text of the current proposal is below. The motivation for this is
> that for complex decorators, even if the type checker can figure out what's
> going on (by taking the signature of the decorator into account), it's
> sometimes helpful to the human reader of the code to be reminded of the
> type after applying the decorators (or a stack thereof). Much discussion
> can be found in the PR. Note that we ended up having `Callable` in the type
> because there's no rule that says a decorator returns a function type (e.g.
> `property` doesn't).
>
> This is a small thing but I'd like to run it by a larger audience than the
> core mypy devs who have commented so far. There was a brief discussion on
> python-ideas (my original
> <https://mail.python.org/pipermail/python-ideas/2017-April/045548.html>,
> favorable reply
> <https://mail.python.org/pipermail/python-ideas/2017-May/045550.html> by
> Nick, my response
> <https://mail.python.org/pipermail/python-ideas/2017-May/045557.html>).
>
> Credit for the proposal goes to Naomi Seyfer, with discussion by Ivan
> Levkivskyi and Jukka Lehtosalo.
>
> If there's no further debate here I'll merge it into the PEP and an
> implementation will hopefully appear in the next version of the typing
> module (also hopefully to be included in CPython 3.6.2 and 3.5.4).
>
> Here's the proposed text (wordsmithing suggestions in the PR please):
>
> +Decorators
> +----------
> +
> +Decorators can modify the types of the functions or classes they
> +decorate. Use the ``decorated_type`` decorator to declare the type of
> +the resulting item after all other decorators have been applied::
> +
> + from typing import ContextManager, Iterator, decorated_type
> + from contextlib import contextmanager
> +
> + class DatabaseSession: ...
> +
> + @decorated_type(Callable[[str], ContextManager[DatabaseSession]])
> + @contextmanager
> + def session(url: str) -> Iterator[DatabaseSession]:
> + s = DatabaseSession(url)
> + try:
> + yield s
> + finally:
> + s.close()
> +
> +The argument of ``decorated_type`` is a type annotation on the name
> +being declared (``session``, in the example above). If you have
> +multiple decorators, ``decorated_type`` must be topmost. The
> +``decorated_type`` decorator is invalid on a function declaration that
> +is also decorated with ``overload``, but you can annotate the
> +implementation of the overload series with ``decorated_type``.
> +
>
> --
> --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170509/76e4dd73/attachment.html>

From naomi at seyfer.org  Tue May  9 19:27:07 2017
From: naomi at seyfer.org (Naomi Seyfer)
Date: Tue, 9 May 2017 16:27:07 -0700
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CAP1=2W5LNxXA+3tn6Nd7zW+cpLEVaNRuwbrvU06x+C+1Tw=K4A@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
 <CAP1=2W5LNxXA+3tn6Nd7zW+cpLEVaNRuwbrvU06x+C+1Tw=K4A@mail.gmail.com>
Message-ID: <92B9FADF-F059-4FD9-91D3-E05DC01E31CE@seyfer.org>

Stay tuned for the pep that allows callable to take keyword args. 

> On May 9, 2017, at 3:59 PM, Brett Cannon <brett at python.org> wrote:
> 
> The idea seems reasonable to me when viewing type hints as a form of documentation as it helps remind people how they are expected to call the final function.
> 
> One worry I do have, though, is Callable doesn't support keyword-only parameters, so declared_type won't work in all cases without Callable gaining such support (for those that don't know, Callable didn't start with that support as Callable has been meant for callback scenarios up to this point).
> 
>> On Tue, 9 May 2017 at 10:21 Guido van Rossum <guido at python.org> wrote:
>> There's a PR to the peps proposal here:
>> https://github.com/python/peps/pull/242
>> 
>> The full text of the current proposal is below. The motivation for this is that for complex decorators, even if the type checker can figure out what's going on (by taking the signature of the decorator into account), it's sometimes helpful to the human reader of the code to be reminded of the type after applying the decorators (or a stack thereof). Much discussion can be found in the PR. Note that we ended up having `Callable` in the type because there's no rule that says a decorator returns a function type (e.g. `property` doesn't).
>> 
>> This is a small thing but I'd like to run it by a larger audience than the core mypy devs who have commented so far. There was a brief discussion on python-ideas (my original, favorable reply by Nick, my response).
>> 
>> Credit for the proposal goes to Naomi Seyfer, with discussion by Ivan Levkivskyi and Jukka Lehtosalo.
>> 
>> If there's no further debate here I'll merge it into the PEP and an implementation will hopefully appear in the next version of the typing module (also hopefully to be included in CPython 3.6.2 and 3.5.4).
>> 
>> Here's the proposed text (wordsmithing suggestions in the PR please):
>> 
>> +Decorators
>> +----------
>> +
>> +Decorators can modify the types of the functions or classes they
>> +decorate. Use the ``decorated_type`` decorator to declare the type of
>> +the resulting item after all other decorators have been applied::
>> +
>> + from typing import ContextManager, Iterator, decorated_type
>> + from contextlib import contextmanager
>> +
>> + class DatabaseSession: ...
>> +
>> + @decorated_type(Callable[[str], ContextManager[DatabaseSession]])
>> + @contextmanager
>> + def session(url: str) -> Iterator[DatabaseSession]:
>> + s = DatabaseSession(url)
>> + try:
>> + yield s
>> + finally:
>> + s.close()
>> +
>> +The argument of ``decorated_type`` is a type annotation on the name
>> +being declared (``session``, in the example above). If you have
>> +multiple decorators, ``decorated_type`` must be topmost. The
>> +``decorated_type`` decorator is invalid on a function declaration that
>> +is also decorated with ``overload``, but you can annotate the
>> +implementation of the overload series with ``decorated_type``.
>> +
>> 
>> -- 
>> --Guido van Rossum (python.org/~guido)
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170509/316431c9/attachment.html>

From ncoghlan at gmail.com  Tue May  9 22:37:42 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 May 2017 12:37:42 +1000
Subject: [Python-Dev] PEP 484 proposal: don't default to Optional if
 argument default is None
In-Reply-To: <CAP1=2W6t7M9Ff0KGoCgm1BwHeOTZGwb3HV3AP8L6TcLhmA9aoQ@mail.gmail.com>
References: <CAP7+vJKFeSVcqEstcPMzk8kJMcae+4SfACcoEAKKJAbb4DokeA@mail.gmail.com>
 <ee01b3c6-19be-f131-cc9e-ebccdecfb46a@oddbird.net>
 <CAP1=2W6t7M9Ff0KGoCgm1BwHeOTZGwb3HV3AP8L6TcLhmA9aoQ@mail.gmail.com>
Message-ID: <CADiSq7f_BDt_TgGxkr7VN+Dv-+3g+t-VC2MGU=EEd4jpS1iyfA@mail.gmail.com>

On 10 May 2017 at 08:51, Brett Cannon <brett at python.org> wrote:
> On Tue, 9 May 2017 at 11:11 Carl Meyer <carl at oddbird.net> wrote:
>> It might be nice to have a less verbose syntax for Optional, but that
>> can be a separate discussion.
>
> You should be able to do that today with `from typing import Optional as Eh`
> or whatever your preferred optional/maybe name is. :)

While "from typing import Optional as Opt" can indeed help, perhaps
PEP 505 should be updated to discuss this point in addition to the
current proposals for None-aware binary operators?

If it included a ? prefix operator as a shorthand for
"typing.Optional[<expr>]", that would shorten affected declarations
back to:

    def handle_employee(e: ?Employee = None) -> None: ...

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From guido at python.org  Tue May  9 23:31:52 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 9 May 2017 20:31:52 -0700
Subject: [Python-Dev] PEP 484 proposal: don't default to Optional if
 argument default is None
In-Reply-To: <CADiSq7f_BDt_TgGxkr7VN+Dv-+3g+t-VC2MGU=EEd4jpS1iyfA@mail.gmail.com>
References: <CAP7+vJKFeSVcqEstcPMzk8kJMcae+4SfACcoEAKKJAbb4DokeA@mail.gmail.com>
 <ee01b3c6-19be-f131-cc9e-ebccdecfb46a@oddbird.net>
 <CAP1=2W6t7M9Ff0KGoCgm1BwHeOTZGwb3HV3AP8L6TcLhmA9aoQ@mail.gmail.com>
 <CADiSq7f_BDt_TgGxkr7VN+Dv-+3g+t-VC2MGU=EEd4jpS1iyfA@mail.gmail.com>
Message-ID: <CAP7+vJ+pAdLW2yaOiPJwnUSArZxGGdWhHQ+ONtrZdNaj6iMy9w@mail.gmail.com>

On Tue, May 9, 2017 at 7:37 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 10 May 2017 at 08:51, Brett Cannon <brett at python.org> wrote:
> > On Tue, 9 May 2017 at 11:11 Carl Meyer <carl at oddbird.net> wrote:
> >> It might be nice to have a less verbose syntax for Optional, but that
> >> can be a separate discussion.
> >
> > You should be able to do that today with `from typing import Optional as
> Eh`
> > or whatever your preferred optional/maybe name is. :)
>
> While "from typing import Optional as Opt" can indeed help, perhaps
> PEP 505 should be updated to discuss this point in addition to the
> current proposals for None-aware binary operators?
>
> If it included a ? prefix operator as a shorthand for
> "typing.Optional[<expr>]", that would shorten affected declarations
> back to:
>
>     def handle_employee(e: ?Employee = None) -> None: ...
>

I really don't want to go there. And this idea should definitely not be a
condition for removing the existing PEP 484 feature. Whatever gets done
syntax-wise won't affect anyone who needs any kind of backward
compatibility anyways, and that's very important for practical adoption of
PEP 484.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170509/2eb996fe/attachment.html>

From ncoghlan at gmail.com  Wed May 10 01:50:18 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 10 May 2017 15:50:18 +1000
Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C
 locale to a UTF-8 based locale
In-Reply-To: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>
References: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>
Message-ID: <CADiSq7e6+SgTPLFf4Yy=-6kyBVScsQvJgDWJnL2nksACg+cX_w@mail.gmail.com>

On 9 May 2017 at 21:57, Nick Coghlan <ncoghlan at gmail.com> wrote:
> The essential related changes to the reference implementation can be seen here:
>
> * Always set "surrogateescape" for coercion target locales,
> independently of whether or not coercion occurred:
> https://github.com/ncoghlan/cpython/commit/188e7807b6d9e49377aacbb287c074e5cabf70c5
> * Stop setting LC_ALL:
> https://github.com/python/peps/commit/2f530ce0d1fd24835ac0c6f984f40db70482a18f

Sorry, I just noticed the copy & paste error when posting that second
link. The correct link is:
https://github.com/ncoghlan/cpython/commit/476a78133c94d82e19b89f50036cecd9b4214e7a

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From levkivskyi at gmail.com  Wed May 10 06:31:20 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 10 May 2017 12:31:20 +0200
Subject: [Python-Dev] PEP 484 proposal: don't default to Optional if
 argument default is None
In-Reply-To: <CAP7+vJ+pAdLW2yaOiPJwnUSArZxGGdWhHQ+ONtrZdNaj6iMy9w@mail.gmail.com>
References: <CAP7+vJKFeSVcqEstcPMzk8kJMcae+4SfACcoEAKKJAbb4DokeA@mail.gmail.com>
 <ee01b3c6-19be-f131-cc9e-ebccdecfb46a@oddbird.net>
 <CAP1=2W6t7M9Ff0KGoCgm1BwHeOTZGwb3HV3AP8L6TcLhmA9aoQ@mail.gmail.com>
 <CADiSq7f_BDt_TgGxkr7VN+Dv-+3g+t-VC2MGU=EEd4jpS1iyfA@mail.gmail.com>
 <CAP7+vJ+pAdLW2yaOiPJwnUSArZxGGdWhHQ+ONtrZdNaj6iMy9w@mail.gmail.com>
Message-ID: <CAOMjWk=u__BzONYGJkp=e-sA1UEbxA37SbtLj5_8zqfkdqrFMQ@mail.gmail.com>

There is a discrepancy now between PEP 484 and PEP 526:

def f(x: int = None): ... # OK
x: int = None     # Error

I think the two rules should be "in sync", in view of this I agree with the
proposal.

Concerning verbosity and a long name Optional there are many reasonable
workarounds.
One is already mentioned from typing import Optional as O. Another
(unrelated to `` = None`` pattern)
is https://github.com/python/typing/issues/420 that allows to avoid
Optional altogether in patterns like this:

def func(x: Optional[X]) -> Optional[Y]:
    if x is None:
        return None
    # do some stuff with 'x'

With @maybe decorator proposed in
https://github.com/python/typing/issues/420
this will be simplified to:

@maybe
def func(x: X) -> Y:
    if x is None:
        return None
    # do some stuff with 'x'

Even if @maybe will not make it to typing, one still can define such (or
similar) decorators
(especially with @decorated_type and extended Callable syntax).

In view of this, I think verbosity is not a problem at all.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170510/518f178d/attachment.html>

From jlehtosalo at gmail.com  Wed May 10 08:23:05 2017
From: jlehtosalo at gmail.com (Jukka Lehtosalo)
Date: Wed, 10 May 2017 13:23:05 +0100
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <92B9FADF-F059-4FD9-91D3-E05DC01E31CE@seyfer.org>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
 <CAP1=2W5LNxXA+3tn6Nd7zW+cpLEVaNRuwbrvU06x+C+1Tw=K4A@mail.gmail.com>
 <92B9FADF-F059-4FD9-91D3-E05DC01E31CE@seyfer.org>
Message-ID: <CAA_f+Lz=TcTcV94+H5G84U51TTQAdwXaE1wmxE+wJUgQae6QiQ@mail.gmail.com>

Even if Callable types will soon support keyword arguments, the syntax for
Callables will look quite different from function definitions and this
inconsistency may hurt readability, at least for more complex signatures.
We could work around this by using the def syntax for the declared type of
a decorator. For example:

@declared_type
def session(url: str) -> ContextManager[DatabaseSession]: ...  # Explicit
'...'

@contextmanager
def session(url: str) -> Iterator[DatabaseSession]:
    s = DatabaseSession(url)
    ...

This would be quite similar to how overloads work, so there is a precedent
for something like this. We could require or recommend that the declared
type comes immediately before the implementation so that the entire
definition of a single function would not be too spread out.

This won't help if the decorated type is not a callable. We could support
this use case by using the normal variable annotation syntax:

thing: Decorated  # Declared type of 'thing'

@decorator
def thing() -> int:
    ...

Jukka

On Wed, May 10, 2017 at 12:27 AM, Naomi Seyfer <naomi at seyfer.org> wrote:

> Stay tuned for the pep that allows callable to take keyword args.
>
> On May 9, 2017, at 3:59 PM, Brett Cannon <brett at python.org> wrote:
>
> The idea seems reasonable to me when viewing type hints as a form of
> documentation as it helps remind people how they are expected to call the
> final function.
>
> One worry I do have, though, is Callable doesn't support keyword-only
> parameters, so declared_type won't work in all cases without Callable
> gaining such support (for those that don't know, Callable didn't start with
> that support as Callable has been meant for callback scenarios up to this
> point).
>
> On Tue, 9 May 2017 at 10:21 Guido van Rossum <guido at python.org> wrote:
>
>> There's a PR to the peps proposal here:
>> https://github.com/python/peps/pull/242
>>
>> The full text of the current proposal is below. The motivation for this
>> is that for complex decorators, even if the type checker can figure out
>> what's going on (by taking the signature of the decorator into account),
>> it's sometimes helpful to the human reader of the code to be reminded of
>> the type after applying the decorators (or a stack thereof). Much
>> discussion can be found in the PR. Note that we ended up having `Callable`
>> in the type because there's no rule that says a decorator returns a
>> function type (e.g. `property` doesn't).
>>
>> This is a small thing but I'd like to run it by a larger audience than
>> the core mypy devs who have commented so far. There was a brief discussion
>> on python-ideas (my original
>> <https://mail.python.org/pipermail/python-ideas/2017-April/045548.html>,
>> favorable reply
>> <https://mail.python.org/pipermail/python-ideas/2017-May/045550.html> by
>> Nick, my response
>> <https://mail.python.org/pipermail/python-ideas/2017-May/045557.html>).
>>
>> Credit for the proposal goes to Naomi Seyfer, with discussion by Ivan
>> Levkivskyi and Jukka Lehtosalo.
>>
>> If there's no further debate here I'll merge it into the PEP and an
>> implementation will hopefully appear in the next version of the typing
>> module (also hopefully to be included in CPython 3.6.2 and 3.5.4).
>>
>> Here's the proposed text (wordsmithing suggestions in the PR please):
>>
>> +Decorators
>> +----------
>> +
>> +Decorators can modify the types of the functions or classes they
>> +decorate. Use the ``decorated_type`` decorator to declare the type of
>> +the resulting item after all other decorators have been applied::
>> +
>> + from typing import ContextManager, Iterator, decorated_type
>> + from contextlib import contextmanager
>> +
>> + class DatabaseSession: ...
>> +
>> + @decorated_type(Callable[[str], ContextManager[DatabaseSession]])
>> + @contextmanager
>> + def session(url: str) -> Iterator[DatabaseSession]:
>> + s = DatabaseSession(url)
>> + try:
>> + yield s
>> + finally:
>> + s.close()
>> +
>> +The argument of ``decorated_type`` is a type annotation on the name
>> +being declared (``session``, in the example above). If you have
>> +multiple decorators, ``decorated_type`` must be topmost. The
>> +``decorated_type`` decorator is invalid on a function declaration that
>> +is also decorated with ``overload``, but you can annotate the
>> +implementation of the overload series with ``decorated_type``.
>> +
>>
>> --
>> --Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> brett%40python.org
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> jlehtosalo%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170510/abcf2b19/attachment-0001.html>

From naomi at seyfer.org  Wed May 10 10:17:41 2017
From: naomi at seyfer.org (Naomi Seyfer)
Date: Wed, 10 May 2017 07:17:41 -0700
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CAA_f+Lz=TcTcV94+H5G84U51TTQAdwXaE1wmxE+wJUgQae6QiQ@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
 <CAP1=2W5LNxXA+3tn6Nd7zW+cpLEVaNRuwbrvU06x+C+1Tw=K4A@mail.gmail.com>
 <92B9FADF-F059-4FD9-91D3-E05DC01E31CE@seyfer.org>
 <CAA_f+Lz=TcTcV94+H5G84U51TTQAdwXaE1wmxE+wJUgQae6QiQ@mail.gmail.com>
Message-ID: <4BE882DB-4436-4669-8226-3740A66557B3@seyfer.org>

I think you're right that the redefinition style is easier to read for complicated stuff... It also seems more complicated in a way I can't put my finger on. 

The overload precedent helps. 

Currently leaning towards Jukka being right, vague worries about extra complication be damned. 

Readability is more important. 

> On May 10, 2017, at 5:23 AM, Jukka Lehtosalo <jlehtosalo at gmail.com> wrote:
> 
> Even if Callable types will soon support keyword arguments, the syntax for Callables will look quite different from function definitions and this inconsistency may hurt readability, at least for more complex signatures. We could work around this by using the def syntax for the declared type of a decorator. For example:
> 
> @declared_type
> def session(url: str) -> ContextManager[DatabaseSession]: ...  # Explicit '...'
> 
> @contextmanager
> def session(url: str) -> Iterator[DatabaseSession]:
>     s = DatabaseSession(url)
>     ...
> 
> This would be quite similar to how overloads work, so there is a precedent for something like this. We could require or recommend that the declared type comes immediately before the implementation so that the entire definition of a single function would not be too spread out.
> 
> This won't help if the decorated type is not a callable. We could support this use case by using the normal variable annotation syntax:
> 
> thing: Decorated  # Declared type of 'thing'
> 
> @decorator
> def thing() -> int:
>     ...
> 
> Jukka
> 
>> On Wed, May 10, 2017 at 12:27 AM, Naomi Seyfer <naomi at seyfer.org> wrote:
>> Stay tuned for the pep that allows callable to take keyword args. 
>> 
>>> On May 9, 2017, at 3:59 PM, Brett Cannon <brett at python.org> wrote:
>>> 
>>> The idea seems reasonable to me when viewing type hints as a form of documentation as it helps remind people how they are expected to call the final function.
>>> 
>>> One worry I do have, though, is Callable doesn't support keyword-only parameters, so declared_type won't work in all cases without Callable gaining such support (for those that don't know, Callable didn't start with that support as Callable has been meant for callback scenarios up to this point).
>>> 
>>>> On Tue, 9 May 2017 at 10:21 Guido van Rossum <guido at python.org> wrote:
>>>> There's a PR to the peps proposal here:
>>>> https://github.com/python/peps/pull/242
>>>> 
>>>> The full text of the current proposal is below. The motivation for this is that for complex decorators, even if the type checker can figure out what's going on (by taking the signature of the decorator into account), it's sometimes helpful to the human reader of the code to be reminded  of the type after applying the decorators (or a stack thereof). Much discussion can be found in the PR. Note that we ended up having `Callable` in the type because there's no rule that says a decorator returns a function type (e.g. `property` doesn't).
>>>> 
>>>> This is a small thing but I'd like to run it by a larger audience than the core mypy devs who have commented so far. There was a brief discussion on python-ideas (my original, favorable reply by Nick, my response).
>>>> 
>>>> Credit for the proposal goes to Naomi Seyfer, with discussion by Ivan Levkivskyi and Jukka Lehtosalo.
>>>> 
>>>> If there's no further debate here I'll merge it into the PEP and an implementation will hopefully appear in the next version of the typing module (also hopefully to be included in CPython 3.6.2 and 3.5.4).
>>>> 
>>>> Here's the proposed text (wordsmithing suggestions in the PR please):
>>>> 
>>>> +Decorators
>>>> +----------
>>>> +
>>>> +Decorators can modify the types of the functions or classes they
>>>> +decorate. Use the ``decorated_type`` decorator to declare the type of
>>>> +the resulting item after all other decorators have been applied::
>>>> +
>>>> + from typing import ContextManager, Iterator, decorated_type
>>>> + from contextlib import contextmanager
>>>> +
>>>> + class DatabaseSession: ...
>>>> +
>>>> + @decorated_type(Callable[[str], ContextManager[DatabaseSession]])
>>>> + @contextmanager
>>>> + def session(url: str) -> Iterator[DatabaseSession]:
>>>> + s = DatabaseSession(url)
>>>> + try:
>>>> + yield s
>>>> + finally:
>>>> + s.close()
>>>> +
>>>> +The argument of ``decorated_type`` is a type annotation on the name
>>>> +being declared (``session``, in the example above). If you have
>>>> +multiple decorators, ``decorated_type`` must be topmost. The
>>>> +``decorated_type`` decorator is invalid on a function declaration that
>>>> +is also decorated with ``overload``, but you can annotate the
>>>> +implementation of the overload series with ``decorated_type``.
>>>> +
>>>> 
>>>> -- 
>>>> --Guido van Rossum (python.org/~guido)
>>>> _______________________________________________
>>>> Python-Dev mailing list
>>>> Python-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/python-dev
>>>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
>> 
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/jlehtosalo%40gmail.com
>> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170510/4c7bad32/attachment.html>

From d.f.hilliard at gmail.com  Wed May 10 10:31:07 2017
From: d.f.hilliard at gmail.com (Jim F.Hilliard)
Date: Wed, 10 May 2017 17:31:07 +0300
Subject: [Python-Dev] mention aenum in the Enum docs?
In-Reply-To: <CAP1=2W5m2=SJVvpSsEnerN1PVr6OLWspwxJr-TwBtTKdYjpS-A@mail.gmail.com>
References: <5911F6E7.2020102@stoneleaf.us> <oet00l$m1i$1@blaine.gmane.org>
 <CAP1=2W5m2=SJVvpSsEnerN1PVr6OLWspwxJr-TwBtTKdYjpS-A@mail.gmail.com>
Message-ID: <CAEZY4u3ymmcCh73wMLgFjyQ=uiNG9ndoim1-X_teAVp9QTUdxg@mail.gmail.com>

Being the one who left the comment, I'm all for adding it.

The reason I wondered why it was not there was because I was aware you are
the author of both so, I believe, the con of it being a 3rd party library
is minimized. I've also seen plenty of people asking around for more
advanced usages of Enums so it's better to direct them to a library that
not only covers many cases but is also reliable due to you authoring it.

I'm afraid that the alternative of adding it to the wiki (as you replied)
would make no tangible difference on its visibility.




On Wed, May 10, 2017 at 1:47 AM, Brett Cannon <brett at python.org> wrote:

>
>
> On Tue, 9 May 2017 at 10:59 Mark Lawrence via Python-Dev <
> python-dev at python.org> wrote:
>
>> On 09/05/2017 18:05, Ethan Furman wrote:
>> > A comment on a recent SO answer [1] wondered why my aenum library wasn't
>> > mentioned in the docs to help guide people that needed/wanted more
>> > advanced Enum options to it.  I responded that Python was not in the
>> > habit of mentioning third-party libraries in the docs.
>> >
>> > However, I thought I would double-check here to see if it might be a
>> > good idea.
>> >
>> > Pros:
>> > - drop-in replacement for the stdlib Enum
>> > - has many advanced features such as
>> >    - auto __init__ building
>> >    - multi-value members
>> >    - duplicate value but non-aliasing members
>> >    - etc.
>> > - I'm the primary/only maintainer for both
>> >
>> > Cons:
>> > - third-party library
>> >
>> > Thoughts?
>> >
>> > --
>> > ~Ethan~
>> >
>> >
>> > [1] http://stackoverflow.com/a/43855536/208880
>>
>> The precedent is all ready set as the third-party regex module gets a
>> mention here https://docs.python.org/3/library/re.html and the requests
>> package here https://docs.python.org/3/library/urllib.request.html.
>>
>
> Mark's right that we already do this for select modules. Since aenum is by
> the author of enum to add things that don't fit in the stdlib and has been
> out for years (i.e. by you, Ethan :) , I see nothing wrong with linking to
> it.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/d.
> f.hilliard%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170510/08bc5059/attachment.html>

From ethan at stoneleaf.us  Wed May 10 10:53:15 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 10 May 2017 07:53:15 -0700
Subject: [Python-Dev] python docs
Message-ID: <5913295B.4060502@stoneleaf.us>

Not sure where to ask about this, so I'm asking here.

In the on-line docs, at the very bottom of a page, in fine print, is a link: _Find a bug?_  Following that link leads to 
a short page with some advice on how to handle it.  Under the second heading [1] is this paragraph:

> If you?re short on time, you can also email documentation bug reports
 > to docs at python.org (behavioral bugs can be sent to python-list at python.org).
 > ?docs@? is a mailing list run by volunteers; your request will be noticed,
 > though it may take a while to be processed.

Why is python-list the place to send behavioral bugs to?  It's been my experience that folks there will (rightly) ask 
the individual to file a bug on the tracker.

--
~Ethan~



[1] https://docs.python.org/3/bugs.html#documentation-bugs




From rosuav at gmail.com  Wed May 10 10:59:34 2017
From: rosuav at gmail.com (Chris Angelico)
Date: Thu, 11 May 2017 00:59:34 +1000
Subject: [Python-Dev] python docs
In-Reply-To: <5913295B.4060502@stoneleaf.us>
References: <5913295B.4060502@stoneleaf.us>
Message-ID: <CAPTjJmpwjUv=Pp-1ojs6iJAN7a6Yf=NrX+iXD-uwzVVvBbxqAA@mail.gmail.com>

On Thu, May 11, 2017 at 12:53 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
> Why is python-list the place to send behavioral bugs to?  It's been my
> experience that folks there will (rightly) ask the individual to file a bug
> on the tracker.
>

How many bug tracker entries do you want to see about mutable default
arguments? :) The mailing list is a great gatekeeper against "not a
bug" bugs.

ChrisA

From gvanrossum at gmail.com  Wed May 10 11:08:27 2017
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Wed, 10 May 2017 08:08:27 -0700
Subject: [Python-Dev] python docs
In-Reply-To: <5913295B.4060502@stoneleaf.us>
References: <5913295B.4060502@stoneleaf.us>
Message-ID: <CAP7+vJJe7fipja-tuek6o8uv1kmjzp_fJGbhfhVHy1ZxdWY77Q@mail.gmail.com>

Presumably so newbies can get help on common issues that aren't bugs, or
guidance on how to file a good bug report. Or search the tracker for
duplicates, or to upgrade to a newer version.

On May 10, 2017 7:58 AM, "Ethan Furman" <ethan at stoneleaf.us> wrote:

> Not sure where to ask about this, so I'm asking here.
>
> In the on-line docs, at the very bottom of a page, in fine print, is a
> link: _Find a bug?_  Following that link leads to a short page with some
> advice on how to handle it.  Under the second heading [1] is this paragraph:
>
> If you?re short on time, you can also email documentation bug reports
>>
> > to docs at python.org (behavioral bugs can be sent to
> python-list at python.org).
> > ?docs@? is a mailing list run by volunteers; your request will be
> noticed,
> > though it may take a while to be processed.
>
> Why is python-list the place to send behavioral bugs to?  It's been my
> experience that folks there will (rightly) ask the individual to file a bug
> on the tracker.
>
> --
> ~Ethan~
>
>
>
> [1] https://docs.python.org/3/bugs.html#documentation-bugs
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170510/094ccecf/attachment.html>

From tjreedy at udel.edu  Wed May 10 14:38:50 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Wed, 10 May 2017 14:38:50 -0400
Subject: [Python-Dev] python docs
In-Reply-To: <CAPTjJmpwjUv=Pp-1ojs6iJAN7a6Yf=NrX+iXD-uwzVVvBbxqAA@mail.gmail.com>
References: <5913295B.4060502@stoneleaf.us>
 <CAPTjJmpwjUv=Pp-1ojs6iJAN7a6Yf=NrX+iXD-uwzVVvBbxqAA@mail.gmail.com>
Message-ID: <oevmnk$sa9$1@blaine.gmane.org>

On 5/10/2017 10:59 AM, Chris Angelico wrote:
> On Thu, May 11, 2017 at 12:53 AM, Ethan Furman <ethan at stoneleaf.us> wrote:
>> Why is python-list the place to send behavioral bugs to?

Where else?  The people who would follow that advice are generally not 
registered on bpo and have no interest in the hassle of becoming so. 
(The verification process sometimes fails.)  Those already registered 
already know about it.

>>  It's been my
>> experience that folks there will (rightly) ask the individual to file a bug
>> on the tracker.

After vetting.  Or sometimes someone else open an issue with more 
coherence or info than the O could.

> How many bug tracker entries do you want to see about mutable default
> arguments? :) The mailing list is a great gatekeeper against "not a
> bug" bugs.

I wish more newbies went to python-list first.

-- 
Terry Jan Reedy


From breamoreboy at yahoo.co.uk  Wed May 10 15:37:12 2017
From: breamoreboy at yahoo.co.uk (Mark Lawrence)
Date: Wed, 10 May 2017 20:37:12 +0100
Subject: [Python-Dev] python docs
In-Reply-To: <5913295B.4060502@stoneleaf.us>
References: <5913295B.4060502@stoneleaf.us>
Message-ID: <oevq58$5ab$1@blaine.gmane.org>

On 10/05/2017 15:53, Ethan Furman wrote:
> Not sure where to ask about this, so I'm asking here.
> 
> In the on-line docs, at the very bottom of a page, in fine print, is a 
> link: _Find a bug?_  Following that link leads to a short page with some 
> advice on how to handle it.  Under the second heading [1] is this 
> paragraph:
> 
>> If you?re short on time, you can also email documentation bug reports
>  > to docs at python.org (behavioral bugs can be sent to 
> python-list at python.org).
>  > ?docs@? is a mailing list run by volunteers; your request will be 
> noticed,
>  > though it may take a while to be processed.
> 
> Why is python-list the place to send behavioral bugs to?  It's been my 
> experience that folks there will (rightly) ask the individual to file a 
> bug on the tracker.
> 
> -- 
> ~Ethan~
> 
> [1] https://docs.python.org/3/bugs.html#documentation-bugs
> 

Saves even more reports on the grounds that "Python can't do floating 
point properly".  I don't know if it's calmed down now, but there used 
to be 3 or 4 a year.  They were, as always, very politely referred to 
the FAQ, or similar.  Multiply that up by a multitude of newbie problems 
and the triagers would be swamped.

-- 
My fellow Pythonistas, ask not what our language can do for you, ask
what you can do for our language.

Mark Lawrence


From ethan at stoneleaf.us  Wed May 10 16:35:13 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 10 May 2017 13:35:13 -0700
Subject: [Python-Dev] python docs
In-Reply-To: <5913295B.4060502@stoneleaf.us>
References: <5913295B.4060502@stoneleaf.us>
Message-ID: <59137981.5040301@stoneleaf.us>

On 05/10/2017 07:53 AM, Ethan Furman wrote:

> Why is python-list the place to send behavioral bugs to?  It's been my experience that folks there will (rightly) ask
> the individual to file a bug on the tracker.

Thank you for the abundance of answers.  I am now totally on-board with such "bug reports" going to Python List.

--
~Ethan~


From steve at pearwood.info  Wed May 10 20:50:54 2017
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 11 May 2017 10:50:54 +1000
Subject: [Python-Dev] mention aenum in the Enum docs?
In-Reply-To: <5911F6E7.2020102@stoneleaf.us>
References: <5911F6E7.2020102@stoneleaf.us>
Message-ID: <20170511005054.GB23636@ando.pearwood.info>

On Tue, May 09, 2017 at 10:05:43AM -0700, Ethan Furman wrote:

> A comment on a recent SO answer [1] wondered why my aenum library wasn't 
> mentioned in the docs to help guide people that needed/wanted more advanced 
> Enum options to it.

I know that the std lib is where good modules go to die :-)

Is the aenum module feature complete? Is it ready to ~~die~~ move to the 
std lib?

If aenum is not going through active development, perhaps the missing 
features should be ported to enum and then there's no need to recommend 
a second module.

> Pros:
> - drop-in replacement for the stdlib Enum
> - has many advanced features such as
>   - auto __init__ building
>   - multi-value members
>   - duplicate value but non-aliasing members
>   - etc.

Those three advanced features sound useful. Is there a reason enum 
doesn't support them?



-- 
Steve

From steve at pearwood.info  Wed May 10 21:39:43 2017
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 11 May 2017 11:39:43 +1000
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CADrxXXkEgVH5APWuaaSBT1tNkDAJ1YWnAM6fjA1F9xUFvUmq9Q@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
 <CADrxXXkEgVH5APWuaaSBT1tNkDAJ1YWnAM6fjA1F9xUFvUmq9Q@mail.gmail.com>
Message-ID: <20170511013943.GC23636@ando.pearwood.info>

On Tue, May 09, 2017 at 11:54:26PM +0400, Abdur-Rahmaan Janhangeer wrote:
> I'm really new to the mailing list. Can someone just summarise the
> preceding message in 5 to 10 lines like what it is, what type is it or when
> does it happen

It is an update to PEP 484, adding support for type-checking decorators.

https://www.python.org/dev/peps/pep-0484/

I don't understand what you mean by "what type is it".

When does it happen: if there are no strong or conclusive objections
(and I don't think there have been) it may have already happened by now.
If not, then soon. Depends on Guido's schedule.

As for when it will actually be visible in Python, the original post 
already answered that: Guido hopes to add it to Python 3.6.2 and 3.5.4. 
I don't think this will have much visible impact unless you're using 
MyPy for type checking.

By the way, the original post was less than fifty lines. That already 
is a summary.


-- 
Steve

From ethan at stoneleaf.us  Wed May 10 22:29:58 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Wed, 10 May 2017 19:29:58 -0700
Subject: [Python-Dev] mention aenum in the Enum docs?
In-Reply-To: <20170511005054.GB23636@ando.pearwood.info>
References: <5911F6E7.2020102@stoneleaf.us>
 <20170511005054.GB23636@ando.pearwood.info>
Message-ID: <5913CCA6.4020301@stoneleaf.us>

On 05/10/2017 05:50 PM, Steven D'Aprano wrote:
> On Tue, May 09, 2017 at 10:05:43AM -0700, Ethan Furman wrote:

>> A comment on a recent SO answer [1] wondered why my aenum library wasn't
>> mentioned in the docs to help guide people that needed/wanted more advanced
>> Enum options to it.
>
> I know that the std lib is where good modules go to die :-)
>
> Is the aenum module feature complete? Is it ready to ~~die~~ move to the
> std lib?

No, it is still under active development.

>> Pros:
>> - drop-in replacement for the stdlib Enum
>> - has many advanced features such as
>>    - auto __init__ building
>>    - multi-value members
>>    - duplicate value but non-aliasing members
>>    - etc.
>
> Those three advanced features sound useful. Is there a reason enum
> doesn't support them?

New features go to aenum first for experimentation and bug squashing.  After I'm happy with them, thay /may/ go to the 
standard library.

Even if all the features end up in the stdlib (unlikely), it would still be a useful recommendation as aenum works as 
far back as 2.7, possibly further).

--
~Ethan~

From p.f.moore at gmail.com  Thu May 11 03:55:02 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Thu, 11 May 2017 08:55:02 +0100
Subject: [Python-Dev] mention aenum in the Enum docs?
In-Reply-To: <20170511005054.GB23636@ando.pearwood.info>
References: <5911F6E7.2020102@stoneleaf.us>
 <20170511005054.GB23636@ando.pearwood.info>
Message-ID: <CACac1F_2BmSkesXyiRtVtHLRYe7ic5bt3tspaq0q+NWpgwizdg@mail.gmail.com>

On 11 May 2017 at 01:50, Steven D'Aprano <steve at pearwood.info> wrote:
> On Tue, May 09, 2017 at 10:05:43AM -0700, Ethan Furman wrote:
>
>> A comment on a recent SO answer [1] wondered why my aenum library wasn't
>> mentioned in the docs to help guide people that needed/wanted more advanced
>> Enum options to it.
>
> I know that the std lib is where good modules go to die :-)
>
> Is the aenum module feature complete? Is it ready to ~~die~~ move to the
> std lib?

The proposal is about mentioning aenum as a 3rd party alternative in
the stdlib docs for enum (which, BTW, I think is perfectly fine and a
useful thing to do), not to bring aenum into the stdlib.

Paul

From victor.stinner at gmail.com  Thu May 11 11:56:31 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 11 May 2017 17:56:31 +0200
Subject: [Python-Dev] New buildbot-status mailing list to be notified on
 buildbot failures
Message-ID: <CAMpsgwYPRvxCFBcLo4ZdrVWU0g_LG2Bv5C=D1NkhQMB4mbqHKg@mail.gmail.com>

Hi,

You can now subscribe to this mailing list to get email notifications
when the state of a buildbot changes from success (green)/warning
(orange) to failure (red):
https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/

Only these state changes (green=>red, orange=>red) send an email notification.

See http://bugs.python.org/issue30325 for the technical details on how
these emails are sent.

Thanks to Zachary Ware for his strong support on creating this list
and fixing buildbot issues ;-)

Victor

From freddyrietdijk at fridh.nl  Fri May 12 09:06:15 2017
From: freddyrietdijk at fridh.nl (Freddy Rietdijk)
Date: Fri, 12 May 2017 15:06:15 +0200
Subject: [Python-Dev] Hash randomization and deterministic bytecode
Message-ID: <CAOQtOH2XSE3UQhD8SKKn8HLEWgnAOZ4F0WzGyKh+4mHnAtqGmg@mail.gmail.com>

Hi,

On Nix we set PYTHONHASHSEED to 0 when building packages, disabling hash
randomization. We do this to improve determinism of the builds because we
store the bytecode next to the code.

When one runs Python directly or via a script PYTHONHASHSEED is not set
thus enabling hash randomization. Am I correct when I say that in this case
Python still uses the reproducibly build bytecode and, because its now
running with a random seed we wouldn't be vulnerable to
http://www.ocert.org/advisories/ocert-2011-003.html ? Or would it also try
to each time also recompile bytecode?

Kind regards,

Freddy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170512/243ecfb0/attachment.html>

From guido at python.org  Fri May 12 11:22:58 2017
From: guido at python.org (Guido van Rossum)
Date: Fri, 12 May 2017 08:22:58 -0700
Subject: [Python-Dev] Hash randomization and deterministic bytecode
In-Reply-To: <CAOQtOH2XSE3UQhD8SKKn8HLEWgnAOZ4F0WzGyKh+4mHnAtqGmg@mail.gmail.com>
References: <CAOQtOH2XSE3UQhD8SKKn8HLEWgnAOZ4F0WzGyKh+4mHnAtqGmg@mail.gmail.com>
Message-ID: <CAP7+vJLbKgKS=oOoKCWVCYE0khsJ7mTwsAWHLrE7KHw9L4t2MQ@mail.gmail.com>

Don't worry, the PYTHONHASHSEED setting does not get recorded in the
bytecode header and the generated bytecode (even if it sometimes differs in
trivial ways) is usable with all hash seed settings.

--Guido

On Fri, May 12, 2017 at 6:06 AM, Freddy Rietdijk <freddyrietdijk at fridh.nl>
wrote:

> Hi,
>
> On Nix we set PYTHONHASHSEED to 0 when building packages, disabling hash
> randomization. We do this to improve determinism of the builds because we
> store the bytecode next to the code.
>
> When one runs Python directly or via a script PYTHONHASHSEED is not set
> thus enabling hash randomization. Am I correct when I say that in this case
> Python still uses the reproducibly build bytecode and, because its now
> running with a random seed we wouldn't be vulnerable to
> http://www.ocert.org/advisories/ocert-2011-003.html ? Or would it also
> try to each time also recompile bytecode?
>
> Kind regards,
>
> Freddy
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170512/3da5ccfd/attachment.html>

From status at bugs.python.org  Fri May 12 12:09:06 2017
From: status at bugs.python.org (Python tracker)
Date: Fri, 12 May 2017 18:09:06 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20170512160906.4B592560C5@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2017-05-05 - 2017-05-12)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5958 (+29)
  closed 36149 (+40)
  total  42107 (+69)

Open issues with patches: 2384 


Issues opened (47)
==================

#30273: The coverage job is broken: distutils build_ext fails on None
http://bugs.python.org/issue30273  reopened by haypo

#30283: [2.7] Backport test_regrtest (partially) on Python 2.7
http://bugs.python.org/issue30283  opened by haypo

#30284: Build CPython out of tree with a read-only source tree
http://bugs.python.org/issue30284  opened by haypo

#30287: cpython and Clang Static Analyzer
http://bugs.python.org/issue30287  opened by dilyan.palauzov

#30290: IDLE: add tests for help_about.py
http://bugs.python.org/issue30290  opened by terry.reedy

#30291: Allow windows launcher to specify bit lengths with & without m
http://bugs.python.org/issue30291  opened by Steve Barnes

#30294: ./configure, pydebug and pymalloc
http://bugs.python.org/issue30294  opened by dilyan.palauzov

#30295: msvcrt SetErrorMode not documented
http://bugs.python.org/issue30295  opened by giampaolo.rodola

#30296: Remove unnecessary tuples, lists, sets, and dicts from Lib
http://bugs.python.org/issue30296  opened by jdufresne

#30299: Display the bytecode when compiled a regular expression in deb
http://bugs.python.org/issue30299  opened by serhiy.storchaka

#30300: asyncio.Controller
http://bugs.python.org/issue30300  opened by barry

#30301: multiprocessing: AttributeError: 'SimpleQueue' object has no a
http://bugs.python.org/issue30301  opened by Daniel Moore

#30302: Improve .__repr__ implementation for datetime.timedelta
http://bugs.python.org/issue30302  opened by musically_ut

#30303: IDLE: Add _utest to textview
http://bugs.python.org/issue30303  opened by louielu

#30304: TestCase.assertMultiLineEqual only registered for Unicode stri
http://bugs.python.org/issue30304  opened by martin.panter

#30306: release arguments of contextmanager
http://bugs.python.org/issue30306  opened by Martin.Teichmann

#30310: tkFont.py assumes that all font families are encoded as ascii 
http://bugs.python.org/issue30310  opened by culler

#30312: Small correction in set code sample
http://bugs.python.org/issue30312  opened by mcocdawc

#30313: Tests of Python 2.7 VS9.0 buildbots must be run with -uall -rw
http://bugs.python.org/issue30313  opened by haypo

#30314: Buildbots: 15 min is too low for test_tools on x86 Tiger 3.6 b
http://bugs.python.org/issue30314  opened by haypo

#30315: test_ftplib.TestTLS_FTPClass: "[Errno 54] Connection reset by 
http://bugs.python.org/issue30315  opened by haypo

#30316: test_default_timeout() of test_threading.BarrierTests: random 
http://bugs.python.org/issue30316  opened by haypo

#30317: test_timeout() of test_multiprocessing_spawn.WithManagerTestBa
http://bugs.python.org/issue30317  opened by haypo

#30318: test_distutils is too verbose on Windows
http://bugs.python.org/issue30318  opened by haypo

#30319: test_invalid_authentication()  of test_imaplib: ConnectionRese
http://bugs.python.org/issue30319  opened by haypo

#30323: concurrent.futures.Executor.map() consumes all memory when big
http://bugs.python.org/issue30323  opened by Klamann

#30325: Buildbot: send email notifications to buildbot-status@
http://bugs.python.org/issue30325  opened by haypo

#30328: test_ssl.test_connect_with_context(): ConnectionResetError on 
http://bugs.python.org/issue30328  opened by haypo

#30329: test_imaplib.test_login_cram_md5(): OSError: [WinError 10022] 
http://bugs.python.org/issue30329  opened by haypo

#30330: test_socket.test_idna(): socket.gaierror: [Errno 11001] getadd
http://bugs.python.org/issue30330  opened by haypo

#30331: TestPOP3_TLSClass: socket.timeout: timed out on AMD64 FreeBSD 
http://bugs.python.org/issue30331  opened by haypo

#30333: test_multiprocessing_forkserver: poll() failed on AMD64 FreeBS
http://bugs.python.org/issue30333  opened by haypo

#30335: Document deprecated alias of assertNotRegex
http://bugs.python.org/issue30335  opened by Jim Fasarakis-Hilliard

#30337: Vague wording of pkgutil.walk_packages parameter 'prefix'
http://bugs.python.org/issue30337  opened by smsilb

#30339: test_multiprocessing_main_handling: "RuntimeError: Timed out w
http://bugs.python.org/issue30339  opened by haypo

#30340: Optimize out non-capturing groups
http://bugs.python.org/issue30340  opened by serhiy.storchaka

#30341: Add an explaining comment in _PyTrash_thread_destroy_chain()
http://bugs.python.org/issue30341  opened by xiang.zhang

#30343: Subclassed json.JSONEncoder does not respect default method fo
http://bugs.python.org/issue30343  opened by Xophmeister

#30344: test_multiprocessing.test_notify_all(): AssertionError: 6 != 5
http://bugs.python.org/issue30344  opened by haypo

#30345: test_gdb fails on Python 3.6 when built with LTO+PGO
http://bugs.python.org/issue30345  opened by haypo

#30346: Odd behavior when unpacking `itertools.groupby`
http://bugs.python.org/issue30346  opened by Matt Gilson

#30347: itertools.groupby() can fail a C assert()
http://bugs.python.org/issue30347  opened by arigo

#30348: IDLE: Add fetch completions and get entity unittest
http://bugs.python.org/issue30348  opened by louielu

#30349: Preparation for advanced set syntax in regular expressions
http://bugs.python.org/issue30349  opened by serhiy.storchaka

#30350: devguide suggests to use VS 2008 to build Python 2.7, but VS 2
http://bugs.python.org/issue30350  opened by haypo

#30351: regrtest hangs on x86 Windows XP 2.7
http://bugs.python.org/issue30351  opened by haypo

#30352: The 'in' syntax should work with any object that implements __
http://bugs.python.org/issue30352  opened by Edouard KLEIN



Most recent 15 issues with no replies (15)
==========================================

#30351: regrtest hangs on x86 Windows XP 2.7
http://bugs.python.org/issue30351

#30349: Preparation for advanced set syntax in regular expressions
http://bugs.python.org/issue30349

#30348: IDLE: Add fetch completions and get entity unittest
http://bugs.python.org/issue30348

#30341: Add an explaining comment in _PyTrash_thread_destroy_chain()
http://bugs.python.org/issue30341

#30339: test_multiprocessing_main_handling: "RuntimeError: Timed out w
http://bugs.python.org/issue30339

#30333: test_multiprocessing_forkserver: poll() failed on AMD64 FreeBS
http://bugs.python.org/issue30333

#30329: test_imaplib.test_login_cram_md5(): OSError: [WinError 10022] 
http://bugs.python.org/issue30329

#30328: test_ssl.test_connect_with_context(): ConnectionResetError on 
http://bugs.python.org/issue30328

#30318: test_distutils is too verbose on Windows
http://bugs.python.org/issue30318

#30314: Buildbots: 15 min is too low for test_tools on x86 Tiger 3.6 b
http://bugs.python.org/issue30314

#30304: TestCase.assertMultiLineEqual only registered for Unicode stri
http://bugs.python.org/issue30304

#30303: IDLE: Add _utest to textview
http://bugs.python.org/issue30303

#30301: multiprocessing: AttributeError: 'SimpleQueue' object has no a
http://bugs.python.org/issue30301

#30290: IDLE: add tests for help_about.py
http://bugs.python.org/issue30290

#30282: object returned by tarfile.extractfile has an incorrect value 
http://bugs.python.org/issue30282



Most recent 15 issues waiting for review (15)
=============================================

#30349: Preparation for advanced set syntax in regular expressions
http://bugs.python.org/issue30349

#30347: itertools.groupby() can fail a C assert()
http://bugs.python.org/issue30347

#30341: Add an explaining comment in _PyTrash_thread_destroy_chain()
http://bugs.python.org/issue30341

#30340: Optimize out non-capturing groups
http://bugs.python.org/issue30340

#30325: Buildbot: send email notifications to buildbot-status@
http://bugs.python.org/issue30325

#30310: tkFont.py assumes that all font families are encoded as ascii 
http://bugs.python.org/issue30310

#30306: release arguments of contextmanager
http://bugs.python.org/issue30306

#30299: Display the bytecode when compiled a regular expression in deb
http://bugs.python.org/issue30299

#30262: Don't expose sqlite3 Cache and Statement
http://bugs.python.org/issue30262

#30248: Using boolean arguments in the _json module
http://bugs.python.org/issue30248

#30246: fix several error messages in struct
http://bugs.python.org/issue30246

#30242: resolve undefined behaviour in struct
http://bugs.python.org/issue30242

#30231: test_imaplib needs a TLS server accepting self-signed certific
http://bugs.python.org/issue30231

#30224: remove outdated checks in struct
http://bugs.python.org/issue30224

#30219: webbrowser.open outputs xdg-open deprecation warning
http://bugs.python.org/issue30219



Top 10 most discussed issues (10)
=================================

#30283: [2.7] Backport test_regrtest (partially) on Python 2.7
http://bugs.python.org/issue30283  14 msgs

#15786: IDLE code completion window can hang or misbehave with mouse
http://bugs.python.org/issue15786  10 msgs

#29243: --enable-optimizations makes common build commands always need
http://bugs.python.org/issue29243  10 msgs

#29447: Add/check os.PathLike support for the tempfile module's 'dir' 
http://bugs.python.org/issue29447   7 msgs

#30181: Correct the parsing of a test case docstring.
http://bugs.python.org/issue30181   7 msgs

#30219: webbrowser.open outputs xdg-open deprecation warning
http://bugs.python.org/issue30219   7 msgs

#30262: Don't expose sqlite3 Cache and Statement
http://bugs.python.org/issue30262   7 msgs

#30273: The coverage job is broken: distutils build_ext fails on None
http://bugs.python.org/issue30273   6 msgs

#30291: Allow windows launcher to specify bit lengths with & without m
http://bugs.python.org/issue30291   6 msgs

#30346: Odd behavior when unpacking `itertools.groupby`
http://bugs.python.org/issue30346   6 msgs



Issues closed (38)
==================

#26336: Expose regex bytecode of compiled pattern object
http://bugs.python.org/issue26336  closed by terry.reedy

#26700: Make digest_size a class variable
http://bugs.python.org/issue26700  closed by gregory.p.smith

#28787: Out of tree --with--dtrace builds fail with a traceback
http://bugs.python.org/issue28787  closed by haypo

#29823: mimetypes guesses XSL mimetype when passed an XML file
http://bugs.python.org/issue29823  closed by martin.panter

#29956: math.exp documentation is misleading
http://bugs.python.org/issue29956  closed by belopolsky

#29979: cgi.parse_multipart is not consistent with FieldStorage
http://bugs.python.org/issue29979  closed by orsenthil

#29990: Range checking in GB18030 decoder
http://bugs.python.org/issue29990  closed by xiang.zhang

#30024: Treat `import a.b.c as m` as `m = sys.modules['a.b.c']`
http://bugs.python.org/issue30024  closed by serhiy.storchaka

#30048: If a task is canceled at the right moment, the cancellation is
http://bugs.python.org/issue30048  closed by inada.naoki

#30168: Class Logger is unindented in the documentation.
http://bugs.python.org/issue30168  closed by vinay.sajip

#30218: shutil.unpack_archive doesn't support PathLike
http://bugs.python.org/issue30218  closed by brett.cannon

#30272: distutils.version.LooseVersion's compare raises exception in c
http://bugs.python.org/issue30272  closed by berker.peksag

#30275: pickle doesn't work in compile/exec
http://bugs.python.org/issue30275  closed by brett.cannon

#30276: import hashlib makes many programs slow
http://bugs.python.org/issue30276  closed by bmwiedemann

#30281: set stop default to -PY_SSIZE_T_MAX-1 in PySlice_Unpack
http://bugs.python.org/issue30281  closed by xiang.zhang

#30285: Optimize case-insensitive regular expressions
http://bugs.python.org/issue30285  closed by serhiy.storchaka

#30286: ctypes FreeLibrary fails on Windows 64-bit
http://bugs.python.org/issue30286  closed by giampaolo.rodola

#30288: ssl.wrap_ssl will fail on do_handshake if default parameters a
http://bugs.python.org/issue30288  closed by botter

#30289: make distclean and Misc/python-config.sh
http://bugs.python.org/issue30289  closed by xiang.zhang

#30292: Why is there a concept "unbound method" in <Descriptor HowTo G
http://bugs.python.org/issue30292  closed by martin.panter

#30293: Peephole binops folding can lead to memory and bytecache ballo
http://bugs.python.org/issue30293  closed by rhettinger

#30297: Recursive starmap causes Segmentation fault
http://bugs.python.org/issue30297  closed by rhettinger

#30298: Weak deprecations for inline regular expression modifiers
http://bugs.python.org/issue30298  closed by serhiy.storchaka

#30305: python 2.7.13 join issue
http://bugs.python.org/issue30305  closed by ezio.melotti

#30307: https://docs.python.org/3/tutorial/introduction.html#strings S
http://bugs.python.org/issue30307  closed by serhiy.storchaka

#30308: Add code coverage for argument in random.shuffle
http://bugs.python.org/issue30308  closed by rhettinger

#30311: random.shuffle pointlessly shuffles dicts
http://bugs.python.org/issue30311  closed by rhettinger

#30320: test_eintr.test_sigwaitinfo(): race condition on AMD64 FreeBSD
http://bugs.python.org/issue30320  closed by haypo

#30321: format() function prints fillchar as backslash twice
http://bugs.python.org/issue30321  closed by zach.ware

#30322: PyObject_GetIter does not behave as documented on dict objects
http://bugs.python.org/issue30322  closed by Sam De Meyer

#30324: Error using newline='' when writing to CSV file
http://bugs.python.org/issue30324  closed by martin.panter

#30326: Version may be out of date in travis.yml
http://bugs.python.org/issue30326  closed by Jensen Taylor

#30327: Version may be out of date in travis.yml
http://bugs.python.org/issue30327  closed by brett.cannon

#30332: Errors that aren't actually defined are purple in the IDLE
http://bugs.python.org/issue30332  closed by terry.reedy

#30334: [Windows] support.rmtree() should retry on WindowsError: [Erro
http://bugs.python.org/issue30334  closed by haypo

#30336: Pull Requests need more detail
http://bugs.python.org/issue30336  closed by brett.cannon

#30338: LC_ALL=en_US + io.open() => LookupError: (osx)
http://bugs.python.org/issue30338  closed by haypo

#30342: [2.7] sysconfig.is_python_build() doesn't work if Python is bu
http://bugs.python.org/issue30342  closed by haypo

From deshpande.v.pranav at gmail.com  Sun May 14 11:04:33 2017
From: deshpande.v.pranav at gmail.com (Pranav Deshpande)
Date: Sun, 14 May 2017 20:34:33 +0530
Subject: [Python-Dev] Python Tests
Message-ID: <CANFkxPLorGZmmupzVHkQmYiz6NhKLk+v7oKiXg3H6+enUreXSw@mail.gmail.com>

Hello, everyone. I wanted to contribute to Python so I began by following
the steps given here: https://docs.python.org/devguide/

WHile executing

./python -m test -j3

I encountered the following error:
383 tests OK.

1 test failed:
    test_re

21 tests skipped:
    test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_kqueue
    test_msilib test_ossaudiodev test_smtpnet test_socketserver
    test_startfile test_timeout test_tix test_tk test_ttk_guionly
    test_urllib2net test_urllibnet test_winconsoleio test_winreg
    test_winsound test_xmlrpc_net test_zipfile64

Total duration: 24 min 25 sec
Tests result: FAILURE

How do I resolve this?

Regards,

Pranav.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170514/209d9b4e/attachment.html>

From storchaka at gmail.com  Sun May 14 12:08:37 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Sun, 14 May 2017 19:08:37 +0300
Subject: [Python-Dev] Python Tests
In-Reply-To: <CANFkxPLorGZmmupzVHkQmYiz6NhKLk+v7oKiXg3H6+enUreXSw@mail.gmail.com>
References: <CANFkxPLorGZmmupzVHkQmYiz6NhKLk+v7oKiXg3H6+enUreXSw@mail.gmail.com>
Message-ID: <of9ve0$qag$1@blaine.gmane.org>

On 14.05.17 18:04, Pranav Deshpande wrote:
> Hello, everyone. I wanted to contribute to Python so I began by
> following the steps given here: https://docs.python.org/devguide/
>
> WHile executing
>
> ./python -m test -j3
>
> I encountered the following error:
> 383 tests OK.
>
> 1 test failed:
>     test_re
>
> 21 tests skipped:
>     test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_kqueue
>     test_msilib test_ossaudiodev test_smtpnet test_socketserver
>     test_startfile test_timeout test_tix test_tk test_ttk_guionly
>     test_urllib2net test_urllibnet test_winconsoleio test_winreg
>     test_winsound test_xmlrpc_net test_zipfile64
>
> Total duration: 24 min 25 sec
> Tests result: FAILURE
>
> How do I resolve this?

Run the failed test in verbose mode:

./python -m test -v test_re

Get the traceback, search on the bug tracker whether this bug already 
was reported (I suppose you have the en_IN locale and this is issue29571 
[1]), and if nothing is found open a new ticket. Now read the code and 
try to find the cause of the bug and write a patch that fixes it. If 
this is issue29571, the cause is known, and several solutions was 
proposed, but the problem is deeper.

[1] https://bugs.python.org/issue29571



From guido at python.org  Sun May 14 12:12:46 2017
From: guido at python.org (Guido van Rossum)
Date: Sun, 14 May 2017 09:12:46 -0700
Subject: [Python-Dev] Python Tests
In-Reply-To: <CANFkxPLorGZmmupzVHkQmYiz6NhKLk+v7oKiXg3H6+enUreXSw@mail.gmail.com>
References: <CANFkxPLorGZmmupzVHkQmYiz6NhKLk+v7oKiXg3H6+enUreXSw@mail.gmail.com>
Message-ID: <CAP7+vJ+ZgAzCshUuWTr1tF=Ah_O72EgQv5TOBcxbu4jMaJn+Cw@mail.gmail.com>

What is it that you wanted to contribute? Unless it affects regular
expressions  you probably needn't worry about that one test. (If you do,
run the test by itself in verbose mode so you can debug why it is failing.)

On Sun, May 14, 2017 at 8:04 AM, Pranav Deshpande <
deshpande.v.pranav at gmail.com> wrote:

> Hello, everyone. I wanted to contribute to Python so I began by following
> the steps given here: https://docs.python.org/devguide/
>
> WHile executing
>
> ./python -m test -j3
>
> I encountered the following error:
> 383 tests OK.
>
> 1 test failed:
>     test_re
>
> 21 tests skipped:
>     test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_kqueue
>     test_msilib test_ossaudiodev test_smtpnet test_socketserver
>     test_startfile test_timeout test_tix test_tk test_ttk_guionly
>     test_urllib2net test_urllibnet test_winconsoleio test_winreg
>     test_winsound test_xmlrpc_net test_zipfile64
>
> Total duration: 24 min 25 sec
> Tests result: FAILURE
>
> How do I resolve this?
>
> Regards,
>
> Pranav.
>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170514/da782ba0/attachment.html>

From deshpande.v.pranav at gmail.com  Sun May 14 14:24:28 2017
From: deshpande.v.pranav at gmail.com (Pranav Deshpande)
Date: Sun, 14 May 2017 23:54:28 +0530
Subject: [Python-Dev] Python Tests
In-Reply-To: <CAP7+vJ+ZgAzCshUuWTr1tF=Ah_O72EgQv5TOBcxbu4jMaJn+Cw@mail.gmail.com>
References: <CANFkxPLorGZmmupzVHkQmYiz6NhKLk+v7oKiXg3H6+enUreXSw@mail.gmail.com>
 <CAP7+vJ+ZgAzCshUuWTr1tF=Ah_O72EgQv5TOBcxbu4jMaJn+Cw@mail.gmail.com>
Message-ID: <CANFkxP+-es3MRjtWCKS=HF=CRmxLAmNqSiGXhea1HQE-wj-z8A@mail.gmail.com>

I just wanted to start contributing to open-source; that is all. I was
going to ask for guidance after I completed the initial steps( search for
beginner friendly bugs) and get a general idea of the code. How do I go
about doing this?

Regards,
Pranav

On Sun, May 14, 2017 at 9:42 PM, Guido van Rossum <guido at python.org> wrote:

> What is it that you wanted to contribute? Unless it affects regular
> expressions  you probably needn't worry about that one test. (If you do,
> run the test by itself in verbose mode so you can debug why it is failing.)
>
> On Sun, May 14, 2017 at 8:04 AM, Pranav Deshpande <
> deshpande.v.pranav at gmail.com> wrote:
>
>> Hello, everyone. I wanted to contribute to Python so I began by following
>> the steps given here: https://docs.python.org/devguide/
>>
>> WHile executing
>>
>> ./python -m test -j3
>>
>> I encountered the following error:
>> 383 tests OK.
>>
>> 1 test failed:
>>     test_re
>>
>> 21 tests skipped:
>>     test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_kqueue
>>     test_msilib test_ossaudiodev test_smtpnet test_socketserver
>>     test_startfile test_timeout test_tix test_tk test_ttk_guionly
>>     test_urllib2net test_urllibnet test_winconsoleio test_winreg
>>     test_winsound test_xmlrpc_net test_zipfile64
>>
>> Total duration: 24 min 25 sec
>> Tests result: FAILURE
>>
>> How do I resolve this?
>>
>> Regards,
>>
>> Pranav.
>>
>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
>> 40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170514/ff8a3abe/attachment.html>

From camillamon at gmail.com  Sun May 14 14:38:55 2017
From: camillamon at gmail.com (Camilla)
Date: Sun, 14 May 2017 19:38:55 +0100
Subject: [Python-Dev] Python Tests
In-Reply-To: <CANFkxP+-es3MRjtWCKS=HF=CRmxLAmNqSiGXhea1HQE-wj-z8A@mail.gmail.com>
References: <CANFkxPLorGZmmupzVHkQmYiz6NhKLk+v7oKiXg3H6+enUreXSw@mail.gmail.com>
 <CAP7+vJ+ZgAzCshUuWTr1tF=Ah_O72EgQv5TOBcxbu4jMaJn+Cw@mail.gmail.com>
 <CANFkxP+-es3MRjtWCKS=HF=CRmxLAmNqSiGXhea1HQE-wj-z8A@mail.gmail.com>
Message-ID: <CAHh6JBx0vn+3iFJiieoOyLJc+EVArBN1gZZVVqeHY9PveEQgHQ@mail.gmail.com>

hi Pranav,
the core-mentorship mailing list is a friendly group that can help you get
started and answer questions. You can find out more about the Python Core
Mentorship <https://www.python.org/dev/core-mentorship/> initiative here.

Kind regards,
Camilla

2017-05-14 19:24 GMT+01:00 Pranav Deshpande <deshpande.v.pranav at gmail.com>:

> I just wanted to start contributing to open-source; that is all. I was
> going to ask for guidance after I completed the initial steps( search for
> beginner friendly bugs) and get a general idea of the code. How do I go
> about doing this?
>
> Regards,
> Pranav
>
> On Sun, May 14, 2017 at 9:42 PM, Guido van Rossum <guido at python.org>
> wrote:
>
>> What is it that you wanted to contribute? Unless it affects regular
>> expressions  you probably needn't worry about that one test. (If you do,
>> run the test by itself in verbose mode so you can debug why it is failing.)
>>
>> On Sun, May 14, 2017 at 8:04 AM, Pranav Deshpande <
>> deshpande.v.pranav at gmail.com> wrote:
>>
>>> Hello, everyone. I wanted to contribute to Python so I began by
>>> following the steps given here: https://docs.python.org/devguide/
>>>
>>> WHile executing
>>>
>>> ./python -m test -j3
>>>
>>> I encountered the following error:
>>> 383 tests OK.
>>>
>>> 1 test failed:
>>>     test_re
>>>
>>> 21 tests skipped:
>>>     test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_kqueue
>>>     test_msilib test_ossaudiodev test_smtpnet test_socketserver
>>>     test_startfile test_timeout test_tix test_tk test_ttk_guionly
>>>     test_urllib2net test_urllibnet test_winconsoleio test_winreg
>>>     test_winsound test_xmlrpc_net test_zipfile64
>>>
>>> Total duration: 24 min 25 sec
>>> Tests result: FAILURE
>>>
>>> How do I resolve this?
>>>
>>> Regards,
>>>
>>> Pranav.
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: https://mail.python.org/mailma
>>> n/options/python-dev/guido%40python.org
>>>
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> camillamon%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170514/12ec4506/attachment.html>

From deshpande.v.pranav at gmail.com  Sun May 14 15:02:02 2017
From: deshpande.v.pranav at gmail.com (Pranav Deshpande)
Date: Mon, 15 May 2017 00:32:02 +0530
Subject: [Python-Dev] Python Tests
In-Reply-To: <CAHh6JBx0vn+3iFJiieoOyLJc+EVArBN1gZZVVqeHY9PveEQgHQ@mail.gmail.com>
References: <CANFkxPLorGZmmupzVHkQmYiz6NhKLk+v7oKiXg3H6+enUreXSw@mail.gmail.com>
 <CAP7+vJ+ZgAzCshUuWTr1tF=Ah_O72EgQv5TOBcxbu4jMaJn+Cw@mail.gmail.com>
 <CANFkxP+-es3MRjtWCKS=HF=CRmxLAmNqSiGXhea1HQE-wj-z8A@mail.gmail.com>
 <CAHh6JBx0vn+3iFJiieoOyLJc+EVArBN1gZZVVqeHY9PveEQgHQ@mail.gmail.com>
Message-ID: <CANFkxP+Nc8Sg9cp_LvzAFQ9AuOFGnmFgLeN1R0YJX3g=RtC3xA@mail.gmail.com>

Thank you. I redirecting my questions over there now.

Regards,
Pranav.

On Mon, May 15, 2017 at 12:08 AM, Camilla <camillamon at gmail.com> wrote:

> hi Pranav,
> the core-mentorship mailing list is a friendly group that can help you get
> started and answer questions. You can find out more about the Python Core
> Mentorship <https://www.python.org/dev/core-mentorship/> initiative here.
>
> Kind regards,
> Camilla
>
> 2017-05-14 19:24 GMT+01:00 Pranav Deshpande <deshpande.v.pranav at gmail.com>
> :
>
>> I just wanted to start contributing to open-source; that is all. I was
>> going to ask for guidance after I completed the initial steps( search for
>> beginner friendly bugs) and get a general idea of the code. How do I go
>> about doing this?
>>
>> Regards,
>> Pranav
>>
>> On Sun, May 14, 2017 at 9:42 PM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> What is it that you wanted to contribute? Unless it affects regular
>>> expressions  you probably needn't worry about that one test. (If you do,
>>> run the test by itself in verbose mode so you can debug why it is failing.)
>>>
>>> On Sun, May 14, 2017 at 8:04 AM, Pranav Deshpande <
>>> deshpande.v.pranav at gmail.com> wrote:
>>>
>>>> Hello, everyone. I wanted to contribute to Python so I began by
>>>> following the steps given here: https://docs.python.org/devguide/
>>>>
>>>> WHile executing
>>>>
>>>> ./python -m test -j3
>>>>
>>>> I encountered the following error:
>>>> 383 tests OK.
>>>>
>>>> 1 test failed:
>>>>     test_re
>>>>
>>>> 21 tests skipped:
>>>>     test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_kqueue
>>>>     test_msilib test_ossaudiodev test_smtpnet test_socketserver
>>>>     test_startfile test_timeout test_tix test_tk test_ttk_guionly
>>>>     test_urllib2net test_urllibnet test_winconsoleio test_winreg
>>>>     test_winsound test_xmlrpc_net test_zipfile64
>>>>
>>>> Total duration: 24 min 25 sec
>>>> Tests result: FAILURE
>>>>
>>>> How do I resolve this?
>>>>
>>>> Regards,
>>>>
>>>> Pranav.
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Python-Dev mailing list
>>>> Python-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/python-dev
>>>> Unsubscribe: https://mail.python.org/mailma
>>>> n/options/python-dev/guido%40python.org
>>>>
>>>>
>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/camillamo
>> n%40gmail.com
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170515/6639ff6e/attachment.html>

From julien at palard.fr  Sun May 14 16:07:43 2017
From: julien at palard.fr (Julien Palard)
Date: Sun, 14 May 2017 16:07:43 -0400
Subject: [Python-Dev] PEP 545: Python Documentation Translations
In-Reply-To: <CAP1=2W4vjKydG82ZcxpGmJsDJ5aw-hoDkEyW5eU6nCbLyUvGYA@mail.gmail.com>
References: <serPZ8e15GRZpFGH3mQaywZvDz-tz4ZYcF2iNhDdrX1njbKzMLfduqvBvzPRTTABOyDCugVKznYNcp8Oay1uU8umLgJKamuCW3mlOAZxGQs=@palard.fr>
 <CAP1=2W5QpyMJAKxtEqCvMGJ8TC+GssAHx3LCX7JgtaE3SfDA8w@mail.gmail.com>
 <6-6zandAUD_HqCXBfrlg0-lw5H9zwipt2GRfqW2ZptmVLdHkeFVyc7lvfIawnnm0qo3pV-CKdvXTmWenV-uA3IvxjGFdHMMSzdPYCcautMU=@palard.fr>
 <CAP1=2W4vjKydG82ZcxpGmJsDJ5aw-hoDkEyW5eU6nCbLyUvGYA@mail.gmail.com>
Message-ID: <A1eY2R3txj6gJS2yeRD871VWg_LGOig9MxivaQFh2sM5nuSOc-brnRUzetRaPRowfMh6sTrjydPBm1EfE01eHmMUWVPAcytPw4t8Ys5Tptg=@palard.fr>

Hi Brett,

On Wed, 29 Mar 2017 at 13:13 Julien Palard <julien at palard.fr> wrote:

What kind of support does Read the Docs have for translations? I have no active plans to push for this but it has been idea in the back of my head for a while so it would be good to know if such a move would make this easier or harder.

[...]

Their language tags are simplified too (redundency removed (fr-FR ? fr)) but not lowercased, and they
use underscore "instead of" dashes as a separator, see for example:

- https://docs.phpmyadmin.net/fr/latest/
- https://docs.phpmyadmin.net/pt_BR/latest/

while the PEP proposes /pt-br/ instead.

.. [1] Project with multiple translations
(https://docs.readthedocs.io/en/latest/localization.html#project-with-multiple-translations)

Should we just match what Read the Docs does then?

As I prefer our notation (/pt-br/) over Read the Docs one (/pt_BR/), I'd prefer if we stick on our notation. They are not against changing and looks like they're OK with our notation: https://github.com/rtfd/readthedocs.org/issues/2763

--
Julien Palard
https://mdk.fr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170514/13d1b9b3/attachment.html>

From k7hoven at gmail.com  Mon May 15 04:48:57 2017
From: k7hoven at gmail.com (Koos Zevenhoven)
Date: Mon, 15 May 2017 11:48:57 +0300
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
Message-ID: <CAMiohoiXuAZmzyJYBx6jVVfWj5=duLuRAMuFT6JfEC=vM+x5gw@mail.gmail.com>

On Tue, May 9, 2017 at 8:19 PM, Guido van Rossum <guido at python.org> wrote:
> There's a PR to the peps proposal here:
> https://github.com/python/peps/pull/242
>
> The full text of the current proposal is below. The motivation for this is
> that for complex decorators, even if the type checker can figure out what's
> going on (by taking the signature of the decorator into account), it's
> sometimes helpful to the human reader of the code to be reminded of the type
> after applying the decorators (or a stack thereof). Much discussion can be
> found in the PR. Note that we ended up having `Callable` in the type because
> there's no rule that says a decorator returns a function type (e.g.
> `property` doesn't).
>
> This is a small thing but I'd like to run it by a larger audience than the
> core mypy devs who have commented so far. There was a brief discussion on
> python-ideas (my original, favorable reply by Nick, my response).
>
> Credit for the proposal goes to Naomi Seyfer, with discussion by Ivan
> Levkivskyi and Jukka Lehtosalo.
>
> If there's no further debate here I'll merge it into the PEP and an
> implementation will hopefully appear in the next version of the typing
> module (also hopefully to be included in CPython 3.6.2 and 3.5.4).
>

So the change would only affect early adopters of this typing feature,
who are likely to upgrade to newer python versions often? Could this
be called a 3.7 feature with a clearly documented bonus that it also
works in 3.6.2+ and 3.5.4+? I mean, to prevent 3rd-party libraries
tested with 3.5(.4) from being broken in 3.5.3?

> Here's the proposed text (wordsmithing suggestions in the PR please):
>
> +Decorators
> +----------
> +
> +Decorators can modify the types of the functions or classes they
> +decorate. Use the ``decorated_type`` decorator to declare the type of
> +the resulting item after all other decorators have been applied::
> +
> + from typing import ContextManager, Iterator, decorated_type
> + from contextlib import contextmanager
> +
> + class DatabaseSession: ...
> +
> + @decorated_type(Callable[[str], ContextManager[DatabaseSession]])
> + @contextmanager
> + def session(url: str) -> Iterator[DatabaseSession]:
> + s = DatabaseSession(url)
> + try:
> + yield s
> + finally:
> + s.close()
> +
> +The argument of ``decorated_type`` is a type annotation on the name
> +being declared (``session``, in the example above). If you have
> +multiple decorators, ``decorated_type`` must be topmost. The
> +``decorated_type`` decorator is invalid on a function declaration that
> +is also decorated with ``overload``, but you can annotate the
> +implementation of the overload series with ``decorated_type``.
> +
>

Would __annotations__ be set by the decorator? To me, not setting them
would seem weird, but cases where the result is not a function could
be weird. I also don't see a mention of this only working in stubs.

I like Jukka's version, as it has a clear distinction between
functions and other attributes. But that might require a language
change to provide __annotations__ in a clean manner? Maybe that
language change would be useful elsewhere.

?Koos



-- 
+ Koos Zevenhoven + http://twitter.com/k7hoven +

From levkivskyi at gmail.com  Mon May 15 09:20:26 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Mon, 15 May 2017 15:20:26 +0200
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CAMiohoiXuAZmzyJYBx6jVVfWj5=duLuRAMuFT6JfEC=vM+x5gw@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
 <CAMiohoiXuAZmzyJYBx6jVVfWj5=duLuRAMuFT6JfEC=vM+x5gw@mail.gmail.com>
Message-ID: <CAOMjWkmmfGSEu1mjVzPYE_BT5g06RSJZB02hz0nouAcJT62J_g@mail.gmail.com>

On 15 May 2017 at 10:48, Koos Zevenhoven <k7hoven at gmail.com> wrote:

> > Here's the proposed text (wordsmithing suggestions in the PR please):
> >
> > +Decorators
> > +----------
> > +
> > +Decorators can modify the types of the functions or classes they
> > +decorate. Use the ``decorated_type`` decorator to declare the type of
> > +the resulting item after all other decorators have been applied::
> > +
> > + from typing import ContextManager, Iterator, decorated_type
> > + from contextlib import contextmanager
> > +
> > + class DatabaseSession: ...
> > +
> > + @decorated_type(Callable[[str], ContextManager[DatabaseSession]])
> > + @contextmanager
> > + def session(url: str) -> Iterator[DatabaseSession]:
> > + s = DatabaseSession(url)
> > + try:
> > + yield s
> > + finally:
> > + s.close()
> > +
> > +The argument of ``decorated_type`` is a type annotation on the name
> > +being declared (``session``, in the example above). If you have
> > +multiple decorators, ``decorated_type`` must be topmost. The
> > +``decorated_type`` decorator is invalid on a function declaration that
> > +is also decorated with ``overload``, but you can annotate the
> > +implementation of the overload series with ``decorated_type``.
> > +
> >
>
> Would __annotations__ be set by the decorator? To me, not setting them
> would seem weird, but cases where the result is not a function could
> be weird. I also don't see a mention of this only working in stubs.
>
> I like Jukka's version, as it has a clear distinction between
> functions and other attributes. But that might require a language
> change to provide __annotations__ in a clean manner? Maybe that
> language change would be useful elsewhere.


With original syntax it is possible to overwrite annotations without
language changes.
However with Jukka's syntax it looks difficult. A possible pure-Python way
could be
to insert an item in an enclosing __annotations__, if an enclosing scope
is a class or module scope.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170515/acd23ce9/attachment.html>

From freddyrietdijk at fridh.nl  Mon May 15 03:05:05 2017
From: freddyrietdijk at fridh.nl (Freddy Rietdijk)
Date: Mon, 15 May 2017 09:05:05 +0200
Subject: [Python-Dev] Hash randomization and deterministic bytecode
In-Reply-To: <CAP7+vJLbKgKS=oOoKCWVCYE0khsJ7mTwsAWHLrE7KHw9L4t2MQ@mail.gmail.com>
References: <CAOQtOH2XSE3UQhD8SKKn8HLEWgnAOZ4F0WzGyKh+4mHnAtqGmg@mail.gmail.com>
 <CAP7+vJLbKgKS=oOoKCWVCYE0khsJ7mTwsAWHLrE7KHw9L4t2MQ@mail.gmail.com>
Message-ID: <CAOQtOH2rRVVwsgPaUYza9wdqCXsGZcmOhFqpVBXiSO=fDEgMXw@mail.gmail.com>

Thanks for the clarification. I'm glad we can keep hash randomization
disabled during our builds.

On Fri, May 12, 2017 at 5:22 PM, Guido van Rossum <guido at python.org> wrote:

> Don't worry, the PYTHONHASHSEED setting does not get recorded in the
> bytecode header and the generated bytecode (even if it sometimes differs in
> trivial ways) is usable with all hash seed settings.
>
> --Guido
>
> On Fri, May 12, 2017 at 6:06 AM, Freddy Rietdijk <freddyrietdijk at fridh.nl>
> wrote:
>
>> Hi,
>>
>> On Nix we set PYTHONHASHSEED to 0 when building packages, disabling hash
>> randomization. We do this to improve determinism of the builds because we
>> store the bytecode next to the code.
>>
>> When one runs Python directly or via a script PYTHONHASHSEED is not set
>> thus enabling hash randomization. Am I correct when I say that in this case
>> Python still uses the reproducibly build bytecode and, because its now
>> running with a random seed we wouldn't be vulnerable to
>> http://www.ocert.org/advisories/ocert-2011-003.html ? Or would it also
>> try to each time also recompile bytecode?
>>
>> Kind regards,
>>
>> Freddy
>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
>> 40python.org
>>
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170515/df24b16d/attachment.html>

From deshpande.v.pranav at gmail.com  Tue May 16 09:55:52 2017
From: deshpande.v.pranav at gmail.com (Pranav Deshpande)
Date: Tue, 16 May 2017 19:25:52 +0530
Subject: [Python-Dev] Issue18299 Change script_helper to use
 universal_newlines=True in _assert_python
Message-ID: <CANFkxP+USJUAer6kJ-kd8rLUoYbo1xeMxzmrDZUX1EU7pAxOJw@mail.gmail.com>

I am a beginner when it comes to open source contribution and have decided
to take up this issue. I did some basic research about the issue found this
file:

cpython/Lib/test/support/script_helper.py

The function _assert_python call run_python_until_end which call
subprocess.Popen which takes the parameter universal_newlines=True.

But I think that I don't understand this issue. Could anyone guide me from
this point forward?

Regards,
Pranav
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170516/c86d4163/attachment.html>

From martin.wimpress at canonical.com  Tue May 16 06:31:42 2017
From: martin.wimpress at canonical.com (Martin Wimpress)
Date: Tue, 16 May 2017 11:31:42 +0100
Subject: [Python-Dev] Snap Python for simple distribution across multiple
 Linux distros
Message-ID: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>

Hi all,

I work at Canonical as part of the engineering team developing Ubuntu
and Snapcraft [1] and I'm a long time Python fan :-)

We've created snaps, a platform that enables projects to directly
control delivery of software updates to users. This video of a
lightning talk by dlang developers at DConf2017 [2] shows how they've
made good use of snaps to distribute their compiler. They found the
release channels particularly useful so their users can track a
specific release.

Is there someone here who'd be interested in doing the same for Python?

[1] https://snapcraft.io/
[2] https://www.youtube.com/watch?v=M-bDzr4gYUU
[3] https://snapcraft.io/docs/core/install
[4] https://build.snapcraft.io/

-- 
Regards, Martin.

From rodrigc at crodrigues.org  Wed May 17 17:41:29 2017
From: rodrigc at crodrigues.org (Craig Rodrigues)
Date: Wed, 17 May 2017 14:41:29 -0700
Subject: [Python-Dev] Format strings, Unicode, and Py2.7: need clarification
Message-ID: <CAG=rPVeak6ZySBdVW1th9K+o=nSSSV6suDTOr8THmSF2qm0i5w@mail.gmail.com>

Hi,

While cleaning up some code during Python 2 -> Python 3 porting,
I switched some code to use str.format(), I found this behavor:

Python 2.7
=========
a = "%s" % "hi"
b = "%s" % u"hi"
c = u"%s" % "hi"
d = "{}".format("hi")
e = "{}".format(u"hi")
f = u"{}".format("hi")

type(a) == str
type(b) == unicode
type(c) == unicode
type(d) == str
type(e) == str
type(f) == unicode

My intuition would lead me to believe that type(b)
and type(e) would be the same (unicode), but they are not.
The confusion for me is why is type(e) of type str, and not unicode?

Can someone clarify this for me?

I understand that in Python 3, all these cases are str, so it is not
as big a problem there, but I am trying to keep things working on
Python 2.7.

Thanks.
--
Craig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170517/383b0176/attachment.html>

From steve at pearwood.info  Wed May 17 20:41:12 2017
From: steve at pearwood.info (Steven D'Aprano)
Date: Thu, 18 May 2017 10:41:12 +1000
Subject: [Python-Dev] Format strings, Unicode,
 and Py2.7: need clarification
In-Reply-To: <CAG=rPVeak6ZySBdVW1th9K+o=nSSSV6suDTOr8THmSF2qm0i5w@mail.gmail.com>
References: <CAG=rPVeak6ZySBdVW1th9K+o=nSSSV6suDTOr8THmSF2qm0i5w@mail.gmail.com>
Message-ID: <20170518004112.GR24625@ando.pearwood.info>

On Wed, May 17, 2017 at 02:41:29PM -0700, Craig Rodrigues wrote:

> e = "{}".format(u"hi")
[...]
> type(e) == str

> The confusion for me is why is type(e) of type str, and not unicode?

I think that's one of the reasons why the Python 2.7 string model is (1) 
convenient to those using purely ASCII, but (2) ultimately broken.

You can see why it's broken if you do this:

py> "{}".format(u"hi?")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character u'\xb5' in 
position 2: ordinal not in range(128)


So it tries to encode the Unicode string to ASCII, and if that succeeds, 
format returns a byte str. I'm not sure if that was a deliberate design 
choice for format, or just a side-effect of it calling str() on its 
arguments by default.

I'm not sure if I've answered your question or not. Are you looking for 
justification of this misfeature, or an explanation of the historical 
reasons why it exists, or something else?


(If you're looking for the same behaviour in Python 3 and 2.7, probably 
the best thing you can do is just religiously use unicode strings u'' in 
both. You might try:

from __future__ import unicode_literals

in 2.7, but I'm not sure that's enough.)


-- 
Steve

From eric at trueblade.com  Wed May 17 21:56:59 2017
From: eric at trueblade.com (Eric V. Smith)
Date: Wed, 17 May 2017 18:56:59 -0700
Subject: [Python-Dev] Format strings, Unicode,
 and Py2.7: need clarification
In-Reply-To: <CAG=rPVeak6ZySBdVW1th9K+o=nSSSV6suDTOr8THmSF2qm0i5w@mail.gmail.com>
References: <CAG=rPVeak6ZySBdVW1th9K+o=nSSSV6suDTOr8THmSF2qm0i5w@mail.gmail.com>
Message-ID: <EA6411AD-78F2-4C67-8BD2-C616B33F9CED@trueblade.com>


> On May 17, 2017, at 2:41 PM, Craig Rodrigues <rodrigc at crodrigues.org> wrote:
> 
> Hi,
> 
> While cleaning up some code during Python 2 -> Python 3 porting,
> I switched some code to use str.format(), I found this behavor:
> 
> Python 2.7
> =========
> a = "%s" % "hi"
> b = "%s" % u"hi"
> c = u"%s" % "hi"
> d = "{}".format("hi")
> e = "{}".format(u"hi")
> f = u"{}".format("hi")
> 
> type(a) == str
> type(b) == unicode
> type(c) == unicode
> type(d) == str
> type(e) == str
> type(f) == unicode
> 
> My intuition would lead me to believe that type(b)
> and type(e) would be the same (unicode), but they are not.
> The confusion for me is why is type(e) of type str, and not unicode?
> 
> Can someone clarify this for me?

I think it's because I wanted to return str if possible, and didn't want to find out that one of the calls to __format__ returned unicode, and then go back and convert all of the previous results to unicode from str.

And, I guess we didn't consider it important enough at the time.

Eric. 

> I understand that in Python 3, all these cases are str, so it is not
> as big a problem there, but I am trying to keep things working on
> Python 2.7.
> 
> Thanks.
> --
> Craig
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com


From hobsonlane at gmail.com  Thu May 18 01:14:47 2017
From: hobsonlane at gmail.com (Hobson Lane)
Date: Wed, 17 May 2017 22:14:47 -0700
Subject: [Python-Dev] Format strings, Unicode,
 and Py2.7: need clarification
In-Reply-To: <CAG=rPVeak6ZySBdVW1th9K+o=nSSSV6suDTOr8THmSF2qm0i5w@mail.gmail.com>
References: <CAG=rPVeak6ZySBdVW1th9K+o=nSSSV6suDTOr8THmSF2qm0i5w@mail.gmail.com>
Message-ID: <CACZ_DodJpTsk12FWf2cWLadV-MD-stE+0Ja7bMw-KPaLUDsSOA@mail.gmail.com>

Because `.format()` is a method on an instantiated `str` object in e and so
must return the same type so additional str methods could be stacked on
after it, like `.format(u'hi').decode()`.  Whereas the % string
interpolation is a binary operation, so, like addition, where the more
general type can be used for the return value, analogous to `1 + 2.0`
returning a float.

--Hobson
(503) 974-6274
gh <https://github.com/hobson/> twtr <https://twitter.com/hobsonlane> li
<https://www.linkedin.com/in/hobsonlane> g+
<http://plus.google.com/+HobsonLane/> so
<http://stackoverflow.com/users/623735/hobs>

On Wed, May 17, 2017 at 2:41 PM, Craig Rodrigues <rodrigc at crodrigues.org>
wrote:

> Hi,
>
> While cleaning up some code during Python 2 -> Python 3 porting,
> I switched some code to use str.format(), I found this behavor:
>
> Python 2.7
> =========
> a = "%s" % "hi"
> b = "%s" % u"hi"
> c = u"%s" % "hi"
> d = "{}".format("hi")
> e = "{}".format(u"hi")
> f = u"{}".format("hi")
>
> type(a) == str
> type(b) == unicode
> type(c) == unicode
> type(d) == str
> type(e) == str
> type(f) == unicode
>
> My intuition would lead me to believe that type(b)
> and type(e) would be the same (unicode), but they are not.
> The confusion for me is why is type(e) of type str, and not unicode?
>
> Can someone clarify this for me?
>
> I understand that in Python 3, all these cases are str, so it is not
> as big a problem there, but I am trying to keep things working on
> Python 2.7.
>
> Thanks.
> --
> Craig
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> hobsonlane%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170517/c1d0351b/attachment.html>

From random832 at fastmail.com  Thu May 18 10:25:24 2017
From: random832 at fastmail.com (Random832)
Date: Thu, 18 May 2017 10:25:24 -0400
Subject: [Python-Dev] Format strings, Unicode,
 and Py2.7: need clarification
In-Reply-To: <CACZ_DodJpTsk12FWf2cWLadV-MD-stE+0Ja7bMw-KPaLUDsSOA@mail.gmail.com>
References: <CAG=rPVeak6ZySBdVW1th9K+o=nSSSV6suDTOr8THmSF2qm0i5w@mail.gmail.com>
 <CACZ_DodJpTsk12FWf2cWLadV-MD-stE+0Ja7bMw-KPaLUDsSOA@mail.gmail.com>
Message-ID: <1495117524.1533735.980904560.208EB6D3@webmail.messagingengine.com>

On Thu, May 18, 2017, at 01:14, Hobson Lane wrote:
> Because `.format()` is a method on an instantiated `str` object in e and
> so
> must return the same type

That it *must* return the same type is overstating the matter. Split
returns a list (and, rather like %, the list is of unicode or str
objects depending on the argument). Join will return a unicode object if
any of the elements of the sequence are unicode. I was honestly
surprised though to see that % returns unicode when formatting a unicode
value, since my mental model of %s was more like {!s} - call str() on
whatever object is at the given position in the right-hand argument.
This kind of ad hoc implementation decision (format always returns str,
other methods can return unicode,  ljust/rjust refuse to accept a
unicode character argument) is what Python 3 moved away from.

From robin.wyb at gmail.com  Thu May 18 14:52:11 2017
From: robin.wyb at gmail.com (Wen Yaobin)
Date: Thu, 18 May 2017 14:52:11 -0400
Subject: [Python-Dev] Why doesn't Python's _multiprocessing.SemLock have
 'name'?
Message-ID: <CAA4bk2ykSyr744oW_0AVequPM_D+No0FV+++7SG0qqdjSEeSJQ@mail.gmail.com>

Hi,

This is not a general Python question but about a specific detail: *Why
doesn't Python's _multiprocessing.SemLock have 'name'?*

I posted the question on Stack Overflow but so far haven't got any
response. The link is here:

http://stackoverflow.com/questions/44036757/why-doesnt-pythons-multiprocessing-semlock-have-name

This mailing group might be a better place to ask because the question is
about the internal implementation.

I don't want to paste the question text in the email body because I
included source code which might get mal-formatted when pasting. Sorry for
the inconvenience.

*Could anyone help me on the question?* I would appreciate it!! This
doesn't affect any of my work but I am really curious about the reason.

Best,
Yaobin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170518/1496c07b/attachment.html>

From hobsonlane at gmail.com  Thu May 18 20:20:58 2017
From: hobsonlane at gmail.com (Hobson Lane)
Date: Thu, 18 May 2017 17:20:58 -0700
Subject: [Python-Dev] Format strings, Unicode,
 and Py2.7: need clarification
In-Reply-To: <1495117524.1533735.980904560.208EB6D3@webmail.messagingengine.com>
References: <CAG=rPVeak6ZySBdVW1th9K+o=nSSSV6suDTOr8THmSF2qm0i5w@mail.gmail.com>
 <CACZ_DodJpTsk12FWf2cWLadV-MD-stE+0Ja7bMw-KPaLUDsSOA@mail.gmail.com>
 <1495117524.1533735.980904560.208EB6D3@webmail.messagingengine.com>
Message-ID: <CACZ_DocbFejw97UuyEdzv7FMRHTAg=pSnE1eDcMSOJvjBtXvLA@mail.gmail.com>

Yea, *must* probably should have been *should (unless indicated by the
method name)*. I forgot about the many methods that don't. But the names of
these methods, like `split()`, imply the types that they return.

Your example reminded me of the convention (outside python core) to name
getters and setters so that they imply their returned type. Similarly for
conversion methods like `to_*()`). But the name `format`, in my mind,
implies that it only changes the *contents* of the str, rather than
morphing its type.

The % example surprised me too, but it clicked for me when I realized % is
a binary operator and not a method.

--Hobson
(503) 974-6274
gh <https://github.com/hobson/> twtr <https://twitter.com/hobsonlane> li
<https://www.linkedin.com/in/hobsonlane> g+
<http://plus.google.com/+HobsonLane/> so
<http://stackoverflow.com/users/623735/hobs>

On Thu, May 18, 2017 at 7:25 AM, Random832 <random832 at fastmail.com> wrote:

> On Thu, May 18, 2017, at 01:14, Hobson Lane wrote:
> > Because `.format()` is a method on an instantiated `str` object in e and
> > so
> > must return the same type
>
> That it *must* return the same type is overstating the matter. Split
> returns a list (and, rather like %, the list is of unicode or str
> objects depending on the argument). Join will return a unicode object if
> any of the elements of the sequence are unicode. I was honestly
> surprised though to see that % returns unicode when formatting a unicode
> value, since my mental model of %s was more like {!s} - call str() on
> whatever object is at the given position in the right-hand argument.
> This kind of ad hoc implementation decision (format always returns str,
> other methods can return unicode,  ljust/rjust refuse to accept a
> unicode character argument) is what Python 3 moved away from.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> hobsonlane%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170518/84e529a6/attachment.html>

From status at bugs.python.org  Fri May 19 12:09:12 2017
From: status at bugs.python.org (Python tracker)
Date: Fri, 19 May 2017 18:09:12 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20170519160912.27B1511A86B@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2017-05-12 - 2017-05-19)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5968 (+10)
  closed 36191 (+42)
  total  42159 (+52)

Open issues with patches: 2380 


Issues opened (42)
==================

#29619: st_ino (unsigned long long) is casted to long long in posixmod
http://bugs.python.org/issue29619  reopened by haypo

#30283: [2.7] Backport test_regrtest (partially) on Python 2.7
http://bugs.python.org/issue30283  reopened by haypo

#30353: ctypes: pass by value for structs broken on Cygwin/MinGW 64-bi
http://bugs.python.org/issue30353  opened by erik.bray

#30354: Change data model documentation to zero-argument super()
http://bugs.python.org/issue30354  opened by csabella

#30355: Unicode symbols crash lib2to3.parse
http://bugs.python.org/issue30355  opened by Yann Grisel

#30356: test_mymanager_context() of test_multiprocessing_spawn: manage
http://bugs.python.org/issue30356  opened by haypo

#30359: A standard convention for annotating a function as returning a
http://bugs.python.org/issue30359  opened by njs

#30360: getpass.getpass() does not accept stdin from an Expect script
http://bugs.python.org/issue30360  opened by brent saner

#30361: Docs example: converting mixed types to floating point
http://bugs.python.org/issue30361  opened by MihaiHimself

#30362: Launcher add list and list with paths options
http://bugs.python.org/issue30362  opened by Steve Barnes

#30364: Outdated function attribute to disable address sanitizer
http://bugs.python.org/issue30364  opened by xiang.zhang

#30367: Cannot build CPython3.6 with module ???testcapimodule??? stati
http://bugs.python.org/issue30367  opened by Nan Zhang

#30368: [2.7] OpenSSL compilation fails on AMD64 Windows7 SP1 VS9.0 2.
http://bugs.python.org/issue30368  opened by haypo

#30369: test_thread crashed with SIGBUS on AMD64 FreeBSD 10.x Shared 3
http://bugs.python.org/issue30369  opened by haypo

#30371: test_long_lines() fails randomly on AMD64 Windows7 SP1 3.x
http://bugs.python.org/issue30371  opened by haypo

#30372: Status of __builtins__ is not totally clear
http://bugs.python.org/issue30372  opened by Malcolm Smith

#30374: Make win_add2path.py take effect without having to log off
http://bugs.python.org/issue30374  opened by neeverett

#30376: Curses documentation refers to incorrect type
http://bugs.python.org/issue30376  opened by Ryan Jarvis

#30377: Unnecessary complexity in tokenize.py around handling of comme
http://bugs.python.org/issue30377  opened by Albert-Jan Nijburg

#30378: SysLogHandler does not support IPv6 destinations
http://bugs.python.org/issue30378  opened by calcheng

#30379: multiprocessing Array create for ctypes.c_char, TypeError unle
http://bugs.python.org/issue30379  opened by jgschaefer

#30380: Sphinx 1.6.1 raising new warnings in docs build
http://bugs.python.org/issue30380  opened by brett.cannon

#30381: test_smtpnet.test_connect_using_sslcontext_verified() randomly
http://bugs.python.org/issue30381  opened by haypo

#30382: test_stdin_broken_pipe() of test_asyncio failed randomly on AM
http://bugs.python.org/issue30382  opened by haypo

#30383: [3.5] Backport regrtest features from master to Python 3.5
http://bugs.python.org/issue30383  opened by haypo

#30384: traceback.TracebackException.format shouldn't format_exc_only(
http://bugs.python.org/issue30384  opened by Aaron.Meurer

#30386: Add a build infrastructure for Android
http://bugs.python.org/issue30386  opened by xdegaye

#30388: ndbm can't iterate through values on OS X
http://bugs.python.org/issue30388  opened by Forest Gregg

#30389: distutils._msvccompiler cannot find VS 2017
http://bugs.python.org/issue30389  opened by steve.dower

#30391: test_socketserver killed after 15 min on AMD64 FreeBSD 10.x Sh
http://bugs.python.org/issue30391  opened by haypo

#30392: default webbrowser fails on macOS Sierra 10.12.5
http://bugs.python.org/issue30392  opened by Andrew.Jaffe

#30393: test_readline hangs
http://bugs.python.org/issue30393  opened by wmayner

#30394: smtplib leaves open sockets around if SMTPConnectError is rais
http://bugs.python.org/issue30394  opened by jhillacre

#30395: deadlocked child process after forking on pystate.c's head_mut
http://bugs.python.org/issue30395  opened by Louis Brandy

#30396: Document the PyClassMethod* C API functions.
http://bugs.python.org/issue30396  opened by Decorater

#30397: Expose regular expression and match objects types in the re mo
http://bugs.python.org/issue30397  opened by serhiy.storchaka

#30398: Add a docstring for re.error
http://bugs.python.org/issue30398  opened by serhiy.storchaka

#30399: Get rid of trailing comma if the repr() of BaseException
http://bugs.python.org/issue30399  opened by serhiy.storchaka

#30400: Race condition in shutil.copyfile()
http://bugs.python.org/issue30400  opened by Preston Moore

#30401: Remove the .bzrignore file
http://bugs.python.org/issue30401  opened by matrixise

#30403: Running extension modules using -m switch
http://bugs.python.org/issue30403  opened by Dormouse759

#30404: Make stdout and stderr truly unbuffered when run with the -u o
http://bugs.python.org/issue30404  opened by serhiy.storchaka



Most recent 15 issues with no replies (15)
==========================================

#30404: Make stdout and stderr truly unbuffered when run with the -u o
http://bugs.python.org/issue30404

#30403: Running extension modules using -m switch
http://bugs.python.org/issue30403

#30401: Remove the .bzrignore file
http://bugs.python.org/issue30401

#30399: Get rid of trailing comma if the repr() of BaseException
http://bugs.python.org/issue30399

#30398: Add a docstring for re.error
http://bugs.python.org/issue30398

#30396: Document the PyClassMethod* C API functions.
http://bugs.python.org/issue30396

#30395: deadlocked child process after forking on pystate.c's head_mut
http://bugs.python.org/issue30395

#30394: smtplib leaves open sockets around if SMTPConnectError is rais
http://bugs.python.org/issue30394

#30393: test_readline hangs
http://bugs.python.org/issue30393

#30391: test_socketserver killed after 15 min on AMD64 FreeBSD 10.x Sh
http://bugs.python.org/issue30391

#30389: distutils._msvccompiler cannot find VS 2017
http://bugs.python.org/issue30389

#30384: traceback.TracebackException.format shouldn't format_exc_only(
http://bugs.python.org/issue30384

#30383: [3.5] Backport regrtest features from master to Python 3.5
http://bugs.python.org/issue30383

#30382: test_stdin_broken_pipe() of test_asyncio failed randomly on AM
http://bugs.python.org/issue30382

#30381: test_smtpnet.test_connect_using_sslcontext_verified() randomly
http://bugs.python.org/issue30381



Most recent 15 issues waiting for review (15)
=============================================

#30404: Make stdout and stderr truly unbuffered when run with the -u o
http://bugs.python.org/issue30404

#30399: Get rid of trailing comma if the repr() of BaseException
http://bugs.python.org/issue30399

#30398: Add a docstring for re.error
http://bugs.python.org/issue30398

#30397: Expose regular expression and match objects types in the re mo
http://bugs.python.org/issue30397

#30394: smtplib leaves open sockets around if SMTPConnectError is rais
http://bugs.python.org/issue30394

#30389: distutils._msvccompiler cannot find VS 2017
http://bugs.python.org/issue30389

#30386: Add a build infrastructure for Android
http://bugs.python.org/issue30386

#30377: Unnecessary complexity in tokenize.py around handling of comme
http://bugs.python.org/issue30377

#30349: Preparation for advanced set syntax in regular expressions
http://bugs.python.org/issue30349

#30347: itertools.groupby() can fail a C assert()
http://bugs.python.org/issue30347

#30346: Odd behavior when unpacking `itertools.groupby`
http://bugs.python.org/issue30346

#30310: tkFont.py assumes that all font families are encoded as ascii 
http://bugs.python.org/issue30310

#30306: release arguments of contextmanager
http://bugs.python.org/issue30306

#30262: Don't expose sqlite3 Cache and Statement
http://bugs.python.org/issue30262

#30248: Using boolean arguments in the _json module
http://bugs.python.org/issue30248



Top 10 most discussed issues (10)
=================================

#30380: Sphinx 1.6.1 raising new warnings in docs build
http://bugs.python.org/issue30380  12 msgs

#30350: devguide suggests to use VS 2008 to build Python 2.7, but VS 2
http://bugs.python.org/issue30350  10 msgs

#29137: Fix fpectl-induced ABI breakage
http://bugs.python.org/issue29137   8 msgs

#29619: st_ino (unsigned long long) is casted to long long in posixmod
http://bugs.python.org/issue29619   7 msgs

#30211: Bdb: add docstrings
http://bugs.python.org/issue30211   7 msgs

#30371: test_long_lines() fails randomly on AMD64 Windows7 SP1 3.x
http://bugs.python.org/issue30371   7 msgs

#25324: Importing tokenize modifies token
http://bugs.python.org/issue25324   6 msgs

#30346: Odd behavior when unpacking `itertools.groupby`
http://bugs.python.org/issue30346   6 msgs

#30377: Unnecessary complexity in tokenize.py around handling of comme
http://bugs.python.org/issue30377   6 msgs

#18233: SSLSocket.getpeercertchain()
http://bugs.python.org/issue18233   5 msgs



Issues closed (39)
==================

#9850: obsolete macpath module dangerously broken and should be remov
http://bugs.python.org/issue9850  closed by haypo

#11681: -b option undocumented
http://bugs.python.org/issue11681  closed by zach.ware

#16283: ctypes.util.find_library does not find all DLLs anymore
http://bugs.python.org/issue16283  closed by eryksun

#28974: Make default __format__ be equivalent to __str__
http://bugs.python.org/issue28974  closed by serhiy.storchaka

#29063: Fixed timemodule compile warnings (gmtoff).
http://bugs.python.org/issue29063  closed by haypo

#29196: Remove old-deprecated plistlib features
http://bugs.python.org/issue29196  closed by serhiy.storchaka

#29252: self in classes missinterpreted as a string.
http://bugs.python.org/issue29252  closed by Decorater

#29570: Windows Buildbot 2.7 is broken
http://bugs.python.org/issue29570  closed by haypo

#29611: TextIOWrapper's write_through option behave differently betwee
http://bugs.python.org/issue29611  closed by serhiy.storchaka

#29651: Inconsistent/undocumented urlsplit/urlparse behavior on invali
http://bugs.python.org/issue29651  closed by orsenthil

#29863: Add a COMPACT constant to the json module
http://bugs.python.org/issue29863  closed by brett.cannon

#29898: PYTHONLEGACYWINDOWSIOENCODING isn't implemented
http://bugs.python.org/issue29898  closed by berker.peksag

#29948: DeprecationWarning when parse ElementTree with a doctype in 2.
http://bugs.python.org/issue29948  closed by serhiy.storchaka

#29950: Rename SlotWrapperType to WrapperDescriptorType
http://bugs.python.org/issue29950  closed by Jim Fasarakis-Hilliard

#30110: test_asyncio reports reference leak
http://bugs.python.org/issue30110  closed by xiang.zhang

#30224: remove outdated checks in struct
http://bugs.python.org/issue30224  closed by xiang.zhang

#30242: resolve undefined behaviour in struct
http://bugs.python.org/issue30242  closed by xiang.zhang

#30291: Allow windows launcher to specify bit lengths with & without m
http://bugs.python.org/issue30291  closed by Steve Barnes

#30296: Remove unnecessary tuples, lists, sets, and dicts from Lib
http://bugs.python.org/issue30296  closed by rhettinger

#30299: Display the bytecode when compiled a regular expression in deb
http://bugs.python.org/issue30299  closed by serhiy.storchaka

#30301: multiprocessing: AttributeError: 'SimpleQueue' object has no a
http://bugs.python.org/issue30301  closed by xiang.zhang

#30323: concurrent.futures.Executor.map() consumes all memory when big
http://bugs.python.org/issue30323  closed by Klamann

#30325: Buildbot: send email notifications to buildbot-status@
http://bugs.python.org/issue30325  closed by haypo

#30340: Optimize out non-capturing groups
http://bugs.python.org/issue30340  closed by serhiy.storchaka

#30341: Add an explaining comment in _PyTrash_thread_destroy_chain()
http://bugs.python.org/issue30341  closed by xiang.zhang

#30344: test_multiprocessing.test_notify_all(): AssertionError: 6 != 5
http://bugs.python.org/issue30344  closed by haypo

#30352: The 'in' syntax should work with any object that implements __
http://bugs.python.org/issue30352  closed by rhettinger

#30357: test_thread should clean threads after each test
http://bugs.python.org/issue30357  closed by haypo

#30358: Profile documentation - include sort argument for runctx
http://bugs.python.org/issue30358  closed by berker.peksag

#30363: Backport warnings in the re module to 2.7
http://bugs.python.org/issue30363  closed by serhiy.storchaka

#30365: Backport warnings in ElementTree/cElementTree modules and fix 
http://bugs.python.org/issue30365  closed by serhiy.storchaka

#30366: Backport tests for test.support to 2.7
http://bugs.python.org/issue30366  closed by serhiy.storchaka

#30370: IPython 5.3.0 debug mode breakpoint issue
http://bugs.python.org/issue30370  closed by berker.peksag

#30373: Incomplete description of re.LOCALE
http://bugs.python.org/issue30373  closed by serhiy.storchaka

#30375: Correct stacklevel of warnings when compile regular expression
http://bugs.python.org/issue30375  closed by serhiy.storchaka

#30385: Segfault on OSX with 3.6.1
http://bugs.python.org/issue30385  closed by ned.deily

#30387: warning of threading_cleanup in test_threading
http://bugs.python.org/issue30387  closed by haypo

#30390: Document how to make classes in the C API.
http://bugs.python.org/issue30390  closed by serhiy.storchaka

#30402: Unexpected and/or inconsistent del behavior
http://bugs.python.org/issue30402  closed by rhettinger

From guido at python.org  Sun May 21 01:54:48 2017
From: guido at python.org (Guido van Rossum)
Date: Sat, 20 May 2017 22:54:48 -0700
Subject: [Python-Dev] PEP 545: Python Documentation Translations
In-Reply-To: <A1eY2R3txj6gJS2yeRD871VWg_LGOig9MxivaQFh2sM5nuSOc-brnRUzetRaPRowfMh6sTrjydPBm1EfE01eHmMUWVPAcytPw4t8Ys5Tptg=@palard.fr>
References: <serPZ8e15GRZpFGH3mQaywZvDz-tz4ZYcF2iNhDdrX1njbKzMLfduqvBvzPRTTABOyDCugVKznYNcp8Oay1uU8umLgJKamuCW3mlOAZxGQs=@palard.fr>
 <CAP1=2W5QpyMJAKxtEqCvMGJ8TC+GssAHx3LCX7JgtaE3SfDA8w@mail.gmail.com>
 <6-6zandAUD_HqCXBfrlg0-lw5H9zwipt2GRfqW2ZptmVLdHkeFVyc7lvfIawnnm0qo3pV-CKdvXTmWenV-uA3IvxjGFdHMMSzdPYCcautMU=@palard.fr>
 <CAP1=2W4vjKydG82ZcxpGmJsDJ5aw-hoDkEyW5eU6nCbLyUvGYA@mail.gmail.com>
 <A1eY2R3txj6gJS2yeRD871VWg_LGOig9MxivaQFh2sM5nuSOc-brnRUzetRaPRowfMh6sTrjydPBm1EfE01eHmMUWVPAcytPw4t8Ys5Tptg=@palard.fr>
Message-ID: <CAP7+vJ+6s0qcD4SuVKBzr1Lf9S1w=nOn+36ckp+=cJgAjBRv2A@mail.gmail.com>

Congratulations!

After a brief review period, and noticing there has not been more
discussion on this PEP, I have *approved* the latest version of PEP 545.

Hopefully one of the authors can update the PEP text in the peps repo with
this status change, linking to this message.

(Note that my approval does not preclude minor textual tweaks and
clarifications, or even updates to specific processes recommended by the
PEP. This PEP seems to be a work in progress in terms of some details, but
I approve of the overall plan, and it is now up to the PEP authors to
implement that plan.)

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170520/f3c034a1/attachment.html>

From victor.stinner at gmail.com  Sun May 21 04:01:22 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Sun, 21 May 2017 01:01:22 -0700
Subject: [Python-Dev] PEP 545: Python Documentation Translations
In-Reply-To: <CAP7+vJ+6s0qcD4SuVKBzr1Lf9S1w=nOn+36ckp+=cJgAjBRv2A@mail.gmail.com>
References: <serPZ8e15GRZpFGH3mQaywZvDz-tz4ZYcF2iNhDdrX1njbKzMLfduqvBvzPRTTABOyDCugVKznYNcp8Oay1uU8umLgJKamuCW3mlOAZxGQs=@palard.fr>
 <CAP1=2W5QpyMJAKxtEqCvMGJ8TC+GssAHx3LCX7JgtaE3SfDA8w@mail.gmail.com>
 <6-6zandAUD_HqCXBfrlg0-lw5H9zwipt2GRfqW2ZptmVLdHkeFVyc7lvfIawnnm0qo3pV-CKdvXTmWenV-uA3IvxjGFdHMMSzdPYCcautMU=@palard.fr>
 <CAP1=2W4vjKydG82ZcxpGmJsDJ5aw-hoDkEyW5eU6nCbLyUvGYA@mail.gmail.com>
 <A1eY2R3txj6gJS2yeRD871VWg_LGOig9MxivaQFh2sM5nuSOc-brnRUzetRaPRowfMh6sTrjydPBm1EfE01eHmMUWVPAcytPw4t8Ys5Tptg=@palard.fr>
 <CAP7+vJ+6s0qcD4SuVKBzr1Lf9S1w=nOn+36ckp+=cJgAjBRv2A@mail.gmail.com>
Message-ID: <CAMpsgwZZnA7tMonV+Aix4tPe5-sbp7y13Gk5F4YgO0703-UYOw@mail.gmail.com>

Champagne! Congrats *Julien*! Julien Palards is the one who really
drives the effort to get a translated doc at python.org. Thanks also
Naoki INADA for the help on that PEP. I didn't do much, except of
pushing to get a PEP and get everything written down to make things
going smoothly and making the discussion more productive.

Victor

2017-05-20 22:54 GMT-07:00 Guido van Rossum <guido at python.org>:
> Congratulations!
>
> After a brief review period, and noticing there has not been more discussion
> on this PEP, I have *approved* the latest version of PEP 545.
>
> Hopefully one of the authors can update the PEP text in the peps repo with
> this status change, linking to this message.
>
> (Note that my approval does not preclude minor textual tweaks and
> clarifications, or even updates to specific processes recommended by the
> PEP. This PEP seems to be a work in progress in terms of some details, but I
> approve of the overall plan, and it is now up to the PEP authors to
> implement that plan.)
>
> --
> --Guido van Rossum (python.org/~guido)

From steve.dower at python.org  Mon May 22 14:17:18 2017
From: steve.dower at python.org (Steve Dower)
Date: Mon, 22 May 2017 11:17:18 -0700
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
Message-ID: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>

One of the main reasons we are stuck with an old libffi fork in CPython 
is because the newer versions do not support protection from calling 
functions with too few/many arguments:

https://docs.python.org/3/library/ctypes.html?highlight=ctypes#calling-functions

There are a number of caveats here, including "this only works on 
Windows", but since it is documented we cannot just remove the behaviour 
without a deprecation period.

I'd like to propose a highly-accelerated deprecation period for this 
specific feature, starting in CPython 3.6.2 and being "completed" in 
3.7.0, when we will hopefully move onto a newer libffi.

In general, the "feature" is a misfeature anyway, since calling a native 
function with incorrect arguments is unsupported and a very easy way to 
cause information leakage or code execution vulnerabilities. There may 
be an argument for removing the functionality immediately, but honestly 
I think changing libffi in a point release is higher risk.

Once the special protection is removed, most of these cases will become 
OSError due to the general protection against segmentation faults. Some 
will undoubtedly fall through the cracks and crash the entire 
interpreter, but these are unavoidable (and really ought to crash to 
avoid potential exploits).

Does anyone have any reasons to oppose this? It already has votes from 
another Windows expert and the 3.6/3.7 Release Manager, but we wanted to 
see if anyone has a concern we haven't thought of.

Cheers,
Steve

From guido at python.org  Mon May 22 14:25:00 2017
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 May 2017 11:25:00 -0700
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
In-Reply-To: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
References: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
Message-ID: <CAP7+vJLNi0wNpAhLShDG=cdLyNwT+HbQcxMBEuDNTWg6jz=MZg@mail.gmail.com>

Sounds good to me.

On Mon, May 22, 2017 at 11:17 AM, Steve Dower <steve.dower at python.org>
wrote:

> One of the main reasons we are stuck with an old libffi fork in CPython is
> because the newer versions do not support protection from calling functions
> with too few/many arguments:
>
> https://docs.python.org/3/library/ctypes.html?highlight=ctyp
> es#calling-functions
>
> There are a number of caveats here, including "this only works on
> Windows", but since it is documented we cannot just remove the behaviour
> without a deprecation period.
>
> I'd like to propose a highly-accelerated deprecation period for this
> specific feature, starting in CPython 3.6.2 and being "completed" in 3.7.0,
> when we will hopefully move onto a newer libffi.
>
> In general, the "feature" is a misfeature anyway, since calling a native
> function with incorrect arguments is unsupported and a very easy way to
> cause information leakage or code execution vulnerabilities. There may be
> an argument for removing the functionality immediately, but honestly I
> think changing libffi in a point release is higher risk.
>
> Once the special protection is removed, most of these cases will become
> OSError due to the general protection against segmentation faults. Some
> will undoubtedly fall through the cracks and crash the entire interpreter,
> but these are unavoidable (and really ought to crash to avoid potential
> exploits).
>
> Does anyone have any reasons to oppose this? It already has votes from
> another Windows expert and the 3.6/3.7 Release Manager, but we wanted to
> see if anyone has a concern we haven't thought of.
>
> Cheers,
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170522/57d970e5/attachment.html>

From p.f.moore at gmail.com  Mon May 22 15:56:11 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Mon, 22 May 2017 20:56:11 +0100
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
In-Reply-To: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
References: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
Message-ID: <CACac1F9v_tfJ6SWNtETHySQvSukmsX7Za74iCY+YmwLzEaaWRw@mail.gmail.com>

On 22 May 2017 at 19:17, Steve Dower <steve.dower at python.org> wrote:
> One of the main reasons we are stuck with an old libffi fork in CPython is
> because the newer versions do not support protection from calling functions
> with too few/many arguments:
>
> https://docs.python.org/3/library/ctypes.html?highlight=ctypes#calling-functions
>
> There are a number of caveats here, including "this only works on Windows",
> but since it is documented we cannot just remove the behaviour without a
> deprecation period.
>
> I'd like to propose a highly-accelerated deprecation period for this
> specific feature, starting in CPython 3.6.2 and being "completed" in 3.7.0,
> when we will hopefully move onto a newer libffi.
>
> In general, the "feature" is a misfeature anyway, since calling a native
> function with incorrect arguments is unsupported and a very easy way to
> cause information leakage or code execution vulnerabilities. There may be an
> argument for removing the functionality immediately, but honestly I think
> changing libffi in a point release is higher risk.
>
> Once the special protection is removed, most of these cases will become
> OSError due to the general protection against segmentation faults. Some will
> undoubtedly fall through the cracks and crash the entire interpreter, but
> these are unavoidable (and really ought to crash to avoid potential
> exploits).
>
> Does anyone have any reasons to oppose this? It already has votes from
> another Windows expert and the 3.6/3.7 Release Manager, but we wanted to see
> if anyone has a concern we haven't thought of.

+1 from me.

Paul

From guido at python.org  Mon May 22 17:48:26 2017
From: guido at python.org (Guido van Rossum)
Date: Mon, 22 May 2017 14:48:26 -0700
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAP7+vJL7gy6i8Z-wShD-r18H3q1fpWR7XD1Zgd7H4FY06MBOCg@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
 <CAK256p02rjxa3=tNU=iiRM10O2OgqTMZy8Cs3zsGY3gO3cC18g@mail.gmail.com>
 <CAMpsgwbA2ViSiDKnYxEg-1RXTmWN_VnXesiSfYd0jgrNaFZUSg@mail.gmail.com>
 <CAP1=2W77F+kHuyTKPc5Dp1adMLPTNWBKwjw6RP6GwpkC4r+TtA@mail.gmail.com>
 <CAP7+vJJrPoPMfMbmPvhCESDvmZb1rKHXW2OHN6zsWG-wF6MkAA@mail.gmail.com>
 <CAP7+vJL7gy6i8Z-wShD-r18H3q1fpWR7XD1Zgd7H4FY06MBOCg@mail.gmail.com>
Message-ID: <CAP7+vJLhjdRbVM3dt-GyTkaz0cxo8KgbUkBx=GFJjQB5ufTo9A@mail.gmail.com>

This story has a happy ending. I sent a request to GitHub user support, who
forwarded a message from me to the owner of the repo, who responded with
apologies and then deleted the repo! Click for yourself:
https://github.com/python-git/python/ is now a 404. Yay!

On Fri, May 5, 2017 at 3:23 PM, Guido van Rossum <guido at python.org> wrote:

> OK I'll contact GitHub.
>
> On Fri, May 5, 2017 at 10:01 AM, Guido van Rossum <guido at python.org>
> wrote:
>
>> Let's coordinate who contacts GitHub. Victor, Brett or myself?
>>
>> On Fri, May 5, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote:
>>
>>>
>>>
>>> On Fri, 5 May 2017 at 09:50 Victor Stinner <victor.stinner at gmail.com>
>>> wrote:
>>>
>>>> 2017-05-05 18:36 GMT+02:00 Jonathan Goble <jcgoble3 at gmail.com>:
>>>> > It appears to me to be an individual user rather than an organization.
>>>>
>>>> Oh nice, glad to meet you :-) So what do you think? Are you ok to
>>>> remove this old clone? Or do you have reasons to keep it?
>>>>
>>>
>>> I don't think Jonathan was claiming ownership of the python-git account,
>>> just pointed out the account is a personal account of somebody's and not an
>>> organization account.
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: https://mail.python.org/mailma
>>> n/options/python-dev/guido%40python.org
>>>
>>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170522/0ce0cbcf/attachment.html>

From victor.stinner at gmail.com  Mon May 22 18:18:44 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 22 May 2017 17:18:44 -0500
Subject: [Python-Dev] Outdated GitHub clone of the old svn repository
In-Reply-To: <CAP7+vJLhjdRbVM3dt-GyTkaz0cxo8KgbUkBx=GFJjQB5ufTo9A@mail.gmail.com>
References: <CAMpsgwYJV4EvL=0uAWsxay36mCrkw9Snrj3zxrxQw+N67=-NSQ@mail.gmail.com>
 <CAK256p02rjxa3=tNU=iiRM10O2OgqTMZy8Cs3zsGY3gO3cC18g@mail.gmail.com>
 <CAMpsgwbA2ViSiDKnYxEg-1RXTmWN_VnXesiSfYd0jgrNaFZUSg@mail.gmail.com>
 <CAP1=2W77F+kHuyTKPc5Dp1adMLPTNWBKwjw6RP6GwpkC4r+TtA@mail.gmail.com>
 <CAP7+vJJrPoPMfMbmPvhCESDvmZb1rKHXW2OHN6zsWG-wF6MkAA@mail.gmail.com>
 <CAP7+vJL7gy6i8Z-wShD-r18H3q1fpWR7XD1Zgd7H4FY06MBOCg@mail.gmail.com>
 <CAP7+vJLhjdRbVM3dt-GyTkaz0cxo8KgbUkBx=GFJjQB5ufTo9A@mail.gmail.com>
Message-ID: <CAMpsgwbaovExhfTMcxDkJBDQv8-+jczF+=KDMRhKd4bUBC1BXw@mail.gmail.com>

Thank you Guido for taking take of that. So I fon't have to ask the same
question over again (since I always forget what I did yesterday :-)).

Happy hacking at the CPython sprint now on GitHub!!!

Victor

Le 22 mai 2017 2:48 PM, "Guido van Rossum" <guido at python.org> a ?crit :

> This story has a happy ending. I sent a request to GitHub user support,
> who forwarded a message from me to the owner of the repo, who responded
> with apologies and then deleted the repo! Click for yourself:
> https://github.com/python-git/python/ is now a 404. Yay!
>
> On Fri, May 5, 2017 at 3:23 PM, Guido van Rossum <guido at python.org> wrote:
>
>> OK I'll contact GitHub.
>>
>> On Fri, May 5, 2017 at 10:01 AM, Guido van Rossum <guido at python.org>
>> wrote:
>>
>>> Let's coordinate who contacts GitHub. Victor, Brett or myself?
>>>
>>> On Fri, May 5, 2017 at 9:52 AM, Brett Cannon <brett at python.org> wrote:
>>>
>>>>
>>>>
>>>> On Fri, 5 May 2017 at 09:50 Victor Stinner <victor.stinner at gmail.com>
>>>> wrote:
>>>>
>>>>> 2017-05-05 18:36 GMT+02:00 Jonathan Goble <jcgoble3 at gmail.com>:
>>>>> > It appears to me to be an individual user rather than an
>>>>> organization.
>>>>>
>>>>> Oh nice, glad to meet you :-) So what do you think? Are you ok to
>>>>> remove this old clone? Or do you have reasons to keep it?
>>>>>
>>>>
>>>> I don't think Jonathan was claiming ownership of the python-git
>>>> account, just pointed out the account is a personal account of somebody's
>>>> and not an organization account.
>>>>
>>>> _______________________________________________
>>>> Python-Dev mailing list
>>>> Python-Dev at python.org
>>>> https://mail.python.org/mailman/listinfo/python-dev
>>>> Unsubscribe: https://mail.python.org/mailma
>>>> n/options/python-dev/guido%40python.org
>>>>
>>>>
>>>
>>>
>>> --
>>> --Guido van Rossum (python.org/~guido)
>>>
>>
>>
>>
>> --
>> --Guido van Rossum (python.org/~guido)
>>
>
>
>
> --
> --Guido van Rossum (python.org/~guido)
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170522/0bfe6692/attachment.html>

From brett at python.org  Mon May 22 19:34:29 2017
From: brett at python.org (Brett Cannon)
Date: Mon, 22 May 2017 23:34:29 +0000
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
Message-ID: <CAP1=2W4UqEG1uod=U5505jhXP_COKv29WE6RT9+Zbp2e+oDyUw@mail.gmail.com>

On Tue, 16 May 2017 at 08:08 Martin Wimpress <martin.wimpress at canonical.com>
wrote:

> Hi all,
>
> I work at Canonical as part of the engineering team developing Ubuntu
> and Snapcraft [1] and I'm a long time Python fan :-)
>
> We've created snaps, a platform that enables projects to directly
> control delivery of software updates to users. This video of a
> lightning talk by dlang developers at DConf2017 [2] shows how they've
> made good use of snaps to distribute their compiler. They found the
> release channels particularly useful so their users can track a
> specific release.
>
> Is there someone here who'd be interested in doing the same for Python?
>
>
So the problem with adding Snap is it's yet one more thing for us to create
at release time. And if we do this for Snap are there competitors from e.g.
Fedora that people would want supported?

-Brett


> [1] https://snapcraft.io/
> [2] https://www.youtube.com/watch?v=M-bDzr4gYUU
> [3] https://snapcraft.io/docs/core/install
> [4] https://build.snapcraft.io/
>
> --
> Regards, Martin.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170522/7104cc18/attachment.html>

From vnjunu at usiu.ac.ke  Mon May 22 15:59:35 2017
From: vnjunu at usiu.ac.ke (Victor Njunu)
Date: Mon, 22 May 2017 19:59:35 +0000
Subject: [Python-Dev] Python is Bae
Message-ID: <AM4PR07MB31866CDAF3FFF017DD5AD670F0F80@AM4PR07MB3186.eurprd07.prod.outlook.com>

Dear Python,

My name is Victor Mshindi and I am a student at United States International University - Africa(USIU - Africa) studying Information Systems Technology and also a member of the growing tech community. I've been working with python for 2 years now, currently using it for machine learning and web development(Django) and would like to first say thank you for this amazing language and the community you've built.

At USIU we realized there is an increasing number of tech students who aren't self learning and depending on classes only, we also noticed that the community was becoming exclusive for top developers and not passing knowledge to new students. As a result we decide to start a python class to help beginners and then a django class to help them transition and become really good developers. We also hope to grow the python sub community and one day hold our very own Nairobi Pycon giving python developers a chance to learn and interact with members form around Kenya , outside Kenya and hopefully some core python members.

In order to improve the quality of the class we would need to provide attendees branded merchandise, snacks & lunch and hopefully a Skype session with one of the core members. This will help excite them and welcome them in the best way possible to the python community. Therefore I would like to request for support from the foundation by providing a Skype session, Branded merchandise and if possible financial support to buy lunch & snacks.

Regards,
Victor Mshindi
United States International University - Africa
Student and IT Club President,
Django Full stack Developer,
+254-712-619-612
Email Hygiene scanner powered by Forefront Online Protection for Exchange.

From matt at vazor.com  Mon May 22 19:56:43 2017
From: matt at vazor.com (Matt Billenstein)
Date: Mon, 22 May 2017 23:56:43 +0000
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
Message-ID: <0101015c32976a82-02f59fbb-9c76-4c8d-a338-4a2ea289f202-000000@us-west-2.amazonses.com>

On Tue, May 16, 2017 at 11:31:42AM +0100, Martin Wimpress wrote:
> Is there someone here who'd be interested in doing the same for Python?

Do snaps support Windows and/or MacOS?

m

-- 
Matt Billenstein
matt at vazor.com
http://www.vazor.com/

From encukou at gmail.com  Tue May 23 04:27:02 2017
From: encukou at gmail.com (Petr Viktorin)
Date: Tue, 23 May 2017 10:27:02 +0200
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <CAP1=2W4UqEG1uod=U5505jhXP_COKv29WE6RT9+Zbp2e+oDyUw@mail.gmail.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
 <CAP1=2W4UqEG1uod=U5505jhXP_COKv29WE6RT9+Zbp2e+oDyUw@mail.gmail.com>
Message-ID: <8981c0dd-fd59-adfe-be79-ff9d93f59cfd@gmail.com>

On 05/23/2017 01:34 AM, Brett Cannon wrote:
> 
> On Tue, 16 May 2017 at 08:08 Martin Wimpress 
> <martin.wimpress at canonical.com <mailto:martin.wimpress at canonical.com>> 
> wrote:
> 
>     Hi all,
> 
>     I work at Canonical as part of the engineering team developing Ubuntu
>     and Snapcraft [1] and I'm a long time Python fan :-)
> 
>     We've created snaps, a platform that enables projects to directly
>     control delivery of software updates to users. This video of a
>     lightning talk by dlang developers at DConf2017 [2] shows how they've
>     made good use of snaps to distribute their compiler. They found the
>     release channels particularly useful so their users can track a
>     specific release.
> 
>     Is there someone here who'd be interested in doing the same for Python?
> 
> 
> So the problem with adding Snap is it's yet one more thing for us to 
> create at release time. And if we do this for Snap are there competitors 
> from e.g. Fedora that people would want supported?

Not Fedora per se, Flatpak [0] (formerly `xdg-app`) is distro-agnostic [1].


[0] http://flatpak.org/
[1] http://flatpak.org/faq.html

From songofacandy at gmail.com  Tue May 23 04:38:42 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Tue, 23 May 2017 17:38:42 +0900
Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C
 locale to a UTF-8 based locale
In-Reply-To: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>
References: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>
Message-ID: <CAEfz+TzVQq+rtD8+S8UbUVxt_ijbBne6xqgFP3C8aGhoZipB1Q@mail.gmail.com>

Hi, Nick.

I read again and I think PEP 538 is mostly ready for accepted,
without waiting PEP 540.

One remaining my concern is setting LANG.

> Setting LANG to C.UTF-8 ensures that even components that only check the LANG fallback for their locale settings will still use C.UTF-8 .
https://www.python.org/dev/peps/pep-0538/#setting-both-lc-ctype-lang-for-utf-8-locale-coercion

I feel setting only LC_CTYPE making PEP 538 simpler.
Is there any real component using LANG for deciding encoding?

For example, date command refers LC_TIME.

$ LANG=ja_JP.UTF-8 LC_CTYPE=C date
2017?  5? 23? ??? 17:31:03 JST

$ LANG=ja_JP.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing only LC_CTYPE
2017?  5? 23? ??? 17:32:58 JST

$ LANG=C.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing both of LC_CTYPE and LANG
Tue May 23 17:31:10 JST 2017

In this case, coercing only LC_CTYPE has less side-effect.

Would you add example demonstrates how coercing LANG helps people?


On Tue, May 9, 2017 at 8:57 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> Hi folks,
>
> Enough changes have accumulated in PEP 538 since the start of the
> previous thread that it seems sensible to me to start a new thread
> specifically covering the current design (which aims to address all
> the concerns raised in the previous thread).
>
> I haven't requoted the PEP in full since it's so long, but will
> instead refer readers to the web version:
> https://www.python.org/dev/peps/pep-0538/
>
> I also generated a diff covered the full changes to the PEP text:
>
> * https://gist.github.com/ncoghlan/1067805fe673b3735ac854e195747493/revisions
> (this is the diff covering the last few days of changes
>
> Summarising the key technical changes:
>
> * to make the runtime behaviour independent of whether or not locale
> coercion took place, stdin and stderr now always have
> "surrogateescape" as their error handler in the potential coercion
> target locales. This means Python will behave the same way regardless
> of whether the locale gets set externally (e.g. by a parent Python
> process or a container image definition) or implicitly during CLI
> startup
> * for the full locales, the interpreter now sets LC_CTYPE and LANG,
> *not* LC_ALL. This means LC_ALL is once again a full locale override,
> and also means that CPython won't inadvertently interfere with other
> locale categories like LC_MONETARY, LC_NUMERIC, etc
> * the reference implementation has been refactored so the bulk of the
> new code lives in the shared library and is exposed to the linker via
> a couple of underscore prefixed API symbols
> (_Py_LegacyLocaleDetected() and _Py_CoerceLegacyLocale()). While the
> current PEP still keeps them private, it would be straightforward to
> make them public for use in embedding applications if we decided we
> wanted to do so.
> * locale coercion and warnings are now enabled by default on all
> platforms that use the autotools-based build chain - the assumption
> that some platforms didn't need them turned out to be incorrect
>
> In addition to being updated to cover the above changes, the Rationale
> section of the PEP has also been updated to explain why it doesn't
> propose setting PYTHONIOENCODING, and to walk through some examples of
> the problems with GNU readlines compatibility when the current locale
> isn't set correctly.
>
> The essential related changes to the reference implementation can be seen here:
>
> * Always set "surrogateescape" for coercion target locales,
> independently of whether or not coercion occurred:
> https://github.com/ncoghlan/cpython/commit/188e7807b6d9e49377aacbb287c074e5cabf70c5
> * Stop setting LC_ALL:
> https://github.com/python/peps/commit/2f530ce0d1fd24835ac0c6f984f40db70482a18f
>
> (There are also some smaller cleanup commits that can be seen by
> browsing that branch on GitHub)
>
> Cheers,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com

From freddyrietdijk at fridh.nl  Tue May 23 06:46:18 2017
From: freddyrietdijk at fridh.nl (Freddy Rietdijk)
Date: Tue, 23 May 2017 12:46:18 +0200
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
Message-ID: <CAOQtOH3teATSF=jYg+=d9f2PbWV4q+Jh9XSpFjANQOuK7H+Zwg@mail.gmail.com>

In my opinion the trend of creating these standalone packages is pretty
bad. Developers that choose to support these kind of packages become
responsible not only for the code they distribute, but also for whether the
dependencies that are bundled in such a standalone package are up to date
and secure.

What libraries would have to be bundled? I guess that depends on how
standalone you want to get. In Nixpkgs we get to 130 MB for Python 3.5, if
it would bundle all its dependencies (glibc, openssl, ncurses, some X11
libs, ...). If I am correct you can assume certain libraries are available
on the host system so you could skip bundling them.

Furthermore, there's now at least three competing formats: Snappy, Flatpak
and AppImage.


On Tue, May 16, 2017 at 12:31 PM, Martin Wimpress <
martin.wimpress at canonical.com> wrote:

> Hi all,
>
> I work at Canonical as part of the engineering team developing Ubuntu
> and Snapcraft [1] and I'm a long time Python fan :-)
>
> We've created snaps, a platform that enables projects to directly
> control delivery of software updates to users. This video of a
> lightning talk by dlang developers at DConf2017 [2] shows how they've
> made good use of snaps to distribute their compiler. They found the
> release channels particularly useful so their users can track a
> specific release.
>
> Is there someone here who'd be interested in doing the same for Python?
>
> [1] https://snapcraft.io/
> [2] https://www.youtube.com/watch?v=M-bDzr4gYUU
> [3] https://snapcraft.io/docs/core/install
> [4] https://build.snapcraft.io/
>
> --
> Regards, Martin.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> freddyrietdijk%40fridh.nl
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/854a58a6/attachment.html>

From martin.wimpress at canonical.com  Tue May 23 11:00:19 2017
From: martin.wimpress at canonical.com (Martin Wimpress)
Date: Tue, 23 May 2017 16:00:19 +0100
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <8981c0dd-fd59-adfe-be79-ff9d93f59cfd@gmail.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
 <CAP1=2W4UqEG1uod=U5505jhXP_COKv29WE6RT9+Zbp2e+oDyUw@mail.gmail.com>
 <8981c0dd-fd59-adfe-be79-ff9d93f59cfd@gmail.com>
Message-ID: <4fcf175c-1bb1-9e41-448a-7a01c8aec7b8@canonical.com>

On 23/05/17 09:27, Petr Viktorin wrote:
> On 05/23/2017 01:34 AM, Brett Cannon wrote:
>>
>> On Tue, 16 May 2017 at 08:08 Martin Wimpress 
>> <martin.wimpress at canonical.com <mailto:martin.wimpress at canonical.com>> 
>> wrote:
>>
>>     Hi all,
>>
>>     I work at Canonical as part of the engineering team developing Ubuntu
>>     and Snapcraft [1] and I'm a long time Python fan :-)
>>
>>     We've created snaps, a platform that enables projects to directly
>>     control delivery of software updates to users. This video of a
>>     lightning talk by dlang developers at DConf2017 [2] shows how they've
>>     made good use of snaps to distribute their compiler. They found the
>>     release channels particularly useful so their users can track a
>>     specific release.
>>
>>     Is there someone here who'd be interested in doing the same for 
>> Python?
>>
>>
>> So the problem with adding Snap is it's yet one more thing for us to 
>> create at release time. And if we do this for Snap are there 
>> competitors from e.g. Fedora that people would want supported?
> 
> Not Fedora per se, Flatpak [0] (formerly `xdg-app`) is distro-agnostic [1].

Snaps are supported on Fedora 24, 25 and 26 [1] as well as other distros 
[2].

> 
> 
> [0] http://flatpak.org/
> [1] http://flatpak.org/faq.html
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: 
> https://mail.python.org/mailman/options/python-dev/martin.wimpress%40canonical.com 
> 

[1] 
https://insights.ubuntu.com/2017/04/11/snap-support-lands-in-fedora-24-25-26/
[2] https://snapcraft.io/docs/core/install

--
Regards, Martin.

From martin.wimpress at canonical.com  Tue May 23 11:02:47 2017
From: martin.wimpress at canonical.com (Martin Wimpress)
Date: Tue, 23 May 2017 16:02:47 +0100
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <0101015c32976a82-02f59fbb-9c76-4c8d-a338-4a2ea289f202-000000@us-west-2.amazonses.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
 <0101015c32976a82-02f59fbb-9c76-4c8d-a338-4a2ea289f202-000000@us-west-2.amazonses.com>
Message-ID: <65856131-5d27-c344-c0f3-6e7e99082454@canonical.com>

Hi,

On 23/05/17 00:56, Matt Billenstein wrote:
> On Tue, May 16, 2017 at 11:31:42AM +0100, Martin Wimpress wrote:
>> Is there someone here who'd be interested in doing the same for Python?
> 
> Do snaps support Windows and/or MacOS?

You can't install snaps on Windows or macOS. Snaps are currently 
targeting various Linux distros, you can create Linux snaps on Windows 
using WSL and we're working on improving the tooling for snap creation 
on macOS and Windows.

> m
> 

-- 
--
Regards, Martin.

From solipsis at pitrou.net  Tue May 23 11:27:02 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 23 May 2017 17:27:02 +0200
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
 <0101015c32976a82-02f59fbb-9c76-4c8d-a338-4a2ea289f202-000000@us-west-2.amazonses.com>
Message-ID: <20170523172702.4aebf72e@fsol>

On Mon, 22 May 2017 23:56:43 +0000
Matt Billenstein <matt at vazor.com> wrote:
> On Tue, May 16, 2017 at 11:31:42AM +0100, Martin Wimpress wrote:
> > Is there someone here who'd be interested in doing the same for Python?  
> 
> Do snaps support Windows and/or MacOS?

If you want a package manager that supports installing binaries on
Linux, Windows and MacOS, I suggest you try out conda:
https://conda.io/docs/
Building conda packages uses a tool named conda-build (both are open
source):
https://conda.io/docs/building/recipe.html

Conda powers the Anaconda software distribution, published by Continuum
Analytics.  Many conda-build recipes are available publicly:
https://github.com/ContinuumIO/anaconda-recipes
For example here is the conda-build recipe for Python 3.6.1:
https://github.com/ContinuumIO/anaconda-recipes/blob/master/python-3.6/meta.yaml
There are also a growing number of community-maintained
recipes and packages:
https://conda-forge.github.io/

Disclaimer: I work for Continuum (though not on Conda nor
Anaconda).

Regards

Antoine.



From solipsis at pitrou.net  Tue May 23 11:28:31 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 23 May 2017 17:28:31 +0200
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
References: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
Message-ID: <20170523172831.309a781e@fsol>

On Mon, 22 May 2017 11:17:18 -0700
Steve Dower <steve.dower at python.org> wrote:
> 
> I'd like to propose a highly-accelerated deprecation period for this 
> specific feature, starting in CPython 3.6.2 and being "completed" in 
> 3.7.0, when we will hopefully move onto a newer libffi.
> 
> In general, the "feature" is a misfeature anyway, since calling a native 
> function with incorrect arguments is unsupported and a very easy way to 
> cause information leakage or code execution vulnerabilities.

Agreed.

> Does anyone have any reasons to oppose this? It already has votes from 
> another Windows expert and the 3.6/3.7 Release Manager, but we wanted to 
> see if anyone has a concern we haven't thought of.

+1 from me.

Regards

Antoine.



From mariatta.wijaya at gmail.com  Tue May 23 11:44:15 2017
From: mariatta.wijaya at gmail.com (Mariatta Wijaya)
Date: Tue, 23 May 2017 08:44:15 -0700
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
In-Reply-To: <20170523172831.309a781e@fsol>
References: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
 <20170523172831.309a781e@fsol>
Message-ID: <CAGbohnaP2x7MpWzHGNDSbJCychGxzghvWwaOzTnjhPWacH7XmQ@mail.gmail.com>

+1

My understanding is this is a documentation change, marking it as
deprecated in 3.6.2 and a Misc News entry.
No actual code change.
Correct?



Mariatta Wijaya

On Tue, May 23, 2017 at 8:28 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:

> On Mon, 22 May 2017 11:17:18 -0700
> Steve Dower <steve.dower at python.org> wrote:
> >
> > I'd like to propose a highly-accelerated deprecation period for this
> > specific feature, starting in CPython 3.6.2 and being "completed" in
> > 3.7.0, when we will hopefully move onto a newer libffi.
> >
> > In general, the "feature" is a misfeature anyway, since calling a native
> > function with incorrect arguments is unsupported and a very easy way to
> > cause information leakage or code execution vulnerabilities.
>
> Agreed.
>
> > Does anyone have any reasons to oppose this? It already has votes from
> > another Windows expert and the 3.6/3.7 Release Manager, but we wanted to
> > see if anyone has a concern we haven't thought of.
>
> +1 from me.
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> mariatta.wijaya%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/294c10e4/attachment-0001.html>

From steve.dower at python.org  Tue May 23 12:20:48 2017
From: steve.dower at python.org (Steve Dower)
Date: Tue, 23 May 2017 09:20:48 -0700
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
In-Reply-To: <CAGbohnaP2x7MpWzHGNDSbJCychGxzghvWwaOzTnjhPWacH7XmQ@mail.gmail.com>
References: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
 <20170523172831.309a781e@fsol>
 <CAGbohnaP2x7MpWzHGNDSbJCychGxzghvWwaOzTnjhPWacH7XmQ@mail.gmail.com>
Message-ID: <0efccd8e-f08f-4676-8d1e-9000c8b9cbbd@python.org>

On 23May2017 0844, Mariatta Wijaya wrote:
> +1
>
> My understanding is this is a documentation change, marking it as
> deprecated in 3.6.2 and a Misc News entry.
> No actual code change.
> Correct?

That's correct. I'm thinking just a :note: box in that section, and then 
you can keep working on the 3.7 change. I'll let you file the bugs.

Cheers,
Steve


From ncoghlan at gmail.com  Tue May 23 12:34:50 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 May 2017 02:34:50 +1000
Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C
 locale to a UTF-8 based locale
In-Reply-To: <CAEfz+TzVQq+rtD8+S8UbUVxt_ijbBne6xqgFP3C8aGhoZipB1Q@mail.gmail.com>
References: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>
 <CAEfz+TzVQq+rtD8+S8UbUVxt_ijbBne6xqgFP3C8aGhoZipB1Q@mail.gmail.com>
Message-ID: <CADiSq7deo_U6LLb6Vw0z398wiiUN_HrCO=oVefEdcG-aivpkEA@mail.gmail.com>

On 23 May 2017 at 18:38, INADA Naoki <songofacandy at gmail.com> wrote:
> Hi, Nick.
>
> I read again and I think PEP 538 is mostly ready for accepted,
> without waiting PEP 540.

Great to hear!

> One remaining my concern is setting LANG.
>
>> Setting LANG to C.UTF-8 ensures that even components that only check the LANG fallback for their locale settings will still use C.UTF-8 .
> https://www.python.org/dev/peps/pep-0538/#setting-both-lc-ctype-lang-for-utf-8-locale-coercion
>
> I feel setting only LC_CTYPE making PEP 538 simpler.
> Is there any real component using LANG for deciding encoding?
>
> For example, date command refers LC_TIME.
>
> $ LANG=ja_JP.UTF-8 LC_CTYPE=C date
> 2017?  5? 23? ??? 17:31:03 JST
>
> $ LANG=ja_JP.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing only LC_CTYPE
> 2017?  5? 23? ??? 17:32:58 JST
>
> $ LANG=C.UTF-8 LC_CTYPE=C.UTF-8 date  # Coercing both of LC_CTYPE and LANG
> Tue May 23 17:31:10 JST 2017
>
> In this case, coercing only LC_CTYPE has less side-effect.
>
> Would you add example demonstrates how coercing LANG helps people?

I'm honestly not sure it does - I think it's an assumption I added to
the PEP early on, and never actually tested. Looking at it more
closely now, all of the interpreter level checks are specifically for
LC_CTYPE, and experimenting with "LANG=C LC_CTYPE=C.UTF-8" indicates
that coercing only LC_CTYPE is still enough to fix the GNU readline
encoding compatibility problem covered in
https://www.python.org/dev/peps/pep-0538/#considering-locale-coercion-independently-of-utf-8-mode

So I'll take another pass through the implementation this weekend, and
simplify it to only set LC_CTYPE regardless of whether it's using
C.UTF-8, C.utf8, or UTF-8 as the target locale. Assuming that doesn't
uncover any hidden problems with the idea, I'll then update the PEP to
match.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From gvanrossum at gmail.com  Tue May 23 12:54:11 2017
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Tue, 23 May 2017 09:54:11 -0700
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <65856131-5d27-c344-c0f3-6e7e99082454@canonical.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
 <0101015c32976a82-02f59fbb-9c76-4c8d-a338-4a2ea289f202-000000@us-west-2.amazonses.com>
 <65856131-5d27-c344-c0f3-6e7e99082454@canonical.com>
Message-ID: <CAP7+vJLE4T-cJnmjGTHOnVxL7L5VEtMPvpBP3Wb7xsL1vpnfAA@mail.gmail.com>

I think the I inevitable conclusion is"thanks, but no thanks."

On May 23, 2017 8:05 AM, "Martin Wimpress" <martin.wimpress at canonical.com>
wrote:

> Hi,
>
> On 23/05/17 00:56, Matt Billenstein wrote:
>
>> On Tue, May 16, 2017 at 11:31:42AM +0100, Martin Wimpress wrote:
>>
>>> Is there someone here who'd be interested in doing the same for Python?
>>>
>>
>> Do snaps support Windows and/or MacOS?
>>
>
> You can't install snaps on Windows or macOS. Snaps are currently targeting
> various Linux distros, you can create Linux snaps on Windows using WSL and
> we're working on improving the tooling for snap creation on macOS and
> Windows.
>
> m
>>
>>
> --
> --
> Regards, Martin.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/437582c8/attachment.html>

From steve at holdenweb.com  Tue May 23 12:25:33 2017
From: steve at holdenweb.com (Steve Holden)
Date: Tue, 23 May 2017 17:25:33 +0100
Subject: [Python-Dev] Python is Bae
In-Reply-To: <AM4PR07MB31866CDAF3FFF017DD5AD670F0F80@AM4PR07MB3186.eurprd07.prod.outlook.com>
References: <AM4PR07MB31866CDAF3FFF017DD5AD670F0F80@AM4PR07MB3186.eurprd07.prod.outlook.com>
Message-ID: <CAMofdRBcJpN_U4XtvS6y8SX5Y94LfHgC8On=UOD8YTB+pgtZdA@mail.gmail.com>

Hi Victor,

Since this request (or at least the bit about branded goods) isn't really
appropriate for python-dev I am taking the liberty of forwarding this email
to the PSF's administrator, who may be better placed to help. I have Bcc'd
python-dev on this reply so they aren't troubled by correspondence on that
topic.

I applaud your efforts to build a Python community from the ground up, and
wish your efforts every success. I imagine there are other developers who
feel similarly.

regards
 Steve

Steve Holden

On Mon, May 22, 2017 at 8:59 PM, Victor Njunu <vnjunu at usiu.ac.ke> wrote:

> Dear Python,
>
> My name is Victor Mshindi and I am a student at United States
> International University - Africa(USIU - Africa) studying Information
> Systems Technology and also a member of the growing tech community. I've
> been working with python for 2 years now, currently using it for machine
> learning and web development(Django) and would like to first say thank you
> for this amazing language and the community you've built.
>
> At USIU we realized there is an increasing number of tech students who
> aren't self learning and depending on classes only, we also noticed that
> the community was becoming exclusive for top developers and not passing
> knowledge to new students. As a result we decide to start a python class to
> help beginners and then a django class to help them transition and become
> really good developers. We also hope to grow the python sub community and
> one day hold our very own Nairobi Pycon giving python developers a chance
> to learn and interact with members form around Kenya , outside Kenya and
> hopefully some core python members.
>
> In order to improve the quality of the class we would need to provide
> attendees branded merchandise, snacks & lunch and hopefully a Skype session
> with one of the core members. This will help excite them and welcome them
> in the best way possible to the python community. Therefore I would like to
> request for support from the foundation by providing a Skype session,
> Branded merchandise and if possible financial support to buy lunch & snacks.
>
> Regards,
> Victor Mshindi
> United States International University - Africa
> Student and IT Club President,
> Django Full stack Developer,
> +254-712-619-612
> Email Hygiene scanner powered by Forefront Online Protection for Exchange.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> steve%40holdenweb.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/4ef84671/attachment.html>

From victor.stinner at gmail.com  Tue May 23 15:12:32 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 May 2017 14:12:32 -0500
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
In-Reply-To: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
References: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
Message-ID: <CAMpsgwb3Ly8Bq2b-XZkDf6x011TO5_WWQ8+3DnYCyrbVriudaQ@mail.gmail.com>

2017-05-22 13:17 GMT-05:00 Steve Dower <steve.dower at python.org>:
> Once the special protection is removed, most of these cases will become
> OSError due to the general protection against segmentation faults.

It didn't know that ctypes on Windows had a special protection against
programming errors. I'm not aware of such protection Linux. If you
call a function with the wrong number of arguments, it's likely to
crash or return random data.

I guess that the point is to help debugging. But since Python 3.6,
faulthandler now registers a Windows exception handler and so it able
to dump the Python traceback on any Windows exception:
https://docs.python.org/dev/library/faulthandler.html#faulthandler.enable

So I think that it's now fine to remove the ctypes protection. Just
advice (remind? ;-)) users to enable faulthandler: python3 -X
faulthandler, or call faulthandler.enable(). (You might want to use a
log file for that on Windows, depends on the use case.)

From njs at pobox.com  Tue May 23 16:41:58 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 23 May 2017 13:41:58 -0700
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
Message-ID: <CAPJVwBk8XX5XqdjmQKSr4xcVhk69mUVZGsycehkwq-8D9doYUQ@mail.gmail.com>

Well, it sounds like you have an answer to the question of whether the
cpython developers are interested in making official snap releases, but of
course this doesn't stop you making unofficial snap releases, and simple
standalone python distributions can be pretty handy. I just wanted to point
out that virtualenv support is a very important feature of python builds,
and it may be a challenge to support this within the app-as-container
paradigm, because it requires that the python interpreter be able to locate
itself, copy itself to arbitrary directories, and then run the copy. (Or
maybe it's fine, dunno. And python 3 has some changes that might make it
friendlier than python 2 in this regard.)

-n

On May 16, 2017 8:09 AM, "Martin Wimpress" <martin.wimpress at canonical.com>
wrote:

> Hi all,
>
> I work at Canonical as part of the engineering team developing Ubuntu
> and Snapcraft [1] and I'm a long time Python fan :-)
>
> We've created snaps, a platform that enables projects to directly
> control delivery of software updates to users. This video of a
> lightning talk by dlang developers at DConf2017 [2] shows how they've
> made good use of snaps to distribute their compiler. They found the
> release channels particularly useful so their users can track a
> specific release.
>
> Is there someone here who'd be interested in doing the same for Python?
>
> [1] https://snapcraft.io/
> [2] https://www.youtube.com/watch?v=M-bDzr4gYUU
> [3] https://snapcraft.io/docs/core/install
> [4] https://build.snapcraft.io/
>
> --
> Regards, Martin.
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> njs%40pobox.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/8ac5ba87/attachment.html>

From victor.stinner at gmail.com  Tue May 23 20:46:38 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 May 2017 19:46:38 -0500
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
Message-ID: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>

Hi,

Would you be ok to backport ssl.MemoryBIO and ssl.SSLObject on Python
2.7? I can do the backport.

  https://docs.python.org/dev/library/ssl.html#ssl.MemoryBIO

Cory Benfield told me that it's a blocking issue for him to implement
his PEP 543 -- A Unified TLS API for Python 2.7:

  https://www.python.org/dev/peps/pep-0543/

And I expect that if a new cool TLS API happens, people will want to
use it on Python 2.7-3.6, not only on Python 3.7. Security evolves
more quickly that the current Python release process, and people wants
to keep their application secure.

>From what I understood, he wants to first implement an abstract
MemoryBIO API (http://sans-io.readthedocs.io/ like API? I'm not sure
about that), and then implement a socket/FD based on top of that.
Maybe later, some implementations might have a fast-path using
socket/FD directly.

He described me his PEP and I strongly support it (sorry, I missed it
when he posted it on python-dev), but we decided (Guido van Rossum,
Christian Heimes, Cory Benfield and me, see the tweet below) to not
put this in the stdlib right now, but spend more time on testing it on
Twisted, asyncio, requests, etc. So publishing an implementation on
PyPI was proposed instead. It seems like we agreed on a smooth plan
(or am I wrong, Cory?).

  https://twitter.com/VictorStinner/status/865467388141027329

I'm quite sure that Twisted will love MemoryBIO on Python 2.7 as well,
to implement TLS, especially on Windows using IOCP. Currently,
external libraries (C extensions) are required.

I'm not sure if the PEP 466 should be amended for that? Is a new PEP
really needed? MemoryBIO/SSLObject are tiny. Nick (Coghlan): what do
you think?

  https://www.python.org/dev/peps/pep-0466/

Victor

From alex.gaynor at gmail.com  Tue May 23 20:49:10 2017
From: alex.gaynor at gmail.com (Alex Gaynor)
Date: Tue, 23 May 2017 20:49:10 -0400
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
Message-ID: <CAFRnB2WdUULa33n2AiC8+zP1r13egnAHgYZ_PRRHFDO2xotW-g@mail.gmail.com>

I'm +1 on this, I even wrote the patch: https://bugs.python.org/issue22559
:-) If you're interested in making sure that still applies and tests still
pass, I'd be a big fan.

In addition to all the benefits you mentioned, it also substantially
reduces the diff between 2.7 and 3.x (or at least it did when I originally
wrote it).

Cheers,
Alex

On Tue, May 23, 2017 at 8:46 PM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi,
>
> Would you be ok to backport ssl.MemoryBIO and ssl.SSLObject on Python
> 2.7? I can do the backport.
>
>   https://docs.python.org/dev/library/ssl.html#ssl.MemoryBIO
>
> Cory Benfield told me that it's a blocking issue for him to implement
> his PEP 543 -- A Unified TLS API for Python 2.7:
>
>   https://www.python.org/dev/peps/pep-0543/
>
> And I expect that if a new cool TLS API happens, people will want to
> use it on Python 2.7-3.6, not only on Python 3.7. Security evolves
> more quickly that the current Python release process, and people wants
> to keep their application secure.
>
> From what I understood, he wants to first implement an abstract
> MemoryBIO API (http://sans-io.readthedocs.io/ like API? I'm not sure
> about that), and then implement a socket/FD based on top of that.
> Maybe later, some implementations might have a fast-path using
> socket/FD directly.
>
> He described me his PEP and I strongly support it (sorry, I missed it
> when he posted it on python-dev), but we decided (Guido van Rossum,
> Christian Heimes, Cory Benfield and me, see the tweet below) to not
> put this in the stdlib right now, but spend more time on testing it on
> Twisted, asyncio, requests, etc. So publishing an implementation on
> PyPI was proposed instead. It seems like we agreed on a smooth plan
> (or am I wrong, Cory?).
>
>   https://twitter.com/VictorStinner/status/865467388141027329
>
> I'm quite sure that Twisted will love MemoryBIO on Python 2.7 as well,
> to implement TLS, especially on Windows using IOCP. Currently,
> external libraries (C extensions) are required.
>
> I'm not sure if the PEP 466 should be amended for that? Is a new PEP
> really needed? MemoryBIO/SSLObject are tiny. Nick (Coghlan): what do
> you think?
>
>   https://www.python.org/dev/peps/pep-0466/
>
> Victor
>



-- 
"I disapprove of what you say, but I will defend to the death your right to
say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
"The people's good is the highest law." -- Cicero
GPG Key fingerprint: D1B3 ADC0 E023 8CA6
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/cba0e025/attachment-0001.html>

From cory at lukasa.co.uk  Tue May 23 20:54:02 2017
From: cory at lukasa.co.uk (Cory Benfield)
Date: Tue, 23 May 2017 17:54:02 -0700
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
Message-ID: <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>


> On 23 May 2017, at 17:46, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> 
> He described me his PEP and I strongly support it (sorry, I missed it
> when he posted it on python-dev), but we decided (Guido van Rossum,
> Christian Heimes, Cory Benfield and me, see the tweet below) to not
> put this in the stdlib right now, but spend more time on testing it on
> Twisted, asyncio, requests, etc. So publishing an implementation on
> PyPI was proposed instead. It seems like we agreed on a smooth plan
> (or am I wrong, Cory?).

Yes, this is correct. There are plans afoot to look at moving Requests to a more event-loop-y model, and doing so basically mandates a MemoryBIO. In the absence of a Python 2.7 backport, Requests is required to basically use the same solution that Twisted currently does: namely, a mandatory dependency on PyOpenSSL.

I?m very much +1 on this.

Cory

From victor.stinner at gmail.com  Tue May 23 20:54:34 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 May 2017 19:54:34 -0500
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CAFRnB2WdUULa33n2AiC8+zP1r13egnAHgYZ_PRRHFDO2xotW-g@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <CAFRnB2WdUULa33n2AiC8+zP1r13egnAHgYZ_PRRHFDO2xotW-g@mail.gmail.com>
Message-ID: <CAMpsgwadgL9TrTaD-2Y=LmOaGum5X2vfwyNjMhPg7=Nh9nX2AQ@mail.gmail.com>

Well, things evolved recently. asyncio is getting more popular, TLS
became even more important, and pyOpenSSL is not more evolving (from
what I heard from his maintainer, Hynek). Does Twister still use
pyOpenSSL?

I like the idea of putting more security stuff into Python to ease
maintainance of applications and try to make it as security as
possible. Python has more resources (ex: CI) than some smaller
projects.

Victor

2017-05-23 19:49 GMT-05:00 Alex Gaynor <alex.gaynor at gmail.com>:
> I'm +1 on this, I even wrote the patch: https://bugs.python.org/issue22559
> :-) If you're interested in making sure that still applies and tests still
> pass, I'd be a big fan.
>
> In addition to all the benefits you mentioned, it also substantially reduces
> the diff between 2.7 and 3.x (or at least it did when I originally wrote
> it).
>
> Cheers,
> Alex
>
> On Tue, May 23, 2017 at 8:46 PM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>>
>> Hi,
>>
>> Would you be ok to backport ssl.MemoryBIO and ssl.SSLObject on Python
>> 2.7? I can do the backport.
>>
>>   https://docs.python.org/dev/library/ssl.html#ssl.MemoryBIO
>>
>> Cory Benfield told me that it's a blocking issue for him to implement
>> his PEP 543 -- A Unified TLS API for Python 2.7:
>>
>>   https://www.python.org/dev/peps/pep-0543/
>>
>> And I expect that if a new cool TLS API happens, people will want to
>> use it on Python 2.7-3.6, not only on Python 3.7. Security evolves
>> more quickly that the current Python release process, and people wants
>> to keep their application secure.
>>
>> From what I understood, he wants to first implement an abstract
>> MemoryBIO API (http://sans-io.readthedocs.io/ like API? I'm not sure
>> about that), and then implement a socket/FD based on top of that.
>> Maybe later, some implementations might have a fast-path using
>> socket/FD directly.
>>
>> He described me his PEP and I strongly support it (sorry, I missed it
>> when he posted it on python-dev), but we decided (Guido van Rossum,
>> Christian Heimes, Cory Benfield and me, see the tweet below) to not
>> put this in the stdlib right now, but spend more time on testing it on
>> Twisted, asyncio, requests, etc. So publishing an implementation on
>> PyPI was proposed instead. It seems like we agreed on a smooth plan
>> (or am I wrong, Cory?).
>>
>>   https://twitter.com/VictorStinner/status/865467388141027329
>>
>> I'm quite sure that Twisted will love MemoryBIO on Python 2.7 as well,
>> to implement TLS, especially on Windows using IOCP. Currently,
>> external libraries (C extensions) are required.
>>
>> I'm not sure if the PEP 466 should be amended for that? Is a new PEP
>> really needed? MemoryBIO/SSLObject are tiny. Nick (Coghlan): what do
>> you think?
>>
>>   https://www.python.org/dev/peps/pep-0466/
>>
>> Victor
>
>
>
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." -- Evelyn Beatrice Hall (summarizing Voltaire)
> "The people's good is the highest law." -- Cicero
> GPG Key fingerprint: D1B3 ADC0 E023 8CA6
>

From victor.stinner at gmail.com  Tue May 23 20:58:48 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 May 2017 19:58:48 -0500
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
Message-ID: <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>

2017-05-23 19:54 GMT-05:00 Cory Benfield <cory at lukasa.co.uk>:
> In the absence of a Python 2.7 backport, Requests is required to basically use the same solution that Twisted currently does: namely, a mandatory dependency on PyOpenSSL.

Last time I looked at requests, it embedded all its dependencies. I
don't like the idea of embedding PyOpenSSL...

Victor

From cory at lukasa.co.uk  Tue May 23 21:13:17 2017
From: cory at lukasa.co.uk (Cory Benfield)
Date: Tue, 23 May 2017 18:13:17 -0700
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
 <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
Message-ID: <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>


> On 23 May 2017, at 17:58, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> 2017-05-23 19:54 GMT-05:00 Cory Benfield <cory at lukasa.co.uk>:
>> In the absence of a Python 2.7 backport, Requests is required to basically use the same solution that Twisted currently does: namely, a mandatory dependency on PyOpenSSL.
> 
> Last time I looked at requests, it embedded all its dependencies. I
> don't like the idea of embedding PyOpenSSL?

There are discussions around Requests unvendoring its dependencies thanks to the improved nature of pip. This might be a year of pretty big changes for Requests.

Cory


From steve.dower at python.org  Tue May 23 21:19:44 2017
From: steve.dower at python.org (Steve Dower)
Date: Tue, 23 May 2017 18:19:44 -0700
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
In-Reply-To: <CAMpsgwb3Ly8Bq2b-XZkDf6x011TO5_WWQ8+3DnYCyrbVriudaQ@mail.gmail.com>
References: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
 <CAMpsgwb3Ly8Bq2b-XZkDf6x011TO5_WWQ8+3DnYCyrbVriudaQ@mail.gmail.com>
Message-ID: <93282162-34be-2eb1-39a6-0f2fa8810ea9@python.org>

On 23May2017 1212, Victor Stinner wrote:
> 2017-05-22 13:17 GMT-05:00 Steve Dower <steve.dower at python.org>:
>> Once the special protection is removed, most of these cases will become
>> OSError due to the general protection against segmentation faults.
>
> It didn't know that ctypes on Windows had a special protection against
> programming errors. I'm not aware of such protection Linux. If you
> call a function with the wrong number of arguments, it's likely to
> crash or return random data.
>
> I guess that the point is to help debugging. But since Python 3.6,
> faulthandler now registers a Windows exception handler and so it able
> to dump the Python traceback on any Windows exception:
> https://docs.python.org/dev/library/faulthandler.html#faulthandler.enable
>
> So I think that it's now fine to remove the ctypes protection. Just
> advice (remind? ;-)) users to enable faulthandler: python3 -X
> faulthandler, or call faulthandler.enable(). (You might want to use a
> log file for that on Windows, depends on the use case.)

faulthandler is already recommended in the docs, and the existing SEH 
protection for access violations will remain (since that is independent 
of libffi).

I'll be honest, I have appreciated the functionality in the past, but it 
really isn't good practice and getting rid of it will be an overall 
benefit. Technically even the segfault protection isn't a great idea, 
since you really do end up in an unknown state with regards to memory 
page allocations, but it's better than crashing all the way out.

Cheers,
Steve

From ncoghlan at gmail.com  Tue May 23 22:46:11 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 May 2017 12:46:11 +1000
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
Message-ID: <CADiSq7cvACBZhDt5j7dTueB3r3SOK3MO2y4tTy+uxyTBCHtUHQ@mail.gmail.com>

On 24 May 2017 at 10:46, Victor Stinner <victor.stinner at gmail.com> wrote:
> Hi,
>
> Would you be ok to backport ssl.MemoryBIO and ssl.SSLObject on Python
> 2.7? I can do the backport.
>
>   https://docs.python.org/dev/library/ssl.html#ssl.MemoryBIO

+1 from me - the last SSL module resync to 2.7 was from 3.4, and
bringing them back closer to feature parity again is genuinely
beneficial in ensuring the Python ecosystem is able to keep up with
evolving network security standards.

Guido requested back when PEP 466 was written that any further
security backports come with their own PEP in order to clearly
communicate what's being backported, and the "What's New in Python 2.7
Maintenance Releases?" section is organised accordingly.

However, the "Why?" section in such a PEP can be a lot shorter than it
was for the original precedent setting one, since it only needs to
describe the benefits of the specific security features being
backported, rather than having to make the case for the idea of
backporting security features in general.

Cheers,
Nick.

P.S. Somewhat related, folks may be interested to know that the
upcoming RHEL 7.4 release finally completes the process of backporting
PEPs 466 & 476 to the RHEL system Python by switching the default
behaviour for new installs to be to verify SSL/TLS certificates
against the system trust store:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7-Beta/html/7.4_Release_Notes/new_features_compiler_and_tools.html

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From dw+python-dev at hmmz.org  Tue May 23 23:34:03 2017
From: dw+python-dev at hmmz.org (David Wilson)
Date: Wed, 24 May 2017 03:34:03 +0000
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
 <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
 <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>
Message-ID: <20170524033403.GB16849@k3>

On Tue, May 23, 2017 at 06:13:17PM -0700, Cory Benfield wrote:

> There are discussions around Requests unvendoring its dependencies
> thanks to the improved nature of pip. This might be a year of pretty
> big changes for Requests.

In which case, what is to prevent Requests from just depending on
pyOpenSSL as usual?

I'm still writing 2.7 code every day and would love to see it live a
little longer, but accepting every feature request seems the wrong way
to go - and MemoryBIO is a hard sell as a security enhancement, it's new
functionality.


David

From victor.stinner at gmail.com  Tue May 23 23:58:49 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 May 2017 22:58:49 -0500
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CADiSq7cvACBZhDt5j7dTueB3r3SOK3MO2y4tTy+uxyTBCHtUHQ@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <CADiSq7cvACBZhDt5j7dTueB3r3SOK3MO2y4tTy+uxyTBCHtUHQ@mail.gmail.com>
Message-ID: <CAMpsgwaSwk5ty=Z2+W=eF9sq88uU1u5oZ-U+y8M=ZYYV7QM-sg@mail.gmail.com>

Le 23 mai 2017 19:46, "Nick Coghlan" <ncoghlan at gmail.com> a ?crit :
(...) to the RHEL system Python by switching the default

behaviour for new installs to be to verify SSL/TLS certificates
against the system trust store


Awesome! Security, I see security everywhere!

Stay safe,
Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/6898c4fa/attachment.html>

From victor.stinner at gmail.com  Tue May 23 23:59:43 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 May 2017 22:59:43 -0500
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CAFRnB2WdUULa33n2AiC8+zP1r13egnAHgYZ_PRRHFDO2xotW-g@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <CAFRnB2WdUULa33n2AiC8+zP1r13egnAHgYZ_PRRHFDO2xotW-g@mail.gmail.com>
Message-ID: <CAMpsgwZ=B=OO4tbNbw8hgyWvrkC-6T9Zg4FLWfS+QBHvb3Oudw@mail.gmail.com>

Le 23 mai 2017 17:49, "Alex Gaynor" <alex.gaynor at gmail.com> a ?crit :

I'm +1 on this, I even wrote the patch: https://bugs.python.org/issue22559
:-) If you're interested in making sure that still applies and tests still
pass, I'd be a big fan.


Oh sorry, I wasn't aware of that one. Sure, will do.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/5d7453ae/attachment.html>

From cory at lukasa.co.uk  Wed May 24 00:08:47 2017
From: cory at lukasa.co.uk (Cory Benfield)
Date: Tue, 23 May 2017 21:08:47 -0700
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <20170524033403.GB16849@k3>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
 <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
 <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>
 <20170524033403.GB16849@k3>
Message-ID: <CD5B24F1-C78C-4D3B-A097-77C8B80FA3AA@lukasa.co.uk>


> On 23 May 2017, at 20:34, David Wilson <dw+python-dev at hmmz.org> wrote:
> 
> On Tue, May 23, 2017 at 06:13:17PM -0700, Cory Benfield wrote:
> 
>> There are discussions around Requests unvendoring its dependencies
>> thanks to the improved nature of pip. This might be a year of pretty
>> big changes for Requests.
> 
> In which case, what is to prevent Requests from just depending on
> pyOpenSSL as usual?

Requests currently does not depend on PyOpenSSL on 2.7: it?s simply an optional dependency for those who need it.

Cory

From victor.stinner at gmail.com  Wed May 24 00:09:31 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 May 2017 23:09:31 -0500
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <20170524033403.GB16849@k3>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
 <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
 <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>
 <20170524033403.GB16849@k3>
Message-ID: <CAMpsgwbj+yeptKx58FTFhEXkoSyMFiFveGus=cGrb10eZT108g@mail.gmail.com>

Le 23 mai 2017 20:43, "David Wilson" <dw+python-dev at hmmz.org> a ?crit :
In which case, what is to prevent Requests from just depending on

pyOpenSSL as usual?


>From what I heard, pyOpenSSL development is slowing down, so I'm not sure
that it's really safe and future-proof (TLS 1.3 anyone?).

I'm still writing 2.7 code every day and would love to see it live a
little longer, but accepting every feature request seems the wrong way
to go - and MemoryBIO is a hard sell as a security enhancement, it's new
functionality.


You are true that they are new features. I disagree on the "accepting every
feature" part: we are talking about two classes and it's restricted to
security. Security matters for me and for practical reasons explained in
thid thread, we need the two classes.

Cory's PEP adds long awaited features (bugfixes?) to TLS, like getting
access to root certificates on macOS and Windows. Not having to provide our
own set of root certificates would make applications hlobally more secure.
It's ttricky to update certificates. It happens that root CA are revoked
after aa break-in or because the CA is no more trusted.

I also understood that getting access to system CA allows admins to
register their company CA and so avoid that users ignore the TLS warning
(unknown CA).

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/c5d47efb/attachment.html>

From victor.stinner at gmail.com  Wed May 24 00:11:53 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 23 May 2017 23:11:53 -0500
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
In-Reply-To: <93282162-34be-2eb1-39a6-0f2fa8810ea9@python.org>
References: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
 <CAMpsgwb3Ly8Bq2b-XZkDf6x011TO5_WWQ8+3DnYCyrbVriudaQ@mail.gmail.com>
 <93282162-34be-2eb1-39a6-0f2fa8810ea9@python.org>
Message-ID: <CAMpsgwbV2vh_8BK91aHM-SBLDrdHXK7tv7L5uBkO5QBMP7CstQ@mail.gmail.com>

Sure, make your change and then update libffi!

Victor

Le 23 mai 2017 18:19, "Steve Dower" <steve.dower at python.org> a ?crit :

> On 23May2017 1212, Victor Stinner wrote:
>
>> 2017-05-22 13:17 GMT-05:00 Steve Dower <steve.dower at python.org>:
>>
>>> Once the special protection is removed, most of these cases will become
>>> OSError due to the general protection against segmentation faults.
>>>
>>
>> It didn't know that ctypes on Windows had a special protection against
>> programming errors. I'm not aware of such protection Linux. If you
>> call a function with the wrong number of arguments, it's likely to
>> crash or return random data.
>>
>> I guess that the point is to help debugging. But since Python 3.6,
>> faulthandler now registers a Windows exception handler and so it able
>> to dump the Python traceback on any Windows exception:
>> https://docs.python.org/dev/library/faulthandler.html#faulthandler.enable
>>
>> So I think that it's now fine to remove the ctypes protection. Just
>> advice (remind? ;-)) users to enable faulthandler: python3 -X
>> faulthandler, or call faulthandler.enable(). (You might want to use a
>> log file for that on Windows, depends on the use case.)
>>
>
> faulthandler is already recommended in the docs, and the existing SEH
> protection for access violations will remain (since that is independent of
> libffi).
>
> I'll be honest, I have appreciated the functionality in the past, but it
> really isn't good practice and getting rid of it will be an overall
> benefit. Technically even the segfault protection isn't a great idea, since
> you really do end up in an unknown state with regards to memory page
> allocations, but it's better than crashing all the way out.
>
> Cheers,
> Steve
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170523/5d61620e/attachment.html>

From njs at pobox.com  Wed May 24 00:27:33 2017
From: njs at pobox.com (Nathaniel Smith)
Date: Tue, 23 May 2017 21:27:33 -0700
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <20170524033403.GB16849@k3>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
 <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
 <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>
 <20170524033403.GB16849@k3>
Message-ID: <CAPJVwBnUpJeZi3tF=dk4+-MzGQfuVgy95Xs3BsXZ=GGuv80KMw@mail.gmail.com>

On Tue, May 23, 2017 at 8:34 PM, David Wilson <dw+python-dev at hmmz.org> wrote:
> On Tue, May 23, 2017 at 06:13:17PM -0700, Cory Benfield wrote:
>
>> There are discussions around Requests unvendoring its dependencies
>> thanks to the improved nature of pip. This might be a year of pretty
>> big changes for Requests.
>
> In which case, what is to prevent Requests from just depending on
> pyOpenSSL as usual?

I thought that requests couldn't have any hard dependencies on C
extensions, because pip vendors requests, and pip needs to be
pure-python for bootstrapping purposes? Cory would know better than me
though, so perhaps I'm wrong...

> I'm still writing 2.7 code every day and would love to see it live a
> little longer, but accepting every feature request seems the wrong way
> to go - and MemoryBIO is a hard sell as a security enhancement, it's new
> functionality.

IIUC, the security enhancement is indirect but real: on Windows/MacOS,
Python's dependence on openssl is a security liability, and to get
away from this we need Cory's new library that abstracts over
different TLS implementations. But for applications to take advantage
of this, they need to switch to using the new library. And before they
can switch to using the new library, it needs to work everywhere. And
for the new library to work on python 2 on unix, it needs MemoryBIO's
in the stdlib ? ideally using an interface that's as-close-as-possible
to what they look like on python 3, so he doesn't have to implement
totally different backends for py2 and py3, because Cory is already a
hero for trying to make this happen and we don't want to waste any
more of his time than necessary. So the end result is that if you have
Python 2 code doing SSL/TLS on Windows/MacOS, and you want proper
trust handling and prompt security updates, then MemoryBIO support is
actually on the critical path for making that happen.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org

From ncoghlan at gmail.com  Wed May 24 02:26:52 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Wed, 24 May 2017 16:26:52 +1000
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CAPJVwBnUpJeZi3tF=dk4+-MzGQfuVgy95Xs3BsXZ=GGuv80KMw@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
 <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
 <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>
 <20170524033403.GB16849@k3>
 <CAPJVwBnUpJeZi3tF=dk4+-MzGQfuVgy95Xs3BsXZ=GGuv80KMw@mail.gmail.com>
Message-ID: <CADiSq7c1KkRjm0NG7ffyrtkK8vXyw+Gc=R5qPiRmTSiAgh1-Zw@mail.gmail.com>

On 24 May 2017 at 14:27, Nathaniel Smith <njs at pobox.com> wrote:
> On Tue, May 23, 2017 at 8:34 PM, David Wilson <dw+python-dev at hmmz.org> wrote:
>> On Tue, May 23, 2017 at 06:13:17PM -0700, Cory Benfield wrote:
>>
>>> There are discussions around Requests unvendoring its dependencies
>>> thanks to the improved nature of pip. This might be a year of pretty
>>> big changes for Requests.
>>
>> In which case, what is to prevent Requests from just depending on
>> pyOpenSSL as usual?
>
> I thought that requests couldn't have any hard dependencies on C
> extensions, because pip vendors requests, and pip needs to be
> pure-python for bootstrapping purposes? Cory would know better than me
> though, so perhaps I'm wrong...

You're not wrong - even if requests itself stops bundling its
dependencies in its releases, pip will still need to bundle requests
and all its dependencies for bootstrapping purposes, which rules out a
hard dependency on PyOpenSSL (since even with manylinux1 available,
the wheel format doesn't offer 100% coverage of all environments where
pip will need to be bootstrapped).

>> I'm still writing 2.7 code every day and would love to see it live a
>> little longer, but accepting every feature request seems the wrong way
>> to go - and MemoryBIO is a hard sell as a security enhancement, it's new
>> functionality.
>
> IIUC, the security enhancement is indirect but real: on Windows/MacOS,
> Python's dependence on openssl is a security liability, and to get
> away from this we need Cory's new library that abstracts over
> different TLS implementations. But for applications to take advantage
> of this, they need to switch to using the new library. And before they
> can switch to using the new library, it needs to work everywhere. And
> for the new library to work on python 2 on unix, it needs MemoryBIO's
> in the stdlib ? ideally using an interface that's as-close-as-possible
> to what they look like on python 3, so he doesn't have to implement
> totally different backends for py2 and py3, because Cory is already a
> hero for trying to make this happen and we don't want to waste any
> more of his time than necessary. So the end result is that if you have
> Python 2 code doing SSL/TLS on Windows/MacOS, and you want proper
> trust handling and prompt security updates, then MemoryBIO support is
> actually on the critical path for making that happen.

I believe you just wrote the "Rationale" section of the backport PEP :)

The other key aspect potentially worth mentioning is that upstream
approving the backport for a future 2.x maintenance release means that
downstream redistributors are also free to add it to their(/our) long
term support releases without introducing vendor specific divergences
(i.e. the feature being missing becomes just a typical "your Python is
too old, you'll have to install PyOpenSSL" problem, rather than a
"your Python is from the wrong redistributor" problem - while the
latter can still happen sometimes for various reasons, it's something
we collectively try to avoid).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From benhoyt at gmail.com  Wed May 24 14:07:30 2017
From: benhoyt at gmail.com (Ben Hoyt)
Date: Wed, 24 May 2017 14:07:30 -0400
Subject: [Python-Dev] Micro-optimizations by adding special-case bytecodes?
Message-ID: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>

Hi folks,

I was looking at some `dis` output today, and I was wondering if anyone has
investigated optimizing Python (slightly) by adding special-case bytecodes
for common expressions or statements involving constants?

For example, I (and, based on a quick grep of the stdlib, many others)
write "x is None" and "x is not None" very often. Or "return True" or
"return None" or "return 1" and things like that. These all expand into two
bytecodes, which seems pretty non-optimal (LOAD_CONST + COMPARE_OP or
LOAD_CONST + RETURN_VALUE). It seems we could get an easy speedup for these
common cases by adding a peephole optimization and some new opcodes (maybe
COMPARE_IS_SMALL_CONST and RETURN_SMALL_CONST for these cases).

I'm not proposing to do this yet, as I'd need to benchmark to see how much
of a gain (if any) it would amount to, but I'm just wondering if there's
any previous work on this kind of thing. Or, if not, any other thoughts
before I try it?

Thanks,
Ben
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170524/0f74a701/attachment.html>

From stephane at wirtel.be  Wed May 24 14:12:15 2017
From: stephane at wirtel.be (Stephane Wirtel)
Date: Wed, 24 May 2017 20:12:15 +0200
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
Message-ID: <94012d1a-0931-ab04-eec7-31e0eefecd79@wirtel.be>

there is the Peephole for that, it's a small optimizer in the compiler.

On 05/24/2017 08:07 PM, Ben Hoyt wrote:
> Hi folks,
> 
> I was looking at some `dis` output today, and I was wondering if anyone has
> investigated optimizing Python (slightly) by adding special-case bytecodes
> for common expressions or statements involving constants?
> 
> For example, I (and, based on a quick grep of the stdlib, many others)
> write "x is None" and "x is not None" very often. Or "return True" or
> "return None" or "return 1" and things like that. These all expand into two
> bytecodes, which seems pretty non-optimal (LOAD_CONST + COMPARE_OP or
> LOAD_CONST + RETURN_VALUE). It seems we could get an easy speedup for these
> common cases by adding a peephole optimization and some new opcodes (maybe
> COMPARE_IS_SMALL_CONST and RETURN_SMALL_CONST for these cases).
> 
> I'm not proposing to do this yet, as I'd need to benchmark to see how much
> of a gain (if any) it would amount to, but I'm just wondering if there's
> any previous work on this kind of thing. Or, if not, any other thoughts
> before I try it?
> 
> Thanks,
> Ben
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/stephane%40wirtel.be
> 

From python-dev at masklinn.net  Wed May 24 14:26:24 2017
From: python-dev at masklinn.net (Xavier Morel)
Date: Wed, 24 May 2017 20:26:24 +0200
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
Message-ID: <D62E97F0-C379-4A96-B4D7-D1124534C0F4@masklinn.net>


> On 2017-05-24, at 20:07 , Ben Hoyt <benhoyt at gmail.com> wrote:
> 
> Hi folks,
> 
> I was looking at some `dis` output today, and I was wondering if anyone has investigated optimizing Python (slightly) by adding special-case bytecodes for common expressions or statements involving constants?

Python 3.6 added an opcode specifically for dicts with constant keys:
https://bugs.python.org/issue27140. Though I guess that's a slightly
different case as it's not a peephole-fused opcode.

From python-dev at masklinn.net  Wed May 24 14:29:11 2017
From: python-dev at masklinn.net (Xavier Morel)
Date: Wed, 24 May 2017 20:29:11 +0200
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <D62E97F0-C379-4A96-B4D7-D1124534C0F4@masklinn.net>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
 <D62E97F0-C379-4A96-B4D7-D1124534C0F4@masklinn.net>
Message-ID: <1D450FC4-8E9C-4243-B97E-FAFD050206FB@masklinn.net>


> On 2017-05-24, at 20:26 , Xavier Morel <python-dev at masklinn.net> wrote:
> 
>> On 2017-05-24, at 20:07 , Ben Hoyt <benhoyt at gmail.com> wrote:
>> 
>> Hi folks,
>> 
>> I was looking at some `dis` output today, and I was wondering if anyone has investigated optimizing Python (slightly) by adding special-case bytecodes for common expressions or statements involving constants?
> 
> Python 3.6 added an opcode specifically for dicts with constant keys:
> https://bugs.python.org/issue27140. Though I guess that's a slightly
> different case as it's not a peephole-fused opcode.

And followup, Python 2.7 did add fused opcodes related to conditional: 
http://bugs.python.org/issue4715.

From python at lucidity.plus.com  Wed May 24 16:14:18 2017
From: python at lucidity.plus.com (Erik)
Date: Wed, 24 May 2017 21:14:18 +0100
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
Message-ID: <dda9cdaf-0bf4-ab0d-d884-a5dc447bca2b@lucidity.plus.com>

Hi Ben,

On 24/05/17 19:07, Ben Hoyt wrote:
> I'm not proposing to do this yet, as I'd need to benchmark to see how 
> much of a gain (if any) it would amount to, but I'm just wondering if 
> there's any previous work on this kind of thing. Or, if not, any other 
> thoughts before I try it?

This is exactly what I looked into just over a year ago. As Stephane 
suggests, I did this by adding new opcodes that the peephole optimizer 
generated and the interpreter loop understood (but the compiler itself 
did not need to know anything about these new opcodes, so that makes 
things much easier).



Adding new opcodes like this at the time wasn't straightforward because 
of issues with the build process (see this thread: 
https://mail.python.org/pipermail/python-dev/2015-December/142600.html - 
it starts out as a question about the bytecode format but ended up with 
some very useful information on the build process).

Note that since that thread, a couple of things have changed - the 
bytecode is now wordcode so some of my original questions aren't 
relevant, and some of the things I had a problem with in the build 
system are now auto-generated with a new 'make' target. So it _should_ 
be easier now than it was then.



In terms of the results I got once I had things building and running, I 
didn't manage to find any particular magic bullets that gave me a 
significant enough speedup. Perhaps I just didn't pick the right opcode 
sequences or the right test cases (though what I was trying to do was 
quite successful in terms of doing things like replacing 
branches-to-RETURN into a single RETURN - so LOAD_CONST/RETURN_VALUE 
became RETURN_CONST and therefore if the target of an unconditional 
branch was to a RETURN_CONST op, the branch op could be replaced by the 
RETURN_CONST).

I figured that one thing every function or method needs to do is return, 
so I tried to make that more efficient. I only had two weeks to spend on 
it though ...

I was trying to do that by avoiding trips-around-the-interpreter-loop as 
that was historically something that would give speedups. However, with 
the new computed-goto version of the interpreter I came to the 
conclusion that it's not at important as it used to be. I was building 
with gcc though and what I *didn't* do was disable the computed-goto 
code (it's controlled by a #define) to see if my changes improved 
performance on platforms that can't use it.


I have other opcode sequences that I identified that might be useful to 
look at further.

I didn't (and still don't) have enough bandwidth to *drive* something 
like this through though, but if you want to do that I'd be more than 
happy to be kept in the loop on what you're doing and can possibly find 
time to write some code too.

Regards, E.

From benhoyt at gmail.com  Wed May 24 16:56:01 2017
From: benhoyt at gmail.com (Ben Hoyt)
Date: Wed, 24 May 2017 16:56:01 -0400
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <dda9cdaf-0bf4-ab0d-d884-a5dc447bca2b@lucidity.plus.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
 <dda9cdaf-0bf4-ab0d-d884-a5dc447bca2b@lucidity.plus.com>
Message-ID: <CAL9jXCFfxF99zo-pCCS7JMr-73CaaU60Xw-PbscMsTnDvULKoA@mail.gmail.com>

Interesting -- thanks very much, Erik. Yeah, I wonder if the wordcode will
change things too. But interesting to know you got not much of a speedup!
Also, the "x is None" and "x is not None" were the opcodes I started out
thinking would be helpful, so I'm interested to try those.

I'm wondering if the simplest way to test would be to add the new opcodes
and handle them in ceval.c, but then do all the bytecode manipulation in a
pure Python peephole optimizer ("bytecode munger"). Then if it worked we
could put those optimizations in the compiler or peephole.c.

-Ben

On Wed, May 24, 2017 at 4:14 PM, Erik <python at lucidity.plus.com> wrote:

> Hi Ben,
>
> On 24/05/17 19:07, Ben Hoyt wrote:
>
>> I'm not proposing to do this yet, as I'd need to benchmark to see how
>> much of a gain (if any) it would amount to, but I'm just wondering if
>> there's any previous work on this kind of thing. Or, if not, any other
>> thoughts before I try it?
>>
>
> This is exactly what I looked into just over a year ago. As Stephane
> suggests, I did this by adding new opcodes that the peephole optimizer
> generated and the interpreter loop understood (but the compiler itself did
> not need to know anything about these new opcodes, so that makes things
> much easier).
>
>
>
> Adding new opcodes like this at the time wasn't straightforward because of
> issues with the build process (see this thread:
> https://mail.python.org/pipermail/python-dev/2015-December/142600.html -
> it starts out as a question about the bytecode format but ended up with
> some very useful information on the build process).
>
> Note that since that thread, a couple of things have changed - the
> bytecode is now wordcode so some of my original questions aren't relevant,
> and some of the things I had a problem with in the build system are now
> auto-generated with a new 'make' target. So it _should_ be easier now than
> it was then.
>
>
>
> In terms of the results I got once I had things building and running, I
> didn't manage to find any particular magic bullets that gave me a
> significant enough speedup. Perhaps I just didn't pick the right opcode
> sequences or the right test cases (though what I was trying to do was quite
> successful in terms of doing things like replacing branches-to-RETURN into
> a single RETURN - so LOAD_CONST/RETURN_VALUE became RETURN_CONST and
> therefore if the target of an unconditional branch was to a RETURN_CONST
> op, the branch op could be replaced by the RETURN_CONST).
>
> I figured that one thing every function or method needs to do is return,
> so I tried to make that more efficient. I only had two weeks to spend on it
> though ...
>
> I was trying to do that by avoiding trips-around-the-interpreter-loop as
> that was historically something that would give speedups. However, with the
> new computed-goto version of the interpreter I came to the conclusion that
> it's not at important as it used to be. I was building with gcc though and
> what I *didn't* do was disable the computed-goto code (it's controlled by a
> #define) to see if my changes improved performance on platforms that can't
> use it.
>
>
> I have other opcode sequences that I identified that might be useful to
> look at further.
>
> I didn't (and still don't) have enough bandwidth to *drive* something like
> this through though, but if you want to do that I'd be more than happy to
> be kept in the loop on what you're doing and can possibly find time to
> write some code too.
>
> Regards, E.
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/benhoyt%
> 40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170524/e8fa26aa/attachment.html>

From mark at hotpy.org  Wed May 24 17:11:50 2017
From: mark at hotpy.org (Mark Shannon)
Date: Wed, 24 May 2017 14:11:50 -0700
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
Message-ID: <124ca628-6f8e-ce0b-4a91-5c2922e3e673@hotpy.org>

On 24/05/17 11:07, Ben Hoyt wrote:
> Hi folks,
>
> I was looking at some `dis` output today, and I was wondering if anyone
> has investigated optimizing Python (slightly) by adding special-case
> bytecodes for common expressions or statements involving constants?

[snip]

Hi Ben,

What you are suggesting is an ad-hoc form of what is known as "static 
superinstructions".
Take a look at http://www.complang.tuwien.ac.at/projects/interpreters.html

Cheers,
Mark.


From python at lucidity.plus.com  Wed May 24 17:22:57 2017
From: python at lucidity.plus.com (Erik)
Date: Wed, 24 May 2017 22:22:57 +0100
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <CAL9jXCFfxF99zo-pCCS7JMr-73CaaU60Xw-PbscMsTnDvULKoA@mail.gmail.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
 <dda9cdaf-0bf4-ab0d-d884-a5dc447bca2b@lucidity.plus.com>
 <CAL9jXCFfxF99zo-pCCS7JMr-73CaaU60Xw-PbscMsTnDvULKoA@mail.gmail.com>
Message-ID: <73d18830-6468-1d90-c340-32d46d03e0cd@lucidity.plus.com>

On 24/05/17 21:56, Ben Hoyt wrote:
> But interesting to know you got not much of a 
> speedup! 

I think that improvements at the hardware level in terms of 
parallelizing instruction and data fetching (and branch prediction) in 
even the cheapest processors these days have largely trivialized the 
amount of time it take the interpreter loop to read another opcode and 
branch to the code that executes it.

I think that the answer to speeding things up is better algorithms at a 
higher level rather than micro-optimizations.

But it's still fun to _try_ this sort of thing isn't it? ;)

> I'm wondering if the simplest way to test would be to add the new 
> opcodes and handle them in ceval.c, but then do all the bytecode 
> manipulation in a pure Python peephole optimizer ("bytecode munger").

There used to be just such a thing in pure Python (but not with _new_ 
opcodes, obviously) - Skip Montanaro wrote it IIRC. Probably about 1997 
or so. I think that may be where the current peephole optimizer 
originated (the way it works is certainly similar to the Python version 
I remember experimenting with back then).

E.

From levkivskyi at gmail.com  Wed May 24 17:31:47 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 24 May 2017 23:31:47 +0200
Subject: [Python-Dev] PEP 544: Protocols - second round
Message-ID: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>

Hi all,

After collecting suggestions in the previous discussion on python-dev
https://mail.python.org/pipermail/python-dev/2017-March/thread.html#147629
and playing with implementation, here is an updated version of PEP 544.

--
Ivan


A link for those who don't like reading long e-mails:
https://www.python.org/dev/peps/pep-0544/

=========================

PEP: 544
Title: Protocols
Version: $Revision$
Last-Modified: $Date$
Author: Ivan Levkivskyi <levkivskyi at gmail.com>, Jukka Lehtosalo <
jukka.lehtosalo at iki.fi>, ?ukasz Langa <lukasz at langa.pl>
Discussions-To: Python-Dev <python-dev at python.org>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 05-Mar-2017
Python-Version: 3.7


Abstract
========

Type hints introduced in PEP 484 can be used to specify type metadata
for static type checkers and other third party tools. However, PEP 484
only specifies the semantics of *nominal* subtyping. In this PEP we specify
static and runtime semantics of protocol classes that will provide a support
for *structural* subtyping (static duck typing).


.. _rationale:

Rationale and Goals
===================

Currently, PEP 484 and the ``typing`` module [typing]_ define abstract
base classes for several common Python protocols such as ``Iterable`` and
``Sized``. The problem with them is that a class has to be explicitly marked
to support them, which is unpythonic and unlike what one would
normally do in idiomatic dynamically typed Python code. For example,
this conforms to PEP 484::

  from typing import Sized, Iterable, Iterator

  class Bucket(Sized, Iterable[int]):
      ...
      def __len__(self) -> int: ...
      def __iter__(self) -> Iterator[int]: ...

The same problem appears with user-defined ABCs: they must be explicitly
subclassed or registered. This is particularly difficult to do with library
types as the type objects may be hidden deep in the implementation
of the library. Also, extensive use of ABCs might impose additional
runtime costs.

The intention of this PEP is to solve all these problems
by allowing users to write the above code without explicit base classes in
the class definition, allowing ``Bucket`` to be implicitly considered
a subtype of both ``Sized`` and ``Iterable[int]`` by static type checkers
using structural [wiki-structural]_ subtyping::

  from typing import Iterator, Iterable

  class Bucket:
      ...
      def __len__(self) -> int: ...
      def __iter__(self) -> Iterator[int]: ...

  def collect(items: Iterable[int]) -> int: ...
  result: int = collect(Bucket())  # Passes type check

Note that ABCs in ``typing`` module already provide structural behavior
at runtime, ``isinstance(Bucket(), Iterable)`` returns ``True``.
The main goal of this proposal is to support such behavior statically.
The same functionality will be provided for user-defined protocols, as
specified below. The above code with a protocol class matches common Python
conventions much better. It is also automatically extensible and works
with additional, unrelated classes that happen to implement
the required protocol.


Nominal vs structural subtyping
-------------------------------

Structural subtyping is natural for Python programmers since it matches
the runtime semantics of duck typing: an object that has certain properties
is treated independently of its actual runtime class.
However, as discussed in PEP 483, both nominal and structural
subtyping have their strengths and weaknesses. Therefore, in this PEP we
*do not propose* to replace the nominal subtyping described by PEP 484 with
structural subtyping completely. Instead, protocol classes as specified in
this PEP complement normal classes, and users are free to choose
where to apply a particular solution. See section on `rejected`_ ideas at
the
end of this PEP for additional motivation.


Non-goals
---------

At runtime, protocol classes will be simple ABCs. There is no intent to
provide sophisticated runtime instance and class checks against protocol
classes. This would be difficult and error-prone and will contradict the
logic
of PEP 484. As well, following PEP 484 and PEP 526 we state that protocols
are
**completely optional**:

* No runtime semantics will be imposed for variables or parameters annotated
  with a protocol class.
* Any checks will be performed only by third-party type checkers and
  other tools.
* Programmers are free to not use them even if they use type annotations.
* There is no intent to make protocols non-optional in the future.


Existing Approaches to Structural Subtyping
===========================================

Before describing the actual specification, we review and comment on
existing
approaches related to structural subtyping in Python and other languages:

* ``zope.interface`` [zope-interfaces]_ was one of the first widely used
  approaches to structural subtyping in Python. It is implemented by
providing
  special classes to distinguish interface classes from normal classes,
  to mark interface attributes, and to explicitly declare implementation.
  For example::

    from zope.interface import Interface, Attribute, implementer

    class IEmployee(Interface):

        name = Attribute("Name of employee")

        def do(work):
            """Do some work"""

    @implementer(IEmployee)
    class Employee:

        name = 'Anonymous'

        def do(self, work):
            return work.start()

  Zope interfaces support various contracts and constraints for interface
  classes. For example::

    from zope.interface import invariant

    def required_contact(obj):
        if not (obj.email or obj.phone):
            raise Exception("At least one contact info is required")

    class IPerson(Interface):

        name = Attribute("Name")
        email = Attribute("Email Address")
        phone = Attribute("Phone Number")

        invariant(required_contact)

  Even more detailed invariants are supported. However, Zope interfaces rely
  entirely on runtime validation. Such focus on runtime properties goes
  beyond the scope of the current proposal, and static support for
invariants
  might be difficult to implement. However, the idea of marking an interface
  class with a special base class is reasonable and easy to implement both
  statically and at runtime.

* Python abstract base classes [abstract-classes]_ are the standard
  library tool to provide some functionality similar to structural
subtyping.
  The drawback of this approach is the necessity to either subclass
  the abstract class or register an implementation explicitly::

    from abc import ABC

    class MyTuple(ABC):
        pass

    MyTuple.register(tuple)

    assert issubclass(tuple, MyTuple)
    assert isinstance((), MyTuple)

  As mentioned in the `rationale`_, we want to avoid such necessity,
especially
  in static context. However, in a runtime context, ABCs are good
candidates for
  protocol classes and they are already used extensively in
  the ``typing`` module.

* Abstract classes defined in ``collections.abc`` module [collections-abc]_
  are slightly more advanced since they implement a custom
  ``__subclasshook__()`` method that allows runtime structural checks
without
  explicit registration::

    from collections.abc import Iterable

    class MyIterable:
        def __iter__(self):
            return []

    assert isinstance(MyIterable(), Iterable)

  Such behavior seems to be a perfect fit for both runtime and static
behavior
  of protocols. As discussed in `rationale`_, we propose to add static
support
  for such behavior. In addition, to allow users to achieve such runtime
  behavior for *user-defined* protocols a special ``@runtime`` decorator
will
  be provided, see detailed `discussion`_ below.

* TypeScript [typescript]_ provides support for user-defined classes and
  interfaces. Explicit implementation declaration is not required and
  structural subtyping is verified statically. For example::

    interface LabeledItem {
        label: string;
        size?: int;
    }

    function printLabel(obj: LabeledItem) {
        console.log(obj.label);
    }

    let myObj = {size: 10, label: "Size 10 Object"};
    printLabel(myObj);

  Note that optional interface members are supported. Also, TypeScript
  prohibits redundant members in implementations. While the idea of
  optional members looks interesting, it would complicate this proposal and
  it is not clear how useful it will be. Therefore it is proposed to
postpone
  this; see `rejected`_ ideas. In general, the idea of static protocol
  checking without runtime implications looks reasonable, and basically
  this proposal follows the same line.

* Go [golang]_ uses a more radical approach and makes interfaces the primary
  way to provide type information. Also, assignments are used to explicitly
  ensure implementation::

    type SomeInterface interface {
        SomeMethod() ([]byte, error)
    }

    if _, ok := someval.(SomeInterface); ok {
        fmt.Printf("value implements some interface")
    }

  Both these ideas are questionable in the context of this proposal. See
  the section on `rejected`_ ideas.


.. _specification:

Specification
=============

Terminology
-----------

We propose to use the term *protocols* for types supporting structural
subtyping. The reason is that the term *iterator protocol*,
for example, is widely understood in the community, and coming up with
a new term for this concept in a statically typed context would just create
confusion.

This has the drawback that the term *protocol* becomes overloaded with
two subtly different meanings: the first is the traditional, well-known but
slightly fuzzy concept of protocols such as iterator; the second is the more
explicitly defined concept of protocols in statically typed code.
The distinction is not important most of the time, and in other
cases we propose to just add a qualifier such as *protocol classes*
when referring to the static type concept.

If a class includes a protocol in its MRO, the class is called
an *explicit* subclass of the protocol. If a class is a structural subtype
of a protocol, it is said to implement the protocol and to be compatible
with a protocol. If a class is compatible with a protocol but the protocol
is not included in the MRO, the class is an *implicit* subtype
of the protocol. (Note that one can explicitly subclass a protocol and
still not implement it if a protocol attribute is set to ``None``
in the subclass, see Python [data-model]_ for details.)

The attributes (variables and methods) of a protocol that are mandatory
for other class in order to be considered a structural subtype are called
protocol members.


.. _definition:

Defining a protocol
-------------------

Protocols are defined by including a special new class ``typing.Protocol``
(an instance of ``abc.ABCMeta``) in the base classes list, typically
at the end of the list. Here is a simple example::

  from typing import Protocol

  class SupportsClose(Protocol):
      def close(self) -> None:
          ...

Now if one defines a class ``Resource`` with a ``close()`` method that has
a compatible signature, it would implicitly be a subtype of
``SupportsClose``, since the structural subtyping is used for
protocol types::

  class Resource:
      ...
      def close(self) -> None:
          self.file.close()
          self.lock.release()

Apart from few restrictions explicitly mentioned below, protocol types can
be used in every context where a normal types can::

  def close_all(things: Iterable[SupportsClose]) -> None:
      for t in things:
          t.close()

  f = open('foo.txt')
  r = Resource()
  close_all([f, r])  # OK!
  close_all([1])     # Error: 'int' has no 'close' method

Note that both the user-defined class ``Resource`` and the built-in
``IO`` type (the return type of ``open()``) are considered subtypes of
``SupportsClose``, because they provide a ``close()`` method with
a compatible type signature.


Protocol members
----------------

All methods defined in the protocol class body are protocol members, both
normal and decorated with ``@abstractmethod``. If any parameters of a
protocol method are not annotated, then their types are assumed to be
``Any``
(see PEP 484). Bodies of protocol methods are type checked.
An abstract method that should not be called via ``super()`` ought to raise
``NotImplementedError``. Example::

  from typing import Protocol
  from abc import abstractmethod

  class Example(Protocol):
      def first(self) -> int:     # This is a protocol member
          return 42

      @abstractmethod
      def second(self) -> int:    # Method without a default implementation
          raise NotImplementedError

Static methods, class methods, and properties are equally allowed
in protocols.

To define a protocol variable, one can use PEP 526 variable
annotations in the class body. Additional attributes *only* defined in
the body of a method by assignment via ``self`` are not allowed. The
rationale
for this is that the protocol class implementation is often not shared by
subtypes, so the interface should not depend on the default implementation.
Examples::

  from typing import Protocol, List

  class Template(Protocol):
      name: str        # This is a protocol member
      value: int = 0   # This one too (with default)

      def method(self) -> None:
          self.temp: List[int] = [] # Error in type checker

  class Concrete:
      def __init__(self, name: str, value: int) -> None:
          self.name = name
          self.value = value

  var: Template = Concrete('value', 42)  # OK

To distinguish between protocol class variables and protocol instance
variables, the special ``ClassVar`` annotation should be used as specified
by PEP 526. By default, protocol variables as defined above are considered
readable and writable. To define a read-only protocol variable, one can use
an (abstract) property.


Explicitly declaring implementation
-----------------------------------

To explicitly declare that a certain class implements a given protocol,
it can be used as a regular base class. In this case a class could use
default implementations of protocol members. ``typing.Sequence`` is a good
example of a protocol with useful default methods. Static analysis tools are
expected to automatically detect that a class implements a given protocol.
So while it's possible to subclass a protocol explicitly, it's *not
necessary*
to do so for the sake of type-checking.

The default implementations cannot be used if
the subtype relationship is implicit and only via structural
subtyping -- the semantics of inheritance is not changed. Examples::

    class PColor(Protocol):
        @abstractmethod
        def draw(self) -> str:
            ...
        def complex_method(self) -> int:
            # some complex code here

    class NiceColor(PColor):
        def draw(self) -> str:
            return "deep blue"

    class BadColor(PColor):
        def draw(self) -> str:
            return super().draw()  # Error, no default implementation

    class ImplicitColor:   # Note no 'PColor' base here
        def draw(self) -> str:
            return "probably gray"
        def comlex_method(self) -> int:
            # class needs to implement this

    nice: NiceColor
    another: ImplicitColor

    def represent(c: PColor) -> None:
        print(c.draw(), c.complex_method())

    represent(nice) # OK
    represent(another) # Also OK

Note that there is little difference between explicit and implicit
subtypes, the main benefit of explicit subclassing is to get some protocol
methods "for free". In addition, type checkers can statically verify that
the class actually implements the protocol correctly::

    class RGB(Protocol):
        rgb: Tuple[int, int, int]

        @abstractmethod
        def intensity(self) -> int:
            return 0

    class Point(RGB):
        def __init__(self, red: int, green: int, blue: str) -> None:
            self.rgb = red, green, blue  # Error, 'blue' must be 'int'

        # Type checker might warn that 'intensity' is not defined

A class can explicitly inherit from multiple protocols and also form normal
classes. In this case methods are resolved using normal MRO and a type
checker
verifies that all subtyping are correct. The semantics of
``@abstractmethod``
is not changed, all of them must be implemented by an explicit subclass
before it can be instantiated.


Merging and extending protocols
-------------------------------

The general philosophy is that protocols are mostly like regular ABCs,
but a static type checker will handle them specially. Subclassing a protocol
class would not turn the subclass into a protocol unless it also has
``typing.Protocol`` as an explicit base class. Without this base, the class
is "downgraded" to a regular ABC that cannot be used with structural
subtyping. The rationale for this rule is that we don't want to accidentally
have some class act as a protocol just because one of its base classes
happens to be one. We still slightly prefer nominal subtyping over
structural
subtyping in the static typing world.

A subprotocol can be defined by having *both* one or more protocols as
immediate base classes and also having ``typing.Protocol`` as an immediate
base class::

  from typing import Sized, Protocol

  class SizedAndClosable(Sized, Protocol):
      def close(self) -> None:
          ...

Now the protocol ``SizedAndClosable`` is a protocol with two methods,
``__len__`` and ``close``. If one omits ``Protocol`` in the base class list,
this would be a regular (non-protocol) class that must implement ``Sized``.
Alternatively, one can implement ``SizedAndClosable`` protocol by merging
the ``SupportsClose`` protocol from the example in the `definition`_ section
with ``typing.Sized``::

  from typing import Sized

  class SupportsClose(Protocol):
      def close(self) -> None:
          ...

  class SizedAndClosable(Sized, SupportsClose, Protocol):
      pass

The two definitions of ``SizedAndClosable`` are equivalent.
Subclass relationships between protocols are not meaningful when
considering subtyping, since structural compatibility is
the criterion, not the MRO.

If ``Protocol`` is included in the base class list, all the other base
classes
must be protocols. A protocol can't extend a regular class, see `rejected`_
ideas for rationale. Note that rules around explicit subclassing are
different
from regular ABCs, where abstractness is simply defined by having at least
one
abstract method being unimplemented. Protocol classes must be marked
*explicitly*.


Generic protocols
-----------------

Generic protocols are important. For example, ``SupportsAbs``, ``Iterable``
and ``Iterator`` are generic protocols. They are defined similar to normal
non-protocol generic types::

  class Iterable(Protocol[T]):
      @abstractmethod
      def __iter__(self) -> Iterator[T]:
          ...

``Protocol[T, S, ...]`` is allowed as a shorthand for
``Protocol, Generic[T, S, ...]``.

User-defined generic protocols support explicitly declared variance.
Type checkers will warn if the inferred variance is different from
the declared variance. Examples::

  T = TypeVar('T')
  T_co = TypeVar('T_co', covariant=True)
  T_contra = TypeVar('T_contra', contravariant=True)

  class Box(Protocol[T_co]):
      def content(self) -> T_co:
          ...

  box: Box[float]
  second_box: Box[int]
  box = second_box  # This is OK due to the covariance of 'Box'.

  class Sender(Protocol[T_contra]):
      def send(self, data: T_contra) -> int:
          ...

  sender: Sender[float]
  new_sender: Sender[int]
  new_sender = sender  # OK, 'Sender' is contravariant.

  class Proto(Protocol[T]):
      attr: T  # this class is invariant, since it has a mutable attribute

  var: Proto[float]
  another_var: Proto[int]
  var = another_var  # Error! 'Proto[float]' is incompatible with
'Proto[int]'.

Note that unlike nominal classes, de-facto covariant protocols cannot be
declared as invariant, since this can break transitivity of subtyping
(see `rejected`_ ideas for details). For example::

  T = TypeVar('T')

  class AnotherBox(Protocol[T]):  # Error, this protocol is covariant in T,
      def content(self) -> T:     # not invariant.
          ...


Recursive protocols
-------------------

Recursive protocols are also supported. Forward references to the protocol
class names can be given as strings as specified by PEP 484. Recursive
protocols are useful for representing self-referential data structures
like trees in an abstract fashion::

  class Traversable(Protocol):
      def leaves(self) -> Iterable['Traversable']:
          ...

Note that for recursive protocols, a class is considered a subtype of
the protocol in situations where the decision depends on itself.
Continuing the previous example::

  class SimpleTree:
      def leaves(self) -> List['SimpleTree']:
          ...

  root: Traversable = SimpleTree()  # OK

  class Tree(Generic[T]):
      def leaves(self) -> List['Tree[T]']:
          ...

  def walk(graph: Traversable) -> None:
      ...
  tree: Tree[float] = Tree()
  walk(tree)  # OK, 'Tree[float]' is a subtype of 'Traversable'


Using Protocols
===============

Subtyping relationships with other types
----------------------------------------

Protocols cannot be instantiated, so there are no values whose
runtime type is a protocol. For variables and parameters with protocol
types,
subtyping relationships are subject to the following rules:

* A protocol is never a subtype of a concrete type.
* A concrete type ``X`` is a subtype of protocol ``P``
  if and only if ``X`` implements all protocol members of ``P`` with
  compatible types. In other words, subtyping with respect to a protocol is
  always structural.
* A protocol ``P1`` is a subtype of another protocol ``P2`` if ``P1``
defines
  all protocol members of ``P2`` with compatible types.

Generic protocol types follow the same rules of variance as non-protocol
types. Protocol types can be used in all contexts where any other types
can be used, such as in ``Union``, ``ClassVar``, type variables bounds, etc.
Generic protocols follow the rules for generic abstract classes, except for
using structural compatibility instead of compatibility defined by
inheritance relationships.


Unions and intersections of protocols
-------------------------------------

``Union`` of protocol classes behaves the same way as for non-protocol
classes. For example::

  from typing import Union, Optional, Protocol

  class Exitable(Protocol):
      def exit(self) -> int:
          ...
  class Quittable(Protocol):
      def quit(self) -> Optional[int]:
          ...

  def finish(task: Union[Exitable, Quittable]) -> int:
      ...
  class DefaultJob:
      ...
      def quit(self) -> int:
          return 0
  finish(DefaultJob()) # OK

One can use multiple inheritance to define an intersection of protocols.
Example::

  from typing import Sequence, Hashable

  class HashableFloats(Sequence[float], Hashable, Protocol):
      pass

  def cached_func(args: HashableFloats) -> float:
      ...
  cached_func((1, 2, 3)) # OK, tuple is both hashable and sequence

If this will prove to be a widely used scenario, then a special
intersection type construct could be added in future as specified by PEP
483,
see `rejected`_ ideas for more details.


``Type[]`` with protocols
-------------------------

Variables and parameters annotated with ``Type[Proto]`` accept only concrete
(non-protocol) subtypes of ``Proto``. The main reason for this is to allow
instantiation of parameters with such type. For example::

  class Proto(Protocol):
      @abstractmethod
      def meth(self) -> int:
          ...
  class Concrete:
      def meth(self) -> int:
          return 42

  def fun(cls: Type[Proto]) -> int:
      return cls().meth() # OK
  fun(Proto)              # Error
  fun(Concrete)           # OK

The same rule applies to variables::

  var: Type[Proto]
  var = Proto    # Error
  var = Concrete # OK
  var().meth()   # OK

Assigning an ABC or a protocol class to a variable is allowed if it is
not explicitly typed, and such assignment creates a type alias.
For normal (non-abstract) classes, the behavior of ``Type[]`` is
not changed.


``NewType()`` and type aliases
------------------------------

Protocols are essentially anonymous. To emphasize this point, static type
checkers might refuse protocol classes inside ``NewType()`` to avoid an
illusion that a distinct type is provided::

  from typing import NewType, Protocol, Iterator

  class Id(Protocol):
      code: int
      secrets: Iterator[bytes]

  UserId = NewType('UserId', Id)  # Error, can't provide distinct type

In contrast, type aliases are fully supported, including generic type
aliases::

  from typing import TypeVar, Reversible, Iterable, Sized

  T = TypeVar('T')
  class SizedIterable(Iterable[T], Sized, Protocol):
      pass
  CompatReversible = Union[Reversible[T], SizedIterable[T]]


.. _discussion:

``@runtime`` decorator and narrowing types by ``isinstance()``
--------------------------------------------------------------

The default semantics is that ``isinstance()`` and ``issubclass()`` fail
for protocol types. This is in the spirit of duck typing -- protocols
basically would be used to model duck typing statically, not explicitly
at runtime.

However, it should be possible for protocol types to implement custom
instance and class checks when this makes sense, similar to how ``Iterable``
and other ABCs in ``collections.abc`` and ``typing`` already do it,
but this is limited to non-generic and unsubscripted generic protocols
(``Iterable`` is statically equivalent to ``Iterable[Any]`).
The ``typing`` module will define a special ``@runtime`` class decorator
that provides the same semantics for class and instance checks as for
``collections.abc`` classes, essentially making them "runtime protocols"::

  from typing import runtime, Protocol

  @runtime
  class Closable(Protocol):
      def close(self):
          ...

  assert isinstance(open('some/file'), Closable)

Static type checkers will understand ``isinstance(x, Proto)`` and
``issubclass(C, Proto)`` for protocols defined with this decorator (as they
already do for ``Iterable`` etc.). Static type checkers will narrow types
after such checks by the type erased ``Proto`` (i.e. with all variables
having type ``Any`` and all methods having type ``Callable[..., Any]``).
Note that ``isinstance(x, Proto[int])`` etc. will always fail in agreement
with PEP 484. Examples::

  from typing import Iterable, Iterator, Sequence

  def process(items: Iterable[int]) -> None:
      if isinstance(items, Iterator):
          # 'items' has type 'Iterator[int]' here
      elif isinstance(items, Sequence[int]):
          # Error! Can't use 'isinstance()' with subscripted protocols

Note that instance checks are not 100% reliable statically, this is why
this behavior is opt-in, see section on `rejected`_ ideas for examples.


Using Protocols in Python 2.7 - 3.5
===================================

Variable annotation syntax was added in Python 3.6, so that the syntax
for defining protocol variables proposed in `specification`_ section can't
be used if support for earlier versions is needed. To define these
in a manner compatible with older versions of Python one can use properties.
Properties can be settable and/or abstract if needed::

  class Foo(Protocol):
      @property
      def c(self) -> int:
          return 42         # Default value can be provided for property...

      @abstractproperty
      def d(self) -> int:   # ... or it can be abstract
          return 0

Also function type comments can be used as per PEP 484 (for example
to provide compatibility with Python 2). The ``typing`` module changes
proposed in this PEP will also be backported to earlier versions via the
backport currently available on PyPI.


Runtime Implementation of Protocol Classes
==========================================

Implementation details
----------------------

The runtime implementation could be done in pure Python without any
effects on the core interpreter and standard library except in the
``typing`` module, and a minor update to ``collections.abc``:

* Define class ``typing.Protocol`` similar to ``typing.Generic``.
* Implement metaclass functionality to detect whether a class is
  a protocol or not. Add a class attribute ``_is_protocol = True``
  if that is the case. Verify that a protocol class only has protocol
  base classes in the MRO (except for object).
* Implement ``@runtime`` that allows ``__subclasshook__()`` performing
  structural instance and subclass checks as in ``collections.abc`` classes.
* All structural subtyping checks will be performed by static type checkers,
  such as ``mypy`` [mypy]_. No additional support for protocol validation
will
  be provided at runtime.
* Classes ``Mapping``, ``MutableMapping``, ``Sequence``, and
  ``MutableSequence`` in ``collections.abc`` module will support structural
  instance and subclass checks (like e.g. ``collections.abc.Iterable``).


Changes in the typing module
----------------------------

The following classes in ``typing`` module will be protocols:

* ``Callable``
* ``Awaitable``
* ``Iterable``, ``Iterator``
* ``AsyncIterable``, ``AsyncIterator``
* ``Hashable``
* ``Sized``
* ``Container``
* ``Collection``
* ``Reversible``
* ``Sequence``, ``MutableSequence``
* ``Mapping``, ``MutableMapping``
* ``ContextManager``, ``AsyncContextManager``
* ``SupportsAbs`` (and other ``Supports*`` classes)

Most of these classes are small and conceptually simple. It is easy to see
what are the methods these protocols implement, and immediately recognize
the corresponding runtime protocol counterpart.
Practically, few changes will be needed in ``typing`` since some of these
classes already behave the necessary way at runtime. Most of these will need
to be updated only in the corresponding ``typeshed`` stubs [typeshed]_.

All other concrete generic classes such as ``List``, ``Set``, ``IO``,
``Deque``, etc are sufficiently complex that it makes sense to keep
them non-protocols (i.e. require code to be explicit about them). Also, it
is
too easy to leave some methods unimplemented by accident, and explicitly
marking the subclass relationship allows type checkers to pinpoint the
missing
implementations.


Introspection
-------------

The existing class introspection machinery (``dir``, ``__annotations__``
etc)
can be used with protocols. In addition, all introspection tools implemented
in the ``typing`` module will support protocols. Since all attributes need
to be defined in the class body based on this proposal, protocol classes
will
have even better perspective for introspection than regular classes where
attributes can be defined implicitly -- protocol attributes can't be
initialized in ways that are not visible to introspection
(using ``setattr()``, assignment via ``self``, etc.). Still, some things
like
types of attributes will not be visible at runtime in Python 3.5 and
earlier,
but this looks like a reasonable limitation.

There will be only limited support of ``isinstance()`` and ``issubclass()``
as discussed above (these will *always* fail with ``TypeError`` for
subscripted generic protocols, since a reliable answer could not be given
at runtime in this case). But together with other introspection tools this
give a reasonable perspective for runtime type checking tools.


.. _rejected:

Rejected/Postponed Ideas
========================

The ideas in this section were previously discussed in [several]_
[discussions]_ [elsewhere]_.

Make every class a protocol by default
--------------------------------------

Some languages such as Go make structural subtyping the only or the primary
form of subtyping. We could achieve a similar result by making all classes
protocols by default (or even always). However we believe that it is better
to require classes to be explicitly marked as protocols, for the following
reasons:

* Protocols don't have some properties of regular classes. In particular,
  ``isinstance()``, as defined for normal classes, is based on the nominal
  hierarchy. In order to make everything a protocol by default, and have
  ``isinstance()`` work would require changing its semantics,
  which won't happen.
* Protocol classes should generally not have many method implementations,
  as they describe an interface, not an implementation.
  Most classes have many method implementations, making them bad protocol
  classes.
* Experience suggests that many classes are not practical as protocols
anyway,
  mainly because their interfaces are too large, complex or
  implementation-oriented (for example, they may include de facto
  private attributes and methods without a ``__`` prefix).
* Most actually useful protocols in existing Python code seem to be
implicit.
  The ABCs in ``typing`` and ``collections.abc`` are rather an exception,
but
  even they are recent additions to Python and most programmers
  do not use them yet.
* Many built-in functions only accept concrete instances of ``int``
  (and subclass instances), and similarly for other built-in classes. Making
  ``int`` a structural type wouldn't be safe without major changes to the
  Python runtime, which won't happen.


Protocols subclassing normal classes
------------------------------------

The main rationale to prohibit this is to preserve transitivity of
subtyping,
consider this example::

  from typing import Protocol

  class Base:
      attr: str

  class Proto(Base, Protocol):
      def meth(self) -> int:
          ...

  class C:
      attr: str
      def meth(self) -> int:
          return 0

Now, ``C`` is a subtype of ``Proto``, and ``Proto`` is a subtype of
``Base``.
But ``C`` cannot be a subtype of ``Base`` (since the latter is not
a protocol). This situation would be really weird. In addition, there is
an ambiguity about whether attributes of ``Base`` should become protocol
members of ``Proto``.


Support optional protocol members
---------------------------------

We can come up with examples where it would be handy to be able to say
that a method or data attribute does not need to be present in a class
implementing a protocol, but if it is present, it must conform to a specific
signature or type. One could use a ``hasattr()`` check to determine whether
they can use the attribute on a particular instance.

Languages such as TypeScript have similar features and
apparently they are pretty commonly used. The current realistic potential
use cases for protocols in Python don't require these. In the interest
of simplicity, we propose to not support optional methods or attributes.
We can always revisit this later if there is an actual need.


Allow only protocol methods and force use of getters and setters
----------------------------------------------------------------

One could argue that protocols typically only define methods, but not
variables. However, using getters and setters in cases where only a
simple variable is needed would be quite unpythonic. Moreover, the
widespread
use of properties (that often act as type validators) in large code bases
is partially due to previous absence of static type checkers for Python,
the problem that PEP 484 and this PEP are aiming to solve. For example::

  # without static types

  class MyClass:
      @property
      def my_attr(self):
          return self._my_attr
      @my_attr.setter
      def my_attr(self, value):
          if not isinstance(value, int):
              raise ValidationError("An integer expected for my_attr")
          self._my_attr = value

  # with static types

  class MyClass:
      my_attr: int


Support non-protocol members
----------------------------

There was an idea to make some methods "non-protocol" (i.e. not necessary
to implement, and inherited in explicit subclassing), but it was rejected,
since this complicates things. For example, consider this situation::

  class Proto(Protocol):
      @abstractmethod
      def first(self) -> int:
          raise NotImplementedError
      def second(self) -> int:
          return self.first() + 1

  def fun(arg: Proto) -> None:
      arg.second()

The question is should this be an error? We think most people would expect
this to be valid. Therefore, to be on the safe side, we need to require both
methods to be implemented in implicit subclasses. In addition, if one looks
at definitions in ``collections.abc``, there are very few methods that could
be considered "non-protocol". Therefore, it was decided to not introduce
"non-protocol" methods.

There is only one downside to this: it will require some boilerplate for
implicit subtypes of ``Mapping`` and few other "large" protocols. But, this
applies to few "built-in" protocols (like ``Mapping`` and ``Sequence``) and
people are already subclassing them. Also, such style is discouraged for
user-defined protocols. It is recommended to create compact protocols and
combine them.


Make protocols interoperable with other approaches
--------------------------------------------------

The protocols as described here are basically a minimal extension to
the existing concept of ABCs. We argue that this is the way they should
be understood, instead of as something that *replaces* Zope interfaces,
for example. Attempting such interoperabilities will significantly
complicate both the concept and the implementation.

On the other hand, Zope interfaces are conceptually a superset of protocols
defined here, but using an incompatible syntax to define them,
because before PEP 526 there was no straightforward way to annotate
attributes.
In the 3.6+ world, ``zope.interface`` might potentially adopt the
``Protocol``
syntax. In this case, type checkers could be taught to recognize interfaces
as protocols and make simple structural checks with respect to them.


Use assignments to check explicitly that a class implements a protocol
----------------------------------------------------------------------

In the Go language the explicit checks for implementation are performed
via dummy assignments [golang]_. Such a way is also possible with the
current proposal. Example::

  class A:
      def __len__(self) -> float:
          return ...

  _: Sized = A()  # Error: A.__len__ doesn't conform to 'Sized'
                  # (Incompatible return type 'float')

This approach moves the check away from
the class definition and it almost requires a comment as otherwise
the code probably would not make any sense to an average reader
-- it looks like dead code. Besides, in the simplest form it requires one
to construct an instance of ``A``, which could be problematic if this
requires
accessing or allocating some resources such as files or sockets.
We could work around the latter by using a cast, for example, but then
the code would be ugly. Therefore we discourage the use of this pattern.


Support ``isinstance()`` checks by default
------------------------------------------

The problem with this is instance checks could be unreliable, except for
situations where there is a common signature convention such as
``Iterable``.
For example::

  class P(Protocol):
      def common_method_name(self, x: int) -> int: ...

  class X:
      <a bunch of methods>
      def common_method_name(self) -> None: ... # Note different signature

  def do_stuff(o: Union[P, X]) -> int:
      if isinstance(o, P):
          return o.common_method_name(1)  # oops, what if it's an X
instance?

Another potentially problematic case is assignment of attributes
*after* instantiation::

  class P(Protocol):
      x: int

  class C:
      def initialize(self) -> None:
          self.x = 0

  c = C()
  isinstance(c1, P)  # False
  c.initialize()
  isinstance(c, P)  # True

  def f(x: Union[P, int]) -> None:
      if isinstance(x, P):
          # static type of x is P here
          ...
      else:
          # type of x is "int" here?
          print(x + 1)

  f(C())   # oops

We argue that requiring an explicit class decorator would be better, since
one can then attach warnings about problems like this in the documentation.
The user would be able to evaluate whether the benefits outweigh
the potential for confusion for each protocol and explicitly opt in -- but
the default behavior would be safer. Finally, it will be easy to make this
behavior default if necessary, while it might be problematic to make it
opt-in
after being default.


Provide a special intersection type construct
---------------------------------------------

There was an idea to allow ``Proto = All[Proto1, Proto2, ...]`` as a
shorthand
for::

  class Proto(Proto1, Proto2, ..., Protocol):
      pass

However, it is not yet clear how popular/useful it will be and implementing
this in type checkers for non-protocol classes could be difficult. Finally,
it
will be very easy to add this later if needed.


Prohibit explicit subclassing of protocols by non-protocols
-----------------------------------------------------------

This was rejected for the following reasons:

* Backward compatibility: People are already using ABCs, including generic
  ABCs from ``typing`` module. If we prohibit explicit subclassing of these
  ABCs, then quite a lot of code will break.

* Convenience: There are existing protocol-like ABCs (that will be turned
  into protocols) that have many useful "mix-in" (non-abstract) methods.
  For example in the case of ``Sequence`` one only needs to implement
  ``__getitem__`` and ``__len__`` in an explicit subclass, and one gets
  ``__iter__``, ``__contains__``, ``__reversed__``, ``index``, and ``count``
  for free.

* Explicit subclassing makes it explicit that a class implements a
particular
  protocol, making subtyping relationships easier to see.

* Type checkers can warn about missing protocol members or members with
  incompatible types more easily, without having to use hacks like dummy
  assignments discussed above in this section.

* Explicit subclassing makes it possible to force a class to be considered
  a subtype of a protocol (by using ``# type: ignore`` together with an
  explicit base class) when it is not strictly compatible, such as when
  it has an unsafe override.


Covariant subtyping of mutable attributes
-----------------------------------------

Rejected because covariant subtyping of mutable attributes is not safe.
Consider this example::

  class P(Protocol):
      x: float

  def f(arg: P) -> None:
      arg.x = 0.42

  class C:
      x: int

  c = C()
  f(c)  # Would typecheck if covariant subtyping
        # of mutable attributes were allowed
  c.x >> 1  # But this fails at runtime

It was initially proposed to allow this for practical reasons, but it was
subsequently rejected, since this may mask some hard to spot bugs.


Overriding inferred variance of protocol classes
------------------------------------------------

It was proposed to allow declaring protocols as invariant if they are
actually
covariant or contravariant (as it is possible for nominal classes, see PEP
484).
However, it was decided not to do this because of several downsides:

* Declared protocol invariance breaks transitivity of sub-typing. Consider
  this situation::

    T = TypeVar('T')

    class P(Protocol[T]):  # Declared as invariant
        def meth(self) -> T:
            ...
    class C:
        def meth(self) -> float:
            ...
    class D(C):
        def meth(self) -> int:
            ...

  Now we have that ``D`` is a subtype of ``C``, and ``C`` is a subtype of
  ``P[float]``. But ``D`` is *not* a subtype of ``P[float]`` since ``D``
  implements ``P[int]``, and ``P`` is invariant. There is a possibility
  to "cure" this by looking for protocol implementations in MROs but this
  will be too complex in a general case, and this "cure" requires abandoning
  simple idea of purely structural subtyping for protocols.

* Subtyping checks will always require type inference for protocols. In the
  above example a user may complain: "Why did you infer ``P[int]`` for
  my ``D``? It implements ``P[float]``!". Normally, inference can be
overruled
  by an explicit annotation, but here this will require explicit
subclassing,
  defeating the purpose of using protocols.

* Allowing overriding variance will make impossible more detailed error
  messages in type checkers citing particular conflicts in member
  type signatures.

* Finally, explicit is better than implicit in this case. Requiring user to
  declare correct variance will simplify understanding the code and will
avoid
  unexpected errors at the point of use.


Support adapters and adaptation
-------------------------------

Adaptation was proposed by PEP 246 (rejected) and is supported by
``zope.interface``, see https://docs.zope.org/zope.interface/adapter.html.
Adapters is quite an advanced concept, and PEP 484 supports unions and
generic aliases that can be used instead of adapters. This can be
illustrated
with an example of ``Iterable`` protocol, there is another way of supporting
iteration by providing ``__getitem__`` and ``__len__``. If a function
supports both this way and the now standard ``__iter__`` method, then it
could
be annotated by a union type::

  class OldIterable(Sized, Protocol[T]):
      def __getitem__(self, item: int) -> T: ...

  CompatIterable = Union[Iterable[T], OldIterable[T]]

  class A:
      def __iter__(self) -> Iterator[str]: ...
  class B:
      def __len__(self) -> int: ...
      def __getitem__(self, item: int) -> str: ...

  def iterate(it: CompatIterable[str]) -> None:
      ...

  iterate(A())  # OK
  iterate(B())  # OK

Since there is a reasonable alternative for such cases with existing
tooling,
it is therefore proposed not to include adaptation in this PEP.


Backwards Compatibility
=======================

This PEP is almost fully backwards compatible. Few collection classes such
as
``Sequence`` and ``Mapping`` will be turned into runtime protocols,
therefore
results of ``isinstance()`` checks are going to change in some edge cases.
For example, a class that implements the ``Sequence`` protocol but does not
explicitly inherit from ``Sequence`` currently returns ``False`` in
corresponding instance and class checks. With this PEP implemented, such
checks will return ``True``.


Implementation
==============

A working implementation of this PEP for ``mypy`` type checker is found on
GitHub repo at https://github.com/ilevkivskyi/mypy/tree/protocols,
corresponding ``typeshed`` stubs for more flavor are found at
https://github.com/ilevkivskyi/typeshed/tree/protocols. Installation steps::

  git clone --recurse-submodules https://github.com/ilevkivskyi/mypy/
  cd mypy && git checkout protocols && cd typeshed
  git remote add proto https://github.com/ilevkivskyi/typeshed
  git fetch proto && git checkout proto/protocols
  cd .. && git add typeshed && sudo python3 -m pip install -U .

The runtime implementation of protocols in ``typing`` module is
found at https://github.com/ilevkivskyi/typehinting/tree/protocols.
The version of ``collections.abc`` with structural behavior for mappings and
sequences is found at https://github.com/ilevkivskyi/cpython/tree/protocols.


References
==========

.. [typing]
   https://docs.python.org/3/library/typing.html

.. [wiki-structural]
   https://en.wikipedia.org/wiki/Structural_type_system

.. [zope-interfaces]
   https://zopeinterface.readthedocs.io/en/latest/

.. [abstract-classes]
   https://docs.python.org/3/library/abc.html

.. [collections-abc]
   https://docs.python.org/3/library/collections.abc.html

.. [typescript]
   https://www.typescriptlang.org/docs/handbook/interfaces.html

.. [golang]
   https://golang.org/doc/effective_go.html#interfaces_and_types

.. [data-model]
   https://docs.python.org/3/reference/datamodel.html#special-method-names

.. [typeshed]
   https://github.com/python/typeshed/

.. [mypy]
   http://github.com/python/mypy/

.. [several]

https://mail.python.org/pipermail/python-ideas/2015-September/thread.html#35859

.. [discussions]
   https://github.com/python/typing/issues/11

.. [elsewhere]
   https://github.com/python/peps/pull/224


Copyright
=========

This document has been placed in the public domain.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170524/c2a630f9/attachment-0001.html>

From greg.ewing at canterbury.ac.nz  Wed May 24 18:33:30 2017
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 25 May 2017 10:33:30 +1200
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <73d18830-6468-1d90-c340-32d46d03e0cd@lucidity.plus.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
 <dda9cdaf-0bf4-ab0d-d884-a5dc447bca2b@lucidity.plus.com>
 <CAL9jXCFfxF99zo-pCCS7JMr-73CaaU60Xw-PbscMsTnDvULKoA@mail.gmail.com>
 <73d18830-6468-1d90-c340-32d46d03e0cd@lucidity.plus.com>
Message-ID: <59260A3A.9090601@canterbury.ac.nz>

When testing things like this, as well as testing whether it speeds
up your target cases, remember to check that it doesn't slow everything
else down due to the increased size of the eval code pushing something
out of instruction cache or some such effect.

-- 
Greg

From martin.wimpress at canonical.com  Thu May 25 06:32:33 2017
From: martin.wimpress at canonical.com (Martin Wimpress)
Date: Thu, 25 May 2017 11:32:33 +0100
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <CAP7+vJLE4T-cJnmjGTHOnVxL7L5VEtMPvpBP3Wb7xsL1vpnfAA@mail.gmail.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
 <0101015c32976a82-02f59fbb-9c76-4c8d-a338-4a2ea289f202-000000@us-west-2.amazonses.com>
 <65856131-5d27-c344-c0f3-6e7e99082454@canonical.com>
 <CAP7+vJLE4T-cJnmjGTHOnVxL7L5VEtMPvpBP3Wb7xsL1vpnfAA@mail.gmail.com>
Message-ID: <81eb1cd3-59f9-259d-3d90-0593919703ef@canonical.com>

Hi,

On 23/05/17 17:54, Guido van Rossum wrote:
> I think the I inevitable conclusion is"thanks, but no thanks."

Can I ask why this the inevitable conclusion? The Python Foundation make 
packages for Windows and macOS, why not snaps for Linux?

> On May 23, 2017 8:05 AM, "Martin Wimpress" 
> <martin.wimpress at canonical.com <mailto:martin.wimpress at canonical.com>> 
> wrote:
> 
>     Hi,
> 
>     On 23/05/17 00:56, Matt Billenstein wrote:
> 
>         On Tue, May 16, 2017 at 11:31:42AM +0100, Martin Wimpress wrote:
> 
>             Is there someone here who'd be interested in doing the same
>             for Python?
> 
> 
>         Do snaps support Windows and/or MacOS?
> 
> 
>     You can't install snaps on Windows or macOS. Snaps are currently
>     targeting various Linux distros, you can create Linux snaps on
>     Windows using WSL and we're working on improving the tooling for
>     snap creation on macOS and Windows.
> 
>         m
> 
> 
>     -- 
>     --
>     Regards, Martin.
>     _______________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     https://mail.python.org/mailman/listinfo/python-dev
>     <https://mail.python.org/mailman/listinfo/python-dev>
>     Unsubscribe:
>     https://mail.python.org/mailman/options/python-dev/guido%40python.org <https://mail.python.org/mailman/options/python-dev/guido%40python.org>
> 

-- 
--
Regards, Martin.

From victor.stinner at gmail.com  Thu May 25 06:47:42 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 25 May 2017 12:47:42 +0200
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
Message-ID: <CAMpsgwaXfRuxZMjRKN0q+4KfA_GJq52-VLfnn=0nTodS05B2Dw@mail.gmail.com>

Hi Ben,

I am not convinced that combining operations will have a significant impact
in term of performance. Mark Shanon implemented that in his HotPy project.

I proposed a RETURN_NONE opcode to combine LOAD_CONST with RETURN_VALUE.
The issue was rejected because I failed to show any speedup.

https://bugs.python.org/issue28800

I would be interested to restart/finish my registervm project to use
register-based bytecode. It allows to implement more optmisations and
reduce the number of instructions. In my experience, less instructions =
faster code.

http://faster-cpython.readthedocs.io/registervm.html

Mark's bytecode uses registers but also a stack.

Victor

Le 24 mai 2017 8:09 PM, "Ben Hoyt" <benhoyt at gmail.com> a ?crit :

> Hi folks,
>
> I was looking at some `dis` output today, and I was wondering if anyone
> has investigated optimizing Python (slightly) by adding special-case
> bytecodes for common expressions or statements involving constants?
>
> For example, I (and, based on a quick grep of the stdlib, many others)
> write "x is None" and "x is not None" very often. Or "return True" or
> "return None" or "return 1" and things like that. These all expand into two
> bytecodes, which seems pretty non-optimal (LOAD_CONST + COMPARE_OP or
> LOAD_CONST + RETURN_VALUE). It seems we could get an easy speedup for these
> common cases by adding a peephole optimization and some new opcodes (maybe
> COMPARE_IS_SMALL_CONST and RETURN_SMALL_CONST for these cases).
>
> I'm not proposing to do this yet, as I'd need to benchmark to see how much
> of a gain (if any) it would amount to, but I'm just wondering if there's
> any previous work on this kind of thing. Or, if not, any other thoughts
> before I try it?
>
> Thanks,
> Ben
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> victor.stinner%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170525/b30edbc0/attachment.html>

From solipsis at pitrou.net  Thu May 25 07:24:00 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 25 May 2017 13:24:00 +0200
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
 <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
 <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>
 <20170524033403.GB16849@k3>
 <CAMpsgwbj+yeptKx58FTFhEXkoSyMFiFveGus=cGrb10eZT108g@mail.gmail.com>
Message-ID: <20170525132400.70a051fd@fsol>

On Tue, 23 May 2017 23:09:31 -0500
Victor Stinner <victor.stinner at gmail.com> wrote:
> Le 23 mai 2017 20:43, "David Wilson" <dw+python-dev at hmmz.org> a ?crit :
> In which case, what is to prevent Requests from just depending on
> 
> pyOpenSSL as usual?
> 
> 
> From what I heard, pyOpenSSL development is slowing down, so I'm not sure
> that it's really safe and future-proof (TLS 1.3 anyone?).

So what? Python 2.7 isn't future-proof either...

> I'm still writing 2.7 code every day and would love to see it live a
> little longer, but accepting every feature request seems the wrong way
> to go - and MemoryBIO is a hard sell as a security enhancement, it's new
> functionality.

Agreed with this.

> You are true that they are new features. I disagree on the "accepting every
> feature" part: we are talking about two classes and it's restricted to
> security.

The new TLS API wouldn't significantly improve security.  It's just a
different API.

> I also understood that getting access to system CA allows admins to
> register their company CA and so avoid that users ignore the TLS warning
> (unknown CA).

System admins can add the company CA at the system level in the
system's CA cert store, they have no need for a Python API.  Actually,
they certainly don't want to modify every Python application to add a
company CA.

Regards

Antoine.



From solipsis at pitrou.net  Thu May 25 07:26:20 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Thu, 25 May 2017 13:26:20 +0200
Subject: [Python-Dev] PEP 544: Protocols - second round
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
Message-ID: <20170525132620.7a80070a@fsol>

On Wed, 24 May 2017 23:31:47 +0200
Ivan Levkivskyi <levkivskyi at gmail.com> wrote:

> Hi all,
> 
> After collecting suggestions in the previous discussion on python-dev
> https://mail.python.org/pipermail/python-dev/2017-March/thread.html#147629
> and playing with implementation, here is an updated version of PEP 544.
> 
> --
> Ivan
> 
> 
> A link for those who don't like reading long e-mails:
> https://www.python.org/dev/peps/pep-0544/
> 
> =========================
> 
> PEP: 544
> Title: Protocols

Can you give this PEP a more explicit title? "Protocols" sound like
network protocols to me.

Regards

Antoine.



From victor.stinner at gmail.com  Thu May 25 09:03:59 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Thu, 25 May 2017 15:03:59 +0200
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CAMpsgwZ_kf7k45=T44H07PqP1D4oXUFusc=sCvQirRzdpUZL5w@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
 <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
 <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>
 <20170524033403.GB16849@k3>
 <CAMpsgwbj+yeptKx58FTFhEXkoSyMFiFveGus=cGrb10eZT108g@mail.gmail.com>
 <20170525132400.70a051fd@fsol>
 <CAMpsgwa_VGgrW2enEJpTbuWkmU9hj+w=VFtBoCAFakiBy1_ePw@mail.gmail.com>
 <CAMpsgwZ_kf7k45=T44H07PqP1D4oXUFusc=sCvQirRzdpUZL5w@mail.gmail.com>
Message-ID: <CAMpsgwZHreaPKrJrn6-cf3yh9uDdrdkwDGAjMzeT3_EmEUYauA@mail.gmail.com>

Le 25 mai 2017 1:26 PM, "Antoine Pitrou" <solipsis at pitrou.net> a ?crit :

System admins can add the company CA at the system level in the
system's CA cert store, they have no need for a Python API.


If I understood correctly, since the Python ssl module is unable to load
system CAs (at least on Python 2.7) on Windows and macOS, bundled CAs are
needed (like certfi package).

Actually,
they certainly don't want to modify every Python application to add a
company CA.


Yes.

Hum, maybe a short PEP is needed for Python 2.7 to clarify the indirect
advantages of the backport.

Victor
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170525/a7233139/attachment.html>

From ncoghlan at gmail.com  Thu May 25 09:22:39 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Thu, 25 May 2017 23:22:39 +1000
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <20170525132400.70a051fd@fsol>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
 <69BAD4EE-D1CF-4D37-B999-E14A44E91DEF@lukasa.co.uk>
 <CAMpsgwY7FNmC1X+bsuOp4yMpEW5ppFe6m72rOGjJVGUnCu8QmA@mail.gmail.com>
 <380D9ADF-ED94-45E1-8CEB-DFD097856072@lukasa.co.uk>
 <20170524033403.GB16849@k3>
 <CAMpsgwbj+yeptKx58FTFhEXkoSyMFiFveGus=cGrb10eZT108g@mail.gmail.com>
 <20170525132400.70a051fd@fsol>
Message-ID: <CADiSq7f838CM0YXn=vHjaZJpM_o-Zr37RX=GWdyG9e+ewuBi2Q@mail.gmail.com>

On 25 May 2017 at 21:24, Antoine Pitrou <solipsis at pitrou.net> wrote:
> The new TLS API wouldn't significantly improve security.  It's just a
> different API.

It isn't just a different API. It's an API with *backends for the
native TLS implementations on WIndows and Mac OS X*.

This means that instead of attempting to extract system certs and
inject them into a bundled copy of OpenSSL, it becomes viable to just
use the operating system provided security services and behave the
same was as any other native application (at least as far as secure
network connections are concerned).

Ideally, it would be good to get to a point where we can stop bundling
OpenSSL entirely on Windows and Mac OS X, such that OpenSSL CVEs stop
automatically turning into CPython CVEs, and we can instead delegate
the task of prompt network security stack updates to the OS provider,
the same way we do for non-Apple *nix systems.

Even if it isn't feasible to get Python 2.7 to that happy state by
2020, we may at least be able to get to a point where most routine TLS
connections from Python 2.7 applications are using an OS provided TLS
implementation, so their network security is less dependent on
receiving updates to the Python level components of their stack.

>> I also understood that getting access to system CA allows admins to
>> register their company CA and so avoid that users ignore the TLS warning
>> (unknown CA).
>
> System admins can add the company CA at the system level in the
> system's CA cert store, they have no need for a Python API.  Actually,
> they certainly don't want to modify every Python application to add a
> company CA.

The state of affairs that you're citing as undesirable is pretty much
the way things *currently* work. Most Python applications still won't
see system level certificates on Windows and Mac OS X, and they often
won't see them even on Linux and *BSD systems (unless they're running
directly in the system Python and using the system-provided requests
module (which gets patched to use the system certs rather than the
default Mozilla bundle).

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu May 25 10:00:21 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 May 2017 00:00:21 +1000
Subject: [Python-Dev] Snap Python for simple distribution across
 multiple Linux distros
In-Reply-To: <81eb1cd3-59f9-259d-3d90-0593919703ef@canonical.com>
References: <CAKn8M0xb7PehJGP7P1qHsejKWbSWwpvFyH+Dc=CH5SR_wt1Yyg@mail.gmail.com>
 <0101015c32976a82-02f59fbb-9c76-4c8d-a338-4a2ea289f202-000000@us-west-2.amazonses.com>
 <65856131-5d27-c344-c0f3-6e7e99082454@canonical.com>
 <CAP7+vJLE4T-cJnmjGTHOnVxL7L5VEtMPvpBP3Wb7xsL1vpnfAA@mail.gmail.com>
 <81eb1cd3-59f9-259d-3d90-0593919703ef@canonical.com>
Message-ID: <CADiSq7fmYzBvLiLF94ntErxeN=J_Dre2HgJbtg4ctph6SxfoMw@mail.gmail.com>

On 25 May 2017 at 20:32, Martin Wimpress <martin.wimpress at canonical.com> wrote:
> Hi,
>
> On 23/05/17 17:54, Guido van Rossum wrote:
>>
>> I think the I inevitable conclusion is"thanks, but no thanks."
>
>
> Can I ask why this the inevitable conclusion? The Python Foundation make
> packages for Windows and macOS, why not snaps for Linux?

Mainly because there's no real pay-off to CPython as a project in
lowering barriers to adoption for end users: if someone is running
Linux, they're almost always going to have ready access to completely
usable pre-built Python binaries through their system package manager.

That means that if we were to start publishing our own docker/OCI
images, or our own snaps, or our own FlatPak runtime environment, we'd
be incurring additional ongoing effort without a comparable increase
in the audience we're able to effectively reach.

It also relates to the fact that when it comes to the interminable
packaging format debates in the Linux world, the typical pattern has
been for groups and organisations aiming to promote the use of a
particular package format to use the availability of Python to lower
the barriers to adoption for their particular offering, rather than
the other way around. Some examples:

* Debian takes care of providing deb packages and docker images
* Fedora provides RPMs and docker images
* OpenSUSE provides RPMs (and maybe docker images?)
* the Nix community provide nix packages
* Continuum Analytics provide conda packages
* Heroku take care of providing build packs
* Red Hat provides the RHEL/CentOS and Software Collections RPMs and
docker images
* Docker provide Alpine Linux based docker images
* ActiveState and Enthought provide binaries in a suitable format for
their users
* FreeBSD/NetBSD/OpenBSD also provide their own binaries

With snaps vs docker/OCI and snaps vs FlatPak vs AppImage emerging as
new variations of the longstanding "deb vs RPM vs something else"
arguments, "Here's a source tarball, y'all have fun now" remains the
most sensible publication approach for relatively low level operating
system components like CPython, while the `manylinux` ABI definition
provides "usefully broad" compatibility of pre-built wheel files
across a range of Linux distributions.

Cheers,
Nick.

P.S. Full disclosure: I do work for Red Hat, but I'd still be opposed
to the idea even if the suggestion was to publish our own RPMs or
docker/OCI images

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Thu May 25 10:19:44 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Fri, 26 May 2017 00:19:44 +1000
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <20170525132620.7a80070a@fsol>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <20170525132620.7a80070a@fsol>
Message-ID: <CADiSq7f2oNe8fhKxAj3EAz2zDhp+LVKiOSQqvp8tYcWdbrJivA@mail.gmail.com>

On 25 May 2017 at 21:26, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Wed, 24 May 2017 23:31:47 +0200
> Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
>
>> Hi all,
>>
>> After collecting suggestions in the previous discussion on python-dev
>> https://mail.python.org/pipermail/python-dev/2017-March/thread.html#147629
>> and playing with implementation, here is an updated version of PEP 544.
>>
>> --
>> Ivan
>>
>>
>> A link for those who don't like reading long e-mails:
>> https://www.python.org/dev/peps/pep-0544/
>>
>> =========================
>>
>> PEP: 544
>> Title: Protocols
>
> Can you give this PEP a more explicit title? "Protocols" sound like
> network protocols to me.

Especially given the existing use of the term in an asyncio context:
https://www.python.org/dev/peps/pep-3156/#transports-and-protocols

Given the abstract, I'd suggest "Structural Subtyping" as a suitable
title for the PEP.

That said, I think it's fine to use "protocol" throughout the rest of
the PEP as is currently the case - object protocols and network
protocols are clearly different things, it's just the bare word
"Protocols" appearing as a PEP title in the PEP index with no other
context that's potentially confusing.

I'm +1 on the general idea of the PEP, and only have one question
regarding the specifics. Given:

    import typing

    class MyContainer:
        def __len__(self) -> int:
            ...
        def close(self) -> None:
            ...

Would that be enough for a static typechecker to consider MyContainer
a structural subtype of both typing.Sized and SupportsClose from the
PEP, even though neither is imported explicitly into the module? I'm
assuming the answer is "Yes", but I didn't see it explicitly stated
anywhere.

Cheers,
Nick.



-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From benhoyt at gmail.com  Thu May 25 10:28:11 2017
From: benhoyt at gmail.com (Ben Hoyt)
Date: Thu, 25 May 2017 10:28:11 -0400
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <CAMpsgwaXfRuxZMjRKN0q+4KfA_GJq52-VLfnn=0nTodS05B2Dw@mail.gmail.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
 <CAMpsgwaXfRuxZMjRKN0q+4KfA_GJq52-VLfnn=0nTodS05B2Dw@mail.gmail.com>
Message-ID: <CAL9jXCGrg8SeH7Hcd12Rmo-04Lun2uTmznXJhCJpU2e7kYztnw@mail.gmail.com>

Thanks, Victor. That's very helpful. So RETURN_NONE (and probably
RETURN_SMALL_CONST) are not worth it, based on your empirical tests. Your
patch shows how (relatively) straight-forward it is to test out new opcodes.

I'm still optimistic about the value of COMPARE_IS_NONE and
COMPARE_IS_NOT_NONE, though. Mainly because I've done a quick expansion of
LOAD_CONST(None) + COMPARE_OP and it's quite a bit more code and many more
instructions than COMPARE_IS_NONE would be:

LOAD_CONST(None)
COMPARE_OP
    PyObject *value = ((PyTupleObject *)(consts))->ob_item[oparg];
    value->ob_refcnt++;
    *stack_pointer++ = value;
    FAST_DISPATCH();
    PyObject *right = *--stack_pointer;
    PyObject *left = stack_pointer[-1]

    // cmp_outcome(), presumably inlined
    int r = 0;
    switch (compare_oparg) {
    case PyCmp_IS:
        r = (left == right);
        break;
    case PyCmp_IS_NOT:
        r = (left != right);
        break;
    case ...
    }
    PyObject *res = r ? Py_True : Py_False;
    res->ob_refcnt++;

    if (--(left)->ob_refcnt == 0)
        _Py_Dealloc(left);
    if (--(right)->ob_refcnt == 0)
        _Py_Dealloc(right);
    stack_pointer[-1] = res;
    if (res == NULL)
        goto error;

    PREDICT(POP_JUMP_IF_FALSE);
    PREDICT(POP_JUMP_IF_TRUE);
    DISPATCH();


COMPARE_IS_NONE
    PyObject* left = stack_pointer[-1];  // TOP()
    PyObject* res = (left == Py_None) ? Py_True : Py_False;
    res->ob_refcnt++;
    if (--(left)->ob_refcnt == 0)
        _Py_Dealloc(left);
    stack_pointer[-1] = res;  // SET_TOP(res)
    PREDICT(POP_JUMP_IF_FALSE);
    PREDICT(POP_JUMP_IF_TRUE);
    DISPATCH();

You don't have to get the const arg, there are fewer increfs/decrefs, you
skip a pop, you don't have to test res==NULL (because it's Py_True or
Py_False, which are never NULL), and if there are separate COMPARE_IS_NONE
and COMPARE_IS_NOT_NONE you don't have to switch on the compare arg (though
I'm not sure if that part will be worth it).

For reference, based on a grep, " is None" occurs 2737 times in the CPython
source tree, and " is not None" 2010 times. And I know personally I often
use them in loops as well is at the start of functions (for mutable default
arg handling).

Still, the performance proof will be in the pudding! I might hack these two
opcodes together and test it at some point.

-Ben

On Thu, May 25, 2017 at 6:47 AM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> Hi Ben,
>
> I am not convinced that combining operations will have a significant
> impact in term of performance. Mark Shanon implemented that in his HotPy
> project.
>
> I proposed a RETURN_NONE opcode to combine LOAD_CONST with RETURN_VALUE.
> The issue was rejected because I failed to show any speedup.
>
> https://bugs.python.org/issue28800
>
> I would be interested to restart/finish my registervm project to use
> register-based bytecode. It allows to implement more optmisations and
> reduce the number of instructions. In my experience, less instructions =
> faster code.
>
> http://faster-cpython.readthedocs.io/registervm.html
>
> Mark's bytecode uses registers but also a stack.
>
> Victor
>
> Le 24 mai 2017 8:09 PM, "Ben Hoyt" <benhoyt at gmail.com> a ?crit :
>
>> Hi folks,
>>
>> I was looking at some `dis` output today, and I was wondering if anyone
>> has investigated optimizing Python (slightly) by adding special-case
>> bytecodes for common expressions or statements involving constants?
>>
>> For example, I (and, based on a quick grep of the stdlib, many others)
>> write "x is None" and "x is not None" very often. Or "return True" or
>> "return None" or "return 1" and things like that. These all expand into two
>> bytecodes, which seems pretty non-optimal (LOAD_CONST + COMPARE_OP or
>> LOAD_CONST + RETURN_VALUE). It seems we could get an easy speedup for these
>> common cases by adding a peephole optimization and some new opcodes (maybe
>> COMPARE_IS_SMALL_CONST and RETURN_SMALL_CONST for these cases).
>>
>> I'm not proposing to do this yet, as I'd need to benchmark to see how
>> much of a gain (if any) it would amount to, but I'm just wondering if
>> there's any previous work on this kind of thing. Or, if not, any other
>> thoughts before I try it?
>>
>> Thanks,
>> Ben
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.
>> stinner%40gmail.com
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170525/1553340e/attachment.html>

From jelle.zijlstra at gmail.com  Thu May 25 10:30:38 2017
From: jelle.zijlstra at gmail.com (Jelle Zijlstra)
Date: Thu, 25 May 2017 07:30:38 -0700
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CADiSq7f2oNe8fhKxAj3EAz2zDhp+LVKiOSQqvp8tYcWdbrJivA@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <20170525132620.7a80070a@fsol>
 <CADiSq7f2oNe8fhKxAj3EAz2zDhp+LVKiOSQqvp8tYcWdbrJivA@mail.gmail.com>
Message-ID: <CAFp3-p93t_s+HfqZ9AHFJ1Peec28HJrO8=uRH17F9rhoBnsBcg@mail.gmail.com>

2017-05-25 7:19 GMT-07:00 Nick Coghlan <ncoghlan at gmail.com>:

> On 25 May 2017 at 21:26, Antoine Pitrou <solipsis at pitrou.net> wrote:
> > On Wed, 24 May 2017 23:31:47 +0200
> > Ivan Levkivskyi <levkivskyi at gmail.com> wrote:
> >
> >> Hi all,
> >>
> >> After collecting suggestions in the previous discussion on python-dev
> >> https://mail.python.org/pipermail/python-dev/2017-
> March/thread.html#147629
> >> and playing with implementation, here is an updated version of PEP 544.
> >>
> >> --
> >> Ivan
> >>
> >>
> >> A link for those who don't like reading long e-mails:
> >> https://www.python.org/dev/peps/pep-0544/
> >>
> >> =========================
> >>
> >> PEP: 544
> >> Title: Protocols
> >
> > Can you give this PEP a more explicit title? "Protocols" sound like
> > network protocols to me.
>
> Especially given the existing use of the term in an asyncio context:
> https://www.python.org/dev/peps/pep-3156/#transports-and-protocols
>
> Given the abstract, I'd suggest "Structural Subtyping" as a suitable
> title for the PEP.
>
> That said, I think it's fine to use "protocol" throughout the rest of
> the PEP as is currently the case - object protocols and network
> protocols are clearly different things, it's just the bare word
> "Protocols" appearing as a PEP title in the PEP index with no other
> context that's potentially confusing.
>
> I'm +1 on the general idea of the PEP, and only have one question
> regarding the specifics. Given:
>
>     import typing
>
>     class MyContainer:
>         def __len__(self) -> int:
>             ...
>         def close(self) -> None:
>             ...
>
> Would that be enough for a static typechecker to consider MyContainer
> a structural subtype of both typing.Sized and SupportsClose from the
> PEP, even though neither is imported explicitly into the module? I'm
> assuming the answer is "Yes", but I didn't see it explicitly stated
> anywhere.
>
> Yes, that should be the case. Specifically, if you pass a MyContainer
object to a function whose argument is annotated as typing.Sized or
SupportsClose, a type checker should accept that call.


> Cheers,
> Nick.
>
>
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> jelle.zijlstra%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170525/452032d8/attachment.html>

From jake at lwn.net  Thu May 25 10:03:22 2017
From: jake at lwn.net (Jake Edge)
Date: Thu, 25 May 2017 08:03:22 -0600
Subject: [Python-Dev] 2017 Python Language Summit coverage
Message-ID: <20170525080322.2f8a8f3b@redtail>


Hola python-dev,

Thanks to Larry and Barry, I was able to sit in on the Python Language
Summit again this year.  The start of the coverage for that is now
available.

The starting point is the overview article, here:
https://lwn.net/Articles/723251/ (for non-subscribers, this link will
get around the paywall:
https://lwn.net/SubscriberLink/723251/2d2cbb3329dd87f8/ )  It will be
updated as I add more sessions over the next week or so.

LWN subscribers can see the content right away, but one week after they
are published in the weekly edition, they become freely available for
everyone.  Until then, though, feel free to share the SubscriberLinks I
am posting here.

Python 3.6.x, 3.7.0, and beyond: https://lwn.net/Articles/723252/ or
https://lwn.net/SubscriberLink/723252/c5bfea8533558bab/

New CPython workflow issues: https://lwn.net/Articles/723418/ or
https://lwn.net/SubscriberLink/723418/62487f893b324500/

The state of bugs.python.org: https://lwn.net/Articles/723513/ or
https://lwn.net/SubscriberLink/723513/7e4e2e626fc93606/

Progress on the Gilectomy: https://lwn.net/Articles/723514/ or
https://lwn.net/SubscriberLink/723514/70df37b92c746d62/

Hopefully I captured things reasonably well -- if you have corrections
or clarifications (or just comments :) , I would recommend posting them
as comments on the article.

I will post an update next week with the next round (with luck, all of
the rest of them).

enjoy!

jake

-- 
Jake Edge - LWN - jake at lwn.net - http://lwn.net

From cesare.di.mauro at gmail.com  Thu May 25 11:00:11 2017
From: cesare.di.mauro at gmail.com (Cesare Di Mauro)
Date: Thu, 25 May 2017 17:00:11 +0200
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <CAL9jXCGrg8SeH7Hcd12Rmo-04Lun2uTmznXJhCJpU2e7kYztnw@mail.gmail.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
 <CAMpsgwaXfRuxZMjRKN0q+4KfA_GJq52-VLfnn=0nTodS05B2Dw@mail.gmail.com>
 <CAL9jXCGrg8SeH7Hcd12Rmo-04Lun2uTmznXJhCJpU2e7kYztnw@mail.gmail.com>
Message-ID: <CAP7v7k6m5so8r5L8y5xOHVkmZiv5qRZAqKybFp815DDksAorkA@mail.gmail.com>

Hi Ben,

for what you're interested in, you might give a look at WPython 1.0 (
https://code.google.com/archive/p/wpython/downloads ) and 1.1 (
https://code.google.com/archive/p/wpython2/downloads ), but they cover a
lot of optimizations (as you can see from a brief look at the slides):
RETURN_CONST and fusing some opcodes for binary operations are only some of
them.
For this reason, it's also very difficult to micro-benchmark every single
change... :-/

Cheers,
Cesare

<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
Mail
priva di virus. www.avast.com
<https://www.avast.com/sig-email?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=webmail>
<#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>

2017-05-25 16:28 GMT+02:00 Ben Hoyt <benhoyt at gmail.com>:

> Thanks, Victor. That's very helpful. So RETURN_NONE (and probably
> RETURN_SMALL_CONST) are not worth it, based on your empirical tests. Your
> patch shows how (relatively) straight-forward it is to test out new opcodes.
>
> I'm still optimistic about the value of COMPARE_IS_NONE and
> COMPARE_IS_NOT_NONE, though. Mainly because I've done a quick expansion of
> LOAD_CONST(None) + COMPARE_OP and it's quite a bit more code and many more
> instructions than COMPARE_IS_NONE would be:
>
> LOAD_CONST(None)
> COMPARE_OP
>     PyObject *value = ((PyTupleObject *)(consts))->ob_item[oparg];
>     value->ob_refcnt++;
>     *stack_pointer++ = value;
>     FAST_DISPATCH();
>     PyObject *right = *--stack_pointer;
>     PyObject *left = stack_pointer[-1]
>
>     // cmp_outcome(), presumably inlined
>     int r = 0;
>     switch (compare_oparg) {
>     case PyCmp_IS:
>         r = (left == right);
>         break;
>     case PyCmp_IS_NOT:
>         r = (left != right);
>         break;
>     case ...
>     }
>     PyObject *res = r ? Py_True : Py_False;
>     res->ob_refcnt++;
>
>     if (--(left)->ob_refcnt == 0)
>         _Py_Dealloc(left);
>     if (--(right)->ob_refcnt == 0)
>         _Py_Dealloc(right);
>     stack_pointer[-1] = res;
>     if (res == NULL)
>         goto error;
>
>     PREDICT(POP_JUMP_IF_FALSE);
>     PREDICT(POP_JUMP_IF_TRUE);
>     DISPATCH();
>
>
> COMPARE_IS_NONE
>     PyObject* left = stack_pointer[-1];  // TOP()
>     PyObject* res = (left == Py_None) ? Py_True : Py_False;
>     res->ob_refcnt++;
>     if (--(left)->ob_refcnt == 0)
>         _Py_Dealloc(left);
>     stack_pointer[-1] = res;  // SET_TOP(res)
>     PREDICT(POP_JUMP_IF_FALSE);
>     PREDICT(POP_JUMP_IF_TRUE);
>     DISPATCH();
>
> You don't have to get the const arg, there are fewer increfs/decrefs, you
> skip a pop, you don't have to test res==NULL (because it's Py_True or
> Py_False, which are never NULL), and if there are separate COMPARE_IS_NONE
> and COMPARE_IS_NOT_NONE you don't have to switch on the compare arg (though
> I'm not sure if that part will be worth it).
>
> For reference, based on a grep, " is None" occurs 2737 times in the
> CPython source tree, and " is not None" 2010 times. And I know personally I
> often use them in loops as well is at the start of functions (for mutable
> default arg handling).
>
> Still, the performance proof will be in the pudding! I might hack these
> two opcodes together and test it at some point.
>
> -Ben
>
> On Thu, May 25, 2017 at 6:47 AM, Victor Stinner <victor.stinner at gmail.com>
> wrote:
>
>> Hi Ben,
>>
>> I am not convinced that combining operations will have a significant
>> impact in term of performance. Mark Shanon implemented that in his HotPy
>> project.
>>
>> I proposed a RETURN_NONE opcode to combine LOAD_CONST with RETURN_VALUE.
>> The issue was rejected because I failed to show any speedup.
>>
>> https://bugs.python.org/issue28800
>>
>> I would be interested to restart/finish my registervm project to use
>> register-based bytecode. It allows to implement more optmisations and
>> reduce the number of instructions. In my experience, less instructions =
>> faster code.
>>
>> http://faster-cpython.readthedocs.io/registervm.html
>>
>> Mark's bytecode uses registers but also a stack.
>>
>> Victor
>>
>> Le 24 mai 2017 8:09 PM, "Ben Hoyt" <benhoyt at gmail.com> a ?crit :
>>
>>> Hi folks,
>>>
>>> I was looking at some `dis` output today, and I was wondering if anyone
>>> has investigated optimizing Python (slightly) by adding special-case
>>> bytecodes for common expressions or statements involving constants?
>>>
>>> For example, I (and, based on a quick grep of the stdlib, many others)
>>> write "x is None" and "x is not None" very often. Or "return True" or
>>> "return None" or "return 1" and things like that. These all expand into two
>>> bytecodes, which seems pretty non-optimal (LOAD_CONST + COMPARE_OP or
>>> LOAD_CONST + RETURN_VALUE). It seems we could get an easy speedup for these
>>> common cases by adding a peephole optimization and some new opcodes (maybe
>>> COMPARE_IS_SMALL_CONST and RETURN_SMALL_CONST for these cases).
>>>
>>> I'm not proposing to do this yet, as I'd need to benchmark to see how
>>> much of a gain (if any) it would amount to, but I'm just wondering if
>>> there's any previous work on this kind of thing. Or, if not, any other
>>> thoughts before I try it?
>>>
>>> Thanks,
>>> Ben
>>>
>>>
>>> _______________________________________________
>>> Python-Dev mailing list
>>> Python-Dev at python.org
>>> https://mail.python.org/mailman/listinfo/python-dev
>>> Unsubscribe: https://mail.python.org/mailma
>>> n/options/python-dev/victor.stinner%40gmail.com
>>>
>>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> cesare.di.mauro%40gmail.com
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170525/6547c0ed/attachment.html>

From ericsnowcurrently at gmail.com  Thu May 25 12:29:33 2017
From: ericsnowcurrently at gmail.com (Eric Snow)
Date: Thu, 25 May 2017 09:29:33 -0700
Subject: [Python-Dev] 2017 Python Language Summit coverage
In-Reply-To: <20170525080322.2f8a8f3b@redtail>
References: <20170525080322.2f8a8f3b@redtail>
Message-ID: <CALFfu7ALvRwfNsdRVC6P_bsNvmXP-49Z-0ncjNL4KZNNSXEQTQ@mail.gmail.com>

On Thu, May 25, 2017 at 7:03 AM, Jake Edge <jake at lwn.net> wrote:
>
> Hola python-dev,
>
> Thanks to Larry and Barry, I was able to sit in on the Python Language
> Summit again this year.  The start of the coverage for that is now
> available.
>
> [snipped]
>
> Hopefully I captured things reasonably well -- if you have corrections
> or clarifications (or just comments :) , I would recommend posting them
> as comments on the article.
>
> I will post an update next week with the next round (with luck, all of
> the rest of them).

Thanks again, Jake, for taking the time to do this.  I haven't read
through the articles yet, but those from past years have been an
excellent view into the what's going on.

-eric

From barry at python.org  Thu May 25 13:32:31 2017
From: barry at python.org (Barry Warsaw)
Date: Thu, 25 May 2017 10:32:31 -0700
Subject: [Python-Dev] 2017 Python Language Summit coverage
In-Reply-To: <20170525080322.2f8a8f3b@redtail>
References: <20170525080322.2f8a8f3b@redtail>
Message-ID: <20170525103231.10708049@presto>

On May 25, 2017, at 08:03 AM, Jake Edge wrote:

>Thanks to Larry and Barry, I was able to sit in on the Python Language
>Summit again this year.  The start of the coverage for that is now
>available.

Thanks so much for your always excellent reporting Jake.  It's unfortunate
that we can't invite everyone who wants to attend the summit, so your articles
are the best way to improve our transparency, reduce the mystery, and let the
rest of the community see what happens when you lock <wink> a bunch of core
devs in a room for a day.

be-fez-zled-ly y'rs,
-Barry

From mark at hotpy.org  Thu May 25 13:49:33 2017
From: mark at hotpy.org (Mark Shannon)
Date: Thu, 25 May 2017 10:49:33 -0700
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
Message-ID: <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>

On 24/05/17 14:31, Ivan Levkivskyi wrote:
> Hi all,
>
> After collecting suggestions in the previous discussion on python-dev
> https://mail.python.org/pipermail/python-dev/2017-March/thread.html#147629
> and playing with implementation, here is an updated version of PEP 544.
>
> --
> Ivan

I really like this PEP in general. I think this brings the type system 
for type-hints closer to Python semantics.

But there are a few points I disagree with.
I don't think Protocol types should be tied to ABCs. It just makes 
things more complex with no obvious benefit.
I also think all references to 'isinstance' and 'issubclass' should be 
removed. Type-hints should not have runtime semantics, beyond those that 
they have as classes. In fact, there is no need for protocol types to be 
classes at all.

Cheers,
Mark.


From mark at hotpy.org  Thu May 25 14:03:51 2017
From: mark at hotpy.org (Mark Shannon)
Date: Thu, 25 May 2017 11:03:51 -0700
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <CAMpsgwaXfRuxZMjRKN0q+4KfA_GJq52-VLfnn=0nTodS05B2Dw@mail.gmail.com>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
 <CAMpsgwaXfRuxZMjRKN0q+4KfA_GJq52-VLfnn=0nTodS05B2Dw@mail.gmail.com>
Message-ID: <85433e92-5985-6c5c-a129-8d82a9c0823f@hotpy.org>



On 25/05/17 03:47, Victor Stinner wrote:
> Hi Ben,
>
> I am not convinced that combining operations will have a significant
> impact in term of performance. Mark Shanon implemented that in his HotPy
> project.

I don't think that I did ;)
HotPy implemented a trace-based optimising interpreter, that preformed 
type specialisation and deferred evaluation of objects, with a "dumb" 
compiler that applied standard optimisations to the resulting traces.
It was technologically similar to PyPy (at the time), but designed for 
ease of engineering,

>
> I proposed a RETURN_NONE opcode to combine LOAD_CONST with RETURN_VALUE.
> The issue was rejected because I failed to show any speedup.
>
> https://bugs.python.org/issue28800
>
> I would be interested to restart/finish my registervm project to use
> register-based bytecode. It allows to implement more optmisations and
> reduce the number of instructions. In my experience, less instructions =
> faster code.
>
> http://faster-cpython.readthedocs.io/registervm.html
>
> Mark's bytecode uses registers but also a stack.
The "registers" were used to hold values across calls and the like when 
optimising traces, but I wouldn't use that design if I were to do it again.

>
> Victor
>
> Le 24 mai 2017 8:09 PM, "Ben Hoyt" <benhoyt at gmail.com
> <mailto:benhoyt at gmail.com>> a ?crit :
>
>     Hi folks,
>
>     I was looking at some `dis` output today, and I was wondering if
>     anyone has investigated optimizing Python (slightly) by adding
>     special-case bytecodes for common expressions or statements
>     involving constants?
>
>     For example, I (and, based on a quick grep of the stdlib, many
>     others) write "x is None" and "x is not None" very often. Or "return
>     True" or "return None" or "return 1" and things like that. These all
>     expand into two bytecodes, which seems pretty non-optimal
>     (LOAD_CONST + COMPARE_OP or LOAD_CONST + RETURN_VALUE). It seems we
>     could get an easy speedup for these common cases by adding a
>     peephole optimization and some new opcodes (maybe
>     COMPARE_IS_SMALL_CONST and RETURN_SMALL_CONST for these cases).
>
>     I'm not proposing to do this yet, as I'd need to benchmark to see
>     how much of a gain (if any) it would amount to, but I'm just
>     wondering if there's any previous work on this kind of thing. Or, if
>     not, any other thoughts before I try it?
>
>     Thanks,
>     Ben
>
>
>     _______________________________________________
>     Python-Dev mailing list
>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>     https://mail.python.org/mailman/listinfo/python-dev
>     <https://mail.python.org/mailman/listinfo/python-dev>
>     Unsubscribe:
>     https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>     <https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com>
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org
>

From greg at krypto.org  Thu May 25 17:16:38 2017
From: greg at krypto.org (Gregory P. Smith)
Date: Thu, 25 May 2017 21:16:38 +0000
Subject: [Python-Dev] 2017 Python Language Summit coverage
In-Reply-To: <20170525103231.10708049@presto>
References: <20170525080322.2f8a8f3b@redtail> <20170525103231.10708049@presto>
Message-ID: <CAGE7PNJVhr2V7=8S2GwDdykg5T0WmPe8zBt2BDNuWhF59e_q+A@mail.gmail.com>

Thanks Jake!

I'm laughing at not being in the photo. That must've happened soon after I
left the room as I disappeared a tad early.  I'll pretend I'm hiding behind
Brett or Thomas. ;)

-gps


On Thu, May 25, 2017 at 10:34 AM Barry Warsaw <barry at python.org> wrote:

> On May 25, 2017, at 08:03 AM, Jake Edge wrote:
>
> >Thanks to Larry and Barry, I was able to sit in on the Python Language
> >Summit again this year.  The start of the coverage for that is now
> >available.
>
> Thanks so much for your always excellent reporting Jake.  It's unfortunate
> that we can't invite everyone who wants to attend the summit, so your
> articles
> are the best way to improve our transparency, reduce the mystery, and let
> the
> rest of the community see what happens when you lock <wink> a bunch of core
> devs in a room for a day.
>
> be-fez-zled-ly y'rs,
> -Barry
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170525/e9d430b7/attachment.html>

From victor.stinner at gmail.com  Thu May 25 19:11:13 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Fri, 26 May 2017 01:11:13 +0200
Subject: [Python-Dev] Micro-optimizations by adding special-case
 bytecodes?
In-Reply-To: <85433e92-5985-6c5c-a129-8d82a9c0823f@hotpy.org>
References: <CAL9jXCEFZW0K=fnaMRi9NKwK7VyNWx9sZ=wHj_UZMqwuCcvwiQ@mail.gmail.com>
 <CAMpsgwaXfRuxZMjRKN0q+4KfA_GJq52-VLfnn=0nTodS05B2Dw@mail.gmail.com>
 <85433e92-5985-6c5c-a129-8d82a9c0823f@hotpy.org>
Message-ID: <CAMpsgwY_-puzW8NZgSy6z0gE+ng8_CBLnBR_DRt0_OryviWK6w@mail.gmail.com>

After reading Cesare Di Mauro's email, I realized that I was thinking to
WPython in fact...

Victor

Le 25 mai 2017 8:11 PM, "Mark Shannon" <mark at hotpy.org> a ?crit :

>
>
> On 25/05/17 03:47, Victor Stinner wrote:
>
>> Hi Ben,
>>
>> I am not convinced that combining operations will have a significant
>> impact in term of performance. Mark Shanon implemented that in his HotPy
>> project.
>>
>
> I don't think that I did ;)
> HotPy implemented a trace-based optimising interpreter, that preformed
> type specialisation and deferred evaluation of objects, with a "dumb"
> compiler that applied standard optimisations to the resulting traces.
> It was technologically similar to PyPy (at the time), but designed for
> ease of engineering,
>
>
>> I proposed a RETURN_NONE opcode to combine LOAD_CONST with RETURN_VALUE.
>> The issue was rejected because I failed to show any speedup.
>>
>> https://bugs.python.org/issue28800
>>
>> I would be interested to restart/finish my registervm project to use
>> register-based bytecode. It allows to implement more optmisations and
>> reduce the number of instructions. In my experience, less instructions =
>> faster code.
>>
>> http://faster-cpython.readthedocs.io/registervm.html
>>
>> Mark's bytecode uses registers but also a stack.
>>
> The "registers" were used to hold values across calls and the like when
> optimising traces, but I wouldn't use that design if I were to do it again.
>
>
>> Victor
>>
>> Le 24 mai 2017 8:09 PM, "Ben Hoyt" <benhoyt at gmail.com
>> <mailto:benhoyt at gmail.com>> a ?crit :
>>
>>     Hi folks,
>>
>>     I was looking at some `dis` output today, and I was wondering if
>>     anyone has investigated optimizing Python (slightly) by adding
>>     special-case bytecodes for common expressions or statements
>>     involving constants?
>>
>>     For example, I (and, based on a quick grep of the stdlib, many
>>     others) write "x is None" and "x is not None" very often. Or "return
>>     True" or "return None" or "return 1" and things like that. These all
>>     expand into two bytecodes, which seems pretty non-optimal
>>     (LOAD_CONST + COMPARE_OP or LOAD_CONST + RETURN_VALUE). It seems we
>>     could get an easy speedup for these common cases by adding a
>>     peephole optimization and some new opcodes (maybe
>>     COMPARE_IS_SMALL_CONST and RETURN_SMALL_CONST for these cases).
>>
>>     I'm not proposing to do this yet, as I'd need to benchmark to see
>>     how much of a gain (if any) it would amount to, but I'm just
>>     wondering if there's any previous work on this kind of thing. Or, if
>>     not, any other thoughts before I try it?
>>
>>     Thanks,
>>     Ben
>>
>>
>>     _______________________________________________
>>     Python-Dev mailing list
>>     Python-Dev at python.org <mailto:Python-Dev at python.org>
>>     https://mail.python.org/mailman/listinfo/python-dev
>>     <https://mail.python.org/mailman/listinfo/python-dev>
>>     Unsubscribe:
>>     https://mail.python.org/mailman/options/python-dev/victor.
>> stinner%40gmail.com
>>     <https://mail.python.org/mailman/options/python-dev/victor.
>> stinner%40gmail.com>
>>
>>
>>
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/mark%
>> 40hotpy.org
>>
>> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.
> stinner%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170526/6c6eebbb/attachment.html>

From python at mrabarnett.plus.com  Thu May 25 20:56:16 2017
From: python at mrabarnett.plus.com (MRAB)
Date: Fri, 26 May 2017 01:56:16 +0100
Subject: [Python-Dev] 2017 Python Language Summit coverage
In-Reply-To: <CAGE7PNJVhr2V7=8S2GwDdykg5T0WmPe8zBt2BDNuWhF59e_q+A@mail.gmail.com>
References: <20170525080322.2f8a8f3b@redtail> <20170525103231.10708049@presto>
 <CAGE7PNJVhr2V7=8S2GwDdykg5T0WmPe8zBt2BDNuWhF59e_q+A@mail.gmail.com>
Message-ID: <67fa4dd2-c88f-b3dc-9122-a1703bc47187@mrabarnett.plus.com>

On 2017-05-25 22:16, Gregory P. Smith wrote:
> Thanks Jake!
> 
> I'm laughing at not being in the photo. That must've happened soon after 
> I left the room as I disappeared a tad early.  I'll pretend I'm hiding 
> behind Brett or Thomas. ;)
> 
 > -gps
 >
Or someone could Photoshop you in. :-)

> 
> On Thu, May 25, 2017 at 10:34 AM Barry Warsaw <barry at python.org 
> <mailto:barry at python.org>> wrote:
> 
>     On May 25, 2017, at 08:03 AM, Jake Edge wrote:
> 
>      >Thanks to Larry and Barry, I was able to sit in on the Python Language
>      >Summit again this year.  The start of the coverage for that is now
>      >available.
> 
>     Thanks so much for your always excellent reporting Jake.  It's
>     unfortunate
>     that we can't invite everyone who wants to attend the summit, so
>     your articles
>     are the best way to improve our transparency, reduce the mystery,
>     and let the
>     rest of the community see what happens when you lock <wink> a bunch
>     of core
>     devs in a room for a day.
> 
>     be-fez-zled-ly y'rs,
>     -Barry
> 

From status at bugs.python.org  Fri May 26 12:09:15 2017
From: status at bugs.python.org (Python tracker)
Date: Fri, 26 May 2017 18:09:15 +0200 (CEST)
Subject: [Python-Dev] Summary of Python tracker Issues
Message-ID: <20170526160915.320E656A46@psf.upfronthosting.co.za>


ACTIVITY SUMMARY (2017-05-19 - 2017-05-26)
Python tracker at http://bugs.python.org/

To view or respond to any of the issues listed below, click on the issue.
Do NOT respond to this message.

Issues counts and deltas:
  open    5999 (+31)
  closed 36241 (+50)
  total  42240 (+81)

Open issues with patches: 2372 


Issues opened (62)
==================

#21074: Too aggressive constant folding
http://bugs.python.org/issue21074  reopened by haypo

#30405: build.bat: register binaries for py launcher
http://bugs.python.org/issue30405  opened by terry.reedy

#30406: async and await should be keywords in 3.7
http://bugs.python.org/issue30406  opened by Jelle Zijlstra

#30410: Documentation for sys.stdout encoding does not reflect the new
http://bugs.python.org/issue30410  opened by paul.moore

#30411: git doesn't support "-C" args under 1.8.5 occurs in configure.
http://bugs.python.org/issue30411  opened by Yuan Liu

#30412: mailbox : add option to prevent platform EOL conversion
http://bugs.python.org/issue30412  opened by adder

#30413: Add fnmatch.filterfalse function
http://bugs.python.org/issue30413  opened by steven.daprano

#30416: constant folding opens compiler to quadratic time hashing
http://bugs.python.org/issue30416  opened by dalke

#30417: [buildbot] Disable the cpu resources on slow buildbots
http://bugs.python.org/issue30417  opened by haypo

#30418: test_communicate_epipe() of test_subprocess fails randomly on 
http://bugs.python.org/issue30418  opened by haypo

#30419: Bdb: Update doc page
http://bugs.python.org/issue30419  opened by csabella

#30420: Clarify kwarg handing for subprocess convenience APIs
http://bugs.python.org/issue30420  opened by ncoghlan

#30421: argparse: relative include of config files
http://bugs.python.org/issue30421  opened by roschi

#30422: Add roadmap.txt section to idlelib
http://bugs.python.org/issue30422  opened by terry.reedy

#30424: make pydoc-topics fails
http://bugs.python.org/issue30424  opened by Jelle Zijlstra

#30425: Python 3.6.1 (32-bit)_20170522020736.log
http://bugs.python.org/issue30425  opened by Chakrit Rakhuang

#30427: isinstance checks in os.path.normcase redundant with os.fspath
http://bugs.python.org/issue30427  opened by wolma

#30428: mailbox.MMDF wrongly adds From_ header to file
http://bugs.python.org/issue30428  opened by ralph.corderoy

#30429: bdb and pdb: Add watchpoint function
http://bugs.python.org/issue30429  opened by louielu

#30430: Cannot write to a tun interface file
http://bugs.python.org/issue30430  opened by R??my L??one

#30431: input function truncates prompt by NULL byte
http://bugs.python.org/issue30431  opened by ?????????? ??????????

#30433: Devguide lacks instructions for building docs
http://bugs.python.org/issue30433  opened by cjrh

#30434: multiprocessing AuthenticationError "digest sent was rejected"
http://bugs.python.org/issue30434  opened by Soon Jeffrey

#30435: Documentation either unclear or incorrect on comparisons betwe
http://bugs.python.org/issue30435  opened by ipatrol

#30436: importlib.find_spec raises AttributeError when parent is not a
http://bugs.python.org/issue30436  opened by tkhyn

#30437: SSL_shutdown can return meaningless SSL_ERROR_SYSCALL
http://bugs.python.org/issue30437  opened by njs

#30438: tarfile would fail to extract tarballs with files under R/O di
http://bugs.python.org/issue30438  opened by Yaroslav.Halchenko

#30439: Expose the subinterpreters C-API in the stdlib.
http://bugs.python.org/issue30439  opened by eric.snow

#30440: document peephole optimizer effects
http://bugs.python.org/issue30440  opened by dalke

#30441: os.environ raises RuntimeError: dictionary changed size during
http://bugs.python.org/issue30441  opened by osantana

#30442: Skip test_xml_etree under coverage
http://bugs.python.org/issue30442  opened by kushal.das

#30444: Add ability to change "-- more --" text in pager module
http://bugs.python.org/issue30444  opened by Gautam krishna.R

#30446: Embedded 3.6.1 distribution cannot find _socket module
http://bugs.python.org/issue30446  opened by ThomasS

#30448: test_subprocess creates a core dump on FreeBSD
http://bugs.python.org/issue30448  opened by haypo

#30449: Improve __slots__ datamodel documentation
http://bugs.python.org/issue30449  opened by Aaron Hall

#30450: Pull Windows dependencies from GitHub rather than svn.python.o
http://bugs.python.org/issue30450  opened by zach.ware

#30451: parse windows path error in webbrowser.get() and lead to webbr
http://bugs.python.org/issue30451  opened by chainly

#30454: Python module cx_Oracle ld installation issue on Solaris11U3 S
http://bugs.python.org/issue30454  opened by sameernatekar

#30455: Generate C code from token.py and not vice versa
http://bugs.python.org/issue30455  opened by serhiy.storchaka

#30456: 2to3 docs: example of fix for duplicates in second argument of
http://bugs.python.org/issue30456  opened by Eli_B

#30457: Allow retrieve the number of waiters pending for most of the a
http://bugs.python.org/issue30457  opened by pfreixes

#30458: CRLF Injection in httplib
http://bugs.python.org/issue30458  opened by orange

#30459: PyList_SET_ITEM  could be safer
http://bugs.python.org/issue30459  opened by espie

#30460: file opened for updating cannot write after read
http://bugs.python.org/issue30460  opened by jkloth

#30462: urllib does not support NO_PROXY environment variable containi
http://bugs.python.org/issue30462  opened by Jiri Hnidek

#30463: Add __slots__ to ABC convenience class
http://bugs.python.org/issue30463  opened by Aaron Hall

#30464: gammavariate has a wrong comment
http://bugs.python.org/issue30464  opened by leodema

#30465: FormattedValue expressions have wrong lineno and col_offset in
http://bugs.python.org/issue30465  opened by lukasz.langa

#30466: Tutorial doesn't explain the use of classes
http://bugs.python.org/issue30466  opened by trey

#30468: Propagate zipfile.py pypy issue #905 patch to CPython 3.7
http://bugs.python.org/issue30468  opened by shubhar

#30469: Inconsistent Execution of Generic Descriptor Attributes
http://bugs.python.org/issue30469  opened by Ryan Morshead

#30470: Deprecate invalid ctypes call protection on Windows
http://bugs.python.org/issue30470  opened by Mariatta

#30473: defaultdict raises SystemError, __missing__ returned NULL in t
http://bugs.python.org/issue30473  opened by Peter Parente

#30475: Docs for PyDict_GetItemWithError() should say it returns a bor
http://bugs.python.org/issue30475  opened by eric.snow

#30476: Add   _GenerateCRCTable() to zipfile.py to pre-compute CRC Tab
http://bugs.python.org/issue30476  opened by shubhar

#30477: tuple.index error message improvement
http://bugs.python.org/issue30477  opened by llllllllll

#30479: improve asyncio debugging
http://bugs.python.org/issue30479  opened by tarek

#30480: samefile and sameopenfile fail for WebDAV mapped drives
http://bugs.python.org/issue30480  opened by eryksun

#30482: socket.getservbyname(), socket.getservbyport(), socket.getprot
http://bugs.python.org/issue30482  opened by dwfreed

#30483: urllib.parse.parse_qsl does not handle unicode data properly
http://bugs.python.org/issue30483  opened by maxking

#30484: Garbage Collector can cause Segfault whilst iterating dictiona
http://bugs.python.org/issue30484  opened by jimwright

#30485: Element.findall(path, dict) doesn't insert null namespace
http://bugs.python.org/issue30485  opened by ben.wainwright



Most recent 15 issues with no replies (15)
==========================================

#30484: Garbage Collector can cause Segfault whilst iterating dictiona
http://bugs.python.org/issue30484

#30483: urllib.parse.parse_qsl does not handle unicode data properly
http://bugs.python.org/issue30483

#30480: samefile and sameopenfile fail for WebDAV mapped drives
http://bugs.python.org/issue30480

#30479: improve asyncio debugging
http://bugs.python.org/issue30479

#30476: Add   _GenerateCRCTable() to zipfile.py to pre-compute CRC Tab
http://bugs.python.org/issue30476

#30475: Docs for PyDict_GetItemWithError() should say it returns a bor
http://bugs.python.org/issue30475

#30470: Deprecate invalid ctypes call protection on Windows
http://bugs.python.org/issue30470

#30462: urllib does not support NO_PROXY environment variable containi
http://bugs.python.org/issue30462

#30458: CRLF Injection in httplib
http://bugs.python.org/issue30458

#30457: Allow retrieve the number of waiters pending for most of the a
http://bugs.python.org/issue30457

#30456: 2to3 docs: example of fix for duplicates in second argument of
http://bugs.python.org/issue30456

#30454: Python module cx_Oracle ld installation issue on Solaris11U3 S
http://bugs.python.org/issue30454

#30428: mailbox.MMDF wrongly adds From_ header to file
http://bugs.python.org/issue30428

#30425: Python 3.6.1 (32-bit)_20170522020736.log
http://bugs.python.org/issue30425

#30419: Bdb: Update doc page
http://bugs.python.org/issue30419



Most recent 15 issues waiting for review (15)
=============================================

#30468: Propagate zipfile.py pypy issue #905 patch to CPython 3.7
http://bugs.python.org/issue30468

#30466: Tutorial doesn't explain the use of classes
http://bugs.python.org/issue30466

#30465: FormattedValue expressions have wrong lineno and col_offset in
http://bugs.python.org/issue30465

#30450: Pull Windows dependencies from GitHub rather than svn.python.o
http://bugs.python.org/issue30450

#30449: Improve __slots__ datamodel documentation
http://bugs.python.org/issue30449

#30441: os.environ raises RuntimeError: dictionary changed size during
http://bugs.python.org/issue30441

#30439: Expose the subinterpreters C-API in the stdlib.
http://bugs.python.org/issue30439

#30438: tarfile would fail to extract tarballs with files under R/O di
http://bugs.python.org/issue30438

#30431: input function truncates prompt by NULL byte
http://bugs.python.org/issue30431

#30427: isinstance checks in os.path.normcase redundant with os.fspath
http://bugs.python.org/issue30427

#30421: argparse: relative include of config files
http://bugs.python.org/issue30421

#30416: constant folding opens compiler to quadratic time hashing
http://bugs.python.org/issue30416

#30413: Add fnmatch.filterfalse function
http://bugs.python.org/issue30413

#30404: Make stdout and stderr truly unbuffered when using -u option
http://bugs.python.org/issue30404

#30403: Running extension modules using -m switch
http://bugs.python.org/issue30403



Top 10 most discussed issues (10)
=================================

#30441: os.environ raises RuntimeError: dictionary changed size during
http://bugs.python.org/issue30441  18 msgs

#30290: IDLE: add tests for help_about.py
http://bugs.python.org/issue30290  16 msgs

#30405: build.bat: register binaries for py launcher
http://bugs.python.org/issue30405  11 msgs

#30413: Add fnmatch.filterfalse function
http://bugs.python.org/issue30413  11 msgs

#30459: PyList_SET_ITEM  could be safer
http://bugs.python.org/issue30459   9 msgs

#12857: Expose called function on frame object
http://bugs.python.org/issue12857   8 msgs

#21074: Too aggressive constant folding
http://bugs.python.org/issue21074   8 msgs

#22257: PEP 432: Redesign the interpreter startup sequence
http://bugs.python.org/issue22257   8 msgs

#30362: Launcher add list and list with paths options
http://bugs.python.org/issue30362   8 msgs

#30392: default webbrowser fails on macOS Sierra 10.12.5
http://bugs.python.org/issue30392   8 msgs



Issues closed (50)
==================

#11410: Use GCC visibility attrs in PyAPI_*
http://bugs.python.org/issue11410  closed by twouters

#11470: Flag inappropriate uses of callable class attributes
http://bugs.python.org/issue11470  closed by twouters

#17411: Build failures with non-NDEBUG, non-Py_DEBUG builds.
http://bugs.python.org/issue17411  closed by twouters

#21056: csv documentation is incorrect
http://bugs.python.org/issue21056  closed by Mariatta

#23894: lib2to3 doesn't recognize rb'...' and f'...' in Python 3.6
http://bugs.python.org/issue23894  closed by lukasz.langa

#25794: __setattr__ does not always overload operators
http://bugs.python.org/issue25794  closed by serhiy.storchaka

#27728: HTTPConnection.requests has a mutable as default value for hea
http://bugs.python.org/issue27728  closed by matrixise

#27945: Various segfaults with dict
http://bugs.python.org/issue27945  closed by serhiy.storchaka

#28567: Bus error / segmentation fault on macOS debug build when using
http://bugs.python.org/issue28567  closed by ned.deily

#28707: add 'directory' option to the http.server module
http://bugs.python.org/issue28707  closed by haypo

#28845: Clean up known issues for AIX
http://bugs.python.org/issue28845  closed by zach.ware

#29102: Add an id field to PyInterpreterState.
http://bugs.python.org/issue29102  closed by eric.snow

#29104: Left bracket remains in format string result when '\' preceeds
http://bugs.python.org/issue29104  closed by serhiy.storchaka

#29321: Wrong documentation (Language Ref) for unicode and str compari
http://bugs.python.org/issue29321  closed by martin.panter

#29335: subprocess module does not check WIFSTOPPED on SIGCHLD
http://bugs.python.org/issue29335  closed by gregory.p.smith

#29340: SSL destructor segfaults in python3.6 threads when an unverifi
http://bugs.python.org/issue29340  closed by christian.heimes

#29851: Have importlib.reload() raise ModuleNotFoundError when a spec 
http://bugs.python.org/issue29851  closed by brett.cannon

#29943: PySlice_GetIndicesEx change broke ABI in 3.5 and 3.6 branches
http://bugs.python.org/issue29943  closed by serhiy.storchaka

#29976: urllib.parse clarify what ' ' in schemes mean
http://bugs.python.org/issue29976  closed by orsenthil

#30003: Fix handling escape characters in HZ codec
http://bugs.python.org/issue30003  closed by xiang.zhang

#30160: BaseHTTPRequestHandler.wfile: supported usage unclear
http://bugs.python.org/issue30160  closed by zach.ware

#30178: Indent methods and attributes of MimeTypes class
http://bugs.python.org/issue30178  closed by Mariatta

#30310: tkFont.py assumes that all font families are encoded as ascii 
http://bugs.python.org/issue30310  closed by serhiy.storchaka

#30355: Unicode symbols crash lib2to3.parse
http://bugs.python.org/issue30355  closed by terry.reedy

#30372: Status of __builtins__ is not totally clear
http://bugs.python.org/issue30372  closed by Mariatta

#30376: Curses documentation refers to incorrect type
http://bugs.python.org/issue30376  closed by kushal.das

#30377: Unnecessary complexity in tokenize.py: comments and newlines
http://bugs.python.org/issue30377  closed by serhiy.storchaka

#30380: Sphinx 1.6.1 raising new warnings in docs build
http://bugs.python.org/issue30380  closed by brett.cannon

#30394: smtplib leaves open sockets around if SMTPConnectError is rais
http://bugs.python.org/issue30394  closed by xiang.zhang

#30401: Remove the .bzrignore file
http://bugs.python.org/issue30401  closed by Mariatta

#30407: ipaddress ver. 1.0.14 IPv4Network off by 1
http://bugs.python.org/issue30407  closed by xiang.zhang

#30408: [defaultdict] default_factory should accept a "key" default pa
http://bugs.python.org/issue30408  closed by namtt

#30409: locale.getpreferredencoding doesn't return result
http://bugs.python.org/issue30409  closed by haypo

#30414: multiprocesing.Queue silently ignore messages after exc in _fe
http://bugs.python.org/issue30414  closed by pitrou

#30415: Improve fnmatch testing
http://bugs.python.org/issue30415  closed by serhiy.storchaka

#30423: [asyncio] orphan future close loop and cause "RuntimeError: Ev
http://bugs.python.org/issue30423  closed by lukasz.langa

#30426: why not use the same style func name. eg: int.to_bytes (with u
http://bugs.python.org/issue30426  closed by eric.smith

#30432: FileInput doesn't accept PathLike objects for file names
http://bugs.python.org/issue30432  closed by lukasz.langa

#30443: datetime.datetime.__add__ does not respect fold.
http://bugs.python.org/issue30443  closed by belopolsky

#30445: test_traceback fails in coverage run
http://bugs.python.org/issue30445  closed by zach.ware

#30447: test_capi.test_subinterps() fails on ARMv7 Ubuntu 3.x
http://bugs.python.org/issue30447  closed by eric.snow

#30452: xml.etree sourcecode bug
http://bugs.python.org/issue30452  closed by serhiy.storchaka

#30453: str.format() method, Rounding off wrong
http://bugs.python.org/issue30453  closed by mark.dickinson

#30461: glob returns results in undeterministic order
http://bugs.python.org/issue30461  closed by rhettinger

#30467: Propagate zipfile.py pypy issue #905 patch to CPython 2.7
http://bugs.python.org/issue30467  closed by serhiy.storchaka

#30471: "as" keyword in comprehensions
http://bugs.python.org/issue30471  closed by rhettinger

#30472: [Selenium 3.4.2-geckodriver 0.16.1] Python 3.6.1 unable to ini
http://bugs.python.org/issue30472  closed by pitrou

#30474: Crash on OS X   EXC_BAD_ACCESS (SIGSEGV)
http://bugs.python.org/issue30474  closed by ned.deily

#30478: Python 2.7 crashes in Linux environment
http://bugs.python.org/issue30478  closed by ncoghlan

#30481: lib/socket.py, line 330, Treated 'file' as a socket object, in
http://bugs.python.org/issue30481  closed by Mohamad amin Khakzadan

From guido at python.org  Fri May 26 16:51:50 2017
From: guido at python.org (Guido van Rossum)
Date: Fri, 26 May 2017 13:51:50 -0700
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CADiSq7f2oNe8fhKxAj3EAz2zDhp+LVKiOSQqvp8tYcWdbrJivA@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <20170525132620.7a80070a@fsol>
 <CADiSq7f2oNe8fhKxAj3EAz2zDhp+LVKiOSQqvp8tYcWdbrJivA@mail.gmail.com>
Message-ID: <CAP7+vJLW_OVCKdAR=XxjO4_0BsMWJCy9g0M58tOAZ-mp0wB64A@mail.gmail.com>

On Thu, May 25, 2017 at 7:19 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> Given the abstract, I'd suggest "Structural Subtyping" as a suitable
> title for the PEP.
>

Maybe even "Structural Subtyping (a.k.a. Duck Typing)"



> That said, I think it's fine to use "protocol" throughout the rest of
> the PEP as is currently the case - object protocols and network
> protocols are clearly different things, it's just the bare word
> "Protocols" appearing as a PEP title in the PEP index with no other
> context that's potentially confusing.
>

Agreed.


> I'm +1 on the general idea of the PEP, and only have one question
> regarding the specifics. Given:
>
>     import typing
>
>     class MyContainer:
>         def __len__(self) -> int:
>             ...
>         def close(self) -> None:
>             ...
>
> Would that be enough for a static typechecker to consider MyContainer
> a structural subtype of both typing.Sized and SupportsClose from the
> PEP, even though neither is imported explicitly into the module? I'm
> assuming the answer is "Yes", but I didn't see it explicitly stated
> anywhere.
>

Yes. Imports don't enter the matter. (Things get tied together at the call
sites.) @Ivan: if there isn't an example that makes this clear we should
add one.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170526/79eef781/attachment.html>

From guido at python.org  Fri May 26 16:56:05 2017
From: guido at python.org (Guido van Rossum)
Date: Fri, 26 May 2017 13:56:05 -0700
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
Message-ID: <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>

On Thu, May 25, 2017 at 10:49 AM, Mark Shannon <mark at hotpy.org> wrote:

> I really like this PEP in general. I think this brings the type system for
> type-hints closer to Python semantics.
>

Thank you.


> But there are a few points I disagree with.
> I don't think Protocol types should be tied to ABCs. It just makes things
> more complex with no obvious benefit.
>

There are backwards compatibility benefits -- we could make e.g. Sequence a
Protocol in Python 3.7 and it would be possible to write code that inherits
from Sequence and works in Python 3.6 and 3.7. For this to work we need
some support for non-abstract methods in Protocols.


> I also think all references to 'isinstance' and 'issubclass' should be
> removed. Type-hints should not have runtime semantics, beyond those that
> they have as classes.


Again, backwards compatibility.


> In fact, there is no need for protocol types to be classes at all.
>

That's pretty much a separate discussion (see
https://github.com/python/typing/issues/432).

-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170526/bf2b7930/attachment.html>

From ncoghlan at gmail.com  Sat May 27 00:26:54 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 27 May 2017 14:26:54 +1000
Subject: [Python-Dev] Aligning the packaging.python.org theme with the rest
 of the docs
Message-ID: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>

Hi folks,

Over on https://github.com/pypa/python-packaging-user-guide/pull/305#issuecomment-304169735
we're looking to update the theming of packaging.python.org to match
that of the language documentation at docs.python.org.

Doing that would also entail updating the documentation of the
individual tools and services (pip, pypi, setuptools, wheel, etc) to
maintain consistency with the main packaging user guide, so Jon has
tentatively broken the theme out as a (not yet published anywhere)
"pypa-theme" package to make it easier to re-use across multiple
projects.

The question that occurred to me is whether or not it might make more
sense to instead call that package "psf-docs-theme", to reflect that
it's intended specifically for projects that are legally backed by the
PSF, and that general Python projects looking for a nice,
high-contrast, theme should consider using an org independent one like
Alabaster instead.

Thoughts? Should we stick with pypa-theme as the name? Switch to
psf-docs-theme? Publish both, with pypa-theme adding PyPA specific
elements to a more general psf-docs-theme?

Cheers,
Nick.

P.S. In case folks aren't aware of the full legal arrangements here:
in addition to the informal "Python Packaging Authority" designation,
there's also a formally constituted PSF Packaging Working Group that
provides the legal connection back to the PSF. That means the
relationship between PyPA and the PSF ends up being pretty similar to
the one between python-dev and the PSF, where there's no direct PSF
involvement in day to day development activities, but the PSF provides
the legal and financial backing needed to sustainably maintain popular
community-supported software and services.

Part of my rationale for suggesting the inclusion of "psf" in the
package name is to make it clear that the intent would be to create a
clear and distinctive "trade dress" for the documentation of directly
PSF backed projects:
https://en.wikipedia.org/wiki/Trade_dress#Protection_for_electronic_interfaces_and_websites

Future requests to use the theme (beyond CPython and the PyPA) could
then be run through the PSF Trademarks committee, as with requests to
use the registered marks.

Whereas if we go with pypa-theme, then that would just be a
non-precedent-setting agreement between PyPA and CPython to share a
documentation theme, without trying to define any form of
documentation trade dress for the PSF in general.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sat May 27 03:19:07 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sat, 27 May 2017 17:19:07 +1000
Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C
 locale to a UTF-8 based locale
In-Reply-To: <CADiSq7deo_U6LLb6Vw0z398wiiUN_HrCO=oVefEdcG-aivpkEA@mail.gmail.com>
References: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>
 <CAEfz+TzVQq+rtD8+S8UbUVxt_ijbBne6xqgFP3C8aGhoZipB1Q@mail.gmail.com>
 <CADiSq7deo_U6LLb6Vw0z398wiiUN_HrCO=oVefEdcG-aivpkEA@mail.gmail.com>
Message-ID: <CADiSq7civ6nvpZUbkH=LsYg0R1-ba2-=s7v06e-suM6ghLT6tA@mail.gmail.com>

On 24 May 2017 at 02:34, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 23 May 2017 at 18:38, INADA Naoki <songofacandy at gmail.com> wrote:
>> Would you add example demonstrates how coercing LANG helps people?
>
> I'm honestly not sure it does - I think it's an assumption I added to
> the PEP early on, and never actually tested. Looking at it more
> closely now, all of the interpreter level checks are specifically for
> LC_CTYPE, and experimenting with "LANG=C LC_CTYPE=C.UTF-8" indicates
> that coercing only LC_CTYPE is still enough to fix the GNU readline
> encoding compatibility problem covered in
> https://www.python.org/dev/peps/pep-0538/#considering-locale-coercion-independently-of-utf-8-mode
>
> So I'll take another pass through the implementation this weekend, and
> simplify it to only set LC_CTYPE regardless of whether it's using
> C.UTF-8, C.utf8, or UTF-8 as the target locale. Assuming that doesn't
> uncover any hidden problems with the idea, I'll then update the PEP to
> match.

I've now gone through this, and as far as I can tell, setting only
LC_CTYPE is sufficient to handle all the scenarios that the PEP aims
to address, and has fewer potential side effects than setting both
LC_CTYPE and LANG.

Accordingly, I've updated both the PEP and the implementation to only
set LC_CTYPE and leave LANG alone:

* PEP: https://github.com/python/peps/commit/12cecb05489e74a36a11c17e8d0b1e36e3768bda
* Implementation:
https://github.com/python/cpython/pull/659/commits/939ba0a77d4b52a04315c129f9db89b90c0532cd

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From mariatta.wijaya at gmail.com  Sat May 27 11:06:17 2017
From: mariatta.wijaya at gmail.com (Mariatta Wijaya)
Date: Sat, 27 May 2017 08:06:17 -0700
Subject: [Python-Dev] Deprecate invalid ctypes call protection on Windows
In-Reply-To: <CAMpsgwbV2vh_8BK91aHM-SBLDrdHXK7tv7L5uBkO5QBMP7CstQ@mail.gmail.com>
References: <7ab62d94-34f7-1bb1-a676-4471e95d7f8a@python.org>
 <CAMpsgwb3Ly8Bq2b-XZkDf6x011TO5_WWQ8+3DnYCyrbVriudaQ@mail.gmail.com>
 <93282162-34be-2eb1-39a6-0f2fa8810ea9@python.org>
 <CAMpsgwbV2vh_8BK91aHM-SBLDrdHXK7tv7L5uBkO5QBMP7CstQ@mail.gmail.com>
Message-ID: <CAGbohnYxpL8r6N4gUfH=hUF=1fo=5S5ojDOL+wP-t1p7eg+sYQ@mail.gmail.com>

Thanks all.

Documentation has been updated in  https://bugs.python.org/issue30470


On May 23, 2017 9:13 PM, "Victor Stinner" <victor.stinner at gmail.com> wrote:

Sure, make your change and then update libffi!

Victor

Le 23 mai 2017 18:19, "Steve Dower" <steve.dower at python.org> a ?crit :

> On 23May2017 1212, Victor Stinner wrote:
>
>> 2017-05-22 13:17 GMT-05:00 Steve Dower <steve.dower at python.org>:
>>
>>> Once the special protection is removed, most of these cases will become
>>> OSError due to the general protection against segmentation faults.
>>>
>>
>> It didn't know that ctypes on Windows had a special protection against
>> programming errors. I'm not aware of such protection Linux. If you
>> call a function with the wrong number of arguments, it's likely to
>> crash or return random data.
>>
>> I guess that the point is to help debugging. But since Python 3.6,
>> faulthandler now registers a Windows exception handler and so it able
>> to dump the Python traceback on any Windows exception:
>> https://docs.python.org/dev/library/faulthandler.html#faulthandler.enable
>>
>> So I think that it's now fine to remove the ctypes protection. Just
>> advice (remind? ;-)) users to enable faulthandler: python3 -X
>> faulthandler, or call faulthandler.enable(). (You might want to use a
>> log file for that on Windows, depends on the use case.)
>>
>
> faulthandler is already recommended in the docs, and the existing SEH
> protection for access violations will remain (since that is independent of
> libffi).
>
> I'll be honest, I have appreciated the functionality in the past, but it
> really isn't good practice and getting rid of it will be an overall
> benefit. Technically even the segfault protection isn't a great idea, since
> you really do end up in an unknown state with regards to memory page
> allocations, but it's better than crashing all the way out.
>
> Cheers,
> Steve
>

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/
mariatta.wijaya%40gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170527/406ca0f4/attachment-0001.html>

From brett at python.org  Sat May 27 14:32:29 2017
From: brett at python.org (Brett Cannon)
Date: Sat, 27 May 2017 18:32:29 +0000
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
In-Reply-To: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
Message-ID: <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>

On Fri, 26 May 2017 at 21:28 Nick Coghlan <ncoghlan at gmail.com> wrote:

> Hi folks,
>
> Over on
> https://github.com/pypa/python-packaging-user-guide/pull/305#issuecomment-304169735
> we're looking to update the theming of packaging.python.org to match
> that of the language documentation at docs.python.org.
>
> Doing that would also entail updating the documentation of the
> individual tools and services (pip, pypi, setuptools, wheel, etc) to
> maintain consistency with the main packaging user guide, so Jon has
> tentatively broken the theme out as a (not yet published anywhere)
> "pypa-theme" package to make it easier to re-use across multiple
> projects.
>
> The question that occurred to me is whether or not it might make more
> sense to instead call that package "psf-docs-theme", to reflect that
> it's intended specifically for projects that are legally backed by the
> PSF, and that general Python projects looking for a nice,
> high-contrast, theme should consider using an org independent one like
> Alabaster instead.
>
> Thoughts? Should we stick with pypa-theme as the name? Switch to
> psf-docs-theme? Publish both, with pypa-theme adding PyPA specific
> elements to a more general psf-docs-theme?
>

If we're going to share the theme beyond docs.python.org it makes sense to
have a shared theme under the Python org that can easily be reused by
multiple sets of documentation.

As for the name, the psf-docs-theme seems fine with me.

-Brett


>
> Cheers,
> Nick.
>
> P.S. In case folks aren't aware of the full legal arrangements here:
> in addition to the informal "Python Packaging Authority" designation,
> there's also a formally constituted PSF Packaging Working Group that
> provides the legal connection back to the PSF. That means the
> relationship between PyPA and the PSF ends up being pretty similar to
> the one between python-dev and the PSF, where there's no direct PSF
> involvement in day to day development activities, but the PSF provides
> the legal and financial backing needed to sustainably maintain popular
> community-supported software and services.
>
> Part of my rationale for suggesting the inclusion of "psf" in the
> package name is to make it clear that the intent would be to create a
> clear and distinctive "trade dress" for the documentation of directly
> PSF backed projects:
>
> https://en.wikipedia.org/wiki/Trade_dress#Protection_for_electronic_interfaces_and_websites
>
> Future requests to use the theme (beyond CPython and the PyPA) could
> then be run through the PSF Trademarks committee, as with requests to
> use the registered marks.
>
> Whereas if we go with pypa-theme, then that would just be a
> non-precedent-setting agreement between PyPA and CPython to share a
> documentation theme, without trying to define any form of
> documentation trade dress for the PSF in general.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170527/3b5792ec/attachment.html>

From gvanrossum at gmail.com  Sat May 27 16:54:58 2017
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Sat, 27 May 2017 13:54:58 -0700
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
In-Reply-To: <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
 <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
Message-ID: <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>

Are you also going to stop others from using the psf theme?

On May 27, 2017 11:35, "Brett Cannon" <brett at python.org> wrote:

>
>
> On Fri, 26 May 2017 at 21:28 Nick Coghlan <ncoghlan at gmail.com> wrote:
>
>> Hi folks,
>>
>> Over on https://github.com/pypa/python-packaging-user-guide/
>> pull/305#issuecomment-304169735
>> we're looking to update the theming of packaging.python.org to match
>> that of the language documentation at docs.python.org.
>>
>> Doing that would also entail updating the documentation of the
>> individual tools and services (pip, pypi, setuptools, wheel, etc) to
>> maintain consistency with the main packaging user guide, so Jon has
>> tentatively broken the theme out as a (not yet published anywhere)
>> "pypa-theme" package to make it easier to re-use across multiple
>> projects.
>>
>> The question that occurred to me is whether or not it might make more
>> sense to instead call that package "psf-docs-theme", to reflect that
>> it's intended specifically for projects that are legally backed by the
>> PSF, and that general Python projects looking for a nice,
>> high-contrast, theme should consider using an org independent one like
>> Alabaster instead.
>>
>> Thoughts? Should we stick with pypa-theme as the name? Switch to
>> psf-docs-theme? Publish both, with pypa-theme adding PyPA specific
>> elements to a more general psf-docs-theme?
>>
>
> If we're going to share the theme beyond docs.python.org it makes sense
> to have a shared theme under the Python org that can easily be reused by
> multiple sets of documentation.
>
> As for the name, the psf-docs-theme seems fine with me.
>
> -Brett
>
>
>>
>> Cheers,
>> Nick.
>>
>> P.S. In case folks aren't aware of the full legal arrangements here:
>> in addition to the informal "Python Packaging Authority" designation,
>> there's also a formally constituted PSF Packaging Working Group that
>> provides the legal connection back to the PSF. That means the
>> relationship between PyPA and the PSF ends up being pretty similar to
>> the one between python-dev and the PSF, where there's no direct PSF
>> involvement in day to day development activities, but the PSF provides
>> the legal and financial backing needed to sustainably maintain popular
>> community-supported software and services.
>>
>> Part of my rationale for suggesting the inclusion of "psf" in the
>> package name is to make it clear that the intent would be to create a
>> clear and distinctive "trade dress" for the documentation of directly
>> PSF backed projects:
>> https://en.wikipedia.org/wiki/Trade_dress#Protection_for_
>> electronic_interfaces_and_websites
>>
>> Future requests to use the theme (beyond CPython and the PyPA) could
>> then be run through the PSF Trademarks committee, as with requests to
>> use the registered marks.
>>
>> Whereas if we go with pypa-theme, then that would just be a
>> non-precedent-setting agreement between PyPA and CPython to share a
>> documentation theme, without trying to define any form of
>> documentation trade dress for the PSF in general.
>>
>> --
>> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
>> _______________________________________________
>> Python-Dev mailing list
>> Python-Dev at python.org
>> https://mail.python.org/mailman/listinfo/python-dev
>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
>> brett%40python.org
>>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170527/0fd4a016/attachment.html>

From ncoghlan at gmail.com  Sun May 28 01:31:35 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 28 May 2017 15:31:35 +1000
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
In-Reply-To: <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
 <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
 <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>
Message-ID: <CADiSq7dn7Ovh_BLzzwbRJMOCvRGoW5gwup4Z=ri4nUMKx4VOcw@mail.gmail.com>

On 28 May 2017 at 06:54, Guido van Rossum <gvanrossum at gmail.com> wrote:
> Are you also going to stop others from using the psf theme?

I think it would definitely make sense to discourage the use of this
particular theme for projects that aren't relatively directly
affiliated with the PSF - there are plenty of other pip-installable
high contrast themes out there that aren't closely associated with a
particular backing organisation.

The one Jon was originally considering for the PyPA docs was
Alabaster, which is now the default theme in Sphinx 1.3+:
https://alabaster.readthedocs.io/en/latest/

So I think it would be appropriate to include a sentence like the
following in the README for psf-docs-theme:

"Please limit use of this theme to projects which are closely
affiliated with the Python Software Foundation, and have permission
from either the CPython core development team or the PSF itself for
such use. For other projects looking for a simple, high contrast, pip
installable Sphinx theme, we recommend the Alabaster theme used by
default in Sphinx 1.3+."

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From songofacandy at gmail.com  Sun May 28 02:46:00 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Sun, 28 May 2017 15:46:00 +0900
Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C
 locale to a UTF-8 based locale
In-Reply-To: <CADiSq7civ6nvpZUbkH=LsYg0R1-ba2-=s7v06e-suM6ghLT6tA@mail.gmail.com>
References: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>
 <CAEfz+TzVQq+rtD8+S8UbUVxt_ijbBne6xqgFP3C8aGhoZipB1Q@mail.gmail.com>
 <CADiSq7deo_U6LLb6Vw0z398wiiUN_HrCO=oVefEdcG-aivpkEA@mail.gmail.com>
 <CADiSq7civ6nvpZUbkH=LsYg0R1-ba2-=s7v06e-suM6ghLT6tA@mail.gmail.com>
Message-ID: <CAEfz+TwoT9zJ+EhwpVrivJtMO-=wAqwO3PLPQBLsN334jMesvg@mail.gmail.com>

Now I approve the PEP 538.

It's side-effect (just set LC_CTYPE envvar) seems simple enough and
moderate enough.

Locale coercion will save people from unwanted mojibake (escaped string)
and locale warning will navigate people to configure locale properly.

And there are configure options and envvar option to disable it for people
who want to continue to use C locale explicitly.

Congrats, Nick!

On Sat, May 27, 2017 at 4:19 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 24 May 2017 at 02:34, Nick Coghlan <ncoghlan at gmail.com> wrote:
>> On 23 May 2017 at 18:38, INADA Naoki <songofacandy at gmail.com> wrote:
>>> Would you add example demonstrates how coercing LANG helps people?
>>
>> I'm honestly not sure it does - I think it's an assumption I added to
>> the PEP early on, and never actually tested. Looking at it more
>> closely now, all of the interpreter level checks are specifically for
>> LC_CTYPE, and experimenting with "LANG=C LC_CTYPE=C.UTF-8" indicates
>> that coercing only LC_CTYPE is still enough to fix the GNU readline
>> encoding compatibility problem covered in
>> https://www.python.org/dev/peps/pep-0538/#considering-locale-coercion-independently-of-utf-8-mode
>>
>> So I'll take another pass through the implementation this weekend, and
>> simplify it to only set LC_CTYPE regardless of whether it's using
>> C.UTF-8, C.utf8, or UTF-8 as the target locale. Assuming that doesn't
>> uncover any hidden problems with the idea, I'll then update the PEP to
>> match.
>
> I've now gone through this, and as far as I can tell, setting only
> LC_CTYPE is sufficient to handle all the scenarios that the PEP aims
> to address, and has fewer potential side effects than setting both
> LC_CTYPE and LANG.
>
> Accordingly, I've updated both the PEP and the implementation to only
> set LC_CTYPE and leave LANG alone:
>
> * PEP: https://github.com/python/peps/commit/12cecb05489e74a36a11c17e8d0b1e36e3768bda
> * Implementation:
> https://github.com/python/cpython/pull/659/commits/939ba0a77d4b52a04315c129f9db89b90c0532cd
>
> Regards,
> Nick.
>
> --
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ncoghlan at gmail.com  Sun May 28 02:49:40 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Sun, 28 May 2017 16:49:40 +1000
Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C
 locale to a UTF-8 based locale
In-Reply-To: <CAEfz+TwoT9zJ+EhwpVrivJtMO-=wAqwO3PLPQBLsN334jMesvg@mail.gmail.com>
References: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>
 <CAEfz+TzVQq+rtD8+S8UbUVxt_ijbBne6xqgFP3C8aGhoZipB1Q@mail.gmail.com>
 <CADiSq7deo_U6LLb6Vw0z398wiiUN_HrCO=oVefEdcG-aivpkEA@mail.gmail.com>
 <CADiSq7civ6nvpZUbkH=LsYg0R1-ba2-=s7v06e-suM6ghLT6tA@mail.gmail.com>
 <CAEfz+TwoT9zJ+EhwpVrivJtMO-=wAqwO3PLPQBLsN334jMesvg@mail.gmail.com>
Message-ID: <CADiSq7csbbvHXuauxvbSSRe377L-B3_BEE47ixbam34YQ3CKRw@mail.gmail.com>

On 28 May 2017 at 16:46, INADA Naoki <songofacandy at gmail.com> wrote:
> Now I approve the PEP 538.
>
> It's side-effect (just set LC_CTYPE envvar) seems simple enough and
> moderate enough.
>
> Locale coercion will save people from unwanted mojibake (escaped string)
> and locale warning will navigate people to configure locale properly.
>
> And there are configure options and envvar option to disable it for people
> who want to continue to use C locale explicitly.
>
> Congrats, Nick!

Thank you! And thank you for your work in reviewing the PEP - I think
the accepted version is a significant improvement over the more
intrusive design I originally proposed downstream in Fedora :)

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From ethan at stoneleaf.us  Sun May 28 03:27:51 2017
From: ethan at stoneleaf.us (Ethan Furman)
Date: Sun, 28 May 2017 00:27:51 -0700
Subject: [Python-Dev] PEP 538 (review round 2): Coercing the legacy C
 locale to a UTF-8 based locale
In-Reply-To: <CAEfz+TwoT9zJ+EhwpVrivJtMO-=wAqwO3PLPQBLsN334jMesvg@mail.gmail.com>
References: <CADiSq7c-8=0fg2bzTi3frxUraLTf5eK0OYG0VGdcP7xhFD0q9A@mail.gmail.com>
 <CAEfz+TzVQq+rtD8+S8UbUVxt_ijbBne6xqgFP3C8aGhoZipB1Q@mail.gmail.com>
 <CADiSq7deo_U6LLb6Vw0z398wiiUN_HrCO=oVefEdcG-aivpkEA@mail.gmail.com>
 <CADiSq7civ6nvpZUbkH=LsYg0R1-ba2-=s7v06e-suM6ghLT6tA@mail.gmail.com>
 <CAEfz+TwoT9zJ+EhwpVrivJtMO-=wAqwO3PLPQBLsN334jMesvg@mail.gmail.com>
Message-ID: <592A7BF7.9000409@stoneleaf.us>

On 05/27/2017 11:46 PM, INADA Naoki wrote:

> Now I approve the PEP 538.

Thank you, Nick and Inada-san!

--
~Ethan~


From solipsis at pitrou.net  Sun May 28 07:42:12 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Sun, 28 May 2017 13:42:12 +0200
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
Message-ID: <20170528134212.0c947462@fsol>

On Sat, 27 May 2017 14:26:54 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Thoughts? Should we stick with pypa-theme as the name? Switch to
> psf-docs-theme? Publish both, with pypa-theme adding PyPA specific
> elements to a more general psf-docs-theme?
> 
[...]
> 
> Future requests to use the theme (beyond CPython and the PyPA) could
> then be run through the PSF Trademarks committee, as with requests to
> use the registered marks.

Did you consider that whoever designed the theme might not agree with
such a drastic policy change?  You should first get their agreement for
this.  And what about derived works (which probably do exist already)?

I don't know what the current legal situation for the theme is.  I'm
assuming it's licensed under an open source license (or perhaps
implicitly believed to be so, which would be embarassing).  If that's
the case, I'm opposed to requiring that previously freely-reusable
resources now need to get through a case-by-case approval process.
This is at odds with the principles of free and open source software.

(by the way I don't think that Alabaster is really comparable.  "High
contrast" isn't the only concern here.  It's actually difficult to
find a theme that ticks all the boxes in terms of being able to
emphasize structure, having a sensible colour theme, displaying
long-winded doc pages in a compact and readable form, etc.)


Alternatively, you could have the PSF commission a designer and
produce a truly specific theme for "official" Python docs.  But perhaps
that would be trying to solve a problem that does not truly exist
(that is, that people would judge a third-party doc "official" just
because it reuses the same grey-ish Sphinx theme).

Or you could deem the use of the trademark-protected Python logo
sufficient to denote "official" docs (which would still allow for
variety in theming, which btw exists even within the Python docs:
witness the theme difference between 2.7 and 3.x), and call it a day.

Regards

Antoine.



From mal at egenix.com  Sun May 28 08:07:54 2017
From: mal at egenix.com (M.-A. Lemburg)
Date: Sun, 28 May 2017 14:07:54 +0200
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
In-Reply-To: <CADiSq7dn7Ovh_BLzzwbRJMOCvRGoW5gwup4Z=ri4nUMKx4VOcw@mail.gmail.com>
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
 <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
 <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>
 <CADiSq7dn7Ovh_BLzzwbRJMOCvRGoW5gwup4Z=ri4nUMKx4VOcw@mail.gmail.com>
Message-ID: <df8962f7-ebdd-44f3-0ede-7abd7765705e@egenix.com>

I'm -1 on going down the suggested route of Apple et al. for an
open source language.

We don't need more trademarks to "protect" ourselves against
fellow open source projects.

I see this whole trademark business that OSS projects are getting
into in recent years in a more and more critical light.

On one hand you have the open source idea, where you want to enable
anyone to reuse your code. The OSS copyright licenses implement this
idea nicely.

On the other hand, you have trademark law which limits
this reusability by imposing strict rules on how you can reuse the
names, designs, colors, etc. trademarked by the OSS project (or
else - TM lawyers tell you - you lose the rights to the TMs).

The latter has to be handled with great care, since while you
do want to protect the trademarks against malicious use by third
parties (e.g. a company registering the same mark and then preventing
use of the trademark in their jurisdiction), you don't want to
hinder the original intent of having an open source license. In
particular, you don't want to bully community projects which try to
further the use of our open source software by using similar marks.

In my role as PSF TM committee member, it's often painful to have to
tell community members that they cannot use e.g. really nice looking
variants of the Python logo for their projects. Let's not add more
pain.

If you want to make PSF related projects be recognized as such,
simply add the PSF logo to the footer of the site and provide
some blurb explaining the relationships to the about page.

We could also have a special "PSF Project" logo for such a
purpose. Use of the logo would then be subject to the TM
requirements, but not the looks and UI of the site.



On 28.05.2017 07:31, Nick Coghlan wrote:
> On 28 May 2017 at 06:54, Guido van Rossum <gvanrossum at gmail.com> wrote:
>> Are you also going to stop others from using the psf theme?
> 
> I think it would definitely make sense to discourage the use of this
> particular theme for projects that aren't relatively directly
> affiliated with the PSF - there are plenty of other pip-installable
> high contrast themes out there that aren't closely associated with a
> particular backing organisation.
> 
> The one Jon was originally considering for the PyPA docs was
> Alabaster, which is now the default theme in Sphinx 1.3+:
> https://alabaster.readthedocs.io/en/latest/
> 
> So I think it would be appropriate to include a sentence like the
> following in the README for psf-docs-theme:
> 
> "Please limit use of this theme to projects which are closely
> affiliated with the Python Software Foundation, and have permission
> from either the CPython core development team or the PSF itself for
> such use. For other projects looking for a simple, high contrast, pip
> installable Sphinx theme, we recommend the Alabaster theme used by
> default in Sphinx 1.3+."
> 
> Cheers,
> Nick.
> 

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 28 2017)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From levkivskyi at gmail.com  Sun May 28 09:19:03 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sun, 28 May 2017 15:19:03 +0200
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
Message-ID: <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>

Thanks everyone for interesting suggestions!

@Antoine @Guido:
Some of the possible options for the title are:
* Protocols (structural subtyping)
* Protocols (static duck typing)
* Structural subtyping (static duck typing)
which one do you prefer?

@Nick:
Yes, explicit imports are not necessary for static type checkers (I will
add a short comment about this).

@Mark:
I agree with Guido on all points here. For example,
collections.abc.Iterable is already a class,
and lots of code uses isinstance(obj, collections.abc.Iterable) and similar
checks with other ABCs
(also in a structural manner, i.e. via __subclasshook__). So that disabling
this will case many breakages.
The question of whether typing.Iterable[int] should be a class is
independent (orthogonal) and
does not belong to this PEP.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170528/1685146e/attachment.html>

From kevinjacobconway at gmail.com  Sun May 28 10:13:22 2017
From: kevinjacobconway at gmail.com (Kevin Conway)
Date: Sun, 28 May 2017 14:13:22 +0000
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
 <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
Message-ID: <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>

> Some of the possible options for the title are
It seems like you're talking about something most other languages would
refer to as "Interfaces". What is unique about this proposal that would
call for not using the industry standard language?

> Type-hints should not have runtime semantics, beyond those that they have
as classes
>  lots of code uses isinstance(obj, collections.abc.Iterable) and similar
checks with other ABCs
Having interfaces defined as something extended from abc doesn't
necessitate their use at runtime, but it does open up a great deal of
options for those of us who want to do so. I've been leveraging abc for a
few years now to implement a lightweight version of what this PEP is
attempting to achieve (https://github.com/kevinconway/iface). Once you
start getting into dynamically loaded plugins you often lose the ability to
strictly enforce the shape of the input until runtime. In those cases, I've
found it exceedingly useful to add 'isinstance' and 'issubbclass' as
assertions to input of untrusted types for the tests and non-production
deployments. For a perf boost in prod you can throw the -O flag and strip
out the assertions to remove the runtime checks. I've found that to be a
valuable pattern.

On Sun, May 28, 2017 at 8:21 AM Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> Thanks everyone for interesting suggestions!
>
> @Antoine @Guido:
> Some of the possible options for the title are:
> * Protocols (structural subtyping)
> * Protocols (static duck typing)
> * Structural subtyping (static duck typing)
> which one do you prefer?
>
> @Nick:
> Yes, explicit imports are not necessary for static type checkers (I will
> add a short comment about this).
>
> @Mark:
> I agree with Guido on all points here. For example,
> collections.abc.Iterable is already a class,
> and lots of code uses isinstance(obj, collections.abc.Iterable) and
> similar checks with other ABCs
> (also in a structural manner, i.e. via __subclasshook__). So that
> disabling this will case many breakages.
> The question of whether typing.Iterable[int] should be a class is
> independent (orthogonal) and
> does not belong to this PEP.
>
> --
> Ivan
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/kevinjacobconway%40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170528/954784bd/attachment.html>

From mertz at gnosis.cx  Sun May 28 11:15:49 2017
From: mertz at gnosis.cx (David Mertz)
Date: Sun, 28 May 2017 08:15:49 -0700
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
In-Reply-To: <df8962f7-ebdd-44f3-0ede-7abd7765705e@egenix.com>
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
 <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
 <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>
 <CADiSq7dn7Ovh_BLzzwbRJMOCvRGoW5gwup4Z=ri4nUMKx4VOcw@mail.gmail.com>
 <df8962f7-ebdd-44f3-0ede-7abd7765705e@egenix.com>
Message-ID: <CAEbHw4YYFkfVmjeqcEy5Sc1QpgoaZog_ozqxdLt4-MUC1KNtVg@mail.gmail.com>

I agree with MAL and have also been on the Trademarks Committee for 8-9
years. Protecting an actual Mark like the logo is fine, as painful as it is
to someone's say no to an attractive derived logo. But trying to protect a
look-and-feel is way too far down the path of evil (it's what some
proprietary companies we want to be different from do). Moreover, when when
that's done it's not generally as trademark but as "trade dress"
copyright... which is a concept I opposed ethically, but know it's the
usual legal instrument.

On May 28, 2017 5:08 AM, "M.-A. Lemburg" <mal at egenix.com> wrote:

> I'm -1 on going down the suggested route of Apple et al. for an
> open source language.
>
> We don't need more trademarks to "protect" ourselves against
> fellow open source projects.
>
> I see this whole trademark business that OSS projects are getting
> into in recent years in a more and more critical light.
>
> On one hand you have the open source idea, where you want to enable
> anyone to reuse your code. The OSS copyright licenses implement this
> idea nicely.
>
> On the other hand, you have trademark law which limits
> this reusability by imposing strict rules on how you can reuse the
> names, designs, colors, etc. trademarked by the OSS project (or
> else - TM lawyers tell you - you lose the rights to the TMs).
>
> The latter has to be handled with great care, since while you
> do want to protect the trademarks against malicious use by third
> parties (e.g. a company registering the same mark and then preventing
> use of the trademark in their jurisdiction), you don't want to
> hinder the original intent of having an open source license. In
> particular, you don't want to bully community projects which try to
> further the use of our open source software by using similar marks.
>
> In my role as PSF TM committee member, it's often painful to have to
> tell community members that they cannot use e.g. really nice looking
> variants of the Python logo for their projects. Let's not add more
> pain.
>
> If you want to make PSF related projects be recognized as such,
> simply add the PSF logo to the footer of the site and provide
> some blurb explaining the relationships to the about page.
>
> We could also have a special "PSF Project" logo for such a
> purpose. Use of the logo would then be subject to the TM
> requirements, but not the looks and UI of the site.
>
>
>
> On 28.05.2017 07:31, Nick Coghlan wrote:
> > On 28 May 2017 at 06:54, Guido van Rossum <gvanrossum at gmail.com> wrote:
> >> Are you also going to stop others from using the psf theme?
> >
> > I think it would definitely make sense to discourage the use of this
> > particular theme for projects that aren't relatively directly
> > affiliated with the PSF - there are plenty of other pip-installable
> > high contrast themes out there that aren't closely associated with a
> > particular backing organisation.
> >
> > The one Jon was originally considering for the PyPA docs was
> > Alabaster, which is now the default theme in Sphinx 1.3+:
> > https://alabaster.readthedocs.io/en/latest/
> >
> > So I think it would be appropriate to include a sentence like the
> > following in the README for psf-docs-theme:
> >
> > "Please limit use of this theme to projects which are closely
> > affiliated with the Python Software Foundation, and have permission
> > from either the CPython core development team or the PSF itself for
> > such use. For other projects looking for a simple, high contrast, pip
> > installable Sphinx theme, we recommend the Alabaster theme used by
> > default in Sphinx 1.3+."
> >
> > Cheers,
> > Nick.
> >
>
> --
> Marc-Andre Lemburg
> eGenix.com
>
> Professional Python Services directly from the Experts (#1, May 28 2017)
> >>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
> >>> Python Database Interfaces ...           http://products.egenix.com/
> >>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
> ________________________________________________________________________
>
> ::: We implement business ideas - efficiently in both time and costs :::
>
>    eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
>     D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
>            Registered at Amtsgericht Duesseldorf: HRB 46611
>                http://www.egenix.com/company/contact/
>                       http://www.malemburg.com/
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> mertz%40gnosis.cx
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170528/49c1f0aa/attachment.html>

From levkivskyi at gmail.com  Sun May 28 11:27:33 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Sun, 28 May 2017 17:27:33 +0200
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
 <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
 <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>
Message-ID: <CAOMjWkkjmrkzws_d-TqQ+CN=CPSwJhyCejgwQMCO7mWd6fP4DA@mail.gmail.com>

On 28 May 2017 at 16:13, Kevin Conway <kevinjacobconway at gmail.com> wrote:

> > Some of the possible options for the title are
> It seems like you're talking about something most other languages would
> refer to as "Interfaces". What is unique about this proposal that would
> call for not using the industry standard language?
>

Well, I would say there is no "industry standard language" about structural
subtyping. There are interfaces, protocols,
traits, mixins, typeclasses, roles, and probably some other terms I am not
aware of - all with subtly different semantics
in different languages. There are several reasons why we use term protocol
and not interface.
Two important reasons for me are:
* The term protocol is already de-facto standard in Python for things like
sequence protocol, iterator protocol,
  descriptor protocol, etc.
* Protocols are very different from Java interfaces in one important
aspect: they don't require explicit
  declaration of implementation, they are mainly oriented on duck-typing.
Maybe we need to add a short section to rejected ideas?

> Type-hints should not have runtime semantics, beyond those that they have
> as classes
> >  lots of code uses isinstance(obj, collections.abc.Iterable) and
> similar checks with other ABCs
> Having interfaces defined as something extended from abc doesn't
> necessitate their use at runtime, but it does open up a great deal of
> options for those of us who want to do so. I've been leveraging abc for a
> few years now to implement a lightweight version of what this PEP is
> attempting to achieve
>

IIUC this is not the main goal of the PEP, the main goal is to provide
support/standard for _static_ structural subtyping.
Possibility to use protocols in runtime context is rather a minor bonus
that exists mostly to provide a seamless transition
for projects that already use ABCs.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170528/d46ea0b7/attachment.html>

From guido at python.org  Sun May 28 13:40:37 2017
From: guido at python.org (Guido van Rossum)
Date: Sun, 28 May 2017 10:40:37 -0700
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAOMjWkkjmrkzws_d-TqQ+CN=CPSwJhyCejgwQMCO7mWd6fP4DA@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
 <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
 <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>
 <CAOMjWkkjmrkzws_d-TqQ+CN=CPSwJhyCejgwQMCO7mWd6fP4DA@mail.gmail.com>
Message-ID: <CAP7+vJLWpeDpZshcMvtK+axz2441TKdHyP+T_MWyikJ=PNuHAQ@mail.gmail.com>

On Sun, May 28, 2017 at 8:27 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> On 28 May 2017 at 16:13, Kevin Conway <kevinjacobconway at gmail.com> wrote:
>
>> > Some of the possible options for the title are
>> It seems like you're talking about something most other languages would
>> refer to as "Interfaces". What is unique about this proposal that would
>> call for not using the industry standard language?
>>
>
> Well, I would say there is no "industry standard language" about
> structural subtyping. There are interfaces, protocols,
> traits, mixins, typeclasses, roles, and probably some other terms I am not
> aware of - all with subtly different semantics
> in different languages. There are several reasons why we use term protocol
> and not interface.
> Two important reasons for me are:
> * The term protocol is already de-facto standard in Python for things like
> sequence protocol, iterator protocol,
>   descriptor protocol, etc.
> * Protocols are very different from Java interfaces in one important
> aspect: they don't require explicit
>   declaration of implementation, they are mainly oriented on duck-typing.
> Maybe we need to add a short section to rejected ideas?
>

If you feel like it.

Regarding the title, I'd like to keep the word Protocol in the title too,
so I'd go with "Protocols: Structural subtyping (duck typing)" -- hope
that's not too long to fit in a PEP title field.


> > Type-hints should not have runtime semantics, beyond those that they
>> have as classes
>> >  lots of code uses isinstance(obj, collections.abc.Iterable) and
>> similar checks with other ABCs
>> Having interfaces defined as something extended from abc doesn't
>> necessitate their use at runtime, but it does open up a great deal of
>> options for those of us who want to do so. I've been leveraging abc for a
>> few years now to implement a lightweight version of what this PEP is
>> attempting to achieve
>>
>
> IIUC this is not the main goal of the PEP, the main goal is to provide
> support/standard for _static_ structural subtyping.
> Possibility to use protocols in runtime context is rather a minor bonus
> that exists mostly to provide a seamless transition
> for projects that already use ABCs.
>

Is something like this already in the PEP? It deserves attention in one of
the earlier sections.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170528/61c4fd96/attachment.html>

From greg.ewing at canterbury.ac.nz  Mon May 29 01:48:07 2017
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Mon, 29 May 2017 17:48:07 +1200
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
In-Reply-To: <df8962f7-ebdd-44f3-0ede-7abd7765705e@egenix.com>
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
 <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
 <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>
 <CADiSq7dn7Ovh_BLzzwbRJMOCvRGoW5gwup4Z=ri4nUMKx4VOcw@mail.gmail.com>
 <df8962f7-ebdd-44f3-0ede-7abd7765705e@egenix.com>
Message-ID: <592BB617.9060900@canterbury.ac.nz>

M.-A. Lemburg wrote:
> In my role as PSF TM committee member, it's often painful to have to
> tell community members that they cannot use e.g. really nice looking
> variants of the Python logo for their projects. Let's not add more
> pain.

But it's always within the PSF's power to give that community
member permission to use that variant if they ask, is it not?
So you don't actually have to tell anyone that they can't
use anything.

-- 
Greg

From levkivskyi at gmail.com  Mon May 29 06:41:42 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Mon, 29 May 2017 12:41:42 +0200
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAP7+vJLWpeDpZshcMvtK+axz2441TKdHyP+T_MWyikJ=PNuHAQ@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
 <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
 <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>
 <CAOMjWkkjmrkzws_d-TqQ+CN=CPSwJhyCejgwQMCO7mWd6fP4DA@mail.gmail.com>
 <CAP7+vJLWpeDpZshcMvtK+axz2441TKdHyP+T_MWyikJ=PNuHAQ@mail.gmail.com>
Message-ID: <CAOMjWk==E-vpphCOADGqFk_UjyQuoaHg1+8m2T++Jh6y9ZzETA@mail.gmail.com>

On 28 May 2017 at 19:40, Guido van Rossum <guido at python.org> wrote:

> On Sun, May 28, 2017 at 8:27 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
> wrote:
>
[...]

> Regarding the title, I'd like to keep the word Protocol in the title too,
> so I'd go with "Protocols: Structural subtyping (duck typing)" -- hope
> that's not too long to fit in a PEP title field.
>

OK, this looks reasonable.


>
> > Type-hints should not have runtime semantics, beyond those that they
>>> have as classes
>>> >  lots of code uses isinstance(obj, collections.abc.Iterable) and
>>> similar checks with other ABCs
>>> Having interfaces defined as something extended from abc doesn't
>>> necessitate their use at runtime, but it does open up a great deal of
>>> options for those of us who want to do so. I've been leveraging abc for a
>>> few years now to implement a lightweight version of what this PEP is
>>> attempting to achieve
>>>
>>
>> IIUC this is not the main goal of the PEP, the main goal is to provide
>> support/standard for _static_ structural subtyping.
>> Possibility to use protocols in runtime context is rather a minor bonus
>> that exists mostly to provide a seamless transition
>> for projects that already use ABCs.
>>
>
> Is something like this already in the PEP? It deserves attention in one of
> the earlier sections.
>

Yes, similar discussions appear in "Rationale and Goals", and "Existing
approaches to structural subtyping". Maybe I need to polish the text there
adding more focus on static typing.

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170529/45a445e8/attachment.html>

From solipsis at pitrou.net  Mon May 29 08:13:47 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Mon, 29 May 2017 14:13:47 +0200
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
Message-ID: <20170529141347.136ea686@fsol>


Hi,

I hope readers won't get bothered by what is mostly a link to blogpost
(well, two of them :-)), but I suspect there at least 2 or 3 people
who might be interested in the following analysis:
https://www.corsix.org/content/compilers-cpython-interpreter-main-loop
http://www.corsix.org/content/micro-optimisations-can-speed-up-cpython

Regards

Antoine.



From victor.stinner at gmail.com  Mon May 29 08:45:19 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Mon, 29 May 2017 14:45:19 +0200
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <20170529141347.136ea686@fsol>
References: <20170529141347.136ea686@fsol>
Message-ID: <CAMpsgwZT=hVMfFSxQWfStnrDFVT1pfvPGjPvijWCuDbsf+qceQ@mail.gmail.com>

Hi,

Thank you for the links.

-            PyObject *value = GETLOCAL(oparg);
+            PyObject *value = GETLOCAL((unsigned)oparg);

Oh, I remember that I proposed to change oparg type to unsigned when
Demur wrote the WORDCODE change. I even wrote a patch, but I got a
segfault and was unable to understand why. It seems like I should try
again, it seems worth it ;-)

Victor

2017-05-29 14:13 GMT+02:00 Antoine Pitrou <solipsis at pitrou.net>:
>
> Hi,
>
> I hope readers won't get bothered by what is mostly a link to blogpost
> (well, two of them :-)), but I suspect there at least 2 or 3 people
> who might be interested in the following analysis:
> https://www.corsix.org/content/compilers-cpython-interpreter-main-loop
> http://www.corsix.org/content/micro-optimisations-can-speed-up-cpython
>
> Regards
>
> Antoine.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com

From storchaka at gmail.com  Mon May 29 10:15:43 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Mon, 29 May 2017 17:15:43 +0300
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <20170529141347.136ea686@fsol>
References: <20170529141347.136ea686@fsol>
Message-ID: <oghaea$jpn$1@blaine.gmane.org>

29.05.17 15:13, Antoine Pitrou ????:
> I hope readers won't get bothered by what is mostly a link to blogpost
> (well, two of them :-)), but I suspect there at least 2 or 3 people
> who might be interested in the following analysis:
> https://www.corsix.org/content/compilers-cpython-interpreter-main-loop
> http://www.corsix.org/content/micro-optimisations-can-speed-up-cpython

Interesting articles, thank you. I wonder why the author doesn't propose 
his patches for CPython. Does he fear that CPython can become faster 
than Lua? ;-)

And the following article should be especially interesting for Victor:

https://www.corsix.org/content/why-are-slots-so-slow

The part of optimizations already are applied in 3.6 and 3.7, but `a + 
b` still is slower than `a.__add__(b)`.


From z+py+pydev at m0g.net  Mon May 29 10:36:08 2017
From: z+py+pydev at m0g.net (Guyzmo)
Date: Mon, 29 May 2017 16:36:08 +0200
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <oghaea$jpn$1@blaine.gmane.org>
References: <20170529141347.136ea686@fsol>
 <oghaea$jpn$1@blaine.gmane.org>
Message-ID: <20170529143608.6uzak54urcve7g2s@BuGz.eclipse.m0g.net>

On Mon, May 29, 2017 at 05:15:43PM +0300, Serhiy Storchaka wrote:
> Interesting articles, thank you. I wonder why the author doesn't propose his
> patches for CPython. Does he fear that CPython can become faster than Lua?
> ;-)

the author's answer: https://twitter.com/corsix/status/869200284261789696

?

Cheers,

-- 
zmo

From kevinjacobconway at gmail.com  Mon May 29 10:51:51 2017
From: kevinjacobconway at gmail.com (Kevin Conway)
Date: Mon, 29 May 2017 14:51:51 +0000
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAOMjWk==E-vpphCOADGqFk_UjyQuoaHg1+8m2T++Jh6y9ZzETA@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
 <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
 <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>
 <CAOMjWkkjmrkzws_d-TqQ+CN=CPSwJhyCejgwQMCO7mWd6fP4DA@mail.gmail.com>
 <CAP7+vJLWpeDpZshcMvtK+axz2441TKdHyP+T_MWyikJ=PNuHAQ@mail.gmail.com>
 <CAOMjWk==E-vpphCOADGqFk_UjyQuoaHg1+8m2T++Jh6y9ZzETA@mail.gmail.com>
Message-ID: <CAKF=+djkHjeYkEga=TqHA=WnGU2JJyXGtPKcXYG2=cZ=kzmQLQ@mail.gmail.com>

>From the PEP:
> The problem with them is that a class has to be explicitly marked to
support them, which is unpythonic and unlike what one would normally do in
idiomatic dynamically typed Python code.
> The same problem appears with user-defined ABCs: they must be explicitly
subclassed or registered.
Neither of these statements are entirely true. The semantics of `abc` allow
for exactly the kind of detached interfaces this PEP is attempting to
provide. The `abc.ABCMeta` provides the `__subclasshook__` which allows a
developer to override the default check of internal `abc` registry state
with virtually any logic that determines the relationship of a class with
the interface. The prior art I linked to earlier in the thread uses this
feature to generically support `issubclass` and `isinstance` in such a way
that the PEPs goal is achieved.

> The intention of this PEP is to solve all these problems by allowing
users to write the above code without explicit base classes in the class
definition
As I understand this goal, you want to take what some of us in the
community have been building ourselves and make it canonical via the
stdlib. What strikes me as odd is that the focus is on 3rd party type
checkers first rather than introducing this as a feature of the language
runtime and then updating the type checker contract to make use of it. I
see a mention of the `isinstance` check support in the postponed/rejected
ideas, but the only rationale given for it being in that category is,
generally, "there are edge cases". For example, the PEP lists this as an
edge case:
>The problem with this is instance checks could be unreliable, except for
situations where there is a common signature convention such as Iterable
However, the sample given demonstrates precisely the expected behavior of
checking if a concrete implements the protocol. It's unclear why this
sample is given as a negative. The other case given is:
> Another potentially problematic case is assignment of attributes after
instantiation
Can you elaborate on how type checkers would not encounter this same issue?
If there is a solution to this problem for type checkers, would that same
solution not work at runtime? Also, it seems odd to use a custom initialize
function rather than `__init__`. I don't think it was intentional, but this
makes it seem like a bit of a strawman that doesn't represent typical
Python code.

> Also, extensive use of ABCs might impose additional runtime costs.
I'd love to see some data around this. Given that it's a rationale for the
PEP I'd expect to see some numbers behind it. For example, is memory cost
of directly registering implementations to abc linear or worse? What is the
runtime growth pattern of isinstance or issubclass when used with heavily
registered or deeply registered abc graphs and is it different than those
calls on concrete class hierarchies? Does the cost affect anything more
than the initial evaluation of the code or, in the absence of
isinstance/issubclass checks, does it continue to have an impact on the
runtime?



On Mon, May 29, 2017 at 5:41 AM Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> On 28 May 2017 at 19:40, Guido van Rossum <guido at python.org> wrote:
>
>> On Sun, May 28, 2017 at 8:27 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
>> wrote:
>>
> [...]
>
>> Regarding the title, I'd like to keep the word Protocol in the title too,
>> so I'd go with "Protocols: Structural subtyping (duck typing)" -- hope
>> that's not too long to fit in a PEP title field.
>>
>
> OK, this looks reasonable.
>
>
>>
>> > Type-hints should not have runtime semantics, beyond those that they
>>>> have as classes
>>>> >  lots of code uses isinstance(obj, collections.abc.Iterable) and
>>>> similar checks with other ABCs
>>>> Having interfaces defined as something extended from abc doesn't
>>>> necessitate their use at runtime, but it does open up a great deal of
>>>> options for those of us who want to do so. I've been leveraging abc for a
>>>> few years now to implement a lightweight version of what this PEP is
>>>> attempting to achieve
>>>>
>>>
>>> IIUC this is not the main goal of the PEP, the main goal is to provide
>>> support/standard for _static_ structural subtyping.
>>> Possibility to use protocols in runtime context is rather a minor bonus
>>> that exists mostly to provide a seamless transition
>>> for projects that already use ABCs.
>>>
>>
>> Is something like this already in the PEP? It deserves attention in one
>> of the earlier sections.
>>
>
> Yes, similar discussions appear in "Rationale and Goals", and "Existing
> approaches to structural subtyping". Maybe I need to polish the text there
> adding more focus on static typing.
>
> --
> Ivan
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170529/a30bfe8a/attachment-0001.html>

From benhoyt at gmail.com  Mon May 29 11:42:33 2017
From: benhoyt at gmail.com (Ben Hoyt)
Date: Mon, 29 May 2017 11:42:33 -0400
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <oghaea$jpn$1@blaine.gmane.org>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
Message-ID: <CAL9jXCGucuBovUpciM8KDvVVAYgRvagatrcbT7u-Nvi-=a8TQA@mail.gmail.com>

That's very interesting -- thanks for sharing, Serhiy and Antoine. Very
relevant!

On Mon, May 29, 2017 at 10:15 AM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> 29.05.17 15:13, Antoine Pitrou ????:
>
>> I hope readers won't get bothered by what is mostly a link to blogpost
>> (well, two of them :-)), but I suspect there at least 2 or 3 people
>> who might be interested in the following analysis:
>> https://www.corsix.org/content/compilers-cpython-interpreter-main-loop
>> http://www.corsix.org/content/micro-optimisations-can-speed-up-cpython
>>
>
> Interesting articles, thank you. I wonder why the author doesn't propose
> his patches for CPython. Does he fear that CPython can become faster than
> Lua? ;-)
>
> And the following article should be especially interesting for Victor:
>
> https://www.corsix.org/content/why-are-slots-so-slow
>
> The part of optimizations already are applied in 3.6 and 3.7, but `a + b`
> still is slower than `a.__add__(b)`.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/benhoyt%
> 40gmail.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170529/87952f5d/attachment.html>

From tjreedy at udel.edu  Mon May 29 16:43:52 2017
From: tjreedy at udel.edu (Terry Reedy)
Date: Mon, 29 May 2017 16:43:52 -0400
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <20170529143608.6uzak54urcve7g2s@BuGz.eclipse.m0g.net>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <20170529143608.6uzak54urcve7g2s@BuGz.eclipse.m0g.net>
Message-ID: <ogi164$8i9$1@blaine.gmane.org>

On 5/29/2017 10:36 AM, Guyzmo via Python-Dev wrote:
> On Mon, May 29, 2017 at 05:15:43PM +0300, Serhiy Storchaka wrote:
>> Interesting articles, thank you. I wonder why the author doesn't propose his
>> patches for CPython. Does he fear that CPython can become faster than Lua?
>> ;-)
> 
> the author's answer: https://twitter.com/corsix/status/869200284261789696

"Because getting patches merged is much effort", which is sometimes true.

Peter Cawley is already bpo user 'Peter Cawley' with linked github 
account 'corsix', but no CLA.  If one or more core developers were to 
commit to reviewing his patches, one of them (or someone else) could 
email him (see bpo user profile) conveying our interest and such 
commitment(s), and request that he sign the CLA, open an issue for one 
patch, and open a git PR for the patch.

-- 
Terry Jan Reedy


From mertz at gnosis.cx  Mon May 29 18:00:41 2017
From: mertz at gnosis.cx (David Mertz)
Date: Mon, 29 May 2017 15:00:41 -0700
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
In-Reply-To: <CAEbHw4ZbNrxfdz8on2vUaCjBVzsVFo3oY1reexYTtm3OewwHeg@mail.gmail.com>
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
 <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
 <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>
 <CADiSq7dn7Ovh_BLzzwbRJMOCvRGoW5gwup4Z=ri4nUMKx4VOcw@mail.gmail.com>
 <df8962f7-ebdd-44f3-0ede-7abd7765705e@egenix.com>
 <592BB617.9060900@canterbury.ac.nz>
 <CAEbHw4ZbNrxfdz8on2vUaCjBVzsVFo3oY1reexYTtm3OewwHeg@mail.gmail.com>
Message-ID: <CAEbHw4bwHd=TuFY7o-t4eAqrV9+Kxj3-NtiFO2u-A=G4K_dOkw@mail.gmail.com>

This is a side issue, do I don't want to go too? long with it. But *NO* we
can't always give permission. The problem isn't how permissive PSF might
like to be in the abstract, but trademark law itself. Trademark is "enforce
it or lose it" ... Even passively allowing dilutive derivatives would cause
us to lose control of the mark, i.e. we would not have legal authority to
prohibit the misleading uses we actually care about preventing.

On May 29, 2017 2:32 AM, "Greg Ewing" <greg.ewing at canterbury.ac.nz> wrote:

M.-A. Lemburg wrote:

> In my role as PSF TM committee member, it's often painful to have to
> tell community members that they cannot use e.g. really nice looking
> variants of the Python logo for their projects. Let's not add more
> pain.
>

But it's always within the PSF's power to give that community
member permission to use that variant if they ask, is it not?
So you don't actually have to tell anyone that they can't
use anything.

-- 
Greg

_______________________________________________
Python-Dev mailing list
Python-Dev at python.org
https://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: https://mail.python.org/mailman/options/python-dev/mertz%40g
nosis.cx
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170529/3c60fc89/attachment.html>

From guido at python.org  Mon May 29 22:27:05 2017
From: guido at python.org (Guido van Rossum)
Date: Mon, 29 May 2017 19:27:05 -0700
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <oghaea$jpn$1@blaine.gmane.org>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
Message-ID: <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>

On Mon, May 29, 2017 at 7:15 AM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> 29.05.17 15:13, Antoine Pitrou ????:
>
>> I hope readers won't get bothered by what is mostly a link to blogpost
>> (well, two of them :-)), but I suspect there at least 2 or 3 people
>> who might be interested in the following analysis:
>> https://www.corsix.org/content/compilers-cpython-interpreter-main-loop
>> http://www.corsix.org/content/micro-optimisations-can-speed-up-cpython
>>
>
> Interesting articles, thank you. I wonder why the author doesn't propose
> his patches for CPython. Does he fear that CPython can become faster than
> Lua? ;-)
>
> And the following article should be especially interesting for Victor:
>
> https://www.corsix.org/content/why-are-slots-so-slow
>

Is the author of that article using non-standard terminology? The article
doesn't appear to be about __slots__ at all.


> The part of optimizations already are applied in 3.6 and 3.7, but `a + b`
> still is slower than `a.__add__(b)`.
>

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170529/3a146953/attachment.html>

From dw+python-dev at hmmz.org  Mon May 29 22:38:15 2017
From: dw+python-dev at hmmz.org (David Wilson)
Date: Tue, 30 May 2017 02:38:15 +0000
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
Message-ID: <20170530023815.GA14698@k3>

On Mon, May 29, 2017 at 07:27:05PM -0700, Guido van Rossum wrote:

> Is the author of that article using non-standard terminology? The article
> doesn't appear to be about __slots__ at all.

They're referred to as slots throughout typeobject.c

From steve at pearwood.info  Mon May 29 22:47:50 2017
From: steve at pearwood.info (Steven D'Aprano)
Date: Tue, 30 May 2017 12:47:50 +1000
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
Message-ID: <20170530024749.GJ23443@ando.pearwood.info>

On Mon, May 29, 2017 at 07:27:05PM -0700, Guido van Rossum wrote:

> > https://www.corsix.org/content/why-are-slots-so-slow
> >
> 
> Is the author of that article using non-standard terminology? The article
> doesn't appear to be about __slots__ at all.

Quoting Armin Ronacher:

    By far my biggest problem with the language is the stupid slot
    system. I do not mean the __slots__ but the internal type slots
    for special methods.

http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see/

Armin shows the history of these "slots" (or however they're called) 
back to Python in 1990!

static number_methods int_as_number = {
    intadd, /*tp_add*/
    intsub, /*tp_subtract*/
    intmul, /*tp_multiply*/
    ... etc

I don't know where the name "slot" comes for from the various tp_* 
members (fields? attributes? slots?) but I'm pretty sure I've heard it 
before. I don't normally pay attention to what happens in the C 
implementation, but I'm fairly sure he's referring to these tp_* 
thingies.

Oh yes, here you go:

https://docs.python.org/2/extending/newtypes.html#type-methods

refers to "tp_* slot" regularly. What does "tp" stand for? Type 
something, I guess.


-- 
Steve

From guido at python.org  Mon May 29 23:41:26 2017
From: guido at python.org (Guido van Rossum)
Date: Mon, 29 May 2017 20:41:26 -0700
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <20170530024749.GJ23443@ando.pearwood.info>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
Message-ID: <CAP7+vJJ=CVwzvD2PU7w0QZ0CFb-uSYxSphQ_6TKKcFOSrK0tcw@mail.gmail.com>

<Blush>

tp_ stands for "type". I wrote all that in 1990 and indeed made up the term
slot -- it's like an entry in a C++ vtable, where apparently they are also
called slots by some folks. I also did a major update to the machinery here
in the early 2000s (when typeobject.c grew from 50 lines to 5000) but
didn't update the terminology.

I feel a bit offended that Armin calls it "the stupid slot system" -- this
architecture an important part of what makes C extensions so flexible. I
will happily see it replaced by something better, if it can be found, but I
don't think it's fair to call it "stupid". However I no longer am
interested in maintaining that code myself, and I can't bear to read those
blog posts (they're rambling and sound like someone's research notes).

On Mon, May 29, 2017 at 7:47 PM, Steven D'Aprano <steve at pearwood.info>
wrote:

> On Mon, May 29, 2017 at 07:27:05PM -0700, Guido van Rossum wrote:
>
> > > https://www.corsix.org/content/why-are-slots-so-slow
> > >
> >
> > Is the author of that article using non-standard terminology? The article
> > doesn't appear to be about __slots__ at all.
>
> Quoting Armin Ronacher:
>
>     By far my biggest problem with the language is the stupid slot
>     system. I do not mean the __slots__ but the internal type slots
>     for special methods.
>
> http://lucumr.pocoo.org/2014/8/16/the-python-i-would-like-to-see/
>
> Armin shows the history of these "slots" (or however they're called)
> back to Python in 1990!
>
> static number_methods int_as_number = {
>     intadd, /*tp_add*/
>     intsub, /*tp_subtract*/
>     intmul, /*tp_multiply*/
>     ... etc
>
> I don't know where the name "slot" comes for from the various tp_*
> members (fields? attributes? slots?) but I'm pretty sure I've heard it
> before. I don't normally pay attention to what happens in the C
> implementation, but I'm fairly sure he's referring to these tp_*
> thingies.
>
> Oh yes, here you go:
>
> https://docs.python.org/2/extending/newtypes.html#type-methods
>
> refers to "tp_* slot" regularly. What does "tp" stand for? Type
> something, I guess.
>
>
> --
> Steve
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> guido%40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170529/8cd6cb4e/attachment.html>

From greg.ewing at canterbury.ac.nz  Tue May 30 01:56:09 2017
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 30 May 2017 17:56:09 +1200
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <20170530023815.GA14698@k3>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530023815.GA14698@k3>
Message-ID: <592D0979.5070307@canterbury.ac.nz>

David Wilson wrote:
> They're referred to as slots throughout typeobject.c

That's probably where he got the term from. But it really refers
to C-level fields in the type object. Magic methods that don't
correspond to C-level type fields are not called slots.

-- 
Greg

From greg.ewing at canterbury.ac.nz  Tue May 30 02:06:39 2017
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Tue, 30 May 2017 18:06:39 +1200
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <20170530024749.GJ23443@ando.pearwood.info>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
Message-ID: <592D0BEF.3040807@canterbury.ac.nz>

Steven D'Aprano wrote:
> What does "tp" stand for? Type something, I guess.

I think it's just short for "type". There's an old tradition
in C of giving member names a short prefix reminiscent of
the type they belong to. Not sure why, maybe someone thought
it helped readability.

-- 
Greg

From storchaka at gmail.com  Tue May 30 02:12:12 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 30 May 2017 09:12:12 +0300
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <oghaea$jpn$1@blaine.gmane.org>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
Message-ID: <ogj2fm$a0s$1@blaine.gmane.org>

29.05.17 17:15, Serhiy Storchaka ????:
> 29.05.17 15:13, Antoine Pitrou ????:
>> I hope readers won't get bothered by what is mostly a link to blogpost
>> (well, two of them :-)), but I suspect there at least 2 or 3 people
>> who might be interested in the following analysis:
>> https://www.corsix.org/content/compilers-cpython-interpreter-main-loop
>> http://www.corsix.org/content/micro-optimisations-can-speed-up-cpython
> 
> Interesting articles, thank you. I wonder why the author doesn't propose 
> his patches for CPython. Does he fear that CPython can become faster 
> than Lua? ;-)
> 
> And the following article should be especially interesting for Victor:
> 
> https://www.corsix.org/content/why-are-slots-so-slow
> 
> The part of optimizations already are applied in 3.6 and 3.7, but `a + 
> b` still is slower than `a.__add__(b)`.

See https://bugs.python.org/issue30509. All optimizations already are 
applied in 3.7 by Victor. In these microbenchmarks 3.7 is much faster 
than 2.7 and 3.5. But still there is some overhead due to excess 
intermediate levels.


From storchaka at gmail.com  Tue May 30 02:16:09 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 30 May 2017 09:16:09 +0300
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <592D0BEF.3040807@canterbury.ac.nz>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz>
Message-ID: <ogj2n3$4kl$1@blaine.gmane.org>

30.05.17 09:06, Greg Ewing ????:
> Steven D'Aprano wrote:
>> What does "tp" stand for? Type something, I guess.
> 
> I think it's just short for "type". There's an old tradition
> in C of giving member names a short prefix reminiscent of
> the type they belong to. Not sure why, maybe someone thought
> it helped readability.

In early ages of C structures didn't create namespaces, and member names 
were globals.


From ncoghlan at gmail.com  Tue May 30 07:49:19 2017
From: ncoghlan at gmail.com (Nick Coghlan)
Date: Tue, 30 May 2017 21:49:19 +1000
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
In-Reply-To: <CAEbHw4YYFkfVmjeqcEy5Sc1QpgoaZog_ozqxdLt4-MUC1KNtVg@mail.gmail.com>
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
 <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
 <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>
 <CADiSq7dn7Ovh_BLzzwbRJMOCvRGoW5gwup4Z=ri4nUMKx4VOcw@mail.gmail.com>
 <df8962f7-ebdd-44f3-0ede-7abd7765705e@egenix.com>
 <CAEbHw4YYFkfVmjeqcEy5Sc1QpgoaZog_ozqxdLt4-MUC1KNtVg@mail.gmail.com>
Message-ID: <CADiSq7eaLiXTv1uP8AdV0MtJJTZPNyXzRXZiihD8pmgtOyiDLw@mail.gmail.com>

On 29 May 2017 at 01:15, David Mertz <mertz at gnosis.cx> wrote:
> I agree with MAL and have also been on the Trademarks Committee for 8-9
> years. Protecting an actual Mark like the logo is fine, as painful as it is
> to someone's say no to an attractive derived logo. But trying to protect a
> look-and-feel is way too far down the path of evil (it's what some
> proprietary companies we want to be different from do). Moreover, when when
> that's done it's not generally as trademark but as "trade dress"
> copyright... which is a concept I opposed ethically, but know it's the usual
> legal instrument.

I'm also fine with leaving the trade dress aspect completely
unenforced - the main question is whether or not folks are OK with
PyPA using a derivative of the CPython docs theme to make PyPA-related
documentation look more like the CPython docs, and hence more
official.

I do think it's appropriate for us to ask that question, as going down
this path means we're deliberately trading on the community trust
earned by the CPython core development team over the past 20+ years
specifically in order to help make PyPA's own documentation seem more
trustworthy to others.

If we're going to borrow somebody else's reputation like that, then it
seems reasonable to me for us to offer the people whose reputation
we're borrowing the courtesy of *asking first*, regardless of what the
law says we're technically allowed to do.

Here's an alternate wording for the README that would focus on those
considerations without explicitly asking folks not to use the theme:

"Note that when adopting this theme, you're also borrowing an element
of the trust and credibility established by the CPython core
developers over the years, as well as the legal credibility arising
from their close association with the Python Software Foundation.
That's fine, and you're welcome to do so for other Python community
projects if you so choose, but please keep in mind that in doing so
you're also choosing to become a co-steward of that collective trust
:)"

Regards,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia

From victor.stinner at gmail.com  Tue May 30 07:49:17 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Tue, 30 May 2017 13:49:17 +0200
Subject: [Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
In-Reply-To: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
References: <CAMpsgwbx+rUCnLH4L5KNCHK53Gq4Vk7CzcW8a_bFjuyc4hUfKQ@mail.gmail.com>
Message-ID: <CAMpsgwbG15juAmu1094F25StG_SssGe9N4OuWKnyUmoU99nWKg@mail.gmail.com>

I wrote a first PEP draft:
https://github.com/python/peps/pull/272

Victor

2017-05-24 2:46 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:
> Hi,
>
> Would you be ok to backport ssl.MemoryBIO and ssl.SSLObject on Python
> 2.7? I can do the backport.
>
>   https://docs.python.org/dev/library/ssl.html#ssl.MemoryBIO
>
> Cory Benfield told me that it's a blocking issue for him to implement
> his PEP 543 -- A Unified TLS API for Python 2.7:
>
>   https://www.python.org/dev/peps/pep-0543/
>
> And I expect that if a new cool TLS API happens, people will want to
> use it on Python 2.7-3.6, not only on Python 3.7. Security evolves
> more quickly that the current Python release process, and people wants
> to keep their application secure.
>
> From what I understood, he wants to first implement an abstract
> MemoryBIO API (http://sans-io.readthedocs.io/ like API? I'm not sure
> about that), and then implement a socket/FD based on top of that.
> Maybe later, some implementations might have a fast-path using
> socket/FD directly.
>
> He described me his PEP and I strongly support it (sorry, I missed it
> when he posted it on python-dev), but we decided (Guido van Rossum,
> Christian Heimes, Cory Benfield and me, see the tweet below) to not
> put this in the stdlib right now, but spend more time on testing it on
> Twisted, asyncio, requests, etc. So publishing an implementation on
> PyPI was proposed instead. It seems like we agreed on a smooth plan
> (or am I wrong, Cory?).
>
>   https://twitter.com/VictorStinner/status/865467388141027329
>
> I'm quite sure that Twisted will love MemoryBIO on Python 2.7 as well,
> to implement TLS, especially on Windows using IOCP. Currently,
> external libraries (C extensions) are required.
>
> I'm not sure if the PEP 466 should be amended for that? Is a new PEP
> really needed? MemoryBIO/SSLObject are tiny. Nick (Coghlan): what do
> you think?
>
>   https://www.python.org/dev/peps/pep-0466/
>
> Victor

From mal at egenix.com  Tue May 30 07:59:30 2017
From: mal at egenix.com (M.-A. Lemburg)
Date: Tue, 30 May 2017 13:59:30 +0200
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
In-Reply-To: <CADiSq7eaLiXTv1uP8AdV0MtJJTZPNyXzRXZiihD8pmgtOyiDLw@mail.gmail.com>
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
 <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
 <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>
 <CADiSq7dn7Ovh_BLzzwbRJMOCvRGoW5gwup4Z=ri4nUMKx4VOcw@mail.gmail.com>
 <df8962f7-ebdd-44f3-0ede-7abd7765705e@egenix.com>
 <CAEbHw4YYFkfVmjeqcEy5Sc1QpgoaZog_ozqxdLt4-MUC1KNtVg@mail.gmail.com>
 <CADiSq7eaLiXTv1uP8AdV0MtJJTZPNyXzRXZiihD8pmgtOyiDLw@mail.gmail.com>
Message-ID: <9d31fb92-a7f7-d5a8-b890-16eeff5f3076@egenix.com>

On 30.05.2017 13:49, Nick Coghlan wrote:
> Here's an alternate wording for the README that would focus on those
> considerations without explicitly asking folks not to use the theme:
> 
> "Note that when adopting this theme, you're also borrowing an element
> of the trust and credibility established by the CPython core
> developers over the years, as well as the legal credibility arising
> from their close association with the Python Software Foundation.
> That's fine, and you're welcome to do so for other Python community
> projects if you so choose, but please keep in mind that in doing so
> you're also choosing to become a co-steward of that collective trust
> :)"

Much better. Thanks :-)

-- 
Marc-Andre Lemburg
eGenix.com

Professional Python Services directly from the Experts (#1, May 30 2017)
>>> Python Projects, Coaching and Consulting ...  http://www.egenix.com/
>>> Python Database Interfaces ...           http://products.egenix.com/
>>> Plone/Zope Database Interfaces ...           http://zope.egenix.com/
________________________________________________________________________

::: We implement business ideas - efficiently in both time and costs :::

   eGenix.com Software, Skills and Services GmbH  Pastor-Loeh-Str.48
    D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg
           Registered at Amtsgericht Duesseldorf: HRB 46611
               http://www.egenix.com/company/contact/
                      http://www.malemburg.com/


From solipsis at pitrou.net  Tue May 30 08:08:27 2017
From: solipsis at pitrou.net (Antoine Pitrou)
Date: Tue, 30 May 2017 14:08:27 +0200
Subject: [Python-Dev] Aligning the packaging.python.org theme with the
 rest of the docs
References: <CADiSq7fWvvh1k4pDHbfLAzQS6_mYt1GKm-hXxVpGBm+RXqpuQw@mail.gmail.com>
 <CAP1=2W7+2W7yg0ST1-Mo=ULh=n-PfWgHCk+5DkpRVqtPMqgCUA@mail.gmail.com>
 <CAP7+vJL3FVVBEyNVToWZAd-GkSy643vvCRp7BTJVHTu1rX5F0g@mail.gmail.com>
 <CADiSq7dn7Ovh_BLzzwbRJMOCvRGoW5gwup4Z=ri4nUMKx4VOcw@mail.gmail.com>
 <df8962f7-ebdd-44f3-0ede-7abd7765705e@egenix.com>
 <CAEbHw4YYFkfVmjeqcEy5Sc1QpgoaZog_ozqxdLt4-MUC1KNtVg@mail.gmail.com>
 <CADiSq7eaLiXTv1uP8AdV0MtJJTZPNyXzRXZiihD8pmgtOyiDLw@mail.gmail.com>
Message-ID: <20170530140827.1ca65f00@fsol>

On Tue, 30 May 2017 21:49:19 +1000
Nick Coghlan <ncoghlan at gmail.com> wrote:
> 
> Here's an alternate wording for the README that would focus on those
> considerations without explicitly asking folks not to use the theme:
> 
> "Note that when adopting this theme, you're also borrowing an element
> of the trust and credibility established by the CPython core
> developers over the years, as well as the legal credibility arising
> from their close association with the Python Software Foundation.

The statement about "legal credibility" sounds wishy-washy and could
lure users into thinking that they're doing something illegal by
borrowing the theme.

Also I'm not sure what is that "legal credibility" you're talking
about.  If it's about the PSF license and the Python CLA then
better to voice that explicitly, IMO.

> That's fine, and you're welcome to do so for other Python community
> projects if you so choose, but please keep in mind that in doing so
> you're also choosing to become a co-steward of that collective trust
> :)"

"Becoming a co-steward of that collective trust" sounds serious enough
(even though I don't understand what it means concretely), so why
the smiley?

Regards

Antoine.



From guido at python.org  Tue May 30 11:38:35 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 May 2017 08:38:35 -0700
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <ogj2n3$4kl$1@blaine.gmane.org>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz> <ogj2n3$4kl$1@blaine.gmane.org>
Message-ID: <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>

On Mon, May 29, 2017 at 11:16 PM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> 30.05.17 09:06, Greg Ewing ????:
>
>> Steven D'Aprano wrote:
>>
>>> What does "tp" stand for? Type something, I guess.
>>>
>>
>> I think it's just short for "type". There's an old tradition
>> in C of giving member names a short prefix reminiscent of
>> the type they belong to. Not sure why, maybe someone thought
>> it helped readability.
>>
>
> In early ages of C structures didn't create namespaces, and member names
> were globals.
>

That's nonsense. The reason is greppability.


-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170530/6e4a232d/attachment.html>

From storchaka at gmail.com  Tue May 30 14:38:55 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Tue, 30 May 2017 21:38:55 +0300
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz> <ogj2n3$4kl$1@blaine.gmane.org>
 <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>
Message-ID: <ogke7p$s6o$1@blaine.gmane.org>

30.05.17 18:38, Guido van Rossum ????:
> On Mon, May 29, 2017 at 11:16 PM, Serhiy Storchaka <storchaka at gmail.com 
> <mailto:storchaka at gmail.com>> wrote:
> 
>     30.05.17 09:06, Greg Ewing ????:
> 
>         Steven D'Aprano wrote:
> 
>             What does "tp" stand for? Type something, I guess.
> 
> 
>         I think it's just short for "type". There's an old tradition
>         in C of giving member names a short prefix reminiscent of
>         the type they belong to. Not sure why, maybe someone thought
>         it helped readability.
> 
> 
>     In early ages of C structures didn't create namespaces, and member
>     names were globals.
> 
> 
> That's nonsense. The reason is greppability.

Good reason!


From dw+python-dev at hmmz.org  Tue May 30 14:49:27 2017
From: dw+python-dev at hmmz.org (David Wilson)
Date: Tue, 30 May 2017 18:49:27 +0000
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <ogke7p$s6o$1@blaine.gmane.org>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz> <ogj2n3$4kl$1@blaine.gmane.org>
 <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>
 <ogke7p$s6o$1@blaine.gmane.org>
Message-ID: <20170530184927.GB14698@k3>

On Tue, May 30, 2017 at 09:38:55PM +0300, Serhiy Storchaka wrote:

> > In early ages of C structures didn't create namespaces, and member
> > names were globals.

> >That's nonsense. The reason is greppability.

> Good reason!

The first time I heard about prefixing struct members was to allow
tricks like this:

    x86_64-linux-gnu/bits/stat.h
    94:# define st_atime st_atim.tv_sec     /* Backward compatibility.  */
    95:# define st_mtime st_mtim.tv_sec
    96:# define st_ctime st_ctim.tv_sec

Which is relatively safe thanks to the prefix.


David

From guido at python.org  Tue May 30 17:02:02 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 May 2017 14:02:02 -0700
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CAOMjWkmmfGSEu1mjVzPYE_BT5g06RSJZB02hz0nouAcJT62J_g@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
 <CAMiohoiXuAZmzyJYBx6jVVfWj5=duLuRAMuFT6JfEC=vM+x5gw@mail.gmail.com>
 <CAOMjWkmmfGSEu1mjVzPYE_BT5g06RSJZB02hz0nouAcJT62J_g@mail.gmail.com>
Message-ID: <CAP7+vJ+dkQRYaza_Zpp+8vm_wR6tC5PXm9mYqRhFw5wY2UW=og@mail.gmail.com>

> On 15 May 2017 at 10:48, Koos Zevenhoven <k7hoven at gmail.com> wrote:
>
>> Would __annotations__ be set by the decorator? To me, not setting them
>> would seem weird, but cases where the result is not a function could
>> be weird. I also don't see a mention of this only working in stubs.
>>
>
It took me a while to realize which annotations we're talking about here.

I like Jukka's version, as it has a clear distinction between
>> functions and other attributes. But that might require a language
>> change to provide __annotations__ in a clean manner? Maybe that
>> language change would be useful elsewhere.
>
>
I'm not interesting in language changes to solve this problem.

On Mon, May 15, 2017 at 6:20 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> With original syntax it is possible to overwrite annotations without
> language changes.
> However with Jukka's syntax it looks difficult. A possible pure-Python way
> could be
> to insert an item in an enclosing __annotations__, if an enclosing scope
> is a class or module scope.
>

I assume we're talking about `session.__annotations__`, which will be
overwritten by the last definition (i.e. the "implementation"). This is
only a runtime problem, and we're actually still a bit better off here than
for @overload -- for @overload, the implementation is often un-annotated or
has very vague annotations (e.g. `*args: Any`), while here the annotations
are in principle preserved by the decorators on the implementation
function. (The main premise of the proposal under discussion is to clarify
the type for the *human* reader, although in some edge cases the explicit
annotation may provide a constraint for the naturally inferred type of the
decorated function.)

But in either case the annotations on the overload variants or the
"declaration" are lost at runtime by the trick of using multiple
definitions, since at runtime the last definition wins. (And we don't want
to go through the contortions used by e.g. @property for collecting getters
and setters belonging to a single attribute.)

The question then is, do we care more about the runtime accessibility of
the type, or do we care more about ease of use in a context of static
analysis? In general with the design of PEP 484 I've tried to balance the
two (and several other constraints like no changes to the core Python
language).

I'm still not decided. Let's compare the two proposals quickly. In Naomi's
version you'd write

@decorated_type(Callable[[str], ContextManager[DbSession]])
@contextmanager
def session(url: str) -> Iterator[DbSession]:
    <body>

while in Jukka's version you'd write

@declared_type
def session(url: str) -> ContextManager[DbSession]: ...

@contextmanager
def session(url: str) -> Iterator[DbSession]:
    <body>

There's also a difference in power between the two: in Naomi's proposal,
session's type becomes exactly

    Callable[[str], ContextManager[DbSession]]

which can only be called with positional arguments, whereas in Jukka's
proposal session will retain the name and clasiffication (e.g.
keyword-only, varargs) of the arguments. If those are important, in Naomi's
case this can be written using the new "argspecs" form of Callable, i.e.

@decorated_type(Callable[[Arg(str, 'url')], ContextManager[DbSession])
@contextmanager
def session <etc.>

which is definitely more verbose and uglier than Jukka's version. OTOH it
does mean that Naomi's proposal has all the *power* of Jukka's version --
and in simple cases it is actually less verbose than Jukka's proposal (in
cases where you want all that power -- if you're happy with a basic
callable it's actually *less* verbose).

Jukka's version also needs another special case where the decorator returns
something that's not a callable. In Naomi's proposal this would simply be

@decorated_type(SomeType)
@some_decorator
def my_thing(url: str) -> int:
    <etc.>

Jukka's proposal *seems* to be to just allow overriding a variable
declaration with a function declararion:

my_thing: SomeType

@some_decorator
def my_thing <etc.>

This seems fragile to me (since it looks like an accidental redefinition,
although typically the static check would find a discrepancy between the
inferred type of the first and the second definition).

All in all I'm still leaning towards Naomi's original proposal -- it looks
simpler to implement as well.

-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170530/a3c73c54/attachment.html>

From guido at python.org  Tue May 30 17:18:34 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 May 2017 14:18:34 -0700
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <20170530184927.GB14698@k3>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz> <ogj2n3$4kl$1@blaine.gmane.org>
 <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>
 <ogke7p$s6o$1@blaine.gmane.org> <20170530184927.GB14698@k3>
Message-ID: <CAP7+vJJrdD=3+dd5iB8+=ta0wpNyA32xjjxr2fVNQ-YwthvfOA@mail.gmail.com>

On Tue, May 30, 2017 at 11:49 AM, David Wilson <dw+python-dev at hmmz.org>
wrote:

> On Tue, May 30, 2017 at 09:38:55PM +0300, Serhiy Storchaka wrote:
>
> > > In early ages of C structures didn't create namespaces, and member
> > > names were globals.
>
> > >That's nonsense. The reason is greppability.
>
> > Good reason!
>
> The first time I heard about prefixing struct members was to allow
> tricks like this:
>
>     x86_64-linux-gnu/bits/stat.h
>     94:# define st_atime st_atim.tv_sec     /* Backward compatibility.  */
>     95:# define st_mtime st_mtim.tv_sec
>     96:# define st_ctime st_ctim.tv_sec
>
> Which is relatively safe thanks to the prefix.
>

Believe me that is not why the prefix convention was introduced.

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170530/45aee90a/attachment.html>

From ja.py at farowl.co.uk  Tue May 30 18:04:41 2017
From: ja.py at farowl.co.uk (Jeff Allen)
Date: Tue, 30 May 2017 23:04:41 +0100
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz> <ogj2n3$4kl$1@blaine.gmane.org>
 <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>
Message-ID: <341a83fd-9456-2986-945b-47a2f7e15e01@farowl.co.uk>

On 30/05/2017 16:38, Guido van Rossum wrote:
> On Mon, May 29, 2017 at 11:16 PM, Serhiy Storchaka 
> <storchaka at gmail.com <mailto:storchaka at gmail.com>> wrote:
>
>     30.05.17 09:06, Greg Ewing ????:
>
>         Steven D'Aprano wrote:
>
>             What does "tp" stand for? Type something, I guess.
>
>
>         I think it's just short for "type". There's an old tradition
>         in C of giving member names a short prefix reminiscent of
>         the type they belong to. Not sure why, maybe someone thought
>         it helped readability.
>
>
>     In early ages of C structures didn't create namespaces, and member
>     names were globals.
>
>
> That's nonsense. The reason is greppability.
>
It does seem that far enough back, struct member names were all one 
space, standing for little more than their offset and type:

"Two structures may share a common initial sequence of members; that is, 
the same member may appear in two different structures if it has the 
same type in both and if all previous members are the same in both. 
(Actually, the compiler checks only that a name in two different 
structures has the same type and offset in both, ... )" -- The C 
Programming Language, K&R 1978 (p197).

With these Python name spaces, you're really spoiling us, Mr BDFL.

Jeff Allen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170530/c40491c8/attachment.html>

From greg.ewing at canterbury.ac.nz  Tue May 30 18:47:00 2017
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Wed, 31 May 2017 10:47:00 +1200
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <ogj2n3$4kl$1@blaine.gmane.org>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz> <ogj2n3$4kl$1@blaine.gmane.org>
Message-ID: <592DF664.2040107@canterbury.ac.nz>

Serhiy Storchaka wrote:
> In early ages of C structures didn't create namespaces, and member names 
> were globals.

That would certainly explain the origins of it, but I'm
pretty sure it wasn't the case by the time Python was
invented. So Guido must have liked it for other reasons.

-- 
Greg

From guido at python.org  Tue May 30 18:58:05 2017
From: guido at python.org (Guido van Rossum)
Date: Tue, 30 May 2017 15:58:05 -0700
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAKF=+djkHjeYkEga=TqHA=WnGU2JJyXGtPKcXYG2=cZ=kzmQLQ@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
 <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
 <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>
 <CAOMjWkkjmrkzws_d-TqQ+CN=CPSwJhyCejgwQMCO7mWd6fP4DA@mail.gmail.com>
 <CAP7+vJLWpeDpZshcMvtK+axz2441TKdHyP+T_MWyikJ=PNuHAQ@mail.gmail.com>
 <CAOMjWk==E-vpphCOADGqFk_UjyQuoaHg1+8m2T++Jh6y9ZzETA@mail.gmail.com>
 <CAKF=+djkHjeYkEga=TqHA=WnGU2JJyXGtPKcXYG2=cZ=kzmQLQ@mail.gmail.com>
Message-ID: <CAP7+vJJY5p8nT+kqj5Mix6Qu_MD6wQNMitPMjL_F=Eahmm+zJQ@mail.gmail.com>

[I added some blank lines to separate the PEP quotes from Kevin's
responses.]

On Mon, May 29, 2017 at 7:51 AM, Kevin Conway <kevinjacobconway at gmail.com>
wrote:

> From the PEP:
> > The problem with them is that a class has to be explicitly marked to
> support them, which is unpythonic and unlike what one would normally do in
> idiomatic dynamically typed Python code.
> > The same problem appears with user-defined ABCs: they must be explicitly
> subclassed or registered.
>
> Neither of these statements are entirely true. The semantics of `abc`
> allow for exactly the kind of detached interfaces this PEP is attempting to
> provide. The `abc.ABCMeta` provides the `__subclasshook__` which allows a
> developer to override the default check of internal `abc` registry state
> with virtually any logic that determines the relationship of a class with
> the interface. The prior art I linked to earlier in the thread uses this
> feature to generically support `issubclass` and `isinstance` in such a way
> that the PEPs goal is achieved.
>

But that doesn't help a static type checker. You can't expect a static
checker to understand the implementation of a particular
`__subclasshook__`. In practice, except for a few "one-trick ponies" such
as Hashable, existing ABCs rely on subclassing or registration to make
isinstance() work, and for statically checking code that uses duck typing
those aren't enough.


> > The intention of this PEP is to solve all these problems by allowing
> users to write the above code without explicit base classes in the class
> definition
>
> As I understand this goal, you want to take what some of us in the
> community have been building ourselves and make it canonical via the
> stdlib.
>

Not really. The goal is to suggest implementation a frequently requested
feature in static checkers based on the type system laid out in PEP 484,
namely checks for duck types. To support this the type system from PEP 484
needs to be extended, and that's what PEP 544 is about.


> What strikes me as odd is that the focus is on 3rd party type checkers
> first rather than introducing this as a feature of the language runtime and
> then updating the type checker contract to make use of it.
>

This seems to be a misunderstanding, or at least an attempt to change the
agenda for PEP 544. The primary goal of the PEP is not to support runtime
checking but static checking. This is not new -- PEP 484 and PEP 526 before
it have also focused on features that are useful primarily for static
checkers. (Also, a bit of history: PEP 484 intentionally focused on static
checking support because there was widespread skepticism about the need for
more runtime checking, but there was a subset of the community that was
very interested in static checking.)


> I see a mention of the `isinstance` check support in the
> postponed/rejected ideas, but the only rationale given for it being in that
> category is, generally, "there are edge cases". For example, the PEP lists
> this as an edge case:
>
> >The problem with this is instance checks could be unreliable, except for
> situations where there is a common signature convention such as Iterable
>
> However, the sample given demonstrates precisely the expected behavior of
> checking if a concrete implements the protocol. It's unclear why this
> sample is given as a negative.
>

I assume we're talking about this example:

  class P(Protocol):
      def common_method_name(self, x: int) -> int: ...

  class X:
      <a bunch of methods>
      def common_method_name(self) -> None: ... # Note different signature

  def do_stuff(o: Union[P, X]) -> int:
      if isinstance(o, P):
          return o.common_method_name(1)  # oops, what if it's an X
instance?

The problem there is that the "state of the art" for runtiming checking
isinstance(o, P) boils down to hasattr(o, 'common_method_name') while the
type checker takes the method signatures into account, so it will consider
X objects not to be instances of P.

The other case given is:
>
> > Another potentially problematic case is assignment of attributes after
> instantiation
>
> Can you elaborate on how type checkers would not encounter this same
> issue? If there is a solution to this problem for type checkers, would that
> same solution not work at runtime? Also, it seems odd to use a custom
> initialize function rather than `__init__`. I don't think it was
> intentional, but this makes it seem like a bit of a strawman that doesn't
> represent typical Python code.
>

Lots of code I've seen initializes variables in a separate function
(usually called from `__init__`). Mypy, at least, considers instance
variables assigned through `self` in all methods of a class to be potential
instance variable declarations, otherwise a lot of code could not be
type-checked.

Again, the example is problematic given that the runtime check for
isinstance(c, P) can't do better than hasattr(c, 'x').  (I think there's a
typo in the PEP here, 'c1' should be 'c'.)

The need to use an explicit class decorator to add isinstance support is
used as a way to encourage developers to think about whether the runtime
instance check will match the picture as seen by the static checker, before
they turn on this decorator. It seems reasonable to me.

> Also, extensive use of ABCs might impose additional runtime costs.
>
> I'd love to see some data around this. Given that it's a rationale for the
> PEP I'd expect to see some numbers behind it. For example, is memory cost
> of directly registering implementations to abc linear or worse? What is the
> runtime growth pattern of isinstance or issubclass when used with heavily
> registered or deeply registered abc graphs and is it different than those
> calls on concrete class hierarchies? Does the cost affect anything more
> than the initial evaluation of the code or, in the absence of
> isinstance/issubclass checks, does it continue to have an impact on the
> runtime?
>

Its commonly known that ABCs are expensive (though if you want to do
precise measurements you're welcome). Here's one data point:
https://github.com/python/mypy/commit/1be4db7ac6e06a162355c3d5f7794d21b89a1056
-- it's a one-line diff that removes `metaclass=ABCMeta` from one class,
and the commit message reads:

Make the AST classes not ABCs

This results in roughly a 20% speedup on the non-parsing steps.
Here are the timings I got from running mypy on itself:

Before the change:
3861.8ms (49.0%) SemanticallyAnalyzedFile
2760.3ms (35.0%) UnprocessedFile
1111.8ms (14.1%) ParsedFile
 142.8ms ( 1.8%) PartiallySemanticallyAnalyzedFile

After the change:
3086.1ms (45.1%) SemanticallyAnalyzedFile
2665.1ms (39.0%) UnprocessedFile
 945.1ms (13.8%) ParsedFile
 139.6ms ( 2.0%) PartiallySemanticallyAnalyzedFile


-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170530/9c4ebd7c/attachment-0001.html>

From gvanrossum at gmail.com  Tue May 30 19:15:50 2017
From: gvanrossum at gmail.com (Guido van Rossum)
Date: Tue, 30 May 2017 16:15:50 -0700
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <592DF664.2040107@canterbury.ac.nz>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz>
 <ogj2n3$4kl$1@blaine.gmane.org> <592DF664.2040107@canterbury.ac.nz>
Message-ID: <CAP7+vJLTBUovZJ4yCgeRyrzPFLfuee3qzkWi7fnARi++=0ST9w@mail.gmail.com>

But I do stand corrected. I had forgotten that.

On May 30, 2017 3:49 PM, "Greg Ewing" <greg.ewing at canterbury.ac.nz> wrote:

> Serhiy Storchaka wrote:
>
>> In early ages of C structures didn't create namespaces, and member names
>> were globals.
>>
>
> That would certainly explain the origins of it, but I'm
> pretty sure it wasn't the case by the time Python was
> invented. So Guido must have liked it for other reasons.
>
> --
> Greg
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170530/d81f4a76/attachment.html>

From jcronin at egh.com  Tue May 30 19:06:10 2017
From: jcronin at egh.com (Jonathan Cronin)
Date: Tue, 30 May 2017 19:06:10 -0400
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz> <ogj2n3$4kl$1@blaine.gmane.org>
 <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>
Message-ID: <C0AC56DC-3E28-43F3-A275-4697BA9BA355@egh.com>

An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170530/028dd5a9/attachment.html>

From v+python at g.nevcal.com  Tue May 30 21:30:56 2017
From: v+python at g.nevcal.com (Glenn Linderman)
Date: Tue, 30 May 2017 18:30:56 -0700
Subject: [Python-Dev] "Micro-optimisations can speed up CPython"
In-Reply-To: <CAP7+vJJrdD=3+dd5iB8+=ta0wpNyA32xjjxr2fVNQ-YwthvfOA@mail.gmail.com>
References: <20170529141347.136ea686@fsol> <oghaea$jpn$1@blaine.gmane.org>
 <CAP7+vJJNVD11m79FhEKsKU_eiBr0_=ZG3TrbRBGQFkGf02Rmuw@mail.gmail.com>
 <20170530024749.GJ23443@ando.pearwood.info>
 <592D0BEF.3040807@canterbury.ac.nz> <ogj2n3$4kl$1@blaine.gmane.org>
 <CAP7+vJJnckZ0Pd1zQ89Y1viBxwhbmpZh2Sg9FTurea=Yat9scQ@mail.gmail.com>
 <ogke7p$s6o$1@blaine.gmane.org> <20170530184927.GB14698@k3>
 <CAP7+vJJrdD=3+dd5iB8+=ta0wpNyA32xjjxr2fVNQ-YwthvfOA@mail.gmail.com>
Message-ID: <77335d1d-0c30-5ee3-a348-d21dca345d9f@g.nevcal.com>

On 5/30/2017 2:18 PM, Guido van Rossum wrote:
> On Tue, May 30, 2017 at 11:49 AM, David Wilson <dw+python-dev at hmmz.org 
> <mailto:dw+python-dev at hmmz.org>> wrote:
>
>     On Tue, May 30, 2017 at 09:38:55PM +0300, Serhiy Storchaka wrote:
>
>     > > In early ages of C structures didn't create namespaces, and member
>     > > names were globals.
>
>     > >That's nonsense. The reason is greppability.
>
>     > Good reason!
>
>     The first time I heard about prefixing struct members was to allow
>     tricks like this:
>
>         x86_64-linux-gnu/bits/stat.h
>         94:# define st_atime st_atim.tv_sec     /* Backward
>     compatibility.  */
>         95:# define st_mtime st_mtim.tv_sec
>         96:# define st_ctime st_ctim.tv_sec
>
>     Which is relatively safe thanks to the prefix.
>
>
> Believe me that is not why the prefix convention was introduced.

Sure, but grepping was not the reason either. Serhiy had the right 
explanation.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170530/9ecd73ee/attachment.html>

From victor.stinner at gmail.com  Wed May 31 03:42:31 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 31 May 2017 09:42:31 +0200
Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to
 Python 2.7
Message-ID: <CAMpsgwY7sZYE1kB3F3AfK9HCeob8ufi_cSwGOYK0QXtO4qwBhQ@mail.gmail.com>

Hi,

I wrote a PEP based on the previous thread "Backport ssl.MemoryBIO on
Python 2.7?". Thanks for Cory Benfield, Alex Gaynor and Nick Coghlan
who helped me to write it!

HTML version:
https://www.python.org/dev/peps/pep-0546/

Inline verison below.

Victor


PEP: 546
Title: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>,
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 30-May-2017


Abstract
========

Backport the ssl.MemoryBIO and ssl.SSLObject classes from Python 3 to Python
2.7 to enhance the overall security of Python 2.7.


Rationale
=========

While Python 2.7 is getting closer to its end-of-support date (scheduled for
2020), it is still used on production systems and the Python community is still
responsible for its security. This PEP will help facilitate the future adoption
of :pep:`543` across all supported Python versions, which will improve security
for both Python 2 and Python 3 users.

This PEP does NOT propose a general exception for backporting new
features to Python 2.7 - every new feature proposed for backporting will
still need to be justified independently. In particular, it will need to
be explained why relying on an independently updated backport on the
Python Package Index instead is not an acceptable solution.


PEP 543
-------

:pep:`543` defines a new TLS API for Python which would enhance Python
security by giving Python applications access to the native TLS implementations
on Windows and macOS, instead of using OpenSSL. A side effect is that it gives
access to the system trust store and certificates installed
locally by system administrators, enabling Python applications to use "company
certificates" without having to modify each application and so to correctly
validate TLS certificates (instead of having to ignore or bypass TLS
certificate validation).

For practical reasons, Cory Benfield would like to first implement an
I/O-less class similar to ssl.MemoryBIO and ssl.SSLObject for
:pep:`543`, and to provide a second class based on the first one to use
sockets or file descriptors.  This design would help to structure the code
to support more backends and simplify testing and auditing, as well as
implementation. Later, optimized classes using directly sockets or file
descriptors may be added for performance.

While :pep:`543` defines an API, the PEP would only make sense if it
comes with at least one complete and good implementation. The first
implementation would ideally be based on the ``ssl`` module of the Python
standard library, as this is shipped to all users by default and can be used as
a fallback implementation in the absence of anything more targetted.

If this backport is not performed, the only baseline implementation that could
be used would be pyOpenSSL. This is problematic, however, because of the
interaction with pip, which is shipped with CPython on all supported versions.


requests, pip and ensurepip
---------------------------

There are plans afoot to look at moving Requests to a more event-loop-y
model, and doing so basically mandates a MemoryBIO. In the absence of a
Python 2.7 backport, Requests is required to basically use the same
solution that Twisted currently does: namely, a mandatory dependency on
`pyOpenSSL <https://pypi.python.org/pypi/pyOpenSSL>`_.

The `pip <https://pip.pypa.io/>`_ program has to embed all its
dependencies for practical reasons: namely, that it cannot rely on any other
installation method being present. Since pip depends on requests, it means
that it would have to embed a copy of pyOpenSSL. That would imply substantial
usability pain to install pip. Currently, pip doesn't support embedding
C extensions which must be compiled on each platform and so require a C
compiler.

Since Python 2.7.9, Python embeds a copy of pip both for default
installation and for use in virtual environments via the new ``ensurepip``
module. If pip ends up bundling PyOpenSSL, then CPython will end up
bundling PyOpenSSL. Only backporting ``ssl.MemoryBIO`` and
``ssl.SSLObject`` would avoid the need to embed pyOpenSSL, and would fix the
bootstrap issue (python -> ensurepip -> pip -> requests -> MemoryBIO).


Changes
=======

Add ``MemoryBIO`` and ``SSLObject`` classes to the ``ssl`` module of
Python 2.7.

The code will be backported and adapted from the master branch
(Python 3).

The backport also significantly reduced the size of the Python 2/Python
3 difference of the ``_ssl`` module, which make maintenance easier.


Links
=====

* :pep:`543`
* `[backport] ssl.MemoryBIO
  <https://bugs.python.org/issue22559>`_: Implementation of this PEP
  written by Alex Gaynor (first version written at October 2014)
* :pep:`466`


Discussions
===========

* `[Python-Dev] Backport ssl.MemoryBIO on Python 2.7?
  <https://mail.python.org/pipermail/python-dev/2017-May/147981.html>`_
  (May 2017)


Copyright
=========

This document has been placed in the public domain.

From levkivskyi at gmail.com  Wed May 31 05:16:10 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 31 May 2017 11:16:10 +0200
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAP7+vJJY5p8nT+kqj5Mix6Qu_MD6wQNMitPMjL_F=Eahmm+zJQ@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
 <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
 <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>
 <CAOMjWkkjmrkzws_d-TqQ+CN=CPSwJhyCejgwQMCO7mWd6fP4DA@mail.gmail.com>
 <CAP7+vJLWpeDpZshcMvtK+axz2441TKdHyP+T_MWyikJ=PNuHAQ@mail.gmail.com>
 <CAOMjWk==E-vpphCOADGqFk_UjyQuoaHg1+8m2T++Jh6y9ZzETA@mail.gmail.com>
 <CAKF=+djkHjeYkEga=TqHA=WnGU2JJyXGtPKcXYG2=cZ=kzmQLQ@mail.gmail.com>
 <CAP7+vJJY5p8nT+kqj5Mix6Qu_MD6wQNMitPMjL_F=Eahmm+zJQ@mail.gmail.com>
Message-ID: <CAOMjWknzvtBpWGHTXmbEUz3AA6yaENONnRaAkq_HjKre8g5xWA@mail.gmail.com>

On 31 May 2017 at 00:58, Guido van Rossum <guido at python.org> wrote:
[...]

Thank you for very detailed answers! I have practically nothing to add.
It seems to me that most of the Kevin's questions stem from unnecessary
focus
on runtime type checking. Here are two ideas about how to fix this:

* Add the word "static" somewhere in the PEP title.
* Add a short note at the start mentioning this is an extension of the type
system proposed in PEP 484 and recommending to read PEP 484 first.

What do you think?

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/96607dca/attachment.html>

From storchaka at gmail.com  Wed May 31 07:00:22 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 31 May 2017 14:00:22 +0300
Subject: [Python-Dev] Put token information in one place
Message-ID: <ogm7o1$qf5$1@blaine.gmane.org>

Currently when you add a new token you need to change a couple of files:

* Include/token.h
* _PyParser_TokenNames in Parser/tokenizer.c
* PyToken_OneChar(), PyToken_TwoChars() or PyToken_ThreeChars() in 
Parser/tokenizer.c
* Lib/token.py (generated from Include/token.h)
* EXACT_TOKEN_TYPES in Lib/tokenize.py
* Operator, Bracket or Special in Lib/tokenize.py
* Doc/library/token.rst

It is possible to generate all this information from a single source. 
Proposed in [1] patch uses Lib/token.py as an initial source. But maybe 
Lib/token.py also should be generated from some file in general format? 
Some information can be derived from Grammar/Grammar, but not all. 
Needed also a mapping between token strings ('(' or '>=') and names 
(LPAR, GREATEREQUAL). Can this be added in Grammar/Grammar or a new file?

There is a related problem, the tokenize module uses three additional 
tokens not used by the C tokenizer. It modifies the content of the token 
module after importing it, that is not good. [2] One of solutions is 
making a copy of tok_names in tokenize before modifying it, but this 
doesn't work, because third-party code search tokenize constants in 
token.tok_names. Other solution is adding tokenize specific constants to 
the token module. Is this good to expose in the token module tokens not 
used in the C tokenizer?

Non-terminal symbols are generated automatically, Lib/symbol.py from 
Include/graminit.h, and Include/graminit.h and Python/graminit.c from 
Grammar/Grammar by Parser/pgen. Is it worth to generate Lib/symbol.py by 
pgen too? Can pgen be implemented in Python?

See also similar issue for opcodes. [3]

[1] https://bugs.python.org/issue30455
[2] https://bugs.python.org/issue25324
[3] https://bugs.python.org/issue17861


From songofacandy at gmail.com  Wed May 31 07:26:30 2017
From: songofacandy at gmail.com (INADA Naoki)
Date: Wed, 31 May 2017 20:26:30 +0900
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAOMjWknzvtBpWGHTXmbEUz3AA6yaENONnRaAkq_HjKre8g5xWA@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
 <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
 <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>
 <CAOMjWkkjmrkzws_d-TqQ+CN=CPSwJhyCejgwQMCO7mWd6fP4DA@mail.gmail.com>
 <CAP7+vJLWpeDpZshcMvtK+axz2441TKdHyP+T_MWyikJ=PNuHAQ@mail.gmail.com>
 <CAOMjWk==E-vpphCOADGqFk_UjyQuoaHg1+8m2T++Jh6y9ZzETA@mail.gmail.com>
 <CAKF=+djkHjeYkEga=TqHA=WnGU2JJyXGtPKcXYG2=cZ=kzmQLQ@mail.gmail.com>
 <CAP7+vJJY5p8nT+kqj5Mix6Qu_MD6wQNMitPMjL_F=Eahmm+zJQ@mail.gmail.com>
 <CAOMjWknzvtBpWGHTXmbEUz3AA6yaENONnRaAkq_HjKre8g5xWA@mail.gmail.com>
Message-ID: <CAEfz+TzO=uogTYAJG9K9aopku0-gDT4ahLAwC9bxHJEZaf6Zew@mail.gmail.com>

Hi,

I'm interested in startup time too, not only execution time.
Here is very rough test:


with open('with_abc.py', 'w') as f:
    print("import abc", file=f)
    for i in range(1, 1001):
        print(f"class A{i}(metaclass=abc.ABCMeta): pass", file=f)

with open('without_abc.py', 'w') as f:
    print("import abc", file=f)
    for i in range(1, 1001):
        print(f"class A{i}: pass", file=f)


$  time python3 -c 'import abc'

real 0m0.051s
user 0m0.035s
sys 0m0.013s

$  time python3 -c 'import with_abc'

real 0m0.083s
user 0m0.063s
sys 0m0.017s

$  time python3 -c 'import without_abc'

real 0m0.055s
user 0m0.042s
sys 0m0.011s


It seems 1000 ABC classes takes less than 30ms but 1000 normal
classes takes less than 10ms.

I don't know this penalty is acceptable or not.
But how about making ABC optional?


I don't want to use ABC so frequently when there is no real requirement of ABC.

ABC implementation is very complex and sometimes ABC cause unexpected
performance issue, like you fixed in https://github.com/python/typing/pull/383

If we start with "Protocol is always ABC" and we face unexpected
performance penalty later, it may be difficult to find and optimize it.

# If we can stop using ABC for io.IOBase, Python startup time will be
few ms faster.
# Maybe, I should implement weakset and abc in  C.

Regards,

From levkivskyi at gmail.com  Wed May 31 09:16:17 2017
From: levkivskyi at gmail.com (Ivan Levkivskyi)
Date: Wed, 31 May 2017 15:16:17 +0200
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CAP7+vJ+dkQRYaza_Zpp+8vm_wR6tC5PXm9mYqRhFw5wY2UW=og@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
 <CAMiohoiXuAZmzyJYBx6jVVfWj5=duLuRAMuFT6JfEC=vM+x5gw@mail.gmail.com>
 <CAOMjWkmmfGSEu1mjVzPYE_BT5g06RSJZB02hz0nouAcJT62J_g@mail.gmail.com>
 <CAP7+vJ+dkQRYaza_Zpp+8vm_wR6tC5PXm9mYqRhFw5wY2UW=og@mail.gmail.com>
Message-ID: <CAOMjWkngwYOWBFJnfFsjzOc2E1MqAR5KF23B6RBvsNudsOruYg@mail.gmail.com>

On 30 May 2017 at 23:02, Guido van Rossum <guido at python.org> wrote:

>
> All in all I'm still leaning towards Naomi's original proposal -- it looks
> simpler to implement as well.
>
>
OK, I think having a bit of verbosity is absolutely fine if we win
simplicity of implementation (for both static and runtime purposes).

--
Ivan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/dd038edf/attachment.html>

From cory at lukasa.co.uk  Wed May 31 09:25:22 2017
From: cory at lukasa.co.uk (Cory Benfield)
Date: Wed, 31 May 2017 14:25:22 +0100
Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to
 Python 2.7
In-Reply-To: <CAMpsgwY7sZYE1kB3F3AfK9HCeob8ufi_cSwGOYK0QXtO4qwBhQ@mail.gmail.com>
References: <CAMpsgwY7sZYE1kB3F3AfK9HCeob8ufi_cSwGOYK0QXtO4qwBhQ@mail.gmail.com>
Message-ID: <ECFACE06-24DF-49DD-B5D3-C82A61A9951F@lukasa.co.uk>


> On 31 May 2017, at 08:42, Victor Stinner <victor.stinner at gmail.com> wrote:
> 
> Hi,
> 
> I wrote a PEP based on the previous thread "Backport ssl.MemoryBIO on
> Python 2.7?". Thanks for Cory Benfield, Alex Gaynor and Nick Coghlan
> who helped me to write it!

It probably goes without saying, given that I helped with the drafting for the PEP, but I?m strongly in favour of this PEP. Just in case it helps to get that reaffirmation here. ;)

Cory


From victor.stinner at gmail.com  Wed May 31 10:11:39 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 31 May 2017 16:11:39 +0200
Subject: [Python-Dev] PEP 7 and braces { .... } on if
Message-ID: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>

Hi,

I have a question on the CPython coding code for C code, the PEP 7:
https://www.python.org/dev/peps/pep-0007/

"""
Code structure: (...); braces are strongly preferred but may be
omitted where C permits, and they should be formatted as shown:

if (mro != NULL) {
    ...
}
else {
    ...
}
"""

This section was updated after the creation of CPython in 1991, so as
you may expect, a lot of existing C code doesn't respect this coding
style. I'm trying to slowly add "missing" braces and indent as the
example *when I have to modify existing code*.

The problem is the "strongly preferred" and "omitted where C permits"
part. I would like to make the PEP 7 more explicit on that part since
in each review, I disagree with Serhiy who wants to omit them.
Example:

if (func == NULL)
    return NULL;

versus

if (func == NULL) {
    return NULL;
}

I now prefer the version with braces. It's more verbose, but it
prevents bugs when the code is modified later for whatever reasons.
IMHO it also makes the code easily to read and to understand.

So I would suggest to modify the PEP 7 to *always* require braces for if.

I would also suggest to require braces on "for(...) { ... }" and
"while(...) { ... }". But only if the code has to be modified, not
only to update the coding style.

Changes which are pure coding style changes should be rejected to
avoid conflicts with other pending pull requests and "code churn".

Victor

From victor.stinner at gmail.com  Wed May 31 10:13:58 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 31 May 2017 16:13:58 +0200
Subject: [Python-Dev] PEP 7 and braces { .... } on if
In-Reply-To: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
References: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
Message-ID: <CAMpsgwb=maZf_jxqd+N18xGUcYSwtvfKRAJg=utiRrojEdkN9w@mail.gmail.com>

Previous discussion which added "strongly preferred" to the PEP 7, January 2016:
https://mail.python.org/pipermail/python-dev/2016-January/142746.html

Victor

2017-05-31 16:11 GMT+02:00 Victor Stinner <victor.stinner at gmail.com>:
> Hi,
>
> I have a question on the CPython coding code for C code, the PEP 7:
> https://www.python.org/dev/peps/pep-0007/
>
> """
> Code structure: (...); braces are strongly preferred but may be
> omitted where C permits, and they should be formatted as shown:
>
> if (mro != NULL) {
>     ...
> }
> else {
>     ...
> }
> """
>
> This section was updated after the creation of CPython in 1991, so as
> you may expect, a lot of existing C code doesn't respect this coding
> style. I'm trying to slowly add "missing" braces and indent as the
> example *when I have to modify existing code*.
>
> The problem is the "strongly preferred" and "omitted where C permits"
> part. I would like to make the PEP 7 more explicit on that part since
> in each review, I disagree with Serhiy who wants to omit them.
> Example:
>
> if (func == NULL)
>     return NULL;
>
> versus
>
> if (func == NULL) {
>     return NULL;
> }
>
> I now prefer the version with braces. It's more verbose, but it
> prevents bugs when the code is modified later for whatever reasons.
> IMHO it also makes the code easily to read and to understand.
>
> So I would suggest to modify the PEP 7 to *always* require braces for if.
>
> I would also suggest to require braces on "for(...) { ... }" and
> "while(...) { ... }". But only if the code has to be modified, not
> only to update the coding style.
>
> Changes which are pure coding style changes should be rejected to
> avoid conflicts with other pending pull requests and "code churn".
>
> Victor

From barry at python.org  Wed May 31 11:04:57 2017
From: barry at python.org (Barry Warsaw)
Date: Wed, 31 May 2017 11:04:57 -0400
Subject: [Python-Dev] PEP 7 and braces { .... } on if
In-Reply-To: <CAMpsgwb=maZf_jxqd+N18xGUcYSwtvfKRAJg=utiRrojEdkN9w@mail.gmail.com>
References: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
 <CAMpsgwb=maZf_jxqd+N18xGUcYSwtvfKRAJg=utiRrojEdkN9w@mail.gmail.com>
Message-ID: <20170531110457.5b4ab7ac@subdivisions.wooz.org>

On May 31, 2017, at 04:13 PM, Victor Stinner wrote:

>Previous discussion which added "strongly preferred" to the PEP 7, January
>2016: https://mail.python.org/pipermail/python-dev/2016-January/142746.html

I had to go back to make sure I wouldn't contradict myself.  +1 then, +1
now for requiring braces. :)

-Barry

From p.f.moore at gmail.com  Wed May 31 11:23:50 2017
From: p.f.moore at gmail.com (Paul Moore)
Date: Wed, 31 May 2017 16:23:50 +0100
Subject: [Python-Dev] PEP 7 and braces { .... } on if
In-Reply-To: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
References: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
Message-ID: <CACac1F_1UB2tDK=fNQQvnZE_F-MiTfQ7FJnt+13EMjaZODxd6A@mail.gmail.com>

On 31 May 2017 at 15:11, Victor Stinner <victor.stinner at gmail.com> wrote:
> So I would suggest to modify the PEP 7 to *always* require braces for if.
>
> I would also suggest to require braces on "for(...) { ... }" and
> "while(...) { ... }". But only if the code has to be modified, not
> only to update the coding style.
>
> Changes which are pure coding style changes should be rejected to
> avoid conflicts with other pending pull requests and "code churn".

As a practical compromise, I'd argue that if you're changing
*existing* code, you should retain the current style. For *new* code,
always add braces. So, changing

if (func == NULL)
    return NULL;
/* further code */

to

if (func1 == NULL)
    return NULL;
/* further code */

you would leave out the braces. But changing

/* Code using func */

to

if (func == NULL) {
    return NULL;
}
/* Code using func */

you include the braces, because it's new code.

I'm not against making the PEP mandate braces, but I don't think it's
necessary to resolve the situation you describe as having triggered
the suggestion :-)

Paul

From orenmn at gmail.com  Wed May 31 11:28:10 2017
From: orenmn at gmail.com (Oren Milman)
Date: Wed, 31 May 2017 15:28:10 +0000
Subject: [Python-Dev] PEP 7 and braces { .... } on if
In-Reply-To: <CACac1F_1UB2tDK=fNQQvnZE_F-MiTfQ7FJnt+13EMjaZODxd6A@mail.gmail.com>
References: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
 <CACac1F_1UB2tDK=fNQQvnZE_F-MiTfQ7FJnt+13EMjaZODxd6A@mail.gmail.com>
Message-ID: <CAAD-+5qmFm_nX-bhi+1nRu+HZHjKyd4eSaWKiJ3tDTjWV2g_eA@mail.gmail.com>

the 'goto fail' bug is a somewhat extreme reminder for why such braces are
a good idea (as Victor said) -
https://www.imperialviolet.org/2014/02/22/applebug.html

On Wed, May 31, 2017 at 6:25 PM Paul Moore <p.f.moore at gmail.com> wrote:

> On 31 May 2017 at 15:11, Victor Stinner <victor.stinner at gmail.com> wrote:
> > So I would suggest to modify the PEP 7 to *always* require braces for if.
> >
> > I would also suggest to require braces on "for(...) { ... }" and
> > "while(...) { ... }". But only if the code has to be modified, not
> > only to update the coding style.
> >
> > Changes which are pure coding style changes should be rejected to
> > avoid conflicts with other pending pull requests and "code churn".
>
> As a practical compromise, I'd argue that if you're changing
> *existing* code, you should retain the current style. For *new* code,
> always add braces. So, changing
>
> if (func == NULL)
>     return NULL;
> /* further code */
>
> to
>
> if (func1 == NULL)
>     return NULL;
> /* further code */
>
> you would leave out the braces. But changing
>
> /* Code using func */
>
> to
>
> if (func == NULL) {
>     return NULL;
> }
> /* Code using func */
>
> you include the braces, because it's new code.
>
> I'm not against making the PEP mandate braces, but I don't think it's
> necessary to resolve the situation you describe as having triggered
> the suggestion :-)
>
> Paul
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/orenmn%40gmail.com
>
-- 

-Oren
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/e494f368/attachment.html>

From jim.baker at python.org  Wed May 31 11:45:39 2017
From: jim.baker at python.org (Jim Baker)
Date: Wed, 31 May 2017 09:45:39 -0600
Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to
 Python 2.7
In-Reply-To: <ECFACE06-24DF-49DD-B5D3-C82A61A9951F@lukasa.co.uk>
References: <CAMpsgwY7sZYE1kB3F3AfK9HCeob8ufi_cSwGOYK0QXtO4qwBhQ@mail.gmail.com>
 <ECFACE06-24DF-49DD-B5D3-C82A61A9951F@lukasa.co.uk>
Message-ID: <CAOhO=aOw=rWtXjeQ7TTL+rwmcjL6Z+Dz8SCZ+VL7G6gdQTZygA@mail.gmail.com>

Jython 2.7.1 is about to be released, with full support of upstream pip
(9.0.1), and corresponding vendored libraries, including requests.

However, this proposed new feature for CPython 2.7, and its usage, will
likely break pip on Jython 2.7.x going forward, given that future versions
of pip will depend on requests requiring MemoryBIO. Or am I wrong in this
analysis? This means we have to get back on the 2.7 development treadmill
just as we were about to focus on finally working on Jython 3 (
https://github.com/jython/jython3 previews this work).

Given that this proposed new feature is for 2.7 to support event loop usage
and not a security fix, I'm -1 on this change. In particular, it runs
counter to the justification policy stated in PEP 466.

- Jim


On Wed, May 31, 2017 at 7:25 AM, Cory Benfield <cory at lukasa.co.uk> wrote:

>
> > On 31 May 2017, at 08:42, Victor Stinner <victor.stinner at gmail.com>
> wrote:
> >
> > Hi,
> >
> > I wrote a PEP based on the previous thread "Backport ssl.MemoryBIO on
> > Python 2.7?". Thanks for Cory Benfield, Alex Gaynor and Nick Coghlan
> > who helped me to write it!
>
> It probably goes without saying, given that I helped with the drafting for
> the PEP, but I?m strongly in favour of this PEP. Just in case it helps to
> get that reaffirmation here. ;)
>
> Cory
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/
> jbaker%40zyasoft.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/a311af93/attachment.html>

From storchaka at gmail.com  Wed May 31 12:02:57 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 31 May 2017 19:02:57 +0300
Subject: [Python-Dev] PEP 7 and braces { .... } on if
In-Reply-To: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
References: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
Message-ID: <ogmpfc$id5$1@blaine.gmane.org>

31.05.17 17:11, Victor Stinner ????:
> I have a question on the CPython coding code for C code, the PEP 7:
> https://www.python.org/dev/peps/pep-0007/
> 
> """
> Code structure: (...); braces are strongly preferred but may be
> omitted where C permits, and they should be formatted as shown:
> 
> if (mro != NULL) {
>      ...
> }
> else {
>      ...
> }
> """
> 
> This section was updated after the creation of CPython in 1991, so as
> you may expect, a lot of existing C code doesn't respect this coding
> style. I'm trying to slowly add "missing" braces and indent as the
> example *when I have to modify existing code*.
> 
> The problem is the "strongly preferred" and "omitted where C permits"
> part. I would like to make the PEP 7 more explicit on that part since
> in each review, I disagree with Serhiy who wants to omit them.
> Example:
> 
> if (func == NULL)
>      return NULL;
> 
> versus
> 
> if (func == NULL) {
>      return NULL;
> }
> 
> I now prefer the version with braces. It's more verbose, but it
> prevents bugs when the code is modified later for whatever reasons.
> IMHO it also makes the code easily to read and to understand.
> 
> So I would suggest to modify the PEP 7 to *always* require braces for if.
> 
> I would also suggest to require braces on "for(...) { ... }" and
> "while(...) { ... }". But only if the code has to be modified, not
> only to update the coding style.

Thank you for opening this discussion Victor. I was going to open it myself.

I strongly prefer using braces around "for" and "while" bodies. I 
dislike unconditional adding braces, but after changing PEP 7 I became 
always adding them in my code and asking this in reviewed code. I even 
made Argument Clinic generating braces around gotos. But when I asked 
Barry to add braces around simple statements to satisfy PEP 7 
requirements, he pointed out that braces are only "strongly preferred". 
Since that time I allowed myself to left braces out in cases when they 
look cluttering.

I think that PEP 7 should be more explicit about cases in what braces 
are required and in what they are optional. I prefer to make them 
optional (or even make the variant without braces preferable) in case of 
"if" (without "else") or "case" body containing only the single "goto", 
"break", "continue" or simple "return". Especially if it follows by an 
empty line or closing brace. Adding braces in these cases decreases 
readability IMHO. This is one peculiarity for which many people like (or 
hate) Python.


From victor.stinner at gmail.com  Wed May 31 12:40:44 2017
From: victor.stinner at gmail.com (Victor Stinner)
Date: Wed, 31 May 2017 18:40:44 +0200
Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to
 Python 2.7
In-Reply-To: <CAOhO=aOw=rWtXjeQ7TTL+rwmcjL6Z+Dz8SCZ+VL7G6gdQTZygA@mail.gmail.com>
References: <CAMpsgwY7sZYE1kB3F3AfK9HCeob8ufi_cSwGOYK0QXtO4qwBhQ@mail.gmail.com>
 <ECFACE06-24DF-49DD-B5D3-C82A61A9951F@lukasa.co.uk>
 <CAOhO=aOw=rWtXjeQ7TTL+rwmcjL6Z+Dz8SCZ+VL7G6gdQTZygA@mail.gmail.com>
Message-ID: <CAMpsgwZ_XgwCL1JKVQ70nT-Doq9MdQ0tbS5Lds8z13H6-f0gLA@mail.gmail.com>

2017-05-31 17:45 GMT+02:00 Jim Baker <jim.baker at python.org>:
> Given that this proposed new feature is for 2.7 to support event loop usage
> and not a security fix, I'm -1 on this change. In particular, it runs
> counter to the justification policy stated in PEP 466.

Hum, it seems like the PEP 546 abstract is incomplete. The final goal
of the PEP is to make Python 3 more secure thanks to all goodness of
the PEP 543. The PEP 546 tries to explain why Python 2.7 is blocking
the adoption of the PEP 543 in practice.

Victor

From guido at python.org  Wed May 31 13:27:46 2017
From: guido at python.org (Guido van Rossum)
Date: Wed, 31 May 2017 10:27:46 -0700
Subject: [Python-Dev] PEP 7 and braces { .... } on if
In-Reply-To: <ogmpfc$id5$1@blaine.gmane.org>
References: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
 <ogmpfc$id5$1@blaine.gmane.org>
Message-ID: <CAP7+vJJvQNv1ktYr4_UGETWHs5bOjPw2bwBdVrnXdkEsSw8f9w@mail.gmail.com>

I interpret the PEP as saying that you should use braces everywhere but not
to add them in code that you're not modifying otherwise. (I.e. don't go on
a brace-adding rampage.) If author and reviewer of a PR disagree I would go
with "add braces" since that's clearly the PEP's preference. This is C
code. We should play it safe.

On Wed, May 31, 2017 at 9:02 AM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> 31.05.17 17:11, Victor Stinner ????:
>
> I have a question on the CPython coding code for C code, the PEP 7:
>> https://www.python.org/dev/peps/pep-0007/
>>
>> """
>> Code structure: (...); braces are strongly preferred but may be
>> omitted where C permits, and they should be formatted as shown:
>>
>> if (mro != NULL) {
>>      ...
>> }
>> else {
>>      ...
>> }
>> """
>>
>> This section was updated after the creation of CPython in 1991, so as
>> you may expect, a lot of existing C code doesn't respect this coding
>> style. I'm trying to slowly add "missing" braces and indent as the
>> example *when I have to modify existing code*.
>>
>> The problem is the "strongly preferred" and "omitted where C permits"
>> part. I would like to make the PEP 7 more explicit on that part since
>> in each review, I disagree with Serhiy who wants to omit them.
>> Example:
>>
>> if (func == NULL)
>>      return NULL;
>>
>> versus
>>
>> if (func == NULL) {
>>      return NULL;
>> }
>>
>> I now prefer the version with braces. It's more verbose, but it
>> prevents bugs when the code is modified later for whatever reasons.
>> IMHO it also makes the code easily to read and to understand.
>>
>> So I would suggest to modify the PEP 7 to *always* require braces for if.
>>
>> I would also suggest to require braces on "for(...) { ... }" and
>> "while(...) { ... }". But only if the code has to be modified, not
>> only to update the coding style.
>>
>
> Thank you for opening this discussion Victor. I was going to open it
> myself.
>
> I strongly prefer using braces around "for" and "while" bodies. I dislike
> unconditional adding braces, but after changing PEP 7 I became always
> adding them in my code and asking this in reviewed code. I even made
> Argument Clinic generating braces around gotos. But when I asked Barry to
> add braces around simple statements to satisfy PEP 7 requirements, he
> pointed out that braces are only "strongly preferred". Since that time I
> allowed myself to left braces out in cases when they look cluttering.
>
> I think that PEP 7 should be more explicit about cases in what braces are
> required and in what they are optional. I prefer to make them optional (or
> even make the variant without braces preferable) in case of "if" (without
> "else") or "case" body containing only the single "goto", "break",
> "continue" or simple "return". Especially if it follows by an empty line or
> closing brace. Adding braces in these cases decreases readability IMHO.
> This is one peculiarity for which many people like (or hate) Python.
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%
> 40python.org
>



-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/c72edd8e/attachment.html>

From brett at python.org  Wed May 31 13:58:10 2017
From: brett at python.org (Brett Cannon)
Date: Wed, 31 May 2017 17:58:10 +0000
Subject: [Python-Dev] Put token information in one place
In-Reply-To: <ogm7o1$qf5$1@blaine.gmane.org>
References: <ogm7o1$qf5$1@blaine.gmane.org>
Message-ID: <CAP1=2W4UJkcMWefnZxDehqKq6=2-fjT+Vz+QECLM2S7vcd4Ptg@mail.gmail.com>

On Wed, 31 May 2017 at 04:01 Serhiy Storchaka <storchaka at gmail.com> wrote:

> Currently when you add a new token you need to change a couple of files:
>
> * Include/token.h
> * _PyParser_TokenNames in Parser/tokenizer.c
> * PyToken_OneChar(), PyToken_TwoChars() or PyToken_ThreeChars() in
> Parser/tokenizer.c
> * Lib/token.py (generated from Include/token.h)
> * EXACT_TOKEN_TYPES in Lib/tokenize.py
> * Operator, Bracket or Special in Lib/tokenize.py
> * Doc/library/token.rst
>
> It is possible to generate all this information from a single source.
> Proposed in [1] patch uses Lib/token.py as an initial source. But maybe
> Lib/token.py also should be generated from some file in general format?
>

I don't think it matters really. Whatever is most convenient.


> Some information can be derived from Grammar/Grammar, but not all.
> Needed also a mapping between token strings ('(' or '>=') and names
> (LPAR, GREATEREQUAL). Can this be added in Grammar/Grammar or a new file?
>

Maybe Grammar/Tokens?


>
> There is a related problem, the tokenize module uses three additional
> tokens not used by the C tokenizer. It modifies the content of the token
> module after importing it, that is not good. [2] One of solutions is
> making a copy of tok_names in tokenize before modifying it, but this
> doesn't work, because third-party code search tokenize constants in
> token.tok_names. Other solution is adding tokenize specific constants to
> the token module. Is this good to expose in the token module tokens not
> used in the C tokenizer?
>

No opinion from me.


>
> Non-terminal symbols are generated automatically, Lib/symbol.py from
> Include/graminit.h, and Include/graminit.h and Python/graminit.c from
> Grammar/Grammar by Parser/pgen. Is it worth to generate Lib/symbol.py by
> pgen too? Can pgen be implemented in Python?
>

I assume there's a build rule for Python/graminit.c and porting pgen to
Python would require Python be installed to do a build from scratch. Have
we made that a requirement yet? If so then rewriting pgen in Python would
make it easier to maintain (although when was the last time anyone touched
that file?).

-Brett


>
> See also similar issue for opcodes. [3]
>
> [1] https://bugs.python.org/issue30455
> [2] https://bugs.python.org/issue25324
> [3] https://bugs.python.org/issue17861
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/brett%40python.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/37186db8/attachment.html>

From storchaka at gmail.com  Wed May 31 14:37:59 2017
From: storchaka at gmail.com (Serhiy Storchaka)
Date: Wed, 31 May 2017 21:37:59 +0300
Subject: [Python-Dev] Put token information in one place
In-Reply-To: <CAP1=2W4UJkcMWefnZxDehqKq6=2-fjT+Vz+QECLM2S7vcd4Ptg@mail.gmail.com>
References: <ogm7o1$qf5$1@blaine.gmane.org>
 <CAP1=2W4UJkcMWefnZxDehqKq6=2-fjT+Vz+QECLM2S7vcd4Ptg@mail.gmail.com>
Message-ID: <ogn2i2$it2$1@blaine.gmane.org>

31.05.17 20:58, Brett Cannon ????:
> I assume there's a build rule for Python/graminit.c and porting pgen to 
> Python would require Python be installed to do a build from scratch. 
> Have we made that a requirement yet? If so then rewriting pgen in Python 
> would make it easier to maintain (although when was the last time anyone 
> touched that file?).

That is why generated C files are added to the repository. There are 
precedences: Include/Python-ast.h, Python/Python-ast.c, 
Include/opcode.h, Python/opcode_targets.h, Objects/typeslots.inc, 
Objects/unicodetype_db.h, Modules/cjkcodecs/mappings_*.h, 
Modules/sre_constants.h, Modules/_ssl_data.h, and a number of 
**/clinic/*.h files are generated by Python scripts.


From jim.baker at python.org  Wed May 31 16:09:20 2017
From: jim.baker at python.org (Jim Baker)
Date: Wed, 31 May 2017 14:09:20 -0600
Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to
 Python 2.7
In-Reply-To: <CAMpsgwZ_XgwCL1JKVQ70nT-Doq9MdQ0tbS5Lds8z13H6-f0gLA@mail.gmail.com>
References: <CAMpsgwY7sZYE1kB3F3AfK9HCeob8ufi_cSwGOYK0QXtO4qwBhQ@mail.gmail.com>
 <ECFACE06-24DF-49DD-B5D3-C82A61A9951F@lukasa.co.uk>
 <CAOhO=aOw=rWtXjeQ7TTL+rwmcjL6Z+Dz8SCZ+VL7G6gdQTZygA@mail.gmail.com>
 <CAMpsgwZ_XgwCL1JKVQ70nT-Doq9MdQ0tbS5Lds8z13H6-f0gLA@mail.gmail.com>
Message-ID: <CAOhO=aPZXtQhKq-QaS2UPb=v7WkJW_N2BtAyyU-ErMVqhy4iCw@mail.gmail.com>

So we have two distinct changes that are proposed here:

1. Support alternative implementations of TLS instead of OpenSSL. In
particular this will enable the use of system trust stores for certificates.

2. Implement ABCs and concrete classes to support MemoryBIO, etc., from 3.7.

Supporting system trust stores is a valid security fix for 2.7, and I have
no such problem with such changes as long as they are narrowed to this
specific change.

But I object to a completely new feature being added to 2.7 to support the
implementation of event loop SSL usage. This feature cannot be construed as
a security fix, and therefore does not qualify as a feature that can be
added to CPython 2.7 at this point in its lifecycle.

The discussion that implementing such new features for 2.7 will improve
their adoption for Python 3 is a red herring. We could enumerate many such
features, but https://www.python.org/dev/peps/pep-0404/#upgrade-path is
rather clear here.

- Jim

On Wed, May 31, 2017 at 10:40 AM, Victor Stinner <victor.stinner at gmail.com>
wrote:

> 2017-05-31 17:45 GMT+02:00 Jim Baker <jim.baker at python.org>:
> > Given that this proposed new feature is for 2.7 to support event loop
> usage
> > and not a security fix, I'm -1 on this change. In particular, it runs
> > counter to the justification policy stated in PEP 466.
>
> Hum, it seems like the PEP 546 abstract is incomplete. The final goal
> of the PEP is to make Python 3 more secure thanks to all goodness of
> the PEP 543. The PEP 546 tries to explain why Python 2.7 is blocking
> the adoption of the PEP 543 in practice.
>
> Victor
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/d70ad0cc/attachment.html>

From barry at python.org  Wed May 31 17:08:34 2017
From: barry at python.org (Barry Warsaw)
Date: Wed, 31 May 2017 17:08:34 -0400
Subject: [Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to
 Python 2.7
In-Reply-To: <CAOhO=aPZXtQhKq-QaS2UPb=v7WkJW_N2BtAyyU-ErMVqhy4iCw@mail.gmail.com>
References: <CAMpsgwY7sZYE1kB3F3AfK9HCeob8ufi_cSwGOYK0QXtO4qwBhQ@mail.gmail.com>
 <ECFACE06-24DF-49DD-B5D3-C82A61A9951F@lukasa.co.uk>
 <CAOhO=aOw=rWtXjeQ7TTL+rwmcjL6Z+Dz8SCZ+VL7G6gdQTZygA@mail.gmail.com>
 <CAMpsgwZ_XgwCL1JKVQ70nT-Doq9MdQ0tbS5Lds8z13H6-f0gLA@mail.gmail.com>
 <CAOhO=aPZXtQhKq-QaS2UPb=v7WkJW_N2BtAyyU-ErMVqhy4iCw@mail.gmail.com>
Message-ID: <20170531170834.51f5124f@subdivisions.wooz.org>

On May 31, 2017, at 02:09 PM, Jim Baker wrote:

>But I object to a completely new feature being added to 2.7 to support the
>implementation of event loop SSL usage. This feature cannot be construed as
>a security fix, and therefore does not qualify as a feature that can be
>added to CPython 2.7 at this point in its lifecycle.

The other problem with this is that there isn't just one CPython 2.7, there
are many.  It's likely that most people get their Python from whatever
distribution/package manager they use.  Looking at active Ubuntu/Debian
releases you can see Python 2.7's from 2.7.3 to 2.7.13.  Few if any of those
will get the new feature backported, so that makes it difficult to rely on
them being present.

I agree with Jim that it makes sense to backport security fixes.  Usually
those are more well contained, and thus easier to cherry pick into stable
releases.  New features are much tougher to justify the potential for
regressions.

Cheers,
-Barry

From guido at python.org  Wed May 31 18:10:23 2017
From: guido at python.org (Guido van Rossum)
Date: Wed, 31 May 2017 15:10:23 -0700
Subject: [Python-Dev] PEP 544: Protocols - second round
In-Reply-To: <CAOMjWknzvtBpWGHTXmbEUz3AA6yaENONnRaAkq_HjKre8g5xWA@mail.gmail.com>
References: <CAOMjWkk8BaQ=pz7cNYU3W8giB6XKDKX5QcMzsi=GXtoBGK8xVQ@mail.gmail.com>
 <8b064822-14bb-6800-810a-c14dc97f565a@hotpy.org>
 <CAP7+vJJbBXuP1zTomj2_g4HFEfWDPfD3fMiUevuwUxOw2cRskw@mail.gmail.com>
 <CAOMjWk=i1irgjwzxT0Gqp1P2TKCET3TdnKPsyW--sUOF1aa0ug@mail.gmail.com>
 <CAKF=+diuFz2ED8Pqqpb5Sn=7+0cfGk8YZcY+i0KVTxZUYeHSPg@mail.gmail.com>
 <CAOMjWkkjmrkzws_d-TqQ+CN=CPSwJhyCejgwQMCO7mWd6fP4DA@mail.gmail.com>
 <CAP7+vJLWpeDpZshcMvtK+axz2441TKdHyP+T_MWyikJ=PNuHAQ@mail.gmail.com>
 <CAOMjWk==E-vpphCOADGqFk_UjyQuoaHg1+8m2T++Jh6y9ZzETA@mail.gmail.com>
 <CAKF=+djkHjeYkEga=TqHA=WnGU2JJyXGtPKcXYG2=cZ=kzmQLQ@mail.gmail.com>
 <CAP7+vJJY5p8nT+kqj5Mix6Qu_MD6wQNMitPMjL_F=Eahmm+zJQ@mail.gmail.com>
 <CAOMjWknzvtBpWGHTXmbEUz3AA6yaENONnRaAkq_HjKre8g5xWA@mail.gmail.com>
Message-ID: <CAP7+vJ+fi5GquWeXeE7_=g=7iipgarVL34tN2ou74quijsniXA@mail.gmail.com>

On Wed, May 31, 2017 at 2:16 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> On 31 May 2017 at 00:58, Guido van Rossum <guido at python.org> wrote:
> [...]
>
> Thank you for very detailed answers! I have practically nothing to add.
> It seems to me that most of the Kevin's questions stem from unnecessary
> focus
> on runtime type checking. Here are two ideas about how to fix this:
>
> * Add the word "static" somewhere in the PEP title.
>

So the title could become "Protocols: Static structural subtyping (duck
typing)" -- long, but not record-setting.

* Add a short note at the start mentioning this is an extension of the type
> system proposed in PEP 484 and recommending to read PEP 484 first.
>

Hm, the Abstract already spells that out. I suspect that many people react
to the discussion without first reading the PEP itself (I do this myself
:-). The only thing that could possibly be confusing about the abstract is
that it claims to specify "static and runtime semantics" -- but that's
reasonable, since the runtime semantics must somehow be specified even if
they're minimal.

-- 
--Guido van Rossum (python.org/~guido <http://python.org/%7Eguido>)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/e6231be5/attachment-0001.html>

From guido at python.org  Wed May 31 18:17:42 2017
From: guido at python.org (Guido van Rossum)
Date: Wed, 31 May 2017 15:17:42 -0700
Subject: [Python-Dev] PEP 484 update proposal: annotating decorated
 declarations
In-Reply-To: <CAOMjWkngwYOWBFJnfFsjzOc2E1MqAR5KF23B6RBvsNudsOruYg@mail.gmail.com>
References: <CAP7+vJ+yu3Y=DELGcu9rv=50EuxbBAFrtFHnqRL2kZX1LzFqng@mail.gmail.com>
 <CAMiohoiXuAZmzyJYBx6jVVfWj5=duLuRAMuFT6JfEC=vM+x5gw@mail.gmail.com>
 <CAOMjWkmmfGSEu1mjVzPYE_BT5g06RSJZB02hz0nouAcJT62J_g@mail.gmail.com>
 <CAP7+vJ+dkQRYaza_Zpp+8vm_wR6tC5PXm9mYqRhFw5wY2UW=og@mail.gmail.com>
 <CAOMjWkngwYOWBFJnfFsjzOc2E1MqAR5KF23B6RBvsNudsOruYg@mail.gmail.com>
Message-ID: <CAP7+vJLstk+Zt3+FkPhGKeOqeSZU7n4N4bR_Sg2Xx=ftXeqKyw@mail.gmail.com>

On Wed, May 31, 2017 at 6:16 AM, Ivan Levkivskyi <levkivskyi at gmail.com>
wrote:

> On 30 May 2017 at 23:02, Guido van Rossum <guido at python.org> wrote:
>
>> All in all I'm still leaning towards Naomi's original proposal -- it
>> looks simpler to implement as well.
>>
>
> OK, I think having a bit of verbosity is absolutely fine if we win
> simplicity of implementation (for both static and runtime purposes).
>

Then I propose to do it this way. We can always add Jukka's way as an
alternative notation later. I'd like to hear from Jukka before I merge the
PR for PEP-484.

In the meantime, Naomi, are you interested in trying to implement this?

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/c5104b35/attachment.html>

From greg.ewing at canterbury.ac.nz  Wed May 31 18:59:17 2017
From: greg.ewing at canterbury.ac.nz (Greg Ewing)
Date: Thu, 01 Jun 2017 10:59:17 +1200
Subject: [Python-Dev] PEP 7 and braces { .... } on if
In-Reply-To: <20170531110457.5b4ab7ac@subdivisions.wooz.org>
References: <CAMpsgwYY6f+=rmt7iPseB5nQ2S5F5QK+1RE5C8OHkvsruMLh=Q@mail.gmail.com>
 <CAMpsgwb=maZf_jxqd+N18xGUcYSwtvfKRAJg=utiRrojEdkN9w@mail.gmail.com>
 <20170531110457.5b4ab7ac@subdivisions.wooz.org>
Message-ID: <592F4AC5.5040802@canterbury.ac.nz>

Seems like a good idea to tighten it up.

If a style guide is going to say "you can either do X or
not do X", it might as well not mention X at all. :-)

-- 
Greg

From guido at python.org  Wed May 31 19:31:18 2017
From: guido at python.org (Guido van Rossum)
Date: Wed, 31 May 2017 16:31:18 -0700
Subject: [Python-Dev] Put token information in one place
In-Reply-To: <ogm7o1$qf5$1@blaine.gmane.org>
References: <ogm7o1$qf5$1@blaine.gmane.org>
Message-ID: <CAP7+vJJ7dkF9QVz_GYUCZhFgubqhPO9290MV+pD-DvjC36+vOw@mail.gmail.com>

On Wed, May 31, 2017 at 4:00 AM, Serhiy Storchaka <storchaka at gmail.com>
wrote:

> [...] Can pgen be implemented in Python?
>

Yes, an implementation already exists in the stdlib under lib2to3/pgen2/...

-- 
--Guido van Rossum (python.org/~guido)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170531/bff3f1d3/attachment.html>